Anthony Bloesch's Web Log : Repository

SQL Server Modeling Services announcement

AnthonyBloesch — Thu, 19 Nov 2009 17:28:00 GMT

The code name “Oslo” repository now has the official name “SQL Server Modeling Services”. SQL Server Modeling Services will be a SQL Server workload like SQL Server Reporting Services. Details will be announced at the PDC where the most relevant session is http://microsoftpdc.com/Sessions/SVR19. You can also read about it online at http://msdn.microsoft.com/en-us/library/dd129586(VS.85).aspx.

In Microsoft, like most organizations, it is hard to get things done when you are in different organizational units and on different schedules. This move makes the repository team firmly part of the SQL Server team and on the same schedule. Thus this change will make alignment and deep integration of the repository with rest of the SQL Server platform easier. Also it provides a simple setup experience and much simpler licensing model for customers. Already, there are many planned changes for the next version of SQL Server that will make for a much stronger metadata store and overall platform.

"Oslo" repository lifecycle/versioning whitepaper

AnthonyBloesch — Tue, 07 Apr 2009 19:44:00 GMT

Repositories face issues to do with versioning schema and data as well as integrating with an organization's lifecycle processes. Here is a whitepaper I wrote on how the "Oslo" team thinks about handling various lifecycle/versioning issues http://msdn.microsoft.com/en-us/library/dd638048.aspx. The whitepaper deals with the following issues:

Application lifecycle management (ALM): Facilitating team development of software in a continuous cycle of steps (that is, definition, design, development, testing, deployment, and management). Traditionally, Source Code Control Systems (SCCS) and build systems, like Visual Studio Team Foundation Server, automate ALM.

Schema evolution: Changing schemas over time in such a way that applications continue to run unchanged or with minimal changes.

"Oslo" product versioning: Upgrading a customer's "Oslo" repository, from version n to version n+1, in such a way that the data is preserved.

Application versioning: The ability to upgrade applications, stored in the "Oslo" repository, to new versions. The upgrade process has the option to retain the data from older versions of the application.

Operational versioning: The ability to understand operational/management changes to a deployed application.

How to make a copy of the repository

AnthonyBloesch — Tue, 07 Apr 2009 02:22:00 GMT

Here is a nice article on how to make a copy of the “Oslo” repository so you can later restore it to its previous state http://blogs.msdn.com/modelcitizen/archive/2009/04/06/how-to-clean-up-the-repository.aspx. This is great for experimenting with the repository and for edit debug cycles with domains. The steps are essentially what we do internally at Microsoft.

SQL Server 2008 error handling best practice

AnthonyBloesch — Tue, 10 Mar 2009 22:45:00 GMT

Error handling in SQL Server 2008 needs careful implementation. The Microsoft “Oslo” Repository’s API has the further problem that we cannot mandate the error handling logic in our callers. Thus a stored procedure call could be in a transaction or not and in a try-catch block or not. Below is the pattern we have chosen based on experimentation and best practice guidance from the SQL Server team and other SQL Server experts. A test script for all the interesting cases can be found at http://blogs.msdn.com/anthonybloesch/attachment/9469577.ashx.

For a good overview of SQL Server error handling see http://www.sommarskog.se/error-handling-I.html and http://www.sommarskog.se/error-handling-II.html. In our design, we had the following main issues and mitigations:

· A transaction rollback will rollback to the outermost transaction but if there is an outer transaction we would like to only rollback the changes to the module’s inner transaction. The solution is to use SQL Server save points if there is an outer transaction.

· A sufficiently severe raiserror will terminate a module if there is an outer try-catch block but if there is none then execution will continue in the module. The solution is to always explicitly return after raising an error.

· Some developers like to use stored procedure return values to encode error states. The solution is to return appropriate error codes.

· SQL Server will raise warnings if the transaction depth entering and leaving a transaction do not match. The solution is to be careful.

· Triggers have an implicit transaction. The solution is to use a simplified pattern for triggers where a transaction is never started.

· Save points need unique names if modules can nest otherwise you can rollback to the wrong save point. The solution is to use a GUID to name the save points.

Here is the pattern for stored procedures (eliding our special error reporting routines):

if parameter error
begin
raiserror(N'…', 16, 0);

return -1;
end
else
begin
begin try
    declare @hasOuterTransaction bit = case when @@trancount > 0 then 1 else 0 end;
    declare @rollbackPoint nchar(32) = replace(convert(nchar(36), newid()), N'-', N'');

    if @hasOuterTransaction = 1
    begin
      save transaction @rollbackPoint;
    end
    else
    begin
      begin transaction @rollbackPoint;
    end;

    Do work;

    if @hasOuterTransaction = 0
    begin
      commit transaction;
    end;
end try
begin catch
    if xact_state() = 1
    begin
      rollback transaction @rollbackPoint;
    end;

    execute Standard module error handler;

    return -error_number();
end catch;
end;

Here is the pattern for triggers (eliding our special error reporting routines):

if parameter error
begin

rollback transaction;

raiserror(N'…', 16, 0);

return;
end
else
begin
begin try

    Do work;

end try
begin catch
    rollback transaction;

    execute Standard module error handler;

    return;
end catch;
end;

Architectural design tenets and corollaries

AnthonyBloesch — Wed, 17 Dec 2008 20:50:00 GMT

Now that the codename “Oslo” repository is public I can discuss its architecture and design.

When designing a new product or component I like to:

1. Understand the business landscape. For example, what is the market opportunity, what are the major competitors, where are the competitors likely to be by the time the product is released, how will the product fit into customer’s business processes. Creating new products is difficult and expensive so it is important to get this part right. Also an in-depth understanding of the business landscape can help enormously in day-to-day decisions. For example, can a feature wait until version 2?

2. Create design tenets. Design tenets are the bedrock on which good architecture is built. Any non-trivial design must make tradeoffs—design tenets help drive decisions by succinctly describing what is important. Without design tenets, scope creep and over-engineering can easily destroy a design.

I did an extensive multi-version business analysis for the “Oslo” repository which, of course, I cannot share with you but here are the “Oslo” repositories design tenets:

Integrate well with Oslo. Oslo should provide a seamless platform. The “Oslo” repository should integrate seamlessly with the rest of the Oslo platform.

Support a wide variety of tools. No toolset can fully anticipate customer needs. Support for ad hoc queries and updates, using familiar tools, will allow customers to circumvent the limitations in our tools. For example, it should be straightforward to build an Access based application over the “Oslo” repository or write a query that finds code vulnerable to a new security exploit.

The need to support traditional tools implies that the “Oslo” repository uses a straightforward relational representation of data as its primary API. Further to make queries easy to write, the metadata stored in the “Oslo” repository should form an integrated and consistent whole.

Scale well. Deployment and code analysis data (e.g. call graphs) can be very large. The “Oslo” repository needs to scale well to provide consistently good performance even when it contains a large enterprise’s metadata. In particular, good geo-scaling is likely to be important in “Oslo” repository scenarios.

Enterprises vary in their scaling strategies, the “Oslo” repository needs to support both scale-up and scale-out strategies. In particular, the “Oslo” repository should be designed to align with the SQL Server’s scale-out strategy since it will offer customers the most cost effective scale-out architecture for the near future. In practice, this means that “Oslo” repository data should be easily partitionable into related segments.

Support high availability. To be credible, the Oslo server will need to be highly available. As a critical component of the Oslo Server, the “Oslo” repository will also need to support high availability.

As well as being highly available itself, the “Oslo” repository should also support building fault resilient components that can survive “Oslo” repository failures. In particular, the “Oslo” repository should be designed to align with the SQL Server scale-out strategy and OCS (Occasionally Connected Systems) changes since they will offer customers the most cost effective high availability architecture for the near future. In practice, this means that “Oslo” repository data should be easily partitionable into related segments.

Be easily extensible. To quickly instantiate the “Oslo” repository with domain models for a wide variety of metadata and allow partners and customers to extend Microsoft tools or create new ones, the “Oslo” repository needs to be easily and cheaply extensible, supporting domain model extension and composition.

Ideally, the “Oslo” repository should make it easy to design and build domain models with, preferably, a wide variety of standard development tools or, less desirably, special purpose tools.

Support schema evolution. As technologies evolve, so will our domain models. The “Oslo” repository must support the easy and robust migration of applications to new domain model versions.

The “Oslo” repository should use good relational database design to ensure easy schema evolution. Because object-oriented meta-models are fragile, fixed object-oriented domain APIs should be avoided.

Oslo repository

AnthonyBloesch — Mon, 05 Nov 2007 21:33:00 GMT

The central challenge of repository implementations is how to provide support for an extensible list of metadata schema. The most common solution is to use a universal schema (e.g. an entity table, a property-value table and a relationship table). This makes writing tools like repository browsers easier but at a significant cost. Repository performance and queriability suffers. Queries run slowly and standard database tools like SQL Server Reporting Services do not work well because they expect well structured data.

Instead of using a universal schema approach, the design of the Oslo repository has focused on making well structured relational APIs work. This makes the engineering effort harder but customers benefit from a performant store that can be easily queried and used with a wide variety of stores. For example, the query to retrieve name and accessibility of a particular CLR method using the universal schema approach would be something like:

select N.[Value] as [Name], A.[Value] as [Accessibility]
from [Properties] as N, [Properties] as A
where N.[Id] = @id and A.[Id] = @id and
N.[Property] = ‘82aba91a-ca89-4e4f-bd39-9b67ec02ab5a’ and -- Name Property
A.[Property] = ‘efad35c8-46bb-439e-8a5d-7bd0f695d27f’ -- Accessibility Property

by contrast the repository query is:

select M.[Name], M.[Accessibility]
from [CLR].[Methods] as M
where M.[Id] = @id

which is a natural SQL query.

Reliance on universal schema are just one of the issues past repository implementations have suffered from. Now that the repository is announced I am free to talk about its design. In subsequent articles I will go through these design tradeoffs and how we resolve these issues.

Last week, as part of the Oslo announcement, Microsoft announced that we are building a metadata repository that will ship in System Center, BizTalk and Visual Studio. If you are interested in reading more about Oslo and the repository then the Oslo site, press release and Directions on Microsoft articles are good starts.

We need great developers and testers to complete the repository team. If you are interested in helping change the way software is built and managed then please apply for one of the following positions:

Development Manager (position #179184)

Software Developer (position #179188)

Modeler (position #213280)

Program Manager (position #185006)