SQL Azure Team Blog : Astoria

Accessing the New Relational SDS with REST

davidrob — Fri, 20 Mar 2009 19:16:45 GMT

Hi Everyone! This is Lev Novik, one of the architects on the SQL Data Services Team.

We get a number of questions about whether or not we have abandoned or downgraded REST access to SDS, and if so, why. To summarize: not at all --- rather than abandoning REST, we intend to support it in precisely the ways and places where we believe most people want it. Allow me to explain.

Consider an average (web) application today. Most often, it consists of a:

client (often running in a browser),
mid-tier business logic (often running in a web server), and
back-end storage (often a relational database).

What are the protocols that are typically used? Well, the protocol between the client and the mid-tier is usually HTTP and often REST; the protocol between the mid-tier and the back-end storage is typically the native protocol of the back-end database.

Well, with SQL Data Services, the situation is quite similar. In these scenarios:

SDS takes the place of the back-end database,
mid-tier business logic is running maybe in Windows Azure, or maybe outside,
and the client, well, the client is unchanged

In this world, we believe that you will continue to want to use the most powerful mechanism to talk from your mid-tier to SDS, together with the nicest database tools; and you will continue to want to be able to easily expose your web application to its clients in the most simple, elegant, and web-friendly way. This is why we concentrate on:

Exposing TDS (SQL Server native protocol) as the full-featured way to talk to SDS
Making it very easy for you to expose REST access to your SDS-based application.

To achieve the latter, we rely on ADO.NET Data Services (aka “Astoria”). Its very purpose in life is to make building data-driven REST services as easy as possible. For those who haven’t played with ADO.NET DS, I highly recommend it --- grab a favorite beverage, watch a quick introductory movie, get hold of a database of your own, and see what it takes to make a REST web service out of it (you might just have some of the beverage still remaining at this point).

But of course, your application is unlikely to be just a one-to-one mapping of a database --- you will want to build your own business logic, your own security policies, your own views, etc. ADO.NET Data Services allows you to implement these, and to make your creation into a real non-toy service (though this will certainly involve a few more trips into the kitchen!)

But that was SQL Server on your own dev box --- how does that work with SDS and Windows Azure? Well, we intend to make it work in much the same way:

Get set-up to develop Windows Azure services.
Create an SDS Database in the cloud, and load it up with your schema.
Create a Windows Azure Web Cloud Service Project in Visual Studio.
Add ADO.NET Data Service project item, and point it at the SDS Database (or at a local database with the same schema).
Develop your ADO.NET Data Service (security, biz logic, templates, etc)
Change the connection string to point to the real SDS Database in the cloud (if you used a local database for development)
Publish your service to Windows Azure.

And at the end, you have an SDS-based REST Service, configured just the way you like it!

Hope this made some sense!

--Lev

SSDS at PDC 2008

Soumitra Sengupta — Thu, 18 Sep 2008 20:53:00 GMT

It is around 40 days to PDC and the excitement is building. We have a ton of work to do between now and then and the team is heads down getting ready for this major event. If you look at the PDC 2008 breakout sessions already announced, it is incredible that we will have 26 sessions on Cloud Services and 6 sessions related to SQL Server Data Services (SSDS). You can look it up here. You can see Mike and Jennifer's interview with Amitabh Srivastava here, where he talks about some of the things his team has been working on. Unfortunately he cannot talk much about it, but I can assure you coming to PDC this year will be well worth it. In my mind this is as big and maybe bigger than when we announced the .Net wave. The cloud wave is a major event and disruption in our industry and you will get to see and hear from the guys driving this wave inside Microsoft at PDC 2008.

That is as much as I can say about our overall Cloud services initiatives. Specifically, as a member of the SQL Server Data Services (SSDS) team, I am psyched to say that we have scheduled 3 talks directly related to SSDS, 1 talk from our Technical Fellow Dave Campbell on how we are evolving and extending our comprehensive Data Platform and the fast growing SQL business to the cloud and another 3-4 talks on technologies like Astoria, Velocity, Sync Framework and Entity Frameworks that are important to our cloud strategy. So here are the key talks and a brief outline of what you can expect from these talks at PDC.

1. Microsoft SQL Server: Data-Driven Applications from Device to Cloud - Dave Campbell. Dave has been through it all. He will present the key trends that are shaping our industry and how we are evolving our Data Platform from devices, desktop, server to the cloud and the services that makes this platform worth betting on. If you ever wondered what software and services have to do with data, you do not want to miss this talk. Dave's talk will set the stage for all the talks to follow on data and tie them together. So this has to be a must attend talk for you if you care about data.

2. A Lap around SQL Server Data Services - Soumitra Sengupta. This will be an advanced technical presentation where I will explain how we got started on SSDS, how it relates to our Cloud and Data Platform strategy, how we see our Cloud services vis-a-vis our on-premises SQL Server offering. You will also learn why SSDS seems like a departure from hosted SQL Server. I get that one a lot. Tables, Columns, T-SQL anyone. I will fly through this material quickly and spend most of the time coding up an application using our REST and SOAP interfaces. I promise there will be one or two surprises during this presentation. I will also explain how we run our service and the trade-offs and questions that should be on developers' mind as they build applications using SSDS.

3. Under the Hood: Building SQL Server Data Services - Istvan Cseri and Gopal Kakivaya. Man what can I say. These guys are going to take the lid off SSDS and really dive under the hood and talk about how this service is built. If you want to see how to build a fault-tolerant, highly available multi-datacenter service, you cannot miss this talk. It will demystify what we have built and how we operate this. This is the first time we will go this deep. These guys are real rock stars and I will not be upset if you miss my talk BUT DO NOT MISS THIS ONE.

4. SQL Server Data Services: Futures - Patric McElroy. If you are curious after listening to the first 3 talks about where we are headed with SSDS, this is the talk for you. There will be lots of surprises as Patric will show you how we are thinking about extending our entire data platform and services to the cloud. This is a real forward looking session but there will be demos on the way to give you a clear idea of the vision and the path ahead.

In addition we will have lunch sessions, lounges, booth and Hand-on-Labs. You will get to meet, greet, ask questions and socialize with the team members who are making it all happen and if there is enough interest, we will try to get a group together and do something fun during PDC. So if you are interested let, David Robinson know. I know he is in touch with Mike Amundsen already about this.

Aside from these 4 talks, we have the following talks where we will establish the linkages between Sync, ADO.Net Data Services (Astoria), Velocity and other Microsoft services and SSDS. So it would be interesting for you to attend these as well.

1. Microsoft Sync Framework: Enterprise Data in the Cloud and on Devices - Liam Cavanagh
2. Microsoft Sync Framework Advances - Lev Novik
3. Developing Applications Using Data Services - Mike Flasko
4. Entity Framework Futures - Tim Mallalieu
5. Project "Velocity": A First Look - Murali Krishnaprasad
6. A Lap around Building Block Services - John Shewchuk
7. Architecture of the Building Block Services - Dennis Pilarinos, John Shewchuk

Obviously there will be the Keynote from Ray Ozzie to set the stage for this wave and you cannot miss that. I also hear Don Box and Chris Anderson are cooking up something using SSDS. These 2 guys have always lit up PDC and I do not think you will be disappointed this time around.

See you all there.

SQL Server Data Services Sprint 3 now Live

Soumitra Sengupta — Fri, 25 Jul 2008 21:18:00 GMT

We completed the rollout of Sprint 3 yesterday. The team is already half way into Sprint 4. In Sprint 4 we are focussed on things we need for PDC 2008, which by the way is in late October in Los Angeles. Ray Ozzie and Steve Ballmer foreshadowed some of what this PDC will be all about here and here. You can also go here to see the already announced sessions. SSDS will be a key part of this year's PDC. We set our technical and product agendas that developers care about at PDC. If you want to understand and learn what Services is all about in our Software plus Services strategy, do not miss this year's PDC.

Overall, this was an interesting deployment as we had new environments come online that had to be configured. The team worked really really hard to make the deployment as smooth as possible. Interestingly we did not hit any issues during deployment yesterday. Wish we could video tape how we do this and share with our readers. Something I have to take up with management. I think it will give everyone an appreciation of the amount of planning, tooling, automation and discipline that goes into these deployments.

Anyway as developers, you care about what new things we are enabling. Things have been leaked on this before by Roger, David Robinson and Jeff Currier. But here are the new features:

Blob Support via the REST interface

In this release, Blobs are limited to 100 megabytes in size.
Blob Support via the SOAP interface will be delivered in a later update.

Version metadata property has changed from a large integer to a container-wide dbtimestamp value:

Version numbers are now based on a dbtimestamp and will increase monotonically on a container-wide basis. No two entities in the same container will ever receive the same version value. The dbtimestamp increases monotonically, however not necessarily sequentially. Clients should no longer assume entity version numbers are sequentially assigned. It does not start at 0, but may be any non-zero integer value for a newly created entity.

ETag support via REST, with similar functionality provided on the SOAP interface

Support for If-Match and If-None-Match via the REST interface. This will allow you to verify that you have the latest version of an Entity, enabling you to use resources more efficiently.
Support for “Accept” header via the REST interface. This will allow you to denote what content you want returned from the service. For instance when retrieving a Blob, the “Accept” header will allow you to choose between the Blob content, or the metadata properties associated with it.
Support for ETag semantics on the SOAP service via the new Version Match object on the Scope object.

HEAD support via REST interface

Support for the “HEAD” verb via the REST interface. This provides a lightweight way of validating existing entity version information without requiring entire entities (or blobs) to be retransmitted to the client.

A new content type – “application/x-ssds+xml” – has been added to the service

This content type is for storing XML content in the service.
Existing “application/xml” content type is being phased out in a future sprint.
The "application/x-ssds+xml" content type should be used for all entities except those containing blob content. Blob entities should instead use a content type value which best reflects the type of blob data stored. It's important to note that the value chosen here will be used later when attempting to retrieve the different parts of an entity by using the "Accept" header.

In addition we did some infrastructure work to enable metering (we will be charging for this service some day :-)) ), throttling (DoS attacks happen and the service needs to work in degraded situation), quota enforcement etc. that developers will not see much of at this time. But once we open our management, debugging and monitoring interfaces, you will be able to plug in and keep up with what is going on with your service.

The primer is updated at our Dev Center here to help you take advantage of the new features. There is one other new "thing" that we are enabling in this rollout that we will talk about a bit later. It is not huge, but it will demonstrate that we are thinking about our building block services very very broadly and deeply. Stay tuned.

Why Astoria alignment is not that trivial

Soumitra Sengupta — Tue, 15 Jul 2008 08:56:00 GMT

I have been meaning to write about this topic for quite some time. But it took me quite a while to understand deeply the key issues in aligning Astoria and SSDS. Thanks to Alan Bush, an architect in our team, I have a much better understanding of the key issues. The tendency for most of us is to focus on the data model and query language differences between Astoria and SSDS. Turns out, while they are important, they are not the most difficult ones to work in. So here is what I have learnt from Alan:

a. Astoria protocol and interaction model deals with only a single consistency domain. Each consistency domain sort of maps to a database. Astoria consistency domain is akin to SSDS container. SSDS data model actually spans multiple consistency domains. It is possible to write a query across containers or even query for container properties. Yes Astoria can be extended to address this and we are working on that with Pablo and the Astoria team.
b. Multi-tenancy is a fundamental concept in SSDS while Astoria has no such concept at this point in time. We have to work through this as well. This has huge implications in the security model for the service.
c. Entities in SSDS do not require schema while default for Astoria is EDM schemas. While on the surface this is an easy problem to solve yet it requires you to think through all the options. If you look at the data model for Google App Engine, you will see how they chose to solve this problem - typed entities and expando entities are 2 separate concepts. It is one way to deal with it but think about updates on a graph where you have both typed and flexible entities, how would the user deal with the semantic differences between the two concepts in the same transaction?

There are host of issues but these are some of the top ones we are grappling with. Rest assured minds lot sharper than mine are working this and if there is an elegant solution, they will find it. I have full faith that by PDC we will have a compelling story to tell around SSDS and Astoria. So stay tuned.

Roger Jennings talks about SSDS in a Visual Studio Magazine Article

Soumitra Sengupta — Wed, 02 Jul 2008 10:34:00 GMT

Roger has an interesting article on SSDS in the latest Visual Studio Magazine here. It is really interesting to see the performance data between the SOAP and the REST end points of SSDS. It would be really interesting to dig into this.

One bit of correction - the SSDS flexible entity model was developed independent of EDM / EF and the Astoria model. At this point in time, both teams are working together to get this aligned.

I thought Dave Campbell in one of his press interviews had mentioned it but I could not find the reference. Sorry Roger. So here is an aswer to your question - SSDS is built using SQL Server 2005, SP2 as the starting code base. That is the starting point and we made changes to it. Over time, some of these changes will make its way into the SQL Server mainline and SQL Server 2008 will make its way into SSDS. We wanted to proceed in parallel as fast as we could and this was the best way to do it.