SQL Azure Team Blog : ADO.Net Data Services

Accessing the New Relational SDS with REST

davidrob — Fri, 20 Mar 2009 19:16:45 GMT

Hi Everyone! This is Lev Novik, one of the architects on the SQL Data Services Team.

We get a number of questions about whether or not we have abandoned or downgraded REST access to SDS, and if so, why. To summarize: not at all --- rather than abandoning REST, we intend to support it in precisely the ways and places where we believe most people want it. Allow me to explain.

Consider an average (web) application today. Most often, it consists of a:

client (often running in a browser),
mid-tier business logic (often running in a web server), and
back-end storage (often a relational database).

What are the protocols that are typically used? Well, the protocol between the client and the mid-tier is usually HTTP and often REST; the protocol between the mid-tier and the back-end storage is typically the native protocol of the back-end database.

Well, with SQL Data Services, the situation is quite similar. In these scenarios:

SDS takes the place of the back-end database,
mid-tier business logic is running maybe in Windows Azure, or maybe outside,
and the client, well, the client is unchanged

In this world, we believe that you will continue to want to use the most powerful mechanism to talk from your mid-tier to SDS, together with the nicest database tools; and you will continue to want to be able to easily expose your web application to its clients in the most simple, elegant, and web-friendly way. This is why we concentrate on:

Exposing TDS (SQL Server native protocol) as the full-featured way to talk to SDS
Making it very easy for you to expose REST access to your SDS-based application.

To achieve the latter, we rely on ADO.NET Data Services (aka “Astoria”). Its very purpose in life is to make building data-driven REST services as easy as possible. For those who haven’t played with ADO.NET DS, I highly recommend it --- grab a favorite beverage, watch a quick introductory movie, get hold of a database of your own, and see what it takes to make a REST web service out of it (you might just have some of the beverage still remaining at this point).

But of course, your application is unlikely to be just a one-to-one mapping of a database --- you will want to build your own business logic, your own security policies, your own views, etc. ADO.NET Data Services allows you to implement these, and to make your creation into a real non-toy service (though this will certainly involve a few more trips into the kitchen!)

But that was SQL Server on your own dev box --- how does that work with SDS and Windows Azure? Well, we intend to make it work in much the same way:

Get set-up to develop Windows Azure services.
Create an SDS Database in the cloud, and load it up with your schema.
Create a Windows Azure Web Cloud Service Project in Visual Studio.
Add ADO.NET Data Service project item, and point it at the SDS Database (or at a local database with the same schema).
Develop your ADO.NET Data Service (security, biz logic, templates, etc)
Change the connection string to point to the real SDS Database in the cloud (if you used a local database for development)
Publish your service to Windows Azure.

And at the end, you have an SDS-based REST Service, configured just the way you like it!

Hope this made some sense!

--Lev

First round of Questions and Answers

davidrob — Thu, 12 Mar 2009 17:38:00 GMT

Tuesday was an exciting day for the team, and more importantly, for the industry. I think one of the best responses I got from someone was that they “stood up from their chair and did a dance of joy”.

Here is the first round of questions I have received and the answers you are looking for.

How will existing accounts be migrated over to the new version?

We haven’t worked out all the details yet, but our current plans are to send invites to all registered users when we launch the CTP by mid-year 2009.

Are we losing some BASE capabilities to grant ACID capabilities?

The whole theory behind BASE (basically available, soft state, eventually consistent) is to gain scalability at the cost of consistency. We have always supported full ACID capabilities in the service and will continue to do so.

Are we losing scalability or partition support to guarantee consistency?

The short answer is no. The foundation of SDS has not changed. All of the guarantees around scalability and consistency still apply. In the ACE model, each authority had one or more containers with each container being the unit of consistency for query and update. The new SDS model hasn’t changed this if you think of each authority now being a server and each container a database. You now get the richness of the T-SQL model over your databases!

When? or to quote JamieT “When do I get to party on this new service”?

We’re on track to deliver a public CTP mid-calendar year 2009 and ship in the second half of calendar year 2009.

If Microsoft can deliver on the features for SQL Data Services (SDS) announced today then they have set the bar for SQL and Cloud database vendors

Yes, we believe we have.

Will SDS support Database Encryption (certificate and key management). Any support for row level versioning?

Database encryption? Not initially, but it’s on the list and as we have demonstrated – if there is sufficient customer demand, it will be one of the first things we add after v1. As far as row level versioning, this can mean a few different things. Do we support statement level snapshot transactions? Yes. Do we support Change Data Capture? It is still being evaluated.

The blog entry states “If it works with SQL Server, it will largely work with SQL Data Services.”. That word “largely” bothers me a little – it suggests the functionality is going to be reduced slightly. Details please?

We will be providing documentation soon on what is and is not supported in SDS. I’ll post an entry to the blog once the guidance is available and you can also keep an eye out for it on our MSDN Dev Center. But, to answer the question – We say *largely* due to the fact that there are things that just don’t apply in a cloud based world like setting the location of a data or log file or making server wide configuration changes. In v1 we expect to deliver a surface area that will support the vast majority of SQL Server database applications.

What artificial limits will be in place?

The database size will be capped. We are still evaluating what the cap will be, but the plan is to ensure that the allowed database size supports most, if not all, departmental and web application workloads.

Do we have to pay per instance, per MB of storage or per bandwidth usage?

The pricing options and levels are still being finalized.

Will it be SQL Server 2005 or 2008? Enterprise or Standard?

The core engine is based on the SQL Server 2008 technology foundation. The feature set does not map to a specific SQL Server SKU – in fact you should think of the service as its own SKU.

Will we be able to connect to our cloud instances from SQL Server Management Studio (SSMS)? And will we be able to buy just the client tools without an on-premises server license?

Our plans are to support SSMS, but it might be a requirement to install a new version or a hotfix or two. As far as buying just the client tools, using SSMS Express is a great option. You can find out more info here.

Will replication between instances be offered to aid with Business Intelligence?

Not initially. After you start working with the service let us know what functionality to include or scenarios to support. We want to hear from you!

Will you offer hosted SSIS/SSAS/SSRS?

It’s on the product roadmap, but I can’t comment on specifics or timing.

The blog entry states that only SQL Server authentication (username/password) will be offered initially. Can we assume that eventually the Microsoft Services Connector will be used to offer Windows Authentication?

Post v1 you can expect a more robust authentication story. I can’t discuss implementation details yet.

Will we have the option to expose a REST head using ADO.NET Data Services as part of SDS’s offerings (i.e. at the flip of a switch) or will we have to implement ADO.NET Data Services separately ourselves on Windows Azure?

During the v1 timeframe you will need to implement this yourself on the Azure Services Platform. This approach is very similar to using ADO.NET Data Services against an on-premises SQL Server. There are many key capabilities of the framework that you specify (security, etc.) and so this does require some coding.

You say, you will no longer support the ACE model since Windows Azure has the same data model. If I’m going to put my application in Windows Azure (as I did) my best option for data is SDS, right? As I understood, Microsoft’s approach for cloud computing was Windows Azure as the platform for developers and SDS to store the data. And now you say I can use Windows Azure storage… what’s the difference? What’s the path to follow?

The best storage option for an Azure Services Platform application depends on your application. At a very high level, if you require the features of a relational database, use SDS. If you require basic blob or “schemaless” storage, then Windows Azure Storage is for you. Both will be key capabilities available to developers in the overall Azure Services platform.

Will the new SDS features support accessing SQL Server in the Cloud using LINQ to Entities and the ADO.NET Entity Framework?

It will work based on our client library story. Interop through ADO.NET is one of our top scenarios.

I just finished reading the latest post on Data Platform Insider Blog. It says that customers who wish to use REST based interfaces to SDS can do it through customized ADO.NET Data Services. Does this mean that customers can consume the REST based ADO.NET Data Services through any platform?

Yes that is correct. I would also like to point out that we have drivers for SQL Server to work with most popular development stacks like PHP, Java and Ruby. Those drivers work with SDS as well.

Your latest blog posting says that the ACE model will no longer be supported – I guess you are referring to the SDS storage, does this also include the Windows Azure sStorage?

These announcements do not affect Windows Azure Storage in any way. That is still a key foundational piece of the Azure Services Platform.

How can an existing applications based on the ACE model be migrated to the newer version?

There is no direct migration path from the ACE based SDS to the relational based SDS. What you can do, however, is use the existing SSIS adaptor to move the data from your containers down to tables within a SQL Server or SQL Server Express database and then start using all of the relational features in the product. When the CTP of SDS goes live mid-calendar year, just deploy your schema and database to SDS via the available tools (e.g. Visual Studio)

Does Windows Azure Storage and SDS both use SQL Server under the hood? Is there really a difference between the two storage options?

I don’t have all the details on how Windows Azure Storage works under the hood, but these are two distinct storage options of the Azure Services Platform – similar to the difference between saving data to your local file system and saving it to a database.

With ADO.NET Data Services compatibility, do you mean I can work against the System.Data.SqlClient and in the theory only change my connection string to switch between a local database and SDS?

That’s exactly what I mean. Pretty cool isn’t it?

SSDS at PDC 2008

Soumitra Sengupta — Thu, 18 Sep 2008 20:53:00 GMT

It is around 40 days to PDC and the excitement is building. We have a ton of work to do between now and then and the team is heads down getting ready for this major event. If you look at the PDC 2008 breakout sessions already announced, it is incredible that we will have 26 sessions on Cloud Services and 6 sessions related to SQL Server Data Services (SSDS). You can look it up here. You can see Mike and Jennifer's interview with Amitabh Srivastava here, where he talks about some of the things his team has been working on. Unfortunately he cannot talk much about it, but I can assure you coming to PDC this year will be well worth it. In my mind this is as big and maybe bigger than when we announced the .Net wave. The cloud wave is a major event and disruption in our industry and you will get to see and hear from the guys driving this wave inside Microsoft at PDC 2008.

That is as much as I can say about our overall Cloud services initiatives. Specifically, as a member of the SQL Server Data Services (SSDS) team, I am psyched to say that we have scheduled 3 talks directly related to SSDS, 1 talk from our Technical Fellow Dave Campbell on how we are evolving and extending our comprehensive Data Platform and the fast growing SQL business to the cloud and another 3-4 talks on technologies like Astoria, Velocity, Sync Framework and Entity Frameworks that are important to our cloud strategy. So here are the key talks and a brief outline of what you can expect from these talks at PDC.

1. Microsoft SQL Server: Data-Driven Applications from Device to Cloud - Dave Campbell. Dave has been through it all. He will present the key trends that are shaping our industry and how we are evolving our Data Platform from devices, desktop, server to the cloud and the services that makes this platform worth betting on. If you ever wondered what software and services have to do with data, you do not want to miss this talk. Dave's talk will set the stage for all the talks to follow on data and tie them together. So this has to be a must attend talk for you if you care about data.

2. A Lap around SQL Server Data Services - Soumitra Sengupta. This will be an advanced technical presentation where I will explain how we got started on SSDS, how it relates to our Cloud and Data Platform strategy, how we see our Cloud services vis-a-vis our on-premises SQL Server offering. You will also learn why SSDS seems like a departure from hosted SQL Server. I get that one a lot. Tables, Columns, T-SQL anyone. I will fly through this material quickly and spend most of the time coding up an application using our REST and SOAP interfaces. I promise there will be one or two surprises during this presentation. I will also explain how we run our service and the trade-offs and questions that should be on developers' mind as they build applications using SSDS.

3. Under the Hood: Building SQL Server Data Services - Istvan Cseri and Gopal Kakivaya. Man what can I say. These guys are going to take the lid off SSDS and really dive under the hood and talk about how this service is built. If you want to see how to build a fault-tolerant, highly available multi-datacenter service, you cannot miss this talk. It will demystify what we have built and how we operate this. This is the first time we will go this deep. These guys are real rock stars and I will not be upset if you miss my talk BUT DO NOT MISS THIS ONE.

4. SQL Server Data Services: Futures - Patric McElroy. If you are curious after listening to the first 3 talks about where we are headed with SSDS, this is the talk for you. There will be lots of surprises as Patric will show you how we are thinking about extending our entire data platform and services to the cloud. This is a real forward looking session but there will be demos on the way to give you a clear idea of the vision and the path ahead.

In addition we will have lunch sessions, lounges, booth and Hand-on-Labs. You will get to meet, greet, ask questions and socialize with the team members who are making it all happen and if there is enough interest, we will try to get a group together and do something fun during PDC. So if you are interested let, David Robinson know. I know he is in touch with Mike Amundsen already about this.

Aside from these 4 talks, we have the following talks where we will establish the linkages between Sync, ADO.Net Data Services (Astoria), Velocity and other Microsoft services and SSDS. So it would be interesting for you to attend these as well.

1. Microsoft Sync Framework: Enterprise Data in the Cloud and on Devices - Liam Cavanagh
2. Microsoft Sync Framework Advances - Lev Novik
3. Developing Applications Using Data Services - Mike Flasko
4. Entity Framework Futures - Tim Mallalieu
5. Project "Velocity": A First Look - Murali Krishnaprasad
6. A Lap around Building Block Services - John Shewchuk
7. Architecture of the Building Block Services - Dennis Pilarinos, John Shewchuk

Obviously there will be the Keynote from Ray Ozzie to set the stage for this wave and you cannot miss that. I also hear Don Box and Chris Anderson are cooking up something using SSDS. These 2 guys have always lit up PDC and I do not think you will be disappointed this time around.

See you all there.

Jon Udell cURL's with SQL Server Data Services

Soumitra Sengupta — Thu, 28 Aug 2008 03:31:00 GMT

Just read Jon Udell's post on SQL Server Data Services (SSDS) here. Happy to hear that Jon was able to explore SSDS with cURL "within minutes of cracking open the SSDS documentation". Readers who are wondering what Jon is talking about, should read Jeff Currier, one of the guys writing real code that we ship in SSDS, talking about using cURL to explore SSDS here.

When few of us crazies started thinking of SSDS, a non-negotiable requirement was that developers should be able to start exploring and using the service quickly. There is so much going on in the cloud services space that if it took too long to experiment with a new service, we were afraid that developers will put it on the shelve and may never give it a spin. I know I do not like to spend months on something to see if I would use it. It was hard for us to "cover up" the SQL Server heritage of SSDS, but we argued that simplicity is a virtue that we should aspire to.

Jon hits on an interesting point when he says 'while the novelty of “just coding to a URL” on a Microsoft platform will undoubtedly attract some tirekickers who otherwise wouldn’t show up, the real draw will be the ability to exercise choice along the whole continuum'. Internally we use the words "reach" and "rich". We believe that to be a platform with large adoption, one needs to provide both "reach" (so that people from all other platforms can use you) as well as "richness" (so that people who use your platform exlusively have a rich set of tools to leverage). Today it is easy to see that SSDS is aiming to "reach" as many developers as possible. To do this we have to address "the continuum of access styles". Jon seems to be saying that we are meeting that goal. What do you think? Are we close? Are we way off base? Or we should not worry about the "continuum of access styles".

It will be interesting to hear from you all.

Why Astoria alignment is not that trivial

Soumitra Sengupta — Tue, 15 Jul 2008 08:56:00 GMT

I have been meaning to write about this topic for quite some time. But it took me quite a while to understand deeply the key issues in aligning Astoria and SSDS. Thanks to Alan Bush, an architect in our team, I have a much better understanding of the key issues. The tendency for most of us is to focus on the data model and query language differences between Astoria and SSDS. Turns out, while they are important, they are not the most difficult ones to work in. So here is what I have learnt from Alan:

a. Astoria protocol and interaction model deals with only a single consistency domain. Each consistency domain sort of maps to a database. Astoria consistency domain is akin to SSDS container. SSDS data model actually spans multiple consistency domains. It is possible to write a query across containers or even query for container properties. Yes Astoria can be extended to address this and we are working on that with Pablo and the Astoria team.
b. Multi-tenancy is a fundamental concept in SSDS while Astoria has no such concept at this point in time. We have to work through this as well. This has huge implications in the security model for the service.
c. Entities in SSDS do not require schema while default for Astoria is EDM schemas. While on the surface this is an easy problem to solve yet it requires you to think through all the options. If you look at the data model for Google App Engine, you will see how they chose to solve this problem - typed entities and expando entities are 2 separate concepts. It is one way to deal with it but think about updates on a graph where you have both typed and flexible entities, how would the user deal with the semantic differences between the two concepts in the same transaction?

There are host of issues but these are some of the top ones we are grappling with. Rest assured minds lot sharper than mine are working this and if there is an elegant solution, they will find it. I have full faith that by PDC we will have a compelling story to tell around SSDS and Astoria. So stay tuned.

Roger Jennings talks about SSDS in a Visual Studio Magazine Article

Soumitra Sengupta — Wed, 02 Jul 2008 10:34:00 GMT

Roger has an interesting article on SSDS in the latest Visual Studio Magazine here. It is really interesting to see the performance data between the SOAP and the REST end points of SSDS. It would be really interesting to dig into this.

One bit of correction - the SSDS flexible entity model was developed independent of EDM / EF and the Astoria model. At this point in time, both teams are working together to get this aligned.

I thought Dave Campbell in one of his press interviews had mentioned it but I could not find the reference. Sorry Roger. So here is an aswer to your question - SSDS is built using SQL Server 2005, SP2 as the starting code base. That is the starting point and we made changes to it. Over time, some of these changes will make its way into the SQL Server mainline and SQL Server 2008 will make its way into SSDS. We wanted to proceed in parallel as fast as we could and this was the best way to do it.