SQL Azure Team Blog : SSDS

Dave Campbell talks about extending the Data Platform to the Cloud

Soumitra Sengupta — Tue, 23 Sep 2008 00:08:00 GMT

Here is a little teaser on what we are upto from a direction perspective. Dave Campbell talks about what to expect at PDC 2008 as we extend our Data Platform to the Cloud. As I said before, Dave has been involved with everything SQL since the very beginning and his talk will set the stage for the next evolution of our platform. You can listen to Dave talk to Mike and Jennifer here on Channel 9.

SSDS at PDC 2008

Soumitra Sengupta — Thu, 18 Sep 2008 20:53:00 GMT

It is around 40 days to PDC and the excitement is building. We have a ton of work to do between now and then and the team is heads down getting ready for this major event. If you look at the PDC 2008 breakout sessions already announced, it is incredible that we will have 26 sessions on Cloud Services and 6 sessions related to SQL Server Data Services (SSDS). You can look it up here. You can see Mike and Jennifer's interview with Amitabh Srivastava here, where he talks about some of the things his team has been working on. Unfortunately he cannot talk much about it, but I can assure you coming to PDC this year will be well worth it. In my mind this is as big and maybe bigger than when we announced the .Net wave. The cloud wave is a major event and disruption in our industry and you will get to see and hear from the guys driving this wave inside Microsoft at PDC 2008.

That is as much as I can say about our overall Cloud services initiatives. Specifically, as a member of the SQL Server Data Services (SSDS) team, I am psyched to say that we have scheduled 3 talks directly related to SSDS, 1 talk from our Technical Fellow Dave Campbell on how we are evolving and extending our comprehensive Data Platform and the fast growing SQL business to the cloud and another 3-4 talks on technologies like Astoria, Velocity, Sync Framework and Entity Frameworks that are important to our cloud strategy. So here are the key talks and a brief outline of what you can expect from these talks at PDC.

1. Microsoft SQL Server: Data-Driven Applications from Device to Cloud - Dave Campbell. Dave has been through it all. He will present the key trends that are shaping our industry and how we are evolving our Data Platform from devices, desktop, server to the cloud and the services that makes this platform worth betting on. If you ever wondered what software and services have to do with data, you do not want to miss this talk. Dave's talk will set the stage for all the talks to follow on data and tie them together. So this has to be a must attend talk for you if you care about data.

2. A Lap around SQL Server Data Services - Soumitra Sengupta. This will be an advanced technical presentation where I will explain how we got started on SSDS, how it relates to our Cloud and Data Platform strategy, how we see our Cloud services vis-a-vis our on-premises SQL Server offering. You will also learn why SSDS seems like a departure from hosted SQL Server. I get that one a lot. Tables, Columns, T-SQL anyone. I will fly through this material quickly and spend most of the time coding up an application using our REST and SOAP interfaces. I promise there will be one or two surprises during this presentation. I will also explain how we run our service and the trade-offs and questions that should be on developers' mind as they build applications using SSDS.

3. Under the Hood: Building SQL Server Data Services - Istvan Cseri and Gopal Kakivaya. Man what can I say. These guys are going to take the lid off SSDS and really dive under the hood and talk about how this service is built. If you want to see how to build a fault-tolerant, highly available multi-datacenter service, you cannot miss this talk. It will demystify what we have built and how we operate this. This is the first time we will go this deep. These guys are real rock stars and I will not be upset if you miss my talk BUT DO NOT MISS THIS ONE.

4. SQL Server Data Services: Futures - Patric McElroy. If you are curious after listening to the first 3 talks about where we are headed with SSDS, this is the talk for you. There will be lots of surprises as Patric will show you how we are thinking about extending our entire data platform and services to the cloud. This is a real forward looking session but there will be demos on the way to give you a clear idea of the vision and the path ahead.

In addition we will have lunch sessions, lounges, booth and Hand-on-Labs. You will get to meet, greet, ask questions and socialize with the team members who are making it all happen and if there is enough interest, we will try to get a group together and do something fun during PDC. So if you are interested let, David Robinson know. I know he is in touch with Mike Amundsen already about this.

Aside from these 4 talks, we have the following talks where we will establish the linkages between Sync, ADO.Net Data Services (Astoria), Velocity and other Microsoft services and SSDS. So it would be interesting for you to attend these as well.

1. Microsoft Sync Framework: Enterprise Data in the Cloud and on Devices - Liam Cavanagh
2. Microsoft Sync Framework Advances - Lev Novik
3. Developing Applications Using Data Services - Mike Flasko
4. Entity Framework Futures - Tim Mallalieu
5. Project "Velocity": A First Look - Murali Krishnaprasad
6. A Lap around Building Block Services - John Shewchuk
7. Architecture of the Building Block Services - Dennis Pilarinos, John Shewchuk

Obviously there will be the Keynote from Ray Ozzie to set the stage for this wave and you cannot miss that. I also hear Don Box and Chris Anderson are cooking up something using SSDS. These 2 guys have always lit up PDC and I do not think you will be disappointed this time around.

See you all there.

Jon Udell cURL's with SQL Server Data Services

Soumitra Sengupta — Thu, 28 Aug 2008 03:31:00 GMT

Just read Jon Udell's post on SQL Server Data Services (SSDS) here. Happy to hear that Jon was able to explore SSDS with cURL "within minutes of cracking open the SSDS documentation". Readers who are wondering what Jon is talking about, should read Jeff Currier, one of the guys writing real code that we ship in SSDS, talking about using cURL to explore SSDS here.

When few of us crazies started thinking of SSDS, a non-negotiable requirement was that developers should be able to start exploring and using the service quickly. There is so much going on in the cloud services space that if it took too long to experiment with a new service, we were afraid that developers will put it on the shelve and may never give it a spin. I know I do not like to spend months on something to see if I would use it. It was hard for us to "cover up" the SQL Server heritage of SSDS, but we argued that simplicity is a virtue that we should aspire to.

Jon hits on an interesting point when he says 'while the novelty of “just coding to a URL” on a Microsoft platform will undoubtedly attract some tirekickers who otherwise wouldn’t show up, the real draw will be the ability to exercise choice along the whole continuum'. Internally we use the words "reach" and "rich". We believe that to be a platform with large adoption, one needs to provide both "reach" (so that people from all other platforms can use you) as well as "richness" (so that people who use your platform exlusively have a rich set of tools to leverage). Today it is easy to see that SSDS is aiming to "reach" as many developers as possible. To do this we have to address "the continuum of access styles". Jon seems to be saying that we are meeting that goal. What do you think? Are we close? Are we way off base? Or we should not worry about the "continuum of access styles".

It will be interesting to hear from you all.

SSDS Software Developer Kit (SDK) Beta Released on MSDN

Soumitra Sengupta — Tue, 19 Aug 2008 20:13:00 GMT

Just learnt from Mohan Vanmane that the SQL Server Data Services (SSDS) SDK Beta is now live on MSDN. You can get it from here. The documentation is available here. The SDK includes the command line tool and the SSDS Explorer that I demonstrated at TechEd 2008. These are beta version of the tools. So the team would appreciate if you can give it a spin and let us know what you like, what you do not like and above all file bugs that you see. I should mention that you do need a SSDS account to use these tools.

The team is seeking feedback that you can provide through our fourm here. Here is an excerpt from the email that the team sent out today:

"As we continue to build on our tools strategy, we would like to hear from you on how we could improve the developer experience when working with SSDS. The feedback we seek is primarily related to what tools and/or libraries would help you most when developing applications for SSDS. These can be targeted for development, data management, performance, debugging or any other area of SSDS for which you would like us to provide better support."

We appreciate all feedback.

On going quiet and transparency in the design process

Soumitra Sengupta — Fri, 15 Aug 2008 19:22:00 GMT

Roger has challenged us to be transparent as we design SQL Server Data Services. He has urged us couple of times to follow the process that the ADO.Net Data Services team has followed. First let me assert again that we are all in favor of an open process that gets the community and experts who want to provide guidance involved. Second I have seen the benefits of the open and transparent design process during my XML days and prior to that my open source days. So personally I am committed to being open and transparent.

Having said that, we are currently working under several constraints. SQL Server Data Services is only one part of a very broad and deep services initiative which will only come to full light at the upcoming PDC. While we want to be open and transparent, we want to make sure we do not disclose anything inadvertently. This limits the degrees of freedom we have when we discuss features we want to put in and designs for those features. I am very hopeful that post PDC, we will be progressively be more open about our features and design.

Regarding going quiet between sprints - this is the result of 2 separate issues. One our team was quite small till about March of this year. As the team has grown, we have lot of new people who are upto their eyeballs catching up and getting up the steep unlearning and learning curves of transitioning from software to a services mindset. This leaves us with very little bandwidth especially during sprints to engage in deep discussions. Second, this has been a very busy year starting with MIX, TechEd, Worldwide Partner Conference, TechReady and now PDC and TechEd EMEA for the team. To top it all, summers are usually slow time in Redmond as people tend to take their vacations during this time of the year. I was out of the country for 2 weeks with no access to cell phones or email. I am officially still on vacation. But this is an important topic of discussion in the forums and in the blog and I appreciate your concern. Just wanted to shoot this out to alleviate your concerns and request for some patience. PDC is only 2 months away and I urge that you all attend PDC and get the full scope of what is cooking, meet all the key people, share your concerns and ideas about software and services. This is a big wave and we cannot ride it without you. I promise it will be a fun, exciting and profitable ride for all of us.

SQL Server Data Services Sprint 3 now Live

Soumitra Sengupta — Fri, 25 Jul 2008 21:18:00 GMT

We completed the rollout of Sprint 3 yesterday. The team is already half way into Sprint 4. In Sprint 4 we are focussed on things we need for PDC 2008, which by the way is in late October in Los Angeles. Ray Ozzie and Steve Ballmer foreshadowed some of what this PDC will be all about here and here. You can also go here to see the already announced sessions. SSDS will be a key part of this year's PDC. We set our technical and product agendas that developers care about at PDC. If you want to understand and learn what Services is all about in our Software plus Services strategy, do not miss this year's PDC.

Overall, this was an interesting deployment as we had new environments come online that had to be configured. The team worked really really hard to make the deployment as smooth as possible. Interestingly we did not hit any issues during deployment yesterday. Wish we could video tape how we do this and share with our readers. Something I have to take up with management. I think it will give everyone an appreciation of the amount of planning, tooling, automation and discipline that goes into these deployments.

Anyway as developers, you care about what new things we are enabling. Things have been leaked on this before by Roger, David Robinson and Jeff Currier. But here are the new features:

Blob Support via the REST interface

In this release, Blobs are limited to 100 megabytes in size.
Blob Support via the SOAP interface will be delivered in a later update.

Version metadata property has changed from a large integer to a container-wide dbtimestamp value:

Version numbers are now based on a dbtimestamp and will increase monotonically on a container-wide basis. No two entities in the same container will ever receive the same version value. The dbtimestamp increases monotonically, however not necessarily sequentially. Clients should no longer assume entity version numbers are sequentially assigned. It does not start at 0, but may be any non-zero integer value for a newly created entity.

ETag support via REST, with similar functionality provided on the SOAP interface

Support for If-Match and If-None-Match via the REST interface. This will allow you to verify that you have the latest version of an Entity, enabling you to use resources more efficiently.
Support for “Accept” header via the REST interface. This will allow you to denote what content you want returned from the service. For instance when retrieving a Blob, the “Accept” header will allow you to choose between the Blob content, or the metadata properties associated with it.
Support for ETag semantics on the SOAP service via the new Version Match object on the Scope object.

HEAD support via REST interface

Support for the “HEAD” verb via the REST interface. This provides a lightweight way of validating existing entity version information without requiring entire entities (or blobs) to be retransmitted to the client.

A new content type – “application/x-ssds+xml” – has been added to the service

This content type is for storing XML content in the service.
Existing “application/xml” content type is being phased out in a future sprint.
The "application/x-ssds+xml" content type should be used for all entities except those containing blob content. Blob entities should instead use a content type value which best reflects the type of blob data stored. It's important to note that the value chosen here will be used later when attempting to retrieve the different parts of an entity by using the "Accept" header.

In addition we did some infrastructure work to enable metering (we will be charging for this service some day :-)) ), throttling (DoS attacks happen and the service needs to work in degraded situation), quota enforcement etc. that developers will not see much of at this time. But once we open our management, debugging and monitoring interfaces, you will be able to plug in and keep up with what is going on with your service.

The primer is updated at our Dev Center here to help you take advantage of the new features. There is one other new "thing" that we are enabling in this rollout that we will talk about a bit later. It is not huge, but it will demonstrate that we are thinking about our building block services very very broadly and deeply. Stay tuned.

Mike Amundsen is putting out lot of samples in his blog

Soumitra Sengupta — Wed, 16 Jul 2008 21:05:00 GMT

I just noticed that Mike has put out interesting code in his "life in lowercase" blog. It is good to see him add caching support to his provisioning client. Just put it on my "to play with during my vacation" list for the summer. Thanks Mike.

By the way, I responded to Mike's comments on SSDS and Astoria alignment. I do not disagree with Mike that one of the core values of SSDS is its simplicity. It is a tribute to the team that they have been able to keep it simple for developers to approach and use right away. I appreciate Mike's concern that aligning SSDS with Astoria could make SSDS lot more complicated. As I said in my response, the alignment does not mean that we are ditching the flex entity model. Once we release the aligned service, as a developer you will have a choice of staying with the flex entity model or add schemas where it makes sense to add schemas. We hope to provide this capability at the container level so you can choose to:

a. Keep all entities as flex entities
b. Keep all entities as typed entities
c. Have both typed and untyped entities and type entities with open or flex properties

The side effect is that the Astoria client library then becomes the default client library for developers to build their applications in Visual Studio. Still does not solve the problem of client libraries for Java, Ruby and PHP. I still got those on my plate to address.

If you look at the model above, you will notice that Amazon SimpleDB and SSDS only allows you to do a. today. GAE lets you do a. and b. We think c. is really very powerful and see all sorts of scenarios can be modeled using c. quite effectively.

At the time of writing this blog, I have not seen a plan for supporting EDM associations post alignment. I also want to point out that this alignment is actually good for EDM as it extends the current EDM model to cover untyped or flex entities. It is also important to point out that it is possible to bet on EDM without using EF. Astoria itself is an example of that.

Why Astoria alignment is not that trivial

Soumitra Sengupta — Tue, 15 Jul 2008 08:56:00 GMT

I have been meaning to write about this topic for quite some time. But it took me quite a while to understand deeply the key issues in aligning Astoria and SSDS. Thanks to Alan Bush, an architect in our team, I have a much better understanding of the key issues. The tendency for most of us is to focus on the data model and query language differences between Astoria and SSDS. Turns out, while they are important, they are not the most difficult ones to work in. So here is what I have learnt from Alan:

a. Astoria protocol and interaction model deals with only a single consistency domain. Each consistency domain sort of maps to a database. Astoria consistency domain is akin to SSDS container. SSDS data model actually spans multiple consistency domains. It is possible to write a query across containers or even query for container properties. Yes Astoria can be extended to address this and we are working on that with Pablo and the Astoria team.
b. Multi-tenancy is a fundamental concept in SSDS while Astoria has no such concept at this point in time. We have to work through this as well. This has huge implications in the security model for the service.
c. Entities in SSDS do not require schema while default for Astoria is EDM schemas. While on the surface this is an easy problem to solve yet it requires you to think through all the options. If you look at the data model for Google App Engine, you will see how they chose to solve this problem - typed entities and expando entities are 2 separate concepts. It is one way to deal with it but think about updates on a graph where you have both typed and flexible entities, how would the user deal with the semantic differences between the two concepts in the same transaction?

There are host of issues but these are some of the top ones we are grappling with. Rest assured minds lot sharper than mine are working this and if there is an elegant solution, they will find it. I have full faith that by PDC we will have a compelling story to tell around SSDS and Astoria. So stay tuned.

Trade-offs all around in SQL Server Data Services (SSDS)

Soumitra Sengupta — Tue, 15 Jul 2008 08:41:00 GMT

David Robinson talks about concurrency here. Roger comments about David's take on concurrency here. In SSDS, we make several important trade-offs:

a. We give up some latency because of quorum based commits in favor of aggregate throughput. Theoretically this approach lets you hyper partition the data to get extreme throughput if that is what the app needs. We have to remember SSDS is a big train or a bus and not a finely tuned Ferrari. I believe that is what David was referring to in his article.
b. Trade-off single system image and backward compatibility of existing SQL Server applications for lights out operations, high scale and availability. I know this raises a lot of eyebrows. But giving up backward compatibility allowed us lots of freedom in choosing a data model and optimize for a completely different cost point.

It is important we understand these nuances when dealing with a large scale distributed data service.

Nice. David Robinson's MSDN Article about SSDS is out

Soumitra Sengupta — Tue, 15 Jul 2008 08:28:00 GMT

Just noticed this. David Robinson and Jeff Currier are the best writers on our team. Check it out.

My TechEd 2008 Talk is now online

Soumitra Sengupta — Tue, 15 Jul 2008 02:30:00 GMT

I presented a talk titled "Introduction to Microsoft SQL Server Data Services" (DAT 251). You can view the session Introduction to Microsoft SQL Server Data Services in its entirety. I got the chance to share some details we have not had the chance to share yet. Roger Jenninngs posted a link as well here. He also included a short wish list. I have taken note and will let all of you know if and when we can post the code for the command line tool and the SSDS Management Studio demos that I showed during this presentation.

Roger Jennings talks about SSDS in a Visual Studio Magazine Article

Soumitra Sengupta — Wed, 02 Jul 2008 10:34:00 GMT

Roger has an interesting article on SSDS in the latest Visual Studio Magazine here. It is really interesting to see the performance data between the SOAP and the REST end points of SSDS. It would be really interesting to dig into this.

One bit of correction - the SSDS flexible entity model was developed independent of EDM / EF and the Astoria model. At this point in time, both teams are working together to get this aligned.

I thought Dave Campbell in one of his press interviews had mentioned it but I could not find the reference. Sorry Roger. So here is an aswer to your question - SSDS is built using SQL Server 2005, SP2 as the starting code base. That is the starting point and we made changes to it. Over time, some of these changes will make its way into the SQL Server mainline and SQL Server 2008 will make its way into SSDS. We wanted to proceed in parallel as fast as we could and this was the best way to do it.

Ryan has posted his Working with Objects Part 2

Soumitra Sengupta — Fri, 27 Jun 2008 10:34:00 GMT

You can find part 2 Ryan's post on working with objects using SSDS here.

Philosophy behind the design of SSDS and some personal thoughts

Soumitra Sengupta — Fri, 27 Jun 2008 09:21:00 GMT

With Sprint 3 winding down, I thought it is a good time for me to share with you some of the philosophy behind the design of SQL Server Data Services (SSDS) and a few personal thoughts about the experience. When we started this project 2 years back, we realized that we had three fairly difficult problems to solve before we could credibly roll out an internet scale data service. The 3 big problems in order of complexity are:

a. Building a scale free, highly available consistent data service that is fault tolerant and self healing
b. Building the service using low cost commonly available hardware
c. Building a service that was also cheap to operate - lights out operation

Solving problems a and b incidentally makes problem c a bit more complex as you end up with lot more hardware to manage and the hardware tend to fail more often. As a team, we made what I think was a wise decision to use technology already proven to solve these problems. The only area where we had to do a ton of heavy lifting was solving problem a. It allowed us to focus the team's energy on the most difficult problem when it comes to scaling out stateful services. I am not going to go into the details of our approach. If you are interested in hearing about this, I urge you to attend PDC 2008 and hear it from the architects who actually solved this problem.

For b and c we mostly used technologies already available within Microsoft and adapted it for stateful services. Initially we thought this would be fairly easy but it turned out to be more complicated than we thought, especially given the fact that we had to put in infrastructure software that allow us to debug problems with the service without attaching debuggers to the machine or touching the machine. We had to put in logging and tracing infrastructure and given that we all got the logging and tracing religion, in one of our sprints early on we inadvertently dumped so much that we shut down the service effectively as there were no resources left to respond to user requests. Some of our early experiences are fodder for some very interesting hallway conversations. But it taught us that there are quite a few Ph.D. level research topics around debugging large scale distributed systems and if you are up for it and interested in working on them, do give us a holler. Even though we have quite a few Ph.D.'s in the team, we could use some more help :-))

Since we made an early decision to limit the number of hard problems we needed to solve, we decided that we would focus less on the features of the service but more on the quality of the service and the cost of standing up and running the service. The less the service does we argued, the easier it would be for us to achieve our objectives. In hindsight, this was probably one of the best decisions we made. Istvan, Tudor and Nigel deserve special credit for keeping us focussed on "less is better". It also allowed us to learn about the pitfalls of running such a service, including upgrading the service without shutting it down. We did not shy away from complex problems, but we made sure if we could limit the surface area without losing a ton of value, we always took that path. We are still in the learning mode and learning every day about workloads that cause "irregular heartbeats" to our service and how to handle such workloads. But the team has definitely come a long way, working with internal partner teams, working very closely with our operations guys (who by the way are absolutely awesome) and now with our beta customers.

While a service is still about software, and the fundamentals still hold, the engineering process, cadence and discipline it requires, I think is quite different from shipping shrink wrapped software. It is easier in some dimensions (like our test matrix is not huge), it is more difficult in others (debugging a large scale distributed system). We had to unlearn quite a few things (like it is better to kill a sick process fast than try to keep it up at all cost) before we could start climbing up the learning curve. It is really quite an experience for us, one that I would not trade for anything else.

If I have to think about this experience as crawl, walk and then run, as Dave Campbell puts it "we are just about getting our knees off the ground so we can start to walk". Yes we have been cautious and yes it is frustrating that the rich capabilities of SQL Server are not accessible to our customers, but I think we are going about it the best way we know how and I am confident we are doing it the right way. Over time, as we learn more about the system we have built, as we roll out more hardware in the datacenters (and find new problems), as we learn what it takes to run a 24x7x365 service (nobody we know of is running a data service using a commercial database system at this scale and cost point) like SSDS, I can assure you we will start to expose capabilities of the underlying engine. How quickly and how much will depend on our ability to provide you, our customers with the quality of service you need to trust your business to SSDS.

Thank you for your patience.

Mike Amundsen has posted his TaskList Demo

Soumitra Sengupta — Thu, 26 Jun 2008 23:10:00 GMT

Found out through Roger Jennings' blog here that Mike Amundsen has posted his TaskList demo. Mike says that the code is not quite ready for live demo but it works for now. You can find it here. Thanks Mike for posting it and Roger for pointing us to it. If you have built any such demos and would like to share it, please send us the link. You can post it in the comments or ping me (soumitrs at <ignore this stuff> microsoft.com).