SQL Azure Team Blog : Query Language

Why Astoria alignment is not that trivial

Soumitra Sengupta — Tue, 15 Jul 2008 08:56:00 GMT

I have been meaning to write about this topic for quite some time. But it took me quite a while to understand deeply the key issues in aligning Astoria and SSDS. Thanks to Alan Bush, an architect in our team, I have a much better understanding of the key issues. The tendency for most of us is to focus on the data model and query language differences between Astoria and SSDS. Turns out, while they are important, they are not the most difficult ones to work in. So here is what I have learnt from Alan:

a. Astoria protocol and interaction model deals with only a single consistency domain. Each consistency domain sort of maps to a database. Astoria consistency domain is akin to SSDS container. SSDS data model actually spans multiple consistency domains. It is possible to write a query across containers or even query for container properties. Yes Astoria can be extended to address this and we are working on that with Pablo and the Astoria team.
b. Multi-tenancy is a fundamental concept in SSDS while Astoria has no such concept at this point in time. We have to work through this as well. This has huge implications in the security model for the service.
c. Entities in SSDS do not require schema while default for Astoria is EDM schemas. While on the surface this is an easy problem to solve yet it requires you to think through all the options. If you look at the data model for Google App Engine, you will see how they chose to solve this problem - typed entities and expando entities are 2 separate concepts. It is one way to deal with it but think about updates on a graph where you have both typed and flexible entities, how would the user deal with the semantic differences between the two concepts in the same transaction?

There are host of issues but these are some of the top ones we are grappling with. Rest assured minds lot sharper than mine are working this and if there is an elegant solution, they will find it. I have full faith that by PDC we will have a compelling story to tell around SSDS and Astoria. So stay tuned.

Roger Jennings talks about SSDS in a Visual Studio Magazine Article

Soumitra Sengupta — Wed, 02 Jul 2008 10:34:00 GMT

Roger has an interesting article on SSDS in the latest Visual Studio Magazine here. It is really interesting to see the performance data between the SOAP and the REST end points of SSDS. It would be really interesting to dig into this.

One bit of correction - the SSDS flexible entity model was developed independent of EDM / EF and the Astoria model. At this point in time, both teams are working together to get this aligned.

I thought Dave Campbell in one of his press interviews had mentioned it but I could not find the reference. Sorry Roger. So here is an aswer to your question - SSDS is built using SQL Server 2005, SP2 as the starting code base. That is the starting point and we made changes to it. Over time, some of these changes will make its way into the SQL Server mainline and SQL Server 2008 will make its way into SSDS. We wanted to proceed in parallel as fast as we could and this was the best way to do it.

Philosophy behind the design of SSDS and some personal thoughts

Soumitra Sengupta — Fri, 27 Jun 2008 09:21:00 GMT

With Sprint 3 winding down, I thought it is a good time for me to share with you some of the philosophy behind the design of SQL Server Data Services (SSDS) and a few personal thoughts about the experience. When we started this project 2 years back, we realized that we had three fairly difficult problems to solve before we could credibly roll out an internet scale data service. The 3 big problems in order of complexity are:

a. Building a scale free, highly available consistent data service that is fault tolerant and self healing
b. Building the service using low cost commonly available hardware
c. Building a service that was also cheap to operate - lights out operation

Solving problems a and b incidentally makes problem c a bit more complex as you end up with lot more hardware to manage and the hardware tend to fail more often. As a team, we made what I think was a wise decision to use technology already proven to solve these problems. The only area where we had to do a ton of heavy lifting was solving problem a. It allowed us to focus the team's energy on the most difficult problem when it comes to scaling out stateful services. I am not going to go into the details of our approach. If you are interested in hearing about this, I urge you to attend PDC 2008 and hear it from the architects who actually solved this problem.

For b and c we mostly used technologies already available within Microsoft and adapted it for stateful services. Initially we thought this would be fairly easy but it turned out to be more complicated than we thought, especially given the fact that we had to put in infrastructure software that allow us to debug problems with the service without attaching debuggers to the machine or touching the machine. We had to put in logging and tracing infrastructure and given that we all got the logging and tracing religion, in one of our sprints early on we inadvertently dumped so much that we shut down the service effectively as there were no resources left to respond to user requests. Some of our early experiences are fodder for some very interesting hallway conversations. But it taught us that there are quite a few Ph.D. level research topics around debugging large scale distributed systems and if you are up for it and interested in working on them, do give us a holler. Even though we have quite a few Ph.D.'s in the team, we could use some more help :-))

Since we made an early decision to limit the number of hard problems we needed to solve, we decided that we would focus less on the features of the service but more on the quality of the service and the cost of standing up and running the service. The less the service does we argued, the easier it would be for us to achieve our objectives. In hindsight, this was probably one of the best decisions we made. Istvan, Tudor and Nigel deserve special credit for keeping us focussed on "less is better". It also allowed us to learn about the pitfalls of running such a service, including upgrading the service without shutting it down. We did not shy away from complex problems, but we made sure if we could limit the surface area without losing a ton of value, we always took that path. We are still in the learning mode and learning every day about workloads that cause "irregular heartbeats" to our service and how to handle such workloads. But the team has definitely come a long way, working with internal partner teams, working very closely with our operations guys (who by the way are absolutely awesome) and now with our beta customers.

While a service is still about software, and the fundamentals still hold, the engineering process, cadence and discipline it requires, I think is quite different from shipping shrink wrapped software. It is easier in some dimensions (like our test matrix is not huge), it is more difficult in others (debugging a large scale distributed system). We had to unlearn quite a few things (like it is better to kill a sick process fast than try to keep it up at all cost) before we could start climbing up the learning curve. It is really quite an experience for us, one that I would not trade for anything else.

If I have to think about this experience as crawl, walk and then run, as Dave Campbell puts it "we are just about getting our knees off the ground so we can start to walk". Yes we have been cautious and yes it is frustrating that the rich capabilities of SQL Server are not accessible to our customers, but I think we are going about it the best way we know how and I am confident we are doing it the right way. Over time, as we learn more about the system we have built, as we roll out more hardware in the datacenters (and find new problems), as we learn what it takes to run a 24x7x365 service (nobody we know of is running a data service using a commercial database system at this scale and cost point) like SSDS, I can assure you we will start to expose capabilities of the underlying engine. How quickly and how much will depend on our ability to provide you, our customers with the quality of service you need to trust your business to SSDS.

Thank you for your patience.

Confusion about Full Text Search in SSDS

Soumitra Sengupta — Sun, 22 Jun 2008 08:48:00 GMT

When I posted the features that we delivered in Sprint 2, I mentioned Full Text Search. This led to this blog statement "In the meantime, I’m waiting for the docs to try full-text search, which was part of Sprint 2. It’s not over until … the docs are done."

I should have mentioned in my post that we rolled out support for Full Text Indexing and Search to the backend of the service. My fault. So here is the clarification. The way things work is we roll out a feature to the backend and then do the work to expose it through the web service which is SSDS. Often this does not get worked on in the subsequent sprint. So from now on when I talk about new features, I will just talk about features that are accessible through SSDS webservice to avoid confusion. Sorry about this.

Cross container queries in SSDS

kellyalt — Wed, 16 Apr 2008 17:27:00 GMT

Eugenio Pace in his 5th article in a series on SSDS gives a pattern for cross container queries. The article can be found here. This is an important scenario since SSDS partitions data into containers to scale out the system. But it actually limits scenarios like cross container queries and transactions. Cross container queries can be loosely grouped into 2 patterns:

Fan out queries: where a query can be run against every container in parallel and the results returned and a union-all performed at the client. This is an important scenario for us and we are looking at how we can make this pattern run efficiently in SSDS.
Queries that have cross container dependencies: join, sort and groupby across containers fall in this category. Map-reduce is one option as it is a technique that is known to scale well when data is partitioned. Again this is an important scenario and you can expect to see us make progress here as well.

Question for you, how far do you think fan out queries get you in your scenario?

Istvan and Nigel talk about SSDS and data services in the Cloud

kellyalt — Tue, 08 Apr 2008 00:04:00 GMT

Ryan Dunn, Technical Evangelist, recently interviewed Istvan Cseri, SSDS Architect and Nigel Ellis, SSDS Development Manager and Architect about SSDS and their vision for Cloud Data Services. He posted his interview at Channel 9. Check it out here. Thanks Ryan for posting this video. Hopefully it will give you the readers a better idea of how these leaders are thinking about data and services in the cloud.

Java code sample for SSDS

kellyalt — Wed, 19 Mar 2008 17:34:00 GMT

Jeff is at it again. This time he has posted some code sample in Java to demonstrate how to issue requests and get response from SSDS. The example he walks through builds SSDS queries and gets the responses back. Check it out.

Jeff Currier has posted some code samples for SSDS

kellyalt — Mon, 17 Mar 2008 20:36:00 GMT

Jeff is one of the developers on the SSDS project. He just sent me a note indicating that he has posted his SSDS code sample here. Very cool.

Good post about SSDS Query Language

kellyalt — Sun, 16 Mar 2008 00:43:00 GMT

Ryan Dunn has a fairly detailed post on the SSDS query model here. Ryan has prototyped some application code already, so his description of the query model is worth a read.