SQL Azure Team Blog : SQL Server

SQL Azure - Same great platform, just a better aligned name

davidrob — Fri, 10 Jul 2009 04:18:00 GMT

Yesterday we announced a branding update to SQL Azure (formerly SQL Services). I think you will agree that our new name is a better fit. If you look across the Windows platform, you will see a consistent branding theme that runs the gamut from Windows Mobile on mobile devices, to Windows Client on the desktop, Windows Server as the leading server operating system and more recently Windows Azure as our soon to be released operating system in the cloud. SQL Data Services is now known as SQL Azure Database. Same SQL Platform, same team, same innovation. SQL Server is the data platform of Windows Server so it made sense to align the name of the relational database platform of Windows Azure, SQL Azure. SQL Azure is built on the SQL Server technology foundation which delivers relational database capabilities in the cloud.

Regardless of what platform your solution targets, your existing SQL Server skills and experience don’t change. That was the goal of the team when we decided to accelerate our relational capabilities and add TDS support. A goal we will be delivering upon when we release SQL Azure later this year.

The next few months are going to be exciting. On the technology front, we are getting ready for our next CTP in August. You can, however, get started today with SQL Express. The team has been working closely with some early adopters and the feedback has been overwhelming positive. Thank you for being so patient, the wait is almost over. Over the next day or two, we will be pushing additional details regarding SQL Azure, including posting our first round of SQL Azure Database documentation to MSDN (once it’s posted, I will post a link to it here).

Stay Tuned,

Dave

SQL Server - Your Data, Any Place, Any Time...and now in the cloud with SQL Azure Database

Dave Campbell talks about extending the Data Platform to the Cloud

Soumitra Sengupta — Tue, 23 Sep 2008 00:08:00 GMT

Here is a little teaser on what we are upto from a direction perspective. Dave Campbell talks about what to expect at PDC 2008 as we extend our Data Platform to the Cloud. As I said before, Dave has been involved with everything SQL since the very beginning and his talk will set the stage for the next evolution of our platform. You can listen to Dave talk to Mike and Jennifer here on Channel 9.

SSDS at PDC 2008

Soumitra Sengupta — Thu, 18 Sep 2008 20:53:00 GMT

It is around 40 days to PDC and the excitement is building. We have a ton of work to do between now and then and the team is heads down getting ready for this major event. If you look at the PDC 2008 breakout sessions already announced, it is incredible that we will have 26 sessions on Cloud Services and 6 sessions related to SQL Server Data Services (SSDS). You can look it up here. You can see Mike and Jennifer's interview with Amitabh Srivastava here, where he talks about some of the things his team has been working on. Unfortunately he cannot talk much about it, but I can assure you coming to PDC this year will be well worth it. In my mind this is as big and maybe bigger than when we announced the .Net wave. The cloud wave is a major event and disruption in our industry and you will get to see and hear from the guys driving this wave inside Microsoft at PDC 2008.

That is as much as I can say about our overall Cloud services initiatives. Specifically, as a member of the SQL Server Data Services (SSDS) team, I am psyched to say that we have scheduled 3 talks directly related to SSDS, 1 talk from our Technical Fellow Dave Campbell on how we are evolving and extending our comprehensive Data Platform and the fast growing SQL business to the cloud and another 3-4 talks on technologies like Astoria, Velocity, Sync Framework and Entity Frameworks that are important to our cloud strategy. So here are the key talks and a brief outline of what you can expect from these talks at PDC.

1. Microsoft SQL Server: Data-Driven Applications from Device to Cloud - Dave Campbell. Dave has been through it all. He will present the key trends that are shaping our industry and how we are evolving our Data Platform from devices, desktop, server to the cloud and the services that makes this platform worth betting on. If you ever wondered what software and services have to do with data, you do not want to miss this talk. Dave's talk will set the stage for all the talks to follow on data and tie them together. So this has to be a must attend talk for you if you care about data.

2. A Lap around SQL Server Data Services - Soumitra Sengupta. This will be an advanced technical presentation where I will explain how we got started on SSDS, how it relates to our Cloud and Data Platform strategy, how we see our Cloud services vis-a-vis our on-premises SQL Server offering. You will also learn why SSDS seems like a departure from hosted SQL Server. I get that one a lot. Tables, Columns, T-SQL anyone. I will fly through this material quickly and spend most of the time coding up an application using our REST and SOAP interfaces. I promise there will be one or two surprises during this presentation. I will also explain how we run our service and the trade-offs and questions that should be on developers' mind as they build applications using SSDS.

3. Under the Hood: Building SQL Server Data Services - Istvan Cseri and Gopal Kakivaya. Man what can I say. These guys are going to take the lid off SSDS and really dive under the hood and talk about how this service is built. If you want to see how to build a fault-tolerant, highly available multi-datacenter service, you cannot miss this talk. It will demystify what we have built and how we operate this. This is the first time we will go this deep. These guys are real rock stars and I will not be upset if you miss my talk BUT DO NOT MISS THIS ONE.

4. SQL Server Data Services: Futures - Patric McElroy. If you are curious after listening to the first 3 talks about where we are headed with SSDS, this is the talk for you. There will be lots of surprises as Patric will show you how we are thinking about extending our entire data platform and services to the cloud. This is a real forward looking session but there will be demos on the way to give you a clear idea of the vision and the path ahead.

In addition we will have lunch sessions, lounges, booth and Hand-on-Labs. You will get to meet, greet, ask questions and socialize with the team members who are making it all happen and if there is enough interest, we will try to get a group together and do something fun during PDC. So if you are interested let, David Robinson know. I know he is in touch with Mike Amundsen already about this.

Aside from these 4 talks, we have the following talks where we will establish the linkages between Sync, ADO.Net Data Services (Astoria), Velocity and other Microsoft services and SSDS. So it would be interesting for you to attend these as well.

1. Microsoft Sync Framework: Enterprise Data in the Cloud and on Devices - Liam Cavanagh
2. Microsoft Sync Framework Advances - Lev Novik
3. Developing Applications Using Data Services - Mike Flasko
4. Entity Framework Futures - Tim Mallalieu
5. Project "Velocity": A First Look - Murali Krishnaprasad
6. A Lap around Building Block Services - John Shewchuk
7. Architecture of the Building Block Services - Dennis Pilarinos, John Shewchuk

Obviously there will be the Keynote from Ray Ozzie to set the stage for this wave and you cannot miss that. I also hear Don Box and Chris Anderson are cooking up something using SSDS. These 2 guys have always lit up PDC and I do not think you will be disappointed this time around.

See you all there.

Trade-offs all around in SQL Server Data Services (SSDS)

Soumitra Sengupta — Tue, 15 Jul 2008 08:41:00 GMT

David Robinson talks about concurrency here. Roger comments about David's take on concurrency here. In SSDS, we make several important trade-offs:

a. We give up some latency because of quorum based commits in favor of aggregate throughput. Theoretically this approach lets you hyper partition the data to get extreme throughput if that is what the app needs. We have to remember SSDS is a big train or a bus and not a finely tuned Ferrari. I believe that is what David was referring to in his article.
b. Trade-off single system image and backward compatibility of existing SQL Server applications for lights out operations, high scale and availability. I know this raises a lot of eyebrows. But giving up backward compatibility allowed us lots of freedom in choosing a data model and optimize for a completely different cost point.

It is important we understand these nuances when dealing with a large scale distributed data service.

Roger Jennings talks about SSDS in a Visual Studio Magazine Article

Soumitra Sengupta — Wed, 02 Jul 2008 10:34:00 GMT

Roger has an interesting article on SSDS in the latest Visual Studio Magazine here. It is really interesting to see the performance data between the SOAP and the REST end points of SSDS. It would be really interesting to dig into this.

One bit of correction - the SSDS flexible entity model was developed independent of EDM / EF and the Astoria model. At this point in time, both teams are working together to get this aligned.

I thought Dave Campbell in one of his press interviews had mentioned it but I could not find the reference. Sorry Roger. So here is an aswer to your question - SSDS is built using SQL Server 2005, SP2 as the starting code base. That is the starting point and we made changes to it. Over time, some of these changes will make its way into the SQL Server mainline and SQL Server 2008 will make its way into SSDS. We wanted to proceed in parallel as fast as we could and this was the best way to do it.

Philosophy behind the design of SSDS and some personal thoughts

Soumitra Sengupta — Fri, 27 Jun 2008 09:21:00 GMT

With Sprint 3 winding down, I thought it is a good time for me to share with you some of the philosophy behind the design of SQL Server Data Services (SSDS) and a few personal thoughts about the experience. When we started this project 2 years back, we realized that we had three fairly difficult problems to solve before we could credibly roll out an internet scale data service. The 3 big problems in order of complexity are:

a. Building a scale free, highly available consistent data service that is fault tolerant and self healing
b. Building the service using low cost commonly available hardware
c. Building a service that was also cheap to operate - lights out operation

Solving problems a and b incidentally makes problem c a bit more complex as you end up with lot more hardware to manage and the hardware tend to fail more often. As a team, we made what I think was a wise decision to use technology already proven to solve these problems. The only area where we had to do a ton of heavy lifting was solving problem a. It allowed us to focus the team's energy on the most difficult problem when it comes to scaling out stateful services. I am not going to go into the details of our approach. If you are interested in hearing about this, I urge you to attend PDC 2008 and hear it from the architects who actually solved this problem.

For b and c we mostly used technologies already available within Microsoft and adapted it for stateful services. Initially we thought this would be fairly easy but it turned out to be more complicated than we thought, especially given the fact that we had to put in infrastructure software that allow us to debug problems with the service without attaching debuggers to the machine or touching the machine. We had to put in logging and tracing infrastructure and given that we all got the logging and tracing religion, in one of our sprints early on we inadvertently dumped so much that we shut down the service effectively as there were no resources left to respond to user requests. Some of our early experiences are fodder for some very interesting hallway conversations. But it taught us that there are quite a few Ph.D. level research topics around debugging large scale distributed systems and if you are up for it and interested in working on them, do give us a holler. Even though we have quite a few Ph.D.'s in the team, we could use some more help :-))

Since we made an early decision to limit the number of hard problems we needed to solve, we decided that we would focus less on the features of the service but more on the quality of the service and the cost of standing up and running the service. The less the service does we argued, the easier it would be for us to achieve our objectives. In hindsight, this was probably one of the best decisions we made. Istvan, Tudor and Nigel deserve special credit for keeping us focussed on "less is better". It also allowed us to learn about the pitfalls of running such a service, including upgrading the service without shutting it down. We did not shy away from complex problems, but we made sure if we could limit the surface area without losing a ton of value, we always took that path. We are still in the learning mode and learning every day about workloads that cause "irregular heartbeats" to our service and how to handle such workloads. But the team has definitely come a long way, working with internal partner teams, working very closely with our operations guys (who by the way are absolutely awesome) and now with our beta customers.

While a service is still about software, and the fundamentals still hold, the engineering process, cadence and discipline it requires, I think is quite different from shipping shrink wrapped software. It is easier in some dimensions (like our test matrix is not huge), it is more difficult in others (debugging a large scale distributed system). We had to unlearn quite a few things (like it is better to kill a sick process fast than try to keep it up at all cost) before we could start climbing up the learning curve. It is really quite an experience for us, one that I would not trade for anything else.

If I have to think about this experience as crawl, walk and then run, as Dave Campbell puts it "we are just about getting our knees off the ground so we can start to walk". Yes we have been cautious and yes it is frustrating that the rich capabilities of SQL Server are not accessible to our customers, but I think we are going about it the best way we know how and I am confident we are doing it the right way. Over time, as we learn more about the system we have built, as we roll out more hardware in the datacenters (and find new problems), as we learn what it takes to run a 24x7x365 service (nobody we know of is running a data service using a commercial database system at this scale and cost point) like SSDS, I can assure you we will start to expose capabilities of the underlying engine. How quickly and how much will depend on our ability to provide you, our customers with the quality of service you need to trust your business to SSDS.

Thank you for your patience.

Confusion about Full Text Search in SSDS

Soumitra Sengupta — Sun, 22 Jun 2008 08:48:00 GMT

When I posted the features that we delivered in Sprint 2, I mentioned Full Text Search. This led to this blog statement "In the meantime, I’m waiting for the docs to try full-text search, which was part of Sprint 2. It’s not over until … the docs are done."

I should have mentioned in my post that we rolled out support for Full Text Indexing and Search to the backend of the service. My fault. So here is the clarification. The way things work is we roll out a feature to the backend and then do the work to expose it through the web service which is SSDS. Often this does not get worked on in the subsequent sprint. So from now on when I talk about new features, I will just talk about features that are accessible through SSDS webservice to avoid confusion. Sorry about this.

Sprint 2 goes live

kellyalt — Mon, 26 May 2008 19:53:00 GMT

Last week our Sprint 2 bits went live. We added several compelling features to our backend service, some of which will show up in the SSDS service in the upcoming Sprints. Key features added include:

a. Full text index and search
b. Multiple B-Trees - this will allow us to deploy schemas as we move toward supporting schemas in SSDS
c. Asynch queries - this will help in exposing fanout queries across multiple containers
d. Logging and tracing improvements to help us manage and debug the service efficiently
e. Backup and restore services
f. Improvements to our Distributed Data Fabric for failover and friendly swapping between primary and secondary partitions

One of the most satisfying part of Sprint 2 deployment was how long it took us to complete deployment. We cut the time taken to deploy and certify deployment by more than half. Kudos to our operations team led by Kyle Patton in achieving this.

Now on to Sprint 3. We are half way through Sprint 3 and the go/no-go is our next big milestone for this sprint. This is when the team will decide which improvements are looking good and will make it to deployment and which improvements we will have to postpone to the next sprint. The long weekend was a good break for all of us as things will get quite hectic as we prepare for TechEd (decks and demos will be locked down and ready to go). If you are at TechEd look us up as we will be presenting at the Developer as well as the IT Pro sessions.

What is going on at SQL Server Data Services

kellyalt — Mon, 19 May 2008 21:20:00 GMT

The SSDS team is taking major steps this week and I thought of sharing the work we have done over the last 2 months with you, our readers. We announced our "restricted beta" at MIX 08 . The response was great and thank you all for your interest in SSDS. I am happy to say that we have been on-boarding registrants over the past 8 weeks and the pace has picked up over the past 3 weeks. You can argue that we are not doing this fast enough and some of us would agree with you. But the team wants to make sure that we provide our beta users with a great experience. It is hard to argue with that. Having said that, we are on our way to clear up the registration backlog. Thank you for your patience. If you have not registered yet but would like to register, please visit us at http://www.microsoft.com/sql/dataservices/default.mspx and click on the Register for Free Beta button and get yourself registered.

We promised to refresh SSDS every 8 to 10 weeks and I am happy to let you know that we are now in the middle of our second refresh since MIX 08. This is a big deal for us. Most of us cut our teeth building great client and server software. We are now learning how to build a great service. We are learning how to deliver new features every 8-10 weeks instead of every 24-30 months. We are learning how to keep our service up and running 24/7. So I am really proud that we landed everything we wanted to in our last sprint. Our operations team will be rolling out all that good work over the next few days. They did a great job rolling out the previous refresh (thanks Kyle, Tracy, Dustin and team) and I am confident that they will do it again. The product team is now in the middle of our next 8 week sprint. Planning is done, coding has begun and you can expect to see the results in another 8 weeks time. I feel great about how fast we have come up to speed on this new rythm of shipping and the amount of value getting added every sprint. You will hear about some of the new features we delivered in the last sprint once we complete our rollout. Stay tuned.

As part of this refresh, we now have the SQL Server Data Service Primer up on MSDN. Thanks to the product team and especially Mohan Vanmane who worked diligently with several people and teams across Microsoft to get this up on MSDN. Now it is up to you our customers and users to let us know how we can improve this primer and provide you with the information and guidance you need to build compelling applications for your customers.

Last but not the least, Nigel Ellis and Niraj Nagrani did a webcast on SSDS recently. Check it out at https://www106.livemeeting.com/cc/mseventsbmo/view?id=1032376634&role=attend&pw=63B495CA. Just enter your name and press return and it will take you to the links for viewing the webcast. It will give you a good introduction to SSDS, the value propositions and how you can take advantage of it. Again, we would like to hear from you what you would like us to cover in future webcasts.

Thanks for your time.

Namashkar,
Soumitra Sengupta
Architect, SSDS

Tudor and Soumitra talk about SSDS value prop with Ryan

kellyalt — Fri, 02 May 2008 00:40:00 GMT

Agility (faster time to market), scale with your need and business ready SLA - Tudor and Soumitra highlight the core value propositions of SSDS on Channel 9. Thanks Ryan for doing the interview.

Cross container queries in SSDS

kellyalt — Wed, 16 Apr 2008 17:27:00 GMT

Eugenio Pace in his 5th article in a series on SSDS gives a pattern for cross container queries. The article can be found here. This is an important scenario since SSDS partitions data into containers to scale out the system. But it actually limits scenarios like cross container queries and transactions. Cross container queries can be loosely grouped into 2 patterns:

Fan out queries: where a query can be run against every container in parallel and the results returned and a union-all performed at the client. This is an important scenario for us and we are looking at how we can make this pattern run efficiently in SSDS.
Queries that have cross container dependencies: join, sort and groupby across containers fall in this category. Map-reduce is one option as it is a technique that is known to scale well when data is partitioned. Again this is an important scenario and you can expect to see us make progress here as well.

Question for you, how far do you think fan out queries get you in your scenario?

Istvan and Nigel talk about SSDS and data services in the Cloud

kellyalt — Tue, 08 Apr 2008 00:04:00 GMT

Ryan Dunn, Technical Evangelist, recently interviewed Istvan Cseri, SSDS Architect and Nigel Ellis, SSDS Development Manager and Architect about SSDS and their vision for Cloud Data Services. He posted his interview at Channel 9. Check it out here. Thanks Ryan for posting this video. Hopefully it will give you the readers a better idea of how these leaders are thinking about data and services in the cloud.

Jeff has posted some SOAP samples

kellyalt — Thu, 03 Apr 2008 17:50:00 GMT

Most of the sample code that Jeff, Jason and David Robinson have posted so far show you how to use the REST interface when coding against SSDS. Jeff has just posted his first SOAP example here. We are interested in finding out how many out there will build against REST and how many will build using SOAP. What are the considerations when you think about using one or the other? How can we make these interfaces better over time? These are some of the questions we have in our mind. Would love to hear from you.

Interesting take from Rick Negris over at AjaxWorld magazine

kellyalt — Fri, 28 Mar 2008 10:35:00 GMT

As Rick points out here, there are very few articles written about SSDS in the technical press. We did not want to make a lot of noise about this service at this time. This is an "invitation only" beta and the service is not generally released. While there has been some coverage, I found Rick's take quite interesting. Give it a read.

Interesting post

kellyalt — Thu, 20 Mar 2008 05:39:00 GMT

Found this through David Robinson's blog. David is a Program Manager in the SSDS team and one of the first guys to write an application using SSDS. I do not know Max but I know what he is talking about. Thanks Max for explaining the value proposition of SSDS in a clear and concise manner.