SQL Azure Team Blog : Software plus Service

SSDS at PDC 2008

Soumitra Sengupta — Thu, 18 Sep 2008 20:53:00 GMT

It is around 40 days to PDC and the excitement is building. We have a ton of work to do between now and then and the team is heads down getting ready for this major event. If you look at the PDC 2008 breakout sessions already announced, it is incredible that we will have 26 sessions on Cloud Services and 6 sessions related to SQL Server Data Services (SSDS). You can look it up here. You can see Mike and Jennifer's interview with Amitabh Srivastava here, where he talks about some of the things his team has been working on. Unfortunately he cannot talk much about it, but I can assure you coming to PDC this year will be well worth it. In my mind this is as big and maybe bigger than when we announced the .Net wave. The cloud wave is a major event and disruption in our industry and you will get to see and hear from the guys driving this wave inside Microsoft at PDC 2008.

That is as much as I can say about our overall Cloud services initiatives. Specifically, as a member of the SQL Server Data Services (SSDS) team, I am psyched to say that we have scheduled 3 talks directly related to SSDS, 1 talk from our Technical Fellow Dave Campbell on how we are evolving and extending our comprehensive Data Platform and the fast growing SQL business to the cloud and another 3-4 talks on technologies like Astoria, Velocity, Sync Framework and Entity Frameworks that are important to our cloud strategy. So here are the key talks and a brief outline of what you can expect from these talks at PDC.

1. Microsoft SQL Server: Data-Driven Applications from Device to Cloud - Dave Campbell. Dave has been through it all. He will present the key trends that are shaping our industry and how we are evolving our Data Platform from devices, desktop, server to the cloud and the services that makes this platform worth betting on. If you ever wondered what software and services have to do with data, you do not want to miss this talk. Dave's talk will set the stage for all the talks to follow on data and tie them together. So this has to be a must attend talk for you if you care about data.

2. A Lap around SQL Server Data Services - Soumitra Sengupta. This will be an advanced technical presentation where I will explain how we got started on SSDS, how it relates to our Cloud and Data Platform strategy, how we see our Cloud services vis-a-vis our on-premises SQL Server offering. You will also learn why SSDS seems like a departure from hosted SQL Server. I get that one a lot. Tables, Columns, T-SQL anyone. I will fly through this material quickly and spend most of the time coding up an application using our REST and SOAP interfaces. I promise there will be one or two surprises during this presentation. I will also explain how we run our service and the trade-offs and questions that should be on developers' mind as they build applications using SSDS.

3. Under the Hood: Building SQL Server Data Services - Istvan Cseri and Gopal Kakivaya. Man what can I say. These guys are going to take the lid off SSDS and really dive under the hood and talk about how this service is built. If you want to see how to build a fault-tolerant, highly available multi-datacenter service, you cannot miss this talk. It will demystify what we have built and how we operate this. This is the first time we will go this deep. These guys are real rock stars and I will not be upset if you miss my talk BUT DO NOT MISS THIS ONE.

4. SQL Server Data Services: Futures - Patric McElroy. If you are curious after listening to the first 3 talks about where we are headed with SSDS, this is the talk for you. There will be lots of surprises as Patric will show you how we are thinking about extending our entire data platform and services to the cloud. This is a real forward looking session but there will be demos on the way to give you a clear idea of the vision and the path ahead.

In addition we will have lunch sessions, lounges, booth and Hand-on-Labs. You will get to meet, greet, ask questions and socialize with the team members who are making it all happen and if there is enough interest, we will try to get a group together and do something fun during PDC. So if you are interested let, David Robinson know. I know he is in touch with Mike Amundsen already about this.

Aside from these 4 talks, we have the following talks where we will establish the linkages between Sync, ADO.Net Data Services (Astoria), Velocity and other Microsoft services and SSDS. So it would be interesting for you to attend these as well.

1. Microsoft Sync Framework: Enterprise Data in the Cloud and on Devices - Liam Cavanagh
2. Microsoft Sync Framework Advances - Lev Novik
3. Developing Applications Using Data Services - Mike Flasko
4. Entity Framework Futures - Tim Mallalieu
5. Project "Velocity": A First Look - Murali Krishnaprasad
6. A Lap around Building Block Services - John Shewchuk
7. Architecture of the Building Block Services - Dennis Pilarinos, John Shewchuk

Obviously there will be the Keynote from Ray Ozzie to set the stage for this wave and you cannot miss that. I also hear Don Box and Chris Anderson are cooking up something using SSDS. These 2 guys have always lit up PDC and I do not think you will be disappointed this time around.

See you all there.

On going quiet and transparency in the design process

Soumitra Sengupta — Fri, 15 Aug 2008 19:22:00 GMT

Roger has challenged us to be transparent as we design SQL Server Data Services. He has urged us couple of times to follow the process that the ADO.Net Data Services team has followed. First let me assert again that we are all in favor of an open process that gets the community and experts who want to provide guidance involved. Second I have seen the benefits of the open and transparent design process during my XML days and prior to that my open source days. So personally I am committed to being open and transparent.

Having said that, we are currently working under several constraints. SQL Server Data Services is only one part of a very broad and deep services initiative which will only come to full light at the upcoming PDC. While we want to be open and transparent, we want to make sure we do not disclose anything inadvertently. This limits the degrees of freedom we have when we discuss features we want to put in and designs for those features. I am very hopeful that post PDC, we will be progressively be more open about our features and design.

Regarding going quiet between sprints - this is the result of 2 separate issues. One our team was quite small till about March of this year. As the team has grown, we have lot of new people who are upto their eyeballs catching up and getting up the steep unlearning and learning curves of transitioning from software to a services mindset. This leaves us with very little bandwidth especially during sprints to engage in deep discussions. Second, this has been a very busy year starting with MIX, TechEd, Worldwide Partner Conference, TechReady and now PDC and TechEd EMEA for the team. To top it all, summers are usually slow time in Redmond as people tend to take their vacations during this time of the year. I was out of the country for 2 weeks with no access to cell phones or email. I am officially still on vacation. But this is an important topic of discussion in the forums and in the blog and I appreciate your concern. Just wanted to shoot this out to alleviate your concerns and request for some patience. PDC is only 2 months away and I urge that you all attend PDC and get the full scope of what is cooking, meet all the key people, share your concerns and ideas about software and services. This is a big wave and we cannot ride it without you. I promise it will be a fun, exciting and profitable ride for all of us.

Why Astoria alignment is not that trivial

Soumitra Sengupta — Tue, 15 Jul 2008 08:56:00 GMT

I have been meaning to write about this topic for quite some time. But it took me quite a while to understand deeply the key issues in aligning Astoria and SSDS. Thanks to Alan Bush, an architect in our team, I have a much better understanding of the key issues. The tendency for most of us is to focus on the data model and query language differences between Astoria and SSDS. Turns out, while they are important, they are not the most difficult ones to work in. So here is what I have learnt from Alan:

a. Astoria protocol and interaction model deals with only a single consistency domain. Each consistency domain sort of maps to a database. Astoria consistency domain is akin to SSDS container. SSDS data model actually spans multiple consistency domains. It is possible to write a query across containers or even query for container properties. Yes Astoria can be extended to address this and we are working on that with Pablo and the Astoria team.
b. Multi-tenancy is a fundamental concept in SSDS while Astoria has no such concept at this point in time. We have to work through this as well. This has huge implications in the security model for the service.
c. Entities in SSDS do not require schema while default for Astoria is EDM schemas. While on the surface this is an easy problem to solve yet it requires you to think through all the options. If you look at the data model for Google App Engine, you will see how they chose to solve this problem - typed entities and expando entities are 2 separate concepts. It is one way to deal with it but think about updates on a graph where you have both typed and flexible entities, how would the user deal with the semantic differences between the two concepts in the same transaction?

There are host of issues but these are some of the top ones we are grappling with. Rest assured minds lot sharper than mine are working this and if there is an elegant solution, they will find it. I have full faith that by PDC we will have a compelling story to tell around SSDS and Astoria. So stay tuned.

Jeff talks about upcoming ETag support in SSDS

Soumitra Sengupta — Sun, 22 Jun 2008 10:39:00 GMT

Jeff just sent me a note indicating that he has a post on the upcoming ETag support in SSDS. To be clear the bits are not yet released to operations. So we will have to wait a bit longer to get our hands on this feature. You can read all about it here. There are lots more cool things coming once Sprint 3 bits get released. So please stay tuned.

Confusion about Full Text Search in SSDS

Soumitra Sengupta — Sun, 22 Jun 2008 08:48:00 GMT

When I posted the features that we delivered in Sprint 2, I mentioned Full Text Search. This led to this blog statement "In the meantime, I’m waiting for the docs to try full-text search, which was part of Sprint 2. It’s not over until … the docs are done."

I should have mentioned in my post that we rolled out support for Full Text Indexing and Search to the backend of the service. My fault. So here is the clarification. The way things work is we roll out a feature to the backend and then do the work to expose it through the web service which is SSDS. Often this does not get worked on in the subsequent sprint. So from now on when I talk about new features, I will just talk about features that are accessible through SSDS webservice to avoid confusion. Sorry about this.

PDC08 Site goes live

kellyalt — Wed, 28 May 2008 18:02:00 GMT

Our premiere conference for professional developers is on this year. The website to register just went live. This is Ray's first PDC and he will be giving a keynote. Some of the sessions are already posted here. There is a deep dive session on SSDS and several sessions on our cloud services. If you want to know how we built SSDS and where we are going with SSDS, you do not want to miss PDC08. Hope to see you there.

July 1st, 2008: New sessions have been uploaded here. An all up data talk titled "SQL Server: From Devices to the Cloud" and the Velocity talks are some of the data talks that show up.

There will be more sessions announced over the next few months. Are there any particular session that you would like to see? Let us know.

Sprint 2 goes live

kellyalt — Mon, 26 May 2008 19:53:00 GMT

Last week our Sprint 2 bits went live. We added several compelling features to our backend service, some of which will show up in the SSDS service in the upcoming Sprints. Key features added include:

a. Full text index and search
b. Multiple B-Trees - this will allow us to deploy schemas as we move toward supporting schemas in SSDS
c. Asynch queries - this will help in exposing fanout queries across multiple containers
d. Logging and tracing improvements to help us manage and debug the service efficiently
e. Backup and restore services
f. Improvements to our Distributed Data Fabric for failover and friendly swapping between primary and secondary partitions

One of the most satisfying part of Sprint 2 deployment was how long it took us to complete deployment. We cut the time taken to deploy and certify deployment by more than half. Kudos to our operations team led by Kyle Patton in achieving this.

Now on to Sprint 3. We are half way through Sprint 3 and the go/no-go is our next big milestone for this sprint. This is when the team will decide which improvements are looking good and will make it to deployment and which improvements we will have to postpone to the next sprint. The long weekend was a good break for all of us as things will get quite hectic as we prepare for TechEd (decks and demos will be locked down and ready to go). If you are at TechEd look us up as we will be presenting at the Developer as well as the IT Pro sessions.

What is going on at SQL Server Data Services

kellyalt — Mon, 19 May 2008 21:20:00 GMT

The SSDS team is taking major steps this week and I thought of sharing the work we have done over the last 2 months with you, our readers. We announced our "restricted beta" at MIX 08 . The response was great and thank you all for your interest in SSDS. I am happy to say that we have been on-boarding registrants over the past 8 weeks and the pace has picked up over the past 3 weeks. You can argue that we are not doing this fast enough and some of us would agree with you. But the team wants to make sure that we provide our beta users with a great experience. It is hard to argue with that. Having said that, we are on our way to clear up the registration backlog. Thank you for your patience. If you have not registered yet but would like to register, please visit us at http://www.microsoft.com/sql/dataservices/default.mspx and click on the Register for Free Beta button and get yourself registered.

We promised to refresh SSDS every 8 to 10 weeks and I am happy to let you know that we are now in the middle of our second refresh since MIX 08. This is a big deal for us. Most of us cut our teeth building great client and server software. We are now learning how to build a great service. We are learning how to deliver new features every 8-10 weeks instead of every 24-30 months. We are learning how to keep our service up and running 24/7. So I am really proud that we landed everything we wanted to in our last sprint. Our operations team will be rolling out all that good work over the next few days. They did a great job rolling out the previous refresh (thanks Kyle, Tracy, Dustin and team) and I am confident that they will do it again. The product team is now in the middle of our next 8 week sprint. Planning is done, coding has begun and you can expect to see the results in another 8 weeks time. I feel great about how fast we have come up to speed on this new rythm of shipping and the amount of value getting added every sprint. You will hear about some of the new features we delivered in the last sprint once we complete our rollout. Stay tuned.

As part of this refresh, we now have the SQL Server Data Service Primer up on MSDN. Thanks to the product team and especially Mohan Vanmane who worked diligently with several people and teams across Microsoft to get this up on MSDN. Now it is up to you our customers and users to let us know how we can improve this primer and provide you with the information and guidance you need to build compelling applications for your customers.

Last but not the least, Nigel Ellis and Niraj Nagrani did a webcast on SSDS recently. Check it out at https://www106.livemeeting.com/cc/mseventsbmo/view?id=1032376634&role=attend&pw=63B495CA. Just enter your name and press return and it will take you to the links for viewing the webcast. It will give you a good introduction to SSDS, the value propositions and how you can take advantage of it. Again, we would like to hear from you what you would like us to cover in future webcasts.

Thanks for your time.

Namashkar,
Soumitra Sengupta
Architect, SSDS

Istvan and Nigel talk about SSDS and data services in the Cloud

kellyalt — Tue, 08 Apr 2008 00:04:00 GMT

Ryan Dunn, Technical Evangelist, recently interviewed Istvan Cseri, SSDS Architect and Nigel Ellis, SSDS Development Manager and Architect about SSDS and their vision for Cloud Data Services. He posted his interview at Channel 9. Check it out here. Thanks Ryan for posting this video. Hopefully it will give you the readers a better idea of how these leaders are thinking about data and services in the cloud.

Invitation to Startups using Open Source to SSDS Design Review

kellyalt — Tue, 01 Apr 2008 01:17:00 GMT

Ryan has sent out an invitation to companies, individuals and startups in the Silicon Valley to attend a Strategic Design Review (SDR) for SSDS through his blog. Since this invitation is going out to people who are not our traditional customers and have not attended a Microsoft SDR before, I thought we should point out that the product / service team shares the current offering and their plans in detail to get feedback from the audience at these events. These reviews are fairly interactive with the product team presenting, holding hands on lab and panel discussions around all the aspects of building a great service. To make it easier for technical and business decision makers from startups to attend, the SSDS team will be conducting these reviews without any NDA. We are really interested in hearing from people who do not use any Microsoft technologies today. Seats are limited so if you are interested, please get in touch with Ryan as soon as you can. The event will be held at our Silicon Valley campus. Hope to see you there.

Interacting with SSDS using Excel

kellyalt — Sat, 22 Mar 2008 19:36:00 GMT

David Robinson describes here how to interact with data stored in SSDS using Excel as the client. Cool. He is off adding Synch to this. That would be real cool.

Answering a few questions from Jeff's blog

kellyalt — Sat, 22 Mar 2008 19:00:00 GMT

Jeff Currier asked the community to send their questions about SSDS. He got quite a few questions. He answered quite a few. I am adding some more clarifications to his answers.

From Senkwe:
"I'm curious to know if you're going to have a service similar to Amazon S3. For the application I have in mind, "eventual consistency" of data would be a huge drawback."

There are 2 questions here:
1. Are we going to support large or small unstructured data (blobs) like S3?
The answer to this is yes. It is in our plans. How soon we will have it, we do not know yet.

2. Is "eventual consistency" the semantics of transactions in SSDS as well?
As Jeff said "We do not use a eventual consistency model. The inserts you do are visible to your on your next query." I should also stress that we do keep multiple copies but we use algorithms that guarantee consistency across the copies.

From JamieT:

Can you bill individual users for their use of the authority rather than billing me, the ultimate owner of the service?
This is the scenario we call "bill on behalf of" a customer. You are calling it the "revenue share" model. Frankly we were not thinking of doing this. But like you, we have heard from a few customers that they would like to:
a. Have their own billing plan but have us service it
b. Bill their customers on their behalf, charge them for overall usage and send the difference to them
As Jeff says, the billing plan is not nailed down right now. It is a scenario we are including in our planning. We will let you know when we have something concrete.

From rogerj:

1. What are your plans and timing for supporting associations/relationships plus lazy- and eager-loading?
Answer to the first question is that we will support the full EDM/EF/ADO.Net V3 model in due time. It is a bit of work. We do not have exact timing for it. Once we do that we will have to address lazy and eager loading, right?

2. When will folks who signed up for the beta know if they've been accepted?
First batch should go out next week. We will keep provisioning on a weekly basis from there on out. By the way once the first batch gets provisioned, everyone else should be able to get to the technical documentation.

3. What's the schedule for an Atom/AtomPub implementation?
We are on this one. Do not have the exact schedule to share at this time.

Keep the questions coming and we will try to answer them to the best of our abilities.

Namashkar,
Soumitra Sengupta

Interesting post

kellyalt — Thu, 20 Mar 2008 05:39:00 GMT

Found this through David Robinson's blog. David is a Program Manager in the SSDS team and one of the first guys to write an application using SSDS. I do not know Max but I know what he is talking about. Thanks Max for explaining the value proposition of SSDS in a clear and concise manner.

Eugenio Pace discussses Litware HR Architecture

kellyalt — Sun, 16 Mar 2008 07:21:00 GMT

If you saw Nigel Ellis' "Introduction to SQL Server Data Services" presentation at MIX08, you will remember the Litware HR application that Nigel demonstrated. This application is built using the SSDS service by Eugenio Pace and members of his team. Eugenio discusses the architecture in his blog here and how he built this multi-tenant application here. He talks about the pros and cons of using SSDS. Good read. He promises to go deeper into how his team built the application and the challenges they faced. Look forward to hearing from you Eugenio. This feedback is very important to us as we build out SSDS. Thanks for the help.

When will we release the Technical Documentation?

kellyalt — Sun, 16 Mar 2008 00:46:00 GMT

I talked to our lead about this and we are working on releasing the technical documentation soon. Keep you eye on this blog as I will post it as soon as the documentation goes live.