SQL Azure Team Blog : Cloud

Updated CTP for SQL Azure Database includes complete feature set for PDC 2009!

davidrob — Wed, 14 Oct 2009 16:58:47 GMT

A few short weeks ago we announced the SQL Azure Database August CTP. Since the announcement, tens of thousands in the community have signed up for the service and have provided us with a tremendous amount of positive feedback. Today, the SQL Azure Database October CTP (CTP 2) update is another major milestone as we get ready for the Professional Developers Conference (PDC) on November 17th - 20th, 2009. We are also excited to announce that this CTP represents the complete feature set that will be available in the SQL Azure Database at PDC.

“We see SQL Azure as the perfect fit for information that needs to be kept in one safe, well-structured, accessible spot. Partitioning the data helps ensure scalability as more and more users interact with the site, and it also isolates tracking resources, helping avoid single points of failure,” said Luigi Rosso, Chief Technology Officer at leading interactive firm, Archetype. “To establish the kind of default redundant installation we get with SQL Azure, we’d have to purchase at least two dedicated servers and hire IT staff to administer them. A reliable system in place that is managed, efficient, and fast is critical for us. With SQL Azure, we don’t have to buy servers or manage and monitor them for capacity to ensure availability for our customers.”

The October CTP has been deployed to one of our go-live production clusters. This production cluster is significantly larger and more powerful than the machine cluster that is supporting the August CTP but is a completely separate machine cluster serviced by a dedicated developer portal (https://sql.azure.com). Accounts for all existing users of the current CTP (August CTP) have been automatically provisioned for access to the new October CTP and environment. Simply go to the developer portal (https://sql.azure.com) to activate your account and create servers on the new environment. Servers you create on this new environment will be reachable through a new address(<servername>.database.windows.net – we’ve dropped the ‘ctp’ moniker from the base address name).

When SQL Azure Database becomes generally available, this environment will automatically roll over into a fully supported production environment and all your databases and data in this environment will be converted into an active subscription to the SQL Azure Database service based on the subscription offer you choose.

Note: Existing servers and databases that you created in the August CTP environment will still be fully accessible and available through the current service URL (<servername>.ctp.database.windows.net), in parallel with the October CTP environment. The developer portal for the August CTP environment will also continue to be available, although through a new URL (https://ctpportal.database.windows.net ). As noted above, the previous URL will be re-used to support the new production environment.

As the August CTP is running on hardware below our production standard, that environment will not roll over into a production environment as part of our go-live plans. That environment will be decommissioned by the end of the year (Dec ’09).

We invite you to begin to use the new cluster to take advantage of all the cool new relational features available as part of the updated service!

The key new features included in this October CTP are listed below.

Firewall Support – The new firewall features allows a customer to specify an allow list of IP address that can access their SQL Azure Server. Security is a concern for companies looking at storing data in the cloud and with this new feature you can rest assured that only hosts you specify will be allowed to connect. Please be aware that your firewall will deny all connections by default, so please go to the SQL Azure Portal and configure your allow list so that existing clients can continue to connect.
Support for Bulk Insert – One of the pain points we heard from customers was around the speed at which they were able to load data into the system. We have taken this feedback and have enabled support for Bulk Insert. This will improve the rate at which you are able to load data into the system by a few orders of magnitude. This change also enables you to use the SqlBulkCopy class from within ADO.Net
Database Edition Selection – This allows you to select which SQL Azure Database edition (Web Edition (up to 1GB relational database) or Business Edition (up to 10GB relational database)) is created during the database provisioning process. This is surfaced both in the SQL Azure Portal and in the T-SQL Create Database statement. For example, to create a Business Edition database the T-SQL command would be as follows:

CREATE DATABASE foo (MAXSIZE = 10GB)

Updates to SQL Azure Portal – We have made numerous functionality changes and bug fixes in the portal. These include database edition selection when creating a database in the portal, viewing the size of your database and also the ability to configure your firewall settings.
Additional T-SQL Support – Due to customer demand, we have enabled support for additional T-SQL statements. The complete list can be found on MSDN and it includes support for items like Synonyms, Types, Table Value Parameters, additional systems views and more.
Updated SQL Azure Server Properties – We have finalized the internal SQL Server engine properties so you can reliably tell if you are connecting to an instance of SQL Server on premises or connecting to SQL Azure in the cloud. The major changes include a new value for Edition of “SQL Azure” and a new Engine Edition value of “5”. For further details please refer to the SQL Azure documentation on MSDN.
Updated SQL Azure Error Messages – We have enhanced the error messages returned should an error occur. Our goal to ensure that should you experience an error, the message received is meaningful and actionable by you.

Consuming data stored in SQL Azure Database is also critical, so as a reminder, the updated SQL Server Driver for PHP 1.1 with support for SQL Azure Database was recently released and can be found here. In addition to SQL Azure Database support, this release includes significant enhancements for PHP development including support for PHP version 5.3, performance improvements, and new features such as scrollable results sets, row count, and support for UTF-8 encoding. If you would like to find out more, you can check out the documentation on MSDN. Also, you should download the refreshed Windows Azure Platform Training Kit that reflects the changes in this October CTP.

A good portion of the new features and enhancements for this CTP came to us as requests from the user community. We listen to and appreciate your feedback as it helps us to build a better platform that will provide you critical capabilities for your database workloads. Please keep the feedback coming.

When we started this journey our goal was to extend the SQL Server Platform to the cloud and to provide our customers with relational database features and availability with cost efficiencies. We are on track to deliver that goal at PDC. There is much more to come as we continue to light up new scenarios and experiences. There will be more on this at PDC ‘09 and the SQL Azure team hopes to see everyone there.

So… What do you think?

davidrob — Mon, 24 Aug 2009 05:40:42 GMT

One of the most important pieces of information a product team can get is an understanding of what people think about their product, and how they are using it. My question for you is…What do you think? How has your experience been thus far? We would love to know – the good and the bad.

So please feel free to either post a comment or drop me an email (david.robinsonATmicrosoft.com).

Also, don’t forget the SQL Azure documentation can be found here and the SQL Azure Getting Started forum is located here.

Dave

Dave Campbell talks about extending the Data Platform to the Cloud

Soumitra Sengupta — Tue, 23 Sep 2008 00:08:00 GMT

Here is a little teaser on what we are upto from a direction perspective. Dave Campbell talks about what to expect at PDC 2008 as we extend our Data Platform to the Cloud. As I said before, Dave has been involved with everything SQL since the very beginning and his talk will set the stage for the next evolution of our platform. You can listen to Dave talk to Mike and Jennifer here on Channel 9.

Jon Udell cURL's with SQL Server Data Services

Soumitra Sengupta — Thu, 28 Aug 2008 03:31:00 GMT

Just read Jon Udell's post on SQL Server Data Services (SSDS) here. Happy to hear that Jon was able to explore SSDS with cURL "within minutes of cracking open the SSDS documentation". Readers who are wondering what Jon is talking about, should read Jeff Currier, one of the guys writing real code that we ship in SSDS, talking about using cURL to explore SSDS here.

When few of us crazies started thinking of SSDS, a non-negotiable requirement was that developers should be able to start exploring and using the service quickly. There is so much going on in the cloud services space that if it took too long to experiment with a new service, we were afraid that developers will put it on the shelve and may never give it a spin. I know I do not like to spend months on something to see if I would use it. It was hard for us to "cover up" the SQL Server heritage of SSDS, but we argued that simplicity is a virtue that we should aspire to.

Jon hits on an interesting point when he says 'while the novelty of “just coding to a URL” on a Microsoft platform will undoubtedly attract some tirekickers who otherwise wouldn’t show up, the real draw will be the ability to exercise choice along the whole continuum'. Internally we use the words "reach" and "rich". We believe that to be a platform with large adoption, one needs to provide both "reach" (so that people from all other platforms can use you) as well as "richness" (so that people who use your platform exlusively have a rich set of tools to leverage). Today it is easy to see that SSDS is aiming to "reach" as many developers as possible. To do this we have to address "the continuum of access styles". Jon seems to be saying that we are meeting that goal. What do you think? Are we close? Are we way off base? Or we should not worry about the "continuum of access styles".

It will be interesting to hear from you all.

On going quiet and transparency in the design process

Soumitra Sengupta — Fri, 15 Aug 2008 19:22:00 GMT

Roger has challenged us to be transparent as we design SQL Server Data Services. He has urged us couple of times to follow the process that the ADO.Net Data Services team has followed. First let me assert again that we are all in favor of an open process that gets the community and experts who want to provide guidance involved. Second I have seen the benefits of the open and transparent design process during my XML days and prior to that my open source days. So personally I am committed to being open and transparent.

Having said that, we are currently working under several constraints. SQL Server Data Services is only one part of a very broad and deep services initiative which will only come to full light at the upcoming PDC. While we want to be open and transparent, we want to make sure we do not disclose anything inadvertently. This limits the degrees of freedom we have when we discuss features we want to put in and designs for those features. I am very hopeful that post PDC, we will be progressively be more open about our features and design.

Regarding going quiet between sprints - this is the result of 2 separate issues. One our team was quite small till about March of this year. As the team has grown, we have lot of new people who are upto their eyeballs catching up and getting up the steep unlearning and learning curves of transitioning from software to a services mindset. This leaves us with very little bandwidth especially during sprints to engage in deep discussions. Second, this has been a very busy year starting with MIX, TechEd, Worldwide Partner Conference, TechReady and now PDC and TechEd EMEA for the team. To top it all, summers are usually slow time in Redmond as people tend to take their vacations during this time of the year. I was out of the country for 2 weeks with no access to cell phones or email. I am officially still on vacation. But this is an important topic of discussion in the forums and in the blog and I appreciate your concern. Just wanted to shoot this out to alleviate your concerns and request for some patience. PDC is only 2 months away and I urge that you all attend PDC and get the full scope of what is cooking, meet all the key people, share your concerns and ideas about software and services. This is a big wave and we cannot ride it without you. I promise it will be a fun, exciting and profitable ride for all of us.

SQL Server Data Services Sprint 3 now Live

Soumitra Sengupta — Fri, 25 Jul 2008 21:18:00 GMT

We completed the rollout of Sprint 3 yesterday. The team is already half way into Sprint 4. In Sprint 4 we are focussed on things we need for PDC 2008, which by the way is in late October in Los Angeles. Ray Ozzie and Steve Ballmer foreshadowed some of what this PDC will be all about here and here. You can also go here to see the already announced sessions. SSDS will be a key part of this year's PDC. We set our technical and product agendas that developers care about at PDC. If you want to understand and learn what Services is all about in our Software plus Services strategy, do not miss this year's PDC.

Overall, this was an interesting deployment as we had new environments come online that had to be configured. The team worked really really hard to make the deployment as smooth as possible. Interestingly we did not hit any issues during deployment yesterday. Wish we could video tape how we do this and share with our readers. Something I have to take up with management. I think it will give everyone an appreciation of the amount of planning, tooling, automation and discipline that goes into these deployments.

Anyway as developers, you care about what new things we are enabling. Things have been leaked on this before by Roger, David Robinson and Jeff Currier. But here are the new features:

Blob Support via the REST interface

In this release, Blobs are limited to 100 megabytes in size.
Blob Support via the SOAP interface will be delivered in a later update.

Version metadata property has changed from a large integer to a container-wide dbtimestamp value:

Version numbers are now based on a dbtimestamp and will increase monotonically on a container-wide basis. No two entities in the same container will ever receive the same version value. The dbtimestamp increases monotonically, however not necessarily sequentially. Clients should no longer assume entity version numbers are sequentially assigned. It does not start at 0, but may be any non-zero integer value for a newly created entity.

ETag support via REST, with similar functionality provided on the SOAP interface

Support for If-Match and If-None-Match via the REST interface. This will allow you to verify that you have the latest version of an Entity, enabling you to use resources more efficiently.
Support for “Accept” header via the REST interface. This will allow you to denote what content you want returned from the service. For instance when retrieving a Blob, the “Accept” header will allow you to choose between the Blob content, or the metadata properties associated with it.
Support for ETag semantics on the SOAP service via the new Version Match object on the Scope object.

HEAD support via REST interface

Support for the “HEAD” verb via the REST interface. This provides a lightweight way of validating existing entity version information without requiring entire entities (or blobs) to be retransmitted to the client.

A new content type – “application/x-ssds+xml” – has been added to the service

This content type is for storing XML content in the service.
Existing “application/xml” content type is being phased out in a future sprint.
The "application/x-ssds+xml" content type should be used for all entities except those containing blob content. Blob entities should instead use a content type value which best reflects the type of blob data stored. It's important to note that the value chosen here will be used later when attempting to retrieve the different parts of an entity by using the "Accept" header.

In addition we did some infrastructure work to enable metering (we will be charging for this service some day :-)) ), throttling (DoS attacks happen and the service needs to work in degraded situation), quota enforcement etc. that developers will not see much of at this time. But once we open our management, debugging and monitoring interfaces, you will be able to plug in and keep up with what is going on with your service.

The primer is updated at our Dev Center here to help you take advantage of the new features. There is one other new "thing" that we are enabling in this rollout that we will talk about a bit later. It is not huge, but it will demonstrate that we are thinking about our building block services very very broadly and deeply. Stay tuned.

Roger Jennings talks about SSDS in a Visual Studio Magazine Article

Soumitra Sengupta — Wed, 02 Jul 2008 10:34:00 GMT

Roger has an interesting article on SSDS in the latest Visual Studio Magazine here. It is really interesting to see the performance data between the SOAP and the REST end points of SSDS. It would be really interesting to dig into this.

One bit of correction - the SSDS flexible entity model was developed independent of EDM / EF and the Astoria model. At this point in time, both teams are working together to get this aligned.

I thought Dave Campbell in one of his press interviews had mentioned it but I could not find the reference. Sorry Roger. So here is an aswer to your question - SSDS is built using SQL Server 2005, SP2 as the starting code base. That is the starting point and we made changes to it. Over time, some of these changes will make its way into the SQL Server mainline and SQL Server 2008 will make its way into SSDS. We wanted to proceed in parallel as fast as we could and this was the best way to do it.

Philosophy behind the design of SSDS and some personal thoughts

Soumitra Sengupta — Fri, 27 Jun 2008 09:21:00 GMT

With Sprint 3 winding down, I thought it is a good time for me to share with you some of the philosophy behind the design of SQL Server Data Services (SSDS) and a few personal thoughts about the experience. When we started this project 2 years back, we realized that we had three fairly difficult problems to solve before we could credibly roll out an internet scale data service. The 3 big problems in order of complexity are:

a. Building a scale free, highly available consistent data service that is fault tolerant and self healing
b. Building the service using low cost commonly available hardware
c. Building a service that was also cheap to operate - lights out operation

Solving problems a and b incidentally makes problem c a bit more complex as you end up with lot more hardware to manage and the hardware tend to fail more often. As a team, we made what I think was a wise decision to use technology already proven to solve these problems. The only area where we had to do a ton of heavy lifting was solving problem a. It allowed us to focus the team's energy on the most difficult problem when it comes to scaling out stateful services. I am not going to go into the details of our approach. If you are interested in hearing about this, I urge you to attend PDC 2008 and hear it from the architects who actually solved this problem.

For b and c we mostly used technologies already available within Microsoft and adapted it for stateful services. Initially we thought this would be fairly easy but it turned out to be more complicated than we thought, especially given the fact that we had to put in infrastructure software that allow us to debug problems with the service without attaching debuggers to the machine or touching the machine. We had to put in logging and tracing infrastructure and given that we all got the logging and tracing religion, in one of our sprints early on we inadvertently dumped so much that we shut down the service effectively as there were no resources left to respond to user requests. Some of our early experiences are fodder for some very interesting hallway conversations. But it taught us that there are quite a few Ph.D. level research topics around debugging large scale distributed systems and if you are up for it and interested in working on them, do give us a holler. Even though we have quite a few Ph.D.'s in the team, we could use some more help :-))

Since we made an early decision to limit the number of hard problems we needed to solve, we decided that we would focus less on the features of the service but more on the quality of the service and the cost of standing up and running the service. The less the service does we argued, the easier it would be for us to achieve our objectives. In hindsight, this was probably one of the best decisions we made. Istvan, Tudor and Nigel deserve special credit for keeping us focussed on "less is better". It also allowed us to learn about the pitfalls of running such a service, including upgrading the service without shutting it down. We did not shy away from complex problems, but we made sure if we could limit the surface area without losing a ton of value, we always took that path. We are still in the learning mode and learning every day about workloads that cause "irregular heartbeats" to our service and how to handle such workloads. But the team has definitely come a long way, working with internal partner teams, working very closely with our operations guys (who by the way are absolutely awesome) and now with our beta customers.

While a service is still about software, and the fundamentals still hold, the engineering process, cadence and discipline it requires, I think is quite different from shipping shrink wrapped software. It is easier in some dimensions (like our test matrix is not huge), it is more difficult in others (debugging a large scale distributed system). We had to unlearn quite a few things (like it is better to kill a sick process fast than try to keep it up at all cost) before we could start climbing up the learning curve. It is really quite an experience for us, one that I would not trade for anything else.

If I have to think about this experience as crawl, walk and then run, as Dave Campbell puts it "we are just about getting our knees off the ground so we can start to walk". Yes we have been cautious and yes it is frustrating that the rich capabilities of SQL Server are not accessible to our customers, but I think we are going about it the best way we know how and I am confident we are doing it the right way. Over time, as we learn more about the system we have built, as we roll out more hardware in the datacenters (and find new problems), as we learn what it takes to run a 24x7x365 service (nobody we know of is running a data service using a commercial database system at this scale and cost point) like SSDS, I can assure you we will start to expose capabilities of the underlying engine. How quickly and how much will depend on our ability to provide you, our customers with the quality of service you need to trust your business to SSDS.

Thank you for your patience.

PDC08 Site goes live

kellyalt — Wed, 28 May 2008 18:02:00 GMT

Our premiere conference for professional developers is on this year. The website to register just went live. This is Ray's first PDC and he will be giving a keynote. Some of the sessions are already posted here. There is a deep dive session on SSDS and several sessions on our cloud services. If you want to know how we built SSDS and where we are going with SSDS, you do not want to miss PDC08. Hope to see you there.

July 1st, 2008: New sessions have been uploaded here. An all up data talk titled "SQL Server: From Devices to the Cloud" and the Velocity talks are some of the data talks that show up.

There will be more sessions announced over the next few months. Are there any particular session that you would like to see? Let us know.

Sprint 2 goes live

kellyalt — Mon, 26 May 2008 19:53:00 GMT

Last week our Sprint 2 bits went live. We added several compelling features to our backend service, some of which will show up in the SSDS service in the upcoming Sprints. Key features added include:

a. Full text index and search
b. Multiple B-Trees - this will allow us to deploy schemas as we move toward supporting schemas in SSDS
c. Asynch queries - this will help in exposing fanout queries across multiple containers
d. Logging and tracing improvements to help us manage and debug the service efficiently
e. Backup and restore services
f. Improvements to our Distributed Data Fabric for failover and friendly swapping between primary and secondary partitions

One of the most satisfying part of Sprint 2 deployment was how long it took us to complete deployment. We cut the time taken to deploy and certify deployment by more than half. Kudos to our operations team led by Kyle Patton in achieving this.

Now on to Sprint 3. We are half way through Sprint 3 and the go/no-go is our next big milestone for this sprint. This is when the team will decide which improvements are looking good and will make it to deployment and which improvements we will have to postpone to the next sprint. The long weekend was a good break for all of us as things will get quite hectic as we prepare for TechEd (decks and demos will be locked down and ready to go). If you are at TechEd look us up as we will be presenting at the Developer as well as the IT Pro sessions.

What is going on at SQL Server Data Services

kellyalt — Mon, 19 May 2008 21:20:00 GMT

The SSDS team is taking major steps this week and I thought of sharing the work we have done over the last 2 months with you, our readers. We announced our "restricted beta" at MIX 08 . The response was great and thank you all for your interest in SSDS. I am happy to say that we have been on-boarding registrants over the past 8 weeks and the pace has picked up over the past 3 weeks. You can argue that we are not doing this fast enough and some of us would agree with you. But the team wants to make sure that we provide our beta users with a great experience. It is hard to argue with that. Having said that, we are on our way to clear up the registration backlog. Thank you for your patience. If you have not registered yet but would like to register, please visit us at http://www.microsoft.com/sql/dataservices/default.mspx and click on the Register for Free Beta button and get yourself registered.

We promised to refresh SSDS every 8 to 10 weeks and I am happy to let you know that we are now in the middle of our second refresh since MIX 08. This is a big deal for us. Most of us cut our teeth building great client and server software. We are now learning how to build a great service. We are learning how to deliver new features every 8-10 weeks instead of every 24-30 months. We are learning how to keep our service up and running 24/7. So I am really proud that we landed everything we wanted to in our last sprint. Our operations team will be rolling out all that good work over the next few days. They did a great job rolling out the previous refresh (thanks Kyle, Tracy, Dustin and team) and I am confident that they will do it again. The product team is now in the middle of our next 8 week sprint. Planning is done, coding has begun and you can expect to see the results in another 8 weeks time. I feel great about how fast we have come up to speed on this new rythm of shipping and the amount of value getting added every sprint. You will hear about some of the new features we delivered in the last sprint once we complete our rollout. Stay tuned.

As part of this refresh, we now have the SQL Server Data Service Primer up on MSDN. Thanks to the product team and especially Mohan Vanmane who worked diligently with several people and teams across Microsoft to get this up on MSDN. Now it is up to you our customers and users to let us know how we can improve this primer and provide you with the information and guidance you need to build compelling applications for your customers.

Last but not the least, Nigel Ellis and Niraj Nagrani did a webcast on SSDS recently. Check it out at https://www106.livemeeting.com/cc/mseventsbmo/view?id=1032376634&role=attend&pw=63B495CA. Just enter your name and press return and it will take you to the links for viewing the webcast. It will give you a good introduction to SSDS, the value propositions and how you can take advantage of it. Again, we would like to hear from you what you would like us to cover in future webcasts.

Thanks for your time.

Namashkar,
Soumitra Sengupta
Architect, SSDS

Tudor and Soumitra talk about SSDS value prop with Ryan

kellyalt — Fri, 02 May 2008 00:40:00 GMT

Agility (faster time to market), scale with your need and business ready SLA - Tudor and Soumitra highlight the core value propositions of SSDS on Channel 9. Thanks Ryan for doing the interview.

Cross container queries in SSDS

kellyalt — Wed, 16 Apr 2008 17:27:00 GMT

Eugenio Pace in his 5th article in a series on SSDS gives a pattern for cross container queries. The article can be found here. This is an important scenario since SSDS partitions data into containers to scale out the system. But it actually limits scenarios like cross container queries and transactions. Cross container queries can be loosely grouped into 2 patterns:

Fan out queries: where a query can be run against every container in parallel and the results returned and a union-all performed at the client. This is an important scenario for us and we are looking at how we can make this pattern run efficiently in SSDS.
Queries that have cross container dependencies: join, sort and groupby across containers fall in this category. Map-reduce is one option as it is a technique that is known to scale well when data is partitioned. Again this is an important scenario and you can expect to see us make progress here as well.

Question for you, how far do you think fan out queries get you in your scenario?

Istvan and Nigel talk about SSDS and data services in the Cloud

kellyalt — Tue, 08 Apr 2008 00:04:00 GMT

Ryan Dunn, Technical Evangelist, recently interviewed Istvan Cseri, SSDS Architect and Nigel Ellis, SSDS Development Manager and Architect about SSDS and their vision for Cloud Data Services. He posted his interview at Channel 9. Check it out here. Thanks Ryan for posting this video. Hopefully it will give you the readers a better idea of how these leaders are thinking about data and services in the cloud.

Dave Robinson at CodeTrip

kellyalt — Wed, 02 Apr 2008 10:04:00 GMT

Here is the link to some quick code Dave wrote up in 20 mins under pressure to demonstrate how easy it is to provision a SSDS Authority and Container and start inserting SSDS Flex Entities into them. I hope this demonstrates that it is quite simple to get started with storing your structured data in SSDS and write an application against it.

Hey Dave, I am hoping you had a taste of Chuck's Irish Bomb :-)) That stuff looks pretty evil.