SQL Azure Team Blog : Scaling Out

SQL Azure at PASS Summit 2009

davidrob — Sun, 01 Nov 2009 22:59:43 GMT

This upcoming week, PASS (Professional Association for SQL Server) will be holding its annual summit in Seattle, WA. The summit runs from November 2nd through the 5th and, as described on the website, provides:

In-depth technical sessions all focused on SQL Server
Unparalleled access to the industry’s top SQL Server experts and the Microsoft SQL Server development team
Unique opportunity to network with your peers, share challenges, and get answers and advice
Return to work with new skills and knowledge to do your job better, faster, easier – right away.

This year the SQL Azure team will be presenting the following sessions:

What’s new in SQL Azure - Patric McElroy
Come and learn how SQL Azure has evolved over the past year based on your feedback. In this session you'll learn how SQL Azure delivers on promise of Database as a Service. You'll see how easy it is to take an existing class of SQL Server applications and extend them to the service using existing SQL Server knowledge, protocols, client libraries and tools. With minimal changes, your application will be running in a highly available and scalable service. Finally we’ll touch on the business model, terms of use and present a roadmap for the service.

Building Applications with SQL Azure and Windows Azure – David Robinson and Liam Cavanagh
Are you looking to reduce the costs of building and maintaining enterprise applications? Do you want to extend the reach of your applications across multiple devices, locations and partners? SQL Azure and Windows Azure provides you a friction free, highly scalable platform for building applications. The scale and reach of the cloud lights up a new class of application scenarios. Come see how easy it is to consume SQL Azure from within Windows Azure. In addition, we will dive into Microsoft’s new Data Hub for businesses and see how this SQL Azure powered synchronization service allows for data aggregation within the Hub to provide straight-forward data sharing between on-premises databases, business partners, remote offices and mobile users.

Roles and Responsibilities Managing a Microsoft SQL Azure Database - Nino Bice
Come and learn about database administration in the new world of SQL Azure. Starting from the creation of an SQL Azure account, learn about tools and techniques for provisioning accounts, servers, and databases. See how billing is managed. Learn how to collaborate with your peers to build an Azure project that uses SQL Azure. We demonstrate the mechanisms for database management such as managing logins and permissions. See how SQL Azure automates the tedious and complex administrative tasks (e.g., machine failures) and allows you to spend more time on the important design functions of a DBA.

Exchange Hosted Archive - A True Testament of Scalability

davidrob — Mon, 08 Jun 2009 18:50:00 GMT

Hi everyone, this is Shankar Pal. I am a Principal Program Manager on the SQL Data Services (SDS) team. I spend my time working on the backend design for large, enterprise applications.

I wanted to share with you some of my experiences on the scalability of SQL Data Services and how this is best exemplified by one of our online services, the Microsoft Exchange Hosted Archive (EHA). This is a very rich service for e-mail archive, e-Discovery and regulatory compliance for corporate customers and large organizations. The next generation EHA uses the same relational database service infrastructure as SDS. I will focus on the section of the service pertaining to the scale aspects of the workload, and discuss how the relational database service addresses the scale requirements of EHA.

First, a brief introduction to the characteristics of the workload. Archived messages accumulate in the system and are governed by the retention policies of the customers. The message lifecycle goes from archival of messages in the system, retention based on retention policies (e.g. 3 years) and purging the messages at the end of the retention period. Inserted messages are full-text indexed on the header, subject line, message body and a variety of common business attachments such as Word documents. E-discovery consists of structured and full-text query of the messages. Examples are searches based on various properties such as the send time, the sender, or full-text search of the message body.

EHA looked for a long-term solution in a relational database service which would scale to a much higher archiving limit per customer than the current system, be easy to administer, provide the required availability and keep pace with the rapid growth of the service. The result is the next generation EHA which is powered by the SDS relational database service platform. The service allows the seat limit per customer to become many fold larger; this is achieved by distributing the archived emails from each customer to a large number of servers rather than to a specific server. Performance enhancements are seen during message insertions, as well as in structured and full-text queries across the system. For more information about the backend architecture, you can view a presentation from Gopal Kakivaya, a Distinguished Engineer in the SDS team, from last year’s PDC. That video can be found at http://mschnlnine.vo.llnwd.net/d1/pdc08/WMV-HQ/BB03.wmv.

Email is a very prolific form of communication. The high volume of incoming data is automatically replicated within the database service to provide fault-tolerance against various types of failures. The platform provides high availability whether storing gigabytes, terabytes, or petabytes of data. Each cluster of machines has a capacity to store hundreds of terabytes for email archive. Together with the replication and the backup requirements, the total capacity of the EHA cluster is petabytes of data, a testament to the scalability of the SDS relational database service platform.

The enormous scale is achieved with surprising simple design principles. Mail messages can be partitioned in several ways, the most obvious being by customer or user. Such segments can grow quite large, so for more parallelism, each customer or user’s messages can be partitioned further, most notably by the send time. A variation of this partitioning scheme is used for the EHA application. The partitions for each customer are scattered over many servers. This increases the throughput of the system for message insertion by distributing the write operations over a large number of physical servers. The net result is much higher insertion rate compared to the current EHA system.

Queries benefit from the physical distribution of the data as well by executing on multiple partitions scattered over many server machines. The process of running queries in parallel, sometimes referred to as fan out, and aggregating the responses pays greater rewards the more complex the query and the bigger and more distributed the overall data set. Customers, especially in heavily regulated industries, frequently perform full-text searches using date ranges and other qualifiers. The more structured the query the more relevant the results. Our measurements with real customer messages show that queries with a high degree of fan-out often execute an order of magnitude faster compared to a single instance of a server.

The new generation archive will be available later this summer. It is very exciting to build a system which uses physical distribution of data to meet the scale and performance requirements of a large enterprise application. The self-managing system simplifies a host of administrative functions and makes those more reliable.

Scaling Out with SQL Data Services

davidrob — Wed, 27 May 2009 19:43:00 GMT

Hi Folks,

Here is another video from TechEd. In this one, Rick Negrin and I talk about scaling out your database with SQL Data Services and the performance gains you will see. The video can be seen here.

Enjoy!!!

Dave