Microsoft Sync Framework : SQL Azure

How to Synchronize Multiple Geographically Distributed SQL Server Databases using SQL Azure Data Sync

liamca — Mon, 14 Dec 2009 16:43:00 GMT

Imagine that you want to have multiple copies of the same SQL Server databases located in different locations around the country or even around the world. Those SQL Server databases could exist in your headquarters, subsidiaries, retail stores, and even in your remote offices. Currently, to accomplish this, there are a number of really great technologies to help you use on-premises software that you manage. Some examples of this include Merge Replication and Sync Framework database providers (previously known as Sync Services for ADO.NET). The most common concern that we hear with these technologies is the difficulty of getting the system up and running quickly and the complexity of the management requirements. Quite often it requires working with IT to open holes in the corporate firewall and setting up web servers to host the synchronization logic. What I would like to show you is an alternate way that you can accomplish this, by using a technique that removes the need to configure corporate firewalls or to install and configure web services. With this technique you can take virtually any SQL Server database and share it with other users via Windows Azure.

To get started let’s begin by looking at a very basic example. Imagine a company we will call Fabrikam that has a SQL Server in their New York headquarters. They would really like to make an exact copy of that SQL Server database and make it available in their London subsidiary. Down the road they will also want to put one in their Tokyo office. Fabrikam wants to be able to have the database local to each of these locations to remove any latency issues. Ultimately, there will be users that make changes to both of these databases, so periodic synchronization will need to take place to move changes to and from each of these locations.

One of the ways that this can be accomplished is through the use of SQL Azure and in particular SQL Azure Data Sync. SQL Azure is a fully managed relational database in the cloud. This database is built on SQL Server technologies. Using SQL Azure Data Sync, we can easily solve the first half of Fabrikam's problem, in that we can set up synchronization from their existing New York database and synchronize it to a SQL Azure database. All of this can be done without any specific configuration to the corporate firewall (other than outbound only access on port 1433) and without the need to set up web services. This is because SQL Azure Data Sync sets synchronization up within the SQL Server as a SQL Agent process that periodically pushes changes to and from the SQL Server and SQL Azure databases. Since it makes outbound calls to the public SQL Azure database service there is no need to open holes in the corporate firewall. The first part of the architecture looks like this:

There is a really good walkthrough document on how to set this up here: Microsoft Sync Framework Power Pack for SQL Azure.

The provisioning tool to set up synchronization between SQL Server and SQL Azure does not currently have support to extend this to other SQL Server databases. However, Sync Framework, which powers SQL Azure Data Sync, does have the capability to support a number of databases, so although this scenario can not be configured automatically through the tool, with a little manual effort we can enable it as seen below.

The first things we need to do are to back up the New York SQL Server database and restore it on the London server. Next, since each database that is part of the synchronization ecosystem has unique attributes assigned to it, we will need to “reset” the synchronization state. To do this we will use the SqlSyncStoreRestore (Sync Framework) class to update the synchronization metadata for the remote endpoint by calling the PerformPostRestoreFixup method. After this you will be able to synchronize the London database as well.
In order to try to keep things a little simpler, I have included a simple C# command line application that allows you to do the SqlSyncStoreRestore against the copied database.

To create this application, you will need to build a C# Visual Studio Project using the following 4 steps:

Open Visual Studio and choose: File | New | Project | Visual C# | Console Application.
After the project is created, add a reference to Microsoft.Synchronization.Data.SqlServer. To do this, within the Solution Explorer right click on Properties | Add Reference | Browse, and enter: C:\Program Files\Microsoft Sync Framework\2.0\Runtime\ADO.NET\V3.0\x86\Microsoft.Synchronization.Data.SqlServer.dll
Add the following code in your Program.cs file and update your ConnectionString to point to the database you restored.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Microsoft.Synchronization.Data.SqlServer;

using System.Data.SqlClient;

namespace ConsoleApplication9

{

class Program

{

static void Main(string[] args)

{

SqlConnection serverConn = new SqlConnection();

serverConn.ConnectionString = "Data Source=localhost;Trusted_Connection=True;Database=MYDATABASE";

SqlConnection.ClearPool(serverConn);

SqlSyncStoreRestore databaseRestore = new SqlSyncStoreRestore(serverConn);

databaseRestore.PerformPostRestoreFixup();

}

Compile and run the application.

That should reset the synchronization information in the London SQL Server database and allow it to synchronize to SQL Azure just as the New York database has. By the way, this code is a great way to create template databases that have data pre-provisioned in them so that you do not need to synchronize the initial data set.

To test this new database, you will need to add the same SQL Agent process on the London machine that you have on the New York machine. If it is a separate machine you will also need to install the 32-bit version of the Microsoft Sync Framework 2.0 SDK (even if you are running on a 64-bit machine) as well as Sync Framework Power Pack for SQL Azure. If you are not familiar with SQL Agent, to find the command line that is being used on the New York database, open SQL Server Management Studio, connect it to the New York database, choose SQL Agent |Jobs, and then right click on the job that is of the format “SyncToSQLAzure-XXXXX” and choose properties. Choose Steps | Edit and copy the contents from the text box. It should look something like this:

"C:\Program Files\Microsoft Sync Framework\Power Pack For SQL Azure November CTP\SyncLocalSqlAzureDatabase.exe" -localServer localhost -localDb Archetype -SqlAzureDb MyDB -scope Sync_MyDB -SqlAzureServer XXXXXXX.database.windows.net -SqlAzureUser sa -SqlAzurePassword passwordaf1786d3-jda5-4e55-9368-9kdffb2bdea3 -ConflictResolutionPolicy SqlAzureWins

You can either manually run this command line from the London machine or create a new SQL Agent that executes the same task.

That’s it! At this point you have two SQL Server databases that can communicate with each other through a SQL Azure database. If you need more, you can follow the same process to add additional SQL Server databases.
I’d love to hear your feedback on this!

For more details on how to get started with SQL Azure Data Sync, please visit our developer page.

Liam

PDC Keynote Recording - Kelley Blue Book's use of SQL Azure Data Sync

liamca — Wed, 25 Nov 2009 19:54:00 GMT

Last week at PDC we announced the availability of our first CTP of SQL Azure Data Sync. In the keynote, Andy Lapin of Kelley Blue Book explained how they keep their community databases up-to-date using this technolgy to synchronize data between their on-premises SQL Server and SQL Azure. The recording is now available for you to view (the Kelley Blue Book portion of the keynote starts at about 1:18:00).

Ultimately, one of the goals of this CTP is to get your feedback. What is it that you like or dislike about it? Are you primarily interested in the Visual Studio integration or would you like us to see us add additional capabilities such as advanced monitoring or logging to the SQL Server provisioning tool. So please send us your thoughts (good or bad) here:

http://social.microsoft.com/Forums/en-US/syncfeedback/threads

Liam

Announcing SQL Azure Data Sync (November CTP) Available for Download

liamca — Tue, 17 Nov 2009 16:42:00 GMT

Today in the opening keynote at PDC we announced the availability of SQL Azure Data Sync – November CTP, an early preview open to the public through a demonstration with Kelley Blue Book. For those of you who have been following our blog, you may be asking yourself, what exactly does this include and how does it compare to Project “Huron” that we have been talking about for some time now? In this post I want to give some additional details.

You can think of SQL Azure Data Sync as the first part of our overall Project “Huron” vision which is to create a Data Hub in the Cloud, or more specifically a place for you to easily consolidate and share all of your information. With SQL Azure Data Sync we have worked to simplify the task of sharing information whether that is from on-premises SQL Server to the cloud or from the cloud, down to mobile users, retails stores or remote offices. All of this being powered by the Microsoft Sync Framework.

SQL Azure Data Sync allows developers and DBA's to:

Link existing on-premises data stores to SQL Azure.
Create new applications in Windows Azure without abandoning existing on-premises applications.
Extend on-premises data to remote offices, retail stores and mobile workers via the cloud.
Take Windows Azure and SQL Azure based web application offline to provide an “Outlook like” cached-mode experience.

All of this is accomplished through

An end-user Data Sync Tool that keeps on-premises SQL Server data in sync with SQL Azure.
Visual Studio templates enabling developers to take Windows Azure and SQL Azure based web application offline within SQL Compact and SQL Server databases.

By downloading Microsoft Sync Framework Power Pack for SQL Azure November CTP along with the Microsoft Sync Framework 2.0, developers can use the built-in providers for SQL, allowing them to synchronize SQL Azure with other stores such as SQL Server and SQL Server Compact. SQL Azure Data Sync is optimized for performance over high latency networks by taking advantage of features like SQL Azure Table Valued Parameters to help reduce latency and significantly boost performance. Moving forward Microsoft we will continue to enhance this technology and focus on providing a linkage between current on-premises data sources and Windows Azure Storage as well as utilizing this technology as a basis of a developer technology that will enable the Windows Azure platform to be the platform for creating cached mode web applications.

To get started with SQL Azure Data Sync, you need the following:

· Microsoft Sync Framework 2.0

· Microsoft Sync Framework Power Pack for SQL Azure – November CTP

· SQL Azure Server Account (register here)

After installation, you will find 2 new components to help you get started:

1) SQL Server Provisioning Wizard: This wizard (launched from Start | Programs | Microsoft Sync Framework | SQL Azure Data Sync Tool for SQL Server) walks you through the process of connecting SQL Server to SQL Azure through data synchronization. Within this wizard you select the tables to provision in SQL Azure, it will create a new database and setup background synchronization to keep the two data stores in-sync.

2) Visual Studio 2008 Template: For users that would like to take an existing SQL Azure database offline, this new template simplifies the task of creating an offline data cache within SQL Compact. Using the template wizard (available by right clicking on a Visual Studio Solution Explorer Application and choosing Add | New Item | SQLAzureDataSyncClient, developers can choose the SQL Azure tables that they would like to be made available offline. After completing the wizard, a SQL Compact database will be created and code will be generated that allows the offline to synchronize changes on-demand between SQL Azure and SQL Compact.

- Liam

SQL Pass Conference & Synchronization to SQL Azure

liamca — Wed, 04 Nov 2009 17:34:00 GMT

Yesterday was the first day of the SQL Server conference, SQL Pass, here in Seattle. It was a really exciting day for me because I had the opportunity to present our new synchronization capabilities to the cloud in the opening day keynote and in a subsequent session.

We introduced the first piece of Project “Huron”, that is a technology based on the Sync Framework enabling people to use Windows Azure as a central Data Hub for all information. To accomplish this, later this month we will make available for download an early preview of a tool that provides the ability to extend a SQL Server on-premises databases to the cloud. Once configured users can then easily extend data from the cloud to mobile users and remote offices.

Using this tool and with the help of one of our partners, Archetype, we showed how companies can easily extend their on-premises SQL Servers to SQL Azure using data synchronization, allowing the data stores to co-exist and interoperate seamlessly.

All of this can be setup using a wizard. Then from an automatically generated SQL Agent, the synchronization process is executed periodically to move the incremental changes between SQL Server and SQL Azure.

Here are a few screenshots of the tool:

Watch for more details over this month at the Professional Developer Conference and in this blog for other new capabilities we are providing to for integration to Windows Azure and SQL Azure.

Liam Cavanagh

SharePoint 2010, Windows 7 and PwC Now Integrate Microsoft Sync Framework v2.0

liamca — Mon, 19 Oct 2009 18:17:00 GMT

As we announced earlier today, Microsoft Sync Framework V2.0 is now available for public download.

Along with this release I wanted to outline a number of companies and partners that are already using Sync Framework v2.0 including Windows 7, Microsoft Office 2010 and PricewaterhouseCoopers (PwC).

SharePoint has also chosen to integrate Microsoft Sync Framework as part of their solution for synchronizing to Office SharePoint 2010. The SharePoint and Office teams will use the Sync Framework as the engine powering synchronization between Office Workspace (Groove) and SharePoint. In addition, the SharePoint server will also expose Sync Framework API’s that will allow any developer to build providers to integrate new data stores into SharePoint.

"The Sync Framework will greatly reduce the amount of code that an ISV needs to create, and this represents probably the most painful code that they otherwise would have had to write".
Maxim Lukiyanov, Program Manager SharePoint Server, Microsoft

For more information on this please see our new case study: Microsoft Sync Framework used to Create Microsoft SharePoint Workspace 2010

The Windows 7 Device Stage gives users a common user interface to manage and synchronize data with their Windows 7 enabled phone. Contacts, calendar, tasks and even notes can be synchronized with any data store. Sync Framework v2.0 is the easiest way for developers to implement data synchronization providers for any store to enable it for the Windows 7 Device synchronization experience.

"The flexibility of Sync Framework and its ability to track and exchange data really set this technology apart. Since Sync Framework is data agnostic this allowed us to develop a framework tailored around contact, calendar, tasks and notes data".
Anna Boyd, Windows 7 Engineering, Microsoft

PricewaterhouseCoopers has put a system in deployment for 60,000+ mobile auditors in 150 countries to allow much of their work to be done remotely at the client site, making disconnected, offline use of their application and friction free collaboration across teams.

"Our implementation of Sync Framework ‘just worked’ for our auditors. If they set automatic synchronization, our software finds peers, verifies access rights to the same engagement database, and updates all databases. They might be connected to each other over a client’s network, or just via ad hoc (no access point) WiFi. If just one laptop has access to the servers, that laptop automatically funnels all changes from the laptops to the server and vice versa. No manual configuration is needed".
Dr. Glenn Ricart, Technology and Development Leader, PricewaterhouseCoopers

Liam Cavanagh

Sync Framework and our upcoming conferences

liamca — Mon, 05 Oct 2009 19:31:00 GMT

The next few months are going to be very busy for us here in the Sync Framework team. Not only do we have Sync Framework v2 coming out shortly, we also have a number of conferences where we will talk about the work we have been doing with some of external companies and internal groups here at Microsoft to allow developers to plug-in synchronization to systems like SQL Server and SQL Azure.

Starting in November, we will be at SQL PASS, and then at the PDC for the launch of SQL Azure. Last week I recorded a Channel 9 video along with Buck Woody and Michael Rys on the upcoming SQL PASS Summit. You can view the video here:

http://channel9.msdn.com/posts/LarryLarsen/Upcoming-SQL-PASS-Summit-1/

Here is a summary of the video:

Join thousands of SQL Server & BI professionals at the Washington State Convention & Trade Center in Seattle from Nov. 2-5 for the largest SQL Server conference in the world! PASS Summit offers 168 technical sessions presented by SQL Server experts, one-on-one time with Microsoft’s SQL Server engineering team, free troubleshooting & design architecture guidance from Microsoft consultants, keynotes from Microsoft’s senior executives and much more!

Senior Data Platform Technology Specialist Buck Woody stopped by the Channel 9 studio to talk to Michael Rys, Principal Program Manager in SQL Server Engineering Team, and Liam Cavanagh, Senior Program Manager, about upcoming sessions you won’t want to miss this year’s PASS Summit conference.

Liam Cavanagh

SQL Services is now SQL Azure

liamca — Thu, 09 Jul 2009 01:58:00 GMT

I thought I would help get the word out on this one. Given that we have been spending so much time on Project "Huron", which is a project to enable data synchronization to the cloud where the data store is ~~SQL Data Services~~ SQL Azure Database, I thought you should all be aware of this new branding. As we announced in the Data Insiders Blog earlier today:

"Effective immediately, SQL Services will be called Microsoft SQL Azure, and SQL Data Services will be Microsoft SQL Azure Database".

Also, I appologize that I have not provided much of an update on Project "Huron" lately. We are currently heads down in engineering and working out the specific details of the sync service to be hosted within Azure. I would like to thank all of you who offered to help us with our Early Adopter Program. The feedback we have been receiving from you has been absolutely invaluable.

Liam