Mahjayar's WebLog. : Sync Services for ADO.NET

SQL Azure Data Sync – Moving Schema from Compact Database to SQL Azure

Mahjayar — Thu, 19 Nov 2009 01:30:00 GMT

Update 12/11: Fixing a typo in the code as pointed by the commenter.

With yesterdays announcement of the Microsoft Sync Framework Power Pack for SQL Azure, we have had questions on how to move sync schema from SQL Compact database up to SQL Azure. The power pack only comes with tools to automate the sync schema setup from SQL Server to SQL Azure (via the SQL Azure Data Sync Tool) and taking SQL Azure data offline to a new compact database (via the VS Add New Item template). There is no UI tool to go from Compact to SQL Azure or even SQL Azure to SQL Server.

Just wanted to point out that its quite simple to achieve the above two non UI supported scenarios via code. Attached is the simple code that shows how to move sync schema from Compact to SQL Azure.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.SqlServerCe;
using Microsoft.Synchronization.Data;
using Microsoft.Synchronization.Data.SqlServerCe;
using Microsoft.Synchronization.Data.SqlAzure;
using System.Data.SqlClient;
namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            SqlCeConnection conn = new SqlCeConnection("c:\temp\abc.sdf");
            using (conn)
            {
                conn.Open();
                DbSyncScopeDescription desc = GetDbSyncDescription(conn);

                //Check and sync enable the Compact database
                SqlCeSyncScopeProvisioning ceScope = 
                    new SqlCeSyncScopeProvisioning(desc);
                if (!ceScope.ScopeExists(scopeName, conn))
                {
                    ceScope.Apply(conn);
                }

                SqlConnection azConn = 
                    new SqlConnection("YourSqlAzureConnectionString");
                using (azConn)
                {
                    azConn.Open();
                    //Check and sync enable the SQL Azure database
                    SqlAzureSyncScopeProvisioning azScope = 
                        new SqlAzureSyncScopeProvisioning(desc);
                    if (!azScope.ScopeExists(scopeName, azConn))
                    {
                        azScope.Apply(azConn);
                    }
                }
            }
        }

        static string[] tableNames = new string[] { "Foo", "Bar" };
        static string scopeName = "FooBarScope";
        private static DbSyncScopeDescription GetDbSyncDescription
            (SqlCeConnection conn)
        {
            DbSyncScopeDescription desc = new DbSyncScopeDescription();

            /***
             * Option 1 for generating schema - Iterate through the list 
             * of tables you need and get description for them
             */ 
            foreach (string table in tableNames)
            {

                desc.Tables.Add(
                    SqlCeSyncDescriptionBuilder.GetDescriptionForTable
                    (table, conn));

            }

            /***
             * Option 2 for generating schema - If you already have a 
             * Compact database provisioned for sync you
             * could just read the whole scope description from it.
             **/
            // desc = SqlCeSyncDescriptionBuilder.GetDescriptionForScope(
            // scopeName, conn);

            return desc;
        }
    }
}

To move from SQL Azure to SQL Server you just need to replace SqlCeSyncScopeDescription type with SqlSyncScopeDescription. To access the new types under Microsoft.Synchronization.Data.SqlAzure just add a reference to the dll of the same name found under %programfiles%\Microsoft Sync Framework\Power pack For Sql azure November CTP\

Maheshwar Jayaraman

Announcing Microsoft Sync Framework Power Pack For SQL Azure CTP

Mahjayar — Tue, 17 Nov 2009 19:15:00 GMT

PDC 2009 starts today and Ray Ozzie and Bob Muglia kicked off todays keynote presentation. It gives me great pleasure to announce the public availability of the MSF power pack for SQL Azure. Its a CTP and we are really excited to see people download this and provide feedback around the direction we are heading w.r.t providing a synchronization story for SQL Azure. Head over to http://tinyurl.com/yhn76rr to download the Power pack.

The power pack provides the following components.

SQL Azure Data Sync Tool for SQL Server - This is a automated tool to migrate your on premise SQL server data to SQL Azure by setting up a strong synchronization relationship. Kelly Blue Book demonstrated this in the key note when they moved a table from their on premise database to SQL Azure.

SqlAzureSyncProvider - Power pack also brings in a new dll, named Microsoft.Synchronization.Data.SqlAzure.dll, that contains a brand new provider, SqlAzureProvider, that enables existing database providers to synchronize with SQL Azure. This also has the new SQL Azure specific Sync metadata provisioning API's.

Sql Azure Offline Visual Studio Plug-In - This adds a new item template for VS 2008 SP1 that enables you to setup offline synchronization relationships with SQL Azure.

The team has worked really hard to bring this to you and we would love any feedback you got. You can use the email feature here or send email to syncfdbk at microsoft dot com with your feedback.

We first discussed about Project "Huron" at last year's PDC and with this CTP we have taken the first steps toward the vision of a Data Hub in the sky. We have a more detailed post about this over at our Sync blog. Refer http://blogs.msdn.com/sync/archive/2009/11/17/announcing-sql-azure-data-sync-november-ctp-available-for-download.aspx.

Mark Scurell, our lead PM, will be demonstrating this whole end to end walkthrough in his talk on Thursday 11/19 at 3 PM. For more information refer http://microsoftpdc.com/Sessions/SVC23. He will take you through the process of synchronizing an on-premise database with SQL Azure, generate an offline cache from SQL Azure down to a SQL Server Compact database and move data between the client, SQL Azure and on-premise server all with just one single line of code.

Maheshwar Jayaraman

MSF V2 DeepDive – Batching Directory and Batch Files

Mahjayar — Mon, 16 Nov 2009 22:15:00 GMT

This is a continuation from the previous post I did explaining the new memory based batching feature for Database providers in MSF V2. Please refer to that post at MSF V2 CTP2 Deep Dive – Memory Based Batching. This is probably a good time to mention that the final version of MSF V2 has been released to web and can be downloaded from http://msdn.microsoft.com/en-us/sync/default.aspx.

Database providers chunks up changes by spooling changes to the file system. These chunks, or batch files, are later transferred to the remote provider which applies them in the correct order. At any given point the runtime will be processing at most one batch file. Notice the emphasis on runtime. Any user specified event operating on the DataSet property would result in more than one batch file live in memory. Batches are spooled to the file system by the enumerating provider and the provider uses the RelationalSyncProvider.BatchingDirectory property on the enumerating provider to detect the base directory to use. For each sync the runtime will create a unique directory inside the base directory. The name of the base directory is of the format Sync_XXXXXXXXXXXX. The directory name is unique for the two providers currently synchronizing and the name does not change for subsequent syncs. This allows the runtime to detect “failed” synchronization attempts so it can resume from the failed point. More on the “Sync Resume” feature later. Inside that directory the runtime will spool one SyncBatchHeaderFile.sync file and one or more .batch files. The .sync file is the metadata file that contains metadata on the current sync session. It holds some key information such as the Version, MadeWithKnowledge and DestinationKnowledge. The .batch files contain the raw change data that needs to be applied on the destination. The .batch file is just a binary serialized version of the DbSyncBatchInfo type. The batch info type contains both the actual data and the metadata corresponding to that data. For faster access to the metadata the runtime serializes the metadata separately from the data. Here is the definition of the DbSyncBatchInfo type.

   1:  public class DbSyncBatchInfo : IDisposable

   2:      {

   3:          public DbSyncBatchInfo();

   4:          public long DataCacheSize { get; set; }

   5:          public DataSetSurrogate DataSetSurrogate { get; set; }

   6:          public string Id { get; set; }

   7:          public bool IsLastBatch { get; set; }

   8:          public uint SequenceNumber { get; set; }

   9:          public Version Version { get; set; }

  10:          public void Dispose();

  11:          protected virtual void Dispose(bool cleanup);

  12:          public byte[] GetLearnedKnowledge();

  13:          public void SetLearnedKnowledge(byte[] knowledgeBytes);

  14:          public override string ToString();

  15:      }

The actual data is contained in the DataSetSurrogate property while the rest of the properties are just the metadata for that data. Some key metadata items are

Version – Batching version of the provider that generated this batch. This enables the destination runtime to apply versioning rules when consuming batches generated from an older version.
Id – Unique id of this batch
SequenceNumber – This is used by the destination to ensure that batches are applied in the same order they were generated in. This will ensure that batch files arriving out of order at the destination are still consumed in the correct sequence.
DataCacheSize – Represents the deserialized size of the data.

With the RTM version we exposed this type out to the user. This change was made to accommodate feedback from users that they wanted to override the default BinarySerializer or in some cases completely move off DataSet as a data transfer medium. With this type public, users can now easily deserialize any .batch file and provider their customizations on top of that. Each batch file contains two binary objects. The first one is the actual DbSyncBatchInfo type sans the actual data followed by the serialized version of DataSertSurrogate. Users customizing this part of the runtime should pay attention to the format. I have attached a simple factory class DbSyncBatchInfoFactory.cs that can serialize and deserialize any batch file.

Resuming Sync Session

I had earlier mentioned that the runtime uses a unique reproducible directory name for storing batch files whenever it enumerates changes for a provider. This is so that the runtime can attempt to restart earlier failed sync session. There are many reasons for sync session abruptions but the most common one users face is transient network disconnections. Flaky network connections is quite a common issue with mobile users. Users would have restart a sync session if their network connectivity gets dropped in between. This meant that they had to download/upload all changes back to the remote server. This also means that the remote Sql Server is wasting precious CPU time enumerating the same changes over and over.

To avoid re-enumerating changes the enumerating provider will check to see if any batch files exist prior to spawning a new “Select” query. If any batch files for the current remote provider exists, the runtime will inspect the files to see if it can reuse those files. Several factors are considered to determine batch reusability such as state of enumeration, destination knowledge etc. The runtime uses the metadata from the SyncBatchHeaderFile.sync to check if existing batch files are relevant for the current state of the destination. If it determines that the files are relevant then it will pick up enumeration from where the older sync left off. This means if a table had changes between the first failed sync session and the restart only those changes would be picked up. If the runtime determines that the existing batch files are not relevant (destination got changes from a different peer or one of the batch files is corrupt of missing or the enumeration was incomplete the first time) all batch files are deleted and a fresh enumeration query is launched.

This batching directory is usually cleaned up by the runtime after it successfully applies all changes. Users can override this behavior by setting RelationalSyncProvider.CleanupBatchingDirectory property to false.

Batch Files Cleanup In Mid Tier

If in case users are using a mid tier to communicate with Sql Server then you would need a background job that constantly monitors the batching folder to delete any files older than a certain time. This is needed because the runtime cleanup of the batch files happens only on the destination provider after a successful application (to enable sync restart feature).

Anti-Virus tools and Batching

One note of caution to users having real time anti-virus scanning tools. Please exclude the batching directory from real time scan so that there is no file contention between the sync runtime and the virus scan runtime.

Maheshwar Jayaraman

Technorati Tags: DbSyncProvider,Sync Framework,Sync Services For ADO.NET

PDC 2009 and Sync related sessions

Mahjayar — Fri, 13 Nov 2009 21:28:00 GMT

PDC 2009 is next week and I wanted to have a quick post on some of the sessions about Microsoft Sync Framework.

Using the Microsoft Sync Framework to Connect Apps to the Cloud

http://microsoftpdc.com/Sessions/SVC23

- Come hear how the Microsoft Sync Framework can be used to bridge on-premises data to SQL Azure and Windows Azure storage. Learn to quickly build applications that use the cloud as a data hub for aggregation of enterprise, partner, desktop and device data.

You should hear more about Sync Framework in some of the Sql Azure sessions as well. Will have more details to share after next Tuesday.

MSF V2 CTP2 Deep Dive – Memory Based Batching

Mahjayar — Thu, 17 Sep 2009 02:24:00 GMT

Its been quite a while since I posted about MSF v2 CTP2. As promised in my earlier posts I will be doing a series of deep-dive of the new SyncServices features added in MSF V2 CTP2. Today I am starting off with the Batching feature.

One of the most common feedback we received with the peer providers shipped for MSF V1 was the inability to sync changes in batches. Customer using DbSyncProvider always complained of OutOfMemory exceptions when synchronizing large data across databases. The MSF workflow did support the notion of batching but the relational peer providers did not take advantage of the feature.

Background:

When we decided to support batching we looked back at how the hub-spoke providers supported batching. For the hub-spoke we supported batched download of data from server by modifying the enumeration query to return rows in batches. Even though this solved the issue of OOM with large data it had some inherent limitations. Some of the problems were

Code has to be hand coded by the user for each sync schema which meant a higher probability of user error.
Batching was not supported for uploads from the clients.
There was no easy way to estimate the number of batches you would need. The solution computed batch size by dividing the current tickcount with the last sent tick count for any table. This meant that the batch size was fixed even if the table had 0 changes between the last sent and the current time instance. This meant that the SyncAgent would return empty batches.
Batching was based on number of rows which made it harder to predict when data will be too big.
In general usability was hindered as the batching SP was complex and highly involved. The SP became more complex if the anchors were non timestamp based.

One other problem unique to the peer providers was the use of SyncKnowledge to compute the changes that needs to be synchronized. In hub-spoke these were simple timestamp anchors that users used to compute what has changed in a table and each table had its own anchor which did not collide with other tables. In peer providers the SyncKnowledge represents the “knowledge” of all tables for a given Sync scope. Any disparity in what the knowledge represents about a replica and the actual data would mean data will not synchronize and result in data non-convergence in your sync topology.

Motivation for Batching:

When we designed batching we wanted to address all the above shortcomings. We also wanted to reduce the probability of user errors when computing batches. The primary notion of batching would be to avoid running out of memory when synchronizing large amounts of data. The most straight forward way to support batching would be to batch by rows. This works for most cases but does the solve the problem mentioned earlier – how does the user compute how many rows can the client handle. A desktop machine might have enough processing power to consume 10,000 rows at a time but a hand held device might only be able to handle 100 rows. Moreover on a desktop the sync session has to share the available memory which meant that on high loads the number of rows that a sync session can handle without going out of memory rapidly declines.

The primary goal of the sync developer is to successfully sync data over from one replica to another without running out of memory. Basically the one and only goal of batching is to ensure that incremental progress can be made with the available memory resource on the machine. Since one of the most deterministic resource a developer has control over is the available memory, we decided to design batching around memory usage. The developer would put a limit on the max amount on the size of data being synchronized. This works wonderfully well as it makes batching an on demand operation. If row based batching was used then the system would have to send ‘N’ batches no matter how small the individual rows were of. In the memory based batching if all changes required to send to the destination fits within the max size specified then batching is bypassed.

We also wanted to make it support batching for all kinds of relational stores (Sql, Sql CE and any ADO.NET capable database). We wanted to achieve all of this with as little input from the user as possible. We did not want customers to reinvent the wheel around batching logic for every sync application they wrote.

Batching Workflow:

With the motivations for batching being clear we decided to enable batching via a simple switch on the RelationalSyncProvider. We added a new property, MemoryDataCacheSize", which users can set to limit the amount of memory used (in KB) by the synchronizing rows. Data cache limit can be set on both providers and the runtime will use the lower size to ensure that one peer does not send data that cannot be handled by the destination. Sync runtime will revert to batched mode even if one peer has not specified a cache size.

Batched Enumeration:

The workflow of batching is quite simple. Destination providers communicate its cache size by returning it to the SyncOrchestrator in its GetSyncBatchParameters() call. This in-turn is passed to the enumerating peer in the GetChangeBatch() call. Here is the simple workflow of the batched enumeration process.

If batching is enabled then start a background thread that will compute the enumeration.
Runtime starts by querying the list of tables mentioned in the DbSyncAdapters list.
For each table it retrieves the ADO.NET DataReader for its corresponding enum query.
- For each row, before reading the column values, it computes the size of the row.
- If current_Data_Cache_Size + Next_Row_Size <= (110% * MemoryDataCacheSize) specified then it reads the row in to memory.
- If pulling in the new row exceeds the specified MemoryDataCacheSize it spools the current in memory rows to a file. It then cleans up its data cache and continues reading changes.
- Batches are spooled in the user specified directory configured on RelationalSyncProvider.BatchingDirectory property.
The above steps are repeated till all the tables have been enumerated.

As you can see batch files are spooled to disk only when the data exceeds the memory size specified. If all changes fit in memory then the system reverts back to non batching mode where all changes are sent to destination in memory.

The runtime will try to stuff a batch with data as long as the in-memory size of those data does not exceed the specified batch size by 10%. This is to guard against sending many number of under populated batches.

There are times when a single row would be greater than 110% of the specified data cache size. In such cases the runtime errors out with an exception message that lists the offending table and the primary key of the row that is too big. Users would need to increase the data cache size to accommodate that row.

Since enumeration happens in the background, we employ a prodcuer/consumer model between the main threads GetChangeBatch() call and the background enumeration thread. The main consumer thread will block till the producer spools a batch (or reverts to non batched mode if all changes fit in memory) and report this batch information to the destination provider.

Batched Apply:

Relational providers uses the type DbSyncContext to communicate changes between the source and the destination providers. With batching we have added three new properties to this type to facilitate working with batches. The three properties added are

public string BatchFileName { get; set; } ==> Points to the actual spooled batch file.

public string IsLastBatch { get; set; }

public string IsDataBatched { get; set; }

When the destination receives a batched DbSyncContext it will queue the context. It will queue all batches till it receives the last batch at which point it will open a transaction, deserialize each batch, apply its contents and then finally commit the transaction. Destination provider will apply the batches in the same FIFO order it received them from the source provider.

Tombstone Application:

Relational providers usually applies changes in the following order. DELETES followed by INSERTS and UPDATES. Order in which these changes are applied is usually dictated by the order in which the users specify their table adapters. The adapter order assumes that related tables (PK/FK) and ordered correctly. For DELETES, the runtime will apply changes in the reverse adapter order to ensure that child table DELETES are applied first before parent table DELETES to avoid RI violations. Since batches are applied in FIFO order its plausible that the current batch may not contain the child table DELETES. Within a given batch the runtime continues to apply DELETES in reverse order. In the case where the child table is not in current batch the runtime will recognize the SQL errors for those tombstones and queue them up for retry. When all batches have been applied, the runtime will revisit this queued up DELETES and retry applying them. At this point all of the parent deletes should succeed as the dependent child entries should have already been applied.

Note that batching does not change any runtime logic related to conflicts handing. Batching only governs the number of rows in memory at any given point of time and nothing else. The only thing you have to guard in your ApplyChangeFailed event handler is the above mentioned case where a PK delete fails.

Batching Assumptions:

Sync runtime has full Read/Write access to the directory specified in the BatchingDirectory property.
MemoryDataCacheSize only applies to the memory used for reading sync rows. It does not guard against overall process memory usage.
Batches are applied in the order they are received by the destination provider.

So, Does It Work?

Moment of truth is whether all this batching thingy works and does it prevent process from running out of memory. I coded up a simple scenario which syncs two tables, Orders and OrderDetails, from a Sql server down to a CE client. The schema for the two tables is quite simple.

//Orders
// orderid - int, PK
// orderdate – datetime

//OrderDetails
            // orderdetailsid - id, PK
            // orderid - int
            // product - varchar(max)
            // quantity –int

I populated the tables with about 25,000 rows each. Each row in OrderDetails is about 8K in size and each row in orders is about 50 bytes in size. The demo computes the max working set of the process for reporting purposes. I did two runs one without batching and one with batching. Here is the comparison between the two runs.

Sync Type	Peak Working Set (MB)	Memory Use percent
Non Batched	446	NA
Batched (1 MB batch Size)	47	10.53
Batched (5 MB batch Size)	59	13.2
Batched (10 MB batch Size)	78	17.45

As you can see that batching uses approximately 10-20% of the non batched memory for the above sample. This is not to be an indication of memory savings for all kinds of schemas. Feel free to try it with your real world schemas and I am sure you will see considerable improvements.

Here is the procmon output for the non-batched case. As you can see the memory utilization keeps climbing till all changes are read in memory.

Here is the procmon for the same demo when batching is enabled. You can see that the runtime goes to 31 MB initially and then settles in to a nice pattern as files are spooled and then consumed by the destination.

Here is the MSDN doc link for the batching feature. http://msdn.microsoft.com/en-us/library/dd918908(SQL.105).aspx. Feel free to comment here is you have any specific questions.

Next- Detailed look at the batch file.

Maheshwar Jayaraman

Announcing Sync Framework 2.0 CTP2

Mahjayar — Fri, 05 Jun 2009 23:31:00 GMT

We just released CTP2 of Microsoft Sync Framework V2. Check out the blog entry at http://blogs.msdn.com/sync/archive/2009/06/04/announcing-sync-framework-2-0-ctp2.aspx for details on major new features.

We have so much to talk about Sync Services for ADO.NET in this release that I cannot fit all of these in one blog. For starters here are the big features we have crammed in to this CTP.

1. New Sql CE and Sql based peer sync provider - SqlCeSyncProvider and SqlSyncProvider.

2. New management API's to automatically provision Sql and Sql Server express to sync enable tables.

3. API's to backup and restore a Sync enabled database with no loss of Sync metadata.

4. Support for 0 code support for enabling batching in all databases providers (Sql and Sql CE providers) - This one majorly cool feature.

5. Snapshot initialization for bootstrapping a new Sql CE peer from an existing peer.

6. Support for shared server sync scopes on Sql server.

7. Improved sync performance on Sql CE.

Thats an summary of the major changes that we are making available in this CTP. Each of these features will require a separate blog post and I will do over the course of this weekend.

Maheshwar Jayaraman

Looking for volunteers for Project Huron Early Adopter Program

Mahjayar — Fri, 01 May 2009 00:55:00 GMT

Ever since we announced Project Huron at PDC last year we have been hard at work trying to scope out the scenarios for our V1 release. Due to resource contstraints we decided to scope out the Access to Cloud publish use case for V1. We instead decided to concentrate more on sharing Sql Server and Sql Server Compact databases via the Data Hub hosted in the sky. Liam, our Project Manager has a post over at our official sync blog asking for some early adopter partners. Please reply back using the contact form at http://blogs.msdn.com/sync/contact.aspx if you would like to be an early adpoter. Head over to the blog post for see some screen mockups of our early Huron Management studio UI.

Maheshwar Jayaraman

SyncServicesADO.NET Survery

Mahjayar — Wed, 08 Apr 2009 02:11:00 GMT

We are looking for some SyncServices for ADO.NET users (both Hub spoke and peer to peer toplogy) to take a survey and answer questions on how they handle custom conflict resolution and business logic processing. Please head over to http://blogs.msdn.com/sync/archive/2009/04/07/custom-conflict-resolution-survey.aspx and leave your responses as comments to that post.

This will go a long way in helping us figure out how best to streamline our event API's.

Maheshwar Jayaraman

DbSyncProvider WCF Based Synchronization– Memory Performance Analysis Of DataSet Binary SerializationFormat Vs DataSet Surrogates

Mahjayar — Tue, 10 Mar 2009 11:02:00 GMT

I had posted earlier about the memory performance when using default DataSet serialization behavior vs. using a Surrogate. Users had some questions on how the DataSet surrogate compared with the DataSet’s Remoting SerializationFormat. With .Net Fx 2.0 DataSet shipped support for serializing DataSet in Binary in addition to the default XML format. I decided to do a quick test comparing the serialization memory usage between the two optimizations. Here is a simple table detailing the memory usage.

I used a Toshiba Tecra laptop running Windows 7 Beta build with 2GB RAM. The DataSet contains one DataTable which contains one string column. Each column is a 5Mb string. Size is Peak working set. The BinaryFormatter serializes the DataSet in to a MemoryStream.

No of Rows	BinaryFormat Serialization (MB)	Surrogate Serialization (MB)
1	34.1	34.5
5	131.6	131.39
10	271.1	271.39
48	1116.1	1118.1
50	Out of Memory	1024
51	Out Of Memory	Out Of Memory

As you can see the memory performance is pretty similar to both the options but the Surrogate approach is able to handle slightly more data than the Binary format. The Binary format peaked at 48 rows while the Surrogate approach peaked at 50 rows. I like the Surrogate approach as it gives the users complete control on the way data is serialized and has option for further user defined optimizations. Further the option to pick DataSet Binary serialization format is not available in the Compact framework which means Surrogates might be the only viable option (as opposed to writing a new serializer) for devices. Next I will compare the wire size of the actual serialized data.

Here is the link for original Microsoft KB article on the Surrogate approach. http://support.microsoft.com/kb/829740.

PDC 08 Summary - Project "Huron" and MSF V2 CTP

Mahjayar — Thu, 06 Nov 2008 21:46:02 GMT

PDC 08 was last week and I wanted to summarize the PDC sync specific announcements. Hope every one had a chance to catch up on the all the impressive videos. First, Windows 7 looks impressive and I cant wait to dogfood the Beta build.

As I posted earlier, we had 3 Microsoft Sync Framework related presentations and here are the direct links to the recorded sessions.

Microsoft Sync Framework Advances – Presenter Lev Novik

Windows 7: Programming Sync Providers That Work Great with Windows – Presenter Jason Roberts

Sync Framework: Enterprise Data in the Cloud and on Devices (Presenter: Liam Cavanagh)

I am involved in Project Huron and wanted to talk briefly about what we discussed at PDC and what’s coming next. Project Huron intends to be a data hub in the cloud enabling data sharing and replication using the Sql Data Services and Microsoft Sync Framework. Here is the full snippet of Project Huron from our Sync blog.

Project Codename “Huron” – Leverage the power of SQL Data Services and Microsoft Sync Framework to build business data hubs in the cloud. Using this cloud based data hub, “Huron” provides a simpler, more convenient and less expensive way to:

· Publish databases to the cloud along with reports, forms and objects

· Subscribe to published data and automatically configure the local database for sync

· Make online changes through SQL Data Services and propagate those changes to subscribed users once they connect

· Enable scheduled and background synchronization of data changes through SQL Data Services and then on to other subscribed users

· Backup and restore of database applications to the cloud

We showed one such scenario of Project Huron at PDC. The PDC demo shows how customers using Access database can scale out and enable collaboration scenarios on the Access database by publishing it to the cloud. Customers interested in sharing or contributing to the data can subscribe to the hosted Access database. All changes are routed and synchronized in a peer to peer fashion through the cloud hub. We also demonstrated the “data hub” nature of the data in the cloud by downloading the same Access data to a Microsoft Compact database with full bi directional sync support. All components showed in the demo were running on live code. We used Microsoft Synchronization Framework V1 to build our AccessProvider, SDSProvider and SqlCESyncProvider. Acess and CE providers run locally on the box while the SDSProvider is running in our sync service running the cloud handling the SDS store. Feedback from PDC is very promising and the Access scale out/collaboration scenario seems to be a very common problem that customers run in to. Infact, we had a couple of guest posts over at the Access team blog and feedback from that blog has also been very promising. You can read the entire post at Announcement- Storing Access apps and data in the cloud and Video demos of Huron - Access and the SQL Server Data Services.

What’s next for Huron

We are looking for early adopters to participate in project Huron which will start very soon. If you are interested in participating then please send an email to DataLabs@Microsoft.com with “Huron beta” in the subject line.

We also announced public availability of the MSF v2 CTP which has some interesting new features built in. More info on MSF V2 CTP1 can be found here. We are very excited about Huron and cant wait to share it with early adopters.

No Re-release of Microsoft Sync Framework V1

Mahjayar — Wed, 22 Oct 2008 21:55:00 GMT

Last week, Mary Jo Foley reported that we re-released Microsoft Sync Framework V1 with updated bits. Since I hadn't seen any internal emails on this I pinged my PM to get more details. Yesterday Liam posted an entry in our Sync blog explaining the "bump" in the published date. Summary: We did not update the bits with any features/bug fixes. Apologize for the confusion.

Maheshwar Jayaraman

PD 2008 – Sync Framework Sessions

Mahjayar — Sun, 12 Oct 2008 20:11:17 GMT

We have 3 sessions planned for PDC. Details of the sessions:

Microsoft Sync Framework Advances

Presenter: Lev Novik

This session shows you how the next version of the Microsoft Sync Framework makes it easier to synchronize distributed copies of data across desktops, devices, services, or anywhere else they may be stored.

Sync Framework: Enterprise Data in the Cloud and on Devices

Presenter: Liam Cavanagh

See how synchronization plays a pivotal role in transitioning to a managed cloud environment by creating a central hub of information in the cloud. Using synchronization, organizations can enable more efficient mobile and enterprise-to-enterprise scenarios.

Windows 7: Programming Sync Providers That Work Great with Windows

Moe Khosravy, Jason Roberts

Learn how you can enable your application to synchronize with other applications that use the Microsoft Sync Framework. This session covers how to implement sync for contacts and other PIM data, how to package sync providers for distribution and installation, and how to register sync provider for use on Windows.

All 3 sessions have excellent content and are shaping very well. I have been involved in the second one being presented by Liam Cavanagh. I will discuss more about the work we did after PDC. PDC is mere 2 weeks away. Excited!

DbSyncProvider: Improving Memory Performance In WCF Based Synchronization

Mahjayar — Thu, 02 Oct 2008 06:16:00 GMT

Users have frequently wanted the ability to remotely synchronize relational nodes in a peer to peer fashion using SyncServices for ADO.NET. We have a sample demonstrating remote synchronization by using Windows Communication Foundation (WCF). I wanted to use this post to provide an easy way to optimize the performance of this WCF based solution. DbSyncProvider enumerates all changes in a DataSet and this gets applied on the destination provider. The moment the two providers are moved from a local 2-tier model to a remote n-tier model this DataSet has to be serialized and transmitted over the network. DataSet’s are very efficient when the amount of data involved is very small. The moment your data exceeds few hundred rows the amount of memory required to serialize/deserialize this DataSet is quite huge. Further the serialized size of the DataSet on disk is quite big as well.

DataSet object is by default serialized in XML format and serializing/deserialzing this XML data creates a lot of transient objects resulting in a spike in your memory usage. When you have enough data in the DataSet, like synchronizing large number of database rows, your app has the potential to go out of memory deserializing it. Infact users of SyncServices for ADO.NET V2 are quite aware of the OutOfMemoryException when they are synchronizing large number of records in the 2-tier model (PS: Solving this is the highest priority for us in the next release). Using the WCF solution increases the likelihood of this error happening even for mid sized data that fits fine in memory.

There is an easy way to optimize this problem and obliviously it requires users to move away from the XML based default DataSet serialization. Since DataSet doesn’t support any other format, the only way is to use a Surrogate to serialize it. Microsoft Knowledge base has a wonderful article detailing this Surrogate and it can be downloaded from http://support.microsoft.com/kb/829740.

Download it and use it in your WCF based Synchronization apps and you should see vast improvement in your memory usage and performance.

I wrote a quick sample checking the process peak memory usage during serialization/deserialization of a DataSet. Here are the comparison numbers between default and surrogate serialization.

The DataSet contains one DataTable which contains one string column. Each column is a 5Mb string. Size is Peak working set.

No of Rows	Default Serialization	Default Deserialization	Surrogate Serialization	Surrogate Deserialization
1	41 Mb	41 Mb	36 Mb	19 Mb
5	184 Mb	165 Mb	61 Mb	60 Mb
10	298 Mb	320 Mb	112 Mb	111 Mb
50	Out Of Memory	NA	524 Mb	523 Mb

Point proven :). As you can see for each 5Mb of data added in memory the default serialization takes anywhere from 6-8 times more memory. For 50 rows (approx 250Mb data size in memory) the serialization step itself fails with OOM.

Announcing the release of Microsoft Sync Framework V1 and Sync Services for ADO.Net V2

Mahjayar — Wed, 06 Aug 2008 21:13:43 GMT

Microsoft just released SQL Server 2008 and with that we have also released the golden versions of Microsoft Sync Framework V1 and Sync Services for ADO.Net V2. With this release Sync services now is built on top of the Sync Framework and supports peer to peer synchronization in addition to hub-spoke topology.

Our Sync blog has a post detailing all new features that we have shipped in this release. Check it out.

With this release we also pleased to announce the stand alone availability of the Sync Framework and Sync Services installer. When SQL Server 2008 RC was released the framework and services were part of SQL server installer and we did not provide a standalone installer. Go grab the new bits from http://www.microsoft.com/downloads/details.aspx?FamilyId=C88BA2D1-CEF3-4149-B301-9B056E7FB1E6&displaylang=en. Now every one should be able to check out our DbSyncProvider hello world sample.

Happy synchronization.

Maheshwar Jayaraman

Getting started with DbSyncProvider (Peer to Peer Synchronization)

Mahjayar — Wed, 25 Jun 2008 10:09:00 GMT

[Note: Source for this sample can be downloaded from http://www.maheshwar.net/projects/Sync/DbSyncProviderSample.zip. Just ensure that you modify the connectionstrings in properties\settings file to point to the correct database. This sample is based off the RC0 build recently released.]

This post is to be a starting guide for people wanting to use Microsoft Sync Framework to synchronize databases. We are shipping a customized database provider, DbSyncProvider, that lets users synchronize tables in a peer to peer fashion. Note that we also ship a DbServerSyncProvider/DbClientSyncProvider combination in Sync Services For ADO.NET API that synchronizes data between Server and Client in a hub-and-spoke topology. DbSyncProvider will allow hub-and-spoke synchronization (if that is what is required) but the real power lies in the peer to peer synchronization.

This post will walk through the steps required to sync an existing table using DbSyncProvider. We will use a simple WinForms application to view/edit and synchronize a Customer table across two DB's.

This post assumes that the table Customer cannot be modified and hence we will use a separate table (DeCoupled Tracking), CustomerMetadata, to track item metadata. To keep our queries simple we will use the default DB metadata column names. I will show at the very end how simple it is to modify the above solution for a table that maintains the metadata in the same table (Coupled Tracking). For sake of simplicity lets approach the database synchronization configuration in these following steps.

DbSyncProvider Specifics
Create Scope Information table
Add metadata for Customer table
Configure Source DbSyncProvider
Specify Synchronization Scope commands
Configure Destination DbSyncProvider
Conclusion: Putting it all together

Step 1. DbSyncProvider Specifics

A DbSyncProvider represents a set of tables within a database defined by a “scope”that are to be synchronized. It contains information such as the connection string and the synchronization scope name, as well as Knowledge information that is used to determine what information is known by the local peer. Each table that needs to be synchronized from the database needs to be represented by a DbSyncAdapter and should be added to the SyncAdapters collection on the provider. Each DbSyncAdapter will contain SqlCommands for the following tasks.

Selecting incremental changes that are not in the destination’s knowledge.
Performing Inserts/Updates/Deletes from remote peers and updating corresponding metadata.
Cleanup tombstone metadata.
Select a particular row with a given id.

At the provider level, users need to specify SqlCommands for the following tasks.

Select the current timestamp for the local provider.
Select/update Scope info for given scope_name.

With our sample we will see the very basic properties that needs to be configured at the provider/syncadapter level to get started. To keep this demo simple, I am going to refrain from using stored procedures and rather depend on plain vanilla select/insert/delete SQL commands. For this demo we will create two databases SampleDB1 and SampleDB2 and will create two DbSyncProviders to represent them. Create the above mentioned databases.

Note: For RC release, DbSyncProviders expects to have snapshot isolation turned on at each database. This also means that this can be used against Sql Server 2005 and higher. So after creating them run the following command in both databases.

Step 2. Create Scope Information Table

As I mentioned earlier the provider needs the synchronization scope name and other scope related commands (update/select). Scope represents a set of tables that are involved in a sync session and each scope will need to maintain knowledge (data and tombstone) information for all peers that this scope has synchronized with in the past. We will create a Scope_Info table to maintain this information. The design for the scope_info table in SampleDB1 database is as follows.

It contains the scope id, sync knowledge, tombstone knowledge and the last updated timestamp to hold all scope related information. Scope Id is a unique identifier Lets create a new scope 'DbSyncDemo' and use this to synchronize our user tables. Initially it will contain null for DbSyncSession.SyncScopeKnowledge and DbSyncSession.SyncScopeCleanupKnowledge. This will tell the core Sync runtime that this database has not synchronized with any peer.

We need the scope table in all peers that will be involved in the synchronization session so create the same table in the SampleDb2 database as well.

Note: All column names of this table matches the constant values for scope metadata columns in DbSyncSession. The provider will look for these column names from the result set when selecting or updating scope.

Step 3. Add metadata for Customer table

Lets take a look at the Customer table. It contains fields for name, age and country of each customer. Design of Customer table is as follows.

We use id as the Primary Key for this table. Next we need to create a CustomerMetadata table that will hold row metadata for each row in Customer table. Each row in the table represents an Item in the Sync world and hence our metadata table needs to maintain the modified timestamp, creation id, creation timestamp, update key, update timestamp and tombstone metadata for each corresponding row in Customer table. We will use the PK column ‘id’ from the base table to correlated entries in the metadata table. Once again we will use the constant metadata names from DbSyncSession class to name the columns in this table. Design of CustomerMetadata table is as follows.

Next we need to update the metadata table each time a row is inserted/updated/deleted. For that we will add triggers on Customer table. We will create 3 triggers, one each for Insert, Update and Delete. Lets take a look at the insert trigger.

CREATE TRIGGER [dbo].[customer_insert_trigger]
   ON [dbo].[Customer]
   For INSERT
AS
BEGIN
    -- Insert statements for trigger here
    insert into CustomerMetadata (id, sync_row_is_tombstone,sync_create_peer_key, sync_create_peer_timestamp, sync_update_peer_key, sync_update_peer_timestamp)
    select id, 0,0,@@DBTS+1,0,@@DBTS+1 from inserted
END

Since in Sync, '0' is the id for the current peer we enter 0 as the create/update key. Similarly we enter 0 for the tombstone column denoting that it is not a tombstone row.

The update trigger is much simpler. In case of updated we only need to update the update key (will always be 0 denoting current peer) and update timestamp.

CREATE TRIGGER [dbo].[customer_update_trigger]
   ON [dbo].[Customer]
   For update
AS
BEGIN
    -- Insert statements for trigger here
    update meta
        set sync_update_peer_key = 0,
        sync_update_peer_timestamp = @@DBTS +1
    from CustomerMetadata meta join inserted i on i.id = meta.id
END

Finally the delete trigger will set the tombstone column to 1 (in addition to modifying the update key/timestamp pair).

CREATE TRIGGER [dbo].[customer_delete_trigger]
   ON [dbo].[Customer]
   For delete
AS
BEGIN
    -- Insert statements for trigger here
    update meta
        set sync_row_is_tombstone = 1,
        sync_update_peer_key = 0,
        sync_update_peer_timestamp = @@DBTS+1
    from CustomerMetadata meta join deleted i on i.id = meta.id
END

Create the above tables, triggers in SampleDb1 and SampleDb2 databases. Next we are ready to configure DbSyncProvider's for each of these endpoints. For starters purpose we will just sync in one direction, i.e SampleDb1 -> SampleDb2. I use this example as opposed to discussing a two way sync as I want to list the specific properties that needs to be set for a "source" vs "destination" provider.

Step 4. Configure Source DbSyncProvider

Lets start by creating a DbSyncProvider object and pass the connection string for SampleDb1. Since we are using Decoupled metadata table we will set the ChangeTracking property on the provider. to ChangeTrackingModel.Decoupled.

Next, create a DbSyncAdapter for Customer table.

Each DbSyncAdapter needs to know the list of Id columns whose value it will use to create a unique SyncId for each row metadata. In our case we will add column "Id" to the RowIdColumns collection. Next the only command that this DbSyncAdapter needs to provide is the SelectIncrementalChanges command. This will return the rows that have been modified/added/deleted since the destination provider last sync'd.

As you can see we specify the SyncMinTimestamp metadata column as an input parameter. This input parameter would be dynamically filled by the provider. The value is computed by merging the local knowledge with the remote knowledge information returned by the destination provider for GetSyncBatchParamters() method call. The above select command will return all new/updated and deleted rows as well. Next we tell the DbSyncAdapter the names of the tracking columns which represent the TombStone flag and the last change timestamp for the row. DbSyncProvider will populate the IsTombstone property on SyncRowMetadata with the value returned from the IsTombstoneColumn and remove the tracking columns from the DataSet returned to the destination provider.

Finally add the DbSyncAdapter to the providers collection.

Next we need to specify Scope commands on the provider.

Step 5. Specify Synchronization Scope commands

Each DbSyncProvider needs certain commands to figure out the current timestamp of the database and to read/update scope information.

We need to set the following 3 properties on DbSyncProvider to enable reading timestamp and scope information.

DbSyncProvider.SelectNewTimestampProvider: This command will enable the provider to determine the current timestamp of the provider (in our sample it will be the current transaction count). Runtime will look for the timestamp in the DbSyncSession.SyncNewTimestamp metadata column so we will specify an output parameter in our query.

DbSyncProvider.SelectScopeInfoCommand:

This command will be used by the provider to select the data knowledge and cleanup knowledge that this provider knows about for this scope. We will retrieve the required information from the query as output parameters. The required output parameters are DbSyncSession.SyncScopeId, SyncScopeKnoeledge, SyncScopeCleanupKnowledge and SyncScopeTimestamp.

DbSyncProvider.UpdateScopeInfoCommand:

This command will be used by the destination provider to update knowledge and cleanup information each time the provider completes a successful sync from a remote provider. This command is also used when users calls DbSyncProvider.CleanupMetadata() function.

The runtime will make sure that it passes the right values for all the input parameters. We need to pass the rowcount back to the system in SyncRowCount metadata parameter so that the runtime can detect when update of the scope fails.

Finally configure the source DbSyncProvider with the above three commands.

Thats it. SampleDb1 is now configured to be a DownloadOnly source provider.

Step 6. Configure Destination DbSyncProvider

Next we need to create DbSyncProvider representing SampleDb2 database. For this we will create a DbSyncProvider and add the scope specific commands we added in step 4 and 5. Since the database schemas are same we will reuse the same SqlCommands.

Then we create a DbSyncAdapter for table Customer.

We will then add the following destination specific properties on the sync adapter.

DbSyncAdapter.InsertCommand:

In this case we are specifying column names we want to insert in Customer table. As you can see the individual column names are specified as input parameters and these parameters will be set with the corresponding column values for each row added on the source provider. We need to explicitly populate the DbSyncSession.SyncRowCount metadata parameter in our insert command as this is how the provider will determine if the insert succeeded or not. Count will always be 1 for a successful insert but will be 0 for either a constraint violation or when a row with the same PK already exists.

In case of a PK violation and other conflict scenarios, the provider would try to run the DbSyncAdapter.SelectRowCommand to see what is currently in the table. So we would provide the SelectRowCommand as follows.

I am going to reserve conflict detection to a separate blog post as the workflow and commands involved are slightly different (based on type of PK conflict such as Local delete, remote update or local update, remote update). Suffice to say if Insert failed and SelectRowCommand returns 0 results then it means that its a constraint violation and it will raise DbSyncProvider.ApplyChangeFailed event if one is registered.

DbSyncAdapter.InsertMetadataCommand

On a successful insert, the provider will then try to insert/update metadata for the currently inserted row. We need to provide the following insert command to insert metadata in CustomerMetadata table.

We need to return the rowcount back to DbSyncProvider so it can raise a ApplyMetadataFailed event in case its 0.

DbSyncAdapter.UpdateMetadataCommand

The update command for our sample is as follows.

On a successful insert of a row(rowcount for InsertCommand > 0), DbSyncProvider assumes the Couple metadata tracking model and hence will first try to execute the UpdateMetadataCommand. In our case the UpdateMetadataCommand will return 0 rowcount and the system will check to see that the model is decoupled and then execute the InsertMetadataCommand. If that fails the ApplyMetadataFailed event will be raised.

DbSyncAdapter.UpdateCommand:

This command will update existing record in Customer table. Similar to the Insert command we need to return the rowcount back as an output parameter. If update count is 0 then we try to detect whether the row exists by running SelectRowMetadata and resolve the error. If SelectRowMetadata returns 0 then runtime will retry the update as an Insert.

On successful update, the provider will then execute UpdateMetadataCommand. If that fails (rowcount < 1) then the same fallback procedure mentioned for InsertCommand will be followed.

DbSyncAdapter.DeleteCommand:

Finally in the delete command we will remove the row from the Customer table.

Once again runtime expects us to return the rowcount as an output parameter. On failure to delete the record it will try execute SelectRowCommand to try to detect conflict as mentioned above. If that fails then it means that the remote peer created and deleted a row before this peer could synchronize. So the system would try to keep tombstone record for that entry by adding a metadata record for that deleted row to guarantee consistent metadata across peers.

DbSyncAdapter.DeleteMetadataCommand

This command is never used directly. Its only used during a conflict during InsertCommand (and the system tries it as an update) or a conflict in UpdateCommand (in that case the system tries it as an insert). In that case DbSyncProvider will first run the DeleteMetadataCommand and if that is successful will then retry the insert/update.

In our sample we will use the following DeleteMetadataCommand.

With the above mentioned 7 commands the destination specific DbSyncAdapter configuration is complete. We will add this table to the destination DbSyncProvider.

Step 7. Conclusion: Putting it all together

With the source and destination providers set we will start Synchronization by creating a SyncOrchestrator object, set the local and remote providers and set the Direction property to SyncDirectionOrder.Download.

Finally we will call Synchronize when the user presses the "Synchronize" button on the winform.

Running the sample:

Lets start off by running the sample. We will add two rows for John and Jane Doe. Add the rows on the left hand side Customer table and press SaveChanges button. Then refresh the tables to see the metadata created by the triggers.

Next press synchronize and see the left side tables getting the values. Also ensure that the create and update key on the left side metadata column points to 1 and not 0.

I will leave the update/delete exercise up to the user. To modify the sample to be a two way synchronization, just add the source level DbSyncAdapter commands to SampleDb2 provider and similarly add the destination specific DbSyncAdapter commands to SampleDb1 provider. Then finally change the Direction property on the SyncOrchestrator to DownloadAndUpload.

Conclusion:

To summarize, we did the following to enable syncing Customer table across peers.

Created a metadata table for DeCoupled tracking in all peers.
Enabled Snapshot isolation on all peers.
Configured a DbSyncProvider/DbSyncAdapter for SampleDb1 peer and added source peer specific commands.
Configured a DbSyncProvider/DbSyncAdapter for SampleDb2 peer and added destination peer specific commands.

Maheshwar Jayaraman