Mahjayar's WebLog.

New year and new beginings

mahjayar — Sat, 16 Apr 2011 21:21:46 GMT

Hi,

I have recently made the decision to start on a new adventure outside Microsoft. With me moving out of Microsoft, I will no longer have access to manage this blog. For the past couple of years I have been cross posting all my posts on my personal blog at http://maheshwar.net/blog and moving forward that will be my primary blog. I know this is not a high traffic blog but if you have me on your subscription lists then please change your feeds over to http://maheshwar.net/Blog/syndication.axd.

I will be posting about my new adventures over on my personal blog soon.

Maheshwar

Sync Framework 4.0 CTP Refresh Now Available

mahjayar — Thu, 18 Nov 2010 19:43:45 GMT

As mentioned in my earlier post the 4.0 CTP refresh is now available. It contains the UI Helper for SyncSvcUtil.exe command line tool in addition to the iPhone sample client for the Lists sample. Please download the bits from here. We have also updated the documentation in this refresh which now shows step by step instructions on how to use the UI tool. Documentation can be found here.

Some presentations that talks and shows off the latest CTP is also available.

PDC 2010 Session: To learn more about this release 4.0 in general, take a look at our PDC session recording "Building Offline Applications using Sync Framework and SQL Azure".

Sql PASS 2010 Session: The recording for this talk is not available online but the presentation is available online. Its over here.

Announcing Sync Framework 4.0 CTP

mahjayar — Wed, 10 Nov 2010 05:29:25 GMT

Back in Mix 10 we showed an application built with the intention of extending the Sync Framework to Silverlight platform. This enabled rich apps built on Silverlight for desktop and Windows Phone 7 to participate in a sync topology. At that point we only showed an application built on our early thinking. Since then we have been off trying to package it for public consumption. I am very glad to announce the public availability of the Sync Framework 4.0 CTP which enables users to now build similar applications. With this release we are announcing the Asymmetric Sync toplogy that enables offline application development on any client platform that has capability of persisting data. Up until Sync Fx 2.1 users had to have Windows to be able to use the Sync framework. One of the most common feature requests were to ship the framework for windows devices (CE, embedded and the WM 6.X) platform. With the advent of Windows Phone 7 we took a long hard look at the device/non windows platform support and the asymmetric sync was one such solution we decided to pursue. Asymmetric sync standardizes on a protocol instead of a sync components on clients and moved ALL sync framework components to the server side. This enabled client platforms such as Silverlight (desktop and Windows Phone 7), HTML5, iPhone, Android, RIM and others have the ability to build offline apps synchronizing with a standard sync service.

Sync Framework 4.0 (yes we skipped 3.0!) CTP is not available. Please visit http://blogs.msdn.com/b/sync/archive/2010/10/28/announcing-sync-framework-4-0-october-2010-ctp.aspx for more information and the scenarios it enables. We are very excitied with this release as this opens the Sync framework to a completely new set of developers and we cant wait to see the interesting scenarios these developers will come up with.

I am presenting at Sql PASS 2010 tomorrow on Sync Framework 4.0. If you are attending Sql PASS then try to attend this session. The session should be available for on demand viewing later and I will update with a direct link to the presentation.

Announcing the availability of the MIX10 Session Planner (Beta)

mahjayar — Mon, 01 Mar 2010 23:49:00 GMT

Sorry for the radio silence the past couple of months. We have been working on our the next version of Microsoft Sync Framework and as part of that we decided to do a little preview of our work at MIX10. For starters we have made the MIX10 Session Planner (Beta) public that provides an offline cache/Synchronization of the MIX sessions data in Silverlight. The synchronization brains for the app is built on top of our early Sync 3.0 code base and provides a glimpse of the direction we are heading towards. We will have lots more to discuss at MIX on the architecture of the service. For the time being visit http://live.visitmix.com/News/MIX10-Session-Planner to get started on using the offline cache experience (note LiveID sign in required).

PS: If the synchronization space is something you are interested in and would like to join our team we are still hiring.

Maheshwar Jayaraman

Software Engineer In Test Opportunities Available In The Microsoft Sync Framework team

mahjayar — Fri, 15 Jan 2010 00:26:00 GMT

If you are interested in the data synchronization space then we have opportunties available on the Microsoft Sync Framework team. Refer https://careers.microsoft.com/JobDetails.aspx?ss=&pg=0&so=&rw=1&jid=11145&jlang=EN. You can apply directly on Microsoft career site or use the contact link in this blog to send me your resume.

SQL Azure Data Sync – Moving Schema from Compact Database to SQL Azure

mahjayar — Thu, 19 Nov 2009 01:30:00 GMT

Update 12/11: Fixing a typo in the code as pointed by the commenter.

With yesterdays announcement of the Microsoft Sync Framework Power Pack for SQL Azure, we have had questions on how to move sync schema from SQL Compact database up to SQL Azure. The power pack only comes with tools to automate the sync schema setup from SQL Server to SQL Azure (via the SQL Azure Data Sync Tool) and taking SQL Azure data offline to a new compact database (via the VS Add New Item template). There is no UI tool to go from Compact to SQL Azure or even SQL Azure to SQL Server.

Just wanted to point out that its quite simple to achieve the above two non UI supported scenarios via code. Attached is the simple code that shows how to move sync schema from Compact to SQL Azure.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.SqlServerCe;
using Microsoft.Synchronization.Data;
using Microsoft.Synchronization.Data.SqlServerCe;
using Microsoft.Synchronization.Data.SqlAzure;
using System.Data.SqlClient;
namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            SqlCeConnection conn = new SqlCeConnection("c:\temp\abc.sdf");
            using (conn)
            {
                conn.Open();
                DbSyncScopeDescription desc = GetDbSyncDescription(conn);

                //Check and sync enable the Compact database
                SqlCeSyncScopeProvisioning ceScope = 
                    new SqlCeSyncScopeProvisioning(desc);
                if (!ceScope.ScopeExists(scopeName, conn))
                {
                    ceScope.Apply(conn);
                }

                SqlConnection azConn = 
                    new SqlConnection("YourSqlAzureConnectionString");
                using (azConn)
                {
                    azConn.Open();
                    //Check and sync enable the SQL Azure database
                    SqlAzureSyncScopeProvisioning azScope = 
                        new SqlAzureSyncScopeProvisioning(desc);
                    if (!azScope.ScopeExists(scopeName, azConn))
                    {
                        azScope.Apply(azConn);
                    }
                }
            }
        }

        static string[] tableNames = new string[] { "Foo", "Bar" };
        static string scopeName = "FooBarScope";
        private static DbSyncScopeDescription GetDbSyncDescription
            (SqlCeConnection conn)
        {
            DbSyncScopeDescription desc = new DbSyncScopeDescription();

            /***
             * Option 1 for generating schema - Iterate through the list 
             * of tables you need and get description for them
             */ 
            foreach (string table in tableNames)
            {

                desc.Tables.Add(
                    SqlCeSyncDescriptionBuilder.GetDescriptionForTable
                    (table, conn));

            }

            /***
             * Option 2 for generating schema - If you already have a 
             * Compact database provisioned for sync you
             * could just read the whole scope description from it.
             **/
            // desc = SqlCeSyncDescriptionBuilder.GetDescriptionForScope(
            // scopeName, conn);

            return desc;
        }
    }
}

To move from SQL Azure to SQL Server you just need to replace SqlCeSyncScopeDescription type with SqlSyncScopeDescription. To access the new types under Microsoft.Synchronization.Data.SqlAzure just add a reference to the dll of the same name found under %programfiles%\Microsoft Sync Framework\Power pack For Sql azure November CTP\

Maheshwar Jayaraman

Announcing Microsoft Sync Framework Power Pack For SQL Azure CTP

mahjayar — Tue, 17 Nov 2009 19:15:00 GMT

PDC 2009 starts today and Ray Ozzie and Bob Muglia kicked off todays keynote presentation. It gives me great pleasure to announce the public availability of the MSF power pack for SQL Azure. Its a CTP and we are really excited to see people download this and provide feedback around the direction we are heading w.r.t providing a synchronization story for SQL Azure. Head over to http://tinyurl.com/yhn76rr to download the Power pack.

The power pack provides the following components.

SQL Azure Data Sync Tool for SQL Server - This is a automated tool to migrate your on premise SQL server data to SQL Azure by setting up a strong synchronization relationship. Kelly Blue Book demonstrated this in the key note when they moved a table from their on premise database to SQL Azure.

SqlAzureSyncProvider - Power pack also brings in a new dll, named Microsoft.Synchronization.Data.SqlAzure.dll, that contains a brand new provider, SqlAzureProvider, that enables existing database providers to synchronize with SQL Azure. This also has the new SQL Azure specific Sync metadata provisioning API's.

Sql Azure Offline Visual Studio Plug-In - This adds a new item template for VS 2008 SP1 that enables you to setup offline synchronization relationships with SQL Azure.

The team has worked really hard to bring this to you and we would love any feedback you got. You can use the email feature here or send email to syncfdbk at microsoft dot com with your feedback.

We first discussed about Project "Huron" at last year's PDC and with this CTP we have taken the first steps toward the vision of a Data Hub in the sky. We have a more detailed post about this over at our Sync blog. Refer http://blogs.msdn.com/sync/archive/2009/11/17/announcing-sql-azure-data-sync-november-ctp-available-for-download.aspx.

Mark Scurell, our lead PM, will be demonstrating this whole end to end walkthrough in his talk on Thursday 11/19 at 3 PM. For more information refer http://microsoftpdc.com/Sessions/SVC23. He will take you through the process of synchronizing an on-premise database with SQL Azure, generate an offline cache from SQL Azure down to a SQL Server Compact database and move data between the client, SQL Azure and on-premise server all with just one single line of code.

Maheshwar Jayaraman

MSF V2 DeepDive – Batching Directory and Batch Files

mahjayar — Mon, 16 Nov 2009 22:15:00 GMT

This is a continuation from the previous post I did explaining the new memory based batching feature for Database providers in MSF V2. Please refer to that post at MSF V2 CTP2 Deep Dive – Memory Based Batching. This is probably a good time to mention that the final version of MSF V2 has been released to web and can be downloaded from http://msdn.microsoft.com/en-us/sync/default.aspx.

Database providers chunks up changes by spooling changes to the file system. These chunks, or batch files, are later transferred to the remote provider which applies them in the correct order. At any given point the runtime will be processing at most one batch file. Notice the emphasis on runtime. Any user specified event operating on the DataSet property would result in more than one batch file live in memory. Batches are spooled to the file system by the enumerating provider and the provider uses the RelationalSyncProvider.BatchingDirectory property on the enumerating provider to detect the base directory to use. For each sync the runtime will create a unique directory inside the base directory. The name of the base directory is of the format Sync_XXXXXXXXXXXX. The directory name is unique for the two providers currently synchronizing and the name does not change for subsequent syncs. This allows the runtime to detect “failed” synchronization attempts so it can resume from the failed point. More on the “Sync Resume” feature later. Inside that directory the runtime will spool one SyncBatchHeaderFile.sync file and one or more .batch files. The .sync file is the metadata file that contains metadata on the current sync session. It holds some key information such as the Version, MadeWithKnowledge and DestinationKnowledge. The .batch files contain the raw change data that needs to be applied on the destination. The .batch file is just a binary serialized version of the DbSyncBatchInfo type. The batch info type contains both the actual data and the metadata corresponding to that data. For faster access to the metadata the runtime serializes the metadata separately from the data. Here is the definition of the DbSyncBatchInfo type.

   1:  public class DbSyncBatchInfo : IDisposable

   2:      {

   3:          public DbSyncBatchInfo();

   4:          public long DataCacheSize { get; set; }

   5:          public DataSetSurrogate DataSetSurrogate { get; set; }

   6:          public string Id { get; set; }

   7:          public bool IsLastBatch { get; set; }

   8:          public uint SequenceNumber { get; set; }

   9:          public Version Version { get; set; }

  10:          public void Dispose();

  11:          protected virtual void Dispose(bool cleanup);

  12:          public byte[] GetLearnedKnowledge();

  13:          public void SetLearnedKnowledge(byte[] knowledgeBytes);

  14:          public override string ToString();

  15:      }

The actual data is contained in the DataSetSurrogate property while the rest of the properties are just the metadata for that data. Some key metadata items are

Version – Batching version of the provider that generated this batch. This enables the destination runtime to apply versioning rules when consuming batches generated from an older version.
Id – Unique id of this batch
SequenceNumber – This is used by the destination to ensure that batches are applied in the same order they were generated in. This will ensure that batch files arriving out of order at the destination are still consumed in the correct sequence.
DataCacheSize – Represents the deserialized size of the data.

With the RTM version we exposed this type out to the user. This change was made to accommodate feedback from users that they wanted to override the default BinarySerializer or in some cases completely move off DataSet as a data transfer medium. With this type public, users can now easily deserialize any .batch file and provider their customizations on top of that. Each batch file contains two binary objects. The first one is the actual DbSyncBatchInfo type sans the actual data followed by the serialized version of DataSertSurrogate. Users customizing this part of the runtime should pay attention to the format. I have attached a simple factory class DbSyncBatchInfoFactory.cs that can serialize and deserialize any batch file.

Resuming Sync Session

I had earlier mentioned that the runtime uses a unique reproducible directory name for storing batch files whenever it enumerates changes for a provider. This is so that the runtime can attempt to restart earlier failed sync session. There are many reasons for sync session abruptions but the most common one users face is transient network disconnections. Flaky network connections is quite a common issue with mobile users. Users would have restart a sync session if their network connectivity gets dropped in between. This meant that they had to download/upload all changes back to the remote server. This also means that the remote Sql Server is wasting precious CPU time enumerating the same changes over and over.

To avoid re-enumerating changes the enumerating provider will check to see if any batch files exist prior to spawning a new “Select” query. If any batch files for the current remote provider exists, the runtime will inspect the files to see if it can reuse those files. Several factors are considered to determine batch reusability such as state of enumeration, destination knowledge etc. The runtime uses the metadata from the SyncBatchHeaderFile.sync to check if existing batch files are relevant for the current state of the destination. If it determines that the files are relevant then it will pick up enumeration from where the older sync left off. This means if a table had changes between the first failed sync session and the restart only those changes would be picked up. If the runtime determines that the existing batch files are not relevant (destination got changes from a different peer or one of the batch files is corrupt of missing or the enumeration was incomplete the first time) all batch files are deleted and a fresh enumeration query is launched.

This batching directory is usually cleaned up by the runtime after it successfully applies all changes. Users can override this behavior by setting RelationalSyncProvider.CleanupBatchingDirectory property to false.

Batch Files Cleanup In Mid Tier

If in case users are using a mid tier to communicate with Sql Server then you would need a background job that constantly monitors the batching folder to delete any files older than a certain time. This is needed because the runtime cleanup of the batch files happens only on the destination provider after a successful application (to enable sync restart feature).

Anti-Virus tools and Batching

One note of caution to users having real time anti-virus scanning tools. Please exclude the batching directory from real time scan so that there is no file contention between the sync runtime and the virus scan runtime.

Maheshwar Jayaraman

Technorati Tags: DbSyncProvider,Sync Framework,Sync Services For ADO.NET

PDC 2009 and Sync related sessions

mahjayar — Fri, 13 Nov 2009 21:28:00 GMT

PDC 2009 is next week and I wanted to have a quick post on some of the sessions about Microsoft Sync Framework.

Using the Microsoft Sync Framework to Connect Apps to the Cloud

http://microsoftpdc.com/Sessions/SVC23

- Come hear how the Microsoft Sync Framework can be used to bridge on-premises data to SQL Azure and Windows Azure storage. Learn to quickly build applications that use the cloud as a data hub for aggregation of enterprise, partner, desktop and device data.

You should hear more about Sync Framework in some of the Sql Azure sessions as well. Will have more details to share after next Tuesday.

MSF V2 CTP2 Deep Dive – Memory Based Batching

mahjayar — Thu, 17 Sep 2009 02:24:00 GMT

Its been quite a while since I posted about MSF v2 CTP2. As promised in my earlier posts I will be doing a series of deep-dive of the new SyncServices features added in MSF V2 CTP2. Today I am starting off with the Batching feature.

One of the most common feedback we received with the peer providers shipped for MSF V1 was the inability to sync changes in batches. Customer using DbSyncProvider always complained of OutOfMemory exceptions when synchronizing large data across databases. The MSF workflow did support the notion of batching but the relational peer providers did not take advantage of the feature.

Background:

When we decided to support batching we looked back at how the hub-spoke providers supported batching. For the hub-spoke we supported batched download of data from server by modifying the enumeration query to return rows in batches. Even though this solved the issue of OOM with large data it had some inherent limitations. Some of the problems were

Code has to be hand coded by the user for each sync schema which meant a higher probability of user error.
Batching was not supported for uploads from the clients.
There was no easy way to estimate the number of batches you would need. The solution computed batch size by dividing the current tickcount with the last sent tick count for any table. This meant that the batch size was fixed even if the table had 0 changes between the last sent and the current time instance. This meant that the SyncAgent would return empty batches.
Batching was based on number of rows which made it harder to predict when data will be too big.
In general usability was hindered as the batching SP was complex and highly involved. The SP became more complex if the anchors were non timestamp based.

One other problem unique to the peer providers was the use of SyncKnowledge to compute the changes that needs to be synchronized. In hub-spoke these were simple timestamp anchors that users used to compute what has changed in a table and each table had its own anchor which did not collide with other tables. In peer providers the SyncKnowledge represents the “knowledge” of all tables for a given Sync scope. Any disparity in what the knowledge represents about a replica and the actual data would mean data will not synchronize and result in data non-convergence in your sync topology.

Motivation for Batching:

When we designed batching we wanted to address all the above shortcomings. We also wanted to reduce the probability of user errors when computing batches. The primary notion of batching would be to avoid running out of memory when synchronizing large amounts of data. The most straight forward way to support batching would be to batch by rows. This works for most cases but does the solve the problem mentioned earlier – how does the user compute how many rows can the client handle. A desktop machine might have enough processing power to consume 10,000 rows at a time but a hand held device might only be able to handle 100 rows. Moreover on a desktop the sync session has to share the available memory which meant that on high loads the number of rows that a sync session can handle without going out of memory rapidly declines.

The primary goal of the sync developer is to successfully sync data over from one replica to another without running out of memory. Basically the one and only goal of batching is to ensure that incremental progress can be made with the available memory resource on the machine. Since one of the most deterministic resource a developer has control over is the available memory, we decided to design batching around memory usage. The developer would put a limit on the max amount on the size of data being synchronized. This works wonderfully well as it makes batching an on demand operation. If row based batching was used then the system would have to send ‘N’ batches no matter how small the individual rows were of. In the memory based batching if all changes required to send to the destination fits within the max size specified then batching is bypassed.

We also wanted to make it support batching for all kinds of relational stores (Sql, Sql CE and any ADO.NET capable database). We wanted to achieve all of this with as little input from the user as possible. We did not want customers to reinvent the wheel around batching logic for every sync application they wrote.

Batching Workflow:

With the motivations for batching being clear we decided to enable batching via a simple switch on the RelationalSyncProvider. We added a new property, MemoryDataCacheSize", which users can set to limit the amount of memory used (in KB) by the synchronizing rows. Data cache limit can be set on both providers and the runtime will use the lower size to ensure that one peer does not send data that cannot be handled by the destination. Sync runtime will revert to batched mode even if one peer has not specified a cache size.

Batched Enumeration:

The workflow of batching is quite simple. Destination providers communicate its cache size by returning it to the SyncOrchestrator in its GetSyncBatchParameters() call. This in-turn is passed to the enumerating peer in the GetChangeBatch() call. Here is the simple workflow of the batched enumeration process.

If batching is enabled then start a background thread that will compute the enumeration.
Runtime starts by querying the list of tables mentioned in the DbSyncAdapters list.
For each table it retrieves the ADO.NET DataReader for its corresponding enum query.
- For each row, before reading the column values, it computes the size of the row.
- If current_Data_Cache_Size + Next_Row_Size <= (110% * MemoryDataCacheSize) specified then it reads the row in to memory.
- If pulling in the new row exceeds the specified MemoryDataCacheSize it spools the current in memory rows to a file. It then cleans up its data cache and continues reading changes.
- Batches are spooled in the user specified directory configured on RelationalSyncProvider.BatchingDirectory property.
The above steps are repeated till all the tables have been enumerated.

As you can see batch files are spooled to disk only when the data exceeds the memory size specified. If all changes fit in memory then the system reverts back to non batching mode where all changes are sent to destination in memory.

The runtime will try to stuff a batch with data as long as the in-memory size of those data does not exceed the specified batch size by 10%. This is to guard against sending many number of under populated batches.

There are times when a single row would be greater than 110% of the specified data cache size. In such cases the runtime errors out with an exception message that lists the offending table and the primary key of the row that is too big. Users would need to increase the data cache size to accommodate that row.

Since enumeration happens in the background, we employ a prodcuer/consumer model between the main threads GetChangeBatch() call and the background enumeration thread. The main consumer thread will block till the producer spools a batch (or reverts to non batched mode if all changes fit in memory) and report this batch information to the destination provider.

Batched Apply:

Relational providers uses the type DbSyncContext to communicate changes between the source and the destination providers. With batching we have added three new properties to this type to facilitate working with batches. The three properties added are

public string BatchFileName { get; set; } ==> Points to the actual spooled batch file.

public string IsLastBatch { get; set; }

public string IsDataBatched { get; set; }

When the destination receives a batched DbSyncContext it will queue the context. It will queue all batches till it receives the last batch at which point it will open a transaction, deserialize each batch, apply its contents and then finally commit the transaction. Destination provider will apply the batches in the same FIFO order it received them from the source provider.

Tombstone Application:

Relational providers usually applies changes in the following order. DELETES followed by INSERTS and UPDATES. Order in which these changes are applied is usually dictated by the order in which the users specify their table adapters. The adapter order assumes that related tables (PK/FK) and ordered correctly. For DELETES, the runtime will apply changes in the reverse adapter order to ensure that child table DELETES are applied first before parent table DELETES to avoid RI violations. Since batches are applied in FIFO order its plausible that the current batch may not contain the child table DELETES. Within a given batch the runtime continues to apply DELETES in reverse order. In the case where the child table is not in current batch the runtime will recognize the SQL errors for those tombstones and queue them up for retry. When all batches have been applied, the runtime will revisit this queued up DELETES and retry applying them. At this point all of the parent deletes should succeed as the dependent child entries should have already been applied.

Note that batching does not change any runtime logic related to conflicts handing. Batching only governs the number of rows in memory at any given point of time and nothing else. The only thing you have to guard in your ApplyChangeFailed event handler is the above mentioned case where a PK delete fails.

Batching Assumptions:

Sync runtime has full Read/Write access to the directory specified in the BatchingDirectory property.
MemoryDataCacheSize only applies to the memory used for reading sync rows. It does not guard against overall process memory usage.
Batches are applied in the order they are received by the destination provider.

So, Does It Work?

Moment of truth is whether all this batching thingy works and does it prevent process from running out of memory. I coded up a simple scenario which syncs two tables, Orders and OrderDetails, from a Sql server down to a CE client. The schema for the two tables is quite simple.

//Orders
// orderid - int, PK
// orderdate – datetime

//OrderDetails
            // orderdetailsid - id, PK
            // orderid - int
            // product - varchar(max)
            // quantity –int

I populated the tables with about 25,000 rows each. Each row in OrderDetails is about 8K in size and each row in orders is about 50 bytes in size. The demo computes the max working set of the process for reporting purposes. I did two runs one without batching and one with batching. Here is the comparison between the two runs.

Sync Type	Peak Working Set (MB)	Memory Use percent
Non Batched	446	NA
Batched (1 MB batch Size)	47	10.53
Batched (5 MB batch Size)	59	13.2
Batched (10 MB batch Size)	78	17.45

As you can see that batching uses approximately 10-20% of the non batched memory for the above sample. This is not to be an indication of memory savings for all kinds of schemas. Feel free to try it with your real world schemas and I am sure you will see considerable improvements.

Here is the procmon output for the non-batched case. As you can see the memory utilization keeps climbing till all changes are read in memory.

Here is the procmon for the same demo when batching is enabled. You can see that the runtime goes to 31 MB initially and then settles in to a nice pattern as files are spooled and then consumed by the destination.

Here is the MSDN doc link for the batching feature. http://msdn.microsoft.com/en-us/library/dd918908(SQL.105).aspx. Feel free to comment here is you have any specific questions.

Next- Detailed look at the batch file.

Maheshwar Jayaraman