We just released CTP2 of Microsoft Sync Framework V2. Check out the blog entry at http://blogs.msdn.com/sync/archive/2009/06/04/announcing-sync-framework-2-0-ctp2.aspx for details on major new features.

 We have so much to talk about Sync Services for ADO.NET in this release that I cannot fit all of these in one blog. For starters here are the big features we have crammed in to this CTP.

1. New Sql CE and Sql based peer sync provider - SqlCeSyncProvider and SqlSyncProvider.

2. New management API's to automatically provision Sql and Sql Server express to sync enable tables.

3. API's to backup and restore a Sync enabled database with no loss of Sync metadata.

4. Support for 0 code support for enabling batching in all databases providers (Sql and Sql CE providers) - This one majorly cool feature.

5. Snapshot initialization for bootstrapping a new Sql CE peer from an existing peer.

6. Support for shared server sync scopes on Sql server.

7. Improved sync performance on Sql CE.

Thats an summary of the major changes that we are making available in this CTP. Each of these features will require a separate blog post and I will do over the course of this weekend.

Maheshwar Jayaraman

Ever since we announced Project Huron at PDC last year we have been hard at work trying to scope out the scenarios for our V1 release. Due to resource contstraints we decided to scope out the Access to Cloud publish use case for V1. We instead decided to concentrate more on sharing Sql Server and Sql Server Compact databases via the Data Hub hosted in the sky. Liam, our Project Manager has a post over at our official sync blog asking for some early adopter partners. Please reply back using the contact form at http://blogs.msdn.com/sync/contact.aspx if you would like to be an early adpoter. Head over to the blog post for see some screen mockups of our early Huron Management studio UI.

Maheshwar Jayaraman

We are looking for some SyncServices for ADO.NET users (both Hub spoke and peer to peer toplogy) to take a survey and answer questions on how they handle custom conflict resolution and business logic processing. Please head over to http://blogs.msdn.com/sync/archive/2009/04/07/custom-conflict-resolution-survey.aspx and leave your responses as comments to that post.

This will go a long way in helping us figure out how best to streamline our event API's.

Maheshwar Jayaraman

I had posted earlier about the memory performance when using default DataSet serialization behavior vs. using a Surrogate. Users had some questions on how the DataSet surrogate compared with the DataSet’s Remoting SerializationFormat. With .Net Fx 2.0 DataSet shipped support for serializing DataSet in Binary in addition to the default XML format. I decided to do a quick test comparing the serialization memory usage  between the two optimizations. Here is a simple table detailing the memory usage.

I used a Toshiba Tecra laptop running Windows 7 Beta build with 2GB RAM. The DataSet contains one DataTable which contains one string column. Each column is a 5Mb string. Size is Peak working set. The BinaryFormatter serializes the DataSet in to a MemoryStream.

As you can see the memory performance is pretty similar to both the options but the Surrogate approach is able to handle slightly more data than the Binary format. The Binary format peaked at 48 rows while the Surrogate approach peaked at 50 rows. I like the Surrogate approach as it gives the users complete control on the way data is serialized and has option for further user defined optimizations. Further the option to pick DataSet Binary serialization format is not available in the Compact framework which means Surrogates might be the only viable option (as opposed to writing a new serializer) for devices. Next I will compare the wire size of the actual serialized data.

Here is the link for original Microsoft KB article on the Surrogate approach. http://support.microsoft.com/kb/829740.

We recently updated our official MSF blog with three new Customer Spotlight case study data detailing how customers are using Microsoft Sync Framework to develop new synchronization scenarios. Here are the links to the posts.

Flip Client: http://blogs.msdn.com/sync/archive/2009/02/10/customer-spotlight-interscape-flip-for-the-mobile-sales-workforce.aspx

Sync2DB: http://blogs.msdn.com/sync/archive/2009/02/06/customer-spotlight-sync2db-outlook-to-database-synchronization.aspx

Windows Live toolbar: http://blogs.msdn.com/sync/archive/2009/02/11/internal-customer-spotlight-synchronization-scalability-and-live-toolbar.aspx

Having Windows Live toolbar build on top of MSF to synchronize favorites is a huge testament to the scability of sync knowledge. Live toolbar is currently synchronizing millions of user favorites.

Maheshwar Jayaraman

Cute..very cute.

This is the second post aimed at providing tips for improving runtime memory footprint when using the P2P DbSyncProvier class in V2 of SyncServices for ADO.NET.

First post link: DbSyncProvider- Improving Memory Performance In WCF Based Synchronization

We have had customers report memory usage surging when DbSyncProvider is applying changes from a peer and encounters conflicts. If the number of conflicts increases the memory usage spikes and pretty soon will run out of memory.

After debugging, we narrowed down the issue down to a property in DbSyncTableProgress class: Conflicts. This property was being used by the runtime to “log” conflicts that were skipped during that Sync session. DbSyncTableProgress holds the running count of progress made during a sync session for each sync adapter configured. Whenever the system encounters a Conflict while applying a row it will try to invoke the optional conflict event handler. The runtime will then decide what to do with the row based on the ApplyAction enum returned by that event handler. The default action is ApplyAction.Continue which means the runtime will assume “LocalWins” resolution and just update the local metadata for that row and move on. That is the best case scenario. Now lets see the worst case scenarios: we have two.

1. No event handler registered but row application failed due to Database error.

2. User registered event handler returned an ApplyAction of RetryNextSync.

The first case can happen for a variety of reasons such as errors in user configured TSQL command, typos in user configured Stored proc names or params, error in executing stored proc or database timeouts or other database related errors. The second is just a normal way of saying “dont worry about this row for this sync and we will try this again in the next sync”. Since the P2P provider is based off of Microsoft Sync Framework choosing RetryNextSync adds an exception for that particular row in the sync knowledge and moves on. This exception will tell the source to resend the row during the next sync.

Whenever the above two scenarios happen the runtime will “log” the conflict in the corresponding DbSyncTableProgress for that table. Logging includes cloning the source DataRow and the local DataRow and storing it in DbSyncTableProgress.Conflicts property. So for each conflict the runtime caches the source and destination row. If the number of conflicts increases or the size of each DataRow is large the system will gobble up available memory eventually leading to OutOfMemory exceptions.

This property was carried over from SyncTableProgress type in SyncServices V1. SyncServices V1 was an anchor based hub-spoke synchronization. V1 too had conflict resolution handling and one such option was to skip applying rows. Since the runtime had no way of storing anchors for those skipped rows, those rows would never ever be synchronized again (until a future changed to the same row happens). So, we had to aggregate and log all such conflicts so users can view it at the end of a sync session and take appropriate action. As you can see with the advent of SyncFramework this property is no longer needed as this bookkeeping is automatically done by the SyncFramework API’s.

Workaround:

So there is a simple workaround to avoid this issue. Workaround is to implement the SyncProgress event handler on DbSyncProvider and clear the Conflicts collection on the TableProgress property.

this.peerProvider.SyncProgress += new EventHandler<DbSyncProgressEventArgs>(peerProvider_SyncProgress);

void peerProvider_SyncProgress(object sender, DbSyncProgressEventArgs e)
{
    e.TableProgress.Conflicts.Clear();
}

This will ensure that the collection is always cleared and no extra memory will be used.

Ps: This bug is on track to be fixed for the next release.

Maheshwar Jayaraman

One of my first posts was a link to my managed UI wrapper that let users view/configure/edit SSL certificates on local machines. I had developed it when I worked on the System.Net team as I found it really hard to use the command line version of Httpcfg.exe to configure SSL certs. The tool was developed with the Beta version of .Net 2.0 and I really didnt have time to update it after Whidbey was released. What I didnt realize was that the tool was the most popular search hit on my blog and that the original download link is dead on codeplex. So, I took some time this weekend and resurrected the tool up to latest release versions of .NET.

The latest bits are available at http://maheshwar.net/projects/httpcfgui/httpcfgui.zip.

From my old blog:

"Some Requirements/Information before you download the tool.

1. Tested on Win2k3, WinXP and Vista(needs to run as admin).

2. Requires .NET 2.0 framework.

3. To open the machine store, you need to be the admin on that machine.

4. You need to have httpcfg.exe in your system path. It can be downloaded here. It comes as a part of support tools for win2k3 and XP.  I could have attached the exe but dont think I can redistribute it.

The readme.txt tells you how to use it."

Maheshwar Jayaraman

Recently a commenter posted that they were unable to get Favorities sync to work (or show as an valid option on the toolbar). Found a recent post by a Microsoft dev who debugged the issue. Read more at http://blogs.msdn.com/john_pollard/archive/2008/12/22/fixing-an-issue-with-windows-live-toolbar-and-favorites-sync.aspx. Hopefully this solves your installation issues.

Windows Live team just released the beta for Windows Live Essentials. I wanted to quickly point out the new Windows Live Favorites Sync feature that is part of the Live Toolbar. It now uses Microsoft Sync Framework (File Sync providers to be specific) to synchronize your favorites folder. This is the first Microsoft product release that uses our framework and naturally we are very excited about this. Download the suite and try out the new favorites sync.

Update: Adding links to download

 Direct Links:

Suite: http://download.live.com

Toolbar: http://download.live.com/toolbar

Maheshwar Jayaraman

I recently got a beta invite for the Windows Azure hosting and storage services and immediately set about writing a simple hello world hosted app. I wanted to test both the Web role and the Worker role so I was looking for a simple scenario for that. I liked the new Windows Live feature of displaying a new background image on http://live.com site. They started doing this for the recently concluded Olympics but have since continued to display different images from around the world. I had written a simple windows service that checks the page every day to download any new images. I decided to port this app over to Windows Azure. I could use the Worker role to host my live image “poller” and store any new images in Windows Azure Blob storage. Then I can write a simple web app that will display all images stored in my blog storage.

Step 1: Download Windows Azure SDK and Visual Studio tools

Downloaded and installed Windows Azure SDK and Windows Azure tools for Visual studio.

Even if you don't have a Windows Azure invite, you can download the SDK and try it out by developing against the development fabric. The SDK contains all needed runtime components to host a local development fabric and storage services.

Step 2: Create a new Windows Azure Cloud Service

Launch VS 2008 and started a new Cloud Service project. From the predefined templates, I chose the Web and Worker Cloud Service template. This gave me a cloud service project with 2 roles in it (Web and Worker roles).

Step 3: Code the live search image poller worker role.

The worker role is intended for background processing jobs. Worker roles cannot accept incoming requests but can make unlimited outgoing requests. The roles are run in a sandbox domain which means it doesn’t have any access to local file systems. All storage requirements are to be solved by using one of the three Windows Azure storage services (Blob, Queue and Table). For this sample all I needed to do was store the images in the blob service and add some metadata such that blobs can be rendered by a browser.

Before downloading all images, I needed to create a unique container where I would be storing these image blobs. The Windows Azure SDK ships a very useful REST based API to programmatically access the storage service. Here is the code to do that.

BlobStorage and BlobContainer types are part of the REST storage access API and RoleManager is Windows Azure logging utility. I set the container visibility to public for ease of access. (Don't bother trying to delete content as only GET requests are  anonymous, all other operations require my unique access key)

Primary job of the poller would be to download http://live.com resource and look for the background image. It will then inspect the blob storage to see if the image already exists and if not add it to the container. We also set the blob’s content type to image/jpeg so browsers can render the blob link as an image.

Done. Its that simple. Now details about blobs in the live search container can be accessed via a simple get request to http://maheshwar.blob.core.windows.net/livesearchimages/?comp=list .Here is the list as of writing this post.

Step 4: Code the Web app to display the images.

I just wanted to have a simple ASPX page that just lists all images and their download timestamp. I once again used the REST API’s to download all blobs from my livesearchimages container and data bound the columns to a simple DataGrid.

Here is the code to retrieve all Blobs. To make data binding easier, I just wrapped the contents of my blob in to a LiveImageMetadata class.

LiveImageMetadata type.

And finally the DataGrid definition.

After testing locally, I updated all references to my cloud services url and published the service.

You can view all images downloaded by visiting link http://maheshwar.cloudapp.net/LiveSearchImages.aspx

That looks simple but it was not so straight forward experience moving it from my development fabric to the cloud fabric. I have some feedback to the Windows Azure team on how to make the developer experience simpler.

Some feedback:

1. Whats up with the dependency on SQL Server Express edition? I had Sql Server on my dev box and had to go through manual steps (not straight forward and not documented) to get it to work with Sql Server.
2. The app settings are in two different places when developing for the dev fabric and the cloud fabric. In dev mode, app settings go in web.config and app.config while for the production mode it has to entered in the service configuration file. It would be nice if the “Publish” button did that on behalf of the users.
3. The ConfigurationSettings has to be defined in two parts. All keys needs to be defined in the ServiceDefinition.csdef file and the actual name value pairs defined in the serviceconfiguration.cscfg file. Seems redundant.
4. Once deployed there is no UI indication of the initialization process. It would be good to have a UI (like the one they demonstrated in PDC that showed what state the VM machines are in)
5. No online mechanism to view logs. you have to click “Copy logs” in the configure button then wait till logs are dumped to the blob container. Then you have to use the CloudDrive sample (the sample rocks btw) to copy the logs locally and then inspect them.
6. No log filter. By default it logs all and hence the log XML files gets very large. Especially when every other second you have the “<EventProperty Name="Message">Entered GetHealthStatus()</EventProperty>” message spamming the logs.
7. A TDS proxy to connect to the storage services so we can inspect our data via Sql management studio.

Having said that I see the huge potential Windows Azure has. Since the hosting services has .NET 3.5 installed it means you can host any (WCF/Silverlight/ASPX/ASMX) services on the cloud. I am going to move my Silverlight projects from http://maheshwar.net/bog/projects over to Azure.

Maheshwar Jayaraman

PDC 08 was last week and I wanted to summarize the PDC sync specific announcements. Hope every one had a chance to catch up on the all the impressive videos. First, Windows 7 looks impressive and I cant wait to dogfood the Beta build.

As I posted earlier, we had 3 Microsoft Sync Framework related presentations and here are the direct links to the recorded sessions.

Microsoft Sync Framework Advances – Presenter Lev Novik

Windows 7: Programming Sync Providers That Work Great with Windows – Presenter Jason Roberts

Sync Framework: Enterprise Data in the Cloud and on Devices (Presenter: Liam Cavanagh)

I am involved in Project Huron and wanted to talk briefly about what we discussed at PDC and what’s coming next. Project Huron intends to be a data hub in the cloud enabling data sharing and replication using the Sql Data Services and Microsoft Sync Framework. Here is the full snippet of Project Huron from our Sync blog.

Project Codename “Huron” – Leverage the power of SQL Data Services and Microsoft Sync Framework to build business data hubs in the cloud. Using this cloud based data hub, “Huron” provides a simpler, more convenient and less expensive way to:

· Publish databases to the cloud along with reports, forms and objects

· Subscribe to published data and automatically configure the local database for sync

· Make online changes through SQL Data Services and propagate those changes to subscribed users once they connect

· Enable scheduled and background synchronization of data changes through SQL Data Services and then on to other subscribed users

· Backup and restore of database applications to the cloud

We showed one such scenario of Project Huron at PDC. The PDC demo shows how customers using Access database can scale out and enable collaboration scenarios on the Access database by publishing it to the cloud. Customers interested in sharing or contributing to the data can subscribe to the hosted Access database. All changes are routed and synchronized in a peer to peer fashion through the cloud hub. We also demonstrated the “data hub” nature of the data in the cloud by downloading the same Access data to a Microsoft Compact database with full bi directional sync support. All components showed in the demo were running on live code. We used Microsoft Synchronization Framework V1 to build our AccessProvider, SDSProvider and SqlCESyncProvider. Acess and CE providers run locally on the box while the SDSProvider is running in our sync service running the cloud handling the SDS store. Feedback from PDC is very promising and the Access scale out/collaboration scenario seems to be a very common problem that customers run in to. Infact, we had a couple of guest posts over at the Access team blog and feedback from that blog has also been very promising. You can read the entire post at Announcement- Storing Access apps and data in the cloud and Video demos of Huron - Access and the SQL Server Data Services.

What’s next for Huron

We are looking for early adopters to participate in project Huron which will start very soon. If you are interested in participating then please send an email to DataLabs@Microsoft.com with “Huron beta” in the subject line.

We also announced public availability of the MSF v2 CTP which has some interesting new features built in. More info on MSF V2 CTP1 can be found here. We are very excited about Huron and cant wait to share it with early adopters.

Last week, Mary Jo Foley reported that we re-released Microsoft Sync Framework V1 with updated bits. Since I hadn't seen any internal emails on this I pinged my PM to get more details. Yesterday Liam posted an entry in our Sync blog explaining the "bump" in the published date. Summary: We did not update the bits with any features/bug fixes. Apologize for the confusion.

Maheshwar Jayaraman

We have 3 sessions planned for PDC. Details of the sessions:

Microsoft Sync Framework Advances

Presenter: Lev Novik

This session shows you how the next version of the Microsoft Sync Framework makes it easier to synchronize distributed copies of data across desktops, devices, services, or anywhere else they may be stored.

Sync Framework: Enterprise Data in the Cloud and on Devices

Presenter: Liam Cavanagh

See how synchronization plays a pivotal role in transitioning to a managed cloud environment by creating a central hub of information in the cloud. Using synchronization, organizations can enable more efficient mobile and enterprise-to-enterprise scenarios.

Windows 7: Programming Sync Providers That Work Great with Windows

Moe Khosravy, Jason Roberts

Learn how you can enable your application to synchronize with other applications that use the Microsoft Sync Framework. This session covers how to implement sync for contacts and other PIM data, how to package sync providers for distribution and installation, and how to register sync provider for use on Windows.

All 3 sessions have excellent content and are shaping very well. I have been involved in the second one being presented by Liam Cavanagh. I will discuss more about the work we did after PDC. PDC is mere 2 weeks away. Excited!

Users have frequently wanted the ability to remotely synchronize relational nodes in a peer to peer fashion using SyncServices for ADO.NET. We have a sample demonstrating remote synchronization by using Windows Communication Foundation (WCF). I wanted to use this post to provide an easy way to optimize the performance of this WCF based solution. DbSyncProvider enumerates all changes in a DataSet and this gets applied on the destination provider. The moment the two providers are moved from a local 2-tier model to a remote n-tier model this DataSet has to be serialized and transmitted over the network. DataSet’s are very efficient when the amount of data involved is very small. The moment your data exceeds few hundred rows the amount of memory required to serialize/deserialize this DataSet is quite huge. Further the serialized size of the DataSet on disk is quite big as well.

DataSet object is by default serialized in XML format and serializing/deserialzing this XML data creates a lot of transient objects resulting in a spike in your memory usage. When you have enough data in the DataSet, like synchronizing large number of database rows, your app has the potential to go out of memory deserializing it. Infact users of SyncServices for ADO.NET V2 are quite aware of the OutOfMemoryException when they are synchronizing large number of records in the 2-tier model (PS: Solving this is the highest priority for us in the next release). Using the WCF solution increases the likelihood of this error happening even for mid sized data that fits fine in memory.

There is an easy way to optimize this problem and obliviously it requires users to move away from the XML based default DataSet serialization. Since DataSet doesn’t support any other format, the only way is to use a Surrogate to serialize it. Microsoft Knowledge base has a wonderful article detailing this Surrogate and it can be downloaded from http://support.microsoft.com/kb/829740.

Download it and use it in your WCF based Synchronization apps and you should see vast improvement in your memory usage and performance.

I wrote a quick sample checking the process peak memory usage during serialization/deserialization of a DataSet. Here are the comparison numbers between default and surrogate serialization.

The DataSet contains one DataTable which contains one string column. Each column is a 5Mb string. Size is Peak working set.

Point proven :). As you can see for each 5Mb of data added in memory the default serialization takes anywhere from 6-8 times more memory. For 50 rows (approx 250Mb data size in memory) the serialization step itself fails with OOM.