Microsoft Sync Framework

3 (more) New Sync Samples - Sync 101 for Sync Framework SDK v1.0

We have a couple more synchronization samples for you to take a look at.  These C# samples are based on Sync101 which is a sample template that allows us to show specific concepts relating to Sync Framework.  All of these samples have been updated to work with the latest v 1.0 release of Sync Framework SDK . I am particularly excited about the remote change application sample (RCA) as it has been a common request to understand how to integrate Sync Framework with technologies like WCF and enable a service to manage the server provider componets of the Sync Framework.   Each of the samples use the the SQL MetadataStore store for the metadata information for each replica database. This greatly simplifies the logic required to implement data synchronization between data stores. In these samples three in memory data stores are used to show how data collaboration can be accomplished and how the Sync Framework helps to minimize P2P complexities such as data conflicts / resolution and data deletions.

The following are the samples that were just added to the Sync Framework Code Gallery.

Sync101 with Metadata Store v1.0
This sample uses the Metadata store to show the basic concepts that are required when building a Sync Framework application and illustrates many of the core concepts such as synchronization providers and the synchronization orchestrator. 

Sync101 RCA with Metadata Store v1.0
Expanding on the previous sample, this samples outlines the "remote change application" (RCA) pattern used over WCF.

Sync101 with Change Units
Expanding on the first samples, this sample illustrates how a data store can use change units to partition data and send only data that has changed. This sample also illustrate explicit transactioning (batched instead of per-update).

Liam

Now Available - SyncToy 2.0 File Synchronization

SyncToy 2.0 has just been released to the Microsoft Download center. SyncToy 2.0 for Windows is available as a free download and is an easy to use, customizable application that helps you copy, move, rename, and delete files between folders and computers. SyncToy is powered by Microsoft Sync Framework and is yet another example of how Sync Framework helps to create end-to-end solution for enabling synchronization of any content or data.

Based on the feedback from the SyncToy Forum and Windows XP Professional Photography web site, the following features were added to this release of SyncToy:

• Dynamic Drive Letter Assignment: Drive letter reassignment will now be detected and updated in the folder pair definition. 
• True Folder Sync: Folder creates, renames and deletes are now synchronized for all SyncToy actions. 
• Exclusion Filtering Based on Name: File exclusion based on name with exact or fuzzy matching. 
• Filtering Based on File Attributes: The ability to exclude files based on one or more file attributes (Read-Only, System, Hidden). 
• Unattended Folder Pair Execution: Addressed issues related to running scheduled folder pairs while logged off. 
• Folder Pairs With Shared Endpoints: Ability for folder pairs associated with the same or different instances of SyncToy to share end-points. 
• Command line enhancements: Added the ability to manage folder pairs via the command line interface. 
• Re-Architect Sync Engine: The SyncToy engine has been rearchitected to provide scalability and the ability to add significant enhancements in future releases. 
• Sync engine is also more robust insomuch that many single, file level errors are skipped without affecting the entire sync operation. 
• Sync Encrypted Files: Sync of Encrypted files works when local folder and files are encrypted, which addresses the common scenario involving sync between local, encrypted laptop PC folder and remote, unencrypted desktop PC folder. 
• 64-Bit Support: SyncToy now has a native 64-bit build (x64 only) for 64-bit versions of Windows. 
• Folder pair rename 
• Sub-folder Exclusion Enhancements: Descendents created under excluded sub-folders are automatically excluded. Usability improvements for the sub-folder exclusion dialog. 
• Folder Pair Metadata Moved: Folder pair metadata removed from MyDocuments to resolve any issues with server-based folder pair re-direction setup. 
• Setup Improvements: Integrated setup with single self-extracting archive file and no extra downloads if you already have .NET Framework 2.0 installed. Enabled silent install for the SyncToy Installer file (see readme.txt file for more information).
• Removed combine and subscribe actions.

SyncToy 2.0 is available for download from: http://www.microsoft.com/downloads/details.aspx?familyid=C26EFA36-98E0-4EE9-A7C5-98D0592D8C52&displaylang=en

Liam

2 New Sync Samples - Download Only & Adapter Builder

We have just posted the following two new samples to the Sync Code Gallery today for Sync Services for ADO.NET v2: Download Only Sync This sample demonstrates two-tier download-only synchronization using Microsoft Sync Services for ADO.NET v2.0. The sample solution is composed of two projects: Client, which includes the UI form, data access code, and synchronization agent code for Sync Services; and SampleServerSyncProvider, which includes server synchronization provider code for Sync Services. For more information, see the code comments that are included in each file.

Sync Adapter Builder
This sample utility demonstrates how to create synchronization commands by using the SqlSyncAdapterBuilder in Microsoft Sync Services for ADO.NET v2. The sample solution is composed of one project that includes the UI form, data access code, and SqlSyncAdapterBuilder code. The SqlSyncAdapterBuilder is typically used at run time to generate synchronization commands, which are in the form of SQL queries. This provides an alternative to writing these commands manually at design time. In this sample, we use the SqlSyncAdapterBuilder to output synchronization commands. You can output commands that work with SQL Server change tracking (available in SQL Server 2008) or a custom change tracking system. Running this application enables you to see how the commands are structured, and to copy those commands for use in applicatons where you will specify commands manually.

Liam

Sync Framework for Windows Mobile (Devices) - CTP1

Right on the heels of our launch of Microsoft Sync Framework v1, we have just posted our first CTP for Microsoft Sync Framework for Devices. As of right now, you can download this CTP for Windows Mobile from the Microsoft Download Center.  As we have shown in previous webcasts and conference demos, this is great for enabling synchronization from devices to data stores such as SQL Server Data Services or even corporate web services.  Perhaps even to write an offline SmugMug application for synchronizing your device photos with your SmugMug account.

Not to be confused with the previous CTP release of Sync Services for ADO.NET for Devices (a component of the Sync Framework) that is specifically meant for synchronizing an ADO.NET enabled server database with a SQL Server Compact device database, this CTP release of Sync Framework for Devices includes the Core Sync Framework runtime that gives the ability to exchange information with any data store or file system on the device, and not just SQL Server Compact databases.

This release contains only the core sync engine you would find in “synchronization.msi” in the PC release. Other features such as the metadata store and file sync provider are not included: some of the omitted features don’t necessarily make sense on devices, others will appear in a future CTP. We are looking for feedback on which are which. In particular, we want to know: what are your scenarios for using the different aspects of the Sync Framework on a device?

We also decided to produce the CTP for a limited platform set: Windows Mobile 5 and 6 for ARM processors. Supporting this subset allows us to give a CTP usable by a very large percentage of mobile developers while still limiting the test matrix for this first CTP. We expect to expand the supported devices and platforms for RTM and we are considering the scope of that as we go forward.

So what exactly is in this release?

Microsoft Sync Framework core runtime

• Managed (.NET Compact Framework) and unmanaged/native (ARM)
• Supported platforms are Windows Mobile 5 and 6 (ARM processors only in CTP1)

Installer with Visual Studio integration

• Support for adding a reference to the Sync Framework in a mobile project using Visual Studio 2008

As always, we look forward to your feedback on the Sync Framework Forum.

Liam

Sync Case Study - ServiceU Corporation & Event Management

One of the best parts about getting our first version of Sync Framework out the door is that we can now start talking about some of the customers we have been working with over the past few months and some of the great solutions that they are building. 

Beyond the customers I mentioned earlier (SmugMug, InterScape, Fujitsu Siemens) I also wanted to point you to another customer called ServiceU Corporation.  This company is a leader in the field of supporting online and on-demand event management. As you will read in the case study:

"Its software services, created and deployed using the Microsoft® Application Platform, are used by churches, schools, universities, theaters, and businesses to manage concerts, conferences, and other events. As a Payment Card Industry Level 1 Service Provider, the company is interested in security and performance, which is why it is upgrading to Microsoft SQL Server® 2008. ServiceU is using a number of new SQL Server 2008 features including enhanced auditing, and Policy-based Management to enforce configuration policies. The company is also using the new Performance Data Collection and Plan Freezing features to enhance performance, and the Data Compression and Backup Compression features to reduce the cost of disk-based storage".

To help power this solution, ServiceU chose Sync Services for ADO.NET:

"Working from Visual Studio 2008, developers can use Microsoft Synchronization Services for ADO.NET 2.0 to synchronize data from disparate sources over multi-tier and service-based architectures. ServiceU likes this feature because it is interested in deploying more solutions that run on mobile and remote devices. “Whenever you have server-based, data-driven applications coordinating data with mobile devices, synchronization is essential,” says Smith. “Synchronization Services will be a great help as we develop new occasionally-connected solutions to offer to our customers.”

Over the next few months we will be talking more and more about some of the great companies that are using Sync Framework and Sync Services for ADO.NET.

Of course, given that our technology is free and readily downloadable from the web, we don't hear from every customer.  As such, if you are using any of these technologies please let me know (at the Email link on the sidebar) so we can share your success with the rest of the community.

For more details on this case study, please go to: http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000001258

Liam

Top New Features in Sync Services for ADO.NET v2

Today’s release of Microsoft Sync Framework includes the latest version of Sync Services for ADO.NET.  A great deal of work has gone into this release and I thought I would take some time to point out some of the top new features in version 2 since our previous release.

• Peer-to-peer synchronization support:

Beyone the traditional hub-and-spoke synchronization support built into Sync Services for ADO.NET.  v2 now supports peer-to-peer synchronization.  The focus with this part of the API is collaborative applications. For example, in an application that allows users to share project notes, project team members often require a local copy of data that they can work with. When they have made changes, they can synchronize with another team member to exchange changes. 

• Sync Services is now based on Microsoft Sync Framework:

Sync Services now uses Microsoft.Synchronization.dll. If you installed Sync Services by using SyncSetup.exe, this DLL is already installed. This enables Sync Services for ADO.NET to take advantage of all the features and capabilities built into the Sync Framework.

• Synchronize with devices: 

Sync Services for ADO.NET 1.0 enabled synchronization between a server database and a SQL Server Compact 3.5 database on the desktop. Now, you can also synchronize between a server database and a SQL Server Compact 3.5 database on a device.

• Support for SQL Server 2008 Integrated Change Tracking:

If you are using SQL Server 2008, Sync Services for ADO.NET has the ability to take advantage of the new SQL Server integrated change tracking feature. This feature addresses many of the issues of custom-tracking systems and provides a straightforward way to track changes. Change tracking is also supported by the Local Database Cache in Visual Studio 2008 SP1. Developers can now specify that the Configure Data Synchronization wizard should enable SQL Server change tracking on the server and generate the commands necessary to select and apply changes to the server database. Unlike custom change tracking systems, SQL Server change tracking does not require any schema changes in the server database.

• Support for the new data types in SQL Server 2008:

Sync Services for ADO.NET has added support for a number of new data types including those added in SQL Server 2008.

• The ability to trace the synchronization process: 

In distributed applications, tracing can be crucial because it enables you to troubleshoot issues that might otherwise be difficult to identify. Sync Services includes tracing for the client and server synchronization providers.

Liam Cavanagh

Announcing Availability of Sync Framework v1 and Sync Services for ADO.NET v2

I am happy to announce the general availability of Sync Framework v1 and Sync Services for ADO.NET v2.  As of today you can download the Sync Framework SDK in 11 languages including Chinese (Hong Kong), Chinese (Simplified), English, German, French, Italian, Japanese, Korean, Portuguese, Russian and Spanish for AMD64, IA64 and x86 processors from the Sync Framework Download Center. 

What is Sync Framework?

Imagine being able to build a solution that seamlessly exchanges contact information between Outlook, a database contact management application, your mobile device and your service based contact management system.  Or how about a mobile device that connects with other devices to exchange pictures and videos.  How about being able to take data from any of your enterprise databases, file or enterprise systems and make it available offline for users to modify and sync back up to the enterprise.  All of these capabilities are possible with the Sync Framework and best of all, it is free on Windows platforms and licensable on non-Windows platforms! 

Just a few examples of companies that are already using the Sync Framework include:

• SmugMug who has added support for the Sync Framework to enable developers to build rich offline applications that can allow photos to be easily shared on friends and families local computers. 
• InterScape have embedded Sync Framework into their Customer Relationship & Management (CRM) solution to enable sales people to synchronize enterprise files and data for offline access. 
• Fujitsu Siemens will embed the SyncToy file synchronization solution (powered by Sync Framework) on their STORAGEBIRD external drives to enable seamless file synchronization between devices and computers. 

Sync Framework Highlights 

Some of the highlights of this release include:

• Sync support to new and existing applications, services, and devices
• Collaboration and offline capabilities for any application
• Roam and share information from any data store, over any protocol, and over any network configuration
• Leverage sync capabilities exposed in Microsoft technologies to create sync ecosystems
• Extend the architecture to support custom data types including files

Sync Services for ADO.NET v2 Highlights 

Sync Services for ADO.NET is a Microsoft Sync Framework powered solution for synchronizing ADO.NET enabled databases in offline and collaboration scenarios. Sync Services for ADO.NET allows developers who are familiar with the concepts of ADO.NET to apply that knowledge to data synchronization through a very similar set of APIs to that of ADO.NET. Sync Services for ADO.NET provides the flexibility of a programming model like offline datasets and a richer synchronization feature set like that found in Merge replication. Sync Services for ADO.NET also supports synchronization over services, such as Windows Communication Foundation (WCF).

Some of the Sync Services for ADO.NET highlights include:

• Offline Database Synchronization: With Sync Services for ADO.NET you are able to build a solution where multiple remote clients connect and synchronize to a central ADO.NET database in a Hub-and-Spoke configuration. This enables occasionally connected devices to periodically connect and synchronize changes with a central ADO.NET database server. This topology is a common solution for remote workers such as sales reps or field service workers.
• Collaboration Between Databases: Sync Services for ADO.NET also includes Peer-to-Peer capabilities. Through a custom Peer provider, collaboration between two or more SQL Server databases can occur (support is not available for SQL Server Compact). Unlike a Hub-and-Spoke architecture, this provider enables a SQL Server database to communicate and exchange information with any other SQL Server database. This type of scenario is useful in group scenarios where users (such as auditors) need to update information and then collaborate those changes with other group members.
• SQL Server 2008 Integrated Change Tracking: If you are using SQL Server 2008, we recommend that you use the SQL Server change tracking feature. This feature addresses many of the issues of custom-tracking systems and provides a straightforward way to track changes. Change tracking is also supported by the Local Database Cache in Visual Studio 2008 SP1. Developers can now specify that the Configure Data Synchronization wizard should enable SQL Server change tracking on the server and generate the commands necessary to select and apply changes to the server database. Unlike custom change tracking systems, SQL Server change tracking does not require any schema changes in the server database. For more information, see the Visual Studio 2008 documentation.

Sync Framework Pricing

Sync Framework will be licensed free on Windows platforms.  In addition, we are also licensing the specifications and a source code porting kit to developers who want to implement Microsoft Sync Framework solutions on non-Windows platforms. 

To learn more, visit http://msdn.microsoft.com/sync

I also want to send a special thank-you to those of you who have provided feedback throughout our CTP's and have helped make this a great release!

Liam Cavanagh

Using WCF for Communcation in a Mobile Sync Application

In the past few months we have seen a number of posts in our sync forum that have shown questions and issues relating to the use of WCF as the transport layer for a Windows Mobile Sync Services for ADO.NET application.   As such, it seemed prudent to post a note with some tips for ways to work around some of the limitations of WCF in the .NET Compact Framework when using devices.

Tips

1)      Force the use of [XmlSerializerFormat] for the contract interfaces, as shown in the Iservice.cs code attached below.

2)      Force the use of basicHttpBinding for your contracts, as shown in the attached web.config file.

3)      Modify the netcfSvcUtil.exe generated service.cs code to remove all the duplicated types that are already defined in the Microsoft.Synchronization.Data.dll. This is most likely the reason users will encounter an InvalidCast exception.

4)      Deal with Collection<string> and string[]. as you can see in the sample code below, the attached sample uses string [] to ensure it is consistent in the GetSchema(). This is most likely the cause of an ArgumentException when calling this method.

5)      Setting the includeExceptionDetailInFaults="true" flag in the web.config can be helpful for debugging, and some logging logic at the server side could be helpful also as shown in the server side code.

6)      When using netcfSvcUtil.exe, just use the default option, to generate the client side code.

7)      The following blog is helpful for more information on using WCF on devices: http://blogs.msdn.com/andrewarnottms/archive/2007/09/13/calling-wcf-services-from-netcf-3-5-using-compact-wcf-and-netcfsvcutil-exe.aspx

Additionally, the following WCF sample code will hopefully provide some help to ease some of your frustration.

Yunwen Bai

Creating an Open Community around Sync

One of the key goals in the Sync Framework team has always been to try to build an open community around synchronization.  As many of you know, the Sync Framework has been built such that anyone can create providers to tie any end points together into a sync ecosystem.  Up until now, most of the providers (database, file system, feedsync) have come included within the Sync Framework and have been written by us here at Microsoft.  I was especially pleased to read a post in one of our Sync forums by Hans Hinnekint who has started a new Codeplex site called File Synchronizer that has the purpose of creating a utility that will synchronize file folders using the Microsoft Sync Framework.  For those of you who do not already know, CodePlex is Microsoft's open source project hosting web site.

In the future, I hope we will see more open source projects like this one that will enable new endpoints to be linked through synchronization and thanks to Hans for kicking of the process!

Liam Cavanagh

P.S. If you know of any other open source projects that have been started based on the Sync Framework, I would love to hear from you.

Using Notifications to Enable Background Synchronization

Once in a while I get asked if the Sync Framework supports background synchronization based on network states.  To this question I usually respond, that there is no direct support within Sync Framework to do this, however there are a lot of existing API's to help you enable this.  One example of this is the State and Notification API's (SNAPI) built into Windows Mobile to monitor network activity.  Using this API, you can embed functionality to monitor the network state and initiate background synchronization based on connectivity.  You can see an example of this in a notification sample I wrote about a year ago.

Using SQL Server 2008 Integrated Change Tracking to Optimize Data Synchronization

I thought I would take the chance to talk to you a little bit about a major enhancements we have made with synchronization in SQL Server 2008.  The feature I am referrring to is called SQL Server 2008 Integrated Change Tracking.  The idea behind this feature is that it enables SQL Server to track data changes to your database rather than relying on you to create a change tracking technique.

Why is this feature so great?  Well in my experience I have found that DBA's are less then enthusiastic when you tell them that in order to track changes, they will need to add triggers to each of the tables.  Then they will either need to add an additional tracking column to that table (or to a separately linked table) and will need to create one tombstone table per table to track deletes.  Oh, and by the way these triggers will need to be fairly complicated because you really need to deal with all of the isolation issues that come along with proper change tracking... and cross your fingers that these changes do not affect your existing applications.  Not a fan of that idea?  I am not surprised.

Well with SQL Server 2008, you can now simply turn on an option to enable change tracking.  Then you tell SQL Server which tables you want it to monitor.  From that point SQL Server will start monitoring these changes and store them in a separate change tracking table which you have access to.  If you want to get the changes since a certain point in time you simply join the ChangeTable to your base table and presto you have all of the inserts, updates and deletes. 

As an added bonus, this feature has been integrated into the existing Visual Studio SP1 beta that is now available for download.  Guy Burstein has a great summary of this feature and I am going to steal one of his screen shots below.

Liam Cavanagh

Sample - SQL Express Client Synchronization using Sync Services for ADO.NET

We have just posted a new sync sample to our Code Gallery that demonstrates how to enable offline data collection applications to synchronize a SQL Express client database to another SQL Server database using the Synchronization Services for ADO.NET library.  In this scenario, SQL Express client databases can be used as client databases to a central ADO.NET enabled database.  Unlike the existing peer-to-peer synchronization SQL Express provider sample which you may have seen, this sample is optimized for a hub-and-spoke topology.

In this sample, two tables (orders and order_details) are on the server database and also on the local client database.  The sample synchronizes edits to these table to keep their data identical.
This sample demonstrates:

• Using SQL Express to cache changes for a client application.
• A customized SQLExpressClientSyncProvider class that wraps around Microsoft.Synchronization.Data.Server.DbServerSyncProvider.

I would like to point out that even though we have done a fair amount of testing in house with this sample, it is still only posted as a sample.  In the future we certainly plan of including a fully supported version of a SQL Express client provider within a future release of Sync Services for ADO.NET. 

With that, we certainly want to hear your feedback on this sample and please feel free to post comments to our Sync Services for ADO.NET forum.

Liam Cavanagh

Extending Visual Studio 2008 SP1 Sync Designer to Support Bi-Directional Synchronization

The following is an excerpt from the documentation updates that will be occurring for Visual Studio 2008 sp1.  We have decided to post this content early due to the significant amount of interest expressed around this particular topic.  Specifically, this section describes how one would go about extending the sync designer generated code to support bi-directional sync scenarios where changes made on the client-side database are pushed back up to the server.  The intent of this documentation is to provide an introductory walk-through and samples. For a more in-depth discussion around bi-directional synchronization, please see the following run-time docs:

http://msdn.microsoft.com/en-us/library/bb726007(SQL.100).aspx
 
Please let us know if you have any feedback.
 
Sean Kelley

Video - Building Offline Enabled Applications

At TechEd 2008 Developers in Orlando a number of our mobility specialists and Microsoft MVP's got together to discuss the concept of building offline applications for mobile and remote users.  In this panel we discussed various concepts including optimizing data synchronization, scalability as well as techniques to improve the user experience of your mobile applications.  You can view this video recording in various formats here:                    Building Offline Enabled Applications [MP4] [WMV Hi] [WMV Lo]

Panel Members:

• Ginny Caughey - Microsoft MVP, Wasteworks
• Steve Lasker - Sr. Program Manager, Microsoft SQL Server Compact
• Rob Tiffany - Microsoft Mobility Architect
• Rob Relyea - Architect, WPF Team
• Liam Cavanagh - Sr. Program Manager, Microsoft Sync Framework

Announcing Sync Framework v1.0 RC0 and Sync Services for ADO.NET v2.0 RC0 [Updated]

With the completion of TechEd 2008 for Developer we have one last big announcement to make.  A Release Candidate (RC0) of Sync Framework v1.0 and Sync Services for ADO.NET v2.0 for Desktops and Servers is now available.    These components can be found as a top level feature within SQL Server 2008 RC0.  RC0 is the final step before SQL Server 2008 RTMs in Q3 of this year. SQL Server 2008 RC0 can be downloaded from the Microsoft Download Center at:   SQL Server 2008 Release Candidate 0

For MSDN and TechNet Plus subscribers, after logging into their respective accounts, subscribers can view Product Keys and download SQL Server 2008 RC0 from the following links:

Download SQL Server 2008 RC0 (MSDN subscribers)

Download SQL Server 2008 RC0 (TechNet Plus subscribers)

As we announced at TechEd, we are also targeting a CTP (Community Technology Preview) of the Sync Framework on devices for Q3 2008.