Microsoft Sync Framework

SyncToy 2.1 Available for Download

2009-11-11T04:21:00Z

SyncToy 2.1 has just been released to the Microsoft Download center. SyncToy 2.1 is a free application that synchronizes files and folders between locations. Typical uses include sharing files, such as photos, with other computers and creating backup copies of files and folders.

SyncToy2.1 is powered by the latest synchronization engine from Microsoft Sync Framework 2.0 and provides better performance and robustness. The new features and improvements included in SyncToy 2.1 release are:

Better Performance: The speed of file copy operations is significantly increased across the board.
Improved Robustness: Much more resilient to transient network and file system errors, and better error reporting that pinpoints which file the sync failed on in case there’s a fatal error that stops the sync.
Folder pair configuration backup: Folder pair configuration is automatically backed up under %localappdata%\microsoft\synctoy\2.0. You can replace SyncTopDirPairs.bin with the backup copy to resolve the last saved configuration.
Bug Fixes:

o Fixed the data corruption issue when using SyncToy with NAS drives.

o Fixed the bug that prevented uploading files to SharePoint when using SyncToy 2.0.

o Fixed the bug that prevented delete changes from being synchronized when the sync option was set to “Echo”.

SyncToy 2.1 is now available for download from: http://www.microsoft.com/downloads/details.aspx?familyid=C26EFA36-98E0-4EE9-A7C5-98D0592D8C52&displaylang=en.

SQL Pass Conference & Synchronization to SQL Azure

2009-11-04T17:34:00Z

Yesterday was the first day of the SQL Server conference, SQL Pass, here in Seattle. It was a really exciting day for me because I had the opportunity to present our new synchronization capabilities to the cloud in the opening day keynote and in a subsequent session.

We introduced the first piece of Project “Huron”, that is a technology based on the Sync Framework enabling people to use Windows Azure as a central Data Hub for all information. To accomplish this, later this month we will make available for download an early preview of a tool that provides the ability to extend a SQL Server on-premises databases to the cloud. Once configured users can then easily extend data from the cloud to mobile users and remote offices.

Using this tool and with the help of one of our partners, Archetype, we showed how companies can easily extend their on-premises SQL Servers to SQL Azure using data synchronization, allowing the data stores to co-exist and interoperate seamlessly.

All of this can be setup using a wizard. Then from an automatically generated SQL Agent, the synchronization process is executed periodically to move the incremental changes between SQL Server and SQL Azure.

Here are a few screenshots of the tool:

Watch for more details over this month at the Professional Developer Conference and in this blog for other new capabilities we are providing to for integration to Windows Azure and SQL Azure.

Liam Cavanagh

New whitepaper: Minimum Database Permissions Requirements for Microsoft Sync Framework

2009-11-02T23:11:00Z

Are you trying to set up a sync-ready database with the lowest (i.e. most secure) permissions possible? Have you set it up, but are worried you might have opened things up too wide?

Perhaps our just-published whitepaper on this very topic can be of assistance: http://msdn.microsoft.com/en-us/library/ee730363.aspx

New Sync Framework Samples

2009-10-26T22:11:00Z

We have lots of new samples on Sync Framework Code Gallery. These samples demonstrate some of the most important and exciting features of Sync Framework 2.0. If you are new to Sync Framework or are looking for ways to write code to solve your synchronization problems, chances are we have a sample that can get you started. Here are some of the highlights:

Sync101 Samples

This set of samples illustrates various concepts and features needed to synchronize a custom data store, such as:

Sync101 with Simple Providers, which shows how to write a basic synchronization application that uses simple custom providers.
Sync101 with Constraint Conflicts, which shows how to write a standard custom provider that uses Windows Communication Foundation (WCF) to synchronize data between replicas on different computers.
Sync101 with Custom Filtering, which shows how to track custom filters and how to use a standard custom provider to send changes from a filter-tracking replica to two different filtered replicas.

Database Synchronization Provider Samples

This set of samples illustrates various concepts around using database synchronization providers to synchronize between SQL Server databases and SQL Server Compact databases, such as:

Database Sync: SQL Server and SQL Server Compact, which shows how to use database synchronization providers to configure and execute peer-to-peer synchronization between a SQL Server database and one or more SQL Server Compact databases.

File Synchronization Provider Sample

This sample shows how to use the file synchronization provider.

File Synchronization Provider Sample, which shows how to use the file synchronization provider, a fully functioning provider that helps an application to synchronize files and folders in NTFS, FAT, and SMB file systems.

Windows 7 is out -- and with it comes Windows Sync, powered by Microsoft Sync Framework!

2009-10-22T09:00:00Z

It's finally here!

Today is the long-awaited launch date of Windows 7, and on the Sync team we couldn't be more excited, as it is the first Windows client OS to ship with Microsoft Sync Framework running out-of-the-box. This special version of Sync Framework is powering a new feature called Windows Sync, which allows Windows users to manage their sync relationships through a user interface.

Under the hood, Sync Framework is running, and each sync relationship is being governed by custom providers that were developed to handle things like keeping contacts, calendars, notes and tasks up-to-date between the user's Windows machine and their smart phone.

Are you a developer that would like to make it easy for users to manage and synchronize with data stored on your product using the brand new Windows Sync feature? Click the below link and get cracking!

http://msdn.microsoft.com/en-us/library/dd317274(VS.85).aspx - Windows Sync Developer Reference

SharePoint 2010, Windows 7 and PwC Now Integrate Microsoft Sync Framework v2.0

2009-10-19T18:17:00Z

As we announced earlier today, Microsoft Sync Framework V2.0 is now available for public download.

Along with this release I wanted to outline a number of companies and partners that are already using Sync Framework v2.0 including Windows 7, Microsoft Office 2010 and PricewaterhouseCoopers (PwC).

SharePoint has also chosen to integrate Microsoft Sync Framework as part of their solution for synchronizing to Office SharePoint 2010. The SharePoint and Office teams will use the Sync Framework as the engine powering synchronization between Office Workspace (Groove) and SharePoint. In addition, the SharePoint server will also expose Sync Framework API’s that will allow any developer to build providers to integrate new data stores into SharePoint.

"The Sync Framework will greatly reduce the amount of code that an ISV needs to create, and this represents probably the most painful code that they otherwise would have had to write".
Maxim Lukiyanov, Program Manager SharePoint Server, Microsoft

For more information on this please see our new case study: Microsoft Sync Framework used to Create Microsoft SharePoint Workspace 2010

The Windows 7 Device Stage gives users a common user interface to manage and synchronize data with their Windows 7 enabled phone. Contacts, calendar, tasks and even notes can be synchronized with any data store. Sync Framework v2.0 is the easiest way for developers to implement data synchronization providers for any store to enable it for the Windows 7 Device synchronization experience.

"The flexibility of Sync Framework and its ability to track and exchange data really set this technology apart. Since Sync Framework is data agnostic this allowed us to develop a framework tailored around contact, calendar, tasks and notes data".
Anna Boyd, Windows 7 Engineering, Microsoft

PricewaterhouseCoopers has put a system in deployment for 60,000+ mobile auditors in 150 countries to allow much of their work to be done remotely at the client site, making disconnected, offline use of their application and friction free collaboration across teams.

"Our implementation of Sync Framework ‘just worked’ for our auditors. If they set automatic synchronization, our software finds peers, verifies access rights to the same engagement database, and updates all databases. They might be connected to each other over a client’s network, or just via ad hoc (no access point) WiFi. If just one laptop has access to the servers, that laptop automatically funnels all changes from the laptops to the server and vice versa. No manual configuration is needed".
Dr. Glenn Ricart, Technology and Development Leader, PricewaterhouseCoopers

Liam Cavanagh

Sync Framework 2.0 Available for Download

2009-10-19T15:27:00Z

I am happy to announce the availability of Sync Framework 2.0. Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

Adds features that cater to new scenarios or scenarios that were difficult to support.
Reduces the amount of work required to develop providers.
Supports more data sources with new built-in providers.

The major new features and improvements included in Sync Framework 2.0 are:

In Core Components:

Simple Providers: Reduce the amount of work required to develop providers, especially when the data source has very few synchronization-related capabilities, such as change-tracking. The goal is to enable a developer to write a provider without having to become a synchronization expert. The majority of the code that is required for a simple provider is limited to that responsible for interacting with the data source. Simple providers support many complex synchronization scenarios, such as filtering, concurrency and constraint conflict handling, and anchor-based enumeration.
Flexible Filtering: Change unit filters and custom filters have been added to the custom provider components. These filters, in addition to the item filters from Sync Framework 1.0, enable you to filter your data in whatever way is most useful. Filter negotiation allows the source and destination providers to negotiate the filter used during the synchronization session. Change unit filters enable a provider to specify that, while the replica it serves stores all items in the synchronization scope, the replica stores a reduced representation of each item. For example, when an item is a contact a device might store only the name and phone number of the contact as compared to the complete contact stored on a PC. Custom filters enable you to define a filter in whatever way is most appropriate, including filters that allow an item to move into or out of the filter as the item data changes. For example, a media storage device stores only songs that are rated as three stars or better. When the rating on a song changes from four stars to two stars, the song moves out of the filter. Filter negotiation allows a destination provider to specify the filter to be used by the source provider during change enumeration; the source provider can accept or reject a filter. If a source provider does not support the requested filter then the destination provider can choose to receive all of the data and do the filtering itself.
Improved Conflict Handling: Additional conflict scenarios are supported, such as reporting constraint conflicts, managing logged conflicts, and resolving conflicts by using a last-writer-wins policy. Constraint conflicts are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. Sync Framework includes components to help resolve constraint conflicts and apply the resolution to the destination replica. Conflict logging is enhanced by Sync Framework components that help manage the log, such as by removing obsolete conflicts. A last-writer-wins conflict resolution policy can be used to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this policy by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change.
Data Conversion between Providers: In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
Change Application Service: Increases the flexibility and usefulness of the change applier component of Sync Framework. By performing the same actions as the standard change applier, but in a more granular way, the change application service gives a destination provider the ability to use only the features that it requires.
Tracing: Enables you to trace the execution of several components, which is useful during application debugging. This download provides the .tmf files that can be used to view a trace, flush the logger, and format the binary trace file. For more information about how to trace Sync Framework execution, see “Tracing Sync Framework Components” in the documentation that installs with the Sync Framework SDK.

In Database Providers:

New Database Providers (SQL Server and SQL Server Compact): Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore.
Robust Memory-Based Batching: Previous versions of Sync Framework and Sync Services for ADO.NET provided a way for developers to define their own batching logic but there were a lot of limitations, including significant complexity, excessive chattiness, out of memory issues, and restrictions on usage. Sync Framework 2.0 addresses all of these issues by providing a more complete and robust batching API. Developers no longer have to write batching logic themselves because Sync Framework divides changes into batches based on several properties in the API. Batches are now defined by memory consumption rather than the number of rows synchronized, which has eliminated out-of-memory issues for most common scenarios.
Provisioning and Management APIs: Provisioning and initialization activities that were previously exposed only through Visual Studio tooling have now been added to the database provider APIs. This includes the ability to provision an existing database by adding the change-tracking tables and triggers that are required by Sync Framework. It also includes the ability to start with an empty database, create the user schema, and provision that schema based on another server or client database that has already been provisioned.
Performance Improvements: The new database providers in this release have been thoroughly tested in large-scale scenarios in which a single server supports thousands of clients with hundreds of concurrent synchronization operations. This testing resulted in a large number of internal performance improvements that enable Sync Framework database providers to perform as well as other Microsoft technologies like Remote Data Access (RDA) while offering a wide range of capabilities that compete with end-to-end solutions like merge replication.

In File Synchronization Provider:

· Data Conversion between Providers: The data transfer interface used by the file synchronization provider is now available in managed code. A custom provider that synchronizes some other data type can use the data transfer interface to convert its data to synchronize with a file synchronization provider.

· Better File Transfer Performance: Improvements to the way file data is copied allow file data to be transferred up to 30% faster than in Sync Framework 1.0.

· More Robust Synchronization: Instead of failing the entire synchronization session when a single file cannot be synchronized (including failures caused by network issues), the failure is flagged and the synchronization session continues.

To learn more about Sync Framework or to download Sync Framework 2.0, please visit the Sync Framework Developer Center.

Sync Framework and our upcoming conferences

2009-10-05T19:31:00Z

The next few months are going to be very busy for us here in the Sync Framework team. Not only do we have Sync Framework v2 coming out shortly, we also have a number of conferences where we will talk about the work we have been doing with some of external companies and internal groups here at Microsoft to allow developers to plug-in synchronization to systems like SQL Server and SQL Azure.

Starting in November, we will be at SQL PASS, and then at the PDC for the launch of SQL Azure. Last week I recorded a Channel 9 video along with Buck Woody and Michael Rys on the upcoming SQL PASS Summit. You can view the video here:

http://channel9.msdn.com/posts/LarryLarsen/Upcoming-SQL-PASS-Summit-1/

Here is a summary of the video:

Join thousands of SQL Server & BI professionals at the Washington State Convention & Trade Center in Seattle from Nov. 2-5 for the largest SQL Server conference in the world! PASS Summit offers 168 technical sessions presented by SQL Server experts, one-on-one time with Microsoft’s SQL Server engineering team, free troubleshooting & design architecture guidance from Microsoft consultants, keynotes from Microsoft’s senior executives and much more!

Senior Data Platform Technology Specialist Buck Woody stopped by the Channel 9 studio to talk to Michael Rys, Principal Program Manager in SQL Server Engineering Team, and Liam Cavanagh, Senior Program Manager, about upcoming sessions you won’t want to miss this year’s PASS Summit conference.

Liam Cavanagh

SQL Services is now SQL Azure

2009-07-09T01:58:00Z

I thought I would help get the word out on this one. Given that we have been spending so much time on Project "Huron", which is a project to enable data synchronization to the cloud where the data store is ~~SQL Data Services~~ SQL Azure Database, I thought you should all be aware of this new branding. As we announced in the Data Insiders Blog earlier today:

"Effective immediately, SQL Services will be called Microsoft SQL Azure, and SQL Data Services will be Microsoft SQL Azure Database".

Also, I appologize that I have not provided much of an update on Project "Huron" lately. We are currently heads down in engineering and working out the specific details of the sync service to be hosted within Azure. I would like to thank all of you who offered to help us with our Early Adopter Program. The feedback we have been receiving from you has been absolutely invaluable.

Liam

New Sample - WCF for Devices

2009-06-17T19:00:00Z

Two of our MVPs (José Miguel Torres and Cesar Fong) recently posted an excellent end to end WCF sample for devices that’s worth a look. They did an excellent job of describing the various components and it is a must see for developers thinking about getting their feet wet building offline applications for Windows Mobile. Thanks again to Jose and Cesar for contributing these samples to the community!

http://synccomm.codeplex.com/

Regards,

Sean Kelley
Program Manager, Microsoft

Announcing Sync Framework 2.0 CTP2

2009-06-05T01:49:00Z

I am extremely happy to announce the availability of Sync Framework 2.0 CTP2 . Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

Adds features that cater to new scenarios or scenarios that were difficult to support.
Reduces the amount of work required to develop providers.
Supports more data sources with new built-in providers.

The major new features included in this CTP are:

New database providers (SqlSyncProvider and SqlCeSyncProvider)
Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore.
Custom filters
Enable a filtered replica to store item data only for items that are in the filter, such as a media storage replica that stores only songs that are rated as three stars or better. As the rating on a song changes, an item might move in or out of the filter. A filter-tracking replica is another replica in the community that can identify which items are in the filter and which have moved in or out of the filter recently. A filter-tracking replica may store item data for items that are not in the filter. Sync Framework supports efficient synchronization between filtered replicas and filter-tracking replicas, and maintains all of the necessary metadata.
Improved conflict handling
Sync Framework added several features in this release that make it easier to handle conflicts.

Constraint conflict handling: Sync Framework now supports constraint conflicts, which are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. A destination provider can report a constraint conflict to Sync Framework, and use Sync Framework components to resolve the conflict and determine how to apply the resolution to the destination replica.
Conflict logging: Sync Framework defines a set of conflict log interfaces and provides components that help manage conflicts in the log. Sync Framework also provides an in-memory implementation of the conflict log interfaces for replicas that do not contain a conflict log. It can be useful to save conflicts in a log so that they can be processed separately from the synchronization session, such as when a user needs to review conflicts and decide how to resolve them.
Last-writer-wins conflict resolution: An application might want to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this resolution option by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change.
Improved simple provider conflict handling: Simple providers include a new set of conflict handling interfaces that make it easier to specify resolution actions for concurrency and constraint conflicts.

Data conversion between providers
In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
Change application service
Sync Framework includes a change applier implementation that most applications use to apply changes to a replica. In this CTP, Sync Framework introduces the change application service, which performs the same actions as the change applier, but in a more granular way. A destination provider that requires greater flexibility than the standard change applier can use the change application service to perform only the set of actions that are required.

For more details please visit the Microsoft download center at:
http://www.microsoft.com/downloads/details.aspx?FamilyID=89adbb1e-53ff-41b5-ba17-8e43a2e66254&displaylang=en

Nina Hu

TechEd 2009 - Building Applications with Microsoft SQL Data Services and Windows Azure (DAT316)

2009-05-05T03:06:00Z

Hopefully a number of you will be attending this years TechEd in Los Angeles. This year I will be co-presenting with David Robinson where we will be discussing SQL Data Services and our "Huron" Data Hub project. I am really looking forward to showing you our latest demo so I hope you can all make it. If you are not able to make the session but are at the conference, I will also be at the SQL Data Services booth for quite a bit of the time, so please stop by so we can talk "sync".

Here is the abstract for the session:

"Are you looking to reduce the costs of building and maintaining enterprise applications? Do you want to extend the reach of your applications across multiple devices, locations and partners? SQL Data Services and Windows Azure provides you a friction free, highly scalable platform for building applications. The scale and reach of the cloud lights up a new class of application scenarios. Come see how easy it is to consume SQL Data Services from within Windows Azure. In addition, we dive into Microsoft's new "Data Hub" for businesses and see how this SQL Data Services powered synchronization service allows for data aggregation within the Hub to provide straightforward data sharing between on-premises databases, business partners, remote offices, and mobile users".

See you there!

Liam Cavanagh

New Webcast Video - Drill Down Into the Next Release of the Microsoft Sync Framework

2009-05-02T03:14:00Z

We recently had a chance to record an MSDN webcast describing a handful of the new features that will ship in the next CTP of Sync Framework v2. This particular session focuses primarily on those features we have built related to our offline and collaboration database scenarios. Please register at the following location:

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032413045&CountryCode=US

This session is a deep dive into the next release of the Microsoft Sync Framework with a focus on our database scenarios. Specifically, this session will discuss some of the work we have done in order to allow developers to synchronize SQL Compact in a peer to peer fashion. In addition, we will discuss the improvements we have made around batching to make it easier to use, expose more control over memory usage and provide resumability over transient networks that go up and down when performing sync operations. Furthermore, we will examine some of the performance related features baked into the runtime to improve the time it takes to initialize your client side database and synchronize large tables that have a less frequent rate of change. This will also include a discussion around some of the lessons we learned when performing scale and performance testing in an environment with 4000 clients and as much as 400 concurrent sync operations on a single server. These best practices were compiled at the end of one of the largest external implementations of the Microsoft Sync Framework in the world. Finally, we will go over some of the new scenarios that we have enabled as a result of building on top of the Sync Framework, which provides much more flexibility around the shape of your topology. These scenarios include taking data stored in SQL Data Services and ADO.NET Data Services offline, USB shuttle scenarios, and traditional scenarios such as branch-office. This session will include demos that provide a deep dive into the use of these features as well as benchmark data around the performance of the Sync Framework in a large distributed environment.

Enjoy and please send any and all feedback our way!

Sean Kelley

Project "Huron" - Early Adopter Program

2009-04-29T19:27:00Z

I wanted to give an update on our Project "Huron". For those new to it, this is an incubation project that we have been working on for some time to provide database "friction free" (this is a one fo those Microsoft terms that basically means simple to setup :-)) synchronization capabilities to the Cloud (SQL Data Service) via a hosted Sync Service sitting within the Azure data centers. The goal is to remove the typical complexities (configuration, scalability, security, etc) involved with sharing database information between local databases such as SQL Server and SQL Compact and provide simple UI tools for configuration and sync components developers can embed in existing applications.

One unfortunate scoping that we have decided to make is to temporarily hold off on direct support of Microsoft Access for v1. Although we realize that there are a LOT of customers that are looking to support this, based on our current resourcing this has been an unfortunate scoping that we have decided to take. The primarily jusitification was that we already had an existing SQL Server and SQL Compact provider from Sync Framework that we are able to easily re-use which greatly simplified our efforts. It is our hope that some of these Access customers will consider initially linking their Access forms to a SQL Server database to then synchronize via the cloud in the short term until we can provide the full Access support.

The good news is that in the short term we are still planning to offer direct support SQL Server and SQL Compact (desktop and devices). In fact we are very close to the point that we can start taking on some early adopter to verify some of our assumptioms and help us solidify the first release. What we are looking for are any customers that are looking to share SQL Server or SQL Compact databases via the cloud and have an existing project that would warrant this functionality. If you are interested please contact us using the "Email" link in the left sidebar and I can provide additional details.

With that I will leave you with a few screen mock-ups we are working with from our Huron Management Studio Dev UI.

Liam

SyncToy 2.0 - 1 Millionth Download

2009-04-17T18:56:00Z

I am excited to annouce that SyncToy 2.0 has reached a major milestone. During the month of March it passed 1 Million downloads! This is exciting both for SyncToy as well as the Microsoft Sync Framework team. Since SyncToy bases synchornization upon the Sync Framework's File Sync Provider, it is always a good test of the technology to have users of this scale. In our forums we regularaly see people use it to keep 10's of thousands of files and folders in sync across desktops and devices. It is also a great test and starting point as we look to expand file synchronization into cloud based services.

Liam Cavanagh