Microsoft Sync Framework

PDC Keynote Recording - Kelley Blue Book's use of SQL Azure Data Sync

2009-11-25T19:54:00Z

Last week at PDC we announced the availability of our first CTP of SQL Azure Data Sync. In the keynote, Andy Lapin of Kelley Blue Book explained how they keep their community databases up-to-date using this technolgy to synchronize data between their on-premises SQL Server and SQL Azure. The recording is now available for you to view (the Kelley Blue Book portion of the keynote starts at about 1:18:00).

Ultimately, one of the goals of this CTP is to get your feedback. What is it that you like or dislike about it? Are you primarily interested in the Visual Studio integration or would you like us to see us add additional capabilities such as advanced monitoring or logging to the SQL Server provisioning tool. So please send us your thoughts (good or bad) here:

http://social.microsoft.com/Forums/en-US/syncfeedback/threads

Liam

New Code Sample: File Synchronization with a Simple Custom Provider

2009-11-23T23:41:00Z

Do you have data in a custom format that you want to synchronize with a directory in the file system? This new code sample shows you how to use Sync Framework to do just that.

This sample shows you how to:

Write a simple custom provider that represents your data store.
Implement a data transfer interface that can send and receive data between your data store and the file synchronization provider.
Configure the file synchronization provider to represent a directory on your computer.
Synchronize between your custom data store and the directory you identify.

Download the sample from MSDN Code Gallery and try it out: File Sync with Simple Custom Provider Sample.

Announcing SQL Azure Data Sync (November CTP) Available for Download

2009-11-17T16:42:00Z

Today in the opening keynote at PDC we announced the availability of SQL Azure Data Sync – November CTP, an early preview open to the public through a demonstration with Kelley Blue Book. For those of you who have been following our blog, you may be asking yourself, what exactly does this include and how does it compare to Project “Huron” that we have been talking about for some time now? In this post I want to give some additional details.

You can think of SQL Azure Data Sync as the first part of our overall Project “Huron” vision which is to create a Data Hub in the Cloud, or more specifically a place for you to easily consolidate and share all of your information. With SQL Azure Data Sync we have worked to simplify the task of sharing information whether that is from on-premises SQL Server to the cloud or from the cloud, down to mobile users, retails stores or remote offices. All of this being powered by the Microsoft Sync Framework.

SQL Azure Data Sync allows developers and DBA's to:

Link existing on-premises data stores to SQL Azure.
Create new applications in Windows Azure without abandoning existing on-premises applications.
Extend on-premises data to remote offices, retail stores and mobile workers via the cloud.
Take Windows Azure and SQL Azure based web application offline to provide an “Outlook like” cached-mode experience.

All of this is accomplished through

An end-user Data Sync Tool that keeps on-premises SQL Server data in sync with SQL Azure.
Visual Studio templates enabling developers to take Windows Azure and SQL Azure based web application offline within SQL Compact and SQL Server databases.

By downloading Microsoft Sync Framework Power Pack for SQL Azure November CTP along with the Microsoft Sync Framework 2.0, developers can use the built-in providers for SQL, allowing them to synchronize SQL Azure with other stores such as SQL Server and SQL Server Compact. SQL Azure Data Sync is optimized for performance over high latency networks by taking advantage of features like SQL Azure Table Valued Parameters to help reduce latency and significantly boost performance. Moving forward Microsoft we will continue to enhance this technology and focus on providing a linkage between current on-premises data sources and Windows Azure Storage as well as utilizing this technology as a basis of a developer technology that will enable the Windows Azure platform to be the platform for creating cached mode web applications.

To get started with SQL Azure Data Sync, you need the following:

· Microsoft Sync Framework 2.0

· Microsoft Sync Framework Power Pack for SQL Azure – November CTP

· SQL Azure Server Account (register here)

After installation, you will find 2 new components to help you get started:

1) SQL Server Provisioning Wizard: This wizard (launched from Start | Programs | Microsoft Sync Framework | SQL Azure Data Sync Tool for SQL Server) walks you through the process of connecting SQL Server to SQL Azure through data synchronization. Within this wizard you select the tables to provision in SQL Azure, it will create a new database and setup background synchronization to keep the two data stores in-sync.

2) Visual Studio 2008 Template: For users that would like to take an existing SQL Azure database offline, this new template simplifies the task of creating an offline data cache within SQL Compact. Using the template wizard (available by right clicking on a Visual Studio Solution Explorer Application and choosing Add | New Item | SQLAzureDataSyncClient, developers can choose the SQL Azure tables that they would like to be made available offline. After completing the wizard, a SQL Compact database will be created and code will be generated that allows the offline to synchronize changes on-demand between SQL Azure and SQL Compact.

- Liam

SyncToy 2.1 Available for Download

2009-11-11T04:21:00Z

SyncToy 2.1 has just been released to the Microsoft Download center. SyncToy 2.1 is a free application that synchronizes files and folders between locations. Typical uses include sharing files, such as photos, with other computers and creating backup copies of files and folders.

SyncToy2.1 is powered by the latest synchronization engine from Microsoft Sync Framework 2.0 and provides better performance and robustness. The new features and improvements included in SyncToy 2.1 release are:

Better Performance: The speed of file copy operations is significantly increased across the board.
Improved Robustness: Much more resilient to transient network and file system errors, and better error reporting that pinpoints which file the sync failed on in case there’s a fatal error that stops the sync.
Folder pair configuration backup: Folder pair configuration is automatically backed up under %localappdata%\microsoft\synctoy\2.0. You can replace SyncTopDirPairs.bin with the backup copy to resolve the last saved configuration.
Bug Fixes:

o Fixed the data corruption issue when using SyncToy with NAS drives.

o Fixed the bug that prevented uploading files to SharePoint when using SyncToy 2.0.

o Fixed the bug that prevented delete changes from being synchronized when the sync option was set to “Echo”.

SyncToy 2.1 is now available for download from: http://www.microsoft.com/downloads/details.aspx?familyid=C26EFA36-98E0-4EE9-A7C5-98D0592D8C52&displaylang=en.

SQL Pass Conference & Synchronization to SQL Azure

2009-11-04T17:34:00Z

Yesterday was the first day of the SQL Server conference, SQL Pass, here in Seattle. It was a really exciting day for me because I had the opportunity to present our new synchronization capabilities to the cloud in the opening day keynote and in a subsequent session.

We introduced the first piece of Project “Huron”, that is a technology based on the Sync Framework enabling people to use Windows Azure as a central Data Hub for all information. To accomplish this, later this month we will make available for download an early preview of a tool that provides the ability to extend a SQL Server on-premises databases to the cloud. Once configured users can then easily extend data from the cloud to mobile users and remote offices.

Using this tool and with the help of one of our partners, Archetype, we showed how companies can easily extend their on-premises SQL Servers to SQL Azure using data synchronization, allowing the data stores to co-exist and interoperate seamlessly.

All of this can be setup using a wizard. Then from an automatically generated SQL Agent, the synchronization process is executed periodically to move the incremental changes between SQL Server and SQL Azure.

Here are a few screenshots of the tool:

Watch for more details over this month at the Professional Developer Conference and in this blog for other new capabilities we are providing to for integration to Windows Azure and SQL Azure.

Liam Cavanagh

New whitepaper: Minimum Database Permissions Requirements for Microsoft Sync Framework

2009-11-02T23:11:00Z

Are you trying to set up a sync-ready database with the lowest (i.e. most secure) permissions possible? Have you set it up, but are worried you might have opened things up too wide?

Perhaps our just-published whitepaper on this very topic can be of assistance: http://msdn.microsoft.com/en-us/library/ee730363.aspx

New Sync Framework Samples

2009-10-26T22:11:00Z

We have lots of new samples on Sync Framework Code Gallery. These samples demonstrate some of the most important and exciting features of Sync Framework 2.0. If you are new to Sync Framework or are looking for ways to write code to solve your synchronization problems, chances are we have a sample that can get you started. Here are some of the highlights:

Sync101 Samples

This set of samples illustrates various concepts and features needed to synchronize a custom data store, such as:

Sync101 with Simple Providers, which shows how to write a basic synchronization application that uses simple custom providers.
Sync101 with Constraint Conflicts, which shows how to write a standard custom provider that uses Windows Communication Foundation (WCF) to synchronize data between replicas on different computers.
Sync101 with Custom Filtering, which shows how to track custom filters and how to use a standard custom provider to send changes from a filter-tracking replica to two different filtered replicas.

Database Synchronization Provider Samples

This set of samples illustrates various concepts around using database synchronization providers to synchronize between SQL Server databases and SQL Server Compact databases, such as:

Database Sync: SQL Server and SQL Server Compact, which shows how to use database synchronization providers to configure and execute peer-to-peer synchronization between a SQL Server database and one or more SQL Server Compact databases.

File Synchronization Provider Sample

This sample shows how to use the file synchronization provider.

File Synchronization Provider Sample, which shows how to use the file synchronization provider, a fully functioning provider that helps an application to synchronize files and folders in NTFS, FAT, and SMB file systems.

Windows 7 is out -- and with it comes Windows Sync, powered by Microsoft Sync Framework!

2009-10-22T09:00:00Z

It's finally here!

Today is the long-awaited launch date of Windows 7, and on the Sync team we couldn't be more excited, as it is the first Windows client OS to ship with Microsoft Sync Framework running out-of-the-box. This special version of Sync Framework is powering a new feature called Windows Sync, which allows Windows users to manage their sync relationships through a user interface.

Under the hood, Sync Framework is running, and each sync relationship is being governed by custom providers that were developed to handle things like keeping contacts, calendars, notes and tasks up-to-date between the user's Windows machine and their smart phone.

Are you a developer that would like to make it easy for users to manage and synchronize with data stored on your product using the brand new Windows Sync feature? Click the below link and get cracking!

http://msdn.microsoft.com/en-us/library/dd317274(VS.85).aspx - Windows Sync Developer Reference

SharePoint 2010, Windows 7 and PwC Now Integrate Microsoft Sync Framework v2.0

2009-10-19T18:17:00Z

As we announced earlier today, Microsoft Sync Framework V2.0 is now available for public download.

Along with this release I wanted to outline a number of companies and partners that are already using Sync Framework v2.0 including Windows 7, Microsoft Office 2010 and PricewaterhouseCoopers (PwC).

SharePoint has also chosen to integrate Microsoft Sync Framework as part of their solution for synchronizing to Office SharePoint 2010. The SharePoint and Office teams will use the Sync Framework as the engine powering synchronization between Office Workspace (Groove) and SharePoint. In addition, the SharePoint server will also expose Sync Framework API’s that will allow any developer to build providers to integrate new data stores into SharePoint.

"The Sync Framework will greatly reduce the amount of code that an ISV needs to create, and this represents probably the most painful code that they otherwise would have had to write".
Maxim Lukiyanov, Program Manager SharePoint Server, Microsoft

For more information on this please see our new case study: Microsoft Sync Framework used to Create Microsoft SharePoint Workspace 2010

The Windows 7 Device Stage gives users a common user interface to manage and synchronize data with their Windows 7 enabled phone. Contacts, calendar, tasks and even notes can be synchronized with any data store. Sync Framework v2.0 is the easiest way for developers to implement data synchronization providers for any store to enable it for the Windows 7 Device synchronization experience.

"The flexibility of Sync Framework and its ability to track and exchange data really set this technology apart. Since Sync Framework is data agnostic this allowed us to develop a framework tailored around contact, calendar, tasks and notes data".
Anna Boyd, Windows 7 Engineering, Microsoft

PricewaterhouseCoopers has put a system in deployment for 60,000+ mobile auditors in 150 countries to allow much of their work to be done remotely at the client site, making disconnected, offline use of their application and friction free collaboration across teams.

"Our implementation of Sync Framework ‘just worked’ for our auditors. If they set automatic synchronization, our software finds peers, verifies access rights to the same engagement database, and updates all databases. They might be connected to each other over a client’s network, or just via ad hoc (no access point) WiFi. If just one laptop has access to the servers, that laptop automatically funnels all changes from the laptops to the server and vice versa. No manual configuration is needed".
Dr. Glenn Ricart, Technology and Development Leader, PricewaterhouseCoopers

Liam Cavanagh

Sync Framework 2.0 Available for Download

2009-10-19T15:27:00Z

I am happy to announce the availability of Sync Framework 2.0. Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

Adds features that cater to new scenarios or scenarios that were difficult to support.
Reduces the amount of work required to develop providers.
Supports more data sources with new built-in providers.

The major new features and improvements included in Sync Framework 2.0 are:

In Core Components:

Simple Providers: Reduce the amount of work required to develop providers, especially when the data source has very few synchronization-related capabilities, such as change-tracking. The goal is to enable a developer to write a provider without having to become a synchronization expert. The majority of the code that is required for a simple provider is limited to that responsible for interacting with the data source. Simple providers support many complex synchronization scenarios, such as filtering, concurrency and constraint conflict handling, and anchor-based enumeration.
Flexible Filtering: Change unit filters and custom filters have been added to the custom provider components. These filters, in addition to the item filters from Sync Framework 1.0, enable you to filter your data in whatever way is most useful. Filter negotiation allows the source and destination providers to negotiate the filter used during the synchronization session. Change unit filters enable a provider to specify that, while the replica it serves stores all items in the synchronization scope, the replica stores a reduced representation of each item. For example, when an item is a contact a device might store only the name and phone number of the contact as compared to the complete contact stored on a PC. Custom filters enable you to define a filter in whatever way is most appropriate, including filters that allow an item to move into or out of the filter as the item data changes. For example, a media storage device stores only songs that are rated as three stars or better. When the rating on a song changes from four stars to two stars, the song moves out of the filter. Filter negotiation allows a destination provider to specify the filter to be used by the source provider during change enumeration; the source provider can accept or reject a filter. If a source provider does not support the requested filter then the destination provider can choose to receive all of the data and do the filtering itself.
Improved Conflict Handling: Additional conflict scenarios are supported, such as reporting constraint conflicts, managing logged conflicts, and resolving conflicts by using a last-writer-wins policy. Constraint conflicts are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. Sync Framework includes components to help resolve constraint conflicts and apply the resolution to the destination replica. Conflict logging is enhanced by Sync Framework components that help manage the log, such as by removing obsolete conflicts. A last-writer-wins conflict resolution policy can be used to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this policy by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change.
Data Conversion between Providers: In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
Change Application Service: Increases the flexibility and usefulness of the change applier component of Sync Framework. By performing the same actions as the standard change applier, but in a more granular way, the change application service gives a destination provider the ability to use only the features that it requires.
Tracing: Enables you to trace the execution of several components, which is useful during application debugging. This download provides the .tmf files that can be used to view a trace, flush the logger, and format the binary trace file. For more information about how to trace Sync Framework execution, see “Tracing Sync Framework Components” in the documentation that installs with the Sync Framework SDK.

In Database Providers:

New Database Providers (SQL Server and SQL Server Compact): Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore.
Robust Memory-Based Batching: Previous versions of Sync Framework and Sync Services for ADO.NET provided a way for developers to define their own batching logic but there were a lot of limitations, including significant complexity, excessive chattiness, out of memory issues, and restrictions on usage. Sync Framework 2.0 addresses all of these issues by providing a more complete and robust batching API. Developers no longer have to write batching logic themselves because Sync Framework divides changes into batches based on several properties in the API. Batches are now defined by memory consumption rather than the number of rows synchronized, which has eliminated out-of-memory issues for most common scenarios.
Provisioning and Management APIs: Provisioning and initialization activities that were previously exposed only through Visual Studio tooling have now been added to the database provider APIs. This includes the ability to provision an existing database by adding the change-tracking tables and triggers that are required by Sync Framework. It also includes the ability to start with an empty database, create the user schema, and provision that schema based on another server or client database that has already been provisioned.
Performance Improvements: The new database providers in this release have been thoroughly tested in large-scale scenarios in which a single server supports thousands of clients with hundreds of concurrent synchronization operations. This testing resulted in a large number of internal performance improvements that enable Sync Framework database providers to perform as well as other Microsoft technologies like Remote Data Access (RDA) while offering a wide range of capabilities that compete with end-to-end solutions like merge replication.

In File Synchronization Provider:

· Data Conversion between Providers: The data transfer interface used by the file synchronization provider is now available in managed code. A custom provider that synchronizes some other data type can use the data transfer interface to convert its data to synchronize with a file synchronization provider.

· Better File Transfer Performance: Improvements to the way file data is copied allow file data to be transferred up to 30% faster than in Sync Framework 1.0.

· More Robust Synchronization: Instead of failing the entire synchronization session when a single file cannot be synchronized (including failures caused by network issues), the failure is flagged and the synchronization session continues.

To learn more about Sync Framework or to download Sync Framework 2.0, please visit the Sync Framework Developer Center.

Sync Framework and our upcoming conferences

2009-10-05T19:31:00Z

The next few months are going to be very busy for us here in the Sync Framework team. Not only do we have Sync Framework v2 coming out shortly, we also have a number of conferences where we will talk about the work we have been doing with some of external companies and internal groups here at Microsoft to allow developers to plug-in synchronization to systems like SQL Server and SQL Azure.

Starting in November, we will be at SQL PASS, and then at the PDC for the launch of SQL Azure. Last week I recorded a Channel 9 video along with Buck Woody and Michael Rys on the upcoming SQL PASS Summit. You can view the video here:

http://channel9.msdn.com/posts/LarryLarsen/Upcoming-SQL-PASS-Summit-1/

Here is a summary of the video:

Join thousands of SQL Server & BI professionals at the Washington State Convention & Trade Center in Seattle from Nov. 2-5 for the largest SQL Server conference in the world! PASS Summit offers 168 technical sessions presented by SQL Server experts, one-on-one time with Microsoft’s SQL Server engineering team, free troubleshooting & design architecture guidance from Microsoft consultants, keynotes from Microsoft’s senior executives and much more!

Senior Data Platform Technology Specialist Buck Woody stopped by the Channel 9 studio to talk to Michael Rys, Principal Program Manager in SQL Server Engineering Team, and Liam Cavanagh, Senior Program Manager, about upcoming sessions you won’t want to miss this year’s PASS Summit conference.

Liam Cavanagh

SQL Services is now SQL Azure

2009-07-09T01:58:00Z

I thought I would help get the word out on this one. Given that we have been spending so much time on Project "Huron", which is a project to enable data synchronization to the cloud where the data store is ~~SQL Data Services~~ SQL Azure Database, I thought you should all be aware of this new branding. As we announced in the Data Insiders Blog earlier today:

"Effective immediately, SQL Services will be called Microsoft SQL Azure, and SQL Data Services will be Microsoft SQL Azure Database".

Also, I appologize that I have not provided much of an update on Project "Huron" lately. We are currently heads down in engineering and working out the specific details of the sync service to be hosted within Azure. I would like to thank all of you who offered to help us with our Early Adopter Program. The feedback we have been receiving from you has been absolutely invaluable.

Liam

New Sample - WCF for Devices

2009-06-17T19:00:00Z

Two of our MVPs (José Miguel Torres and Cesar Fong) recently posted an excellent end to end WCF sample for devices that’s worth a look. They did an excellent job of describing the various components and it is a must see for developers thinking about getting their feet wet building offline applications for Windows Mobile. Thanks again to Jose and Cesar for contributing these samples to the community!

http://synccomm.codeplex.com/

Regards,

Sean Kelley
Program Manager, Microsoft

Announcing Sync Framework 2.0 CTP2

2009-06-05T01:49:00Z

I am extremely happy to announce the availability of Sync Framework 2.0 CTP2 . Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

Adds features that cater to new scenarios or scenarios that were difficult to support.
Reduces the amount of work required to develop providers.
Supports more data sources with new built-in providers.

The major new features included in this CTP are:

New database providers (SqlSyncProvider and SqlCeSyncProvider)
Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore.
Custom filters
Enable a filtered replica to store item data only for items that are in the filter, such as a media storage replica that stores only songs that are rated as three stars or better. As the rating on a song changes, an item might move in or out of the filter. A filter-tracking replica is another replica in the community that can identify which items are in the filter and which have moved in or out of the filter recently. A filter-tracking replica may store item data for items that are not in the filter. Sync Framework supports efficient synchronization between filtered replicas and filter-tracking replicas, and maintains all of the necessary metadata.
Improved conflict handling
Sync Framework added several features in this release that make it easier to handle conflicts.

Constraint conflict handling: Sync Framework now supports constraint conflicts, which are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. A destination provider can report a constraint conflict to Sync Framework, and use Sync Framework components to resolve the conflict and determine how to apply the resolution to the destination replica.
Conflict logging: Sync Framework defines a set of conflict log interfaces and provides components that help manage conflicts in the log. Sync Framework also provides an in-memory implementation of the conflict log interfaces for replicas that do not contain a conflict log. It can be useful to save conflicts in a log so that they can be processed separately from the synchronization session, such as when a user needs to review conflicts and decide how to resolve them.
Last-writer-wins conflict resolution: An application might want to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this resolution option by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change.
Improved simple provider conflict handling: Simple providers include a new set of conflict handling interfaces that make it easier to specify resolution actions for concurrency and constraint conflicts.

Data conversion between providers
In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
Change application service
Sync Framework includes a change applier implementation that most applications use to apply changes to a replica. In this CTP, Sync Framework introduces the change application service, which performs the same actions as the change applier, but in a more granular way. A destination provider that requires greater flexibility than the standard change applier can use the change application service to perform only the set of actions that are required.

For more details please visit the Microsoft download center at:
http://www.microsoft.com/downloads/details.aspx?FamilyID=89adbb1e-53ff-41b5-ba17-8e43a2e66254&displaylang=en

Nina Hu

TechEd 2009 - Building Applications with Microsoft SQL Data Services and Windows Azure (DAT316)

2009-05-05T03:06:00Z

Hopefully a number of you will be attending this years TechEd in Los Angeles. This year I will be co-presenting with David Robinson where we will be discussing SQL Data Services and our "Huron" Data Hub project. I am really looking forward to showing you our latest demo so I hope you can all make it. If you are not able to make the session but are at the conference, I will also be at the SQL Data Services booth for quite a bit of the time, so please stop by so we can talk "sync".

Here is the abstract for the session:

"Are you looking to reduce the costs of building and maintaining enterprise applications? Do you want to extend the reach of your applications across multiple devices, locations and partners? SQL Data Services and Windows Azure provides you a friction free, highly scalable platform for building applications. The scale and reach of the cloud lights up a new class of application scenarios. Come see how easy it is to consume SQL Data Services from within Windows Azure. In addition, we dive into Microsoft's new "Data Hub" for businesses and see how this SQL Data Services powered synchronization service allows for data aggregation within the Hub to provide straightforward data sharing between on-premises databases, business partners, remote offices, and mobile users".

See you there!

Liam Cavanagh