Microsoft Sync Framework

New Sample - WCF for Devices

Two of our MVPs (José Miguel Torres and Cesar Fong) recently posted an excellent end to end WCF sample for devices that’s worth a look.  They did an excellent job of describing the various components and it is a must see for developers thinking about getting their feet wet building offline applications for Windows Mobile.  Thanks again to Jose and Cesar for contributing these samples to the community!

http://synccomm.codeplex.com/

Regards,

Sean Kelley
Program Manager, Microsoft

Announcing Sync Framework 2.0 CTP2

I am extremely happy to announce the availability of Sync Framework 2.0 CTP2 .  Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

• Adds features that cater to new scenarios or scenarios that were difficult to support.
• Reduces the amount of work required to develop providers.
• Supports more data sources with new built-in providers.

The major new features included in this CTP are:

• New database providers (SqlSyncProvider and SqlCeSyncProvider)
  Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore. 
• Custom filters
  Enable a filtered replica to store item data only for items that are in the filter, such as a media storage replica that stores only songs that are rated as three stars or better. As the rating on a song changes, an item might move in or out of the filter. A filter-tracking replica is another replica in the community that can identify which items are in the filter and which have moved in or out of the filter recently. A filter-tracking replica may store item data for items that are not in the filter. Sync Framework supports efficient synchronization between filtered replicas and filter-tracking replicas, and maintains all of the necessary metadata.
• Improved conflict handling
  Sync Framework added several features in this release that make it easier to handle conflicts. 
• Constraint conflict handling: Sync Framework now supports constraint conflicts, which are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. A destination provider can report a constraint conflict to Sync Framework, and use Sync Framework components to resolve the conflict and determine how to apply the resolution to the destination replica. 
• Conflict logging: Sync Framework defines a set of conflict log interfaces and provides components that help manage conflicts in the log. Sync Framework also provides an in-memory implementation of the conflict log interfaces for replicas that do not contain a conflict log. It can be useful to save conflicts in a log so that they can be processed separately from the synchronization session, such as when a user needs to review conflicts and decide how to resolve them. 
• Last-writer-wins conflict resolution: An application might want to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this resolution option by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change. 
• Improved simple provider conflict handling: Simple providers include a new set of conflict handling interfaces that make it easier to specify resolution actions for concurrency and constraint conflicts.
• Data conversion between providers
  In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
• Change application service
  Sync Framework includes a change applier implementation that most applications use to apply changes to a replica. In this CTP, Sync Framework introduces the change application service, which performs the same actions as the change applier, but in a more granular way. A destination provider that requires greater flexibility than the standard change applier can use the change application service to perform only the set of actions that are required.

For more details please visit the Microsoft download center at:
http://www.microsoft.com/downloads/details.aspx?FamilyID=89adbb1e-53ff-41b5-ba17-8e43a2e66254&displaylang=en

Nina Hu

TechEd 2009 - Building Applications with Microsoft SQL Data Services and Windows Azure (DAT316)

Hopefully a number of you will be attending this years TechEd in Los Angeles.  This year I will be co-presenting with David Robinson where we will be discussing SQL Data Services and our "Huron" Data Hub project.  I am really looking forward to showing you our latest demo so I hope you can all make it.  If you are not able to make the session but are at the conference, I will also be at the SQL Data Services booth for quite a bit of the time, so please stop by so we can talk "sync". 

Here is the abstract for the session:

"Are you looking to reduce the costs of building and maintaining enterprise applications? Do you want to extend the reach of your applications across multiple devices, locations and partners? SQL Data Services and Windows Azure provides you a friction free, highly scalable platform for building applications. The scale and reach of the cloud lights up a new class of application scenarios. Come see how easy it is to consume SQL Data Services from within Windows Azure. In addition, we dive into Microsoft's new "Data Hub" for businesses and see how this SQL Data Services powered synchronization service allows for data aggregation within the Hub to provide straightforward data sharing between on-premises databases, business partners, remote offices, and mobile users".

See you there! 

Liam Cavanagh

New Webcast Video - Drill Down Into the Next Release of the Microsoft Sync Framework

We recently had a chance to record an MSDN webcast describing a handful of the new features that will ship in the next CTP of Sync Framework v2.  This particular session focuses primarily on those features we have built related to our offline and collaboration database scenarios. Please register at the following location:

 

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032413045&CountryCode=US

 

This session is a deep dive into the next release of the Microsoft Sync Framework with a focus on our database scenarios. Specifically, this session will discuss some of the work we have done in order to allow developers to synchronize SQL Compact in a peer to peer fashion. In addition, we will discuss the improvements we have made around batching to make it easier to use, expose more control over memory usage and provide resumability over transient networks that go up and down when performing sync operations. Furthermore, we will examine some of the performance related features baked into the runtime to improve the time it takes to initialize your client side database and synchronize large tables that have a less frequent rate of change. This will also include a discussion around some of the lessons we learned when performing scale and performance testing in an environment with 4000 clients and as much as 400 concurrent sync operations on a single server. These best practices were compiled at the end of one of the largest external implementations of the Microsoft Sync Framework  in the world. Finally, we will go over some of the new scenarios that we have enabled as a result of building on top of the Sync Framework, which provides much more flexibility around the shape of your topology. These scenarios include taking data stored in SQL Data Services and ADO.NET Data Services offline, USB shuttle scenarios, and traditional scenarios such as branch-office. This session will include demos that provide a deep dive into the use of these features as well as benchmark data around the performance of the Sync Framework in a large distributed environment.

 

Enjoy and please send any and all feedback our way!

 

Sean Kelley

Project "Huron" - Early Adopter Program

I wanted to give an update on our Project "Huron".  For those new to it, this is an incubation project that we have been working on for some time to provide database "friction free" (this is a one fo those Microsoft terms that basically means simple to setup :-)) synchronization capabilities to the Cloud (SQL Data Service) via a hosted Sync Service sitting within the Azure data centers.  The goal is to remove the typical complexities (configuration, scalability, security, etc) involved with sharing database information between local databases such as SQL Server and SQL Compact and provide simple UI tools for configuration and sync components developers can embed in existing applications.

One unfortunate scoping that we have decided to make is to temporarily hold off on direct support of Microsoft Access for v1.  Although we realize that there are a LOT of customers that are looking to support this, based on our current resourcing this has been an unfortunate scoping that we have decided to take.  The primarily jusitification was that we already had an existing SQL Server and SQL Compact provider from Sync Framework that we are able to easily re-use which greatly simplified our efforts.  It is our hope that some of these Access customers will consider initially linking their Access forms to a SQL Server database to then synchronize via the cloud in the short term until we can provide the full Access support.

The good news is that in the short term we are still planning to offer direct support SQL Server and SQL Compact (desktop and devices).  In fact we are very close to the point that we can start taking on some early adopter to verify some of our assumptioms and help us solidify the first release.  What we are looking for are any customers that are looking to share SQL Server or SQL Compact databases via the cloud and have an existing project that would warrant this functionality.  If you are interested please contact us using the "Email" link in the left sidebar and I can provide additional details.

With that I will leave you with a few screen mock-ups we are working with from our Huron Management Studio Dev UI.

 

Liam

SyncToy 2.0 - 1 Millionth Download

I am excited to annouce that SyncToy 2.0 has reached a major milestone.  During the month of March it passed 1 Million downloads!   This is exciting both for SyncToy as well as the Microsoft Sync Framework team.  Since SyncToy bases synchornization upon the Sync Framework's File Sync Provider, it is always a good test of the technology to have users of this scale.  In our forums we regularaly see people use it to keep 10's of thousands of files and folders in sync across desktops and devices.  It is also a great test and starting point as we look to expand file synchronization into cloud based services.

Liam Cavanagh

Custom Conflict Resolution Survey

We are considering making some changes to the patterns currently leveraged to apply custom conflict resolution policies and business logic.   During our design discussions a handful of questions came up, so we thought it would be best to post them on our blog in an effort to collect feedback.  Please provide your feedback in the comments attached to this post. 

Our questions are as follows:

1. Can you give a handful of examples when you have had to apply some sort of custom conflict resolution policy and/or your own business logic when syncing changes?  Specifically, we are interested in examples that require data to be changed in the middle of synchronization either before/after getting changes or before/after applying changes.
2. Do you typically leverage the Sync Framework or Sync Services events  in order to do so?  If so, which events do you most commonly use?  If not, can you describe alternate approaches you have taken?
3. Do you leverage SyncContext to make changes to the DataSet being sent/received or some other approach?

Thank you for your time!

Regards,

Sean Kelley
Program Manager
Microsoft Sync Framework

The Clouds are clearing in Redmond

After a healthy dumping of March snow here in Redmond yesterday, the clouds have changed and it is now a clear and sunny day.  Kind of appropriate I think given the announcement from our sister team SQL Data Services this morning.  In today’s announcement it was stated that SQL Data Services: "will become the first relational database service in the market to provide customers with the ability to use existing investments in T-SQL development and use a full relational data model in the Cloud" We in the Sync Framework team have been patiently waiting for this big announcement.   Just like the customers that have asked for this change, for our project "Huron" this has been an incredibly positive change in our goal to allow databases to be shared via the cloud.  For us it has:

• Greatly simplified the development effort.  Now we can base even more of our architecture on Sync Services for ADO.NET technology.  Specifically the server provider in Sync Services for ADO.NET, used to powers companies with scales of 80,000 mobile workers can now also be used within our hosted sync service.  
• Provided a well known and proven architecture and allows us to spend more resources on making the client side experience and sync service even better --  my personal goal is that an Access, SQL Compact or SQL Server user can start sharing their database via the cloud in under 3 minutes.  
• Allowed an even greater level of scalability.  Previously, the architecture of SQL Data Services required us to transform from relational to entity format.  Now, we will be able to maintain the relational format of the data which will have a big impact on performance.
• Enabled the use of SQL Server integrated change tracking capabilities within our sync service for even greater sync and change tracking performance

In fact, we have been working for some time on a prototype of “Huron” using a new sync service hosted within Azure against relational cloud storage.  Keep an eye out here for a webcast where I will walk through this and some of our other project goals.

While I am at it, I also wanted to send a thank-you to all of those who took the time to join our “Huron” early adopter program at SQLServicesLabs.com.  As some of you may have noticed we have stopped accepting applications (for now).  This is simply because we could not keep up with the responses and did not feel we could support any more customers in the early adopter stages of our program.  If anything, we have heard loud and clear your requirements in this area, but please feel free to continue to send me your thoughts and ideas. 

I am not quite ready to talk about the public release date other than to state that the goal is to get something into your hands to play with by the end of this year. 

Stay tuned for more!

Liam Cavanagh

(Internal) Customer Spotlight - Synchronization Scalability and Live Toolbar

One of the most common (and most dreaded) questions anyone who writes a synchronization technology will get is “How well does it scale”?  I say dreaded because it is an incredibly difficult thing to state with 100% certainty that a product will scale to X number of concurrent users for all customer scenarios.  There are just too many variables, such as the amount of data being transferred, how fast the machines are, how much memory there is, what is the throughput of the network connection, etc.  To help us answer these questions, sometimes we can point to specific benchmarks or customer implementation examples.   One of the most recent examples of high scale sync that we can look at is Live Toolbar.   For those who have not tried it, Live Toolbar is a technology used by many millions of users across the world.  Live Toolbar works as a plug-in within your Internet Explorer browser and gives you quick access to Windows Live and Live Search.  In the latest version of Live Toolbar, you can now share your favorite websites with people in your network on Windows Live. You can also sync your Internet Explorer favorites across multiple PCs—sign in and access them from any computer where you’ve installed the Windows Live Toolbar.  This favorite synchronization capability is accomplished through the use of the Microsoft Sync Framework.  The favorite sync feature consolidates your favorites into a central Windows Live storage server in the cloud allowing you to then share them and view them from anywhere.  This feature is rolling out across the world and is currently only available in US and Taiwan, however, it still shows a potential of being able to being able to handle a huge scale of millions of synchronization users through the use of Microsoft Sync Framework. Liam Cavanagh

Customer Spotlight – Interscape Flip for the Mobile Sales Workforce

Continuing with our Sync Framework customer spotlight, I wanted to introduce another company called Interscape who has developed a simple to use sales tool called FLIP Client.  Flip was built to incorporate features like scheduling, pipeline reporting, outlook and more.  I had a chance to talk with Matt Francis, CEO of Interscape and this is what he had to say about the Sync Framework and Occasionally Connected Applications: “About 2 years ago I noticed sales professionals using a slew of applications to manage their sales process. Some were online, some were disconnected… None of them let people manage data and documents in a “sync” framework. The idea is that no matter what the data store is, we need an easy way to keep it in sync”. “We started researching and found Microsoft’s new SYNC Framework to look like a good solutions for our new FLIP sales client to keep documents on the team in sync with each other. It worked exactly the way we wanted, we created a SYNC provider and the SYNC Framework handled the rest”. “We then implemented an interface to keep our data in sync as well, between SQL CE and SQL2005. No problem, just create a sync process and interface and now our remote data store is syncable (not a real word… yet!)” “We then had another challenge… How do we keep the events in the client in-sync with Outlook? The answer was SYNC Framework. We worked with the SYNC Framework group and created an Outlook sync interface. The framework managed the rest”! “Overall, the idea of occasionally connected applications is perfect for the mobile workforce. Helping them sale thousands of dollars on wireless cards, reduce frustration of slow online applications, and keep them working when they need to be, is the goal of any organization. The SYNC Framework helped us out and we have accomplished the goals using this technology”. “Now our sales team and other companies are using the FLIP client to work when and where they want to”!

One of the things that I really like about Flip Client and something I can appreciate from discussions with other customers is the need to make synchronization solutions simpler.  I believe we are approaching a time in technology where data synchronization is not just limited to large enterprises that have the resources to implement and manage these solutions.  We are now starting to see solutions that incorporate cloud based data storage and sync services such as Project “Huron” that offload the complexities of data synchronization scalability, security, availability and management to the cloud.  As these technologies and others like Flip Client mature, I believe small and mid-size businesses will begin to see the advantage of data synchronization typically only available to large organizations. For more information on Interscape and Flip Client, please visit: http://www.interscapeinc.com/ or for a closer look at the client please visit: http://www.perfectsalesclient.com/

Liam Cavanagh

Customer Spotlight – Sync2DB (Outlook to Database Synchronization)

If you were to ask me what the three most commonly requested synchronization targets are, I think I would have to say Outlook to Database, Database to Database and SharePoint.  Today I would like to highlight one of our partners who have worked to solve some of this.  Sync2DB is a simple to use Windows application that enables synchronization between Microsoft Outlook and your database.   Best of all, it is all based on the Microsoft Sync Framework! When talking to Sergej Klimuk, of Sync2DB, he explained to me that “All other synchronization solutions between Microsoft Outlook and databases are complicated because they require configuration and an Outlook add-on installation on each user account and PC and allow only one connection at a time. Sync2DB is the first tool that helps to avoid or minimize the user interaction with the program, and helps developers create safe data subscriptions, providing synchronization and deploy the application onto thousands of users' PCs in a matter of days”. One example of a typical use case involves an employee who logs in to the corporate website and clicks the “subscribe/ synchronize/ upload Outlook data” button. The Application will automatically get downloaded and installed on the user’s PC. The data and fields selected by the administrator in a Microsoft Outlook folder will be automatically uploaded to the selected database with pre-defined parameters. The process of synchronization can be fully silent without any user interactions, or fully displayed with all the required program interfaces and notifications, along with the status of upload/synchronization. Synchronization processes that will follow can be fully automated /scheduled or requested manually from corporate website after logging in, as well as from the desktop shortcut that will require logon. In the current implementation, Sync2DB has made it possible to synchronize any Microsoft SQL database with Microsoft Outlook (versions 2002-2007).  In addition they are targeting support of Microsoft SharePoint Server Data as well as other database systems such as MySQL which will help customers solve a number or other common requests.  For more details, please visit http://www.sync2db.com Liam Cavanagh

 

New Sample - Sync101 for Windows Mobile Devices

We have just posted a new Sync Framework sample for devices to our code gallery entitled "Sync101 - (Windows Mobile) Device Synchronization".

This C# sample is meant to show how to use Microsoft Sync Framework on smart device such as Windows Mobile and to illustrate the "remote change application" (RCA) pattern over a web service (between a desktop PC and a smart device).

The current release of "Microsoft Sync Framework for Devices" does not contain the Metadata Store service. Because of this the provider on a device needs to implement the necessary metadata management functions in a type named "MyProviderMetadata". This sample shows how this can be accomplished.  Using this sample the provider on desktop will still uses the Metadata Store for an easy approach.

For more code gallery samples please visit our Sync Framework Code Gallery page.

Liam

New File Sync Provider Webcast

We have posted a new video webcast of the File Sync Provider now available for download (we will make it available on demand within the Sync Framework dev center as a featured video soon). 

In this webcast, Ashish Shah, Principal Development Lead in the Sync Framework team walks through an overview of the files sync provider provided within the Sync Framework. 

During the session, Ashish discusses a variety of scenarios supported by this provider as well as a summary of the feature set and a look at some of the programmatic requirements associated with creating a true file sync solution such as SyncToy.

Liam Cavanagh

What's New with Sync Framework

As a friend pointed out to me a few days ago, I have not provided much of an update on what is going on here in the Sync Framework team since last November.  I think sometimes when I am so heads down on a product I can occasionally forget to give you an update on where we are.  So I thought I would take a few minutes and let you know what we have been working on (or at least what I can say so far publiclyJ).

Sync Framework v2

Late last year we released a CTP (Community Technology Preview) of Sync Framework v2 and this release had support for things like filtering, simple provider creation and a number of other new features and performance improvements.  We are making a lot of progress on this release along with additional new device support.  You should expect to see another CTP of Sync Framework v2 before mid-year 2009.

Sync Services for ADO.NET v2

This product has also been a big focus for our group.  We had previously released Sync Services for ADO.NET v2 along with Sync Framework v1.  In addition we released the device support of Sync Services for ADO.NET.  With this we are working quite hard on the next version release of Sync Services for ADO.NET along with some impressive performance testing results that we will be sharing along with this release.  I am also dying to tell you about some of the large customer successes that we have been working with, but unfortunately I have to wait until their product actually ships.  L

Project “Huron”

This project has been a large focus of mine for quite some time now.  I am really pleased at how things are progressing with this project and hope to present some more of this to you in the next few months.  I love the concept of being able to take any database on any device and with no work (or development) publish that data to the cloud and then invite others to subscribe to that data into a database of their choosing (on the device of their choice) and then share and contribute to it without having to worry about the authentication, security or scalability of your “data hub”.  As you might imagine we have been working quite closely with the SQL Data Services team on this project to become our customer data hub in the cloud.

SyncToy 2.0

SyncToy is a free file synchronization utility that uses Sync Framework under the covers.  This product along with the related “Anchorage” project is managed within our Sync Framework team.  SyncToy just had its biggest month ever with a total of 136,610 downloads for the month of December putting it in the top 50 downloads of all Microsoft products!

 

So that is just a few of the things that have been keeping our team busy.   So to Mike and the others who kindly read this blog on a regular basis, I will try to better on keeping you up-to-date on where we are!

Liam

TechEd 2008 EMEA Presentations

Christian Liensberger and I will be presenting at TechEd EMEA in Barcelona Spain next week.  For those of you who will be attending, I hope we can get a chance to talk.  Although Christian and I will be spending a lot of time in the "Ask the Experts" section, please feel free to email me (using the Email link on the left sidebar) to set up a one-on-one meeting. 

These are the talks that we have lined up for the conference.  I hope to see you there!

WIN201 - Introduction to Microsoft Sync Framework (Liam Cavanagh)
Tuesday, November 11 09:00 - 10:15
In this session we will introduce Microsoft Sync Framework. Attendees will learn about the capabilities of the synchronization platform and see a demonstration of data collaboration across devices and services. We will use this demonstration as a guide in our walk-through of the key components involved in building an application using Microsoft Sync Framework that will enable any data store to exchange information.

WIN04-IS Sync Framework - Deep Dive (Christian Liensberger)
Tuesday, November 11 10:45 - 12:00
tive Session
Learn about the newest features that have been build into Microsoft Sync Framework version 1.0 and how they help you to synchronize your data with various devices, computers (PC to PC) or even services in the cloud. In the beginning of this session we are going to deliver a short overview of the Microsoft Sync Framework’s 101:

• What are the advantages of Microsoft Sync Framework
• How does synchronization work with the Microsoft Sync Framework
• How does Microsoft Sync Framework make synchronization easier and more efficient

After this short dive we will write a custom provider for the Microsoft Sync Framework to showcase some of the hooks that you might find interesting when getting started with writing own providers. After the session you should be able to write custom providers for the Microsoft Sync Framework to enable the synchronization scenarios in your own applications. 

- Liam Cavanagh