What's in Store : Synchronization

Synchronizing Data between WinFS Stores

WinFS Team — Thu, 26 Jan 2006 03:12:00 GMT

Hi, my name is Mark Scurrell and I’m a Program Manager on the WinFS Sync team. I’d like to give you an overview of the functionality we provide to allow applications to synchronize data between WinFS stores.

If you haven’t already done so, I would recommend you first read a previous post where Neil Padgett provided an overview of synchronization and described the scenario where data is synchronized between a WinFS store and a non-WinFS application store.

As well as providing a set of services to synchronize with non-WinFS stores, as Neil described, we also want to allow developers to build peer-to-peer applications. So, as part of the WinFS Sync platform we provide services that will allow applications to synchronize their data between multiple WinFS stores without application developers having to design and code the sophisticated synchronization algorithms that are required for peer-to-peer applications.

A good example of a peer-to-peer application that was built using WinFS Sync is Microsoft Rave, a sample application included in the WinFS Beta 1 SDK. Rave allows WinFS data to be shared between users without the need for a server – each user can directly synchronize folders on their computer with other user's peer-to-peer. The developer of Rave discussed his experience producing this application in a previous post.

Let’s start with the simplest use of WinFS-to-WinFS synchronization which is to synchronize two WinFS stores.

So, what happens when synchronization is initiated? WinFS Sync will do the following:

Determine the changes made to each store since the last sync (change enumeration)
Communicate those changes to the other store and update the other store (change application)
When applying the changes, determine if there is any conflicting data – the same data has been changed on both stores since the last synchronization (conflict detection)
Either log the conflicting data for later resolution or have it resolved immediately (conflict resolution).

WinFS stores will contain many different types of data such as files, custom item types, links between items, and so on. WinFS Sync will synchronize all data stored in WinFS; no custom code is ever required, for example, even if a new item type is defined.

Synchronization between two stores is useful; however more interesting scenarios involve multiple stores. With multiple stores there are different ways the stores can be configured to communicate – the sync topology.

The Microsoft Rave sample application, for example, allows multiple users to synchronize, with each user being able to synchronize with every other user in a full mesh topology. Another application, for example, may require a topology with all changes being communicated through a central “hub”. The point that I want to emphasize here is that WinFS Sync is very flexible and has been designed to cater for any topology; there does not need to be a “master” node and true peer-to-peer applications can be built.

I’ve described how WinFS Sync was built to handle peer-to-peer scenarios and different topologies, but how do we handle the fact that sync applications will need to operate in many different network configurations? Here are some possible scenarios for peer-to-peer sync applications:

There are multiple computers on a home workgroup network; a user could synchronize their data between computers so it can be accessed and updated on any computer in the home.
Users are members of a domain on a corporate network and can configure folders for sharing and collaborating with other invited users.
Family members are located in different parts of the country or in different countries; they can share photos with other members of their family by having them synchronize their computers over the Internet.

WinFS Sync has no knowledge of the network configuration or the transport that will be available for communication of the changes. The application developer must provide a transport over which the sync protocol will operate. We define an interface for the transport and the sync application must supply an implementation of that interface to WinFS Sync. The developers of the transport do not have to concern themselves with the complex logic of synchronization; they implement simple methods such as ReadMessage and WriteMessage and WinFS Sync does the rest.

Those of you who have had a close look at our Beta 1 release may be confused as we provided a facility called Store Synchronizer that synchronized data between WinFS stores and included a transport suitable for a local network. For Beta 2 we have decided to focus on providing the peer-to-peer synchronization platform and will not provide any specific transport implementations. We have therefore modified Store Synchronizer so it requires a transport implementation. In the Beta 2 SDK we will of course provide sample code for a transport, guidance on how to build a transport, and reference material for the Store Synchronizer classes.

My main goal with this article is to raise awareness of the set of synchronization services that WinFS provides to support peer-to-peer sync application development and also to highlight the flexibility we allow in terms of diverse sync scenarios, topologies and network configurations.

I hope you found this post useful. There is further information available in our Beta 1 SDK documentation if you want to dig deeper. I would be interested to hear about any scenarios where you would utilize our WinFS-to-WinFS synchronization capabilities.

Author: Mark Scurrell

Getting Data Into WinFS with WinFS Synchronization

WinFS Team — Fri, 18 Nov 2005 04:45:00 GMT

One of the first topics developers ask about once they start learning about WinFS is “How do I get existing data in?” We tend to think of this in terms of a bigger problem: “How do I move a lot of data in and out of WinFS?” Our answer here is WinFS Synchronization.

My name is Neil Padgett and I’m a program manager working on the WinFS Synchronization APIs. The goal of the API set is to provide access to all of the services WinFS provides for developers building sync solutions. But, before we delve too deeply into that, let’s talk for a bit about what exactly synchronization is.

The simplest idea that pops into most people’s minds when they want to get data into WinFS is to write an importer. That, is, they plan to just write a simple application to pull data from some application store and then use the WinFS API to create WinFS entities to represent their data. This is a one way importer.

So, this seems great, right? You’ve got your data and it is moved into WinFS. And this works well assuming you aren’t going to use the non-WinFS application to update the data anymore. But what happens if you want to update the data in the non-WinFS application? Let’s make this a bit more specific, let’s assume we have a contact in the application store and we’ve imported it into WinFS. And then let’s assume that we’re going to go ahead and keep using the non-WinFS application to update the contact.

So this works fine? Right? We rerun the importer periodically and update the WinFS contact from the application store.

And, this will work. It works because we never update the data in one of the two places. This means that we can just overwrite the data in WinFS every time. However, WinFS is a shared data store – that contact is available in a well-known schematized format. And the user may choose to make it available to their other applications. So, others may update it. But, if we run our simple importer, we’re going to lose data. How can we solve this?

The answer lies in detecting (and later resolving) this conflict. But further to that, we want to try and merge together changes that happened on the different stores. This means we’ll need to be able to figure out what changed on each store so that we can try and apply those changed to the corresponding item in the other store.

So, let’s consider our contact again. And now, let’s be more specific about what we’re changing.

Let’s assume that we did some initial sync to ensure we had our contact in both stores. (We can talk about what this exactly means later, but for now we can think of it to be like running our importer.)

And let’s imagine that, after we sync we make some updates:

In the application store, we’ll update:
-Home address
-Telephone

And, in WinFS we’ll update:
-Home address
-Cell

So, now imagine we want to bring our two stores back into sync. So, how do we do that? We know we were in sync before, so we need to figure out what changed on each store, and then apply those changes to the other store. We can call these processes change enumeration and change application, respectively, and we want to do them in both directions. (In fact, WinFS does the hard work of figuring out what changed for us and of making sure that remote changes brought to WinFS are not echoed back to us later.)

Considering our example, we have some changes that are straightforward – the telephone numbers were each changed on the one store, but not on the other. We call these non-conflicting changes. For these non-conflicting changes, we can simply apply them each to the other store. The difficulty comes with the home address – we made changes on both stores – so-called conflicting changes. We’ve detected a conflict and we’ll need to resolve it, either by prompting the user or, more likely, according to some policy (for example, keeping the latest change.) Then, we can bring our two stores back into sync.

WinFS Synchronization is fundamentally about providing services in the store that do just these things: Change Enumeration, Change Application, Conflict Detection, Conflict Resolution either by deferring for manual resolution or via automatic resolution, and other things. We also generalize these services for the case of many stores with arbitrary topologies, and we provide specialized solutions for common cases like synchronizing files or synchronizing several WinFS stores. In upcoming posts I’ll talk about the services WinFS Synchronization provides and how they can be used to solve interesting data moving problems. We’ll also talk about some more interesting scenarios involving multiple synchronized stores with interesting topologies (such as in peer-to-peer) scenarios.

In the comments for this post, I’m interested to hear about how you think you might use WinFS Synchronization and what you’d like me to focus on first in the upcoming posts.

Author: Neil Padgett

Synchronize your WinFS data with Microsoft Rave

WinFS Team — Fri, 09 Sep 2005 03:48:00 GMT

As some of you may have discovered there is a set of applications on the WinFS CD. Some of these are officially “unsupported”, but are included because they demonstrate some of the capabilities of the WinFS platform.

I’m going to describe Microsoft Rave, which is one such application. A little background – I work on WinFS Synchronization, and last year was tasked with building an application that the WinFS team could use to “self host” the sync features. Self-hosting is what product development teams do by internally deploying our code so that we have a day-to-day feel for the product. The idea is to have the product team go through the experiences a customer is likely to encounter, and so help ship a better product.

Microsoft Rave is designed to make it easy to synchronize your data using WinFS. You can synchronize between multiple computers running WinFS, and also between different users, while granting access to only those you want to synchronize your data with. WinFS has the ability to synchronize any of your data (Contacts, Email, Documents, Photos etc) regardless of which application created it. Rave allows you to select your data using Windows Explorer, and set it up for synchronization. You can then invite your friends and colleagues who have also installed WinFS to synchronize your data. Thus, you can share out your photos, contacts, and documents easily and have changes to them automatically replicated to others.

The documentation for Microsoft Rave is included on the CD, so I won’t talk a lot about how to actually use Rave. Instead, I’ll focus on what the experience was like developing Rave on top of WinFS. First, we recognized the importance of providing a secure sharing experience between the various users. Next, we wanted to make sure that replication occurs automatically when changes to the data occur on any of the computers. In addition, there was the need to grant various levels of access to the data. The key, though, is that once synchronization is set up, no further management should be necessary, and that the data should always be kept up to date on all computers.

WinFS Synchronization handles all the platform level details of peer to peer change enumeration and application, conflict detection, automatic conflict resolution and conflict logging. These are described in the WinFS SDK. This represents a huge investment by Microsoft that makes it easy for an application to distribute its data. Rave was built on top of the platform level synchronization services. Rave implements a full mesh network topology to connect every user to every other user of a synchronized folder using WinFS Synchronization. The list of users and computers (shares, really) that synchronize with each other is itself synchronized through the “community.” So, if I invite Shishir to synchronize with a WinFS folder, the membership list is updated to include Shishir. This membership list is replicated to all the existing users that synchronize with that folder. (For those interested in the details, there is a hidden WinFS Item – a Replica – that stores all the sync metadata. This item is stored underneath your synchronized folder. Rave stores the membership list and other configuration data under this Replica.)

Briefly, the replicated data is secured as follows. The first time synchronization is enabled for a folder, Rave creates a new WinFS share (similar to a Windows share) for that folder. It also creates three security groups, one for each of the roles Rave supports (Reader, Writer, and Manager – these are described in the Rave application documentation.) If the membership list changes (either locally or because a change to the list was synchronized in), Rave detects the change (using WinFS Watcher notifications) and updates the memberships to the security groups for the appropriate folder. If you have installed Rave, you can use the WinFS Stores tool to view the shares and their security permissions. You can also use the Computer Management tool to view the membership of each security group, and verify that Rave sets it up correctly. Finally, Rave also sets up item level security to grant access to just these security groups. As with Windows shares, WinFS access checks are done both at the share level and at the data level. Rave handles all these details automatically.

In addition, Rave waits for data changes to occur using the WinFS Watcher notification API. This API make it extremely simple to detect when changes to data occur. I expect this (along with the WinFS Synchronization services) to enable a new breed of easy to build workflow type applications that can react to changes that are made on other computers and then synchronized locally. As an example, consider a factory in which each unit periodically records its status into a local WinFS and then synchronizes this data into a management computer that can react to various situations and possibly even synchronize configuration changes back down to the individual units. Such applications can of course be written without WinFS and do in fact exist today. However, WinFS with its rich type system, notifications and synchronization services, makes this type of application a whole lot easier to build.

There were several things that could have been added to Rave that didn’t make it onto the beta. Enabling synchronizing between WinFS and external data, such as the file system, by using the WinFS File System synchronization adapter, was one such. Another area was a rich conflict policy and resolution user interface. WinFS offers very detailed control over what happens when a conflict occurs. Rave currently uses a last-writer-wins policy in the case of concurrent updates (and a merge policy in case of concurrent creates, which is a topic I won’t go into.)

When I consider what it would take to develop a peer-to-peer application like Rave in the absence of a powerful storage platform like WinFS, I think “several years” rather than the several months that it actually took. Hopefully this post helped describe why – no other general purpose store or file system provides the synchronization support, data change notifications, and security that WinFS does. Implementing a peer-to-peer data synchronization application like Rave in absence of these services would have been a daunting undertaking indeed.

If you’ve started using Rave and have ideas for what you’d like to see, we’d really like to hear from you! Post your comments and we’ll be sure to read them.

Author: Vivek Jhaveri