Andrei's Blog

Microsoft Sync Framework, Part 3: Sync Knowledge

AndreiMaksimenka — Sun, 18 Nov 2007 20:45:09 GMT

Before we proceed to an overview of the synchronization process and we'll walk through a simple sync provider implementation, it's important to understand how sync metadata works and what knowledge operations are used by the sync engine and sync providers. This understanding, which can be difficult in the beginning, will make things much more clear going forward when you'll realize how different pieces of Microsoft Sync Framework interact with each other.

In this post I'll briefly walk through the most basic knowledge operations and explain where to use them.

Sync knowledge structure

In the previous post, I've described the sync metadata required to implement the sync solution. Part of the sync metadata which sync endpoint needs to store and maintain is the sync knowledge. Sync knowledge is the compact representation of all the changes which a particular sync endpoint knows about which is used during change enumeration and conflict detection phases of the sync process. As you could deduce from the sync knowledge operations, it has the following components:

Scope clock vector.
Range exceptions.
Single item or change unit exceptions.

Scope clock vector defines the most compact knowledge representation where its size is proportional to the number of endpoints which synchronize with each other. Scope clock vector consists of components which are called clock vector elements. Each clock vector element currently is a replica key associated with the maximum tick count of a change which our endpoint knows about.

Each clock vector contains at least a single element dedicated to the local replica (or sync endpoint which is local for the given instance of sync knowledge). Because different replicas have different and unique Ids, there are no collisions between elements for local replicas for each replica.

Range exceptions are used to associate a clock vector with a range of items, i.e. for items which have item Ids falling into the range the knowledge will be equal to the clock vector of the range exception. As you can imagine the range exceptions consume less space in the sync knowledge structure because they typically specify clock vector for more than a single item.

Single item exceptions are used to associated a clock vector with a single item or change unit. This is the least compact sync knowledge representation and used only in the cases where it's impossible to use other size optimization techniques like scope or ranges.

When life is good and there are no conflicts, no errors during sync or sync interruptions, the sync knowledge remains compact and has just a scope clock vector. When life is not that good and there are errors, conflicts or sync interruptions, knowledge becomes fragmented and single item exceptions or range exceptions appear in it. Those exceptions identify "deviations" for certain items, change units or ranges of items from the scope clock vector. Note that these exceptions are not permanent and go away after subsequent successful sync or when conflicts get resolved and knowledge returns to its normal very compact representation.

Note that in the managed Microsoft Sync Framework API knowledge exceptions are called overrides.

Replica key/map

Replica key/map is defined in unmanaged API as an object which implements IReplicaKeyMap interface. Managed API uses ReplicaKeyMap class. Replica key/map is used to associate replica Ids assigned to sync endpoints with the 32-bit replica keys which are used internally in the sync knowledge to save the space allocated by the sync knowledge. The replica/key map can be serialized and deserialized along with the sync knowledge and typically is not required to be directly used by the sync solution (although it's used internally by the sync knowledge).

Operations on sync knowledge

Sync knowledge implements set of operations which allow it to be serialized (for storage), combined with another knowledge, project the knowledge onto a given item or change unit, exclude an item or change unit change from the knowledge, perform a check whether this knowledge instance knows about a given item or change unit change, discover knowledge structure and some other operations. The knowledge operations are also described in the following MSDN article.

Mapping of remote knowledge to local

The ISyncKnowledge::MapRemoteToLocal method in the unmanaged API (SyncKnowledge.MapRemoteKnowledgeToLocal method in the managed API) creates an ISyncKnowledge knowledge instance which is compatible with the sync knowledge this method is called upon. The reason why this may be required is because as you probably remember the sync knowledge references a replica key/map object and internally uses 32-bit replica keys to identify replica Ids of sync endpoints in the community. If we don't map foreign sync knowledge which uses a different replica key/map the operations on that knowledge may return incorrect results because those 32-bit replica keys may refer to different replica Ids in different replica/key maps. This operation is typically required to be done by the source of sync changes to correctly do containment checks on the destination knowledge.

Change containment check

This operation is used during change enumeration and conflict detection. For example, if an item has last update version associated with it A5 where A is the replica key and 5 is a tick count, the fact whether a given knowledge instance knows about this change can be established with the help of IBasicKnowledge::ContainsChange method in the unmanaged API (SyncKnowledge.Contains method in the managed API). For the purposes of the change enumeration the sync solution should return a change to the sync destination when destination's knowledge doesn't contain the change. Symmetrically, if a destination version for an item or change unit whose change we're trying to apply is not contained by the source's made-with knowledge, this indicates that we have an independent update to an item or change unit done both on source and destination otherwise known as a conflict.

Knowledge combining

The ISyncKnowledge::Union method in the unmanaged API (SyncKnowledge.Combine method in the managed API) merges information from another sync knowledge into the current one. This operation is typically used during the change application.

Item exclusion

The ISyncKnowledge::ExcludeItem method in the unmanaged API (SyncKnowledge.ExcludeItem method in the managed API) is used to indicate in the sync knowledge that it doesn't know anything about the item specified. This operation is used during change application to create exceptions in the knowledge.

Summary

I've just given a quick overview of some sync knowledge operations. Those operations are sufficient to implement a simple sync solution. There are many other sync knowledge operations useful in advanced scenarios which we'll cover later. Also, it's much easier to better understand knowledge operations described above during the practical walkthrough of the sync process which we'll be covered in the next post.

Microsoft Sync Framework, Part 2: Sync Metadata

AndreiMaksimenka — Sat, 10 Nov 2007 10:34:22 GMT

Sync metadata is the cornerstone of Microsoft Sync Framework just like it would be of any other sync solution. The reason why we need to have the sync metadata is obvious -- we need to track data changes and detect conflicts. The ability to track data changes (or change detection) is based on simple comparison of current data state against a previously recorded data state. Similarly, the ability to detect conflicts is based on comparison of the current state of the data against the state of data being applied to the local data store.

Sync versions

Microsoft Sync Framework associates a data state with a version. The version is essentially a tuple consisting of a sync endpoint Id which made a change and logical clock of the time when the change was made. The logical clock is the replica's own clock which is monotonically increasing and thus we a have a guarantee that each subsequent change made to the data on that replica will have a logical clock value associated with the version greater than a previous change to the data made on that replica. For example, if we know that replica A made a change to an item I1 at the logical clock time 5, we record this as the last update version for item I1 as A5. I'll use this notation throughout other posts in this blog. Different replicas in the community can have different logical clock values and different rules of incrementing them (in other words, a logical clock values can be incremented arbitrarily as long as the ever increasing value requirement holds true).

Items can be changed on different replicas. For example, the latest change to item I1 can be done on replica A at its logical time 5 whereas the latest change on item I2 can be done on another replica B at its logical time 12. Therefore, when we receive changes from replica B we should be able to receive the change for item I2 if we don't know this change yet and after receiving of that change our local sync metadata will indicate that we know versions A5 for item I1 and B12 for item I2.

Introduction to the sync knowledge

Typical sync operations like change enumeration and conflict detection require comparison of a sync version for an item against other sync versions. When we do change enumeration, we need to compare sync versions of items on the source against the sync versions on the destination, and send the changes on the source for those items which destination doesn't know about. With the set of sync versions, the typical solution would be sending of all sync versions from the destination to the source. This approach is very inefficient - you need to send the number of versions proportional to the number of items which destination knows about. Clearly, we need to have an optimization.

The optimization here is that we compress all sync versions in the sync endpoint into the single compact data structure which we call knowledge. The knowledge encompasses all changes (in other words, versions of all items) which a particular sync endpoint knows about. It's important to understand that the sync knowledge used in Microsoft Sync Framework is very compact in the normal case - its size is proportional to the number of sync endpoints participating in sync, not a number of items synchronized. Sync knowledge is the great optimization of the sync metadata of a particular replica, however it doesn't completely replace per-item sync metadata. We still need to know when and where (by the virtue of a sync version which answers both questions) a given item was changed.

Tombstones

Just like the name suggests, the tombstones are used to track dead (or deleted) items. When an item gets deleted, the sync endpoint shouldn't simply get rid of an item, rather it should create a tombstone for it. The tombstone for an item has its own version assigned to it and is used to propagate deletions across the sync community. Clearly, over time tombstones can accumulate however luckily Microsoft Sync Framework supports the tombstone cleanup scenarios as well (more on those in subsequent topics). To store cleaned up tombstones, Microsoft Sync Framework suggests sync endpoints to store forgotten knowledge which tracks cleaned up tombstones (or in other words, forgotten deletes).

Per-item sync metadata

What sync metadata do we need to store per an item? First, it's the last update version or the sync version which is assigned by the endpoint which made the latest change to an item we know about. Second, it's the item Id -- we need to identify an item in the Microsoft Sync Framework. Third, we need to track item's creation version which is particularly useful during item resurrection scenarios. And finally, we need to be able to tell whether an item is alive or not.

Summary

So far, we've categorized the sync metadata which sync endpoints need to track into per-item sync metadata such as last update version and creation version, global sync endpoint metadata such as knowledge and optional forgotten knowledge, and tombstones which track deleted items. In some scenarios, tombstones can be stored along with the rest of sync metadata and just identified by a simple IsTombstone flag:

ItemId	Last update version	Creation version	IsTombstone
I1	A5	B1	False
I2	C12	A5	True

This table represents what is commonly called a sync metadata storage. Microsoft Sync Framework offers built-in support for a sync metadata storage based on SQL Server Compact Edition.

In the next post, I'll delve into the sync knowledge and various operations it supports. Stay tuned.

Microsoft Sync Framework, Part 1: Introduction

AndreiMaksimenka — Tue, 06 Nov 2007 07:46:10 GMT

Introducing Microsoft Sync Framework

Microsoft has already or will shortly announce Microsoft Sync Framework at TechEd Europe in Barcelona and at DevConnections in Las Vegas. Microsoft Sync Framework is the revolutionary product which makes possible multi-master data synchronization to be relatively easy implemented and supported by various applications. MSDN web site has just launched the sync dev center which can be used as a starting point for your exploration of Microsoft Sync Framework.

Why Microsoft Sync Framework

Sync is hard. Correction: right sync is hard. Most people, when they think about sync, first think about the synchronization of data between two endpoints. Next, they think about conflicts - particularly about conflict detection, propagation of conflicts, conflict resolution and propagation of conflict resolution. Then people think about scenarios where more than two endpoints synchronize with each other and that's where the problems typically appear. The sync dev center has a great illustration of the multi-master sync:

In general, it's natural for a good data sync technology to satisfy the following requirements:

Be efficient. Change detection and conflict detection should be as efficient as possible.
Sync metadata should be compact. Sync metadata is required to implement any sync solution, however ideally it shouldn't introduce a significant overhead.
Conflicts must not be over- and under-detected. This is a particularly important requirement - no one wants fake conflicts detected or even worse conflicts undetected which lead to data loss.

Microsoft Sync Framework addresses requirements above in the most balanced way in terms of efficiency and sync metadata size. The conflicts over- and under-detection problem is completely solved by Microsoft Sync Framework and is done automatically and transparently for users.

Sync integration levels

There are several sync integration levels possible:

The UI integration level doesn't have a common implementation or store, all sync solutions are simply integrated by the common UI. This approach is the most flexible in terms of store/protocol independence, however different sync solutions cannot inter-operate.
The Sync Metadata integration level requires different sync solutions to negotiate on a sync metadata level (obviously, there are other parts in this equation - sync solutions also need to negotiate on data representation and schema as well). This is the most balanced level and easiest to achieve.
The Protocol level requires sync solutions to negotiate on details of data exchange. This level is also quite flexible however it can be difficult to use by solutions which use different protocols.
The Store level demands all data to be stored in the same way and sync support is built into the store. It's not a particularly demanded level because applications tend to have their own data stores and it's also very to hard to develop a flexible store which could be used by various applications. Still, there can be scenarios where this particular level could be useful, for example databases.

As we'll see, Microsoft Sync Framework currently enables levels 2 thru 4. It definitely supports Sync Metadata level and partially Store level with Sync Services for ADO.NET, and in the future we can expect work done to better support Protocol level (right now Microsoft Sync Framework is protocol agnostic).

What is Microsoft Sync Framework?

Microsoft Sync Framework is a set of services which help customers develop efficient, flexible, protocol agnostic and correct sync solutions. Microsoft Sync Framework offers a set of services to do multi-master data synchronization between arbitrary stores and protocols, it has built-in efficient file sync and synchronization of database records. Microsoft Sync Framework has built-in support for Simple Sharing Extensions.

Microsoft Sync Framework offers both COM APIs and .NET APIs (with the caveat that some services are only available in .NET APIs like Sync Services for ADO.NET).

In future posts I'll cover some parts of Microsoft Sync Framework like Metadata services, Synchronization Engine, SSE interoperability and various tips for sync provider writers. There are services which I probably won't cover because other people in our team can do it much better - Rafik for example is an expert in other areas of Microsoft Sync Framework like Sync Services for ADO.NET which he'll cover in his blog.

Stay tuned

I've started this blog to keep Microsoft Sync Framework users informed about the platform, tricks and techniques to utilize it in the most effective way in various scenarios as well as to get an input from users, so if you're interested - speak up, ask questions and better yet download Microsoft Sync Framework and give it a try. Let us know what you think, what else you want us to support in Microsoft Sync Framework and what can be done differently. Our team believes in a great potential of this technology and we're very interested in making it better.

In the next post I'll write about the sync metadata which is the heart of Microsoft Sync Framework.

Microsoft Sync Framework > Google Gears

AndreiMaksimenka — Mon, 05 Nov 2007 22:13:00 GMT

It looks like Microsoft Sync Framework has officially gone public and a lot of blogs on the Web started associating Microsoft Sync Framework with Google Gears: for example http://www.teknobites.com/2007/11/05/microsoft-sync-framework-google-gears-competitor/ or http://visitmix.com/Blogs/Joshua/microsoft-sync-framework-sse-and-google-gears/. This association is not correct - it's true that Microsoft Sync Framework enables offline scenarios for Web services, however it's important to understand that it also enables much more - unlike Google Gears which is just a browser extension new Microsoft Sync Framework is a complete multi-master synchronization platform which has the runtime and set of APIs which allow anyone implement something like Google Gears and even better - that application would be able to synchronize its data with any client, device or cloud/Web service which supports Microsoft Sync Framework. Moe has a great article in CoDe Magazine which explains how Microsoft Sync Framework has been conceived, implemented and what features it supports (such as built-in file sync, ASP.NET integration and Simple Sharing Extensions for RSS/Atom).

First post

AndreiMaksimenka — Thu, 01 Nov 2007 08:24:00 GMT

This is my first post here. My name is Andrei Maksimenka, I'm a developer in Microsoft Corporation, SQL Data Replication and Synchronization team and work on Microsoft Sync Framework which is going to be announced very soon at TechEd Europe and DevConnections in Las Vegas. In this blog I'm going to cover various things about Microsoft Sync Framework, various interesting technical topics, and also non-technical topics like skiing, cycling, hiking and other stuff which I like.

You can also check out the Rafik's blog entry where he mentions various sessions at DevConnections which will introduce Microsoft Sync Framework to the masses.

Starting on November 5th I'll start publishing interesting stuff. Stay tuned.