The Synchronizer : p2p

Metadata is the Hart and Sole of Peer Sync

Rafik — Fri, 30 May 2008 01:09:00 GMT

Occasionally, I receive questions about using SQL Change Tracking with PeerSyncProvider. People who examine SQL Change Tracking in preview version of SQL Server 2008 have become fans rather quickly and see the value of using it. Using change tracking with ServerSyncProvider scenarios is a piece of cake and will get easier with the VS2008 SP1. Now how can you do the same with PeerSyncProvider?
The short answer is that it is not so easy. Not as easy as with ServerSyncProvider. The reasons might not be so obvious and I see many attempts of writing the commands for PeerSyncProvider adapters to work against SQL Change Tracking. While it might work for very limited scenarios, it is largely incorrect.

Again, the question is why?

Well, let’s think a little bit about the following SyncAdapter commands for the PeerSyncProvider:

InsertMetadataCommand
UpdateMetadataCommand
DeleteMetadataCommand

Notice that these commands are intended to make changes to the tracking metadata. That’s rather interesting and clearly different from the commands you supply to for ServerSyncProvider. Now it is becoming clear that we have a problem since SQL Change Tracking metadata (i.e. CHANGETABLE) are all read-only!

General Purpose Tracking Infrastructure

Remember that SQL Change Tracking is a general purpose tracking mechanism. You enable change tracking for a table with the sole purpose to be able to ask “what have changed since given version?” If applications are allowed to make changes to the metadata then this question is not reliable and thus useless. Making change tracking metadata read only makes all the sense in the world for its general purpose nature.

Direct Metadata Manipulation

An obvious question here is: Why make direct changes to the metadata? Changes to the base table will be reflected in the CHANGETABLE, wouldn’t that achieve the same effect?
Well, for peer to peer data synchronization… you need more control than that. Let’s take the following two cases:

Synchronizing Deletes

Let’s consider the following example, three nodes A, B, C. The nodes are connected as shown in the diagram. Follow the following scenario:

Make a change R on node A
Let A sync with B, B now has R
B deletes R
Let B sync with C. B will send the deleted row metadata to C

As B received the metadata for the deleted row, it needs to persist is. R never make it to B thus a normal delete command won’t generate the metadata we desire as it simply won’t do anything.

As a side note, the delete metadata must be persisted on B. Since B could be connected to another node (A, for example) which have R and thus needs to pass on the delete during subsequent sync. In P2P environment, one cannot make assumption about the other nodes and must maintain metadata for every deletes.

At a first glance one can come up with a clever work around for this scenario which is insert R then delete it right away. SQL Change tracking should record that, right? That’s right but the workaround cannot survive two or more tables with PK-FK relations which is the reality of most databases. Thus this workaround is short lived.

Resolving Conflicts

When conflicts are detected and a winner is chosen, one needs to update the metadata such that this winner will propagate across the topology of nodes. That said, the version of the row needs to be pumped. Now consider the case where the local row was selected as the winner. With SQL Change Tracking, there is no way to make changes to the version as the metadata is not editable.

My point in this post is to answer a common question and illustrate the underlying technical reasons behind the limitation. That being said, you can still leverage SQL Change Tracking for Peer Scenarios albeit you need to use a side table to store the changes to the metadata. I am sure some of the readers will find this as a worthwhile challenge to tackle. Happy Sync’ing!

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Top 10 Hotels in Myrtle Beach, Best Hotels in New York City, Romantic Hotels in Seattle, Hotels with a kitchen or kitchenette and Top 10 Hotels in San Francisco. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Unleash the power of Peer Sync Provider

Rafik — Mon, 25 Feb 2008 20:22:00 GMT

The term 'peer' or more commonly 'peer to peer' leads people to think about sharing and collaboration scenarios. That's good. One of the most common peer to peer applications that quickly jumps to mind is file sharing which became rather popular in the recent years and led to illegal sharing of copy righted material which resulted in large number of law suits that you are well aware of. That's the bad part. This kind of thought when comes to mind discourages people from exploring the technology and uncovering its strength. My feel is that you won't fall in this unfortunate trap and I am just plain wrong.

Let me tell you the main scenarios for which DbPeerSyncProvider is a key enabler:

Collaboration

Two people in a project could exchange information between each other’s without the need for a third node or, in other words a server node. In this kind of setup, data flows in multiple directions and loops are likely to occur between different members of the project. By that I mean, P1 sync changes with P2, P2 sync changes with P3, and P3 sync changes with P1. In collaboration scenarios multiple people are working on the same resources and synchronize their changes between each others. A server node is not required in this scenario but typically added for centralized backup and reporting needs.

Scale Out

When the load increases on your server, you would typically upgrade your server to more powerful one. Buy more RAM, change the processor, or even replace the machine altogether. These are different types of scaling up your system. But if you get a new machine and add it to the existing server such that the load is distributed on both, then that is scale out.

While DbServerSyncProvider scalability is limited to scale up techniques, DbPeerSyncProvider gives you scale out architecture that you might be looking for. In scale out scenarios, your client sync applications are not talking to each others, they are not synchronizing peer-to-peer, they only talking with the server. However, the server is not a single machine any more. It could be a server farm with some scheme of distributed load balancing. Changes on one server are synchronized with the other servers in peer to peer fashion using DbPeerSyncProvider. The client application too will have to use DbPeerSyncProvider despite that fact that it is only synchronizing data with the server.

This is a fascinating setup and not so obvious especially when all we talk about is peer-to-peer sync. Many variations of these two key scenarios are definitely possible. It is all up to your imagination.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in NYC, Best Hotels in Seattle, Top 10 Hotels in Myrtle Beach, Kid Friendly Hotels in Chicago, Hotels with in-room jacuzzi and Best Hotels in Miami. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Sharing Application Samples

Rafik — Tue, 15 Jan 2008 08:08:00 GMT

I originally authored the offline application line of samples to show off the different features of Sync Services. My main goal back then was to provide you with a quick start guide such that you can get into sync quickly. The series of demos has grown to up to eight demos, so as the rate of positive feedback that I’ve been receiving from you and even internally inside the company. Well, for that I thank you!

Now, with the release of our new DbPeerSyncProvider which is included in CTP1 of Microsoft Sync Framework, a new series of demos is also released with it. This time it is part of the SDK. On my machine, the samples are located at: C:\Program Files\Microsoft Sync Framework\v1.0\Runtime\ADO.NET\V2.0\Samples\

I called this new line of demos “Sharing Application” to emphasize the key feature of the peer provider which enables different nodes to share and collaborate on a common data set without the need for central authority. Not to say that having a central database is discouraged, it is just not required.

The SDK ships with five different flavors of the sharing application sample. They share the same user interface and act against the same database schema of orders and order_details tables. If you have tried the offline application demos, you will feel right at home with this new set of samples. I want to give you an overview of each sample and what features of the provider it shows off. But before going there, let’s check out the common user interface for these samples

The first thing to pay attention to in the main form as shown above is the three peers (peer1, peer2, peer3) settings in multiple places. On the top right you configure the machine name where the database is located for each peer (I have all three databases on one machine). On the bottom left, you direct the follow of synchronization through to and from boxes. The UI won’t allow you to select to sync from a given peer to itself. The three conflict generation buttons on the left of the Synchronize button are specific to conflict sample. Other samples might introduce new buttons.

In building multi-master sync application, three nodes are required to exercise almost all possible scenarios and that’s why you see three peers. You can extend the demo and add as many peers as you want; but as far as the demo goes, three peers should suffice.

In the setup directory with every sample there is a bunch of .sql files that you should use to create the peer databases along with all necessary stored procedures and triggers. The peer provider requires snapshot isolation transaction while enumerating changes, which means that you need any flavor of SQL Server 2005. SQL Server 2005 Express should work just fine.

So now you are eager to give it a try, the question is where to start. Let me give you a quick overview for each sample, ordered by complexity, starting with the simplest of course:

SharingAppDemo – Coupled Tracking
Following the most common model used in the offline application demos, here too you start with your base table and add necessary tracking columns to it. For recording deletes, you create a tombstone table for that.
SharingAppDemo – Decoupled Tracking
You soon realize, after trying the first demo, that tracking metadata for p2p sync is rather demanding and adds many columns to the base table that makes you uncomfortable. Well, I’ve gotten the same feeling myself and this demo shows you how to leave your base table schema intact and move all tracking metadata to a side table called “tracking table”. The rest of the demos are built on top of the decoupled tracking technique so you should give a little bit of time to this demo.
SharingAppDemo – Tombstone Cleanup
In the peer to peer sync world metadata cleanup is not taken lightly and the peer provider has a special logic for handling the cleanup for you. You just need to call the right method on the provider interface when you need to cleanup the metadata stored for deleted rows. This demo shows you how to do that.
SharingAppDemo - Column Mapping
In many situations the table schema is different on each node. Despite the diversion in columns names, columns data types are identical which make sync possible. If all nodes decide to agree on a global schema definition, they all could sync without problems. This demo shows you the idea of global schema. It does not use different schema on each node but it maps column name on each node to a global schema name and shows you how that works. If you don’t have this kind of scenario, you could skip this demo.
SharingAppDemo – Conflicts
It is always fun to show conflict detection and resolution logic. It is my favorite anyways. Once you get comfortable with the peer provider normal sync logic, your treat is to check out how conflicts are handled in peer to peer environment. This demo shows you different types of conflicts and different types of conflict resolution including the ability to retry applying the conflicting row in the next sync.

I am very much exited about this new sharing demo series. I think you going to like it too. As always, your comments are welcomed and you feedback is much appreciated.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in NYC, Best Hotels in Seattle, Top 10 Hotels in Myrtle Beach, Kid Friendly Hotels in Chicago, Hotels with in-room jacuzziand Best Hotels in Miami. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.