The Synchronizer : SyncServices

What’s new in Sync Services for ADO.NET 2.0?

Rafik — Fri, 14 Dec 2007 01:44:00 GMT

As you know we’ve released the V1.0 of Sync Services for ADO.NET as a standalone package. It is also bundled with Visual Studio 2008 with additional Sync Designer experience. Sync Services is one component in a comprehensive sync platform offering from Microsoft called Microsoft Sync Framework that we unleashed recently. The Sync Framework is publicly available as an early CTP release. With it, you will find a new version of Sync Services for ADO.NET that adds on top of V1 components a new provider called DbPeerSyncProvider. Well, let’s talk a little bit about this new guy.

DbPeerSyncProvider Overview

Sync applications tend to evolve over time; let’s say you started by adding offline capabilities to your application, your customers loved it, but then they ask for more. They wanted to have the same experience on the device; well, no big deal, you tweaked your app to work on devices as well as desktop. Both flavors of your app take data offline and synchronize with some useful service you have on the cloud. Now, some savvy customers will come to you and say, “My data plan is expensive, can I synchronize my mobile device with the desktop instead of going to the web?” others might say, “we need to be able to synchronize with one another in an ad-hook fashion!” ... So now you have a growing demand to support a different model than that of a central server database. You need to build peer to peer sync system. Your central database will become yet another peer(s) in an ever growing topology!

This scenario is not uncommon, but technology seems to lack big time when you move to loosely-coupled multi master sync scenarios. This is where DbPeerSyncProvider comes in to fill this gap.

DbPeerSyncProvider Architecture

You will feel at home when looking at the DbPeerSyncProvider interface. It follows the same programming model introduced with DbServerSyncProvider. Let’s see what commands we have on the provider and sync adapter:

Provider Commands

The provider interface contains thee commands that you need to define. These commands are used by the provider to manage key information about this node including metadata summary for the contents on this node. This summary is formally known as ‘knowledge’ object in the sync framework. Using the knowledge object, one can tell very quickly if two nodes are in sync or not. The provider commands are:

SelectScopeInfoCommand
UpdateScopeInfoCommand
SelectNewTimestampCommand

The ‘scope’ refers to the set of tables that this provider synchronizes. A given node could have multiple scopes. This is similar to the table group concept in the client\server provider terminology.

Adapter Commands

Each adapter represents a table in the scope on the node. The sync adapter interface carries nine commands that can be further categorized as follows:

Change Enumeration
Just one command is used to enumerate incremental changes from the database. This is different from the DbServerSyncProvider where we have three commands instead.
Change Application
As you might expect, there are three commands for applying inserts, updates and deletes. In addition there is another command for getting a hold of a given row or its metadata. This command is mostly used during change application.
Tombstone Cleanup
The peer provider needs to record in its summary the fact that tombstones have been deleted. That said, the provider will use this command in which you, the developer, defines the criteria for deleted rows to be cleanup (tombstones older than 10 days, for example), to find those tombstones and record the needed metadata for you then delete the rows. No worries, this command is just a simple select statement, the provider will do the heavy lifting for you.
Metadata Management
In the p2p sync world, metadata is king! By this statement I am trying to underline the importance of the metadata management and the ability of the provider to tweak them as it sees fit. There are three metadata management commands that give the provider the control to insert, update and delete metadata for a given row.

This just a quick introduction to our new provider, there is a lot to talk about as we move forward with building sharing applications using this provider. Check out the documentation on the sync dev center, and give the bits a test drive!

More to come…

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Top 10 Hotels in Myrtle Beach, Best Hotels in NYC,Romantic Hotels in Seattle, Kid Friendly Hotels in Orlando and Top 10 Hotels in San Diego. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Stay Away From Auto Increment Columns

Rafik — Thu, 28 Jun 2007 20:26:00 GMT

I had few conversations on the sync services forum about that ability to synchronize a table with auto-increment column, also known as identity column, as the primary key or unique id for the row. It is rather common to see this pattern of primary key set to auto increment integer type in every table and then establish foreign key relationships across different tables in the database. Unfortunately, auto-increment columns are not designed with sync in mind.

Unique ID

Before we go any further, let me state this clearly that sync services must be able to uniquely identity each row undergoing sync. By definition, the unique key must not be reintroduced in the system by any other node.

You might get the feeling that this requirement is too restrictive and it would hinder many scenarios. Well, you will be surprised to know that this is such a basic requirement that all sync and replication systems are built upon! For example, Merge replication adds a GUID column to every table it synchronizes, you cannot delete that column (try it); file replication service in windows uses FID that NTFS generates to uniquely identify each file; active directory replication requires a GUID for every object …etc

Data Caching Scenarios

For download only or caching scenarios where the data is cached on the client for easy reference, there is no problem in synchronizing auto increment columns. This is because the server is the only node generating row IDs which is guaranteed to be unique across the system. Clients are not changing the data thus eliminating any ID collisions.

Bidirectional Sync Scenarios

When you have many clients generating same ID and uploading data to the server, the problem manifests itself clearly. Many clients will end up generating a row with ID of 17, for example. These rows are actually different objects yet have the same identity.. This scenario does not work. I know some people tried few tricks to get this scenario to work, which I acknowledge is possible for simple cases, but once you add foreign key relationships to the mix the complexity will shot high up, take my word for it.

Possible Solutions

Use Unique Keys
The best solution and will save you a lot of effort on the long run, is to eliminate the risk of reusing the same key elsewhere in the system by generating a globally unique key for every new row. Few ideas of a unique key generation:

Use Guid
Use a hash of machine unique MAC address + local incrementing number (i.e. timestamp, or application controlled number)

Online Insertions
If the row inserts goes to the server directly then the server will be the only node that generates IDs which guarantees uniqueness. That said, if you must use auto-increment columns and your clients does not make inserts very often, then you might want to consider making inserts directly to the server database followed by a sync to get the rows back to the client database. Updates and deletes will remain offline.
ID Ranges
This is the same technique used by merge replication today. It defines a unique range of IDs for each client and thus avoids any overlaps between clients. Sync Services allows you to do that only once when the schema is created on the client. Unlike merge, clients cannot go back to the server to request a new range. This might be addressed in future versions of the sync services based on the feedback we receive from developers.

This is a important problem for us and we are looking into addressing this in future versions with new and innovative techniques but for now try to avoid auto increment columns for your offline applications.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Top 10 Hotels in Myrtle Beach, Best Hotels in New York City, Romantic Hotels in Seattle, Hotels with a kitchen or kitchenette and Top 10 Hotels in San Francisco. Your feedback is welcome on twitter.com/raveable, raveable blog.

Sync Events for Conflict Handling, Progress Reporting, Business Logic …

Rafik — Wed, 06 Jun 2007 18:23:00 GMT

Let me just say that synchronization services eventing model is so rich. It is designed to enable you to build many of the features that business applications demand. Some of those features that come to mind are:

Monitoring and progress reporting
Conflict detection and resolution
Tracing and debug logging
Business logic
Auditing
Schema creation tweaks

So what are those events that I am talking about? well, let me go through them here. To begin, I'd like to divide the events into two categories, provider events and agent events.

Sync Provider Events

Change Enumeration Events

SelectingChanges - This event is fired prior selecting changes in any of the tables in a group. The event is at group level.

ChangesSelected - This event is fired after the provider concludes enumeration of changes for all the tables in the group. This event is at group level.

Change Application Events

ApplyingChanges - This event is fired prior applying changes in any of the tables in the group. The event is at group level. You can access the changes dataset through this event and apply any kind of business logic your scenario demands.

ChangesAppliedEvent - This event is fired after the provider concludes application of changes for all the tables in the group. This event is at group level. The dataset of changes along with the set of conflicts encountered while applying are also available from the event arguments. This way you could log them to a file for auditing or tracing purposes.

ApplyChangeFailed - This event is fired when an attempt to apply a row resulted in a failure. This event is at row level.

Progress Event

SyncProgress - This event is fired at the table level after selecting changes from a given table. The event is also fired at the row level after applying a given row.

Schema Creation Events (Client Provider only)

CreatingSchema - This event is fired prior to creating table schema on the client database. This event is at the table level. The schema is represented by a DataSet object and using this event you could tweak the schema definition if needed.

SchemaCreated – This event is fired after creating a table on the client database. This event is at the table level. The connection and transaction objects are also available through this event so that you could execute extra commands in the same transaction. This is useful in case you want to add an index or a constraint to the table that just got created.

Sync Agent Events

Session Events

SessionProgress – This event is fired in the context of Synchronize() call. It fires at pre-defined checkpoints in the synchronization process: reading metadata, creating schema, uploading changes, downloading changes and writing metadata. If you are trying to show synchronization progress bar in a form, for example, then this event is the one you would want to use.

SessionStateChanged – This event is fired to indicate the state of the agent session which is either ready of synchronizing. The event is fired in the context of Synchronize() call.

As you can see, you have many points to plug in your code and perform myriad of tasks while synchronization is in progress. Many of these events were driven by the experience we have working with SQL Server merge replication customers. I'd be interested to know what you think and if these events would help you building your custom logic.

One final note before I go is that not all the events listed here are available in the current beta 1 bits; few are coming up in beta 2.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in NYC, Best Hotels in Seattle, Top 10 Hotels in Myrtle Beach, Kid Friendly Hotels in Chicago, Hotels with in-room jacuzziand Best Hotels in Miami. Your feedback is welcome on twitter.com/raveable, raveable blog.

Avoiding False Conflicts Resulting From Interrupted Sync

Rafik — Wed, 09 May 2007 01:52:00 GMT

A great thread on the sync services forum yesterday brought up this interesting question on what happens if the sync session is interrupted due to connection failure or client application crash …etc. To visualize the scenario in your mind, picture an n-tier sync app which uses a web service to upload changes to the server. At sync time, the web method was called to upload changes to the server; the server applies the changes but prior to returning to the client the connection was dropped. In this case, the client has not received the acknowledgement from the server and thus didn’t save the new sent anchor. In the next sync, the client will attempt to upload the changes that it sent last time along with any new changes to the server. Now, the question is how can you, the developer, deal with duplicate application of same rowa and what can the DbServerSyncProvider do to help you out?

Let me answer this by going back to the tracking infrastructure that you have seen in the samples. Recall from the demos that one of the tracking columns added on the server was originator_id column. This column records the id of the client which makes the last change to the row. If the change was done on the server (outside of sync), the value of this column is reset to the server id instead (which is typically 0)

The originator_id column is needed for two reasons:

1- Breaking the Loop
Changes uploaded by a given client should not come down when the same client download changes. originator_id column helps you filter out those changes

2- Interrupted Sync
When the client app crashes or the connection to the server is dropped the sync is interrupted. Some of the in-flight changes might have been processed already but the sync anchor which is persisted after successful sync have not been saved yet which means that in the next sync some of the already-processed changes will be resent. This situation results in false conflicts. However, you can avoid these conflicts using originator_id knowing that a client change cannot conflict with its own pervious change. In other words, if client A uploaded a change to row R, then the same change (or even new change) to R was uploaded again; there should be no conflict as long as the row on the server has A as the originator_id. If R was updated on the server (i.e. by different client) in the mean time, then conflict is unavoidable.

How can you implement this logic in your commands?

Actually, this logic is already implemented in all demos and also generated by the SyncAdapterBuilder. Here is a quick recap:

For update and delete commands, the where clause skips the conflict detection check (update_timestamp <= @sync_last_received_anchor) if the pervious originator_id is same as the synchronizing client (update_originator_id = @sync_client_id) as show below:

Update Command:

update [orders]

set [order_date] = @order_date,

[update_originator_id] = @sync_client_id
where (update_timestamp <= @sync_last_received_anchor or update_originator_id = @sync_client_id) and [order_id] = @order_id
set @sync_row_count = @@rowcount

Delete Command:

delete from [orders] where (update_timestamp <= @sync_last_received_anchor or update_originator_id = @sync_client_id) and [order_id] = @order_id
set @sync_row_count = @@rowcount

Insert Command:

There are two ways to avoid false conflicts when inserting a row:

(1) Optimistic: Let it fail hard and deal with the conflict later on (as explained below)

(2) Pessimistic: Before applying the insert, make sure that the row does not exist already. There is an obvious race condition between the first check and the insert though

The demos typically show simple insert command. For interrupted sync scenario, the second insert of the same row will fail and throws exception; the DbServerSyncProvider will surface the error as ApplyRowFailedEvent. With this event you could decide to either skip the row or retry with the force write flag set. In the latter case, your command has to execute an update instead of insert when the flag is set, as follows:

if @sync_force_wirte = 0

insert into [orders] ([order_id], [order_date], [update_originator_id])

values (@order_id, @order_date, @sync_client_id)

else

update [orders]

set [order_date] = @order_date,

[update_originator_id] = @sync_client_id
where (update_timestamp <= @sync_last_received_anchor or update_originator_id = @sync_client_id) and [order_id] = @order_id
set @sync_row_count = @@rowcount

I tend to prefer the optimistic approach since such failures are typically very low percentage. However, the pessimistic approach would perform better if the probability of interruptions is high.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in Myrtle Beach, Best Hotels in Seattle,Top 10 Hotels in Miami, Hotels with in-room hot tub and Kid friendly hotels in San Francisco. Your feedback is welcome on twitter.com/raveable, raveable blog.

Sync Services: When should you author your own table schema for your offline clients?

Rafik — Thu, 26 Apr 2007 04:35:00 GMT

The question of creating the schema on the client store always pops up in almost any sync related conversation. Flexibility is desired but simplicity is a must! I can understand that and I believe that the sync framework has a good balance between both (feel free to tell me otherwise)…

In one of my pervious posts I touched on schema creation logic that is implemented by the DbServerSyncProvider today. Automatic generation of the schema based on the SelectIncrementInserts\Updates command provides the ‘simplicity’ aspect, while authoring the SyncSchema object and pass it to the provider reflects the ‘flexibility’ aspect of the framework. Now, I want to talk a little bit more about when you would want to author you own SyncSchema object:

Custom Schema Scenarios

Incomplete automatic schema creation
In some cases, the schema generated using the select command is not complete. This typically the case when ADO.NET provider for your backend database is unable to get needed schema information. For example, the Oracle ADO.NET provider does not return the precision and scale information for numeric columns. See for yourself how I addressed this in Oracle demo.
DataType mapping
Different database brands define different datatypes.Now, if you are using SqlCe on your offline clients, then you must somehow map forigen data types to those that SqlCe understands. You do that through SyncSchema.
Customizing the schema for each client
If you decided to implement some sort of identity ranges for your client such that each client new inserts will generate different identity value that is garanteeded to be unique, then you would need to customize the schema for each client. I should not that unlike SQL Server Merge Replication, sync services does not define a way to get a new range once the existing one expires. But anyways, you get the idea; if you need to pass some extra info for each client during schema creation you could author the schema yourself, or intercept the auto generated schema and edit it prior to passing it to the client.
Custom server provider
If you don’t have access to the relational store directly (for example, you have to use API or web service to get to the data) or you have non-relational store and thus cannot use ADO.NET; you would need to implement a custom server provider. The only way to describe the schema for your clients in this case is through SyncSchema object.

I should note that in the case you know the schema in advance and you are developing the client app too, then you have the option of just creating the tables on the client database when you app is setup or even ship the app with a SqlCe file that has the initial content which could include the schema along with some initial data or just the schema..

This is a very rich topic and I think I will talk about it some more later on, stay tuned …

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in Myrtle Beach, Best Hotels in Seattle,Top 10 Hotels in Miami, Kid friendly hotels in San Diego, Hotels with a kitchen or kitchenette and 5 Star Hotels in Las Vegas. Your feedback is welcome ontwitter.com/raveable, raveable blog.

Sync Services: How to partition data for your offline clients?

Rafik — Sat, 07 Apr 2007 10:01:00 GMT

Offline clients are not born equal; one client would like to sync a lot of server data while another client might just wish for its small share of rows that pertains to the task at hand. Sync services framework recognizes the need for data partitioning and so in this post I’ll try to give some details on how to do that. Let’s start by identifying two type of partitioning: vertical and horizontal.

Vertical Partitioning

In vertical partitioning a sub set of the columns is synchronized with the client application. So let’s say you have a table with 20 columns and you decided to select only 5 columns for synchronization purposes. That can simply be achieved on the SyncAdapter commands for that table. Note that for bidirectional sync, columns that are not synchronized need to have either a default value or the insert and delete commands would set the value for those columns using some custom logic.

To change the horizontal partitioning for different clients, your commands could find what type of client is synchronizing (manager, employee, etc) and branch to different column selection statement.

Horizontal Partitioning

This is the more interesting one; let’s say that each sales person is assigned to a given state. In this case, sales person in state of CA would be interested to get the customer information located on that state and not the rest. This is also known as filtering. There are multiple ways to implement that:

The assignment information is already known to the server, then the adapter commands can get the state value first for the synchronizing client and use it in the where clause of the select incremental commands.
The client can pass the filter value to the server directly, this can be done using the CustomParameter collection on the SyncAgent interface. The parameter value is then sent to the server during the sync session and can be used directly in the where clause to partition the data.

There are few interesting issues in this scenario though:

What if a customer relocated from CA to OR, how can that be reflected on the sales person overseeing CA? This goes back to the tracking infrastructure that you have on the server, your triggers can catch such update to the customer and record it such that the next time CA sales guy synchronizes he would see a delete for that customer info. The OR client won’t have any problem when the customer row surfaces in his partition.
What if the sales person changes his coverage area from CA to OR? Well, the client application needs to reset its anchors and data, this can happen by simply deleting the database and let the client synchronizes again.

This was just a quick overview; I am sure many of sync developers would need to do something along these same lines. Give it a try and post all your questions to the public forum, my blog, or email. Have fun!

Deep in Sync: Handling PK-FK Constraints

Rafik — Sat, 24 Mar 2007 02:04:00 GMT

Roger Jennings in his most recent blog post asked an interesting question on how sync services deal with one-to-many relationship between tables undergoing synchronization? Roger is the most active blogger I know of and he is watching sync services closely. It is no secret that his blogs is one of my favorites. I check it at least once a day if not more! I cannot wait to see his upcoming article on sync services next May along with the test harness that he is working on.

Now back to the question, the one-to-many relationship in relational databases is implemented as PK-FK constraint. In my demos, I showed two tables but did not have PK-FK constraint established. The reason is that I wanted to do some rodom actions on the database and having such relation makes it rather complex (and I am lazy too ;-) ). I should consider fixing that in later demos though.

So let’s imagine for a minute that ‘order_details’ table has a foreign key that references ‘orders’ table’s primary key. Now, how sync services deals with such setup and will it be able to synchronize the data without constraint failures? The quick answer is yes, sync services is capable of dealing with PK-FK relationships between tables. However, as you experienced from the demos, the sync framework is very open model and correct implementation is required to get sync to work properly. To achieve that you need to understand few basic sync concepts. Nothing new here since the demos have already shown all that but did not elaborate on why they are needed; now it is the time to dive deep in sync:

Grouping PK-FK Tables Together
The framework provides SyncGroup type for this very reason. With SyncGroup, you instruct the runtime to move changes of set of tables as one unit. This is particularly important when the tables have PK-FK relationships. This way, FK rows in one table will go with their PK rows in the other table.

Tracking Row Creation
Recall that in all the demos I added ‘create_timestamp’ column on every table along with ‘update_timestamp’. ‘create_timestamp’ is needed to indicate when the row was created. This is important to distinguish inserts from updates but more importantly it guarantees that a given row will be enumerated as an insert even if it is beeing updated very frequently. That said, hot rows won’t escape enumeration at least once. This is critical for PK rows because its is required for subsequent FK rows.

Change Application Order
As you can see, the two things I mentioned above guarantees that PK-FK changes will either come togather in one dataset or at least PK rows will come before their FKs. The only thing left is to ensure that PKs are inserted prior to FKs. For deletes, FKs are deleted first then PKs. This is done by the sync providers. On the client side, the order of SyncTable objects in the agent’s SyncTableCollection defines the order for applying inserts. On the server, the order of SyncAdapter objects in the provider SyncAdapterCollection defines the order for applying inserts. Deletes are applied in reverse order on both sides.

Hopefully, I was able to give some insights on how the framework handles PK-FK relationships. I would encourage you to experiment the scenarios you have in mind and share your feedback with the sync team through the forum.

Sync Services: Thoughts on Schema Creation

Rafik — Wed, 28 Feb 2007 20:09:00 GMT

In typical database to database sync scenarios where the backend database is SQL Server, the sync framework creates the schema for the tables undergoing synchronization on the client database. All that is needed is to have the SelectIncrementalInsertsCommand or SelectIncrementalUpdatesCommand supplied on the SyncAdapter object corresponding to the table in question. The sync runtime queries the database for the schema and provide it to the client as a dataset. This logic also works with none SQL Server backend however this process is subject to failure. The reason is that the schema obtained from the server might, and most likely will, have different data types which SqlCe on the client does not know how to map to its supported data types. Even with SQL Server as the backend, some data types are not supported on SqlCe and will lead to the same failure.

In no so typical scenarios where the server is not a relational store, the runtime won’t be able to find the schema in the first place.

There has to be a way to address table schema creation issues. The SyncServerProvider has a property called SyncSchema. This property is set to null by default. However, if your scenario fall one of the above scenarios, then you ought to take a look at the SyncSchema type. SyncSchema has the same look and feel of a DataSet with few more additions to simplify setting up a table schema. By passing a SyncSchema instance to the SyncServerProvider, you explicitly disable the dynamic schema query logic and instruct the provider to pass your custom made schema instead.

While dynamic generation of schema is an attractive feature, it does not come for free. Therefore hand crafted SyncSchema object is still a good practice even if there is no type difference issues between server and client tables.

A nice gift from SQL Server 2005 SP2 to sync developers

Rafik — Tue, 20 Feb 2007 22:55:00 GMT

If you haven’t heard the news yet, SQL Server 2005 SP2 is out! Now if you are like me, thinking about data synchronization for the most part of your day, then you should be really excited about SP2. Let me tell you why.

Recall that in all sync services demos in this blog, we used timestamp tracking column to know when the row was updated. This works fine if there were no uncommitted transactions on the SQL server backend. So what would happen if there was a pending transaction? Well, changes made in that transaction will not be enumerated by the SelectIncrememental commands, that’s expected since dirty data is not visible in read committed isolation level in which we typically operate. When the transaction is committed sometime later on, the timestamp value of the changed rows could be older than the one the client stores for the next sync. Timestamp value is captured at statement execution time and that poses the problem for our simple tracking approach. Basically, next time the client synchronizes it will miss those changes with older timestamp. This is not cool and not acceptable of course!

To clarify this point more, let’s look at a timeline diagram:

As the diagram shows, timestamp is monotonically increasing with every change made in the database regardless if the change is committed or not. Transaction T2 is committed therefore all its changes (from 53-68) are visible. Transaction T1 on the other hand has not committed yet, changes at 50, 51, 52 and 60, 61, 62, 63 are not visible. In the diagram, the sync request for a new anchor came back with timestamp value of 61; therefore all sync enumerations will go up until this point for committed rows only. After T1 commits, all its changes will be visible. In the next sync, changes from 61 to N (70 for example) will see T1 changes of 60, 62 and 63 but will miss 50, 51 and 52. Too bad!

But hey, sync team is part of SQL Server and the SQL engine guys are next door. So we asked politely, of course, for a little favor to somehow be able to get the timestamp of the oldest uncommitted transaction. Well, guess what? Our request was granted and SP2 has this little function (min_active_rowversion()) for sync services developers.
Let’s put the new function to test. Below is a simple SQL commands to show the min_active_rowversion in action in the same database and table I used in the demos:

select min_active_rowversion()

select @@DBTS

select * from orders

begin tran

insert into orders(order_id, order_date) values (5, GetDate())

insert into orders(order_id, order_date) values (6, GetDate())

commit tran

The result I got on my test server is as follows:

	Select @@DBTS	Select min_active_rowversion()
Before Tran	0x0000000000000BC4	0x0000000000000BC5
Begin Tran + Inserting 2 rows	0x0000000000000BC6	0x0000000000000BC5
Commit Tran	0x0000000000000BC6	0x0000000000000BC7

Also try opening another query window to experiment when an older tran is committed the min_active_rowversion function will return the timestamp of the next oldest pending tran in line! Beautiful!

Back to sync services; to take advantage of this new function simply update the implementation of GetNewAnchorCommand on the server provider to “Select min_active_rowversion() - 1” with this little change you’ve solved a serious data consistency problem! Viva SP2!

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle,Best Hotels in Miami, Best Hotels in San Francisco, Hotels with in-room jacuzzi and 5 Star Hotels in Los Angeles. Your feedback is welcome on twitter.com/raveable, raveable blog.

Sync Services: What can SyncAnchor do for you, today?

Rafik — Sun, 18 Feb 2007 10:32:00 GMT

If you had time to browse the documentation for Synchronization Services for ADO.NET, you might have seen a class called SyncAnchor. You don’t deal with it directly and none of the demos in my blog interact with it directly except the server command called GetNewAnchor which returns a new timestamp value. However, as you build up your sync knowledge you will appreciate what this type can do for you. Hmm, have you noticed that too? Truly powerful types appear low profile from far!

The anchor in sync services is that marker that the client stores after every successful session in order to use it next time to tell the server where it left off in the last sync session. The server provider generates a new one for the new sync session. The changes between the old and the new anchors are enumerated and sent to the client. In simple scenarios, sync anchor will be something like timestamp, datetime or any monotonically increasing number that your server store maintains. However, it could carry much more than a number. You could overload the anchor to hold the date and time of when the sync took place; this is useful for detecting rouge client who did not sync for very long time that the server has lost track of some old changes - tombstones for example - thus synchronizing with such client could result in data loss or non convergence. In addition, you could also store a token for changes that the server could not pass on to the client in the current sync and would like to try aging in the next sync. The possibilities are endless.

One thing you need to be aware of is that the anchor value is serialized by the server provider prior to sending them to the client and desterilized before they are used at the server. Your application needs to follow this protocol to plug smoothly into the sync runtime and leverage the power of the sync anchor.

Tip: I had very interesting discussions in the forum, make sure to check it out

Sync Services: Periodic Tombstone Cleanup

Rafik — Fri, 16 Feb 2007 09:12:00 GMT

One of the important topics in data synchronization is managing metadata used to track changes on the store. Just a quick reminder of the different kinds of metadata used in the demos so far:

Adding inline columns to each table we plan to synchronize. Sync services runtime along with database triggers maintains the values for theses metadata columns.
Adding tombstone table to track deletes

As time goes by the deletes accumulate in the tombstone table and the size of that table grows. This is a problem and you should be concerned about it. Pretty much all sync systems have to deal with this problem in one way or another; sync services is no exception. So the question now is how can we manage the size of the tombstone tables?

A common approach for tombstone cleanup is setting up an “expiration policy”. Simply add a datetime column to the tombstone table and setup a job that runs every couple of hours (or at non peak times) to cleanup all records that are older than a couple of weeks for example. This is one of the knobs that you would expose to your system admins to tweak based on their environment workload.

The tombstone expiration policy has some side effects though. Consider this; what happened to a client synchronizes with the server after a period longer than that of the expiration policy? Basically, deletes for some of the rows are gone. There is only two options here, either fail the client sync request and ask the client to reinitialize its local database, or let it proceed without synchronizing the deletes. Most sync solutions fails the client sync request thus force it to reinitialize. This option guarantees conversion.

Synchronization Services for ADO.NET Vs SQL Merge Replication

Rafik — Sun, 04 Feb 2007 08:51:00 GMT

One of the common questions I get from developers once they get their hands on the new sync framework for ADO.NET is: How is it different from SQL Merge Replication? When should I use merge as oppose to sync services? Can sync services do filtering, ddl replication etc …? These are all interesting questions, so let’s talk about the scope of each product:

Synchronization Services for ADO.NET is a development framework for developing occasionally connected systems that:

Enables developers to build offline capabilities into their data applications easily and efficiently.
Provides an open model, allowing developers to sync data with different backend databases or any other relational data store
Uses SQLCE to provide small footprint, in-proc client database.
Supports unidirectional and bidirectional sync with conflict detection and resolution.

SQL Server Merge Replication is a powerful replication solution for synchronizing data between SQL Server Databases; it evolved overtime to cover very complex and demanding replication scenarios; today it provides the following features:

Join Filters
Partition Groups
Logical Records
Schema Changes Replication
Automatic data realignment
Column Tracking \ Conflicts detection
Backward Compatibility
Backup and Restore

Obviously, the scope of each product and the target audience is quite different. While developers use Merge Replication today, they don’t have a lot of control on how it works. Sync framework, on the other hand, puts developers on the driver seat, but takes away many of the high-end features to keep the framework simple and applicable to the most common scenarios.

Download the CTP: Microsoft Synchronization Services for ADO.NET CTP release
Feedback: Visit Sync Services Forum