SQL Server Compact - Compact & Capable : SyncServices

Merge replication vs. Sync Services for Compact

SQLCEBLOG — Mon, 09 Nov 2009 14:12:00 GMT

SQL Server Compact supports predominantly two different sync technologies. “Merge replication” and “Sync Services”. Users might be using one or the other, and interested in knowing the differences between the two technologies, from a use-case point of view. Still, there might be others interested in knowing, which technology better suites their need, by comparing and choosing the right one. One simple rule of thumb here: Merge replication is designed keeping enterprises in mind, where Sync Services is a framework for developer community/social networking/hobbyist programmers etc… (everything else). “Merge replication” is a solution, which is suitable for Enterprises mostly, in preference to Sync services. However, Sync services is an open-ended, extensible framework on top of which, one can do almost everything that Merge replication does. This article concentrates mostly on the merge replication angle in terms of where it makese more sense and where it doesn't.

Where does "Merge" make more sense?

In the enterprise scenario. In my opinion, enterprises like things faster, better (without any issues), and easily do-able/repeatable. If a solution (like “Merge replication”) allows them to do things like “setting up a publication, subscribers etc…”, although with significant coding, people are not going to like it. So, tools and wizards for enterprises are mandatory (Also give them a way to re-do the setups, typically outputting scripts from these tools/wizards). To avoid the setup/upgrade glitches, provide an out-of-the-box integration story, with other often used components (like Windows, IIS, SQL Server, HTTP etc…). Also, tighter integration helps performance. Usually, in enterprise contexts, setting up a “data server and clients” is a long-term commitment, so, tighter integration is fine there. All these are basic requirements for the “Enterprise” oriented “merge replication” technology.

Note: There are many ways in which a tighter integration with other components helps deployments.

1. Efficiency/performance

2. Automation possibility and tooling

3. Better end-2-end support/solution story for the customers

4. Ability to monitor and troubleshoot parts of the system

5. Essentially, you leave all the hardwork to us, and in the end, expect something that automatically works J (Means, reduced cost of development and testing for you)

Also, my take is that “Enterprises like rich (and relevant) features”. Give them something that can easily model/extend their business logic, and that could be of great value to them. Some of the features of “Merge replication”, in this category are,

1. Automatic partition management

2. Custom conflict resolution hooks

3. Business logic plugins to do custom processing, in the process of merge replication

a. Example is, if one wants to charge the client for synching

b. One wants to compute his share whenever a salesman syncs to the server

4. Retention cleanup, which can be used to (weakly) set a stringent policy about incremental syncs.

a. Like, every salesman must sync at least once in a week or every branch office, must sync at least once a day.

5. Incremental schema changes replication

6. Integration with SQL Server mirroring

7. Integration with SQL Server backup and restore.

Other valuable features are, “Automatic identity range management” etc. I hope now it is very clear as to, why “merge replication” technology is good for enterprise businesses. Enterprises do not mind, buying into a special architecture, if that provides value. Enterprises can setup a particular set (at the least) of windows machines, with IIS and SQL Server, and run merge replication. So, a tailored solution, like merge replication is best suited for them.

Where does Sync play better role compared to Merge?

Now, let’s examine another use-case. Incidentally, in this use-case, the features of merge replication, do not seem much relevant. You want to write a stock-alert application, and link it with a stock-tick source on a website. On closer examination, this is also a “data sync” scenario, so, merge replication could be used here. Trying to use merge replication here, elicits many incompatibilities/redundancies, that are not relevant to this use-case.

First of all, the nature of the data source is unknown, it can change while the app is running too. The web-site might use SQL backend now, and something else tomorrow. The transport is not known, it is over HTTP, but, can have any format (plain HTML/Json etc…). There is no need of rich data semantics here, as one is just comparing a single piece time-series data, and deciding to throw the alert or not. Also, one does not need a big database at client, all he does is read and discard (or read, alert and discard). Clearly, using merge replication here is an overkill, and unsuitable. So, you should use “Sync services”.

If you want to write a very quick sync app, and use it anywhere*, “Sync services” is the answer. Because Sync Services, is not tied to any particular, server architecture or transport mechanism, one can use it anywhere*. For quick development of sync apps, Sync Services is integrated with Visual Studio. But, this is only to develop the app quickly, not to deploy it. Visual Studio is not even needed for plain development of any sync app. It is just there to make it easy and fast.

Sync services is amazingly extensible, componentized, customizable. While merge replication answers enterprise use-case, which is a typical use-case for data synchronization, virtually all other use-cases are addressable by “Sync services”. If for some reason, people do not want to use a particular server/transport architecture, (or can’t use a single architecture), Sync services is the way to go. Besides, Sync services is a free platform to develop apps on. It is a great tool to enable non-(traditional)-enterprise related, businesses/users to realize the new models of interaction cropping up almost everywhere now. Sync services can be made to have feature-parity with “merge replication” solution through coding, although, I hope you would agree that it is not the intent of Sync services. I quoted that here, only to make you realize the potential of Sync services platform.

When an open-ended sync solution needs to be developed, Sync Services Framework is the one to use. Many times in usage, applications to sync data, need heterogeneous servers/transports etc. This is a very important capability, for example, for news generation and propagation web-site. News could be present in various web-sites, and also inside documents/databases of different formats. Also, the solution should be open-ended and extensible, so that, it can be easily tailored for a new data source. The importance of “Sync services” in enabling such scenarios, should not be underestimated. Sync Services, is coming out strong, and there is more to watch out in this field, going forward. With this, we conclude the explanations of the technologies, from the use-cases point of view.

The list of features of merge replication and Sync services are tabulated below:

Feature	Supported in merge replication	Supported in Sync Services
Enterprise-centric
Type of the technology	Solution	Framework
Target users	Enterprises with DBAs	Developers/Hobbyists/Social networking Communities
Integration with DBA tools (SQL Server)	Yes	No
More tooling support	Yes	No
Pluggable business logic hooks	Yes	No
Conflict resolution support	Built-in + support for custom***	Built-in + extensible**
Schema propagation support	Initial + any changes	Limited, available by extension**
Partitioning support	Built-in	Available by extension
Server type (State-full or State-less)	State-full	State-less
Integration with Server mirroring (SQL Server)	Yes	No
Integration with Server backup/restore (SQL Server)	Yes	No
Identity ranges management	Yes	No

Developer centric
Type of the technology	Solution	Framework
Target users	Enterprises with DBAs	Developers/Hobbyists/Social networking Communities
Developer platform support	No	Yes, Visual Studio integration provided
Pluggable transport	No (HTTP with IIS server only)	Yes (Web service model is possible, by extension)
Heterogeneous server	No (Only SQL Server)	Any server is good (by extension).
Network architectures	Fixed, 3-tier	Variable, 2-tier to N-tier
Supports web services model	No	Yes
Ability to work with other sync platforms/frameworks	No	Yes (by extension)
Exposed API surface for tracking****	No	Yes

Common functionality provided
Subscribed database deployment	Yes	Yes
Type of change tracking used	ROWGUID	PK or ROWGUID
Sync directions allowed	All	All
Programmability layer	Native & managed	Managed

Semantics provided
Auto-management of dependent tables	Yes	No
Sync granularity enforced	Yes (Publication level)	No (table level)
Schema propagation	Yes	Limited extent
Built-in conflict resolution strategies (also customizable)	Yes	Only some limited number of built-ins

* Well, almost everywhere.

** Whenever we use, “available by extension” or simply “by extension”, that means developer should write code to achieve the desired behavior.

*** (For this document,) the difference between customizable and extensible is the following: customization is a type of extension, where, the application architecture is not affected majorly. Like, register a COM dll, to do conflict resolution etc…

**** Tracking is a mechanism used to tag all changes. Changes could be Inserts/Updates/Deletes on tracked tables. This module is used to detect changes between successive syncs, so that the data can be forwarded to the other party. Exposing the tracking API (enable/disable tracking etc…), helps write applications like peer-peer sync, transaction notifications etc…

Wrap-up:

We have looked at the use-cases where merge replication technology is suitable and where it is not. There are scenarios where “Sync services” comes out as a candidate for solution, and there are scenarios for merge replication too. When considering the right technology for using, there is no silver bullet; there is no panacea that works for all needs. The requirements of your use determine the “right” technology for you. Merge replication provides a solution, but, there are many scenarios that it can’t help you with. Sync services can be made to work in “any” scenario (including the one that merge replication provides out-of-the-box J), but, are you fine with building such a (mammoth) architecture yourself, and incurring the various costs involved? In merge replication, we take the pains and give you a solution, although, for a special, tailored need, is also happens to be the most common and justified one. In sync services, you are on your own, but, it gives many benefits that you can’t do without in many situations.

Contributor: Udaya Bhanu Goteti

Sync Services optimizations

SQLCEBLOG — Mon, 06 Oct 2008 15:40:00 GMT

There are some optimizations that can be done for an "Sync Services" sync scenario. These are, (might not be an exhaustive list)

Small sync chunks (using smaller SyncGroups which are only absolutely necessary)
Index creation on tracking and filtering columns
Transfer encryption of data payload on web
Batching the changes
Filtering the data at various stages to reduce network load

    Sync (synonym for "Sync Services sync", in this post) interfaces with the providers using DataSet objects. That is, sync uses DataSet to store and retrieve changes happening at client and server. DataSets are known to be not very memory efficient, and the amount of memory consumed by these, is a bit of a concern when one is using these in device clients in Sync Services.

    Sync happens in chunks, using the SyncGroups that one supplies. That is, all data in a SyncGroup is logically one sync, and corresponds to one DataSet. Since sync DataSets are one per SyncGroup, and these are needed only till the corresponding SyncGroup commits, we can optimize the memory taken by DataSets, using smaller SyncGroups. Divide sync tables into various SyncGroups, so that, each group has it’s own (smaller) DataSet, and it is disposed (by garbage collector) once the SyncGroup changes are committed. However, to ensure logical consistency, you should always keep the tables which are related (by a foreign key constraint, for example), in the same SyncGroup.

Example:
Tables Order and OrderDetails are related.
Tables Customers and ShippingDetails are related. But, these are unrelated to the first set.

SyncGroup customerSyncGroup = new SyncGroup("Customers");

SyncTable customerSyncTable = new SyncTable("Customer");
customerSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
customerSyncTable.SyncDirection = SyncDirection.DownloadOnly;
customerSyncTable.SyncGroup = customerSyncGroup;

SyncTable orderShipSyncTable = new SyncTable("ShippingDetails");
orderShipSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderShipSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderShipSyncTable.SyncGroup = customerSyncGroup;

SyncGroup orderSyncGroup = new SyncGroup("Orders");

SyncTable orderSyncTable = new SyncTable("Order");
orderSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderSyncTable.SyncGroup = orderSyncGroup;

SyncTable orderDetailSyncTable = new SyncTable("OrderDetails");
orderDetailSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderDetailSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderDetailSyncTable.SyncGroup = orderSyncGroup;

    Rather than having a single SyncGroup with all 4 tables, one can create two sync groups. One for the first set, and the second for the other set.

    Also, to increase the performance of enumerating process at server, one can create indexes on the tracking and filtering columns. As an example, let us say one has two tracking timestamp columns __CreateTime and __UpdateTime. Since, our server side changes enumerating queries always have filtering based on these columns, one should create indexes on these, so that, enumerating server changes becomes faster, and, sync performance increases.

    In case of device syncs, transferring the data on the network/internet is also going to cost more time/resources. Since, transfer of data from a web service/server machine to a device is time/bandwidth consuming, one should look at compressing data sent/received. This is mostly the case, when data being synched is large (Something like, initial synching SQL Compact with SQL Server). One can use System.IO.Compression or your own implementation of compression/decompression algorithm. (More details at http://msdn.microsoft.com/en-us/library/system.io.compression.aspx). Another blog post that refers to this issue and proposes solutions is at, (http://blogs.msdn.com/mahjayar/archive/2008/10/01/dbsyncprovider-improving-memory-performance-in-wcf-based-synchronization.aspx)

    Sync Services Server side supports batching. Please refer to http://msdn.microsoft.com/en-us/library/bb902828.aspx for more details.

    Sometimes, we may have server tables holding and synching data, that is alien to clients. For example, server table has a large GPS map of some place (in custom format) per row, that is associated with the data in other columns in the row. The map is stored as an image at client side. But, since, the data is of proprietary format, client can’t interpret the data, and it is not used anyway at client side. And also, since, it is large data, lot of network bandwidth is consumed and response time is compromised for transferring this across. It is better, if one can filter these columns out at server side, rather than sending to client.

    Another scenario is as follows: The client devices of a Sync application are used by Salesmen of a company, and the server side is the company office, where data pertaining to each salesman is stored in a SQL Server database. When the client devices sync, they are mostly concerned about the data relevant to them alone, and not to other sales persons. Here also, one needs to filter rows from the table.

    To facilitate filtering of rows/columns at client/server sides, one can use a number of strategies. For filtering data sent from client (or at the client side), one can use the callback architecture of "Sync Services". For server side, since you have more control there, one can choose to strip down the DataSet, when one is sending it over the wire/use callbacks to strip off DataSet data/client can supply queries to server side adapters, which filter data.

Explaining each of these options:

Callback architecture of sync: Sync provides various callbacks during the process of Synching. One can hook in the callbacks for accessing/modifying the dataset generated at every stage of sync. For example, one can add/drop some columns to the dataset, after changee enumeration is done at client side. After changing the dataset, effectively, the changes being applied to client database are changed. The same provision is present at server side also.

This callback architecture can be used to modify the DataSet applied/sent to server, on the fly. This is one type of filtering, possible at both client and server.

Server side gives more options when sending data over the wire. First of all, server chnages are enumerated within the "Sync Services" application itself. That is because, server change enumeration is done by queries supplied by the Sync Services client. So, which columns to select from server, can be controlled from the client side, when specifying queries like, SelectIncrementalInserts etc... This is another way of programmatically filtering server data.

One more option to filter data in a N-tier architecture is that, filtering the wire transferred content. In an N-tier architecture, client and server are in two different, but connected machines. Typically, client machine has a proxy server provider, which delegates calls to enumeraion/application of changes etc... to a web service. Web service talks to a backend data store, and services the data and operation requests. In this setting, interesting possibilities like, use of compression over the wire, and filtering over the wire etc... come up. When server side data store gives data, it can do some filtering. Also, the client side proxy can be used to filter irrelevant data. This is another type of filtering.

Example for callback architecture:
For client side DataSet filtering (client side), use SelectingChanges before the enumeration and ChangesSelected after the enumeration
For server’s data set filtering (client side), ApplyingChanges before the application of changes and ChangesApplied after the application
For filtering server side changes (at server side):SelectingChanges before the enumeration and ChangesSelected after the enumeration
For client’s data filtering (at server side):ApplyingChanges before the application of changes and ChangesApplied after the application

In every one of these methods, eventArgs.GroupMetadata and eventArgs.Context.DataSet are the ones need to be changed.

Thanks

Udaya Bhanu,

SDE II, SQL Compact