SQL Server Compact - Compact & Capable : RDA

Merge replication vs. Sync Services for Compact

SQLCEBLOG — Mon, 09 Nov 2009 14:12:00 GMT

SQL Server Compact supports predominantly two different sync technologies. “Merge replication” and “Sync Services”. Users might be using one or the other, and interested in knowing the differences between the two technologies, from a use-case point of view. Still, there might be others interested in knowing, which technology better suites their need, by comparing and choosing the right one. One simple rule of thumb here: Merge replication is designed keeping enterprises in mind, where Sync Services is a framework for developer community/social networking/hobbyist programmers etc… (everything else). “Merge replication” is a solution, which is suitable for Enterprises mostly, in preference to Sync services. However, Sync services is an open-ended, extensible framework on top of which, one can do almost everything that Merge replication does. This article concentrates mostly on the merge replication angle in terms of where it makese more sense and where it doesn't.

Where does "Merge" make more sense?

In the enterprise scenario. In my opinion, enterprises like things faster, better (without any issues), and easily do-able/repeatable. If a solution (like “Merge replication”) allows them to do things like “setting up a publication, subscribers etc…”, although with significant coding, people are not going to like it. So, tools and wizards for enterprises are mandatory (Also give them a way to re-do the setups, typically outputting scripts from these tools/wizards). To avoid the setup/upgrade glitches, provide an out-of-the-box integration story, with other often used components (like Windows, IIS, SQL Server, HTTP etc…). Also, tighter integration helps performance. Usually, in enterprise contexts, setting up a “data server and clients” is a long-term commitment, so, tighter integration is fine there. All these are basic requirements for the “Enterprise” oriented “merge replication” technology.

Note: There are many ways in which a tighter integration with other components helps deployments.

1. Efficiency/performance

2. Automation possibility and tooling

3. Better end-2-end support/solution story for the customers

4. Ability to monitor and troubleshoot parts of the system

5. Essentially, you leave all the hardwork to us, and in the end, expect something that automatically works J (Means, reduced cost of development and testing for you)

Also, my take is that “Enterprises like rich (and relevant) features”. Give them something that can easily model/extend their business logic, and that could be of great value to them. Some of the features of “Merge replication”, in this category are,

1. Automatic partition management

2. Custom conflict resolution hooks

3. Business logic plugins to do custom processing, in the process of merge replication

a. Example is, if one wants to charge the client for synching

b. One wants to compute his share whenever a salesman syncs to the server

4. Retention cleanup, which can be used to (weakly) set a stringent policy about incremental syncs.

a. Like, every salesman must sync at least once in a week or every branch office, must sync at least once a day.

5. Incremental schema changes replication

6. Integration with SQL Server mirroring

7. Integration with SQL Server backup and restore.

Other valuable features are, “Automatic identity range management” etc. I hope now it is very clear as to, why “merge replication” technology is good for enterprise businesses. Enterprises do not mind, buying into a special architecture, if that provides value. Enterprises can setup a particular set (at the least) of windows machines, with IIS and SQL Server, and run merge replication. So, a tailored solution, like merge replication is best suited for them.

Where does Sync play better role compared to Merge?

Now, let’s examine another use-case. Incidentally, in this use-case, the features of merge replication, do not seem much relevant. You want to write a stock-alert application, and link it with a stock-tick source on a website. On closer examination, this is also a “data sync” scenario, so, merge replication could be used here. Trying to use merge replication here, elicits many incompatibilities/redundancies, that are not relevant to this use-case.

First of all, the nature of the data source is unknown, it can change while the app is running too. The web-site might use SQL backend now, and something else tomorrow. The transport is not known, it is over HTTP, but, can have any format (plain HTML/Json etc…). There is no need of rich data semantics here, as one is just comparing a single piece time-series data, and deciding to throw the alert or not. Also, one does not need a big database at client, all he does is read and discard (or read, alert and discard). Clearly, using merge replication here is an overkill, and unsuitable. So, you should use “Sync services”.

If you want to write a very quick sync app, and use it anywhere*, “Sync services” is the answer. Because Sync Services, is not tied to any particular, server architecture or transport mechanism, one can use it anywhere*. For quick development of sync apps, Sync Services is integrated with Visual Studio. But, this is only to develop the app quickly, not to deploy it. Visual Studio is not even needed for plain development of any sync app. It is just there to make it easy and fast.

Sync services is amazingly extensible, componentized, customizable. While merge replication answers enterprise use-case, which is a typical use-case for data synchronization, virtually all other use-cases are addressable by “Sync services”. If for some reason, people do not want to use a particular server/transport architecture, (or can’t use a single architecture), Sync services is the way to go. Besides, Sync services is a free platform to develop apps on. It is a great tool to enable non-(traditional)-enterprise related, businesses/users to realize the new models of interaction cropping up almost everywhere now. Sync services can be made to have feature-parity with “merge replication” solution through coding, although, I hope you would agree that it is not the intent of Sync services. I quoted that here, only to make you realize the potential of Sync services platform.

When an open-ended sync solution needs to be developed, Sync Services Framework is the one to use. Many times in usage, applications to sync data, need heterogeneous servers/transports etc. This is a very important capability, for example, for news generation and propagation web-site. News could be present in various web-sites, and also inside documents/databases of different formats. Also, the solution should be open-ended and extensible, so that, it can be easily tailored for a new data source. The importance of “Sync services” in enabling such scenarios, should not be underestimated. Sync Services, is coming out strong, and there is more to watch out in this field, going forward. With this, we conclude the explanations of the technologies, from the use-cases point of view.

The list of features of merge replication and Sync services are tabulated below:

Feature	Supported in merge replication	Supported in Sync Services
Enterprise-centric
Type of the technology	Solution	Framework
Target users	Enterprises with DBAs	Developers/Hobbyists/Social networking Communities
Integration with DBA tools (SQL Server)	Yes	No
More tooling support	Yes	No
Pluggable business logic hooks	Yes	No
Conflict resolution support	Built-in + support for custom***	Built-in + extensible**
Schema propagation support	Initial + any changes	Limited, available by extension**
Partitioning support	Built-in	Available by extension
Server type (State-full or State-less)	State-full	State-less
Integration with Server mirroring (SQL Server)	Yes	No
Integration with Server backup/restore (SQL Server)	Yes	No
Identity ranges management	Yes	No

Developer centric
Type of the technology	Solution	Framework
Target users	Enterprises with DBAs	Developers/Hobbyists/Social networking Communities
Developer platform support	No	Yes, Visual Studio integration provided
Pluggable transport	No (HTTP with IIS server only)	Yes (Web service model is possible, by extension)
Heterogeneous server	No (Only SQL Server)	Any server is good (by extension).
Network architectures	Fixed, 3-tier	Variable, 2-tier to N-tier
Supports web services model	No	Yes
Ability to work with other sync platforms/frameworks	No	Yes (by extension)
Exposed API surface for tracking****	No	Yes

Common functionality provided
Subscribed database deployment	Yes	Yes
Type of change tracking used	ROWGUID	PK or ROWGUID
Sync directions allowed	All	All
Programmability layer	Native & managed	Managed

Semantics provided
Auto-management of dependent tables	Yes	No
Sync granularity enforced	Yes (Publication level)	No (table level)
Schema propagation	Yes	Limited extent
Built-in conflict resolution strategies (also customizable)	Yes	Only some limited number of built-ins

* Well, almost everywhere.

** Whenever we use, “available by extension” or simply “by extension”, that means developer should write code to achieve the desired behavior.

*** (For this document,) the difference between customizable and extensible is the following: customization is a type of extension, where, the application architecture is not affected majorly. Like, register a COM dll, to do conflict resolution etc…

**** Tracking is a mechanism used to tag all changes. Changes could be Inserts/Updates/Deletes on tracked tables. This module is used to detect changes between successive syncs, so that the data can be forwarded to the other party. Exposing the tracking API (enable/disable tracking etc…), helps write applications like peer-peer sync, transaction notifications etc…

Wrap-up:

We have looked at the use-cases where merge replication technology is suitable and where it is not. There are scenarios where “Sync services” comes out as a candidate for solution, and there are scenarios for merge replication too. When considering the right technology for using, there is no silver bullet; there is no panacea that works for all needs. The requirements of your use determine the “right” technology for you. Merge replication provides a solution, but, there are many scenarios that it can’t help you with. Sync services can be made to work in “any” scenario (including the one that merge replication provides out-of-the-box J), but, are you fine with building such a (mammoth) architecture yourself, and incurring the various costs involved? In merge replication, we take the pains and give you a solution, although, for a special, tailored need, is also happens to be the most common and justified one. In sync services, you are on your own, but, it gives many benefits that you can’t do without in many situations.

Contributor: Udaya Bhanu Goteti

HTTP/Internet Proxy Detection - Using WINCE API

SQLCEBLOG — Tue, 19 Jun 2007 23:28:00 GMT

SqlCeReplication and SqlCeRemoteDataAccess have InternetProxyServer, InternetProxyLogin, InternetProxyPassword properties which the application can set to make these sync protocols communicate via/thru the desired proxy.

However, then the question comes on how to programmatically detect the HTTP/Internet proxy settings from the app to set the properties on SQL CE classes. Here is the short story of how to detect the settings on Windows CE/Mobile devices.

DLL/Module Name and Reference

API

Data Structures

CellCore.dll

Connection Manager API

ConnMgrMapURL

ConnMgrEstablishConnectionSync

ConnMgrProviderMessage

ConnMgrReleaseConnection

CONNMGR_CONNECTIONINFO

PROXY_CONFIG

1) Call ConnMgrMapURL function to map the InternetUrl to a network Guid and an index

a. This function takes the URL you specify and determines the best network connection to use to access that URL.

2) Fill CONNMGR_CONNECTIONINFO with the retrieved network Guid

a. dwParam: CONNMGR_PARAM_GUIDDESTNET

b. dwPriority: CONNMGR_PRIORITY_USERINTERACTIVE

3) Call ConnMgrEstablishConnectionSync with the above filled structure to get a connection HANDLE

a. This function uses ConnMgrEstablishConnection and ConnMgrConnectionStatus internally to attempt to fully establish a connection before returning to the caller. The ConnMgrEstablishConnectionSync function does not provide an internal message pump; therefore, the message queues of any user interface threads that call into this function are blocked until the function returns.

4) Call ConnMgrProviderMessage with the received connection handle and Guid provider as IID_ConnPrv_IProxyExtension, to get PROXY_CONFIG

a. Make sure to set PROXY_CONFIG.dwType: CONNMGR_FLAG_PROXY_HTTP before calling

5) Call ConnMgrReleaseConnection to close the connection HANDLE

Now, PROXY_CONFIG has all the required details of the HTTP/Internet Proxy for ready use.

Thanks,

Laxmi Narsimha Rao ORUGANTI

Annoying error - 28559 - Encountered problems when opening the database

SQLCEBLOG — Wed, 07 Feb 2007 08:14:00 GMT

We have been hearing lot of complaints on how to get more information for error 28559 when doing RDA or Merge Replication.

If you are using managed API (SqlCeReplication OR SqlCeRemoteDataAccess), then it should be straight forward for you as you would be receiving SqlCeErrorCollection in SqlCeException.Errors which would contain list of errors. This list would tell you more on why there was a problem opening the database.

However, if you are native programmer and using ISSCEMerge, ISSCERDA interfaces, then you need to do some extra steps in getting that information. We have written a sample to help you get the error information much like how our managed wrappers get it. Please have a quick look at our sample app attached to this blog post for more details.

Thanks,

Pawan Kumar

RDA Subscriptions

SQLCEBLOG — Tue, 23 Jan 2007 20:06:00 GMT

Q1. How do I know whether the RDA Pulled table is tracked or not? OR What an RDA Subscribed table means?

A1. RDA Subscriptions are maintained in a system table __sysRDASubscriptions. If the table is pulled with tracking ON (that is RdaTrackOption.TrackingOn or RdaTrackOption.TrackingOnWithIndexes), then it means it is RDA Subscribed. Hence, an entry will be made in this system table. To conclude, if the pulled table is listed in __sysRDASubscriptions, then it is tracked. Also, if the table is not pulled with tracking ON then there will NOT be any RDA subscription for it.

Q2. Why do I get error “SSCE_M_RESTRICTEDDDL – 28605 - Internal error: DDL operations are not allowed on system tables. [,,,Table name,,]” and What does it mean?

A2. When you RDA pull a table with Tracking On (or when a table is RDA Subscribed), the DDL operations on that table are restricted so that we maintain the schema integrity with the master table to which we are a replica. Hence, when ever you try to perform DDL operations on RDA Subscribed table, you would hit this error.

Q3. How to get around the error “SSCE_M_TABLEALREADYEXISTS – 28573 - The LocalTableName parameter is already specified. [,,,Table name,,]”

A3. This error is thrown when there exists a table with the name supplied in either of the parameters local table name or error table name for RDA Pull command. To get around this error, there are two ways: 1) Drop that table 2) Change the RDA Pull arguments

Q4. How to get around the error "Duplicate value cannot be inserted into a unique index. [Table name = __sysRDASubscriptions, Constraint name = c_LocalTableName]"

A4. Ideally, this error should not occur. But due to a bug this has started showing up. It is discussed well in detail in KB Article 920272. If you are using Whidbey SP1+ (i.e. SQL CE v3.1), then you should not encounter this anymore. If you do encounter this, go with a work around of putting all the binaries that access database (including product binaries, customer application) in a single directory and preferably \Windows.

Q5. How to change IDENTITY information after doing RDA Pull as RDA does not support Identity Range Management.

A5. Though the RDA Pulled tables are DDL restricted, changing the default values and identity information is not restricted. You can use ALTER TABLE <Table Name> ALTER COLUMN <Column Name> <int | bigint> IDENTITY(<New Seed>, <New Step>).

Thanks,

Laxmi Narsimha Rao ORUGANTI