Steve Lasker's Web Log - www.SteveLasker.com/Blog : Microsoft Synchronization Platform

Evolution or Revolution for moving to offline architectures

Steve.Lasker — Sat, 18 Oct 2008 03:15:00 GMT

There's been a lot of talk, momentum and questioning for where and how occasionally connected designs fit into the application landscape. There have been a lot of questions for "if and how" developers can take their existing apps "offline". There have been a number of efforts for enabling offline apps and some longer term plans. In VS 2008 (Orcas) we introduced Sync Services for ADO.NET as a developer focused way to synchronize data. The roadmap continues with the Microsoft Sync Framework releases. We've started to show "Astoria Offline" as a means to enable sync over REST based HTTP protocols enabling more generic services. But do any of these "solve the problem". It's been a while since I've done a brain dump on some thoughts related to the evolution vs. revolution of app models, so what better way to dump thoughts than a blogicle.

Can you take your current app and make it offline?
The reality is "it depends". There are so many app designs, it's hard to say definitely what works and what doesn't. There are some well established things that have worked and haven't, but it's not fair to say these apply 100% of the time. One size doesn't fit all.

Intercept/fall back
Do you go with the intercept model? When online and connected, the app communicates directly to the service. (Note I'm using "service" here to represent any remote source. It could be a database "service" a Web Service, a WCF service, or some other API that's only available when your device is connected to "the network"). When the network falls away, the app "falls back" to a local "cache".

This is generally a bad (opinion), or complicated (fact) model. This assumes one of two things. Either your user knows they're going offline and they synchronize before they yank the cable or they predict when the network fails. Of course the application can constantly synchronizing data in the background. Crystal ball? When do you really know or have time to sync beforehand? You're working in your office, and Outlook pops up reminder. Or worse, you get an IM or Text message asking "where are you". You grab your laptop and run... oh, wait, you forgot to sync. Do you have the info needed for the meeting?

For the background sync, one of two things could be happening. When you "fall offline", your view of data suddenly changes because your local copy is so out of date compared to the data you were just looking at, or your synching often enough that there's no difference, asking why you're even trying to work online in the first place. Or, you've effectively created a denial of service attack by asking, has anything changed, has anything changed, has anything changed?

Outlook 2003 (I think that was the version) tried this fallback model. It was awful. You're sitting in your office, writing an email, looking at your inbox. The network goes down, your inbox view completely changes, or worse Outlook froze, and you lost that message you were carefully wording.

Always work locally
In this model, you synchronize everything you need, and make all your interactions against the local store. This is sort of the Outlook model today (Outlook 2007). There are equal problems here. Is it realistic to have all the data locally? Even Outlook data can be significantly large, and that's clearly your data. What about public folders? By default, these are only available online. If you're online, they work. If you're not, they don't. Fairly straight forward; but what about historical data for all the products the company has shipped? What about your list of customers, or the current list of stocks and their prices?

The reality is data volume is growing at an exponential rate. Storage continues to get cheaper and faster. 1TB disks and Solid State storage are becoming mainstream. The problem is while connectivity is getting better; the bandwidth isn't increasing at the same rate. And, even as it does, is it realistic to have everything local? Do you keep one of everything that Home Depot or Safeway keeps in stock? Would be really nice, but not all that practical.

One size fits nobody
The reality is there's no easy answer. This is really an architectural and user model change is required to make this happen. We've been through these before. In the "old times", families ate what they grew. Corn, milk, meat, etc. Others only had Ice or Milk that was delivered to their house daily. You could think of this as the "online model". Today, we go shopping to Safeway, D'Agostino, or your local grocery store. The stores don't grow anything locally. They stock goods they get from somewhere else. You take stuff home and stock it in your pantry, frig, freezer. When you throw out the garbage it goes into the can. The can gets picked up on Thursday. It gets taken to the transfer station, and then off to the dump. The point is we have lots of "store and forward"/sync models already in place. What's really interesting is the "online" things are actually more expensive. It turns out it's cheaper to stock goods that may be used, and accept a percentage of loss for things that don't sell than it is to create on demand. Why do we think electronic bits and bytes should be "online" only? Why are 1's and 0's more expensive than physical goods?

Evolutionary approach: Mapping concepts
If we look at the concepts we have today in web/service/stateless models, they break down into two common, and one advanced models:

Get
Get maps very easily to HTTP:Get. Essentially you make a request to a domain, application and method. You likely pass parameters and of course all the authentication goo. The result of the get is returned and you now have a copy of information relevant to that point in time, and the specific question you've asked. Most likely, but no guarantee it's accurate by the time you quickly display it on your 64bit, quad laptop with a gigabit connection to the internet. Even historical data is subject to "correctness"- remember the Florida elections J But, we live with "copies" all the time. Copies are tangible, things that are "just there". There's a separate question of how "out of date" the copy is.

Submit
You have some information you're sending back. You expect some action to be taken on the data you're submitting. That submit can be accepted, rejected or modified. You get an immediate response, but no guarantee what you've submitted will be accepted. That might happen later on, it might not. You hope, or expect to be notified if something changes. But is that concept built into our app models? Or, do we just assume the world is globally connected, and everything is always running so things can be asked and answered immediately, with 100% accuracy, in one big transaction so no two people can allocate the same "goods".... Whew, that's as ridiculous to assume as it is to type.

Can Get and Submit evolve?
With Get, you can implement a "caching" model. Rather than Get directly from the source, imagine you Get through an abstraction layer. For each Get the abstraction layer keys off the domain/app/method and name/value pairs for parameters. Let's say I make a service call to: http://AcmeServices.com/Books/GetBookByName.asmx. I pass in: Title=SQL Server Compact

In the GetCache, it looks for the service and name/value pairs that match. If it finds a result, it may not have to actually make the network call. It could just return the cached result. But what about stale data? You can put an expiration policy on the cache. For a list of states, a day is likely more than generous. For pricing, maybe an hour. The point is you can customize this per source. For many things, one could argue most queries, the data is fairly static and the app could benefit immensely by caching the "gets". If, however, you were to look online first, than only if the resource wasn't available look in the local cache than what benefit are you getting? Sure, the app would "work offline", but you don't see any benefit in the online mode. Each request still takes the time to roundtrip. If you implement a smart caching model you can provide dramatically faster user responses. ...and in most cases, the data is perfectly accurate. Now it would sure be nice to know when something changes, so if California does sink, Iraq or Canada become the 51^st state, you don't have to watch CNN and ask all your clients to either wait for the cache to update, or "invalidate" the cache. I talked about notifications a bit here, so I'll keep going.

Submit
This one gets interesting as well. You have an order to submit for some widgets. Sure, you can submit directly to a "Web Service", or web page. But in many situations the order submission doesn't actually mean the order will ship. Accounting needs to verify your credit. The products that were supposed to be in stock may have been routed to a bigger customer, or "fell off the truck". Again, the point is just because you may get an immediate acknowledgement the order was accepted, doesn't mean it will be fulfilled. So, why not accept this as the model, and when it works, it just works. When a problem occurs, it's not really a problem because it's built into the system. What if you "queued it up" for sending? In this case we save the message locally, in a "logical queue", and when the service is available, we send it on its way. I used the term "logical queuing" as I'm not implying you must use something like SQL Server Broker, MSMQ or WCF Queues, etc. These tend to be a bit over complicated for what you need on the client, and it makes it difficult to act, query or evaluate the data in its transient state. A while back I wrote a blogicle on and some samples for Logical Queuing.

Sync
This is what some might assume is the answer to all of these problems. If I can synchronize changes from other nodes (the server and other clients) with my local store, and I can make changes locally and have them synchronize back "up" to the other nodes, doesn't that solve everything? In the land of demos, sure. But in the real world there are just too many complications. The volume of data, requirements for business logic, and workflow make it difficult to assume Sync can solve all the problems. Where sync does do really well is for synchronizing reference data. That list of states, codes, or even larger volumes like the product catalog. The Logical Queuing blogicle and sample demonstrate some thoughts here. I recently saw some work the Microsoft Mobile Line of Business CRM team, and it also uses a similar model.

Revolution is on the horizon
With all the complications and half way solutions, you have to wonder what's the real answer to all this mess. Will this problem just go away because we'll have high speed internet available everywhere? Even if you could be guaranteed internet access everywhere, do we think that every service will be available all the time? As the world gets more connected, will systems be more reliable and capable of enlisting in massive distributed transactions? Or, will computing systems evolve to what every other large scale system as evolved to. Queuing, Caching, Reserving, Buffering, Acknowledgement, Conflict Detection & Resolution and Notifications.

While I talked a bit about the food supply earlier, this is also a bit like the VB Classic to Web transition. For those old enough to remember the great 90's there was a big transition from "fat clients" to "thin clients". No subtle suggestion of which is better there J When developers first started building ASP Classic (pre .net) apps they simply moved their session state to the ASP Session object. Which meant web servers could serve about 5-10 people before falling over. Not exactly the promise we had all hoped for. It took a while to realize the app models had to change. But, moving from Client apps to Web apps was a bit more obvious. Mostly because the UI changed (for the worse), but at least users could get their work done because we couldn't get the VB apps deployed. (yes, I'm over simplifying things, but run with me for a minute).

Since developers had to re-write their UI anyway, it was an obvious change. Making the transition from online to offline mode apps isn't as obvious. The only UI required is the UI really telling them when problems occur. Who wants that?

The reality is things are a changing. Nothing to announce, nothing to promise, but there is a revolution on the way. Revolutions take time for people to live through denial that it's coming, but it comes as a wave of reality that just smacks you with "duhh, I guess that was a stupid assumption".

While we all wait for the revolution, you can look at things like ADO.NET Sync Services, the Sync Framework, Astoria Offline, and lots of goodies the Patterns and Practices team has done.

Steve

First look at the Visual Studio Orcas Sync Designer

Steve.Lasker — Thu, 22 Mar 2007 03:32:00 GMT

Here's part 1 of the new Sync Designer. In this screen cast I walk through how to cache lookup tables locally. In this screencast I use the 2 tier sync, starting simple. In the next screencast I'll show how to take the same designer, and split it across N tier to keep the intimate knowledge of the server, connection string, queries, passwords, etc. from the client.

http://channel9.msdn.com/posts/SteveLasker/Visual-Studio-Orcas-Sync-Designer-for-Caching-Data-in-SQL-Server-Compact-Edition/

Just as a note, this screencast shows an internal build of Visual Studio Orcas between the Feb CTP and Beta 1.

Steve

Additional Q&A on the Visual Studio Orcas Sync Designer

Steve.Lasker — Wed, 21 Mar 2007 20:54:00 GMT

Q: Why does the Orcas Feb CTP Typed DataSet designer not work on Vista?

A: Visual Studio Orcas is not yet fully compatible with Vista. In the Feb CTP there are several issues with something we call "red bits". These are changes to the runtime that shipped in .NET 3.0 which shipped with Vista. For our own internal testing, we do have fixes allowing us to test on Vista and all the scenarios are working. However, due to the way the dlls are signed and shipped in Vista, and the time crunch we're working under, we weren't able to get the work done in such a way that we could service a Vista box that had the Feb CTP installed. This was a very tough decision as we do not want to set the expectation that Visual Studio Orcas won't be Vista compatible. But, it was felt more important to make sure that anything we ship, including the CTP must be serviceable. As we all know, we're working on to maintain our schedule, and since CTP's are not meant to reflect the final quality, it was determined to ship the Feb CTP without full Vista support. As those that have seen my demos, we do have Vista working, and we will have full support for Vista. I believe, but not sure, that B1 will be fully Vista compatible. So, for now, if you're using Vista, the Feb CTP should be run in a VPC. In general, it's not a bad practice anyway.

Q: Will the Sync Designer generate time based sync?

A: This feature was unfortunately cut. We had some great designs, and even some prototypes that I've demo'd previously. However, we just didn't have the time to do all the background task, network resource detection, and timer work. Rather than hack something together we didn't have time to fully design, dev and test, we've had to cut this feature altogether. Of course this doesn't mean developers aren't smart enough to do this work with a combination of the BackgroundWorker, Timer and Network events. It just means we don't have a checkbox to do all the work automatically. We will have samples on how to do this, and we're in discussion with Patterns & Practices group on filling these gaps ‘till we have fully integrated runtime components that make this "just work".

Q: Will tombstone records be automatically cleaned up?

A: This was another cut we made in the sync designer with Visual Studio Orcas. Tombstone records are how we track deletes. If you sync on Monday and get the full product catalog, and then sync again on Wednesday, how can the server answer the question of "what was deleted". There are many ways to handle this. You can mark the main record with a status of Active = False, or you can physically delete the row and in the Deletion Trigger copy the PK to a [Table]_Tombstone record. The later is what the Visual Studio Orcas Sync Designer does. We had planned to create a Scheduled Task to automatically purge these deleted records, but since SQL Server Express doesn't have the Agent, we felt this was going to be too confusing. We also expect developers will do some additional work between their development and production boxes, so for now we punted this. We're not yet talking in detail about our SQL Server Katmai release, but you can expect we're doing some cool things here. So, for now, you'll have to create a scheduled task to purge the tombstone records that are say older than 30 days. Stay tuned, and we'll have more exciting things to discuss about the upcoming Katmai rlease.

Q: How do I get my cached tables to be synchronized in a single transaction?

A: The Orcas Sync Designer is focused on download scenarios. You can absolutely extend the sync designer generated code to do uploads as well, and in fact we even generate the upload SyncAdapter commands. For download only scenarios it was felt we should keep things simple by default and enable the ability to easily recover from dropped connections. So, by default, the Sync Designer synchronizes each table individually. As one table is finished, the next table is synchronized. This means if table 1 finishes, and the connection drops during table 2, that the next time a sync occurs, Table1 is likely already complete. To put all the tables in a single transaction, expand the [advanced] expando button and choose the sync all tables in a single transaction option.

For the Feb CTP, the Sync Designer generated a SyncGroup for each table. This wasn't necessary as the SyncAgent knows to place each in it's own SyncGroup when one isn't specified. This has since been changed so you'll only get a SyncGroup when placing all in a single transaction.

Q: Once all the tables are placed in a single transaction, how do I control the order the tables are updated to handle parent/child relationships?

A: The Sync Runtime will apply inserts, updates and deletes based on the order of the SyncTables and SyncAdapters in their respective collections. This isn't fully implemented in the Orcas Feb CTP, or the Jan Sync CTP. However, we do now have that updated internally and will be releasing it soon in the CTP. In an upcoming screen cast you'll see the current updates to the Sync Designer, including the ability to move items in the collection up and down.

Q: Does the sync runtime create relationships locally within SQLce?

A: No, I touched on this here: http://blogs.msdn.com/stevelasker/default.aspx#constraints

Q: Does the sync runtime work with server side identities for PK's?

A: No, no, and maybe. In general, despite the fact that many of our own samples depend on the Identity feature to manage the primary key, identities are an overall poor design. Building offline apps is only the extreme of the problem. How do you create a new row on the client, assign it a primary key when the client won't actually know the PK until it posts it to the server? However, the problem exists in "connected" stacks as well. Say you're creating a Order with a collection of OrderDetails. The standard model is to have the OrderDetails table reference the PK of it's parent, than add another sub key, say line number. You can't actually create the OrderItem records until the Order record is posted first. This creates complications for connected development. Sure, you can use the @@scope_identity value, but you still have to do a bunch of updates. Sure, you can wrap this in a sproc, but it's still a PITA. Now, consider you're building a high availability system. You have multiple server nodes managing the creation of data. Who owns the master identity range? We see this in rollup scenarios all the time. Each branch maintains their own data. Each uses identities. When the data is rolled up to corporate, you have collisions, requiring you to add the branch id to the corporate database. Now, you could partition the identity values, which is what we do in Merge Replication, but now we've introduced additional problems. Prior to joining Microsoft we deployed many mobile applications. If a person is in the field, using Merge Replication, and is working with their partition of IDs, what happens when they use them up? They have to re-synchronize with the server to get a new allocation, which they may not be able to do at the time they've run out. So, the app is dead. Another client doesn't use up their allocation and leaves "holes" in the server.

So, what's the solution? There are two ways, the simple but not always the most efficient, and the complex. The simple is to simply use GUIDs. SQL Server and SQLce both have UniqueIdentifier data types. You can either set the IsRowGuid property, or create your own GUID with Guid.NewGuid(). The other is to create your own hashing. Each client is given a hash when the initially connect to the server. They then create incremental counters, possibly even with Identity within their hash. Now when you push up the new values, voila, no conflicts. If you still want incremental counters, say for that single, simple OrderId, you could still use the Identity feature, just don't make it the primary key. You upload a new order from the client. The client assigned a primary key using it's hash. When that order is uploaded to the server, the server applies the identity value to the non-pk column. The update is then brought back down to the client. Life is good.

Steve

Q&A on OCS & Sync Services for ADO.NET

Steve.Lasker — Sun, 18 Mar 2007 17:25:00 GMT

Not surprisingly we've been get a lot of great questions about specific features and scenarios for our new Sync Services for ADO.NET (OCS). Rafik has been fielding most of these on the Sync Services forums. Since the Q&A for SQLce seemed popular, I thought I'd do the same here.

Q: How does Sync Services compare to Merge Replication and RDA?

A: Merge replication is our developer oriented, feature rich, SQL Server based database "replication" product. It's designed for a DBA to configure a publication on the database exposing a set of tables (articles) enabling filtering and rules. Clients then "subscribe" to the publication and receive a local database reflecting the publication. It's a very powerful and mature product that we will continue to invest in. From the server side (publications), Merge Replication is a SQL Server only feature that is available in our SQL Server Workgroup, Standard and Enterprise SKUs. Merge Replication, as a publisher, is not available in the free SQL Server Express SKU. Merge replication supports 2tier sync and sync over HTTP(s) to enable internet synchronization. However, I wouldn't say that Merge is SOA oriented in that you don't have much control of what's sent or received across the wire, nor do you have the ability to support additional transports. It's a fantastic end to end replication product that handles a lot of complicated scenarios.

Remote Data Access (RDA) is a developer oriented, simple but RAD sync technology. It works over HTTP, so it enables internet protocol sync, but again, it's not very SOA friendly. While the client doesn't directly open a connection to SQL Server, it does require the client provide the connection string and query to be executed on the server. So, it pretty much violates the notion of keeping the server information from the client. While it is limited, you can't deny it's popularity for it's simplicity. We continually hear many people implementing RDA rather than merge as theire requirements are simple and RDA just worked great. Which brings me to Sync Services for ADO.NET

Sync Services for ADO.NET is our answer to this dilemma of having to choose between the dba focused powerful features of Merge, the simplicity of RDA, and the developer motivation for them to write their own. When designing Sync Services we used RDA as our user model. Debra Dove and her team did an excellent job with RDA, and we wanted to take it to the next level. We used the provider and developer programming model of ADO.NET and wanted to leverage all the great transport, security and protocol work others were doing. Essentially, rather than build additional solutions to existing problems, we wanted to scope the solution to what we needed to solve, and leverage others who were experts in their field. The Sync Services for ADO.NET features are built by the same team that delivers Merge Replication. I've been truly amazed at the knowledge these guys have. It's because of their experience that I have a hard time calling Sync Services a 1.0 product. It's really a culmination of all the great work from Merge, RDA, file synchronization, and even the WinFS sync work.

The quick bullet for Sync Services is it's a componentized synchronization framework, built on ADO.NET, but factored to provide a common sync platform for Entities, Files, and other formats. Sync Services for ADO.NET is our first delivery in our Microsoft Synchronization Platform.

The key advantages of Sync Services is its developer focus, w/DBA empowerment. Rather than force your DBA to understand all the details of how you're going to synchronize data within your application, you engage the DBA for the portions that touch the server database. The transports are up to you, and we have a great WCF designer integrated story, but you can use other transports as well. The local database schema/structure is up to the developer to decide as it's their apps database. Of course you can engage your DBA as well, but you don't get what you get as the result of a synch operation. You can use just the client components and use Java and Oracle on the server, or you can use just the server components and other technologies on the client. If your data is locked up in Oracle, you can still use Sync Services to synchronize that data directly from Oracle to your client. You don't have to put SQL Server in the middle just to enable sync.

The point here is regardless of what cards you've been dealt, we want to help you get your app done to make your users happy. The more Microsoft products you use, the better the experience we can provide, but we're not locking you in, or out of our platform just because something is out of your control.

Q: What's the roadmap for Merge, RDA and Sync Services

Merge replication will continue to be our database replication product and will have Katmai investments in the next release. Merge will be the DBA tool for replicating a database, and those that are already using Merge shouldn't feel like we're abandoning them by any means. If Merge is working for you, don't worry, we're continuing to take feature work and making improvements. In fact, many of the Sync Services features will work into Merge as well. That's about all I can say for now. We'll announce more about Katmai later on.

RDA will eventually be phased out. We truly believe the Sync Services features address the simplicity of RDA, but don't have any of the limits imposed by RDA. Specifically, incremental changes, ability to synchronize several tables in one transaction handling all the interleaving of inserts, updates and deletes. Support for other ADO.NET Providers, etc. If you're already using RDA, we're not killing it yet, but we won't be doing any new work there either. We do expect to deprecate it within the next release or so as Sync Services are released. If you haven't yet deployed RDA, but are looking into it, definitely look at the Sync Services CTP. If you need to go into production now and can't wait ‘till Sync Services are released, than by all means use RDA. It's not like it has that much functionality that you won't be able to easily switch to Sync Services later on. <g>

Sync Services is where we're doing a lot of our investments. It's just the first delivery in our new Microsoft Synchronization Platform. I'll write a different blog post on our naming. (I love discussing naming, ...not). Sync Services are our developer oriented, SOA enabled data synchronization features. We won't have all the database replication features of Merge, as we really focus on synchronizing data. Sync Services provides and end to end story, but is a componentized model allowing you to get in the middle, or completely replace one end.

Q: Can I use Merge and Sync Services together?

A: As we've all seen with our SQLce and Express discussions, one size doesn't fit all. But we also don't believe you need 10 ways to solve the same problem. Just as with Express being our entry point to our Data Service platform and SQLce being our client/embedded platform, we expect Merge and Sync Services to address the needs of two types of scenarios. It's likely that some scenarios, such as a branch office may actually use both Merge and Sync Services. Merge to replicate data between the branch office and the corporate office, and Sync Services to enable branch workers to go out into the field.

Q: Does Sync Services support N Tier?

A: Absolutely. The beauty about the N, is it could imply 1 to many. With Sync Services we have Server and Client Providers. Actually, we like to think of them as local and remote providers as we currently focus on hub/spoke, but we'll be enabling p2p as well in the future. You can start with 2 tier and move to N tier. You can intermix 2 tier and N tier based on where the clients are connecting from. When you first start working with Sync Services you may ask why you have to specify the tables your synchronizing on the client and server providers. That's because we want to make sure you can easily split the client and server code to different tiers. All the "intimate" knowledge of the server can easily be moved to the mid tier, while the client provider configuration is maintained on the client. Even the Sync Designer allows you to easily split the client and server provider code. It's quite sweet when you see it. Screen cast coming soon...

Q: What transports do you support?

A: What do you want? As noted above, the sync team wants to get out of the transport business. We have many bright people in Microsoft thinking about those problems. In Orcas the Visual Studio designers will focus on enabling WCF, but you can use Web Services, SSE, SyncML or if you can figure out how to convert Jelly Beans to .NET objects, you can use those as well. Essentially, you simply plug in a matching service and a proxy, and you're good to go.

Q: Do you support column level tracking?

A: Not yet. For this first release, we only support row level tracking. We are working on column level tracking, but it does require additional functionality on the server, and we wanted to make it easy for developers to start synchronizing data from their existing databases. As with any good database design, you should think about how you partition your tables. For various reasons, it's good to separate images, or other large blobs from your primary table into other tables and maintain a 1:1 relationship. It's not meant to be an excuse, and we will be implementing column level tracking in the future. Using features like ADO.NET V3 entities you can roll up the 1:1 mappings into a single object allowing your developers to work in a normal fashion, while managing performance and isolation in your database.

Q: Do you support custom conflict resolvers?

A: Of course. How could we not? On both the server and client providers we expose a conflict event. In that event you'll get the client and server rows that represent the conflict. You can simply say force overwrite, or implement very custom logic. Such as determining if the person who made the change is a manager, or owner of that particular customer account. Based on that custom logic, you simply make the logical change. All this is in managed code, with your favorite .NET language.

Q: Do you support partitioning/filtering?

A: Of course. We don't really expect people to synchronize terabytes of their data to all their clients. The partitioning is the normal horizontal and vertical partitioning. You simply provide the query that represents the filter you wish to support. You can do joins, etc. It's just a query. The client sends up as many parameter values as you need. There's no limitation on the number of parameters. In fact, you can even intercept calls on the server and set sync parameter values based on other logic. In WCF you can determine who the client is, and based on that info, send them their customers without exposing the SalesPersonId to the client to substitute another value.

Q: Does the server track each client individually?

A: No. This is one of the major differences when compared to Merge. One of the powerful features of Merge is it knows all its clients, so when the client connects, it already has the data ready for it, and easily supports "data repartitioning". In Sync Services, the server has no idea who all the clients are. The benefit here is Sync Services don't have the same scalability constraints. You can synchronize as many clients as your server can handle queries. The server doesn't necessarily know it's a publisher, but rather it's just answering queries.

Q: Does Sync Services support multiple publications

A: Yes/no. Sync Services doesn't utilize the pub/sub model per se. You can configure the server provider to offer 20 tables you want to synchronize. The client simply say it cares about 3. Another client cares about a different 3. Another client cares about 4, which overlap the first two clients. In fact, we also support a client dynamically adding tables. A sales person may cover another sales person for a week and needs to bring in an additional product line. Within the app, the developer can change the filtering query, and off they go.

Q: How are schema changes handled?

A: Unlike merge which is geared around replicating a database, Sync Services is geared around synchronizing data. I'm not a big believer that generally speaking the DBA simply adds a column to the server and the UI automatically updates on the client and life is good. While it can be done, most of the time I'd bet you want some control over where and how the new element is displayed, add some interaction logic to the client, tab order etc. We really treat schema updates as an app update. It's a holistic update of the app overall. The model we've gone with the Sync Services are the following:

A new requirement is defined, say AddressLine3. The DBA would add the column to the server. All the normal rules apply. If the column is non-nullable, than a default should be provided.
The developer involved with the sync layer would most likely create a new version of the Sync Service, say v2. This means that apps that were using v1 can be slowly migrated, or at least be migrated within some level of control. If the user is in the middle of an important deal, the last thing they need is a forced software upgrade. Ever been in the middle of something important and IT forces an update that reboots your computer or app? Software is an enabler, it should help me achieve my goals, not fight me because IT thinks its important now.
The app developer updates their service proxy to point to v2 of the sync service, exposing the extra column.
In the version check code, the app author can either choose to reset the table, or they can execute the alter table script locally adding the additional column. They may even bring down a single data call to retrieve the values for the new column on all the existing rows.
The developer than decides what they want to do with the new element, updating their ui, logic etc.

So, while we didn't implement something as simple as point click, we think it tends to fit the SOA model where apps may consume services from other apps, and they should have control over how and when they consume new schema.

Q: How are constraints, keys, and other db objects brought down to the client?

A: This again falls in the category of Sync Services is about synchronizing data, not replicating a database. Sync Services does do some schema and even database creation with SQL Server Compact Edition. If you're starting from scratch, and you first synchronize, the SQLce database will be created based on the connection string properties, name, encryption, password, etc. It will then create all the tables the client has said they're interested in. Remember, just because the server exposes 20 tables, doesn't mean the client must use all of them. The client determines which tables it wants to consume with the SyncTable collection. When the tables are created, primary keys are created, datatypes are mapped to the clients datatypes, and nullability is applied. No additional indexes, constraints, defaults, etc. are applied. There are SchemaCreating/ed events fired where you can either initial create the schema to be used, or alter the schema after the tables are created.

Q: Does sync services handle parent/child/grandchild relationships?

A: Yes. Unlike RDA where you can only sync one table at a time, Sync Services handles the hierarchical nesting of inserts, updates and deletes. In fact you can even control it seperatly on the server from the client. On the server, tables are placed in the SyncAdapter collection. The order of the SyncAdapters defines the order by which updates will be applied. Inserts and Updates are done from the top down, while deletes are done from the bottoms up. The same is done on the client, in the SyncTables collection. This allows the server to control its order, while allowing the client to control its order of updates.

Q: Can I update only a few tables at a time?

A: Yes. Say you want to only synchronize your lookup tables, states, codes, etc. once a day. You can create a specific service just for your lookups, and another for your product catalog.

Q: Can I update everything in a single operation, or can I control things more granularly?

A: Within the SyncAgent, you can utilize the SyncGroup to determine the grouping of updates. In the previous example, you may choose to put all the lookup tables in their own individual groups. If the connection drops while you're synchronizing your lookups, it can pickup where it left off next time it synchs. However, when synchronizing Orders, you probably don't want Orders to ever go up/down without OrderDetails. Simply put the Orders and OrderDetails table in the same SyncGroup, and you're all set.

Q: Does Sync Services support batching for large data sets?

A: Yes, but not quite yet. We initially scoped this out of the first release, but we believe we'll be able to get it in, so look for it sometime around March 07.

Q: How does Sync Services track changes?

A: Sync Services uses an Anchor based model. Each time a sync operation occurs it gets a reference mark from the server. It could be the servers DateTime, or a TimeStamp (RowVersion). The client saves that value for the next sync operation. Each time the client synchronizes a particular SyncGroup, it first requests the server anchor. It than executes the queries on the server using the last anchor as the low range, and the new anchor as the high range. This gets a consistent set of changes across several queries. In future releases of the Microsoft Synchronization Platform we'll be supporting a knowledge based sync model as well as the anchor based model discussed here. Rafik does a great job explaining it in his blog

Q: How are deletes purged?

A: On the server, deletes are either kept in a tombstone table, or simply tracked by some sort of active/status flag in the primary table. Since this version of Sync Services isn't tightly coupled to SQL Server, we actually don't do anything. In general, we'd expect the DBA to write a scheduled task to purge tombstone records on their determined interval. You can expect us to do more in the "future", tease, tease, tease...

On the client, SQLce purges deleted records once it confirms data has been sent to the server.

Q: Can I purge old data on the client without triggering a delete on the server?

A: Yes. While we don't have a simple API to do this today, you can delete a bunch of rows on the client based on what ever criteria you decide, than simply "AcceptChanges" on the client prior to these changes being sent to the server. Of course you could intercept these on the server an toss deletes as well to protect your server data.

Q: Does Sync Services support low bandwidth type sync scenarios?

A: By low bandwidth, I mean can I synchronize only the important things now, and catch up later. Yes. You can either upload only, download only, or synchronize just a particular SyncGroup based on your own logic at the time.

Q: When will Sync Services ship?

A: Sync Services for ADO.NET will ship at the same time Visual Studio Orcas ships. This is currently scheduled for Q4 2007. Note: This is not meant to be the official place to get the timeframe for Orcas, but rather just saying that our current plan is to ship Sync Services within SQL Server Compact Edition 3.5, which will ship with Orcas.

Q: Will Sync Services ship on both the desktop framework and the .NET Compact Framework?

A: Yes, but at different times. At current, March 16th '07, we are scheduled to ship for the full framework, but we are not planning on shipping Sync Services for the device platform in the Orcas product. We do plan to ship the client components for Sync Services soon after Orcas, but are still working out the schedule. The problem is the various .NET Compact Framework teams, including the Visual Studio for Devices teams and Sync teams have a lot of work to manage with many different device platforms and a very short schedule, and we haven't been able to get all the appropriate ship level test coverage complete. We have designed, and done preliminary testing with the client components working and synching over Web Services. We are still hopeful we can pull it in, but at this point, we're just not ready to commit to Orcas, but rather shortly thereafter.

Q: Will Sync Services or SQL Server Compact Edition be in the .NET Framework, or .NET Compact Framework?

A: No, we are not shipping within either framework, but rather shipping as an add-on component. Why? Because we wanted more flexibility with our ship schedule. SQLce will ship 2-3 times between .NET 2.0 and .NET 3.5. While it would be nice to ride the distribution of the frameworks, with the embedded/private deployment options of SQLce and Sync Services, we felt it was better to have more flexibility with our schedule.

Q: Are these the only questions?

A: I doubt it. So, keep them coming, and I'll update this FAQ as I receive them.

Thanks for all the great questions. Keep them coming as they help us make sure we're shipping the right features in the right order.

Steve