Steve Lasker's Web Log - www.SteveLasker.com/Blog : OCS

Demos, Presentations, Links, Screencasts and Videos for SQL Server Compact

Steve.Lasker — Wed, 26 Nov 2008 00:57:00 GMT

I often find it hard to find my own demos scattered throughout my blog, so figured I'd just create a single posting that had all my current demos and powerpoints.

Recent Presentations at PDC and Tech Ed

SQL Server Compact Overview (lots of demos with supporting slides)
This was a very demo heavy deck with supporting slides. Covers many of our SQL Server Compact goals and non-goals, including the tooling scenarios, split between VS and Management Studio. Slides and demo on ASP.NET soft block, programming models with DataSet and SqlCeResultSet. (This deck was before we had LINQ to Entity support available). Covers a quick overview of optimized online, offline enabled architectures, with demos of Sync Services for ADO.NET, including 2 tier and N Tier. Then wraps up with Logical Queuing as the overall architecture for client based apps. I had a lot of fun with this presentation, but it was hard to fit in an hour.

Unlocking the Power of SQL Server Compact (Tech Ed EMEA 08)
Covered several topics, such as the questions on how fast can SQL Server Compact insert data. For Tech Ed EMEA, I added the section discussing stored procedures, and how a common DAL layer can actually provide more functionality than sprocs for both local data and server data. (yes, I'm sure there will be lots more to debate on this topic, but here's our thought process J)

With some coverage on private deployment, including 64bit, deep dive on perf and the managed sprocs demos, I bubbled back up to cover several interesting aspects for using SQL Server Compact, including a TraceListener to capture information from the field and sync it back with Sync Services for ADO.NET. Then covered read only databases on DVD's and using Compact as a custom Document format.

Associated Demos:

SQL Server Compact Bulk Insert Performance Test Harness (Compares Compact & Express)
Demonstrates the power of an embedded database and how much faster it can run than SQL Server for many operations. By implementing a few things, such as "connection pooling" and caching your SqlCeCommand objects, you can achieve similar functionality provided by SQL Server's connection pooling and Query Plan caching. However, because Compact is "in-proc" with your application, you can get the benefits of not having to work through additional networking, process and security layers required by SQL Server which is designed to work over the network.
Managed Stored Procedures (A Common DLL for data access to Compact and SQL Server)
SQL Server Compact Trace Listener
Demo for using Compact as a TraceListener to capture the output of System.Diagnostics.Trace.WriteLine and then synchronize that info back to the server providing visibility into what's happening out on the client. This deck includes animation, which looks a bit like Missile Command, demonstrating how this sample works.
Running Compact from Read Only Media (DVD, CD, Locked USB Key)
Using Compact as a Custom Doc Format (Recipe Manager)

Managing and Versioning Client Databases (Tech Ed EMEA 08)

Discusses the pros and cons of managing database deployment on the client. Why deploying databases as files, through MSI's, ClickOnce or other technologies can be easy for V1, but could lead to data loss in V2. Why a little investment up front can yield you productive results long term, and make for a better user experience.

Associated Demos:

Creating & Managing Local Database Versions through Scripts
When deploying applications with a local database, deploying the database with the app can seem easy at first, but once the user starts making changes to the local database, it gets messy quickly. If you're not careful, you can even lose the local data created/updated by the user. In this presentation, I created several animated slides that explain the complications and why script based deployment and versioning is the only practical way to manage local databases. This sample walks through the creation and versioning of local tables. To focus on the script based versioning aspects, I excluded any interaction with synchronized tables as that just gets even more complicated.
Versioning Local Databases with Scripts when using Sync Services for ADO.NET
In this demo we see how to version schemas when using Sync Services for ADO.NET. This includes two projects. One for the initial installation of V1, and another for the upgrade, or initial creation of V2. V1 includes creating a set of tables that aren't synchronized, such as state management for submitting orders to an existing service. Once sync completes, there are a set of additional scripts to create foreign keys between the synched tables, and between the synched tables and the local tables.
V2 enhances the synchronized tables with additional columns. The setup of this demo is a bit more complex as I've configured the server side sync components in a dll, then hosted the dll in a WCF Web Project. The WCF Web Project is published to an IIS instance, but uses different sub directories: for V1-http://localhost/VersionSchema/V1/LookupSyncService.svc, for V2-http://localhost/VersionSchema/V2/LookupSyncService.svc. Using this versioning scheme we can publish V2, allowing apps to slowly migrate over to V2, while the V1 apps continue to function and the business continues to function. The animation in this presentation helps explain this somewhat complex subject.

Just the demos, and nothing but the demos...

SQL Server Compact Trace Listener
Demo for using Compact as a TraceListener to capture the output of System.Diagnostics.Trace.WriteLine and then synchronize that info back to the server providing visibility into what's happening out on the client. This deck includes animation, which looks a bit like Missile Command, demonstrating how this sample works.
Using SQL Server Compact under ASP.NET
Demonstrates how to "unblock" SQL Server Compact for usage under ASP.NET. I discussed our reasoning in this post. A quick recap: The reason we blocked Compact for ASP.net scenarios was simply to not set expectations Compact is meant to be a replacement for SQL Server. It performs better than Jet. For the most part, we unblocked it for pre-building a SQL Server Compact database for sync scenarios where it's faster to download the database as a stream, then to sync the initial snapshot of data. It's also perfectly fine for small websites or other services. The main intention is strictly to make sure developers know, this is not meant to compete with SQL Server. There are coding differences between Compact and SQL Server, so we really didn't want developers to start with Compact, only to later find out it doesn't scale to thousands of concurrent website/service users for updating data. As long as developers understand the scope, it's perfectly fine to use for ASP.NET, WCF, or other service scenarios...
SqlCeResultSet Demos - Scrollable, Updateable, ISAM like cursors
Demonstrates how developers can take advantage of embedded a database directly into your app. The best thing about the SqlCeResultSet is it works directly over the data in your local database. No queries that bring the results into another copy in memory that you must mange potential conflicts with the local store, or potential conflicts between the local store and the remote server. With the SqlCeResultSet you can scroll directly over the local table, even setting the index to filter and order the rows you're viewing. The SqlCeResultSet is a scrollable, updateable cursor and can really improve the perf and overall working set of your application.

Direct Access (ISAM) with the SqlCeResultSet
Why do we need to make copies of the data, update it, then send it back? In a shared database environment, where you're likely connecting to the database across the network, and sharing with potentially thousands of others, because it's the only way we can scale. However, when the database is local, in-proc with the app, things can be much simpler. In this demo, you can easily update data, and as soon as you move off the row, it's committed to the database. To get the most of this demo, run the same app twice (CTRL+F5). Make changes in one of the instances, move off the row. In the other instance, move to the row that was just updated, (note: if you're already on the row, you'll need to move off then back). You'll see the change made by the other app and can make the change back. The one piece missing from this scenario is an implementation of eventing so that instance 2 would see the changes made by instance one. This is why you must move off, then back to the row to see the change. As you move across the rows, the DataGridView requests the data. The data is fetched directly from the local database/table.
Another ISAM/SqlCeResultSet demo
In this demo we see how the index can be set to control the order of the data. Using a system view, we can populate the ListBox with the list of potential indexes. As you change the index, the view is updated to reflect the data sorted by that index. What isn't covered in this demo is the ability to "filter" the values using the SqlCeCommand.SetRange() api which uses the index to directly filter the visible rows.

SQL Server Compact Bulk Insert Performance Test Harness (Compares Compact & Express)
Demonstrates the power of an embedded database and how much faster it can run than SQL Server for many operations. By implementing a few things, such as "connection pooling" and caching your SqlCeCommand objects, you can achieve similar functionality provided by SQL Server's connection pooling and Query Plan caching. However, because Compact is "in-proc" with your application, you can get the benefits of not having to work through additional networking, process and security layers required by SQL Server which is designed to work over the network.
Privately deploying both the 32 and 64bit versions of SQL Server Compact with your app
Do your users know whether they have 32bit or 64bit machines? If they are 64bit, are they running a 64bit version of the OS? Should they have to figure this out? Using private deployment, developers can include both the 32 and 64bit versions of Compact so the user never has to make a choice. Yes, it's a bit more k to include in your app, but if it "just works", is that a small price to pay?
Running Compact from Read Only Media (DVD, CD, Locked USB Key)
Ever get a CD or DVD in the mail. Pop it in the drive to see what's on it, and it prompts you to install some software - yikes! Let's see if you can use this as a Frisbee and get it in the trash across the room, J With this demo, you can deploy a large amount of content, but stored in a Compact database so your users can run the app, search the data, and with private deployment, you can include the Compact engine so your users can "view the data" without having to install anything.
The demo is simply an extension of private deployment, but modifies the connection string to tell Compact to open the database in ReadOnly mode, and specifies a temp directory:
connStr = String.Format("Data Source = {0}\Northwind.sdf;Mode = Read Only;Temp Path={1} ", Windows.Forms.Application.StartupPath, My.Computer.FileSystem.SpecialDirectories.Temp)
Using Compact as a Custom Doc Format (Recipe Manager)
Need to save data from your app in a "document"? Want to post that "document" to a SharePoint site or email it to another team member? Want to double click your document and launch the associated app? Since SQL Server Compact uses a single file, code free, file format, and since it doesn't care what extension you use, developers can use custom extensions for the database file to launch their app. In this demo I use Click Once enhancements in .NET FX 3.5 SP1 to associate the custom extension with my ClickOnce installed application.
Managed Stored Procedures (A Common DLL for data access to Compact and SQL Server)
The great debate will forever continue on the debate of sprocs and its religion or science. I wrote about this here, but rather than debate the topic, here's a sample that shows how you can write a common data access layer to target against SQL Server (Express included) and Compact (SQLce) As outlined in this presentation, you can get far better scalability from your database by moving the logic out of the database as you can create several front end services to a single database, but you can only scale up a database so far before you need to scale out, which today is a difficult task.
Logical Queuing
In a service oriented world, is it really practical to assume all "services" can be synchronized? Or, is it a mixture of synchronizing some data, and submitting to existing, non synchable services that really touches on reality? In this demo, associated by this blog post, I demonstrate why using a database as your "queue" to services provides much more practical and realistic methods for working with existing services than expecting all the existing services to rev to support "sync". Sync is a very important technology, but no one technology serves all. In this blog post I noted down some thoughts, but not as polished as I was hoping to post.
Using Compact within Excel
Ginny Caughey and I co-presented at Tech Ed US in 08. Ginny assembled some great demos on using SQL Server Compact within VSTO, particularly within Excel.
Database Viewer
Every wanted to just look inside a SQL Server Compact Database, but didn't want to install Visual Studio or SQL Server Management Studio Express? With this little utility you can see the list of tables, and the data within each table. Not a full featured app, but rather a utility I use to show encryption or the ability to open a database across multiple users. It does show how to query the system views to get the list of tables.
Creating & Managing Local Database Versions through Scripts
When deploying applications with a local database, deploying the database with the app can seem easy at first, but once the user starts making changes to the local database, it gets messy quickly. If you're not careful, you can even lose the local data created/updated by the user. In this presentation, I created several animated slides that explain the complications and why script based deployment and versioning is the only practical way to manage local databases. This sample walks through the creation and versioning of local tables. To focus on the script based versioning aspects, I excluded any interaction with synchronized tables as that just gets even more complicated.
Versioning Local Databases with Scripts when using Sync Services for ADO.NET
In this demo we see how to version schemas when using Sync Services for ADO.NET. This includes two projects. One for the initial installation of V1, and another for the upgrade, or initial creation of V2. V1 includes creating a set of tables that aren't synchronized, such as state management for submitting orders to an existing service. Once sync completes, there are a set of additional scripts to create foreign keys between the synched tables, and between the synched tables and the local tables.
V2 enhances the synchronized tables with additional columns. The setup of this demo is a bit more complex as I've configured the server side sync components in a dll, then hosted the dll in a WCF Web Project. The WCF Web Project is published to an IIS instance, but uses different sub directories: for V1-http://localhost/VersionSchema/V1/LookupSyncService.svc, for V2-http://localhost/VersionSchema/V2/LookupSyncService.svc. Using this versioning scheme we can publish V2, allowing apps to slowly migrate over to V2, while the V1 apps continue to function and the business continues to function. The animation in this presentation helps explain this somewhat complex subject.

· Deploying Express with Scripts
Similar to the Compact deployment model, but with the additional steps required to create the physical mdf/ldf combination.

Private Deployment using the ADO.NET Entity Provider
One of the most popular features of SQL Server Compact is its ability to privately deploy the Compact runtime as a set of dlls directly within your application. But when using the ADO.NET Entity framework, things get a little more difficult as the Entity Framework uses the Provider Factory model for loading the individual data providers. I discussed this in more detail in this post, but here's a sample showing how to include the dlls and changes in app.config to work properly.
Common Programming between Desktop & Device
While SQL Server Compact provides the same apis between desktop and device (.NET Framework and the .NET Compact Framework), it can get a little more complex as the surrounding apis, such as the lack of the |DataDirectory| macro, and the lack of settings make it more difficult. This project shows a strategy for filling in the missing pieces of the .NET Compact framework so you can compile the same sourcecode for desktop, then device.

Some great partner tools

Data Port Console by Primeworks
Great tool for scripting tables from SQL Server Compact. For only $50, it's hard to beat
Remote SQLce
Great tool for managing a Compact database on the device, from the desktop
CodePlex
Codeplex is starting to collect several great code projects related to SQL Server Compact. I'm sure this will continue to grow, so it's a great place to look for what pain someone else has already endured

Great books

HitchhikerGuides.net - Bill Vaughn
Bill has a great, no BS style that just helps you get to the facts. It's a quick read, and at only $10, it's hard to pass up when you want to get up to speed quickly on Compact.
Microsoft SQL Server 2005 Compact - Prashant Dhingra
Prashant originally worked in the Compact team, and provides a great insiders look at SQL Server Compact
SQL Server Compact (In Spanish)- Jose M. Torres
If you're looking for a book on Compact in Spanish, here you go. I wish I could say I've read it, but, it's been years since I was at all fluent in Spanish. From what I have flipped through, it looks like it's covered all the topics. Jose has been active in the community, so I'd say it's a great bet.
Windows Mobile Data Synchronization - Rob Tiffany
Need some help, or want to know about Merge Replication? Might as well get it from the mast of Merge - Rob Tiffany. Rob spends his time traveling around the world helping customers scale and configure Merge. If it's been done, Rob has seen it, and likely has helped get it working.

Other Support/Info Channels

SQL Server Compact Blogs & Forums

Steve Lasker's Blog
The blog you got this document from, :) Lots of information on how we got here, demos and powerpoints. But, since I've moved off the team, won't be able to maintain the information going forward, so I'd recommend the following blogs going forward.

SQL Server Compact Team Blog
Where most of the team posts info. Everything from our encryption details, Entity support, query processor details, etc. You can think of it as the authoritative source. Even Laxmi posts to the Team Blog as well as his own...

Laxmi's Blog
One of great and passionate developers on the SQL Server Compact team. Be careful what you ask for, ‘cause Laxmi will tell you J. Seriously, Laxmi spends a lot of time in the forums listening and answering the issues and has a lot of experience with the product.

SQL Server Compact Forums
Where the team and our MVPs hang out to answer your questions. In addition to the product team, you'll also find answers from those that are actually using the product. We've gotten some great info here as well for how the product is actually used, as opposed to what we thought developers would do with Compact <g>

Sync Framework/Sync Services for ADO.NET Blogs & Forums

Sync Development Center
The home for the sync team and discusses the Sync Framework as well as Sync Services for ADO.NET.

Sync Team Blog
Less formal information on Sync Services and the Sync Framework

Sync Framework/Sync Services for ADO.NET Forums
A great place to post questions and find answers from the Sync team

Syncmaster Rafik website and Blog
Rafik Robeal was one of our core developers for Sync Services for ADO.NET and has all the nuts and bolts knowledge for what we originally built. Unfortunately, Rafik has moved onto other ventures. However, he's still got some great content on his site.

Enjoy the holidays,

Steve

Evolution or Revolution for moving to offline architectures

Steve.Lasker — Sat, 18 Oct 2008 03:15:00 GMT

There's been a lot of talk, momentum and questioning for where and how occasionally connected designs fit into the application landscape. There have been a lot of questions for "if and how" developers can take their existing apps "offline". There have been a number of efforts for enabling offline apps and some longer term plans. In VS 2008 (Orcas) we introduced Sync Services for ADO.NET as a developer focused way to synchronize data. The roadmap continues with the Microsoft Sync Framework releases. We've started to show "Astoria Offline" as a means to enable sync over REST based HTTP protocols enabling more generic services. But do any of these "solve the problem". It's been a while since I've done a brain dump on some thoughts related to the evolution vs. revolution of app models, so what better way to dump thoughts than a blogicle.

Can you take your current app and make it offline?
The reality is "it depends". There are so many app designs, it's hard to say definitely what works and what doesn't. There are some well established things that have worked and haven't, but it's not fair to say these apply 100% of the time. One size doesn't fit all.

Intercept/fall back
Do you go with the intercept model? When online and connected, the app communicates directly to the service. (Note I'm using "service" here to represent any remote source. It could be a database "service" a Web Service, a WCF service, or some other API that's only available when your device is connected to "the network"). When the network falls away, the app "falls back" to a local "cache".

This is generally a bad (opinion), or complicated (fact) model. This assumes one of two things. Either your user knows they're going offline and they synchronize before they yank the cable or they predict when the network fails. Of course the application can constantly synchronizing data in the background. Crystal ball? When do you really know or have time to sync beforehand? You're working in your office, and Outlook pops up reminder. Or worse, you get an IM or Text message asking "where are you". You grab your laptop and run... oh, wait, you forgot to sync. Do you have the info needed for the meeting?

For the background sync, one of two things could be happening. When you "fall offline", your view of data suddenly changes because your local copy is so out of date compared to the data you were just looking at, or your synching often enough that there's no difference, asking why you're even trying to work online in the first place. Or, you've effectively created a denial of service attack by asking, has anything changed, has anything changed, has anything changed?

Outlook 2003 (I think that was the version) tried this fallback model. It was awful. You're sitting in your office, writing an email, looking at your inbox. The network goes down, your inbox view completely changes, or worse Outlook froze, and you lost that message you were carefully wording.

Always work locally
In this model, you synchronize everything you need, and make all your interactions against the local store. This is sort of the Outlook model today (Outlook 2007). There are equal problems here. Is it realistic to have all the data locally? Even Outlook data can be significantly large, and that's clearly your data. What about public folders? By default, these are only available online. If you're online, they work. If you're not, they don't. Fairly straight forward; but what about historical data for all the products the company has shipped? What about your list of customers, or the current list of stocks and their prices?

The reality is data volume is growing at an exponential rate. Storage continues to get cheaper and faster. 1TB disks and Solid State storage are becoming mainstream. The problem is while connectivity is getting better; the bandwidth isn't increasing at the same rate. And, even as it does, is it realistic to have everything local? Do you keep one of everything that Home Depot or Safeway keeps in stock? Would be really nice, but not all that practical.

One size fits nobody
The reality is there's no easy answer. This is really an architectural and user model change is required to make this happen. We've been through these before. In the "old times", families ate what they grew. Corn, milk, meat, etc. Others only had Ice or Milk that was delivered to their house daily. You could think of this as the "online model". Today, we go shopping to Safeway, D'Agostino, or your local grocery store. The stores don't grow anything locally. They stock goods they get from somewhere else. You take stuff home and stock it in your pantry, frig, freezer. When you throw out the garbage it goes into the can. The can gets picked up on Thursday. It gets taken to the transfer station, and then off to the dump. The point is we have lots of "store and forward"/sync models already in place. What's really interesting is the "online" things are actually more expensive. It turns out it's cheaper to stock goods that may be used, and accept a percentage of loss for things that don't sell than it is to create on demand. Why do we think electronic bits and bytes should be "online" only? Why are 1's and 0's more expensive than physical goods?

Evolutionary approach: Mapping concepts
If we look at the concepts we have today in web/service/stateless models, they break down into two common, and one advanced models:

Get
Get maps very easily to HTTP:Get. Essentially you make a request to a domain, application and method. You likely pass parameters and of course all the authentication goo. The result of the get is returned and you now have a copy of information relevant to that point in time, and the specific question you've asked. Most likely, but no guarantee it's accurate by the time you quickly display it on your 64bit, quad laptop with a gigabit connection to the internet. Even historical data is subject to "correctness"- remember the Florida elections J But, we live with "copies" all the time. Copies are tangible, things that are "just there". There's a separate question of how "out of date" the copy is.

Submit
You have some information you're sending back. You expect some action to be taken on the data you're submitting. That submit can be accepted, rejected or modified. You get an immediate response, but no guarantee what you've submitted will be accepted. That might happen later on, it might not. You hope, or expect to be notified if something changes. But is that concept built into our app models? Or, do we just assume the world is globally connected, and everything is always running so things can be asked and answered immediately, with 100% accuracy, in one big transaction so no two people can allocate the same "goods".... Whew, that's as ridiculous to assume as it is to type.

Can Get and Submit evolve?
With Get, you can implement a "caching" model. Rather than Get directly from the source, imagine you Get through an abstraction layer. For each Get the abstraction layer keys off the domain/app/method and name/value pairs for parameters. Let's say I make a service call to: http://AcmeServices.com/Books/GetBookByName.asmx. I pass in: Title=SQL Server Compact

In the GetCache, it looks for the service and name/value pairs that match. If it finds a result, it may not have to actually make the network call. It could just return the cached result. But what about stale data? You can put an expiration policy on the cache. For a list of states, a day is likely more than generous. For pricing, maybe an hour. The point is you can customize this per source. For many things, one could argue most queries, the data is fairly static and the app could benefit immensely by caching the "gets". If, however, you were to look online first, than only if the resource wasn't available look in the local cache than what benefit are you getting? Sure, the app would "work offline", but you don't see any benefit in the online mode. Each request still takes the time to roundtrip. If you implement a smart caching model you can provide dramatically faster user responses. ...and in most cases, the data is perfectly accurate. Now it would sure be nice to know when something changes, so if California does sink, Iraq or Canada become the 51^st state, you don't have to watch CNN and ask all your clients to either wait for the cache to update, or "invalidate" the cache. I talked about notifications a bit here, so I'll keep going.

Submit
This one gets interesting as well. You have an order to submit for some widgets. Sure, you can submit directly to a "Web Service", or web page. But in many situations the order submission doesn't actually mean the order will ship. Accounting needs to verify your credit. The products that were supposed to be in stock may have been routed to a bigger customer, or "fell off the truck". Again, the point is just because you may get an immediate acknowledgement the order was accepted, doesn't mean it will be fulfilled. So, why not accept this as the model, and when it works, it just works. When a problem occurs, it's not really a problem because it's built into the system. What if you "queued it up" for sending? In this case we save the message locally, in a "logical queue", and when the service is available, we send it on its way. I used the term "logical queuing" as I'm not implying you must use something like SQL Server Broker, MSMQ or WCF Queues, etc. These tend to be a bit over complicated for what you need on the client, and it makes it difficult to act, query or evaluate the data in its transient state. A while back I wrote a blogicle on and some samples for Logical Queuing.

Sync
This is what some might assume is the answer to all of these problems. If I can synchronize changes from other nodes (the server and other clients) with my local store, and I can make changes locally and have them synchronize back "up" to the other nodes, doesn't that solve everything? In the land of demos, sure. But in the real world there are just too many complications. The volume of data, requirements for business logic, and workflow make it difficult to assume Sync can solve all the problems. Where sync does do really well is for synchronizing reference data. That list of states, codes, or even larger volumes like the product catalog. The Logical Queuing blogicle and sample demonstrate some thoughts here. I recently saw some work the Microsoft Mobile Line of Business CRM team, and it also uses a similar model.

Revolution is on the horizon
With all the complications and half way solutions, you have to wonder what's the real answer to all this mess. Will this problem just go away because we'll have high speed internet available everywhere? Even if you could be guaranteed internet access everywhere, do we think that every service will be available all the time? As the world gets more connected, will systems be more reliable and capable of enlisting in massive distributed transactions? Or, will computing systems evolve to what every other large scale system as evolved to. Queuing, Caching, Reserving, Buffering, Acknowledgement, Conflict Detection & Resolution and Notifications.

While I talked a bit about the food supply earlier, this is also a bit like the VB Classic to Web transition. For those old enough to remember the great 90's there was a big transition from "fat clients" to "thin clients". No subtle suggestion of which is better there J When developers first started building ASP Classic (pre .net) apps they simply moved their session state to the ASP Session object. Which meant web servers could serve about 5-10 people before falling over. Not exactly the promise we had all hoped for. It took a while to realize the app models had to change. But, moving from Client apps to Web apps was a bit more obvious. Mostly because the UI changed (for the worse), but at least users could get their work done because we couldn't get the VB apps deployed. (yes, I'm over simplifying things, but run with me for a minute).

Since developers had to re-write their UI anyway, it was an obvious change. Making the transition from online to offline mode apps isn't as obvious. The only UI required is the UI really telling them when problems occur. Who wants that?

The reality is things are a changing. Nothing to announce, nothing to promise, but there is a revolution on the way. Revolutions take time for people to live through denial that it's coming, but it comes as a wave of reality that just smacks you with "duhh, I guess that was a stupid assumption".

While we all wait for the revolution, you can look at things like ADO.NET Sync Services, the Sync Framework, Astoria Offline, and lots of goodies the Patterns and Practices team has done.

Steve

Application Services - Optimizing Online, Enabling Offline clients

Steve.Lasker — Sun, 20 May 2007 18:23:00 GMT

Saurab, a PM on the UiFX team posted a screencast on the new application services coming in Visual Studio Orcas enabling settings to be cached locally, and synchronized with the profile store in ASP.net. He also demonstrates caching authentication and roles as well.

http://blogs.msdn.com/saurabh/archive/2006/11/27/live-from-redmond-client-application-services-in-orcas.aspx

The screencast is a bit old, and seems sluggish, so for those that prefer a walkthrough, Saurab and Karl have created this step by step info:

http://blogs.msdn.com/winformsue/archive/2007/05/20/client-application-services-in-windows-forms-end-to-end-walkthrough-available.aspx

Steve

First look at the Visual Studio Orcas Sync Designer

Steve.Lasker — Thu, 22 Mar 2007 03:32:00 GMT

Here's part 1 of the new Sync Designer. In this screen cast I walk through how to cache lookup tables locally. In this screencast I use the 2 tier sync, starting simple. In the next screencast I'll show how to take the same designer, and split it across N tier to keep the intimate knowledge of the server, connection string, queries, passwords, etc. from the client.

http://channel9.msdn.com/posts/SteveLasker/Visual-Studio-Orcas-Sync-Designer-for-Caching-Data-in-SQL-Server-Compact-Edition/

Just as a note, this screencast shows an internal build of Visual Studio Orcas between the Feb CTP and Beta 1.

Steve

Additional Q&A on the Visual Studio Orcas Sync Designer

Steve.Lasker — Wed, 21 Mar 2007 20:54:00 GMT

Q: Why does the Orcas Feb CTP Typed DataSet designer not work on Vista?

A: Visual Studio Orcas is not yet fully compatible with Vista. In the Feb CTP there are several issues with something we call "red bits". These are changes to the runtime that shipped in .NET 3.0 which shipped with Vista. For our own internal testing, we do have fixes allowing us to test on Vista and all the scenarios are working. However, due to the way the dlls are signed and shipped in Vista, and the time crunch we're working under, we weren't able to get the work done in such a way that we could service a Vista box that had the Feb CTP installed. This was a very tough decision as we do not want to set the expectation that Visual Studio Orcas won't be Vista compatible. But, it was felt more important to make sure that anything we ship, including the CTP must be serviceable. As we all know, we're working on to maintain our schedule, and since CTP's are not meant to reflect the final quality, it was determined to ship the Feb CTP without full Vista support. As those that have seen my demos, we do have Vista working, and we will have full support for Vista. I believe, but not sure, that B1 will be fully Vista compatible. So, for now, if you're using Vista, the Feb CTP should be run in a VPC. In general, it's not a bad practice anyway.

Q: Will the Sync Designer generate time based sync?

A: This feature was unfortunately cut. We had some great designs, and even some prototypes that I've demo'd previously. However, we just didn't have the time to do all the background task, network resource detection, and timer work. Rather than hack something together we didn't have time to fully design, dev and test, we've had to cut this feature altogether. Of course this doesn't mean developers aren't smart enough to do this work with a combination of the BackgroundWorker, Timer and Network events. It just means we don't have a checkbox to do all the work automatically. We will have samples on how to do this, and we're in discussion with Patterns & Practices group on filling these gaps ‘till we have fully integrated runtime components that make this "just work".

Q: Will tombstone records be automatically cleaned up?

A: This was another cut we made in the sync designer with Visual Studio Orcas. Tombstone records are how we track deletes. If you sync on Monday and get the full product catalog, and then sync again on Wednesday, how can the server answer the question of "what was deleted". There are many ways to handle this. You can mark the main record with a status of Active = False, or you can physically delete the row and in the Deletion Trigger copy the PK to a [Table]_Tombstone record. The later is what the Visual Studio Orcas Sync Designer does. We had planned to create a Scheduled Task to automatically purge these deleted records, but since SQL Server Express doesn't have the Agent, we felt this was going to be too confusing. We also expect developers will do some additional work between their development and production boxes, so for now we punted this. We're not yet talking in detail about our SQL Server Katmai release, but you can expect we're doing some cool things here. So, for now, you'll have to create a scheduled task to purge the tombstone records that are say older than 30 days. Stay tuned, and we'll have more exciting things to discuss about the upcoming Katmai rlease.

Q: How do I get my cached tables to be synchronized in a single transaction?

A: The Orcas Sync Designer is focused on download scenarios. You can absolutely extend the sync designer generated code to do uploads as well, and in fact we even generate the upload SyncAdapter commands. For download only scenarios it was felt we should keep things simple by default and enable the ability to easily recover from dropped connections. So, by default, the Sync Designer synchronizes each table individually. As one table is finished, the next table is synchronized. This means if table 1 finishes, and the connection drops during table 2, that the next time a sync occurs, Table1 is likely already complete. To put all the tables in a single transaction, expand the [advanced] expando button and choose the sync all tables in a single transaction option.

For the Feb CTP, the Sync Designer generated a SyncGroup for each table. This wasn't necessary as the SyncAgent knows to place each in it's own SyncGroup when one isn't specified. This has since been changed so you'll only get a SyncGroup when placing all in a single transaction.

Q: Once all the tables are placed in a single transaction, how do I control the order the tables are updated to handle parent/child relationships?

A: The Sync Runtime will apply inserts, updates and deletes based on the order of the SyncTables and SyncAdapters in their respective collections. This isn't fully implemented in the Orcas Feb CTP, or the Jan Sync CTP. However, we do now have that updated internally and will be releasing it soon in the CTP. In an upcoming screen cast you'll see the current updates to the Sync Designer, including the ability to move items in the collection up and down.

Q: Does the sync runtime create relationships locally within SQLce?

A: No, I touched on this here: http://blogs.msdn.com/stevelasker/default.aspx#constraints

Q: Does the sync runtime work with server side identities for PK's?

A: No, no, and maybe. In general, despite the fact that many of our own samples depend on the Identity feature to manage the primary key, identities are an overall poor design. Building offline apps is only the extreme of the problem. How do you create a new row on the client, assign it a primary key when the client won't actually know the PK until it posts it to the server? However, the problem exists in "connected" stacks as well. Say you're creating a Order with a collection of OrderDetails. The standard model is to have the OrderDetails table reference the PK of it's parent, than add another sub key, say line number. You can't actually create the OrderItem records until the Order record is posted first. This creates complications for connected development. Sure, you can use the @@scope_identity value, but you still have to do a bunch of updates. Sure, you can wrap this in a sproc, but it's still a PITA. Now, consider you're building a high availability system. You have multiple server nodes managing the creation of data. Who owns the master identity range? We see this in rollup scenarios all the time. Each branch maintains their own data. Each uses identities. When the data is rolled up to corporate, you have collisions, requiring you to add the branch id to the corporate database. Now, you could partition the identity values, which is what we do in Merge Replication, but now we've introduced additional problems. Prior to joining Microsoft we deployed many mobile applications. If a person is in the field, using Merge Replication, and is working with their partition of IDs, what happens when they use them up? They have to re-synchronize with the server to get a new allocation, which they may not be able to do at the time they've run out. So, the app is dead. Another client doesn't use up their allocation and leaves "holes" in the server.

So, what's the solution? There are two ways, the simple but not always the most efficient, and the complex. The simple is to simply use GUIDs. SQL Server and SQLce both have UniqueIdentifier data types. You can either set the IsRowGuid property, or create your own GUID with Guid.NewGuid(). The other is to create your own hashing. Each client is given a hash when the initially connect to the server. They then create incremental counters, possibly even with Identity within their hash. Now when you push up the new values, voila, no conflicts. If you still want incremental counters, say for that single, simple OrderId, you could still use the Identity feature, just don't make it the primary key. You upload a new order from the client. The client assigned a primary key using it's hash. When that order is uploaded to the server, the server applies the identity value to the non-pk column. The update is then brought back down to the client. Life is good.

Steve

Q&A on OCS & Sync Services for ADO.NET

Steve.Lasker — Sun, 18 Mar 2007 17:25:00 GMT

Not surprisingly we've been get a lot of great questions about specific features and scenarios for our new Sync Services for ADO.NET (OCS). Rafik has been fielding most of these on the Sync Services forums. Since the Q&A for SQLce seemed popular, I thought I'd do the same here.

Q: How does Sync Services compare to Merge Replication and RDA?

A: Merge replication is our developer oriented, feature rich, SQL Server based database "replication" product. It's designed for a DBA to configure a publication on the database exposing a set of tables (articles) enabling filtering and rules. Clients then "subscribe" to the publication and receive a local database reflecting the publication. It's a very powerful and mature product that we will continue to invest in. From the server side (publications), Merge Replication is a SQL Server only feature that is available in our SQL Server Workgroup, Standard and Enterprise SKUs. Merge Replication, as a publisher, is not available in the free SQL Server Express SKU. Merge replication supports 2tier sync and sync over HTTP(s) to enable internet synchronization. However, I wouldn't say that Merge is SOA oriented in that you don't have much control of what's sent or received across the wire, nor do you have the ability to support additional transports. It's a fantastic end to end replication product that handles a lot of complicated scenarios.

Remote Data Access (RDA) is a developer oriented, simple but RAD sync technology. It works over HTTP, so it enables internet protocol sync, but again, it's not very SOA friendly. While the client doesn't directly open a connection to SQL Server, it does require the client provide the connection string and query to be executed on the server. So, it pretty much violates the notion of keeping the server information from the client. While it is limited, you can't deny it's popularity for it's simplicity. We continually hear many people implementing RDA rather than merge as theire requirements are simple and RDA just worked great. Which brings me to Sync Services for ADO.NET

Sync Services for ADO.NET is our answer to this dilemma of having to choose between the dba focused powerful features of Merge, the simplicity of RDA, and the developer motivation for them to write their own. When designing Sync Services we used RDA as our user model. Debra Dove and her team did an excellent job with RDA, and we wanted to take it to the next level. We used the provider and developer programming model of ADO.NET and wanted to leverage all the great transport, security and protocol work others were doing. Essentially, rather than build additional solutions to existing problems, we wanted to scope the solution to what we needed to solve, and leverage others who were experts in their field. The Sync Services for ADO.NET features are built by the same team that delivers Merge Replication. I've been truly amazed at the knowledge these guys have. It's because of their experience that I have a hard time calling Sync Services a 1.0 product. It's really a culmination of all the great work from Merge, RDA, file synchronization, and even the WinFS sync work.

The quick bullet for Sync Services is it's a componentized synchronization framework, built on ADO.NET, but factored to provide a common sync platform for Entities, Files, and other formats. Sync Services for ADO.NET is our first delivery in our Microsoft Synchronization Platform.

The key advantages of Sync Services is its developer focus, w/DBA empowerment. Rather than force your DBA to understand all the details of how you're going to synchronize data within your application, you engage the DBA for the portions that touch the server database. The transports are up to you, and we have a great WCF designer integrated story, but you can use other transports as well. The local database schema/structure is up to the developer to decide as it's their apps database. Of course you can engage your DBA as well, but you don't get what you get as the result of a synch operation. You can use just the client components and use Java and Oracle on the server, or you can use just the server components and other technologies on the client. If your data is locked up in Oracle, you can still use Sync Services to synchronize that data directly from Oracle to your client. You don't have to put SQL Server in the middle just to enable sync.

The point here is regardless of what cards you've been dealt, we want to help you get your app done to make your users happy. The more Microsoft products you use, the better the experience we can provide, but we're not locking you in, or out of our platform just because something is out of your control.

Q: What's the roadmap for Merge, RDA and Sync Services

Merge replication will continue to be our database replication product and will have Katmai investments in the next release. Merge will be the DBA tool for replicating a database, and those that are already using Merge shouldn't feel like we're abandoning them by any means. If Merge is working for you, don't worry, we're continuing to take feature work and making improvements. In fact, many of the Sync Services features will work into Merge as well. That's about all I can say for now. We'll announce more about Katmai later on.

RDA will eventually be phased out. We truly believe the Sync Services features address the simplicity of RDA, but don't have any of the limits imposed by RDA. Specifically, incremental changes, ability to synchronize several tables in one transaction handling all the interleaving of inserts, updates and deletes. Support for other ADO.NET Providers, etc. If you're already using RDA, we're not killing it yet, but we won't be doing any new work there either. We do expect to deprecate it within the next release or so as Sync Services are released. If you haven't yet deployed RDA, but are looking into it, definitely look at the Sync Services CTP. If you need to go into production now and can't wait ‘till Sync Services are released, than by all means use RDA. It's not like it has that much functionality that you won't be able to easily switch to Sync Services later on. <g>

Sync Services is where we're doing a lot of our investments. It's just the first delivery in our new Microsoft Synchronization Platform. I'll write a different blog post on our naming. (I love discussing naming, ...not). Sync Services are our developer oriented, SOA enabled data synchronization features. We won't have all the database replication features of Merge, as we really focus on synchronizing data. Sync Services provides and end to end story, but is a componentized model allowing you to get in the middle, or completely replace one end.

Q: Can I use Merge and Sync Services together?

A: As we've all seen with our SQLce and Express discussions, one size doesn't fit all. But we also don't believe you need 10 ways to solve the same problem. Just as with Express being our entry point to our Data Service platform and SQLce being our client/embedded platform, we expect Merge and Sync Services to address the needs of two types of scenarios. It's likely that some scenarios, such as a branch office may actually use both Merge and Sync Services. Merge to replicate data between the branch office and the corporate office, and Sync Services to enable branch workers to go out into the field.

Q: Does Sync Services support N Tier?

A: Absolutely. The beauty about the N, is it could imply 1 to many. With Sync Services we have Server and Client Providers. Actually, we like to think of them as local and remote providers as we currently focus on hub/spoke, but we'll be enabling p2p as well in the future. You can start with 2 tier and move to N tier. You can intermix 2 tier and N tier based on where the clients are connecting from. When you first start working with Sync Services you may ask why you have to specify the tables your synchronizing on the client and server providers. That's because we want to make sure you can easily split the client and server code to different tiers. All the "intimate" knowledge of the server can easily be moved to the mid tier, while the client provider configuration is maintained on the client. Even the Sync Designer allows you to easily split the client and server provider code. It's quite sweet when you see it. Screen cast coming soon...

Q: What transports do you support?

A: What do you want? As noted above, the sync team wants to get out of the transport business. We have many bright people in Microsoft thinking about those problems. In Orcas the Visual Studio designers will focus on enabling WCF, but you can use Web Services, SSE, SyncML or if you can figure out how to convert Jelly Beans to .NET objects, you can use those as well. Essentially, you simply plug in a matching service and a proxy, and you're good to go.

Q: Do you support column level tracking?

A: Not yet. For this first release, we only support row level tracking. We are working on column level tracking, but it does require additional functionality on the server, and we wanted to make it easy for developers to start synchronizing data from their existing databases. As with any good database design, you should think about how you partition your tables. For various reasons, it's good to separate images, or other large blobs from your primary table into other tables and maintain a 1:1 relationship. It's not meant to be an excuse, and we will be implementing column level tracking in the future. Using features like ADO.NET V3 entities you can roll up the 1:1 mappings into a single object allowing your developers to work in a normal fashion, while managing performance and isolation in your database.

Q: Do you support custom conflict resolvers?

A: Of course. How could we not? On both the server and client providers we expose a conflict event. In that event you'll get the client and server rows that represent the conflict. You can simply say force overwrite, or implement very custom logic. Such as determining if the person who made the change is a manager, or owner of that particular customer account. Based on that custom logic, you simply make the logical change. All this is in managed code, with your favorite .NET language.

Q: Do you support partitioning/filtering?

A: Of course. We don't really expect people to synchronize terabytes of their data to all their clients. The partitioning is the normal horizontal and vertical partitioning. You simply provide the query that represents the filter you wish to support. You can do joins, etc. It's just a query. The client sends up as many parameter values as you need. There's no limitation on the number of parameters. In fact, you can even intercept calls on the server and set sync parameter values based on other logic. In WCF you can determine who the client is, and based on that info, send them their customers without exposing the SalesPersonId to the client to substitute another value.

Q: Does the server track each client individually?

A: No. This is one of the major differences when compared to Merge. One of the powerful features of Merge is it knows all its clients, so when the client connects, it already has the data ready for it, and easily supports "data repartitioning". In Sync Services, the server has no idea who all the clients are. The benefit here is Sync Services don't have the same scalability constraints. You can synchronize as many clients as your server can handle queries. The server doesn't necessarily know it's a publisher, but rather it's just answering queries.

Q: Does Sync Services support multiple publications

A: Yes/no. Sync Services doesn't utilize the pub/sub model per se. You can configure the server provider to offer 20 tables you want to synchronize. The client simply say it cares about 3. Another client cares about a different 3. Another client cares about 4, which overlap the first two clients. In fact, we also support a client dynamically adding tables. A sales person may cover another sales person for a week and needs to bring in an additional product line. Within the app, the developer can change the filtering query, and off they go.

Q: How are schema changes handled?

A: Unlike merge which is geared around replicating a database, Sync Services is geared around synchronizing data. I'm not a big believer that generally speaking the DBA simply adds a column to the server and the UI automatically updates on the client and life is good. While it can be done, most of the time I'd bet you want some control over where and how the new element is displayed, add some interaction logic to the client, tab order etc. We really treat schema updates as an app update. It's a holistic update of the app overall. The model we've gone with the Sync Services are the following:

A new requirement is defined, say AddressLine3. The DBA would add the column to the server. All the normal rules apply. If the column is non-nullable, than a default should be provided.
The developer involved with the sync layer would most likely create a new version of the Sync Service, say v2. This means that apps that were using v1 can be slowly migrated, or at least be migrated within some level of control. If the user is in the middle of an important deal, the last thing they need is a forced software upgrade. Ever been in the middle of something important and IT forces an update that reboots your computer or app? Software is an enabler, it should help me achieve my goals, not fight me because IT thinks its important now.
The app developer updates their service proxy to point to v2 of the sync service, exposing the extra column.
In the version check code, the app author can either choose to reset the table, or they can execute the alter table script locally adding the additional column. They may even bring down a single data call to retrieve the values for the new column on all the existing rows.
The developer than decides what they want to do with the new element, updating their ui, logic etc.

So, while we didn't implement something as simple as point click, we think it tends to fit the SOA model where apps may consume services from other apps, and they should have control over how and when they consume new schema.

Q: How are constraints, keys, and other db objects brought down to the client?

A: This again falls in the category of Sync Services is about synchronizing data, not replicating a database. Sync Services does do some schema and even database creation with SQL Server Compact Edition. If you're starting from scratch, and you first synchronize, the SQLce database will be created based on the connection string properties, name, encryption, password, etc. It will then create all the tables the client has said they're interested in. Remember, just because the server exposes 20 tables, doesn't mean the client must use all of them. The client determines which tables it wants to consume with the SyncTable collection. When the tables are created, primary keys are created, datatypes are mapped to the clients datatypes, and nullability is applied. No additional indexes, constraints, defaults, etc. are applied. There are SchemaCreating/ed events fired where you can either initial create the schema to be used, or alter the schema after the tables are created.

Q: Does sync services handle parent/child/grandchild relationships?

A: Yes. Unlike RDA where you can only sync one table at a time, Sync Services handles the hierarchical nesting of inserts, updates and deletes. In fact you can even control it seperatly on the server from the client. On the server, tables are placed in the SyncAdapter collection. The order of the SyncAdapters defines the order by which updates will be applied. Inserts and Updates are done from the top down, while deletes are done from the bottoms up. The same is done on the client, in the SyncTables collection. This allows the server to control its order, while allowing the client to control its order of updates.

Q: Can I update only a few tables at a time?

A: Yes. Say you want to only synchronize your lookup tables, states, codes, etc. once a day. You can create a specific service just for your lookups, and another for your product catalog.

Q: Can I update everything in a single operation, or can I control things more granularly?

A: Within the SyncAgent, you can utilize the SyncGroup to determine the grouping of updates. In the previous example, you may choose to put all the lookup tables in their own individual groups. If the connection drops while you're synchronizing your lookups, it can pickup where it left off next time it synchs. However, when synchronizing Orders, you probably don't want Orders to ever go up/down without OrderDetails. Simply put the Orders and OrderDetails table in the same SyncGroup, and you're all set.

Q: Does Sync Services support batching for large data sets?

A: Yes, but not quite yet. We initially scoped this out of the first release, but we believe we'll be able to get it in, so look for it sometime around March 07.

Q: How does Sync Services track changes?

A: Sync Services uses an Anchor based model. Each time a sync operation occurs it gets a reference mark from the server. It could be the servers DateTime, or a TimeStamp (RowVersion). The client saves that value for the next sync operation. Each time the client synchronizes a particular SyncGroup, it first requests the server anchor. It than executes the queries on the server using the last anchor as the low range, and the new anchor as the high range. This gets a consistent set of changes across several queries. In future releases of the Microsoft Synchronization Platform we'll be supporting a knowledge based sync model as well as the anchor based model discussed here. Rafik does a great job explaining it in his blog

Q: How are deletes purged?

A: On the server, deletes are either kept in a tombstone table, or simply tracked by some sort of active/status flag in the primary table. Since this version of Sync Services isn't tightly coupled to SQL Server, we actually don't do anything. In general, we'd expect the DBA to write a scheduled task to purge tombstone records on their determined interval. You can expect us to do more in the "future", tease, tease, tease...

On the client, SQLce purges deleted records once it confirms data has been sent to the server.

Q: Can I purge old data on the client without triggering a delete on the server?

A: Yes. While we don't have a simple API to do this today, you can delete a bunch of rows on the client based on what ever criteria you decide, than simply "AcceptChanges" on the client prior to these changes being sent to the server. Of course you could intercept these on the server an toss deletes as well to protect your server data.

Q: Does Sync Services support low bandwidth type sync scenarios?

A: By low bandwidth, I mean can I synchronize only the important things now, and catch up later. Yes. You can either upload only, download only, or synchronize just a particular SyncGroup based on your own logic at the time.

Q: When will Sync Services ship?

A: Sync Services for ADO.NET will ship at the same time Visual Studio Orcas ships. This is currently scheduled for Q4 2007. Note: This is not meant to be the official place to get the timeframe for Orcas, but rather just saying that our current plan is to ship Sync Services within SQL Server Compact Edition 3.5, which will ship with Orcas.

Q: Will Sync Services ship on both the desktop framework and the .NET Compact Framework?

A: Yes, but at different times. At current, March 16th '07, we are scheduled to ship for the full framework, but we are not planning on shipping Sync Services for the device platform in the Orcas product. We do plan to ship the client components for Sync Services soon after Orcas, but are still working out the schedule. The problem is the various .NET Compact Framework teams, including the Visual Studio for Devices teams and Sync teams have a lot of work to manage with many different device platforms and a very short schedule, and we haven't been able to get all the appropriate ship level test coverage complete. We have designed, and done preliminary testing with the client components working and synching over Web Services. We are still hopeful we can pull it in, but at this point, we're just not ready to commit to Orcas, but rather shortly thereafter.

Q: Will Sync Services or SQL Server Compact Edition be in the .NET Framework, or .NET Compact Framework?

A: No, we are not shipping within either framework, but rather shipping as an add-on component. Why? Because we wanted more flexibility with our ship schedule. SQLce will ship 2-3 times between .NET 2.0 and .NET 3.5. While it would be nice to ride the distribution of the frameworks, with the embedded/private deployment options of SQLce and Sync Services, we felt it was better to have more flexibility with our schedule.

Q: Are these the only questions?

A: I doubt it. So, keep them coming, and I'll update this FAQ as I receive them.

Thanks for all the great questions. Keep them coming as they help us make sure we're shipping the right features in the right order.

Steve

Steve Lasker's Web Log - www.SteveLasker.com/Blog : OCS

Demos, Presentations, Links, Screencasts and Videos for SQL Server Compact

Recent Presentations at PDC and Tech Ed

Just the demos, and nothing but the demos...

Other interesting links to previous posts:

Some great partner tools

Great books

Other Support/Info Channels

SQL Server Compact Blogs & Forums

Sync Framework/Sync Services for ADO.NET Blogs & Forums

Evolution or Revolution for moving to offline architectures

Application Services - Optimizing Online, Enabling Offline clients

First look at the Visual Studio Orcas Sync Designer

Additional Q&A on the Visual Studio Orcas Sync Designer

Q&A on OCS & Sync Services for ADO.NET