The Synchronizer

Sync Designer in VS2008 SP1

Rafik — Fri, 30 May 2008 23:43:00 GMT

I am sure you already know that SP1 Beta for Visual Studio 2008 is available to download. There are plenty of enhancements that I am sure the VS folks are covering all over the net with different blogs. As a sync guy, I am on the lookout for everything new in sync and SP1 has made a lot of improvement to the Sync Designer introduced for the first time with Visual Studio 2008.

The designer was revamped quite a bit and it much more developer friendly now. The biggest improvement however is the ability to tap into SQL Server 2008 Change Tracking feature. The designer gives you the ability to use Change Tracking instead of the traditional trigger based model (a.k.a. coupled tracking as tracking columns are coupled with data columns in the base table).

Under the covers, the Sync Designer uses a newer version of the SqlAdapterBuilder to generate the adapter that works against SQL Change Tracking. The new class is part of the newer beta bits of the Sync Services for ADO.NET. Developers can dynamically generate the commands be configuring the builder to run against an instance of SQL Server 2008 with Change Tracking enabled on the target table.

Guy Burstein has written a really good post on the new changes in the sync designer with nice snapshots including the one I posted here. Make sure to check it out.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle, Best Hotels in Miami, Best Hotels in San Francisco, Hotels with in-room jacuzzi and 5 Star Hotels in Los Angeles. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Metadata is the Hart and Sole of Peer Sync

Rafik — Fri, 30 May 2008 01:09:00 GMT

Occasionally, I receive questions about using SQL Change Tracking with PeerSyncProvider. People who examine SQL Change Tracking in preview version of SQL Server 2008 have become fans rather quickly and see the value of using it. Using change tracking with ServerSyncProvider scenarios is a piece of cake and will get easier with the VS2008 SP1. Now how can you do the same with PeerSyncProvider?
The short answer is that it is not so easy. Not as easy as with ServerSyncProvider. The reasons might not be so obvious and I see many attempts of writing the commands for PeerSyncProvider adapters to work against SQL Change Tracking. While it might work for very limited scenarios, it is largely incorrect.

Again, the question is why?

Well, let’s think a little bit about the following SyncAdapter commands for the PeerSyncProvider:

InsertMetadataCommand
UpdateMetadataCommand
DeleteMetadataCommand

Notice that these commands are intended to make changes to the tracking metadata. That’s rather interesting and clearly different from the commands you supply to for ServerSyncProvider. Now it is becoming clear that we have a problem since SQL Change Tracking metadata (i.e. CHANGETABLE) are all read-only!

General Purpose Tracking Infrastructure

Remember that SQL Change Tracking is a general purpose tracking mechanism. You enable change tracking for a table with the sole purpose to be able to ask “what have changed since given version?” If applications are allowed to make changes to the metadata then this question is not reliable and thus useless. Making change tracking metadata read only makes all the sense in the world for its general purpose nature.

Direct Metadata Manipulation

An obvious question here is: Why make direct changes to the metadata? Changes to the base table will be reflected in the CHANGETABLE, wouldn’t that achieve the same effect?
Well, for peer to peer data synchronization… you need more control than that. Let’s take the following two cases:

Synchronizing Deletes

Let’s consider the following example, three nodes A, B, C. The nodes are connected as shown in the diagram. Follow the following scenario:

Make a change R on node A
Let A sync with B, B now has R
B deletes R
Let B sync with C. B will send the deleted row metadata to C

As B received the metadata for the deleted row, it needs to persist is. R never make it to B thus a normal delete command won’t generate the metadata we desire as it simply won’t do anything.

As a side note, the delete metadata must be persisted on B. Since B could be connected to another node (A, for example) which have R and thus needs to pass on the delete during subsequent sync. In P2P environment, one cannot make assumption about the other nodes and must maintain metadata for every deletes.

At a first glance one can come up with a clever work around for this scenario which is insert R then delete it right away. SQL Change tracking should record that, right? That’s right but the workaround cannot survive two or more tables with PK-FK relations which is the reality of most databases. Thus this workaround is short lived.

Resolving Conflicts

When conflicts are detected and a winner is chosen, one needs to update the metadata such that this winner will propagate across the topology of nodes. That said, the version of the row needs to be pumped. Now consider the case where the local row was selected as the winner. With SQL Change Tracking, there is no way to make changes to the version as the metadata is not editable.

My point in this post is to answer a common question and illustrate the underlying technical reasons behind the limitation. That being said, you can still leverage SQL Change Tracking for Peer Scenarios albeit you need to use a side table to store the changes to the metadata. I am sure some of the readers will find this as a worthwhile challenge to tackle. Happy Sync’ing!

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Top 10 Hotels in Myrtle Beach, Best Hotels in New York City, Romantic Hotels in Seattle, Hotels with a kitchen or kitchenette and Top 10 Hotels in San Francisco. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

HTML5 - The Offline Web

Rafik — Sat, 26 Apr 2008 00:55:00 GMT

We hear every day that more and more people are coming online with high speed internet connections, WiFi hotspots are increasingly popular and getting faster, unlimited mobile data plan are cheaper than ever and the reception is better than ever. Everyone and every device appear to be always connected.

Ironically, when you check out W3C latest HTML 5 specs, you notice a key new APIs:

Persistent storage. Both key / value and a SQL database are supported.
An API that enables offline Web applications.

Isn’t interesting that despite the current networking revolution people are building the next web to work offline!? It is not strange to me but for many people this is bizarre.

I attend most of the design review meetings in my group and I can tell you that even software professionals don’t get the value of offline access when the application they are building is connected all the time. To make a point, I always use outlook analogy. Outlook in every office connected all the time, yet it is fully offline application. A stronger argument to make in the next few years will be that the browser itself with HTML5 is another flavor of an offline application. Can you beat that!?

To me, offline is an experience and programming model. Offline experience is all about richness, responsiveness, and better use of the data pipe. An offline programming model is what I’ve been showing people with Sync Services.

Once the need for offline or caching on the client crosses the chasm in people minds, another debate starts about how to implement that effectively. Most of the time I see developers thinking of offline access as a cache of random elements that will get cleared based on least accessed algorithm. This is fine for a brower cache or file cache like Windows Client Side Caching where you are randomly accessing remote content. But what about accessing a well defined data set remotely, like a given directory or a data table. In this case, an offline access with sync semantics could be a better fit. Suffice to say that it is not that obvious to many people.

It is so much easier to build application to run against a local store and in the back ground the store is synchronizing with another end point than building a cache hit and misses logic where the application needs to connect to the end point and get the content, deal with all sorts of network issues, and maintain a seamless user experience … etc. Well, it is sad to say that most people underestimate the complexity with the latter model until it is too late.

The good news is that HTML5 will bring a new store to browser near you. Use it wisely and go party offline!

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in Myrtle Beach, Best Hotels in Seattle, Top 10 Hotels in Miami, Hotels with in-room hot tub and Kid friendly hotels in San Francisco. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

The Mesh – A New Synchronization Backbone

Rafik — Fri, 25 Apr 2008 02:52:00 GMT

A lot of excitement about the announcement of Live Mesh this week. It is interesting to see this collection of services put together tying user content, devices, and social circle together in a web enabled experience. My favorite part is the news pane where you can see recent activity stream.

Since the announcement, I’ve been asking myself, what is the key problem that Live Mesh is trying to solve?

Is it remote access?
I am not sure if that many people want to access their machines remotely. I see the need for anywhere access to content but not to given physical machine. If the content is available somewhere else that would suffice. Also remember, not all computers are left running 24/7 or connected through cable internet, this is definitely not the case in many countries
Is it data sharing?
There are a lot of services that does that today. You upload something to SkyDrive then share it with others. The news part is neat though.
Is it data collaboration?
Hmm, I think we are on to something here. An internet scale collaboration service is one attractive feature of the Mesh. Enterprises have it easy with SharePoint, but when it comes to end users and small businesses. Having data delivered to different people to work on is needed.
Is it data synchronization platform?
While Microsoft Sync Framework gives you that tools to build something like the Mesh if you wish, no body with limited resources can build such service. It is a huge undertaken for even large companies. That said, Live Mesh builds the backbone for a global sync infrastructure that enable application developers to work against local store and leave to the Mesh the task of moving the data to other end points. I’ve not looked deeply into the details of the platform yet, though.

I am really interested to see more details coming out in the next few months about the Mesh development platform. Let’s keep watching the Mesh as it unfolds…

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle, Kid Friendly Hotels in Miami, Best Hotels in San Francisco, Hotels with in-room jacuzzi and Best Hotels in Chicago. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

New Release: Windows Mobile Line of Business Solution Accelerator 2008

Rafik — Sun, 16 Mar 2008 22:42:00 GMT

Rob Tiffany, the mobile accelerator guy, just announced the new release of Windows Mobile Line of Business Solution Accelerator 2008. It a collection of windows mobile centric samples that showcase different new technologies and help developers greatly accelerate building their applications, thus the name “Accelerator”. Rob also has a great blog that I read regularly. Here is a quick overview of what this new release contains:

Adapt your App
Create a single binary that runs unchanged on Windows Mobile Standard or Pro, Portrait or Landscape, Rectangle or Square. No more wasting time building separate executables to accommodate different screen sizes or input methods.
Sync Services for ADO.NET
Synchronize your data between SQL Server 2008 and SQL Server Compact 3.5 using the new Sync Framework. Keep all your occasionally-connected mobile workers on the same page.
Windows Communication Foundation (WCF) Store and Forward
Reliably push messages to servers or other devices via Exchange Server 2007. Programmatically notify peer devices that they have new orders waiting for them and need to sync.
MapPoint
Guide delivery drivers to their customers via either the shortest or quickest route. Integrated mapping means you’ll never get lost again.
LINQ
Use the new Language Integrated Query to filter results from Generic Object Collections. Query both your objects and XML using a familiar, SQL-like syntax to boost developer productivity.
Custom Controls
Capture signatures and dazzle your end-users with 3D and Alpha-blended controls that alter their behavior depending on the platform they’re running on.
Managed Stored Procedures and Triggers
The pluggable data layer allows you to say goodbye to compiling Dynamic SQL inside your code and fires events to react to INSERT, UPDATE, and DELETE operations.
Notifications and Online Help
Formerly only supported on Pro, say hello to Popup Notifications and Online Help on Standard. Popup Notifications, also known as “toast,” display an HTML message and then disappear after a pre-determined amount of time. Using Online Help on every screen reduces your application training costs.
Language Switching and Localization
Change Language/Regional Settings inside your app and watch text and Online Help speak a different language. Don’t wait until your application is finished to realize that it needs to be world-ready.
Time to Market
Stop reinventing the wheel and use this Accelerator as the foundation for your next Windows Mobile development effort. If you don’t want to use the whole thing, pick and choose the components that are the best fit for your project.

I am playing with it right now.. pretty cool stuff. Download Now

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Top 10 Hotels in Myrtle Beach, Best Hotels in NYC, Romantic Hotels in Seattle, Kid Friendly Hotels in Orlando and Top 10 Hotels in San Diego. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

MIX08 – Unified Web Platform in the Making

Rafik — Fri, 14 Mar 2008 06:59:00 GMT

Apart from MIX08 big announcements about Silverlight 2.0, Silverlight for mobile and IE 8.0, there were an interesting collection of announcements that if you put them together a picture of a unified API layer on top of Windows Live services will emerge. In my opinion, this is huge.

There are three main components in realizing this new unified platform:

Cloud Data Services

Live: Spaces Photos, Application Data
DB: SQL Server, SQL Server Data Service

Protocols

AtomPub
Atom

Programming Frameworks

The really really new announcement in MIX08 was the SQL Server Data Services (SSDS) which is in private beta right now. ADO.NET Data Services is the new name of the project Astoria that was announced for the first time back in MIX07. Microsoft Sync Framework was first unleashed back in November of 2007. What was so exciting though is the adaption of these technologies by Windows Live Services. With that you can reuse the same tools and techniques to access new services. Your focus won’t be on how to make calls to a certain services (Data Access) but on what to do with the data you get (Business Logic).

One might ask, will this go all the way to cover services like Live ID and Messenger, for example? The answer is NO. This unified model suites better the type of services that is solely data centric. In which, you main purpose is to access a store with basic operations of Get, PUT, POST and DELETE. Services like Live ID are operation centric thus the model does not apply.

I am pretty excited about the direction of the API and the adaption of the model by Windows Live Services. It won’t take long to see cool new apps built quickly and easily against the Live Platform. I want to write one myself!

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle, 5 Star Hotels in Miami, Hotels with in-room jacuzzi, Family Friendly hotels in Anaheim and Best Hotels in Fort Lauderdale. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

It is about time: Sync Services for Windows Mobile

Rafik — Mon, 10 Mar 2008 10:01:00 GMT

I remember the very beginnings when Sync Services came along and I started blogging that people asked to have the runtime works on mobile devices. That is a reasonable and expected request. After all, mobile devices are the true occasionally connected and sync services can make real difference. I am pleased to tell you that you that the initial CTP for the sync services for mobile is ready to download and try out:

Sync Services for ADO.NET for Devices supporting Windows Mobile

The runtime is almost identical to the desktop version with one exception. The mobile version does not support 2-tier model where the device can execute commands directly on the backend database server. While it is possible to have a mobile device VPN to the network thus be able to connect directly to the DB, it is does appear to be neither a common scenario nor the direction where the industry is heading. Having been at MIX08 this year, I can tell you that even device cradle scenarios are becoming the theme of the past.

You asked for it, here you have it. It is about time to give it a try and tell us what you think.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in Myrtle Beach, Best Hotels in Seattle, Top 10 Hotels in Miami, Kid friendly hotels in San Diego, Hotels with a kitchen or kitchenette and 5 Star Hotels in Las Vegas. Your feedback is welcome on twitter.com/raveable, raveable blog.

Unleash the power of Peer Sync Provider

Rafik — Mon, 25 Feb 2008 20:22:00 GMT

The term 'peer' or more commonly 'peer to peer' leads people to think about sharing and collaboration scenarios. That's good. One of the most common peer to peer applications that quickly jumps to mind is file sharing which became rather popular in the recent years and led to illegal sharing of copy righted material which resulted in large number of law suits that you are well aware of. That's the bad part. This kind of thought when comes to mind discourages people from exploring the technology and uncovering its strength. My feel is that you won't fall in this unfortunate trap and I am just plain wrong.

Let me tell you the main scenarios for which DbPeerSyncProvider is a key enabler:

Collaboration

Two people in a project could exchange information between each other’s without the need for a third node or, in other words a server node. In this kind of setup, data flows in multiple directions and loops are likely to occur between different members of the project. By that I mean, P1 sync changes with P2, P2 sync changes with P3, and P3 sync changes with P1. In collaboration scenarios multiple people are working on the same resources and synchronize their changes between each others. A server node is not required in this scenario but typically added for centralized backup and reporting needs.

Scale Out

When the load increases on your server, you would typically upgrade your server to more powerful one. Buy more RAM, change the processor, or even replace the machine altogether. These are different types of scaling up your system. But if you get a new machine and add it to the existing server such that the load is distributed on both, then that is scale out.

While DbServerSyncProvider scalability is limited to scale up techniques, DbPeerSyncProvider gives you scale out architecture that you might be looking for. In scale out scenarios, your client sync applications are not talking to each others, they are not synchronizing peer-to-peer, they only talking with the server. However, the server is not a single machine any more. It could be a server farm with some scheme of distributed load balancing. Changes on one server are synchronized with the other servers in peer to peer fashion using DbPeerSyncProvider. The client application too will have to use DbPeerSyncProvider despite that fact that it is only synchronizing data with the server.

This is a fascinating setup and not so obvious especially when all we talk about is peer-to-peer sync. Many variations of these two key scenarios are definitely possible. It is all up to your imagination.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in NYC, Best Hotels in Seattle, Top 10 Hotels in Myrtle Beach, Kid Friendly Hotels in Chicago, Hotels with in-room jacuzzi and Best Hotels in Miami. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

The MIX08 Factor

Rafik — Tue, 12 Feb 2008 20:55:00 GMT

I am not going to miss MIX this year, period.

Last year, MIX07 was a phenomenal event. Unfortunately, I did not get to attend. I kept following the announcements every day and watched some of the really good recorded sessions as they became available on the mix web site. I kept saying to myself, “Man, I should have been there”!

This year, it might be even better, a lot of folks are preparing demos, announcements ...etc. I cannot disclose anything I know coming. But I know so little, despite being at Microsoft. Yes, strange but true :)
Conferences tend to be rather confusing, too much activity that you cannot follow it all. For this reason, I like to have a plan. So, let me share my plan with you:

The keynotes this year are of particular interest. Ray is speaking this year and I am keen to listen to what he has to say. Steve keynotes are always good too.
Windows Live … what are the next cool things that are coming up
Sync Framework … I know few things are coming to those who love sync
FeedSync … I personally need to understand to what extend FeedSync is going to be used across Live properties, if any
Mobile development … what’s new in VS2008 and next coming up

It is sure to be fun. See you there.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle,Best Hotels in Miami, Best Hotels in San Francisco, Hotels with in-room jacuzzi and Best Hotels in Los Angeles. Your feedback is welcome on twitter.com/raveable, raveable blog.

SQL Server Change Tracking vs min_active_rowversion()

Rafik — Wed, 06 Feb 2008 23:02:00 GMT

In a bit old post I wrote about min_active_rowversion() as the best way to get the new anchor showing you that @@DBTS has its flaws and thus should be avoided. Min_active_rowversion() was introduced in SP2 for SQL Server 2005. If you have code that still uses @@DBTS, you are running the risk of missing changes from the server, do yourself a favor and change it now.

Despite the a fact that min_active_rowversion() addressed a major problem for sync developers, it did not solve it. Check out the diagram below:

T1 made a change to R1 row which received a timestamp of 50. The transaction did not commit and when you call min_active_rowversion() you will end up with the value of 50. That’s quite what we expect; sync should then synchronize up to 50-1 = 49 anchor. So what if we have few more transactions, like T2, that started after T1 and committed soon thereafter. If T1 did not commit, then min_active_rowversion() will remain at 50 until T1 commits. This could be a problem in many situations. You see, min_active_rowversion() did address the problem but did not solve it. Still much better that @@DBTS.

If we dig deeper into what the problem really is, we find that timestamp value is assigned at the statement execution time and not at transaction commit time. Had the row version be assigned at the transaction commit time, we would have a real solution for this problem. Fortunately, that’s exactly what SQL Server Change Tracking does for you in SQL Server 2008. Each transaction gets a version at the commit time and all the rows affected by the transaction will get that version. Check out the diagram below:

SQL Change Tracking will assign version at transaction commit time and all the rows affected by the transaction will have the same version. Now, when you try to sync before T1 or T2 starts, you should get 49, whereas after T2 commits, you will receive a value of 50. You see, now you are not stuck waiting for that long transaction to commits to sync changes that happened in subsequent transactions. Pretty cool, huh! …

SQL Server Change Tracking is a reliable way to track changes and designed from the ground up to address the needs of data synchronization applications!

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Opportunity Knocks

Rafik — Tue, 29 Jan 2008 02:23:00 GMT

This blog has been phenomenal. Through it I was introduced to so many of you whom I have corresponded with for building sync solutions. Now it is time for me to invite you to work with me in my new tenure at Microsoft. I recently joined Mobile Services team. This team is responsible for bringing windows live experiences to the windows mobile platform which includes Live Contacts, Live Messenger, Live Mail, Live Spaces, Live Search along with new interesting stuff that we are working on and I cannot tell you about just yet. You might ask where ‘sync’ in all of that!? Well, it is all over the place. In fact, mobile is sync play ground since mobile devices are the true occasionally connected systems. Mail, contacts … are synchronized with Live servers in the cloud and the same concepts that I’ve been advocating are materialized in a large scale system. But sync is just one enabler for great offline experience, there a lot of other skills that come into play, think about UI and user experience, seamless integration, performance, battery life … The new wave of innovation is coming in small form factors. It is really a great opportunity and I am very excited about it.

Interested ? Check out this list of openings below. Email your resume to me and let’s get the process started

Software Development Engineers (in no particular order)

· Position 1

· Position 2

· Position 3

· Position 4

· Position 5

· Position 6

· Position 7

· Position 8

· Position 9

Software Development Engineers in Test (in no particular order)

· Position 1

· Position 2

· Position 3

· Position 4

Lead Software Development Engineer in Test

· Position 1

The links above are likely to be unavailable as time goes by as we hire people and remove the openings from the careers database. Don’t be shy and apply now. The opportunity is knocking .. :)

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle,Kid Friendly Hotels in Miami, Best Hotels in San Francisco, Hotels with in-room jacuzzi and Best Hotels in Chicago. Your feedback is welcome on twitter.com/raveable,raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Change Tracking for the Masses

Rafik — Fri, 25 Jan 2008 03:36:00 GMT

I admit that writing your own change tracking logic for each table you want to synchronize is a little bit annoying and suboptimal experience at best. Not just that, adding triggers affects the performance of DML operation negatively. You also need to store information about deleted rows (tombstones) which we typically do in a separate table. At some point, you would need to clean up the oldest tombstones that have been sent to your existing sync clients since there is no point in keeping them anymore. All of this adds up to the complexity of your system.

While nobody claims that build sync application is any easy task, there is nothing stopping us from reducing the complexity especially for common operations like change tracking.

Introducing SQL Server Change Tracking

The good news is that better solutions are in the pipeline. In the latest CTP5 of SQL Server 2008 there is a cool new feature called SQL Server Change Tracking. This feature is all what you wish for from a sync-friendly store: A lightweight, reliable change tracking infrastructure! With SQL Server Change tracking, you can achieve the following:

Enable change tracking for a given table without making any schema changes to your data table. The tracking is happening deep inside the engine that is extremely fast and won’t degrade the performance of your DML compared to triggers.
Configure automatic cleanup of the tracking information
Track changes at column level
Enumerate changes starting from a given version. SQL Server Change Tracking record changes per transaction scope which works better than timestamp based approach.

Looks promising, right?! I personally believe that SQL Change Tracking is the best feature coming up in SQL Server for data sync and replication scenarios. But for full disclosure, I am biased toward anything sync :) there are loads of new features in SQL Server 2008.

You might be wondering about what it means to “record changes per transaction scope” and why is that better than using @@DBTS or min_active_rowversion(). Well, I owe you a detailed explanation; it just needs a separate post. Stay Tuned …

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle,Best Hotels in Miami, Best Hotels in San Francisco, Hotels with in-room jacuzzi and Best Hotels in Los Angeles. Your feedback is welcome on twitter.com/raveable, raveable blog. .

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Sharing Application Samples

Rafik — Tue, 15 Jan 2008 08:08:00 GMT

I originally authored the offline application line of samples to show off the different features of Sync Services. My main goal back then was to provide you with a quick start guide such that you can get into sync quickly. The series of demos has grown to up to eight demos, so as the rate of positive feedback that I’ve been receiving from you and even internally inside the company. Well, for that I thank you!

Now, with the release of our new DbPeerSyncProvider which is included in CTP1 of Microsoft Sync Framework, a new series of demos is also released with it. This time it is part of the SDK. On my machine, the samples are located at: C:\Program Files\Microsoft Sync Framework\v1.0\Runtime\ADO.NET\V2.0\Samples\

I called this new line of demos “Sharing Application” to emphasize the key feature of the peer provider which enables different nodes to share and collaborate on a common data set without the need for central authority. Not to say that having a central database is discouraged, it is just not required.

The SDK ships with five different flavors of the sharing application sample. They share the same user interface and act against the same database schema of orders and order_details tables. If you have tried the offline application demos, you will feel right at home with this new set of samples. I want to give you an overview of each sample and what features of the provider it shows off. But before going there, let’s check out the common user interface for these samples

The first thing to pay attention to in the main form as shown above is the three peers (peer1, peer2, peer3) settings in multiple places. On the top right you configure the machine name where the database is located for each peer (I have all three databases on one machine). On the bottom left, you direct the follow of synchronization through to and from boxes. The UI won’t allow you to select to sync from a given peer to itself. The three conflict generation buttons on the left of the Synchronize button are specific to conflict sample. Other samples might introduce new buttons.

In building multi-master sync application, three nodes are required to exercise almost all possible scenarios and that’s why you see three peers. You can extend the demo and add as many peers as you want; but as far as the demo goes, three peers should suffice.

In the setup directory with every sample there is a bunch of .sql files that you should use to create the peer databases along with all necessary stored procedures and triggers. The peer provider requires snapshot isolation transaction while enumerating changes, which means that you need any flavor of SQL Server 2005. SQL Server 2005 Express should work just fine.

So now you are eager to give it a try, the question is where to start. Let me give you a quick overview for each sample, ordered by complexity, starting with the simplest of course:

SharingAppDemo – Coupled Tracking
Following the most common model used in the offline application demos, here too you start with your base table and add necessary tracking columns to it. For recording deletes, you create a tombstone table for that.
SharingAppDemo – Decoupled Tracking
You soon realize, after trying the first demo, that tracking metadata for p2p sync is rather demanding and adds many columns to the base table that makes you uncomfortable. Well, I’ve gotten the same feeling myself and this demo shows you how to leave your base table schema intact and move all tracking metadata to a side table called “tracking table”. The rest of the demos are built on top of the decoupled tracking technique so you should give a little bit of time to this demo.
SharingAppDemo – Tombstone Cleanup
In the peer to peer sync world metadata cleanup is not taken lightly and the peer provider has a special logic for handling the cleanup for you. You just need to call the right method on the provider interface when you need to cleanup the metadata stored for deleted rows. This demo shows you how to do that.
SharingAppDemo - Column Mapping
In many situations the table schema is different on each node. Despite the diversion in columns names, columns data types are identical which make sync possible. If all nodes decide to agree on a global schema definition, they all could sync without problems. This demo shows you the idea of global schema. It does not use different schema on each node but it maps column name on each node to a global schema name and shows you how that works. If you don’t have this kind of scenario, you could skip this demo.
SharingAppDemo – Conflicts
It is always fun to show conflict detection and resolution logic. It is my favorite anyways. Once you get comfortable with the peer provider normal sync logic, your treat is to check out how conflicts are handled in peer to peer environment. This demo shows you different types of conflicts and different types of conflict resolution including the ability to retry applying the conflicting row in the next sync.

I am very much exited about this new sharing demo series. I think you going to like it too. As always, your comments are welcomed and you feedback is much appreciated.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in NYC, Best Hotels in Seattle, Top 10 Hotels in Myrtle Beach, Kid Friendly Hotels in Chicago, Hotels with in-room jacuzziand Best Hotels in Miami. Your feedback is welcome on twitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Hey You, Care to Share!?

Rafik — Sat, 05 Jan 2008 08:50:00 GMT

I spent years building sync solutions in Microsoft! I loved it! Sync and replication logic is so challenging and that is very appealing to me. In learning about sync algorithms, I’ve gone into several phases. First, I learned about how to debug customer reported issues and create fixes that shipped with windows services packs. After that, I went into building services and features around a sync engine trying to improve monitoring and debug-ability of a multi-master system. Finally, I directed my attention to the sync engine itself trying to ensure that it does not impose unnecessary assumptions on the developers of the layers above.

In my journey I found that the best way to better grasp sync logic is by getting the issues that bothers you out in the open. By that I mean: asking questions; talking to people; reviewing designs that others have written; building prototypes; collecting feedback and act on it; and learning from past mistakes. Simply put, sync is not easy that one person can fully comprehend. But two will get closer, three will round out most mistakes and four will do…Make no mistake about this, in developing Microsoft Sync Framework we have a selected group of very smart people getting together day in and day out discussing designs, features and issues for weeks. Different prospective add value and improve the product. You can do that too!

In the recent weeks I started collecting links to articles and projects that were authored by internal Microsoft staff or by external developers who just felt like sharing what they have learned. The resources page on my web site is growing fairly quickly. Just yesterday, I added two links for one sample code using sync services against Oracle backed shared by Microsoft Solution Architect Memi Lavi, and the other for a sample file provider that syncs file over the internet shared by Bryant Likes.

This is what I am talking about. Sharing will make us all learn. If you are experimenting with the sync framework and happen to build a neat sample or demo, then don’t just put it aside to catch the dust. Share it. Put it on your blog or on one of those code sharing sites like CodePlex or CodeProject, or even on a web share like SkyDrive and post a link to it along with a readme file to the forums. And please send me a link so that I can add it to the resources page.

So here you go. A New Year resolution for 2008: Share your sync experiments!

Happy New Year!

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See examples: Romantic Hotels in Myrtle Beach, Best Hotels in Seattle,Top 10 Hotels in Miami, Kid friendly hotels in San Diego, Hotels with a kitchen or kitchenette and 5 Star Hotels in Las Vegas. Your feedback is welcome ontwitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.

Year in Review: Social Networking, RIA Frameworks, and iPhone

Rafik — Fri, 28 Dec 2007 00:34:00 GMT

It is a rather slow week as most people are off on vacations. A quite time at the end of one year and beginning of another gives me an opportunity to pose a few questions and share my thoughts with you. One question is about the key highlights of this year; in other words, what were the most significant tech events of 2007? Well, I think the consumer space was the winner this year. It is interesting, few years back most of the advances were in the enterprise space but somehow it is getting saturated and the consumer technologies are the next big thing. That’s all good. In many ways technology is not really limited to a certain segment; it transfers in amazing ways. Recall instant messaging clients few years back, it was all for home users, now it is a productivity\collaboration tool in enterprises.

This year social networking was king. Facebook in particular was the cleverest player of all. If you think about it, social networking is a natural evolution of email, instant messaging, blogs and feeds put together. The new medium for people to communicate, share, and let information about themselves flow freely in the system where new level of experiences can be realized. The new development frameworks for social networking are a very significant development in this space. The potential is not fully realized yet. My feeling is that in 2008 we will witness amazing new experiences introduced in this space.

The Rich Internet Application or RIA is yet another significant development. Companies like Microsoft, Google and Adobe are working hard trying to win this market. Call me biased; but taking data offline was a big hit this year. In case you are not following, Microsoft Sync Framework, Adobe AIR, and Google Gear are new offerings that just came out this year almost at the same time. It appears that all players come to realization that offline or rather better sync technology is a key element for true RIA. I am convinced that RIA space will grow faster in the coming year.

I am maybe stating the obvious here, but mobile communication segment needed a real change. Powerful devices are still used mainly for voice and SMS. Realizing the full potential of the computing powers that we carry in our pockets needed a new device that is data centric rather than voice centric. iPhone did just that. It is benefiting all of us really. More devices will go the same route and mobile operating systems will change to get the best experience possible. The effects of iPhone will be realized in the coming year as the industry transforms.

Year 2007 was great, 2008 seems more interesting though. Have a great new year and I will see you next year.

Update: Just to let you know, I left Microsoft to start a new company, Raveable Hotel Reviews. See Examples: Romantic Hotels in Myrtle Beach, Top 10 Hotels in Seattle, 5 Star Hotels in Miami, Hotels with in-room jacuzzi, Family Friendly hotels in Anaheimand Best Hotels in Fort Lauderdale. Your feedback is welcome ontwitter.com/raveable, raveable blog.

I am not actively blogging about Sync Technologies. Please see Sync Team Blog for more updated content.