Microsoft Sync Framework : Microsoft Sync Framework

Looking Forward - Offline Capable Applications Using SilverLight and Sync Framework

liamca — Mon, 14 Dec 2009 17:47:00 GMT

Right from the first release of Microsoft Sync Framework, one of the most common requests we have received is for the ability to take SilverLight applications offline, meaning that users can continue to access their applications and data regardless of connectivity to the backend web application. Using this architecture, developers can build much richer applications and also take advantage of the local machine resources to provide fast data access. In addition, the ability to easily deploy SilverLight applications either within a browser of within an out-of-browser application makes this platform a very popular choice when building rich internet applications.

In the past there were a few issues that we had when trying to enable this functionality, but now with the latest release of SilverLight, much of what we require to provide offline capabilities is available. At this year’s Professional Developer Conference (PDC), Mark Scurrell introduced everyone to this set of capabilities in his session: "Using the Microsoft Sync Framework to Connect Apps to the Cloud" (overview starts at 44:40 minutes into session). Using the PDC conference session web site he showed how we built a rich SilverLight application that allows a user to take the sessions data offline into isolated storage, and while the user is offline they can continue to view sessions, add sessions to their calendar, rate attended sessions and then, once back online, the application automatically takes the changes and uploads them to the main agenda web site (demo starts at 48:00 minutes into session). The screenshot to the left shows an example of what this application looks like.

In order to truly take advantage of SilverLight's cross platform capability we had to come up with a solution where we could still support clients that do not have a Sync Framework runtime. To do this what we did was build the architecture such that all of the core Sync Framework logic resides on the service (in this case a Windows Azure Webrole), and that is where all of the heavy lifting of the synchronization process is executed. From the service we exposed a simple HTTP-based synchronization protocol that the client consumed, allowing any client to participate in synchronization. Currently the best offline store within SilverLight is isolated storage, and as you can see in Mark's demonstration of filtering data, this is sufficient to build a rich and very quick data-access application.

Over the next year you will see more of how we plan to enable this scenario and we will provide additional details on what you can do to get started to build these offline-capable applications using SilverLight.

Liam

Synchronizing Files to Windows Azure Storage Using Microsoft Sync Framework

liamca — Thu, 03 Dec 2009 21:48:00 GMT

I wanted to kick off this post by introducing myself. I’m Michael Clark. I lead the Sync Framework team. I don’t normally post to this blog (Liam Cavanagh does a fantastic job of that) but wanted a forum to get the information below out for now and decided that blog would be a good venue. One of the things that we’d like to know is if the community would like to see more posts of this nature. If so let us know by giving us some feedback and we’ll figure out some appropriate venue to get this kind of info out on a more regular basis.

At PDC09 we talked quite a bit about how to synchronize data to the cloud. Most of this revolved around synchronizing structured data to and from SQL Azure. If you are interested in that, check out our Developer Quick Start, which includes a link to download Microsoft Sync Framework Power Pack for SQL Azure November CTP.

For this post, however, I want to augment that information to answer another question that we are frequently asked and were asked a number of times at PDC, “How can I synchronize things like files with Azure Blob Storage?” The answer at this point is that you’ll have to build a provider. The good news is that it’s not too hard. I built one that I’ll describe as part of this post. The full sample code is available here.

So how does it work? The sample itself consists of three major portions: the actual provider, some wrapper code on Azure Blob Storage, and a simple console application to run it. I’ll talk in a bit more depth about both the provider and the Azure Blob Storage integration. On the client side the sample uses Sync Framework’s file synchronization provider. The file synchronization provider solves a lot of the hard problems for synchronizing files, including moves, renames, etc., so it is a great way to get this up and going quickly.

The azure provider is implemented as a FullEnumerationSimpleSyncProvider, using the simple provider components of Sync Framework. Simple providers are a way to create Sync Framework providers for data stores that do not have built-in synchronization support.

The provider itself derives from the FullEnumerationSimpleSyncProvider class, which is used for stores that don’t support any form of change detection whatsoever. It’s extremely useful because this is actually the category that most off-the-shelf stores fall into (another great example of this type of store is the FAT file system). Sync Framework also contains the notion of an AnchorEnumerationSimpleSyncProvider for stores that have the ability to enumerate changes based on some type of anchor (timestamp, tick count, opaque blob of goo, whatever). But for now I’m going to focus on full enumeration as that is what is required for Azure Blob Storage at this point.

The basic idea behind a full enumeration synchronization provider is that you need to tell Sync Framework some basic information about the items you'll be synchronizing, how to identify them and how to detect a version change, and then give Sync Framework the ability to enumerate through the items looking for changes. To tell Sync Framework about the items, override the MetadataSchema property of the FullEnumerationSimpleSyncProvider class. When you build the metadata schema you’ll specify a set of custom fields to track, and an IdentityRule. Together these things make up the set of data required to track and identify changes for objects in the store. For the Azure Blob synchronization provider this property looks like this:

public override ItemMetadataSchema MetadataSchema

{

get

{

CustomFieldDefinition[] customFields = new CustomFieldDefinition[2];

customFields[0] = new CustomFieldDefinition(ItemFields.CUSTOM_FIELD_NAME, typeof(string), AzureBlobStore.MaxFileNameLength);

customFields[1] = new CustomFieldDefinition(ItemFields.CUSTOM_FIELD_TIMESTAMP, typeof(ulong));

IdentityRule[] identityRule = new IdentityRule[1];

identityRule[0] = new IdentityRule(new uint[] { ItemFields.CUSTOM_FIELD_NAME });

return new ItemMetadataSchema(customFields, identityRule);

}

Next, in order to let Sync Framework detect changes you need to override the EnumerateItems method. In your EnumerateItems implementation you’ll create a List of ItemFieldDictionary objects to tell Sync Framework about all of the metadata properties you have specified in your MetadataSchema. Sync Framework uses this information to track the state of objects in the store, looking for adds, updates, and deletes and then produces the proper synchronization metadata for those changes so that they can be synchronized with any Sync Framework provider. The implementation for EnumerateItems in this sample looks like this:

// Enumerate all items in the store

public override void EnumerateItems(FullEnumerationContext context)

{

List<ItemFieldDictionary> items = DataStore.ListBlobs();

context.ReportItems(items);

}

There is obviously some hidden detail here because I’ve put some of the work in the store wrapper that I mentioned previously. I’ll talk more about the store wrapper in a moment but I’ll include its ListBlobs method here since it is relevant.

internal List<ItemFieldDictionary> ListBlobs()

{

List<ItemFieldDictionary> items = new List<ItemFieldDictionary>();

BlobRequestOptions opts = new BlobRequestOptions();

opts.UseFlatBlobListing = true;

opts.BlobListingDetails = BlobListingDetails.Metadata;

foreach (IListBlobItem o in Container.ListBlobs(opts))

{

CloudBlob blob = Container.GetBlobReference(o.Uri.ToString());

ItemFieldDictionary dict = new ItemFieldDictionary();

dict.Add(new ItemField(ItemFields.CUSTOM_FIELD_NAME, typeof(string), o.Uri.ToString()));

dict.Add(new ItemField(ItemFields.CUSTOM_FIELD_TIMESTAMP, typeof(ulong), (ulong)blob.Properties.LastModifiedUtc.ToBinary()));

items.Add(dict);

}

return items;

}

What this ListBlobs implementation does is simply walk through all of the blobs in the container (container is an Azure Blob Storage concept; in this sample I treat it like a root directory). Then, for each blob get out the information that was specified for the MetadataSchema and build a list of that information to give back to Sync Framework.

The last crucial bit for the providers is to give Sync Framework a way to add, update, or delete items from the store. To do this, override InsertItem, UpdateItem, and DeleteItem respectively. I won’t include the source for those methods here for brevity but you can check them out in the sample.

There are a couple of other things that are needed for the synchronization provider but those are mostly bookkeeping and I’ll let you look at the sample to get the details.

At this point, I want to talk briefly about the store wrapper class. The store wrapper class utilizes objects in the Microsoft.WindowsAzure.StorageClient namespace. To get that you’ll need to download Windows Azure Tools for Microsoft Visual Studio. I’m not going to go into too much detail about the store wrapper itself. It has most of the methods you’d expect, such as the ListBlobs method seen above, and the corresponding methods for adds, updates, and deletes. I do want to talk a little bit about one important detail with the store and that is optimistic concurrency. Optimistic concurrency is the thing that will allow multiple synchronization clients to work with the store at the same time without overwriting each other unwittingly and corrupting data. The Sync Framework simple providers are designed to work well with stores that support optimistic concurrency so that they can provide correct synchronization.

The great news is that Windows Azure Blob Storage supports optimistic concurrency well. If you are using the StorageClient API, it does this by using a BlobRequestOptions object. You can see an example of how this works in the DeleteFile method of the store wrapper:

internal void DeleteFile(

string name,

DateTime expectedLastModified

)

{

CloudBlob blob = Container.GetBlobReference(name);

try

{

blob.FetchAttributes();

}

catch (StorageClientException e)

{

// Someone may have deleted the blob in the meantime

if (e.ErrorCode == StorageErrorCode.BlobNotFound)

{

throw new ApplicationException("Concurrency Violation", e);

}

throw;

}

BlobProperties blobProperties = blob.Properties;

...

BlobRequestOptions opts = new BlobRequestOptions();

opts.AccessCondition = AccessCondition.IfNotModifiedSince(expectedLastModified);

try

{

blob.Delete(opts);

}

catch( StorageClientException e )

{

// Someone must have modified the file in the meantime

if (e.ErrorCode == StorageErrorCode.BlobNotFound || e.ErrorCode == StorageErrorCode.ConditionFailed)

{

throw new ApplicationException("Concurrency Violation", e);

}

throw;

}

Note that by specifying the AccessCondition property of the BlobRequestOptions object, the code tells Azure Blob Storage not to touch the file if the file has been modified since the last time we looked at it. If the file has been touched, the CloudBlog object from the StorageClient library throws StorageClientException. For a couple of specific errors, the store wrapper converts that into ApplicationException to let other parts of the code know that they should treat this as a temporary error and temporarily skip it from the perspective of synchronization. That code is in the DeteleItem method of the provider and looks like this:

try

{

DataStore.DeleteFile(name, expectedLastUpdate);

}

catch (ApplicationException e)

{

recoverableErrorReportingContext.RecordRecoverableErrorForChange(new RecoverableErrorData(e));

}

What this does is cause Sync Framework to temporarily exclude that particular item from synchronization. The item will be picked up again later.

Now, to run this sample you’ll need to download the Microsoft Sync Framework v2 SDK as well as the Windows Azure Tools for Microsoft Visual Studio. The sample synchronizes to a Windows Azure Blob Storage project so you’ll need to make sure that you have an account and project set up. To get started on that, go to http://www.microsoft.com/windowsazure/. Finally, when running the sample you’ll need to modify the app.config file for the sample and specify the AccountName and AccountSharedKey properties for the storage project that you created. When you actually run it from the command line, the application expects to be given a container name (this is an Azure Blob Storage concept mentioned previously) and a local path. It will synchronize everything from the local path, including subdirectories, up to the container specified.

So that’s basically it. Please give this a try. If you do this with photos you can easily view them in your web browser by going to https://<accountName>.blobs.core.windows.net/<containerName>/<fileName>.jpg. Check out the code, and let us know what you think by sending mail to syncfdbk@microsoft.com

One final note that is worth covering: A great question to ask here is, “Why not just use Live Mesh to synchronize the files?” The answer to that is that if Live Mesh fits your scenario and requirements, then you absolutely should use it. Live Mesh, part of the Windows Live Platform, is a great product and will allow you to synchronize files between any set of PCs and mobile devices. It is perfect for a lot of synchronization scenarios involving files. Most of the customers that have asked us about how to accomplish this with Sync Framework need something special. For instance, they are creating an end-to-end application and want explicit control over everything (instead of leaving that up to the end user) including where the files are synchronized, etc. Other examples are customers who specifically want the files in Azure Blob Storage so that they can use them with their Azure Web Application. The bottom line is that if Live Mesh meets your needs, then great. If not, Sync Framework is a perfect alternative for meeting the synchronization needs of your application.

So there it is. This is a simple example of how to synchronize files with Azure Blob Storage. You can take this a lot further by for instance hosting in a Windows Azure Web Role and storing the item metadata directly in the file metadata. But using the method described above is fast to get up and running and performs well for a number of scenarios. Let us know if you have any questions or comments.

The full sample code is available here.

Mike

Synchronizing a Remote Replica By Using a Proxy Provider

LarenC — Mon, 30 Nov 2009 21:01:00 GMT

In a typical synchronization scenario, the two data stores are on different computers or devices that are connected over a network. A controlling application runs on a third computer or on the same computer as one of the data stores. A common way to perform synchronization is for the synchronization providers to run on the computer local to the controlling application. In a typical implementation, each provider communicates directly with the remote data store to send and retrieve data in a piecemeal fashion. This approach typically means lots of local-remote round trips to retrieve items and to apply changes, and relies on the local computer to perform the bulk of the synchronization processing. There's a better way! A more efficient method is to use a proxy provider on the local computer. The proxy provider sends metadata and data to a provider that runs on the remote computer, allowing the bulk of synchronization processing to be distributed to the remote computer. When change data caching is added, only one round trip is needed to process each change batch.

I've written an article that shows you how to implement a proxy provider that uses Windows Communication Foundation (WCF) to send data and commands over the Internet to a remote provider running on a Web server. The pattern used in this article can be easily adapted to work with any protocol over any network.

The article can be read here: Synchronizing a Remote Replica By Using a Proxy Provider.

The article draws heavily on the Sync101 with Remote Change Application over WCF sample on the MSDN Code Gallery. If you'd rather skip the reading and just look at the code, click here: Sync101 with Remote Change Application over WCF.

PDC Keynote Recording - Kelley Blue Book's use of SQL Azure Data Sync

liamca — Wed, 25 Nov 2009 19:54:00 GMT

Last week at PDC we announced the availability of our first CTP of SQL Azure Data Sync. In the keynote, Andy Lapin of Kelley Blue Book explained how they keep their community databases up-to-date using this technolgy to synchronize data between their on-premises SQL Server and SQL Azure. The recording is now available for you to view (the Kelley Blue Book portion of the keynote starts at about 1:18:00).

Ultimately, one of the goals of this CTP is to get your feedback. What is it that you like or dislike about it? Are you primarily interested in the Visual Studio integration or would you like us to see us add additional capabilities such as advanced monitoring or logging to the SQL Server provisioning tool. So please send us your thoughts (good or bad) here:

http://social.microsoft.com/Forums/en-US/syncfeedback/threads

Liam

SQL Pass Conference & Synchronization to SQL Azure

liamca — Wed, 04 Nov 2009 17:34:00 GMT

Yesterday was the first day of the SQL Server conference, SQL Pass, here in Seattle. It was a really exciting day for me because I had the opportunity to present our new synchronization capabilities to the cloud in the opening day keynote and in a subsequent session.

We introduced the first piece of Project “Huron”, that is a technology based on the Sync Framework enabling people to use Windows Azure as a central Data Hub for all information. To accomplish this, later this month we will make available for download an early preview of a tool that provides the ability to extend a SQL Server on-premises databases to the cloud. Once configured users can then easily extend data from the cloud to mobile users and remote offices.

Using this tool and with the help of one of our partners, Archetype, we showed how companies can easily extend their on-premises SQL Servers to SQL Azure using data synchronization, allowing the data stores to co-exist and interoperate seamlessly.

All of this can be setup using a wizard. Then from an automatically generated SQL Agent, the synchronization process is executed periodically to move the incremental changes between SQL Server and SQL Azure.

Here are a few screenshots of the tool:

Watch for more details over this month at the Professional Developer Conference and in this blog for other new capabilities we are providing to for integration to Windows Azure and SQL Azure.

Liam Cavanagh

New Sync Framework Samples

LarenC — Mon, 26 Oct 2009 22:11:00 GMT

We have lots of new samples on Sync Framework Code Gallery. These samples demonstrate some of the most important and exciting features of Sync Framework 2.0. If you are new to Sync Framework or are looking for ways to write code to solve your synchronization problems, chances are we have a sample that can get you started. Here are some of the highlights:

Sync101 Samples

This set of samples illustrates various concepts and features needed to synchronize a custom data store, such as:

Sync101 with Simple Providers, which shows how to write a basic synchronization application that uses simple custom providers.
Sync101 with Constraint Conflicts, which shows how to write a standard custom provider that uses Windows Communication Foundation (WCF) to synchronize data between replicas on different computers.
Sync101 with Custom Filtering, which shows how to track custom filters and how to use a standard custom provider to send changes from a filter-tracking replica to two different filtered replicas.

Database Synchronization Provider Samples

This set of samples illustrates various concepts around using database synchronization providers to synchronize between SQL Server databases and SQL Server Compact databases, such as:

Database Sync: SQL Server and SQL Server Compact, which shows how to use database synchronization providers to configure and execute peer-to-peer synchronization between a SQL Server database and one or more SQL Server Compact databases.

File Synchronization Provider Sample

This sample shows how to use the file synchronization provider.

File Synchronization Provider Sample, which shows how to use the file synchronization provider, a fully functioning provider that helps an application to synchronize files and folders in NTFS, FAT, and SMB file systems.

SharePoint 2010, Windows 7 and PwC Now Integrate Microsoft Sync Framework v2.0

liamca — Mon, 19 Oct 2009 18:17:00 GMT

As we announced earlier today, Microsoft Sync Framework V2.0 is now available for public download.

Along with this release I wanted to outline a number of companies and partners that are already using Sync Framework v2.0 including Windows 7, Microsoft Office 2010 and PricewaterhouseCoopers (PwC).

SharePoint has also chosen to integrate Microsoft Sync Framework as part of their solution for synchronizing to Office SharePoint 2010. The SharePoint and Office teams will use the Sync Framework as the engine powering synchronization between Office Workspace (Groove) and SharePoint. In addition, the SharePoint server will also expose Sync Framework API’s that will allow any developer to build providers to integrate new data stores into SharePoint.

"The Sync Framework will greatly reduce the amount of code that an ISV needs to create, and this represents probably the most painful code that they otherwise would have had to write".
Maxim Lukiyanov, Program Manager SharePoint Server, Microsoft

For more information on this please see our new case study: Microsoft Sync Framework used to Create Microsoft SharePoint Workspace 2010

The Windows 7 Device Stage gives users a common user interface to manage and synchronize data with their Windows 7 enabled phone. Contacts, calendar, tasks and even notes can be synchronized with any data store. Sync Framework v2.0 is the easiest way for developers to implement data synchronization providers for any store to enable it for the Windows 7 Device synchronization experience.

"The flexibility of Sync Framework and its ability to track and exchange data really set this technology apart. Since Sync Framework is data agnostic this allowed us to develop a framework tailored around contact, calendar, tasks and notes data".
Anna Boyd, Windows 7 Engineering, Microsoft

PricewaterhouseCoopers has put a system in deployment for 60,000+ mobile auditors in 150 countries to allow much of their work to be done remotely at the client site, making disconnected, offline use of their application and friction free collaboration across teams.

"Our implementation of Sync Framework ‘just worked’ for our auditors. If they set automatic synchronization, our software finds peers, verifies access rights to the same engagement database, and updates all databases. They might be connected to each other over a client’s network, or just via ad hoc (no access point) WiFi. If just one laptop has access to the servers, that laptop automatically funnels all changes from the laptops to the server and vice versa. No manual configuration is needed".
Dr. Glenn Ricart, Technology and Development Leader, PricewaterhouseCoopers

Liam Cavanagh

Sync Framework 2.0 Available for Download

LarenC — Mon, 19 Oct 2009 15:27:00 GMT

I am happy to announce the availability of Sync Framework 2.0. Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

Adds features that cater to new scenarios or scenarios that were difficult to support.
Reduces the amount of work required to develop providers.
Supports more data sources with new built-in providers.

The major new features and improvements included in Sync Framework 2.0 are:

In Core Components:

Simple Providers: Reduce the amount of work required to develop providers, especially when the data source has very few synchronization-related capabilities, such as change-tracking. The goal is to enable a developer to write a provider without having to become a synchronization expert. The majority of the code that is required for a simple provider is limited to that responsible for interacting with the data source. Simple providers support many complex synchronization scenarios, such as filtering, concurrency and constraint conflict handling, and anchor-based enumeration.
Flexible Filtering: Change unit filters and custom filters have been added to the custom provider components. These filters, in addition to the item filters from Sync Framework 1.0, enable you to filter your data in whatever way is most useful. Filter negotiation allows the source and destination providers to negotiate the filter used during the synchronization session. Change unit filters enable a provider to specify that, while the replica it serves stores all items in the synchronization scope, the replica stores a reduced representation of each item. For example, when an item is a contact a device might store only the name and phone number of the contact as compared to the complete contact stored on a PC. Custom filters enable you to define a filter in whatever way is most appropriate, including filters that allow an item to move into or out of the filter as the item data changes. For example, a media storage device stores only songs that are rated as three stars or better. When the rating on a song changes from four stars to two stars, the song moves out of the filter. Filter negotiation allows a destination provider to specify the filter to be used by the source provider during change enumeration; the source provider can accept or reject a filter. If a source provider does not support the requested filter then the destination provider can choose to receive all of the data and do the filtering itself.
Improved Conflict Handling: Additional conflict scenarios are supported, such as reporting constraint conflicts, managing logged conflicts, and resolving conflicts by using a last-writer-wins policy. Constraint conflicts are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. Sync Framework includes components to help resolve constraint conflicts and apply the resolution to the destination replica. Conflict logging is enhanced by Sync Framework components that help manage the log, such as by removing obsolete conflicts. A last-writer-wins conflict resolution policy can be used to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this policy by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change.
Data Conversion between Providers: In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
Change Application Service: Increases the flexibility and usefulness of the change applier component of Sync Framework. By performing the same actions as the standard change applier, but in a more granular way, the change application service gives a destination provider the ability to use only the features that it requires.
Tracing: Enables you to trace the execution of several components, which is useful during application debugging. This download provides the .tmf files that can be used to view a trace, flush the logger, and format the binary trace file. For more information about how to trace Sync Framework execution, see “Tracing Sync Framework Components” in the documentation that installs with the Sync Framework SDK.

In Database Providers:

New Database Providers (SQL Server and SQL Server Compact): Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore.
Robust Memory-Based Batching: Previous versions of Sync Framework and Sync Services for ADO.NET provided a way for developers to define their own batching logic but there were a lot of limitations, including significant complexity, excessive chattiness, out of memory issues, and restrictions on usage. Sync Framework 2.0 addresses all of these issues by providing a more complete and robust batching API. Developers no longer have to write batching logic themselves because Sync Framework divides changes into batches based on several properties in the API. Batches are now defined by memory consumption rather than the number of rows synchronized, which has eliminated out-of-memory issues for most common scenarios.
Provisioning and Management APIs: Provisioning and initialization activities that were previously exposed only through Visual Studio tooling have now been added to the database provider APIs. This includes the ability to provision an existing database by adding the change-tracking tables and triggers that are required by Sync Framework. It also includes the ability to start with an empty database, create the user schema, and provision that schema based on another server or client database that has already been provisioned.
Performance Improvements: The new database providers in this release have been thoroughly tested in large-scale scenarios in which a single server supports thousands of clients with hundreds of concurrent synchronization operations. This testing resulted in a large number of internal performance improvements that enable Sync Framework database providers to perform as well as other Microsoft technologies like Remote Data Access (RDA) while offering a wide range of capabilities that compete with end-to-end solutions like merge replication.

In File Synchronization Provider:

· Data Conversion between Providers: The data transfer interface used by the file synchronization provider is now available in managed code. A custom provider that synchronizes some other data type can use the data transfer interface to convert its data to synchronize with a file synchronization provider.

· Better File Transfer Performance: Improvements to the way file data is copied allow file data to be transferred up to 30% faster than in Sync Framework 1.0.

· More Robust Synchronization: Instead of failing the entire synchronization session when a single file cannot be synchronized (including failures caused by network issues), the failure is flagged and the synchronization session continues.

To learn more about Sync Framework or to download Sync Framework 2.0, please visit the Sync Framework Developer Center.

Sync Framework and our upcoming conferences

liamca — Mon, 05 Oct 2009 19:31:00 GMT

The next few months are going to be very busy for us here in the Sync Framework team. Not only do we have Sync Framework v2 coming out shortly, we also have a number of conferences where we will talk about the work we have been doing with some of external companies and internal groups here at Microsoft to allow developers to plug-in synchronization to systems like SQL Server and SQL Azure.

Starting in November, we will be at SQL PASS, and then at the PDC for the launch of SQL Azure. Last week I recorded a Channel 9 video along with Buck Woody and Michael Rys on the upcoming SQL PASS Summit. You can view the video here:

http://channel9.msdn.com/posts/LarryLarsen/Upcoming-SQL-PASS-Summit-1/

Here is a summary of the video:

Join thousands of SQL Server & BI professionals at the Washington State Convention & Trade Center in Seattle from Nov. 2-5 for the largest SQL Server conference in the world! PASS Summit offers 168 technical sessions presented by SQL Server experts, one-on-one time with Microsoft’s SQL Server engineering team, free troubleshooting & design architecture guidance from Microsoft consultants, keynotes from Microsoft’s senior executives and much more!

Senior Data Platform Technology Specialist Buck Woody stopped by the Channel 9 studio to talk to Michael Rys, Principal Program Manager in SQL Server Engineering Team, and Liam Cavanagh, Senior Program Manager, about upcoming sessions you won’t want to miss this year’s PASS Summit conference.

Liam Cavanagh

Announcing Sync Framework 2.0 CTP2

liamca — Fri, 05 Jun 2009 01:49:00 GMT

I am extremely happy to announce the availability of Sync Framework 2.0 CTP2 . Sync Framework 2.0 expands on the capabilities offered by Sync Framework 1.0:

Adds features that cater to new scenarios or scenarios that were difficult to support.
Reduces the amount of work required to develop providers.
Supports more data sources with new built-in providers.

The major new features included in this CTP are:

New database providers (SqlSyncProvider and SqlCeSyncProvider)
Enable hub-and-spoke and peer-to-peer synchronization for SQL Server, SQL Server Express, and SQL Server Compact. Sync Framework automatically creates all of the commands that are required to communicate with each database. You do not have to write synchronization queries as you do with other providers. The providers support: flexible initialization options; batching of changes based on data size; and maintenance tasks, such as metadata cleanup and server database restore.
Custom filters
Enable a filtered replica to store item data only for items that are in the filter, such as a media storage replica that stores only songs that are rated as three stars or better. As the rating on a song changes, an item might move in or out of the filter. A filter-tracking replica is another replica in the community that can identify which items are in the filter and which have moved in or out of the filter recently. A filter-tracking replica may store item data for items that are not in the filter. Sync Framework supports efficient synchronization between filtered replicas and filter-tracking replicas, and maintains all of the necessary metadata.
Improved conflict handling
Sync Framework added several features in this release that make it easier to handle conflicts.

Constraint conflict handling: Sync Framework now supports constraint conflicts, which are conflicts that violate constraints that are put on items or change units, such as the relationship of folders or the location of identically named data within a file system. A destination provider can report a constraint conflict to Sync Framework, and use Sync Framework components to resolve the conflict and determine how to apply the resolution to the destination replica.
Conflict logging: Sync Framework defines a set of conflict log interfaces and provides components that help manage conflicts in the log. Sync Framework also provides an in-memory implementation of the conflict log interfaces for replicas that do not contain a conflict log. It can be useful to save conflicts in a log so that they can be processed separately from the synchronization session, such as when a user needs to review conflicts and decide how to resolve them.
Last-writer-wins conflict resolution: An application might want to resolve concurrency conflicts (in which an item is updated by more than one replica) by keeping the most recently made change, regardless of where the change was made. Sync Framework supports this resolution option by enabling an application to retrieve the time a change was made on the source and destination replicas. The application can then compare the two times and apply the last change.
Improved simple provider conflict handling: Simple providers include a new set of conflict handling interfaces that make it easier to specify resolution actions for concurrency and constraint conflicts.

Data conversion between providers
In some scenarios, synchronization providers synchronize the same type of data (such as sales data), but the data format that each provider requires is different. To address this scenario, Sync Framework enables you to implement interfaces that convert data to the format that each provider requires. In addition to data conversion APIs that can be used for any type of custom provider, Sync Framework also includes conversion APIs specifically for the file synchronization provider.
Change application service
Sync Framework includes a change applier implementation that most applications use to apply changes to a replica. In this CTP, Sync Framework introduces the change application service, which performs the same actions as the change applier, but in a more granular way. A destination provider that requires greater flexibility than the standard change applier can use the change application service to perform only the set of actions that are required.

For more details please visit the Microsoft download center at:
http://www.microsoft.com/downloads/details.aspx?FamilyID=89adbb1e-53ff-41b5-ba17-8e43a2e66254&displaylang=en

Nina Hu

TechEd 2009 - Building Applications with Microsoft SQL Data Services and Windows Azure (DAT316)

liamca — Tue, 05 May 2009 03:06:00 GMT

Hopefully a number of you will be attending this years TechEd in Los Angeles. This year I will be co-presenting with David Robinson where we will be discussing SQL Data Services and our "Huron" Data Hub project. I am really looking forward to showing you our latest demo so I hope you can all make it. If you are not able to make the session but are at the conference, I will also be at the SQL Data Services booth for quite a bit of the time, so please stop by so we can talk "sync".

Here is the abstract for the session:

"Are you looking to reduce the costs of building and maintaining enterprise applications? Do you want to extend the reach of your applications across multiple devices, locations and partners? SQL Data Services and Windows Azure provides you a friction free, highly scalable platform for building applications. The scale and reach of the cloud lights up a new class of application scenarios. Come see how easy it is to consume SQL Data Services from within Windows Azure. In addition, we dive into Microsoft's new "Data Hub" for businesses and see how this SQL Data Services powered synchronization service allows for data aggregation within the Hub to provide straightforward data sharing between on-premises databases, business partners, remote offices, and mobile users".

See you there!

Liam Cavanagh

New Webcast Video - Drill Down Into the Next Release of the Microsoft Sync Framework

liamca — Sat, 02 May 2009 03:14:00 GMT

We recently had a chance to record an MSDN webcast describing a handful of the new features that will ship in the next CTP of Sync Framework v2. This particular session focuses primarily on those features we have built related to our offline and collaboration database scenarios. Please register at the following location:

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032413045&CountryCode=US

This session is a deep dive into the next release of the Microsoft Sync Framework with a focus on our database scenarios. Specifically, this session will discuss some of the work we have done in order to allow developers to synchronize SQL Compact in a peer to peer fashion. In addition, we will discuss the improvements we have made around batching to make it easier to use, expose more control over memory usage and provide resumability over transient networks that go up and down when performing sync operations. Furthermore, we will examine some of the performance related features baked into the runtime to improve the time it takes to initialize your client side database and synchronize large tables that have a less frequent rate of change. This will also include a discussion around some of the lessons we learned when performing scale and performance testing in an environment with 4000 clients and as much as 400 concurrent sync operations on a single server. These best practices were compiled at the end of one of the largest external implementations of the Microsoft Sync Framework in the world. Finally, we will go over some of the new scenarios that we have enabled as a result of building on top of the Sync Framework, which provides much more flexibility around the shape of your topology. These scenarios include taking data stored in SQL Data Services and ADO.NET Data Services offline, USB shuttle scenarios, and traditional scenarios such as branch-office. This session will include demos that provide a deep dive into the use of these features as well as benchmark data around the performance of the Sync Framework in a large distributed environment.

Enjoy and please send any and all feedback our way!

Sean Kelley

Project "Huron" - Early Adopter Program

liamca — Wed, 29 Apr 2009 19:27:00 GMT

I wanted to give an update on our Project "Huron". For those new to it, this is an incubation project that we have been working on for some time to provide database "friction free" (this is a one fo those Microsoft terms that basically means simple to setup :-)) synchronization capabilities to the Cloud (SQL Data Service) via a hosted Sync Service sitting within the Azure data centers. The goal is to remove the typical complexities (configuration, scalability, security, etc) involved with sharing database information between local databases such as SQL Server and SQL Compact and provide simple UI tools for configuration and sync components developers can embed in existing applications.

One unfortunate scoping that we have decided to make is to temporarily hold off on direct support of Microsoft Access for v1. Although we realize that there are a LOT of customers that are looking to support this, based on our current resourcing this has been an unfortunate scoping that we have decided to take. The primarily jusitification was that we already had an existing SQL Server and SQL Compact provider from Sync Framework that we are able to easily re-use which greatly simplified our efforts. It is our hope that some of these Access customers will consider initially linking their Access forms to a SQL Server database to then synchronize via the cloud in the short term until we can provide the full Access support.

The good news is that in the short term we are still planning to offer direct support SQL Server and SQL Compact (desktop and devices). In fact we are very close to the point that we can start taking on some early adopter to verify some of our assumptioms and help us solidify the first release. What we are looking for are any customers that are looking to share SQL Server or SQL Compact databases via the cloud and have an existing project that would warrant this functionality. If you are interested please contact us using the "Email" link in the left sidebar and I can provide additional details.

With that I will leave you with a few screen mock-ups we are working with from our Huron Management Studio Dev UI.

Liam

SyncToy 2.0 - 1 Millionth Download

liamca — Fri, 17 Apr 2009 18:56:00 GMT

I am excited to annouce that SyncToy 2.0 has reached a major milestone. During the month of March it passed 1 Million downloads! This is exciting both for SyncToy as well as the Microsoft Sync Framework team. Since SyncToy bases synchornization upon the Sync Framework's File Sync Provider, it is always a good test of the technology to have users of this scale. In our forums we regularaly see people use it to keep 10's of thousands of files and folders in sync across desktops and devices. It is also a great test and starting point as we look to expand file synchronization into cloud based services.

Liam Cavanagh

Custom Conflict Resolution Survey

liamca — Wed, 08 Apr 2009 01:27:00 GMT

We are considering making some changes to the patterns currently leveraged to apply custom conflict resolution policies and business logic. During our design discussions a handful of questions came up, so we thought it would be best to post them on our blog in an effort to collect feedback. Please provide your feedback in the comments attached to this post.

Our questions are as follows:

Can you give a handful of examples when you have had to apply some sort of custom conflict resolution policy and/or your own business logic when syncing changes? Specifically, we are interested in examples that require data to be changed in the middle of synchronization either before/after getting changes or before/after applying changes.
Do you typically leverage the Sync Framework or Sync Services events in order to do so? If so, which events do you most commonly use? If not, can you describe alternate approaches you have taken?
Do you leverage SyncContext to make changes to the DataSet being sent/received or some other approach?

Thank you for your time!

Regards,

Sean Kelley
Program Manager
Microsoft Sync Framework