-or-

Sync’ing OData to Local Storage in Windows Phone (Part 3)

If you have been following this series, you know that I have been demonstrating a way to persist data from an OData feed to a local Windows Phone (“Mango”) device. Just to refresh, here are the benefits of maintaining a local cache of data:

  1. Reduced amount of network traffic. This is the most important benefit to caching data locally. Otherwise, every time the app starts it has to hit the OData feed to load initial data, which is a huge waste of bandwidth for OData feeds that are relatively static.
  2. Improved load time. It’s much faster to load data from the device than to make an HTTP request to the OData service.
  3. App don’t need network access to start. When the app is dependent on remote data, it can’t really start well without a connection.
  4. Puts the user in control. Users can decide on if and when to sync the local data to the OData service, since most of them pay for their data.
  5. Reduced tombstoning serialization. When the entities are stored in local database, they can be retrieved from there, which means they don’t need to be serialized and tombstoned.

Note that I have published my completed project to MSDN Code Gallery as Using Local Storage with OData on Windows Phone To Reduce Network Bandwidth. This project contains my generated proxy classes, but not the T4 templates that I used to generate them (I’m holding the template code a little longer to make sure that I have the licensing and copyright stuff correct).

As of the last post, I was working on getting a nice solution to requesting and storing media resources, in this case binary image files. It turns out that, in general, the OData client library and Silverlight APIs make it rather easy to request binary data as streams from the data service and use this stream to create an image to store locally in isolated storage.

Requesting and Storing the Media Resource Stream

As I mentioned in an earlier post, a great solution (as long as you aren’t sending updates to the data service) is to create a new binding property in a partial class of the entity that returns the read stream URI. This time, because you are storing and retrieving of binary image data yourself, you need the extension property to return the actual BitmapImage for binding, which means you must deal with streams and not just the URI (and let the binding do the work).

Here is the partial class that defines the DefaultImage property:

// Extend the Title class to bind to the media resource URI.
public partial class Title
{
    private BitmapImage _image;

    // Returns the media resource URI for binding. 
    public BitmapImage DefaultImage 
    { 
        get 
        { 
            if (_image == null) 
            { 
                // Get the URI for the media resource stream. 
                return App.ViewModel.GetImage(this); 
            } 
            else 
            { 
                return _image; 
            } 
        } 
        set 
        { 
            _image = value; 
            OnPropertyChanged("DefaultImage"); 
        } 
    } 
}

The GetImage method on the ViewModel first checks isolated storage for the image, and if it’s not there it makes an asynchronous BeginGetReadStream call to data service:

// Calls into the DataServiceContext to get the URI of the media resource.
public BitmapImage GetImage(object entity)
{
    // First check for the image stored locally.
    // Obtain the virtual store for the application.
    IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication();

    Title title = entity as Title;
    MergeOption cacheMergeOption;
    Uri entityUri;           

    // Construct the file name from the entity ID.
    string fileStorageName = string.Format("{0}\\{1}.png", isoPathName, title.Id);

    if (!isoStore.DirectoryExists(isoPathName))
    {
        // Create a new folder if it doesn't exist.
        isoStore.CreateDirectory(isoPathName);
    }

    // We need to handle the case where we have stored the entity but not the image.
    if (!isoStore.FileExists(fileStorageName))
    {
        // Try to get the key of the title entity; if it's not in the DataServiceContext,
        // then the entity comes from
        // the local database and it is not in the DataServiceContext, which means that
        // we need to request it again to get the URI of the media resource.
        if (!_context.TryGetUri(entity, out entityUri))
        {
            // We need to attach the entity to request it from the data service.
            _context.AttachTo("Titles", entity);

            if (_context.TryGetUri(entity, out entityUri))
            {
                // Cache the current merge option and change it to overwrite changes.
                cacheMergeOption = _context.MergeOption;
                _context.MergeOption = MergeOption.OverwriteChanges;

                // Request the Title entity again from the data service.
                _context.BeginExecute<Title>(entityUri, OnExecuteComplete, entity);

                // Reset the merge option.
                _context.MergeOption = cacheMergeOption;
            }
        }
        else
        {
            DataServiceRequestArgs args = new DataServiceRequestArgs();

            // If the file doesn't already exist, request it from the data service.
            _context.BeginGetReadStream(title, args, OnGetReadStreamComplete, title);                   
        }

        // We don't have an image yet to set.
        return null;
    }
    else
    {
        using (var fs = new IsolatedStorageFileStream(fileStorageName, FileMode.Open, isoStore))
        {
            // Return the image as a BitmapImage.
            // Create a new bitmap image using the memory stream.
            BitmapImage imageFromStream = new BitmapImage();
            imageFromStream.SetSource(fs);

            // Return the bitmap.
            return imageFromStream;
        }
    }           
}

When the request is completed, the OnGetReadStream callback method is invoked, where EndGetReadStream is called first to write the stream to local storage and then to set the DefaultImage property of the specific Title entity, which causes the binding to be updated with the image.

private void OnGetReadStreamComplete(IAsyncResult result)
{
    // Obtain the virtual store for the application.
    IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication();

    Title title = result.AsyncState as Title;

    if (title != null)
    {
        // Use the Dispatcher to ensure that the
        // asynchronous call returns in the correct thread.
        Deployment.Current.Dispatcher.BeginInvoke(() =>
            {
                try
                {
                    // Get the response.
                    DataServiceStreamResponse response =
                        _context.EndGetReadStream(result);

                    // Construct the file name from the entity ID.
                    string fileStorageName = string.Format("{0}\\{1}.png",
                        isoPathName, title.Id);

                    // Specify the file path and options.
                    using (var isoFileStream =
                        new IsolatedStorageFileStream(fileStorageName,
                            FileMode.Create, isoStore))
                    {
                        //Write the data
                        using (var fileWriter = new BinaryWriter(isoFileStream))
                        {
                            byte[] buffer = new byte[1000];
                            int count = 0;

                            // Read the returned stream into the new file stream.
                            while (response.Stream.CanRead && (0 < (
                                count = response.Stream.Read(buffer, 0, buffer.Length))))
                            {
                                fileWriter.Write(buffer, 0, count);
                            }
                        }
                    }

                    using (var bitmapFileStream =
                        new IsolatedStorageFileStream(fileStorageName,
                            FileMode.Open, isoStore))
                    {
                        // Return the image as a BitmapImage.
                        // Create a new bitmap image using the memory stream.
                        BitmapImage imageFromStream = new BitmapImage();
                        imageFromStream.SetSource(bitmapFileStream);

                        // Return the bitmap.                                   
                        title.DefaultImage = imageFromStream;
                    }
                }

                catch (DataServiceClientException)
                {
                    // We need to eat this exception so that loading can continue.
                    // Plus there is a bug where the binary stream gets
                    /// written to the message.
                }
            });
    }
}

Note that there currently is a bug where the client doesn’t correctly handle a 404 response from BeginGetReadStream (which Netflix returns) when that response contains an image stream, and it tries to write the binary data to the Message property of the DataServiceClientException that is generated. During debug, this did cause my VS to hang when I moused-over the Message property, so watch out for that.

Getting the Media Resource for a Stored Entity

Loading the media resource actually gets a little complicated for the case where you have stored the entity in local database, but for some reason you don’t also have the image file stored. The issue is that you only store the entity itself in the local database, but each tracked entity in the DataServiceContext has a companion EntityDescriptor object that contains non-property metadata from the entry in the OData feed, including the read stream URI. You first need to call AttachTo, which starts tracking the entity and creates a new EntityDescriptor. However, only the key URI value gets set in this new EntityDescriptor, which is immutable and is inferred from the metadata. The context has no way to guess about the read stream URI, which can be changed by the data service at any time. This means to get the read stream, you need to first call BeginExecute<T> to get the complete MLE (with the missing entry info including the read stream URI) from the data service. the following section from a previous code snippet  is the part that checks the context for the entity, and if it’s not there, it requests it again from the data service:

// Try to get the key of the title entity; if it's not in the DataServiceContext,
// then the entity comes from  the local database and it is not in the
// DataServiceContext, which means that we need to request it again to get
// the URI of the media resource.
if (!_context.TryGetUri(entity, out entityUri))
{
    // We need to attach the entity to request it from the data service.
    _context.AttachTo("Titles", entity);

    if (_context.TryGetUri(entity, out entityUri))
    {
        // Cache the current merge option and change it to overwrite changes.
        cacheMergeOption = _context.MergeOption;
        _context.MergeOption = MergeOption.OverwriteChanges;

        // Request the Title entity again from the data service.
        _context.BeginExecute<Title>(entityUri, OnExecuteComplete, entity);

        // Reset the merge option.
        _context.MergeOption = cacheMergeOption;
        }
    }
    else 
    { 
        DataServiceRequestArgs args = new DataServiceRequestArgs();

        // If the file doesn't already exist, request it from the data service. 
        _context.BeginGetReadStream(title, args, OnGetReadStreamComplete, title);  
     }

     // We don't have an image yet to set. 
     return null;
}

Because this method requires two call to the data service (one to refresh the entity and the second to get the stream), it might be better to store a generic not found image rather than make two calls. Another option would be to further extend the entity type on the client to include a property that can be used to store the read stream URI. Then, you can just make a regular HttpWebRequest to the URI , now stored with the entity, to get the image from the data service—one request instead of two. (This would be another property that must be removed from a MERGE/PATCH/POST request for entities that are not read-only.) Less request, but a bit more complex still.

Tombstoning with Locally Stored Entity Data

I mentioned that one of the benefits of storing entity data in local database was (potentially) simplified tombstoning, and in this exact scenario of read-only data, this is the case. By always trying to load from the local database (and isolated storage) first, we don’t need to serialize entity data during tombstoning. Notice that in the SaveState and RestoreState methods we are only persisting the current page number and the selected title, instead of using the DataServiceState to serialize out all the in-memory data tracked by the DataServiceContext:

// Return a collection of key-value pairs to store in the application state.
public List<KeyValuePair<string, object>> SaveState()
{
    // Since we are storing entities in local database,
    // we don't need to store the OData client objects.              
    List<KeyValuePair<string, object>> stateList
        = new List<KeyValuePair<string, object>>();

    stateList.Add(new KeyValuePair<string, object>("CurrentPage", CurrentPage));
    stateList.Add(new KeyValuePair<string, object>("SelectedTitle", SelectedTitle));

    return stateList;
}

// Restores the view model state from the supplied state dictionary.
public void RestoreState(IDictionary<string, object> storedState)
{          
    // Restore view model data.
    _currentPage = (int)storedState["CurrentPage"];
    this.SelectedTitle = storedState["SelectedTitle"] as Title;
}

Of course, this also get more complicated when we need to support data updates because we need to attach objects from the local database to the DataServiceContext before we can send changes to the data service, and DataServiceContext is where changes are tracked.

Conclusion

The sample that I put together is based on the OData client library for Windows Phone and is limited to download-only. This solution for persisting entity data from an OData service in local database (and isolated storage for blobs) is probably best for reference data or for data that is relatively static. For data that changes frequently or that must be updated by the client, you probably need to continue to request fresh data from the data service on startup. You could also use timestamp properties in the data model to implement a more incremental kind of downloading of updated entities, but you will be unable to detect deletes from the data services. Also, you will need to first re-attach stored entities to the DataServiceContext to leverage the conflict detection and identity management facilities provided by the client. Then you can request all entities in the feed with a timestamp value greater than the last download date, and use a MergeOption value of OverwriteChanges to make sure that the client is updated with server values (of course the DataContext will need to be updated with the new values too). Another option might be to use the DataServiceState to serialize and persist an entire context and collections in local storage as serialized XML. How you handle this will depend greatly on your scenario, how often the data changes, and how much data your application must deal with to run.

As you can see, compared to the total problem set of maintaining OData entities offline, this solution is rather basic (it does what it does by leveraging the local database), but it’s not truly “sync,” despite the title of the series. In the next post, I plan to discuss another option that provide a much more traditional and comprehensive kind of bi-directional synchronization between data on the Windows Phone device and data in the cloud by using an OData-based sync service.

Stay tuned…

Glenn Gailey