Carlos Figueira MSDN blog

Preserving date time offsets in Azure Mobile Services

CarlosFigueira — Mon, 13 May 2013 21:49:08 GMT

In my last post I mentioned that due to the heterogeneous nature of Azure Mobile Services (JavaScript in the server, other languages / frameworks in the client), some data types have some problems with the conversion between the two environments. Numbers and dates are the most common, but this post from the MSDN forum pointed out that DateTimeOffset values also have issues. Basically, DateTimeOffset values are interpreted in the server side as JavaScript Date objects, and as such they don’t have any information about offsets (in JS dates are represented as a number of milliseconds since the “Unix 0”, 1970/01/01:00:00:00 UTC). The result was that the offset was lost in translation, and that was a problem for the forum poster.

There are ways to work around this limitation, however. The simplest alternative would be to split the DateTimeOffset property into two properties in the client side – one for the DateTime itself, and one for the offset from UTC. In this case, the class would have two properties equivalent to the DateTimeOffset property – and the type can even continue with the original property (properly marked with [JsonIgnore] to prevent it from being sent to the server and creating a new column – notice that this code is already using the version 0.3.x of the client SDK, the Portable Library-based SDK from the NuGet feed). The code below shows the before and after version of a type with a DateTimeOffset property.

 public class Event
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Location { get; set; }
    public DateTimeOffset Time { get; set; }
}
 
public class ModifiedEvent
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Location { get; set; }
    public DateTime UtcTime { get; set; }
    public int UtcTimeOffsetMinutes { get; set; }
 
    [JsonIgnore]
    public DateTimeOffset Time
    {
        get
        {
            return new DateTimeOffset(
                DateTime.SpecifyKind(this.UtcTime.ToUniversalTime(), DateTimeKind.Unspecified),
                TimeSpan.FromMinutes(UtcTimeOffsetMinutes));
        }
 
        set
        {
            this.UtcTime = value.DateTime.Kind == DateTimeKind.Unspecified ?
                DateTime.SpecifyKind(value.DateTime, DateTimeKind.Utc) :
                value.DateTime.ToUniversalTime();
            this.UtcTimeOffsetMinutes = (int)value.Offset.TotalMinutes;
        }
    }
}
 

That solution works fine, but it forces us to change the domain object on the client side to deal with a distributed problem. If other people are using this type, they will get affected by it (we can mitigate it by using something like decorating the UtcTime and UtcTimeOffsetMinutes with [EditorBrowsable(EditorBrowsableState.Never)], but that’s again just working around the problem. What we can do is to change the way we serialize the original Time field, so that only during the serialization (and deserialization) of the type we need to worry about deconstructing (and reconstructing) the value. I posted before about how we can do it with the original managed client SDK, and the idea with the new version of the SDK (based on JSON.NET) is similar, as I’ll show below.

First, we don’t need to split the parameter in multiple values – all we need to do is to decorate the property with the [JsonConverter] attribute to tell the client runtime that we want to use our own code to convert that value to and from JSON.

 public class Event
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Location { get; set; }
    [JsonConverter(typeof(DtoPreservingOffsetConverter))]
    public DateTimeOffset Time { get; set; }
}
 

We can now go about implementing the converter – a type deriving from the JsonConverter class from JSON.NET, with three abstract methods. The implementation of CanConvert is trivial, as shown below.

 public class DtoPreservingOffsetConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return objectType == typeof(DateTimeOffset);
    }
 
    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        return null;
    }
 
    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
    }
}
 

To write the object into JSON we could call the appropriate methods into the JsonWriter parameter, but the simplest way is to wrap them into a dictionary, and let the JSON.NET serializer deal with that for us. What we’re doing here is to write the DateTimeOffset value as a JSON object with two members, one for the UTC value date component, one for the offset component.

 public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
    DateTimeOffset dto = (DateTimeOffset)value;
    DateTime dt = dto.DateTime.Kind == DateTimeKind.Unspecified ?
        DateTime.SpecifyKind(dto.DateTime, DateTimeKind.Utc) :
        dto.DateTime.ToUniversalTime();
 
    Dictionary<string, object> temp = new Dictionary<string, object>
    {
        { "DateTimeUTC", dt },
        { "OffsetMinutes", dto.Offset.TotalMinutes }
    };
 
    serializer.Serialize(writer, temp);
}
 

And since we write it, we need to read it back, so here’s the implementation of ReadJson:

 public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
    Dictionary<string, object> temp = serializer.Deserialize<Dictionary<string, object>>(reader);
    DateTime dateTimeUTC = ((DateTime)temp["DateTimeUTC"]).ToUniversalTime();
    int offsetMinutes = Convert.ToInt32(temp["OffsetMinutes"]);
    return new DateTimeOffset(
        DateTime.SpecifyKind(dateTimeUTC, DateTimeKind.Unspecified),
        TimeSpan.FromMinutes(offsetMinutes));
}
 

At this point, the client is all set. However, if we try to insert any instance of the Event class into an Azure Mobile Service, we’ll have an error at the runtime – the runtime doesn’t know how to insert objects into the database, so we need to update the insert (and update and read) scripts to “tell” it what to do with those. The full explanation can be found in the post about supporting complex types in mobile services. The insert script would look similar to the one below – if the incoming item has a ‘Time’ property of type object, that (likely) means that our converter is the one supplying the values, so we know which fields to look for. And after the insertion is completed, we recreate the original object, which will then be read by the converter at the client.

 function insert(item, user, request) {
    var dto = item.Time;
    if (typeof dto === 'object') {
        delete item.Time;
        item.Time = new Date(dto.DateTimeUTC);
        item.Offset = dto.OffsetMinutes;
    } else {
        // Either null or old clients; assuming object, and setting offset to 0
        item.Offset = 0;
    }
 
    request.execute({
        success: function () {
            // Need to recreate the object back
            var dtoObject = { DateTimeUTC: item.Time, OffsetMinutes: item.Offset };
            delete item.Offset;
            item.Time = dtoObject;
            request.respond();
        }
    });
}
 

The update handler is exactly the same – break the object before the operation, recreate it afterwards:

 function update(item, user, request) {
    var dto = item.Time;
    if (typeof dto === 'object') {
        delete item.Time;
        item.Time = new Date(dto.DateTimeUTC);
        item.Offset = dto.OffsetMinutes;
    } else {
        // Either null or old clients; assuming object, and setting offset to 0
        item.Offset = 0;
    }
 
    request.execute({
        success: function () {
            // Need to recreate the object back
            var dtoObject = { DateTimeUTC: item.Time, OffsetMinutes: item.Offset };
            delete item.Offset;
            item.Time = dtoObject;
            request.respond();
        }
    });
}
 

Finally, for reading we need, for all objects to be returned, to reassemble the object from the two columns in the database, which can be easily implemented by ‘forEach’-ing the results as shown below.

 function read(query, user, request) {
    request.execute({
        success: function (results) {
            results.forEach(function (item) {
                var dtoObject = { DateTimeUTC: item.Time, OffsetMinutes: item.Offset };
                delete item.Offset;
                item.Time = dtoObject;
            });
 
            request.respond();
        }
    });
}
 

That’s it. We don’t need to update the delete script since it is based on the object id.

This post is almost an update to the post about supporting complex types in mobile services using the new client API (based on JSON.NET), and hopefully it has shown that although some times the “impedance mismatch” between the runtime and the various client SDKs can cause problems, the Azure Mobile Services platform is extensible enough that we can, without a huge amount of code, work around the issues caused by the framework differences.

And as usual, please raise any concerns and suggestions in our MSDN forums or in the comments for those blogs.

Large numbers and Azure Mobile Services

CarlosFigueira — Mon, 22 Apr 2013 08:00:00 GMT

In any distributed system with different runtime environments, the way that data is represented in all nodes of the system, and transferred between them, can cause some problems if there is a mismatch between the nodes. In Azure Mobile Services we often have this issue - the server runtime runs on a JavaScript (or more precisely, node.js) engine, while the client can run in many different platforms (CLR managed code, Objective-C, Java, JavaScript, or any other client using the REST interface). With JavaScript - the mobile service runtime - there are two data types which usually cause problems: dates and numbers. Let's look at them in this post.

Dates problems aren't common to JavaScript - dealing with conversions between local time (what people would most like to see in their applications) and standard time (usually UTC, how we'd store data) has been a problem in many frameworks, including .NET, and many smart people have written about it. Besides the framework-specific issues with dates, the main problem with the fact that Azure Mobile Services as a distributed system uses JavaScript in the backend is that dates in JS are represented as the number of milliseconds from the Unix zero date (1970-01-01T00:00:00.000 UTC) is that dates with sub-millisecond precision are truncated, which is rarely a big problem.

Numbers, on the other hand, tend to cause some problem with heterogeneous systems with JavaScript in one side and another language on another. In JS, all numbers are represented as 64-bit (double precision) floating point values. In the managed world, that means that every number would be represented as a Double. But in the managed (or other languages with strong typing), other numeric types exist and are often used (with good reason) in defining the data types used by the application. Integers (usually 32 and 64 bits, but also in other sizes), single and double precision floating point numbers and fixed-point (with fixed or arbitrary precision) numbers are represented by a large variety of types in different languages. That means that there are many numbers which cannot be represented, without loss of precision, in JavaScript, so any time one of those numbers is sent from the client to the service (e.g., as part of an object being inserted into the database), when it’s sent back to the client, its value will be different. For example, any odd numbers beyond 2^53 cannot be represented as a 64-bit floating point value.

So what do the client SDKs for Azure Mobile Services do when facing with numbers which can potentially be corrupted by going to the server? The answer depends on the how the application is interacting with the SDK, or more specifically, the data types which are being stored into / retrieved from the backend service. In many languages, there are two possible ways for an application save data, so let’s look at how the SDK deals with numbers on those two separately.

Typed mode

In the typed mode, we use “regular”, user-defined data types (e.g., User, Product, Order, TodoItem, etc.), and the SDK handles the serialization / deserialization of those types into / from the format used in the wire (JSON). The clients for managed platforms (Windows Store, Windows Phone, Desktop .NET 4.5) and Android both have this mode. JavaScript-based clients (where there really are no user-defined data types – and I’m not going here into the argument of prototypes versus real object-orientation) doesn’t have this mode (and it really doesn’t matter for this specific post, since there’s no difference in number representation between the JavaScript on the client and on the server). The iOS client SDK also doesn’t have it, since there’s no widely-used, generic serialization mechanism to translate between Objective-C @interfaces and JSON.

In the typed mode, the SDK does a lot of data manipulation under the covers, so it was coded in a way that, if data loss were to happen, an exception is thrown to the user. The idea is that the developer is trusting the SDK with its data, and we don’t want to corrupt it without warning the user. Let’s take the code below.

 public sealed partial class MainPage : Page
{
    public static MobileServiceClient MobileService = new MobileServiceClient(
        "https://YOUR_SERVICE_HERE.azure-mobile.net/",
        "YOUR_APPLICATION_KEY_HERE"
    );
 
    public MainPage()
    {
        this.InitializeComponent();
    }
 
    private async void btnStart_Click_1(object sender, RoutedEventArgs e)
    {
        try
        {
            var table = MobileService.GetTable<Test>();
            Test item = new Test { Str = "hello", Value = (1L << 53) + 1 };
            await table.InsertAsync(item);
            AddToDebug("Inserted: {0}", item);
        }
        catch (Exception ex)
        {
            this.AddToDebug("Error: {0}", ex);
        }
    }
 
    void AddToDebug(string text, params object[] args)
    {
        if (args != null && args.Length > 0) text = string.Format(text, args);
        this.txtDebug.Text = this.txtDebug.Text + text + Environment.NewLine;
    }
}
 
public class Test
{
    public int Id { get; set; }
    public string Str { get; set; }
    public long Value { get; set; }
 
    public override string ToString()
    {
        return string.Format("Id={0},Str={1},Value={2}", Id, Str, Value);
    }
}
 

When the btnStart_Click_1 handler is invoked, the code tries to insert a typed item (Test) into a table with a long value which would be corrupted if the operation were to complete. Instead, the code throws the following exception

System.InvalidOperationException: The value 9007199254740993 for member Value is outside the valid range for numeric columns.

The validation ensures that integers have to fall in the range [-2^53, +2^53]; numbers outside that range are rejected, and the exception is thrown.

Now, what if you really want to use numbers beyond the allowed range? There are a few possibilities. In the .NET-based SDKs, you can actually remove the validation, which is made by a JSON.NET converter, by using a code similar to the one below. Notice that this will cause data corruption, but if precision can be sacrificed for a wider range of numbers, then that’s an option.

 var settings = MobileService.SerializerSettings;
var mspcc = settings.Converters.Where(c => c is MobileServicePrecisionCheckConverter).FirstOrDefault();
if (mspcc != null)
{
    settings.Converters.Remove(mspcc);
}
 
var table = MobileService.GetTable<Test>();
Test item = new Test { Str = "hello", Value = (1L << 53) + 1 };
await table.InsertAsync(item);
AddToDebug("Inserted: {0}", item);
 

Another alternative is to change the data type for the value. Double is represented exactly like numbers in the server, so all numbers that can be represented in the client can be transferred to the server and back. But double values may lose precision as the numbers grow big as well.

Yet another alternative is to use strings instead of numbers. With strings you can actually have arbitrary precision, but you may lose the ability to do relational queries on the data (unless you use some sort of zero-left-padding to normalize the values), and they take up more storage on the server.

The main take away is that if you’re dealing with large numbers and user-defined types, there will be cases where those numbers won’t be able to be represented in the server. The client SDK will try its best to warn the user (via exceptions) that a data loss would occur, but there are alternatives if the application really requires large numbers to be stored.

Untyped mode

The second way for an application to exchange data with the service is via an “untyped” model, where instead of dealing with “user types”, the application works with simpler types (dictionaries, arrays, primitives) which map directly to JSON. The untyped model appears in different ways in different platforms:

for the SDKs based on the managed code, they’re represented by types in the Newtonsoft.Linq.Json namespace from JSON.NET, the JSON serializer used by the managed clients (in the original client SDK for Windows Store / managed languages the types were from the Windows.Data.Json namespace, but in the latest portable library-based SDK, which will replace the original one, it was replaced with JSON.NET)
for JavaScript clients (both Windows Store and “plain” HTML), they’re represented by JavaScript arrays, objects and primitives (strings, Boolean, numbers and dates)
for the Android client they’re represented by the JsonElement class (and its subclasses) on the com.google.gson package.
on the iOS client they’re represented by the following classes: NSDictionary (maps to JSON objects), NSArray (maps to JSON arrays), NSNull (maps to the JSON null value), NSString (maps to JSON strings), NSNumber (maps to JSON Boolean or numbers, depending on its value) and NSDate (maps to JSON strings, with a format which is recognized in the server as a date object)

Unlike on the typed mode, where there is a step which is taken to convert the object into the JSON-like structure which is sent to the server, this step is unnecessary in the untyped mode. Therefore, we had to make a choice: validate that the numbers could be faithfully represented in the server and return an error (such as returning exceptions or the appropriate error callback), and incur the penalty of the additional validation for a scenario which isn’t too common; or bypass the validation, and let the user (in the scenarios where it’s applicable) deal with the large numbers themselves. After some internal discussion, we made the second choice (I don’t think there’s really a right or wrong approach, just some decision that had to be made – but if you disagree, you can always send a suggestion in our UserVoice page and we can consider it for the updates to the client SDKs).

What that decision entails is that if you try to run the following code, you’ll not get any error:

 JObject item = new JObject();
item.Add("Str", "hello");
item.Add("Value", 1234567890123456789L);
var table = MobileService.GetTable("Test");
var inserted = await table.InsertAsync(item);
AddToDebug("Original: {0}", item);
AddToDebug("Inserted: {0}", inserted);
 

What will happen instead is that the output will be shown as follows:

Original: {
"Str": "hello",
"Value": 1234567890123456789
}
Inserted: {
"Str": "hello",
"Value": 1234567890123456800,
"id": 36
}

Similarly, this Objective-C code shows the same result

- (IBAction)clickMe:(id)sender {
    MSTable *table = [client tableWithName:@"Test"];
    NSDictionary *item = @{@"Str" : @"Hello", @"Value" : @(1234567890123456789L)};
    [table insert:item completion:^(NSDictionary *inserted, NSError *error) {
        NSLog(@"Original: %@", item);
        NSLog(@"Inserted: %@", inserted);
    }];
}

And the logs:

2013-04-10 13:36:18.009 MyApp[9289:c07] Original: {
    Str = Hello;
    Value = 1234567890123456789;
}
2013-04-10 13:36:18.009 MyApp[9289:c07] Inserted: {
    Str = Hello;
    Value = 1234567890123456800;
    id = 58;
}

Now, what if we actually want to enforce the limit checks on untyped operations? One simple alternative is to, prior to making the CRUD call, to traverse the object to see if it contains a long value which cannot be represented at the server side. Another alternative is to add a message handler (on the managed clients) or a filter (on the other platforms) which will look at the outgoing JSON request and fail if it has some numbers which can cause trouble if sent to the server side. This is one simple implementation of the validation for the managed client:

       bool WillRoundTripWithNoDataLoss(JToken item)
      {
          if (item == null) return true;
          switch (item.Type)
          {
              case JTokenType.Array:
                  JArray ja = (JArray)item;
                  return ja.All(jt => WillRoundTripWithNoDataLoss(jt));
              case JTokenType.Object:
                  JObject jo = (JObject)item;
                  return jo.PropertyValues().All(jt => WillRoundTripWithNoDataLoss(jt));
              case JTokenType.Boolean:
              case JTokenType.Float:
                caseJTokenType.Null:
              case JTokenType.String:
                  return true;
              case JTokenType.Integer:
                  JValue jv = (JValue)item;
                  long value = jv.Value<long>();
                  long maxAllowedValue = 0x0020000000000000;
                  long minAllowedValue = 0;
                  unchecked
                  {
                      minAllowedValue = (long)0xFFE0000000000000;
                  }
 
                  return minAllowedValue <= value && value <= maxAllowedValue;
              default:
                  throw new ArgumentException("Validation for type " + item.Type + " not yet implemented");
          }
      }
 

In summary, it’s possible that you’ll never need to worry about this “impedance mismatch” between the client and the server for large numbers, and all values will just work. But it’s always nice to go into a framework knowing some of its pitfalls, and this one is in my opinion one which could be hard to identify.

Updates (some API changes) to the Azure Mobile Services iOS SDK

CarlosFigueira — Wed, 10 Apr 2013 21:50:00 GMT

Another quick one today. After some feedback from the community, we released a new version of the Azure Mobile Services iOS SDK which contains, in addition to a few bug fixes, some changes in the method names of some of the classes. Those changes were made to make the iOS SDK more in sync with the design and naming guidelines for other iOS libraries. The new SDK is live, and the Quickstart app which can be downloaded from the Azure Portal is also using the new APIs as well. Finally, our tutorials have also been updated to reflect the naming changes.

The changes are listed below. You can also find the changes to the SDK in our public GitHub repository. The changes were made in this commit.

MSClient interface
- Class method clientWithApplicationURLString:withApplicationKey: ==> clientWithApplicationURLString:applicationKey:
- Class method clientWithApplicationURL:withApplicationKey: ==> clientWithApplicationURL:applicationKey:
- Initializer initWithApplicationURL:withApplicationKey: ==> initWithApplicationURL:applicationKey:
- Instance method loginWithProvider:onController:animated:completion: ==> loginWithProvider:controller:animated:completion:
- Instance method loginWithProvider:withToken:completion: ==> loginWithProvider:token:completion:
- Instance method getTable: ==> tableWithName:
- Instance method clientwithFilter: ==> clientWithFilter: (capitalization of the ‘W’)
MSClientConnection interface (mostly internal usage)
- Initializer initWithRequest:withClient:completion: ==> initWithRequest:client:completion:
MSFilter protocol
- Method handleRequest:onNext:onResponse: ==> handleRequest:next:response:
MSLogin interface (mostly internal usage)
- Instance method loginWithProvider:onController:animated:completion: ==> loginWithProvider:controller:animated:completion:
- Instance method loginWithProvider:withToken:completion: ==> loginWithProvider:token:completion:
MSQuery interface
- Instance method initWithTable:withPredicate: ==> initWithTable:predicate:
MSTable interface
- Initializer initWithName:andClient: ==> initWithName:client:
- Instance method readWhere:completion: ==> readWithPredicate:completion:
- Instance method queryWhere: ==> queryWithPredicate:

That’s it. As usual, please let us know if you have any suggestions / comments / question either by using our MSDN forum or our UserVoice page.

Azure Mobile Services Windows Store / JavaScript SDK – now also from NuGet

CarlosFigueira — Mon, 08 Apr 2013 23:39:46 GMT

Quick post today. About a month ago we released a NuGet version of the managed client SDK. Last week we released the SDK for Windows Store applications / JavaScript also on NuGet, so you can develop applications using that platform without needing any installation (MSI). Building Windows Store / JavaScript applications with the NuGet package is almost the same as building with the current, MSI-based SDK. After creating a new project, right-click the project and select the “Manage NuGet Packages…” option.

On the search window, type “mobileservices.winjs”, and that will show the NuGet package for the Windows Store / JS. The current “stable” version (0.2.0) is exactly the same one as the one which is on the current MSI-based SDK. We also have one version (0.3.0-alpha) which is a preview of the next version. It’s similar to the 0.2.0, with some bug fixes. It will appear if you select the “include prerelease” option as shown below; after a few weeks of testing we’ll mark that package “stable”). Select the package and click “Install”.

That’s it. On the version 0.2.0, one file will be added to your project: MobileServices.js; on the next version, two files will be added to your project: MobileServices.js and MobileServices.pri – the first being the code for the SDK itself, and the second one is a compiled file containing the resources for the library (basically, exception strings). To use the library, open the file “default.html”, then click and drag the file “MobileServices.js” into the list of app references. That’s very similar to what was done with the MSI-based SDK, but the file was dragged from the references folder.

And that’s it. As always, please let us know if you have any comments / suggestions on the new features which we’re shipping.

Azure Mobile Services managed client – portable libraries and breaking changes

CarlosFigueira — Thu, 14 Mar 2013 00:35:00 GMT

Among the requested features in the various forums for Azure Mobile Services, the usage of portable libraries has popped up quite often. We now have that support, which unifies the managed clients for the Windows Store and Windows Phone 8 platforms – and we’re also adding support for Windows Phone 7.5 as well. Johan Laanstra wrote a great post explaining the architecture of the new packages, so I won’t repeat what he wrote.

However, since there were some differences between the Windows Phone 8 and the Windows Store SDKs, unifying them meant some breaking changes. We felt that the grief we’ll take for them is worth, given the gains we can get by the platform unification, and we also used this opportunity to make some other changes to the client SDK to make it more polished before the general availability of the service. This post will try to list to the best of my knowledge the changes which will need to be addressed for apps using the new client. Notice that we’ll still support the “non-portable” clients for some more time, but it will be discontinued likely before the Azure Mobile Services goes out of preview mode and into a general availability,

Breaking changes / functionality changes

This is the grouped list of changes, along with the updates one needs to make to their projects to account for them.

Serialization changes

This is the place where the majority of the changes were made. For the Windows Store SDK, conversion between the typed objects and the JSON which was actually transferred over the wire was done via a custom serializer which converted between the two formats. The serializer was fairly simple, and could only deal with simple types (numbers, strings, dates and,Boolean values). To support more complex types (unsupported primitives, arrays, complex objects), we’d need to provide a custom implementation of the IDataMemberJsonCoverter interface, and decorate the appropriate member with the [DataMemberJsonConverter] attribute. Even simple things such as enumerations were not supported. With this new client, all the custom serialization code goes away, as we’ll be using the JSON.NET serializer, and by that move we gain all of the support of that great serializer. That means that complex types, arrays, enumerations will just work. The new client also exposes the serializer settings which allow the developer to fine-tune the output data, including adding custom converters and changing policies such as default value serialization.

Now for the list of changes:

JSON.NET: that’s the serializer used for all platforms (not only for Windows Phone 8 as before). That causes the following behavioral changes from the current Windows Store SDK

JSON bool values cannot be parsed as integers; false was interpreted as 0, and true as 1; now an exception will be thrown. This has since been modified to preserve the original behavior.
Better / consistent exception messages for failed JSON parsing: there was a lot of code in the custom serializer for dealing with bad data; now this is all taken care of by JSON.NET, and its exception messages include more data such as the path in the JSON document, and position of the error
All floating point members (even those with integer values) will be serialized with a decimal point: a JSON.NET behavior. Before a float / double with value 2 would be serialized as {“number”:2}; now it will be serialized as {“number”:2.0}.

To maintain the previous behavior, one possibility is to use a custom JsonConverter class that first casts the values to integers if they are indeed integers.

Float (System.Single) will be serialized with its “native” precision: before a member of type float (Single) would be serialized as if it had double precision. For example, the object new MyType { fltValue = float.MinValue } would be serialized as {“fltValue”:1.4012984643248171E-45}; now it will be serialized as {"fltValue":1.401298E-45}.

To maintain the previous behavior, use a JsonConverter class that serializes float values as doubles (or change the property type to double)

Float and double values less than their respective Epsilon now deserialize as 0.0: before it used to throw an exception. Now deserializing something like {“dblValue:1.23E-1000} will cause the dblValue member to be set to zero, instead of an exception being thrown.
Fields are now serialized before properties: before properties were serialized before fields; now is the opposite. Since JSON objects are supposed to be unordered sets of members, this shouldn’t affect existing apps.
Finer control of serialization via settings: the MobileServiceClient now exposes the JsonSerializerSettings object which is used by the JSON.NET serializer, so you can now have greater control of the serialization.
Miscellaneous bugs fixed:

Tab / newline characters can now be deserialized (before it would result in an exception)
Integer values can now be deserialized to members of type char.

Miscellaneous scenarios which are now supported:

Uri / TimeSpan members don’t need a special converter anymore: they’re serialized as JSON strings
Arrays / objects also now just work. They’re serialized as JSON array / object respectively. Notice that the server runtime still needs to process that value, since they cannot be inserted into the database directly.

Non JSON.NET specific changes:

Enumeration support: it was enabled by the move to JSON.NET, but our code now serializes the string value of the enumerations. Enums can also be used in ‘Where’ clauses while querying data from the service.
Loss of precision handling: the runtime for Azure Mobile Services is based on JavaScript (node.js), which only stores numbers in a double-precision floating representation. That means that there are some integer and decimal values which cannot be represented accurately. Basically, any long number less than 2^53 can be represented, without loss of precision, in the server, so those are serialized correctly; any values larger than that (or smaller than -2^53) will cause an exception to be thrown. Notice that this is the current behavior, although it was a little buggy in which it would report some numbers within the valid range as invalid as well. Decimal values can also lose precision, and now this validation (exception thrown if precision would be lost) is applied to them as well. If you want to bypass the validation, either change the data type of the properties to double, or remove the converter (MobileServicePrecisionCheckConverter) from the serializer settings.

One minor breaking change: before the exception thrown for long values outside the non-precision-loss range was ArgumentOutOfRangeException; now the exception type is InvalidOperationException.

IDataMemberJsonConverter, [DataMemberJsonConverter] removed: now they either are not necessary anymore, or if you still want full control over the serialization of a property, you can use the [JsonConverterAttribute] applied to that property.
ICustomMobileTableSerialization removed: if a type needs special serialization, decorate it with [JsonConverterAttribute].
Only one member with name ‘id’ – regardless of casing – can exist in a type being serialized: before an object with members called “id”, “Id” and “ID” would be serialized to the server – where it would fail, since only one id is allowed. Now it will throw during serialization.
[DataMember] attributes in members now require a [DataContract] attribute in the class: the contract of [DataMember] for serialization is tied with [DataContract], but we had a bug in which data members were being honored even if the containing class itself didn’t opt in to the data contract model (by using the [DataContract] attribute). Now if the SDK finds such a scenario it will throw an exception to signal that this is not a valid scenario. The typical scenario for using [DataMember] would be to change the serialized name of the member; now it can be accomplished via the [JsonPropertyAttribute] instead.
The [DataTableAttribute] now takes the table name in the constructor: before: [DataTable(Name = “MyTable”)]; after:: [DataTable(“MyTable”)]

If I remember more or if you find one which is not listed here, please add a comment and I’ll update this post.

Operations with untyped (i.e., JSON) data

When performing CRUD operations in a managed client, usually we’d create a data type to store our information (e.g., Person, Order, etc.) and pass them to the Insert/Read/Update/Delete operations. We could also work directly with JSON data, and the table would support those as well. In the Windows Store client the SDK used the classes in the Windows.Data.Json namespace, native to the WinRT platform. However, those classes were not present in the Windows Phone 8, so that SDK was already using the JSON.NET classes from the Newtonsoft.Json.Linq namespace (JToken / JArray / JValue / JObject). To move to a common set of classes, now all managed libraries use the JSON.NET classes, including the Windows Store apps.

And the list of changes:

[Windows Store only] Windows.Data.Json –> Newtonsoft.Json.Linq: there’s an added advantage that the JSON.NET classes have an API which is way more user-friendly than the “dry” WinRT one.
Insert / update operations do not change the input object (untyped data only): This is a behavioral change which was the outcome of a lengthy discussion within the team. Before, the insert / operations would mutate the JsonObject (or JObject) value which was passed to it, by “patching” it with whatever the server returned. The most common case was the “id” in insert operations, but the server scripts are were free to mutate the object, even changing the shape of the value returned by the client (i.e., in an insert operation, the service could receive an object and return an array, or a primitive value). Before that would mean that the client had no way to retrieve that value. Now, the object passed to InsertAsync (and UpdateAsync) is not mutated, and the return of the server operation is returned as a new object instead. Notice that for typed operations (e.g., inserting / updating an Order object), the original object is still patched in place.

This may require some code changes. For example, the code below:

var table = MobileService.GetTable(“Clients”);
var obj = JsonObject.Parse("{\"name\":\"John Doe\",\"age\":33}");
await table.InsertAsync(obj);
var id = (int)obj["id"].GetNumber();

Would need to be rewritten as (notice changes in italics)

var table = MobileService.GetTable(“Clients”);
var obj = JObject.Parse("{\"name\":\"John Doe\",\"age\":33}");
var inserted = await table.InsertAsync(obj);
var id = inserted["id"].Value<int>();

Again, I’ll update the post if I can remember (or if you can add a comment) of any additional changes.

Service filters –> HttpClient primitives

Service filters are an advanced concept which implement a pipeline over the request / response. There is already such a pipeline in the HttpClient libraries – and since it has recently been made available for downlevel platforms (it originally only worked for .NET 4.5 and .NET 4.0) the Mobile Services SDK can now use this existing code instead of creating yet another pipeline.

And the changes:

Goodbye IServiceFilter, hello DelegatingHandler: this is a sample implementation of a service filter, and the equivalent implementation with a delegating handler. Notice that there’s no IAsyncOperation<T> anymore, so the code is cleaner. Below you can see an example of the before / after the change.
Message handlers are passed to the MobileServiceClient constructor, not to a method (WithFilter): that method created a clone of the client, and applied the filters to it. That name caused some confusion among users, with many people thinking that it would modify the client. By moving it to a constructor the API now makes it clear.

Here’s an example of a service filter converted to a message handler. The filter will add a custom header to the request, and change the response status code (useful for testing the behavior on unexpected responses from the server). First the service filter.

publicasyncTask CallFilteredClient()
{
    var client = newMobileServiceClient(appUrl, appKey).WithFilter(newMyFilter());
    var table = client.GetTable<Person>();
    var p = newPerson { Name = "John Doe" };
    await table.InsertAsync(p);
}
 
publicclassMyFilter : IServiceFilter
{
    publicIAsyncOperation<IServiceFilterResponse> Handle(IServiceFilterRequest request, IServiceFilterContinuation continuation)
    {
        request.Headers.Add("x-my-header", "my value");
        return continuation.Handle(request).AsTask().ContinueWith<IServiceFilterResponse>(t => {
            HttpStatusCode newStatusCode = HttpStatusCode.ServiceUnavailable;
            var response = t.Result;
            var newResponse = newMyResponse(response.Content, response.ContentType, response.Headers,
                response.ResponseStatus, (int)newStatusCode, newStatusCode.ToString());
            return newResponse;
        }).AsAsyncOperation();
    }
 
    classMyResponse : IServiceFilterResponse
    {
        string content;
        string contentType;
        IDictionary<string, string> headers;
        ServiceFilterResponseStatus responseStatus;
        int statusCode;
        string statusDescription;
 
        public MyResponse(string content, string contentType, IDictionary<string, string> headers,
            ServiceFilterResponseStatus responseStatus, int statusCode, string statusDescription)
        {
            this.content = content;
            this.contentType = contentType;
            this.headers = headers;
            this.responseStatus = responseStatus;
            this.statusCode = statusCode;
            this.statusDescription = statusDescription;
        }
 
        publicstring Content
        {
            get { returnthis.content; }
        }
 
        publicstring ContentType
        {
            get { returnthis.contentType; }
        }
 
        publicIDictionary<string, string> Headers
        {
            get { returnthis.headers; }
        }
 
        publicServiceFilterResponseStatus ResponseStatus
        {
            get { returnthis.responseStatus; }
        }
 
        publicint StatusCode
        {
            get { returnthis.statusCode; }
        }
 
        publicstring StatusDescription
        {
            get { returnthis.statusDescription; }
        }
    }
}

And the same code implemented with a delegating handler:

publicasyncTask CallClientWithHandler()
{
    var client = newMobileServiceClient(appUrl, appKey, newMyHandler());
    var table = client.GetTable<Person>();
    var p = newPerson { Name = "John Doe" };
    await table.InsertAsync(p);
}
 
publicclassMyHandler : DelegatingHandler
{
    protectedoverrideasyncTask<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.Add("x-my-header", "my value");
        var response = awaitbase.SendAsync(request, cancellationToken);
        response.StatusCode = HttpStatusCode.ServiceUnavailable;
        return response;
    }
}

For more information on the HttpClient primitives, check the blog post from the BCL team.

Miscellaneous changes

Those are the other minor changes in the library:

MobileServiceClient.LoginInProgress has been removed
MobileServiceClient.LoginAsync(string) has been removed. This is now called LoginWithMicrosoftAccountAsync, which is added as an extension method to the class.
MobileServiceTable, MobileServiceTable<T> and MobileServiceTableQuery<T> are now internal classes: all methods which used to return a table object now return the interface type IMobileServiceTable (untyped) or IMobileServiceTable<T> (typed). Typed queries are now exposed as the IMobileServiceTableQuery<T> interface.
MobileServiceTable.<T>ToCollectionView() is now ToCollection(): the collection view implementation had some bugs, and it has been rewritten.
Multiple Take operations now considers the minimum value: before it would use the last value. The call table.Take(5).Take(3).Take(7).ToListAsync() would issue a request with $top=7; now it sends a request with $top=3.
Multiple Skip operations now add the values: before it would use the last value. The call table.Skip(3).Skip(5).Skip(4) would issue a request with $skip=4; now it sends a request with $skip=12.

That should be it. Please start using the new libraries and let us know what you think! We still have some (although not much) time to react to user feedback before we have to lock the API down as we move to the general release.

Azure Mobile Services managed client – now also from NuGet

CarlosFigueira — Sat, 09 Mar 2013 00:10:28 GMT

Currently if you want to create a Windows Store application which uses Azure Mobile Services, the process would be to go to either the quickstart page of your application in the Azure Portal (or the Windows Azure downloads page) and install the Mobile Services SDK:

After installing the SDK, you can then go to the Visual Studio project, and add a reference to the library, in the Windows Extensions tab:

That’s simple, but we can make it simpler – and with the added bonus of avoiding the installation of code in your machine. NuGet is a Visual Studio extension (installed by default in VS 2012) that makes it easy to install and update libraries to Visual Studio project. To install the required references using NuGet:

1. Select “Manage NuGet Packages” in the project

2. Search for “Windows Azure Mobile”, select the “Windows Azure Mobile Services” package, and click install

That’s it! NuGet downloads the necessary assemblies, and adds reference to it in the project. The package supports both Windows Store (managed) applications and Windows Phone 8 applications.

Quick caveat

Just one quick note: to use the Windows Azure Mobile Services NuGet package, you need to have the latest (2.2). If you have an older version, you may see this error:

If you get this error, go to the “Extensions and Updates” option of the Tools menu:

And select the update for the NuGet Package Manager

After the update the package installation should work.

Enabling Single Sign-On for Windows 8 Azure Mobile Apps

CarlosFigueira — Tue, 05 Feb 2013 01:04:44 GMT

One of the common complaints about the authentication support in the Windows 8 SDK for Azure Mobile Services is that regardless of whether the user selects the “save password” / “remember me” checkbox when entering their credentials, every time the application starts and the user needs to authenticate, they have to enter their credentials all over again. Currently the only way to enable the single sign-on (SSO) for Win8 apps would be to use the Live SDK (or some native SDK from the other providers, such as Facebook or Google) and then only authenticate with Azure Mobile Services with the token received from that SDK.

We recently released an update of the Azure Mobile Services client SDK for Windows 8 (along with an update to the service runtime), and this feature is now available. To enable single sign-on, we need to use a new overload to the LoginAsync (or login from the JavaScript client), and we also need to set the package SID of the Windows 8 application from the store. Let’s see how I can change my application to use this new feature. This is my current app. It currently uses Facebook authentication (a simple app which gets data from the graph API about the logged in user), but if I were using the other three authentication providers the steps to enable SSO would be the same.

The code for the application (in one method) is the following:

 private async void btnGetData_Click_1(object sender, RoutedEventArgs e)
{
    try
    {
        var user = await MobileService.LoginAsync(MobileServiceAuthenticationProvider.Facebook);
        AddToDebug("Logged in as {0}", user.UserId);
        var table = MobileService.GetTable("UserIdentity");
        var item = new JsonObject();
        await table.InsertAsync(item);
        AddToDebug("Item: {0}", item.Stringify());
        var fbAccessCode = item["identities"].GetObject()["facebook"].GetObject()["accessToken"].GetString();
        var httpClient = new HttpClient();
        var resp = await httpClient.GetAsync("https://graph.facebook.com/me?access_token=" + fbAccessCode);
        var body = await resp.Content.ReadAsStringAsync();
        JsonObject bodyJson = JsonObject.Parse(body);
        foreach (var key in bodyJson.Keys)
        {
            AddToDebug("{0}: {1}", key, bodyJson[key].Stringify());
        }
    }
    catch (Exception ex)
    {
        AddToDebug("Error: {0}", ex);
    }
}
 

First, the change in the client, which is the simple one: just use the new overload of the LoginAsync method – see the change below. Everything else remains exactly the same.

 var user = await MobileService.LoginAsync(MobileServiceAuthenticationProvider.Facebook, true);
AddToDebug("Logged in as {0}", user.UserId);
var table = MobileService.GetTable("UserIdentity");
 

Now, for the required server change. To enable this feature, we actually need to have a package id for the application in the Windows 8 store. If you still haven’t created an application on the Windows Store, you can follow the instructions in the tutorial for enabling push notifications. So far I hadn’t needed to do that, although once my app was ready for publishing to the store I’d had to do that anyway. So let me create one now:

After the application is created (I’ll call mine “blogpost20130204”) we can associate the VS project with the application in the store:

Next, we need to find the package SID for the store application. I actually find the dashboard for the Live Connect Developer Center easier to navigate than the Windows Store dashboard, so I’ll go from there:

After selecting the application, select “Edit Settings” and “API Settings” and it should show the package SID for my app:

Now that we have that value, we need to set it on the Azure Mobile Service portal for our application. Currently the only place where we can set it is on the “PUSH” tab, so that’s where we go, even through we’ll use it for authentication, and not push (in the future I’m sure there will be an update to the portal and that will be in its proper place).

Ok, now we have both the client and the server changes made. And that should be it. The username / password information should be now remembered by the client application!

More info / known issues

The old behavior and the new behavior are a byproduct of our usage of the Web Authentication Browser control in our Windows 8 SDK. Over the next couple of days I’ll publish a post with the internal details of our authentication, and how it relates to that control.

There’s currently one issue that we’re aware of: if you try to use both the Live SDK to authenticate the user to the app (a.k.a. client-side auth flow) and the single sign-on for Microsoft accounts (a.k.a. server-side auth flow), then the server-side flow won’t work. Notice that this isn’t something that one would normally do, as both do the same thing. Just something to be aware.

Inserting multiple items at once in Azure Mobile Services

CarlosFigueira — Wed, 02 Jan 2013 22:29:28 GMT

With the Azure Mobile Services client SDK, it’s fairly simple to insert an item into a table in Azure. Inserting multiple items can be also done fairly easily, simply inserting one at a time. But there are some scenarios where we want to actually do multiple insertions at once, to minimize the number of networking requests made from the client to the service. In the E2E Test Application that we have for our iOS client, the first thing we do in our query tests is to pre-populate a table with some data (if the table was empty), and since we don’t that to affect the test performance (we use about 250 items), we decided to insert them all at once. This has also been asked in our forums, so I decided to post how I went out to implement it.

The idea is fairly simple: instead of sending one item, we send a list of items which we want to be inserted. At the server side, instead of letting the default behavior for the insert operation kick in, we instead loop through the items and insert them one-by-one. Since the database and the instance running the mobile services are often located in the same datacenter, the latency between the two components is a lot smaller than from the client making the insert call and the mobile service, so we have some significant performance gains. And by having the logic to do multiple insertions at the server side, we don’t need to implement that at the client (doing it in managed code is trivial with the async / await keywords; doing that in iOS or in JavaScript not so much).

The way that I chose to implement the logic was that the service would receive not an array of items directly, but an object with one of its members being the array to be inserted. That’s easier to implement in the managed client (using an IDataMemberJsonConverter instead of an ICustomMobileServiceTableSerialization). For JavaScript it really doesn’t matter, since we’re dealing with JS objects in either case. And for the iOS client, we currently can only do that, since the insert operation only takes a dictionary (not an array) as an argument.

Let’s look at the server script. In this example, we’re adding populating our table called ‘iosMovies’, and the data to be inserted is an array in the “movies” property of the received item. The first thing we do is to check whether we actually need to do the insertion (that’s something specific to the scenario of my app; maybe in your case you always want to do the insertion, so that step won’t be necessary). By reading only the first element of the table, we can check whether the table is populated or not (since this script runs in the “insert’ operation, we assume that the table is either fully populated or it needs the items to be inserted). If the data is already there, we respond with an appropriate status. Otherwise, we proceed with the multiple insert operations.

 function insert(item, user, request) {
    var table = tables.getTable('iosMovies');
    table.take(1).read({
        success: function (items) {
            if (items.length > 0) {
                // table already populated
                request.respond(200, { id: 1, status: 'Already populated' });
            } else {
                // Need to populate the table
                populateTable(table, request, item.movies);
            }
        }
    });
}
 

Populating the table with multiple items is done by looping through the items to be inserted, and inserting them one by one (*). Since the insert operation in the table object is asynchronous, we can’t use a simple sequential loop; instead, we’ll insert each element when the callback for the previous one is called. After the last one is inserted, we can then respond to the client with an appropriate status.

There’s one extra step that we had to do in this example, which caught me by surprise at first – updating a field of type Date (or DateTime in the managed client, or NSDate in the iOS client). When I first ran that code, the table was created, and the data was seemingly correct. But when I checked the type of the columns in the database, my field ‘ReleaseDate’ had been translated into a column of type ‘string’. ‘datetime‘ is a supported primitive type in Azure Mobile Services, so I was expecting the column to have the appropriate value. The problem is that dates and strings are transmitted using the same type in JSON, string (there’s no such a thing as a “JSON date”). If a string value arrives with a specific format (the ISO 8601 date format, with millisecond precision), then the Mobile Service runtime converts that value into a JavaScript Date object. However, that conversion is only done in top-level objects – it doesn’t traverse the whole incoming object graph (for performance reasons), so in my case my “date” value ended up as a normal string. To fix that, prior to inserting the data we fixed that by converting the ‘ReleaseDate’ field in in the input array.

 function populateTable(table, request, films) {
    var index = 0;
    films.forEach(changeReleaseDate);
    var insertNext = function () {
        if (index >= films.length) {
            request.respond(201, { id: 1, status: 'Table populated successfully' });
        } else {
            var toInsert = films[index];
            table.insert(toInsert, {
                success: function () {
                    index++;
                    if ((index % 20) === 0) {
                        console.log('Inserted %d items', index);
                    }
 
                    insertNext();
                }
            });
        }
    };
 
    insertNext();
}
 
function changeReleaseDate(obj) {
    var releaseDate = obj.ReleaseDate;
    if (typeof releaseDate === 'string') {
        releaseDate = new Date(releaseDate);
        obj.ReleaseDate = releaseDate;
    }
}
 

So that’s the script for the server. Now for the client code. Let’s look at the flavors we have. First, managed code, where we have the classes for that data:

 [DataTable(Name = "w8Movies")]
public class AllMovies
{
    [DataMember(Name = "id")]
    public int Id { get; set; }
    [DataMember(Name = "status")]
    public string Status { get; set; }
    [DataMember(Name = "movies")]
    [DataMemberJsonConverter(ConverterType = typeof(MovieArrayConverter))]
    public Movie[] Movies { get; set; }
}
 
public class MovieArrayConverter : IDataMemberJsonConverter
{
    public object ConvertFromJson(IJsonValue value)
    {
        // unused
        return null;
    }
 
    public IJsonValue ConvertToJson(object instance)
    {
        Movie[] movies = (Movie[])instance;
        JsonArray result = new JsonArray();
        foreach (var movie in movies)
        {
            result.Add(MobileServiceTableSerializer.Serialize(movie));
        }
 
        return result;
    }
}
 
[DataTable(Name = "w8Movies")]
public class Movie
{
    public int Id { get; set; }
    public string Title { get; set; }
    public int Duration { get; set; }
    public DateTime ReleaseDate { get; set; }
    public int Year { get; set; }
    public bool BestPictureWinner { get; set; }
    public string Rating { get; set; }
}
 

And to insert multiple movies at once, we create an instance of our type which holds the array, and call insert on that object:

 Func<int, int, int, DateTime> createDate = 
    (y, m, d) => new DateTime(y, m, d, 0, 0, 0, DateTimeKind.Utc);
AllMovies allMovies = new AllMovies
{
    Movies = new Movie[]
    {
        new Movie { 
            BestPictureWinner = false, 
            Duration = 142, Rating = "R", 
            ReleaseDate = createDate(1994, 10, 14), 
            Title = "The Shawshank Redemption", 
            Year = 1994 },
        new Movie { 
            BestPictureWinner = true,
            Duration = 175, Rating = "R", 
            ReleaseDate = createDate(1972, 3, 24),
            Title = "The Godfather", 
            Year = 1972 },
        new Movie { 
            BestPictureWinner = true, 
            Duration = 200, Rating = "R",
            ReleaseDate = createDate(1974, 12, 20), 
            Title = "The Godfather: Part II", 
            Year = 1974 },
        new Movie { 
            BestPictureWinner = false, 
            Duration = 168, Rating = "R", 
            ReleaseDate = createDate(1994, 10, 14), 
            Title = "Pulp Fiction", 
            Year = 1994 },
    }
};
try
{
    var table = MobileService.GetTable<AllMovies>();
    await table.InsertAsync(allMovies);
    AddToDebug("Status: {0}", allMovies.Status);
}
catch (Exception ex)
{
    AddToDebug("Error: {0}", ex);
}
 

For Objective-C, we don’t need to create the types, so we can use the NSDictionary and NSArray classes directly. The implementation of the ‘getMovies’ method can be found in our GitHub repository.

 NSArray *movies = [ZumoQueryTestData getMovies];
NSDictionary *item = @{@"movies" : movies};
MSTable *table = [client getTable:queryTestsTableName];
[table insert:item completion:^(NSDictionary *item, NSError *error) {
    if (error) {
        NSLog(@"Error populating table: %@", error);
    } else {
        NSLog(@"Table is populated and ready for query tests");
    }
}];
 

Similarly for JavaScript, we can just use “regular” objects and arrays to insert the data:

 var table = client.getTable('w8Movies');
var allMovies = {
    movies: [
        {
            BestPictureWinner: false,
            Duration: 142,
            Rating: "R",
            ReleaseDate: new Date(Date.UTC(1994, 10, 14)),
            Title: "The Shawshank Redemption",
            Year: 1994
        },
        {
            BestPictureWinner: true,
            Duration: 175,
            Rating: "R",
            ReleaseDate: new Date(Date.UTC(1972, 3, 24)),
            Title: "The Godfather",
            Year: 1972
        },
        {
            BestPictureWinner: true,
            Duration: 200,
            Rating: "R",
            ReleaseDate: new Date(Date.UTC(1974, 12, 20)),
            Title: "The Godfather: Part II",
            Year: 1974
        },
        {
            BestPictureWinner: false,
            Duration: 168,
            Rating: "R",
            ReleaseDate: new Date(Date.UTC(1994, 10, 14)),
            Title: "Pulp Fiction",
            Year: 1994
        }
    ]
};
table.insert(allMovies).done(function (inserted) {
    document.getElementById('result').innerText = inserted.status || allMovies.status;
});
 

That’s about it. This is one of the ways we have to prevent multiple networking requests between the client and the mobile service to perform multiple insertions at once.

(*) Bonus info: As I mentioned before, this still does multiple calls, between the mobile service and the database, but since they’re co-located, the latency is small. You also can make really only one call, from the service to the database, by using the mssql object, and creating one insert request for multiple rows at once. You’d need to do the translation between the data types and the SQL expression yourself, and use the “union all” trick to create a temporary table and insert from that table – see an example below. Notice that you’d also need to create the table columns (if they don’t exist yet), since the dynamic schema feature doesn’t work with the mssql object – once you go down to that level, you’re in full control of the database communication.

 INSERT INTO w8Movies (Title, [Year], Duration, BestPictureWinner, Rating, ReleaseDate)
SELECT 'The Shawshank Redemption', 1994, 142, FALSE, 'R', '1994-10-14'
UNION ALL
SELECT 'The Godfather', 1972, 175, TRUE, 'R', '1972-03-24'
UNION ALL
SELECT 'The Godfather: Part II', 1974, 200, TRUE, 'R', '1974-12-20'
UNION ALL
SELECT 'Pulp Fiction', 1994, 168, TRUE, 'R', '1994-10-14'
 

So which one to use? I really didn’t see much advantage in going full SQL for my scenario. Once a communication between the mobile service and the database is successful, chances are that over the next second or so the subsequent ones will be as well, so doing ~250 insert operations, although not really an atomic transaction, has a very good chance of being one. As usual, it may vary according to the scenario of your application.

Getting user information on Azure Mobile Services

CarlosFigueira — Thu, 25 Oct 2012 05:39:27 GMT

With the introduction of the server-side authentication flow (which I mentioned in my last post), it’s now a lot simpler to authenticate users with Windows Azure Mobile Services. Once the LoginAsync / login / loginViewControllerWithProvider:completion: method / selector completes, the user is authenticated, the MobileServiceClient / MSClient object will hold a token that is used for authenticating requests, and it can now be used to access authentication-protected tables. But there’s more to authentication than just getting a unique identifier for a user – we can also get more information about the user from the providers we used to authenticate, or even act on their behalf if they allowed the application to do so.

With Azure Mobile Services you can still do this. However, the property is not available at the user object stored at the client – the only property it exposes is the user id, which doesn’t give the information that the user authorized the providers to give. This post will show, for the supported providers, how to get access to some of their properties, using their specific APIs.

User identities

The client objects doesn’t expose any of that information to the application, but at the server side, we can get what we need. The User object which is passed to all scripts has now a new function, getIdentities(), which returns an object with provider-specific data which can be used to query their user information. For example, for a user authenticated with a Facebook credential in my app, this is the object returned by calling user.getIdentities():

{
    "facebook":{
        "userId":"Facebook:my-actual-user-id",
        "accessToken":"the-actual-access-token"
    }
}

And for Twitter:

{
    "twitter":{
        "userId":"Twitter:my-actual-user-id",
        "accessToken":"the-actual-access-token",
        "accessTokenSecret":"the-actual-access-token-secret"
    }
}

Microsoft:

{
    "microsoft":{
        "userId":"MicrosoftAccount:my-actual-user-id",
        "accessToken":"the-actual-access-token"
    }
}

Google:

{
    "google":{
        "userId":"Google:my-actual-user-id",
        "accessToken":"the-actual-access-token"
    }
}

Each of those objects has the information that we need to talk to the providers API. So let’s see how we can talk to their APIs to get more information about the user which has logged in to our application. For the examples in this post, I’ll simply store the user name alongside the item which is being inserted

Talking to the Facebook Graph API

To interact with the Facebook world, you can either use one of their native SDKs, or you can talk to their REST-based Graph API. To talk to it, all we need is a HTTP client, and we do have the nice request module which we can import (require) on our server scripts. To get the user information, we can send a request to https://graph.facebook.com/me, passing the access token as a query string parameter. The code below does that. It checks whether the user is logged in via Facebook; if so, it will send a request to the Graph API, passing the token stored in the user identities object. If everything goes right, it will parse the result (which is a JSON object), and retrieve the user name (from its “name” property) and store in the item being added to the table.

 function insert(item, user, request) {
    item.UserName = "<unknown>"; // default
    var identities = user.getIdentities();
    var req = require('request');
    if (identities.facebook) {
        var fbAccessToken = identities.facebook.accessToken;
        var url = 'https://graph.facebook.com/me?access_token=' + fbAccessToken;
        req(url, function (err, resp, body) {
            if (err || resp.statusCode !== 200) {
                console.error('Error sending data to FB Graph API: ', err);
                request.respond(statusCodes.INTERNAL_SERVER_ERROR, body);
            } else {
                try {
                    var userData = JSON.parse(body);
                    item.UserName = userData.name;
                    request.execute();
                } catch (ex) {
                ��   console.error('Error parsing response from FB Graph API: ', ex);
                    request.respond(statusCodes.INTERNAL_SERVER_ERROR, ex);
                }
            }
        });
    } else {
        // Insert with default user name
        request.execute();
    }
}
 

With the access token you can also call other functions on the Graph API, depending on what the user allowed the application to access. But if all you want is the user name, there’s another way to get this information: the userId property of the User object, for users logged in via Facebook is in the format “Facebook:<graph unique id>”. You can use that as well, without needing the access token, to get the public information exposed by the user:

 function insert(item, user, request) {
    item.UserName = "<unknown>"; // default
    var providerId = user.userId.substring(user.userId.indexOf(':') + 1);
    var identities = user.getIdentities();
    var req = require('request');
    if (identities.facebook) {
        var url = 'https://graph.facebook.com/' + providerId;
        req(url, function (err, resp, body) {
            if (err || resp.statusCode !== 200) {
                console.error('Error sending data to FB Graph API: ', err);
                request.respond(statusCodes.INTERNAL_SERVER_ERROR, body);
            } else {
                try {
                    var userData = JSON.parse(body);
                    item.UserName = userData.name;
                    request.execute();
                } catch (ex) {
                    console.error('Error parsing response from FB Graph API: ', ex);
                    request.respond(statusCodes.INTERNAL_SERVER_ERROR, ex);
                }
            }
        });
    } else {
        // Insert with default user name
        request.execute();
    }
}
 

The main advantage of this last method is that it can not only be used in the client-side as well.

 private async void btnFacebookLogin_Click_1(object sender, RoutedEventArgs e)
{
    await MobileService.LoginAsync(MobileServiceAuthenticationProvider.Facebook);
    var userId = MobileService.CurrentUser.UserId;
    var facebookId = userId.Substring(userId.IndexOf(':') + 1);
    var client = new HttpClient();
    var fbUser = await client.GetAsync("https://graph.facebook.com/" + facebookId);
    var response = await fbUser.Content.ReadAsStringAsync();
    var jo = JsonObject.Parse(response);
    var userName = jo["name"].GetString();
    this.lblTitle.Text = "Multi-auth Blog: " + userName;
}
 

That’s it for Facebook; let’s move on to another provider.

Talking to the Google API

The code for the Google API is fairly similar to the one for Facebook. To get user information, we send a request to https://www.googleapis.com/oauth2/v1/userinfo, again passing the access token as a query string parameter.

 function insert(item, user, request) {
    item.UserName = "<unknown>"; // default
    var identities = user.getIdentities();
    var req = require('request');
    if (identities.google) {
        var googleAccessToken = identities.google.accessToken;
        var url = 'https://www.googleapis.com/oauth2/v1/userinfo?access_token=' + googleAccessToken;
        req(url, function (err, resp, body) {
            if (err || resp.statusCode !== 200) {
                console.error('Error sending data to Google API: ', err);
                request.respond(statusCodes.INTERNAL_SERVER_ERROR, body);
            } else {
                try {
                    var userData = JSON.parse(body);
                    item.UserName = userData.name;
                    request.execute();
                } catch (ex) {
                    console.error('Error parsing response from Google API: ', ex);
                    request.respond(statusCodes.INTERNAL_SERVER_ERROR, ex);
                }
            }
        });
    } else {
        // Insert with default user name
        request.execute();
    }
}
 

Notice that the code is so similar that we can just merge them into one:

 function insert(item, user, request) {
    item.UserName = "<unknown>"; // default
    var identities = user.getIdentities();
    var url;
    if (identities.google) {
        var googleAccessToken = identities.google.accessToken;
        url = 'https://www.googleapis.com/oauth2/v1/userinfo?access_token=' + googleAccessToken;
    } else if (identities.facebook) {
        var fbAccessToken = identities.facebook.accessToken;
        url = 'https://graph.facebook.com/me?access_token=' + fbAccessToken;
    }
 
    if (url) {
        var requestCallback = function (err, resp, body) {
            if (err || resp.statusCode !== 200) {
                console.error('Error sending data to the provider: ', err);
                request.respond(statusCodes.INTERNAL_SERVER_ERROR, body);
            } else {
                try {
                    var userData = JSON.parse(body);
                    item.UserName = userData.name;
                    request.execute();
                } catch (ex) {
                    console.error('Error parsing response from the provider API: ', ex);
                    request.respond(statusCodes.INTERNAL_SERVER_ERROR, ex);
                }
            }
        }
        var req = require('request');
        var reqOptions = {
            uri: url,
            headers: { Accept: "application/json" }
        };
        req(reqOptions, requestCallback);
    } else {
        // Insert with default user name
        request.execute();
    }
}
 

And with this generic framework we can add one more:

Talking to the Windows Live API

Very similar to the previous ones, just a different URL. Most of the code is the same, we just need to add a new else if branch:

 if (identities.google) {
    var googleAccessToken = identities.google.accessToken;
    url = 'https://www.googleapis.com/oauth2/v1/userinfo?access_token=' + googleAccessToken;
} else if (identities.facebook) {
    var fbAccessToken = identities.facebook.accessToken;
    url = 'https://graph.facebook.com/me?access_token=' + fbAccessToken;
} else if (identities.microsoft) {
    var liveAccessToken = identities.microsoft.accessToken;
    url = 'https://apis.live.net/v5.0/me/?method=GET&access_token=' + liveAccessToken;
}
 

And the user name can be retrieved in the same way as the others – notice that this is true because all three providers seen so far return the user name in the “name” property, so we didn’t need to change the callback code.

Getting Twitter user data

Twitter is a little harder than the other providers, since it needs two things from the identity (access token and access token secret), and one of the request headers needs to be signed. For simplicity here I’ll just use the user id trick as we did for Facebook:

 if (identities.google) {
    var googleAccessToken = identities.google.accessToken;
    url = 'https://www.googleapis.com/oauth2/v1/userinfo?access_token=' + googleAccessToken;
} else if (identities.facebook) {
    var fbAccessToken = identities.facebook.accessToken;
    url = 'https://graph.facebook.com/me?access_token=' + fbAccessToken;
} else if (identities.microsoft) {
    var liveAccessToken = identities.microsoft.accessToken;
    url = 'https://apis.live.net/v5.0/me/?method=GET&access_token=' + liveAccessToken;
} else if (identities.twitter) {
    var userId = user.userId;
    var twitterId = userId.substring(userId.indexOf(':') + 1);
    url = 'https://api.twitter.com/users/' + twitterId;
}
 

And since the user name is also stored in the “name” property of the Twitter response, the callback doesn’t need to be modified.

Accessing provider APIs from the client

So far I’ve shown how you can get user information from the script, and some simplified version of it for the client side (for Facebook and Twitter). But what if we want the logic to access the provider APIs to live in the client, and just want to retrieve the access token which is stored in the server? Right now, there’s no clean way of doing that (no “non-CRUD operation” support on Azure Mobile Services), so what you can do is to create a “dummy” table that is just used for that purpose.

In the portal, create a new table for your application – for this example I’ll call it Identities, set the permissions for Insert / Delete and Update to “Only Scripts and Admins” (so that nobody will insert any data in this table), and for Read set to “Only Authenticated Users”

Now in the Read script, return the response as requested by the caller, with the user identities stored in a field of the response. For the response: if a specific item was requested, return only one element; otherwise return a collection with only that element:

 function read(query, user, request) {
    var result = {
        id: query.id,
        identities: user.getIdentities()
    };
    if (query.id) {
        request.respond(200, result);
    } else {
        request.respond(200, [result]);
    }
}
 

And we can then get the identities on the client as a JsonObject by retrieving data from that “table”.

 var table = MobileService.GetTable("Identities");
var response = await table.ReadAsync("");
var identities = response.GetArray()[0].GetObject();
 

Notice that there’s no LookupAsync method on the “untyped” table, so the result is returned as an array; it’s possible that this will be added to the client SDK in the future, so we won’t need to get the object from the (single-element) array, receiving the object itself directly.

Wrapping up

The new multi-provider authentication support added in Azure Mobile Services made it quite easy to authenticate users to your mobile application, and it also gives you the power to access the provider APIs. If you have any comments or feedback, don’t hesitate to send either here on in the Azure Mobile Services forum.

Troubleshooting authentication issues in Azure Mobile Services

CarlosFigueira — Tue, 23 Oct 2012 02:28:15 GMT

With the announcement last week in ScottGu’s blog, Azure Mobile Service now supports different kinds of authentication in addition to authentication using the Windows Live SDK which was supported at first. You can now authenticate the users of your applications using Facebook, Google, Twitter and even Microsoft Accounts (formerly known as Live IDs) without any native SDK for those providers, just like on web applications. In fact, the authentication is done exactly by the application showing an embedded web browser control which talks to the authentication provider’s websites. In the example below, we see an app using Twitter to authenticate its user.

The tutorial Getting Started with Users shows how to set up the authentication, including creating applications in each of the supported providers. It’s a great tutorial, and when everything works fine, then great, we’ll add the authentication code to the client (quite small, especially for managed Windows Store apps), users can login and we’re all good. However, there are cases where we just don’t get the behavior we want, and we end up with a client which just can’t correctly authenticate.

There are a number of issues which may be causing this problem, and the nature of authentication of connected mobile applications, with three distinct components (the mobile app itself, the Azure Mobile Service, and the identity provider) makes debugging it harder than simple applications.

There is, however, one nice trick which @tjanczuk (who actually implemented this feature) taught me and can make troubleshooting such problems a little easier. What we do essentially is to remove one component of the equation (the mobile application), to make debugging the issue simpler. The trick is simple: since the application is actually hosting a browser control to perform the authentication, we’ll simply use a real browser to do that. By talking to the authentication endpoints of the mobile service runtime directly, we can see what’s going on behind the scenes of the authentication protocol, and hopefully fix our application.

The authentication endpoint

Before we go into broken scenarios, let’s talk a about the authentication endpoint which we have in the Azure Mobile Services runtime. As of the writing of this post the REST API Reference for Windows Azure Mobile Services has yet to be updated for the server-side (web-based) authentication support, so I’ll cover it briefly here.

The authentication endpoint for an Azure Mobile Service responds to GET requests to https://<service-name>.azure-mobile.net/login/<providerName>, where <providerName> is one of the supported authentication providers (currently “facebook”, “google”, “microsoftaccount” or “twitter”). When a browser (or the embedded browser control) sends a request to that address, the Azure Mobile Service runtime will respond with a redirect (HTTP 302) response to the appropriate page on the authentication provider (for example, the twitter page shown in the first image of this post). Once the user enters valid credentials, the provider will redirect it back to the Azure Mobile Service runtime with its specific authentication token. At that time, the runtime will validate those credentials with the provider, and then issue its own token, which will be used by the client as the authentication token to communicate with the service.

The diagram above shows a rough picture of the authentication flow. Notice that the client may send more than one request to the authentication provider, as it’s often first asks the user to enter its credentials, then (at least once per application) asks the user to allow the application to use its credentials. What the browser control in the client does is to monitor the URL to where it’s navigating, and when it sees that it’s navigating to the /login/done endpoint, it will know that the whole authentication “dance” has finished. At that point, the client can dispose the browser control and store the token to authenticate future requests it sends.

This whole protocol is just a bunch of GET requests and redirect responses. That’s something that a “regular” browser can handle pretty well, so we can use that to make sure that the server is properly set up. So we’ll now see some scenarios where we can use a browser to troubleshoot the server-side authentication. For this scenario I prefer to use either Google Chrome or Mozilla Firefox, since they can display JSON payloads in the browser itself, without needing to go to the developer tools. With Internet Explorer you can also do that, but by default it asks you to save the JSON response in a file, which I personally find annoying. Let’s move on to some problems and how to identify them.

Missing configuration

Once the mobile service application is created, no identity provider credentials are set in the portal. If the authentication with a specific provider is not working, you can try browsing to it. In my application I haven’t set the authentication credentials for Google login, so I’ll get a response saying so in the browser.

To fix this, go to the portal, and enter the correct credentials. Notice that there was a bug in the portal until last week where the credentials were not being properly propagated to the mobile service runtime (it has since been fixed). If you added the credentials once this was released, you can try removing them, then adding them again, and it should go through.

Missing redirect URL

In all of the providers you need to set the redirect URL so that the provider knows to, after authenticating the user, redirect it back to the Azure Mobile Service login page. By using the browser we can check see that error clearer than when using an actual mobile application. For example, this is what we get when we forget to set the “Site URL” property on Facebook, after we browse to https://my-application-name.azure-mobile.net/login/facebook:

And for Windows Live (a.k.a. Microsoft Accounts), after browsing to https://my-application-name.azure-mobile/login/microsoftaccount:

Not as clear an error as the one from Facebook, but if you look at the URL in the browser (may need to copy/paste to a text editor to see better):

https://login.live.com/err.srf?lc=1033
    #error=invalid_request&
              error_description=The%20provided%20value%20for%20the%20input%20parameter%20'redirect_uri'%20is%20not%20valid.%20The%20expected%20value%20is%20'https://login.live.com/oauth20_desktop.srf'%20or%20a%20URL%20which%20matches%20the%20redirect%20URI%20registered%20for%20this%20client%20application.&
               state=13cdf4b00313a8b4302f6

It will have an error description saying that “The provided value for the input parameter 'redirect_uri' is not valid”.

For other providers the experience is similar.

Invalid credentials

Maybe when copying the credentials from the provider site to the Windows Azure Mobile Services portal, the authentication will also fail, but only after going through . In this case, most providers will just say that there is a problem in the request, so one spot to look for issues is on the credentials to see if the ones in the portal match the ones in the provider page. Here are some examples of what you’ll see in the browser when that problem happens. Twitter will mention a problem with the OAuth request (OAuth being the protocol used in the authentication):

Facebook, Microsoft and Google accounts have different errors depending on whether the error is at the client / app id or the client secret. If the error is at the client id, then the provider will display an error right away. For example, Microsoft accounts will show its common error page

But the error description parameter in the URL shows the actual problem: “The client does not exist. If you are the application developer, configure a new application through the application management site at https://manage.dev.live.com/.” Facebook isn’t as clear, with a generic error.

Google is clearer, showing the error right on the first page:

Now, when the client / app id is correct, but the problem is on the app secret, then all three providers (Microsoft, Facebook, Google) will show the correct authentication page, asking for the user credentials. Only when the authentication with the provider is complete, and it redirects it back to the Azure Mobile Service (step 5 in the authentication flow diagram above), and the runtime tries to validate the token with the provider is that it will show the error. Here are the errors which the browser will show in this case. First, Facebook:

Google:

Microsoft:

Other issues

I’ve shown the most common problems which we’ve seen here which we can control. But as usual, there may be times where things just don’t work – network connectivity issues, blackouts on the providers. As with all distributed systems, those issues can arise from time to time which are beyond the control of the Azure Mobile Services. For those cases, you should confirm that those components are working correctly as well.

When everything is fine

Hopefully some of the troubleshoots steps I’ve shown here you’ve been able to fix your server-side authentication with the Azure Mobile Services. If that’s the case, you should see this window.

And with that window (code: 200), you’ll know that, at least the server / provider portion of the authentication dance is ready. With the simple client API, hopefully that will be enough to get your application authentication support.