In my previous post on the protocol versioning scheme for Astoria, I've explained the two HTTP headers DataServiceVersion and MaxDataServiceVersion. These headers are used by the server and client library to determine how and what to communicate with each other. This mechanism is defined by the Astoria Protocol and our library's implementation does not require any user interactions. There is no public APIs to give users direct control of mechanism either. In fact, in the most common scenario where the server and the client versions match, one can simply ignore the existence of version control. However, if you ever need to setup a data service where the server and the consumer are built from two different versions of Astoria, you'll likely want to know what is expected to work and what is not.

With the release of CTP2, we have introduced a couple of new features on the server. In the table below I'll list these features and their behavior when it comes to cross-version calls. In the table, minimum requesting DataServiceVersion means the client must send a DSV header that's greater than or equal to this value, and the response DSV means the server will response with this version (which also means the MDSV header sent by the client must be less than or equal to this value).

The rule of thumb here is we require the requesting DSV to be 2.0 for all new URI constructs ($inlinecount, $count, $skipToken, $select), and the response DSV will be 2.0 if the content is incompatible with V1 response. For example, for Server Driven Paging, when the server truncates the result set, we will bump the response DSV to 2.0, because V1.0 client will ignore the continuation token and thus unable to receive the full results.

The Max Protocol Version configuration

For folks out there wanting to use V2 service and want their service to be V1 protocol compliant, you can "turn off" the features that may require a DSV of 2.0 by setting the MaxProtocolVersion configuration*. This setting is available only during service initialization:

public static void InitializeService(DataServiceConfiguration config)
{
    config.SetEntitySetAccessRule("*", EntitySetRights.All);
    config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V1;
}  

You can think of this as the service-side enforced "MaxDataServiceVersion" header. When this is set, any request that requires a DSV greater than 1.0 will fail even if the proper headers are present. The error message you get in this case is "The response requires that version 2 of the protocol be used, but the MaxProtocolVersion of the data service is set to DataServiceProtocolVersion.V1", and the status code of the reply is BadRequest(400).

*NOTE: We have decided to ship V2 with MaxProtocolVersion default to V1, for safety and compatibility reasons. You MUST explicitly set this to V2 in order to use any of the V2 features.

Although the Astoria client is really designed to query data existing in some remote service location, it does keep a local copy of all entities it has seen. If the MergeOption is not set to NoTracking, you can find all of the entities currently been tracked in the DataServiceContext.Entities property. The property is defined as:

ReadOnlyCollection<EntityDescriptor> Entities

where EntityDescriptor is a class that gives you access to the entity instance, the ETag and the state of the entity (changed, deleted, etc). The easiest way to query local copies of entities is to use LINQ to Objects. For example, if you are looking for a Customer who’s CustomerID is 0, and you want to see if it exists locally first, then you can write:

var q = from ed in Context.Entities
        where ed.Entity is Customer 
              && ((Customer)ed.Entity).ID == 0
        select (Customer)ed.Entity;

Customer cust = q.FirstOrDefault();

In the above example, when cust is not null, you can work with the copy and still have the context track the changes you make to the entity. If cust is null, you can then query the remote service and the next time you run this again, a local copy will exist.

The key to this is, of course, is to have a singleton DataServiceContext for all of your data access needs. In a sense, you are keeping a local "data store” and when possible, use it’s data instead of the remote one. Ultimately, you would want to have it retrieve remote data on demand.

A simple data store may look like this:

public class DataStore
{
    static DataServiceContext ctx = new DataServiceContext(new Uri(“ServiceUri”, UriKind.Absolute));

    public static DataServiceContext Context
    {
        get { return ctx; }
    }

    public static IQueryable<T> QueryLocal<T>()
    {
        return (from ed in ctx.Entities.AsQueryable()
                where ed.Entity is T
                select (T)ed.Entity);
    }

    public static IQueryable<T> QueryLocal<T>(Expression<Func<T,bool>> predicate)
    {
        return (from ed in ctx.Entities.AsQueryable()
                where ed.Entity is T
                select (T)ed.Entity).Where(predicate);
    }
}

With this setup, you can query the remote store with DataStore.Context.CreateQuery<T>(), and you can query the local store with DataStore.QueryLocal<T>(). The above Customer example simply become:

Customer cust = DataStore.QueryLocal<Customer>(c=>c.ID==0).FirstOrDefault();

Taking it one step further, you can imagine having a unified “QueryStore<T>()” function that queries local data first, and if it didn’t find anything, it would go ahead and ask the service for data. This approach can significantly reduce the amount of data you need to move across the wire for your service.

Of course, just like everything else in life, this approach also has trade-offs. The biggest one comes from the fact that the local copies are not “partitioned” or “indexed” in anyway. This means even if all you are looking for is one customer, you may have to loop through thousands of other irrelevant entities. In situation like this, you must make a decision to either minimize the bandwidth or maximize the performance – it could be faster if you just query the remote service instead. In short, whether having a local data store suites your needs depends on the structure of your data. Applied currently, it can significantly optimize your data service experience.

Now that CTP1 of Astoria has been made public, we have two different versions of server and client that can interact with each other. How does Astoria handle the versioning for the various possible interactions? While this question probably does not concern normal users who sticks to our client library, it is something worth to know especially when you run into a bad request that tells you “your version is not supported”.

If you have ever looked at the headers of the HTTP request and response between the client and the server in V1, you’d likely noticed two headers in there: DataServiceVersion and MaxDataServiceVersion. These two headers are the magical tags that controls what request we would accept and what request we would fail. In short, DataServiceVersion defines the protocol version of the particular request/response it is associated with, and MaxDataServiceVersion is a request-only header that tells the server the maximum protocol version of the response the client is willing to accept.

So take our V1 client for example. It can only understand Data Service protocol version “1.0”, so for all the requests, it will put:

DataServiceVersion: 1.0;
MaxDataServiceVersion: 1.0;

Any Astoria service will be able to look at these headers and comply to the specified protocol. For example, in this case a request to “/Customers/$count” would fail, because the “$count” segment does not exist in protocol version 1.0. Another example is “Friendly Feeds”, when turned on, it will require that the client can at least understand protocol version “2.0” regardless of the request’s version (since the format of the feed changes and thus will break the V1 client). In this case the request will also fail, but because of the MaxDataServiceVersion is set to 1.0.

The idea behind defining a protocol version for each request is to eliminate possible ambiguous situations if one ever pops up in the future. For example, let’s say the behavior of “$count” end point is changed in Astoria V5. In this situation, to avoid breaking every lower version clients, the service will rely on the DataServiceVersion header to figure out what is the actual meaning when it sees a request to “$count”.

Not every client choose to set these headers. IE for example, can browse our services and it doesn’t understand a thing about Astoria protocol. So when the one of the headers is not present, Astoria’s implementation will set it to the newest version it can understand.

Going back from the service to the client is easier. The server will only write the header “DataServiceVersion”, which tells the client the protocol associated with the particular response. Of course, the server will only do so if the response version number is less than or equal to the MaxDataServiceVersion of the request. So in theory the response is “version-proof” in a sense that the client should at least be able to understand the result. However, this information becomes useful in the ambiguous situation mentioned above, where the client should actively check the response version and figure out the correct meaning for it.

update: the following is not longer true. We now use protocol version "2.0" for all new features. See Versioning Protocol of Astoria Part II for details.

One side note on the current protocol version used in V1.5 CTP1: you’ll notice that all versioning headers is either 1.0 or 2.0, but our product version is “1.5”. This mismatch could be confusing to many people and was unintentional. In the final release of V1.5, all version 2.0 headers will be changed to “1.5” instead.

There are many situations where you may encounter an undesired 404 “Resource Not Found” exception while querying against an ADO.net Data Service. For example, you may be writing a Linq query where the filter criteria is specified at runtime by the end users. When your query becomes input sensitive, there is no guarantee that the resource you are querying for exists on the server. In such situations, you may want your application to just ignore the exception thrown by the web request. In fact, when what you are looking for is not there, you probably expect an empty set rather than a big 404 on your screen.

As of V1.5, Astoria’s client can be set to silently “eat” the 404 exception, and just give back an empty collection. The switch is on the DataServiceContext itself, and it’s called “IgnoreResourceNotFoundException” (pretty self-explanatory name). When set to true, the client library will not longer throw DataServiceClientException when the server returns status code 404.

Consider the following example, where in a WinForm application, the user is allowed to query against the northwind database sitting at a remote service end point. The user can input any string in a textBox and that’ll be the CustomerID he’s looking for. You can now write:

context.IgnoreResourceNotFoundException = true;

var q = from c in context.CreateQuery<Customers>
        ("Customers")
        where c.CustomerID == textBox1.Text
        select c;

Customers cust = q.FirstOrDefault();
if (cust != null)
{
    textBox2.Text = cust.CompanyName;
}
else
{
    MessageBox.Show("The specified customer cannot be found");
}

Be careful when you are setting this property, since sometimes (especially during development) this can make your code harder to debug. A simple typo on the entity set name, for example, also causes 404 exceptions, and having IgnoreResourceNotFoundException turned on may just trick you into thinking the problem is elsewhere.

The recent CTP release of ADO.Net Data Services (Astoria) V1.5 contains a nice feature that allows result set counting on the server side. In this blog post, I’ll go over the various ways you can benefit from this feature in your data service.

Type of Row Count

Before going into details on how to use the feature, we should first take a look at what the server will count for you. There are two scenarios here: counting the total result set, and counting what the server actually put on the wire.

Counting the total result set is helpful when you need to know exactly how many entities exist in a set, regardless of any paging constrains you put on the query. Take the classic Northwind database for example. There are 91 Customers in the database, if you ask the server “How many customers are there in total”, you are asking for the total count on the set “Customers”, and the result will always be 91, even if you instruct the server to only give you the first 10 customers back (via the $top=10 query). This type of counting is very useful when you want to do client-driven paging. Since you must know the total number of entities before you can calculate how many page links you need to render on the control. Astoria supports counting the total result set via the $inlinecount=allpages option. As the name suggests, the count value will be returned inline with the actual result set.

The other type of counting is when you want to know before hand how many entities the server will (eventually) put on the wire when you execute the query. The reason I put “(eventually)” in that sentence will be explained later. This type of counting happens when you ask the server “How many orders will you give back if I want the first 100 orders made by customers ’ALFKI’”. The server will answer “6” in this case because there are only 6 orders under Customer ‘ALFKI’, but if you change the question to “how many orders will you give back if I want the first 5 orders made by ‘ALFKI’”, then the server will tell you “5”. This type of counting is supported in Astoria by the $count segment, and since you are only interested in the count value, the result is just a plaintext number.

In other words, $inlinecount=allpages is a query option that gives you the count neglect any paging effects ($skip, $top etc.), while $count segment will take explicitly stated paging effect into account.

Inline Counting

When you specify $inlinecount=allpages in the query option string, the server will respond will the normal syndication of result set (or a JSON block) plus the count value embedded in the results. In ATOM serialization, the count is returned in a feed level tag called “count”, under the “Metadata V2” namespace. For example, the query

"/Customers?$inlinecount=allpages&$top=5”

will give you the following tag plus 5 customers:

<m2:count xmlns:m2=”http://schemas.microsoft.com/ado/2008/11/ dataservices/metadata”>91</m2:count>

Inline Counting on the Client

On our client, if the resulting feed contains the count tag, you can extract the value by accessing the “TotalCount” property on the QueryOperationResponse object. Of course, if the count tag is not present, getting the property value will cause an InvalidOperationException to be thrown.

There are many ways for the client to generate a request to cause the server to respond with the count tag. We have provided a new API for the ALinq users called IncludeTotalCount(). The method exists on the DataServiceQuery class, and returns a new instance of DataServiceQuery that will cause the $inlinecount query option to be generated. For example, you can write:

var q = (from c in ctx.CreateQuery<Customers>(“/Customers”).IncludeTotalCount().Take(5) select c) as DataServiceQuery<Customers>;

var results = q.Execute() as QueryOperationResponse<Customers>;

long countValue = results.TotalCount;

An other way to achieve this is to use the AddQueryOption API and manually add “inlinecount” option with value “allpages”. Of course, if you just specify an URI with the correct counting option and directly execute it from the context, you will also be able to use the TotalCount property to access the count tag in the result.

It is also possible to batch requests generated by IncludeTotalCount. The individual count value can be accessed in each of the QueryOperationResponse object in the batch response.

Value Counting

The $count segment can be added to any entity sets on the server, the result is a plaintext response that represents the count of entities in that set. Different from inline counting , query options can be added to this segment to modify the number of entity to be counted. For example, given that there are 91 customers in total, the following table illustrates the behavior of $count

Since the response is in PlainText format, the Accept header should be compatible with “text/plain”. For example, “*/*” will work, but “application/json” will cause an exception to the thrown.

Value Counting on the Client

In V1, calling Count and LongCount on a DataServiceQuery<> will throw NotSupportedException. These two APIs are now implemented and mapped to the $count segment on the corresponding entity set. Note that these APIs will cause the query to execute immediately and returns the count value, hence it’s a synchronized operation and thus not available on the Silverlight Client. Here’s an example of how you can use these APIs:

var q = (DataServiceQuery<Customers>)(from c in ctx.CreateQuery<Customers>("Customers") select c);

long countValue = q.LongCount();

You can call Count or LongCount on any DataServiceQuery<> that doesn’t already have a counting option (i.e., one that’s generated by IncludeTotalCount).

Counting on Links Endpoint

The query option "inlinecount” and the “$count” segment can also be applied to $links end points. However there is no equivalent APIs on the client side. Here are some examples illustrating how:

/Customers(‘ALFKI’)/$links/Orders?$inlinecount=allpages

returns a PlainXML format with the <m2:count> tag embedded.

/Customers(‘ALFKI’)/$links/Orders/$count

returns a PlainText response “6” (there are 6 links to customer ALFKI’s orders)

Counting with Expansion

When you specify inline counting together with expansions (via $expand), the count value represents the number of entities that exists in the outermost set. Each entity will be expanded as normal, but there won’t be a count tag for each of the expanded set.

When you specify value counting ($count segment), expansion is entirely ignored.

Counting with Server Driven Paging

Server driven paging is a new feature in V1.5 that allows server side enforced paging. This means on SDP enabled services, you may be given back a partial set for any request you make, together with a link on where to get the next part of the set. Ideally, SDP will not affect counting, hence in the beginning of the article I wrote how many entities the server will "eventually" give back. However this is not the case right now. When you enable SDP, you will find that both $inlinecount and $count will be affected by the partial set effect. This however is a known issue and we are tracking it right now.

The ADO.net Data Services (Code name Astoria) V1.5 CTP1 bits has been released on download center, you can get to it here. While the bits are downloading, you can find out what's new on this release by reading the team blog.

Meanwhile, I'm planning on writing about some of the new features here soon. Check back for more :)