Vitek's Blog

Custom paging provider

2012-06-04T14:08:28Z

The OData protocol supports a feature called Server driven paging. It is used to limit the amount of data client can query with a single request while still providing a way from the client to get all the data (in multiple requests). A more detailed explanation is for available for example in this blog post.

WCF Data Services implements this in two possible ways. The easy to use built-in server driven paging implementation and the more complicated but powerful custom paging. The built-in server driven paging implementation simply puts a limit on the number of results returned for a specified entity set. The way to set it up and use it is described for example here or here.

But the protocol allows the server to split the results into multiple pages in any way, not just number of results. In fact clients don’t know how the server decided when to split the results. So you may want to limit the time it takes to execute the query, or the amount of memory consumed or maybe even limit the query on a subset of nodes (in a distributed system) and return a page per node, and so on. To support these scenarios the WCF Data Services allows custom implementation of the server driven paging. It is done by implementing the IDataServicePagingProvider interface. Let’s take a look how that might work.

Prerequisites

Custom paging can only be implemented if our service already implements custom metadata and query providers (IDataServiceMetadataProvider and IDataServiceQueryProvider). For samples on these take a look at this great series of posts.

The custom query provider will also have to implement custom IQueryable. We typically already have this when we implement custom metadata and query provider, but it’s not a requirement. For custom paging this is absolutely a requirement as there’s no practical way how to tweak the existing IQueryable implementations (like LINQ to EF, LINQ to SQL, LINQ to Objects, …) for this purpose. On the other hand, there’s nothing preventing us from implementing the IQueryable as a layer over an existing implementation.

And finally we implement the IDataServicePagingProvider interface and return its implementation from our IServiceProvider. The easiest way to describe how to implement the two methods the interface has is to describe the steps which the WCF Data Services takes to process paged queries.

How does it work

Let’s walk through a query processing for a simple query to /Customers.

1 – Request for the entity set

Client sends a request to the service for /Customers.

2 – Query processing

On the server the query gets processed and eventually is passed to the custom IQueryable implementation. This implementation must know that for Customers set the service should return paged results and should produce a query for the first page of the results.

This part depends on the paging behavior we need to implement. We may modify the query to prepare the underlying storage for the paging. We may also need to inject some filter or limit into the query. For example the built-in paging implementation will add sorting to the query by key properties, so that it gets stable order of results and then it will request only the first page of results (by taking first n results).

3 – Enumeration of results

When the query executes the IQueryable.GetEnumerator gets called. Since we have a custom IQueryable implementation we get called here. We need to return a custom enumerator which will only return the first page of results. The end of the page is marked by simply reporting no more results from the enumeration.

Depending on how we chose to limit the page of results this may be a trivial operation (if the limit was built into the query by our IQueryable) or this may actually do the real work. For example here we could measure the time it takes to compute the results and decide to end the page if it takes too long. Note that it’s perfectly fine to return no results (clients will expect this).

No matter which approach we take the important part is that the enumeration must somehow track where it ended in the bigger result set (the one with all the results, the non-paged one).

4 – Getting the continuation token

Once the IEnumerator returns no more results, the WCF Data Services will call IDataServicePagingProvider.GetContinuationToken and pass in the enumerator. Now we need to extract the information of how far we got through the results from the enumerator and serialize it as an array of primitive values and return it.

The GetContinuationToken returns an array of objects. These need to be primitive values (as recognized by OData) and they should be reasonably small since since all of those will be written into the URL.

The continuation token must fulfill several requirements:

- It must not be temporary, clients can in theory hold on to it indefinitely. We may choose to only support certain lifetime, but it needs to be rather long (typical user sessions or something similar). The best would be to have indefinite lifetime of the continuation tokens.

- It must be persisted – that means it should not rely on any transient data on the server side. For example, storing some table in memory on the server. There’s no guarantee that the client will not ask for the next page after the server process was restarted, or maybe the next request will be processed by a different server in the farm and so on.

- It must “make progress”. If we don’t return any results for any given page, the continuation token should contain enough information so that next time we can pick up from where we ended without trying to compute the entire page again. Otherwise this may lead to endless loop. So for example if the page is limited by the amount of time it took to process the query, the continuation token must somehow remember how far the query processing got even if it didn’t return any results.

- It should be reusable. There’s nothing prevent the client from asking for the same next page multiple times.

5 – Response to the client

The service serializes the returned continuation token into the $skiptoken query option and returns it as part of the next link to the client. Let’s say that we produce the next link as /Customers?$skiptoken=1234.

some time later…

6 – Client requests the next page

The client sends a request for the next page using the next link it got. So for example /Customers?$skiptoken=1234.

7 – Processing the next page query

The services processes the query as usual building our custom IQueryable as usual.

8 – Applying the continuation token

At one point during the query processing the service will call the IDataServicePagingProvider.SetContinuationToken. It will pass in the IQueryable it built so far and the continuation token. The continuation token is deserialized from the URL and will contain the values returned by some previous call to GetContinuationToken.

The method needs to “apply” the continuation token to the query. The goal is to modify the query such that it will continue from where it left of as described by the continuation token and return the next page of results.

Couple of notes on implementing this:

- The SetContinuationToken doesn’t return a new IQueryable, the implementation is expected to modify the existing IQueryable.

- Because the IQueryable might be used for further query construction, the best approach is to just stick the continuation token onto the IQueryable at this time without applying it to the query. And then apply it before the query gets executed and once we are sure we have the entire query.

9 – Enumeration of next page results

The service executes the query and enumerates its results. This should be again limited to a one page of results and the enumerator should stop after it hits the page limit.

10 – Producing next-next page link

As in #4 above the GetContinuationToken is called on the enumerator. If it returns a continuation token, the entire process repeats itself just like in #4 above. If the method returns null (no continuation token), the service will not produce any next link into the results and the client will determine that it got all the results for the query.

Additional considerations

The query might be more complicated that just a simple /Customers. It may contain filters, sorting, projections and so on. The WCF Data Service will include all of these query options in the next link automatically, so the continuation token doesn’t have to remember these.

The paged entity set might not be the top-most entity set of the results for a given query. For example a query like /Regions?$expand=Customers will also returns custom result set. It is a responsibility of the IQueryable to recognize this and apply the paging limits to the expansions as well. And as above once it returns the enumerator for the expanded set and the enumerator returns no more results, the service will call GetContinuationToken on it to get the continuation token for the inner set.

So the GetContinuationToken might get called multiple times during a single query processing. Once for the top-most result set and once for each expanded result set. The SetContinuationToken is only ever called once for a query, and only if it did contain the $skiptonen query option.

Once the IDataServicePagingProvider is implemented, the built-in paging provider is disabled. The SetEntitySetPageSize methods will do nothing then. So it’s either all custom, or all built-in.

If we don’t want to implement paging on a specified entity set, we just return null from the GetContinuationToken when called on the enumeration of such entity set.

All in all, implementing the custom paging provider is a complex task. But it opens very interesting and powerful techniques for the service.

Projections in custom providers – Simple solution

2012-01-07T18:17:09Z

Implementing custom provider for WCF Data Services is complicated enough. The fact that for the really interesting providers it’s also necessary to implement a custom IQueryable makes things just so much more complicated. Based on feedback from multiple parties trying to implement the IQueryable for WCF Data Services one of the most challenging tasks is to support projections and expansions. These are hard not only because the expression trees generated for them are complicated but also because they require the query to return results in different shape than usual. In this post we’ll walk through a simple case of dealing with projection and expansions (with some limitations).

First let’s assume that you’re familiar with the expression trees generated by WCF Data Services for projections and expansions. These are described in the Data Services Expressions series part 8 and part 9. Also some familiarity with IQueryable, its behavior and the general approaches to its implementation is required. There are multiple places to learn about these, but the classic source is Matt’s blog series.

Scenarios

Since the solution we’re going to try out is simplistic it only solves certain limited scenarios. Projections ($select) are meant to limit the number of properties returned by a query for any given entity in it. This is definitely useful for the client since it can download only the properties it’s going to need. But the server can take advantage of this as well, since if the client only asks for subset of properties the server can also only load a subset of properties from the underlying data store. The simple solution described in this post assumes that the second optimization is not necessary, that it’s OK to load all properties for each entity from the data store. The projections then only happen inside the service and are easier to deal with.

If your provider wants to support expansions as well, these are even more complicated. This simple solution will only support expansions if the entity instances can load the target of any given navigation property on demand (when the navigation property is accessed). Some data sources might be able to do this easily, but for other it might be hard or expensive. I will discuss possible solutions to those in some later post.

Simplification

The general idea of dealing with projections in our provider will be this:

Take the entire query and split it in two parts, the projection and expansion itself and the source query for the projection and expansion which returns the results as plain entities.
Then run the source query against the underlying data source (we will assume we already have a solution for this).
Take the results of the source query and apply projections and expansions on it inside the service.
Return the new results to WCF Data Services.

For the purposes of this sample we will use a LINQ to SQL based data source. So we create a new web application, add a database to it with two tables Products and Categories (similar schema as in the sample service on odata.org). Then we create a LINQ to SQL classes over these tables which will give us a SampleDataContext class with two properties Products and Categories.

Note: I chose LINQ to SQL since it can deal with all the other parts of the query (filters, sorting and so on). It also supports on-demand loading of navigation properties by just accessing them. And last but not least, it generates the proxy classes, so we can use reflection provider to setup the metadata for us. In custom provider implementations this would obviously not be the case and most of these would have to solve in our code, but for the purposes of this sample such an approach is enough.

Custom query provider

We need a custom query provider, that is IQueryable and IQueryProvider implementations. Implementation of IQueryable is pretty simple (as usual):

public class ProjectionQuery<T> : IOrderedQueryable<T>

    private Expression expression;

    private ProjectionQueryProvider provider;

    public ProjectionQuery(ProjectionQueryProvider provider, Expression expression)

        this.provider = provider;

        this.expression = expression;

    public IEnumerator<T> GetEnumerator()

        return this.provider.ExecuteQuery<T>(this.expression);

    IEnumerator IEnumerable.GetEnumerator()

        return this.provider.ExecuteQuery<T>(this.expression);

    public Type ElementType { get { return typeof(T); } }

    public Expression Expression { get { return this.expression; } }

    public IQueryProvider Provider { get { return this.provider; } }

When a query is to be execute this only calls ExecuteQuery method on the query provider. Now let’s add a stock implementation of the query provider like this:

public class ProjectionQueryProvider : IQueryProvider

    private IQueryProvider dataSourceQueryProvider;

    private ProjectionQueryProvider(IQueryProvider dataSourceQueryProvider)

        this.dataSourceQueryProvider = dataSourceQueryProvider;

    public static IQueryable<T> WrapDataSourceQuery<T>(IQueryable<T> dataSourceQuery)

        return new ProjectionQuery<T>(

            new ProjectionQueryProvider(dataSourceQuery.Provider),

            dataSourceQuery.Expression);

    public IQueryable<TElement> CreateQuery<TElement>(Expression expression)

        return new ProjectionQuery<TElement>(this, expression);

    public IQueryable CreateQuery(Expression expression)

        return (IQueryable)Activator.CreateInstance(

            typeof(ProjectionQuery<>).MakeGenericType(TypeSystem.GetIEnumerableElementType(expression.Type)),

            this,

            expression);

    public TResult Execute<TResult>(Expression expression)

        // This is the case of for example .Count() query

        // - we don't support it for now.

        // One option would be to pass it directly to the underlying provider

        // if it can handle it.

        throw new NotSupportedException("We don't support expressions returning simple values yet.");

    public object Execute(Expression expression)

        // This is the case of for example .Count() query

        // - we don't support it for now.

        // One option would be to pass it directly to the underlying provider

        // if it can handle it.

        throw new NotSupportedException("We don't support expressions returning simple values yet.");

    // ExecuteQuery implementation missing here.

Our query provider is based on a query provider for the underlying data source. As noted above this assumes that we already have an IQueryable implementation which can deal with filters, sorting and alike.

Projections

And now to the real interesting part, how to implement the ExecuteQuery:

public IEnumerator<TElement> ExecuteQuery<TElement>(Expression expression)

    // Simple solution, more robust solution would be using expression visitor

    // to process the projection and separate the projection source

    // from the rest of the query.

    // For now assume that the projection is the last operator on the query

    // (which is mostly true for WCF DS Expressions)

    // Match the .Select call (this is a helper method which determines

    // if the method call is a .Select and extracts the parameters

    // to the Select call in a nicely consumable way,

    // it doesn't modify the expression in any way).

    var selectMatch = ExpressionUtils.MatchSelectCall(expression);

    if (selectMatch == null)

        // No projection - just run the source query.

        return this.dataSourceQueryProvider.CreateQuery<TElement>(expression)

            .GetEnumerator();

    else

        // Projection - separate the source query, and run that

        // against the data source provider.

        // Call a private method so that we can use generics

        // for easier manipulation of results

        // (otherwise we would have to use a lot of reflection)

        // The source item type is the item type of the source expression.

        MethodInfo methodInfo = typeof(ProjectionQueryProvider)

            .GetMethod(

                "ProcessProjection",

                BindingFlags.Instance | BindingFlags.NonPublic)

            .MakeGenericMethod(

                TypeSystem.GetIEnumerableElementType(selectMatch.Source.Type),

                typeof(TElement));

        return (IEnumerator<TElement>)methodInfo.Invoke(

            this,

            new object[] { selectMatch });

private IEnumerator<TResultElement>

     ProcessProjection<TSourceElement, TResultElement>(

        ExpressionUtils.SelectCallMatch selectMatch)

    // Source is the query without projection/expansion

    // So we will run it right here against the original provider

    // - this is the query without projections.

    IQueryable<TSourceElement> dataSourceQuery =

         this.dataSourceQueryProvider.CreateQuery<TSourceElement>(selectMatch.Source);

    // This turns the query into enumerable which effectively executes it.

    // All other LINQ operations we perform on top of it

    // are performed on LINQ to Objects and not the underlying query provider.

    // (We could also cache the results here if needed).

    IEnumerable<TSourceElement> dataSourceQueryResults =

        dataSourceQuery.AsEnumerable(); ;

    // Now we have the results of the query as whole entities

    // (no projection applied).

    // This also assumes that for $expand to work the entities returned

    // already have or can lazy load the navigation properties.

    // Get the projection lambda and compile it into a delegate.

    Func<TSourceElement, TResultElement> projectionFunc =

        (Func<TSourceElement, TResultElement>)selectMatch.Lambda.Compile();

    // And now just run the projection function on each item

    // from the results we have to create the real results.

    IEnumerable<TResultElement> results =

         dataSourceQueryResults.Select(sourceItem => projectionFunc(sourceItem));

    return results.GetEnumerator();

}

The ExecuteQuery implementation only determines if the query contains projections/expansions and if so it invokes the ProcessProjection method. That method takes the source of the projection and, executes it against the data source (in our case this would query IQueryable against the LINQ to SQL and run it). Then it takes the results and applies the projections/expansions in-memory on it.

The main trick used here is to take a lambda expression (System.Linq.Expressions.LambdaExpression) and call the Compile method on it. This generates a lightweight dynamic method which is the body of the .Select and can be called just like any other method. Then we just simply call this method on each result and return to WCF Data Services.

Sample

And now in our sample we simply wrap the IQueryable returned by LINQ to SQL with the one we implemented above. Like this:

// Data context which wraps the LINQ to SQL data context,

// so that we can wrap the IQueryable properties.

public class ProjectionDataContext

    private SampleDataContext sampleDataContext = new SampleDataContext();

    public IQueryable<Product> Products {

         get { return ProjectionQueryProvider.WrapDataSourceQuery(

                  this.sampleDataContext.Products); } }

    public IQueryable<Category> Categories {

         get { return ProjectionQueryProvider.WrapDataSourceQuery(

                  this.sampleDataContext.Categories); } }

public class SampleService : DataService<ProjectionDataContext>

    public static void InitializeService(DataServiceConfiguration config)

        config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);

        config.DataServiceBehavior.MaxProtocolVersion =

            DataServiceProtocolVersion.V2;

And that’s it. Give it a try and run a query for example like service/Products?$select=Name. It should only return products with Name property. To verify that it really worked, set a breakpoint into the ProcessProjection method on the line which create dataSourceQueryResults and run the query again. The dataSourceQuery is the IQueryable which will execute against the data source. In this sample it should show in the debug window a simple table scan for table Products (this is provided by LINQ to SQL for us). If the query would contain filters or sorting this query would contain all of those and LINQ to SQL would generate the appropriate SQL for it.

If we would like to use this solution as part of a bigger custom provider the query provider would need to be able to handle all of our custom provider needs on top of this. There are also some technical limitations. In particular the solution requires that IDataServiceQueryProvider.NullPropagationRequire returns true, otherwise the function generated by lambda.Compile would not correctly handle null values. This solution is also not suitable for wrapping EF providers since the expression trees generated for EF provider are not fully compatible with lambda.Compile either. And last, if our custom provider uses untyped properties, those would have to be “resolved” in the lambda expression before the .Compile is called. Not to mention how to solve the expansions for real and don’t lazy load the navigation properties one by one. All this will have to wait for another post.

And here’s the full sample as a VS 2010 solution.

Adding Multi-Value properties to untyped providers

2010-11-09T10:40:59Z

With the recent release of the WCF Data Services Oct 2010 CTP1 for .NET4 the WCF Data Services now has the ability to expose Multi-Value properties (formerly called “Bags”). These can come very handy when you want to include short collections of primitive or complex types on your entities (or other complex types). In this blog we’ll take a look at what does it take to add the Multi-Value properties to an untyped provider.

Naming note: Sometimes in the CTP1 APIs the Multi-Value properties are still referred to as Bags, or Bag properties. Also the EDM type name is still Bag. I will refer to these as Multi-Value properties in the text, but the code might talk about Bags to be consistent with existing APIs.

Setup

I’m going to be using the untyped read-only provider sample from the OData Provider Toolkit which you can download here: http://www.odata.org/developers/odata-sdk.

You will also need the WCF Data Services Oct 2010 CTP1 for .NET4 as announced on this blog: http://blogs.msdn.com/b/astoriateam/archive/2010/10/26/announcing-wcf-data-services-oct-2010-ctp1-for-net4-amp-sl4.aspx

In your VS 2010 open the Untyped\RO\ODataDemo.sln solution file and let VS convert it to 2010 format and convert all the projects to .NET4.

Since the CTP is a side-by-side release it doesn’t replace the WCF Data Services assemblies, so you will need to update the references. Simply open each of the three projects and remove the references to System.Data.Services and System.Data.Services.Client. Then add references to Microsoft.Data.Services.dll and Microsoft.Data.Services.Client.dll which can be found in your installation folder of the CTP under .NETFramework sub-folder. (You will have to browse for the files as the assemblies are not listed in the framework list).

To verify everything works as expected you should be able to rebuild the solution and run all the tests without issues. You can also browse to the demo service to see it working.

Metadata

In order to expose Multi-Value property, such property must be added to the metadata. To define a Multi-Value property a special Multi-Value resource type is required. This type is used to define the type of each item of the Multi-Value. The class which represents this type is called BagResourceType, it derives from the ResourceType class and it adds a single property ItemType. To create an instance of this type a new method ResourceType.GetBagResourceType was added.

Let’s see how this works on the untyped provider toolkit sample. To add a Multi-Value of complex or primitive type add this method to the DSPMetadata class:

/// <summary>Adds a bag of complex or primitive items property.</summary>
/// <param name="resourceType">The resource type to add the property to.</param>
/// <param name="name">The name of the property to add.</param>
/// <param name="itemType">The resource type of the item in the bag.</param>
public void AddBagProperty(ResourceType resourceType, string name, ResourceType itemType)
{
    ResourceProperty property = new ResourceProperty(
        name, 
        ResourcePropertyKind.Bag, 
        ResourceType.GetBagResourceType(itemType));
    property.CanReflectOnInstanceTypeProperty = false;
    resourceType.AddProperty(property);
}

Please note that the property must be defined with a special kind Bag as well as have the special BagResourceType type. Also since we’re creating an untyped provider the property is set to not use reflection (as any other property in this provider).

For a Multi-Value of primitive types we can also add a helper method like this:

/// <summary>Adds a bag of primitive items property.</summary>
/// <param name="resourceType">The resource type to add the property to.</param>
/// <param name="name">The name of the property to add.</param>
/// <param name="itemType">The primitive CLR type of the item in the bag.</param>
public void AddBagProperty(ResourceType resourceType, string name, Type itemType)
{
    ResourceType itemResourceType = ResourceType.GetPrimitiveResourceType(itemType);
    this.AddBagProperty(resourceType, name, itemResourceType);
}

One other thing to note about the Multi-Value properties is their instance type. The InstanceType property of the BagResourceType will always return IEnumerable<T> where T is the InstanceType of the ItemType. So for example a Multi-Value property of strings (primitive type), will have an instance type IEnumerable<string>.

In the untyped provider in the sample all complex types use the same instance type called DSPResource. So a Multi-Value of complex types will have instance type IEnumerable<DSPResource>.

The instance type is important because it’s the type the WCF Data Services will assume as the value of the Multi-Value property. Most importantly the IDataServiceQueryProvider.GetPropertyValue method should return an instance of IEnumerable<T> for a Multi-Value property. Note that other than being IEnumerable<T> there are no other requirements on the value returned. It can be really any instance which implements the required interface. For most cases List<T> works great, but if you have some special requirements, you can implement your own and it will work just fine.

Data

And that leads us to how to specify instance data. Let’s use the sample ODataDemoService for this. In the DemoDSPDataService.svc.cs file there is a method which defines the metadata for the demo service called CreateDSPMetadata. For now, we will tweak the metadata by adding a Tags property to the Product entity. This can be done by adding just a single line like this:

metadata.AddBagProperty(product, "Tags", typeof(string));

Add this line after the other property additions for the Product entity type.

Now we need to add the instance values for the Tags property. Important thing to note is that Multi-Value properties can not have null value. That means the IEnumerable<T> must never be null (if it is, the service will fail to serialize the results). It is also invalid to have items in the Multi-Value of null value, so the enumeration of the IEnumerable<T> must not return nulls either (if the T actually allows nulls).

The demo service has a method called CreateDataSource which initializes the instance data. So let’s add some instance data for our new Multi-Value property there. Anywhere into the part which sets property values for the productBread add a line like this:

productBread.SetValue("Tags", new List<string>() { "Bakery", "Food" });

Then add similar lines for the productMilk and productWine as well. If you omit these lines the value of the Multi-Value property will be null and it will cause failures (since nulls are not allowed). It is perfectly valid to add empty lists though.

Result

To run the demo service, one more thing has to be done. The Multi-Value property is a new construct which older clients/servers might not understand so it requires an OData protocol version 3.0. So we need to allow the V3 for our service by changing the InitializeService method to include line like this (replace the line there which sets it to V2):

config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V3;

And that’s it. Just build the solution and run the demo service. If you request the product with ID 0 with URL like http://service/DemoDSPDataService.svc/Products(0) in your browser you should see this property in the output:

<d:Tags m:type="Bag(Edm.String)">
    <d:element>Bakery</d:element>
    <d:element>Food</d:element>
</d:Tags>

This is an XML representation of a Multi-Value of primitive type, the JSON representation is similar and uses JSON array to represent it.

Adding a Multi-Value of complex types is similar to the primitive types, just remember that the T in IEnumerable<T> is the instance type of the complex type, so in this sample provider it would be the DSPResource type.

Data Services Expressions – Part 9 – Expansions

2010-07-15T12:09:33Z

Series: This post is the ninth part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

In the last post we looked at projections, which leaves us with just one last big feature, expansions.

The $expand query option

Query which targets a certain entity by default returns just primitive and complex properties of that entity. Navigation properties are only represented as links in the response. Expansions allow clients to ask the server to return the entity or entities referred to by a specified navigation property along with the base entity of the query. For a sample let’s take a look at this query:

http://host/service.svc/Categories

Such query returns category entities. Each category will contain its ID, Name and other primitive or complex properties in the payload. But the Products navigation property will only be represented in the payload as a link. If the client needs the products for each category as well, it would have to download the feed this link points to for each of the categories. Or it can use expansions and ask the server to include the Products in the response as well, like this:

http://host/service.svc/Categories?$expand=Products

This query will return the same Categories as the previous one, but the Products navigation property will now also include all the product entities for each category in the response. This feature can be very useful for clients to minimize the number of requests issued against the server.

Expressing expansions in expressions

Now let’s take a look at how such query is executed on the server. The simplest approach would be to simply assume that the server can ask for the value of the Products property and it would get back the product entities it needs. But this approach would be inefficient and rather complicated to implement for many providers.

As an example let’s consider a provider which reads its data from a database. In the database navigation properties are usually stored as foreign keys. A query which returns categories would simply return all the rows from the table with categories. At the time the service would ask for all products for a given category the provider would have to issue a new query against the database to get such data (from different tables). And this would be repeated for each category in the results. This could quickly become very expensive, not counting that your favorite DBA would not like your application at all.

Much better solution would be to ask the provider to get categories and their products in one query so that it could issue a join query. But how to tell the provider that Products are needed as well? We could invent some new way (special method or so), but that would go against the idea that all queries are passed to the provider as LINQ queries.

The way this is resolved is by explicitly projecting the navigation property which needs to be expanded. So a simple example of the query above could look something like this:

categories.Select(category => new
{
    Category = category,
    Products = category.Products
});

You can try this with any existing LINQ provider and it should work as expected, that is retrieve the categories as well as all their products from the data store.

The above approach can’t be used directly by WCF Data Services. The main problem is that it requires generation of a new type for each set of expanded properties (the return type of the query). So instead a special type is used, which is called ExpandedWrapper. It’s rather similar to the ProjectedWrapper which we discussed in the previous post and it serves very similar purpose.

ExpandedWrapper is a generic type (in fact there are 12 such types) where the generic parameters specify the type of the entity being expanded and then types of each of the expanded properties. Each ExpandedWrapper then has a property ExpandedElement which will store the entity being expanded (the category in our sample) and a certain number of properties called ProjectedProperty0, ProjectedProperty1 and so on to store the expanded navigation properties. So in our sample above using ExpandedWrapper the query would look like:

categories.Select(category => new ExpandedWrapper<Category, IEnumerable<Product>> { 
    ExpandedElement = category, 
    Description = "Products", 
    ProjectedProperty0 = category.Products 
});

I didn’t mention the property Description, which servers the same purpose as ProjectedWrapper.PropertyNameList and stores a comma delimited list of names of navigation properties expanded.

The real expression

To be able to look at the real query generated by WCF Data Services our simple reflection based service we used so far won’t do. The reflection provider (which is used if you just provide classes as the service definition) assumes the exact thing we want to avoid. That is being able to access a navigation property any time without any consequences. For in-memory data, this is a very valid assumption as all the data is readily available without a need to issue expensive queries. So the expression generated by the reflection provider for a simple expansion like the one in our sample will not try to tell the LINQ provider to get the expanded properties, since there’s no need. For a custom provider WCF Data Services can’t make such assumption, and so it will generate the expansion expression like shown above.

So instead of our simple provider we used so far we will use a sample from the OData Provider Toolkit, which you can download here: http://www.odata.org/developers/odata-sdk. From the downloaded archive extract the Typed\RWNavProp folder and open the solution in it. Add the InterceptingProvider toolkit (as mentioned in our second post) to the solution and open the DSPResourceQueryProvider.cs file from the DataServiceProvider project. In it find the method GetTypedQueryRootForResourceSet<TElement> and wrap the call returned IQueryable into the intercepting query, something like this:

private System.Linq.IQueryable GetTypedQueryRootForResourceSet<TElement>(ResourceSet resourceSet)
{
    return Toolkit.InterceptingProvider.Intercept(
        this.dataSource.GetResourceSetEntities(resourceSet.Name).Cast<TElement>().AsQueryable(),
        (expression) =>
        {
            System.Diagnostics.Trace.WriteLine(expression.ToString());
            return expression;
        });
}

Now simply start the ODataDemoService and issue our sample query. You should see an expression like this:

System.Linq.Enumerable+<CastIterator>d__b1`1[ODataDemo.CategoryEntity].Select(p =>
    new ExpandedWrapper`2() {
        ExpandedElement = p,
        Description = "Products",
        ProjectedProperty0 = p.Products})

Which is exactly like the one we constructed above, so it works as it should. The above text doesn’t actually show the real type of the ExpandedWrapper, it just shows that it’s a generic type with two parameters (that’s what the `2 means in CLR). The real type if you look at it in the debugger is ExpandedWrapper<CategoryEntity, IEnumerable<ProductEntity>>.

Note that unlike the ProjectedWrapper expressions, there’s no need for Convert operators in this one since all the properties are strongly typed due to the fact that the ExpandedWrapper is a generic class.

The p.Products expression above is yet another standard property access expression as described in this post, and as such might differ based on the property definition you use in your provider.

During serialization WCF Data Services will now recognize the results as being ExpandedWrapper objects, and it will access the ExpandedElement property and use its value as the entity instance. Then when it needs to get the value of the navigation property to expand it will access the right ProjecteProperty property (based on the comma delimited names in the Description property).

That’s it for a simple expansion. It gets much more complicated if several features are combined together. When both expansions and projections are combined the query will use a combination of ExpandedWrapper and ProjectedWrapper objects and it will get considerably more complex. Also using server driven paging will influence the expression tree a lot. But those are all topics for some future posts.

Data Services Expressions – Part 8 – Projections

2010-07-07T15:30:47Z

Series: This post is the eighth part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

Projections feature is visible in the URL through the $select query option. It allows the client to ask the server to only return a subset of properties on the returned entities. In this post we’ll look into how simple projections are handled by the URL parsing and how are these translated into expression executed against the provider.

Properties projected

A simple projection sample in the URL:

http://host/service.svc/Products?$select=Name

This asks the server to return all product entities, but only include the Name property in the payload. Internally the server will construct a list of properties to request from the data source. So at first the server adds all the properties listed in the $select query option to the list of properties to be projected. For the sample above we have { Name } so far.

The client still expects to get back the usual feed of entries (in case of ATOM payload), and for this to work correctly each entity must have a unique identifier in the payload. The WCF Data Services server uses the key properties to construct the identification. So the server will include all key properties in the list of properties to be projected. In our sample above the Product has just one key property called ID, so we have { Name, ID } as the list of properties to project.

Note: If ETag is defined for the projected entity, all the ETag properties will be added into the list as well. Also, if there is any EntityPropertyMappingAttribute defined for the projected entity, all the properties mapped by these attributes will be included in the list as well.

ProjectedWrapper

Once we have the complete list of properties we will need for each entity from the data source, we need to define the type of each entity to be returned. This is determined by the LINQ expression we will use to project the properties. Projections are usually expressed through the Select operator like this:

products.Select(product => new Product()
{
    Name = product.Name,
    ID = product.ID,
});

But this approach doesn’t work if the provider doesn’t use CLR types which reflect the shape of the entity as defined by the provider’s metadata. Some providers use generic CLR types for their entity representation (for example a Dictionary or some other custom object). Accessing properties on such entities uses method calls as described in this post. So instead WCF Data Services uses a special types to project to, the ProjectedWrapper.

There are in fact ten ProjectedWrapper types. ProjectedWrapper0 to ProjectedWrapper8 are used when 0 to 8 properties need to be projected. ProjectedWrapperMany is used when more then 8 properties are needed, in which case the ProjectedWrapperMany has a Next link to the next ProjectedWrapperMany instance forming a linked list, where each item stores up to eight properties.

Each ProjectedWrapper type has a property ResourceTypeName which stores the full type name of the entity this projected wrapper represents. This is necessary as it’s not possible to tell the type of the entity based on the few properties projected. We will discuss in detail how type hierarchies are handled in projections in some later post. There’s also another property PropertyNameList on each ProjectedWrapper. This property stores a comma delimited list of names of properties projected into this wrapper.

The projected properties are then stored in ProjectedWrapper members ProjectedProperty0 to ProjectedProperty7. The first name in the PropertyNameList specifies the name of the property projected into the ProjectedProperty0, the second specifies the name of the ProjectedProperty1 and so on.

So in our sample above, where we are to project properties Name and ID, using a ProjectedWrapper would look like this:

products.Select(product => new ProjectedWrapper2() { 
    ResourceTypeName = “TestNamespace.Product”, 
    PropertyNameList = “Name,ID”, 
    ProjectedProperty0 = p.Name, 
    ProjectedProperty1 = p.ID 
});

As a result the type of the query is now IQueryable<ProjectedWrapper2> (instead of IQueryable<Product> which would be the case without projections). The WCF Data Services will recognize this and will not assume each result to be of the entity type, but instead will use the ProjectedWrapper and the information in it to write the results to the response.

Note: The reason such a generic looking type like ProjectedWrapper is used instead of something more friendly is performance. Generating new type for each shape of the projection result would be rather expensive.

Note 2: Currently the data source must return instances of the ProjectedWrapper public type if such type was requested by the query. It is not supported for the provider to change the type of query even if it decides to ignore projections and return the original entity for example. In the future the product might support some looser contract in this place, for example an interface similar to the IExpandedResult.

Projection expression

Knowing about ProjectedWrappers we can finally look at the expression generated by WCF Data Services. Let’s take our sample query from above, the expression generated would look like this:

System.Collections.Generic.List`1[DataServiceExpression.Product]
    .Select(p => IIF((p == null), null, new ProjectedWrapper2() {
        ResourceTypeName = "TestNamespace.Product",
        PropertyNameList = "Name,ID",
        ProjectedProperty0 = Convert((p As Product).Name),
        ProjectedProperty1 = Convert(Convert((p As Product).ID))}))

This is pretty close to what we came up with above. The first difference is the null check (the IIF((p == null), null, …)). This is so called “null propagation” and in detail I will discuss it at some later time. In short it makes sure that if the result being projected is null, the expression would still work and would project a null and wouldn’t crash. This part of the query can be influenced by modifying the value returned from IDataServiceQueryProvider.IsNullPropagationRequired, assuming you implement a custom provider. I will describe this in more detail in some later post.

The second difference is the type cast operation “p As Product”. This is needed when there’s type inheritance involved. In the simple case where there’s just one type of entity they are pretty much redundant as the parameter p is already of type Product anyway.

The third difference is the Convert operator. There are actually two types of these. The outer ones which are present on each property regardless of its type are conversions to System.Object. This is necessary since the ProjectedProperty0 is defined as having System.Object type, and thus the value assigned to it needs to be converted to such type. Note that in most programming languages the compiler will generated this conversion for you automatically, so you don’t usually see this in your code. The second type of Convert operators, the inner one, is only needed on value types and it converts the value to a nullable version of such type. So in the case above it converts the System.Int32 type of the ID property to a Nullable<System.Int32> type. This is again necessary when there’s type inheritance involved, in this simple case the value of the property will never be null.

So to makes all of the above more clear, here’s the same expression as above rewritten to what it would look like in C#:

products.Select(p => (p == null) ? null : new ProjectedWrapper2() { 
    ResourceTypeName = “TestNamespace.Product”, 
    ProductNameList = “Name,ID”, 
    ProjectedProperty0 = (object)((p as Product).Name), 
    ProjectedProperty1 = (object)(int?)((p as Product).ID) 
});

And that’s all. We covered all the basics you might need to understand simple projection queries. As noted above it gets more complicated when type inheritance is involved. Also null propagation has some effects on the expressions generated in certain cases. Yet another interesting topic is expansions and their interaction with projections. But I will get to those in some later post.

Data Services Expressions – Part 7 – Navigation

2010-06-22T15:39:20Z

Series: This post is the seventh part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

In this post we will examine how navigations are expressed by the WCF Data Services and what the provider needs to do to correctly support them.

What is navigation

In the OData model, entities can have so called “navigation” properties. These properties are references to other entities in the model. There are two types of navigation properties. Resource reference property references a single resource (entity) and can have a null value. Resource reference property has cardinality of 0..1. Resource set reference property references a collection of entities, must not be null but can evaluate to an empty collection. Resource set reference property has cardinality 0..N.

In the URL, navigations are expressed as traversal of the specified property in the path. For samples in this post we will use the Product entity we have from previous posts. A class-like description of the entities is here:

public class Product
{
    public int ID { get; set; }
    public string Name { get; set; }
    public string Description { get; set; }
    public double Price { get; set; }
    public DateTime ReleaseDate { get; set; }
    public DateTime? DiscontinueDate { get; set; }
    public int Rating { get; set; }
    public Category Category { get; set; }
}

public class Category
{
    public int ID { get; set; }
    public string Name { get; set; }
    public List<Product> Products { get; set; }
}

We added a resource reference property (0..1) Category on the Product and then a resource set reference property (0..N) Products on Category. So a sample query with navigation is for example:

http://host/service.svc/Products(1)/Category

This query returns the Category for Product with ID 1. The other way round navigation like this:

http://host/services.svc/Categories(1)/Products

returns all products which belong to category with ID 1. Navigations can be chained (just like directory traversal), but it is only possible to navigate on properties of a single result. So right after a navigation over resource reference property, we can put another navigation like this:

http://host/services.svc/Products(1)/Category/Products

But if the navigation is over a resource set reference property a key lookup must be inserted to specify which entity to use and then another navigation can be used, like this:

http://host/services.svc/Categories(1)/Products(2)/Category

Resource reference property navigation

Navigating over a resource reference property means to traverse from one entity instance to another over the navigation property. As we’ve discussed in the key lookup description post, a single entity is expressed by appending a call to Where method which filters based on the key properties to the query root. So in order to navigate over a resource reference property, we need to project the value of the navigation property on the entity in question. Since the query is constructed to return multiple results (even though it never will), the projection needs to be done through a call to the Select method. So a query for /Products(1)/Category will look like:

System.Collections.Generic.List`1[TypedService.Product]
.Where(element => (element.ID == 1))
.Select(element => element.Category)

Note that the body of the Select method is a property access to the resource reference property Category. This property access follows all the same rules as other property accesses, as described in this post. The Select method is in fact an extension method on the class Queryable. It takes two parameters, the implicit IQueryable<Product> which is the source query to project from and a lambda expression which for each Product returns the respective Category entity. All of this can be seen in the detailed view of the expression:

.Call System.Linq.Queryable.Select(
    .Call System.Linq.Queryable.Where(
        .Constant<System.Linq.EnumerableQuery`1[TypedService.Product]>(System.Collections.Generic.List`1[TypedService.Product]),
        '(.Lambda #Lambda1<System.Func`2[TypedService.Product,System.Boolean]>)),
    '(.Lambda #Lambda2<System.Func`2[TypedService.Product,TypedService.Category]>))

.Lambda #Lambda1<System.Func`2[TypedService.Product,System.Boolean]>(TypedService.Product $element) {
$element.ID == 1
}

.Lambda #Lambda2<System.Func`2[TypedService.Product,TypedService.Category]>(TypedService.Product $element) {
$element.Category
}

The provider is expected to return enumeration of Category entities from such query with a single result. If the above query returns no result or more than one result, the WCF Data Services will fail to process the request (with 500 status code). The single instance returned can be null though (the case where product has no category), in which case the response for such query will be 404 Resource Not Found. The 404 status code is the equivalent of null in the HTTP/URL world.

Resource set reference property navigation

Navigating over a resource set reference property is similar in that it must start on a single entity instance. But it returns a collection of entities as its result. Since the expression tree for the single entity is in fact an enumeration, the way to return another set of results is to perform a join. Join queries are expressed by a call to method SelectMany. So a query for /Categories(1)/Products will look like:

System.Collections.Generic.List`1[TypedService.Category]
.Where(element => (element.ID == 1))
.SelectMany(element => Convert(element.Products))

The SelectMany is again in fact an extension method on class Queryable. It takes two parameters, the first implicit one is the source query and the second is a lambda which takes the category entity and returns an enumeration of products. The result of the SelectMany call is enumeration of products, which is a concatenation of all the enumerations of products returned for all the categories in the source query. In our case the source query will always return exactly one result, so the result of the whole query is a list of products which belong to the category of ID 1.

Note that the Convert expression in the lambda converts the value of Products (which in our case is of type List<Product>) to the instance type of the navigation property which in case of resource set reference property is always IEnumerable<T>. In this particular case it is IEnumerable<Product>. Again this is in fact a property access expression.

For completeness here is the detailed view of the query:

.Call System.Linq.Queryable.SelectMany(
    .Call System.Linq.Queryable.Where(
        .Constant<System.Linq.EnumerableQuery`1[TypedService.Category]>(System.Collections.Generic.List`1[TypedService.Category]),
        '(.Lambda #Lambda1<System.Func`2[TypedService.Category,System.Boolean]>)),
    '(.Lambda #Lambda2<System.Func`2[TypedService.Category,System.Collections.Generic.IEnumerable`1[TypedService.Product]]>))

.Lambda #Lambda1<System.Func`2[TypedService.Category,System.Boolean]>(TypedService.Category $element) {
$element.ID == 1
}

.Lambda #Lambda2<System.Func`2[TypedService.Category,System.Collections.Generic.IEnumerable`1[TypedService.Product]]>(TypedService.Category $element)
{
(System.Collections.Generic.IEnumerable`1[TypedService.Product])$element.Products
}

A result of such query is expected to return enumeration of Product entities. Such enumeration can be empty or can contain multiple results.

Multiple navigations

Navigations can be chained and the expression generated for these operations are quite predictable. They are basically also results of chaining the single navigations together. So instead of describing again all the details, an example is much easier to understand. A URL query like /Products(1)/Category/Products yields an expression:

System.Collections.Generic.List`1[TypedService.Product]
.Where(element => (element.ID == 1))
.Select(element => element.Category)
.SelectMany(element => Convert(element.Products))

In this case since the first navigation is over resource reference property, the second navigation immediately follows the first one. And the second example with key lookup in the middle with URL query like /Categories(1)/Products(2)/Category yields an expression:

System.Collections.Generic.List`1[TypedService.Category]
.Where(element => (element.ID == 1))
.SelectMany(element => Convert(element.Products))
.Where(element => (element.ID == 2))
.Select(element => element.Category)

Here since the first navigation is over a resource set reference property, it must be followed by a key lookup (a filter expressed by a call to Where method) which then can be followed by the second navigation.

And that’s all there is to navigations. Since the queries in the URL can compose navigations with other operations like sorting, the expression tree generated may also contain such parts. The important thing to remember is that all the query options are applied to the end result of all the navigations.

Data Services Expressions – Part 6 – Key lookup

2010-06-16T20:21:39Z

Series: This post is the sixth part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

We will be looking into navigations in the next couple of posts. To be able to do that we first have to explain so called key lookups. With a query to a given entity set, the service returns all the entities in that set. We can use filters and such to limit the entities returned, but such queries will always assume that the result may contain multiple items.

In order to specify a single entity, the query must somehow identify just one instance. In WCF Data Services each entity has key properties, which serve this purpose. By specifying values for all of the key properties, any given entity instance is uniquely identified. Key lookup is a query (or part of query) which specifies the values for the key properties. So using key lookup we can ask for just one entity. Simple key lookup is like this:

http://host/service.svc/Products(1)

The above query returns a single entity or 404 response if the product specified doesn’t exist. Our Product entity has just one key property called ID and thus it’s not necessary to specify the name of the key property. The query above asks for product with its key property (ID) of value 1 (integer).

When such query is processed and translated into expression tree WCF Data Services will actually generate a query which asks for all products with ID equal to 1. Once that query returns its results, the service will verify that only one result was returned. So the above URL gets translated into expression tree like this one:

System.Collections.Generic.List`1[TypedService.Product].Where(element => (element.ID == 1))

It is pretty straightforward, it’s the query root followed by a filter which picks only products with ID equal to 1.

Note that we would get the exact same expression by running a URL query like:

http://host/service.svc/Products?$filter=ID eq 1

But the response for this query is different, it returns a list with a single item instead of the item itself directly.

Now let’s see what happens if the given entity has more than one key property. We will use an entity called Record which has two key properties; PartitionID which is an integer and RowID which is a string. A query for a single record looks like this:

http://host/service.svc/Records(PartitionID=1,RowID=’id0’)

The expression tree which is generated for this query is a bit different than what we could expect. WCF Data Services will use a separate Where call for each key property it needs to filter the results on, so it looks like this:

System.Collections.Generic.List`1[TypedService.Record]
.Where(element => (element.PartitionID == 1))
.Where(element => (element.RowID == "id0"))

Note again that it is expected that such query returns at most one result. If that’s not true the service will fail with a 500 status code.

And that’s it, key lookups are really not complicated. In the next post we will look at our first navigations over navigation properties.

Data Services Expressions – Part 5 – Sorting

2010-06-04T13:00:23Z

Series: This post is the fifth part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

This time we will look at how WCF Data Services translates sorting expressions. Sorting expression are used when the URL uses a query option $orderby.

Simple sorting

In the expression trees, sorting is expressed by a call to the OrderBy method. In the case of WCF Data Services, since the query is expressed as IQueryable, the exact method is System.Linq.Queryable.OrderBy extension method. First parameter of this method is the implicit IQueryable and the second parameter is a lambda expression which returns the value according to which the results should be sorted.

Let’s take a sample query like:

http://host/service.svc/Products?$orderby=Price

WCF Data Services will translate this into an expression like:

System.Collections.Generic.List`1[TypedService.Product]
.OrderBy(element => element.Price)

So it’s a query root (see this blog post for details) followed by a call to the OrderBy with a lambda which takes the product and extracts its Price property. Note that for sorting expressions WCF Data Services uses the name “element” as the name of parameter expression for the lambda expression.

The lambda body in this case is a simple property access and as such it will follow all the rules described in my previous post about accessing properties.

More complex sorting expression

That was a really simple sample, but in reality it usually gets more complicated. WCF Data Services allows usage of expressions in the $orderby query options. So we can use for example arithmetic like in this query:

http://host/services.svc/Products?$orderby=5 sub Rating

The expression generated looks like this:

System.Collections.Generic.List`1[TypedService.Product]
.OrderBy(element => (5 - element.Rating))

It can obviously get even more complicated, but the expression tree generated would just follow along, without any real surprises.

Multiple sorting expressions

If we want to specify that the results should be sorted according to multiple sorting expressions, in the query we add more expressions into the $orderby query option and separate them using comma characters. So for example:

http://host/services.svc/Products?$orderby=Rating,Price

Unfortunately the OrderBy method which is called in the expression tree to perform the sorting only takes one lambda expression. So to express more sorting expressions another method called ThenBy is used. As with OrderBy the ThenBy in our case is in fact System.Linq.Queryable.ThenBy extension method and it looks exactly the same as OrderBy.

The expression tree for the above query will look like:

System.Collections.Generic.List`1[TypedService.Product]
.OrderBy(element => element.Rating)
.ThenBy(element => element.Price)

Note that if there’s more sorting expressions, other ThenBy calls would be simple appended to the expression tree here. As a result a single $orderby query option will be translated as a call to OrderBy method followed by possibly multiple calls to ThenBy method.

Ascending or Descending

WCF Data Services also allows us to specify the sorting direction. By default sorting is done in the ascending order (that is 1, 2, 3, …), we can change that to the descending order by adding the keyword “desc” to the sorting expression:

http://host/services.svc/Products?$orderby=Rating desc

In the expression tree the change of sorting direction is expressed by calling yet another method OrderByDescending. Other than its name everything else about this method is the same as for the OrderBy method. So the expression tree looks like:

System.Collections.Generic.List`1[TypedService.Product]
.OrderByDescending(element => element.Rating)

There’s also a ThenByDescending which is used when more sorting options are specified some of which use the descending order. Again ThenByDescending is exactly the same as ThenBy, except for its name.

And to show it all of, let’s take our ultimate sorting example:

http://host/services.svc/Products?$orderby=Rating desc,Price,ReleaseDate desc

The expression tree generated is this:

System.Collections.Generic.List`1[TypedService.Product]
    .OrderByDescending(element => element.Rating)
    .ThenBy(element => element.Price)
    .ThenByDescending(element => element.ReleaseDate)

The last note for this blog post is about the OrderBy, ThenBy, OrderByDescending and ThenByDescending methods. These methods are defined as returning an IOrderedQueryable interface and so if you’re writing your own query provider be sure to support it correctly as described here.

Data Services Expressions – Part 4 – Accessing properties

2010-04-16T11:24:00Z

Series: This post is the fourth part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

In this part we will talk about accessing properties in the expressions. WCF Data Services may need to access a property value in many different places in the query, but it always uses the same way to do so. Below I’ll use the $filter as an example, but the same would apply to any other place in the query (for example $orderby, $select and so on).

Property metadata

In the WCF Data Services world the shape of the data is defined using metadata classes like ResourceType and ResourceProperty. To get a general idea how to do that, take a look at this series by Alex. Note that even if you’re not using a custom provider, underneath both the built-in providers will define the metadata in the same way, so there’s no hidden magic there.

Currently WCF Data Services uses three kinds of resource types: Entity types, Complex types and Primitive types. Each resource type is defined in the metadata by an instance of ResourceType class.

Both entity and complex types can define properties. A property can be of any resource type (with some limitations, but that’s for another topic). Each property is defined in the metadata by an instance of the ResourceProperty class.

The value of a property is either a primitive type, instance of an entity type, instance of a complex type or null. The actual CLR type of the property value is determined (and defined) by the resource type of the property. Each resource type has an instance type which is the CLR type used to store an instance of that resource type. For example a primitive resource type Edm.String, has instance type System.String. For entity and complex types, the instance type is defined by you in the metadata, usually it’s some class.

Accessing a property value in an expression

We’ll use $filter as a sample of an expression which accesses properties. If you want to know more about the filters please take a look at my previous post.

We’ll use the same sample query here:

http://host/service.svc/Products?$filter=Rating gt 3

In the metadata the entity type Product defines a primitive property Rating of type Int32. The filter expression needs to get the value of the Rating property for any given Product and compare it to number 3.

In a filter expression, which is a lambda expression which takes a parameter called “product” which holds an instance of the Product entity type, such property access would look like:

product.Rating

No magic, exactly what you would expect. But …

Typed and untyped properties

In the metadata definition of a property, which is represented by an instance of a ResourceProperty type, there’s a setting called CanReflectOnInstanceProperty. By default this value is set to true, which means that WCF Data Services can use normal CLR way of accessing the property value. In short we’ll call these properties “typed” properties meaning they have a backing CLR type which that property defined.

But if you change this setting to false, it tells WCF Data Services that it must not use the CLR way of accessing such property. For short we’ll call these properties “untyped”, meaning that the CLR type of the resource (entity, complex) doesn’t have such property on it, or it should not be used. Defining a property as untyped gives you full control over how the value of such property is determined for any given resource. But this ability comes with a price, it’s more complicated (more code).

Accessing a typed property

Let’s assume we’re constructing a filter expression as the sample above and thus we have defined a parameter expression which will have the instance of the Product entity we’re filtering:

ParameterExpression productParameter = 
    Expression.Parameter(typeof(Product), "product");

Accessing a typed property is simple as we’ve shown above. WCF Data Services uses the normal CLR way of accessing the property. In expression trees this means that accessing a property Rating on our product resource is generated like this:

var rating = Expression.Property(productParameter, "Rating");

Note that this is a shortcut for what actually happens underneath which is:

var rating = Expression.MakeMemberAccess(
    productParameter, 
    productParameter.Type.GetProperty("Rating"));

The result (in either case) is a MemberExpression where Expression is the productParameter and the Member is the CLR PropertyInfo for the Rating property.

Note that since we’re using reflection to access the property, the CLR type of the property access expression is the same type as the instance type of the Rating CLR property. In our case it’s Itn32.

This mechanism is used to access any “typed” property, be it a primitive, complex or a reference to a resource (or multiple resources).

Accessing an untyped property

If we specify CanReflectOnInstanceProperty = false for our Rating property, the expression generated will be completely different. In this case WCF Data Services must not use CLR reflection to access the property and as a result it doesn’t have a typical way to access the value.

It could use a call to IDataServiceQueryProvider.GetPropertyValue to get the value (as that is the method called to access the property value during serialization for example). But it turns out that it’s not the best way to do so in an expression. The main reason is that such call would require the instance of the IDataServiceQueryProvider to be either specified in the expression as a constant or passed in as a parameter. In either case it complicates the expression and makes it dependant on the actual instance of the IDataServiceQueryProvider. This is not suitable for some providers as they are not always going to use the IDataServiceQueryProvider to access the property value, for example they might translate the entire query into a SQL statement, in which case it doesn’t make sense to have a query provider in there.

So instead WCF Data Services uses special method calls in the expression as basically a placeholder for saying “I want the value of this property here”. Our sample above would then look like:

(Int32)DataServiceProviderMethods.GetValue(
    product, 
    ratingResourceProperty);

DataServiceProviderMethods is a class which holds all these placeholder methods WCF Data Services uses. The GetValue method is used to specify the intent to get a value of a property on a resource. The ratingResourceProperty is the instance of the ResourceProperty class which was used in the metadata to define the Rating property. It identifies the property to access. And the product is an instance of the resource (in our case the Product entity) for which to get the value of a property.

In expression this will be constructed like this:

var rating = Expression.Call(
    typeof(DataServiceProviderMethods).GetMethod("GetValue"),
    productParameter,
    Expression.Constant(ratingResourceProperty));

Note that all the methods on the DataServiceProviderMethods class are static, and that their implementation simply throws a NotImplementedException.

This forces the custom provider to replace such expression with some custom way of accessing the property. What this means to us is that it’s more work as we need to replace the expression in the tree with something which will actually get the value. But it also gives us great power. We can do really whatever we want to get the value of the property.

For providers which translate the entire query into a different language (like SQL for example), this provides an easy way to spot the property access in the tree (calls to these special methods are easy to recognize) and it gives them the ResourceProperty instance on which they can store some custom data to help in generating the query.

But even for providers which will execute the LINQ query eventually like a LINQ query (either using LINQ to Objects or something else), this provides the ability to specify a custom way of getting the property value. Using this mechanism we could for example rename a property, change its type, get its value not from a CLR property but from some property bag (Dictionary) or compute it from something completely different.

Untyped property access – the details

There are few more details we need to be aware of though.

First one is that the GetValue method is declared as returning a type System.Object. That alone is not very useful for the expression as it can’t for example be compared to a number. As a result most of the occurrences of GetValue call will be surrounded by a type conversion which casts the object to the actual instance type of the property. So the expression would in fact be constructed by using one more line of code like this:

rating = Expression.Convert(
    rating, 
    ratingResourceProperty.ResourceType.InstanceType);

The Rating property is of a primitive resource type which has an instance type System.Int32.

The second detail to be aware of is that not all properties are accessed using the GetValue method. Resource set reference properties (properties which represent a one to many relationships) are accessed in a similar fashion through a call to GetSequenceValue. We won’t go into the details of why is it like that, let’s just say that it’s useful since the GetSequenceValue returns IEnumerable<T> which allows for further query composition.

The last detail is that if the given property is an open property, the way to access its value is completely different and we’re not going to describe it here either as it’s a topic for a much larger post.

Data Services Expressions – Part 3 – Filters

2010-04-15T12:02:01Z

Series: This post is the third part of the Data Service Expressions Series which describes expressions generated by WCF Data Services.

In this part we will look at how filters are represented in the expressions.

The Queryable.Where method

Filters are meant to filter the entities (rows) returned from a given resource set. The client specifies the filter using the $filter query option in the URL. For example (using the sample service from odata.org):

http://services.odata.org/OData/OData.svc/Products?$filter=Rating gt 3

The intention of this query is to evaluate the filter expression (the part after the equal sign) for each product in the Products resource set, convert the result to a boolean and only return those products which return true. So the above query should return only products with rating greater than 3.

WCF Data Services takes this query and translates it into an expression tree. In expression trees, the way to express filters is to use the Where method. So the query would look like this:

products.Where(product => product.Rating > 3);

This is a nice C# way of writing what in fact looks like this:

Queryable.Where(products, product => product.Rating > 3);

The Queryable.Where is an extension method for IQueryable<T>, so that’s why it can be called on the products as if it was its member method. The other thing is that it’s a generic method which takes the T (the type of one resource) as a generic parameter. So it actually looks like this:

Queryable.Where<Product>(products, product => product.Rating > 3);

The first parameter is the IQueryable<T> which returns the resources to filter. The second parameter is the filter expression. It is an instance of

Expression<Func<Product, bool>>

So it’s an expression which represents a function which takes a single parameter of type Product and returns a boolean. In expression trees functions like this are expressed using lambda expressions.

Lambda expressions

A lambda expression takes a list of parameters (the parameters of the function) and a lambda body. Just like typical function takes parameters and has some body which computes the result.

Parameters are simply an array of instances of ParameterExpression expression node. Each such instance represents a parameter (you give it a type and a name).

The lambda body is an expression tree which can reference these parameters whenever it needs to access the value of the given parameter.

So the above filter expression is just a nice C# way of writing:

ParameterExpression productParameter = Expression.Parameter(typeof(Product), "product");
Expression body = Expression.GreaterThan(
    Expression.Property(productParameter, "Rating"),
    Expression.Constant(3));
var lambda = (Expression<Func<Product, bool>>)Expression.Lambda(
    body,
    productParameter);

For now let’s ignore how exactly the body is constructed and its parts, we’ll talk about these in some later post, instead let’s focus on how the lambda expression itself is used.

Filter expressions

And now back to our filter, using the way to view expressions generated by WCF Data Services from the previous blog post, let’s take a look at expression generated for our sample query:

http://host/service.svc/Products?$filter=Rating gt 3

The expression looks like this (result of the Expression.ToString() method):

System.Collections.Generic.List`1[TypedService.Product].Where(it => (it.Rating > 3))

Note that this is using the nice C# notation in which the Where call is treated as instance method and it shows the filter expression as the C# lambda expression. WCF Data Services calls the parameter expression for the filter expression which holds the resource instance “it”. It’s the same thing as in our samples above where we were using “product”.

To see the real expression, let’s look at the Expression.DebugView:

.Call System.Linq.Queryable.Where(
.Constant<System.Linq.EnumerableQuery`1[TypedService.Product]>(System.Collections.Generic.List`1[TypedService.Product]),
'(.Lambda #Lambda1<System.Func`2[TypedService.Product,System.Boolean]>))

.Lambda #Lambda1<System.Func`2[TypedService.Product,System.Boolean]>(TypedService.Product $it) {
$it.Rating > 3
}

Here we can see that the expression is actually a Call to the Queryable.Where method, which takes two parameters. The first parameter is the query root (see the previous blog post for discussion about the query root expressions). The second parameter is the lambda expression called #Lambda1 which represents the filter expression. As you can see above that lambda expression takes one parameter of type Product called $it and its body is $it.Rating > 3.

This was just the basic structure of how filters are handled by WCF Data Services. In the next post we’ll talk about the filter expression itself and specifically about the part of access property values.

Data Services Expressions – Part 2 – The query root

2010-03-03T13:51:04Z

Series: This post is the second part of the Data Services Expressions Series which describes expressions generated by WCF Data Services.

In this part we will first look how to watch the expressions generated by Data Services. Then we’re going to discuss our first expression, the query root.

How to see the expression?

To see the expressions generated by Data Services we need to get into the query processing pipeline. To do that, we need to implement IQueryable as that is the only public interface Data Services use to construct and execute all its queries as discussed in the Part 1.

Fortunately for me Alex just recently posted a perfect IQueryable implementation for this purpose, the InterceptedQuery<T> and InterceptingProvider. So instead of writing one from scratch, let’s just use that one.

public IQueryable<Product> Products
{
    get
    {
        return Toolkit.InterceptingProvider.Intercept(
            products.AsQueryable(),
            (expression) => {
                Trace.WriteLine(expression.ToString()); 
                return expression; 
            });
    }
}

If you put this into your sample service, every time you issue a query against the service it will write the expression into the debug output. The simplest way to watch these is to run your service under Visual Studio debugger, the debug output will show up in the Output window.

For detailed viewing you can put a breakpoint into the above code and inspect the expression from the Watch window.

And now to our first expression.

The query root

Each query starts with a resource set. To build a query the Data Services gets the query for the resource set root and then builds on top of it. Data Services calls the IDataServiceQueryProvider.GetQueryRootForResourceSet method to get an instance of IQueryable for the given resource set. Each IQueryable has an expression which can be accessed through the IQueryable.Expression property. The expression which is in the IQueryable returned from the GetQueryRootForResourceSet is called the query root.

The query root expression is interesting to us, because we get to specify it completely. Data Services will not modify the expression, it will just use it as the starting point. The nice thing about this is that we can use even expression which Data Services would not recognize, we can use our own stuff here.

There’s only one assumption which Data Services makes about this expression and that is it’s CLR type, which is the value of Expression.Type property. This type must implement IQueryable<T> where T is the instance type of the base resource type of the resource set (ResourceSet.ResourceType.InstanceType). In the above example that would be an IQueryable<Product>.

In the sample above the query root is the “products” variable, which is an instance of List<Product>. You can see this by watching the expression generated for a query “TypedService.svc/Products”.

It looks like the query root is the List<Product> itself, but it’s not. The actual expression node is a constant expression with the value of IQueryable implementation over the list of products. To see this we’ll take a look in the debugger.

We can see that the NodeType of the expression is Constant and the value is the List<Product>. If you’re using Visual Studio 2010 and .NET 4.0 there’s a new property on each Expression called DebugView which returns more detailed text representation of the expression. In this case it returns:

.Constant<System.Linq.EnumerableQuery`1[TypedService.Product]>
    (System.Collections.Generic.List`1[TypedService.Product])

This shows us that the Type of the expression is System.Linq.EnumerableQuery<Product> which is the LINQ to Object’s implementation of IQueryable<Product>. So it fulfills the requirement of Data Services to be of type IQueryable<Product>.

Custom query providers usually return some expression which represents the resource set root. This expression then holds information like the Table to get the data from, or the list of resources and so on.

In all our future visits to Data Services Expressions we will see this query root expression at the beginning of each query.

Data Services Expressions – Part 1 - Intro

2010-02-25T09:28:22Z

WCF Data Services uses Data Service Providers to determine the shape and actual values of the data to expose. There are two providers implemented already for you (Entity Framework provider and Reflection provider) and you can also implement a custom provider.

Alex James is writing a great series about how to implement such custom provider. You should read it to understand how the custom provider works.

Custom Data Service Provider gives you the ability to define the metadata, the update behavior and lot of other extensibility points. The portion I will focus on in this series is the ability to provide custom query provider.

Whenever Data Services need to query the data source to get some data values, it uses the query provider. The most important method of that provider is the IDataServiceQueryProvider.GetQueryRootForResourceSet. This method returns IQueryable which Data Services uses to construct and execute the query.

There are many descriptions of how to implement IQueryable interface, so I won’t try to produce yet another one. I will just point you to the one written by Matt Warren here. You should read the first few posts of that series to understand the basics of IQueryable and expression trees.

In short IQueryable is an interface that allows the caller to construct a query and then run it. The caller (in our case Data Services) builds the query by constructing an expression tree which describes the intent of the query.

Implementing your IQueryable and supporting all the expression trees it can get in full is hard. In fact it’s so hard that I would say there’s only one complete implementation, LINQ to Objects. All other LINQ providers have some limitations. It’s so hard because the space of possible expression trees is infinite and very complex.

The good news for you, who I hope is trying to implement a Data Service Provider, is that Data Services will use only a very small portion of all possible Expression trees. The goal of this series will be to describe this small portion, so that you can implement your own IQueryable which will support enough expressions for Data Services to work correctly.

There’s even better news which is that depending on your requirements and features of Data Services you use, you can implement only part of the possible expression trees and still work correctly.

In this series we will take some portion or feature of the query language Data Services support, for example the $filter query option, and then we’ll describe what type of expression trees Data Services generates to describe the query to the underlying Data Service Provider.

In the next part we’ll talk about the query root and the expressions used by Data Services to access property values.

Data Services Expressions Series

This series describes expressions generated by WCF Data Services

Part 1 – Intro

Part 2 – The query root

Part 3 – Filters

Part 4 – Accessing properties

The beginning

2010-02-23T20:51:00Z

It seems that I’ve finally got to start this blogging thing, exciting times…

Let me introduce myself - I’m Vitek a developer on the WCF Data Services (aka Astoria) team. In the past I’ve worked on Microsoft’s XML technologies like XmlLite and System.Xml.

What will you find on my blog? Technical stuff that I find interesting, some samples of things I’m working on and also information that I want to have a permanent link to.

So, welcome to my blog!