Meta-Me

CQRS with OData and Actions?

Alex D James — Fri, 03 Feb 2012 23:23:35 GMT

I love the Actions feature in OData – which is hardly surprising given I was one of its designers.

Here’s the main reason why I love it: Actions allow you move from a CRUD architecture style, where you query and modify data using the same model, to an architectural style where you have a clear separation between the model used to query and the model used to update. To me this feels a lot like Greg Young’s baby CQRS, or Command Query Responsibility Segregation.

I’ll admit I’m taking some liberties here because these two models are actually ‘merged’ into a single metadata document ($metadata) that describes them both, and you can share types between these two models… however this feels insignificant because the key benefits remain.

Why would you want to move from a CRUD style application to a CQRS style one?

Let’s look at a simple scenario, imagine you have Products that look like this:

public class Product
{
   public int ID {get;set;}
   public string Name {get;set;}
   public Decimal Cost {get;set;}
   public Decimal Price {get;set;}
}

And imagine you want to Discount Marmite (a product in your catalog) by 15%. Today using the CRUD style, the default in OData before Actions, there is only one option: you PUT a new version of the Marmite resource with the new Price to the URL that represents Marmite, i.e. something like this:

POST ~/Products(15) HTTP/1.1
Content-Type: application/json

{
   // abbreviated for readability
   “ID”: 15,
   “Name”: “Marmite”,
   “Cost”: 3.50,
   “Price”: 4.25    // ($5 – 15%)
}

Notice to support this you have to allow PUT for Products. And this has some real issues:

People can now make changes that we don’t necessarily want to allow, i.e. modifying the Name & Cost or changing the Price too much.
- Basically “Update Product” is NOT the same as “Discount Product”.
When a change comes through we don’t actually know it is a Discount. It just looks like an attempt to update a Product.
If you need information that is not part of Product to perform a Discount (perhaps a justification) there is no where to put that information.

More generally the CRUD model is painful because:

If you want to update lots of resources simultaneously, imagine for example that you want to discount every product in a particular category, you first have to retrieve every product in that category, and then you have to do a PUT for each of them. This of course introduces a lot of unnecessary latency and introduces consistency challenges (it is hard to maintain a transaction boundary across requests & it the longer the ‘transaction’ lasts the more likely a concurrency check will fail).
If you want to update something you have to allow it to be read somewhere.

Back to our scenario, it would be much better to disable PUT completely and create a Discount action, and advertise it’s availability in the Marmite resource (to keep your system as Hypermedia driven as possible):

{
“__metadata”: {
     // abbreviated for simplicity
     “Actions”: {
          “#MyActions.Discount”: [{ “title”: “Discount Marmite”, “target”: “Products(15)/Discount”}]
     }
},
“ID”: 15,
“Name”: “Marmite”,
“Cost”: 3.50,
“Price”: 5.00
}

The name of the Action (i.e. #MyActions.Discount) is an ‘anchor’ into the metadata document that can be found at ~/$metadata that says you need to provide a percentage.

POST ~/Products(15)/Discount HTTP/1.1
Content-Type: application/json

{
“percentage”: 15
}

This is much better. Notice this doesn’t allow me to modify the Cost or the Name, and indeed can easily be validated to make sure the percentage is within an acceptable range, and it is semantically much clearer what is happening.

In fact by moving from a CRUD style architecture to one inspired by CQRS but based on actions you can:

Give targeted update capabilities that:
- Allow only certain parts of the ‘Read’ model to be modified.
- Allow things that are not even in the ‘Read’ model to be modified or provided if needed.
Selectively give users permissions to only the ‘Actions’ or Commands they need.
Log every Action and replay them at a later date to rebuild your data (i.e. Event Sourcing).
Capture what is requested (i.e. Discount the product) and respond immediately before the change has actually been made, safe in the knowledge you will eventually process the request and achieve “Eventual Consistency”.
Capture more information about what is happening (i.e. User X discounted Marmite by 15% is much better than User X updated Marmite).
Create Actions that manipulate a lot of entities simultaneously (i.e. POST ~/Categories(‘Yeast Spreads’)/Products/Discount…)

Of course simply separating the read/write models in OData doesn’t give you all of these advantages immediately, but at least it creates a foundation that can scale to support things like Event Sourcing or Eventual Consistency later if required.

Some of you may be thinking you can achieve many of these goals by having a more granular model that makes things like “Discount” a resource, and that would be true. However for most people using OData that way of thinking is foreign and more importantly the EDM foundations of OData get in the way a little too. So for me Actions seems like the right approach in OData.

I love this.
But what do you think?
-Alex

Judgment

Alex D James — Tue, 07 Jun 2011 06:20:00 GMT

In my last post I blithely said 'This is where judgment comes in...'.

Unfortunately judgment isn't easy.

Now when I was younger - say 10 years ago - I was sure about everything - not least my latest harebrained API design. I chalk that up to inexperience and not knowing better.

But now after 15 years of making design mistakes I have a lot more experience. So much so that I can literally see problems with almost every design. A big part of this is experience morphing into instinct.

Instinct is very useful because it helps weed out absurd ideas quickly. But instinct has limits. It is not much use if you are feeling equally uneasy about 3 possible designs.

You need to realize your instincts are based on all your experiences, and no one project encompasses all your experiences. Which means you can easily can fooled into thinking something matters when it really doesn't.

For example if you feel uneasy about one of the possible solutions because it looks inefficient. It might be your 'performance is important instinct' kicking in. If you are working on a website expected to get millions of hits a day, that is a valid concern. But if this is a departmental app with 10 users who each hit the site no more than 10 times a day, then probably not so much.

I know this performance example is contrived, but believe me often it won't be so obvious. You need a reliable way to discard irrelevant concerns and focus on the real issues.

The best way to do this is to have a set of guiding principles for every project you work on. Things like:

Simplicity is a feature
Predictability & Uniformity
Embrace existing standards
Payload Driven (HATEOS)

Creating a list like this gives you something more concrete than instincts to test when you get that uneasy feeling.

If you have an uneasy feeling you simple ask yourself, Does this violate any of my projects guiding principles?

I generally find that uneasy feelings will vanish or become showstoppers when you do this.
Enough so that you'll be much more comfortable making a choice...

... perhaps even the right choice!

How do I do design?

Alex D James — Fri, 03 Jun 2011 21:14:00 GMT

For a while now I've been thinking that the best way to get better at API & protocol design is to try to articulate how you do design.

Articulating your thought process has a number of significant benefits:

Once you know your approach you can critique and try to improve it.
Once you know what you'd like to do, in a collaborative setting you can ask others for permission to give it a go. In my experience phrases like "I'd like to explore how this is similar to LINQ ..." work well if you want others to brainstorm with you.
Once you understand what you do perhaps you can teach or mentor.
Or perhaps others can teach you.

Clearly these are compelling.

So I thought I'd give this a go by trying to document an idealized version of the thought process for Any/All in OData...

The Problem

OData had no way to allow people to query entities based on properties of a related collection, and a lot of people, myself included, wanted to change this...

Step 1: Look for inspiration

I always look for things that I can borrow ideas from. Pattern matching, if you will, with something proven. Here you are limited only by your experience and imagination, there is an almost limitless supply of inspiration. The broader your experience the more source material you have and the higher the probability you will be able to see a useful pattern.

If you need it – and you shouldn’t - this is just another reason to keep learning new languages, paradigms and frameworks!

Sometimes there is something obvious you can pattern match with, other times you really have to stretch...

In this case the problem is really easy. Our source material is LINQ. LINQ already allows you to write queries like this:

from movie in Movies
where movie.Actors.Any(a => a.Name == 'Zack')
select movie;

Which filters movies by the actors (a related collection) in the movie, and that is exactly what we need in OData.

Step 2: Translate from source material to problem domain

My next step is generally to take the ideas from the source material and try to translate them to my problem space. As you do this you'll start to notice the differences between source and problem, some insignificant, some troublesome.

This is where judgment comes in, you need to know when to go back to the drawing board. For me I know I'm on thin ice if I start saying things like 'well if I ignore X and Y, and imagine that Z is like this' .

But don't give up too early either. Often the biggest wins are gained by comparing things that seem quite different until you look a little deeper and find a way to ignore the differences that don't really matter. For example Relational Databases and the web seem very different, until you focus on ForeignKeys vs Hyperlinks, pull vs push seems different until you read about IQueryable vs IObservable.

Notice not giving up here is all about having tolerance for ambiguity, the higher your tolerance, the further you can stretch an idea. Which is vital if you want to be really creative.

In the case of Any/All it turns out that the inspiration and problem domain are very similar, and the differences are insignificant.

So how do you do the translation?

In OData predicate filters are expressed via $filter, so we need to convert this LINQ predicate:

(movie) => movie.Actors.Any(actor => actor.Name == 'Zack')

into something we can put in an OData $filter.

Let's attack this systematically from left to right. In LINQ you need to name the lambda variable in the predicate, i.e. movie, but in OData there is no need to name the thing you are filtering, it is implicit, for example this:

from movie in Movies
where movie.Name == "Donnie Darko"
select movie

is expressed like this in OData:

~/Movies/?$filter=Name eq 'Donnie Darko'

Notice there is no variable name, we access the Name of the movie implicitly.

So we can skip the ‘movie’.

Next in LINQ the Any method is a built-in extension method called directly off the collection using '.Any'. In OData '/' is used in place of '.', and built-in methods are all lowercase, so that points are something like this:

~/Movies/?$filter=Actors/any(????)

As mentioned previously in OData we don't name variables, everything is implicit. Which means we can ignore the actor variable. That leaves only the filter, which we can convert using existing LINQ to OData conversion rules, to yield something like this:

~/Movies/?$filter=Actors/any(Name eq 'Zack')

Step 3: Did you lose something important?

There is a good chance you lost something important in this transformation, so my next step is generally to assess what information has been lost in translation. Paying particular attention to things that are important enough that your source material had specific constructs to capture them.

As you notice these differences you either need to convince yourself the differences don’t matter, or you need to add something new to your solution to bring it back.

In our case you'll remember that in LINQ the actor being tested in the Any method had a name (i.e. 'actor'), yet in our current OData design it doesn't.

Is this important?

Yes it is! Unlike the root filter, where there is only one variable in scope (i.e. the movie), inside an Any/All there are potentially two variables in scope (i.e. the movie and the actor). And if neither are named we won't be able to distinguish between them!

For example this query, which finds any movies with the same name as any actors who star in the movie, is unambiguous in LINQ:

from movie in Movies
where movie.Actors.Any(actor => actor.Name == movie.Name)
select movie;

But our proposed equivalent is clearly nonsensical:

~/Movies/?$filter=Actors/any(Name eq Name)

It seems we need a way to refer to both the inner (actor) and outer variables (movie) explicitly.

Now we can't change the way existing OData queries work - without breaking clients and servers already deployed - which means we can't explicitly name the outer variable, we can however introduce a way to refer to it implicitly. This should be a reserved name so it can't collide with any properties. OData already uses the $ prefix for reserved names (i.e. $filter, $skip etc) so we could try something like $it . This results in something like this:

~/Movies/?$filter=Actors/any(Name eq $it/Name)

And now the query is unambiguous again.

But unfortunately we aren't done yet. We need to make sure nesting works calls works too, for example this:

from movie in Movies
where movie.Actors.Any(actor => actor.Name == movie.Name && actor.Awards.Any(award => award.Name == 'Oscar'))
select movie;

If we translate this, with the current design we get this:

~/Movies/?$filter=Actors/any(Name eq $it/Name AND Awards/any(Name eq 'Oscar'))

But now it is unclear whether Name eq 'Oscar' refers to the actor or the award. Perhaps we need to be able to name the actor and award variables too. Here we are not restricted by the past, Any/All is new, so it can include a way to explicitly name the variable. Again we look at LINQ for inspiration:

award => award.Name == 'Oscar'

Cleary we need something like '=>' that is URI friendly and compatible with the current OData design. It turns out ':' is a good candidate, because it works well in querystrings, and isn’t currently used by OData, and even better there is a precedent in Python lambda’s (notice the pattern matching again). So the final proposal is something like this:

~/Movies/?$filter=Actors/any(actor: actor/Name eq $it/Name)

Or for the nested scenario:

~/Movies/?$filter=Actors/any(actor: actor/Name eq $it/Name AND actor/Awards/any(award: award/Name eq 'Oscar'))

And again nothing is ambiguous.

Step 4: Rinse and Repeat

In this case the design feels good, so we are done.

But clearly that won't always be the case. However you should at least know whether the design is looking promising or not. If not it is back to the drawing board.

If it does look promising, I would essentially repeat steps 1-3 again using some other inspiration to inform further tweaking of the design, hopefully to something that feels complete.

Conclusion

While writing this up I definitely teased out a number of things that had previously been unconscious, and I sure this will help me going forward, hopefully you too?

Next time I do this I'll explore something that involves more creativity... and I'll try to tease out more of the unconscious tools I use.

What do you think about all this? Do you have any suggested improvements?

-Alex

TechEd 2010 Videos

Alex D James — Sat, 12 Jun 2010 16:05:11 GMT

If you didn't get to TechEd - no problem - videos of all the sessions are available here.

Including my Best Practices: Creating an OData Service using WCF Data Services session.

If you watch that video and want to follow try it out - you can download all the source code I uploaded previously.

Enjoy.

Tip 56 - Writing an OData Service using the Reflection Provider

Alex D James — Fri, 11 Jun 2010 18:05:00 GMT

At TechEd I got a lot of questions about how to expose Data as OData

By now you probably know you can use Data Services and the Entity Framework to expose data from a database as an OData Service. You might even know you can use Data Services with a custom Data Service Provider to expose arbitrary data from anywhere.

But did you know about the Data Services Reflection provider?.

Turns out reflection provider is VERY simple to use.

To show you just how simple I am going to create an odata service to expose some in memory data.

First you need some data, so how about this topical data:

[EntityPropertyMapping("Name", 
                        SyndicationItemProperty.Title, 
                        SyndicationTextContentKind.Plaintext, 
                        false)]
[EntityPropertyMapping("Odds", 
                        SyndicationItemProperty.Summary, 
                        SyndicationTextContentKind.Plaintext, 
                        false)]
public class Team {     
   public static List<Team> Teams = new List<Team>{
         new Team {ID = 1, Name = "New Zealand", Odds = "1:1000"},
         new Team {ID = 2, Name = "Paraquay", Odds = "1:50"},
         …     }; 

    public int ID{get;set;}
    public string Name {get;set;}
    public string Odds { get; set; }
}
[DataServiceKey("Name")]
[EntityPropertyMapping("Name", 
                        SyndicationItemProperty.Title,
                        SyndicationTextContentKind.Plaintext, 
                        false)]
public class Group
{
    public static List<Group> Groups = new List<Group>{ 
      new Group { 
         Name = "A", 
         Teams = new List<Team>{ 
             Team.Teams[0],
             Team.Teams[1],
             Team.Teams[2],
             Team.Teams[3]
         }
       },      
      new Group {
         Name = "B",
         Teams = new List<Team>{
             Team.Teams[4],
             Team.Teams[5],
             Team.Teams[6],
             Team.Teams[7]
          }
       },
      …
    };
    public string Name {get;set;}
    public List<Team> Teams {get;set;}
}

Notice the use of the [DataServiceKey] and [EntityPropertyMapping] attributes:

[DataServiceKey] tells Data Services which property(s) is the key. This is only required if the key isn’t ID or {ClassName}ID.
[EntityPropertyMapping] tells Data Services to map the property to a standard atom element. This is not required – but it is a best practice – it makes the resulting feed easier to read in a browser.

Next you need a class to act as your Data Source. Data Services will expose all the IQueryable properties as Feeds and infer types for all the types exposed by those Feeds.

So if I want two feeds, to expose Groups and Teams from the world cup, I’d do it like this:

public class WorldCupData
{
     public IQueryable<Team> Teams{
        get{
            return Team.Teams.AsQueryable();
        }
     }
     public IQueryable<Group> Groups{
        get{ 
           return Group.Groups.AsQueryable();

       }
     }
}

Now all you need to do is create the Data Service, and expose our sets. Simply add a WCF Data Service to you web application and modify the generated code to look like this:

public class WorldCup : DataService<WorldCupData>
{
     // This method is called only once to initialize service-wide policies.
     public static void InitializeService(DataServiceConfiguration config)     {
         config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
         config.SetEntitySetPageSize("*", 100);
         config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
     }
}

That’s it you are done:

A couple of points worth noting:

This service is read-only. If you want to make it read-write you have to implement IDataServiceUpdateProvider.
In this example the data comes from in-memory arrays, however if you have an IQueryable that supports it your data can come from anywhere.
Yes I know my data is all wrong, I’m writing this on a SouthWest flight from New Orleans to Denver with no internet. But thankfully there is a real OData Service for the World Cup here.

Enjoy.

OData - WCF Data Services Best Practices from TechEd

Alex D James — Thu, 10 Jun 2010 23:01:00 GMT

Yesterday I promised to share all the code from my Best Practices – Creating an OData Service using WCF Data Services session at TechED.

Note: you can find a recording of that session here.

So here goes, essentially this is what I did:

Downloaded, unzipped, opened and ran the MVC Music Store Sample.
Added an album to my cart, registered and ordered the album.
Added a data service to that project to expose the Entity Framework model already in the Music Store sample:

public class MusicService : DataService<MusicStoreEntities>
Added specific EntitySetAccessRules:

// We don’t want the Data service to expose carts at all config.SetEntitySetAccessRule(
   "Carts",
    EntitySetRights.None
);
// You can only get 1 OrderDetail at a time
config.SetEntitySetAccessRule(
   "OrderDetails",
    EntitySetRights.ReadSingle
);
// Everything else you can read & query.
config.SetEntitySetAccessRule(
   "*",
   EntitySetRights.AllRead
);
Added Server Driven Paging limits:

config.SetEntitySetPageSize("*", 5);
Added a query interceptor to only allow users to see just the own orders:

[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> OrdersFilter()
{
    var user = HttpContext.Current.User.Identity.Name;
    if (string.IsNullOrEmpty(user))
        return (Order o) => false;
    else if (user== "Administrator")
        return (Order o) => true;
    else
        return (Order o) => o.Username == user;
}
Made my service web browser friendly by Configuring the EDMX to map Genre.Name to the Entry/Title and Genre.Description to the Entry/Summary.

See the EDMX in the final copy of the source to see how.
Added a ClientAccessPolicy.xml so that Silverlight apps hosted on different sites can interact with our Data Service.

NOTE: The two sites in question need to be in the same internet zone!
Demoed using some JQuery JSON code that accesses the MusicService from the same site.

NOTE: this doesn’t work x-site, for that you need JSONP.
Added support for JSONP by applying the [DataServicesJSONP.JSONPSupportBehavior] attribute to our MusicService.

You can download the source from CodeGallery.
Demoed some JQuery JSONP code that accesses the MusicService from a remote site.
Configured ASP.NET to expose an authentication service, so non-browser agents can easily connect and logon to the Forms Authentication service, by adding this to the web.config:

<system.web.extensions>
<scripting>
    <webServices>
      <authenticationService enabled="true" requireSSL="false"/>
    </webServices>
</scripting>
</system.web.extensions>

Tested the authentication service using Fiddler by sending this RAW request to authenticate:

POST http://localhost:1397/Authentication_JSON_AppService.axd/Login HTTP/1.1
Content-Type: application/json

{ "userName": "Alex", "password": "password", "createPersistentCookie":false}
Opened this Data Services client application to show how to authenticate against the forms authentication service.

You can download the finally copy of the Music Service code if you want.

Enjoy and good luck.

Creating a Data Service Provider – Part 9 – Un-typed

Alex D James — Tue, 02 Mar 2010 21:14:57 GMT

Background info: This post builds on Parts 1 thru 8 which teach you how to create a typed Data Service Provider.

But now the part we’ve all been waiting for- an un-typed DSP – which is what I expect most real world DSP implementations will be based on.

What do we mean by Un-typed?

An un-typed DSP is needed whenever there isn’t a matching CLR class for each ResourceType. Typically you there is some general purpose class used for every ResourceType, something like a Dictionary for instance.

Over the course of the rest of this Post we will look at converting our DSP to expose product and category data stored in two Lists of Dictionaries, i.e. a List for each ResourceSet/Feed and a Dictionary for each Resource.

Why Un-typed?

Un-typed DSPs have a lot of interesting advantages:

You don’t need CLR classes for all the different ResourceTypes we want to expose
You don’t need to recompile / restart the service if the ResourceTypes changes in some way, i.e. you add a new Property.
You can store a pointer to the metadata for the ResourceType directly in the ResourceType itself.
You can store extra data along with a Resource very easily, which potentially allows us to take advantage of Data Services support for OpenTypes and OpenProperties.
You can evolve you model at runtime to expose more ResourceTypes/Sets

So now we know why, lets get started.

Metadata Changes:

The first thing we need to do create all our ResourceTypes:

var productType = new ResourceType(
    // CLR type backing this Resource
    typeof(Dictionary<string,object>),
    // Entity, ComplexType etc
    ResourceTypeKind.EntityType,
    null, // BaseType
    "Namespace", // Namespace
    "Product", // Name
    false // Abstract?
);
productType.CanReflectOnInstanceType = false;

Note there are two big differences between this and how we did things for Strongly Typed Resource Types (see part 3):

We tell Data Service that Products are stored in a Dictionary<string,object>
We tell Data Service that Dictionary<string,object> doesn’t have the same CLR inheritance hierarchy as the ResourceType itself – via CanReflectOnInstanceType.
This means whenever Casts are required rather than injecting standard CLR casts into the generated Query Expression, Data Services will inject methods calls the underlying injecting query provider is expected to translate.

Next we create our properties just like before:

var prodKey = new ResourceProperty(
   "ProdKey",
   ResourcePropertyKind.Key |
   ResourcePropertyKind.Primitive,
   ResourceType.GetPrimitiveResourceType(typeof(int))
);

But we add one extra call:

prodKey.CanReflectOnInstanceTypeProperty = false;

This tells DataServices that the declaring type – in this case Dictionary<string,object> - doesn’t actually have a “ProdKey” property, so a PropertyAccessor expression can’t be used.

With this setup this:

GET ~/Products(2)

Will no longer produce an (pseudo) expression like this:

from p in _queryProvider.GetQueryRootForResourceSet(productsSet)
where p.ProdKey == 2
select p;

Which would be invalid because ProdKey is not a property of Dictionary<string,object>, instead it will produce something like this:

from p in _queryProvider.GetQueryRootForResourceSet(productsSet)
where ((int) DataServiceProviderMethods.GetValue(p,"ProdKey"))
== 2
select p;

Which gives your query provider the opportunity to re-write the expression as:

from p in _queryProvider.GetQueryRootForResourceSet(productsSet)
where ((int) p["ProdKey"]) == 2
select p;

Which for example LINQ to Objects can handle.

NOTE: If your data isn’t in memory – maybe it is in a database – you still have to translate this expression into something like SQL, so it might just be easier to do that in one visit.

Once you’ve updated all your types accordingly you should have no problem getting ~/root.svc or ~/root.svc/$metadata working.

Data Source Changes:

Next we need to make some changes to our Data Sources so that the data is stored in Dictionaries.

If you remember we had this abstract base class from which our Data Source should inherit:

public abstract class DSPContext
{
    public abstract IQueryable GetQueryable(
       ResourceSet resourceSet);

    public abstract object CreateResource(
       ResourceType resourceType);

    public abstract void AddResource(
       ResourceSet resourceSet, object resource);

    public abstract void DeleteResource(
       object resource);

    public abstract void SaveChanges();
}

Here is an implementation that works over dictionaries:

public class SampleUntypedContext: DSPContext
{
    public const string RESOURCE_TYPE_KEY = "_ResourceType";

    static Dictionary<string, List<Dictionary<string, object>>>
       dataSources =
       new Dictionary<string, List<Dictionary<string, object>>>{
        {"Products", new List<Dictionary<string,object>>()},
        {"Categories", new List<Dictionary<string,object>>()}
    };

    public override IQueryable GetQueryable(
        ResourceSet resourceSet)
    {
        return GetDataSource(resourceSet.Name).AsQueryable();
    }

    public override void AddResource(
        ResourceSet resourceSet, object resource)
    {
        var resourceInstance = GetResourceInstance(resource);
        var instanceTypeName = GetResourceType(resourceInstance);
        if (instanceTypeName != resourceSet.ResourceType.FullName)
            throw new InvalidOperationException("Unexpected Resource Type");

        var dataSource = GetDataSource(resourceSet.Name);
        dataSource.Add(resourceInstance);
    }

    public override void DeleteResource(object resource)
    {
        var resourceInstance = GetResourceInstance(resource);
        var resourceInstanceType = GetResourceType(resource);

        List<Dictionary<string,object>> datasource = null;
        if (resourceInstanceType == "Product")
            datasource = GetDataSource("Products");
        else if (resourceInstanceType == "Category")
            datasource = GetDataSource("Categories");
        else
            throw new Exception("ResourceSet not found");
        datasource.Add(resourceInstance);
    }
    public override void SaveChanges()
    {
        var products = GetDataSource("Products");
        var categories = GetDataSource("Categories");

        var prodKey = products.Max(p => (int)p["ProdKey"]);
        foreach (var prod in products
                    .Where(p => ((int) p["ProdKey"])== 0))
        {
            prod["ProdKey"] = ++prodKey;
        }

Again you can see that this has some repetitive code that we could be generalize.

We hold our Lists of Dictionary<string,object> in a Dictionary keyed on the ResourceSet name.

And as before our SaveChanges() function mimic server generated values for primary keys.

Perhaps the most interesting function is the implementation of CreateResource which adds all the properties to the Dictionary and assigns default values for each property, mimicking a strongly typed class constructor.

Now that we have a Data Source we need to fill it with some data. In the typed example we did this by overriding CreateDataSource, so no surprises, we need to do that again:

protected override SampleUntypedContext CreateDataSource()
{
    var products = SampleUntypedContext
                        .GetDataSource("Products");
    var categories = SampleUntypedContext
                        .GetDataSource("Categories");
    if (products.Count == 0)
    {
        var bovril = new Dictionary<string,object>
        {
            {"_ResourceType", "Namespace.Product"},
            {"ProdKey", 1},
            {"Name", "Bovril"},
            {"Cost", 4.35M},
            {"Price", 6.49M}
        };
        var marmite = new Dictionary<string, object>
        {
            {"_ResourceType", "Namespace.Product"},
            {"ProdKey", 2},
            {"Name", "Marmite"},
            {"Cost", 4.97M},
            {"Price", 7.21M}
        };
        var food = new Dictionary<string, object>
        {
            {"_ResourceType", "Namespace.Category"},
            {"ID", 1},
            {"Name", "Food"}
        };
        // Build bi-directional relationships
        food["Products"] = new List<Dictionary<string, object>> {
            marmite,
            bovril
        };
        marmite["Category"] = food;
        bovril["Category"] = food;

        // Store resources
        products.Add(marmite);
        products.Add(bovril);
        categories.Add(food);
    }
    return base.CreateDataSource();
}

IDataServiceMetadataProvider changes:

There are no real changes required here, because essentially this is just returning metadata, and the same code we wrote for the Typed provider will work still.

IDataServiceQueryProvider changes:

Here we need to change three things.

The way we work out the ResourceType for a resource:

public ResourceType GetResourceType(object target)
{
return GetResourceType(GetResource(target));
}

private Dictionary<string, object> GetResource(object target)
{
    var dict = target as Dictionary<string, object>;
    if (dict == null)
    {
        throw new InvalidCastException(
            "Resource is not a dictionary"
        );
    }
    return dict;
}

private ResourceType GetResourceType(
    Dictionary<string, object> resource)
{
    ResourceType type = null;
    if (!resource.ContainsKey("_ResourceType"))
        throw new InvalidOperationException(
            "ResourceType not known"
        );

    if (!_metadata.TryResolveResourceType(
        resource["_ResourceType"].ToString(),
        out type)
    )
        throw new InvalidOperationException(
            "ResourceType not found"
        );
    return type;
}

The way we get property values:

public object GetPropertyValue(
    object target,
    ResourceProperty resourceProperty)
{
    var resource = GetResource(target);
    if (resource.ContainsKey(resourceProperty.Name))
        return resource[resourceProperty.Name];
    throw new InvalidOperationException("Property not found");
}

And finally the way we return query roots.

If we continue to do what we did in part 5:

public IQueryable GetQueryRootForResourceSet(
ResourceSet resourceSet)
{
return _currentDataSource.GetQueryable(resourceSet);
}

It will work for things URLs like
~/Products
~/Categories

But as soon as we do anything interesting like

~/Products(1)
~/Products(1)/Category
~/Categories(1)/Products
~/Products/?$filter=Name eq 'Marmite'

It will all fall over in a crashing heap.

Because we are Un-typed, DataServices will inject calls to DataServiceProviderMethods.XXX into the query, and then LINQ to Objects will just blindly call those methods – which unfortunately can’t be called directly.

We need to write a wrapped IQueryable that wraps the LINQ to Objects queryable and rewrites the expression so LINQ to Objects can deal with it.

Turns out I wrote a blog post just yesterday on this :)

In that blog post I created an InterceptedQuery<> and InterceptingProvider that wrap an IQueryable and allow you to visit and alter the expression – which is exactly what we need to do – before it passing it on to the underlying query provider.

So our method now looks like this:

public IQueryable GetQueryRootForResourceSet(
    ResourceSet resourceSet)
{
    var underlying = _currentDataSource.GetQueryable(resourceSet);
    return InterceptingProvider.Intercept(underlying, dspVisitor);
}

Visitor Implementation

The interesting piece of course is dspVisitor. This is an instance of a class derived from System.Linq.Expressions.ExpressionVisitor that looks like this:

public class DSPExpressionVisitor: ExpressionVisitor
{
    static readonly MethodInfo GetValueMethodInfo =
        typeof(DataServiceProviderMethods)
        .GetMethod(
            "GetValue",
            BindingFlags.Static | BindingFlags.Public,
            null,
            new Type[] {
                typeof(object),
                typeof(ResourceProperty)
            },
            null
        );

    static readonly MethodInfo GetSequenceValueMethodInfo =
        typeof(DataServiceProviderMethods)
        .GetMethod(
            "GetSequenceValue",
            BindingFlags.Static | BindingFlags.Public,
            null,
            new Type[] {
                typeof(object),
                typeof(ResourceProperty)
            },
            null
        );

    static readonly MethodInfo ConvertMethodInfo =
        typeof(DataServiceProviderMethods)
        .GetMethod(
           "Convert",
           BindingFlags.Static | BindingFlags.Public
        );

    static readonly MethodInfo TypeIsMethodInfo =
        typeof(DataServiceProviderMethods)
        .GetMethod(
            "TypeIs",
            BindingFlags.Static | BindingFlags.Public
        );

// Replace calls to DataServiceProviderMethods.GetValue
// with this expression.
// we cast to a dictionary, and then use the dictionary
// item accessor to get the property.
static readonly Expression<Func<object, ResourceProperty, object>> GetValueReplacement =
(o, rp) => (o as Dictionary<string,object>)[rp.Name];

// To check that something is a particular type we cast to
// Dictionary and compare the _ResourceType field with the
// fullname of the ResourceType.
// Obviously this approach won’t handle inheritance.
static readonly Expression<Func<object, ResourceType, bool>> TypeIsReplacement =
(o, rt) => ((string)(o as Dictionary<string, object>)["_ResourceType"]) == rt.FullName;

   protected override Expression VisitMethodCall(
       MethodCallExpression node
   )
   {
        if (node.Method == GetValueMethodInfo)
        {
            // Arguments[0] - the resource to get property from
            // Arguments[1] - the ResourceProperty to get
            // Invoke the replacement expression, passing the
            // appropriate parameters.
            return Expression.Invoke(
               Expression.Quote(GetValueReplacement),
               node.Arguments[0],
               node.Arguments[1]
            );
        }
        else if (node.Method.IsGenericMethod &&
            node.Method.GetGenericMethodDefinition() ==
           GetSequenceValueMethodInfo)
        {
            // Arguments[0] - the resource
            // Arguments[1] - the Property that is a sequence
            // Just call the GetValueReplacement(0,1) and
            // cast it to IEnumerable<T> which is the
            // correct return type
            return Expression.Convert(
                Expression.Invoke(
                    Expression.Quote(GetValueReplacement),
                    node.Arguments[0],
                    node.Arguments[1]
                ),
                node.Method.ReturnType);
        }
        else if (node.Method == TypeIsMethodInfo)
        {
            // Arguments[0] – the resource
            // Arguments[1] – the ResourceType
            // Invoke the TypeIsReplacement expression
            // binding to the resource & resourceType
            return Expression.Invoke(
               Expression.Quote(TypeIsReplacement),
               node.Arguments[0],
               node.Arguments[1]
            );
        }
        else if (node.Method == ConvertMethodInfo)
        {
            // Arguments[0] – the resource
            // Arguments[0] – the ResourceType
            // no need to do anything, so just
            // return the argument
            return this.Visit(node.Arguments[0]);
        }
        return base.VisitMethodCall(node);
    }
}

This visitor class grabs hold of MethodInfo’s for each of the DataServiceProviderMethods methods, and stores them once for performance reasons.

Then it overrides VisitMethodCall looking for each of those MethodInfo’s in turn, replacing them with an expression that LINQ to Objects can handle.

To do this we use a little trick to feed in our replacementExpressions:

Expression.Invoke(
   Expression.Quote(replacementLambda),
   argument0,
   argument1
)

Using this approach we don’t have to manually construct the lambda expression, which I think makes our visitor easier to read.

Remember while you are doing this translation / replacement that you need to create expression that your underlying IQueryable provider can actually handle.

For example LINQ to Objects can handle these expressions without any problem, but the Entity Framework would choke because it can’t translate Invoke Expressions in to SQL.

NOTE: if you using .NET 3.5 – where System.Linq.Expressions.ExpressionVisitor isn’t available – you can still use a visitor derived IQToolkit’s base visitor or from this one.

IDataServiceUpdateProvider Changes:

This one is not too tricky: you just need to change each methods so they work against dictionaries rather than strongly typed classes.

Basically we need to take the IDataServiceUpdateProvider implementation in part 7 and update each method in turn to work against a dictionary.

Generally this actually makes things easier (and faster) because we no longer need to use reflection.

For example SetValue is a lot easier now:

public void SetValue(
   object targetResource,
   string propertyName,
   object propertyValue)
{
     try
     {
         _actions.Add(() => ReallySetValue(
             targetResource,
             propertyName,
             propertyValue)
         );
     }
     catch { }
}

public void ReallySetValue(object targetResource, string propertyName, object propertyValue)
{
var resource = targetResource as Dictionary<string, object>;
resource[propertyName] = propertyValue;
}

Putting it altogether

If you make all these changes, you should now have a fully functional, Read/Write Un-typed Data Service Provider!

Congratulations, the world is now officially your oyster…

Tip 55 - How to extend an IQueryable by wrapping it.

Alex D James — Mon, 01 Mar 2010 21:11:00 GMT

Over the last couple of years I’ve found myself in lots of situations where I’ve wanted to get ‘under the hood’ and see what is happening inside an IQueryable, but I haven’t had an easy solution, at least until now.

Getting down and dirty like this is interesting because it means you can:

Log Queries before they are executed
Rewrite expressions, for example replacing expressions that are not supported by a provider - EF, LINQ to SQL, LINQ to Objects etc - with expressions that are.

Anyway, I was very pleased a while back when reviewing some sample code by Vitek, who has a new blog by the way, and I realized I could generalize it to create an InterceptedQuery<> and an InterceptingProvider.

The basic idea is that you can use it like this:

public IQueryable<Customer> Customers
{
   get{
      return InterceptingProvider.CreateQuery(_ctx.Customers, visitor);
   }
}

Here visitor, is an ExpressionVisitor that will visit, and potentially rewrite, any queries composed with Customers before it is handed off to the underlying queryable (in this case the Entity Framework).

Implementation

Thanks to Vitek, the implementation is actually pretty simple.

Lets start with the implementation of the InterceptedQuery<> which is trivial:

public class InterceptedQuery<T> : IOrderedQueryable<T>
{
private Expression _expression;
private InterceptingProvider _provider;

    public InterceptedQuery( 
       InterceptingProvider provider, 
       Expression expression) 
    { 
        this._provider = provider; 
        this._expression = expression; 
    }  
    public IEnumerator<T> GetEnumerator() 
    { 
        return this._provider.ExecuteQuery<T>(this._expression); 
    }  
    IEnumerator IEnumerable.GetEnumerator() 
    { 
        return this._provider.ExecuteQuery<T>(this._expression); 
    }  
    public Type ElementType 
    { 
        get { return typeof(T); } 
    }  
    public Expression Expression 
    { 
        get { return this._expression; } 
    }  
    public IQueryProvider Provider 
    { 
        get { return this._provider; } 
    } 
}

Next lets look at InterceptingProvider which is much more interesting:

public class InterceptingProvider : IQueryProvider
{
private IQueryProvider _underlyingProvider;
private Func<Expression,Expression>[] _visitors;

    private InterceptingProvider(
        IQueryProvider underlyingQueryProvider,
        params Func<Expression,Expression>[] visitors)
    {
        this._underlyingProvider = underlyingQueryProvider;
        this._visitors = visitors;
    }

    public static IQueryable<T> Intercept<T>(
        IQueryable<T> underlyingQuery,
        params ExpressionVisitor[] visitors)
    {
        Func<Expression, Expression>[] visitFuncs =
            visitors
            .Select(v => (Func<Expression, Expression>) v.Visit)
            .ToArray();
        return Intercept<T>(underlyingQuery, visitFuncs);
    }

    public static IQueryable<T> Intercept<T>(
        IQueryable<T> underlyingQuery,
        params Func<Expression,Expression>[] visitors)
    {
        InterceptingProvider provider = new InterceptingProvider(
            underlyingQuery.Provider,
            visitors
        );
        return provider.CreateQuery<T>(
            underlyingQuery.Expression);
    }
    public IEnumerator<TElement> ExecuteQuery<TElement>(
        Expression expression)
    {
        return _underlyingProvider.CreateQuery<TElement>(
            InterceptExpr(expression)
        ).GetEnumerator();
    }
    public IQueryable<TElement> CreateQuery<TElement>(
        Expression expression)
    {
        return new InterceptedQuery<TElement>(this, expression);
    }
    public IQueryable CreateQuery(Expression expression)
    {
        Type et = TypeHelper.FindIEnumerable(expression.Type);
        Type qt = typeof(InterceptedQuery<>).MakeGenericType(et);
        object[] args = new object[] { this, expression };

        ConstructorInfo ci = qt.GetConstructor(
            BindingFlags.NonPublic | BindingFlags.Instance,
            null,
            new Type[] {
                typeof(InterceptingProvider),
                typeof(Expression)
            },
            null);

        return (IQueryable)ci.Invoke(args);
    }
    public TResult Execute<TResult>(Expression expression)
    {
        return this._underlyingProvider.Execute<TResult>(
            InterceptExpr(expression)
        );
    }
    public object Execute(Expression expression)
    {
        return this._underlyingProvider.Execute(
            InterceptExpr(expression)
        );
    }
    private Expression InterceptExpr(Expression expression)
    {
        Expression exp = expression;
        foreach (var visitor in _visitors)
            exp = visitor(exp);
        return exp;
    }
}

Notice that whenever the Query is executed, we Intercept the current Expression, which involves calling all registered ‘visitors’ in turn, and then executing the resulting expression against the underlying provider.

Implementation Notes:

At its foundation our implementation uses Func<Expression,Expression> rather than .NET 4.0’s System.Linq.Expressions.ExpressionVisitor, mainly because .NET was a little late to the party, so there are lots of visitors that don’t derive from System.Linq.Expressions.ExpressionVisitor today.

You need look no further than Matt Warren’s excellent IQToolkit, for many such examples.

Nevertheless we want to encourage the use of System.Linq.Expressions.ExpressionVisitor so there is a convenience overload for that too.

Remember also that if you are wrapping the Entity Framework and doing any non-trivial expression rewrites you have to avoid any Invoke Expressions - see Colin’s blog post.

One of IQToolkit useful visitors is called ExpressionWriter and with a few minor mods – making both its constructor and the base Visit method public – you can use it to write the expression to the console before any Entity Framework query is executed:

CustomersContext _ctx = new CustomersContext();
ExpressionWriter _writer = new ExpressionWriter(Console.Out);

public IQueryable<Customer> Customers{
   get{
        return InterceptingProvider.Intercept(_ctx.Customers, _writer.Visit);
   }
}

You’ll also notice that we are using the IQToolkit’s useful TypeHelper class in our untyped CreateQuery method, it really helps us create the correct generic InterceptedQuery<> instance.

Thanks again Matt!

Putting it all together:

Lets show an end to end example.

Here I mimic what a WCF/ADO.NET Data Service has to do to handle this request:

GET ~/People/?$filter=Surname eq ‘James’

if it is backed by an un-typed DSP.

// Create some data
List<Dictionary<string, object>> data = new List<Dictionary<string, object>>();
data.Add(
    new Dictionary<string, object>{{"Surname", "James"}, {"Firstname", "Alex"}}
);
data.Add(
    new Dictionary<string, object>{{"Surname", "Guard"}, {"Firstname", "Damien"}}
);
data.Add(
    new Dictionary<string, object>{{"Surname", "Meek"}, {"Firstname", "Colin"}}
);
data.Add(
    new Dictionary<string, object>{{"Surname", "Karas"}, {"Firstname", "Vitek"}}
);
data.Add(
    new Dictionary<string, object>{{"Surname", "Warren"}, {"Firstname", "Matt"}}
);

// Create a couple of visitors
var writer = new ExpressionWriter(Console.Out);
var dspVisitor = new DSPExpressionVisitor();

// Intercept queries to the L2O IQueryable
var queryRoot = InterceptingProvider.Intercept(
   data.AsQueryable(), // L2O’s iqueryable
   writer.Visit, // What does the expression look like first?
   dspVisitor.Visit, // Replace GetValue().
   writer.Visit // What does the expression look like now?
);

// Create a Data Services handle for the Surname property
ResourceProperty surname = new ResourceProperty(
   "Surname",
   ResourcePropertyKind.Primitive,
   ResourceType.GetPrimitiveResourceType(typeof(string))
);

// Create a query without knowing how to access the Surname
// from x.
var query =
from x in queryRoot
where ((string) DataServiceProviderMethods.GetValue(x, surname))
== "James"
select x;

// Execute the query and print some results
foreach (var x in query)
    Console.WriteLine("Found Match:{0}",
       x["Firstname"].ToString()
    );

As can see we have some People data in a list of dictionaries, and we are trying to find just the People with a Surname of ‘James’.

The problem is that Data Services doesn’t know how to get the Surname from a Dictionary. So it injects a call to DataServiceProviderMethods.GetValue(..).

Fine.

Unfortunately at this point the LINQ to Objects query provider doesn’t have enough context to process this query - blindly calling GetValue like we are doing in this query will fail.

So we intercept the query, and the DSPExpressionVisitor (which I won’t go into here) simply replaces things like this:

DataServiceProviderMethods.GetValue(x, surname)

with this

x[surname.Name]

which if you look at surname you can see is the same as:

x["Surname"]

So when the whole expression is visited you end up with something like this:

var query =
from x in queryRoot
where ((string) x[surname.Name]) == "James"
select x;

Which Linq to Objects can handle just fine!

Summary

This is a general purpose solution that allows you to layer one IQueryable over another, and translate / rewrite / log the query expression before it is passed to the underlying provider.

Enjoy.

Creating a Data Service Provider – Part 8 - Relationships

Alex D James — Wed, 24 Feb 2010 02:00:00 GMT

In parts 1 thru 7 we made a Read / Write typed data service provider, although it was a little simplistic because it didn’t have relationships.

So lets rectify that right now by adding a relationship to make this more 'real-world'

Changes to our Classes

If you remember from Part 3 we had just a Product class, so lets add a Category and a one to many relationship between the two:

public class Category
{
private List<Product> _products = new List<Product>();

    public int ID { get; set; }
    public string Name { get; set; }
    public List<Product> Products { get { return _products; } }
}

public class Product
{
    public int ProdKey { get; set; }
    public string Name { get; set; }
    public Decimal Price { get; set; }
    public Decimal Cost { get; set; }
    public Category Category { get; set; }
}

Metadata Changes

Building relationships through metadata.

Now that our classes are ready, the next step is to tell Data Services about Category, and the relationships between then two classes:

Most of this is simple, except perhaps the bit that actually adds the relationship.

You start by creating the ‘navigation’ properties:

var productType = …; // Product and its primitive properties
var categoryType = …; // Category and its primitive properties

// Tell Data Services about product.Category
var prodCategory = new ResourceProperty(
    "Category",
    ResourcePropertyKind.ResourceReference,
    categoryType
);
productType.AddProperty(prodCategory);

// Tell Data Services about category.Products
var catProducts = new ResourceProperty(
    "Products",
    ResourcePropertyKind.ResourceSetReference,
    productType
);
categoryType.AddProperty(catProducts);

Because of the way the API works – you need the target ResourceType to create a property – both Resource Types need to be at least partially created before you can create and add the navigation properties.

This essentially means you need to do two passes over each ResourceType: One to create the type and its primitive properties and another to add it’s navigation properties.

You should also notice that product.Category is a ResourceReference, because there is just one category, whereas category.Products is a ResourceSetReference, because it is a collection.

Next we need a ResourceSet for both Category and Product:

var productsSet = new ResourceSet("Products", productType);
var categoriesSet = new ResourceSet("Categories", categoryType);

Once you done that we need to create an AssociationSet that links those two navigation properties and resource sets together:

ResourceAssociationSet productCategoryAssociationSet = new ResourceAssociationSet(
    "ProductCategory",
    new ResourceAssociationSetEnd(
       productsSet,
       productType,
       prodCategory
    ),
    new ResourceAssociationSetEnd(
       categoriesSet,
       categoryType,
       catProducts
    )
);

i.e. this is the bit that tells Data Services that setting:

product.Category = category;

is equivalent to:

category.Products.Add(product);

Next we store a pointer from these properties back to the ResourceAssociationSet using the CustomState property.

prodCategory.CustomState = productCategoryAssociationSet;
catProducts.CustomState = productCategoryAssociationSet;

If you remember the good old days of VB forms development CustomState will remind you of your old friend the Tag property. You can use it for all sorts of tricks.

Actually storing the ResourceAssociationSet in the ResourceProperty.CustomState is not strictly necessary, but it will make our lives easier, and our code quicker, as you’ll see very soon.

Finally we need to tell the DSPMetadataProvider, about both ResourceTypes, both ResourceSets and the ResourceAssociationSet:

metadata.AddResourceType(productType);
metadata.AddResourceSet(productsSet);
metadata.AddResourceType(categoryType);
metadata.AddResourceSet(categoriesSet);
metadata.AddAssociationSet(productCategoryAssociationSet);

IDataServiceMetadataProvider Changes

For this to work we need to make some changes to our DSPMetadataProvider to actually support Relationships:

First we need a place to hold AssociationSets:

List<ResourceAssociationSet> _associationSets
= new List<ResourceAssociationSet>();

Then we need a way to register new AssociationsSets:

public void AddAssociationSet(ResourceAssociationSet aset)
{
_associationSets.Add(aset);
}

And finally we need to update our implementation of IDataServiceMetadataProvider.GetResourceAssociationSet(..) like this:

public virtual ResourceAssociationSet GetResourceAssociationSet(
   ResourceSet resourceSet,
   ResourceType resourceType,
   ResourceProperty resourceProperty
)
{
    return resourceProperty.CustomState as ResourceAssociationSet
}

See I told you putting the ResourceAssociationSet in CustomState was going to come in handy!

The alternative is to write a LINQ query over the _associationSets list, using the method arguments in the predicate, but that is a little more complicated and a lot slower.

Metadata should be working by now…

By this point both the service document: ~/Sample.svc and metadata: ~/Sample.svc/$metadata should be working.

Next we move on to getting query and update working, and to do that we so we need to update our Data Source, and our implementation of IDataServiceQueryProvider.

Query Changes

Updating our DataSource

This just involves adding a List for my categories to ProductsContext, and updating the existing methods to expose Categories too:

public class ProductsContext: DSPContext
{
    private static List<Product> _products
       = new List<Product>();

    private static List<Category> _categories
       = new List<Category>();

    public override IQueryable GetQueryable(
       ResourceSet resourceSet)
    {
        if (resourceSet.Name == "Products")
           return Products.AsQueryable();
        else if (resourceSet.Name == "Categories")
           return Categories.AsQueryable();

        throw new NotSupportedException(
           string.Format("{0} not found", resourceSet.Name));
    }

    public static List<Product> Products{
       get { return _products; }
    }

    public static List<Category> Categories {
       get { return _categories; }
    }

    public override void AddResource(
       ResourceType resourceType, object resource)
    {
        if (resourceType.InstanceType == typeof(Product))
        {
            Product p = resource as Product;
            if (p != null)
            {
                Products.Add(p);
                return;
            }
        }
        else if (resourceType.InstanceType == typeof(Category))
        {
            Category c = resource as Category;
            if (c != null)
            {
                Categories.Add(c);
                return;
            }
        }
        throw new NotSupportedException(
           string.Format("{0} not found", resourceType.FullName));
    }

    public override void DeleteResource(object resource)
    {
        if (resource.GetType() == typeof(Product))
        {
            Products.Remove(resource as Product);
            return;
        }
        else if (resource.GetType() == typeof(Category))
        {
            Categories.Remove(resource as Category);
            return;
        }
        throw new NotSupportedException("Resource Not Found");
    }

    public override object CreateResource(
       ResourceType resourceType)
    {
        if (resourceType.InstanceType == typeof(Product))
            return new Product();
        else if (resourceType.InstanceType == typeof(Category))
            return new Category();
        throw new NotSupportedException(
           string.Format("{0} not found", resourceType.FullName));
    }

    public override void SaveChanges()
    {
        var prodKey = Products.Max(p => p.ProdKey);
        foreach (var prod in Products.Where(p => p.ProdKey == 0))
            prod.ProdKey = ++prodKey;

        var catKey = Categories.Max(p => p.ID);
        foreach (var cat in Categories.Where(c => c.ID == 0))
            cat.ID = ++catKey;
    }
}

As you can see this is mostly just a mechanical modification of the code we already had for Products. If I was doing this for real, I’d be looking are refactoring this code, to make adding another type a more pain free.

But I’ll leave that as an exercise for you guys.

The only thing remaining to get query working is to actually have some data:

protected override ProductsContext CreateDataSource()
{
    if (ProductsContext.Products.Count == 0)
    {
        var bovril = new Product
        {
            ProdKey = 1,
            Name = "Bovril",
            Cost = 4.35M,
            Price = 6.49M
        };
        var marmite = new Product
        {
            ProdKey = 2,
            Name = "Marmite",
            Cost = 4.97M,
            Price = 7.21M
        };
        var food = new Category
        {
            ID = 1,
            Name = "Food"
        };
        food.Products.Add(bovril);
        bovril.Category = food;
        food.Products.Add(marmite);
        marmite.Category = food;
        ProductsContext.Categories.Add(food);
        ProductsContext.Products.Add(bovril);
        ProductsContext.Products.Add(marmite);
    }
    return base.CreateDataSource();
}

Notice that I am building the relationships in both directions, sometimes this logic might be built into the classes themselves – it is what the Entity Framework calls relationship fix-up – if so this would be superfluous.

Either way if you try running your code at this point you should be able to get both query AND relationship traversal working.

i.e. getting the Category for a particular Product:

and getting the Products in a particular Category:

Its also interesting to notice that we didn’t need to make any changes to our implementation of IDataServiceQueryProvider to get Query working.

Update Changes

Our last challenge is getting Update working.

Which basically means implementing these three methods that we didn’t need last time:

public void SetReference(
   object targetResource,
   string propertyName,
   object propertyValue)
{
    throw new NotImplementedException();
}
public void AddReferenceToCollection(
   object targetResource,
   string propertyName,
   object resourceToBeAdded)
{
    throw new NotImplementedException();
}
public void RemoveReferenceFromCollection(
   object targetResource,
   string propertyName,
   object resourceToBeRemoved)
{
    throw new NotImplementedException();
}

Remember (from part 7) that our Update Providers need to delay doing updates until IDataServiceUpdateProvider.SaveChanges() is called, so we record our intent to SetReference as an action.

To make this easy we’ll create methods that do the real work and call theme inside actions we can invoke later:

public void SetReference(
   object targetResource,
   string propertyName,
   object propertyValue)
{
    _actions.Add(() => ReallySetReference(
                     targetResource,
                     propertyName,
                     propertyValue));
}

public void AddReferenceToCollection(
   object targetResource,
   string propertyName,
   object resourceToBeAdded)
{
    _actions.Add(() => ReallyAddReferenceToCollection(
                     targetResource,
                     propertyName,
                     resourceToBeAdded));
}

public void RemoveReferenceFromCollection(
   object targetResource,
   string propertyName,
   object resourceToBeRemoved)
{
    _actions.Add(() => ReallyRemoveReferenceFromCollection(
                      targetResource,
                      propertyName,
                      resourceToBeRemoved));
}

So what do those ReallyXXX() methods look like?

Well something to bear in mind here is that our classes are pretty naive – they don’t do any relationship fix-up – yet the relationships themselves are bi-directional, so our code needs to keep both ends of the relationship in sync.

This is what ReallySetReference(..) looks like:

public void ReallySetReference(
   object targetResource,
   string propertyName,
   object propertyValue)
{
    // Get the resource type.
    var targetType = targetResource.GetType();

    // Figure out the other end...
    var targetResourceType =
         _metadata.Types.Single(t => t.InstanceType==targetType);

    var targetResourceTypeProperty = targetResourceType
        .Properties
        .Single(p => p.Name == propertyName);

    var associationSet = targetResourceTypeProperty
        .CustomState as ResourceAssociationSet

    // Get the relatedEnd
    var relatedAssociationSetEnd = associationSet.End1
        .ResourceProperty == targetResourceTypeProperty ?
        associationSet.End2 : associationSet.End1;

    var relatedResourceTypeProperty = relatedAssociationSetEnd
                                         .ResourceProperty;

    // Get the relatedType and all Properties
    Type relatedType = relatedAssociationSetEnd
         .ResourceType
         .InstanceType;

    var targetTypeProperty = targetType
        .GetProperties()
        .Single(p => p.Name == propertyName);

    var relatedTypeProperty =
        relatedResourceTypeProperty == null ?
        null :
        relatedType.GetProperties()
        .Single(p => p.Name == relatedResourceTypeProperty.Name);

    // We need to fix up the other end
    // but only if this is a two way relationship
    if (relatedResourceTypeProperty != null)
    {
        // Get the other end and clears its
        // relationship to targetResource
        object originalValue = targetTypeProperty
           .GetPropertyValueFromTarget(targetResource);

        if (originalValue != null)
        {
            if (relatedAssociationSetEnd.ResourceProperty.Kind ==
                ResourcePropertyKind.ResourceReference)
            {
                // the other end is a reference so we check
                // that it is pointing to the targetResource
                // before nulling out.
                var backPointer =
                   relatedTypeProperty
                   .GetPropertyValueFromTarget(originalValue);

                if (object.ReferenceEquals(
                       backPointer,
                       targetResource)
                )
                    relatedTypeProperty.SetPropertyValueOnTarget(
                        originalValue, null);
            }
            else if (relatedAssociationSetEnd
                     .ResourceProperty.Kind ==
                     ResourcePropertyKind.ResourceSetReference)
            {
                // the other end is a collection
                // (i.e. a List in our implementation) so we can
                // safely remove targetResource without checking
                // whether it is in that list or not
                (relatedTypeProperty
                   .GetPropertyValueFromTarget(originalValue)
                   as IList
                ).Remove(targetResource);
            }
        }
        // Add the relationship to the other end...
        if (propertyValue != null)
        {
            if (relatedAssociationSetEnd.ResourceProperty.Kind ==
                ResourcePropertyKind.ResourceReference)
            {
                relatedTypeProperty.SetPropertyValueOnTarget(
                   propertyValue, targetResource);
            }
            else if (relatedAssociationSetEnd
                     .ResourceProperty.Kind ==
                      ResourcePropertyKind.ResourceSetReference)
            {
                (relatedTypeProperty
                    .GetPropertyValueFromTarget(propertyValue)
                    as IList
                ).Add(targetResource);
            }
        }
    }
    // actually set the reference !
    targetTypeProperty.SetPropertyValueOnTarget(
       targetResource, propertyValue
    );
}

As you can see this is pretty complicated, but if you look closely you'll notice most of the complexity comes from the fix-up logic. Indeed if the Product and Category classes did their own fix-up logic this would suffice:

public void ReallySetReference(
   object targetResource,
   string propertyName,
   object propertyValue)
{
    // Get the resource type.
    var targetType = targetResource.GetType();

    var targetTypeProperty = targetType
        .GetProperties()
        .Single(p => p.Name == propertyName);

    // actually set the reference !
    targetTypeProperty.SetPropertyValueOnTarget(
       targetResource, propertyValue
    );
}

As you can see that is much better, so if you know the classes your DSP exposes do their own fixup remember to take advantage of that.

In the above code I’m also using a couple of extension methods to make this code ‘easier’ (a relative term) to read:

public static void SetPropertyValueOnTarget(
   this PropertyInfo property,
   object target,
   object value)
{
    property.GetSetMethod()
            .Invoke(target,new object[] { value });
}
public static object GetPropertyValueFromTarget(
   this PropertyInfo property,
   object target)
{
    return property.GetGetMethod()
                   .Invoke(target, new object[] { });
}

The next step is implementing both ReallyRemoveReferenceFromCollection(..) and ReallyAddReferenceToCollection(..).

But I’ll leave that to you the astute reader, because it is more about implementing fixup than implementing a DSP at this point.

When you’ve got all those methods implemented you should have a working Strongly Typed Read/Write Data Service with Relationships

So congratulations are in order!!

Next up we’ll look at doing this all over again, this time un-typed.

Along the way you are going to learn some nifty IQueryable tricks :)

Is hiding lower level APIs bad?

Alex D James — Mon, 22 Feb 2010 22:28:00 GMT

There are many good arguments for hiding away the lower layers in your stack – essentially the building blocks upon which your public API is built - however the ‘we don’t have the time and resources to test that layer completely’ argument seems a little flawed to me.

Surely if you are sitting above a layer you have to test it thoroughly anyway?

And if the lower layer is all locked away, private or internal, that is going to be a LOT harder.

Imagine somebody trying to paint the bottom of a can through an open stay tab.

Not exactly the easiest thing in the world. But that is exactly what testing a private lower level API is like.

It would be much easier if you had complete and unfettered access to the inside of the can / lower level.

So if you have to test that surface anyway why not just make it public?

Counter Arguments?

Well, like all programming decisions, it’s not quite that simple. There are many nuances.

What for example if exposing your lower level API…

Just isn’t that useful to your target audience?
Requires too much orchestration and non-obvious call orders?
Exposes you to more legal obligations?
Would require a lot of supporting work like documentation etc?
Might be wrong, and you don’t want to be locked into the current design forever?
Means your software is no longer opinionated?

I’m sure you can think of many similar pivots.

That there are good reasons to keep things internal is beyond debate, what I hope everyone realizes though is that test cost alone probably isn’t reason enough.

Tip 54 – How to improve performance using Statement Expressions

Alex D James — Sat, 13 Feb 2010 17:27:00 GMT

Background:

While writing the update post in my Data Service Provider series I ended up writing this block of reflection code to copy properties values from one object to another:

foreach (var prop in resourceType
         .Properties
         .Where(p => (p.Kind & ResourcePropertyKind.Key)
                         != ResourcePropertyKind.Key))
{
    var clrProp = clrType
           .GetProperties()
           .Single(p => p.Name == prop.Name);
    var defaultPropValue = clrProp
           .GetGetMethod()
           .Invoke(resetTemplate, new object[] { });
    clrProp
      .GetSetMethod()
      .Invoke(resource, new object[] { defaultPropValue });
}

Problem:

This code has at least two major problems.

It is finding properties and methods via reflection in a loop
It is invoking those methods using reflection in a loop.

We could address (1) by storing all the property get / set methods in some cacheable data-structure.

But fixing (2) is a little more tricky, to make this code truly generic you need to use something like lightweight code-gen, or worse.

Solution:

Thankfully .NET 4.0 adds statement expressions. Which means you can create expressions that describe multi-line statements now, and yes those statements can do things like assignments.

Picture Borat saying ‘Nice…’

A quick search of the interweb revealed this excellent post on statement expressions by Bart, and that was enough to get me – and I hope you – to get excited.

Step 1 – Getting familiar with the API

Armed with my new found enthusiasm I decided to dip my toes in the water with something simple, namely trying to convert this into an expression:

Func<int> func = () => {
    int n;
    n=2;
    return n;
};

It would be awesome if you could just do this:

Expression<Func<int>> expr = () => {
    int n;
    n = 2;
    return n;
};

But unfortunately C# doesn’t support this today, instead you have to manually construct the expression, like this:

var n = Expression.Variable(typeof(int));
var expr = Expression.Lambda<Func<int>>
(
    Expression.Block(
        // int n;
        new[] { n },
        // n = 2;
        Expression.Assign(
            n,
            Expression.Constant(2)
        ),
        // return n;
        n
    )
);

Pretty easy huh.

Step 2 – Proving we can assign to a property

Now we’ve got a feel for the API, it’s time to start applying it to our problem.

What we need is a function that will modify an Object (in this case a product) by resetting one or more of its properties, like this:

Action<Product> baseReset = (Product p) => { p.Name = null; };

This code creates an equivalent action using the new expression APIs:

var parameter = Expression.Parameter(typeof(Product));
var resetExpr = Expression.Lambda<Action<Product>>(
        Expression.Block(
            Expression.Assign(
                Expression.Property(parameter,"Name"),
                Expression.Constant(null, typeof(string))
            )
        ),
        parameter
    );
var reset = resetExpr.Compile();
var product = new Product { ID = 1, Name = "Foo" };
reset(product);

And sure enough after reset(product) is called the product Name is null.

Step 3 – Assigning to all non key properties

Now all we need to do is create a function that when given a particular CLR type will create an expression to reset all the non-key properties.

Actually collecting the list of properties and their intended values, isn’t interesting for this discussion, so lets imagine we’ve already got that information in a dictionary, like this:

var properties = new Dictionary<PropertyInfo, object>();
var productProperties = typeof(Product).GetProperties();
var nameProp = productProperties.Single(p => p.Name == "Name");
var costProp = productProperties.Single(p => p.Name == "Cost");
properties.Add(nameProp, null);
properties.Add(costProp, 0.0M);

Given this data-structure our job is to create an expression that has the same affect as this action:

Action<Product> baseReset = (Product p) => {
p.Name = null;
p.Cost = 0.0M;
};

First we need to create all the assignment expressions:

var parameter = Expression.Parameter(typeof(Product));
List<Expression> assignments = new List<Expression>();
foreach (var property in properties.Keys)
{
    assignments.Add(Expression.Assign(
        Expression.Property(parameter, property.Name),
        Expression.Convert(
            Expression.Constant(
                properties[property],
                property.PropertyType
            )
        )
    );
}

Next we feed the assignment expressions into a block inside a lambda, compile the whole thing and test out our nifty new function:

var resetExpr = Expression.Lambda<Action<Product>>(
    Expression.Block(
        assignments.ToArray()
    ),
    parameter
);
var reset = resetExpr.Compile();
var product = new Product { ID = 1, Name = "Foo", Cost = 34.5M };
reset(product);
Debug.Assert(product.Name == null);
Debug.Assert(product.Cost == 0.0M);

As expected this works like a charm.

To solve the problem I had in my Update Post, we’d need a dictionary keyed on type, that we can used to store the reset action for a particular type. Then if a type’s reset action it isn’t found we just create it…

And our performance problem should be a thing of the past :)

Tip 53 – How to debug EF POCO mapping issues

Alex D James — Fri, 12 Feb 2010 00:50:00 GMT

If you are trying to use POCO classes in EF 4.0, it is relatively easy to run into problems with the mapping from your model to your CLR classes.

If you get any nasty problems here, the best way to get to the bottom of things is to try to explicitly load the metadata for your POCO types.

For example if Product is a POCO type and you are having problems getting it working, you can try this code to see what the problem is:

using (MyContext ctx = new MyContext())
{
   ctx.MetadataWorkspace.LoadFromAssembly(
      typeof(Product).Assembly,
      MappingWarning
   );
}

Where MappingWarning could be any action / method that takes a string and returns void, like for example:

public void MappingWarning(string warning)
{
Console.WriteLine(warning);
}

Now when you do this, EF will loop over the types in your assembly trying to see if they map to types in the Conceptual Model, if some reason a CLR type is identified and subsequently disqualified – isn’t a valid match – your action with be called, and in our example the warning will be pumped out to the Console.

Creating a Data Service Provider – Part 7 – Update

Alex D James — Thu, 11 Feb 2010 23:06:00 GMT

In Parts 1 thru 6 we created a custom Read/Only provider over an in memory list of Products.

Now it’s time to add update support.

To do that we need to add an implementation of IDataServiceUpdateProvider.

But first lets talk about the…

Implications of Batching

The IDataServiceUpdateProvider interface is designed to support Batching, allowing customers to update many Resources in one transaction.

Indeed because the interface itself is low level – there is a API call needed to update every individual property – even just updating a single resource can result in a batch of API calls that should happen atomically: if an API call fails we need to rollback / abort whatever happened earlier.

All this means you can expect to see a number of calls to methods like IDataServiceUpdateProvider.SetValue(..), before each SaveChanges() call.

This seems simple enough, but it has profound implications on how you implement the interface.

You can’t just blindly apply the requested change as it is made. You need a way of recording and then committing or aborting all requested changes as a group.

If your data is in a database this typically means recording a series of commands and issuing them in a single transaction. Which something like the Entity Framework does for you automatically.

If however your data is memory, like the sample we’ve been building, you basically have to record intent, and apply that intent only when SaveChanges() is called.

This is why our implementation creates and records Actions whenever one of these low level changes is made…

Making our DSPContext Updatable.

Another prerequisite to implementing IDataServiceUpdateProvider is having a data source that is updatable.

You can do this in a million different ways, but for the purpose of illustration I’ve decided to add some new abstract methods, to our DSPContent class, for Creating Resources, Adding Resources, Deleting Resources and Saving Changes, like this:

public abstract object CreateResource(ResourceType resourceType);

public abstract void AddResource(ResourceType resourceType,
object resource);

public abstract void DeleteResource(object resource);

public abstract void SaveChanges();

Then in our derived ProductsContext we implement these new methods like this:

public override object CreateResource(ResourceType resourceType)
{
    if (resourceType.InstanceType == typeof(Product))
    {
        return new Product();
    }
    throw new NotSupportedException(
       string.Format("{0} not found", resourceType.FullName)
    );
}

public override void AddResource(
   ResourceType resourceType,
   object resource)
{
    if (resourceType.InstanceType == typeof(Product))
    {
        Product p = resource as Product;
        if (p != null){
           Products.Add(p);
           return;
        }
    }
    throw new NotSupportedException("Type not found");
}

public override void DeleteResource(
   object resource)
{
    if (resource.GetType() == typeof(Product))
    {
        Products.Remove(resource as Product);
        return;
    }
    throw new NotSupportedException("Type not found");
}

public override void SaveChanges()
{
    var prodKey = Products.Max(p => p.ProdKey);
    foreach (var prod in Products.Where(p => p.ProdKey == 0))
        prod.ProdKey = ++prodKey;
}

As you can see each of these methods are pretty simple, except for SaveChanges which mimics a database Identity column by assigning every new Product (ProdKey == 0) a Unique sequential ID.

Which means we are finally ready to…

Implement IDataServiceUpdateProvider

Our implementation is called DSPUpdateProvider and looks like this:

public class DSPUpdateProvider<T>: IDataServiceUpdateProvider
where T: DSPContext

As you can see it is a generic class where T is the type of our DataSource.

The IDataServiceUpdateProvider interface doesn’t have any methods to give us the DSPContext - so we’ll need to get it via the constructor somehow – remember we are in charge of constructing the provider through our implementation of IServiceProvider. :)

Metadata would be handy too, so the DSPUpdateProvider constructor takes both metadata and query providers as arguments:

public DSPUpdateProvider(
   IDataServiceMetadataProvider metadata,
   DSPQueryProvider<T> query)
{
    _metadata = metadata;
    _query = query;
    _actions = new List<Action>();
}

The _actions list is there to batch a list of actions to invoke when IDataServiceUpdateProvider.SaveChanges() is called.

As you remember we can’t ‘really’ update our Data Source until Data Services tells us to.

Getting the DataSource

Next we are going to need a way to get to the DataSource i.e. the class that derives from DSPContext. So we need something like this:

public T GetContext()
{
return (_query.CurrentDataSource as T);
}

Which simply gets the CurrentDataSource from the IDataServiceQueryProvider, and casts it to T which, if you remember, is a class derived from DSPContext.

Creating Resources

Next whenever someone does an insert (aka POST) Data Services needs a way of creating the object backing the Resource – CreateResource – our implementation looks like this:

public object CreateResource(
   string containerName,
   string fullTypeName)
{
    ResourceType type = null;
    if (_metadata.TryResolveResourceType(fullTypeName, out type))
    {
        var context = GetContext();
        var resource = context.CreateResource(type);
        _actions.Add(
           () => context.AddResource(type, resource)
        );
        return resource;
    }
    throw new Exception(
        string.Format("Type {0} not found", fullTypeName)
    );
}

Here we try to create an instance of the CLR type used for a particular ResourceType.

Notice that we create a Resource, but we don’t Add it immediately to our Data Source, instead we create and queue an action that will add the resource to the Data Source when invoked.

Getting a Resource

When Data Services tries to Update or Delete a Resource, it first needs to retrieve it (or a representation of it). Which is where IDataServiceUpdateProvider.GetResource(..) comes in.

Data Services passes an IQueryable that when invoked should return exactly one resource.

You might ask why Data Services doesn’t just get the resource itself?

Well the reason is that this method provides and extra layer of indirection which allows you to return something that ‘represents’ the resource, aka a proxy.

You could for example use this ‘proxy’ to record changes and then committing them in batch when SaveChanges() is called, thereby avoid Actions altogether.

But in our implementation we’ve chosen the Action approach, so we can simply return the resource directly.

So we get the resource, check that there is only one matching resource and if the ResourceType is known – which it isn’t for deletes – we also check that the resource has the expected CLR type.

public object GetResource(IQueryable query, string fullTypeName)
{
   var enumerator = query.GetEnumerator();
   if (!enumerator.MoveNext())
      throw new Exception("Resource not found");
   var resource = enumerator.Current;
   if (enumerator.MoveNext())
      throw new Exception("Resource not uniquely identified");

   if (fullTypeName != null)
   {
      ResourceType type = null;
      if (!_metadata.TryResolveResourceType(
         fullTypeName, out type))
         throw new Exception("ResourceType not found");
      if (!type.InstanceType.IsAssignableFrom(resource.GetType()))
         throw new Exception("Unexpected resource type");
   }
   return resource;
}

Closely related is the IDataServiceUpdateProvider.ResolveResource(..) method.

When you implement this method you are passed whatever your returned from GetResource(..) – which remember might have been a proxy – and are expected to return the real resource.

Because we don’t use proxies, our implementation is essentially a no-op:

public object ResolveResource(object resource)
{
return resource;
}

Updating Property Values

Once Data Services has the resource (or proxy for the resource), it will start updating it, by calling IDataServicesUpdateProvider.SetValue(..) to update each property in turn.

By now you know the drill, we are going to record an action to set the property value rather than setting it immediately.

public void SetValue(
   object targetResource,
   string propertyName,
   object propertyValue)
{
    // TODO: add some asserts!!!
    _actions.Add(
       () => ReallySetValue(
          targetResource,
          propertyName,
          propertyValue)
    );
}
public void ReallySetValue(
   object targetResource,
   string propertyName,
   object propertyValue)
{
    targetResource
       .GetType()
       .GetProperties()
       .Single(p => p.Name == propertyName)
       .GetSetMethod()
       .Invoke(targetResource, new[] { propertyValue });
}

As you can see the ReallySetValue (sorry for the unimaginative name) uses reflection to set the property.

You might be tempted to remove this Reflection code to can speed this code up, but before you do all that work remember the real latency is related to OData / Atom serialization and deserialization and network latency, so it might not actually be worth the effort.

Getting Property Values

Occasionally during updates Data Services needs to Get the value of a property, if so it will use the IDataServiceUpdateProvider.GetValue(..) method. Because this GetValue is non-mutating we don’t can just go and get the value directly using a little reflection code:

public object GetValue(object targetResource, string propertyName)
{
   var value = targetResource
      .GetType()
      .GetProperties()
      .Single(p => p.Name == propertyName)
      .GetGetMethod()
      .Invoke(targetResource, new object[] { });
   return value;
}

Deleting Resources

Next we need to handle Deletes. This code is pretty self explanatory, we record an action to delete the resource later.

public void DeleteResource(object targetResource)
{
    _actions.Add(() =>
        GetContext().DeleteResource(targetResource)
    );
}

Resetting Resources

By default there is no need to reset a resource, but a client can request this, for example by using:

SaveChanges(SaveChangesOptions.ReplaceOnUpdate);

If this happens Data Services will ask your IDataServiceUpdateProvider implementation to reset all the non-key properties of the Resource by calling ResetResource(..).

Here's our implementation:

public object ResetResource(object resource)
{
_actions.Add(() => ReallyResetResource(resource));
return resource;
}

Which uses this to do the actual work:

public void ReallyResetResource(object resource)
{
    // Create an new 'blank' instance of the resource
    var clrType = resource.GetType();
    ResourceType resourceType =
       _metadata.Types.Single(t => t.InstanceType == clrType);
    var resetTemplate = GetContext().CreateResource(resourceType);

    // Copy non-key property values from the 'blank' resource
    foreach (var prop in resourceType
             .Properties
             .Where(p => (p.Kind & ResourcePropertyKind.Key)
                         != ResourcePropertyKind.Key))
    {
        // Obviously for perf reasons you could might want to
        // cache the result of these reflection calls.
        var clrProp = clrType
           .GetProperties()
           .Single(p => p.Name == prop.Name);

        var defaultPropValue = clrProp
           .GetGetMethod()
           .Invoke(resetTemplate, new object[] { });

        clrProp
           .GetSetMethod()
           .Invoke(resource, new object[] { defaultPropValue });
    }
}

As you can see the approach used to reset non-key properties, is to create an new ‘blank’ resource, and copy property values across.

Saving Changes

Finally if everything goes according to plan, once all the property updates etc have been queued up you will see a request to IDataServiceUpdateProvider.SaveChanges().

Because of how we’ve implemented things, this one is trivial for us:

public void SaveChanges()
{
    foreach (var a in _actions)
        a();
    GetContext().SaveChanges();
}

We simply invoke all the queued actions, and then ask the DSPContext to Save its changes, which if you remember, simply ensures that each new Product has a new ProdKey.

If anything goes wrong you’ll get a ClearChanges() call, which is even simplier:

public void ClearChanges()
{
_actions.Clear();
}

And now we are done implementing IDataServiceUpdateProvider.

Yippee!

Missing methods

But wait a second … you might have noticed that we haven’t implemented a number of the methods on the interface:

SetReference(..)
AddReferenceToCollection(..)
RemoveReferenceToCollection(..)
SetConcurrencyValues(..)

That’s because we don’t have any Relationships or ETags, so these methods will never be called, meaning we can safely ignore them for now.

Don’t worry though I’ll cover them in a future post.

Hooking it all together

The last step is to modify our IServiceProvider implementation to construct our DSPUpdateProvider and returned it whenever Data Services asked for the IDataServiceUpdateProvider interface.

Summary

The key to implementing IDataServiceUpdateProvider is realizing that you need to be able to delay execution. You can do that using either Proxy objects or Actions, like the above implementation.

Once you know how you are going to handle the atomicity requirements, and you understand what each method is supposed to do, it is actually a pretty mechanical exercise of going through an implementing each method in turn.

Next up we’ll look at adding relationships.

Tip 52 – How to re-use Types with the Data Services client

Alex D James — Wed, 27 Jan 2010 22:52:00 GMT

By default when you add a Data Service Service Reference you get automatic code-generation, which produces a strongly typed DataServiceContext and classes for all your ResourceTypes.

You can have a look at this generated code if you ‘show all files’ in your project:

And then expand your Data Service Reference, it’s dependant Reference.datasvcmap and open up the Reference.cs file:

Step 1 – Turn off Code-Gen

If however you want to re-use some existing classes you’ll have to turn this off Code Gen.

Which is pretty easy – select the Reference.datasvcmap file and go to its Properties, and then clear the Custom Tool, which by default says ‘DataServiceClientGenerator’.

Step 2 – Create a strongly typed DataServiceContext:

When we’ve turned off code-gen we also lost the strongly typed DataServiceContext which just makes programming a little more convenient.

Well it’s pretty simple code that you can easily write yourself like this:

public class SampleServiceCtx: DataServiceContext
{
    public SampleServiceCtx(Uri serviceRoot) : base(serviceRoot) {
        base.ResolveName = ResolveNameFromType;
        base.ResolveType = ResolveTypeFromName;
    }
    protected Type ResolveTypeFromName(string typeName)
    {
        if (typeName.StartsWith("Sample."))
        {
            return this.GetType().Assembly.GetType(
               typeName.Replace("Sample.", "Tip52.")
               false);
        }
        return null;
    }
    protected string ResolveNameFromType(Type clientType)
    {
        if (clientType.Namespace.Equals("Tip52"))
        {
            return "Sample." + clientType.Name;
        }
        return null;
    }
    public DataServiceQuery<Product> Products {
        get {
            return base.CreateQuery<Product>("Products");
        }
    }
}

Notice that the Products property simple returns DataServiceQuery<Product> where Product is the type we are trying to re-use.

The key to making all this work is the code that maps from a Data Service Resource typeName to a client-side Type and visa versa.

This mapping is handled by two Functions, which we tell the DataServiceContext about in our constructor. In this example you can see we are simply going from the ‘Tip52’ namespace on the client to the ‘Sample’ namespace on the server.

Step 3 – Try it out:

Once you’ve set up the resolvers you should be able to re-use your existing types with quite easily:

That’s it. Nifty huh.

Caveats:

This only works if the property names are the same on the client and server, because there is no way to rename properties.

Also because of the way object materialization works in the Astoria client this won't work well if your class has a Reference property with a backing FK property AND the class does automatic fix-up to keep both values consistent.

Tip 51 – How to load EF metadata from arbitrary streams

Alex D James — Wed, 27 Jan 2010 00:18:00 GMT

In Tip 45 I showed you how to build a connection string at runtime, which is pretty nifty.

The problem with that was that it relies on having metadata files (.csdl .ssdl and .msl) on local disk.

But what if they live on a web-server or something and you don’t even have access to the local file system to copy them locally?

Well it turns out you can load the metadata from streams too, and this Tip shows you how.

Step 1: Get XmlTextReaders for the CSDL, MSL and SSDL:

This can be as simple as ‘new XmlTextReader(url)’.

But in this example I’m going to show you how to do it from string variables, which of course you could have got from anywhere:

string csdl = "…";
string ssdl = "…";
string msl = "…";

var csdlReader = new StringReader(csdl);
var ssdlReader = new StringReader(ssdl);
var mslReader = new StringReader(msl);

var csdlXmlReader = new XmlTextReader(csdlReader);
var ssdlXmlReader = new XmlTextReader(ssdlReader);
var mslXmlReader = new XmlTextReader(mslReader);

Step 2: Create metadata ItemCollections:

Next you need an EdmItemCollection for the CSDL, a StoreItemCollection for the SSDL and a StorageMappingItemCollection for the MSL:

The only one that is even remotely interesting is the StorageMappingItemCollection, which along with MSL needs the other ItemCollections to validate the mapping.

Step 3: Create a MetadataWorkspace:

Next you need to group the ItemCollections together in a MetadataWorkspace:

var mdw = new MetadataWorkspace();
mdw.RegisterItemCollection(edmItemCollection);
mdw.RegisterItemCollection(storeItemCollection);
mdw.RegisterItemCollection(storeMappingItemCollection);

Step 4: Create an EntityConnection:

Finally we need an EntityConnection. To create an EntityConnection we need a native database connection - generally this is a SqlConnection, but of course because EF has a provider model it could be something else like Oracle:

var sqlConnectionString = @"Data Source=.\SQLEXPRESS;Ini…";
var sqlConnection = new SqlConnection(sqlConnectionString);
var entityConnection = new EntityConnection(
metadataWorkspace,
sqlConnection
);

Step 5: Create ObjectContext and use as Required

The last step is to use the EntityConnection you just created to construct your ObjectContext and begin using as just like normal:

using (var ctx = new ProductCategoryEntities(entityConnection))
{
foreach (var product in ctx.Products)
Console.WriteLine(product.Name);
}

That’s it… not necessarily simple but easy enough once you know how.

Creating a Data Service Provider – Part 6 – Query Interactions

Alex D James — Mon, 25 Jan 2010 21:39:00 GMT

Whenever I find myself implementing a series of interfaces to plug into a framework or server, I always find myself wondering how the server will call my implementations.

For me this is about forming a mental model to simplify how I think about what I’m doing.

In fact as I’ve been doing this DSP series I’ve come up with a sort of mental model for how Queries are handled in a Data Service.

Here is some PSEUDO CODE that captures my mental model of how these interfaces interact.

Scenario – Basic Query

Imagine a client issues a GET request for /Sample.svc/Products(1)

The first thing that happens (in my model) is that the data service is initialized.

// Locate the IDSP interfaces
var dataservice = …;
IServiceProvider sp = dataservice as IServiceProvider;
if (sp == null)
{
// some other code I’ll go into another day !!!
}

// Get the various DSP interface implementations
IDataServiceMetadataProvider mdp =
   sp.GetService(typeof(IDataServiceMetadataProvider));
IDataServiceQueryProvider qp =
   sp.GetService(typeof(IDataServiceQueryProvider));

// Set the CurrentDataSource (if necessary)
if (qp.CurrentDataSource == null)
   qp.CurrentDataSource = dataservice.CreateDataService();

// Find the Products resourceSet
// (note we actually try for ServiceOperations first
// but lets keep this simple)
var resourceSet = null;
if (!mdp.TryResolveResourceSet(“Products”, out resourceSet))
   throw new Exception(“404”);

// Get the queryable for the Products resourceSet
IQueryable queryRoot = qp.GetQueryRootForResourceSet(resourceSet);

// Compose expressions onto the IQueryable to represent the
// options ($filter/$select etc) specified in the URL
queryRoot = Compose(options, queryRoot);

// Start writing response
WriteStartODataFeed();

// Enumerate results
foreach (object resource in queryRoot)
{
    // Get the ResourceType for resource
    // NOTE: because of inheritance it might be a resourceType
    // derived from resourceSet.ResourceType
    ResourceType type = qp.GetResourceType(type);
    WriteResource(resource,type);
}
WriteEndODataFeed();

That’s it take it or leave it.

Hopefully you found this ‘completely made up’ code useful for forming your own mental model of how your DSP will fit into the Data Services framework.

Next time

There are some things I’ve left out of the above ‘mental model’ for now, like how ServiceOperations and QueryInterceptors complicate things.

We’ll flesh more of those complications out in future posts.

Next time though it’s time to take our Read/Only strongly typed ResourceSet and make it Read/Write.

Creating a Data Service Provider – Part 5 - Query

Alex D James — Tue, 19 Jan 2010 23:49:00 GMT

In Part 4 of our series showing how to implement a Custom Data Service Provider we hooked up an incomplete implementation of IDataServiceQueryProvider, just enough to get the ServiceDocument and $metadata working.

In this part we’ll get query working too.

To do that we need to know where the data is going to come from – the DataSource – and we need to really implement IDataServiceQueryProvider.

Creating a DataSource:

So where is our Product data going to live?

How about a Context, a little like EF or L2S but in memory. First we need a base class.

public abstract class DSPContext
{
public abstract IQueryable GetQueryable(ResourceSet set);
}

Next we need a strongly typed context to expose a Products resourceSet:

public class ProductsContext: DSPContext
{
    private List<Product> _products = new List<Product>();
    public override IQueryable GetQueryable(ResourceSet set)
    {
        if (set.Name == "Products") return Products.AsQueryable();
        throw new NotSupportedException(
            string.Format("{0} not found", set.Name)
        );
    }
    public List<Product> Products
    {
        get {
            return _products;
        }
    }
}

Nothing too tricky here at all. As you can see the ResourceSet is backed by a simple List.

The only thing worth noting is that, because our data is in memory, we use .AsQueryable() to get the LINQ to Objects implementation of IQueryable.

Next we need to change the Sample service to use the ProductsContext as its DataSource:

public class Sample : DSPDataService<ProductsContext>
{

Then we override CreateDataSource in our Sample service, so that we can pre-populate the Products list with some data:

protected override ProductsContext CreateDataSource()
{
    ProductsContext context = new ProductsContext();
    context.Products.Add(
        new Product {
            ProdKey = 1,
            Name = "Bovril",
            Cost = 4.35M,
            Price = 6.49M
        });
    context.Products.Add(
        new Product {
            ProdKey = 2,
            Name = "Marmite",
            Cost = 4.97M,
            Price = 7.21M
        });
    return context;
}

Finally we override the GetQueryProvider method we added last time to look like this:

public override IDataServiceQueryProvider GetQueryProvider(
IDataServiceMetadataProvider metadata)
{
return new DSPQueryProvider<ProductsContext>(metadata);
}

IDataServiceQueryProvider for strongly typed data:

Lets throw away our ‘running service only’ implementation from Part 4.

It didn’t do anything useful anyway.

What we need is a strongly typed in memory Query Provider:

By ‘strongly typed’ I mean for every ResourceType and its Properties there needs to be a matching CLR type with matching properties.

There isn’t that much code, so here it is:

public class DSPQueryProvider<T> : IDataServiceQueryProvider where T : DSPContext
{
T _currentDataSource;
IDataServiceMetadataProvider _metadata;

    public DSPQueryProvider(IDataServiceMetadataProvider metadata)
    {
        _metadata = metadata;
    }
    public object CurrentDataSource
    {
        get { return _currentDataSource;}
        set { _currentDataSource = value as T; }
    }
    public IQueryable GetQueryRootForResourceSet(
       ResourceSet resourceSet)
    {
        return _currentDataSource.GetQueryable(resourceSet);
    }
    public ResourceType GetResourceType(object target)
    {
        Type type = target.GetType();
        return _metadata.Types
           .Single(t => t.InstanceType == type);
    }
    public bool IsNullPropagationRequired
    {
        get {return true; }
    }
    public object GetOpenPropertyValue(
        object target,
        string propertyName)
    {
        throw new NotImplementedException();
    }
    public IEnumerable<KeyValuePair<string, object>> GetOpenPropertyValues(object target)
    {
        throw new NotImplementedException();
    }
    public object GetPropertyValue(
        object target,
        ResourceProperty resourceProperty)
    {
        throw new NotImplementedException();
    }
    public object InvokeServiceOperation(
       ServiceOperation serviceOperation,
       object[] parameters)
    {
        throw new NotImplementedException();
    }
}

What we’ve implemented:

CurrentDataSource holds a reference to the current ProductsContext in _currentDataSource.

Next in GetQueryRootForResourceSet() we delegate to _currentDataSource.GetQueryable() to find the IQueryable for the specified ResourceSet.

We implement GetResourceType(object), by getting the CLR type of the current object and looking in our IDataServiceMetadataProvider.Types to find the the one with a matching InstanceType.

Finally we return true from IsNullPropagationRequired, because we need Data Services to compensate for the lack of NullPropagation in LINQ to Objects.

NullPropagation?

If you take a LINQ query like this:

var results = from x in queryRoot
select x.Customer.Name;

This could easily throw a NullReferenceException, if x.Customer is ever null.

To avoid this Data Services can inject null checks (aka inject NullPropagation logic) into the query. All you need to do is return true from IsNullPropagationRequired.

NOTE: some IQueryables like L2S and L2E send queries to database which seamlessly support null propagation. Hence the option.

What we didn’t implement:

Our Data Service Provider is strongly typed, so Data Services will never call GetPropertyValue, because it knows how to get a property values directly from the resource object.

The model we expose from our Data Service doesn’t have any ServiceOperations or OpenTypes with OpenProperties so we can safely throw NotImplemented Exception from all of those methods.

Conclusion:

Finally we have a complete Read/Only data service up an running

We’ve got a long way to go though. We still need to add support for Update, Relationships, Service Operations, Streaming, Paging and Open Types and I want to show you how to do this all without backing CLR types too.

In Part 6 we’ll talk a little about the interaction between Data Services and your Customer Data Service Provider during a typical GET request.

Tip 50 – How to query a Data Service using JQuery

Alex D James — Mon, 18 Jan 2010 19:23:00 GMT

Recently I’ve been spending some of my time playing with JQuery.

And because Data Services can expose data in JSON format, I thought I’d use JQuery to grab some data from a Data Service.

Turns out it is pretty simple.

This example isn’t going to win any beauty awards, but it will show you the basics and help you get started.

HTML:

First I put together this page:

<html xmlns="http://www.w3.org/1999/xhtml">
   <head>
      <title></title>
   </head>
   <script src="Scripts/jquery-1.3.2.js" type="text/javascript">
   </script>
   <script type="text/javascript">
   // JSON CODE IS GOING TO GO HERE
   </script>
   <body>
      <a href='#' id="aShowProducts">Show Products</a><br />
      <a href='#' id="aShowCategories">Show Categories</a><br />
      <div id="divResults" />
   </body>
</html>

As you can see it is trivial.

JQuery Code

Next comes the JQuery code.

The first step as always with JQuery is to hook up some document ready code:

var divResults;
var aShowProducts;
var aShowCategories;
var ajaxRequest;

$(document).ready(function () {
    divResults = $('#divResults');
    aShowCategories = $('#aShowCategories');
    aShowProducts = $('#aShowProducts');
    aShowProducts.click(function () {
        GetData('Products');
    });
    aShowCategories.click(function () {
        GetData('Categories');
    });
});

When the document is ready first I grab hold of some globals:

divResults (i.e. <div id=”divResults” />): to hold the results of our query.
aShowProducts (i.e. <a href=”#” id=”aShowProducts” …</a>): the ‘Show Products’ link.
aShowCategory: the ‘Show Categories’ hyperlink.
ajaxRequest: is a variable for holding onto the current ‘request’ so we can abort it if necessary.

Then I hook up the GetData method to the click event of the two hyperlinks, passing in the appropriate Data Service resource set name.

Getting Results

The GetData function looks like this:

function GetData(set) {
    var materializer;
    switch (set) {
        case 'Categories':
            materializer = GetRowForCategory;
            break;
        case 'Products':
            materializer = GetRowForProduct;
            break;
        default:
            alert('problems');
    }

if (ajaxRequest!= null)
ajaxRequest.abort();

    ajaxRequest= $.getJSON(
        "ProductsService.svc/" + set,
        function (data) {
            var array = [];
            array.push("<table>");
            $.each(data.d, function (i, item) {
                materializer(array, item)
            });
            array.push("</table>");
            divResults.html(array.join(""));
        }
    );
}

What this does is based on the ‘set’ argument it chooses a function to use as the materializer, i.e. the function that will produce a <tr> for the corresponding resource type.

So for the ‘Categories’ set I use this function:

function GetRowForCategory(array, item)
{
    array.push("<tr><td>");
    array.push(item.Id);
    array.push("</td><td>");
    array.push(item.Name);
    array.push("</td></tr>");
}

And for the ‘Products’ set this:

function GetRowForProduct(array,item)
{
    array.push("<tr><td>");
    array.push(item.Id);
    array.push("</td><td>");
    array.push(item.Name);
    array.push("</td></tr>");
}

Once we’ve chosen the correct materializer next we abort any in progress AJAX requests.

And initiate a new AJAX request using $.getJSON(…) to get JSON from either "ProductsService.svc/Products" or "ProductsService.svc/Categories".

The resulting JSON (i.e. data) is passed to my callback function that builds some html for a table by:

using the array.push(…) approach to build the html string.
and $.each(data.d) to enumerate over the entities and call the specified materializer for each entity to create a new <tr />

Finally I insert the html – array.join("") – into the div.

Trying it out for yourself

You can Save As the complete html here.

All you need to do is drop the html file in the same website as your DataService change the url to point to your DataService and modify the materializer chooser code and the materializers themselves to match your ResourceSets and ResourceTypes.

Not too hard.

The next step would be to experiment with something to build more interesting queries, see Tip 44 for some ideas.

Oh an BTW I’m new to JQuery so if you spot a newbie mistake please let me know.

Creating a Data Service Provider – Part 4 – Minimal Running Service

Alex D James — Fri, 15 Jan 2010 00:01:00 GMT

This is Part 4 of my ongoing series of posts showing how to create an Custom Data Service Provider.

NOTE: I’ve done a little refactoring to the code in Part 3 to simplify the code in this post, so be sure to check that out if you’ve been following along.

In Parts 2 & 3 we hooked up our implementation of IServiceProvider and IDataServiceMetadataProvider.

Exposing IDataServiceQueryProvider

Now we need to implement IDataServiceQueryProvider, and refactor our IServiceProvider implementation to expose it:

public abstract class DSPDataService<T> :
    DataService<T>, IServiceProvider
{
    public IDataServiceMetadataProvider _metadata;
    public IDataServiceQueryProvider _query;

    public DSPDataService()
    {
        _metadata = GetMetadataProvider(typeof(T));
        _query = GetQueryProvider(_metadata);
    }
    public object GetService(Type serviceType)
    {
        if (serviceType == typeof(IDataServiceMetadataProvider))
            return _metadata;
        else if (serviceType == typeof(IDataServiceQueryProvider))
            return _query;
        else
            return null;
    }

    public abstract IDataServiceMetadataProvider
       GetMetadataProvider(Type dataSourceType);

    public abstract IDataServiceQueryProvider
       GetQueryProvider(IDataServiceMetadataProvider metadata);
}

Notice there is a new GetQueryProvider(…) method that your DataService must override.

The IDataServiceMetadataProvider might be needed by the IDataServiceQueryProvider so it is passed as an argument.

public class Sample : DSPDataService<object>
{
    … see part 3 for the rest of code …

    public override IDataServiceQueryProvider
        GetQueryProvider(IDataServiceMetadataProvider metadata)
    {
        return new DSPQueryProvider<object>();
    }
}

Implementing IDataServiceQueryProvider

If you only want to be able to see the ServiceDoc and $metadata you only need to implement the CurrentDataSource property, the rest of the methods are only called when you actually query one of the ResourceSets:

public class DSPQueryProvider<T> : IDataServiceQueryProvider
{
    T_currentDataSource;
    public object CurrentDataSource
    {
        get {
            return _currentDataSource;
        }
        set {
            _currentDataSource= value as T;
        }
    }
    public object GetOpenPropertyValue(
        object target,
        string propertyName)
    {
        throw new NotImplementedException();
    }
    public IEnumerable<KeyValuePair<string, object>>
        GetOpenPropertyValues(object target)
    {
        throw new NotImplementedException();
    }
    public object GetPropertyValue(
        object target,
        ResourceProperty resourceProperty)
    {
        throw new NotImplementedException();
    }
    public IQueryable GetQueryRootForResourceSet(
        ResourceSet resourceSet)
    {
        throw new NotImplementedException();
    }
    public ResourceType GetResourceType(object target)
    {
        throw new NotImplementedException();
    }
    public object InvokeServiceOperation(
         ServiceOperation serviceOperation,
         object[] parameters)
    {
        throw new NotImplementedException();
    }
    public bool IsNullPropagationRequired
    {
        get { throw new NotImplementedException(); }
    }
}

We’ve now done the minimum you possible can to get a custom Data Service Provider based Service actually running!

If you browse to your http://localhost/sample.svc now you will see something like this:

And this http://localhost/sample.svc/$metadata should look like this:

But if you navigate to the Products ResourceSet you will get an error:

That’s because of all those NotImplementedExceptions.

In Part 5 we will flesh out our implementation so query works too...

Creating a Data Service Provider – Part 3 - IDataServiceMetadataProvider

Alex D James — Fri, 08 Jan 2010 01:52:00 GMT

UPDATE: I’ve made a few updates to the code / write-up to reflect refactors I’ve made as I’ve implemented more of the interfaces.

This is Part 3 of my ongoing series on Creating a Data Service Provider, and in this post we’ll look at how to implement IDataServiceMetadataProvider

IDataServiceMetadataProvider is responsible for describing the shape of the Resources and ResourceSets available from your DataSource (see Part 2).

And what is available from your DataSource is really up to you:

It could be completely static. For example if you are exposing data from an off the shelve software package or something where the model is fixed – perhaps Twitter, Stackoverflow or something similar.
It could be strongly typed. For example your implementation might reflect over the Properties of the DataSource itself and create ResourceSets for each property that returns IQueryable<T> and ResourceTypes for all Ts. This is essentially the way the built-in Reflection and Entity Framework providers work.
It could be completely dynamic. For example you could go off to some database somewhere and read a catalog.

Your options are almost endless.

However whichever option you chose the first step is to create a class that implements the interface.

public class DSPMetadata: IDataServiceMetadataProvider
{

The interface has 6 methods and 5 properties, which sounds terribly complicated until you actually try to implement it. I got my first example working in less than half an hour, and I’m not even a developer anymore!

Here is how I’ve hooked it together:

public abstract class DSPDataService<T>: DataService<T>
{
   IDataServiceMetadataProvider _metadata;

   public DSPMetadata()
   {
      _metadata = GetMetadataProvider(typeof(T));
   }

   public object GetService(Type serviceType)
   {
      if (serviceType == typeof(IDataServiceMetadataProvider))
      {
         return _metadata;
      }
      …
   }

   public abstract IDataServiceMetadataProvider GetMetadataProvider(Type dataSourceType);
}

As you can see this code has an abstract method to get the actual metadata, which subclasses must implement.

We are almost ready to implement the interface, but first lets learn about…

Creating Metadata

If you look at the methods on the interface you will see ResourceTypes, ResourceProperties, ServiceOperations, ResourceSets and ResourceAssociationSets.

These are the public classes in the System.Data.Services assembly that you can construct and manipulate yourself to describe your model.

Essentially these methods just walk over this metadata, exposing different information about the model.

Imagine if you have this CLR class:

public class Product
{
   public int ProdKey {get;set;}
   public string Name {get;set;}
   public Decimal Price {get;set;}
   public Decimal Cost {get;set;}
}

And you want to create a ResourceType for Product, this is how you would do it:

var productType = new ResourceType(
    typeof(Product), // CLR type backing this Resource
    ResourceTypeKind.EntityType, // Entity, ComplexType etc
    null, // BaseType
    "Namespace", // Namespace
    "Product", // Name
    false // Abstract?
);
var prodKey = new ResourceProperty(
   "ProdKey",
   ResourcePropertyKind.Key |
   ResourcePropertyKind.Primitive,
   ResourceType.GetPrimitiveResourceType(typeof(int))
);
var prodName = new ResourceProperty(
   "Name",
   ResourcePropertyKind.Primitive,
   ResourceType.GetPrimitiveResourceType(typeof(string))
);
var prodPrice = new ResourceProperty(
   "Price",
   ResourcePropertyKind.Primitive,
   ResourceType.GetPrimitiveResourceType(typeof(Decimal))
);
productType.AddProperty(prodKey);
productType.AddProperty(prodName);
productType.AddProperty(prodPrice);

Notice that we aren’t exposing the Cost property, because that is ‘sensitive’ information.

Next to create a ResourceSet called Products to expose our ResourceType, you simply do this:

var productsSet = new ResourceSet("Products", productType);

As you can see building metadata is not hard.

Freezing your metadata

The final step before you can use the Metadata is to freeze it:

productsSet.SetReadonly();
productType.SetReadonly();

This means it can no longer be modified, at least not for the current Request.

NOTE: Data Services will ask for a IDataServiceMetadataProvider implementation for every request so even though you need to Freeze your metadata per-request, it is completely possible to reconstruct the metadata for each request if necessary.

Exposing Metadata via IDataServiceMetadataProvider

Once you’ve built your metadata you just need to expose it via your implementation.

Here is what your implementation might look like if you didn’t support inheritance, relationships or service operations.

public class DSPMetadataProvider
{
    private Dictionary<string, ResourceType> _resourceTypes
       = new Dictionary<string, ResourceType>();
    private Dictionary<string, ResourceSet> _resourceSets
       = new Dictionary<string, ResourceSet>();

public DSPMetadataProvider(){}

    public void AddResourceType(ResourceType type)
    {
       type.SetReadOnly();
       _resourceTypes.Add(type.FullName, type);
    }

    public void AddResourceSet(ResourceSet set)
    {
       set.SetReadOnly();
       _resourceSets.Add(set.Name, set);
    }

    public string ContainerName
    {
        get { return "Container"; }
    }

    public string ContainerNamespace
    {
        get { return "Namespace"; }
    }

    public IEnumerable<ResourceType> GetDerivedTypes(
        ResourceType resourceType
    )
    {
        // We don't support type inheritance yet
        yield break;
    }

    public ResourceAssociationSet GetResourceAssociationSet(
        ResourceSet resourceSet,
        ResourceType resourceType,
        ResourceProperty resourceProperty)
    {
        throw new NotImplementedException("No relationships.");
    }

    public bool HasDerivedTypes(ResourceType resourceType)
    {
        // We don’t support inheritance yet
        return false;
    }

    public IEnumerable<ResourceSet> ResourceSets
    {
        get { return this.resourceSets.Values; }
    }

    public IEnumerable<ServiceOperation> ServiceOperations
    {
        // No service operations yet
        get { yield break; }
    }

    public bool TryResolveResourceSet(
        string name,
        out ResourceSet resourceSet)
    {
        return resourceSets.TryGetValue(name, out resourceSet);
    }

    public bool TryResolveResourceType(
        string name,
        out ResourceType resourceType)
    {
        return resourceTypes.TryGetValue(name, out resourceType);
    }

    public bool TryResolveServiceOperation(
        string name,
        out ServiceOperation serviceOperation)
    {
        // No service operations are supported yet
        serviceOperation = null;
        return false;
    }

    public IEnumerable<ResourceType> Types
    {
        get { return this.resourceTypes.Values; }
    }
}

The idea here is that your implementation of GetMetadataProvider you do something like this:

public override IDataServiceMetadataProvider GetMetadataProvider(Type dataSourceType)
{
    DSPMetadataProvider metadata = new DSPMetadataProvider();
    var productType = new ResourceType(
        typeof(Product), // CLR type backing this Resource
        ResourceTypeKind.EntityType, // Entity, ComplexType etc
        null, // BaseType
        "Namespace", // Namespace
        "Product", // Name
        false // Abstract?
    );
    var prodKey = new ResourceProperty(
       "ProdKey",
       ResourcePropertyKind.Key |
       ResourcePropertyKind.Primitive,
       ResourceType.GetPrimitiveResourceType(typeof(int))
    );
    var prodName = new ResourceProperty(
       "Name",
       ResourcePropertyKind.Primitive,
       ResourceType.GetPrimitiveResourceType(typeof(string))
    );
    var prodPrice = new ResourceProperty(
       "Price",
       ResourcePropertyKind.Primitive,
       ResourceType.GetPrimitiveResourceType(typeof(Decimal))
    );
    productType.AddProperty(prodKey);
    productType.AddProperty(prodName);
    productType.AddProperty(prodPrice);

    metadata.AddResourceType(productType);
    metadata.AddResourceSet(
       new ResourceSet("Products", productType)
    );
    return metadata;
}

Notice that as you add ResourceTypes and ResourceSets to the metadata they are indexed and frozen (SetReadOnly) for you.

Of course as you add support for inheritance, relationships and service operations, things get a little bit more complicated. But you can handle each of these methods using LINQ to Objects pretty easily.

Performance Considerations

You might have noticed that there are two kinds of functions/properties on this interface, those that return enumerations and those that try to find a specific instance.

This is for performance reasons.

Most requests to the server use the TryResolveXXX instance methods. This means that even if you have a lot of metadata you can often avoid creating most of it.

The other methods are called much more sparingly: for example when someone gets $metadata or the root service document.

Next time

Now we’ve implemented hooked up our DataService to our Custom DSP, and we’ve implemented IDataServiceMetadataProvider, all we need to do to finish our custom read-only DSP is implement IDataServiceQueryProvider.

We’ll start to cover that in Part 4.

Custom Data Service Providers

Alex D James — Thu, 07 Jan 2010 22:33:00 GMT

Introduction

Data Services sits above a Data Service Provider, which is responsible for interacting with the underlying Data Source on behalf of the Data Service.

Data Services ships with some internal providers, and makes it possible for you to create custom providers too.

So the obvious question is...

Do you need a Custom Data Provider?

Creating a Custom Data Service Provider is a fair amount of work, which of course provides big dividends like, allowing people to:

Query and manipulate your data via the Data Services Client in:
- WPF
- WinForms
- SilverLight
- etc
Query you data directly via a browser
Query and manipulate your data via Javascript and associated frameworks like JQuery
Query your data via knowledge worker tools like PowerPivot.
etc.etc.etc.

But before you jump into creating a custom provider, see if you fall into one of the following camps, all of which have a much easier solution:

Entity Framework

If you plan on putting a Data Service over the Entity Framework the answer is no you don't need to write a custom provider.

Data Services has an implementation that talks to the Entity Framework in the box.

You simply point your DataService at your strongly typed ObjectContext, and that’s it:

public class NorthwindDataService:
DataService<NorthwindEntities>

In this example NorthwindEntities is your strongly typed ObjectContext representing the Northwind database.

For a good example of an OData Service built using the Entity Framework check out this post.

LINQ to SQL

If you want to create a Data Service over a LINQ to SQL data source you should take a look at this project on Code Gallery.

It includes sample code you can use to extend your strongly typed DataContext class to implement the IUpdatable interface.

This in conjunction with the Reflection Provider means you can use your LINQ to SQL DataContext just like an Entity Framework ObjectContext:

public class NorthwindDataService:
DataService<NorthwindDataContext>

Reflection Provider

If you have your own class that can represent your Data Source, and it has a number of strongly typed IQueryable properties like this:

public class MyDataSource
{
public IQueryable<Product> Products { get {…} }
public IQueryable<Categories> Categories { get {…} }
}

The built-in reflection provider can turn this into a read-only service and infer ResourceSets, Types and Properties automatically.

You can even make this read-write by extending your class to implement IUpdatable:

public class MyDataSource: IUpdatable

In fact this is exactly how the LINQ to SQL example above works.

For a good example of an OData Service created using the reflection provider check out Tip 56.

So when do you actually need a custom provider?

The reflection provider is very useful for some scenarios.

But is has a number of limitations that might disqualify it for your scenario:

It is static, i.e. the shape of the service can’t evolve over time.
It needs CLR classes for each ResourceType, which you might not have.
It needs to find Id or {Type}Id properties to use as keys.
It blindly exposes all public properties of the CLR classes.
It gives you less control over streaming or paging.
You can’t take advantage of advanced features like Open Types, which allows resources to optionally include more Open Properties.
You have less control, for example you can't log requests as easily or modify metadata or rename properties
etc. etc. etc.

If any of these are important it’s time to create a Custom Data Service Provider…

Creating a Data Service Provider Series

This series of posts will grow to cover all of the major DSP interfaces and show case lots of scenarios

Part 1 – Intro
Part 2 – IServiceProvider and DataSources
Part 3 – IDataServiceMetadataProvider
Part 4 – Minimal Running Service
Part 5 – Query
Part 6 – Query Interactions
Part 7 – Updates
Part 8 - Relationships
Part 9 - Untyped

possible future installments…

Part 10 – ETags
Part 11 – Friendly Feeds
Part 12 – Streaming
Part 13 – Advanced Paging

Creating a Data Service Provider – Part 2 – IServiceProvider & DataSources

Alex D James — Thu, 07 Jan 2010 21:03:00 GMT

To create a read-only Data Service Provider (or DSP) you need to implement two Data Services interfaces: IDataServiceMetadataProvider to provide the metadata and IDataServiceQueryProvider to handle the queries.

But first Data Services needs a way to find your implementations.

IServiceProvider

Data Services locates these interfaces using the IServiceProvider interface.

Probably the best way to hook into this is to create your own class that derives from DataService<T>, and implement the IServiceProvider interface so Astoria can use that to locate your DSP interfaces:

public class DSPDataService<T>: DataService<T>, IServiceProvider
{
   public object GetService(Type serviceType)
   {
      if (serviceType == typeof(IDataServiceMetadataProvider))
      {
         return …
      }
      else if (serviceType == typeof(IDataServiceQueryProvider))
      {
         return …
      }
      else
      {
         return null;
      }
   }
}

Of course how this method actually works depends a lot upon your implementation. Which we will come back to later.

But this is the starting point, and then you, or your customer, can add a Data Service to your web application or WCF app, and simply change it to derive from your class rather than the standard DataService<> like this:

public class DemoDSPDataService : DSPDataService<DSPContext>
{
   // This method is called only once to initialize
   // service-wide policies.
   public static void InitializeService(
       DataServiceConfiguration config
   )
   {
      // TODO: replace this sample configuration with
      // real configuration
      config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
      config.DataServiceBehavior.MaxProtocolVersion =
         DataServiceProtocolVersion.V2;
      config.DataServiceBehavior.AcceptProjectionRequests = true;
   }
}

Here we've hooked the Data Service up to our custom DSP.

Great.

Of course we’ve got some work to do to actually make it work.

But before we go any further lets cover data sources....

Where does the data actually come from?

If you look closely at this code:

public class DemoDSPDataService : DSPDataService<DSPContext>
{

You might wonder what DSPContext is exactly.

Well it is the thing that actually represents your underlying data source, so if could be called anything.

For example if you wrote a custom DSP for the Entity Framework*, DSPContext would be a class derived from System.Data.Objects.ObjectContext.

*NOTE: There is no reason to do this, because Data Services includes an implementation for EF in the box.

If your DSP allows for the underlying data source to be customized in a strongly typed way like EF you should create a generic DSPDataService class as above, and add a generic constraint, limiting T to classes that derive from one of your Data Sources:

public class DSPDataService<T>: DataService<T>, IServiceProvider
where T: MyDataSource

If however the data-source is fixed and you don’t need a strongly typed sub-class, then something like this would be all you need:

public class DSPDataService: DataService<MyDataSource>,
IServiceProvider

As you can see how you write this class depends upon your capabilities.

Next time

In Part 3 we will look at implementing IDataServiceMetadataProvider…

You can find links to all the posts in the series here.

Creating a Data Service Provider – Part 1 - Intro

Alex D James — Mon, 04 Jan 2010 23:57:00 GMT

One of the the coolest things about Data Services is its provider model.

Any data-source can be exposed as an OData Data Service simply by implementing a few interfaces. SharePoint 2010 implements these interfaces to expose list data, and you can too, to expose just about anything: Facebook? Twitter? You name it.

Once you've done that you can query the data using any of the OData clients, including PowerPivot, LINQ over Data Services etc etc.

Now Data Services ships a couple of providers in the box, including one for the Entity Framework and one that works against in memory objects using reflection.

In this series of posts I will walk through creating a Custom Data Service Provider (DSP) slowly layering in more and more capabilities.

DSP Interfaces

There are 5 new interfaces:

IDataServiceMetadataProvider:
Astoria uses this interface to reason about available ResourceTypes, Properties, Keys, NavigationProperties and ResourceSets.
IDataServiceQueryProvider:
Astoria converts all GET requests into calls against this interface.
IDataServiceUpdateProvider:
If your Data Source is Read/Write, you need to implement this interface because Astoria uses it for all PUT, POST and DELETE requests.
IDataServicePagingProvider:
If you want to gain 'fine grained control' over Server Driven Paging you need to implement this interface.
IDataServiceStreamProvider:
If you want to support Media Link Entries you need to need to implement this interface because Astoria uses it to manipulate the underlying stream.

If you simply want a read-only service you only need to implement IDataServiceMetadataProvider and IDataServiceQueryProvider.

Next time

In Part 2 we'll see how Data Services learns about your custom DSP and gets hold of your implementation of these interfaces...

Tip 49 – How to find your Data Service bug

Alex D James — Sat, 12 Dec 2009 01:41:49 GMT

I’ve been playing around with creating a custom Astoria Data Service Provider, aka DSP, for a while now.

So far I’d managed to get my metadata all setup, so browsing to $metadata works great. I’d got a few simple queries work.

Basically everything was going fine.

At least until I tried this URL:

I wasn’t sure what was going on exactly, and I couldn’t get any help from Fiddler, because it was having problems with the built-in VS web server.

And because my DSP was by design loosely typed, if I needed to spot the difference between this (bad):

And this (good):

Can you see the problem?

If you can good on ya, you’ve got sharper eyes than me…

Turns out that you don’t need sharp eyes, you just need to know about HandleException on DataService<> which you can override something like this:

Once I did this it was easy to spot that one of my dictionaries had an entry keyed on ID rather than Id.

After making the fix my query worked just great:

Knowing about HandleException is absolutely vital when developing an DSP, but it is more generally useful too, for things like auditing, error logging, you name it.

Oh and did you notice that my DSP exposes data from a list of dictionaries?

That is a story for another day ;)

Tip 48 – How to host a Data Service in WCF

Alex D James — Fri, 11 Dec 2009 06:17:40 GMT

Every wonder if you can host an Astoria Data Service in WCF?

Well turns out the answer is yes, in fact once you’ve got your references set up etc it is pretty easy.

Step 1 – Setting up your project

Here is what my working project looks like:

In this example I am using VS 2010 beta 2, but this should work with VS 2008 SP1 too.

As you can see I have a Console app with an EF Entity Data Model called ProductsContext and a Data Service called ProductsCatalog.svc that exposes the ProductsContext.

Actually getting my ProductsCatalog.svc into my Console App was a little tricky – If you try Add New Item you won’t find Add Data Service because that dialog is project type sensitive, so it filters out options it considered unlikely in a Console App – I got around this by creating the Data Service in temporary Web Application project and copying it into my Console App project.

Then you need references to the appropriate assemblies System.Data.Entity (EF), System.Data.Services & System.Data.Services.Client (Astoria) and System.ServiceModel & System.ServiceModel.Web (WCF).

Now we have our project all ready to go, the next step is…

Step 2 – Exposing your Data Service

And now, courtesy of Phani, this code is all you need to expose the Data Service:

string uriBaseAddress = "http://localhost:998";
Uri[] uriArray = { new Uri(uriBaseAddress) };
Type serviceType = typeof(ProductsCatalog);

using(WebServiceHost host = new WebServiceHost(serviceType, uriArray))
{
    try
    {
        host.Open();
        Console.ReadKey();
    }
    catch (Exception ex)
    {
        Console.WriteLine("An exception occurred:");
        Console.WriteLine(ex.ToString());
        host.Abort();
    }
    Console.WriteLine("Aborting");
    Console.ReadKey();
}

As you can see we simply create a WebServiceHost for our Data Service class with the URL you want the service to be bound to. Simple.

And now you should be able to do things like this:

All exposed through your own little console app.

Nifty huh?