Entity Framework Design : Providers

Code Only – Further Enhancements

efdesign — Mon, 12 Oct 2009 17:31:00 GMT

We’ve come a long way since the last post on Code-Only. So it’s high time for another update.

We’ve been working really hard on Code-Only revving the design, and spotting missing capabilities and responding to feedback both internal and external etc.

The current plan still holds. Code-Only will not be in .Net 4.0 with the possible exception of the DDL features described in the last post. For that portion, the implementation and requirements are more clear to us, and because making the DDL stuff work requires changes to things already in the .NET framework, and getting provider writers lined up, we are still working hard to get the DDL changes into .NET 4.0. The rest of Code-Only will continue to evolve, and we will ship another preview or two, before getting it rolled into .NET as soon as we can after 4.0 ships.

API Refactoring

As hinted above, we’ve spent a lot of time validating code only against real world scenarios, and thinking about how the customer code hangs together etc. As a results we’ve done some API refactoring.

Mappings are now part of Configurations

In the last version Mappings were derived from Configurations, which had some strange side-effects on the API.

We’ve re-arranged things now so that Configurations hold mappings, which are terminated by assignment to a table.

builder.Entity<Customer>()
   .HasKey(c => c.ID)
   .MapSingleType(c => new {
         cid = c.ID,
         nme = c.Name
      }
   )
   .ToTable(“dbo.custs”);

This maps the Customer entity to the ‘dbo.Custs’ table, the ID property to the cid column, the Name property to the nme column, and registers the ID property as the EntityKey / Primary Key.

As you can see we’ve also added a helper method called Entity<TEntity>() so that you can fluently create configurations and mappings. You might also have noticed that we have removed RegisterKey and replaced it with HasKey which we feel is more inline with our goal of having an intentional API.

TPH mapping syntax no longer violates DRY

The previous syntax for specifying a TPH mapping forced you to repeat mappings for columns that could easily have been inherited.

This was unfortunate because we normally encourage TPH because generally it has the best performance characteristics.

This is where MapHierarchy() and Case() come in.

Imagine if we want to write these classes using TPH:

public class Person{
   public virtual int ID {get;set;}
   public virtual string Firstname {get;set;}
   public virtual string Surname {get;set;}
}
public class Employee: Person
{
   public virtual Employee Manager {get;set;}
   public virtual List<Employee> Reports {get;set;}
}

You would now do it like this:

builder
   .Entity<Person>()
   .HasKey(p => p.ID)
   .MapHierarchy()
   .Case<Person>(
      p => new {
         p.ID,
         p.Firstname,
         p.Surname,
         thisIsADiscriminator = “P”
      }
   )
   .Case<Employee>(
       e => new {
         manager = e.Manager.Id
         thisIsADiscriminator = “E”
       }
   )
   .ToTable(“dbo.People”);

By using Case<> mappings are re-used for derived type, so there is no need to repeat column mappings for ID, Firstname and Surname.

Which means the Employee Case statement is only responsible for mapping properties and references declared by the Employee class, and for specifying a new discriminator value.

The discriminator column and values are there so the Entity Framework can distinguish between rows in the dbo.People table that represent Person objects and those that represent Employee objects.

MapSingleType and MapHierarchy

Because Mappings are now part of the configuration, there is no need to create them independently and assign them.

You now simply MapXXX() methods on the EntityConfiguration.

The available mapping methods are:

MapSingleType(λ)
MapHierarchy(λ)
MapHierarchy().Case<TEntity>(λ).Case<TDerived>(λ).

If you know the Entity Frameworks MSL files these correspond to:

Type
OfType
OfType with an addition requirement for a type discriminator

These mappings can be combined in many interesting ways, but in general this table shows the recommended way to do most common mappings:

Scenario	Code	Notes
No Inheritance	builder.Entity<A>() .MapSingleType(λ) .ToTable(“dbo.A”);	Columns not mapped are not part of the Entity.
TPH	builder.Entity<A>() .MapHierarchy() .Case<A>(λ) .Case<B>(λ) .Case<C>(λ) .ToTable(“dbo.ABC”);	Generally each Case expression only maps properties declared by the current type and the discriminator for that type. But it is possible to override the mapping for inherited properties if required.
TPT	builder.Entity<A>() .MapHierarchy(λ) .ToTable(“dbo.A”); builder.Entity<B>() .MapHierarchy(λ) .ToTable(“dbo.B”); builder.Entity<C>() .MapHierarchy(λ) .ToTable(“dbo.C”);	Each MapHierarchy expression only maps properties declared by the current type, and properties that make up the entity key.
TPC	builder.Entity<A>() .MapSingleType(λ) .ToTable(“dbo.A”); builder.Entity<B>() .MapSingleType(λ) .ToTable(“dbo.B”); builder.Entity<C>() .MapSingleType(λ) .ToTable(“dbo.C”);	Each MapSingleType expression maps all properties, both those declared by the current type and those inherited.

Foreign Keys

In Beta2 of .NET 4.0 we added FK Association Support, so Code-Only needs a way to link a FK property and an Navigation Property together.

Given this class:

public class Product{
   public virtual int ID {get;set;}
   public virtual string Name {get;set;}
   public virtual Category Category {get;set;}
   public virtual int CategoryID {get;set;}
}

You need to be able to tell code-only that Category.ID and CategoryID should have the same value.

You do it like this:

builder.Entity<Product>()
.Relationship(p => p.Category)
.HasConstraint((p,c) => p.CategoryID == c.ID);

This says that c.ID (i.e. p.Category.ID) must equal p.CategoryID, which tells code-only that p.CategoryID is an FK property and p.Category is a navigation property backed by this FK property.

HasConstraint(λ) can also be used in conjunction with Relationship(λ).FromProperty(λ) like this:

builder.Entity<Product>()
   .Relationship(p => p.Category)
   .FromProperty(c => c.Products)
   .HasConstraint((p,c) => p.CategoryID == c.ID);

Which tells code only that Product.Category and Category.Products are inverses and that Product.CategoryID and Product.Category.ID must be the same, which implies Product.CategoryID is an FK property.

Missing Navigation Properties

Sometimes you don’t have a navigation property or FK property in the Entity which would naturally hold the FK in the database. For example imagine this scenario:

public class Product{
public virtual int ID {get;set;}
public virtual string Name {get;set;}
}

public class Category{
   public virtual int ID {get;set;}
   public virtual string Name {get;set;}
   public virtual List<Product> Products {get;set;}
}

Here there is ‘probably’ a one to many relationship between Categories and Products, and that is probably best modeled using an FK in the products table.

If you start mapping the Product entity you need a way to map the FK column, but there is no reference (i.e. Product.Category) and no FK property (i.e. Product.CategoryID) to map.

So we added the ability to create a fake navigation property to help out:

builder.Entity<Product>().MapSingleType(
   p => new {
       pid = p.ID,
       nme = p.Name
       cid = EntityMap.Related<Category>(c => c.Products).ID
   }
).ToTable(“dbo.Products”);

Here the interesting part is EntityMap.Related, it creates a fake navigation property, just so you can use it in the mapping.

The method signature looks like this:

public static TEntity Related<TEntity>(
Expression<Func<TEntity, object>
);

Notice that this function simply returns TEntity, so in the above mapping fragment, we are mapping the ID property of the Category class to the cid column, in the ‘dbo.Products’ table.

Complex Types

Code-Only’s default behavior is to ignore properties that are not recognized as either a primitive type or an EntityType. So to support Complex Types we need a mechanism to register one, like this:

var addressConf = builder.ComplexType<Address>();

This returns a ComplexTypeConfiguration through which you can configure the properties of the ComplexType in the same way you configure the properties of an Entity:

addressConf.Property(a => a.Street).HasMaxLength(100);
addressConf.Property(a => a.Zip).HasMaxLength(10);

Now if an entity is mapped by convention and it has a ComplexType property we will automatically introduce columns for each property of the ComplexType.

If you want to explicitly map the Entity mapping the ComplexType property is pretty simple too:

builder.Entity<Person>().MapSingleType(
   p => new {
      p.ID,
      fn = p.Firstname,
      sn = p.Surname,
      street = p.Address.Street,
      zip = p.Address.Zip
   }
);

Registering Entity Sets

Some of our early CodeOnly adopters wanted to use Code Only without having an specialized ObjectContext, and in theory with is possible:

builder = contextBuilder.Create<ObjectContext>();
builder.Entity<Person>().HasKey(p => p.ID);

But what we found is you have no control over the name of the set generated. In this case we will generate an EntitySet called PersonSet, but what if I the name should be People?

To address this issues we have added this method:

builder.RegisterSet<Person>(“People”);

Not only is this useful for specifying names, it also allows you to intentional specify what EntitySets you need. The alternative would be to encode your intent into calls to Entity<TEntity> and Relationship<TRelated>(..).

Association Mapping

We also need to add support for mapping associations that are not co-located with the Entity. For example take the relationship between Jobs and Candidates. If a candidate can apply for multiple jobs this is a classic example of a many to many relationship.

If you want to take control of how this relationship is mapped you need a way to specify the mapping for the relationship, like this:

builder
.Entity<Job>()
.Relationship(job => job.Applicants)
.FromProperty(candidate => candidate.Applications)
.Map(“dbo.JobApplications”,
           (job,candidate) => new {
               applicantId = candidate.Id,
               jobId = job.Id
            }
       );

This does a number of things, first it indicates that Job.Candidates and Candidate.Applications are opposite ends of the same relationship.

Then the Map() call indicates that the relationship should be stored in the “dbo.JobApplications” join table, with a compound PK made up of the applicantId and jobId columns.

The applicantId holds the candidate.Id, and the jobId holds the job.Id, and in both cases Code Only will emit an FK constraint to ensure referential integrity.

Interesting Column Names

The existing mapping syntax that uses anonymous types for the table and column definitions, limits you to column names that are valid CLR identifiers.

So for example if you need to map to a column with a space in the name you can’t.

To rectify this we added an alternate mapping syntax:

builder.Entity<Product>().MapSingleType(
   p => EntityMap.Row(
      EntityMap.Column(p.ID, “p i d”),
      EntityMap.Column(p.Name),
      EntityMap.Column(p.CategoryId, “c i d”)
   )
).ToTable(“dbo.Products”);

This snippet of code, maps the Product entity to the “dbo.Products” table and the ID property to a ‘p i d’ column, the Name property to a Name column and the CategoryId property to the ‘c i d’ column.

This alternate API has another advantage too: it makes it easier to create mapping expressions programmatically, primarily because you no longer need to create an anonymous type when building the expression. Which is really useful if you want to configure code-only at runtime.

Extracting the EDMX

We’ve also added the ability to get the EDMX that Code Only is producing internally, either via an XmlWriter or as an XDocument:

DbConnection connection = // some code to get a connection.
XDocument document = builder.GetEdmx(connection);

// or using XmlWriter
var swriter = new StringWriter();
var xwriter = new XmlTextWriter(swriter);
builder.WriteEdmx(connection, xwriter);

This is pretty handy if you want to use Code Only to build your model and mappings but then pull them out so you work with the XML and our designer tooling.

Setting the StoreType

When producing the Storage Model Code Only asks the current database provider to return an appropriate store type for the CLR type and facets specified.

But some CLR types and facet combinations can map to multiple possible database types. For example fixed length byte[] can map to both binary and timestamp. So in this situation SqlClient returns binary by default.

But you might need a timestamp:

builder
   .Entity<Order>()
   .Property(o => o.Version)
   .HasStoreType(“timestamp”)
   .IsComputed();

This tells Code Only that the Version property of Order (which is a byte[]) is computed in the database, and is actually a ‘timestamp’ column, which is computed in the database after every insert / update.

DDL Provider Model

In our last post we covered our plans around the DDL provider model, so check that out.

Summary

As you can see Code Only is now looking much more complete.

But it isn’t completely finished yet, we are still working on the rough edges, so as always we are very interested to get your feedback.

In particular are there things you think we should add / rename / simplify?

Alex James
Program Manager, Entity Framework Team, Microsoft.

This post is part of the transparent design exercise in the Entity Framework Team. To understand how it works and how your feedback will be used please look at this post.

Extending the Entity Framework Provider Model to support DDL

efdesign — Tue, 06 Oct 2009 21:24:18 GMT

As part of the first previews of Code-Only we shared some code to create a database:

// Create a context using code-only
using (var mycontext = builder.Create(dbConnection))
{
    // Create the database if it doesn’t already exist
    if (!myContext.DatabaseExists())
        myContext.CreateDatabase();
    // Standard EF code goes here.
}

But in the first preview of Code-Only this code only worked against SQL Server.

The problem is that our public Provider Model (i.e. DbProviderServices) has no Database Definition Language or DDL features.

Provider Model Changes

To support DDL features across different databases, we plan to extend DbProviderServices, with DDL specific services.

These services will be accessed through these public methods:

public string CreateDatabaseScript(
string providerManifestToken,
StoreItemCollection storeItemCollection);

public void CreateDatabase(
   DbConnection connection,
   int? commandTimeout,
   StoreItemCollection storeItemCollection);

public bool DatabaseExists(
   DbConnection connection,
   int? commandTimeout,
   StoreItemCollection storeItemCollection);

public void DeleteDatabase(
   DbConnection connection,
   int? commandTimeout,
   StoreItemCollection storeItemCollection);

Now internally those methods will call through to the following ‘protected virtual’ methods which will do the actual work:

protected virtual string DbCreateDatabaseScript(
string providerManifestToken,
StoreItemCollection storeItemCollection);

protected virtual void DbCreateDatabase(
   DbConnection connection,
   int? commandTimeout,
   StoreItemCollection storeItemCollection);

protected virtual bool DbDatabaseExists(
   DbConnection connection,
   int? commandTimeout,
   StoreItemCollection storeItemCollection);

protected virtual void DbDeleteDatabase(
   DbConnection connection,
   int? commandTimeout,
   StoreItemCollection storeItemCollection);

And the base implementations of these methods in DbProviderServices will simply throw ProviderIncompatibleExceptions.

Which means the provider writers job will be to override these ‘protected virtual’ methods with an implementation that makes sense for their backend database.

The key is to understand that the StoreItemCollection (aka the SSDL or StorageModel part of the EDMX) represents the intended shape of the database.

This means the provider writer will need to iterate over the EntitySets (tables) and the corresponding EntityTypes (table structures) in the StoreItemCollection and create / drop / script the database and tables as required.

Provider writers will be expected to override these functions so that:

DbCreateDatabaseScript: creates a native text command to create the tables and foreign key constraints defined in the StoreItemCollection. I.e. for SqlClient this would be the contents of a .sql DDL file.
DbCreateDatabase: is similar to DbCreateDatabaseScript except it should actually goes ahead and create the database, tables and foreign key constraints.
DbDatabaseExists: checks to see if the database exists. The SqlClient provider will simply check that the database itself exists, but custom provider writers could get more fancy and check to see if every table / foreign key constraint is found too.
DbDeleteDatabase: should go ahead and delete the database, or if the database server has a single database model (like Oracle) the provider writer should delete just the tables defined in the StoreItemCollection.

Simplifying Wrapping Providers

We are also planning something to simplify writing Wrapping Providers. A wrapping provider is just a provider that wraps an existing provider (i.e. SqlClient) and adds additional services (i.e. Auditing, Logging, Caching etc).

Jarek has some some sample wrapping providers if you are interested.

Today writing a wrapping provider is a little tricky, in fact one ‘protected’ method is impossible to wrap without reflection. So to help we plan to add one public wrapper method:

public DbCommandDefinition CreateCommandDefinition(
DbProviderManifest providerManifest,
DbCommandTree commandTree);

One of the reasons we plan on doing this, is we think people might take a ‘basic’ provider that has no DDL support and wrap it to add DDL support.

End User API

Now so far we’ve been looking at the planned extensions to Provider Services, but Provider Services is a very low level API that few developers will ever program against.

Most people will work directly against the ObjectContext, to which we plan to add these methods:

public void CreateDatabase()

public void DeleteDatabase();

public bool DatabaseExists();

public String CreateDatabaseScript();

This little snippet shows how easy it will be to script, create, check and delete a database:

MyContext ctx = new MyContext();
String sql = ctx.CreateDatabaseScript();
ctx.CreateDatabase();
Assert.True(ctx.DatabaseExists());
ctx.DeleteDatabase();
Assert.False(ctx.DatabaseExists());

As you it could hardly be easier to use.

Summary:

While Code-Only provides the catalyst to add DDL support to the Entity Framework’s Provider model, this feature is about more than just Code-Only.

In fact we think this feature will add significantly to the usability of the Entity Framework.

But as always we are keen to hear what you think.

Transparent Caching Support in the Entity Framework

efdesign — Wed, 09 Jul 2008 23:43:00 GMT

The Entity Framework's provider model makes it possible for it to work over different database's.

The idea being that someone, either the database vendor or a third party, can write a provider that allows the Entity Framework to work with that database.

The Entity Framework asks the provider to convert queries and update operations from a canonical, database agnostic form, into native commands. I.e. T-SQL for SQLServer.

This Provider model however has an interesting side-effect. It makes it possible to write wrapping providers, providers that wrap another provider, layering in additional services.

Examples of possible wrapping providers include: logging providers, auditing providers, security providers, optimizing providers and caching providers. The latter is the subject of the following one-pager put together by Jarek

----

Introduction

Business applications use various kinds of Reference Data that does not change at all during the lifetime of an application or changes very infrequently. Examples may include: countries, cities, regions, product categories, etc.

Applications that present data (such as ASP.NET applications) tend to run the same or similar queries for reference data very often resulting in a significant database load. Developers typically use ASP.NET cache and/or custom caching techniques to minimize number of queries but implementing caching manually adds additional complexity and maintenance cost to an existing solution.

Entity Framework can be extended to handle data caching in a transparent way, so that any application using it can take advantage of caching with little or no modification. In Entity Framework V1 it is possible to implement transparent caching using a custom provider as demonstrated in EFCachingProvider sample (TBD). We are considering adding caching as a first-class concept in Entity Framework V2 so that it will no longer be necessary to use a wrapper provider approach.

Requirements

We are designing a query caching layer for Entity Framework that:

· Will be transparent (existing code will automatically take advantage of caching without modification other than defining caching policy).

· Will cache query results, not entity objects so arbitrary ESQL queries can also benefit from it

· Will be optimized for read-only or mostly-read-only data. Caching of frequently changing data will also be supported, but we are not optimizing for that scenario.

· Will handle cache eviction automatically on updates.

· Will be extensible (it should be easy to use with ASP.NET cache or ^3rd party caching solutions including local and distributed caches)

Implementation

To implement query caching we need to be able to intercept query and update execution.

All queries in Entity Framework (regardless of their origin: Entity SQL queries, Object Query<T>, LINQ queries or internal queries generated by object layer) are processed in the Query Pipeline which at some point passes Canonical Query Tree (CQT) to the provider to get the result set of a query. We will cache query results in such a way that when the same query is used over and over again (as determined by the CQT and parameter values), the results will be assembled from the cache instead of a database.

Updates are also centralized in Entity Framework (Update Pipeline) and handled in a similar way. At some point update commands (Update CQTs) are sent to the provider. We can add cache maintenance routines at this point that ensure that proper cache invalidation happens each time an update occurs.

Cache entries and dependencies

Query results stored in the cache will be represented by opaque data structures, which are immutable and serializable, so that they can be easily passed over the wire. An example of such structure may be:

[Serializable]
public class DbQueryResults
{
public List<object[]> Rows = new List<object[]>();
}

When caching query results care must be taken to make sure that returned data is not stale. To be able to detect that, we associate a list of dependencies with each query result. Whenever any of the dependencies change, the query results should be evicted from cache. In the proposed approach, dependencies will be simply store-level entity set names (tables or views) that are used in the query.

For example:

· SELECT c.Id FROM Customers AS c is dependent on “Customers” entity sets

· SELECT c.Id, c.Orders FROM Customers as c is dependent on Customers and Orders entity sets

· 1+2 is not dependent on any entity sets

When adding items to the cache, we will be passing a list of dependent entity sets to the cache provider. After EF makes changes to the database, it will notify the cache about list of entity sets that have changed. All query results relying on any of those entity sets have to be removed from the cache. Dependency names will be represented as strings and collections of dependent entity sets will be IEnumerable<string>.

In the first implementation we will likely use query text or some derivative of it (such as cryptographic hash) as a cache key, but cache implementations should not rely upon cache keys being meaningful.

Interface

To be able to work with EF, cache must implement the following interface:

public interface ICache
{
    bool TryGetEntryByKey(string key, out object value);
    void Add(string key, object value,
             IEnumerable<string> dependentEntitySets);
    void Invalidate(IEnumerable<string> entitySets);
}

As you can see, the values are passed as objects instead of DbQueryResults.

· TryGetEntryByKey(key,out value) tries to get cache entry for a given key. If the entry is found it is returned in queryResults and the function returns true. If the entry is not found, the entry returns false and value of queryResults is not determined.

· Add (key, value, dependentEntitySets) adds the specified query results to the cache with and sets up dependencies on given entity sets.

· Invalidate(sets) – will be invoked after changes have been committed to the database. “sets” is a list of sets that have been modified. The cache should evict ALL queries whose dependencies include ANY of the modified sets.

Cache providers will typically define some specific retention policies, limits and automatic eviction policies (using LRU, LFU or some other criteria). ICache interface does not define that. Based on the user feedback we may want to extend ICache to specify parameters such as retention timeout for each item or add a new interface for configuring cache behavior in an abstract way.

Using the interface defined above, the pseudo-C#-code implementation of query execution operation may look like this:

DbQueryResults GetResultsFromCache(DbCommandDefinition query)
{
    if (CanBeCached(query))
    {
        // calculate cache key
        string cacheKey = GetCacheKey(query);
        DbQueryResults results;
        // try to look up the results in the cache
        if (!Cache.TryGetEntryByKey(cacheKey, out results))
        {
            // results not found in the cache - perform a database query
            results = GetResultsFromDatabase(query);
            // add results to the cache
            Cache.Add(cacheKey, results, GetDependentEntitySets(query));
        }
        return results;
    }
   else
    {
        return GetResultsFromDatabase(query);
    }
}

----

As always we are keen to hear your comments.

Alex James
Program Manager,
Entity Framework Team

This post is part of the transparent design exercise in the Entity Framework Team. To understand how it works and how your feedback will be used please look at this post.