Entity Framework Design : N-Tier

Foreign Keys in the Entity Framework

efdesign — Mon, 16 Mar 2009 23:47:00 GMT

Background

A number of months ago we asked whether Foreign Keys (FKs) in Conceptual and Object models were important.

The feedback we got indicated that there are a real mix of opinions on the topic. Some people are all for them while others think that FKs pollute the conceptual model.

The fact that our customers were so divided meant we thought it was important to provide options here. Those who want FKs in their Entities should be able to do so, so they can gain all the benefits that having FKs in your Entities undoubtedly provide. On the other hand customers who are concerned that have FKs in their Entities in someway pollutes their model can continue to use the type of associations we had in .NET 3.5 SP1.

In the past we've called the .NET 3.5 SP1 style Associations "first class" associations. Which while accurate somehow implies that associations based on FKs in Entities are "second class". In fact when we talked about this new work recently with our MVPs one of the first questions was "are these second class?", which they most definitely are not.

The fact is these new associations are different but first class nevertheless.

So what's been added?

In .NET 4.0 we will add support for a new type of association called "FK Associations".

The way this works is we will allow you to include your FK columns in your entities as "FK properties", and once you have FK Properties in your entity you can create an "FK Association" that is dependent upon those properties.

Okay so we will have "FK Associations", what are we going to call the older style associations? Obviously saying something like ".NET 3.5 SP1 style Associations" every time isn't ideal. So we are going to call them "Independent Associations".

The term "Independent Association" resonates for us because they are independently mapped, whereas FK Associations need no mapping, simply mapping the Entity(Set) is sufficient.

How do I use an "FK Association"?

The real reason so many customers and partners are asking for "FK Associations" is that they significantly simplify some key coding patterns. So much so that we are convinced that for most people "FK Associations" will be an automatic choice going forward.

Some of the things that are hard with "Independent Associations" are trivially easy using FK Associations. Scenarios as diverse as DataBinding, Dynamic Data, Concurrency Control, ASP.NET MVC Binders, N-Tier etc are all positively impacted.

Let's have a look at some code snippets that will work with "FK Associations":

1) Create a new Product and FK Association to an existing Category by setting the FK Property directly:

using (var context = new Context())
{
    //Create a product and a relationship to a known category by ID
    Product p = new Product
    {
        ID = 1,
        Name = "Bovril",
        CategoryID = 13
    };
    //Add the product (and create the relationship by FK value)
    context.Products.AddObject(p);
    context.SaveChanges();
}

This sort of approach works both for insert and update, and is particular useful for things like databinding, where you often have the new Value of the FK in a grid or something but you don't have the corresponding object, and you don't want to wear the cost of a query to pull back the principal object (i.e. the Category).

2) Create a new Product and a new FK Association to an existing Category by setting the reference instead:

public void Create_new_Product_in_existing_Category_by_reference()
{
    using (var context = new Context())
    {
        //Create a new product and relate to an existing category
        Product p = new Product
        {
            ID = 1,
            Name = "Bovril",
            Category = context.Categories
                        .Single(c => c.Name == "Food")
        };
        // Note: no need to add the product, because relating
        // to an existing category does that automatically.
        // Also notice the use of the Single() query operator
        // this is new to EF in .NET 4.0 too.
        context.SaveChanges();
    }
}

This programming pattern is not new to the Entity Framework, you could do this in .NET 3.5 SP1. I called it out because it shows you can still write the sort of code you write with Independent Associations even when using FK Associations.

3) Update an existing Product without informing the Entity Framework about the original value of the CategoryID (not supported with Independent Associations):

public void Edit(Product editedProduct)
{
    using (var context = new Context())
    {
        // Create a stand-in for the original entity
        // by just using the ID. Of the editedProduct
        // Note: you don't have to provide an existing Category or CategoryID
        context.Products.Attach(
            new Product { ID = editedProduct.ID });

        // Now update with new values including CategoryID
        context.Products.ApplyCurrentValues(editedProduct);
        context.SaveChanges();
    }
}

In this example "editedProduct" is a product which has been edited somewhere, this is exactly the sort of the code you might write in the Edit method of a ASP.NET MVC controller, and is a great improvement over the code you have to write using Independent Associations.

There is much more you can do with FK Associations, but these 3 samples give you a flavor of the sort of coding patterns FK Associations allow.

Keeping FKs and References in Sync

One thing that hasn't been mentioned so far is that the Entity Framework tries to keep related References and FKs in sync as much as possible.

When this synchronization occurs depends upon when the Entity Framework is notified of changes, which depends upon the type of Entity Classes involved: be they POCO with Proxies, POCO without Proxies, IPOCO or EntityObjects etc.

Another post will cover this in more detail.

How do I create an "FK Association"?

There are a number of ways.

One mainline scenario is when using the tools to infer a model from a database. We have added an option to choose whether "FK Associations" or "Independent Associations" are generated by default. The same is true if you use EdmGen.exe (i.e. our command line tool).

The second mainline scenario for creating FK Associations is creating a model from scratch, aka Model First which is new to .NET 4.0.

Here are the steps involved:

Create two Entities (say Product and Category) that look something like this:
Add a property that will be the FK Property: i.e. CategoryID
Create an association between the two EntityTypes with the correct end multiplicities and NavigationProperty names:
Double Click the line between the two Entities, that represents the Association, and add a Referential Integrity Constraint:

This is the step that tells the Entity Framework that the CategoryID is an "FK Property" and that the "CategoryProduct" association is an "FK Association".
You end up with something that looks like this:

And you are done you've set up an FK Association. Notice that this association doesn't need to be mapped you simply need to make sure you map all the properties of both Entities.

Very easy.

Summary

In .NET 4.0 we will add support for FK Properties and FK Associations to the Entity Framework. The existing style Independent Associations will still be possible, but we expect that FK Associations will become most peoples automatic choice because they simplify so many common Entity Framework coding tasks.

As always we'd love to hear your thoughts on this work.

Alex James,
Program Manager, Entity Framework Team, Microsoft.

This post is part of the transparent design exercise in the Entity Framework Team. To understand how it works and how your feedback will be used please look at this post.

N-Tier Improvements for Entity Framework

efdesign — Thu, 20 Nov 2008 08:33:00 GMT

The first version of Entity Framework provides convenient ways to load, manipulate and persist objects and relationships. As with many other O/RMs, Entity Framework has a state manager that tracks every change made. Existing objects are typically loaded first from the database, later modified, and finally the changes are saved back to the store.

Another feature, full graph serialization, makes it very easy for developers to ship around object graphs representing snapshots of the current state of the world, across execution boundaries.

The next version of Entity Framework will also support Persistence Ignorance. Therefore object graphs can now be made of POCO instances, which WCF now also supports.

Nonetheless, there is a task that belongs in any N-Tier application that still requires a great amount of work for developers using EF: decoding messages that represent state changes performed by the client, meant to be processed or persisted by the service.

The major pain point in the first version is the number of intricate steps a program needs to perform in order to setup the state manager in the appropriate state according to the changes encoded in the message. For instance, for any but the simplest scenarios, in order to use the basic graph manipulations APIs in ObjectContext, like AddObject and AttachTo, it is necessary to “shred” the object graphs contained in the message, which destroys important information about relationships between object instances that therefore has to be kept somewhere else (as an illustration of this approach, see Danny Simmons’s EntityBag in project Perseus).

In addition, given that the message typically contains all the necessary information and that concurrency is usually a concern, the program shouldn’t have to re-query the database. Instead, it should be easier to “tell” the state manager about the original state (that came from the database previously) and about what changes were made.

There are a few approaches to N-Tier that are very popular in the industry. Customers using Entity Framework for the first time often carry the expectation that it will support a similar experience:

DataSet

The original ADO.NET data access technology includes the DataSet, a versatile data cache that can be serialized as XML and be used to transport collections of rows and relationships and, also diffgrams representing changes.

DataSets make many N-Tier scenarios extremely easy to implement, and over time, have proven helpful for many customers. In spite of this, they are not appropriate for some scenarios and architectural patterns:

The lack of a widely accepted standard serialization format has hindered the development of implementations that could produce or consume datasets in other platforms.
Diffgrams represent sets of CREATE, UPDATE and DELETE operations of arbitrary depth, which is not adequate for architectural patterns that conceive operations with more constrained semantics (i.e. PlaceOrder, CreateCustomer).
No simple ways to control the application of batch CUD operations contained in diffgrams makes them unsuitable any time there is a trust boundary between the client and the server.
Also, the data-centered perspective of DataSets leads to an anti-pattern known as the “anemic domain model”, which is characterized by a separation between the data aspect and the behavioral aspect of domain objects.

Data Transfer Objects

The approach typically used in the DDD and SOA communities to address multi-tier scenarios is to hand-build a service with well defined operation semantics together with the a conceptual message format, usually composed of data transfer objects (DTOs). Implementing DTOs often also involves writing the logic that translates DTOs back and forth to persistent objects.

Here is a simplified set of rules that characterizes the use of DTOs:

Actual entities are not serialized, just DTOs, which are value-only objects (without behavior) are exposed in the boundaries.
The client does not depend on the types defined on the server. The shapes of the DTOs are the sole contract, and the types used in the client are for the client only.
The context or container is not sent together with the message.
The service decides what to do with the contents of the message. It is not the message who decides (as an extreme counter example, imagine a message that tells the service to delete all the contents of the database).

Those rules are especially important in SOA scenarios, in which the client and the server can be separated by a trust boundary, such as the one that exists between two different companies.

There are other permutations of layered architectures, though, in which the level of decoupling that DTOs provide may not justify the increased complexity, for example:

A multi-tier application in which the client is fully trusted.
A rich internet application in which an infrastructure service is purely used for data persistence (is by design a CRUD-only service) and therefore types exposed do not contain behavior anyway.
Any architecture in which the persistence framework is actually used to provide and manage DTOs and not the actual entities.

REST

ADO.NET Data Services provides a rich framework that developers can easily use to expose any Entity Data Model as a REST-style resource collection. Simple HTTP verbs like GET, PUT, POST and DELETE are used in combination with URIs to represent CRUD operations against the persistence service.

This pattern is suitable for many applications and can easily be combined with service operations with richer semantics in the same application.

ADO.NET Data Services, however, may not be compelling for those customers who want to define their services as operations with richer semantics entirely.

Goals

ADO.NET Data Services goes a long way addressing the needs of customers doing RESTful services, and so there is no need for us to improve Entity Framework experience on that space.

With the N-Tier improvements for Entity Framework, we want to address some of the same problem space as DataSet, but we want to avoid the primary issues with it.

Ideally, we would like to provide building blocks that are appealing for developers building solutions on a wide range of architectures. For instance, we would like to provide fine enough control for DTO proponents, but at the same time reduce the level of pain that those trying to address simpler scenarios experience today.

Design

If we just wanted to provide a DataSet like experience on top of Entity Framework, the most straightforward (although very expensive to test) way would probably be based on full serialization of the state manager. Basically, the internal representation of ObjectStateManager includes collections of objects to be deleted, inserted, and modified, and also relationships to be added and deleted, as well as objects and relationships that should remain unchanged.

Such a model would lead to very fine grain control of the state manager, but you would lose some of the benefits of operating on the domain objects and the higher level APIs. For instance, it would make it extremely easy for the program to set the state manager in a completely invalid state.

An alternative representation for such an experience would be to ship a “change log” containing the history of all operations performed on the client. This representation has the disadvantage of being more verbose than the previous one, but it would make “point in time” rollbacks pretty easy to implement.

Besides these two, there are a few more interesting generic representations for changes in a graph, but in general, they all suffer from the same disadvantage: providing a solution for them does not give the user the level of control that the most complicated scenarios and sophisticated patterns require.

Based on this fact, we decided to adopt the following goal for the design:

Entity Framework won’t define its own unique representation for the set of changes represented in an N-Tier application. Instead, it will provide basic building block APIs that will facilitate the use of a wide range of representations.

This has the desirable consequence that Entity Framework won’t impose a pattern for N-Tier. Both DTO-style and DataSet-like experiences can be built on top of a minimal set of building blocks APIs. It is up to the developer to select the pattern that better suits the application.

Here is a first cut of the new methods that we are proposing to add to the ObjectContext. The new APIs will work with EntityObjects, IPOCO and POCO entities:

public partial class ObjectContext 
{ 
    /// <summary> 
    /// Apply property values to original state of the entity 
    /// </summary> 
    /// <param name="entitySetName">name of EntitySet of root</param> 
    /// <param name="original">entity with original values</param> 
    public void ApplyOriginalValues( 
        string entitySetName, 
        object original) { } 
    
    /// <summary> 
    /// Changes the state of the entity and incident relationships 
    /// </summary> 
    /// <param name="entity">entity to change state of</param> 
    /// <param name="state">new state</param> 
    public void ChangeObjectState( 
        object entity, 
        EntityState state) { } 

    /// <summary> 
    /// Changes state of relationship 
    /// </summary> 
    /// <param name="source">source object of relationship</param> 
    /// <param name="target">target object of relationship</param> 
    /// <param name="relationshipName">name of relationship</param> 
    /// <param name="sourceRole">role of source object</param> 
    /// <param name="targetRole">role of target object </param> 
    /// <param name="state">new state</param> 
    public void ChangeRelationshipState( 
        object source, 
        object target, 
        string relationshipName, 
        string sourceRole, 
        string targetRole, 
        EntityState state) { } 

    /// <summary> 
    /// Changes state of relationship represented by navigation 
    /// property 
    /// </summary> 
    /// <param name="source">source object of relationship</param> 
    /// <param name="target">target object of relationship</param> 
    /// <param name="navigationProperty">navigation property</param> 
    /// <param name="state">new state</param> 
    public void ChangeRelationshipState( 
        object source, 
        object target, 
        string navigationProperty, 
        EntityState state) { } 
    
    /// <summary> 
    /// Changes state of relationships represented by lambda 
    /// expression 
    /// </summary> 
    /// <typeparam name="TSource">type of source entity</typeparam> 
    /// <param name="source">source entity of relationship</param> 
    /// <param name="target">target entity of relationship</param> 
    /// <param name="selector">property selector expression</param> 
    /// <param name="state">new state</param> 
    public void ChangeRelationshipState<TSource>( 
        TSource source, 
        object target, 
        Expression<Func<TSource, object>> selector, 
        EntityState state) { } 
  
} 

ApplyOriginalValues

Similar to the existing ApplyPropertyChanges, but this API keeps the tracked entity intact, only affecting the corresponding original values. This operation only works for entities in Modified, Unchanged or Deleted states. Added entities by definition don’t have original state.

ChangeObjectState

This method will transition the state of a tracked entity passed as argument, but will also have side-effects on incident relationships. In case the new state is modified, it will also mark all properties as modified, regardless of the original and current values.

ChangeRelationshipState

Similar to ChangeObjectState, but sets the relationship between two tracked entities to the provided state. Relationships cannot be in the modified state, but this method can be used to report which relationship should be added, deleted, or unchanged.

The relationship doesn’t need to exist in the context, but the entities do. If the relationship doesn’t exist, it can be created in the new state. For instance, when this code is invoked, a new relationship can be created between customer1 and order1 in the added state:

context.ChangeRelationship( 
    customer1, 
    order1, 
    c=>c.Orders, 
    EntityState.Added); 

The relationship is typically represented by the navigation property but the metadata names of the relationship and roles can also be used in a different overload.

Design notes

All parameters named “entity”, “source” and “target” accept either entities or EntityKeys.
We could choose to deprecate the ApplyPropertyChanges API and replace it with ApplyCurrentValues, which is more consistent name-wise with the new ApplyOriginalValues method.
For convenience, we should add similar APIs in other types like ObjectStateEntry, RelatedEnd and others as appropriate. For instance, the ObjectStateEntry will also have a ChangeState API that will only take the target state parameter.
We could choose to put ChangeObjectState() and the various ChangeRelationshipState() in ObjectStateManager rather than in ObjectContext, since this is a lower level API than everything else currently in the ObjectContext.

Scenario tests

To get a sense of how the new API can be used, we have tried several different approaches and representations. We also have plans to release sample code that will show in more detail different ways to use the new API. In the meanwhile, here are a few simple cases.

General purpose Attach with state resolution delegate

In this scenario between a client and mid-tier, the intention is to allow a client to query for a Customer entity with the Customer’s Order entities. The client then makes a variety of changes to the Customer entity graph including modifying the Customer, and adding or removing Orders. The client would like to send the updated Customer entity graph to the mid-tier so that it can be processed and persisted.

This system will use a simple mechanism to help with change tracking on DTO entities. Each DTO will implement a small interface to help determine the state of the entity using three properties: IsNew, IsModified, and IsDeleted. In this example, original values are not tracked by the client.

public interface IEntityWithChanges 
{ 
    bool IsNew { get; } 
    bool IsModified { get; } 
    bool IsDeleted { get; } 
} 

When the client makes changes to the Customer entity graph, the only requirement in this scenario is that when an entity changes, the appropriate property will return true to help determine the state of the entity.

The mid-tier will expose a service method that implements the logic to update the Customer entity graph. Rather than traverse the Customer, her Orders and OrderLines looking for changes, the service method will use a convention based mechanism that knows how to deal with the IEntityWithChanges interfaces. In this scenario, this mechanism takes the form of an extension method to the ObjectContext that will attach an entity graph to the context, and use a delegate to map each entity that implements IEntityWithChanges to an EntityState so that it can be used with the new ChangeObjectState APIs.

First we can define a method to map from an IEntityWithChanges to an EntityState:

/// <summary> 
/// A mapping from an IEntityWithChanges to an EntityState 
/// </summary> 
public EntityState GetEntityState(object entity) 
{ 
    IEntityWithChanges entityWithChanges = 
       entity as IEntityWithChanges; 
    if (entityWithChanges != null) 
    { 
        if (entityWithChanges.IsNew) 
        { 
            return EntityState.Added; 
        } 
        else if (entityWithChanges.IsModified) 
        { 
            return EntityState.Modified; 
        } 
        else if (entityWithChanges.IsDeleted) 
        { 
            return EntityState.Deleted; 
        } 
    } 
    return EntityState.Unchanged; 
} 

Next we will define the extension method on an ObjectContext that can attach an entity graph and use a mapping delegate to determine the state of each entity in the graph.

/// <summary> 
/// Attach a graph of entities using a supplied mapping function to 
/// determine the entity's state. 
/// In this example, the state of relationships between entities is 
/// determined by the rules of ChangeObjectState: 
/// - if one entity in the relationship is Added, the relationship is 
///   Added 
/// - if an entity is Deleted or Detached, all incident relationships 
///   are Deleted or Detached 
/// </summary> 
/// <param name="entitySetName">name of EntitySet of root</param> 
/// <param name="entityGraph">graph of entities to attach</param> 
/// <param name="entityStateMap">EntityState resolver</param> 
public static void Attach( 
    this ObjectContext context, 
    string entitySetName, 
    object entityGraph, 
    Func<object, EntityState> entityStateMap) 
{ 
    context.AttachTo(entitySetName, entityGraph); 

    // Create a list of unique entities in the graph 
    var allEntities = 
        context 
        .ObjectStateManager 
        .GetObjectStateEntries(EntityState.Unchanged) 
        .Where(entry => !entry.IsRelationship) 
        .Select(entry => entry.Entity); 
        
    // Apply the entityStateMap to each entity 
    foreach (object entity in allEntities) 
    { 
        context.ChangeObjectState(entity, entityStateMap(entity)); 
    } 
} 

Finally we can define our service method that can update a Customer entity graph using the mapping delegate and the Attach extension method.

/// <summary> 
/// A service method to update a Customer's entity graph which 
/// can include modifications to the Customer entity, 
/// new Orders, and removal of Orders 
/// </summary> 
/// <param name="c"></param> 
public void UpdateCustomerOrder(Customer customer) 
{ 
    using (NorthwindEntities context = new NorthwindEntities()) 
    { 
        // Call the Attach extenion method with the mapping delegate 
        // that goes from IEntityWithChanges to EntityState 
        context.Attach("Customers", customer, GetEntityState); 
        context.SaveChanges(); 
    } 
} 

ChangeOrder Service Operation

In this scenario, the mid-tier exposes a method to perform a very specific operation: changing the Customer that owns an Order. The client can set the Order’s Customer to a new Customer or to one that already exists in the store. In either case, the service method uses the old Customer so that it can remove the relationship between the Order and that Customer.

/// <summary> 
/// Updates an Order by changing the relationship between the Order 
/// and a Customer in this case 
/// </summary> 
/// <param name="updatedOrder">order to update</param> 
/// <param name="oldCustomer">previous customer</param> 
public static void ChangeOrder( 
    Order updatedOrder, 
    Customer oldCustomer) 
{ 
    using (NorthwindEntities context = new NorthwindEntities()) 
    { 
        // Tell the context about the entities to persist 
        context.AttachTo("Orders", updatedOrder); 
   
        // Change the state of the new customer 
        // In this example, this is done by the convention: 
        // - Customer.CustomerID is null --> customer is 'Added' 
        // - Customer.CustomerID is not null --> it already exists 
        if(updatedOrder.Customer.CustomerID == "") 
        { 
            // When state of customer changes to 'Added', state 
            // of the relationship between updatedOrder and 
            // pdatedOrder.Customer is also changed to 'Added' 
            context.ChangeObjectState( 
                updatedOrder.Customer, 
                EntityState.Added); 
        } 
        else 
        { 
            if (updatedOrder.Customer.CustomerID != 
                oldCustomer.CustomerID) 
            { 
                // only relationship needs to be marked as 'Added' 
                context.ChangeRelationshipState( 
                    updatedOrder, 
                    updatedOrder.Customer, 
                    o => o.Customer, 
                    EntityState.Added); 

                // Report removal of relationship between 
                // pdatedOrder and oldCustomer 
                context.ChangeRelationshipState( 
                    updatedOrder, 
                    oldCustomer, 
                    "Customer", 
                    EntityState.Deleted); 
             } 
        } 
        
        // Persist the changes to the store 
        context.SaveChanges(); 
    } 
} 

As is always the case, your feedback on this topic is welcome!

Jeff Derstadt
Jaroslaw Kowalski
Diego Vega
and The Object Services Team

This post is part of the transparent design exercise in the Entity Framework Team. To understand how it works and how your feedback will be used please look at this post.