system.data.objects dev guy : ADO.Net

Why use the Entity Framework?

dsimmons@microsoft.com — Sat, 17 May 2008 22:38:00 GMT

There are a number of places where you can read an introduction to the Entity Framework, listen to a podcast about it, or watch a screen cast or video of an interview. Even with these various resources, though, there are so many different data access technologies out there that it's not uncommon for me to get the question: Why should I use the Entity Framework? Or what differentiates it from other options like just using ADO.Net SqlClient and friends, LINQ to SQL or something like nHibernate? I like the second question better, because the truth is that different problems merit different solutions. So here's just a quick take on my perspective about these:

Entity Framework vs. traditional ADO.Net
All of the standard ORM arguments apply here. The highlights are that you can write code against the Entity Framework and the system will automatically produce objects for you as well as track changes on those objects and simplify the process of updating the database. The EF can therefore replace a large chunk of code you would otherwise have to write and maintain yourself. Further, because the mapping between your objects and your database is specified declaratively instead of in code, if you need to change your database schema, you can minimize the impact on the code you have to modify in your applications--so the system provides a level of abstraction which helps isolate the app from the database. Finally, the queries and other operations you write into your code are specified in a syntax that is not specific to any particular database vendor--in ado.net prior to the EF, ado.net provided a common syntax for creating connections, executing queries and processing results, but there was no common language for the queries themselves; ado.net just passed a string from your program down to the provider without manipulating that string at all, and if you wanted to move an app from Oracle to SQL Server, you would have to change a number of the queries. With the EF, the queries are written in LINQ or Entity SQL and then translated at runtime by the providers to the particular back-end query syntax for that database.

Entity Framework vs. LINQ to SQL
The first big difference between the Entity Framework and LINQ to SQL is that the EF has a full provider model which means that as providers come online (and there are several in beta now and many which have committed to release within 3 months of the EF RTM), you will be able to use the EF against not only SQL Server and SQL CE but also Oracle, DB2, Informix, MySQL, Postgres, etc.

Next there is the fact that LINQ to SQL provides very limited mapping capabilities. For the most part L2S classes must be one-to-one with the database (with the exception of one form of inheritance where there is a single table for all of the entity types in a hierarchy and a discriminator column which indicates which type a particular row represents). In the case of the EF, there is a client-side view engine which can transform queries and updates made to the conceptual model into equivalent operations against the database. The mapping system will produce those views for a variety of transformations.

You can apply a variety of inheritance strategies: Assume you have an inheritance model with animal, dog:animal & cat:animal. You can not only do what L2S does and create a single table with all the properties from animal, dog & cat plus a column that indicates if a particular row is just a generic animal or a dog or a cat, but you can also have 3 tables where each table has all of the properties of that particular type (the dog table has not only dog-specific columns but also all the same columns as animal), or 3 tables such that the dog and cat tables have only the key plus those properties specific to their type of animal and retrieving a dog object would involve a join between the animal table and the dog table. And you can further combine these strategies so some parts of a hierarchy might live in one table and some parts in separate tables.

In addition you can do what we call "entity splitting" where a single type has properties which are drawn from two separate tables, and you can model complex types where there is a type which is nested within a larger entity and which doesn't have its own separate identity--it just groups some properties together. The best example of this is something like address where the street, city, state and zip properties go together logically, but they don't have independent identity. The address is only interesting as a set of properties that are part of a customer or whatever. As you have noticed, for v1 you can't create complex types with the designer in the EF--you have to code them by hand in the XML files.

Entity Framework vs. nHibernate
Because nHibernate is a rather full-featured ORM, the distinguishing features between the EF and it are not as large. In fact, it is certainly true that nHibernate is a more mature product and in many ways has more ORM features than the EF. The big difference between the EF and nHibernate is around the Entity Data Model (EDM) and the long-term vision for the data platform we are building around it. The EF was specifically structured to separate the process of mapping queries/shaping results from building objects and tracking changes. This makes it easier to create a conceptual model which is how you want to think about your data and then reuse that conceptual model for a number of other services besides just building objects. Long-term we are working to build EDM awareness into a variety of other Microsoft products so that if you have an Entity Data Model, you should be able to automatically create REST-oriented web services over that model (ADO.Net Data Services aka Astoria), write reports against that model (Reporting Services), synchronize data between a server and an offline client store where the data is moved atomically as entities even if those entities draw from multiple database tables on the server, create workflows from entity-aware building blocks, etc. etc. Not only does this increase the value of the data model by allowing it to be reused for many parts of your overall solution, but it also allows us to invest more heavily in common tools which will streamline the development process, make developer learning apply to more scenarios, etc. So the differentiator is not that the EF supports more flexible mapping than nHibernate or something like that, it's that the EF is not just an ORM--it's the first step in a much larger vision of an entity-aware data platform.

A little more about SP1 Beta and the EF

dsimmons@microsoft.com — Wed, 14 May 2008 08:45:00 GMT

Just a quick note to point out a few interesting things about the updated version of the EF which shipped this week with the beta of VS 2008 / .Net 3.5 SP1:

The initial announcement didn't include links to updated versions of the documentation or our samples, but you can now find the updated docs here (including docs for new things like the entity data source) and at least some of the samples here.
I just made a pass through the EF FAQ updating things for SP1 Beta, etc. There's still a lot of work needed (please help), but I hope you will find this a useful resource.
I highly encourage you to look over the post on the ado.net team blog for more info about what's new. There should also be a post listing breaking changes there very soon.
Finally, let me highlight a couple of the more interesting new features for those of you who have been following my blog and things like my adventures with web services and EntityBag... This release of the EF works in conjunction with updates to WCF made in SP1 to enable out of the box serialization of entity graphs (including cycles). This will make creating web services that return graphs of entities brain-dead simple. It also deals with the known types problem so that inheritance works seemlessly with EF models and WCF services. So a couple of the issues I had to deal with in EntityBag are just solved for you out of the box.

What it doesn't do, though, is handle change tracking on the client and the like. These are still important topics which we'll work on addressing in v2 of the EF, but in the meantime, I'm going to try to put together an updated version of EntityBag which will work with the new EF bits. It's very likely, though, that this won't appear right away. There's just too many things to do not only around finishing up v1 but we're starting to think deeply about v2, etc.

- Danny

Lazy Loading

dsimmons@microsoft.com — Mon, 12 May 2008 21:49:00 GMT

There's a lot going on today! Not only has VS 2008 SP1 Beta been released (with the EF fully integrated into setup and including a number of new features and bug fixes), but also Jarek Kowalski, who is a really smart colleague on the EF team, put up the first blog post in a series he has written which shows some exciting ways that you can layer some code on top of the EF and achieve lazy loading. The goals for his effort are pretty ambitious, and he has achieved them admirably. You might find this code very useful, and it's also a great sign of the flexibility of the EF. I'm so excited to see these kinds of things popping up both from folks inside Microsoft and those on the outside as well. This is how we translate the work of the EF team into real value for folks on many projects.

- Danny

Did you know that connection string builders will parse connection strings not just create them?

dsimmons@microsoft.com — Thu, 10 Jan 2008 02:15:00 GMT

OK. So this one was new on me. Thanks to my colleague down the hall, Alazel Acheson, (being able to say that, by the way, has got to be one of the coolest things about working at Microsoft -- it's great to be surrounded by very smart people) I was able to quickly solve a little problem that has been bugging me lately. It was such an easy solution and a bit unexpected that I thought I'd share it.

Background: The EF uses a connection string for many things including a way to specify the location of your metadata, which store provider you want to use and the connection string for that stored provider. If you really want to talk to the database, you need all those things, but if all you want to do is play with some objects in the ObjectContext/ObjectStateManager, then all you really need is the metadata. Well, actually you also need to know which provider as well for some esoteric reasons we won't discuss right now, but you very specifically do NOT need to know the provider connection string--which is a good thing in some scenarios because provider connection strings can contain sensitive information (like a username and password).

I've got just one of those scenarios that I'm playing around with these days. Part of my solution runs on the middle tier and has a full connection string, but for another part of my solution I want to serialize some data down to a client box which has the entity framework and everything on it but doesn't need or have access to the database. To make my solution work, I'd like to remove the provider connection string from my overall connection string and only serialize that in order to make sure I don't send any sensitive info to the client.

Problem: I really didn't want to write a bunch of code to parse the connection string. Sure we have just that kind of code down in EntityClient, but this solution I'm building lives outside the entity framework, so I needed something publically accessible and easy to use.

Solution: EntityConnectionStringBuilder (and all the other connection string builders) has a constructor overload that takes a connection string. If you use that constructor, then it will parse everything out for you so that you can easily modify things and then reconstitute the connection string. In my case, the code looks something like this:

var csBuilder = new EntityConnectionStringBuilder(context.Connection.ConnectionString);

csBuilder.ProviderConnectionString = null;

this.connectionString = csBuilder.ConnectionString;

Couldn't be easier.

EF Beta 3 is finally available!

dsimmons@microsoft.com — Thu, 06 Dec 2007 19:29:00 GMT

It has been a long time in coming (especially for those of you who have installed VS 2008 RTM and then been frustrated about the inability of beta 2 to work with it), but beta 3 of the Entity Framework and CTP 2 of the EF Designer are finally available!

There are four steps to get everything installed:

0) Uninstall any previous versions of the EF and designer, and install VS 2008 RTM.

1) Install the EF Beta 3 runtime.

2) Install a patch for the XML Editor which is necessary for the Designer to work.

3) Install the EF Tools CTP 2.

In addition to being updated so that they work with the RTM versions of .Net Framework 3.5 and VS 2008, these versions include some great improvements. I'll have to defer to the designer team to give more details about that release, but I wanted to share my quick take on what's new in the EF runtime. I’ll insert my take on things in this color after the new features summary lines I've copied from the download page…

Performance improvements

A HUGE emphasis for us this milestone was on performance. I can’t begin to tell you how much time and effort we spent on these things. The above three represent the biggest, most obvious improvements but effort was made throughout the stack.

Much quicker object query execution

By far and away the most expensive part of executing a query with the EF prior to beta 3 was the time taken to assemble the results into the correctly final format for the conceptual model and then construct object instances from that data. A major effort was made to change the strategy for this process (which we generally refer to as Object Materialization) by compressing a number of layers and doing dynamic code generation of a larger part of the whole process. The result is a noticeable improvement in query speed.

Simpler generated SQL

Another key part of the overall query process is in the SQL generated in the first place. A number of individuals remarked in beta 2 and before about the complexity of the queries generated and sent to the server. This was another major investment area. The result is simpler queries for the database to evaluate (which are also easier for mere mortals like me to understand).

Faster view generation

View generation is a part of the EF process which is necessary for the system to be able to query or update the database. Essentially this is a step which takes the declarative description of the mapping along with the metadata describing the storage model and the conceptual model and transforms them into an internal data structure which is used when the query or update is actually executed. This can happen either at runtime the first time a query or update statement is executed or at compile time. Prior to beta 3, for some complex models the time to generate the views could be quite long (it’s really nasty if you find that booting your app takes 5 minutes with the CPU pegged). During beta 2 I would solve this kind of problem by using compile-time viewgen and checking in the results of the generation so that I only had to pay the price when I actually changed the model, but now it’s enough faster that this generally isn’t necessary.

Easier disconnected operation

Many folks have observed that it can be difficult to use the EF in disconnected scenarios like when building / consuming a webservice or an ASP.Net solution. We spent some time this milestone thinking through the scenarios and adding several key improvements to object services to facilitate these scenarios.

Public, serializable EntityKey property on EntityReference

This is probably the biggest improvement. It accomplishes several things: 1) Directly exposes relationship stubs (ie. Relationship information which the object state manager knows about but isn’t otherwise exposed in the object graph because only one of the two related objects is in memory). 2) Makes it possible to manually set relationship information in the state manager without having both entities in memory. 3) Because the EntityKey is serialized along with the entity by default, this means that for most graphs (all except many-to-many relationships), if you serialize each of the entities in the graph and remote them to a different location and then attach all of them to a state manager at that location, the relationship information will be filled in through the state manager and this will cause the graph to be re-created even though only shallow serialization was used for each separate entity.

ApplyPropertyChanges

Given an entity which is already attached to a state manager and another object which represents a new, updated version of that same entity, you can use this method to compare the two entities and record differences as changes.

Attach on EntityReference

If you have two entity instances which you know are related in the store but the state manager is unaware of that relationship, then you can call Attach on the EntityReference to notify the state manager—just like you can call attach on the context to notify the state manager of an entity that already exists in the database.

Improvements to EntityKey serialization

There was a bug in beta 2 (now fixed) which caused entities with EntityKeys not to serialize the EntityKey along with the rest of the entity data. A variety of additional improvements were made to increase the reliability of EntityKey serialization.

Extensibility and business logic enhancements

Because of some very unpleasant object lessons associated with other components we’ve worked on where too many events were added without thinking them all the way through, we are taking a pretty conservative approach to extensibility in the EF. Events and other extensibility mechanisms have been added as we could convince ourselves that they were both critical and well thought-out.

Partial methods in code generation for property changing and property changed events

Now generated entity classes include declarations of partial methods and calls to those methods before and after setting each property value. The partial method mechanism is great because the compiler completely optimizes away the method calls if no implementation is supplied. The naming convention we use is On<PropertyName>Changing and On<PropertyName>Changed.

Load with MergeOption

This makes it possible to refresh the set of related entities (and the portion of an entity graph which models that—a reference or collection) with well-defined semantics.

AssociationChanged event

This event is exposed on both collections and references—think “collection changed” (fires for adds and deletes) but on both types of relationship objects.

Query improvements

Additional canonical functions for LINQ to Entities

The canonical functions are those which all providers are expected to support.

Apply operator elimination (makes more operations work in SQL Server 2000 and other databases)

The apply operator is a query capability which was added in SQL Server 2005. It’s very effective in some scenarios but unfortunately I’m not aware of any other databases which implement it (as soon as I say that someone will point one out, but in any case most databases do not). Prior to beta 3 this operator was used in a great many scenarios making the provider story for the EF surprisingly complicated. So, we worked to rewrite queries with this operator so that it isn’t required except in a very few scenarios (like when you use apply yourself in an eSQL query).

Compiled LINQ query

This really ought to be listed under “Performance Improvements” as well because it can make a big difference in the performance of LINQ queries. We list it here, though, because it also has an impact on the semantics of the query by making the binding to variables in the surrounding context more explicit. Check this post by Alex James out for more details.

ToTraceString() method on ObjectQuery<T> and EntityCommand to facilitate debugging

Now you can get the translated query text which will execute on the database from an EntityCommand or an ObjectQuery<T> by just calling this method.

Other

Connection management refinements

Prior to beta 3 the object context would open the underlying connection right away when it was constructed in order to gather some key metadata from it. Now that open operation is delayed until it is required. The overall summary of connection management is that by default the context will open the connection before a query or update operation and automatically close it afterwards. If you want to keep a connection open across multiple operations, you can open it explicitly (context.Connection.Open()) in which case you will need to close it yourself, or it will be closed when the context is disposed if the context constructed the connection (rather than it being created by your code and passed into the context’s constructor).

Provider interface allows better reasoning about primitive types

Essentially this is a matter of the EF admitting that primitive types and the details of the semantics around them are determined by the backend database/provider and the EF should avoid reasoning about them as much as possible—when it does need to reason about them it does so using information which the provider supplies.

In addition to the above list there were, of course, a number of bug fixes and other assorted adjustments. See the ADO.Net team blog for the breaking changes doc for details about the ones you are most likely to encounter right away if you have code written against beta 2 that you want to move forward.

Here's hoping that you will enjoy the improvements. As always, I'd love to hear about your experiences with the EF, criticisms, thoughts for how we can improve it, etc.

- Danny

Question: Deep serialization of an object graph--how deep should it go?

dsimmons@microsoft.com — Tue, 20 Nov 2007 10:18:00 GMT

So, I've been thinking lately about serializing/remoting object graphs. The entity framework currently serializes an entire object graph when binary serialization is used but only serializes one entity at a time in XML/DataContract scenarios. I'm working on a sample designed to show how graphs can be serialized, and we're looking into ways to make this even simpler/more automatic. In the process, though, an issue has come up that has me concerned: What if the graph in memory can vary in size? Might you want to serialize only a subset of the graph, and if so how should that subset be specified? Would it always be the same or might you want to specify different subsets for different operations?

To give some context, let's take an example: Assume we have a model with Customers, Orders, OrderLines and Products. Now let's assume that the reason we're serializing things is that we have a web mehtod which returns a customer. With the EF today and a method that just directly returns a customer you would only get the customer, but let's assume for the moment that you could easily indicate that a whole graph should be returned rather than just a single entity. If that were the case, then there are a few possibilities:

1) The entire graph connected to the customer is returned every time. If you are building a stateless webservice, then you would likely construct a new ObjectContext instance each time the method is called, retrieve from the DB just those entities you want to return and then return them. In this scenario, returning the entire graph every time works just fine because the entire graph contains exactly what you want to return.

What if the context is maintained across multiple operations, though? Then everything would still be fine as long as other data retrieved into that context is disjoint from the graph containing the customer you want to return. You could, for instance, retrieve customer1, all of that customer's orders and all of those orders' orderlines as well as customer2 and all of their orders and orderlines and returning customer1 would be unaffected by the fact that customer2 had been loaded into the context. The moment you retrieve into that context a product which has been ordered by both customer1 and customer2, though, those two subgraphs become part of a larger graph, and returning either customer would actually cause the full graph including both customers and all their orders to be returned.

2) One way to address the potential issue with option 1 would be to remove certain navigation properties or annotate some of them to indicate that serialization should stop at that point. So, in the specific example above, the relationship from product back to the orderlines containing that product could be marked so that serializaiton wouldn't travel over it. This would allow a customer graph including products to be returned without that ever leading to the graph for multiple customers being returned all at once.

The problem with this approach, though, is that you might want some web methods to serialize different subgaphs than others. What if you wanted to add to the method that returns a customer a different method which returns a product and all of the orders that contain an instance of that product? In that case, the annotation indicating that products should not serialize the order lines that reference them (necessary to make the customer returning method work correctly) would prevent the products returning method from working as intended.

3) So, another approach altogether would be to have some mechanism to indicate on a method-by-method basis what subgraph to return. Naturally, this kind of mechanism provides the most flexibility, but it's also the most complicated to build and to explain, and it generates other questions like: Is it OK to always serialize all members of a collection as long as that collection is included, or are there scenarios where you would want to perform a filtered serialization where only part of a collection is serialized even though the whole thing is present in memory?

So, what do you think? I could really use the feedback. Binary serialization already uses option #1 above, and part of me thinks that option #1 may well be good enough for almost all scenarios. There's no doubt that it would be a LOT simpler to build and to explain, but if there are important, common scenarios where it isn't good enough, then maybe we need to take on #3.

Thanks,
Danny

An Interesting Relationship Mapping Scenario

dsimmons@microsoft.com — Sun, 18 Nov 2007 08:47:00 GMT

This week on the ADO.Net pre-release forum, someone brought up an interesting relationship mapping scenario in this thread. Essentially the scenario is this: They have a table which represents an entity, call it Foo, and along with the other columns, the Foo table has a column which is an foreign key ID and another column which denotes a type. Foo entities have relationships with three other entities, call them A, B & C, but they can only have one relationship at any one time. The type indicates which relationship is in use, and the ID is then treated as a foreign key to which ever table the type indicates.

On one level, this makes perfect sense even if it's not the same kind of pattern we usually think about, but on another level it's not hard to see why the EF (and other ORMs) do not support this out of the box. The problem is that there's no good way to describe in the conceptual model the fact that an instance of Foo can only have one of the three relationships at any particular time. The object model would actually end up with references to all three types, so it would be all too easy to set more than one of those references to a non-null value and then not be able to save changes because there's only one ID column.

Nevertheless, here's a customer whose database funcitons this way today with hand-written code, and it seems like we ought to be able to do something with the EF. So after some thought, I came up with a few alternatives:

Option #1: Use inheritance and relationships between subtypes.
It's not hard to figure that this solution won't work in many situations, but if it will work for a particular situaiton, then it's the most natural option. Basically the idea is to use the type field to create a TPH inheritance hierarchy of FOO types. You could call them FooA, FooB & FooC, where each of these subtypes has a relationship to one of the other types. The nice thing about it is that the conceptual model now understands that only one relationship can be in place at a time, and the object model matches as well since each type only has one reference to the other type which it can be related to. There are two major problems, though: First off, Foo might already have or be part of another inheritance hierarchy in which case inheritance isn't really an option. Secondly, this mechanism assumes that any one Foo instance has the same relationship throughout its life. It would not be possible to take a FooA and decide later to relate it to B instead, because that would require destroying the instance of FooA and creating a new instance of FooB.

Option #2: Use a view to transform the storage table plus sprocs for updates.
A more creative approach would be to create a view in the database which does not expose the ID & type columns directly the way the underlying tables does but instead exposes 3 different ID columns--call them IdA, IdB, & IdC--each of which has the ID value only if the type indicates a realtionship to the corresponding entity and otherwise is null. Once you do that, Foo can have three null-able relationships where the association sets are mapped only if the value is non-null. That way, only one of the relationships would be active at a time. The Foo object model would have 3 references but two of them would be null at any one time. No inheritance hierarchy would be needed, and instances could change which other entity they are related to just by setting the appropriate reference to a non-null value and the other ones to nulls. The downside is that you have to write the view and, more importantly, that all updates would have to go through stored procs since such a view would not be directly updateable. Essentially this mechanism would fool the entity framework into thinking that the database looks differently than it actually does, and it would depend on the sprocs to enforce validation of the relationships when updating rather than doing the enforcement itself.

Option #3 (in a future EF release): Model the relationships as containment.
One last possibility which I'll mention is that we have talked about adding to the EF in a future release the idea of a containment relationship. The basic idea of this construct is that "child" entities maybe related to one or more "parent" entities but only one parent at a time. This would then allow the conceptual model to more directly represent the idea in the database, and we would then be able to create a declarative mapping for the scenario. I will point out, though, that this definitely won't make v1, and there are still a number of details to work out in the design.

While probably none of these is exactly what the original requester had in mind, I was pleased to discover that the EF is flexible enough to provide some decent alternatives.

- Danny

First draft of an EF FAQ posted

dsimmons@microsoft.com — Thu, 08 Nov 2007 14:02:00 GMT

One of the PMs on the team, Asad Khan, sent me a list of questions and answers which he culled from looking through a whole bunch of forum postings and blogs, and the content seemed like such a useful resource that I've been hacking on it for several hours tonight trying to fill in a few holes, organize a bit more and generally get it in a form suitable for posting... You can find the fruits of that effort at: http://blogs.msdn.com/dsimmons/pages/entity-framework-faq.aspx

It's now 3am and sitting here in my hotel room at DevConnections I just don't have the patience to figure out how to get word to properly format the table of contents I was working on so that you can navigate from a list of topics and quesitons to the answers. Sorry. So I've posted the doc as is without a table of contents and that will have to do for now. Clearly the document as a whole is just a starting point. There's lots of additional data needed, but hopefully some of you will find this useful. If you find something that isn't there which should be, please post a comment or drop me an email so we can revise the doc. Even better, if you've got an answer that you'd like to add, please send that!

- Danny

Two specific ways to take more control of your object model

dsimmons@microsoft.com — Wed, 10 Oct 2007 03:52:00 GMT

There was an interesting discussion on the LINQ Project General forum today (scroll down toward the end where we start talking about what the EF can and cannot do), and there were a couple of key points I wanted to capture since we've had so many discussions about persistence ignorance. Two things which are requested fairly often when folks start to get serious about the object model they expose with their entities are:

Making the default constructor private / forcing consumers to use factory methods.
Creating read-only properties.

Neither of these happens by default with the code generated from the EF designer or from edmgen, but both are possible (even fairly straightforward) to do with the EF, so I want to make sure folks know about this.

Getting rid of the public parameterless constructor
First off, it's important to realize that the EF does require that there be a parameterless constructor for each entity type (as described in this previous post). This constructor is used by the system when materializing the results of queries. This constructor does not, however, have to be public. So if you want other consumers of your object model (such as an app developer--not the EF itself) to always use a constructor which takes a set of required parameters or to use a factory method, then you can just declare a private parameterless constructor in your partial class (or your own class directly if you are using IPOCO rather than the generated code). This constructor can have business logic for initializing your entity if necessary, and it will be called when materializing objects from the database. Consumers of your class will have to call some other public constructor or factory method you supply.

Another request which has been made, by the way, is that the mapping should make it possible to specify a constructor with parameters or a factory method which even the EF will use instead of the parameterless constructor. This is something we will consider for future releases, but it won't be a part of the first release of the EF.

Creating read-only properties
This one is a bit harder because there's no straightforward way to do it with the automatically generated code. If you are writing your own class, though, it's simple. Just make the property which the EF persists private and then create a new public property with only a getter. Unfortunately the EF will not allow you to attribute a field directly--it must be a property, and that property must (in the first version of the EF) have the exact same name as the corresponding property in your conceptual model, but there's nothing in the system which forces you to make that property public.

So, if you want to try this out real quick, you can just take the output of codegen and change the property from public to private and then add a new property to your partial class which is public and has only a getter which access the private property (or the field directly). The EF will use the getter and setter of the property which matches the conceptual model when materializing query results or reading data to write back to the database, but the consumers of your object model will only have access to your public property which can be read-only and/or have various additional business logic in it.

NOTE: I do not normally recommend editting the output of the code generator because if you ever need to regenerate your code these changes will be lost. This is just a quick way to try out this capability of the EF. If you need to make your persisted properties private, then you will need to switch to maintaining your own data classes either by taking a one-time snapshot of the output of codegen and then manually modifying the class from there on or outright writing your own IPOCO classes.

- Danny

Concepts Part I: Getting an entity model up and running

dsimmons@microsoft.com — Sat, 15 Sep 2007 19:18:00 GMT

I've been thinking for a while about a series of blog posts I'd like to write explaining various Entity Framework concepts, particularly those related directly to writing code using the framework--by that I mean that I will concentrate more on using the API to write object-oriented programs which do data access and less on some other kinds of topics like writing and debugging the XML mapping, using the designer, etc. Some of this will be pretty basic, but many of them are the kinds of things that come up fairly often in our own internal discussions as well as on the forums and such. Hopefully this will be valuable, please let me know what you think, if there are topics you would especially like me to cover, or if there's anything I can do to make it more useful.

I'm going to use a common model/database for my samples which I call DPMud. It's something that a few friends and I created and maintain as a fun side-project which is an old-style multi-user text adventure (MUD) built using the entity framework. The architecture is not something that will scale well (every real-time event in the system persists to the database, and users learn about events by querying the database), but it works reasonably well for a few users at a time, it's an environment which is very convenient for us to develop in, and it has turned out to be a decent example for exercising all sorts of entity framework functionality. It even does a reasonable job of stressing the system.

In order to help you get a feel for the model, here's an abbreviated version of the diagram:

Essentially I have a many-to-many self relationship for rooms, but because I need a "payload" on that relationship (some properties on the relationship itself rather than the rooms), I chose to create another entity type called Exit and then have two 1-many relationships between Rooms and Exits--one of those relationships represents all of the exits out of a room, and the other relationship represents all of the entrances to the room. The other parts of the model are actors which are related to rooms, and items which can be related either to a room or an actor. The model doesn't have a good way to enforce that an item must be related either to a room or an actor but not both or neither, but I've added check constraints to the DB which handle that enforcement. In the real game there are also events which relate to each of the other entity types as well as a large inheritance hierarchy of actor types and event types among other things, but those make the model diagram very hard to read, so I've left them out.

This diagram corresponds to my CSDL file which is the XML file describing my conceptual model. If you'd like to look that over, you can find it here. Well, that CSDL file actually has a few things not present in the diagram, but it is simplified considerably from the full model used in our game. For completeness, you can also have a look at the SSDL and the MSL. I know I'm a bit "old school" when it comes to these things since I've been working on the EF since well before we had a designer (funny to say that about a not-yet-released project), but I've authored each of these files by hand rather than using the designer. I'll also point out, in case some of you are unaware, that the designer uses a file called EDMX to store all of the metadata about an EF model--this one file includes the CSDL, MSL and SSDL just in separate sections within it. When it comes to runtime, the system requires the three separate files, and the designer projects automatically create them in your application out directory. So, I will talk about the three files separately and even play some tricks that don't work with the designer today rather than describing the default designer experience.

The one other thing I'll say about DPMud is that it was built from the beginning as rich windows client which talks direclty to the database. So most of our initial apps will work that way. This is nice, because these are some of the easiest apps to write, but as we go through the series we probably will also spend some time exploring other app architectures like web services and web apps.

OK. So enough about my funny little sample. Let's cover some EF concepts, shall we? There are three things I'd like to talk about in this post which are all related to getting any basic app up and running (once you are done creating the metadata for the model, generating the basic entity classes, etc.):

Storing your metadata in resource files. One of the things that folks often find annoying is the fact that EF applications require these three separate metadata files at runtime, and that complicates deployment, etc. Fortunately, it's really quite easy to store these files in the assembly itself as resources and then just tell the EF to find them there. The way to do this is to add the CSDL, MSL and SSDL files to your project and set the "Build Action" property on each one to "Embedded Resource". Then when you specify your conneciton string, rather than explicitly listing the metadata files, you can just add "metadata=res://*/;" which tells the EF to look for appropriate resources in your apps statically linked assemblies any that it finds.
Connection strings. When you use the designer or EdmGen.exe to generate classes from your model, part of what gets generated is a strongly-typed ObjectContext which provides a convenient façade for working with your model. Part of the standard pattern is that the connection string which was originally used to access the database in order to generate the model is embedded in an EntityClient connection string which is placed into the app.config file under the same name as your entity container (specified in the CSDL and used as the name of the strongly-typed ObjectContext type in the generated code). In my sample, this name is "DPMudDB". If the app.config file exists and has a connection string whose name matches the container, then the generated classes have a nice, parameterless constructor which will automatically pick up that connection string and use it.

This mechanism is great for many scenarios--it makes for nice clean code, and it follows a great best-practice of putting the connection string out into the config file where it's easier to swap out in situations like where you might use one connection string for development, another for testing/staging and a third for final deployment. That said, there are cases where you might want to more dynamically generate the connection string--in the case of DPMud, we have multiple database servers for different environments and wanted to provide the user with the opportunity to specify the server as well as credential information at runtime. So, it's important to understand what the parts of the connection string are and how they go together.

If you create a connection string at runtime, then it's still quite straightforward to use. Instead of using the parameterless constructor for the strongly-typed context, you can just use the constructor overload which takes a connection string. This connection string is the exact same connection string that you use either with an ObjectContext (the base type or one of the strongly typed classes that inherit from it) or directly with EntityClient. It follows the same model as all other connection strings which is a set of name value pairs separated by semi-colons, and it has three parts:

A specification for where to find the metadata (csdl, msl & ssdl) which EntityClient will use for mapping between the conceptual model and the database. The keyword for this section is “metadata=”, and there are several different kinds of values you can give it. I mentioned the one I use most often above which is just to specify that the metadata lives in resources within the assembly and that the system should look through all the resources looking for appropriate ones. You can do this with “metadata=res://*/;”, or you can specify a pipe delimited list of directories or files (either absolute or relative to your app directory) which contain the metadata.
The name of the database provider which EntityClient should use to actually talk to the database. For example, “Provider=System.Data.SqlClient;”.
The connection string that you would use with the database provider to access your database. Since this value itself has a series of keywords separated by semi-colons, the string is just enclosed in double quotes. In my sample model it looks like this, “provider connection string="MultipleActiveResultSets=true;database=dpmr;server=.;Integrated Security=true;".

If you are going to build up these parts dynamically, then naturally the recommended way to do so is using a connection string builder, and EntityConnectionBuilder makes this easy. For the provider connection string, you can of course do a similar thing using something like SqlConnectionBuilder. So, the whole thing together would look like this:

SqlConnectionStringBuilder sqlBuilder = new SqlConnectionStringBuilder();

sqlBuilder.MultipleActiveResultSets = true;

sqlBuilder.DataSource = ".";

sqlBuilder.InitialCatalog = "dpmr";

sqlBuilder.IntegratedSecurity = true;

EntityConnectionStringBuilder entityBuilder = new EntityConnectionStringBuilder();

entityBuilder.ProviderConnectionString = sqlBuilder.ToString();

entityBuilder.Metadata = "res://*/";

entityBuilder.Provider = "System.Data.SqlClient";

Context creation and lifetimes. This is a broad topic and obviously the right answer varies a lot depending on the overall architecture of the app you are building—if you are building a web service or an asp.net page then you almost certainly want the context only to stay alive during each request and then be destroyed so that your server can be as stateless as possible, but if you are building a rich client application, then your context will typically stay alive for a much longer period of time—in fact the same context probably stays alive for the life of your app. The reason for this is that the context performs identity resolution and provides the path from entities to the database for when you want to do things like deferred loading. If you destroy the context or detach the entities from it, then deferred loading won’t work.

The other thing you have to remember, though, is that the EF, like most of the rest of the .Net framework, is by and large NOT thread-safe. So if you want to interact with the EF or your entity classes from multiple threads, then you are going to have to do your own locking. One simple model which works for some cases is for each thread to maintain its own context so that no locking is required. This means, of course, that the interactions between the threads tends to be quite limited (you can’t really pass an entity from one thread to another without being very careful), but you can do some pretty useful things even so.

In the case of DPMud, we originally designed the system to have completely separate processes for players and for the system functions (like a process which manages all of the computer controlled AI actors in the system and another process which monitors the system and performs periodic cleanup, etc.). Recently, though, we’ve begun working on some modifications to move the system processes into background threads on the clients. This makes it possible just to run a client and have it find other clients working against the database and automatically elect a process to handle the AI or whatever. (It also means that when it really gets going DPMud can nicely peg both cores on my dual core machine <grin>.)

Happily for us, this migration to a system which uses multiple threads with a separate context on each thread was actually quite easy because some time ago we had adopted a pattern of using a static property as the entry point for accessing the context in various parts of our code. That way we didn’t have to pass the context around everywhere we needed it, and we could also centralize the logic for building the connection string, etc. I want to caution you, though, to be very careful with this kind of pattern. It’s critical that you think about your context lifetime, the lifetime of the objects in it, and the implication of various methods you can call on the context to be sure that this one effectively global context (at least per thread) approach is the right one for you.

One of the biggest potential surprises is around the SaveChanges method which will save ALL changes made to entity objects attached to that context. So, if you have methods which modify your objects but do not immediately save (not all that unusual for many situations where you want to build up a set of changes that are part of a reasonably broad task before saving those changes all at once), and you introduce another method which makes a single change and saves (because its task is more limited and focused), then there is a risk that one of your first set of methods (just modify things but don’t save because it’s part of a bigger unit of work) could end up calling your second type of method and inadvertently saving half of your intended unit of work rather than waiting for the whole thing to complete. In the case of DPMud, we’re building a system which saves to the database as frequently as possible because the database is effectively a real-time communication system, so this wasn’t a problem for us.

In any case, all of that was a long wind-up to sharing some code which follows the same pattern that we use in DPMud not only to consolidate connection string building, but also creating a common, static entry point to the context, and automatically maintaining a separate context for each thread by putting the context into thread-local storage. One other quick point: I put the static properties onto the DPMudDB (strongly typed context) partial class since it’s all about that class so why create another type for that purpose.

Anyway, the code looks like this:

using System;

using System.Data.EntityClient;

using System.Data.SqlClient;

using DPMud;

namespace DPMud

{

public partial class DPMudDB

{

[ThreadStatic]

static DPMudDB _db;

static string _connectionString;

public static string ConnectionString

{

get

{

if (_connectionString == null)

{

// insert code from above which uses connection string builders here //

_connectionString = entityBuilder.ToString();

}

return _connectionString;

}

public static DPMudDB db

{

get

{

if ((_db == null) && (ConnectionString != null))

{

_db = new DPMudDB(ConnectionString);

}

return _db;

}

So what do we get from all this? Well, once we have this foundation laid, writing a program which accesses the database using our strongly typed context becomes pretty darn simple (even if we have multiple threads), and that program can itself just be a single EXE with the entire model and all of its metadata in that one assembly (this is what we do for DPMud—nothing quite like drag-and-drop deployment).

Here’s the code for a simple program which prints out a list of all the rooms and items in the DPMud database but does so from two threads running concurrently. Note: you can set break points in each of the loops and step through the program seeing things interleave nicely, but if you actually run the program there’s a good chance all of the rooms will print together and the items the same because the total time required is relatively small so there may not be that much time-slicing between the threads.

using System;

using System.Threading;

using DPMud;

namespace sample1

{

class Program

{

static void Main(string[] args)

{

Thread thread = new Thread(new ThreadStart(Program.PrintItems));

thread.SetApartmentState(ApartmentState.STA);

thread.Start();

foreach (Room room in DPMudDB.db.Rooms)

{

Console.WriteLine("Room {0}", room.Name);

}

static void PrintItems()

{

foreach (Item item in DPMudDB.db.Items)

{

Console.WriteLine("Item {0}", item.Name);

}

Fun huh? OK. Yes I admit it, I’m a total geek, but I think it’s pretty fun.

Until next time,
Danny

August CTP of the Entity Framework released

dsimmons@microsoft.com — Tue, 28 Aug 2007 11:15:00 GMT

Sorry for the dearth of blog posting folks, but now that the August CTP is out, it seems like I really ought to provide some more info on the features added in this CTP. First, I'll point you to the official announcement which contains links to the download point for the framework, the first CTP of the designer (yeah!), the online docs and samples, etc.

Now here's my short summary of each of the features listed in the announcement:

· Events to customize code generation

Events that fire during code generation whenever types or properties are generated. The event handlers can be used to customize the generated code. Of course there are other mechanisms like partial classes which can be used to add custom logic into the classes or IPOCO which can be used to author the classes altogether by hand rather than using code gen at all, but this mechanism does provide some extensibility points for more advanced code generation scenarios.

· Abstract types in EDM models

As the name implies, this makes it possible to declare types in an EDM model (csdl file) which are explicitly abstract—no entity instances can be created for these types, but types can be derived from them.

· Complex types

“Complex types” is the Entity Framework name for value properties which have more intricate structure than scalars. The canonical example is an Address type which contains several parts (street, city, state, etc.) Complex types are somewhat like entities except that they do not have any identity of their own (they are value types). This means that a complex type instance is always a part of some other enclosing entity—it can’t stand on its own, it doesn’t have relationships, etc. In this release, the mapping scenarios for complex types are significantly limited: inheritance is not supported, complex type properties cannot be null and they can only occur in single instances, not collections.

· <Using> support in metadata files

Models can be built from multiple separate CSDL files—somewhat like a “using” statement in a C# file which brings a namespace into the file or a #include statement in a C/C++ program except with greater validation than you get in the purely textual preprocessing of #include.

· Entity key serialization

In previous CTPs EntityKey objects were not serializable, now they are.

· Increased persistence ignorance in entity data classes

We are continuing down the path toward true POCO and persistence ignorance. To that end we have split the original IPOCO interface (IEntity) into two separate, more specific interfaces (IEntityWithKey and IEntityWithChangeTracker), we made IEntityWithKey optional (not implementing it has performance implications, but the scenario is supported), and many of the ObjectServices signatures have been modified to take instances of Object rather than IEntity which sets the stage for further progress here in future releases.

· Improved connection management in ObjectContext

In previous CTPs the underlying provider connection was opened when the ObjectContext was constructed and held open for the life of the context—this would create issues, for instance, with databases where licensing was based on the number of concurrently open connections since the connection might be held open for an extended period of time—even when the connection is not being used. The new model keeps the connection closed as much as possible while still appropriately dealing with transactions and avoiding promotion of those transactions from local to DTC where possible.

· Improved DataBinding usability

Now ObjectQuery<T> and EntityCollection<T> implement IListSource which makes databinding more automatic rather than requiring explicit calls to the ToBindingList() method. In fact, since the automatic mechanism is so much better, ToBindingList was removed.

· Metadata annotations

CSDL files can now include custom annotations which the framework ignores but can be useful for tools or other systems built around the EDM.

· Better support for span over LINQ to Entities queries

The span feature makes it possible to pre-fetch related entities. In previous CTPs span was specified through a property on ObjectQuery<T>, but in this CTP span has been changed to work as a builder method, Include, which makes it simpler to specify span rules in LINQ to Entities queries. Include can be called on the ObjectQuery used in the from clause of the LINQ query.

· Improvements to LINQ queries: additional canonical functions and automatic mapping from CLR functions to server functions

This means that more CLR functions are recognized and mapped to store functions in the generated SQL queries.

· A new event for extensibility of SaveChanges

The OnSavingChanges event fires at the beginning of the SaveChanges method making it possible to perform validation on entities which will be saved, etc.

· Polymorphic results from stored procedures

Stored procedures can that retrieve entity instances can now return polymorphic results rather than being restricted only to results of a single type.

The other major feature, of course, is that it works with Orcas Beta 2!

If you have any questions or feedback, please don't hesitate to send it along.

Thanks,

Danny

"June" CTP of the Entity Framework is finally out

dsimmons@microsoft.com — Tue, 03 Jul 2007 05:13:00 GMT

Hey folks,

It has been longer in arriving than I would have liked, but the June CTP of the Entity Framework is finally out. This includes a HUGE set of changes. I can't tell you how many times lately I've responded to a question on the forum or something like that with "that's fixed in the next CTP." Finally, it's here. Here's a partial list of the features (largely plagarized from the ADO.Net team blog post on the topic)

IPOCO -- there's a lot more coming here to bring us closer to true POCO, but this is the first step in that direction and makes the key change to allow the creation of Entity classes that do not inherit from our base class.
ObjectContext.Detach() -- will detach an object from the ObjectStateManager associated with an object context so that it can be attached to some other context or reclaimed by the garbage collector
Multiple entity sets per type -- this is relatively esoteric, but it does exactly what its name says. You can define a model which has more than one entity set with the same base type. This is important because it enables models which work more naturally with various database models. If you have two tables with the same schema (say "accounts" and "expired_accounts" or something), then you can define entitysets for each of them which use the same entity type rather than needing to create a different entity type which just happens to have all the same properties. This resulted in a change to a number of the object context APIs -- strongly typed contexts, for instance, now have AddToEntitySetName methods (where EntitySetName is the actual name of each entity set).
Support for referential integrity constraints -- This allows you to define a key to one entity type which is made up, in part, of properties which are also in the key of another entity type. The modeling scenario in the database is where a column is both a foreign key and part of a primary key.
Span -- this makes it possible to specify that a query not only return an entity but return some set of related entities in a single round trip and automatically hook-up the graph. So, for instance, you could say that your customer query should also return the orders for each customer in a single round trip. We also have an automatic query re-write feature for the object layer that enables a feature we call Relationship Span where queries that retrieve entities will under-the-covers also return relationship information if the relationship is an EntityReference (rather than a collection). This is usually a quick operation since typically this relationship information is co-located in the relational database as a foreign key property, and it has the very useful property that for many cases entity graphs will automatically be hooked up as long as the relevant entities on either side of the relationship have been loaded. So, for example, if I query for a set of customers and then I separately do a query for a bunch of orders--any of those orders that are related to those customers will automatically bring along the relationship info and connect the orders up to the customers they go with.
Transactions -- specifically integration with System.Transactions.
Serialization -- that is, entities are automatically generated with attributes that enable binary serialization. Unfortunately EntityKeys do not yet serialize but that will be coming in a future CTP.
No more default constructors in code-generated classes -- I've already written about this in an earlier blog post.
Improvements to stored procedure support -- honestly I can't remember the details of this one just now, and since I'm sitting in the Denver airport where I have limited battery life on my laptop I'll delay looking it up for you. If anyone is wondering, drop me a note and I'll get back to you.
Access to the underlying store connection -- You can now easily access the store connection from an EntityConnection which makes it much easier to go around the covers if you need to. (But I must say I don't have to do that nearly so often now as I used to -- this framework is actually starting to work pretty well for writing apps, IMHO <grin>.)
Directory macros -- Makes it easier to specify where the metadata lives in hosted scenarios
Native SQL read-only views -- You've got to see one of the demos Tim Mallalieu runs where he shows the entity framework accessing sharepoint or some other schema that otherwise really doesn't fit the model.
UNICODE support in Entity SQL
Query plan caching and I'm pretty sure we do some metadata caching across app domains as well, but that may not be fully online yet in this CTP release.
Canonical functions in Entity SQL
Associations between sub-types -- This one is a biggy (as if the others aren't). Before this change, the entity types that are on the ends of relationships had to be the base type of an entityset, but after this change you can define relationships between any types in the type hierarchy even if they aren't the base types. So, for instance, if I have a hierarchy that has Customer and BigAccountCustomer, then I could create an entity type DiscountPolicy and a relationship that only relates BigAccountCustomers to DiscountPolicies not just regular customers.

I'm sure there's more in there, but that should give you a taste of things. That's all the good news. The bad news is that all these changes brought along with them a number of API and metadata schema changes, so if you have apps you've been working on with beta 1, a number of changes will be needed to get your apps working again with this CTP. If your experience is anything like mine, though, the real major changes in your app will come from taking advantage of all the new features to completely remodel things with associations between subtypes and RI constraints, to use span, etc.

As always, comments and feedback are more than welcome. We're really looking forward to hearing what you think.

- Danny

P.S. One other small bit of bad news: Complex Types didn't make it into this CTP. So if you are holding out for them, unfortunately you'll have to wait for the next round.

Non-scalar Value Objects aka "Complex Types"

dsimmons@microsoft.com — Sun, 17 Jun 2007 01:06:00 GMT

Yesterday there was a post on the ADO.Net Orcas forum asking if it is possible to map two entities to a single row in a single table using the entity framework. The motivation/scenario behind this request was an entity which had an address as part of its properties plus the fact that there were other entity types which have very similar properties. The default mapping generated by the wizard just adds these as properties of the entity, but the preferred model would be that the entity has a property which is another class that groups the address properties together. That class would then be shared and used to model the address properties on other entity types as well.

The initial expectation of the person asking the question was that the address should be modeled as an entity type of its own, but they ran into two problems:

The address type doesn't have any appropriate property (or properties) to be its key.
Even if you did have appropriate properties for the key, the system doesn't let you map two entities to the same row in the same table.

The moment I saw this post, I was reminded of a discussion I had with Atul Adya early on in the entity framework project when we were trying to decide whether or not to support a modelling concept we call "complex types". Initially we had hoped not to introduce another modelling concept--aren't scalars, entities and relationships enough? Atul's viewpoint on the matter was: "If we don't support complex types, then someday, someone is going to ask us to map two entities to the same row in the database, and we won't ever be able to support that." Well, here we are. At the moment the EF CTPs do not support complex types, and as a result someone has asked for the other feature Atul was referring to (two entities mapped to the same row) which we just can't support.

Why can't we support two entities mapped to the same row? Well, first off the lack of a property or properties that can be the key is fundamental. Secondly, if you think about it, what would insert and delete operations that affect one but not both of the entities in a row really mean? Things get crazy really quick.

So the answer is to make sure the EDM supports the idea of complex types which Eric Evans calls value objects in his Domain Driven Design book. They have no identity on their own--their identity is slaved to the entity of which they are a part, but they do get their own class where you can add methods, validation, etc. and share it among all instances of that value type regardless of which entity they are a part of.

Fortunately, we did decide to implement complex types, and they will appear in an upcoming CTP.

- Danny

Persistence Ignorance: OK, I think I get it now.

dsimmons@microsoft.com — Sat, 02 Jun 2007 07:54:00 GMT

Let me apologize up front for how long it has taken me to get back to this topic. What I can say, though, is that in the intervening interval I have devoted real time to studying agile design principles and domain driven design.

Initially I thought that some of the requests for *complete* persistence ignorance were the products just of dogma rather than fully informed and reasoned arguments, but when so many people give such passionate feedback it was clear that I needed to investigate more before I could claim to have any sort of an informed opinion. The product of my research is that I am now truly convinced of the importance of complete persistence ignorance for some scenarios. As a result, the team is now working on a series of product changes aimed at ensuring that a later move to complete persistence ignorance will not be a fundamental breaking change.

There's no question, though, that complete persistence ignorance comes at a price--both in the performance of applications built with "pure POCO", persistence ignorant domain models and as a result in the complexity of the entity framework which enables them. The additional complexity of the framework is not just about enabling such applications to function but a product of the fact that the performance degradation which results means that the framework must support a spectrum of flexible options between complete persistence ignorance on one hand and prescriptive architectures which build some persistence information into the domain model to improve performance on the other hand.

To make this more concrete, we have identified 4 elements of the existing entity framework design which block persistence ignorance. That is, 4 ways in which entities are currently required to have knowledge of the framework:

They must store a copy of an EntityKey.
They must provide change tracking notifications through a prescriptive interface.
They must be decorated with a series of attributes which provide additional metadata.
If they participate in relationships, the requirements are even more stringent as indicated by the requirement to implement the interface IEntityWithRelationships.

The first two requirements are enforced with the IEntity interface and must be addressed if we are to enable persistence ignorance at any point in the future. In fact a series of key object services APIs currently . At the same time each of these has significant performance implications. Choosing not to store EntityKeys on the entities, for instance, means that navigating from an entity to the ObjectStateEntry which matches it either requires a brute-force search of the ObjectStateManager or for the ObjectStateManager to maintain a dictionary mapping from CLR reference to ObjectStateEntry which is a significnat expense. Not supporting the change tracking mechanism means that the ObjectStateManager must cache a copy of the original values for each entity (all original values and they must be cache whether or not the entity is modified).

While I do believe that there are circumstances in which the loss of performance is a reasonable tradeoff for persistence ignorance, I also believe that for a great many applications, even where persistence ignorance is highly valued, there will come a time in the performance tuning process when one or another of these two compromises must be made. As a result, we are working on replacing IEntity with two separate interfaces and making both of them optional--which means the object services API signatures will change from taking parameters of type IEntity to type "object" and the framework must support multiple mechanisms for each of these.

The 3rd and 4th issues are, on the one hand, harder to correct, but on the other hand things which can be corrected later. At some point we will add an alternative mechanism for specifying metadata about the entity objects (thus freeing us from the hard requirement for attributes). Similarly, we can envision ways to enable POCO collection and reference mechanisms for relationships which can be added later as a complement to the existing prescriptive EntityCollection<T>, EntityReference<T> and RelationshipManager classes.

So, to sum it all up, message received: persistance ignorance is needed, and we're now working to deliver it. The first release of the entity framework likely will not have full support for it (there's just not enough time), but we will work to get into the first release the key changes which could not be made later so that the stage is set for the next release.

- Danny

Where’s the model? Is it the code or something else?

dsimmons@microsoft.com — Fri, 04 May 2007 09:24:00 GMT

A few weeks ago I had a very interesting email exchange with Jeffrey Palermo in which he raised some important questions about the Entity Framework. Here’s an excerpt from his original query (used by permission):

I've been thinking a lot about the model of the problem domain for a piece of software and the question of "what is the model". In domain driven design, the domain model in code is the canonical representation of the thoughts and concepts of the business domain (the model). Even so, the code is a very good _representation_ or _translation_ of the model as discussed by business analysts (it encorporates the concepts (entities) and behavior and rules of the business domain). The database schema is another translation of this model, but only geared toward storage of the concepts or entities. The behavior is absent here, just storage (for the most part).

This new EDM concept your team is introducing seems to be aimed at being the new canonical representation of the model. Here, there is an Xml file which describes the model or _represents_ the model, but is not the model itself. I remember when UML was really big, and folks were calling that the model. It could be described so well that it was called the model and all else was secondary. In the end, the UML wasn't executable, wasn't verifyable and wasn't testable, so it lost mindshare because when it came to working software that handled the business domain correctly, it still didn't deliver. There were tools that sought to generate software from the UML, but again, it didn't work.

I'm having a problem with your EDM xml file because I respect history. The Xml file that represents the model: how do we test it? How do I _know_ that it's correct? that is contains the right concepts (entities)? that every bit of it's behavior is correct? That every one of its rules is right?

I respect the coded domain model so much because it can't lie to me. I run my unit tests, and they pass or fail. I can demonstrate it. I can objectively prove its behavior and rules. The best representation of the model has to work 100%.

Maybe you've thought about this and have an answer. I've heard said that this new Xml artifact will be called "the model". Is that really the intended noun?

My response was that I hear and share concerns about the parallels to UML and the failed efforts of the past to make UML the center of the universe. (I personally think codegen from UML is so evil that I have taken to calling UML "the enemy" not because it's original purpose and ideas are bad—it can be a great tool for doing some sketches to communicate some ideas—but because it has been perverted to evil ends.)

We do talk about the EDM for an application as "the model", but that's just a verbal short-hand for "data model", which I believe is a critical distinction. We are explicitly NOT trying to define behavior in the EDM. The code is the right way to define that. We are, however, trying to define enough about the structure and constraints on the data that we can provide a rich set of services in a general fashion that still honor those constraints. If you think about it, this is very much like what the relational model does today vs. just doing your own file format. The relational model allows you to describe some things about the structure of your data so that the database and other parts of the system can provide services which help you manage that data. The idea of the EDM is to provide a higher-level of abstraction that gives more information about the structure of the data so that we can provide more services. This doesn't replace the object model or reduce its critical role; it just augments it with new services.

Once we start generating code from the EDM, things definitely become blurred a bit, but the explicit goal of the generated code is not to replace your object model—it's to save you some typing of repetitive things when you are building that object model. Further, the intent is that we would not require this generated code—you can always build your model completely on your own and just align the EDM with the shape of that model so that you can take advantage of the EDM-enabled services we provide (right now that means persistence/query, but eventually that means other things like reporting, replication, remoting, etc.).