system.data.objects dev guy : Entity Framework

D3: Rides Again – Now on VS2010 Beta 2

dsimmons@microsoft.com — Sun, 06 Dec 2009 01:19:18 GMT

And you thought I might never post on D3 again. I know. I know. Jedi knights don’t wait 6 months for a follow-up blog post.¹Well, the wait is finally over. I’ve put a new release of D3 up on code gallery. In this release the code has been updated to work with VS2010/.Net 4 beta 2. The release also includes a reorganization of project names and some new functionality, but I’ll talk about those in a future post or two. For this post, let’s look at the three broad steps involved in upgrading the project since they illuminate EF features and changes you might need to make to move some other project from beta 1 to beta 2.

Step 1 : Basic Compilation

When I loaded the solution in beta 2 and tried to recompile, the first problems I encountered were issues with stylecop—there were a number of errors referring to a file called ".NETFramework,Version=v4.0.AssemblyAttributes." I found a solution in this thread.

The next change needed was to react to the EF’s breaking change from beta 1 to beta 2 where the ContextOptions property DeferredLoadingEnabled was renamed to LazyLoadingEnabled. The semantics of this property didn’t change—just it’s name, so this was a simple fix.

Once compilation was passing, the next step was to start leveraging new EF features in beta 2.

Step 2 : Adding FKs to the Model

Beta 2 not only added runtime support for FKs in the model, but the designer is now smarter about them, and they are turned on by default. So I had a few options:

I could do everything by hand:

Manually add the FK properties.
Go to each association and add the referential integrity constraint between the FK property on the dependent entity and the primary key on the principal entity.
Remove the association set mappings for the associations since the mapping will now be handled by mapping the FK property on the entity just like any other entity.
Finally, either manually add the mapping for that property or regenerate the database from the model.

Since the model is not yet all that big, I could recreate it from scratch, requesting FK properties with each association.
Given that I already had a database with foreign key columns (just not exposed on the entities), I could reverse engineer the model from the database with the option turned on to generate FK properties on the entities.

I choose option #3 for no reason except that it was interesting to come full circle from a model-first database to a DB first model and then back to model first when further changes are needed. :-) In the process I encountered one may difficulty which was that reverse engineering from the database uses a simplistic algorithm for naming things that works reasonably well when you only have one relationship between any two entities, but when you have more than one relationship between the same pair of entities things get a bit confusing. In D3 I have two identical relationships between Room and Exit--one for leaving a room by way of an exit and the other for entering a room from an exit. The one tricky thing is determining which association was connected to which navigation and FK property. First you need to right click on the navprop and choose the "select association” menu item. Then you can double click on the association to see a dialog which describes the referential constraint including the FK property.

Step 3 : Updating Model First Workflow & Templates

The final step was to update the D3 model first workflow and templates both because the bug mentioned in this previous post was fixed and because of other changes and improvements made by the designer team in beta 2. You can read the details of those improvements on the ado.net team blog, but the summary is that there are now two properties available in the EDM designer when you click on an empty place on the designer surface. The “Database Generation Workflow” property allows you to specify which workflow xaml file will be used when you choose to generate the database from the model. The default value picks up the file from one of the VS installation directories, but it can also point to a per user directory (where a new xaml file could be installed by a VSIX package for instance) or to a location relative to your project directory. The “DDL Generation Template” property specifies the T4 template used in the step for generating the data definition language based on the SSDL.

I was able to keep the overall strategy the same—for the most part just comparing the original versions of the files to the new versions and reacting to things like the renaming of the namespace Microsoft.Data.Entity.Design.Pipeline.EdmExtension to Microsoft.Data.Entity.Design.DatabaseGeneration.Activities, or the change in xml namespace for the StoreGeneratedAttribute in SSDL from the SSDL namespace to the annotations namespace.

In the case of the SsdlToCode template there were more extensive changes, so I took the new template from the system and reapplied the same customization I originally made. Once the updates were complete, I diff’d the new template against the old one and the changes generally fall into the following buckets:

Updating assembly and namespace names
Moving re-usable utility code out of the template and into a shared ttinclude file
Supporting multiple sql server schemas in the same ssdl file
Supporting SQLCE by skipping things it doesn't allow like specific schema names

While I was at it, I also applied a couple of improvements which a colleague suggested could be made to the default template including removing the “WITH NOCHECK” clauses when creating constraints and adding statements to create indexes for each foreign key.

Final Result

The final result of all this is that D3 is up and running on beta 2. We’re back to the functionality we had last June with beta 1 plus we now have FK properties and a somewhat more efficient database. This is just the foundation needed so that we can start filling in more of the overall application.

- Danny

Footnotes
1. For some of us increasingly old geezers, the cartoon Bloom County by Berke Breathed was a major part of our lives once upon a time. So I can’t help occasionally throwing out obscure references to it. You can see the specific strip I’m referring to here.

It’s time to rip up EntityBag and throw it away

dsimmons@microsoft.com — Thu, 05 Nov 2009 18:36:05 GMT

A couple of years ago when the overall shape of what we would ship in the first release of the entity framework became apparent, I realized that one of the biggest issues users of that release would fight with was creating N-tier applications. So I started a project to explore the space by creating something I called EntityBag. In spite of the fact that I wasn’t too thrilled with the overall approach, I thought it would be a good exercise and if nothing else it would help illuminate the issues with building this kind of application on the EF and demonstrate to folks what techniques they could use with the EF to solve these problems. The result of this investigation was a series of blog posts and a sample project which I uploaded to code gallery. You can check out the project at: http://code.msdn.microsoft.com/entitybag/ That site also has links to each of the relevant blog posts which is where the real interesting data lives.

At the time that I created EntityBag (mostly January 2008), the version of the Entity Framework was called “beta 3”, and it was before the first RTM of the product. Unfortunately, when the first version was finally released we included a change to by default generate classes which would serialize an entire graph of related entities as one (at beta 3 only one entity would serialize at a time) which ended up breaking EntityBag and the fix involved fairly major surgery. From the beginning this project had never been intended as a full-quality production solution—I just wanted to teach people how they could use the EF to build the right solutions for their particular projects. In addition I wanted to work with the team to produce a much better, long-term solution for N-tier apps, so whatever time I had available for this topic I poured into the next release of the EF (EF4) rather than into updating the EntityBag project.

If not EntityBag, then what have we been doing for N-tier?

The team has been working hard to make N-Tier much easier with EF4, and the result is several key features that are available now when you combine Beta 2 of VS2010/.NET 4 and yesterday’s release of what we call the EF Feature CTP 2.

The first critical addition to the product are core changes to the ObjectContext and ObjectStateManager APIs which make working with entities which spend some time detached from a context MUCH easier. These changes were described about a year ago in this post to the EF Design blog. There have been a few small adjustments to the APIs as we went through the experience of the first and second betas for this release, but the core changes are all there along with much of the reasoning behind them.

Next, we added support for foreign-key relationships which allow FK properties to be added to entities, and the system does intelligent fixup between the FK properties and the collection and reference “navigation” properties which were the only representation of a relationship which the EF had in its first release. FK relationships are important to N-Tier apps for two reasons: First off, they make it easier to support scenarios where you don’t want the perf hit (magnified in N-tier applications) of retrieving an entire object in order to establish a relationship. With FK properties you can just set the key value for the relationship. This was possible before by setting an EntityKey property, but that was harder than it should have been and MUCH harder (and sometimes impossible) when you have POCO entities. Even more interesting for N-tier applications, though, is the fact that FK relationships give the EF the knowledge that a relationship’s persistence is fundamentally tied to a particular entity rather than being conceptually completely separate, and this is important because it allows the EF to use concurrency checks for the entity to cover concurrency checks for the relationship as well. In the first release of the EF, the framework had no way to know from the conceptual model which if any entity “contained” the FK for the relationship so it performed concurrency checks for the relationships separately from the entities which meant that you needed the original value of a relationship not just the original value of whatever concurrency token you used for the entity. Uggg… That’s a little long-winded and obtuse, but the key point to realize is that having FK-relationships makes n-tier applications more efficient and intuitive in at least a couple of ways.

Finally, we had resounding feedback in response to the initial N-tier post on the EF Design blog and in other places saying that while the core APIs were flexible and made things easier, there was still just too much work required to build end-to-end N-tier solutions. So we created the self-tracking entities template which ships for now in Feature CTP 2 but will be in the box when .NET 4 launches on March 22, 2010. Self-tracking entities won’t be right for everyone, and for those folks where they aren’t a great fit as is there will either be the option to modify the template (Long live T4!) or to write code by hand using the core API improvements on which the STE template is built. If they do match your situation, though, they make things MUCH easier. The amount of code required is dramatically less, and the functionality is really pretty great. You can write simple service methods that return entities. On your client (which does NOT require .Net 4 or any version of the entity framework—it works on silverlight, for instance) you can manipulate entity graphs by adding, deleting and modifying entities as well as splicing more than one graph together if you like, and the entities will keep track of the minimum required set of original values as well as whatever changes are made to them all by themselves. NO BAGS OR OTHER EQUIPMENT REQUIRED. The only constraint is that you must use the generated self-tracking entity code on your client (not a simple proxy). Then back on your service, the code to reattach that graph to the context with appropriate original values and all the tracking information is literally one method call: ApplyChanges.

How can I learn more about all this?

In addition to the posts referenced above, I recommend taking a look at the Self Tracking Entities walkthrough which was published after the first release of the feature CTP. The current release of the STE template is a little different, but this is a great starting point, and more information will be available on the ado.net team blog soon. The other resource to consider is that over the course of this summer I wrote a short series of articles in MSDN magazine about the topic of N-Tier in general and especially around using it with the Entity Framework. The third article in that series just came out in the November issue, and it contains samples both of using STEs and using the core APIs to build other N-tier solutions. You can find the articles online here:

So, death to EntityBag! Long live EF4!

- Danny

D3: Fun with Concurrency and Model First – Part 1

dsimmons@microsoft.com — Sat, 27 Jun 2009 03:10:00 GMT

The second D3 release is now up at: d3-0.0026.zip It’s the fruit of my labors pretty much full-time this week because it has been “App Week” on the EF team this week.

What a great time! My kids got out of school for summer break last Friday (which means I have the option of sleeping in rather than waking up early to drive the car pool), and this week I got to almost completely clear my calendar and spend time working on DPMud. App week is a time we set aside periodically to devote as much of our attention as possible to using the EF and related technologies to build applications so that we can experience some of the joys (and pain) of our valued customers. Most everyone looks forward to it, and we invariably find lots of great opportunities to improve the product.

While most other folks have been building new applications from scratch, I devoted my week to concentrated time on my long-running favorite app and just picked the next task off my to-do list and went to work. That task was to add a concurrency-token property to each of my entities and a simple test to verify that optimistic concurrency checks are working correctly. Seems pretty simple right? The model currently only has 4 entities, so how hard could it be? Can you say “working all week on this one task”? :-(

I won’t claim to be a test-first kind of developer, and the test code below is something I wrote after I had things working rather than first, but I’ll show it to you first so you can capture the goal. Then we’ll talk through the challenges I encountered and how I conquered them (in the hope that I can save you some pain). If you remember from my post about the last release, each entity currently in the model has a similar set of scalar properties but different relationships. My idea was to just add a Version property which would be marked as a concurrency token and setup to be automatically updated in the database any time any property on the entity changes so that we would get automatic concurrency checks. This property should actually have protected visibility because I don’t want to interact with it directly when using my entities—I just want the EF to use it and maintain it for me. The result would enable this test helper method:

private void VerifyConcurrency<T>(Func<T> create, Action<T> modify) where T : class
{
    using (var ctx = Utilities.CreateTestContext())
    {
        var objectSet = ctx.CreateObjectSet<T>();
        var obj = create();
        objectSet.AddObject(obj);
        ctx.SaveChanges();

        using (var ctx2 = Utilities.CreateTestContext())
        {
            var key = ctx.ObjectStateManager.GetObjectStateEntry(obj).EntityKey;
            var obj2 = (T)ctx2.GetObjectByKey(key);
            modify(obj2);
            ctx2.SaveChanges();
        }

        modify(obj);
        ctx.SaveChanges();
    }
}

I made the method generic and gave it two delegate parameters so that I can easily re-use it for each entity type I currently have plus extend it as I add more types over time. I also designed the code so that it will work with POCO types since I anticipate switching D3 to use POCO classes in the near future. As a part of this release I also introduced a simple utility method which creates the D3Context with a connection string to my little test database—this way it’s quick and easy to type since intellisense finds things for me.

The basic pattern in this method is that it uses one context to add an entity to the database. I use the create delegate because different entities have different requirements as far as what must be set on them to make them persist successfully. Each of our current entity types requires a non-null Name property, for instance, but while the Room entity doesn’t have any other required properties, an Actor must have a non-null Room reference, etc.

Once the new entity is added, I then create another context instance (which means another connection to the DB, etc.) and on it retrieve another copy of that same entity. I do this retrieval by having the state manager in the first context lookup the entity for me by reference (works for POCO) and then give me back the EntityKey corresponding to it (also works for POCO even though the entity itself doesn’t have an EntityKey property). Then I can use that key with the second context to load another copy of the entity, and call the modify delegate which makes a unique update to some property on the entity and saves that update to the database.

At this point the first instance of the entity is being tracked by the first context, and it’s concurrency token (assuming everything is configured correctly) represents the state of the entity before the second context updated anything. So we call the modify delegate on this entity and SaveChanges which throws an OptimisticConcurrencyException because the concurrency token in the database doesn’t match what’s in this context.

In the actual test, I create a list of Action types which call this method specifying the appropriate type and passing in lambdas for the create and modify delegates. Then it just loops over that list running the action and verifying that each call produces the appropriate exception. You can look at the file Tests\Concurrency.cs in the release if you want the full details.

The point of all this, though, is that to make this test pass I needed my entities to have concurrency properties that were configured properly in the database. With recent versions of SQL Server, the best way to accomplish that is with a RowVersion column (previously called a timestamp). This special column type is stored as an 8 byte binary, and the server guarantees that it will have a unique value any time any part of a row in its table is modified. The trick is that this special type only exists in the server—it’s not available in the CLR or the EDM conceptual model. If I were reverse engineering a model from the database, that would be no big deal. The tools would automatically map the RowVersion column in the database to an 8 byte binary property in my model. In this case, though, I’m using model first to generate the database from my model, and when I create the property, model first has no way to know that it should create a RowVersion type rather than a binary type in the DB.

So it was time to customize the model first process again, but before I got even that far I ran into a bug. Ahhh… The joy of AppWeek and of beta software. The issue is that if you set Concurrency Mode to Fixed on any of your entities in VS 2010 beta 1, then when you choose Generate Database Script from Model, the tool will display an error rather than generating the script. Considerable debugging and discussion with the developer who did most all of the work for model first later, we were able to track the problem down to a one line problem where model first was assuming that all property facets that could be set in the conceptual model were at least valid possibilities for the storage model, when in fact Concurrency Mode is the one property which can only be in the conceptual model and not the storage model. So the workflow activity that generates the SSDL from the CSDL was triggering an exception from the metadata system.

This bug will be fixed in beta 2, but that doesn’t help us much for D3 right now. So my final solution was to extract a small piece of the model-first workflow into a separate DLL, make a fix to the code and then modify the DbGen.xaml file which describes the workflow to call out to that DLL rather than the core version that ships with the product. The final result is part of this release, and it’s called modelfirst-bugfix.dll. If you go the misc directory under the release and run the batch file install-bugfix.cmd it will copy the DLL to a directory in your VS installation (make sure you run this from a VS command prompt where the DevEnvDir environment variable is set). Naturally this little hack is completely unsupported, but it works for me, and it might work for you too if you need to use model first with concurrency and beta 1 of EF4.

Now at least I can run model first when I have concurrency configured, but I’ve still got the problem that my properties come through as binary rather than RowVersion type in the DB. That’s where I’ll pick up the next post… Until then, happy coding.

- Danny

D3: Release 0.0010

dsimmons@microsoft.com — Thu, 11 Jun 2009 02:22:57 GMT

OK folks, after way too much delay, I’ve finally gotten all my ducks in a row and begun the process of rewriting DPMud. Since my goal is to completely rebuild it from scratch, the process is going to take a while, and I intend to let you all look over my shoulder a bit as I go. So don’t expect too much from this first release—all we have is a basic solution with a very simple model and some unit tests. You can download the release from: d3-0.0010

First, let’s take a look at a few decisions that came up along the way, and then we’ll dig into some specific code areas.

Decisions

Source Control and Licensing: It is very important to me to make this project as relevant and useful as possible for folks trying to build applications using the EF. So we decided to release the code via MS-PL which is a very permissive license. I’m no lawyer, but my understanding is that this basically means you can copy the source and use any part or all of it in your own projects. Originally it was my hope that I could just put the project on codeplex and use it for source control so that anyone interested could enlist and just sync down changes to their local machine as the project progresses, but there were various hang-ups with this approach, and in the end I decided to just use local source control and then periodically push releases up to code gallery which has a simple site system and where everything is automatically licensed with MS-PL.
Version Numbers: One of those mundane but required decisions along the way is how we will create version numbers for the releases. Here’s what I decided: Until we have the thing up and running, everything is version 0.something. The last four digits of the version are in the form MMDD where MM is the number of months since I started releasing (so 00 is June ‘09), and DD is the day of the month for that release. I haven’t done it yet, but that allows me to have a very simple algorithm to automate version stamping and such as I make subsequent releases.
EF Versions and Dependencies: One critical area of decisions was around what EF features do I take dependencies on—particularly when we’re in the state where EF4 is not yet complete. In the end I decided that the main principle was to avoid taking dependencies on EF features that aren’t yet available outside the team so that you can stay up with me if you like. This means, for instance, that I’m not going to take a dependency on foreign-key support even though I have access to an internal build where it’s working great. When FKs are publically available, we’ll take a look at the project at that point and evaluate whether or not it’s worthwhile to refactor the app to take advantage of them (almost certainly we will do that refactoring). Similarly, I’m not yet going to take a dependency on self-tracking entities because it isn’t yet available. It will become available much sooner than FKs, though, so I will plan on taking advantage of it soon. A harder question was code-only. The first CTP of it will also be available soon (in the same download with self-tracking entities), but I already know of some areas where I’ll need more control over the model and database generation than code-only will support in its first CTP, so we’ll have to wait on that refactoring until a later CTP of code-only. So the basic approach will be to use EF4 beta 1 with model-first to make it efficient to design the model and generate the database and code from that model.

Initial Model

Like every other part of the system, we’re going to start with a simple model and expand. Here’s a first look:

A few things to notice:

Each entity has an Id property which is a server-generated integer.
Each of these entities is a “real” thing which players will interact with in the virtual environment of the game, and as such we have given them all a few common properties including Name and Description. Eventually we’ll define an interface for these common properties so that we can reason about the commonality of these objects. We could have an abstract base class for each of these entities, but we want them each in their own entity sets, and the designer doesn’t support more than one entityset with the same base type, so using an interface gets the key aspects we want while allowing us to still use the designer (we don’t really need implementation inheritance here anyway).
The two associations between Room and Exit have each had one of the navigation properties removed so that Room only has one collection of “Exits” (it doesn’t also have a collection of entrances, so to speak), and Exit has only the TargetRoom navigation property. This makes our entities follow the intended model where exits go only one way. If you want two rooms to be connected in a symmetrical fashion, you need two exits one going from room A to room B, and another one going back from room B to room A. This is important because it means that we can easily give different names to the exits (from room A you go East to room B, but from room B you have to go West to get back to room A), and it means that you don’t always have symmetrical paths (when you jump off a cliff you can’t necessarily get back to the top of the cliff using the same way you got down, etc.).
We don’t yet have version fields for concurrency checking. That will come later, because it requires some additional customization to the database generation.
Item has a relationship with Actor and one with Room, but logically an item can only be related to one of them at a time (and it must always be related to one or the other so that items have location). These constraints we will represent in the database rather than the conceptual model. They aren’t currently enforced, but they will be once we make additional customizations to database generation.

Customizing “Model First”

The feature we call model first shows up in the entity designer (aka Escher) as the option “Generate Database Script from Model” on the context menu when you right click on an open part of the designer surface. Out of the box this will generate a SQL DDL script that you can use to create a database for persisting the entities in your model. It also automatically generates the SSDL and MSL which corresponds to that model. This is great because it means that I can concentrate on the thing that is important to my program, the conceptual model. If I make a change to the model, once that change is complete, I’m done. I don’t have to also modify the database and the mapping to reflect that change. Since I’m also generating entity classes from the model, I get a nice “DRY” (Don’t Repeat Yourself) development experience even though various parts of the system do have repetition, I don’t have to engage in that repetition myself. Naturally we want to have an even more DRY experience, and code-first will lead us there for folks who want to do everything in code, but for the design-time experience this is not half bad.

Naturally, though, there are times where I want more control over the database generation process, which is why there are customization hooks built-in. The process of generating the database is orchestrated by a customizable workflow which has a couple of activities that manage generating an MSL and SSDL for the CSDL and then generating SQL from that SSDL. This second step (SSDL->SQL) is further customizable because it is accomplished with a T4 template. Eventually we will take advantage of these customization mechanisms to allow us to automatically generate rowversion columns for concurrency, constraints that keep item relationships straight and other things.

All of this seemed somewhat academic, though, for the first few phases of the project until I started thinking through my testing strategy. Eventually we’ll test parts of the system using fakes and mocks and the like, but we also need integration tests that send data all the way down to the database and back, and I wanted to start writing those tests right away and deliver them with my initial stab at the model. To make that work I really wanted APIs for creating and dropping the database somewhat like we can get from L2S so that my tests can create a temp version of the database (separate storage but same schema) and test pushing data in and out of it. Eventually this will also be useful for deployment, import/export of data, version-to-version upgrades and the like.

So, I decided to customize the db generation process now by modifying the SSDL->SQL template to generate a C# method for creating the database schema rather than a SQL script. The process to set things up for customization looks like this:

Copy the workflow xaml file from %ProgramFiles\Microsoft Visual Studio 10.0\Extensions\EntityFrameworkTools\Workflows\DbGen.xaml and the SQL gen template from %ProgramFiles\Microsoft Visual Studio 10.0\Extensions\EntityFrameworkTools\Templates\SsdlToSql10.tt to my project.
I renamed the template to SsdlToCode.tt and modified the xaml file’s TemplatePath to just have that name rather than the full path to the original template.
Both files I added to my project in VS, but because they aren’t actually used at compile time (they are just used by the designer when I choose the menu option to generate the database) I made sure that the Custom Tool property for each of them is blank and the Build Action property is set to None.
I clicked on a blank part of the Entity Designer surface and then set the “Generate Database Script Workflow” property to just DbGen.xaml (just the file name because the xaml file is in the same project and directory with the EDMX).

Now when I choose the option to generate the database it runs the workflow as specified in my project which then uses the template from my project rather than the default versions.

Next, I modified the template file to output C# code rather than SQL. You can find the template file in the release and do your own diff against the default template, but the basic approach was to wrap each batch of SQL statements up to a “GO” in an ado.net command with a C# @”” string and execute it. I eliminated a chunk of the template that would drop existing tables because my approach is to drop the whole database and recreate it, and I put all of these execute statements into a method on an internal partial class that I added to my model DLL. That way most of that class is defined outside the template but this one method can be defined by the code generated from the template. Then I further created a partial class file for my generated EF context and added methods to check for existence of the database, drop the database, and create it (using the generated method).

One current limitation in the designer is that the generate database command always by default creates a file with a name based on the name of the EDMX file and with SQL for the extension. I’ve already started discussions with the designer folks about adding some customization hooks here, but for now you just have to remember that whenever you generate the database you should rename the file.

All of this enabled me to write the following simple test:

[TestMethod]
public void DropAndCreate()
{
    using (var ctx = new D3Context("name=D3TestContext"))
    {
        ctx.DropDatabase();
        Assert.IsFalse(ctx.DatabaseExists());
        ctx.CreateDatabase();
        Assert.IsTrue(ctx.DatabaseExists());
    }
}

Not only is it simple to perform these operations, but also everything is based on the connection string given. So I just created two different connection strings in my app.config file—one called D3Context (the default) and the D3TestContext which is the exact same as the other except that it has a different initial-catalog name for my test database.

The SqlDb Internal Class

It’s probably also worthwhile to take a brief look at the internal class I called SqlDb which does the heavy lifting for these database interactions. The first interesting piece of the puzzle is that in order to create or drop the database with SQL Server (or in this case Sql Express) we need to make sure that we don’t use a connection string with the database in question as the initial-catalog. So, we extract the connection string from the StoreConnection property on the EntityClient connection instance created by the context and create a SqlConnectionBuilder from it. SqlDb’s constructor takes a DbConnection parameter since that’s what ObjectContext exposes and creates the builder like this:

var builder = new SqlConnectionStringBuilder(((EntityConnection)entityConnection).StoreConnection.ConnectionString);

Then we store the initial-catalog property into a string field so we know the name of the database we should be working with, and then we replace the initial-catalog with “master” since that database is present on every SQL instance.

The constructor then extracts the new connection string from the builder and uses it to create a new SqlConnection instance. We use this connection to execute commands without directly connecting to the database. The class also implements IDisposable so that its Dispose method can Dispose of the SqlConnection.

The other interesting trick which came up was the fact that while my initial executions of the above test succeeded while stepping through in the debugger, running the whole test straight through failed because the first part of the DropDatabase method executes a command to check if the database exists, and after that the drop command would fail with an error saying that the database was in use. Eventually I fixed this by adding a ClearAllPools method to the class which closes the connection, calls SqlClient.ClearAllPools() to make sure that there are no connections in the connection pool holding onto the database and then re-opens the connection. This method is called after the existence check and before the command to drop the database.

Conclusions

We’re finally off and running! We have an initial model, a simple way to deploy the database and some initial tests. (Those tests have already found a couple of issues I had in my first crack at the model, by the way. So the testing is already paying off. Yay!) Next I’ll probably tackle some further customizations of the generated database. Soon we’ll look at replacing the default code gen with a POCO template, and in the not distant future I hope to fill out more pieces of the architecture so we can get a small end-to-end slice going with a WCF service and Silverlight, but all in good time…

- Danny

Links: EF4 Podcast, N-Tier Anti-Patterns, and some thoughts about DDD

dsimmons@microsoft.com — Wed, 03 Jun 2009 10:56:50 GMT

Here are a few more resources I hope you will find useful:

Recently I had a conversation about EF4 with Carl and Richard of .Net Rocks, and it went up on their site today. Have a listen at http://www.dotnetrocks.com/default.aspx?showNum=451.

In addition, I have an article about design/architecture decisions related to n-tier applications in the latest issue of MSDN Magazine. This article is actually the first part of a two-part series. Originally it was a single longer article that wasn’t so depressing as this one came out (all about what *not* to do without really giving any help on what you should do). There are some important concepts, though, so hopefully you’ll find something useful even in this first part.

At the same time that I’ve been busy with these, some other folks have been trying out the beta of EF4 and some very interesting posts are starting to appear like this dive into producing a POCO repository with the EF by Keith Patton and a great discussion of LINQ Expression Trees and the Specification Pattern by the EF team’s own David DeWinter.

- Danny

Some Great EF Resources

dsimmons@microsoft.com — Tue, 26 May 2009 02:33:57 GMT

One of the things that’s cool about working on the EF team and getting the chance to talk to customers about what we do and answer questions and such is that I get to hear about lots of things that folks outside the EF team do with the EF. It’s especially neat when folks tell me about books, tools and other resources which help folks get more out of the EF or related technologies. Unfortunately I’m often so buried under various responsibilities that I don’t always share these great resources as quickly as I should. I also try to make sure that I give things a look over before I share them so that what I point you to are only the best, most valuable things which of course means that it takes even longer… Sigh.

The only reason I’m wasting your time with all the apologies today, is that I have three particularly good resources which I’m way overdue sharing. I assure you (and the creators of these resources) that the delay has nothing to do with their quality. If you haven’t seen these things, I highly recommend you check them out.

The first one is Julie Lerman’s book, Programming Entity Framework. Julie’s tireless efforts on behalf of the EF and its users hit a new height with the release of this book. There’s an incredible volume of material covering all parts of the first release of the EF, and all samples are written in both C# and VB.Net.

Next is the product of the Swiss MSDN team which put together a hand’s on lab for learning about the EF and even building an n-tier application using it. It’s all done using the first release of the EF (.net 3.5 sp1) so it doesn’t take advantage of the new features we have added in EF4 beta1, but it will help you learn the fundamentals, and if you are working on an application using the EF which must deploy before EF4 releases, then it will be an especially useful tool.

Finally, I’ve done some blogging about T4 and how you can use it to customize EF code generation in the first release. In addition you may have heard that EF4 uses T4 not only for all the built-in code generation but also for database generation. So learning to work with T4 is a good idea for anyone programming with the EF. Unfortunately, though, VS doesn’t have great built-in support for editing T4 templates. You can edit in VS, but it just treats them as a basic text file with no syntax coloring or intellisense support. Hopefully that will be fixed in a future release of VS, but in the meantime there are two companies which have released T4 editors that plug into VS which make the experience of working with T4 much more pleasant. Check out Clarius Consulting’s Visual T4 and Tangible Engineering’s T4 Editor. Particularly interesting is the fact that Tangible’s editor already has a version which works with VS 2010 beta 1, and you can download and install it from the extension manager within VS 2010.

If you give these a try, I’d love to hear about your experiences with them, and if you know of something else which makes it better or easier to work with the EF, I’m always looking for information about those things as well.

- Danny

EF4 Beta 1 now available!

dsimmons@microsoft.com — Wed, 20 May 2009 21:21:40 GMT

In case you haven’t heard yet, VS 2010 beta 1 and .Net 4 beta 1 are now available for public download, and that means EF4 beta 1 is available as well:

http://msdn.microsoft.com/en-us/vstudio/dd582936.aspx

Lots of goodies! Check it out, and let us know what you think.

- Danny

Online TechTalk about POCO and other EF4 Features

dsimmons@microsoft.com — Fri, 15 May 2009 09:47:31 GMT

Yesterday, while we were here at TechEd, Alex James and I sat down and recorded a short discussion about POCO and other features coming EF4. Check it out at: http://www.msteched.com/online/view.aspx?tid=ef76e96c-7471-44a6-aa88-1ada037471ca

- Danny

Code Sample from my Applied Entity Framework talk at TechEd 2009 now available

dsimmons@microsoft.com — Fri, 15 May 2009 08:51:14 GMT

This week I gave a talk at TechEd focusing on architectural considerations as well as some useful tips for creating n-tier applications with the Entity Framework v3.5SP1. The code from the simple sample application is now available at http://code.msdn.microsoft.com/dtl401sample/. It includes:

an Entity Framework-based Business Object and DAL
some rudimentary validation implemented using a T4-template to create customized code generation with an OnValidate partial method much like is available from LINQ to SQL
a repository which abstracts the UI from data access & a fake implementation of that repository
an asp.net MVC-based UI with URL routing that allows you to specify either to run on the fake repository or on the real one
a WCF implementation of that repository which remotes calls from the web tier to the midtier
several unit tests

Have a look and let me know what you think. Of course this is not the end-all-be-all of anything. It’s just a sample designed to illustrate a few concepts, but maybe someone will find it useful.

- Danny

Cool things are coming in EF4!

dsimmons@microsoft.com — Tue, 12 May 2009 04:36:05 GMT

I’m off at TechEd this week talking to customers about the EF—especially about ways to be successful using the EF right now with 3.5sp1. Meanwhile, though, we’re getting close to releasing the first beta of EF4 (yeah!). As we gear up for that release, the team has begun a series of posts on the ado.net team blog about all the great new features.

Here are the posts released so far:

Update on the Entity Framework in .NET 4 and Visual Studio 2010: This has a quick overview of some of the most exciting new features—sort of a brochure if you will.
Sneak Preview- Persistence Ignorance and POCO in Entity Framework 4.0

There are a lot more posts to come over the coming weeks. So stay tuned to the blog, and let us know what you think.

- Danny

Lots of EF Fun at TechEd Next Week

dsimmons@microsoft.com — Wed, 06 May 2009 07:45:22 GMT

I’m off to TechEd North America in LA next week where I’ll be presenting a 400-level session titled “Applied Entity Framework”. The session will mostly be focused on helping folks be more successful using EF3.5sp1 but also with some info about features coming in EF4 and how to prepare for them. If you are going to be at TechEd, you might want to check out DTL401.

There are quite a few other sessions that are either completely about the EF or contain EF content. So I created the following I-Cal file which makes it real easy to view them all in Outlook or your favorite calendar system: http://www.the-simmons.net/teched-2009-ef.ics In addition to the break out sessions, there are some birds-of-a-feather sessions that sound like they will be very interesting. A helpful resource for folks unable to make the conference in person or who want TechEd info before or after the conference is TechEd Online.

While I’m there, I’ll also be spending plenty of time in the technical learning center and just generally hanging out trying to meet with folks to hear about what you are doing with the EF, to help solve any dilemmas you may be struggling with, and to help folks start getting excited about EF4. While I’m on the topic of EF4 the first beta is coming pretty soon, but if you are at TechEd there will be a hands-on-lab where you can get an even earlier preview of the bits. I know I’m biased, but I think EF4 is really cool. I’m hoping it will really blow your socks off!

One last thing… Julie Lerman recently posted about the fact that we have been referring to the next release of the EF as EF4 to help people keep straight the fact that this is the version of EF going out with .net 4, and that led to an email thread where she suggested that we get some t-shirts made which sounded like lots of fun to me. I would have loved to get a whole bunch of these to give away at TechEd but given tight budgets that wasn’t in the cards (and make no mistake, I’d rather see Microsoft funds going into people and products than shirts). So I did some hunting on the web and found a place that will make custom shirts in small lots and ordered a few to commemorate the fact that EF4 feels like a lot more than the average second release (in spite of the fact that we didn’t have a full release worth of time to devote). The shirts look something like this:

In the off chance that you are crazy like me and interested in such a shirt, you can get your own here: http://www.zazzle.com/v2_squared_ef4_tshirt-235150309421493680

The way this site works, if you buy one of these, they give me a 10% kick-back, but I’m certainly not trying to make any money off of these shirts. So any royalties that come my way from folks buying these will be donated to a local worthy cause: Seattle Union Gospel Mission which does a lot for the homeless and others dealing with hard times in the Seattle area.

In any case, here’s hoping to see you next week!

- Danny

Building N-Tier applications with the EF – The Basics

dsimmons@microsoft.com — Thu, 09 Apr 2009 08:08:00 GMT

Today I was looking at a post in the forums where someone asked a very natural and common question about the EF that I end up answering pretty frequently. So I decided to put my answer here on my blog to make it easier to refer back to the answer in the future. The question was essentially this:

I’m building a silverlight client (but it could just as easily be a rich client or an ajax app or any other client of a WCF service) with a WCF service on the mid-tier that uses the EF to persist changes to a database. I can easily write a WCF service method to retrieve entities, and I can also create one which will add new entities to the database by just calling AddObject and then SaveChanges, but I can’t figure out how to do an update. I was hoping there would just be an UpdateObject method or something like that. What do I do?

Unfortunately this is more difficult than we'd like it to be in EF3.5 (the first release). It's a bit difficult because the problem space is more complicated under the covers than it seems to be on the surface, so you have to think carefully about a few things. That said, the EF API could help more in these scenarios, so in EF4 we're working on a series of improvements which you can read about in posts on the EF Design blog such as this one: http://blogs.msdn.com/efdesign/archive/2009/03/24/self-tracking-entities-in-the-entity-framework.aspx
In the meantime, there are three important things you need to think about:

Keeping track of concurrency tokens for the purpose of doing concurrency checks when you update (you don't want to blindly overwrite a change that some other client has made to the same part of the database between the time when you read the data and the time when you try to write back your changes).
Keeping track of which properties you have modified on an entity. This is important if you want to make sure that the update to the database only includes the modified properties. If you are OK with sending an update statement that includes even unmodified properties (which by the way can sometimes even be faster because it might allow the database to reuse a query plan rather than generating a new one for each combination of modified props), then it may be OK not to worry about this one.
If you have a graph of related entities then you need to worry about tracking what kinds of operations happened to what parts of the graph--one entity modified, another one is a newly added entity, another one deleted, etc.

All of this amounts to tracking this information on the client side, communicating it back to your mid-tier through your WCF service methods, and then replaying information to the ObjectContext so the changes can be persisted. If you skip option #3 for now (one of the hardest parts with EF3.5 and something which can occasionally be skipped if you are willing to just have WCF service methods which operate on one entity or which treat all entities in the graph the same way--so you have separate methods for add, update and delete), then you already know how to do the add method. The delete method is also pretty straight forward, you just need to call Attach on the entity you pass to the method to bring it into the context and then call DeleteObject on it and SaveChanges. The interesting method, as you mention, is update...

The simplest but not terribly efficient way to solve the update method is to make a copy of the entity on the client before you make any changes so that at the time you want to update things you have both a copy of the original version of the entity and the modified one. Then your update service method can look something like this:

void Update(MyEntity original, MyEntity updated)
{
    using (var context = new MyEntities())
    {
        context.Attach(original);
        context.ApplyPropertyChanges(updated);
        context.SaveChanges();
    }
}

You can improve on this, though, if you are willing to make a few assumptions. First, if you can guarantee that the client will not change the property used for concurrency checks, then you don’t need the original value of the concurrency property. Secondly, if you are willing to always send all properties to the server on an update rather than just those properties which have changed, then you don’t need an original value for all the other properties to compare against to see if they are changed. In this case you can get away with only sending a single, updated entity from the client back to the mid-tier, and you just have to use some trickier APIs to get the entity attached to the context in a form so that the EF will treat it as modified. I wrote a previous blog post about how to do that. With the extension method from that post, you can get rid of the original entity argument to the update method and just call the AttachAsModified method followed by SaveChanges().

- Danny

D3: Using T4 to Generate Entity and Context Classes

dsimmons@microsoft.com — Wed, 08 Apr 2009 10:00:00 GMT

Sorry for the DPMud hiatus. Here’s another installment for those interested in the process. I’m still trying to get a public source repository up. Until that’s functional I’m not going to do a lot of development because I want to share the process with you. In the meantime I’m still laying groundwork where I can, and one aspect of that is using T4 to generate classes for the entities and for the data access layer.

Side Note: We’ve been trying hard to refer to EF versions using the corresponding .net version rather than “v1” and “v2” since talking in terms of .net versions makes it clear when and how the corresponding versions ship. So don’t be surprised when I refer to EF4 or EF3.5.

The idea behind using T4 for code generation is that I’d like to create the DPMud model using the EF Designer and then generate both the classes and the database from the model (database generation has been discussed previously on the EF Design blog). There are two key architectural requirements for these classes that we generate:

The entities should be completely persistence ignorant. That is, they should live in an assembly which doesn’t have any dependency on Entity Framework or ADO.Net DLLs. One concrete reason for this is that I want to reuse the entity classes in Silverlight. Happily, EF4 adds extensive support for POCO entity classes.
The DAL should be designed in a way that supports easy introduction of a fake implementation. That is, it should be a “seam” in the terminology of design for testability. Another feature of EF4 which supports this is the introduction of a new interface, IObjectSet<T> which is implemented by the ObjectSet<T> class which provides both a starting point for queries and a place to add and remove entities of that type.

So I created a couple T4 templates. They share some infrastructure for helping to read EF metadata, escaping identifiers, generating multiple files, one per class, etc. We’re working on getting that infrastructure into the core product, so I’ll concentrate on the unique parts of these templates.

The entity template iterates over each entity type in the metadata and generates a partial class with each property marked virtual so that an automatic EF proxy can be generated if necessary. For navigation properties, it generates properties that are simple CLR references or ICollection<T> instances with an implementation that does fixup. The fixup implementation is generated so that it can be included in the entity DLL and not introduce any other requirements to the entity classes.

The context template generates a class which inherits from ObjectContext and has properties that return ObjectSet<T> instances. It also generates an interface for the seam with IObjectSet<T> properties and a fake context implementation which has fake object set instances which have a very simple wrapper around hashset<T>.

This way I can put the entities into one DLL and the context, interface and fake context into a separate DLL for the DAL.

- Danny

“Local” Queries

dsimmons@microsoft.com — Sat, 21 Feb 2009 10:26:00 GMT

Quite some time ago I wrote a blog post about the fact that EF queries execute at the database rather than locally which means that if you add an object to the context or you modify an object already attached to the context, then a query won’t be based on the local data.

Earlier today another blogger called this a “disappointing behavior” . Since I hate for the EF to disappoint folks, I thought I’d whip up a small extension method to help:

public static IEnumerable<T> Local<T>(this ObjectContext context,

string entitySetName) where T : class

{

return from stateEntry in context.ObjectStateManager

.GetObjectStateEntries(EntityState.Added |

EntityState.Modified |

EntityState.Unchanged)

where stateEntry.Entity != null && stateEntry.EntitySet.Name == entitySetName

select stateEntry.Entity as T;

}

To use this, you just need to supply the type and the entity set name:

foreach (Customer c in ctx.Local<Customer>("Customers"))

{

Console.WriteLine(c.ContactName);

}

In .Net 4.0 we will be adding a new type ObjectSet<T> which inherits from ObjectQuery<T> and provides methods for things like adding, attaching, deleting, etc. entities to the entity set which it represents. The default codegen will put instances of ObjectSet<T> on the context rather than ObjectQuery<T>. So, once .Net 4.0 ships we can change the extension method to:

public static IEnumerable<T> Local<T>(this ObjectSet<T> objectSet) where T : class

{

return from stateEntry in objectSet.Context.ObjectStateManager

.GetObjectStateEntries(EntityState.Added |

EntityState.Modified |

EntityState.Unchanged)

where stateEntry.Entity != null && stateEntry.EntitySet == objectSet.EntitySet

select stateEntry.Entity as T;

}

And then you won’t need to specify either the type or the entity set name but rather call the extension method on the object set so that you get a nicer usage pattern:

foreach (Customer c in ctx.Customers.Local())

{

Console.WriteLine(c.ContactName);

}

- Danny

D3: Modeling Part 1 – Real-World Relationships

dsimmons@microsoft.com — Wed, 18 Feb 2009 11:58:00 GMT

While there are a great many scenarios for “standard” associations in the EF, there are cases where an application requires something a little more sophisticated. DPMud is no different. It has two cases.

Relationships with Payload

A key part of the D3 model is the map which consists of rooms which have a many-many relationship. Each room can have exits which lead to other rooms. This would be pretty straightforward except we need to attach properties to the relationship for the name of the exit, whether or not it is locked, etc. In UML terms this would be called an association class. Ideally the EF would be able to directly support this kind of relationship, but there are a number of interesting semantics, and in the end the properties on the relationship amount to an entity with all the same sorts of behaviors. The main problem, though, is that you need both the ability to navigate directly across the relationship skipping the properties and the ability to access the entity “in the middle”.

The only real option here is to replace two entities and one relationship plus payload with three entities and two relationships:

You can easily enumerate the exits from a room, and if you want to transit from that room through an exit to an adjacent room, then you can navigate from the room to the exit and then to the destination room.

(You might notice, by the way, that there’s only one navigation property on Room and only one on Exit even though each of these participates in two associations. I accomplished this by manually editing the XML of the EDMX to get rid of the extra navigation properties. I call them extra because they confuse the common scenarios—it’s interesting to enumerate the exits leading out from a room but not to list the entrances, and similarly it’s much more useful to look at the destination of an exit rather than the source.)

Conditional Relationships

There are lots of kinds of conditions you might want to put on a relationship—maybe the customer entity only has a salesman if their purchases total more than $5000 over the last month or something. In general the EDM doesn’t provide facilities for describing arbitrary conditions on associations, behaviors of that sort should be implemented in business logic on the entity classes. There is a pattern which comes up occasionally, including in D3, which is worth looking at: Sometimes an entity might participate in multiple relationships but where only one of the relationships can be present at any one time.

In the case of D3, this shows up with Items. An Item can be in a Room, it can be carried by an Actor, or (at least in theory, we haven’t added this feature to DPMud yet) inside another Item like a backpack. While the EDM can’t directly capture this kind of constraint in the conceptual model, this can be accomplished fairly easily in business logic in the entities, and it’s quite easy to add constraints in the database itself that require that only one of the three FK properties on Item be non-null at any one time.

You might say that we could create a base type for Items, Rooms and Actors, called “ItemContainer” or something like that and then a relationship between Item and ItemContainer, and that approach could certainly work. There are consequences, though. First of all, we would either have to put all three entity types in the same entity set which is less usable (default context pattern would have one ItemContainers set rather than three sets: Items, Rooms and Actors). Also there could be some perf implications since deeper inheritance hierarchies with relationships affecting subtypes can cost more during view generation and verification and the default database schema would put all three entity types into a single table which probably doesn’t give the ideal locality of reference for common queries, etc. If we try to address these concerns by creating separate entity sets for the three subtypes rather than one top-level entity set, then the EF runtime could handle it, but the EF designer could not, and we would be forced to do all subsequent modeling by editing the XML directly which is certainly less than ideal. In the end, three separate relationships with a constraint just seems to be a smoother path, so that’s the approach we take in D3.

I suppose that’s all for tonight. Next time we’ll continue looking at D3 modeling decisions. Hopefully I’ll be able to work through the issues so that I can set up a source repository and start sharing out code before too long.

- Danny

P.S. As we walk through this process together, if a better approach occurs to you about one of these design decisions or if I gloss over something that needs more explanation, please don’t hesitate to say so. It’s my hope that we can all learn from one another.