system.data.objects dev guy : D3

D3: Rides Again – Now on VS2010 Beta 2

dsimmons@microsoft.com — Sun, 06 Dec 2009 01:19:18 GMT

And you thought I might never post on D3 again. I know. I know. Jedi knights don’t wait 6 months for a follow-up blog post.¹Well, the wait is finally over. I’ve put a new release of D3 up on code gallery. In this release the code has been updated to work with VS2010/.Net 4 beta 2. The release also includes a reorganization of project names and some new functionality, but I’ll talk about those in a future post or two. For this post, let’s look at the three broad steps involved in upgrading the project since they illuminate EF features and changes you might need to make to move some other project from beta 1 to beta 2.

Step 1 : Basic Compilation

When I loaded the solution in beta 2 and tried to recompile, the first problems I encountered were issues with stylecop—there were a number of errors referring to a file called ".NETFramework,Version=v4.0.AssemblyAttributes." I found a solution in this thread.

The next change needed was to react to the EF’s breaking change from beta 1 to beta 2 where the ContextOptions property DeferredLoadingEnabled was renamed to LazyLoadingEnabled. The semantics of this property didn’t change—just it’s name, so this was a simple fix.

Once compilation was passing, the next step was to start leveraging new EF features in beta 2.

Step 2 : Adding FKs to the Model

Beta 2 not only added runtime support for FKs in the model, but the designer is now smarter about them, and they are turned on by default. So I had a few options:

I could do everything by hand:

Manually add the FK properties.
Go to each association and add the referential integrity constraint between the FK property on the dependent entity and the primary key on the principal entity.
Remove the association set mappings for the associations since the mapping will now be handled by mapping the FK property on the entity just like any other entity.
Finally, either manually add the mapping for that property or regenerate the database from the model.

Since the model is not yet all that big, I could recreate it from scratch, requesting FK properties with each association.
Given that I already had a database with foreign key columns (just not exposed on the entities), I could reverse engineer the model from the database with the option turned on to generate FK properties on the entities.

I choose option #3 for no reason except that it was interesting to come full circle from a model-first database to a DB first model and then back to model first when further changes are needed. :-) In the process I encountered one may difficulty which was that reverse engineering from the database uses a simplistic algorithm for naming things that works reasonably well when you only have one relationship between any two entities, but when you have more than one relationship between the same pair of entities things get a bit confusing. In D3 I have two identical relationships between Room and Exit--one for leaving a room by way of an exit and the other for entering a room from an exit. The one tricky thing is determining which association was connected to which navigation and FK property. First you need to right click on the navprop and choose the "select association” menu item. Then you can double click on the association to see a dialog which describes the referential constraint including the FK property.

Step 3 : Updating Model First Workflow & Templates

The final step was to update the D3 model first workflow and templates both because the bug mentioned in this previous post was fixed and because of other changes and improvements made by the designer team in beta 2. You can read the details of those improvements on the ado.net team blog, but the summary is that there are now two properties available in the EDM designer when you click on an empty place on the designer surface. The “Database Generation Workflow” property allows you to specify which workflow xaml file will be used when you choose to generate the database from the model. The default value picks up the file from one of the VS installation directories, but it can also point to a per user directory (where a new xaml file could be installed by a VSIX package for instance) or to a location relative to your project directory. The “DDL Generation Template” property specifies the T4 template used in the step for generating the data definition language based on the SSDL.

I was able to keep the overall strategy the same—for the most part just comparing the original versions of the files to the new versions and reacting to things like the renaming of the namespace Microsoft.Data.Entity.Design.Pipeline.EdmExtension to Microsoft.Data.Entity.Design.DatabaseGeneration.Activities, or the change in xml namespace for the StoreGeneratedAttribute in SSDL from the SSDL namespace to the annotations namespace.

In the case of the SsdlToCode template there were more extensive changes, so I took the new template from the system and reapplied the same customization I originally made. Once the updates were complete, I diff’d the new template against the old one and the changes generally fall into the following buckets:

Updating assembly and namespace names
Moving re-usable utility code out of the template and into a shared ttinclude file
Supporting multiple sql server schemas in the same ssdl file
Supporting SQLCE by skipping things it doesn't allow like specific schema names

While I was at it, I also applied a couple of improvements which a colleague suggested could be made to the default template including removing the “WITH NOCHECK” clauses when creating constraints and adding statements to create indexes for each foreign key.

Final Result

The final result of all this is that D3 is up and running on beta 2. We’re back to the functionality we had last June with beta 1 plus we now have FK properties and a somewhat more efficient database. This is just the foundation needed so that we can start filling in more of the overall application.

- Danny

Footnotes
1. For some of us increasingly old geezers, the cartoon Bloom County by Berke Breathed was a major part of our lives once upon a time. So I can’t help occasionally throwing out obscure references to it. You can see the specific strip I’m referring to here.

D3: Fun with Concurrency and Model First – Part 2

dsimmons@microsoft.com — Sat, 27 Jun 2009 11:14:37 GMT

In my last post I began the story of how I added concurrency checking capabilities to D3—we conquered a bug in the model first workflow activities in VS 2010 beta 1. Next we need to look into ways to customize the SSDL generated so that the concurrency column will end up with the RowVersion type that SQL Server will automatically update when any part of the row changes.

This lead to the next surprise. At first I thought that I would add a new custom activity to the workflow which would fix-up the SSDL, but this turned out to have its own complexities. As it turns out, the model first process runs the workflow within the visual studio process which means the DLLs containing activities for the workflow are loaded in the same way other VS DLLs are loaded, namely from the VS install directories or from the GAC. This also means if the DLL changes, you have to shutdown VS and restart it before model first will reflect the change. Conclusion: If you want to create a general purpose activity that many folks will reuse, then this is a reasonable approach You can carefully debug an activity and set something up to install it centrally. But, if you need a one-off customization, this is just too hard.

So we’re looking into options to help here, but in the meantime I’ve discovered another approach which seems to work well. The default workflow described in DbGen.xaml has two activities. The first one takes the conceptual model and produces an MSL and SSDL string, while the second one takes the SSDL and produces something which will create the database (normally a SQL file, but after my first round of customizations for D3 it will produce a c-sharp file with a method that will create it).

It turns out, though, that the second activity under the covers is a general-purpose mechanism that can be configured to take in a string, run it through a T4 template and produce an output string. The nice thing about using a template is that it allows you to write arbitrary code as a one-off in your solution and easily iterate on your design, modifying and rerunning without any need to restart VS, etc.

Eureka! The second activity can be used twice. In between the two default activities, it can be inserted again and configured to take the SSDL in, transform it with a different template and send it on to the final activity. In addition model first makes some useful bits of information like a path to the EDMX file available to the template, so we can access the CSDL in order to make better decisions about how to modify the SSDL.

The relevant part of the resulting XAML looks like this:

<Sequence>
    ...
    <edm1:SsdlToDdlActivity ExistingSsdlInput="[ExistingSsdl]"
                            SsdlInput="[Ssdl]"
                            DdlOutput="[Ssdl]"
                            TemplatePath="DbGen\SsdlUseRowVersion.tt" />
    ...
</Sequence>

For D3, the fix-up we need is to find properties in the SSDL which correspond to properties in the CSDL with the following characteristics: type is binary, maxlength is 8, concurrency mode is fixed and store generation pattern is computed. So we read the EDMX into an XElement tree in memory and then use LINQ to XML to gather the names of properties that need to be changed in the SSDL grouped by the name of the table in which they appear:

var query = from property in csdl.Elements(edm + "EntityType").SelectMany(t => t.Elements(edm + "Property"))
            where (string)property.Attribute("Type").Value == "Binary" &&
                  (string)property.Attribute("MaxLength").Value == "8" &&
                  (string)property.Attribute("ConcurrencyMode").Value == "Fixed" &&
                  (string)property.Attribute(store + "StoreGeneratedPattern").Value == "Computed"
            group (string)property.Attribute("Name").Value
            by entitySetName(property.Parent, csdl, edm);

This query depends on knowledge of the algorithm used by the CsdlToSsdlAndMslActivity to map conceptual model properties to the database—namely that the column names are the same as the property names on the entities and (at least for the base types of entity sets) the name of the table is the same as the name of the entity set that contains the type. So we query the csdl for all the elements that represent properties which have the required facets and extract the name of the property, then we group those property names by the name of the entity set containing the entity type to which the property belongs using these two functions:

string entitySetName(XElement entityType, XElement csdl, XNamespace edm)
{
    return (from set in csdl.Element(edm + "EntityContainer")
                            .Elements(edm + "EntitySet")
            where entityTypeName(set) == entityType.Attribute("Name").Value
            select (string)set.Attribute("Name").Value).Single();
}

string entityTypeName(XElement entitySet)
{
    XElement schema = entitySet.Parent.Parent;
    string nsPrefix = schema.Attribute("Namespace").Value + ".";
    string aliasPrefix = schema.Attribute("Alias").Value + ".";
    string typeName = ((string)entitySet.Attribute("EntityType").Value).Trim();

    if (typeName.StartsWith(nsPrefix))
    {
        return typeName.Substring(nsPrefix.Length);
    }
    else if (typeName.StartsWith(aliasPrefix))
    {
        return typeName.Substring(aliasPrefix.Length);
    }
    else
    {
        return "UNEXPECTED TYPENAME -- Where is the namespace?";
    }
}

The result of this query is an IGrouping<string, string> where the first string is the name of the entity set and the second string is a collection of property names from the base type within that set. With that information it’s a simple matter to load the SSDL into memory, search for the relevant entity types (in the case of the SSDL, each table is represented by an entity type and each column within the table is a property of that type) and properties within them. Then we swap out the attributes on the XElement for that property and set the name, the type to rowversion and the store generated pattern to computed. Finally, the template serializes the XML for the SSDL to its output.

But what about derived types?

The above algorithm is all well and good for a model with no inheritance—every entity’s corresponding SSDL can be fixed up properly, but what if I want to put a concurrency property on an inherited type? As it turns out the EF doesn’t allow that; concurrency properties can only be present on the base type for an entity set. While I generally don’t like limitations of any kind, I can’t really think of any scenario where I’d want a concurrency property only on a derived type, and if I do have them on a base type, then I don’t need a different or additional property on any of the derived types.

In fact, for standard TPH and TPC scenarios all of the properties for a particular entity instance appear in a single row of a single table. With TPH all of the instances are in a single table, and in TPC there are multiple tables involved but each type has its own table with all of the properties both from the base type and added in the derived type in that table. So the concurrency property will appear on each table and a rowversion will work just fine because any property changing in that row will cause the concurrency property to change.

I got to thinking about TPT, though, and became a little worried. First off, TPT is the default inheritance strategy used by model first. In this strategy the properties from a base type all live in one table which has a row for every single entity in the set, while extra properties added by derived types are placed in their own tables with just those properties and the primary key. So for example, if I have entity type A with an id, scalar properties a and b, and a version, and I have type B which inherits from A and adds scalar property c, then the database would have table A with columns for id, a, b and version PLUS table B with columns for id and c. (A similar sort of case exists with entity splitting or mixed mode mapping, but it’s easier to think about the pure TPH, TPC and TPT scenarios.) In this kind of mapping, what happens if I have an instance of type B and only property c is modified? Then only table B needs an update, and the rowversion would not be updated since SQL Server only maintains it for changes to the row of the table where it resides. Doh!

After banging my head against the wall for a little while, I decided to check with some folks on the team to see what might be done to solve this problem. Could we relax the restriction on concurrency properties only being allowed in the base type (in which case maybe for TPT scenarios we could add a second concurrency property on each derived type which would track changes on the extra tables)? As it turns out Colin Meek, one of the brightest guys I know and a key member of the team, was way ahead of me. He was able to tell me that the design for the update code that handles concurrency checks already takes this condition into account, and whenever any part of an entity which has a concurrency token is modified, the update system of the EF guarantees that the table with the concurrency value is “touched”. In cases where there isn’t a real modification being made, a fake update statement is issued which doesn’t really change anything but does force the rowversion to update.

So, in the end there was nothing to worry about. The SSDL modification algorithm described above works just fine, and once the updated workflow XAML and template were put into place in the project, model first sales through cleanly and produces code which creates a database with rowversion properties for each entity type. And the whole thing is verified by the successful completion of the unit test described in the first part of the article.

Conclusions

AppWeek has come and gone, and in the end I spent the whole week getting my one test to run, but we have that much more of a stable foundation for D3 work to build on. In addition at least one key bug was found and fixed, and some extensibility scenarios were explored and opportunities for improvement identified. The really good news, though, is that the EF in general is starting to become flexible enough that even if it doesn’t work out of the box in just the way I want, I can customize it for my needs.

- Danny

D3: Fun with Concurrency and Model First – Part 1

dsimmons@microsoft.com — Sat, 27 Jun 2009 03:10:00 GMT

The second D3 release is now up at: d3-0.0026.zip It’s the fruit of my labors pretty much full-time this week because it has been “App Week” on the EF team this week.

What a great time! My kids got out of school for summer break last Friday (which means I have the option of sleeping in rather than waking up early to drive the car pool), and this week I got to almost completely clear my calendar and spend time working on DPMud. App week is a time we set aside periodically to devote as much of our attention as possible to using the EF and related technologies to build applications so that we can experience some of the joys (and pain) of our valued customers. Most everyone looks forward to it, and we invariably find lots of great opportunities to improve the product.

While most other folks have been building new applications from scratch, I devoted my week to concentrated time on my long-running favorite app and just picked the next task off my to-do list and went to work. That task was to add a concurrency-token property to each of my entities and a simple test to verify that optimistic concurrency checks are working correctly. Seems pretty simple right? The model currently only has 4 entities, so how hard could it be? Can you say “working all week on this one task”? :-(

I won’t claim to be a test-first kind of developer, and the test code below is something I wrote after I had things working rather than first, but I’ll show it to you first so you can capture the goal. Then we’ll talk through the challenges I encountered and how I conquered them (in the hope that I can save you some pain). If you remember from my post about the last release, each entity currently in the model has a similar set of scalar properties but different relationships. My idea was to just add a Version property which would be marked as a concurrency token and setup to be automatically updated in the database any time any property on the entity changes so that we would get automatic concurrency checks. This property should actually have protected visibility because I don’t want to interact with it directly when using my entities—I just want the EF to use it and maintain it for me. The result would enable this test helper method:

private void VerifyConcurrency<T>(Func<T> create, Action<T> modify) where T : class
{
    using (var ctx = Utilities.CreateTestContext())
    {
        var objectSet = ctx.CreateObjectSet<T>();
        var obj = create();
        objectSet.AddObject(obj);
        ctx.SaveChanges();

        using (var ctx2 = Utilities.CreateTestContext())
        {
            var key = ctx.ObjectStateManager.GetObjectStateEntry(obj).EntityKey;
            var obj2 = (T)ctx2.GetObjectByKey(key);
            modify(obj2);
            ctx2.SaveChanges();
        }

        modify(obj);
        ctx.SaveChanges();
    }
}

I made the method generic and gave it two delegate parameters so that I can easily re-use it for each entity type I currently have plus extend it as I add more types over time. I also designed the code so that it will work with POCO types since I anticipate switching D3 to use POCO classes in the near future. As a part of this release I also introduced a simple utility method which creates the D3Context with a connection string to my little test database—this way it’s quick and easy to type since intellisense finds things for me.

The basic pattern in this method is that it uses one context to add an entity to the database. I use the create delegate because different entities have different requirements as far as what must be set on them to make them persist successfully. Each of our current entity types requires a non-null Name property, for instance, but while the Room entity doesn’t have any other required properties, an Actor must have a non-null Room reference, etc.

Once the new entity is added, I then create another context instance (which means another connection to the DB, etc.) and on it retrieve another copy of that same entity. I do this retrieval by having the state manager in the first context lookup the entity for me by reference (works for POCO) and then give me back the EntityKey corresponding to it (also works for POCO even though the entity itself doesn’t have an EntityKey property). Then I can use that key with the second context to load another copy of the entity, and call the modify delegate which makes a unique update to some property on the entity and saves that update to the database.

At this point the first instance of the entity is being tracked by the first context, and it’s concurrency token (assuming everything is configured correctly) represents the state of the entity before the second context updated anything. So we call the modify delegate on this entity and SaveChanges which throws an OptimisticConcurrencyException because the concurrency token in the database doesn’t match what’s in this context.

In the actual test, I create a list of Action types which call this method specifying the appropriate type and passing in lambdas for the create and modify delegates. Then it just loops over that list running the action and verifying that each call produces the appropriate exception. You can look at the file Tests\Concurrency.cs in the release if you want the full details.

The point of all this, though, is that to make this test pass I needed my entities to have concurrency properties that were configured properly in the database. With recent versions of SQL Server, the best way to accomplish that is with a RowVersion column (previously called a timestamp). This special column type is stored as an 8 byte binary, and the server guarantees that it will have a unique value any time any part of a row in its table is modified. The trick is that this special type only exists in the server—it’s not available in the CLR or the EDM conceptual model. If I were reverse engineering a model from the database, that would be no big deal. The tools would automatically map the RowVersion column in the database to an 8 byte binary property in my model. In this case, though, I’m using model first to generate the database from my model, and when I create the property, model first has no way to know that it should create a RowVersion type rather than a binary type in the DB.

So it was time to customize the model first process again, but before I got even that far I ran into a bug. Ahhh… The joy of AppWeek and of beta software. The issue is that if you set Concurrency Mode to Fixed on any of your entities in VS 2010 beta 1, then when you choose Generate Database Script from Model, the tool will display an error rather than generating the script. Considerable debugging and discussion with the developer who did most all of the work for model first later, we were able to track the problem down to a one line problem where model first was assuming that all property facets that could be set in the conceptual model were at least valid possibilities for the storage model, when in fact Concurrency Mode is the one property which can only be in the conceptual model and not the storage model. So the workflow activity that generates the SSDL from the CSDL was triggering an exception from the metadata system.

This bug will be fixed in beta 2, but that doesn’t help us much for D3 right now. So my final solution was to extract a small piece of the model-first workflow into a separate DLL, make a fix to the code and then modify the DbGen.xaml file which describes the workflow to call out to that DLL rather than the core version that ships with the product. The final result is part of this release, and it’s called modelfirst-bugfix.dll. If you go the misc directory under the release and run the batch file install-bugfix.cmd it will copy the DLL to a directory in your VS installation (make sure you run this from a VS command prompt where the DevEnvDir environment variable is set). Naturally this little hack is completely unsupported, but it works for me, and it might work for you too if you need to use model first with concurrency and beta 1 of EF4.

Now at least I can run model first when I have concurrency configured, but I’ve still got the problem that my properties come through as binary rather than RowVersion type in the DB. That’s where I’ll pick up the next post… Until then, happy coding.

- Danny

D3: Release 0.0010

dsimmons@microsoft.com — Thu, 11 Jun 2009 02:22:57 GMT

OK folks, after way too much delay, I’ve finally gotten all my ducks in a row and begun the process of rewriting DPMud. Since my goal is to completely rebuild it from scratch, the process is going to take a while, and I intend to let you all look over my shoulder a bit as I go. So don’t expect too much from this first release—all we have is a basic solution with a very simple model and some unit tests. You can download the release from: d3-0.0010

First, let’s take a look at a few decisions that came up along the way, and then we’ll dig into some specific code areas.

Decisions

Source Control and Licensing: It is very important to me to make this project as relevant and useful as possible for folks trying to build applications using the EF. So we decided to release the code via MS-PL which is a very permissive license. I’m no lawyer, but my understanding is that this basically means you can copy the source and use any part or all of it in your own projects. Originally it was my hope that I could just put the project on codeplex and use it for source control so that anyone interested could enlist and just sync down changes to their local machine as the project progresses, but there were various hang-ups with this approach, and in the end I decided to just use local source control and then periodically push releases up to code gallery which has a simple site system and where everything is automatically licensed with MS-PL.
Version Numbers: One of those mundane but required decisions along the way is how we will create version numbers for the releases. Here’s what I decided: Until we have the thing up and running, everything is version 0.something. The last four digits of the version are in the form MMDD where MM is the number of months since I started releasing (so 00 is June ‘09), and DD is the day of the month for that release. I haven’t done it yet, but that allows me to have a very simple algorithm to automate version stamping and such as I make subsequent releases.
EF Versions and Dependencies: One critical area of decisions was around what EF features do I take dependencies on—particularly when we’re in the state where EF4 is not yet complete. In the end I decided that the main principle was to avoid taking dependencies on EF features that aren’t yet available outside the team so that you can stay up with me if you like. This means, for instance, that I’m not going to take a dependency on foreign-key support even though I have access to an internal build where it’s working great. When FKs are publically available, we’ll take a look at the project at that point and evaluate whether or not it’s worthwhile to refactor the app to take advantage of them (almost certainly we will do that refactoring). Similarly, I’m not yet going to take a dependency on self-tracking entities because it isn’t yet available. It will become available much sooner than FKs, though, so I will plan on taking advantage of it soon. A harder question was code-only. The first CTP of it will also be available soon (in the same download with self-tracking entities), but I already know of some areas where I’ll need more control over the model and database generation than code-only will support in its first CTP, so we’ll have to wait on that refactoring until a later CTP of code-only. So the basic approach will be to use EF4 beta 1 with model-first to make it efficient to design the model and generate the database and code from that model.

Initial Model

Like every other part of the system, we’re going to start with a simple model and expand. Here’s a first look:

A few things to notice:

Each entity has an Id property which is a server-generated integer.
Each of these entities is a “real” thing which players will interact with in the virtual environment of the game, and as such we have given them all a few common properties including Name and Description. Eventually we’ll define an interface for these common properties so that we can reason about the commonality of these objects. We could have an abstract base class for each of these entities, but we want them each in their own entity sets, and the designer doesn’t support more than one entityset with the same base type, so using an interface gets the key aspects we want while allowing us to still use the designer (we don’t really need implementation inheritance here anyway).
The two associations between Room and Exit have each had one of the navigation properties removed so that Room only has one collection of “Exits” (it doesn’t also have a collection of entrances, so to speak), and Exit has only the TargetRoom navigation property. This makes our entities follow the intended model where exits go only one way. If you want two rooms to be connected in a symmetrical fashion, you need two exits one going from room A to room B, and another one going back from room B to room A. This is important because it means that we can easily give different names to the exits (from room A you go East to room B, but from room B you have to go West to get back to room A), and it means that you don’t always have symmetrical paths (when you jump off a cliff you can’t necessarily get back to the top of the cliff using the same way you got down, etc.).
We don’t yet have version fields for concurrency checking. That will come later, because it requires some additional customization to the database generation.
Item has a relationship with Actor and one with Room, but logically an item can only be related to one of them at a time (and it must always be related to one or the other so that items have location). These constraints we will represent in the database rather than the conceptual model. They aren’t currently enforced, but they will be once we make additional customizations to database generation.

Customizing “Model First”

The feature we call model first shows up in the entity designer (aka Escher) as the option “Generate Database Script from Model” on the context menu when you right click on an open part of the designer surface. Out of the box this will generate a SQL DDL script that you can use to create a database for persisting the entities in your model. It also automatically generates the SSDL and MSL which corresponds to that model. This is great because it means that I can concentrate on the thing that is important to my program, the conceptual model. If I make a change to the model, once that change is complete, I’m done. I don’t have to also modify the database and the mapping to reflect that change. Since I’m also generating entity classes from the model, I get a nice “DRY” (Don’t Repeat Yourself) development experience even though various parts of the system do have repetition, I don’t have to engage in that repetition myself. Naturally we want to have an even more DRY experience, and code-first will lead us there for folks who want to do everything in code, but for the design-time experience this is not half bad.

Naturally, though, there are times where I want more control over the database generation process, which is why there are customization hooks built-in. The process of generating the database is orchestrated by a customizable workflow which has a couple of activities that manage generating an MSL and SSDL for the CSDL and then generating SQL from that SSDL. This second step (SSDL->SQL) is further customizable because it is accomplished with a T4 template. Eventually we will take advantage of these customization mechanisms to allow us to automatically generate rowversion columns for concurrency, constraints that keep item relationships straight and other things.

All of this seemed somewhat academic, though, for the first few phases of the project until I started thinking through my testing strategy. Eventually we’ll test parts of the system using fakes and mocks and the like, but we also need integration tests that send data all the way down to the database and back, and I wanted to start writing those tests right away and deliver them with my initial stab at the model. To make that work I really wanted APIs for creating and dropping the database somewhat like we can get from L2S so that my tests can create a temp version of the database (separate storage but same schema) and test pushing data in and out of it. Eventually this will also be useful for deployment, import/export of data, version-to-version upgrades and the like.

So, I decided to customize the db generation process now by modifying the SSDL->SQL template to generate a C# method for creating the database schema rather than a SQL script. The process to set things up for customization looks like this:

Copy the workflow xaml file from %ProgramFiles\Microsoft Visual Studio 10.0\Extensions\EntityFrameworkTools\Workflows\DbGen.xaml and the SQL gen template from %ProgramFiles\Microsoft Visual Studio 10.0\Extensions\EntityFrameworkTools\Templates\SsdlToSql10.tt to my project.
I renamed the template to SsdlToCode.tt and modified the xaml file’s TemplatePath to just have that name rather than the full path to the original template.
Both files I added to my project in VS, but because they aren’t actually used at compile time (they are just used by the designer when I choose the menu option to generate the database) I made sure that the Custom Tool property for each of them is blank and the Build Action property is set to None.
I clicked on a blank part of the Entity Designer surface and then set the “Generate Database Script Workflow” property to just DbGen.xaml (just the file name because the xaml file is in the same project and directory with the EDMX).

Now when I choose the option to generate the database it runs the workflow as specified in my project which then uses the template from my project rather than the default versions.

Next, I modified the template file to output C# code rather than SQL. You can find the template file in the release and do your own diff against the default template, but the basic approach was to wrap each batch of SQL statements up to a “GO” in an ado.net command with a C# @”” string and execute it. I eliminated a chunk of the template that would drop existing tables because my approach is to drop the whole database and recreate it, and I put all of these execute statements into a method on an internal partial class that I added to my model DLL. That way most of that class is defined outside the template but this one method can be defined by the code generated from the template. Then I further created a partial class file for my generated EF context and added methods to check for existence of the database, drop the database, and create it (using the generated method).

One current limitation in the designer is that the generate database command always by default creates a file with a name based on the name of the EDMX file and with SQL for the extension. I’ve already started discussions with the designer folks about adding some customization hooks here, but for now you just have to remember that whenever you generate the database you should rename the file.

All of this enabled me to write the following simple test:

[TestMethod]
public void DropAndCreate()
{
    using (var ctx = new D3Context("name=D3TestContext"))
    {
        ctx.DropDatabase();
        Assert.IsFalse(ctx.DatabaseExists());
        ctx.CreateDatabase();
        Assert.IsTrue(ctx.DatabaseExists());
    }
}

Not only is it simple to perform these operations, but also everything is based on the connection string given. So I just created two different connection strings in my app.config file—one called D3Context (the default) and the D3TestContext which is the exact same as the other except that it has a different initial-catalog name for my test database.

The SqlDb Internal Class

It’s probably also worthwhile to take a brief look at the internal class I called SqlDb which does the heavy lifting for these database interactions. The first interesting piece of the puzzle is that in order to create or drop the database with SQL Server (or in this case Sql Express) we need to make sure that we don’t use a connection string with the database in question as the initial-catalog. So, we extract the connection string from the StoreConnection property on the EntityClient connection instance created by the context and create a SqlConnectionBuilder from it. SqlDb’s constructor takes a DbConnection parameter since that’s what ObjectContext exposes and creates the builder like this:

var builder = new SqlConnectionStringBuilder(((EntityConnection)entityConnection).StoreConnection.ConnectionString);

Then we store the initial-catalog property into a string field so we know the name of the database we should be working with, and then we replace the initial-catalog with “master” since that database is present on every SQL instance.

The constructor then extracts the new connection string from the builder and uses it to create a new SqlConnection instance. We use this connection to execute commands without directly connecting to the database. The class also implements IDisposable so that its Dispose method can Dispose of the SqlConnection.

The other interesting trick which came up was the fact that while my initial executions of the above test succeeded while stepping through in the debugger, running the whole test straight through failed because the first part of the DropDatabase method executes a command to check if the database exists, and after that the drop command would fail with an error saying that the database was in use. Eventually I fixed this by adding a ClearAllPools method to the class which closes the connection, calls SqlClient.ClearAllPools() to make sure that there are no connections in the connection pool holding onto the database and then re-opens the connection. This method is called after the existence check and before the command to drop the database.

Conclusions

We’re finally off and running! We have an initial model, a simple way to deploy the database and some initial tests. (Those tests have already found a couple of issues I had in my first crack at the model, by the way. So the testing is already paying off. Yay!) Next I’ll probably tackle some further customizations of the generated database. Soon we’ll look at replacing the default code gen with a POCO template, and in the not distant future I hope to fill out more pieces of the architecture so we can get a small end-to-end slice going with a WCF service and Silverlight, but all in good time…

- Danny

D3: Using T4 to Generate Entity and Context Classes

dsimmons@microsoft.com — Wed, 08 Apr 2009 10:00:00 GMT

Sorry for the DPMud hiatus. Here’s another installment for those interested in the process. I’m still trying to get a public source repository up. Until that’s functional I’m not going to do a lot of development because I want to share the process with you. In the meantime I’m still laying groundwork where I can, and one aspect of that is using T4 to generate classes for the entities and for the data access layer.

Side Note: We’ve been trying hard to refer to EF versions using the corresponding .net version rather than “v1” and “v2” since talking in terms of .net versions makes it clear when and how the corresponding versions ship. So don’t be surprised when I refer to EF4 or EF3.5.

The idea behind using T4 for code generation is that I’d like to create the DPMud model using the EF Designer and then generate both the classes and the database from the model (database generation has been discussed previously on the EF Design blog). There are two key architectural requirements for these classes that we generate:

The entities should be completely persistence ignorant. That is, they should live in an assembly which doesn’t have any dependency on Entity Framework or ADO.Net DLLs. One concrete reason for this is that I want to reuse the entity classes in Silverlight. Happily, EF4 adds extensive support for POCO entity classes.
The DAL should be designed in a way that supports easy introduction of a fake implementation. That is, it should be a “seam” in the terminology of design for testability. Another feature of EF4 which supports this is the introduction of a new interface, IObjectSet<T> which is implemented by the ObjectSet<T> class which provides both a starting point for queries and a place to add and remove entities of that type.

So I created a couple T4 templates. They share some infrastructure for helping to read EF metadata, escaping identifiers, generating multiple files, one per class, etc. We’re working on getting that infrastructure into the core product, so I’ll concentrate on the unique parts of these templates.

The entity template iterates over each entity type in the metadata and generates a partial class with each property marked virtual so that an automatic EF proxy can be generated if necessary. For navigation properties, it generates properties that are simple CLR references or ICollection<T> instances with an implementation that does fixup. The fixup implementation is generated so that it can be included in the entity DLL and not introduce any other requirements to the entity classes.

The context template generates a class which inherits from ObjectContext and has properties that return ObjectSet<T> instances. It also generates an interface for the seam with IObjectSet<T> properties and a fake context implementation which has fake object set instances which have a very simple wrapper around hashset<T>.

This way I can put the entities into one DLL and the context, interface and fake context into a separate DLL for the DAL.

- Danny

D3: Modeling Part 1 – Real-World Relationships

dsimmons@microsoft.com — Wed, 18 Feb 2009 11:58:00 GMT

While there are a great many scenarios for “standard” associations in the EF, there are cases where an application requires something a little more sophisticated. DPMud is no different. It has two cases.

Relationships with Payload

A key part of the D3 model is the map which consists of rooms which have a many-many relationship. Each room can have exits which lead to other rooms. This would be pretty straightforward except we need to attach properties to the relationship for the name of the exit, whether or not it is locked, etc. In UML terms this would be called an association class. Ideally the EF would be able to directly support this kind of relationship, but there are a number of interesting semantics, and in the end the properties on the relationship amount to an entity with all the same sorts of behaviors. The main problem, though, is that you need both the ability to navigate directly across the relationship skipping the properties and the ability to access the entity “in the middle”.

The only real option here is to replace two entities and one relationship plus payload with three entities and two relationships:

You can easily enumerate the exits from a room, and if you want to transit from that room through an exit to an adjacent room, then you can navigate from the room to the exit and then to the destination room.

(You might notice, by the way, that there’s only one navigation property on Room and only one on Exit even though each of these participates in two associations. I accomplished this by manually editing the XML of the EDMX to get rid of the extra navigation properties. I call them extra because they confuse the common scenarios—it’s interesting to enumerate the exits leading out from a room but not to list the entrances, and similarly it’s much more useful to look at the destination of an exit rather than the source.)

Conditional Relationships

There are lots of kinds of conditions you might want to put on a relationship—maybe the customer entity only has a salesman if their purchases total more than $5000 over the last month or something. In general the EDM doesn’t provide facilities for describing arbitrary conditions on associations, behaviors of that sort should be implemented in business logic on the entity classes. There is a pattern which comes up occasionally, including in D3, which is worth looking at: Sometimes an entity might participate in multiple relationships but where only one of the relationships can be present at any one time.

In the case of D3, this shows up with Items. An Item can be in a Room, it can be carried by an Actor, or (at least in theory, we haven’t added this feature to DPMud yet) inside another Item like a backpack. While the EDM can’t directly capture this kind of constraint in the conceptual model, this can be accomplished fairly easily in business logic in the entities, and it’s quite easy to add constraints in the database itself that require that only one of the three FK properties on Item be non-null at any one time.

You might say that we could create a base type for Items, Rooms and Actors, called “ItemContainer” or something like that and then a relationship between Item and ItemContainer, and that approach could certainly work. There are consequences, though. First of all, we would either have to put all three entity types in the same entity set which is less usable (default context pattern would have one ItemContainers set rather than three sets: Items, Rooms and Actors). Also there could be some perf implications since deeper inheritance hierarchies with relationships affecting subtypes can cost more during view generation and verification and the default database schema would put all three entity types into a single table which probably doesn’t give the ideal locality of reference for common queries, etc. If we try to address these concerns by creating separate entity sets for the three subtypes rather than one top-level entity set, then the EF runtime could handle it, but the EF designer could not, and we would be forced to do all subsequent modeling by editing the XML directly which is certainly less than ideal. In the end, three separate relationships with a constraint just seems to be a smoother path, so that’s the approach we take in D3.

I suppose that’s all for tonight. Next time we’ll continue looking at D3 modeling decisions. Hopefully I’ll be able to work through the issues so that I can set up a source repository and start sharing out code before too long.

- Danny

P.S. As we walk through this process together, if a better approach occurs to you about one of these design decisions or if I gloss over something that needs more explanation, please don’t hesitate to say so. It’s my hope that we can all learn from one another.

D3: Basic Architecture Decisions

dsimmons@microsoft.com — Mon, 16 Feb 2009 01:36:00 GMT

In order to decide on the basic architecture for DPMud³ (I’ll just call it D3 from here on out), I first needed to build an understanding of the domain, then consider the goals and synthesize a basic approach.

The good news for me is that this is the third major version of this particular application, and in years past I spent way too many hours spending other MUDs and similar games, so I’m pretty well versed in the domain. To give a very quick overview: The key parts are that we model a virtual world in terms of rooms, passages between those rooms, items and actors which can either be players or non-player characters controlled by the game. Interaction happens in real-time via text which means that players express their intentions through simple text commands which must be parsed, and they learn about their environment through textual descriptions of things and events at their locale. The system needs a way to maintain a shared state for all the players, to communicate events as they occur (initiated by players or by other parts of the system such as computer controlled actors) and a framework for creating the AI that brings the non-player actors to life.

The first goal for this exercise is to build a real application not just a collection of samples artificially glued together but at the same time to try out new EF features such as POCO, model-first, mockable-interfaces, the n-tier APIs, T4 for code generation, etc. Another key goal is to use good architectural principles and popular app building patterns which means carefully separating concerns, using Inversion of Control to facilitate mocking and test driven (or at least very carefully tested) development, to create an n-tier architecture with WCF-services and a Silverlight client, and to use a model-view-controller pattern to simplify the creation of multiple views and controllers which share a common model.

The biggest challenge in synthesizing a design out of this domain and these goals is reconciling the multi-player real-time communication aspects with the goal to use and demonstrate EF (and to do so in a way that will be educational to folks building business apps rather than old-style multi-user games). The DPMud solution from the beginning has been to ask that you suspend disbelief about one architectural decision: we are going to use transacted database updates as a real-time communication mechanism. While this is almost certainly not the most efficient, scalable way to implement real-time communication, it does mesh very well with the EF and with the rest of our goals. It also has the potential to be a nice stress-test for the systems involved, and we do have the option to apply more hardware and more sophisticated build-outs to address any scalability problems we encounter.

Given that basic decision and the architectural pattern goals listed above, we end up with a 10,000 foot architecture diagram that looks something like this:

Next time… We’ll build up the model digging into some of the particular modelling difficulties that come up along the way.

- Danny

DPMud

dsimmons@microsoft.com — Fri, 13 Feb 2009 11:04:00 GMT

Back when the Entity Framework first started coming together as a product—well before any sane person outside the product team would actually try to use it to build an end-to-end solution—we had an application building contest within the team which we called ADO.NET Idol. The idea was to get a number of people to explore what it’s like to use the product in the way we intend for our customers to use it.

I'm always up for a challenge, and a couple friends of mine from elsewhere in Microsoft had been bugging me for a while to write a game together. So I dropped them an email and that was the beginning of a project we unimaginatively named DPMud. DP, of course, was for Data Programmability, and Mud was because the game we decided to build was a multi-user dungeon (see this page on wikipedia for some background on MUDs). One of my friends, Josh, was working on AI in the FlightSim/TrainSim team, and he was excited to help out not only with general coding, but especially with adding some AI. My other buddy, Jay, is a pretty serious gamer and a really creative guy, so I knew that he would be a great resource for story-line and content for the game. Jay got the process off to a good start by whipping up some concept art that has been part of DPMud ever since:

I believe the phrase "zombie chicks with shotguns" went with the diagram. Thus, in early April 2006, began a project which has off and on occupied a lot of my spare time ever since. The first version which had all of the basic elements working was completed on May 24, 2006. We continued to improve and expand the game for some time after that, and meanwhile the Entity Framework was expanded and revised to the point that a year after we started the effort, in April 2007, we began version 2, "DPMud Rarloaded" which was a pretty major rewrite to take advantage of new EF features.

Now, after a hiatus of quite some time, the major features of the second release of the EF are coming together, which creates the opportunity for some significant architectural improvements. So I've decided it's time for version 3 which we're going to call DPMud3: The Zombies Search for Spock.

So why am I telling you about all this? This time my plan is to share the project with all of you. In fact, I intend to build the app from scratch and journal the process as I go in the hope that some of you may find the journey educational. I'm sure that I will learn a lot, and I trust that if you spot something I could do better, you will let me know.

I guess that's all for tonight. Next time I'll share some more details about project goals and architecture ideas, but before I can actually start sharing source code, I've got to engage in one of those tasks that tend to show up in the early stages of a great many projects: working out some details with legal.

- Danny