compiler padawan

compiler padawan moves on

Daniel Simmons - MSFT — Wed, 01 Feb 2012 22:35:13 GMT

After nearly 15 years at Microsoft I've decided to leave the company and explore some programming adventures elsewhere. This Friday, February 3, 2012, will be my last day at Microsoft. It has been an absolutely GREAT experience, I have worked with some wonderful people and learned a lot, but I'm excited for a change.

If you are interested in following my adventures as I step out of the reality distortion bubble surrounding 1 Microsoft Way and enter some new form of reality distortion bubble, I'm now going to blog at http://blogs.the-simmons.net/danny

system.data.objects dev guy becomes a compiler padawan

Daniel Simmons - MSFT — Thu, 02 Dec 2010 23:46:22 GMT

More than 5 years ago I joined what became the Entity Framework team and started an awesome adventure which taught me a lot and has given me many memories which I will treasure: Everything from a room of MVPs yelling at me (so much that one of them later apologized on his blog) and a certain well-publicized “vote” … to the v2^2 t-shirts and talk about magic unicorns, and that’s just the public stuff. I won’t mention the crazy antics that went on behind closed doors just like pretty much any other hard-driving software team, or the experience of being part of a high-profile, at times controversial project in the world’s largest software company. It wasn’t all fun and games, but it has been great, and I’ve had the honor of working with an outstanding group of people both inside and outside Microsoft.

One of the best parts about working at Microsoft is the opportunity it gives you to explore a variety of things and work on multiple different projects without the impact on the rest of your life associated with changing companies. So I’ve decided the time has come for me to take advantage of this unique employment situation and explore some new challenges elsewhere in the company. Thankfully, I know that I’m leaving the EF in very good hands and that it has a wonderfully bright future ahead of it.

In case you are wondering where I’m going, I’m joining an incubation team which is in charge of, among other things, the phoenix project. Specifically I’ll be working on ahead-of-time compilation for c# in order to solve some really challenging problems in system programming. One of the things that has come out of the last several years for me is a real love of programming in c#, and this project has the potential to help extend the use of c# into domains where it can’t be used today. The team I’m joining faces some challenges which will benefit from my past experience, and at the same time there is a vast new domain to learn about where I have no experience (compilers).

So today I lay down my claim to the title “system.data.objects dev guy”, and take up the mantle of “compiler padawan.” (My wife asks if I’m going to grow a small padawan braid at the back of my head, but I really don’t think anyone wants to see that, so I suppose I’ll pass.)

- Danny

Simple WPF 2D Graphics: DrawingVisual

Daniel Simmons - MSFT — Sun, 28 Nov 2010 02:47:35 GMT

And now for something completely different… Over the last little while I’ve been working on a side project just for fun which involves some simple 2D graphics using WPF. Because I’m calculating the image I want to draw rather than it being a static thing, one of my criteria for the basic approach is that it will all be code oriented. I’m not by any means opposed to xaml, and there are places in this program where I could use some xaml, but it would only be for the most basic top-level windows and such. So I decided to just do the whole thing in C# and avoid any xaml to keep all my code in one place and to force myself to learn how all the pieces go together.

For my first attempt I did the easiest thing (and the one most frequently shown in simple WPF graphics demos) which is to use a Canvas and create on it a series of Line objects. The result looked like this:

[STAThread]
public static void Main()
{
    var app = new Application();
    var window = new Window();
    var canvas = new Canvas();

    window.Content = canvas;
    canvas.Children.Add(new Line
        {
            X1 = 0,
            Y1 = 0,
            X2 = 400,
            Y2 = 400,
            Stroke = Brushes.Black
        });
    canvas.Children.Add(new Line
        {
            X1 = 0,
            Y1 = 400,
            X2 = 400,
            Y2 = 0,
            Stroke = Brushes.Black
        });

    app.Run(window);
}

While this approach worked for my initial drawings and was quite easy, I became concerned when I wanted to introduce some animation because I had cases where I wanted to animate a figure made up of a large number of lines as though it was one thing, and it began to get a bit ugly to setup. In addition I started to worry a bit about perf, and even though I didn’t have the full program running yet so I wasn’t able to actually measure the perf, my reading on the net led me to think those worries were justified.

One fairly commonly recommended alternative approach is to use a DrawingVisual which as I understand it is basically a lower level construct which allows you to compose a number of drawing commands as though they were a single control in the visual hierarchy. Rather than each part of the drawing being full-fledged controls with their own events and everything, you create a series of drawing commands and put them inside a single object that is the only thing which has events and such. This is supposed to be faster, and it’s also supposed to allow you to use things like DrawingGroup which can cause all its contents to animate together as one thing—just what I wanted!

When I started to try this, though, everything I did seemed to just display a blank screen. My app would start, and the window would come up, but the graphics weren’t drawn into it. I looked at several samples—many of which were all in xaml—and you’d think that they would convert easily to code since xaml is just a straightforward mapping onto object creation after all, but I just couldn’t make it work. After much banging my head on the wall and searching on the net I eventually discovered that one of the main places I was looking, MSDN, had a sample with code shown on the help page that was incomplete. There was a small link to another page where you could go download the full sample, which worked fine. So a fair amount of scratching my head and trial and error later led me to the discovery of what was in that sample which was missing from my code. Hopefully this post will help some other poor wpf graphics programmer figure out their problem sooner.

It turns out that the key to my whole mess was creating the “visual host” control which would contain my drawing. This control inherits from FrameworkElement and provides the infrastructure for event handling and other things. It plays a role much like the canvas in the sample above except that with the canvas, each child is also something that inherits from FrameworkElement whereas when using a VisualDrawing this is the only thing that does that and all of its children are just simple drawing components. In addition to inheriting from FrameworkElement, the visual host needs to contain a VisualCollection object which is a container for its children. That much I had working. The part that had escaped me was that you are also required to override a property and a method from FrameworkElement. The property is VisualChildrenCount. It needs a getter and it should just return the count value from the VisualCollection. The method is GetVisualChild and it just returns the child with the specified index from the VisualCollection.

Unfortunately these aren’t abstract so the compiler won’t catch the issue and tell you that you must override them, but if you don’t, then the framework never becomes aware of your child objects and the result is that nothing is displayed.

Once over that hurdle, everything works great. The final program (equivalent to the one above) looks like this:

[STAThread]
public static void Main()
{
    var app = new Application();
    var window = new Window();
    var host = new MyVisualHost();

    window.Content = host;
    app.Run(window);
}

public class MyVisualHost : FrameworkElement
{
    private VisualCollection children;
        
    public MyVisualHost()
    {
        children = new VisualCollection(this);

        var visual = new DrawingVisual();
        children.Add(visual);

        using (var dc = visual.RenderOpen())
        {
            dc.DrawLine(new Pen(Brushes.Black, 1), new Point(0, 0), new Point(400, 400));
            dc.DrawLine(new Pen(Brushes.Black, 1), new Point(0, 400), new Point(400, 0));
        }
    }

    protected override int VisualChildrenCount
    {
        get { return children.Count; }
    }

    protected override Visual GetVisualChild(int index)
    {
        if (index < 0 || index >= children.Count)
        {
            throw new ArgumentOutOfRangeException();
        }

        return children[index];
    }
}

As you can see, the structure of the program is very similar to the first sample except that instead of creating a canvas and then adding child line objects to that canvas, I create a new “MyVisualHost” object which in its constructor creates a DrawingVisual, uses that object to get a DrawingContext and then calls methods on the DrawingContext to draw lines. Everything but the last two methods is fairly easy to find in samples on the net, but if you omit them, your program will compile—it just won’t work. Lovely.

- Danny

EF FAQ Relocation and Overhaul

Daniel Simmons - MSFT — Mon, 08 Nov 2010 22:21:53 GMT

It has been quite some time since I last updated the EF Frequently Asked Questions which makes them dramatically less useful to everyone. At its core, I think the reason for this delay is not just that I have been busy doing other things but rather that the way the FAQ was setup and posted made it difficult for others to help keep it up to date. To have any hope of continuing to be a great and useful resource in the long run, I believe this needs to be a community effort rather than one run by a single individual or by the EF team.

In an attempt to improve matters, I’ve spent some time over the last few weeks relocating the FAQ to http://www.ef-faq.org (a domain that I registered personally, not something officially owned or maintained by Microsoft) and changing its source format into a series of text files with simple wiki-like markup maintained under mercurial-based source control at the codeplex project http://effaq.codeplex.com/. In addition, some folks on the EF team have put a lot of effort into adding additional questions and answers and updating things based on EF4 so hopefully the FAQ is in a better place than it has been for a while.

Now my hope is that some of you will take a few minutes to help me out in the following ways:

Look over the FAQ—maybe you will learn something.
Next time you have a question or answer a question, check the FAQ, and if the answer is already there publish a link so that others can more easily find and benefit from it. Every question has a fragment name based on the section and question #. So if you want a link, for instance to Code Generation Question 12.2 (“Is there any way to use public, easily modifiable templates for EF code generation?”), the link would be http://www.ef-faq.org/code-generation.html#12.2
If the question you have is NOT answered in the FAQ, and you think it is something others may also run into. Please, PLEASE, take the time to enlist in the codeplex project, edit the source, make a fork and send a pull request so that we can incorporate your change and push it up to the site. This is designed to be a pretty easy process, and it can really benefit your fellow programmers.

For the curious, here’s a little background about how the new EF FAQ site is put together: After casting about for a bit, I decided that I wanted to have a static site with pretty simple HTML so that it is simple, quick and search-engine friendly. I wanted more formatting than I could get from strict plain text, but I wanted something simpler and more constrained/uniform than what I could get out of word or something like that. In the end I decided to take the markdownsharp library (thanks Jeff and others), extend it a little bit to add the concept of “sections” and “questions” which get auto-numbered, have some specific HTML markup and help to produce an auto-generated index of sections for navigation. I also wrote a simple program to wrap the generated content HTML in a high-level template so that every page gets a header, footer and navigation bar and to manage the workflow of turning the various source files into all the HTML for the site. You can find the source for my “faqbldr” app as part of the codeproject as well, and soon I hope to push my extensibility hooks for markdownsharp back into its repository so that I can remove its source from the codeplex project and just check in a binary nuget-style.

By the way, if you are an HTML and CSS guru with a strong stomach, you might want to take a look at template.html in the codeplex project where I have created a horrendous mess that no doubt violates every rule of good HTML design but somehow seems to have landed in a place where the site works and looks OK as long as I don’t touch it any more (If I do I’m pretty sure the whole house of cards will come tumbling down). Someone who wasn’t such an awful html hacker like me could no doubt make it cleaaner, nicer looking and faster.

One last comment, while the FAQ has originally been produced by members of the EF team (and likely some EF team members will continue to contribute to it), it is definitely my intention that this not just be a Microsoft effort. This isn’t official documentation. Ideally it should be a helpful resource that contains whatever info you need to help get your tasks programming with the EF done (or more frequently pointers to where to find the more complete details on what you need). This means that if you have negative comments about the EF in some area or positive comments about some other product that you would use in conjunction with the EF or instead of it in some scenario, those belong here as well—as long as they are constructive and help achieve the goal of making people successful at building great software. Happily Craig Stuntz has agreed to start contributing to the project, and it’s my hope that we’ll have others join in soon.

- Danny

P.S. If you visit the site in the next day or two, you will find that the bing search box in the upper left isn’t fully working yet because the site is so new and bing hasn’t indexed it, but that should fix itself in the next few days.

EF Query Tuning

Daniel Simmons - MSFT — Fri, 23 Jul 2010 00:31:23 GMT

Continuing on the topic of application performance tuning, let’s turn our attention specifically to the EF and take a look at some of what’s going on under the covers with EF queries.

Question #1: Where is the hot path?

The first thing to think about in broad terms is where is the time really going. Often it’s relatively easy to introduce a few counters or make some other small hack to find the worst offender in your application. My first bit of advice to you on this front is: Don’t be afraid to make temporary changes to your code. You can sometimes get great results by making a fresh enlistment in your source control system, checking out a bunch of files and hacking on the code in ways that you would never ship but which help to isolate things so you can figure out what’s really going on.

My next suggestion is: Get a profiler and figure out how to use it. Visual studio has a great profiler if you have a sufficiently advanced version, and that’s what I use, but there are other products on the market. There are even some profilers specifically aimed at working with applications built on top of the EF. In any case, if you care about perf, a profiler is hugely important. I’m no expert on any of these, but you can find some great resources on the web. Some of the the articles I’ve found useful for working with the VS profiler are: http://msdn.microsoft.com/en-us/magazine/cc337887.aspx and http://www.wintellect.com/CS/blogs/jrobbins/archive/2009/10/19/vs-2010-beta-2-sampling-and-instrumentation-profiling-in-depth-first-look.aspx.

However you decide to measure things, the first level of investigation which I’d recommend when you are looking at an EF-based application is to determine where your biggest bottleneck is:

your application
EF one-time startup
EF query code
network transmission
or the database

If you can figure this out, then you can much more quickly home in on where to investigate further and what experiments you might want to try to improve performance.

What happens during the execution of an EF query?

In order to get to the next level of your investigation, it can help a lot to have a general idea of what all is happening and in what order when you execute an EF query. To that end, here’s a rough list:

Startup: When you first create an ObjectContext, it will load metadata describing your model. The first time a context is created in an app domain, that metadata is loaded either from resources in your assemblies or from files on disk. On subsequent context creations the data is loaded from an in-memory cache making it much faster.
Startup: The first time you execute a query on the first ObjectContext instance for an app-domain, the EF will generate an in-memory structure (called the query views) which it uses for query translation. For every query after that, it will simply retrieve the views from an in-memory cache. This view generation step can be very time consuming—especially for large, complex models.
Query: Next the query is translated from LINQ or ESQL into your backend database command (TSQL in the case of SQL Server) and the results assembly plan is created.
Query: The connection is opened if it isn’t already. By default the EF will open the connection before each operation and close it when the operation is complete. If you open the connection manually, though, the EF will leave it alone so it can be kept open if you are performing multiple operations in a row, and this can sometimes be a significant perf win.
Query: If a TransactionScope is active, then the connection is enlisted in the transaction. The effect of this depends on the ado.net provider—for sql server 2008 this is usually cheap, but for earlier versions of sql server if the connection has been previously opened and closed and was just opened again all within the same transaction (or for any provider if more than one transaction participant is involved), then the transaction is promoted to a distributed transaction which can be very expensive.
Network: Send query to the database. Typically this part of the process is inexpensive because the amount of data required to send the query to the database is usually small. Network time is more of a factor when results come back later.
Database: Execute the query. What this entails obviously depends on the complexity of the query, the indexes that are configured and the particular database server involved among other things. Most modern databases have their own fairly sophisticated query optimization system as well as execution plan caches and other mechanisms to help here. In the case of sql server, the complexity of the query being executed is only one of the tools available for tuning, indexes and statistics can be added to the database and even plan guides can be used to give the database hints about how to execute the query efficiently.
Network: Return results from the database. This can become a significant factor if your query returns extra data which isn’t actually needed by the app. In addition, things like eager loading can help perf by reducing the number of network round trips, but they can also hurt performance by causing redundant data to be returned since the way entities from different tables are returned is through the use of a join which often causes some data to be repeated—this repeated data is accounted for during the result assembly process so you end up with accurate results, but there is still overhead in the data returned over the network.
Query: Check the ObjectStateManager by key to see if the entity in question is already present (unless MergeOption is NoTracking).
Startup: The first time a particular entity type / query result shape is returned, dynamic code is generated which extracts the relevant data from the DataReader, constructs the object and initializes it with data (called materialization).
Query: Objects are actually materialized.
Query: Objects are added to the ObjectStateManager (unless MergeOption is NoTracking).
Query: Relationships between entities are fixed up.
Application: Results are processed potentially causing additional queries (especially through lazy loading, etc.)

OK. Now you know what goes on under the covers, and you’ve found something that seems to be taking more time than necessary. That brings us to…

What can be done to improve EF query perf?

Here’s a general rule-of-thumb checklist for EF query perf tuning in the order I would tend to investigate (since this is usually the order of highest bang-for-the-buck).

DANGER! DANGER! DANGER! If you read my post about perf tuning philosophy, all kinds of bells should be going off in your head about now. You should be saying something like, “I thought perf was a science and I should measure what exactly is the issue, create my own prioritization and then do a series of carefully controlled experiments.” Of course all those things are true. You SHOULD measure the reality in your particular situation. That said, I get asked these kinds of questions quite often, and some of these issues show up frequently enough that it if it’s not too expensive to try one of these fixes, you might just try them and measure the results—this could be cheaper to try than deep profiling or the like.

Pre-compiled views. Usually a very easy to implement change is to turn on pre-compiled views. This will move the time of generating views from the first query in your app-domain to design time.
Avoid distributed transactions. If your app is triggering distributed transactions, getting rid of that can be a MAJOR perf win. So even though this can be more involved, it’s worth looking for. If you are using SQL Server 2000 or 2005, this can be caused just by opening and closing the connection more than once inside the same transaction, and if that’s the case the fix is as simple as explicitly opening the connection before the first operation inside the transaction and then either explicitly closing it when the transaction is done or letting it be closed when the context is disposed.
Avoid extra connection opens. Even if you aren’t getting distributed transactions, if you have a number of operations all in a row, just opening the connection can be expensive. As with the best case of distributed transactions, it can be easily addressed by explicitly opening the connection before the first operation.
Reduce database round trips. Eager loading through the Include method can cause graphs of related entities to be retrieved in a single round trip rather than several. This can be an especially big win if you are using lazy loading to retrieve a graph of related entities across a 1-many relationship like a set of customers and their orders. With lazy loading this will take one round trip for the customers + one round trip for each order, while lazy loading could return the entire set in a single round trip. Be careful, though, to balance the cost of round trips against the cost of extra network traffic and query complexity. If you have a large graph of related entities (across several relationships), using eager loading to retrieve the whole thing may not be the most efficient. Often the best answer is a combination with some eager loading and some lazy loading or some other combination. The only way to tell is to try things and measure.
Retrieve less data. Often this is the first thing your DBA will tell you: Quit retrieving data that you don’t actually use in your app. If you only want to display a list of customers and their zip codes, then retrieving the entire customer object might be a bad idea. On the other hand, it may be that querying for the customer object up front will allow you to avoid later queries for the rest of the data, so it’s a balance and you have to try things and measure.
Reduce post DB-query client processing costs. Often the simplest way to do this is to set the MergeOption of the query to NoTracking. When you do this the EF will no longer check the ObjectStateManager to see if the entity is already present there, and after the objects are constructed, it will not add the entity to the ObjectStateManager. This can often produce a significant perf improvement, but you have to keep in mind the side effects this will cause since entity change tracking and identity resolution will no longer happen, etc.
Reduce pre DB-query client processing costs. If you only execute this particular query once, then there’s really not much you can do, but if you execute the same query multiple times and you use ESQL, then the EF will automatically cache the translated query. For LINQ, though, the EF doesn’t automatically cache the query which is where LINQ compiled queries come in. They allow you to explicitly do the translation ahead of time, hold onto a token representing the query and then use that token to execute it each time.
Reduce complexity of the DB query. You may find opportunities to do this by adjusting the shape of the LINQ or ESQL that you write in your application. More often, though, you should look for ways to adjust the model shape. Deep inheritance hierarchies with lots of relationships, and especially if they use TPC-style inheritance tend to create complex queries. If you can reduce the depth of your inheritance hierarchy, write queries that return fewer different types in a single query, or switch from TPC to TPH or something like that, the result will tend to be less complex. While the EF often does a great job in the database queries that it generates, occasionally it will produce a query which is less than ideal. If you determine this is your bottleneck and you know how to hand-write a better query, then you can consider writing the query by hand as a stored procedure and calling that in order to take complete control over the query rather than letting the EF generate it.

- Danny

Application Performance Tuning Philosophy

Daniel Simmons - MSFT — Fri, 23 Jul 2010 00:01:35 GMT

I’m in the process of embarking on some investigation and discussion with folks about performance tuning EF apps (in particular I’m going to dive into a critical area or two on my D3 project), and the more I thought about this, the more convinced I became that there is a fair amount of background that needs to be set out before diving into the heart of the matter. At least for me, there are a number of things about perf tuning that just weren’t obvious when I first began considering it. To be successful at this endeavor, just as with any other, we need to really know what we’re about and something of the underlying ideas that lead to a workable approach. So here’s my off the cuff description of the way I think about performance.

There Are Two Ways to Improve Performance

In my opinion, there are basically two different ways to actually improve the performance of an application (improving the performance of a framework or a server overlaps a lot but each has some unique challenges that I’m not going to consider today). The first technique is all about your application design, and the second is a matter of tuning individual operations.

Without a doubt the biggest wins come from making good design decisions up front. What I’m talking about here are coarse-grained things that affect the entire way your app works. Getting these right is generally either a measure of understanding your problem space well through a lot of experience, being lucky or writing your application, discovering what you got wrong and then completely re-writing large portions of it. An example of this kind of issue would be the difference between a client-server application where the client is essentially just a dumb terminal that requires every single key-stroke to travel to the server to be handled and one where you have a smarter client that does a fair amount of display, validation and other logic on the client giving great responsiveness and then just sends larger batch operations to the server. If you happen to choose the first approach, you might discover huge perf problems, and if that happens it will be a major issue to redesign your application to the second approach.

While the first approach is largely about forecasting where you are going and making sure you create an architecture that leads you down the right path for your largest decisions, the second approach looks at how to improve much finer-grained individual scenarios. The techniques you use for this approach are much different than those used for the first one. In fact, often if you try to think carefully and optimize your fine grained operations up front in the way you do for the overall architecture, you will introduce a ton of complexity which makes your application much harder to develop, test and maintain and even after all that experience shows that most often you will not have actually improved the performance in a way that has a meaningful positive effect on the end user experience—you may actually have made it worse.

So while design is very important, tuning is also critical to get a great experience from your app.

Tuning is Experimental Science

While lots of what we do as software developers is very different from experimental science (resembling something closer to craftsmanship involving a mix of art & science, engineering & design), performance tuning done right is ALL SCIENCE. We need to have an understanding of the underlying concepts, to have some intuition about what will and will not work, but just like an experimental scientist we have to train ourselves to thoroughly harness that intuition and slave it to a process that will lead us to results we can prove and depend on.

I like to tell developers on my team, “If you think you know what will improve the perf, you are almost certainly wrong.” As developers we often have that feeling that we just know what is wrong with the code, and we want desperately to dive in and implement our fix. In practice, though, often we’re wrong. Oh sure, we will improve the performance of some part of the code, but unless we’ve done a very careful investigation first we may have made an improvement in an area of the code that is already so fast it doesn’t matter while ignoring or maybe even making worse the hot path (the thing that is the current slowest part which has the most effect on the user experience). Further, we often have a tendency to take a shotgun approach and make a lot of changes before measuring again with the result that if the perf gets better, worse or stays approximately the same we have no way to know what change we made is responsible.

The only real answer is to make a baseline measurement, look at the data, create a hypothesis and then make the smallest possible change to test that hypothesis. Further, it’s super important that we carefully prioritize our experiments. We want to go after the lowest hanging fruit. That is to tackle those problems which will have the biggest impact on the end user experience for the lowest possible cost in terms of development cost and final code complexity. Obviously this means we have a tendency to look for the hot path through the application as the first step, but we also have to apply a complexity filter since it may be that the hot path is something that would be very hard to change, while there may be 2 or 3 other things that are almost as significant as the hot path but much easier to change. Just always remember: measure, measure, measure. When we’re done we want to KNOW the app will be faster and that we’ve gotten there as quickly and easily as possible.

Tuning Improvements Tend to be the Enemy of Clean Code

To add further strength to the comment that we want to make the cheapest possible changes, I’ll point out that while top-level architecture design style perf improvements can often make your application even cleaner and easier to understand and work with at the same time that they improve performance, the kinds of changes you make when tuning tend to have the opposite effect. So you want to start with the simplest, cleanest possible implementation for every part of your app. Remember: You never know in advance where the tuning will really be needed so wait until you can measure and find out where you have to introduce the hacks rather than putting them in up front.

One example of this principle comes to mind from when I used to work on Excel a number of years ago. Excel is a product that has been around for a LONG time in software years, and it’s been through a lot of changes. When I worked on the product it had already been around a long time so there was a lot of legacy. One aspect of this legacy was a perf optimization they had put into the product back when it was being used on MUCH slower systems than we have today. There was this code which was used to interpret the core data structures representing cells in the spreadsheet. The structure itself was highly optimized to make cells take as little memory as possible with the result that the bytes in memory could have one of several meanings so the code would look at a few bits up front to figure out how to interpret the rest. Not only was this data structure very complicated, but to make matters worse the code for interpreting the structure was repeated all over the code base. They went to great lengths to make sure it stayed the same everywhere, but nevertheless it was repeated. They had done this for very good reasons. At the time it was first done that way, the cost of calling to a single central routine to interpret the structure in all the many places that needed to interact with it was just too high. If they had to make that jump every time and then return, recalcs would take forever. The problem was that by the time I was working on the project, there really wasn’t anyone left on the team who knew every place that might have a copy of that code, and when we wanted to introduce a change to the data structure, the cost was astronomical to hunt down every case and fix it and then to test the product like maniacs to make sure we hadn’t missed anything. At about the same time, a related small project rewrote the core spreadsheet engine from scratch in a small component designed for plugging into web pages. Since this was written from scratch in an era where computers had sped up considerably, this little component was able to centralize its code for the same task with the result that it had more features in some areas than the big brother excel product for several years until excel could finally be updated.

The moral of the story? Start clean, measure carefully, tune in biggest-bang-for-the-buck order. Stop tuning as soon as you can. (Which implies that you need to know just how fast is fast enough.)

- Danny

D3: Implementing the WCF Service

Daniel Simmons - MSFT — Thu, 22 Jul 2010 23:08:50 GMT

D3 release 0.1322 is now available. The work that went into this release is all about building out the WCF service which D3 will use as the interface between our mid-tier and the client. I encountered a grab bag of issues along the way, so here’s a list of tips & tricks as well as random thoughts on the topic of building a WCF service with the EF.

Authentication

Somewhere along the way between the last D3 release and this one I had knee surgery. My recovery was actually pretty quick, but I did end up spending most of a week sitting at home with my knee in the air and a computer on my lap. I spent a frighteningly large amount of time trying to read up on ways to do authentication using WCF for the D3 project. One of the best blog posts I found was this:

http://blogs.infosupport.com/blogs/alexb/archive/2009/10/02/silverlight-3-securing-your-wcf-service-with-a-custom-username-and-password-authentication-mechanism.aspx

The overall approach really seemed to match my general goals in the sense that I want the service to be essentially secure and WCF to handle that security in a relatively transparent way, but I also want to have my own username and password tracking in my EF model (I don’t want to create windows accounts on a domain controller for every user of D3 or something like that). The issue that drove me nuts is the fact that this approach absolutely requires SSL and at the time I was working on it there really wasn’t a good way to get SSL running (even with just a test cert) inside of VS because Cassini (the mini-webserver built into VS) doesn’t support it.

Since then the IISExpress project seems to have gotten off the ground, and it may well be the right long-term answer, but after much banging of my head against this particular wall, I decided that for the current interim release I would completely fake all of this in a hacky way that would leave us setup for the right thing but be clear to implement now. What I did was to add a username and password parameter to every method on the service. We’ll return to this in a later release and make things more secure and nicer to use.

Service Contract Design

As I have written in other places (such as this short series of MSDN Magazine Articles: Anti-Patterns To Avoid In N-Tier Applications, N-Tier Application Patterns & N-Tier Apps and the Entity Framework: Building N-Tier Apps with EF4), there are lots of patterns/approaches that can and should be considered when designing a service. While especially typical intranet line-of-business applications will benefit from the developer productivity and simplicity of an approach like self-tracking entities or RIA services, in this case my goal is to create a game that will run over the internet which means that I’d like to take more control over the contract.

That led me to the decision that D3’s service should be fairly strictly SOA—at least for the most commonly used methods (issuing commands from the client and retrieving events from the server). For these two core commands, I decided that the service should neither accept entities as parameters nor return them as results. We just want the simplest, most basic possible interface, and the result is the following two method signatures:

[OperationContract, FaultContract(typeof(CommandFault)), ApplyProxyResolver]
void ExecuteCommand(string userName, string password, int playerId, string command);

[OperationContract]
Tuple<IEnumerable<string>, int> GetEventMessages(string userName, string password, 
                                                 int playerId, int lastSeenEventId);

Ignoring the authentication parameters (which as discussed above should go away in a later release), the first method takes a player ID and a string representing the command that the player wants to execute—typically this will be something like “go north”, “get fancy sword” or “attack purple snorklewacker”. You’ll notice that this method does not return any results—the only interesting result is if the command is invalid in which case the service will return a CommandFault with the error message.

The way the service returns the results of commands as well as the observable actions of others in the game is through the GetEventMessages method. It also takes a player ID to determine the perspective of the player from which the events are observed, and it takes the ID of the last event which was returned by a previous call to GetEventMessages. This method returns a tuple which contains an enumerable of strings which are messages to display on the client plus the ID of the last event which produced a member of that last event of messages.

The idea is that the client will have two asynchronous UI interaction streams going on. First, at any time the player can type a command and send it to the server, and assuming the command was valid the server will process the command using its business logic and create events representing the result of that command. The second thing going on is that the client will periodically poll the server for new event messages observed by the player. The events which produce these messages are both those created as a part of processing this player’s commands as well as events caused by everyone else in the game. Each time the client calls the GetEventMessages method, it will get back any messages that it hasn’t yet seen plus an ID used in the next call to the method to indicate the “high-water mark” of what it has already seen.

Obviously the GetEventMessages method will be the most frequently called top-level method in the entire application (every client will poll it frequently in order to maintain its UI). So it will be very perf sensitive. In subsequent blog posts, I’ll take a look at the topic of tracking and tuning the perf of this method.

Unit Testing vs. Integration Testing

If you’ve been reading my blog, you probably know by now that I’m a huge fan of unit testing, and I’ve been putting a lot of work into appropriately unit testing D3 as we go. This has led to finding and fixing a lot of bugs early on, increasing my confidence in my ability to refactor the project frequently without much risk of introducing new subtle bugs and generally a better overall structure for the project. Unit testing, though, isn’t everything. When unit testing we try to isolate small parts of the application so we can focus what we test on a very small piece to make it fast and make sure we know what is responsible when a test fails (so it’s easy to fix). Sometimes, though, you’ve got to move to a coarser granularity and make sure that you have some tests to verify that all the units go together properly.

This lesson came home to me in a big way when I decided to write a small integration test that would simulate the most common client / service interactions. I had been unit testing various parts including directly calling the service method implementations, etc., but I hadn’t yet put it all together and run it through WCF. The code I wanted to get working wasn’t exactly huge or hugely complicated. It looked like this:

[TestMethod]
public void KickoffScenario_Works()
{
    using (var svc = new PlayerServiceClient(
        new BasicHttpBinding(), 
        new EndpointAddress("http://localhost:9999/PlayerService")))
    {
        svc.CreateUser("simmdan", "password");

        var id = svc.CreatePlayer("simmdan", "password", "joe");
        svc.GetPlayers("simmdan", "password").Single(p => p.Id == id);
        var logOnEventId = svc.LogOn("simmdan", "password", id);

        svc.ExecuteCommand("simmdan", "password", id, "get item1");
        var eventMessages = svc.GetEventMessages("simmdan", "password", id, logOnEventId);

        var message = eventMessages.Item1.Single();
        Assert.AreEqual("You pick up the Item1.\n", message);
    }
}

As I discovered, though, there were a surprising number of issues that came up including some things that were quick and easy to find and some others that were pretty subtle and took a fair amount of hunting. Along the way, one thing I found very helpful was this post about turning on WCF tracing: http://www.c-sharpcorner.com/UploadFile/shivprasadk/6876788978904302009095012AM/68767889789.aspx Following the directions on that site I added the following XML to the app.config file in D3.Web.Tests:

<system.diagnostics>
  <sources>
    <source name="System.ServiceModel" switchValue="Information, ActivityTracing">
      <listeners>
        <add name="log" type="System.Diagnostics.XmlWriterTraceListener" initializeData="c:\Traces.svclog" />
      </listeners>
    </source>
  </sources>
</system.diagnostics>

Common Issues When Using the EF & WCF Together

So all of that leads me to the following short list of the most common issues I encounter in my own code or when talking with others specific to using the EF and WCF together.

Not understanding the basic design patterns involved. This is by far and away the most common issue. For this I recommend the magazine articles I referred to in the Service Contract Design section above.

(Side Note: It used to be that at least as common as this one was the issue of trying to figure out how EF’s disconnected APIs work—even if you did understand the basic patterns, it could be very hard to implement them correctly with the EF. Happily, since EF4 shipped the APIs are much easier to use, so this is not so common any more.)
Disposing the context before enumerating a query which is returned from the service method. This one will get you almost every time. The general recommended pattern is that each service method should create a context instance in a using block so that it (and its underlying connection) will be disposed properly as soon as possible to free up precious server resources. If you have a method that returns an enumerable of some type which comes from a query to the database, though, you have to make sure that you fully enumerate that query before the context is disposed. If you just return the query itself, in a regular CLR method call where the context lifetime is managed some other way, that query would be cast to IEnumerable and the CLR would lazily evaluate it when the results were actually needed. This is simple and efficient, but in a case where the context is being disposed before the method returns, though, the query results won’t be enumerated until the WCF service’s serializer attempts to process things after the method returns, and in that case the context will first be disposed and then later the query will be enumerated, and it will throw an exception because its connection to the database will be gone. The fix? Just call .ToList() on your query before returning.
Using POCO without thinking through WCF’s requirements for objects it serializes. If you have a WCF service which accepts entities as parameters or returns them as results and you use POCO entities (maybe by using the POCO T4 template or maybe just writing them by hand), then you have to think about WCF’s requirements for its objects because your POCO objects can be almost anything and they may well not be setup properly to work with WCF. The three ways this comes up most commonly for me are if you have dynamic proxies (either just for lazy loading or for full change tracking), if you have inheritance in your model which the DataContract serializer is unaware of, or if you want to return entire graphs of entities which have cycles. You can read about the first two issues (and how to address them) here: http://blogs.msdn.com/b/adonet/archive/2010/01/05/poco-proxies-part-2-serializing-poco-proxies.aspx the key concepts are the DataContractResolver and KnownTypes. For the last issue, the answer is in the IsReference=true parameter to the DataContract attribute. This is a feature that was added in .Net 3.5sp1 which makes the DataContract serializer smarter about graphs of related objects. You can read more about that here: http://zamd.net/2008/05/20/datacontract-serializer-and-isreference-property/

One thing to watch out for is that if you have POCO entities with no DataContract attributes, the DataContract serializer will handle them fine up to a point, but if you start having more advanced scenarios like graphs of related entities or dynamic proxies, then the default conventions which the DataContract serializer uses will no longer work, and in that case the answer is to add the attributes. If you are using the POCO template to generate your entities, for instance, it’s pretty easy to hack these into the T4 template which is what I did for D3. If you are using the default codegen, then all of these things are taken care of for you automatically, but of course there are other advantages to POCO—it’s all a matter of tradeoffs.

OK. I suppose that’s enough for now. Happy WCF-service building.

- Danny

EntityRoots – An EF Extensibility Exercise

Daniel Simmons - MSFT — Mon, 12 Jul 2010 06:21:00 GMT

A little while back I was working on a project for one of the other teams in the Data and Modeling Group where we were building an EDM which was not huge but nevertheless had a number of entity sets, and I was trying various strategies to make the model more usable and easier to understand. It occurred to me that one thing which would help would be to remove some of the entity sets which would normally be exposed as ObjectSet properties on the context since it was very unusual to want to reference those sets directly at the top level—normally all queries and other operations would be initiated using other sets/types. Those types effectively were the “roots” of interaction with the model. This concept is certainly weaker but somewhat reminiscent of aggregate roots as defined by practitioners of domain driven design.

This is not all that different from some samples we have put together before. Sanjay, for instance, wrote a post quite some time ago showing how to enable an extension which would cause code generation to add an arbitrary attribute to classes or properties. The core capability of adding annotations to the conceptual model metadata as described in that post is still the same. Since EF4 has shipped, however, we have some additional options which make the overall task a whole lot easier than it was before. First off, this seemed like a great opportunity to play a bit with the EDM Designer Extension Starter Kit which I had not previously given the time that it deserves. Of course this is a perfect application for T4 templates which make it much easier to customize code generation (I’ll not give a link for that since we have talked about it a lot in the past—you can easily find more info with a quick search), and the VS2010 VSIX extension format makes it easy to wrap all of this up in a neat package and manage installation and versioning of it.

The final result of this effort is a VSIX package I call EntityRoots which adds an extension to the designer making it possible to set in the properties for an entity type whether or not it is a root which will then show up as an annotation in the CSDL for that entity type as well as a new T4 code generation item template which looks at that annotation in order to decide whether or not to output an ObjectSet property for that type. In addition to just downloading and using the VSIX package, if you want you can download the solution I use to build it.

For the designer extension, the exercise was mostly just a matter of removing parts of the starter sample that do other things than just adding a property to entity types and renaming things, so I won’t spend much time describing how to do it. You can follow the example and easily adapt it to whatever other scenario you might have. Happily the starter kit also automatically creates a VSIX project to install the extension without much effort. The more interesting part (although also pretty easy) is creating the T4 template, turning it into a VS item template and adding it to the VSIX package. To do that, my first step was to decide that I wanted to base my template on the POCO template. You could, of course, do the same thing with the default code gen template, the self-tracking entities template or any other EF code gen template you might have.

I created a small sample project, added the POCO template, and then just added a few lines to the portion of the template which outputs the ObjectSet properties:

        var entityRootsExist = container.BaseEntitySets.OfType<EntitySet>()
            .Where(s => s.ElementType.MetadataProperties
                .Contains("http://blogs.msdn.com/dsimmons/schemas/EntityRoot:EntityRoot"))
            .Any();

        foreach (EntitySet entitySet in container.BaseEntitySets.OfType<EntitySet>())
        {
            if (!entityRootsExist || (entitySet.ElementType.MetadataProperties
                .Contains("http://blogs.msdn.com/dsimmons/schemas/EntityRoot:EntityRoot") &&
                ((XElement) entitySet.ElementType
                .MetadataProperties["http://blogs.msdn.com/dsimmons/schemas/EntityRoot:EntityRoot"].Value).Value == "True"))
            {
#>

    <#=Accessibility.ForReadOnlyProperty(entitySet)#> ObjectSet<<#=code.Escape(entitySet.ElementType)#>> <#=code.Escape(entitySet)#>
    {
        get { return <#=code.FieldName(entitySet) #>  
                    ?? (<#=code.FieldName(entitySet)#> = CreateObjectSet<<#=code.Escape(entitySet.ElementType)#>>("<#=entitySet.Name#>")); }
    }
    private ObjectSet<<#=code.Escape(entitySet.ElementType)#>> <#=code.FieldName(entitySet)#>;
<#

            }
        }

Essentially we just check to see if any of the entity sets have the annotation. If so, then we only output an ObjectSet if that set has the annotation and it is set to true.

I then put the modified templates into a project which creates a VS item template (which is just a zip file with the T4 template files plus a vstemplate file giving information about how the templates should be added to the project). In order to make this item template show up in the EF designer’s add code generation item dialog, the vstemplate file’s name must begin with “ADONETArtifactGenerator_”. You can read more about VS item templates here.

- Danny

Generating Business Logic “Hooks” for EF4 Entities

Daniel Simmons - MSFT — Thu, 22 Apr 2010 22:30:18 GMT

Once again a question in the EF MSDN Forum has prompted a blog post where I can give a more complete answer. If I understand things correctly, the person asking the question wanted a simple way to add business logic hooks to their entities which would be called whenever they did SaveChanges. In EF4 we made available the basic building blocks for this kind of thing, but unfortunately it’s not quite as easy and discoverable as it ought to be.

Since I’m stuck sitting at home with my knee elevated as I recover from minor knee surgery, I thought I’d take a little time today to create something which should make this much easier. You can find the result here. It’s a visual studio extension file (VSIX) which installs a new T4 item template called EntityHooks. If you right click on the EF designer background and choose “Add Code Generation Item…”, you should see a new EF EntityHooks Code Generator in the list of templates. If you choose that option, it will add a new TT file to your project that implements the hooks.

It’s important to realize that this complements whatever other code generation you have going on—it doesn’t replace it. So if you have already added a code generation item, this will just add another one which generates partial classes that add functionality to your existing partial classes. Unfortunately, if this is the first code gen artifact you have added to your project, the designer doesn’t realize it doesn’t generate the full entities so it turns off the default codegen, and you will either need to add another codegen artifact or in the properties for your EDMX file set the codegen strategy property from None back to default.

Once this is done you can write your own partial class for any of your entities and implement any or all of the OnAdded, OnModified and OnDeleted partial methods. These methods will be called whenever an instance of that entity type is in the appropriate state at the time SaveChanges is called, and the call will happen before the framework does any other part of SaveChanges. So, if your method throws an exception, no save will happen. This also means that you can modify other entities if you want, or cancel an operation on the particular entity by calling AcceptChanges on its ObjectStateEntry (which is passed as an argument to the partial method).

Here’s a simple example I used to test out the extension. It’s a hook which takes Customer entities marked for deletion and instead of deleting them just prepends “D:” to the front of the company name as a signal that they are deleted but you still want to keep them around for historical info or something like that:

public partial class Customer
{
    partial void OnDeleted(System.Data.Objects.ObjectStateEntry entry)
    {
        entry.ChangeState(EntityState.Modified);
        CompanyName = "D:" + CompanyName;
    }
}

With this class in place, any attempt to delete a Customer will instead become just a modification.

The extension should work with whatever code generation strategy (default, self-tracking, poco, any custom strategy you create) as long as you generate partial classes both for your entities and your context. You will notice, though, that it introduces a dependency on the EF since the partial methods receive an ObjectStateEntry argument. if you want this kind of thing with a truly POCO experience, then you would need to create some other type to abstract away this dependency. For this exercise I just took the shortcut of passing a state entry because it makes it easy to change the state of the entity or perform other actions in a general way.

The way I implemented this was to output an IEntityHooks interface which declares three methods OnAddedHook, OnModifiedHook and OnDeletedHook, plus a partial class for each entity which implements the interface by having those methods call the corresponding OnAdded, OnModified or OnDeleted partial method which the partial class also declares (can’t have a partial method implement an interface method because the compiler will completely optimize away the partial method if no one implements it), and finally a partial class for the context which overrides the SaveChanges virtual method. The new SaveChanges method retrieves entries from the ObjectStateManager which represent entities in the various states, casts the entities to the interface and then executes the appropriate hook.

If you want more details, install the VSIX file linked above, add the hook template to one of your projects and have a look. Pretty straight-forward really. Let me know if you have any questions.

- Danny

D3: Test Fixture & Using a Fake Context While Testing a WCF Service

Daniel Simmons - MSFT — Wed, 14 Apr 2010 12:10:00 GMT

First off, I’ll point out that D3 Release 0.1012 is now available on code gallery. In addition to a foray into a custom, attribute-based text parser (which I’ll not discuss in detail here unless someone really wants me to since it has little bearing on the EF), this release includes the initial outline of the WCF service which should someday soon make it possible to actually have a working game. In the course of putting that together, I continued to invest in testing improvements which will be the primary focus of this post.

Before going into those details, though, I have to commemorate the 4 year anniversary of DPMud’s first formal check-in. One of the original members on the project, Josh Lindquist, sent me an email recently which includes a copy of the first auto-generated check-in mail for the project which had these headers:

From: Daniel Simmons
Sent: Friday, April 14, 2006 12:00 AM
Subject: [DPMUD] REDMOND\dsimmons: #9: Initial checkin of dpmud.

It’s hard to believe that 4 years later I’m still working on this crazy little project, but I am, and if anything I’m enjoying it more than ever!

Enough nostalgia, now back to EF testability.

Simplifying Test Code with a Test Fixture

As we started filling in more tests for business logic on entities in the D3 model, I found that over and over I had code which setup several entities and relationships between them, put that data into a fake context and then used it to perform some action and verify the results. This code was repetitive and prone to small errors. In addition, because it was a bit of custom code each time, it was harder to look at a bunch of tests to pick out what was the core idea of the tests and what patterns were the same or different between them. So I decided to centralize this code and configure my fake context in such a way that I could easily have a common set of initial data.

The first step was to add a new constructor to the D3DB object which I use to provide a simple IOC container. Previously the only constructor was one which take an ID3Context instance, but now I’ve added a default constructor and a property on D3DB which is a Func used for creating a D3Context instance if needed:

public static Func<ID3Context> CreateAction { private get; set; }

public D3DB() : this(CreateAction != null ? CreateAction() : null) { }

That way I could add a constructor to each of my test classes with the line:

D3DB.CreateAction = () => new FakeD3Context();

and then the body of any tests needing to interact with the context would just be surrounded by:

using (new D3DB())
{
    ...
}

It also means that I can add a static constructor to the real D3Context implementation which sets D3DB up to create a real context any time it isn’t overridden like in the test code above.

static D3Context()
{
    D3DB.CreateAction = () => new D3Context();
}

The reason we have to configure this in a static constructor on the real D3Context rather than just in D3DB itself as a default value for the property is that D3DB is part of D3.Model so that entities can have a dependency on it, but that means it can’t have any dependency on any implementation of ID3Context. So we define the basic mechanism in D3DB, setup the default configuration in D3Context and then override that configuration to point to a fake in the tests.

Once I had this mechanism in place, my fake context creation was centralized and I could expand on it by creating a test fixture with a standard set of test data. I did that by adding a factory method to FakeD3Context:

public static FakeD3Context CreateWithTestData()
{
    Room room1, room2;
    Exit exit1, exit2;
    Actor actor1, actor2;
    Player player1;
    Item item1, item2;
    User user1;

    var ctx = new FakeD3Context
    {
        Rooms = new FakeD3ObjectSet<Room>(
            (room1 = new Room { Id = 1, Name = "Room1", Exits = new List<Exit>() }),
            (room2 = new Room { Id = 2, Name = "Room2", Exits = new List<Exit>() })),
        Exits = new FakeD3ObjectSet<Exit>(
            (exit1 = new Exit { Id = 1, Name = "Exit1", SourceRoomId = 1 }),
            (exit2 = new Exit { Id = 2, Name = "Exit2", SourceRoomId = 2 })),
        Actors = new FakeD3ObjectSet<Actor>(
            (actor1 = new Actor { Id = 1, Name = "Actor1" }),
            (actor2 = new Actor { Id = 2, Name = "Actor2" }),
            (player1 = new Player { Id = 3, Name = "Player1" })),
        Items = new FakeD3ObjectSet<Item>(
            (item1 = new Item { Id = 1, Name = "Item1" }),
            (item2 = new Item { Id = 2, Name = "Item2" })),
        Users = new FakeD3ObjectSet<User>(
            (user1 = new User { Id = 1, Name = "User1" }))
    };

    room1.Exits.Add(exit1);
    room2.Exits.Add(exit2);

    exit1.TargetRoom = room2;
    exit2.TargetRoom = room1;

    actor1.Room = room1;
    actor2.Room = room1;

    player1.Room = room1;
    player1.UserId = user1.Id;

    item1.Room = room1;
    item2.Room = room1;

    return ctx;
}

and then I just modified the definition for D3DB.CreateAction set in the constructor for my test classes to call this factory method rather than constructing the fake context directly. Viola! Now each of my tests can automatically get a fake context with a small amount of fresh data available for each run.

Oh yeah, and while I was at it I moved the FakeContext code and related bits into a new D3.TestUtils DLL so I could easily share this code across multiple test projects since I wanted to use it not only for the original purpose of testing the model but also in the new tests I created for the WCF service which takes player commands, parses them and then executes appropriate methods on the player object to carry out the action. Which brings us to the next section…

Using a Fake Context to Test a WCF Service

Given the code above, it’s easy to create a fake context with some test data, call a method in the product that retrieves that context from the thread-local static property D3DB.Current and uses it, and then access the same fake context to verify appropriate results. This is great, but what if I want to write tests for a method that creates its own context instance rather than assuming one has been created in the surrounding environment? Given that the normal pattern for my WCF service methods is to create a context instance at the start of each method and dispose of it at the end so that the service calls can be independent and essentially stateless, this is a particularly relevant question.

Not only do I need to intercept the creation of the context in order to make sure it’s a fake (easily accomplished given the new D3DB default constructor and CreateAction method described above), but I also need to be able to get access to that same context instance at least after the method call has completed so that I can validate the results and maybe even beforehand in order to do some special initialization. For this second trick, I expanded on D3DB’s default constructor so that constructing a new D3DB instance with the default constructor will just re-use any existing context rather than creating a new one or throwing an exception. Since the D3DB objects are disposable, as long as you put them in a using block, constructing a new D3DB when one is already active can only happen in a way that is completely nested so that the inner one will dispose before the outer one. So this process is basically safe, and the overall pattern should be familiar if you have worked with TransactionScope or something of the kind.

The resulting code for the D3DB default constructor looks like this:

public D3DB()
{
    if (current == null)
    {
        if (CreateAction == null)
        {
            throw new InvalidOperationException("Must set a CreateAction if you are going to use the default constructor.");
        }

        Current = CreateAction();
    }
    else
    {
        disposeNeeded = false;
    }
}

And the code for the dispose pattern undergoes a corresponding modification like this:

private bool disposeNeeded = true;

public void Dispose()
{
    if (disposeNeeded)
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }
}

The result of all this is that I can write a WCF service method whose body is surrounded by a using block which creates a new D3DB instance using the default constructor and that method will create the real context when running in the program directly, but I can also create a test method whose body has a using block creating a new D3DB instance with the result that it will create a fake context, give me an opportunity to interact with it if necessary, and then let me call the service method whose D3DB constructor call will see that a context is already available and just use that rather than making another. Since this same context is used for the whole test, I can then also verify that the service call had the intended effect upon the database.

ExpectedExceptionEx

For my last testing trick of the day, let me tell you about a small new addition to my testing arsenal: ExpectedExceptionExAttribute. The motivating scenario for me was that I added a couple new constraints to the database to enforce that actor and user names were unique even though they aren’t part of the primary key for the entity.

Adding those constraints was easy—I just put a couple additional calls to ExecuteStoreCommand in the database creation code on the context, but the more difficult part comes when testing them. it’s easy enough to create two actors with the same name, try to save them both to the database and verify an exception is thrown, but what if that exception was actually the result of some other, subtly different thing and in the end the intended behavior wasn’t being enforced correctly? I really needed a way to verify more than just that an exception was thrown of a particular type—especially since a broad class of failures when saving changes to the database all result in an “UpdateException” being thrown.

Happily, the MsTest library has a great extension point for just this sort of thing. I was able to create a new attribute which I can place on my tests that does more than just verify the exception type, it can also optionally check the inner exception type and/or search for strings within the exception message or inner exception message. So now my test for one of the new constraints looks like this:

[TestMethod, ExpectedExceptionEx(typeof(UpdateException), InnerExceptionType = typeof(SqlException), 
    InnerExceptionMessageContains = "Violation of UNIQUE KEY constraint")]        
public void Users_Name_MustBeUnique()
{
    using (var ctx = Utilities.CreateTestContext())
    {
        ctx.Users.AddObject(new User { Name = "foo" });
        ctx.SaveChanges();

        ctx.Users.AddObject(new User { Name = "foo" });
        ctx.SaveChanges();
    }
}

The entire code to enable this new exception looks like this:

namespace D3.TestUtils
{
    using System;
    using Microsoft.VisualStudio.TestTools.UnitTesting;

    [AttributeUsageAttribute(AttributeTargets.Method, AllowMultiple = false)]
    public sealed class ExpectedExceptionExAttribute : ExpectedExceptionBaseAttribute
    {
        public Type ExceptionType { get; private set; }

        public Type InnerExceptionType { get; set; }
        public string ExceptionMessageContains { get; set; }
        public string InnerExceptionMessageContains { get; set; }

        public ExpectedExceptionExAttribute(Type exceptionType)
        {
            ExceptionType = exceptionType;
        }

        protected override void Verify(Exception exception)
        {
            Assert.IsNotNull(exception);

            RethrowIfAssertException(exception);
            Assert.IsInstanceOfType(exception, ExceptionType);

            if (InnerExceptionType != null)
            {
                Assert.IsInstanceOfType(exception.InnerException, InnerExceptionType);
            }

            if (ExceptionMessageContains != null)
            {
                Assert.IsTrue(
                    exception.Message.Contains(ExceptionMessageContains), 
                    String.Format("ExceptionMessage does not contain '{0}'.  Message value: {1}", 
                                   ExceptionMessageContains, exception.Message));
            }

            if (InnerExceptionMessageContains != null)
            {
                Assert.IsTrue(
                    exception.InnerException.Message.Contains(InnerExceptionMessageContains),
                    String.Format("InnerExceptionMessage does not contain '{0}'.  Message value: {1}", 
                                   InnerExceptionMessageContains, exception.InnerException.Message));
            }
        }
    }
}

That’s it for today. Have fun playing with the EF4 final bits and testing your code as you go. If you have any questions or feedback, as always I’d love to hear it.

- Danny

D3: Testing, TFS and a Teenager

Daniel Simmons - MSFT — Mon, 15 Feb 2010 09:58:00 GMT

It has been a couple months since I’ve put up a new release of D3, but that doesn’t mean I haven’t been working on it. Release 0.0814 is now available. This release is designed to work with VS2010 RC (available starting last week), but it should also work just fine or with very minimal changes with VS2010 beta 2 if that’s what you have.

The Teenager

My 14-year old son, Keith, is developing into quite the programmer. You may have heard the .Net Rocks guys read a letter he wrote them in episode 520 about the new support for complex numbers in .Net 4. Let me assure you that he did that all on his own, and I didn’t even hear about it until he mentioned that he’d gotten a response from them asking for his shirt size so they could send him a .Net Rocks hoodie (which he now wears all the time). You can see a few projects on his website, including the picture to the right which is a computer generated image that he created. Anyway, he has been watching me work on DPMud through each of its versions over the last several years, and he keeps me on the straight and narrow by regularly asking what I’ve done on it lately. One of his long-standing requests has been for us to find a way that he could work on the project with me, so in the time since the last release of D3 not only have I been working to evolve the codebase, but I also installed TFS on a server at home and got things setup so that he can begin contributing. The code in the 0.0814 release is still all from me, but he has begun doing code reviews, and in the next release some of the bits will have originated with him. (What a blast it is to code with my son! Go Keith!)

TFS

Before we dive into a discussion of the changes in this release of D3, let me take a moment to mention what a great experience I’ve had using TFS 2010. The new basic installation is AMAZING. I repurposed an old single-core AMD PC I had sitting around the house, upgraded it’s RAM a little so that it now has 2GB, reformatted it with a clean copy of Win7, and installed TFS 2010 beta2 plus VS2010 beta2, and then followed the instructions in Jason Zander’s blog post and the post from Brian Harry that he links to. Setting things up was super smooth, which is a huge change. The last time I tried to set up TFS for myself I couldn’t even figure out what software or licenses I would need and eventually got disgusted and just decided to use something else for my source control. This time it was as easy as could be, and the final result is a server I can access from home, work or anywhere on the internet, full integration of source control with VS including simple offline support, a build server that does a continuous integration build whenever one of us makes a checkin and even a bug/work item tracking system which we haven’t fully started using yet, but is ready to go. Oh yeah, and not only can I use the built-in client in visual studio, but if I need to check on something from a machine that doesn’t have VS installed I can just connect to the web interface. Now there’s no excuse for not having good source control.

D3 Changes in this Release

OK. So what have I been doing with D3 this time around? Well, as usual I’ve been making general investments in the code including things like refactoring and renaming the projects so that everything follows a three-part naming convention starting with D3, ensuring that every project had source analysis, stylecop and warnings as errors turned on and then correcting any issues which turned up, etc. The really interesting things, though, are that I switched from the standard codegen to POCO classes generated using a T4 template, separated the POCO entities into one project and the context and other database specific bits into another project, and then began porting over some events and actor actions and writing tests for them.

Most of this is pretty straightforward and you can look at the code in the release to see what’s up. I started with one of the POCO templates we’ve release as part of the EF Feature CTP, but because some things have been changing between beta 2 and RC and such, I copied not only the core template but also the EF.Utility.ttinclude file into the project so that I’d be isolated from changes there. Once those things completely settle down, I’ll look at getting ride of the ttinclude file and just using the standard one. On a similar note, I’m still using my customizations to the model-first workflow to generate C# code for creating the database rather than just a SQL file, but .net 4 RC now has methods on ObjectContext which will create a database or check for its existence based on just the SSDL used in the context, so I’ll probably switch to using that in a future release of D3. (It’s always great when I can get rid of some code and just use the EF’s built-in mechanisms).

The really interesting part, though, was when I started porting over the first core DPMud capabilities from the previous version code base. Most of what makes DPMud really work is in the Actor actions and the Event entities. This is how we simulate things happening in the virtual world of DPMud—methods are called on the Actor class for various actions like entering or leaving a room, picking up an item, etc. Those methods do some validation, modify the database state as appropriate and then create one or more Event entities which notify other actors about what happened. The normal workflow for a player client in DPMud is a combination of parsing string input to call these action methods and periodically querying the database for new Events and then displaying a string representation of the Events from the perspective of the player. The Event classes contain the business logic to compute the string describing what the event means so that when I see the event that I picked something up it will return a string saying “You picked up foo.” But when someone else sees that same Event, they will see “Danny picked up foo.” or something like that.

The key point is that these action methods and the events that go with them are really the core business logic of the system, and doing a good job of testing them is a critical mission for the D3 rewrite of DPMud. Since we’re going to need a lot of tests, and since we want to isolate the part of the system under test, it seems clear that we need to create a fake data access layer. Further, while I’m a big fan of the new IObjectSet<T> interface which is designed to make this process simpler, in this case we’ve got some tricky dependencies. The action methods are members of entity classes which we need to make sure have no dependencies on the DAL, but at the same time those methods need services from the DAL such as the ability to add new events or modify entity state.

Inversion of Control

The standard way to handle this kind of problem is an inversion of control / dependency injection pattern which has three parts:

We define interfaces as part of the same assembly containing the entities which describe the capabilities that the entities require from the DAL.
The DAL implements those interfaces, and we can create fake implementations of those interfaces for use by unit tests.
We create some mechanism to make an implementation of the interfaces available to the entities at runtime—either the “real” DAL implementation when the app is in production, or the fake implementation for the tests.

The first two aren’t terribly difficult… T4 to the rescue! I created a new interface ID3ObjectSet which is essentially the same as IObjectSet but specific to D3 and part of the D3.Model assembly rather than in System.Data.Entity.dll. Then it was easy to make a T4 template which generates an interface representing the context which has an ID3ObjectSet property for each top-level entity set. It was also a simple exercise to modify the template generating the context so that the generated context implements that interface except that ObjectSet<T> which provides the core capabilities for working with the sets in the database unfortunately does not implement the new ID3ObjectSet interface. So I created a simple class which wraps an ObjectSet<T> instance and implements the interface.

The third part of the pattern can be done several ways. Sometimes there’s a dependency injection mechanism which supplies the implementation whenever the objects that need it are constructed, but in this case the EF generally creates entity instances as part of materializing queries which makes it harder to inject dependencies at this time. So the approach I took instead was to make a simple IoC container the entities can call to get an implementation from. Often this is done with a general purpose IoC container like Unity or something along those lines, but in this case I created a small special purpose implementation just for the context which I call D3DB.

It has two parts. The core piece is just a thread-local-storage static property whose type is the context interface (ID3Context):

[ThreadStatic]
private static ID3Context current;

public static ID3Context Current 
{
    get
    {
        if (current == null)
        {
            throw new InvalidOperationException("Cannot access context without setting it first.");
        }
        return current;
    }
    set
    {
        current = value;
    }
}

This means entities can interact with the DAL (or a fake of it) by calling methods on “D3DB.Current”. I have a support method for actions, for example, which helps with creating new events that looks like this:

public void Log(Event e)
{
    if (e == null)
    {
        throw new ArgumentNullException("e");
    }

    e.SourceActor = this;
    e.SourceRoomId = this.RoomId;
    D3DB.Current.Events.AddObject(e);
}

The second part is a disposable object with a constructor that sets the static property and clears it when the object is disposed. That way we can have a “using” block that sets the implementation and clears it when the block goes out of scope. Also really simple except for the fact that correctly implementing the dispose pattern has several parts:

public class D3DB : IDisposable
{
    // thread-local-static property here...

    public D3DB(ID3Context context)
    {
        if (context == null)
        {
            throw new ArgumentNullException("context");
        }
        if (current != null)
        {
            throw new InvalidOperationException("There already is a current DB.");
        }

        Current = context;
    }

    //
    // Dispose pattern
    //

    private bool disposed = false;

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (!this.disposed)
        {
            if (disposing)
            {
                var context = Current;
                Current = null;
                context.Dispose();
            }
            disposed = true;
        }
    }

    ~D3DB()
    {
        Dispose(false);
    }
}

This means, for instance, that I can create a fake context which uses HashSets instead of real ObjectSets, and then it’s easy to write a test where the action methods will use the fake context in the test but still have them use the real context in production. Something like this:

[TestMethod]
public void Enter_ValidRoom_SetsActorRoomAndLogsEnterEvent()
{
    using (new D3DB(new FakeD3Context()))
    {
        var actor = new Actor { Id = 1, RoomId = 1 };
        var room = new Room { Id = 2 };
        Assert.AreNotSame(room, actor.Room);

        actor.Enter(room);

        Assert.AreSame(room, actor.Room);
        Assert.AreEqual(room.Id, actor.RoomId);
        var e = (EnterEvent) D3DB.Current.Events.Single();
        Assert.AreEqual(actor.Id, e.SourceActorId);
        Assert.AreEqual(room.Id, e.SourceRoomId);
    }
}

I guess that’s more than enough for one post. Have a look, and let me know if you have questions or feedback. Next release, hopefully we’ll get more of the end-to-end infrastructure together so we can begin to have a working app rather than just a bunch of tests…

- Danny

Cascade Delete and 1-to-1 (or 1-to-0..1) Relationships

Daniel Simmons - MSFT — Sat, 06 Feb 2010 01:29:00 GMT

Here’s another episode in my ongoing mission to help folks be successful using the Entity Framework, as well as to learn more about how folks use the Entity Framework so that each release will be better than the last. Today’s task: Using cascade delete with 1-to-1 relationships.

The problem

I got an email from a customer asking for help with getting cascade delete to work properly in his application. After going through the normal obvious potential issues, I discovered that he seemed to be doing all the right things but it still wasn’t working. He was getting exception something like this:

System.Data.UpdateException: A relationship is being added or deleted from an AssociationSet 'FK_MyObjectDetail_MyObject'. With cardinality constraints, a corresponding 'MyObjectDetail' must also be added or deleted.

It turns out, he wasn’t the only one running into this problem. Some links (chances are there are others):

EF Forum: Cascade Delete Fails on One to One Relationships
stackoverflow: Deleting an entity that has an un-loaded child entity
stackoverflow: Entity Framework – Different multiplicity in SSDL and CSDL <- scan toward the bottom of this thread to see the issue come up

Given that this is obviously an issue, I asked around on the team, and the result is that this is the “collision” of a couple of features. Urrggg. Let me see if I can explain what’s going on, and then I’ll provide a work around which seems to be solving the problem for the customer who originally contacted me.

Some background

The two colliding features are cascade delete and an important related concept we call “relationship span.” To learn more you can take a look at these blog posts:

Cascade delete --

Key lessons for us:

Cascade delete in the model just deletes things that are in memory. If you want to delete things that the database has which haven’t been loaded in memory, then you need to configure cascade delete in the database as well as in the model.

Cascade delete in the model kicks in when you delete the principal entity – NOT when you remove the relationship.

Relationship span --

Key lessons for us:

When you have independent associations, unless you use MergeOption.NoTracking the EF will automatically retrieve the relationship info (ie. the EntityKey) for relationships where the other side has multiplicity 1 or 0..1.

When you delete an entity, the EF deletes all relationships involving that entity.

When you delete a relationship, if there is a “1” side to that relationship, the EF expects the entity on that side to be deleted. (In most cases if you use EF generated classes, this will be done for you automatically.) When you think about this expectation in terms of 1-to-1 relationships, it implies that the EF expects you to delete both entities.

So what the heck is going on?

Why do 1-many relationships work, but 1-to-1 relationships don’t? The problem here is that when I have a 1-to-1 relationship with cascade delete turned on, if I retrieve one of the entities in that relationship, the EF will automatically bring along the relationship to the other entity (that’s relationship span at work). Then if I delete the entity I have retrieved without having loaded the other entity in the relationship, the EF won’t do the cascade itself since it will only do cascade delete on things in memory—it depends on the database to do the cascade for things that are related in the DB but haven’t been loaded in memory—but the EF WILL delete the relationship because when you delete an entity the EF always deletes all relationships to that entity which it has in memory.

So far so good. I now have an ObjectStateManager which has an entry for the entity saying it should be deleted and an entry for the relationship saying it should be deleted. When I call SaveChanges, though, the EF first validates the set of things that it has to do as recorded by the state manager and it discovers that a relationship is being deleted which has a “1” side where the entity is not being deleted. This is the side of the relationship for the entity we haven’t loaded yet. Since this relationship is 1-to-1, it doesn’t really matter which side we’re talking about, if either side is not in memory, then the EF will throw the exception which is the cause of all our pain.

The workaround

Of course the frustrating thing here is that we have already setup the database to cascade for us. So if the EF would just delete the entity, then the database would detect that and delete the other entity and everything would be fine. This brings us to the workaround: Since the thing triggering the validation problem is the deleted relationship, if we just get that relationship out of the state manager before we do SaveChanges, then the validation step of SaveChanges will succeed because it will only find the entity to delete. When that delete goes through to the database, it will delete the related entity and everything will be fine.

So all we need to do is find ObjectStateEntries representing relationships in this and only this state and call AcceptChanges on them each time right before we do SaveChanges.

public partial class MyContext
{
    partial void OnContextCreated()
    {
        this.SavingChanges += (s, e) => FixCascadeDeleteForSingularAssociations();
    }

    public void FixCascadeDeleteForSingularAssociations()
    {
        foreach (var entry in this.ObjectStateManager.GetObjectStateEntries(EntityState.Deleted)
            .Where(e => ((e.IsRelationship == true) && AssociationIsSingular(e))))
        {
            if ((CascadeDelete(entry, 0) && EndIsPresentAndDeleted(entry, 0) && !EndIsPresentAndDeleted(entry, 1))
                || (CascadeDelete(entry, 1) && EndIsPresentAndDeleted(entry, 1) && !EndIsPresentAndDeleted(entry, 0)))
            {
                // AcceptChanges on this state entry so the update pipeline ignores it -- depend on the database to do the cascade
                entry.AcceptChanges();
            }
        }
    }

    private bool CascadeDelete(ObjectStateEntry entry, int index)
    {
        var dataRecordInfo = ((IExtendedDataRecord)entry.OriginalValues).DataRecordInfo;
        var associationEndMember = dataRecordInfo.FieldMetadata[index].FieldType as AssociationEndMember;
        return (associationEndMember.DeleteBehavior == OperationAction.Cascade);
    }

    private bool EndIsPresentAndDeleted(ObjectStateEntry entry, int index)
    {
        ObjectStateEntry endEntry = null;
        if (!this.ObjectStateManager.TryGetObjectStateEntry(entry.OriginalValues[index], out endEntry))
            return false;
        return (endEntry.State == EntityState.Deleted);
    }

    private bool AssociationIsSingular(ObjectStateEntry entry)
    {
        // IsSingular in this sense means it's 1-to-1 or 1-to-0..1, not many-many or 1-many
        var dataRecordInfo = ((IExtendedDataRecord)entry.OriginalValues).DataRecordInfo;
        return (EndIsSingular(dataRecordInfo, 0) && EndIsSingular(dataRecordInfo, 1));
    }

    private bool EndIsSingular(DataRecordInfo dataRecordInfo, int index)
    {
        var associationEndMember = dataRecordInfo.FieldMetadata[index].FieldType as AssociationEndMember;
        return (associationEndMember.RelationshipMultiplicity != RelationshipMultiplicity.Many);
    }
}

The key ideas behind this code are:

We put these methods in the partial class for our context and use the OnContextCreated partial method to make sure we register a handler for the SavingChanges event every time a context instance is created.
The main part of the event handler is a LINQ to Objects query over the ObjectStateEntries which are deleted 1-to-1 relationships.
We get metadata about the relationships by looking at the DataRecordInfo.FieldMetadata for the OriginalValues data record on the ObjectStateEntry. This data record has two columns – each containing the EntityKey of an entity participating in this relationship. The field metadata can tell us if cascade delete is defined as well as the multiplicity of that end of the relationship.
Since the relationship entry is in the deleted state, if we call its AcceptChanges method, then it will be detached from the context as though it never existed.

Once we put this code in place, each time SaveChanges is called, the offending relationship entries will be hunted down and killed before a save happens.

Oh, and if you prefer VB.net, here’s the VB version of this code. Note that I have not tested the VB version, but others have, and it seems to be OK…

Partial Public Class CascadeContainer

    Private Sub CascadeContainer_SavingChanges(ByVal sender As Object, ByVal e As System.EventArgs) _
         Handles Me.SavingChanges
        Me.FixCascadeDeleteForSingularAssociations()
    End Sub

    Public Sub FixCascadeDeleteForSingularAssociations()
        For Each entry In Me.ObjectStateManager.GetObjectStateEntries(EntityState.Deleted).Where( _
                Function(e) ((e.IsRelationship = True) AndAlso AssociationIsSingular(e)))
            If (CascadeDelete(entry, 0) AndAlso EndIsPresentAndDeleted(entry, 0) _
               AndAlso Not EndIsPresentAndDeleted(entry, 1)) _
               OrElse (CascadeDelete(entry, 1) AndAlso EndIsPresentAndDeleted(entry, 1) _
               AndAlso Not EndIsPresentAndDeleted(entry, 0)) Then
                ' AcceptChanges on this state entry so the update pipeline ignores it -- 
                ' depend on the database to do the cascade
                entry.AcceptChanges()
            End If
        Next
    End Sub

    Private Function CascadeDelete(ByVal entry As ObjectStateEntry, ByVal index As Integer) As Boolean
        Dim dataRecordInfo = DirectCast(entry.OriginalValues, IExtendedDataRecord).DataRecordInfo
        Dim associationEndMember = TryCast(dataRecordInfo.FieldMetadata(index).FieldType, AssociationEndMember)
        Return (associationEndMember.DeleteBehavior = OperationAction.Cascade)
    End Function

    Private Function EndIsPresentAndDeleted(ByVal entry As ObjectStateEntry, ByVal index As Integer) As Boolean
        Dim endEntry As ObjectStateEntry = Nothing
        If Not Me.ObjectStateManager.TryGetObjectStateEntry(entry.OriginalValues(index), endEntry) Then
            Return False
        End If
        Return (endEntry.State = EntityState.Deleted)
    End Function

    Private Function AssociationIsSingular(ByVal entry As ObjectStateEntry) As Boolean
        ' IsSingular in this sense means it's 1-to-1 or 1-to-0..1, not many-many or 1-many
        Dim dataRecordInfo = DirectCast(entry.OriginalValues, IExtendedDataRecord).DataRecordInfo
        Return (EndIsSingular(dataRecordInfo, 0) AndAlso EndIsSingular(dataRecordInfo, 1))
    End Function

    Private Function EndIsSingular(ByVal dataRecordInfo As DataRecordInfo, ByVal index As Integer) As Boolean
        Dim associationEndMember = TryCast(dataRecordInfo.FieldMetadata(index).FieldType, AssociationEndMember)
        Return (associationEndMember.RelationshipMultiplicity <> RelationshipMultiplicity.Many)
    End Function

End Class

- Danny

Deleting Foreign-Key Relationships in EF4

Daniel Simmons - MSFT — Sun, 31 Jan 2010 05:07:20 GMT

I try to keep up with blog posts on the net which involve the entity framework, and this afternoon I came across this post where someone had been experimenting with EF4 and encountered some cases where things didn’t behave as they expected. I started to respond in a comment to that blog but the comment starting getting awfully long. So here’s a more complete response (which I’ll then refer to in a short comment). Please take a moment now to go read the other post for context. OK. Now that you are back, here’s my response:

When you use FK-relationships (the default with EF4), there are 3 main kinds of configurations you can encounter which each have slightly different behaviors. Let me see if I can explain so that you can understand better why the EF behaves the way it does.

The first case is actually what appears in the first and last examples. It's where the relationship is backed by a foreign key property on one of the entities, and where that foreign key is non-nullable but also not part of the primary key of the entity. In this case, if you delete the "dependent" entity (namely the one with the FK property--typically on the many side of a 1-many relationship), then the EF knows that the entity going away means the relationship must go away as well. If, however, you just remove the relationship (either by removing the entity from a collection on the other side or by clearing a reference navigation property or something like that), then the EF attempts to make the foreign key property null which is what will remove the relationship without removing the entity. The problem is that since the FK property is non-nullable, when you attempt to save changes, the EF will throw an exception. The EF will allow you to be temporarily in this state which makes it easier to write code where you want to switch an entity to be related to some other entity, but it won't allow you to save when you are in that state. As a side note, the EF actually goes to significant lengths to support this temporary “conceptual null” so that it’s more convenient to program against entities with non-nullable foreign keys.

This kind of configuration is typically used to model scenarios where for a particular entity a relationship is required (you can’t have an order without a customer or something like that), and the EF's approach is to say that removing the relationship without deleting the entity is an error in your business logic.
The second configuration is a simple variation on the first one where everything is the same except that the foreign-key *is* nullable. In this case, when you remove the relationship, the foreign key property is set to null, and when you save everything works fine. This configuration didn’t come up in the samples mentioned in the blog post, but it is something we encounter periodically. Typically you would use something like this to model an optional relationship where an entity might have a related entity or might not—for example, I might have a salesperson for an order or maybe the order came in over the internet and there’s no salesperson.
Finally, there’s the case where the foreign-key property on one entity is also part of that entity’s primary key. In this case the entity’s identity is fundamentally dependent on the relationship. If you remove the relationship, then the entity must be removed. If you change the relationship to some other entity, then you still have to remove the entity and then insert a new one because the entity with a particular primary key went away and a new entity which may be mostly the same (but which has a different primary key so it’s a different entity) has been added. So when you have this kind of case, if you remove the relationship then the EF will remove the entity. This scenario is typically used to model a containment sort of relationship—like the order details lines in an order. If you remove the relationship between an order and its order detail, then the order detail line should go away. It’s also true, by the way, that this configuration implies cascade delete behavior from the principal entity to the dependent entity (or entities). So if you delete the order, then all the order detail entities associated with it will also be deleted.

Another variation to be aware of here in the first two cases is that you can have cascade delete behavior turned on. If you do, then deleting the principal entity will cause the dependent entity to also be deleted. Cascade delete does NOT, however, affect what happens when you remove the relationship. The only case in current releases of the EF where removing the relationship will cause the dependent entity to be removed is #3 above where the primary key of the dependent includes a foreign-key to the principal. An additional option we’d like to add for a future release of the EF is the ability to opt-in to an auto-delete orphans or cascade relationship or whatever you might want to call it behavior where case #1 above could have an additional option turned on so that it would delete the dependent rather than throwing an exception, but for now this is logic you will have to write yourself.

- Danny

Simple Query Results Caching with EF4

Daniel Simmons - MSFT — Thu, 28 Jan 2010 06:39:00 GMT

Recently I spent some time working with a customer who was working on converting an existing application which had originally been written against an old 4GL database technology to .Net and EF4. The programming patterns used in the app were a lot different than what we would normally expect from an app written from scratch against the EF. In one of the typical scenarios, for instance, a single web service call made more than 200 database queries, so you can imagine that they were very focused on eking out every bit of performance they could. As we started working on the problem, they tried one of my standard recommendations which was to switch to no tracking queries, but we were surprised to find that it made no appreciable difference in performance. This was explained when we took a look at the context at the end of running all the queries with the default AppendOnly merge option and discovered that all 200+ queries only retrieved something like 5 different entities. The code they were migrating retrieved the same entities over and over again, so this app was a perfect candidate for a simple caching solution.

In this post I’ll use the solution we put together for this customer to illustrate a few things about the EF and another interesting feature or two added to .Net 4. This is NOT a comprehensive, general purpose caching solution. That’s something which is decidedly beyond the scope of a couple blog posts and that the EF team will look at for a future release. For this customers’ particular scenario, though, the solution we’re going to look at was very effective.

The key observations that made this possible were:

1) The idea that a programmer can easily give a query a string constant name and use it to rendezvous on the same query elsewhere in their code, and if that’s done, the query name makes a good key for caching. One of the tricks about compiling LINQ queries is that the expression tree of a LINQ query can’t be used as a key in a dictionary or the like, and trying to create a hash or something like that from the expression is itself quite expensive. So just specifying a string name makes a lot of things easier.

2) If we specify LINQ queries as a Func taking an IQueryable of entities as its first argument and taking query parameters as the remaining arguments, then that Func is easy to author, it provides something which has the right shape to appease the compiler so that we get great usability in our new caching query API, and that same Func can be executed both against the EF and sent to the database and against an in memory collection with LINQ to Objects.

3) In this particular scenario, query results only need to be cached for the lifetime of a particular ObjectContext instance—we aren’t trying to build something that caches results for queries across multiple threads or multiple mid-tier servers. This isn’t a second-level cache, but more like a first level one.

4) If we are going to add query caching, it’s critical that we also add query compilation, because for LINQ compilation of queries makes them *dramatically* faster. A caching solution without compilation would be pretty much pointless.

Usage

What all this means is that we can write code which looks like this:

var cust = ctx.QueryWithCaching("query1", "ALFKI", 
    (IQueryable<Customer> customers, string id) => 
        customers.First(c => c.CustomerID == id));

and if I were to issue the same query again against the same context with the same parameter values, then there would be no roundtrip to the server—instead I would just get back the same results. Without this new method, because the second run of the query is against the same context, identity resolution would cause me to receive the same object instance back, but the system would require a roundtrip to the database to determine those results.

Similarly, if we just want compilation but not caching, we can write:

var projList = ctx.QueryAutoCompiled("query2", "UK", 
    (IQueryable<Customer> customers, string country) =>
        from c in customers
        where c.Country == country
        select new CustomerProjection() 
        {
            Contact = c.ContactName,
            Country = c.Country
        });

Running this same query again, against this context or any other context, would still require a roundtrip to the database, but it would not require the system to translate the LINQ expression twice—that would happen only on the first call, and then the translation would be cached in a form which can be re-used. This translation is one of the most expensive parts of performing a LINQ query (after making the roundtrip to the database), so even just this step can often be very useful.

Implementing QueryAutoCompiled

First let’s look at what it takes to implement QueryAutoCompiled, then we’ll take on caching (which uses this method when it actually executes a query against the database). The auto-compiling task has three parts:

Appeasing the Compiler - creating a set of flexible overloads
Transforming the Lambda - changing the expression from what is passed to our method into the form required by CompiledQuery.Compile
Compiling, Caching & Executing

Let’s take these in reverse order so that we can move from the easiest to the hardest. Yes, in this case appeasing the compiler really is the hardest part—in fact, it’s appeasing the compiler which forces us to do part #2 at all, but we’ll get to that later.

Compiling, Caching & Executing

Assuming we can get the right lambda expression and have the appropriate parameter values, this process is pretty easy except for one thing—normally the pattern we recommend for compiled LINQ queries is to put the result into a static variable. That way you only pay the price of translation once for your entire program execution. Any thread anywhere in your program can access the one static variable which does not need to change and use it to execute a query. In our case, we want to handle this in a central way, and we’ve got a great key in the form of the query name, so the easiest thing is to use a Dictionary, but we want to put that Dictionary in a static and make it thread safe. Happily, .Net 4 now includes the parallel extensions and that means we have just the ticket: ConcurrentDictionary.

To make this work we add to the partial class for our strongly typed context a private static ConcurrentDictionary<string, object> and use that to cache the compiled query. The key type is obviously a string, for the value type we just use object because different compiled query expressions will have different types since they take different numbers of a parameters and produce different results. I’m not sure, maybe there’s some way we could make this type actually be an expression base type, but it doesn’t provide a lot of extra type-safety because we need to cast the expression to a more specific version in order to execute it anyway.

private static ConcurrentDictionary<string, object> compiledQueryCache = 
    new ConcurrentDictionary<string, object>();

Given the existence of this variable, checking the dictionary for a cached value and doing the compile and caching it if that’s not found is pretty much what you would expect. Leaving out the method signature and the query rewrite code which we will look at in the Transforming the Lambda section below, here’s the code for a scenario where we take one parameter (arg1) and where the type of the context this is associated with is NorthwindEntities:

object query = null;
if (!compiledQueryCache.TryGetValue(queryName, out query))
{
    // ... rewrite the query -- we’ll replace this comment with code in part 2 below ...
    query = CompiledQuery.Compile(rewritten);
    compiledQueryCache[queryName] = query;
}
var compiledQuery = (Func<NorthwindEntities, TArg1, TResult>)query;
return compiledQuery(this, arg1);

The last line actually executes the query and returns the results.

Transforming the Lambda

Next, let’s fill in the missing code above. The whole reason for this code comes from the fact that we want our method to take a lambda (or a Func really—we don’t care if it’s source is a lambda or not) which has an IQueryable<Entity> as its first argument, but CompiledQuery.Compile which does the actual compilation for us instead takes a lambda whose first argument is an ObjectContext, and then it looks for ObjectQuery properties in the LINQ expression and assumes they are associated with that context. One approach to solve this problem would be to just take a lambda of the same signature that Compile requires—the reason Compile works that way is so that you can use the same compiled query with multiple different context instances. As you saw in the code above, executing the query amounts to invoking the delegate returned from Compile and passing in a context instance (in our case “this” since the QueryAutoCompiled method we are writing is a member of the context we want to execute the query on). The problem with changing our method to take a Func of that form is first off that having a Func which only takes an ObjectContext as its first parameter makes things even messier with the compiler as we’ll see in the next section. Secondly having a Func whose first parameter is an IQueryable of an Entity type makes it possible for us to apply that same Func to an in-memory collection of entities with LINQ to Objects as well as to remotely execute it against a database. This can be handy for testing and other caching scenarios, so switching our code to use Funcs of this type not only simplifies our interactions with the compiler, but it can be handy for other reasons as well.

The trick here is to actually do this transformation. When I initially ran into this problem I suspected this was possible, but I was a bit concerned that the code would be really nasty. So I once again took advantage of one of the best perqs of working at Microsoft—especially on a team like the EF. I got in a room with the dev who implemented most of the LINQ translation for LINQ to Entities, and he showed me what’s involved. Given the public expression visitor in recent versions of .Net, it turns out that this is really quite easy.

The first requirement is that the argument to our method needs to be an Expression<Func<…>> rather than Func<…>. All I have to do to make that happen is change my method signature. The same exact calling code works with either signature as long as the caller restricts themselves to Funcs which are just expressions resulting in a value not statement blocks. The difference between an Expression and just having a Func is that the Expression is a data structure representing the code while the Func is basically a pointer to the actual IL which will calculate the expression. In fact, you can get a Func from an Expression by calling the .Compile method on the expression. This will translate the expression into appropriate IL so that you can then invoke it. In our case, we will create a new Expression based on the one passed in, and then let CompiledQuery.Compile do the compilation so that the Func we get back will be translated into the TSQL or other database query information needed to run the query against the DB rather than into IL which will use LINQ to Objects to run the query in memory.

To create that new expression, we first create a small class called ParameterSwapper which inherits from ExpressionVisitor. In the class we override the VisitParameter method so that when we call the Visit method on the base class, it will iterate over the expression tree and call our overridden method on each node in the tree which represents a parameter. When it finds a particular parameter value it will swap it out for a different paramter value. In our case we will setup the ParameterSwapper to look for instances of the IQueryable<T> passed into the func and swap them out for an ObjectSet<T> created from our context.

sealed class ParameterSwapper : ExpressionVisitor
{
   readonly Expression replacement;
   readonly ParameterExpression parameter;

   internal ParameterSwapper(Expression replacement, ParameterExpression parameter)
   {
       this.replacement = replacement;
       this.parameter = parameter;
   }

   protected override Expression  VisitParameter(ParameterExpression node)
   {
       if (node == this.parameter)
       {
           return replacement;
       }
       return base.VisitParameter(node);
   }
}

Given that class, the code to rewrite the expression is just these three lines:

ObjectSet<TEntity> objectSet = this.CreateObjectSet<TEntity>();

Expression newBody = new ParameterSwapper(
        Expression.Constant(objectSet, typeof(ObjectSet<TEntity>)), 
        expression.Parameters[0])
    .Visit(expression.Body);

var rewritten = Expression.Lambda<Func<NorthwindEntities, TArg1, TResult>>(newBody, 
    Expression.Parameter(typeof(NorthwindEntities)), expression.Parameters[1]);

First we create an ObjectSet from the context which matches the entity type of the IQueryable. Then we use the ParameterSwapper to visit the entire body of the lambda expression and create a new body with the first parameter of the lambda replaced by a constant expression referssing to the ObjectSet we just created. Finally we construct a new lambda whose signature takes the context type as its first argument rather than the IQueryable and whose body is the result of the expression visitor operation.

With this little bit of magic we now have a new Func which is exactly like whatever was passed in except that it has the signature needed by CompiledQuery.Compile. Those three lines can be pasted in place of the comment in the compiling, caching and execution section above. Which brings us to the third and most difficult part of QueryAutoCompiled…

Appeasing the Compiler

Now that we have the core logic for all this down, we come to an interesting point. To make this method as useful as possible, we want to be able to call it with a wide variety of lambdas returning a wide variety of results. If we were in a very dynamic language, we could probably do this in a snap, but we’re not. We’re using C# which for the most part requires strong typing. Happily with generics, type inference, Funcs, Expressions, T4 and all the other things added in recent versions of c#, visual studio and .net, we have a strong set of tools to work with, and by playing a few tricks we can end up with something that is flexible in all the ways we want it to be but no others.

To do that, the first thing we encounter is that we need a family of overloads for our method where each overload takes a different number of parameters. This mirrors the various versions of Func which take different numbers of parameters. We could just cut, paste and edit the method as many times as needed to support various numbers of overloads, but not only would that be extremely tedious and error prone to produce the first time, but maintenance of the whole thing would be a nightmare if we ever want to change it. So, T4 to the rescue! We’ll just create a small T4 template that contains one definition of our method and then put some template code around it that outputs the method as many times as necessary with the appropriate parts replaced for the various parameters. As a side benefit, this also means that adding this caching mechanism to an existing solution gets easier because we can just drop the template into a solution, edit one line at the top to point it to the EDMX, and then it will generate a partial class for the context with the appropriate methods on it.

<#
for (int numArgs = 0; numArgs<15; numArgs++)
{
#>
    public TResult QueryAutoCompiled<<#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>TEntity, TResult>
            (string queryName, 
             <#=GetArgDeclarations(numArgs)#>Expression<Func<IQueryable<TEntity>, 
             <#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>TResult>> expression)
        where TEntity : class
    {
	// ... do all that caching and compiling stuff described above ...		
        var compiledQuery = (Func<<#=code.Escape(container)#>, 
		                  <#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>TResult>)query;
       	return compiledQuery(this<#=numArgs > 0 ? ", " : ""#><#=GetArgs(numArgs)#>);
    }

<#
}
#>

Naturally this bit of code needs to be surrounded in the template by code that looks in the EDMX to figure out the name of the context type and things like that, but this represents the core interesting part. You can see that the QueryAutoCompiled method is surrounded by a loop which iterates from 0 to 14. The method output for the 4th iteration (which takes 3 parameters) looks like this:

public TResult QueryAutoCompiled<TArg1, TArg2, TArg3, TEntity, TResult>
        (string queryName, 
         TArg1 arg1, TArg2 arg2, TArg3 arg3, 
         Expression<Func<IQueryable<TEntity>, TArg1, TArg2, TArg3, TResult>> expression)
    where TEntity : class
{
    // ... do all that caching and compiling stuff described above ...
    var compiledQuery = (Func<NorthwindEntities, TArg1, TArg2, TArg3, TResult>) query;
   	return compiledQuery(this, arg1, arg2, arg3);
}

Basically we take places where the code needs to change based on the number of arguments and replace them with calls out to simple methods written in the template that take the number of arguments as a parameter and return an appropriate string. Things like GetArgTypes which returns a series of TArg1, TArg2, etc. and GetArgDeclarations which returns TArg1 arg1, TArg2 arg2, etc. We also have places which output the name of the context type where needed.

Of course you will notice that this method is a giant generic with many type arguments. Happily the c# compiler is now pretty darn good at type inference so in almost every case you can ignore all those type arguments when calling the method, and they just provide the right bits needed when we need to refer to those types in the implementation of the method. If you did have to specify all the generic parameters, the code for using this method gets really, really ugly—especially in cases where you have multiple query parameters, and sadly this is the core of the reason why we needed the visitor code to transform the expression above. If the first argument to the Func is just ObjectContext, then the compiler can’t properly infer some of the types, and you end up having to manually specify all the generic parameters. By making the argument an IQueryable, the compiler ends up with enough information to figure everything out for us.

You may also notice that there is a type constraint on the TEntity generic parameter which requires that it be a class rather than a value type. This is needed because one of the places we use that type in the body of the method has the same constraint. If we didn’t have it, then the compiler would complain about that type.

Once we have the multiple arguments thing figured out, we run into the next problem which is that in a few small ways we want to handle passed in expressions that return a set of values differently from those which return exactly one value. The most important of these cases comes from the fact that if you have a compiled query which returns a set of things, and the resulting delegate’s return type is an IQueryable, then further refinements of the query will cause it to be recompiled transparently—which means the whole refined query will execute on the server but every time you execute it there will be a recompile step. For me this is usually not what I want. So if I have a query returning a set of things, after I compile it, I want my execution method to call AsEnumerable on the result and convert it to an IEnumerable rather than an IQueryable so that further refinements will happen using LINQ to Objects rather than be remoted to the database. If the query just returns a single object, though, you can’t call AsEnumerable. To allow us to separate out these differences, I put two overloads of the method into the body of the T4 template loop. One overload is the one above which takes a Func that just returns TResult, and the other overload takes a Func which returns IQueryable<TResult>. When faced with these two overloads, compiler is smart enough to select the more precise IQueryable<TResult> overload for cases where the Func returns an IQueryable even though if you only had the first overload everything (including things returning IQueryable) would be directed to it.

So the second overload (in T4 form) looks like this:

public IEnumerable<TResult> QueryAutoCompiled<<#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>TEntity, TResult>
	(string queryName, 
	 <#=GetArgDeclarations(numArgs)#>Expression<Func<IQueryable<TEntity>, 
	 <#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>IQueryable<TResult>>> expression)
   		where TEntity : class
{
    // ... do all the caching and compiling stuff ...
    var compiledQuery = (Func<<#=code.Escape(container)#>, 
			      <#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>IQueryable<TResult>>)query;
    return compiledQuery(this<#=numArgs > 0 ? ", " : ""#><#=GetArgs(numArgs)#>).AsEnumerable();
}

If you compare this to the other overload, you will see that the only differences are that the overall method result is IEnumerable<TResult>, the Func’s result type parameter is IQueryable<TResult> and the last line which returns the results calls AsEnumerable() after invoking the delegate.

Once we put both of these overloads inside the loop in the template, we end up generating our QueryAutoCompiled method with 30 overloads (0-14 parameters x 2) which is a bit crazy, but the result is a simple way to execute a linq query and have it sort of auto-magically just run faster the second time. Given that you have to pass the same query name and Func each time you want to call it, I’d recommend wrapping up the call to QueryAutoCompiled in a method of your own so that you can easily find and call it from wherever in your code.

Finally… Implementing QueryWithCaching

Now that we’ve built up this method and all the techniques used in it, creating the code to enable caching is pretty simple. The QueryWithCaching method takes the same signature as QueryAutoCompiled, and we use the same T4 template with dual overrides for single results and sets of things described above.

In this case we only want to cache the results for the lifetime of the context (not in a static that can be used across multiple contexts). So we just add a regular Dictionary field to the context:

private Dictionary<object, object> queryResultCache = new Dictionary<object, object>();

The big trick with this dictionary is that the key to it cannot just be the query name string like we used for the compiled query delegate caching above. In this case we want a separate entry in the dictionary not just for each query but for each combination of parameter values and each query. To accomplish that we take advantage of another new .Net 4 feature: Tuple. Tuple is basically a simple generic type that holds a series of values. Like Func, Tuple comes in a whole series of variants which take different numbers of generic types, but a single Tuple will only hold up to 8 types. Of course, since the types in a Tuple can be anything, you can always get more things in a Tuple by making the last Type of the Tuple another Tuple, and that’s exactly what we do. So we create a key for the query result cache by building a Tuple where the first member is the string query name and the remaining members are from 0 to 14 objects—one for each of the query parameters. Once we have an instance of such a tuple, we can use it as the key for our Dictionary, and the beauty of Tuple is that its equality comparison will make sure that each of the members are equal.

The portion of the T4 template for the single result overload of the method ends up looking like this:

public TResult QueryWithCaching<<#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>TEntity, TResult>
	(string queryName, 
	 <#=GetArgDeclarations(numArgs)#>Expression<Func<IQueryable<TEntity>, 
	 <#=GetArgTypes(numArgs)#><#=numArgs > 0 ? ", " : ""#>TResult>> expression) 
    where TEntity : class 
{
    var queryKey = <#=GetQueryKey(numArgs)#>;
	
    object result;
    if (!queryResultCache.TryGetValue(queryKey, out result))
    {
        result = this.QueryAutoCompiled(
            queryName<#=numArgs > 0 ? ", " : ""#><#=GetArgs(numArgs)#>, expression);
        queryResultCache[queryKey] = result;
    }
    return (TResult) result;
}

This template fragment depends especially on the following support function added elsewhere in the template:

string GetQueryKey(int numArgs)
{
    if (numArgs == 0)
    {
        return "queryName";
    }
	
    if (numArgs < 8)
    {
        return "new Tuple<string, " + GetArgTypes(numArgs) +  ">(queryName, " + GetArgs(numArgs) + ")";
    }
	
    return "new Tuple<string, " + GetArgTypes(6) + ", Tuple<" + GetArgTypes(7, numArgs) + ">>" + 
        "(queryName, " + GetArgs(6) + ", " + 
        "new Tuple<" + GetArgTypes(7, numArgs) + ">(" + GetArgs(7, numArgs) + "))";
}

The method just captures the three cases: 0 parameters returns just the query name, less than 8 parameters returns a single Tuple with the query name and the parameters, and more than 8 parameters returns a Tuple containing the query name, the first 6 parameters, and then another Tuple with the remaining parameters.

The resulting overload for two parameters, for example, looks like this:

public TResult QueryWithCaching<TArg1, TArg2, TEntity, TResult>
    (string queryName, 
     TArg1 arg1, TArg2 arg2, Expression<Func<IQueryable<TEntity>, 
     TArg1, TArg2, TResult>> expression) 
    where TEntity : class 
{
   var queryKey = new Tuple<string, TArg1, TArg2>(queryName, arg1, arg2);
    
    object result;
    if (!queryResultCache.TryGetValue(queryKey, out result))
    {
        result = this.QueryAutoCompiled(
            queryName, arg1, arg2, expression);
       queryResultCache[queryKey] = result;
   }
   return (TResult) result;
}

As you can see, this code just follows the same basic pattern as the code for dealing the compiled query cache except in this case when it can’t find the result in the cache, it just calls QueryAutoCompiled to run the query (with compilation if needed) and get the result.

The one last wrinkle is the overloads which return sets of results rather than single results. These overloads need to be different from the single result versions, because once we execute the query we need to copy the results into a List by calling ToList() rather than just sticking the IEnumerable in the cache. If we put the IEnumerable in the cache, then we wouldn’t have a copy of the query result values but rather sort of a promise to get them when asked, and if one bit of calling code enumerated them, the next bit would find the results already “used up” and not get back what they expected. So these versions just add a call to ToList at the end of the call to QueryAutoCompiled, looking something like this:

result = this.QueryAutoCompiled(queryName, arg1, arg2, expression).ToList();

That’s it! Queries with simple caching, and all you have to do is drop a template into your project, edit the line at the top to point to your EDMX file, and then change your query execution code to call one of these methods.

If you want the whole solution strung together, you can find it here. But don’t forget the caveats: This isn’t full production tested code, and the caching only works for the lifetime of a single context instance. Real, full-featured caching will have to wait for a future release of the EF.

- Danny

Check surroundings for safety

Daniel Simmons - MSFT — Tue, 26 Jan 2010 05:43:36 GMT

And now for something completely different… My wife took this picture recently, and every time I see it I can’t decide whether I want to laugh or run away screaming “NOT SAFE! NOT SAFE!”

I’ll leave figuring out the context of the picture as an exercise for the reader.

- Danny

EF Merge Options and Compiled Queries

Daniel Simmons - MSFT — Tue, 12 Jan 2010 23:35:21 GMT

Recently there have been some questions about compiled queries and how they relate to merge options. As I looked into it I realized that I didn’t fully understand the details of how it all worked, so I walked down the hall and spent a little while chatting with one of the devs who has done the most work on this part of the code. He was able to give me a more complete story, and I wanted to get it out to all of you—especially since this area of the EF just isn’t as clean as we would like. This is definitely another thing on the list for improvement in a future release (*after* .Net 4), but for now at least I hope to help you understand how things work so you can figure out how to get by until we can make it better.

Part 1: Merge Options

The MergeOption enum property of ObjectQuery<T> has a significant effect on the way the EF processes a query. The default is MergeOption.AppendOnly, and it sort of does the magic identity resolution pattern which is the default for most ORMs—that is, new entities retrieved by the query are attached to the ObjectContext, and if an entity which has the same key as an incoming entity is already attached to the context, then that object is returned as is rather than the incoming entity. Two of the other merge options are primarily used for conflict resolution: MergeOption.PreserveChanges will update the original values that the context has for an entity with the current data from the database so that the changes you have made in the context will be saved to the database and replace whatever is there, while MergeOption.OverwriteChanges will update the current values the context has so that your entity is changed to match the current state of the database. The most interesting one of the merge options, though, is MergeOption.NoTracking which says essentially “just give me whatever entities come from the database without trying to attach or identity resolve them against the context.” The NoTracking option is, understandably, the fastest one. What’s a little less obvious is that NoTracking in many cases will also produce a different query than the others because it is the simplest, most streamlined version that does the least magic so things like automatically retrieving EntityKey properties for entities related with independent associations do not happen with NoTracking queries—only with the other merge options.

For the purpose of today’s post, however, let’s home in on another aspect of MergeOptions which is that they are a property of a particular query instance NOT a property of the ObjectContext. So, for instance, I can use LINQ with the EF and create two separate queries which retrieve data from the same entity set but have different merge options, and to make things clear I would do that by getting the query instance and setting the MergeOption property on that query. Unfortunately, since IQueryable is a fixed interface for LINQ, and it’s what is returned when I use LINQ syntax, I typically have to cast my LINQ query to ObjectQuery before I can set the merge option. The result is code that looks something like this:

// executing a query directly from the context will get the default AppendOnly option
var query1 = from c in ctx.Customers
             where c.Country == "UK"
             select c;
var customer1 = query1.First();
Debug.Assert(customer1.EntityState == EntityState.Unchanged);

// if I want a different merge option, then I set the MergeOption on the query instance
var query2 = from c in ctx.Customers
             where c.Country == "UK"
             select c;
((ObjectQuery)query2).MergeOption = MergeOption.NoTracking;
var customer2 = query2.First();
Debug.Assert(!Object.ReferenceEquals(customer1, customer2));
Debug.Assert(customer2.EntityState == EntityState.Detached);

One pattern you will sometimes see is someone setting a merge option on the ObjectQuery (or ObjectSet in the case of EF4) property on the context and then writing a query which uses that query. This actually works, to a point, because the getter for the ObjectQuery properties on the context will cache the instance of the ObjectQuery (at least for the default codegen – if someone uses a custom template or writes the context by hand, then all bets are off). So it’s important to remember that this is a property of the query instance not the context. Let me show you how you could get tripped up if you weren’t careful:

var query1 = from c in ctx.Customers
             where c.Country == "UK"
             select c;
ctx.Customers.MergeOption = MergeOption.NoTracking;
var customer1 = query1.First();
Debug.Assert(customer1.EntityState == EntityState.Unchanged);

In the code above, what do you expect? Would the Assert fire or not? What about with this code:

ctx.Customers.MergeOption = MergeOption.NoTracking;
var query2 = from c in ctx.Customers
             where c.Country == "UK"
             select c;
var customer2 = query2.First();
Debug.Assert(customer2.EntityState == EntityState.Detached);

As it turns out, neither assert will fire, because in the first case, the merge option wasn’t set until after the query was created and since the merge option comes from the query not the context, query1 just uses the default merge option so the entity that is retrieved is attached and ends up in the UnChanged state. In the second case, the query is created after the merge option was set, so that query is based on the Customers query returned from the property and inherits its merge option with the result that when the customer is retrieved from the database it uses NoTracking and the entity ends up Detached.

If you look at the generated code for the Customers property on the context you will see the ObjectQuery caching in action:

public ObjectSet<Customer> Customers
{
    get
    {
        if ((_Customers == null))
        {
            _Customers = base.CreateObjectSet<Customer>("Customers");
        }
        return _Customers;
    }
}
private ObjectSet<Customer> _Customers;

If this code had instead been written to just always return the result of calling CreateObjectSet (a shortcut I sometimes take if I write an ObjectContext by hand for a simple POCO example or something), then both of the above queries would have used the default merge option because the line above which sets the MergeOption would set it on a newly created ObjectSet which is then thrown away because no one holds onto the reference, and the line below it that creates query2 would use a different newly created ObjectSet.

Are you with me so far? Put that info on the backburner, and let’s look at how compiled queries work.

Part 2: Compiled Queries

The idea of compiled queries is to reduce the cost associated with executing a particular query the first time by making sure that you pay that cost only once if you want to execute that query multiple times. The way it works is that you create your LINQ query in advance and call the Compile method to get back a special delegate that you can use later to execute that query. There are three steps. First, you declare a static and initialize it to the delegate returned by calling the compile method.

static Func<NorthwindEntities, string, IQueryable<Customer>> compiledQuery = 
    CompiledQuery.Compile((NorthwindEntities ctx, string country) =>
        (from c in ctx.Customers
         where c.Country == country
         select c));

When you are ready to execute the query, you invoke the delegate and pass in the context and parameters in order to get back the ObjectQuery.

var query = compiledQuery(ctx, "UK");

This query can then be used like any other:

foreach (var customer in query)
{
    // do some stuff
}

As it turns out, though, there are a few very unexpected things about the way compiled queries work. Again, I can only say that I’m sorry that this is such a tricky part of the EF, that we’ll work on it in a later release, and that in the meantime if you know how it works, you can at least figure out how to get your app to function:

No real work happens until the first time the query is actually executed. CompiledQuery.Compile is NOT like a prepare method where the heavy lifting happens when you call prepare. The hard work doesn’t even happen when the delegate is invoked to return the ObjectQuery. It only happens when the query’s execute method is called or more commonly when the query is enumerated. After the first time it is executed, though, all the hard work is cached so that future executions are faster.
If you create a new query based on the compiled query, it will work—you just won’t get any benefit from the compilation. ANYTHING that changes the query which will be sent to the server, will produce a new query which isn’t precompiled. Calling .First() or .Count() or .Any(), for instance, will change the query. You need to keep the query the EXACT same. One way to accomplish this is to call the .AsEnumerable() method on the compiled query and then any additional methods you call will be executed by LINQ to Objects so you will get the benefit of the compilation for the part that goes against the database and then do other steps in memory once the database query is done.

The third unexpected behavior deserves a whole section of its own…

Part 3: Combining Merge Options and Compiled Queries

The real tricky thing here comes when you combine compiled queries and merge options. Because the merge option is a property of the ObjectQuery instance, and with compiled queries you don’t actually get that instance until you invoke the delegate, you can’t specify the merge option when you initially call the compile method or when you invoke the delegate. The merge option does, however, have an effect on the actually generated query, so at the time when the query is first executed (that’s when all the interesting work happens, remember) the merge option is locked in. When the query is first executed, the EF examines the merge option on the ObjectQuery property of the context passed into the delegate. After that first execution, though, the merge option will be the same for any subsequent execution of the query regardless of the merge option set on the ObjectQuery of the context used for any subsequent execution.

So if you take the following code (using the same compiled query delegate created in the example in part 2 above):

using (var ctx1 = new NorthwindEntities())
{
    ctx1.Customers.MergeOption = MergeOption.NoTracking;
    var query1 = compiledQuery(ctx1, "UK");
    var customer1 = query1.AsEnumerable().First();
    Debug.Assert(customer1.EntityState == EntityState.Detached);
}

Then the compiled query will use the no tracking merge option so the assert doesn’t fail. If you follow that with this code…

using (var ctx2 = new NorthwindEntities())
{
    var query2 = compiledQuery(ctx2, "France");
    var customer2 = query2.AsEnumerable().First();
    Debug.Assert(customer2.EntityState == EntityState.Detached);
}

The second execution will also use the NoTracking merge option even though the ObjectQuery on its context has AppendOnly as its merge option.

To make matters a little more complicated, keep in mind unexpected behavior #2 which means if I had left out the AsEnumerable calls, then the query executions will be using new queries rather than reusing the compiled query so the second execution would NOT have the locked-in merge option and would instead pick up the merge option from the query on the context. So if the second execution code looked like this:

using (var ctx2 = new NorthwindEntities())
{
    var query2 = compiledQuery(ctx2, "France");
    var customer2 = query2.First();
    Debug.Assert(customer2.EntityState == EntityState.Detached);
}

Then the assert would fail because the AppendOnly merge option would be used and the entity state would end up Unchanged rather than Detached.

Summary

What can I say? Compiled queries are tricky, and when you combine them with merge options they get even trickier, but the performance benefit can be huge, so it’s worth learning about how they work. Keep in mind these three potentially unexpected behaviors:

No real work happens until the first time the query is actually executed.
If you create a new query based on the compiled query, it will work—you just won’t get any benefit from the compilation.
The merge option used with a compiled query is determined by the merge option specified on the ObjectQuery used as the basis for the compiled query at the time the query is first executed.

Now, back to trying to find time to complete the next phase of D3 and get that posted. :-)

- Danny

Handling Concurrency: It’s a Matter of Trust

Daniel Simmons - MSFT — Sun, 03 Jan 2010 22:06:17 GMT

(with apologies to Billy Joel <grin>)

Last summer I wrote a few articles for MSDN Magazine about N-Tier patterns with the Entity Framework (Entity Framework- Anti-Patterns To Avoid In N-Tier Applications, Entity Framework- N-Tier Application Patterns and Building N-Tier Apps with EF4). One of the most fun parts of the project has been the emails I’ve gotten from various readers with follow-up questions and interesting discussions. Today I responded to one of those messages and after writing up the email I thought the discussion might be of interest to more than just that one reader, so I’m repurposing the content here. Further thoughts and discussion are welcome.

The message was specifically regarding Anti-Pattern #3: Mishandled Concurrency from the first article, and essentially the issue was around how to handle concurrency given the design principle that a good service should not assume trust with its clients. The contention was that in order to avoid trusting the client, the service should either re-load data before doing an update in order to check what was really changed and validate consistency with business rules or it should digitally sign the original version of the entity, send it to the client and then verify it when the result comes back before relying on that data.

Here’s my response:

I agree that trust is a significant issue, but I'm not sure that your two options are the only ones or even that they are the preferable ones in many situations. The critical questions, I think, are: What are we trying to protect? And what kinds of things are we protecting from?

First off, we have the issue of whether or not the client is who we think it is (a question of authentication). If it's possible for someone with evil intent to make a call to the service and pretend to be someone other than who we think, then the service might allow that person to accomplish a task that should not be allowed, which of course would be a major issue, but that's a concern of other parts of the solution--not what we're talking about when we are looking at concurrency. If a request is modified between the client and the service (maybe a man-in-the-middle attack or something like that), then that has the same kind of problem, but again that's not so much a matter for the persistence or service implementation part of the application so much as it is for the messaging substrate of WCF or the like.

So more relevant to this case we have the issue of whether or not the operation being requested by the client should be allowed (authorization / validation). It seems to me that this has three parts:

1) Is the client allowed to change a particular value or not? Most of the time this kind of check is relatively simple / static, and unless I'm missing something it's never really affected by a question of concurrency. In any case we're not really looking at whether or not the client has really changed something compared to what's in the database--it's just a matter of whether the client is allowed to change the values at all.

2) Do the requested changes validate? Are they self-consistent? Do they violate any of our business rules? Again, this is a matter just of whether or not the request makes sense. It's not really a matter of whether the client is lying about what it changed or didn't or anything of the kind. The only time this would be an issue is if we had a business rule that said something like "the client is allowed to increase the value of their insurance coverage by up to 5% but no more than that." If that were the case, then of course we would not be able to trust the original value of the insurance coverage sent by the client--we would either need to re-query the database or send round-trip signed data or something, but this kind of rule is much less common, and again, the question is not one of concurrency so much as it is about reference data. (I'll also point out that for this kind of business rule we'd probably want it to actually be that it can't be increased by more than 5% during some particular time period or something like that rather than per request or else we'd be open to other kinds of attacks.)

3) What about concurrency? Finally we get to the heart of the matter. Can we trust the client to supply the correct original value for the concurrency token? What if there is an evil or buggy client which gets past the authentication checks and then makes calls to the service with a concurrency token that does not match the original value sent? If the value is modified, there are two possibilities. Either the value is some random thing that doesn't match what's in the database in which case the request will fail (keep in mind that the EF will use that concurrency token when it attempts to update the database so it is checked against the current value in the DB before any changes go through--just more efficiently because that check doesn't require an extra round-trip/it's part of the update statement), or someone else has modified the database in the meantime and the value happens to match the new concurrency token so the update goes through when it shouldn't have. This last one is the only case we really have to worry about, but we do have to keep it in perspective. We're talking about a request that passed the other checks--it's something the client should be allowed to do--just not if someone else has modified the data between when it was read and when the update went through. Further, the client has to correctly anticipate the next value of the concurrency token as well as how many times it has been updated, etc.

Yes, I can imagine some cases where the last case could cause a problem if my data is very sensitive or my service is exposed on the internet instead of an intranet, and I allow public access or something like that. Someone might write a client that randomly tries things as a vandalism/denial of service type thing, and in that case I might go to the length of signing things sent to the client or of caching the original values so that I can use them instead of what I get back from the client, but I'd say that in the majority of cases exploiting this kind of concurrency issue isn't realistic and trusting the original value of the concurrency token sent from the client is a good, pragmatic approach.

- Danny

P.S. Happy New Year! May we all have a blessed 2010.

D3: Rides Again – Now on VS2010 Beta 2

Daniel Simmons - MSFT — Sun, 06 Dec 2009 01:19:18 GMT

And you thought I might never post on D3 again. I know. I know. Jedi knights don’t wait 6 months for a follow-up blog post.¹Well, the wait is finally over. I’ve put a new release of D3 up on code gallery. In this release the code has been updated to work with VS2010/.Net 4 beta 2. The release also includes a reorganization of project names and some new functionality, but I’ll talk about those in a future post or two. For this post, let’s look at the three broad steps involved in upgrading the project since they illuminate EF features and changes you might need to make to move some other project from beta 1 to beta 2.

Step 1 : Basic Compilation

When I loaded the solution in beta 2 and tried to recompile, the first problems I encountered were issues with stylecop—there were a number of errors referring to a file called ".NETFramework,Version=v4.0.AssemblyAttributes." I found a solution in this thread.

The next change needed was to react to the EF’s breaking change from beta 1 to beta 2 where the ContextOptions property DeferredLoadingEnabled was renamed to LazyLoadingEnabled. The semantics of this property didn’t change—just it’s name, so this was a simple fix.

Once compilation was passing, the next step was to start leveraging new EF features in beta 2.

Step 2 : Adding FKs to the Model

Beta 2 not only added runtime support for FKs in the model, but the designer is now smarter about them, and they are turned on by default. So I had a few options:

I could do everything by hand:

Manually add the FK properties.
Go to each association and add the referential integrity constraint between the FK property on the dependent entity and the primary key on the principal entity.
Remove the association set mappings for the associations since the mapping will now be handled by mapping the FK property on the entity just like any other entity.
Finally, either manually add the mapping for that property or regenerate the database from the model.

Since the model is not yet all that big, I could recreate it from scratch, requesting FK properties with each association.
Given that I already had a database with foreign key columns (just not exposed on the entities), I could reverse engineer the model from the database with the option turned on to generate FK properties on the entities.

I choose option #3 for no reason except that it was interesting to come full circle from a model-first database to a DB first model and then back to model first when further changes are needed. :-) In the process I encountered one may difficulty which was that reverse engineering from the database uses a simplistic algorithm for naming things that works reasonably well when you only have one relationship between any two entities, but when you have more than one relationship between the same pair of entities things get a bit confusing. In D3 I have two identical relationships between Room and Exit--one for leaving a room by way of an exit and the other for entering a room from an exit. The one tricky thing is determining which association was connected to which navigation and FK property. First you need to right click on the navprop and choose the "select association” menu item. Then you can double click on the association to see a dialog which describes the referential constraint including the FK property.

Step 3 : Updating Model First Workflow & Templates

The final step was to update the D3 model first workflow and templates both because the bug mentioned in this previous post was fixed and because of other changes and improvements made by the designer team in beta 2. You can read the details of those improvements on the ado.net team blog, but the summary is that there are now two properties available in the EDM designer when you click on an empty place on the designer surface. The “Database Generation Workflow” property allows you to specify which workflow xaml file will be used when you choose to generate the database from the model. The default value picks up the file from one of the VS installation directories, but it can also point to a per user directory (where a new xaml file could be installed by a VSIX package for instance) or to a location relative to your project directory. The “DDL Generation Template” property specifies the T4 template used in the step for generating the data definition language based on the SSDL.

I was able to keep the overall strategy the same—for the most part just comparing the original versions of the files to the new versions and reacting to things like the renaming of the namespace Microsoft.Data.Entity.Design.Pipeline.EdmExtension to Microsoft.Data.Entity.Design.DatabaseGeneration.Activities, or the change in xml namespace for the StoreGeneratedAttribute in SSDL from the SSDL namespace to the annotations namespace.

In the case of the SsdlToCode template there were more extensive changes, so I took the new template from the system and reapplied the same customization I originally made. Once the updates were complete, I diff’d the new template against the old one and the changes generally fall into the following buckets:

Updating assembly and namespace names
Moving re-usable utility code out of the template and into a shared ttinclude file
Supporting multiple sql server schemas in the same ssdl file
Supporting SQLCE by skipping things it doesn't allow like specific schema names

While I was at it, I also applied a couple of improvements which a colleague suggested could be made to the default template including removing the “WITH NOCHECK” clauses when creating constraints and adding statements to create indexes for each foreign key.

Final Result

The final result of all this is that D3 is up and running on beta 2. We’re back to the functionality we had last June with beta 1 plus we now have FK properties and a somewhat more efficient database. This is just the foundation needed so that we can start filling in more of the overall application.

- Danny

Footnotes
1. For some of us increasingly old geezers, the cartoon Bloom County by Berke Breathed was a major part of our lives once upon a time. So I can’t help occasionally throwing out obscure references to it. You can see the specific strip I’m referring to here.

AttachAsModified revisited

Daniel Simmons - MSFT — Mon, 09 Nov 2009 23:28:13 GMT

It seems like on a fairly regular basis these days I encounter a question or an issue about the EF to which the answer is “I know I wrote a blog post about that…” Then when I go search my blog and find the post I discover that the post was written before EF4 and things are now a lot easier than they used to be. Here’s another one of those situations:

About a year ago I wrote a post presenting a little extension method AttachAsModified which was designed to make a few small n-tier scenarios easier to write by hand. You can read the post for more background on the situation, but the issue which came up today is that in EF4 the code in that post can be made much simpler. The hardest part of the code before was iterating over an object’s properties and telling the ObjectStateManager to mark each property as modified. It wasn’t all that many lines of code, but it was pretty obtuse. The reason for this code is that the ObjectStateManager had two relevant methods—one marks the whole entity as modified but not any of its properties, and the other marks a single property as modified. What was missing was something that would set all of the properties to modified, and the trick for that was to use metadata from the EF to get the list of persisted property names.

As part of our effort to improve n-tier support in general for EF4, though, we added the method ChangeObjectState to ObjectStateManager, and if you use that method to change an object’s state to “Modified”, then it will do the work of mark each property as modified for you. So my AttachAsModified method becomes trivial—just one call to attach the entity and another call to change its state.

In addition, the earlier version of this method actually had two overloads—one for the case where your entity implemented IEntityWithKey so it could use the key to Attach the entity and another which did not require that interface but did require you to supply the EntitySet name. With the introduction of ObjectSet<T> this can be further simplified because we can just add our AttachAsModified extension method to ObjectSet<T> and remove the need for the overload which takes an entityset name while still supporting a method signature that does not require IEntityWtihKey. Even better, the signature is strongly typed to match the type of the set so you get nicer intellisense support.

The final code for the new method is just:

public static void AttachAsModified<T>(this ObjectSet<T> objectSet, T entity) where T : class
{
    objectSet.Attach(entity);
    objectSet.Context.ObjectStateManager.ChangeObjectState(entity, EntityState.Modified);
}

At this point the whole extension method seems pretty superfluous, which I guess is the goal of improving the EF in the first place. :-)

- Danny

It’s time to rip up EntityBag and throw it away

Daniel Simmons - MSFT — Thu, 05 Nov 2009 18:36:05 GMT

A couple of years ago when the overall shape of what we would ship in the first release of the entity framework became apparent, I realized that one of the biggest issues users of that release would fight with was creating N-tier applications. So I started a project to explore the space by creating something I called EntityBag. In spite of the fact that I wasn’t too thrilled with the overall approach, I thought it would be a good exercise and if nothing else it would help illuminate the issues with building this kind of application on the EF and demonstrate to folks what techniques they could use with the EF to solve these problems. The result of this investigation was a series of blog posts and a sample project which I uploaded to code gallery. You can check out the project at: http://code.msdn.microsoft.com/entitybag/ That site also has links to each of the relevant blog posts which is where the real interesting data lives.

At the time that I created EntityBag (mostly January 2008), the version of the Entity Framework was called “beta 3”, and it was before the first RTM of the product. Unfortunately, when the first version was finally released we included a change to by default generate classes which would serialize an entire graph of related entities as one (at beta 3 only one entity would serialize at a time) which ended up breaking EntityBag and the fix involved fairly major surgery. From the beginning this project had never been intended as a full-quality production solution—I just wanted to teach people how they could use the EF to build the right solutions for their particular projects. In addition I wanted to work with the team to produce a much better, long-term solution for N-tier apps, so whatever time I had available for this topic I poured into the next release of the EF (EF4) rather than into updating the EntityBag project.

If not EntityBag, then what have we been doing for N-tier?

The team has been working hard to make N-Tier much easier with EF4, and the result is several key features that are available now when you combine Beta 2 of VS2010/.NET 4 and yesterday’s release of what we call the EF Feature CTP 2.

The first critical addition to the product are core changes to the ObjectContext and ObjectStateManager APIs which make working with entities which spend some time detached from a context MUCH easier. These changes were described about a year ago in this post to the EF Design blog. There have been a few small adjustments to the APIs as we went through the experience of the first and second betas for this release, but the core changes are all there along with much of the reasoning behind them.

Next, we added support for foreign-key relationships which allow FK properties to be added to entities, and the system does intelligent fixup between the FK properties and the collection and reference “navigation” properties which were the only representation of a relationship which the EF had in its first release. FK relationships are important to N-Tier apps for two reasons: First off, they make it easier to support scenarios where you don’t want the perf hit (magnified in N-tier applications) of retrieving an entire object in order to establish a relationship. With FK properties you can just set the key value for the relationship. This was possible before by setting an EntityKey property, but that was harder than it should have been and MUCH harder (and sometimes impossible) when you have POCO entities. Even more interesting for N-tier applications, though, is the fact that FK relationships give the EF the knowledge that a relationship’s persistence is fundamentally tied to a particular entity rather than being conceptually completely separate, and this is important because it allows the EF to use concurrency checks for the entity to cover concurrency checks for the relationship as well. In the first release of the EF, the framework had no way to know from the conceptual model which if any entity “contained” the FK for the relationship so it performed concurrency checks for the relationships separately from the entities which meant that you needed the original value of a relationship not just the original value of whatever concurrency token you used for the entity. Uggg… That’s a little long-winded and obtuse, but the key point to realize is that having FK-relationships makes n-tier applications more efficient and intuitive in at least a couple of ways.

Finally, we had resounding feedback in response to the initial N-tier post on the EF Design blog and in other places saying that while the core APIs were flexible and made things easier, there was still just too much work required to build end-to-end N-tier solutions. So we created the self-tracking entities template which ships for now in Feature CTP 2 but will be in the box when .NET 4 launches on March 22, 2010. Self-tracking entities won’t be right for everyone, and for those folks where they aren’t a great fit as is there will either be the option to modify the template (Long live T4!) or to write code by hand using the core API improvements on which the STE template is built. If they do match your situation, though, they make things MUCH easier. The amount of code required is dramatically less, and the functionality is really pretty great. You can write simple service methods that return entities. On your client (which does NOT require .Net 4 or any version of the entity framework—it works on silverlight, for instance) you can manipulate entity graphs by adding, deleting and modifying entities as well as splicing more than one graph together if you like, and the entities will keep track of the minimum required set of original values as well as whatever changes are made to them all by themselves. NO BAGS OR OTHER EQUIPMENT REQUIRED. The only constraint is that you must use the generated self-tracking entity code on your client (not a simple proxy). Then back on your service, the code to reattach that graph to the context with appropriate original values and all the tracking information is literally one method call: ApplyChanges.

How can I learn more about all this?

In addition to the posts referenced above, I recommend taking a look at the Self Tracking Entities walkthrough which was published after the first release of the feature CTP. The current release of the STE template is a little different, but this is a great starting point, and more information will be available on the ado.net team blog soon. The other resource to consider is that over the course of this summer I wrote a short series of articles in MSDN magazine about the topic of N-Tier in general and especially around using it with the Entity Framework. The third article in that series just came out in the November issue, and it contains samples both of using STEs and using the core APIs to build other N-tier solutions. You can find the articles online here:

So, death to EntityBag! Long live EF4!

- Danny

D3: Building Great Software is a Battle, Don’t Leave Any Assets on the Sidelines

Daniel Simmons - MSFT — Tue, 30 Jun 2009 09:01:00 GMT

Another D3 release is up. This time I continued my focus on fundamentals. Some of this is tedious, but as I establish solid mechanics I’m starting to gain some momentum. The theme for today is using every last asset at your disposal to fight the battle to build great software. When I first came to Microsoft a number of years ago, I was given the book Writing Solid Code to help get me on the path to building world-class, reliable software. While the book is a bit dated these days, the key message I took away is as important to me now as it was then:

Writing software is hard. So hard, in fact, that we need to play every trick in the book (and then some) if we’re going to have any hope of really getting it right.

The previous two versions of DPMud were just hacks, but this time around I’m trying to build something that demonstrates how the EF can be used as a key component in software to be proud of. So, I ought to do things right—at least as much as I can given time constraints and everything else.

Often Overlooked Asset #1: Analysis Tools

This weekend I spent some time enabling fxcop and stylecop for D3. Actually I’m using the VSTS code analysis feature rather than fxcop because it’s built in to the VSTS download of VS2010 with really nice integration. Turning it on was a breeze, I just right clicked on each of my projects, chose properties, clicked on the code analysis tab and then checked the Enable Code Analysis on Build box. Actually, the first step was to download and install the VSTS flavor of VS2010 since I had initially just installed VS-Pro. I was happy to find, though, that the web install was able to install VSTS on top of my existing pro install without any trouble and was even smart enough to skip all the minor components that were already on my machine…

The hard part was discovering just how many issues it could find even with the very small amount of code present in D3 so far. As you can see above I’m just using the minimum recommended rules right now, and even with that I had to fix quite a few issues. Most of them were essentially cosmetic things, but it did catch a place where I was doing string concatenation of a SQL query that should have been using a parameter. There wasn’t any user input in sight, so I didn’t really have to worry about an injection attack, but it’s still a much better practice to use parameters wherever possible so I was happy to fix it. At some point I expect that I’ll go back and crank the rule-level up even higher, but there were enough issues when I did that which were more about personal preference than really being super important (like requiring that assembly names begin with a capital letter) that I decided it wasn’t worth the effort just yet.

I also downloaded stylecop and turned it on. It’s a little more work since it’s not part of the core product, but like fxcop it really was surprisingly easy to turn on and then the real work was making the project comply. I’m so glad that I did it now rather than later in the project, because keeping things going as I make incremental changes appears quite easy—the hard part is in the conversion. If you have a large amount of code already, I would still recommend making the switch to these tools, but you might want to checkout the stylecop team blog where there is a post about ways to make sure all new code is checked and then gradually convert older code. Fxcop is probably a bit more difficult but still worth it.

In order to turn on stylecop, I copied the binaries for the tool to a directory under my source control, modified the batch file I load for my development environment to set an environment variable pointing to the directory where stylecop is present. Then I unloaded each csproj, edited the XML and added one import element pointing to the stylecop targets file (I put mine next to the Import element for the standard csharp targets):

<Import Condition="'$(StyleCopPath)' != ''" Project="$(StyleCopPath)\Microsoft.StyleCop.targets" />

I put the condition attribute on it so if you don’t have the environment variable defined, it won’t try to load anything. That way everything builds and works without stylecop. (Code analysis also has the feature that if you don’t have it installed because you have VS-pro or something instead of VSTS then the configuration is just ignored and everything builds fine.) Unfortunately, this same edit needs to be made to each csproj file so the trick is to make sure not to forget this when adding new assemblies in the future.

By default both code analysis and stylecop report things as warnings, but I wanted to make certain everything was fixed so I also added a couple properties to the csproj files that change them into errors rather than warnings:

<CodeAnalysisTreatWarningsAsErrors>true</CodeAnalysisTreatWarningsAsErrors>
<StyleCopTreatErrorsAsWarnings>false</StyleCopTreatErrorsAsWarnings>

One interesting issue that I ran into is that the stylecop VS integration doesn’t seem to work for VS2010. Most of the time that’s fine because the VS integration is really only about editing the stylecop settings to turn off certain checks and the like. The build integration that I describe above is all you need to make sure your source code is checked and warnings or errors appear in the VS IDE, but if you do have something you want to turn off (and in my case I decided to turn off the requirement to add documentation comments to everything as well as a few other small cosmetic things) then you need to edit the stylecop settings. Fortunately if you double click on a Settings.Stylecop file in windows explorer, it will launch the UI for editing the settings independent of VS. Probably there’s some other way to bootstrap this process, but for me the easiest way was to just launch the VS integration in VS2008 and then copy the settings file to my other project. Once you have the right settings file, you just need to put it in the root directory of your project.

Asset #2: Check Constraints in the DB

I don’t recommend trying to add check constraints for everything, but there are places where the database is in a good position to enforce constraints on your data. In many cases the EDM itself will enforce constraints—in the case of D3, the association between Actor and Room has cardinality 1:* rather than 0..1:* which means that every Actor must have a Room, and the EF will enforce that. In the case of the association between Item and Room and the association between Item and Actor, we need there to be one and only one of those associations for each Item at any one time, and the EDM doesn’t have any way to represent that. For both of these kinds of issues we will want to add business logic to the classes to make sure these things are prevented well before the time of saving changes, but it’s still good to enforce them in depth.

In the case of the item associations, I decided to use a check constraint in the database. So I added a partial method call to the generated database creation code so that it’s easy to do this:

internal partial class SqlDb
{
	partial void Customize()    
	{        
		// Check Constraint which requires that Items live in either a room or on an actor.        
		this.ExecuteCommand(@"ALTER TABLE [dbo].[Items]
		                      WITH CHECK ADD CONSTRAINT [CK_Item_one_and_only_one_location]
		                      CHECK ((([Room_Id] IS NULL OR [Actor_Id] IS NULL) AND
		                          NOT ([Room_Id] IS NULL AND [Actor_Id] IS NULL)))");
		this.ExecuteCommand(@"ALTER TABLE [dbo].[Items] CHECK CONSTRAINT [CK_Item_one_and_only_one_location]");
	}
}

Which brings me to the last asset in my list for today.

Asset #3: Unit Testing Your Brains Out

I’m sure this isn’t anything new to most of you, but like many others I’d say that I probably can’t emphasize enough the impact really drinking the unit testing kool-aide can have on your development. D3 truly doesn’t have all that much code yet, and every time I add more tests I discover more issues. Testing makes your code better as you go, plus it creates both documentation and automated verification which will help prevent regressions later.

Conclusions

There are always things you can do to make your code better. Even more important, though, is to find things you can do to make CERTAIN that your code is better and will stay better. Your software will benefit from it—D3 already has.

- Danny

D3: Fun with Concurrency and Model First – Part 2

Daniel Simmons - MSFT — Sat, 27 Jun 2009 11:14:37 GMT

In my last post I began the story of how I added concurrency checking capabilities to D3—we conquered a bug in the model first workflow activities in VS 2010 beta 1. Next we need to look into ways to customize the SSDL generated so that the concurrency column will end up with the RowVersion type that SQL Server will automatically update when any part of the row changes.

This lead to the next surprise. At first I thought that I would add a new custom activity to the workflow which would fix-up the SSDL, but this turned out to have its own complexities. As it turns out, the model first process runs the workflow within the visual studio process which means the DLLs containing activities for the workflow are loaded in the same way other VS DLLs are loaded, namely from the VS install directories or from the GAC. This also means if the DLL changes, you have to shutdown VS and restart it before model first will reflect the change. Conclusion: If you want to create a general purpose activity that many folks will reuse, then this is a reasonable approach You can carefully debug an activity and set something up to install it centrally. But, if you need a one-off customization, this is just too hard.

So we’re looking into options to help here, but in the meantime I’ve discovered another approach which seems to work well. The default workflow described in DbGen.xaml has two activities. The first one takes the conceptual model and produces an MSL and SSDL string, while the second one takes the SSDL and produces something which will create the database (normally a SQL file, but after my first round of customizations for D3 it will produce a c-sharp file with a method that will create it).

It turns out, though, that the second activity under the covers is a general-purpose mechanism that can be configured to take in a string, run it through a T4 template and produce an output string. The nice thing about using a template is that it allows you to write arbitrary code as a one-off in your solution and easily iterate on your design, modifying and rerunning without any need to restart VS, etc.

Eureka! The second activity can be used twice. In between the two default activities, it can be inserted again and configured to take the SSDL in, transform it with a different template and send it on to the final activity. In addition model first makes some useful bits of information like a path to the EDMX file available to the template, so we can access the CSDL in order to make better decisions about how to modify the SSDL.

The relevant part of the resulting XAML looks like this:

<Sequence>
    ...
    <edm1:SsdlToDdlActivity ExistingSsdlInput="[ExistingSsdl]"
                            SsdlInput="[Ssdl]"
                            DdlOutput="[Ssdl]"
                            TemplatePath="DbGen\SsdlUseRowVersion.tt" />
    ...
</Sequence>

For D3, the fix-up we need is to find properties in the SSDL which correspond to properties in the CSDL with the following characteristics: type is binary, maxlength is 8, concurrency mode is fixed and store generation pattern is computed. So we read the EDMX into an XElement tree in memory and then use LINQ to XML to gather the names of properties that need to be changed in the SSDL grouped by the name of the table in which they appear:

var query = from property in csdl.Elements(edm + "EntityType").SelectMany(t => t.Elements(edm + "Property"))
            where (string)property.Attribute("Type").Value == "Binary" &&
                  (string)property.Attribute("MaxLength").Value == "8" &&
                  (string)property.Attribute("ConcurrencyMode").Value == "Fixed" &&
                  (string)property.Attribute(store + "StoreGeneratedPattern").Value == "Computed"
            group (string)property.Attribute("Name").Value
            by entitySetName(property.Parent, csdl, edm);

This query depends on knowledge of the algorithm used by the CsdlToSsdlAndMslActivity to map conceptual model properties to the database—namely that the column names are the same as the property names on the entities and (at least for the base types of entity sets) the name of the table is the same as the name of the entity set that contains the type. So we query the csdl for all the elements that represent properties which have the required facets and extract the name of the property, then we group those property names by the name of the entity set containing the entity type to which the property belongs using these two functions:

string entitySetName(XElement entityType, XElement csdl, XNamespace edm)
{
    return (from set in csdl.Element(edm + "EntityContainer")
                            .Elements(edm + "EntitySet")
            where entityTypeName(set) == entityType.Attribute("Name").Value
            select (string)set.Attribute("Name").Value).Single();
}

string entityTypeName(XElement entitySet)
{
    XElement schema = entitySet.Parent.Parent;
    string nsPrefix = schema.Attribute("Namespace").Value + ".";
    string aliasPrefix = schema.Attribute("Alias").Value + ".";
    string typeName = ((string)entitySet.Attribute("EntityType").Value).Trim();

    if (typeName.StartsWith(nsPrefix))
    {
        return typeName.Substring(nsPrefix.Length);
    }
    else if (typeName.StartsWith(aliasPrefix))
    {
        return typeName.Substring(aliasPrefix.Length);
    }
    else
    {
        return "UNEXPECTED TYPENAME -- Where is the namespace?";
    }
}

The result of this query is an IGrouping<string, string> where the first string is the name of the entity set and the second string is a collection of property names from the base type within that set. With that information it’s a simple matter to load the SSDL into memory, search for the relevant entity types (in the case of the SSDL, each table is represented by an entity type and each column within the table is a property of that type) and properties within them. Then we swap out the attributes on the XElement for that property and set the name, the type to rowversion and the store generated pattern to computed. Finally, the template serializes the XML for the SSDL to its output.

But what about derived types?

The above algorithm is all well and good for a model with no inheritance—every entity’s corresponding SSDL can be fixed up properly, but what if I want to put a concurrency property on an inherited type? As it turns out the EF doesn’t allow that; concurrency properties can only be present on the base type for an entity set. While I generally don’t like limitations of any kind, I can’t really think of any scenario where I’d want a concurrency property only on a derived type, and if I do have them on a base type, then I don’t need a different or additional property on any of the derived types.

In fact, for standard TPH and TPC scenarios all of the properties for a particular entity instance appear in a single row of a single table. With TPH all of the instances are in a single table, and in TPC there are multiple tables involved but each type has its own table with all of the properties both from the base type and added in the derived type in that table. So the concurrency property will appear on each table and a rowversion will work just fine because any property changing in that row will cause the concurrency property to change.

I got to thinking about TPT, though, and became a little worried. First off, TPT is the default inheritance strategy used by model first. In this strategy the properties from a base type all live in one table which has a row for every single entity in the set, while extra properties added by derived types are placed in their own tables with just those properties and the primary key. So for example, if I have entity type A with an id, scalar properties a and b, and a version, and I have type B which inherits from A and adds scalar property c, then the database would have table A with columns for id, a, b and version PLUS table B with columns for id and c. (A similar sort of case exists with entity splitting or mixed mode mapping, but it’s easier to think about the pure TPH, TPC and TPT scenarios.) In this kind of mapping, what happens if I have an instance of type B and only property c is modified? Then only table B needs an update, and the rowversion would not be updated since SQL Server only maintains it for changes to the row of the table where it resides. Doh!

After banging my head against the wall for a little while, I decided to check with some folks on the team to see what might be done to solve this problem. Could we relax the restriction on concurrency properties only being allowed in the base type (in which case maybe for TPT scenarios we could add a second concurrency property on each derived type which would track changes on the extra tables)? As it turns out Colin Meek, one of the brightest guys I know and a key member of the team, was way ahead of me. He was able to tell me that the design for the update code that handles concurrency checks already takes this condition into account, and whenever any part of an entity which has a concurrency token is modified, the update system of the EF guarantees that the table with the concurrency value is “touched”. In cases where there isn’t a real modification being made, a fake update statement is issued which doesn’t really change anything but does force the rowversion to update.

So, in the end there was nothing to worry about. The SSDL modification algorithm described above works just fine, and once the updated workflow XAML and template were put into place in the project, model first sales through cleanly and produces code which creates a database with rowversion properties for each entity type. And the whole thing is verified by the successful completion of the unit test described in the first part of the article.

Conclusions

AppWeek has come and gone, and in the end I spent the whole week getting my one test to run, but we have that much more of a stable foundation for D3 work to build on. In addition at least one key bug was found and fixed, and some extensibility scenarios were explored and opportunities for improvement identified. The really good news, though, is that the EF in general is starting to become flexible enough that even if it doesn’t work out of the box in just the way I want, I can customize it for my needs.

- Danny

D3: Fun with Concurrency and Model First – Part 1

Daniel Simmons - MSFT — Sat, 27 Jun 2009 03:10:00 GMT

The second D3 release is now up at: d3-0.0026.zip It’s the fruit of my labors pretty much full-time this week because it has been “App Week” on the EF team this week.

What a great time! My kids got out of school for summer break last Friday (which means I have the option of sleeping in rather than waking up early to drive the car pool), and this week I got to almost completely clear my calendar and spend time working on DPMud. App week is a time we set aside periodically to devote as much of our attention as possible to using the EF and related technologies to build applications so that we can experience some of the joys (and pain) of our valued customers. Most everyone looks forward to it, and we invariably find lots of great opportunities to improve the product.

While most other folks have been building new applications from scratch, I devoted my week to concentrated time on my long-running favorite app and just picked the next task off my to-do list and went to work. That task was to add a concurrency-token property to each of my entities and a simple test to verify that optimistic concurrency checks are working correctly. Seems pretty simple right? The model currently only has 4 entities, so how hard could it be? Can you say “working all week on this one task”? :-(

I won’t claim to be a test-first kind of developer, and the test code below is something I wrote after I had things working rather than first, but I’ll show it to you first so you can capture the goal. Then we’ll talk through the challenges I encountered and how I conquered them (in the hope that I can save you some pain). If you remember from my post about the last release, each entity currently in the model has a similar set of scalar properties but different relationships. My idea was to just add a Version property which would be marked as a concurrency token and setup to be automatically updated in the database any time any property on the entity changes so that we would get automatic concurrency checks. This property should actually have protected visibility because I don’t want to interact with it directly when using my entities—I just want the EF to use it and maintain it for me. The result would enable this test helper method:

private void VerifyConcurrency<T>(Func<T> create, Action<T> modify) where T : class
{
    using (var ctx = Utilities.CreateTestContext())
    {
        var objectSet = ctx.CreateObjectSet<T>();
        var obj = create();
        objectSet.AddObject(obj);
        ctx.SaveChanges();

        using (var ctx2 = Utilities.CreateTestContext())
        {
            var key = ctx.ObjectStateManager.GetObjectStateEntry(obj).EntityKey;
            var obj2 = (T)ctx2.GetObjectByKey(key);
            modify(obj2);
            ctx2.SaveChanges();
        }

        modify(obj);
        ctx.SaveChanges();
    }
}

I made the method generic and gave it two delegate parameters so that I can easily re-use it for each entity type I currently have plus extend it as I add more types over time. I also designed the code so that it will work with POCO types since I anticipate switching D3 to use POCO classes in the near future. As a part of this release I also introduced a simple utility method which creates the D3Context with a connection string to my little test database—this way it’s quick and easy to type since intellisense finds things for me.

The basic pattern in this method is that it uses one context to add an entity to the database. I use the create delegate because different entities have different requirements as far as what must be set on them to make them persist successfully. Each of our current entity types requires a non-null Name property, for instance, but while the Room entity doesn’t have any other required properties, an Actor must have a non-null Room reference, etc.

Once the new entity is added, I then create another context instance (which means another connection to the DB, etc.) and on it retrieve another copy of that same entity. I do this retrieval by having the state manager in the first context lookup the entity for me by reference (works for POCO) and then give me back the EntityKey corresponding to it (also works for POCO even though the entity itself doesn’t have an EntityKey property). Then I can use that key with the second context to load another copy of the entity, and call the modify delegate which makes a unique update to some property on the entity and saves that update to the database.

At this point the first instance of the entity is being tracked by the first context, and it’s concurrency token (assuming everything is configured correctly) represents the state of the entity before the second context updated anything. So we call the modify delegate on this entity and SaveChanges which throws an OptimisticConcurrencyException because the concurrency token in the database doesn’t match what’s in this context.

In the actual test, I create a list of Action types which call this method specifying the appropriate type and passing in lambdas for the create and modify delegates. Then it just loops over that list running the action and verifying that each call produces the appropriate exception. You can look at the file Tests\Concurrency.cs in the release if you want the full details.

The point of all this, though, is that to make this test pass I needed my entities to have concurrency properties that were configured properly in the database. With recent versions of SQL Server, the best way to accomplish that is with a RowVersion column (previously called a timestamp). This special column type is stored as an 8 byte binary, and the server guarantees that it will have a unique value any time any part of a row in its table is modified. The trick is that this special type only exists in the server—it’s not available in the CLR or the EDM conceptual model. If I were reverse engineering a model from the database, that would be no big deal. The tools would automatically map the RowVersion column in the database to an 8 byte binary property in my model. In this case, though, I’m using model first to generate the database from my model, and when I create the property, model first has no way to know that it should create a RowVersion type rather than a binary type in the DB.

So it was time to customize the model first process again, but before I got even that far I ran into a bug. Ahhh… The joy of AppWeek and of beta software. The issue is that if you set Concurrency Mode to Fixed on any of your entities in VS 2010 beta 1, then when you choose Generate Database Script from Model, the tool will display an error rather than generating the script. Considerable debugging and discussion with the developer who did most all of the work for model first later, we were able to track the problem down to a one line problem where model first was assuming that all property facets that could be set in the conceptual model were at least valid possibilities for the storage model, when in fact Concurrency Mode is the one property which can only be in the conceptual model and not the storage model. So the workflow activity that generates the SSDL from the CSDL was triggering an exception from the metadata system.

This bug will be fixed in beta 2, but that doesn’t help us much for D3 right now. So my final solution was to extract a small piece of the model-first workflow into a separate DLL, make a fix to the code and then modify the DbGen.xaml file which describes the workflow to call out to that DLL rather than the core version that ships with the product. The final result is part of this release, and it’s called modelfirst-bugfix.dll. If you go the misc directory under the release and run the batch file install-bugfix.cmd it will copy the DLL to a directory in your VS installation (make sure you run this from a VS command prompt where the DevEnvDir environment variable is set). Naturally this little hack is completely unsupported, but it works for me, and it might work for you too if you need to use model first with concurrency and beta 1 of EF4.

Now at least I can run model first when I have concurrency configured, but I’ve still got the problem that my properties come through as binary rather than RowVersion type in the DB. That’s where I’ll pick up the next post… Until then, happy coding.

- Danny

D3: Release 0.0010

Daniel Simmons - MSFT — Thu, 11 Jun 2009 02:22:57 GMT

OK folks, after way too much delay, I’ve finally gotten all my ducks in a row and begun the process of rewriting DPMud. Since my goal is to completely rebuild it from scratch, the process is going to take a while, and I intend to let you all look over my shoulder a bit as I go. So don’t expect too much from this first release—all we have is a basic solution with a very simple model and some unit tests. You can download the release from: d3-0.0010

First, let’s take a look at a few decisions that came up along the way, and then we’ll dig into some specific code areas.

Decisions

Source Control and Licensing: It is very important to me to make this project as relevant and useful as possible for folks trying to build applications using the EF. So we decided to release the code via MS-PL which is a very permissive license. I’m no lawyer, but my understanding is that this basically means you can copy the source and use any part or all of it in your own projects. Originally it was my hope that I could just put the project on codeplex and use it for source control so that anyone interested could enlist and just sync down changes to their local machine as the project progresses, but there were various hang-ups with this approach, and in the end I decided to just use local source control and then periodically push releases up to code gallery which has a simple site system and where everything is automatically licensed with MS-PL.
Version Numbers: One of those mundane but required decisions along the way is how we will create version numbers for the releases. Here’s what I decided: Until we have the thing up and running, everything is version 0.something. The last four digits of the version are in the form MMDD where MM is the number of months since I started releasing (so 00 is June ‘09), and DD is the day of the month for that release. I haven’t done it yet, but that allows me to have a very simple algorithm to automate version stamping and such as I make subsequent releases.
EF Versions and Dependencies: One critical area of decisions was around what EF features do I take dependencies on—particularly when we’re in the state where EF4 is not yet complete. In the end I decided that the main principle was to avoid taking dependencies on EF features that aren’t yet available outside the team so that you can stay up with me if you like. This means, for instance, that I’m not going to take a dependency on foreign-key support even though I have access to an internal build where it’s working great. When FKs are publically available, we’ll take a look at the project at that point and evaluate whether or not it’s worthwhile to refactor the app to take advantage of them (almost certainly we will do that refactoring). Similarly, I’m not yet going to take a dependency on self-tracking entities because it isn’t yet available. It will become available much sooner than FKs, though, so I will plan on taking advantage of it soon. A harder question was code-only. The first CTP of it will also be available soon (in the same download with self-tracking entities), but I already know of some areas where I’ll need more control over the model and database generation than code-only will support in its first CTP, so we’ll have to wait on that refactoring until a later CTP of code-only. So the basic approach will be to use EF4 beta 1 with model-first to make it efficient to design the model and generate the database and code from that model.

Initial Model

Like every other part of the system, we’re going to start with a simple model and expand. Here’s a first look:

A few things to notice:

Each entity has an Id property which is a server-generated integer.
Each of these entities is a “real” thing which players will interact with in the virtual environment of the game, and as such we have given them all a few common properties including Name and Description. Eventually we’ll define an interface for these common properties so that we can reason about the commonality of these objects. We could have an abstract base class for each of these entities, but we want them each in their own entity sets, and the designer doesn’t support more than one entityset with the same base type, so using an interface gets the key aspects we want while allowing us to still use the designer (we don’t really need implementation inheritance here anyway).
The two associations between Room and Exit have each had one of the navigation properties removed so that Room only has one collection of “Exits” (it doesn’t also have a collection of entrances, so to speak), and Exit has only the TargetRoom navigation property. This makes our entities follow the intended model where exits go only one way. If you want two rooms to be connected in a symmetrical fashion, you need two exits one going from room A to room B, and another one going back from room B to room A. This is important because it means that we can easily give different names to the exits (from room A you go East to room B, but from room B you have to go West to get back to room A), and it means that you don’t always have symmetrical paths (when you jump off a cliff you can’t necessarily get back to the top of the cliff using the same way you got down, etc.).
We don’t yet have version fields for concurrency checking. That will come later, because it requires some additional customization to the database generation.
Item has a relationship with Actor and one with Room, but logically an item can only be related to one of them at a time (and it must always be related to one or the other so that items have location). These constraints we will represent in the database rather than the conceptual model. They aren’t currently enforced, but they will be once we make additional customizations to database generation.

Customizing “Model First”

The feature we call model first shows up in the entity designer (aka Escher) as the option “Generate Database Script from Model” on the context menu when you right click on an open part of the designer surface. Out of the box this will generate a SQL DDL script that you can use to create a database for persisting the entities in your model. It also automatically generates the SSDL and MSL which corresponds to that model. This is great because it means that I can concentrate on the thing that is important to my program, the conceptual model. If I make a change to the model, once that change is complete, I’m done. I don’t have to also modify the database and the mapping to reflect that change. Since I’m also generating entity classes from the model, I get a nice “DRY” (Don’t Repeat Yourself) development experience even though various parts of the system do have repetition, I don’t have to engage in that repetition myself. Naturally we want to have an even more DRY experience, and code-first will lead us there for folks who want to do everything in code, but for the design-time experience this is not half bad.

Naturally, though, there are times where I want more control over the database generation process, which is why there are customization hooks built-in. The process of generating the database is orchestrated by a customizable workflow which has a couple of activities that manage generating an MSL and SSDL for the CSDL and then generating SQL from that SSDL. This second step (SSDL->SQL) is further customizable because it is accomplished with a T4 template. Eventually we will take advantage of these customization mechanisms to allow us to automatically generate rowversion columns for concurrency, constraints that keep item relationships straight and other things.

All of this seemed somewhat academic, though, for the first few phases of the project until I started thinking through my testing strategy. Eventually we’ll test parts of the system using fakes and mocks and the like, but we also need integration tests that send data all the way down to the database and back, and I wanted to start writing those tests right away and deliver them with my initial stab at the model. To make that work I really wanted APIs for creating and dropping the database somewhat like we can get from L2S so that my tests can create a temp version of the database (separate storage but same schema) and test pushing data in and out of it. Eventually this will also be useful for deployment, import/export of data, version-to-version upgrades and the like.

So, I decided to customize the db generation process now by modifying the SSDL->SQL template to generate a C# method for creating the database schema rather than a SQL script. The process to set things up for customization looks like this:

Copy the workflow xaml file from %ProgramFiles\Microsoft Visual Studio 10.0\Extensions\EntityFrameworkTools\Workflows\DbGen.xaml and the SQL gen template from %ProgramFiles\Microsoft Visual Studio 10.0\Extensions\EntityFrameworkTools\Templates\SsdlToSql10.tt to my project.
I renamed the template to SsdlToCode.tt and modified the xaml file’s TemplatePath to just have that name rather than the full path to the original template.
Both files I added to my project in VS, but because they aren’t actually used at compile time (they are just used by the designer when I choose the menu option to generate the database) I made sure that the Custom Tool property for each of them is blank and the Build Action property is set to None.
I clicked on a blank part of the Entity Designer surface and then set the “Generate Database Script Workflow” property to just DbGen.xaml (just the file name because the xaml file is in the same project and directory with the EDMX).

Now when I choose the option to generate the database it runs the workflow as specified in my project which then uses the template from my project rather than the default versions.

Next, I modified the template file to output C# code rather than SQL. You can find the template file in the release and do your own diff against the default template, but the basic approach was to wrap each batch of SQL statements up to a “GO” in an ado.net command with a C# @”” string and execute it. I eliminated a chunk of the template that would drop existing tables because my approach is to drop the whole database and recreate it, and I put all of these execute statements into a method on an internal partial class that I added to my model DLL. That way most of that class is defined outside the template but this one method can be defined by the code generated from the template. Then I further created a partial class file for my generated EF context and added methods to check for existence of the database, drop the database, and create it (using the generated method).

One current limitation in the designer is that the generate database command always by default creates a file with a name based on the name of the EDMX file and with SQL for the extension. I’ve already started discussions with the designer folks about adding some customization hooks here, but for now you just have to remember that whenever you generate the database you should rename the file.

All of this enabled me to write the following simple test:

[TestMethod]
public void DropAndCreate()
{
    using (var ctx = new D3Context("name=D3TestContext"))
    {
        ctx.DropDatabase();
        Assert.IsFalse(ctx.DatabaseExists());
        ctx.CreateDatabase();
        Assert.IsTrue(ctx.DatabaseExists());
    }
}

Not only is it simple to perform these operations, but also everything is based on the connection string given. So I just created two different connection strings in my app.config file—one called D3Context (the default) and the D3TestContext which is the exact same as the other except that it has a different initial-catalog name for my test database.

The SqlDb Internal Class

It’s probably also worthwhile to take a brief look at the internal class I called SqlDb which does the heavy lifting for these database interactions. The first interesting piece of the puzzle is that in order to create or drop the database with SQL Server (or in this case Sql Express) we need to make sure that we don’t use a connection string with the database in question as the initial-catalog. So, we extract the connection string from the StoreConnection property on the EntityClient connection instance created by the context and create a SqlConnectionBuilder from it. SqlDb’s constructor takes a DbConnection parameter since that’s what ObjectContext exposes and creates the builder like this:

var builder = new SqlConnectionStringBuilder(((EntityConnection)entityConnection).StoreConnection.ConnectionString);

Then we store the initial-catalog property into a string field so we know the name of the database we should be working with, and then we replace the initial-catalog with “master” since that database is present on every SQL instance.

The constructor then extracts the new connection string from the builder and uses it to create a new SqlConnection instance. We use this connection to execute commands without directly connecting to the database. The class also implements IDisposable so that its Dispose method can Dispose of the SqlConnection.

The other interesting trick which came up was the fact that while my initial executions of the above test succeeded while stepping through in the debugger, running the whole test straight through failed because the first part of the DropDatabase method executes a command to check if the database exists, and after that the drop command would fail with an error saying that the database was in use. Eventually I fixed this by adding a ClearAllPools method to the class which closes the connection, calls SqlClient.ClearAllPools() to make sure that there are no connections in the connection pool holding onto the database and then re-opens the connection. This method is called after the existence check and before the command to drop the database.

Conclusions

We’re finally off and running! We have an initial model, a simple way to deploy the database and some initial tests. (Those tests have already found a couple of issues I had in my first crack at the model, by the way. So the testing is already paying off. Yay!) Next I’ll probably tackle some further customizations of the generated database. Soon we’ll look at replacing the default code gen with a POCO template, and in the not distant future I hope to fill out more pieces of the architecture so we can get a small end-to-end slice going with a WCF service and Silverlight, but all in good time…

- Danny

Links: EF4 Podcast, N-Tier Anti-Patterns, and some thoughts about DDD

Daniel Simmons - MSFT — Wed, 03 Jun 2009 10:56:50 GMT

Here are a few more resources I hope you will find useful:

Recently I had a conversation about EF4 with Carl and Richard of .Net Rocks, and it went up on their site today. Have a listen at http://www.dotnetrocks.com/default.aspx?showNum=451.

In addition, I have an article about design/architecture decisions related to n-tier applications in the latest issue of MSDN Magazine. This article is actually the first part of a two-part series. Originally it was a single longer article that wasn’t so depressing as this one came out (all about what *not* to do without really giving any help on what you should do). There are some important concepts, though, so hopefully you’ll find something useful even in this first part.

At the same time that I’ve been busy with these, some other folks have been trying out the beta of EF4 and some very interesting posts are starting to appear like this dive into producing a POCO repository with the EF by Keith Patton and a great discussion of LINQ Expression Trees and the Specification Pattern by the EF team’s own David DeWinter.

- Danny