system.data.objects dev guy : Web Services

It’s time to rip up EntityBag and throw it away

dsimmons@microsoft.com — Thu, 05 Nov 2009 18:36:05 GMT

A couple of years ago when the overall shape of what we would ship in the first release of the entity framework became apparent, I realized that one of the biggest issues users of that release would fight with was creating N-tier applications. So I started a project to explore the space by creating something I called EntityBag. In spite of the fact that I wasn’t too thrilled with the overall approach, I thought it would be a good exercise and if nothing else it would help illuminate the issues with building this kind of application on the EF and demonstrate to folks what techniques they could use with the EF to solve these problems. The result of this investigation was a series of blog posts and a sample project which I uploaded to code gallery. You can check out the project at: http://code.msdn.microsoft.com/entitybag/ That site also has links to each of the relevant blog posts which is where the real interesting data lives.

At the time that I created EntityBag (mostly January 2008), the version of the Entity Framework was called “beta 3”, and it was before the first RTM of the product. Unfortunately, when the first version was finally released we included a change to by default generate classes which would serialize an entire graph of related entities as one (at beta 3 only one entity would serialize at a time) which ended up breaking EntityBag and the fix involved fairly major surgery. From the beginning this project had never been intended as a full-quality production solution—I just wanted to teach people how they could use the EF to build the right solutions for their particular projects. In addition I wanted to work with the team to produce a much better, long-term solution for N-tier apps, so whatever time I had available for this topic I poured into the next release of the EF (EF4) rather than into updating the EntityBag project.

If not EntityBag, then what have we been doing for N-tier?

The team has been working hard to make N-Tier much easier with EF4, and the result is several key features that are available now when you combine Beta 2 of VS2010/.NET 4 and yesterday’s release of what we call the EF Feature CTP 2.

The first critical addition to the product are core changes to the ObjectContext and ObjectStateManager APIs which make working with entities which spend some time detached from a context MUCH easier. These changes were described about a year ago in this post to the EF Design blog. There have been a few small adjustments to the APIs as we went through the experience of the first and second betas for this release, but the core changes are all there along with much of the reasoning behind them.

Next, we added support for foreign-key relationships which allow FK properties to be added to entities, and the system does intelligent fixup between the FK properties and the collection and reference “navigation” properties which were the only representation of a relationship which the EF had in its first release. FK relationships are important to N-Tier apps for two reasons: First off, they make it easier to support scenarios where you don’t want the perf hit (magnified in N-tier applications) of retrieving an entire object in order to establish a relationship. With FK properties you can just set the key value for the relationship. This was possible before by setting an EntityKey property, but that was harder than it should have been and MUCH harder (and sometimes impossible) when you have POCO entities. Even more interesting for N-tier applications, though, is the fact that FK relationships give the EF the knowledge that a relationship’s persistence is fundamentally tied to a particular entity rather than being conceptually completely separate, and this is important because it allows the EF to use concurrency checks for the entity to cover concurrency checks for the relationship as well. In the first release of the EF, the framework had no way to know from the conceptual model which if any entity “contained” the FK for the relationship so it performed concurrency checks for the relationships separately from the entities which meant that you needed the original value of a relationship not just the original value of whatever concurrency token you used for the entity. Uggg… That’s a little long-winded and obtuse, but the key point to realize is that having FK-relationships makes n-tier applications more efficient and intuitive in at least a couple of ways.

Finally, we had resounding feedback in response to the initial N-tier post on the EF Design blog and in other places saying that while the core APIs were flexible and made things easier, there was still just too much work required to build end-to-end N-tier solutions. So we created the self-tracking entities template which ships for now in Feature CTP 2 but will be in the box when .NET 4 launches on March 22, 2010. Self-tracking entities won’t be right for everyone, and for those folks where they aren’t a great fit as is there will either be the option to modify the template (Long live T4!) or to write code by hand using the core API improvements on which the STE template is built. If they do match your situation, though, they make things MUCH easier. The amount of code required is dramatically less, and the functionality is really pretty great. You can write simple service methods that return entities. On your client (which does NOT require .Net 4 or any version of the entity framework—it works on silverlight, for instance) you can manipulate entity graphs by adding, deleting and modifying entities as well as splicing more than one graph together if you like, and the entities will keep track of the minimum required set of original values as well as whatever changes are made to them all by themselves. NO BAGS OR OTHER EQUIPMENT REQUIRED. The only constraint is that you must use the generated self-tracking entity code on your client (not a simple proxy). Then back on your service, the code to reattach that graph to the context with appropriate original values and all the tracking information is literally one method call: ApplyChanges.

How can I learn more about all this?

In addition to the posts referenced above, I recommend taking a look at the Self Tracking Entities walkthrough which was published after the first release of the feature CTP. The current release of the STE template is a little different, but this is a great starting point, and more information will be available on the ado.net team blog soon. The other resource to consider is that over the course of this summer I wrote a short series of articles in MSDN magazine about the topic of N-Tier in general and especially around using it with the Entity Framework. The third article in that series just came out in the November issue, and it contains samples both of using STEs and using the core APIs to build other N-tier solutions. You can find the articles online here:

So, death to EntityBag! Long live EF4!

- Danny

Links: EF4 Podcast, N-Tier Anti-Patterns, and some thoughts about DDD

dsimmons@microsoft.com — Wed, 03 Jun 2009 10:56:50 GMT

Here are a few more resources I hope you will find useful:

Recently I had a conversation about EF4 with Carl and Richard of .Net Rocks, and it went up on their site today. Have a listen at http://www.dotnetrocks.com/default.aspx?showNum=451.

In addition, I have an article about design/architecture decisions related to n-tier applications in the latest issue of MSDN Magazine. This article is actually the first part of a two-part series. Originally it was a single longer article that wasn’t so depressing as this one came out (all about what *not* to do without really giving any help on what you should do). There are some important concepts, though, so hopefully you’ll find something useful even in this first part.

At the same time that I’ve been busy with these, some other folks have been trying out the beta of EF4 and some very interesting posts are starting to appear like this dive into producing a POCO repository with the EF by Keith Patton and a great discussion of LINQ Expression Trees and the Specification Pattern by the EF team’s own David DeWinter.

- Danny

Code Sample from my Applied Entity Framework talk at TechEd 2009 now available

dsimmons@microsoft.com — Fri, 15 May 2009 08:51:14 GMT

This week I gave a talk at TechEd focusing on architectural considerations as well as some useful tips for creating n-tier applications with the Entity Framework v3.5SP1. The code from the simple sample application is now available at http://code.msdn.microsoft.com/dtl401sample/. It includes:

an Entity Framework-based Business Object and DAL
some rudimentary validation implemented using a T4-template to create customized code generation with an OnValidate partial method much like is available from LINQ to SQL
a repository which abstracts the UI from data access & a fake implementation of that repository
an asp.net MVC-based UI with URL routing that allows you to specify either to run on the fake repository or on the real one
a WCF implementation of that repository which remotes calls from the web tier to the midtier
several unit tests

Have a look and let me know what you think. Of course this is not the end-all-be-all of anything. It’s just a sample designed to illustrate a few concepts, but maybe someone will find it useful.

- Danny

Building N-Tier applications with the EF – The Basics

dsimmons@microsoft.com — Thu, 09 Apr 2009 08:08:00 GMT

Today I was looking at a post in the forums where someone asked a very natural and common question about the EF that I end up answering pretty frequently. So I decided to put my answer here on my blog to make it easier to refer back to the answer in the future. The question was essentially this:

I’m building a silverlight client (but it could just as easily be a rich client or an ajax app or any other client of a WCF service) with a WCF service on the mid-tier that uses the EF to persist changes to a database. I can easily write a WCF service method to retrieve entities, and I can also create one which will add new entities to the database by just calling AddObject and then SaveChanges, but I can’t figure out how to do an update. I was hoping there would just be an UpdateObject method or something like that. What do I do?

Unfortunately this is more difficult than we'd like it to be in EF3.5 (the first release). It's a bit difficult because the problem space is more complicated under the covers than it seems to be on the surface, so you have to think carefully about a few things. That said, the EF API could help more in these scenarios, so in EF4 we're working on a series of improvements which you can read about in posts on the EF Design blog such as this one: http://blogs.msdn.com/efdesign/archive/2009/03/24/self-tracking-entities-in-the-entity-framework.aspx
In the meantime, there are three important things you need to think about:

Keeping track of concurrency tokens for the purpose of doing concurrency checks when you update (you don't want to blindly overwrite a change that some other client has made to the same part of the database between the time when you read the data and the time when you try to write back your changes).
Keeping track of which properties you have modified on an entity. This is important if you want to make sure that the update to the database only includes the modified properties. If you are OK with sending an update statement that includes even unmodified properties (which by the way can sometimes even be faster because it might allow the database to reuse a query plan rather than generating a new one for each combination of modified props), then it may be OK not to worry about this one.
If you have a graph of related entities then you need to worry about tracking what kinds of operations happened to what parts of the graph--one entity modified, another one is a newly added entity, another one deleted, etc.

All of this amounts to tracking this information on the client side, communicating it back to your mid-tier through your WCF service methods, and then replaying information to the ObjectContext so the changes can be persisted. If you skip option #3 for now (one of the hardest parts with EF3.5 and something which can occasionally be skipped if you are willing to just have WCF service methods which operate on one entity or which treat all entities in the graph the same way--so you have separate methods for add, update and delete), then you already know how to do the add method. The delete method is also pretty straight forward, you just need to call Attach on the entity you pass to the method to bring it into the context and then call DeleteObject on it and SaveChanges. The interesting method, as you mention, is update...

The simplest but not terribly efficient way to solve the update method is to make a copy of the entity on the client before you make any changes so that at the time you want to update things you have both a copy of the original version of the entity and the modified one. Then your update service method can look something like this:

void Update(MyEntity original, MyEntity updated)
{
    using (var context = new MyEntities())
    {
        context.Attach(original);
        context.ApplyPropertyChanges(updated);
        context.SaveChanges();
    }
}

You can improve on this, though, if you are willing to make a few assumptions. First, if you can guarantee that the client will not change the property used for concurrency checks, then you don’t need the original value of the concurrency property. Secondly, if you are willing to always send all properties to the server on an update rather than just those properties which have changed, then you don’t need an original value for all the other properties to compare against to see if they are changed. In this case you can get away with only sending a single, updated entity from the client back to the mid-tier, and you just have to use some trickier APIs to get the entity attached to the context in a form so that the EF will treat it as modified. I wrote a previous blog post about how to do that. With the extension method from that post, you can get rid of the original entity argument to the update method and just call the AttachAsModified method followed by SaveChanges().

- Danny

Part 2 of dnrTV about the Entity Framework now up

dsimmons@microsoft.com — Tue, 29 Jul 2008 02:16:00 GMT

Check it out at http://www.dnrtv.com/default.aspx?showNum=118. In this episdoe we build on the first EF dnrTV by looking at using the EF in a variety of scenarios including ASP.Net web pages with the Entity Data Source Control and with ASP.Net Dynamic Data and web services with ADO.Net Data Services and written by hand using WCF.

- Danny

EntityBag – Wrap-up and Future Directions

dsimmons@microsoft.com — Mon, 28 Jan 2008 10:49:00 GMT

Over the course of quite a few posts during the last several weeks I’ve shared source code that adds up to an implementation for EntityBag<T>. Piecing together a project from all those snippets, though, would be a pretty painful task, so I put the whole thing together in a zip file and posted it up at the new MSDN Code Gallery site: http://code.msdn.microsoft.com/entitybag/

Since I finished the series, though, I’ve done a bit more testing on the project and ran into a small bug. ContextSnapshot includes a copy of the connection string from the context which is the initial source of the EntityBag, and it uses that connection string to create its own internal context for tracking changes to the graph. In my initial testing, I always constructed the context by passing a connection string to the constructor, and everything worked fine, but an even more common pattern is to use the parameterless constructor which causes the context to have a connection string of the form: name=”containerName” and the full connection string is actually in the application’s config file. The problem with this is that on the client the config file would not be present. So, we add the following code to the ContextSnapshot.ConnectionString setter after the line which constructs the connection string builder:

if ((csBuilder.Name != null) && (csBuilder.Name != ""))

{

// lookup the actual connection string from the config file

string connectionString = ConfigurationManager.ConnectionStrings[csBuilder.Name].ConnectionString;

csBuilder = new EntityConnectionStringBuilder(connectionString);

}

The project file on code gallery includes this fix. I’ve also had some questions from folks who were having trouble getting a WCF service working using EntityBag, so I included a sample service as well.

Creating an EntityBag-based Service

I’m no WCF expert, but I’ve been able to muddle my way through. It seems like it might be useful to point out some of the trickier steps involved in putting together the sample:

Known types for the service methods are an issue—especially since EntityBag is generic and can contain a graph of many entity types. This is something which will be addressed automatically in the next release, but for now this is worked around with a static method (GetKnownEntityTypes) and an operation contract attribute which points to it as described in this previous post.
If you create your entities in a separate DLL like I’ve done in the sample, then you need to copy the connection string from the model.dll’s app.config file to the web.config for the service.
If you are going to use regular metadata files, you need to make sure they are in the right directory--otherwise, you can embed them as resources, but then you need to deal with a beta 3 bug by building with the metadata not embedded first and then rebuilding set to embed.
One common and frustrating problem is that it’s easy for the size of the message to grow beyond the default maximum message size. This can be addressed by changing the maxReceivedMessageSize in the app.config for the client and in the web.config for the service.
Finally, it’s important to make sure the service is set to re-use referenced types. This is turned on by default with VS 2008, but it’s worthwhile to make sure that it is on, and of course you need to reference system.data.entity.dll, perseus and your model on the client before adding the service reference.

Future Directions

With all of this we have a decent proof of concept, but clearly there’s a lot of room for additional improvements. Here are a few ideas both to give you an idea of current limitations and future possibilities:

· It needs to be tested against a wider array of models.

o Entities with complex types will likely present problems because there are currently limitations in the CurrentValues and OriginalValues records on ObjectStateEntry when dealing with properties that are complex types.

o Associations with referential integrity constraints likely need careful ordering which the system doesn’t yet impose.

· There are various limitations due to bugs or not yet implemented features in the Entity Framework.

o Entities used with EntityBag must implement IEntityWithKey. Classes generated by the system implement this interface, but IPOCO classes are not required to implement it. Unfortunately, an API change in beta 3 makes it impossible to do certain operations in a general-purpose component unless the entities in question implement the interface. A future release will remedy the problem, and it will be possible to remove the requirement.

o Currently the system uses reflection to set the EntityKey property on IRelatedEnd when the end in question is an EntityReference. Future releases will remove that requirement. For the most part this isn’t an issue, but it could create partial trust issues—I haven’t yet had a chance to investigate fully.

· The message size could be further optimized.

o Right now the system can’t distinguish between the serialization used when transmitting from the service to the client and that used when transmitting from the client back to the service. This causes unnecessarily large messages because when transmitting to the client you clearly want to include everything, but when transmitting back to the service in most cases you could skip all unchanged entities.

o The component could also take advantage of work underway for the Entity Data Source Control to store and transmit only the minimum set of original values which the system needs rather than a whole copy of the original entity. Of course in some cases you will need the whole entity anyway because business logic may prevent constructing an entity with only this minimum set of properties, but in many cases this would be a useful optimization.

· Additional steps could be taken to strengthen a web service’s contract.

o Events or some other mechanism could be added to allow validation when an EntityBag is passed from the client back to the service for updates.

o Even better, code could be generated to create a stronger contract and just use EntityBag as an internal storage/serialization mechanism.

I’m sure there are other things we could do as well, but that’s probably enough to set you thinking. As always, if you have questions or ideas, please don’t hesitate to share them.

- Danny

EntityBag Part VI – RelationshipEntry

dsimmons@microsoft.com — Thu, 24 Jan 2008 22:52:00 GMT

Here’s the last piece in the EntityBag saga. RelationshipEntry is a small, DataContract serializeable class which wraps an ObjectStateEntry that represents a pair of related entities. It contains the name of the relationship, the state of the entry, and the key or index of the entity for each end organized by the role of that entity in the relationship. In addition to acting as a serializable container, this class provides methods which simplify applying the relationship to a context.

Fields and Properties

[DataContract]

internal class RelationshipEntry

{

// serializable properties

[DataMember] string RelationshipName { get; set; }

[DataMember] EntityState State { get; set; }

[DataMember] string Role1 { get; set; }

[DataMember] EntityKey Key1 { get; set; }

[DataMember] int AddedEntityIndex1 { get; set; }

[DataMember] string Role2 { get; set; }

[DataMember] EntityKey Key2 { get; set; }

[DataMember] int AddedEntityIndex2 { get; set; }

// non-serializable properties

IEntityWithRelationships Entity1 { get; set; }

IEntityWithRelationships Entity2 { get; set; }

Constructor

The interesting part of this constructor is the fact that it requires a Dictionary mapping from EntityKey to an index be passed in containing entries for any entities which are in the added state. This dictionary is created during the construction of the ContextSnapshot and the index is the position with the snapshot’s list of added entities. As described in previous posts, temp keys are unique based on object identity rather than values in the key, so their identity is lost in serialization and we have to use this other mechanism to make sure the identity is preserved.

// constructor

internal RelationshipEntry(ObjectStateEntry stateEntry, ObjectContext context,

Dictionary<EntityKey,int> addedEntityKeyToIndex)

{

Debug.Assert(stateEntry.IsRelationship);

this.RelationshipName = stateEntry.EdmType().FullName;

this.State = stateEntry.State;

this.Role1 = stateEntry.UsableValues().GetName(0);

this.Key1 = (EntityKey)stateEntry.UsableValues().GetValue(0);

if (context.GetEntityState(this.Key1) == EntityState.Added)

{

this.AddedEntityIndex1 = addedEntityKeyToIndex[this.Key1];

this.Key1 = null;

}

this.Role2 = stateEntry.UsableValues().GetName(1);

this.Key2 = (EntityKey)stateEntry.UsableValues().GetValue(1);

if (context.GetEntityState(this.Key2) == EntityState.Added)

{

this.AddedEntityIndex2 = addedEntityKeyToIndex[this.Key2];

this.Key2 = null;

}

ResolveEntitiesAndKeys

This is a private method used in each of the public methods after the relationship entry has deserialized to lookup the keys and entities for the relationship (takes care of the added entity indexes, etc.).

void ResolveEntitiesAndKeys(ObjectContext context, List<IEntityWithKey> addedEntities)

{

if (this.Key1 == null)

{

this.Entity1 = (IEntityWithRelationships)addedEntities[this.AddedEntityIndex1];

this.Key1 = ((IEntityWithKey)this.Entity1).EntityKey;

}

else

{

this.Entity1 = (IEntityWithRelationships)context.GetEntityByKey(this.Key1);

}

if (this.Key2 == null)

{

this.Entity2 = (IEntityWithRelationships)addedEntities[this.AddedEntityIndex2];

this.Key2 = ((IEntityWithKey)this.Entity2).EntityKey;

}

else

{

this.Entity2 = (IEntityWithRelationships)context.GetEntityByKey(this.Key2);

}

Methods

The three methods for use by calling classes will attach, add or delete a relationship using the relationship manager on one of the related entities. The framework will automatically handle fixup for the other end.

// methods

internal void AddRelationship(ObjectContext context, List<IEntityWithKey> addedEntities)

{

Debug.Assert(this.State == EntityState.Added);

ResolveEntitiesAndKeys(context, addedEntities);

IRelatedEnd relatedEnd = this.Entity1.RelationshipManager.GetRelatedEnd(this.RelationshipName,

this.Role2);

if (!relatedEnd.Contains(this.Key2))

{

relatedEnd.Add(this.Entity2);

}

internal void AttachRelationship(ObjectContext context)

{

Debug.Assert((this.State == EntityState.Deleted) || (this.State == EntityState.Unchanged));

// Unchanged and deleted relationships cannot involve added entities, so no need for

// addedEntities list.

ResolveEntitiesAndKeys(context, null);

IRelatedEnd relatedEnd = this.Entity1.RelationshipManager.GetRelatedEnd(this.RelationshipName,

this.Role2);

if (!relatedEnd.Contains(this.Key2))

{

relatedEnd.Attach(this.Entity2);

}

internal void DeleteRelationship(ObjectContext context)

{

Debug.Assert(this.State == EntityState.Deleted);

// DeletedRelationships cannot involve added entities, so no need for addedEntities list

ResolveEntitiesAndKeys(context, null);

IRelatedEnd relatedEnd = this.Entity1.RelationshipManager.GetRelatedEnd(this.RelationshipName,

this.Role2);

if (relatedEnd.Contains(this.Key2))

{

relatedEnd.Remove(this.Entity2);

}

Not too bad is it? I guess this is a testament to the power of decomposing a problem. If each piece is small enough and cohesive enough, the overall story can become much less complicated.

- Danny

EntityBag Part V – ContextSnapshot Constructing and Applying

dsimmons@microsoft.com — Wed, 23 Jan 2008 20:58:00 GMT

For this post, I’m going to set a new personal record for least prose/most code. We’re going to look at the core of ContextSnapshot, and the code includes extensive comments so we’ll mostly let it speak for itself.

Constructors

We have two different constructors because in some scenarios with EntityBag it’s most convenient to construct the snapshot by just passing an ObjectContext, while in others the connection string in the ObjectContext has been cleared, so it’s necessary to explicitly pass the connection string in addition to the context. Regardless of the passed in parameters, the core piece of the constructor creates the lists of entities and relationships by state. The code below makes extensive use of the methods I described previously in my Extension Methods Extravaganza posts and in my posts about creating original values objects.

// constructors

public ContextSnapshot(ObjectContext context) : this(context, context.Connection.ConnectionString)

{

}

public ContextSnapshot(ObjectContext context, string connectionString)

{

this.ConnectionString = connectionString;

// For unchanged entities we only need the current values.

this.unchangedEntities = new List<IEntityWithKey>(context.GetEntities(EntityState.Unchanged));

// For modified entities we need current plus the original values object along with EntityReferences

// (the addition of references to the original values object here is critical since this is how

// 1-1 and 1-many relationships are serialized for modified entities since it's the original values

// object which is attached on modified entities. In the case of deleted entities, the

// relationships don't exist, and in the cases of added or unchanged entities the current values

// objects bring along the relationships and they are what are added or attached.

this.modifiedEntities = new List<IEntityWithKey>();

this.modifiedOriginalEntities = new List<IEntityWithKey>();

foreach (var entity in context.GetEntities(EntityState.Modified))

{

this.modifiedOriginalEntities.Add((IEntityWithKey)context

.CreateOriginalValuesObjectWithReferences(entity));

this.modifiedEntities.Add(entity);

}

// For deleted, just the original values.

this.deletedEntities = new List<IEntityWithKey>();

foreach (var entity in context.GetEntities(EntityState.Deleted))

{

this.deletedEntities.Add((IEntityWithKey)context.CreateOriginalValuesObject(entity));

}

// For added entities we need the current values, plus we need to make an index mapping from

// the key to where the entity lands in the added entities list so that we can serialize those

// index values where we would have serialized the added entity key.

this.addedEntities = new List<IEntityWithKey>();

this.addedEntityKeyToIndex = new Dictionary<EntityKey, int>();

int index = 0;

foreach (var entity in context.GetEntities(EntityState.Added))

{

this.addedEntities.Add(entity);

this.addedEntityKeyToIndex.Add(entity.EntityKey, index);

index++;

}

// Also serialize lists of added, deleted relationships.

this.addedRelationships = new List<RelationshipEntry>();

foreach (var stateEntry in context.GetRelationships(EntityState.Added))

{

this.addedRelationships.Add(new RelationshipEntry(stateEntry, context,

this.addedEntityKeyToIndex));

}

this.deletedRelationships = new List<RelationshipEntry>();

foreach (var stateEntry in context.GetRelationships(EntityState.Deleted))

{

this.deletedRelationships.Add(new RelationshipEntry(stateEntry, context,

this.addedEntityKeyToIndex));

}

// Serialize unchanged many-to-many relationships

this.unchangedManyToManyRelationships = new List<RelationshipEntry>();

foreach (var stateEntry in context.GetUnchangedManyToManyRelationships())

{

this.unchangedManyToManyRelationships.Add(new RelationshipEntry(stateEntry, context,

this.AddedEntityKeyToIndex));

}

ApplyToContext

The final piece of this class is the method which will take the lists of entities and relationships that the constructor created and play them back into a context. Original values are attached, added entities added, modified entities have their property changes “applied” and deleted entities are deleted. Similar operations are performed for explicitly added and deleted relationships.

public void ApplyToContext(ObjectContext context)

{

// Attach all the unchanged entities.

foreach (IEntityWithKey entity in this.unchangedEntities)

{

context.Attach(entity);

}

// Attach & apply changes to modified entities

foreach (IEntityWithKey entity in this.modifiedOriginalEntities)

{

context.Attach(entity);

}

foreach (IEntityWithKey entity in this.modifiedEntities)

{

context.ApplyPropertyChanges(entity.EntityKey.EntityContainerName + "." +

entity.EntityKey.EntitySetName, entity);

}

// Add entities & relationships

foreach (IEntityWithKey entity in this.addedEntities)

{

// have to remove the entity key from the object so add will succeed

string fullEntitySetName = entity.EntityKey.GetFullEntitySetName();

entity.EntityKey = null;

context.AddObject(fullEntitySetName, entity);

}

foreach (RelationshipEntry re in this.addedRelationships)

{

re.AddRelationship(context, this.addedEntities);

}

// to deal with deleting entities & relationships properly, we need a special dance:

// - attach all the deleted entities & the deleted relationships

// - delete the entities

// - delete the relationships

foreach (IEntityWithKey entity in this.deletedEntities)

{

context.Attach(entity);

}

foreach (RelationshipEntry re in this.deletedRelationships)

{

re.AttachRelationship(context);

}

foreach (IEntityWithKey entity in this.deletedEntities)

{

context.DeleteObject(entity);

}

foreach (RelationshipEntry re in this.deletedRelationships)

{

re.DeleteRelationship(context);

}

foreach (RelationshipEntry re in this.unchangedManyToManyRelationships)

{

re.AttachRelationship(context);

}

// Clear lists. Once a snapshot has been applied, you cannot re-apply it

// since the entities have been attached.

this.unchangedEntities = null;

this.modifiedOriginalEntities = null;

this.modifiedEntities = null;

this.deletedEntities = null;

this.addedEntities = null;

}

That’s it. Hopefully you’ll find this all clear—if not, don’t hesitate to ask some questions. Next post we’ll wrap up EntityBag with a look at the RelationshipEntry class.

- Danny

EntityBag Part IV – ContextSnapshot Fields and Properties

dsimmons@microsoft.com — Tue, 22 Jan 2008 21:34:00 GMT

Well, we’re moving right along—at the fourth part in this series already. I imagine, though, that some of you may be chomping at the bit because you’ve waded through three posts worth of my ramblings without really getting to the hard part yet. At the end of the day we’re basically looking at how to serialize the ObjectContext, and in my last post I mentioned that EntityBag just delegates responsibility for that to another class. Well, in this post we’re going to start looking at ContextSnapshot which is DataContract serializable and can transport the entire contents of an ObjectContext over the wire. (On a side note, have any of you spent much time working in a Literate Programming system—something like Donald Knuth’s Web? If so, maybe this series of posts with code and prose intermingled will generate some of the same nostalgia for you that it does for me. <grin>)

The DataMembers and Other Fields

Essentially the way we manage serialization of the context is just by transporting a number of lists of entities and of relationships organized by states. For unchanged and added entities, only the current values are kept. With deleted entities, we keep the original values only, and with modified entities of course we need both an original and a current version. In each case we automatically serialize the EntityKey of related entities if the relationship is an EntityReference and this ends up bringing along enough information for all 1-1 and 1-many relationships. For many-to-many relationships, however, we have to explicitly serialize the relationship info as we need also to do for added and deleted relationships so that we can effectively recreate the changes. Beyond these lists, we also serialize the connection string which is required when it later comes time to construct a local ObjectContext for change tracking, and the class maintains a dictionary which maps from the EntityKey of added entities to their index in the added entity list. This dictionary is not serialized, but is used to help with constructing anything which must reference added entities in a way that is preserved across serialization (like relationship entries or the root entity for EntityBag).

[DataContract]

public class ContextSnapshot

{

// serialized members

[DataMember] string connectionString;

[DataMember] List<IEntityWithKey> unchangedEntities;

[DataMember] List<IEntityWithKey> modifiedEntities;

[DataMember] List<IEntityWithKey> modifiedOriginalEntities;

[DataMember] List<IEntityWithKey> deletedEntities;

[DataMember] List<IEntityWithKey> addedEntities;

[DataMember] List<RelationshipEntry> addedRelationships;

[DataMember] List<RelationshipEntry> deletedRelationships;

[DataMember] List<RelationshipEntry> unchangedManyToManyRelationships;

// non-serialized members

Dictionary<EntityKey, int> addedEntityKeyToIndex;

Public Properties and Related Method

One interesting bit here is that when the ConnectionString property is set on the context snapshot we clear the provider connection string portion of it for a few reasons: First, it may contain sensitive information like a username and password which we don’t want travelling over the web service or available on the client. Secondly, this part of the overall connection string is not needed to successfully construct the context (what is needed is the location of the metadata and the name of the store provider). Finally, even if we were willing to send that information to the client, the expectation is that the client context won’t have line-of-sight to the database anyway, so it’s better to clear the connection string and get an immediate error if any operation is attempted on the client which would require a connection to the database.

// properties

public string ConnectionString

{

get

{

return this.connectionString;

}

private set

{

// Clear the store provider connection string because it could contain a password

// or something else we don't want to send to the client.

var csBuilder = new EntityConnectionStringBuilder(value);

csBuilder.ProviderConnectionString = null;

this.connectionString = csBuilder.ConnectionString;

}

public Dictionary<EntityKey, int> AddedEntityKeyToIndex

{

get

{

return this.addedEntityKeyToIndex;

}

public IEntityWithKey GetAddedEntity(int addedEntityIndex)

{

return this.addedEntities[addedEntityIndex];

}

That sets the stage for the core logic of ContextSnapshot which is the subject of the next installment in this series.

- Danny

EntityBag Part III – Public Surface and Serialization

dsimmons@microsoft.com — Mon, 21 Jan 2008 21:24:00 GMT

Now that we can construct an EntityBag, we’ll continue pulling apart the top level class.

Public Surface

First, there are the public properties. The Mode is trivial, but in the case of the Root property, there are a couple of interesting parts: One is that retrieving the Root will automatically move the mode from Deserialized to LocalContext, and the other is that while EntityBag exposes a root entity, the backing field for the property is just the key not the whole entity—that way we don’t inadvertently serialize the root entity twice.

public Mode Mode { get; private set; }

public T Root

{

get

{

if (this.Mode == Mode.Deserialized)

{

SwitchToLocalContext();

}

return (T) context.GetEntityByKey(this.rootKey);

}

internal set

{

this.rootKey = value.EntityKey;

}

The SwitchToLocalContext method creates a private ObjectContext instance and plays changes from the ContextSnapshot into it. The other interesting bit is how keys for entities in the added state are serialized—because added entities always have temp keys, those keys don’t serialize well—all that comes across is the entity set name. So, for EntityBag we serialize added entities in an ordered list, and when we need to serialize a reference to a particular added entity we just store an index into that list.

void SwitchToLocalContext()

{

Debug.Assert(this.Mode == Mode.Deserialized);

this.context = new ObjectContext(this.snapshot.ConnectionString);

this.snapshot.ApplyToContext(this.context);

// Set the root key if root entity was added (in which case the root key is null)

if (this.rootKey == null)

{

this.Root = (T)this.snapshot.GetAddedEntity(this.rootAddedEntityIndex);

}

this.Mode = Mode.LocalContext;

}

The Delete method is fairly straightforward—the only trick being to account for the mode:

public void Delete(object entity)

{

switch (this.Mode)

{

case Mode.Deserialized:

SwitchToLocalContext();

break;

case Mode.LocalContext:

break;

default:

throw new InvalidOperationException("Can't delete when Mode == Constructed || Unwrapped.");

}

this.context.DeleteObject(entity);

}

UnwrapInto is a little trickier but the comments explain things fairly well.

public void UnwrapInto(ObjectContext context)

{

switch(this.Mode)

{

case Mode.Deserialized:

// This is the normal case for unwrap into -- just apply the snapshot

this.snapshot.ApplyToContext(context);

break;

case Mode.LocalContext:

// If we are operating on the local context, then we need to take a new snapshot,

// detach things that will conflict and then apply.

this.snapshot = new ContextSnapshot(this.context);

// Detach unchanged and added entities from the internal context so the snapshot

// can be applied elsewhere. Don't have to worry about modified entities because

// only the modifiedOriginal versions are attached during the apply, and those

// have been synthesized from the original values--they aren't attached.

foreach (var entity in this.context.GetEntities(EntityState.Unchanged))

{

this.context.Detach(entity);

}

foreach (var entity in this.context.GetEntities(EntityState.Added))

{

// Detaching an added entity will clear its key -- so save before detaching

// and then restore. The key is needed since it is what stores the entityset.

EntityKey key = entity.EntityKey;

this.context.Detach(entity);

entity.EntityKey = key;

}

this.snapshot.ApplyToContext(context);

this.snapshot = null;

break;

default:

// mode == constructed || unwrapped

throw new InvalidOperationException("Can only unwrap if Mode == Deserialized || LocalContext.");

}

// If we make it this far, then we have successfully unwrapped.

this.snapshot = null;

this.Mode = Mode.Unwrapped;

}

Serialization

When it comes to serialization, the EntityBag class is pretty straightforward because we delegate the hard parts to the ContextSnapshot class which we will examine in a future post. The first things to look at are the fields in the class... Given the previous method explanations, these should all be fairly obvious.

[DataMember] ContextSnapshot snapshot;

[DataMember] EntityKey rootKey;

[DataMember] int rootAddedEntityIndex;

ObjectContext context;

string connectionString;

The more interesting part is the custom code we add which gets run immediately before serialization and immediately after de-serialization. This code basically has two purposes: One is to deal with the multi-mode nature of the class by preparing the snapshot if it hasn’t already been taken, setting the mode to Deserialized after de-serializaiton, etc. The other is to handle serializing the added entity index for the root instead of just the EntityKey if the root entity is in the added state.

[OnSerializing]

void OnSerializing(StreamingContext streamingContext)

{

switch(this.Mode)

{

case Mode.Unwrapped:

throw new InvalidOperationException("Cannot serialize when Mode == Unwrapped.");

case Mode.Constructed:

this.snapshot = new ContextSnapshot(this.context, this.connectionString);

this.PrepareRootEntityForSerialization();

break;

case Mode.LocalContext:

this.snapshot = new ContextSnapshot(this.context);

this.PrepareRootEntityForSerialization();

break;

default:

// Mode.Deserialized -- nothing to do

break;

}

void PrepareRootEntityForSerialization()

{

if (this.context.ObjectStateManager.GetObjectStateEntry(this.rootKey).State == EntityState.Added)

{

this.rootAddedEntityIndex = this.snapshot.AddedEntityKeyToIndex[this.rootKey];

this.rootKey = null;

}

[OnDeserialized]

void OnDeserialized(StreamingContext streamingContext)

{

this.Mode = Mode.Deserialized;

}

That’s basically it for the EntityBag class itself. I know it’s a little hard to see the class as a whole when we’re looking at all the separate pieces, so once we’ve completed our survey of the whole system, I’ll see what I can do to make a zip file or something available with the source all in one place for your perusal. In the mean time, next post we’ll move on to looking at ContextSnapshot.

- Danny

EntityBag Part II – Modes and Constructor

dsimmons@microsoft.com — Mon, 21 Jan 2008 00:34:00 GMT

In my last post I described the way the EntityBag class can be used. This time around we’ll take a look at the implementation of the top-level EntityBag class itself, and in subsequent posts we’ll dig into some supporting classes.

The first thing to understand about EntityBag is the fact that it has several different modes of operation. We need this one class as the bridge between mid-tier and client operations, and as such it has to operate different ways in different situations. Normally I try to avoid classes with modes since it’s easier to know what to expect of a class if it basically operates the same way regardless of its state. Further, if class does have modes I much prefer the transitions between modes to be very explicit. That said, after a fair amount of thought I came to the conclusion that the standard programming patterns for using EntityBag would just be a lot cleaner and easier if it had different modes of operation and switched between them automatically. Once that was decided, I figured the least that I could do to help would be to create an enum for the mode and use it both to add clarity to the implementation and to expose publicly what mode the class is in so that calling code can operate differently depending on the mode if necessary.

EntityBag has four modes:

public enum Mode

{

Constructed,

Deserialized,

LocalContext,

Unwrapped

}

When it is first constructed (typically on the mid-tier after retrieving some data to return to the client), it is in the “Constructed” mode. In this mode, the bag maintains a reference to an ObjectContext which is owned by the calling code, and as such the Delete and UnwrapInto methods are not allowed. In a typical scenario, the next step for the bag is that it is serialized, transported across a web service and then deserialized on the other side ending up in “Deserialized” mode. When in this mode, the bag has no ObjectContext—instead the entities and their state are stored in a DataContract compatible ContextSnapshot object. Also, when in this mode an attempt to access the Root entity or to call the Delete method will cause the bag to automatically switch to the next mode: LocalContext. The “LocalContext” mode is where the bag creates its own ObjectContext instance and replays the ContextSnapshot into it. The normal use in this mode is to make changes to the EntityGraph and have them automatically tracked by that local context, once these changes are complete, the bag will be serialized, transported back to the mid-tier and returned to Deserialized mode. Normally the next operation would be to UnwrapInto a context created in the mid-tier outside the bag—this operation pushes all of the changes into that context and in the process transfers ownership of the entities so that they may no longer be change tracked by the bag, etc. So the bag has a final “Unwrapped” mode which is something like being disposed. Once a bag is unwrapped it basically will no longer function at all.

The next interesting decision about EntityBag was its construction pattern. In order to create a new EntityBag, you need three things: an ObjectContext, the type of the Root object and the Root object instance. So I decided to make the class generic (to track the type) and to make the construction pattern a factory extension method to ObjectContext which takes the root object as a parameter. The result looks like this:

public static class EntityBagExtensionMethods

{

// This public extension method is what you use to create an EntityBag:

// var bag = ctx.CreateEntityBag<MyType>(myObject);

public static EntityBag<T> CreateEntityBag<T>(this ObjectContext context, T root) where T : IEntityWithKey

{

return new EntityBag<T>(context, root);

}

[DataContract]

public class EntityBag<T> where T : IEntityWithKey

{

internal EntityBag(ObjectContext sourceContext, T root)

{

this.context = sourceContext;

// NOTE: We need to capture the connection string now so we'll have it when serializing--in many

// scenarios the connection will be closed and the connection string lost by the time we serialize.

this.connectionString = sourceContext.Connection.ConnectionString;

this.Root = root;

this.Mode = Mode.Constructed;

}

OK. There’s a start. In my next post we’ll look at the rest of the public surface and how serialization works.

- Danny

EntityBag Part I – Goals

dsimmons@microsoft.com — Sun, 20 Jan 2008 03:04:00 GMT

Well, I guess it’s high time that I get down to business of sharing and explaining the “general purpose container object” for transporting graphs of entities along with change tracking information over WCF web services which I mentioned in this previous post. As it turns out, there’s a fair amount of code involved, so we’ll build up a series of routines / classes until we can get to the top-level object which I call EntityBag<T>.

To start us off, though, let’s take a look at the top-level interface for EntityBag and how it would be used. The goal is to write a couple of mid-tier web methods: one which will retrieve a graph of entities in a single call and instantiate them on the client in full graph form, plus one which will take a modified graph back and persist the changes to the database while honoring the optimistic concurrency contract based on the original values retrieved and without requiring extra round trips to the DB in order to save. To make the whole thing work, we need:

1) A way to send the entire graph in spite of the fact that the EF and WCF by default only shallowly serialize entities (that is related entities are not automatically serialized).

2) Something which will track changes on the client—both changes to regular properties of the entities as well as changes to the graph (new entities, deleted entities, modified relationships).

3) A serialization format that can include not only the current values of an entity but original values and information about the state of the entities, etc.

The strategy I adopted for EntityBag is to create a DataContract serializable object which will effectively transmit an entire ObjectStateManager and to identity a single object as the “root” of the graph. This way on the mid-tier a graph of related entities can be retrieved from the database into an ObjectContext, an EntityBag may be constructed to hold that context, and then that bag can be serialized to the client. On the client, the EntityBag will expose a single public “Root” property and internally will create a private ObjectContext which is used both to help reconstruct the graph and to transparently track changes. When it’s time to persist the changes, the EntityBag can be serialized back to the mid-tier and the updated entity graph along with all original values can be reconstructed into a mid-tier context which will then be used to save the changes to the DB.

Code for the web methods might look something like this:

public EntityBag<Room> GetRoomAndExits(string roomName)

{

using (DPMudDB db = DPMudDB.Create())

{

roomName = '%' + roomName + '%';

var room = db.Rooms.Where("it.Name like @name", new ObjectParameter("name", roomName))

.First();

room.Exits.Load();

return db.CreateEntityBag<Room>(room);

}

public void UpdateRoomAndExits(EntityBag<Room> bag)

{

using (DPMudDB db = DPMudDB.Create())

{

bag.UnwrapInto(db);

db.SaveChanges();

}

As you can see, the code in these two methods is uncluttered by serialization considerations—just one call (CreateEntityBag) in the first method and one (UnwrapInto) in the second method encapsulate the plumbing. The client side is similarly straight-forward. Essentially all that is necessary is to use the Root property on the EntityBag to access the graph:

EntityBag<Room> bag = client.GetRoomAndExits("Calm");

foreach (Exit e in bag.Root.Exits)

{

Console.WriteLine("\t" + e.Name);

e.Name .= "***";

}

bag.Delete(bag.Root.Exits.First());

var newRoom = new Room();

newRoom.Name = "NotTheRoomYouAreLookingFor";

var newExit = new Exit();

newExit.Name = "NotTheExitYouAreLookingFor";

newExit.TargetRoom = newRoom;

bag.Root.Exits.Add(newExit);

client.UpdateRoomAndExits(bag);

While I like the simplicity of this interaction, it is super important to keep in mind the restrictions imposed by this approach. First off, there’s the fact that this requires us to run .Net and the EF on the client—in fact it requires that the code for your object model be available on the client, so it is certainly not interoperable with Java or something like that. Secondly, because we are sending back and forth the entire ObjectContext, the interface of the web methods imposes no real contract on the kind of data that will travel over the wire. The retrieval method in our example is called GetRoomAndExits, but there’s absolutely no guarantee that the method might not return additional data or even that it will return a room and exits at all. This is even scarier for the update method where the client sends back an EntityBag which can contain any arbitrary set of changes and they are just blindly persisted to the database. The ease of use we have achieved comes at a very high price. Of course if you are writing web services that transport instances of the DataSet, you already live in that world, and for some scenarios this might be acceptable.

OK. Now that I have your attention, in future posts we can dig into the implementation of EntityBag.

- Danny

When deserialize(serialize(x)) != x

dsimmons@microsoft.com — Thu, 10 Jan 2008 04:53:00 GMT

So here’s another random little piece of information that might help someone else out (or me 6 months from now)… As I’ve been rumbling about, lately I’ve been playing quite a bit with webservices and serialization—especially DataContract serialization—and in one of my recent projects I was really fighting with what appeared to be a very simple unit test. The test was designed to verify that a particular structure (guess what structure…a graph of entities) would serialize properly across a webservice. As a simple first approximation, I thought I’d write a little routine which would take an object serialize it with the DataContract serializer and then immediately deserialize it so that I could compare the results to the original. This is quicker and easier than building out the full web service, so it seemed like a good idea. In fact, the idea got even better when I found some code laying around which someone else had written just for this purpose. The code looked something like this:

public static string Serialize<T>(T o)

{

return Serialize<T>(o, new Type[0]);

}

public static string Serialize<T>(T o, IEnumerable<Type> knownTypes)

{

DataContractSerializer dcs = new DataContractSerializer(typeof(T), knownTypes);

StringBuilder sb = new StringBuilder();

XmlWriter writer = XmlWriter.Create(sb);

dcs.WriteObject(writer, o);

writer.Close();

string xml = sb.ToString();

return xml;

}

public static T Deserialize<T>(string xml)

{

return Deserialize<T>(xml, new Type[0]);

}

public static T SerializeAndDeserialize<T>(T o)

{

return Deserialize<T>(Serialize<T>(o));

}

Beautiful… Now all I have to do is call the generic method SerializeAndDeserialize and I will get back a new object (or ideally a graph of objects) which should be an accurate copy. At least as accurate as the serialization will be, right? WRONG.

Everything was working fine until I happened to run this little baby on some data that had strings with embedded carriage returns (that’s \r for us c# dudes—in my case the data was RTF). Suddenly my comparisons were failing, and it took me quite some time to realize that it had nothing to do with the serializer or the code I was testing. The problem was in the above code. Apparently if you serialize using an XmlWriter over a StringBuilder, these carriage returns are lost. If you actually build a web service with WCF, though, everything goes through fine. ARRRGGGG.

The fix? Use a memory stream instead of XmlWriter/StringBuilder. So the updated code looks like this:

public static Stream Serialize<T>(T o, IEnumerable<Type> knownTypes)

{

DataContractSerializer dcs = new DataContractSerializer(typeof(T), knownTypes);

MemoryStream stream = new MemoryStream();

dcs.WriteObject(stream, o);

return stream;

}

public static T Deserialize<T>(Stream stream)

{

return Deserialize<T>(stream, new Type[0]);

}

Other methods look the same. Wouldn’t you know it, the new and improved Serialize method is even shorter and simpler than the old one.

Here’s hoping the next guy finds this post instead of banging their head against the wall for 3 or 4 hours like me. ;-)

- Danny

Filtered association loading and re-creating an entity graph across a web service boundary...

dsimmons@microsoft.com — Sat, 22 Dec 2007 02:13:00 GMT

Roger Jennings, in his recent post Controlling the Depth and Order of EntitySets for 1:Many Associations, makes a case for the importance of two features in an O/RM if you want to build data-centric web services using it: the ability to do a filtered relationship navigation, and the ability to either serialize a graph or at least re-create the graph on the other side of a web service boundary. Well, to be fair, he probably is asking for more than these two features, but there are two key things which are possible with the entity framework today (as of beta 3). So I thought I'd take a few minutes to explain how. I’m going to take these two topics out of order, though, because there are some important concepts related to re-creating an entity graph which I hope will help clarify things when it comes to filtered association loading.

Re-creating an entity graph across a web service boundary
While this is by no means a simple problem, for an important sub-set of the overall scenarios it’s pretty easy to solve this with the entity framework today. The thing which makes this possible is the beta 3 feature of serializable EntityReferences. In my previous post about Entity Relationship Concepts, I gave some background about relationship “stub” entries in the object state manager, and these are effectively what gets serialized when an EntityReference is serialized. Stubs are very versatile because they automatically upgrade to real entities either if the real entity is already present when the system tries to load a stub into the state manager or if a stub is present and a real entity is loaded. When you add to this the fact that the framework ensures that the state manager and its entries are automatically kept in sync with the collections and references on the objects, the result is a pretty simple mechanism for breaking a graph into pieces and then cleanly reassembling the whole.

Maybe a picture will help:

So, if you want to remote an object graph across a web service and you do not have many-many relationships, then you can just remote each of the entities which participate in the object graph individually (maybe just as an array of objects) and then attach all of them to an ObjectContext on the other side at which point the EF will recreate the graph for you. This is, in fact, part of how my general purpose container sample (promised as an illustration of strategy #4 in this post but not quite yet ready for sharing) works.

Filtered Association Loading
While it might be nice if there were a way to pass a predicate to either the Load method on EntityCollection and EntityReference or to the Include method on ObjectQuery, it is possible to accomplish these things today. If you want to filter a relationship load, one way to do it is with EntityCollection’s Attach method. As a simple example, if I wanted to load into a customer’s orders collection all orders with date > January 1, 2007, then I could do something like this:

customer.Orders.Attach(customer.Orders.CreateSourceQuery().Where(order => order.Date >= new DateTime(2007, 1, 1)));

The CreateSourceQuery method returns an ObjectQuery<T> which will retrieve the set of entities that Load would retrieve, I then refine that query to filter to the subset of orders we really want and call Attach which tells the collection to incorporate the retrieved orders as though they were Loaded.

Another interesting trick is the fact that object services automatically rewrites ObjectQueries (except those with merge option of NoTracking) to automatically bring along the EntityKey for EntityReferences and then creates the relationship entries and stubs as needed when the results of the query are attached to the context. What this means in practice is that the above operation could also have been accomplished without the attach call just by creating the filtered query and enumerating its results—as the results are enumerated, objects are materialized and attached to the state manager and (like in the diagram above) the orders bring along the keys of their corresponding customers and cause the graph to be fixed up to match.

What this means for the include statement is that there are multiple ways to load a set of customers with a filtered set of related orders:

1) You could query the customers and as you iterate over each one you could use the trick above to query the filtered set of orders for that customer and load them in. This would, unfortunately, result in n+1 queries (where n is the number of customers).

foreach(Customer c in db.Customers)

{

c.Orders.Attach(c.Orders.CreateSourceQuery().Where(o => o.Date >= new DateTime(2007, 1, 1)));

// do stuff

}

2) You could query the customers and then once you were done iterating over all of them you could just query for all orders since January 1, 2007. This would require only 2 queries, and it would recreate the graph for you automatically. The only downside is that if you filtered to a subset of all the customers in your first query, then you would have to apply a similar filter to your orders query or else you would retrieve orders for customers you weren’t interested in.

var customers = new List<Customer>(db.Customers);

var orders = new List<Order>(db.Orders.Where(o => o.Date >= new DateTime(2007, 1, 1)));

foreach (Customer c in customers)

{

// do stuff

}

3) You could create a projection query which retrieved customers in the first column and collections of orders for that customer in the second column and then attach the collection in the second column to the collection property on the customers. This would use only one query.

foreach (var customerAndOrders in from c in db.Customers

select new

{

Customer = c,

Orders = (from o in c.Orders

where o.Date >= new DateTime(2007, 1, 1)

select o)

})

{

customerAndOrders.Customer.Orders.Attach(customerAndOrders.Orders);

// do stuff

}

Well, it’s growing late in the afternoon on the Friday before Christmas, and my family is calling me home. So I’ll leave you with this. Here’s hoping that all of you have a wonderful holiday and new year!

- Danny

So they're hard, but what if I need them...

dsimmons@microsoft.com — Thu, 20 Dec 2007 09:45:00 GMT

In my last post I started a survey of problems with building data-centric web services. When we left our hero (you my intrepid entity framework programmer) things were looking pretty bleak. So far we’ve talked about the challenges. Now let’s talk about some possible approaches for dealing with those challenges.

1) The traditional SOA / DTO approach. This is decidedly more work but also the most flexible and for many users the best option. It allows you to impose control at the web service juncture, and it makes true interoperability possible. With some patterns this isn’t as hard as it sounds at first, but it is more work—you have to maintain the objects which represent your contract as well as the objects which are in your data model, but the objects which represent your contract can be thin wrappers that don’t re-implement your business logic; they just form a tree of the separate entities (which does serialize nicely unlike graphs) with the required granularity, and they constrain the operations to those you want to allow over the web service.

At another extreme you can:

2) Constrain your scenarios and use standardized operations. This is the approach that the REST community takes, and it’s what we’re doing with project Astoria (aka ADO.Net Data Services). The idea here is that you can create a set of standard operations which is not as broad as the full-featured direct data access APIs but still can operate over arbitrary data that you define in your model. Then you create a few general purpose extension/control points where you can enforce security and filter the operations. With this approach the basic work of creating the web service and such can all be automatic. The result is less service-oriented in a classic sense because the operations really are data access operations rather than messages which describe a higher-level operation, but it’s much less work and will address a number of scenarios. You can, of course, extend this model with some more targeted traditional service operations.

There are a number of other possible compromises/stopping points between the above two extremes. A few notable examples:

3) With graph-serialization you could roll-your own SOA system without separate DTOs. If you do have general-purpose graph serialization then you can automatically solve one of the problems in approach #1, but as I mentioned in the previous post, it doesn’t take much time on this path before you begin to realize that you either still need to write DTOs to deal with change tracking and concurrency issues, or you have to constrain your scenarios in various ways to make it easier to automatically determine the intent of the operations. For example, you might decide that “last write wins” is your policy and drop optimistic concurrency checks—this means that you can just send the graph to the client and then when it is modified you can send it back without having to also include original values. A second constraint comes from the question of what kinds of operations you can do in one round trip: Can you modify two entities that are not connected by a single graph? Can you unhook an object from one graph and relate it to another? What are the transaction semantics? All of these things are subtleties that are addressed in a clear, uniform way by approach 1 or 2 above, but will begin to haunt any sophisticated application that starts to roll their own. My point is that just handling graphs truly isn’t enough.

4) General-purpose container object. This container will hold an entity graph, serialize over the web service, recreate the graph on the other side, perform change tracking on the client & replay that change tracking back on the mid-tier in a form that enables persistence with concurrency checks, etc. If you stare at this description for a moment, you will realize that this is what the dataset does (particularly the typed dataset). While this is appealing because it is quite simple to use and very flexible, it also introduces serious problems when it comes to interoperability and to maintaining the abstraction which the web service represents. If you are not careful, you end up allowing pretty much any operation through your webservice which was in part supposed to constrain the set of allowed operations.

Some have gone so far as to declare this truly evil. I would moderate that a bit and say that there may well be a time and a purpose for this approach, but I am concerned about the fact that the ease of use will entice well-meaning developers down a path from which it is hard to return and which will cause them pain in the end. Nevertheless, I have spent the last week working almost full-time on a sample that provides this sort of functionality over the entity framework. It’s going to take a little while to clean it up and get it in a form where I can share it, but I found it a useful exercise for exploring the space, and it may be helpful to some other folks. I can even see the possibility of us formalizing something around this for a future release of the entity framework, but that’s yet to be determined.

Another idea which has been suggested seems to me to have the potential of being the very best approach yet, but it will take some work to flesh out, and I don’t know of anyone who has put this into practice yet. Maybe we’ll be able to experiment some with this approach for future releases of the EF:

5) Automatically generate DTOs from a declarative description of the contract. The idea is to annotate a conceptual model or maybe separately describe interesting sub-graphs of appropriate granularity and service-oriented operations which constrain what manipulations of those sub-graphs are allowed. In theory this could achieve the advantages of approach #1 above while removing the drudgery of maintaining the DTOs manually and maybe even produce a contract description which could be leveraged for other purposes.

At any rate, my overall point is that there are subtleties here. There are multiple approaches and no one approach will be right for every situation. In general I’d encourage you to avoid approach #3, and I don’t think we fully understand approach #5 yet, but we’ll be working on it for future releases. You can use Astoria today for approach #2, which just leaves approaches 1 & 4. In future posts I’ll work on putting together some examples for each of them. In the meantime, if you’ve got other approaches to suggest, I’d love to hear about them.

- Danny