Tim Mallalieu's Blog.

Just a PM's random musings on data, models, services...

Vote of No Confidence

Vote of No Confidence

Rate This
  • Comments 37

So,

It's been a long, long time since I have posted anything on my blog. Reality is I tried to maintain a blog where I thought I could come up with wonderfully profound things to share with the world but clearly that was not the case. Having said that a few events have happened that prompted me to start blogging again.

1: We are about to ship the first version of EF
2: We have an interesting thread going on about a "
vote of no confidence" about our product
3: We are starting work on our next version of the product

Seems like this would be a good time to throw myself out there. First, to respond to the general thread in the community and secondly to be more available for conversations around the current and future versions of the product.

Anyway... I will attempt to be more available and present. People can always just ping me timmall@microsoft.com directly about any of this.

In response to the community threads I will paste a response here that I made, yesterday, on an internal thread on the topic....

The unfortunate reality is that these are scenarios that we care deeply about but do not fully support in V1.0. I can go into some more detail here. One point to note is that the choices on these features were heavily considered, but we had to deal with the tension between trying to add more features vs. trying to stay true to our initial goal which was to lay the core foundation for a multiple-release strategy for building out a broader data platform offering. Today, coincidentally, marked the start of our work on the next version of the product, and we are determined to address this particular developer community in earnest while still furthering the investment in the overall data platform. Here is my take on the points below and things we are thinking about moving forward:

INORDINATE FOCUS THE DATA ASPECT OF ENTITIES LEADS TO DEGRADED ENTITY ARCHITECTURES

In v1.0 we chose to support default scenarios where one would have a distinguished based class (our EntityObject) and we would provide a mapping solution from these classes to our EDM model and from the EDM model to the database. The reason for this was because the investment we were making was in terms of the EDM – a new conceptual data model that allowed people to describe the shape of their data in terms of their domain instead of the layout in the database. Our goal is to allow people, in the fullness of time to be able to have a set of common services in terms of the Entity Data Model. Today they have query (ESQL) and REST based Services (Astoria describes its metadata using the EDM), they also get a first generation Object Persistence facility with the Object Layer component in the Entity Framework. The Object Layer, of course, also comes with our first class LINQ implementation.

There are a couple of interesting things here…

1: We expose the start of a pure value based API using ADO.NET (Entity Client) so that you can issue ESQL queries against the database in terms of your domain model, in a store agnostic manner (i.e. using ESQL instead of TSQL of PLSQL…) . There are many partners in the building for whom this is the only way they would want to leverage the EF and the EDM. Partners that fall in this camp tend to be interested in queries and presentation, data transformation or stores where the schema is frequently changing (Universal Table implementations) so that relying on CLR classes becomes intractable.

2: The focus on the EDM for the ORM developer can appear to be a red herring, as we heard from customers. The EDM describes the shape of the domain objects but does not allow for encapsulation of behavior and provides limited constraints and no specific grammar for action-semantics. In order to define a domain model where one wanted to leverage the EDM and EF today one would have to do one of the following:

  • Define an EDM model, code-gen classes and implement logic in partial classes.
  • Define an EDM model and use this as a pure DAO with a parallel set of domain objects – you could abstract the EF via many patterns… the common one we here is the repository pattern (gotchas will be pointed out in the discussion of points below).
  • Define an EDM model for query and update and then use LINQ to project into Domain Objects…   You get a case for simpler query expression and use LINQ for a projection facility to retrieve and transform from the EDM objects to the true Domain Objects.

I think there are a couple of things we are trying to do in the next version that can help here, and we actively want the feedback of folks in the community to help make sure we do the right thing:

  • Code-First scenarios…  In the code first scenarios we allow people to define their own classes (POCO) which they can work with as they see fit. We then infer a model from the classes. For situations where one desires to have a model that cannot be inferred directly from the classes, developers will be able to use new CLR attributes or an external mapping specification.
  • Code-Gen Pipeline… For some framework scenarios it is desirable to start with an EDM model (possibly with custom annotations) and then generate either the DB, the classes or both. We are looking at a general pipeline that developers can plug into that will give a lot of flexibility in generating classes and databases as well as in the way a model can be derived from the database, CLR classes or some other form (maybe a model repository built by a third party, a UML tool, or some other artificat like sharepoint).
  • Adding action-semantics to the EDM...  We have looked at things we can do here. There are things that would be quite nice but likely they would come at a cost of addressing particular feedback we have heard or our other investments. We will share thoughts and designs on these and see what people think regardless.

EXCESS CODE NEEDED TO DEAL WITH LACK OF LAZY LOADING

The EF, today, does support lazy loading. It does not support IMPLICIT lazy loading… this distinction is subtle but important. In EF we, by default, do not load ends of references or collections we wait until asked to load these. Today, however, one must explicitly ask by calling a .Load method. We took a fairly conservative approach in v1.0, because we wanted developers to be aware of when they were asking the framework to make a roundtrip to the database… our take on “boundaries are explicit”. When one builds out a repository pattern or starts to abstract the EF in some other form of data access abstraction this can become an issue because the need to call an explicit method to perform a lazy load operation causes the EF abstractions to bleed up and now you are struggling with an intersection of concerns as opposed to a clean separation of concerns.

We have heard this feedback and are looking at supporting optional implicit lazy loading as well as other strategies around eager loading (such as general LoadOptions ala LINQ to SQL) in a future release of the EF  Until then, it is possible to build implicit lazy loading on top of the explicit mechanisms which the EF support in v1, and one of our team members has published to his blog and code gallery a sample which demonstrates how this can be done.

SHARED, CANONICAL MODEL CONTRADICTS SOFTWARE BEST PRACTICES

There seems to be confusion around this topic. We are not recommending that folks return to the days where we were evangelizing the use of XSD for “canonical schemas”. I don’t believe that people think that this is tractable. What we do believe, however, is that it is desirable to have a single meta-model (EDM if you will) with which you can describe many domain models and that by having a single grammar we can provide a set of common services on any given domain model. For example, consider an application that is to be written against a database with 600 tables. Do I believe that this app should have a single model with 600 Entity Types in it? No… Furthermore, do I believe that any given domain entity (say Customer) has only one shape in that app and that this shape must be the canonical shape for the entire Enterprise?… Heck no.

I would expect, however, that with a common way to describe these models I could do some interesting things in the fullness of times…

  • I could use reporting services over any instance of one of these models and define reports in terms of the domain entities instead of the underlying tables.
  • I could perform ETL tasks between two stores in terms of entities.
  • I could write sync services between a local store and a remote store where the sync contract and programming model on both sides are in terms of EDM.
  • If I invested in learning how to build a model to use with some tool like reporting services, I would not have to learn a new tool and model description language when it comes time to build a model for one of the other tools.

This is the world that we are working towards… We are not there yet, but a number of these scenarios are what we are aggressively pursuing now.

LACK OF PERSISTENCE IGNORANCE CAUSES BUSINESS LOGIC TO BE HARDER TO READ, WRITE, AND MODIFY, CAUSING DEVELOPMENT AND MAINTENANCE COSTS TO INCREASE AT AN EXAGGERATED RATE

Agreed. There are different developer segments for whom PI is either important or not. We do not have a good story about PI today. We are working on one. Based on customer feedback from over a year ago, we started work on PI, but we said from the first that we would be unable to complete that work in v1 given the other demands of the release.  We made some initial steps for v1, and then began planning future steps for the next release.  At the last MVP summit we presented an early look at one of the techniques we were investigating which involved performing IL rewrite tricks to support these scenarios. We got strong feedback from many of the signatories on the “vote of no confidence” letter that this was not the right approach, and as a result we are not doing the IL rewrite in the next release. Instead we are doing a full POCO implementation that we hope to get feedback from the community on.

Tomorrow we will post some of the initial thinking on the new “EF Design Blog” including a little video of some of the initial dev work we are doing here. It is worthwhile pointing out that this work is a direct response to the feedback we have gotten from this community.

EXCESSIVE MERGE CONFLICTS WITH SOURCE CONTROL IN TEAM ENVIRONMENT

At the end of the day, this is feedback that we have heard, tried to address and are actively addressing moving forward. We are taking steps that we think should help in general:

1: We intend to have a more transparent design process.
2: We have a new advisory council to be more proactive in getting thoughts on our work and our directions.  This advisory council has some key community folks participating:

    Eric Evans - http://www.domainlanguage.com/about/ericevans.html 
    Stephen Forte -
http://www.stephenforte.net/ 
    Martin Fowler -
http://martinfowler.com/ 
    Pavel Hruby -
http://www.phruby.com/ 
    Jimmy Nilsson -
http://jimmynilsson.com/

3: We intend to drop more frequent interim builds. These builds will be unsupported interim builds where people can see what we have rolled into the product earlier and more often. We are still working the logistics of this out.
4: For our internal development we will be doing significantly shorter iterations.  We had quite long milestones in Orcas which made it more difficult to respond to customer feedback, and we hope that the transparent design process, the advisory council and the frequent drops will give us more feedback, and that with shorter iterations we will be more capable of responding.

Here is the thing though, even though we attempt to address the feedback, we know we won’t get everything.  From a product ownership perspective it pains me greatly to read the “no confidence” letter.  Especially as we have spent so much of the last couple of months working internally on how we can get better.  It is ironic that we find out about this letter the same week that we were already planning to roll out our goals for our engineering process for this next release, but such is life. Many of the signatories have given us great feedback, and I only hope that they will continue to do so as we proceed.

 

 

Leave a Comment
  • Please add 2 and 1 and type the answer here:
  • Post
  • I can't say I agree with some of the points made in the VOC, although I'm not saying I'm happy with the EF either. I believe there are other area's such as a very poor story around stateless interaction which are of more concern. Also M:1 association by key isn't particularly great either. I dont buy the Lazy load, I agree with MS that it should be explicit. Granted the implementation could be worked on, but the concept is right.

  • The new Microsoft Entity Framework is the latest in a long line of very impressive, yet tragic failures in Microsoft's data access strategy...

  • I don't understand why the EF doesn't support the native SQL, and some generacal and direct DML methods.

  • I have used Hibernate and NHibernate on many projects and worked with ORM technology since 2003. When reading some of these comments I wonder if developers in the .Net community have experience with other ORM tools before testing EF. It is very difficult to see what you are missing without knowledge about what features the other ORMs can offer.

    So, if I were to lead a conversion from NHibernate to EF in one of my current projects, then this technology replacement would cause a high cost to the project in terms of loss of functionality and a lot of code needed to be written just to make EF work like NH does now. This also include changes in the architetcure, which seems like changes for the worse with no gain as far as I can see. The points below illustrates what I loose by switching to EF:

    - A default fetch plan through mappings specifying what to fetch by default. This must be replaced with a lot of boiler-plate code.

    - Eager loading and implicit lazy-loading, as everything is lazy by default and cannot be configured otherwise. This leads to excessive code just to get all the data needed for a use-case.

    - PONO domain classes, must inherit from EntityObject base class. This also ties the domain classes strongly to EF, not very invasive.

    - Rich domain model (MS encourages the Anemic Domain Model anti-pattern while rich models are supported through partial classes, which I also think is awkward.)

    - Testability, to unit test reverse-engineered EF classes you must test against the DB at all time. This is simply to slow and inconvenient, and probably another reason for advocating the anemic domain model approach.

    - Domain driven development, cannot start with my classes and their behaviour first. This also makes TTD less attractive.

    - Seamless team collaboration. EF causes a lot of friction in team collaboration scenarios due to too many merge conflicts happen to frequent.

    Resolving lazy-eager problems and enable to define a default fetch plan would be good starting points as well as getting rid of the EntityObject base class. Try to make EF as invasive as possible. Remember that the domain model is the heart of any application. Domain models should thus be pure and simple, if possibly PONO/POCO. Technologies around the domain model change frequently, and to be able to embrace change, these concerns should be separated away from the heart of your application.

    Also, be careful with trying to solve all problems with Attributes. Now WCF also want you to endorse your entities with DataContract and DataMember attribtues. The code would read awfully if domain classes must worry about serialization and db-mapping issues. In stead of the well known XML-hell we might get a new Attribute-hell!

    Cheers,

    Steinar.

  • I really appreciate the power of LINQ and EF as a former FoxPro programmer.

    Great step, keep it up.

  • EDM/EF is a data technology abortion. Unforunately I stuck supporting an application written using this technology. I'm really not sure where your/MSFT head is at but obviously you are completely out of touch with the needs of your development community. The funny part is what you really need to incorporate into VS is a data centric language. The FoxPro core has been around 20+ years which MSFT owns the source. Microsoft's data technology you currently are developing last 1 or 2 years tops. Given you are completely clueless when it come to data why not steal ideas from something that works, the VFP model, instead of reinventing the wheel since you suck so bad at it. After all isn't stealing from apple how windows came to be.

  • @ Mark Gordon: God, I thought you Fox Pro relics had all gone the way of the dinosaurs. Technology has moved on since the 1980s. You're ilk and also that WinForms lot have become outdated and no longer in demand thank god. Programming has become more of a profession that takes knowledge and the application of learned disciplines and skills, instead of the cowboy infested drag-n-drop from toolbox mishmash of imperative spaghetti code it used to be in the last decades.

Page 3 of 3 (37 items) 123