Vote of No Confidence
So,
It's been a long, long time since I have posted anything on my blog. Reality is I tried to maintain a blog where I thought I could come up with wonderfully profound things to share with the world but clearly that was not the case. Having said that a few events have happened that prompted me to start blogging again.
1: We are about to ship the first version of EF
2: We have an interesting thread going on about a "vote of no confidence" about our product
3: We are starting work on our next version of the product
Seems like this would be a good time to throw myself out there. First, to respond to the general thread in the community and secondly to be more available for conversations around the current and future versions of the product.
Anyway... I will attempt to be more available and present. People can always just ping me timmall@microsoft.com directly about any of this.
In response to the community threads I will paste a response here that I made, yesterday, on an internal thread on the topic....
The unfortunate reality is that these are scenarios that we care deeply about but do not fully support in V1.0. I can go into some more detail here. One point to note is that the choices on these features were heavily considered, but we had to deal with the tension between trying to add more features vs. trying to stay true to our initial goal which was to lay the core foundation for a multiple-release strategy for building out a broader data platform offering. Today, coincidentally, marked the start of our work on the next version of the product, and we are determined to address this particular developer community in earnest while still furthering the investment in the overall data platform. Here is my take on the points below and things we are thinking about moving forward:
INORDINATE FOCUS THE DATA ASPECT OF ENTITIES LEADS TO DEGRADED ENTITY ARCHITECTURES
In v1.0 we chose to support default scenarios where one would have a distinguished based class (our EntityObject) and we would provide a mapping solution from these classes to our EDM model and from the EDM model to the database. The reason for this was because the investment we were making was in terms of the EDM – a new conceptual data model that allowed people to describe the shape of their data in terms of their domain instead of the layout in the database. Our goal is to allow people, in the fullness of time to be able to have a set of common services in terms of the Entity Data Model. Today they have query (ESQL) and REST based Services (Astoria describes its metadata using the EDM), they also get a first generation Object Persistence facility with the Object Layer component in the Entity Framework. The Object Layer, of course, also comes with our first class LINQ implementation.
There are a couple of interesting things here…
1: We expose the start of a pure value based API using ADO.NET (Entity Client) so that you can issue ESQL queries against the database in terms of your domain model, in a store agnostic manner (i.e. using ESQL instead of TSQL of PLSQL…) . There are many partners in the building for whom this is the only way they would want to leverage the EF and the EDM. Partners that fall in this camp tend to be interested in queries and presentation, data transformation or stores where the schema is frequently changing (Universal Table implementations) so that relying on CLR classes becomes intractable.
2: The focus on the EDM for the ORM developer can appear to be a red herring, as we heard from customers. The EDM describes the shape of the domain objects but does not allow for encapsulation of behavior and provides limited constraints and no specific grammar for action-semantics. In order to define a domain model where one wanted to leverage the EDM and EF today one would have to do one of the following:
- Define an EDM model, code-gen classes and implement logic in partial classes.
- Define an EDM model and use this as a pure DAO with a parallel set of domain objects – you could abstract the EF via many patterns… the common one we here is the repository pattern (gotchas will be pointed out in the discussion of points below).
- Define an EDM model for query and update and then use LINQ to project into Domain Objects… You get a case for simpler query expression and use LINQ for a projection facility to retrieve and transform from the EDM objects to the true Domain Objects.
I think there are a couple of things we are trying to do in the next version that can help here, and we actively want the feedback of folks in the community to help make sure we do the right thing:
- Code-First scenarios… In the code first scenarios we allow people to define their own classes (POCO) which they can work with as they see fit. We then infer a model from the classes. For situations where one desires to have a model that cannot be inferred directly from the classes, developers will be able to use new CLR attributes or an external mapping specification.
- Code-Gen Pipeline… For some framework scenarios it is desirable to start with an EDM model (possibly with custom annotations) and then generate either the DB, the classes or both. We are looking at a general pipeline that developers can plug into that will give a lot of flexibility in generating classes and databases as well as in the way a model can be derived from the database, CLR classes or some other form (maybe a model repository built by a third party, a UML tool, or some other artificat like sharepoint).
- Adding action-semantics to the EDM... We have looked at things we can do here. There are things that would be quite nice but likely they would come at a cost of addressing particular feedback we have heard or our other investments. We will share thoughts and designs on these and see what people think regardless.
EXCESS CODE NEEDED TO DEAL WITH LACK OF LAZY LOADING
The EF, today, does support lazy loading. It does not support IMPLICIT lazy loading… this distinction is subtle but important. In EF we, by default, do not load ends of references or collections we wait until asked to load these. Today, however, one must explicitly ask by calling a .Load method. We took a fairly conservative approach in v1.0, because we wanted developers to be aware of when they were asking the framework to make a roundtrip to the database… our take on “boundaries are explicit”. When one builds out a repository pattern or starts to abstract the EF in some other form of data access abstraction this can become an issue because the need to call an explicit method to perform a lazy load operation causes the EF abstractions to bleed up and now you are struggling with an intersection of concerns as opposed to a clean separation of concerns.
We have heard this feedback and are looking at supporting optional implicit lazy loading as well as other strategies around eager loading (such as general LoadOptions ala LINQ to SQL) in a future release of the EF Until then, it is possible to build implicit lazy loading on top of the explicit mechanisms which the EF support in v1, and one of our team members has published to his blog and code gallery a sample which demonstrates how this can be done.
SHARED, CANONICAL MODEL CONTRADICTS SOFTWARE BEST PRACTICES
There seems to be confusion around this topic. We are not recommending that folks return to the days where we were evangelizing the use of XSD for “canonical schemas”. I don’t believe that people think that this is tractable. What we do believe, however, is that it is desirable to have a single meta-model (EDM if you will) with which you can describe many domain models and that by having a single grammar we can provide a set of common services on any given domain model. For example, consider an application that is to be written against a database with 600 tables. Do I believe that this app should have a single model with 600 Entity Types in it? No… Furthermore, do I believe that any given domain entity (say Customer) has only one shape in that app and that this shape must be the canonical shape for the entire Enterprise?… Heck no.
I would expect, however, that with a common way to describe these models I could do some interesting things in the fullness of times…
- I could use reporting services over any instance of one of these models and define reports in terms of the domain entities instead of the underlying tables.
- I could perform ETL tasks between two stores in terms of entities.
- I could write sync services between a local store and a remote store where the sync contract and programming model on both sides are in terms of EDM.
- If I invested in learning how to build a model to use with some tool like reporting services, I would not have to learn a new tool and model description language when it comes time to build a model for one of the other tools.
This is the world that we are working towards… We are not there yet, but a number of these scenarios are what we are aggressively pursuing now.
LACK OF PERSISTENCE IGNORANCE CAUSES BUSINESS LOGIC TO BE HARDER TO READ, WRITE, AND MODIFY, CAUSING DEVELOPMENT AND MAINTENANCE COSTS TO INCREASE AT AN EXAGGERATED RATE
Agreed. There are different developer segments for whom PI is either important or not. We do not have a good story about PI today. We are working on one. Based on customer feedback from over a year ago, we started work on PI, but we said from the first that we would be unable to complete that work in v1 given the other demands of the release. We made some initial steps for v1, and then began planning future steps for the next release. At the last MVP summit we presented an early look at one of the techniques we were investigating which involved performing IL rewrite tricks to support these scenarios. We got strong feedback from many of the signatories on the “vote of no confidence” letter that this was not the right approach, and as a result we are not doing the IL rewrite in the next release. Instead we are doing a full POCO implementation that we hope to get feedback from the community on.
Tomorrow we will post some of the initial thinking on the new “EF Design Blog” including a little video of some of the initial dev work we are doing here. It is worthwhile pointing out that this work is a direct response to the feedback we have gotten from this community.
EXCESSIVE MERGE CONFLICTS WITH SOURCE CONTROL IN TEAM ENVIRONMENT
At the end of the day, this is feedback that we have heard, tried to address and are actively addressing moving forward. We are taking steps that we think should help in general:
1: We intend to have a more transparent design process.
2: We have a new advisory council to be more proactive in getting thoughts on our work and our directions. This advisory council has some key community folks participating:
Eric Evans - http://www.domainlanguage.com/about/ericevans.html
Stephen Forte - http://www.stephenforte.net/
Martin Fowler - http://martinfowler.com/
Pavel Hruby - http://www.phruby.com/
Jimmy Nilsson - http://jimmynilsson.com/
3: We intend to drop more frequent interim builds. These builds will be unsupported interim builds where people can see what we have rolled into the product earlier and more often. We are still working the logistics of this out.
4: For our internal development we will be doing significantly shorter iterations. We had quite long milestones in Orcas which made it more difficult to respond to customer feedback, and we hope that the transparent design process, the advisory council and the frequent drops will give us more feedback, and that with shorter iterations we will be more capable of responding.
Here is the thing though, even though we attempt to address the feedback, we know we won’t get everything. From a product ownership perspective it pains me greatly to read the “no confidence” letter. Especially as we have spent so much of the last couple of months working internally on how we can get better. It is ironic that we find out about this letter the same week that we were already planning to roll out our goals for our engineering process for this next release, but such is life. Many of the signatories have given us great feedback, and I only hope that they will continue to do so as we proceed.
