ADO.NET team blog : ADO.NET vNext

ADO.NET Entity Framework Update

dpblogs — Sun, 29 Apr 2007 08:09:00 GMT

Microsoft is deepening its investment in the ADO.NET Entity Framework as a critical piece of Microsoft’s Data Platform vision. Based on the need to align with requirements from key internal partners that are building on the Entity Framework, along with the need for a better tool experience, we have decided to ship the ADO.NET Entity Framework and Tools during the first half of 2008 as an update to the Orcas release of the .NET Framework and Visual Studio. We will continue to ship CTPs and Betas of the ADO.NET Entity Framework that align with Orcas throughout the remainder of this year.

Microsoft shipped the first CTP of the ADO.NET Entity Framework in August of 2006. Customers immediately grasped the power of being able to work with their data through a conceptual model, rather than in terms of database schemas optimized around storage concerns like normalization and data partitioning. In February we added the framework into the CTP and Beta 1 builds of the next version of the .NET Framework and Visual Studio codenamed Orcas.

We’ve collected great feedback from you in these early releases, the most significant being that we need better tools to aid in defining a data model and mapping that model to the database. Although we have tools for generating a direct mapping to the storage schema, the real power of the ADO.NET Entity Framework comes in its ability to flexibly map a variety of relational schema representations to a more appropriate conceptual application model. This has reinforced the need to have a graphical designer experience available for the ADO.NET Entity Framework.

At the same time, Microsoft will be leveraging the Entity Data Model in future versions of Microsoft products such as SQL Server. This Data Platform vision enables customers to leverage their investment in a common conceptual entity model across product lines. Stay tuned for announcements starting next week at Mix around new products building on the ADO.NET Entity Framework.

For more information on the ADO.NET Entity Framework and Microsoft’s Data Access Vision, see the Microsoft Data Access Strategy post on the Data blog.

Mike Pizzo
Architect, Data Programmabilty

Event: ADO.NET Provider Writer Tech Preview, February 6 - 8, 2007

dpblogs — Wed, 03 Jan 2007 03:42:00 GMT

What:

The ADO.NET Provider Writer Tech Preview is an opportunity for ADO.NET provider writers to:

- Gain an in-depth understanding of the features planned for the next major release of ADO.NET

- Work with an early version of the upcoming community tech preview bits

- Review product plans and specs

- Meet the Data Programmability team

- Make sure your data source is fully functional on the next generation .NET platform

The event is also an opportunity for Microsoft to hear YOUR feedback for this release!

The provider writer event held in August was a major success. During the event, every provider writer in attendance was able to build a prototype of their provider that worked with the Entity Framework features in ADO.NET Orcas. We're building on that momentum, covering the changes and new features added since the last event. We have also increased our event capacity to accommodate more provider writers.

As with the prior event, this event is geared towards helping provider writers learn how to further enhance their providers to work with the next release of ADO.NET. The event will include overviews of the features, along with plenty of time for you to work in small labs to start enhancing your providers. Each provider writer will have their own space in our labs. Representatives from the development team will be available at various times during the day to answer questions that arise when working with the tech preview binaries. Attendees will also have the option to review specs and start working with a recent drop of the binaries prior to the event.

Q: Can I attend the event if I did not attend the previous event?

A: Absolutely! Attending the prior event is not a prerequisite.

Q: If I attended the previous event, is there any reason to attend this one?

A: Absolutely! As with most pre-release technologies, ADO.NET Orcas has undergone changes since the last event. We will cover these changes along with newly added features during the event, such as:

- Designer integration and command line tool support

- DML queries to submit pending changes to the data store

- Paging queries

- Provider manifest (supports custom functions and type semantics to provide a layer of abstraction between EDM data types and store specific data types)

There will be a more detailed schedule provided to attendees upon your attendance confirmation.

All information communicated during this event is covered by a mutual NDA (Non-Disclosure Agreement) that we have in place with your organization (or you as an individual). If we do not have an NDA with you or your organization, we will ask you to sign one.

When:

The 3-day event will take place on Tuesday February 6th through Thursday February 8th.

Where:

This event will take place in the customer labs on Microsoft's main Redmond campus, which includes private areas for each provider writer. We will send you a list of recommended hotels and MS Rate prices when you respond with your Mini Form (below, but you can find some basic information here.)

What is covered:

Microsoft will provide a continental breakfast and lunches on the campus during the event, as well as one dinner during the week. The cost of travel and lodging is not covered by Microsoft.

Please Respond Quickly:

While we are making more spaces available for this event, capacity is still limited. If we receive interest from more provider writers than we can accommodate, preference will be given to vendors who have previously shipped ADO.NET providers. Please respond to (DavidSc@microsoft.com) as soon as possible with the information in the Response Mini-Form below. We expect that most provider writers will want to send 2 or 3 representatives.

Response Mini Form (one for each attendee):

Contact Name:
IT or Business Role/Title:
Email Address:
Phone Number:
Please provide some background on your provider:

We hope you can join us for this and look forward to your response.

Best regards,

David Sceppa

ADO.NET Program Manager

Microsoft Corporation

425-705-9985

ADO.NET Orcas and Samples

dpblogs — Tue, 07 Nov 2006 16:37:00 GMT

I hope you guys enjoyed the August CTP of ADO.NET vNext. As we make progress on the Orcas release of Visual Studio, the various teams –including ourselves- are working hard at integrating everything in a single product, Visual Studio Orcas. For the ADO.NET Entity Framework this means that you won’t see component-specific CTPs any more for Orcas, and instead you’ll see Orcas CTPs that have all these technologies incorporated. We may still do separate CTPs of related technologies if we see that it would help at some point in time.

Also, since the ADO.NET Entity Framework is in the Orcas builds, we don’t need the generic “vNext” token for it; from now on we’ll refer to it simply as ADO.NET Orcas, which will include the ADO.NET Entity Framework and LINQ to ADO.NET in its various flavors (LINQ to DataSet, LINQ to SQL and LINQ to Entities).

The Orcas October CTP includes an updated version of ADO.NET. The API and the XML file formats for mapping and schemas changed a bit from the August CTP, and the one you’ll see now should stay stable for the most part (we do have a few changes coming, but they should not be very disruptive).

Now for the fine print: the October CTP contains a newer version of all the internal machinery of ADO.NET Entity Framework/LINQ to ADO.NET, but some of the things that go on the top of the stack won’t be there. Specifically, LINQ to Entities will not be supported in this CTP (just timing, waiting for the right compiler to come in, etc.) and the tools that go with the runtime will not be complete by then. You’ll still see VS integration for creating items and setting up projects so they work with ADO.NET and stuff like that, but automatic generation of model/mapping from a database, as well as visual editing of EDM schemas and mappings won’t be there.

Since we won’t have support for generating schemas and mapping files automatically in this CTP, in order to help you get started with these bits we’ve updated and extended all of the examples that we included in the August CTP so they work in the new CTP. You can use these examples as a starting point for your own applications. You can download the samples here.

I know that not having a visual editor is a bit painful; don’t worry, our tools team is working hard at tools to make sure you don’t have to go through the “fun” experience of hand-editing mapping and schema files directly in XML. (and no, the nice tools that we shipped on top of the August CTP will not work with the October CTP, because we’re including our very latest bits there and we don’t have tools that go with them yet).

Which CTP should you be using?

For people who just want to get an understanding of how the ADO.NET Entity Framework, LINQ to Entities and LINQ to DataSet the best bet for now is still the August ADO.NET vNext CTP.
For those who want to see the very latest in the Entity Framework runtime and can live without the tools integration that you get in the August CTP, check out the October 2006 Orcas CTP and download the ADO.NET Samples.

As always, feedback is very welcome.

Pablo Castro
ADO.NET Technical Lead

ADO.NET vNext: Entity Data Model (EDM) Designer Prototype

alexbarn — Thu, 28 Sep 2006 02:12:00 GMT

Folks,

To aid brainstorming and to better understand typical developer scenarios, the ADO.NET team created an internal prototype of a designer to let developers model the ADO.NET vNext “Conceptual” layer.

If you recall, Pablo Castro demoed this prototype in this Channel 9 video and shortly after that we received many requests from users asking for this prototype so they could take it out for a test run. Based on the numerous requests received, we have decided to release the “ADO.NET vNext Entity Data Model Designer Prototype CTP” as an unsupported tool to showcase our upcoming ideas for ADO.NET vNext tooling.

You can download the Entity Data Model (EDM) Designer Prototype from here.

The designer in this CTP is an early prototype and only supports designing the “Conceptual” layer. The designer can also generate an Entity Data Model with 1:1 mappings from an existing database that you can use right away in your application while also graphically visualizing the “Conceptual” layer. Please take a look at the walkthroughs in the Overview document included with the EDM Designer prototype for more details.

Do take it out for a test drive and let us know what you think over at the ADO.NET vNext Technical Preview Forum on MSDN.

More ADO.NET vNext Resources:

Pablo’s announcement about the ADO.NET vNext August CTP has tons of great information about this exciting new technology.
Plenty more links to whitepapers, Chanel 9 videos, screencasts and blog posts from ADO.NET vNext reference page on the MSDN Data Developer Center.

Regards,

Sanjay Nagamangalam

Entity Mapping in ADO.NET vNext

Adya — Thu, 14 Sep 2006 01:24:00 GMT

Hi folks,

My name is Atul Adya and I am a software architect in the ADO.NET team. After working on persistent object stores, object caching, distributed file systems, and diagnostics in wireless networking, I have been spending time on a very important problem faced by programmers - reshaping/transformation/mapping of data from one data model to another.

We have been designing and implementing an innovative and powerful solution to address the mapping problem in ADO.NET vNext. In this blog post, I will describe the overall design of this entity mapping architecture. You can play with our mapping technology by downloading the CTP bits.

1. Model to Model Mapping in ADO.NET vNext

Developers typically use object-oriented technology for writing applications and they use relational databases to store their data. However, because of the "impedance mismatch" between the data models of these two technologies, there is a considerable amount of effort that is spent in many projects to address this mismatch. To bridge this gap, a number of Object-Relational (OR) Mapping solutions have been invented that lessen the burden on the programmer.

In addition to applications, a number of data services can also take advantage of a higher level of abstraction offered by a conceptual model of the data rather than the store model (See MSDN article). To cater to the need of applications and data services, ADO.NET vNext provides a more general mapping solution - OR mapping being just one, albeit a very important, instance of the general mapping solution. ADO.NET vNext provides a conceptual model - the Entity Data Model (EDM) - that allows applications and data services to model, communicate and manipulate data at a higher level of abstraction. Thus, the bridging technology that we need is one that maps store side concepts (i.e., relations) to a higher level of abstraction, i.e., the EDM.

Before we dive into the mapping strategy in ADO.NET vNext, it is important to recall the different layers in the stack - the Object Services layer, the Map Provider layer and the Store layer. The ADO.NET mapping approach is different than previous OR mappings in several ways:

(1) It separates the Map Provider layer (described by the EDM) from the Object Services layer where CLR objects are built. The mappings between these layers are relatively trivial - they are essentially one-to-one mappings although we do take care of relationship and navigation property fix-ups in this mapping component.

(2) The EDM to the store mapping component does the powerful structural transformation between the EDM layer and the store layer. It allows developers to map using different inheritance strategy mappings, split entities across tables, mapping with equality filter conditions, etc.

(3) As discussed below, the mapping transformations are internally expressed as eSQL views which provide significant benefits for querying and update.

Actually, the mapping engine that transforms between the EDM and store space performs mappings from one EDM model to another EDM model - note the store can be modeled in EDM terms. Our current implementation assumes that the store has a flat relational model but that restriction can be easily removed in later versions.

In this blog, I will focus on the EDM-to-EDM mapping engine (a.k.a. Entity Mapping engine). This engine takes the CSDL, SSDL and MSL files and produces mapping information for consumption by the update and query pipelines. Recall that these XML files have the following information:

CSDL file: The Entity Data Model for the application
SSDL file: The database as expressed in EDM terms
MSL file: The mapping information about how various entity/association sets and tables connect with each other.

2. Where we came from?

The main goal of the mapping infrastructure in ADO.NET vNext is to translate various operations from the application space to the relational space:

Queries: Given an eSQL or LINQ query in EDM terms, the mapping engine must provide information to allow the query to be translated into store terms and shipped to the database server.
Updates: Given a set of objects that have been inserted, modified or deleted in the object cache, the mapping engine needs to have sufficient information to help transform these changes into INSERT, DELETE and UPDATE database statements.

Along with these requirements, the mapping engine has to help in propagating server-generated keys and other server-generated values to the EDM space. Furthermore, it also has to provide sufficient information to allow concurrency control checks to be performed.

When we started looking at this problem, we realized that there were a number of important (and sometimes conflicting) requirements that made the problem quite difficult:

a) Flexibility of the mappings: Looking at various scenarios, we observed that people really wanted to map application data to tables in very powerful ways. With a large amount of data present in existing databases, we cannot tell users to change the data or add/remove columns according to the needs of the mapping engine!

For non-legacy database scenarios, existing data is not issue. However, developers may want to have really flexible mappings to the store, i.e., decoupling the thinking of how the store is laid out and how the entities are modeled is important. For example, developers may want to model complex inheritance hierarchies and then map them according to a variety of mapping strategies to database tables, i.e., the three well-known strategies and combinations of them. An enterprise-quality mapping engine needs to be powerful enough to support a large number of scenarios.

b) Robustness of design: Given the plethora of desired scenarios, one way to handle them is to write code that explicitly handles all the different cases. To our knowledge, most OR mapping solutions use this strategy. Even though this approach can be made to work for queries, it is much more difficult to make it work for updates to cached entities and set-based/DML updates. Since every scenario has to be carefully programmed for, the system can become somewhat fragile as more and more scenarios are supported by the mapping solution - clearly, this is undesirable. Thus, robustness of the design as the scenarios scale is critical.

Furthermore, a "case-by-case" programmed design has huge development and maintenance costs - every scenario has to be explicitly programmed for and when a new scenario is added, it needs to be accounted for in the code base. For example, if there is support for the Table-Per-Hierarchy mapping scenario and we want to allow combination of this strategy with a different inheritance approach, we have to explicitly code it. Thus, it is undesirable to have a solution that supports different scenarios via case-by-case reasoning.

c) Correct round-tripping of data: We need to maintain fidelity of the EDM data as changes are made to it, i.e., when an application's changes in the EDM space are propagated to the store and back to the EDM space, the data should be re-created in full fidelity (assuming no concurrent modifications). We need correctness guarantees in the design to ensure that such data round-trips.

3. To View or not to View - that is the question

To address these concerns, we took a radically different approach, thanks to our colleagues in the Database Group at Microsoft Research - Sergey Melnik and Phil Bernstein have been working on these issues for the past few years (see http://research.microsoft.com/~melnik/ for the latest papers).

One of the key insights that we had was that there are two aspects to mapping - the transformation/shaping aspects (e.g., mapping am entity to multiple tables) and the runtime aspects (e.g., translating queries, updates, concurrency control). Wouldn't it be a good idea if we could separate the two problems? Well, we did precisely that. The shaping issues are handled by the View Generator (a.k.a. Mapping Compiler) and the rest is handled by the runtime. The View Generator (as we see later why it is called so) takes the input Mapping Specification Language or MSL files and converts them into Mapping Views that are easily consumable by the update and query pipelines.

Let us consider queries first. To handle data shaping from tables to the rich EDM model, we can express the mapping as a Query Mapping View or QM View that expresses entity sets and association sets in terms of tables using eSQL.

For example, suppose entity type ECustomer is a subtype of entity type EPerson and we have an entity set EPersons of type EPerson, i.e., it can contain EPerson or ECustomer entities. Suppose this entity set is mapped to two tables SPersons and SCustomers such that the former contains all the persons and the latter contains all the customers - this inheritance strategy is also known as Table-Per-Concrete-Class or Horizontal Partitioning. Thus, we have the following (key fields are underlined; ECustomer has an extra field - "credit").

EDM Types: EPerson(pid, name) ECustomer(credit) <inherits from> EPerson
Tables:         SPersons(pid, name) SCustomers(pid, name, credit)

For this scenario, the QM View for EPersons is:

SELECT VALUE
    CASE WHEN _from0 THEN EPerson(p.pid, p.name)
                ELSE ECustomer(p.pid, p.name, p.credit)
    END
FROM
    (SELECT pid, name, null, true as _from0 FROM SPersons
     UNION ALL
     SELECT pid, name, credit, null as _from0 FROM SCustomers) as p

Essentially, this view is saying that EPersons can be populated from SPersons and SCustomers by performing a union of the tables and then constructing EPerson entities for tuples that came from the SPersons table and ECustomer entities for tuples from the SCustomers table (using the _from0 "artificial" field to determine where the tuple came from).

Thus, as explained by Murali in an earlier posting, when the query pipeline receives a query

SELECT VALUE p FROM EPersons AS p WHERE p.pid = 17

it substitutes EPersons with the view above (i.e. "view unfolding") so that all the "leaf" expressions in the query tree are in database terms - at this point, the query pipeline can refactor the query tree such that the relational operators are at the bottom of the tree (which can be shipped off to the server).

Thus, query mapping views give us a nice way of thinking about the problem in a r, i.e., the whole shaping/transformation exercise has been reduced to coming up with the appropriate eSQL view! Furthermore, shaping has nothing to do with the runtime artifacts like delay loading, concurrency control, etc.

This declarative approach gives us a lot of power, composability, flexibility, elegance, and, most importantly, the ability to reason about the correctness of the approach in a very clean manner. As stated above, from the query pipeline's perspective, mapping has just been reduced to view unfolding - there are no complicated mapping data structures or state management to keep track of mapping information! Isn't that cool? It gets even better. Read on …

4. Fine! That was a nice Jedi mind trick. But what about updates?

If we are expressing entity sets as views on tables, then propagating updates to the store essentially boils down to solving the well-known View Update problem in databases. Yay! We can leverage all the knowledge from that field and we are done. Well, not quite. It turns out that there are a number of limitations on view updates, e.g., most systems allow updates only to single-table views, or at best to a single table in a multi-table view - if you modify a view involving a join of two tables, databases typically do not allow updates to both tables. Of course, in mapping scenarios, a user may modify multiple fields of an entity that has been mapped to multiple tables - it is unacceptable to inform the user that such modifications are disallowed. Thus, the view update solution does not work for our environment.

So the question is - how do we handle translation of updates from the EDM space to the Store space in a declarative manner? We solve this problem by introducing a novel concept of Update Mapping Views or UM Views. These are views in which tables are expressed in terms of entity sets and association sets. Essentially, they are the "reverse" of the QM Views. For example, for the EPersons, SPersons, SCustomers example discussed above, the UM View for SPersons is (for SCustomers, the view is similar):

SELECT VALUE SPerson(p.pid, p.name)
FROM EPersons as p
WHERE p IS OF (ONLY EPerson)

The UM View simply says that the SPersons table contains all entities of type EPerson - similarly, for SCustomers. So what do we do with modifications of entities? Well, the modification problem for the above example is - given changes to EPersons, propagate them to SPersons and SCustomers. That is, given changes to the "base set" EPersons we need to propagate the updates to the "view" SPersons and the "view" SCustomers. Wait a minute - that is nothing but the Materialized View Maintenance problem in databases!! Unlike the View Update problem, Materialized View Maintenance is not as restrictive, e.g., you can have a single change to the base relation be propagated to multiple materialized views (recall that this would have corresponded to a multi-table update if we used the View Update approach on the QM Views).

Thus, by having "reverse" views we have converted our View Update problem with limited solutions into the much more flexible View Maintenance problem. Not only do we gain flexibility, it turns out that we can use the knowledge of Incremental Materialized View Maintenance algorithms that have been developed in the literature for propagating small changes in the EDM space to the table space. Of course, these algorithms were developed for the relational model for very general scenarios. Instead, we had to develop optimized algorithms for a non-relational model (i.e., the EDM) for our specific settings (i.e., we have keys on both sides, etc).

Thus, in the above example, given a change to the EPersons entity set, we can propagate the changes to the SPersons or SCustomers table depending on the type of the entity (how update propagation is done is something we will blog later).

5. Data Roundtripping

Our bidirectional views are generated in such a manner that data can roundtrip from the EDM to the store and back. To achieve this goal, the QM Views and UM Views need to be inverses of each other - let's see what that means. Suppose that an entity set E is spread across tables S1, S2, … Sn. That is,

E = QMView(S1, S2, …Sn)

We know that S1 = UMView1(E), S2 = UMView2(E), Sn = UMViewn(E).
(Note that he above views may involve other entity or association sets, e.g., S2 = UMView2(E2). For simplicity, we are ignoring such a scenario in this discussion)
So we get the round-tripping condition that must be satisfied for the views related to E:

E = QMView(UMView1(E), UMView2(E), … UMViewn(E))

That is, applying the update mapping views on E (sending the data to the store) and then applying the query mapping views on that result (querying data in E from the store) must yield E itself. The views used in ADO.NET vNext must not violate this condition. For the example above, it must be the case that:

EPersons = QMView(SPersons, SCustomers)
EPersons = QMView(UMView1(EPersons), UMView2(EPersons))

The reader can verify that this is indeed true.

6. Summary

To summarize our approach:

1. We separate out shaping/transformation issues from runtime aspects

2. We use a declarative style for expressing these transformations - for queries, we use Query Mapping Views that express entity/association sets in terms of tables. For updates, tables are expressed in terms of entity/association sets using Update Mapping Views.

3. During query execution, translation of an EDM query to the store space is done by view unfolding the Query Mapping views.

4. During update propagation, translation of entity changes to tables is performed by using incremental view maintenance algorithms over the Update Mapping Views.

5. The bidirectional views declaratively say that data will roundtrip from the EDM to the store and back (assume a correct implementation :)

The component that converts the MSL files into eSQL views for each table, entity set, and association set is called the View Generator or Mapping Compiler.

7. Future posts

Hopefully, this post has piqued your interest about the mapping technology. There are a lot of things that I did not talk about today. They are topics for future postings:

1. Generation of views - You must be wondering how we get these views. Does the end-user write them? That seems painful! The answer is - we generate them from the XML mapping specification given to us. How we do it is actually quite interesting.

2. How does one express mappings? We have an XML format for specifying mappings (in an MSL file). It turns out that we have formal semantics for each of those mapping concepts in the MSL file. Understanding these semantics really helps in better understanding the mapping input and various mapping issues.

3. Validation of input mappings - We do extensive validation on the MSL file to make sure that the mapping is valid, i.e., to ensure data can roundtrip from the EDM to the Store and back. There is a ton of interesting constraint propagation work that we have done there.

4. Capabilities of mappings - The power of ADO.NET vNext's mapping capabilities can be appreciated by looking at some examples and the corresponding views.

5. Details on the update pipeline - How does propagation of updates happen via view maintenance? There are a number of interesting and subtle issues in the update pipeline that one has to take care of.

Please let us know if you have questions, criticisms, problems w.r.t. our Entity Mapping approach.
Thanks.

Atul

ADO.NET vNext Provider Writer Event Summary

dpblogs — Fri, 25 Aug 2006 16:10:00 GMT

The ADO.NET team recently held an event for ADO.NET provider writers. Representatives from DataDirect, IBM, MySQL, SQLite and Sybase converged on Microsoft headquarters in Redmond to attend the three day event.

The goal for the event was to help provider writers understand how they can enhance their providers to support the next version of ADO.NET and to get feedback on our provider model. Each provider writer had their own lab space to work on their providers. Members of the ADO.NET team presented material on the provider model and query tree structure to the attendees and answered various questions on those topics.

The event was a major success. Representatives from five provider writers attended. By the end of the event, each provider writer was able to execute basic queries using all three layers in the ADO.NET vNext Entity Framework - the Map Provider, Object Services and LINQ to Entities.

Here are a couple quotes taken from blog posts of two of the attendees:

"It's clear Microsoft has made a big commitment to supporting 3rd party providers in the new Entity Framework. ... When the next LINQ CTP hits the streets, I'll be releasing a corresponding SQLite alpha provider for it." – Robert Simpson, SQLite. See the SQLite blog post about the event here: http://sqlite.phxsoftware.com/blogs/sqlite/archive/2006/08/03/1725.aspx

"Microsoft's strenuous efforts to closely involve third-party vendors provides a better foundation for the Entity Framework." – Jonathan Bruce, DataDirect. See the DataDirect blog post about the event here: http://jonathanbruceconnects.com/jonathan_bruce/2006/08/recap_adonet_providerpalooza.html

We look forward to working more closely with provider writers going forward and will keep the community informed of our progress.

David Sceppa
ADO.NET Program Manager

Queries in ADO.NET vNext

dpblogs — Mon, 21 Aug 2006 18:51:00 GMT

Hi folks,

There’s been a lot of interest in the ADO.NET CTP that we posted a few days back, and lots of questions about the internal details. I wanted to briefly describe how we handle queries in the ADO.NET CTP – and hopefully address some of your questions in this area.

The Lay of the Land

The ADO.NET VNext stack provides for two broad programming surfaces – the Object Services layer and the Map Provider layer – in addition to the existing provider programming surface. The Map Provider is a regular .NET data provider that accepts connections, commands, and returns DataReaders. The Map Provider allows you to describe your store data in terms of a conceptual model (the EDM) and allows you to specify client-side views over your store tables. The query language for the Map Provider is Entity SQL (eSQL).

The Object Services layer provides a strongly typed object abstraction, and allows you to program and reason in terms of programming language objects. The Object Services layer provides you a cache of objects, allows you to perform CRUD operations on these objects, allows you to query your data via LINQ or eSQL. The Object Services layer is stacked on top of the Map Provider, and gives you rich objects on top of the client views described by the conceptual model over store tables.

How do I express queries?

At the Object Services layer, you have a choice of expressing queries either via LINQ or via eSQL. At the Map Provider, you need to express queries in terms of eSQL. We may also allow – in a future version – queries to be specified in terms of canonical query trees (CQTs).

Ok, what are Canonical Query Trees (CQTs)?

We use Canonical Query Trees (CQTs) as a declarative representation for queries in ADO.NET. We use this tree representation for multiple purposes – as a common representation for all queries (regardless of how they were expressed); as a provider-neutral mechanism of communicating queries to stores (since every store has its own dialect of SQL); and in the future, as a means of programmatically constructing queries, especially in tooling and query-editing scenarios. More details on CQTs in a follow-on post

The Query Flow

A user query is always converted into a CQT. If the query is expressed in eSQL, the eSQL parser is invoked, the query is validated against the conceptual model, and then converted into a CQT. If the query is expressed in LINQ terms, the LINQ expression tree is converted into a CQT. All further processing of the query happens on the CQT.

We first “expand” (unfold) all references to client views (aka Conceptual Entity Sets) in the query. The client views themselves are simply queries against one or more Store-level Entity Sets (aka tables) and, even possibly other client views; and the view expansion process is identical to what databases do with database views. These views are generated by a mapping compiler (a subject for another blog post – but suffice it to say the mapping compiler is very powerful, and handles a number of different mappings in a very clean fashion) based on a user-supplied mapping specification. The query now targets store tables, but may have references to one or more conceptual types (entity types, complex types etc.).

The query is then run through a number of transformations to produce one or more queries that can be executed at the store. Two of the most important pieces here are the elimination of EDM type constructs (entity types, row types etc.) and the elimination of nesting operations from the query – these require some significant tweaking of the original queries – if you see some funny constants (‘0X1X2X’ etc.) in the generated SQL queries, this is where they come from. We need these pieces of information – as well as some information that we maintain outside the query - to tell us how to reassemble results later. At the end of this stage, we have queries that are entirely in terms of what the store providers support (ie) flat relational queries. We perform a few simplification operations on the query to produce “better” and more compact SQL. In the bits that you’re seeing now, some of this may still be a bit raw, but we’re working on them, and by RTM, we’ll be much better in terms of the queries we generate.

We now have one or more queries (in the form of CQTs) that can be executed by the underlying store provider. Each of these is then sent to the provider to convert into its native SQL dialect (more precisely, the provider provides a mechanism to produce a native DbCommand object from the query tree). We had a ProviderFest earlier this month, where we had provider writers for a number of databases come in to Redmond for a couple of days, and work with us on how to produce their native dialects of SQL from the CQTs – and all of them were able to get their apps up and running very quickly against their respective databases via ADO.NET. We anticipate that by RTM, many databases would have upgraded their providers to plug in easily into ADO.NET VNext.

The queries are then executed, and the results from the different readers are then reassembled (using information that’s been gathered earlier) to produce a single data reader. If the query originated at the Object Services layer, then there’s one additional step. The records from the DataReader are transformed into objects – we call this object materialization – based on the mapping information maintained by our metadata system.

Note that query execution happens entirely at the store. The client side performs result assembly (shaping operations, really), and object materialization; but the real guts of query execution is entirely in the store.

So why eSQL?

SQL has been around for a number of years, and there’s a vast number of folks that are familiar with it. SQL’s great for relational systems, but starts getting a little hairier for more advanced concepts. Large parts of SQL are still interesting though; so we designed Entity SQL (aka eSQL) to be an extension of SQL – very SQL-like in most cases, but more appropriate for the conceptual model (EDM) we’re building. We borrowed heavily from SQL-99, OQL, and earlier internal Microsoft incarnations like WinQL. You’ll see most of your friendly SQL constructs (select, from, where), some new EDM-specific constructs (type constructors, multiset constructors, relationship navigation etc.). More eSQL details in a follow-on post.

I mentioned earlier that every database has a slightly different variant of SQL that it supports. On similar lines, every database supports a slightly different set of builtin functions, and a different set of builtin types. We tried hard to keep the eSQL language agnostic about the builtin functions supported by a database. In fact, eSQL has no builtin functions – it has a very small number of builtin operators (things like +, -, <, >, = etc.). Instead, we rely on a Provider Manifest (I’ll talk about this in a follow-on post) to define the set of builtin functions and builtin types supported by the provider, and these are automatically available to you when you write eSQL queries. We also have the notion of a small set of canonical functions (I’ll describe this in a later post as well) that every provider is expected to support; and you can use these to write provider-independent queries.

When do I use LINQ? When do I use eSQL?

If you’re using the Map Provider, you will need to use eSQL. If you’re using the Object Services layer (Query<T>), you have a choice of using eSQL or LINQ. In a future release, we may also allow queries to be specified in terms of (publicly constructible) CQTs.

Some queries are easier to express in eSQL than in LINQ; some queries are easier to express in LINQ. LINQ queries give you stronger compile-time checking and IntelliSense. On the other hand, eSQL may be more appropriate for dynamic queries. Some constructs of the EDM, and some provider-specific constructs can only be exercised via eSQL. We expect to add a lot more capabilities to eSQL as part of future releases – including query-based DMLs, analytic functions, better aggregation (cube/rollup) support, native span support etc. At the end of the day, however, use the approach that is most suitable for your application. As I described earlier, both eSQL and LINQ queries are first converted into a common tree representation (the CQT) - from that point on, all processing is identical. You can mix eSQL and LINQ queries in your application logic without any trouble.

I hope that gave you all a somewhat reasonable understanding of the way we handle queries in ADO.NET. We’d love to hear what you think. What are we doing right? What parts can we do better? Where are we just plain wrong ?

We also want to get your thoughts on some of these questions. When would you use eSQL, and when would you use LINQ? What kind of constructs do you think are missing from eSQL? Would you like to program against CQTs directly (ie) would you want to build queries directly in terms of CQTs?

Things I will talk about in subsequent posts

· eSQL

· CQTs

· Provider Manifests

· More details on the query pipeline

Thanks

Murali

Announcing: The ADO.NET vNext August CTP

alexbarn — Tue, 15 Aug 2006 22:30:00 GMT

A couple of months ago we committed to shipping a very early technology preview of the ADO.NET Entity Framework and the LINQ integration in ADO.NET, and we said we were "shooting for August". Well, here is it, with a whole 2 weeks to go before August ends :)

As of right now the ADO.NET vNext August CTP is available for download.

This CTP includes a really broad set of features. The CTP documentation has a more detailed feature matrix, but here is a high-level summary of what you’ll find in this preview:

The ADO.NET Entity Framework, which includes:
- The Entity Data Model (EDM), which allows developers to model data at a higher level of abstraction
- A powerful client-views/mapping engine to map to and from store schemas
- Full query support over EDM schemas using Entity SQL and LINQ
- An object services layer that allows you to choose to present query results as rows and columns or as objects. When using .NET objects, the system will transparently do identity resolution, change tracking and update processing for you.
- An open provider model that allows other stores to plug into the ADO.NET Entity Framework
Language-Integrated Query (LINQ) integration in ADO.NET for this CTP includes:
- LINQ to Entities: formulate queries using LINQ against your EDM schemas
- LINQ to DataSet: the DataSet finally gets full query capabilities! You can formulate LINQ queries that go against one or more DataTable objects. The LINQ to DataSet implementation will even optimize certain query patterns for better execution performance.
Tools
- We're working hard to include a rich set of tools that integrate naturally with the runtime components to provide a great end-to-end experience. Some early tools are included in this CTP, and more will come soon!

In the CTP package you'll find:

The actual ADO.NET CTP assemblies that will be installed in your system
A few new project templates to create new ADO.NET Entity Framework-based applications and entity model libraries
Various examples that illustrate how to use the ADO.NET Entity Framework and the LINQ integration in many different scenarios
A couple of documents that describe the Entity Framework & LINQ integration.

I'd like to encourage everyone that writes data-centric applications to take a look at this and send your questions, suggestions and complaints this way; we'll be eagerly waiting for feedback!

If you want some more extra background before downloading and installing this, you can watch this Channel 9 video, and read an introduction to the Entity Framework (which is a previous version of the document installed with the CTP), an introduction to the Entity Data Model, and the Next Generation Data Access Vision document.

Looking forward to hear your thoughts on the CTP!

Pablo Castro
ADO.NET Technical Lead
Microsoft Corporation

ADO.NET Entity Framework on Channel 9

dpblogs — Wed, 19 Jul 2006 23:36:00 GMT

A few weeks ago we released a couple of whitepapers that discuss the future directions of the Microsoft Data Platform and in particular the specifics around the next version of ADO.NET and the ADO.NET Entity Framework.

Recently the Channel 9 folks came by Building 35 (SQL Server building at Microsoft) and we shot an episode with Mike Pizzo, Britt Johnston and myself chatting about why we chose to work on the Entity Framework for the next version of ADO.NET, what scenarios we’re targeting with it and some drill downs into what the code and designers actually look like in the current bits.

You can find the Channel 9 video here. As always, feedback is welcome.

Pablo Castro
ADO.NET Technical Lead
Microsoft Corporation

Attention all ADO.NET provider writers!

dpblogs — Wed, 12 Jul 2006 19:20:00 GMT

Microsoft is hosting an event for ADO.NET provider writers from July 31 - August 2.

We’re looking for provider writers who want to get a head start on their vNext providers to come out to Redmond, WA. Bring your laptops. We’ll supply the caffeine!

For more information, see the following post to the ADO.NET MSDN Forum: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=542887&SiteID=1

David Sceppa
Program Manager

ADO.NET vNext - feedback so far

alexbarn — Tue, 11 Jul 2006 20:31:00 GMT

(this post was originally posted here)

Since announcing our ADO.NET vNext plans at TechEd last week, the team has been on the lookout for your feedback on where we heading with the next release. This post is a round up of some of the comments / feedback we've heard. It's certainly not all of it, but should give you a flavor of what we are hearing. Some of it positive, some if it less so, but all very valuable...Lots of good questions and clarification being asked for.

TechEd

Let's start with Andres Aguiar. He sat through Dave Campbell's session on Microsoft's Data Platform Vision (I'll post a link to the webcast once it's up. In the meantime, Kent Tegels has some useful notes).

Andres enjoyed the Entity Framework news:

"It was a good session. I totally buy MS's Data Vision. The most interesting stuff for me is the Entity Framework. At last there will be a conceptual data model that all MS products will share (Reporting, Replication, Analysis Services, ADO.NET). This is a big and important improvement in the way we used data. The 'Data Dude' tool looks cool. If they manage to integrate it with the Entity Framework, it would rock."

Pablo Castro, technical lead on ADO.NET also presented at TechEd. He provided a talk called 'Next-generation Data Access in .NET Applications with ADO.NET vNext' (summarized at DevX here, Kent's notes here and Jason Coyne's notes here). We had good face time with customers giving us plenty of feedback after the talks. While at TechEd, the team also had meet up with a few ADO.NET friends and gurus.

(Dave Sceppa and Pablo also presented 'Integrated Innovation: Using ADO.NET 2.0 with SQL Server 2005'. You can view the webcast here - registration required)

This week

This week, Pablo followed up TechEd with this post providing an outline of the planned improvements with links to more detailed documentation. He asked for more feedback - you've been keeping us busy by providing lots of it on your blogs and comments at Pablo's post.

Techworld covered the news too:

"Entities will help improve communication between business people and developers, said Lyn Robison, an analyst with the application platforms group at the Burton Group. "It will raise the level of abstraction so that you can begin to think of data from a business perspective, not just from a rows-and-tables-in-a-database perspective," Robison said."

Some of the early feedback on the documents included this post on Ayende Rahien's blog on the topic of LINQ for Entities, specifically: many-to-many relationships; the extensibility of the data model; and Indexed and Custom collections. You can read Pablo's response in the post's comments. Wagnerblog also had some thoughts to share in the context of ADO.NET Entities and ORMs. Tim Mallalieu (a PM on the ADO.NET team) also blogged his thoughts on Entities, sparking off some further discussion there.

A recurring theme of some of the feedback is summed up by Andres Aguiar.

"OK, it actually happened. We'll have two mapping technologies in .NET v.next.

LinQ for SQL, previously known as DLinQ is the 'simple' mapping technology.

LinQ for Entities, will be on top of the new ADO.NET Entity Framework, and will be the 'complex' (we could say 'real') mapping technology.

Now, does this make sense? How will someone decide to use one or the other?"

Ayende again:

"Oh joy! Three ORM frameworks. Linq to SQL, Linq to DataSet, Linq to Entities."

This was echoed by David Hayden and others. Clearly we have work to do in this area - this point was acknowledged by Somasegar (VP of Microsoft's Developer Division) where he blogged this week about LINQ and ADO.NET Entities:

"Our teams are busy working on the next version of Visual Studio and incorporating these technologies in a consolidated way to ensure the very best experience for our developers."

Another common question we heard was about the ADO.NET vNext CTP and its timing. In this Forum post by Lance Olson, the ADO.NET Group PM, he provides an August 2006 target date. This was later blogged by Erwyn Van Der Meer and Alex at Base4 after the team confirmed it.

Getting more into developer experience, Shyam posted up a 'great' screencast showing ADO.NET vNext in action, again prompting yet more feedback and questions.

More feedback

As we move toward the CTP later this year, we want to make sure we have a systematic, scalable process around capturing your feedback (Bugs, Suggestions and Other) on the bits we release. We also want to provide you with status on these as the product development goes on. We'll be using the Microsoft Connect platform developed for beta programs at Microsoft (IE is one of the many teams using it). Using Connect, the feedback provided goes directly into our team's bug tracking and development systems. It has recently been upgraded to replace the Product Feedback Center (Ladybug). Look out for more info with respect to ADO.NET vNext later in the year.

In the meantime, keep the feedback coming!

Oh, and don't forget to check out the new Data blog. If you want to get your thoughts heard by Sam Druker, Product Unit Manager (PUM) for Microsoft's Data Programmability team, then this is the place to do it.

Thanks!

Alex Barnett, Community PM

ADO.NET vNext: The Entity Framework, LINQ and more

alexbarn — Tue, 11 Jul 2006 19:57:00 GMT

(this post was originally posted here)

Once you are done with shipping large products such as SQL Server 2005 and Visual Studio 2005 you’d expect to have a quiet time for a while, slow down a bit, that kind of stuff…

Turns out that it wasn’t the case this time. Right after SQL Server 2005 and Visual Studio 2005 were ready to ship (and actually even before they were completely done) we started to work hard on the next version of the data programming technologies; we worked on a broad vision that spans various releases and various technologies, and also specifically on how ADO.NET plays in that vision.

In the context of the vision for data programmability and ADO.NET, we decided that the next big step for ADO.NET was to move to a higher-level of abstraction. Connections, commands and readers are great for low-level stuff, but it’s not exactly what you want to be dealing with continuously when writing business logic. There are other aspects that are similarly “low level” and applications have to deal with, like the actual database schemas (e.g. did you even wonder why you have to do a 3-way join just to navigate a relationship between entities instead of just saying “traverse the relationship”?).

Now we’re making public our vision on data programmability, it’s a great read, I highly recommend it.

We’re also making public the specific plans for the next version of ADO.NET, a bit more technical, less formal, but with all the details of how we’re moving the technology forward and describes the ADO.NET Entity Framework, as well as the ADO.NET Entity Data Model (EDM).

The first of a series of Channel 9 videos has been posted - this one features Sam and Anders talking about Entities, LINQ and a few details about how all the stuff fits together.

Feedback on all of this stuff is welcome. I highly encourage folks to check out all of the content we’re putting out there and write us with your thoughts.

Pablo Castro
ADO.NET Technical Lead
Microsoft Corporation