Data : LINQ

Data Development Bonus Track - Devconnections Fall 2007

dpblogs — Thu, 20 Sep 2007 23:59:00 GMT

There's a new Bonus Track at at DevConnections this Fall!

Come and check out sessions in the new Data Access Bonus track to learn more about the many new, and not so new, Microsoft products providing developers with a more positive and efficient experience developing data-aware applications regardless of the technology that you choose as the basis for your app (Windows Forums, ASP.NET, WPF, Silverlight, etc.).

Learn about Sync Services for ADO.NET, the upcoming ADO.NET Entity Framework, how to take advantage of new features in SQL Server 2008, the new Project Astoria technology, and a brand new project that Microsoft is introducing in this space.

Data Technologies at Mix '07

dpblogs — Tue, 08 May 2007 19:39:00 GMT

There has been a lot of recent coverage on the participation of data technologies at Mix 07 and the announcements that we, the Data Programmability team at Microsoft, made regarding two new incubation projects that we are working on.

So, if you didn't get a chance to attend Mix, or just didn't get a chance to attend our sessions, here's a quick breakdown of Data at Mix 07.

The Data Programmability Team, including the ADO.NET and XML teams, presented 4 sessions and hosted 2 Sandbox sessions during the course of Mix 07.

· Pablo Castro presented 2 sessions (one repeat) on Accessing Data Services in the Cloud (Project Codename 'Astoria')

· Tim Scudder & Aaron Dunnington presented a Deep Dive session on Data Driven Experiences (LINQ to XML and Silverlight)

· Sam Druker & Shyam Pather presented a session Rapidly Building Data Driven Web Pages with Dynamic ADO.NET (Project Codename ‘Jasper’)

To learn more about these technologies, all sessions are available to view at the above links. You can also checkout more information on Project Astoria and Project Jasper on the MSDN Data and Storage Developers Center.

Elisa Johnson

Community Program Manager, Data Programmability

Microsoft’s Data Access Strategy

dpblogs — Sun, 29 Apr 2007 08:12:00 GMT

Over the last 12 months, Microsoft has been talking a lot about two major innovations related to representing and querying data. The first is the new Entity Data Model exposed as part of the ADO.NET Entity Framework, and the second is a set of extensions to the .NET Framework for integrating queries into the programming language known as LINQ. What are these technologies, how do they relate to one another, and what role do they play in Microsoft’s Data Access Strategy? Let’s start with Microsoft’s Data Access Strategy.

Does Microsoft have a Data Access Strategy?

Yes, it turns out we do. Microsoft envisions an Entity Data Platform that enables customers to define a common Entity Data Model across data services and applications. The Entity Data Platform is a multi-release vision, with future versions of reporting tools, replication, data definition, security, etc. all being built around a common Entity Data Model.

Within the .NET Framework, the ADO.NET Entity Framework is integral to this vision. The ADO.NET Entity Framework builds on our mutual investment in ADO.NET by enabling applications to write to a conceptual data model with strong notions of type, inheritance, and relationships. Through the use of “Client Views”, this conceptual model can be flexibly mapped to existing relational schemas, enabling the creation of new business structures when the underlying database schema cannot be changed and providing a level of indirection that helps isolate applications from naming as well as structural changes of the storage schema (for example, changes in the degree of storage schema normalization). The ability to work with a conceptual model greatly simplifies forms design and web service development where the underlying data store represents the model in a complex way.

Client Views are implemented within a new ADO.NET Data Provider called the Entity Client which supports an extended SQL grammar called Entity SQL (“ESQL”). ESQL provides a common query language across providers by extending the existing SQL language with constructs to work with strong notions of type, inheritance, and relationships from the Entity Data Model. By exposing the conceptual model through traditional ADO.NET Data Provider APIs (Connections, Commands, and Data Readers), existing ADO.NET programmers can immediately benefit from the conceptual model, common SQL language, and flexible mapping of the ADO.NET Entity Framework.

The ADO.NET Entity Framework also supports Object Services for building typed queries and returning, manipulating, and updating results as Business Objects. These Business Objects can be queried using ESQL, or through a new language innovation called LINQ.

What is LINQ?

Microsoft’s new Language Integrated Query (LINQ) is a set of extensions to the .NET Framework that provide common capabilities within the programming language for querying against in-memory data as well as external data sources. LINQ complements our vision for an Entity Data Platform by providing language extensions for querying data as objects within the programming language.

LINQ will ship as part of the next version of Visual Studio and the .NET Framework, codenamed Orcas. At the time that Orcas first ships, the .NET Framework will include support for LINQ over in-memory objects, LINQ over XML (XLINQ), LINQ over ADO.NET DataSets (LINQ to DataSet), and LINQ queries directly mapped to Microsoft SQL Server schemas (LINQ to SQL).

A few months after the shipment of Orcas, and within the first half of 2008, Microsoft will release the ADO.NET Entity Framework as an extension to the Orcas version of the .NET Framework. The ADO.NET Entity Framework will fully support LINQ through a feature known as “LINQ to Entities”. LINQ to Entities combines the developer experience of having query integrated into the programming language and the richness of being able to define an Entity Data Model with flexible mapping to relational stores.

What is the difference between LINQ to SQL and LINQ to Entities?

LINQ to SQL supports rapid development of applications that query Microsoft SQL Server databases using objects that map directly to SQL Server schemas. LINQ to Entities supports more flexible mapping of objects to Microsoft SQL Server and other relational databases through extended ADO.NET Data Providers.

If you are writing an application that requires any of the following features, you should use the ADO.NET Entity Framework:

· The ability to define more flexible mapping to existing relational schema, for example:

o Mapping a single class to multiple tables

o Mapping to different types of inheritance

o Directly Modeling Many to Many relationships

o Mapping to an arbitrary query against the store

· The ability to query relational stores other than the Microsoft SQL Server family of products.

· The ability to share a model across Replication, Reporting Services, BI, Integration Services, etc.

· A full textual query language

· The ability to query a conceptual model without materializing results as objects

If you do not require any of these features, LINQ to SQL may provide a simpler solution for rapid development. LINQ to SQL is targeted at rapid-development scenarios against Microsoft SQL Server and provides a simpler approach and richer design experience in the Orcas timeframe for applications with data classes that map directly to the schema of your Microsoft SQL Server database.

Microsoft is defining a migration plan for customers that start with LINQ to SQL and require additional functionality, such as richer mapping capabilities or access to other stores, to migrate to the ADO.NET Entity Framework.

Sounds great; how do I get started?

Both LINQ to SQL and the ADO.NET Entity Framework, including LINQ to Entities, are available now as part of the Orcas Beta 1. LINQ over Objects, LINQ to XML, LINQ to SQL, and LINQ to DataSet will all ship as part of the Orcas release of the .NET Framework. We will continue to ship CTPs and Betas of the ADO.NET Entity Framework that align with Orcas throughout the remainder of this year.

Mike Pizzo,
Architect, Data Programmability

Updated 101 LINQ Samples for Orcas March CTP

dpblogs — Wed, 14 Mar 2007 00:54:00 GMT

We've updated the 101 LINQ Samples code to include the 101 LINQ to Entities Samples! Check out the updated samples here.

Data Access API of the Day - Part IV (Programming to the Conceptual Model…)

dpblogs — Wed, 24 Jan 2007 02:01:00 GMT

Welcome to Part IV of Data Access API of the Day; a brief history of the evolution of Microsoft’s Data Access APIs.

In Part I we looked at ODBC as Microsoft’s C-based Relational API, and the DAO and RDO automation interfaces that made relational databases exposed through ODBC available to languages like VB. In Part II we looked at OLE DB as Microsoft’s first-class Data Access API for componentized data access within Microsoft’s Component Object Model (COM) environment. In Part III we looked at the introduction of ADO.NET, a managed API for the .NET Framework that revolutionized the relationship between connected data access and working with disconnected sets of data, and started to rationalize the relationship between relational data and XML.

ADO.NET Entities builds upon our mutual investment in ADO.NET by adding the ability to write applications against a rich conceptual "Entity Data Model" schema, rather than a flat relational database schema. The Entity Data Model (EDM) extends the relational data model with Entity-Relationship (ER) constructs for modeling real-world concepts such as Inheritance (cars and trucks are vehicles), Relationships (customers have orders), and complex members (street, city, region, and postal code composed as a single "address" property within a customer). An extended-SQL grammar called "Entity SQL" allows you to directly query your conceptual schema, leveraging inheritance, accessing complex members, and navigating relationships. In many cases, building these concepts into the conceptual schema and query language removes the need for complex joins, unions, and subqueries to do conceptually simple operations.

These rich conceptual schemas are exposed and queried through an "Entity Client". The Entity Client is an ADO.NET Data Provider that builds queries against storage-specific providers using client-side read/write "views". Queries and updates written against these conceptual views are expanded by the Entity Client and executed as queries against underlying storage-specific providers. All the actual query execution is done in the store (not on the client), and the results are assembled into possibly hierarchical, polymorphic, results with nesting and composite members. This separation between the conceptual model that the application targets and the storage schema of the database is an extremely powerful concept that we believe will greatly simplify the authoring and maintenance of database applications.

Exposing client views through an ADO.NET Data Provider allows us to retain the familiar ADO.NET programming model, leveraging investments in code, tools, and knowledge built around ADO.NET. The fact that the Entity Client consumes existing ADO.NET Data Providers (extended to support a new canonical query tree representation) builds on the growing community of popular, as well as custom, ADO.NET Data Providers.

Many of the constructs added we added to the Entity Data Model also exist in popular object-oriented programming environments, including the .NET Framework. This is not by accident.

Since version 1.0, customers have sought an Object/Relational solution within the .NET Framework. Microsoft has made several attempts at an O/R solution for .NET, most notably "ObjectSpaces" which, though never released, was premiered as a "technical preview" at PDC when we launched the .NET Framework, and pretty much every conference there-after.

Like most O/R solutions today, ObjectSpaces attempted to support a rich set of mappings and scenarios through custom query generation. Adding support for a new type of inheritance mapping, for example, meant adding code to a query generator to insert the necessary join conditions in all the right places. Understanding how this new construct composed with other joins, projections, unions, and predicates added throughout the query to model other object-like concepts made the code complex and somewhat brittle. Trying to understand how to generate updates against such complex queries, or if such updates were even possible, was even more difficult. Worse of all was trying to verify that all possible combinations of the constructs composed into a query were handled correctly.

ADO.NET Entities takes a different approach. By modeling a rich conceptual schema through client-side query and update views, the Entity Client leverages the significant investment and research that has gone into relational database view theory. Updating, for example, is done by applying well-defined view maintenance techniques to the update views in order to produce a set of delta expressions that are combined with the query views to produce update expressions. The resulting query views and update processing are both composable and verifiable.

For those that prefer to work with data as strongly typed CLR objects rather than untyped result records, ADO.NET Entities includes "Object Services" which build on top of the Entity Client and allows you to query and retrieve results in terms of application "Data Classes" whose identity and changes are managed for you.

So what about LINQ?

"LINQ" stands for "Language INtegrated Query". As the name implies, LINQ integrates query concepts directly into the programming languages, enabling data access code to be type-checked by the language complier, and developer tools like Intellisense to make it easier for developers to write queries. This, along with higher level conceptual models (like Entities), contributes to reducing the impedance mismatch between applications and data.

LINQ is supported as a first-class citizen within the ADO.NET Entity Framework through "LINQ to Entities". LINQ to Entities is part of the Object Services layer which enables you to build queries through strongly typed language expressions and built-in query comprehensions (in C# and VB) as well as textual Entity SQL statements. This means that the same conceptual client views are available through existing ADO.NET provider programming patterns or through consuming Object Services using either ad-hoc textual queries or integrated language queries.

ADO.NET Entities, along with LINQ, will be featured in the upcoming February Orcas CTP. In the meantime, more on the ADO.NET Entity Framework can be found here, as well as in this Channel 9 video.

It’s been interesting for me to see our data access APIs evolve over the years from ODBC, a C-level interface for accessing data in a SQL database that is still popular today, to a broad Entity Framework that supports modeling data as rich conceptual objects, querying through a common extended SQL grammar or query constructs embedded within the language, and interacting with the data as business objects with identity management and change tracking.

What’s next for Microsoft in Database APIs? Time will tell, but the bet on the Entity Framework, and the Entity Data Model in particular, is big. You can expect to see more and more services within SQL Server, as well as technologies throughout the company, embrace and leverage the Entity Data Model as the natural way to describe data in terms of real-world concepts. Although I don’t admit it to many people, after almost 20 years at Microsoft I still find working with data interesting, and I look forward to continuing the journey with you, wherever it may take us.

Mike Pizzo
Architect, Data Programmabilty

Data Access API of the Day, Part I

dpblogs — Wed, 06 Dec 2006 03:50:00 GMT

Why does it seem like Microsoft is coming up with a new Data Access API every time you turn around? First there was DbLib, then ODBC, then DAO, RDO, ODBCDirect, OLEDB, ADO, ADO.NET -- When is Microsoft going to run out of letters? And now what's this LINQ thing? And what about these "Entities" someone mentioned the other day? And who is the person writing this BLOG?

I'll answer the last question first. My name is Mike Pizzo, and I'm an Architect in the Data Programmability team at Microsoft – yes, the same team which brought you that “alphabet soup” of Data Access interfaces. In fact, I've been in the Data Programmability Team (under various names) for better than 15 years, and personally contributed to each of those APIs (with the exception of DbLib).

Following is a short history of the evolution of Microsoft Data Access APIs in four parts (so far…)

Part I – The Early Years (ODBC, DAO, RDO, ODBCDirect)
Part II – Componentizing Data Access (OLE DB)
Part III – Disconnected Programming in a Managed Environment (ADO.NET)
Part IV – Programming at the Conceptual Model (ADO.NET Entities and LINQ)

My goal is to describe the industry trends, platform shifts, and usage scenarios that shaped our API landscape over the past 15 years (and, in the retelling, perhaps receive some absolution for my sins…)

So, without further “ado”…

Part I – The Early Years…

First, let me apologize. I know – Microsoft knows – that changing something as fundamental as the way your program accesses and works with data is a tremendous cost in terms of existing codebase, tools, components and expertise. Every new option increases the decision matrix for writing new programs. I hate the very thought of mentioning a new data access API to customers. Some days I wish we could have stayed just with ODBC all of these years.

But what if we had? What if we had simply extended ODBC to support building a federated database out of individually reusable components? What if we had extended ODBC to support a disconnected programming model? What if we had extended ODBC to work over XML? What if we made ODBC work with CLR Objects?

The answer is; it wouldn't be ODBC. ODBC (which stands for Open DataBase Connectivity) was designed for a specific purpose; to be a common Call Level Interface (CLI) to a relational database. Through several years, many Standards meetings, and a good deal of excellent Indian food (a favorite of Jim Melton’s, the editor of the ANSI SQL Specification) we even made ODBC into a SQL/CLI addendum to the SQL 92 specification.

In our ODBC 2.0 design preview back in the early 90s I presented a set of proposed ODBC ISAM (Indexed Sequential Access Method) extensions for working with sets of indexed data that didn't support Query Processing. The idea was that we could provide a common query processor over non-relational sources like text, excel, dBase and Paradox files. The feedback we got was positive – that we should support ISAMs, but unequivocal that it NOT be part of ODBC. ODBC's strength, we were told, was its direct tie to relational stores. This feedback totally surprised us, but in retrospect they were absolutely right, and I believe that's why ODBC is still popular today.

And then something happened. Visual Basic became popular as a scriptable "automation language". ODBC, being a C-style interface, was not directly consumable from VB. However, some of you clever folks figured out that Microsoft Access supported executing queries against ODBC Datasources, and that Access did support scriptable automation through its Data Access Object (DAO) API. Voila! Now you could write applications against ODBC sources using VB.

However, DAO went through Access's internal "Jet" (Joint Engine Technology) database engine, which defaulted to building local keysets for each result in order to do advanced query processing and cursoring against the remote data. This was fine if you needed that functionality, but significant performance overhead and additional round trips when you didn't.

Enter the Visual Basic team who, responding to customer demand for better performance against ODBC sources, came up with something called Remote Data Objects (RDO). RDO implemented the same DAO programming patterns directly against ODBC, rather than going through Jet. RDO was extremely popular among VB developers, but the fact that we had two different sets of automation objects for accessing ODBC sources caused confusion.

But apparently not enough confusion, because our solution was to introduce "ODBCDirect". Despite its name, ODBCDirect was not a new API; it was just a mode we added to DAO that set defaults in such a way as to avoid the overhead of building keysets and such.

And then something happened. The industry made a big push toward object-oriented, distributed, componentized software. With the Object Management Group (OMG) pushing their Common Object Request Broker Architecture (CORBA), Microsoft needed first-class support for data access for its Component Object Model (COM)…

Next: Part II – Componentizing Data Access (OLE DB)

Mike Pizzo
Architect, Data Programmability