Jaroslaw Kowalski

Using EFProviderWrappers with precompiled views

One of the users has reported a problem with using EFProviderWrappers and precompiled views together.

When you pre-compile views, Entity Framework calculates a hash of metadata (which includes csdl,ssdl and msl information), stores it with the generated views and compares it later when metadata is loaded. When loaded metadata doesn’t match the hash stored in pre-compiled views, an exception is thrown.

Injecting a provider into provider chains involves changing SSDL file and that invalidates the hash.

Fortunately there’s a workaround, which enables you to use pre-compiled views and wrapper providers together, but you have to use the Alternative injection method as described in original blog post.

Let’s assume you have followed the steps outlined in the post and have your modified SSDL file ready. Injecting is a bit tricky because of additional registration required for EdmGen.exe to work. The first part involves creating a modified version of EdmGen.exe which can handle wrapper providers:

1. Create a working directory (say “C:\ViewGeneration”)
2. Copy %WINDIR%\Microsoft.NET\Framework\v3.5\EdmGen.exe and %WINDIR%\Microsoft.NET\Framework\v3.5\EdmGen.exe.config to that directory
3. Open C:\ViewGeneration\EdmGen.exe.config using notepad
4. Add Provider Registration section before the end of the configuration file so that it looks like this:

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <runtime>
       <generatePublisherEvidence enabled="false"/>
     </runtime>
     <!-- declare provider factories -->
     <system.data>
       <DbProviderFactories>
         <add name="EF Caching Data Provider"
              invariant="EFCachingProvider"
              description="Caching Provider Wrapper"
              type="EFCachingProvider.EFCachingProviderFactory, EFCachingProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b" />
         <add name="EF Tracing Data Provider"
              invariant="EFTracingProvider"
              description="Tracing Provider Wrapper"
              type="EFTracingProvider.EFTracingProviderFactory, EFTracingProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b" />
         <add name="EF Generic Provider Wrapper"
              invariant="EFProviderWrapper"
              description="Generic Provider Wrapper"
              type="EFProviderWrapperToolkit.EFProviderWrapperFactory, EFProviderWrapperToolkit, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b" />
       </DbProviderFactories>
     </system.data>
   </configuration>

5. Copy EFTracingProvider.dll, EFCachingProvider.dll and EFProviderWrapperToolkit.dll to C:\ViewGeneration

(Alternatively you can just gac all the providers and put provider registration section in machine.config, but I think the described method is cleaner as it has only local impact and doesn’t require you to be an administrator).

Once you have the modified EdmGen.exe, you can use it for your view generation as usual, except that you pass modified ssdl file as an argument to /inssdl parameter.

C:\ViewGeneration\EdmGen.exe 
    /mode:viewgeneration 
    /incsdl:NorthwindEFModel.csdl 
    /inssdl:NorthwindEFModel.WithTracing.ssdl 
    /inmsl:NorthwindEFModel.msl 
    /outviews:NorthwindEF.views.cs

This should create you NorthwindEF.views.cs which you can compile along with the rest of your application to get better startup performance.

Now, one more cool trick:

Wrapper-enabled EdmGen.exe can be used not just for view generation. For example it is possible to automatically generate a model from a database using EFTracingProvider! (this will create SSDL file which takes advantage of tracing automatically).

C:\ViewGeneration\EdmGen.exe 
    /mode:FullGeneration
    /provider:EFTracingProvider
    /connectionString:"wrappedProvider=System.Data.SqlClient;server=.\SQLEXPRESS;database=NorthwindEF;integrated security=sspi" /project:NW

You can even use tracing provider during reverse-engineering process, to see what SQL statements are executed. All you need to do is add this section to C:\ViewGeneration\EdmGen.exe.config:

<appSettings>
  <!-- this setting causes all log messages to be written to the console. -->
  <add key="EFTracingProvider.logToConsole" value="true" />
</appSettings>

Enjoy!

Tracing and Caching for Entity Framework available on MSDN Code Gallery

We have just released a sample that shows how to extend Entity Framework in interesting ways by plugging into ADO.NET provider interface. The sample provides two extensions:

• EFTracingProvider – which adds the ability to log all SQL commands that are executed (similar to LINQ to SQL’s DataContext.Log
• EFCachingProvider – which adds transparent query results cache to EF

The sample comes with implementation of distributed cache which uses Velocity CTP 3 as well as an adapter for ASP.NET and simple in-memory cache implementation.

Because the sample is quite large and uses many advanced techniques, it’s impossible to fully explain it all in one blog post. In this first post I’ll briefly explain the idea of wrapper providers and describe the new the APIs exposed by EFTracingProvider and EFCachingProvider. In future posts I’ll try to explain more technical detail details and provide advanced logging/caching tips.

Provider Wrapers

Entity Framework has a public provider model which makes it possible for provider writers to support 3^rd-party databases, such as Oracle, MySQL, PostreSQL, Firebird. The provider model provides uniform way for EF to query the capabilities of the database and execute queries and updates using canonical query tree representation (as opposed to textual queries).

Whenever you issue a LINQ or Entity SQL query through an ObjectContext instance, the query passes through a series of layers (see the picture below). At high level we can say that all queries and updates from ObjectContext are translated and executed through EntityConnection, which in turns talks to server-specific data provider such as SqlClient or Sql Server CE client.

Provider interface used by Entity Framework is stackable, which means it’s possible to write a provider which will wrap another provider and intercept communication between Entity Framework and the original provider.

The wrapper provider gets a chance do interesting things, such as:

• Examining query trees and commands before they are executed
• Controlling connections, commands, transactions, data readers, etc.

EFTracingProvider intercepts DbCommand.ExecuteReader(), ExecuteScalar() and ExecuteNonQuery() and sends details about the command (including command text and parameters) to configured outputs.

EFCachingProvider is a bit more complex. It uses external caching implementation and caches results of all queries queries that are executed in DbCommand.ExecuteReader(). Whenever update is detected (either UPDATE, INSERT or DELETE) the provider invalidates affected cache entries by evicting all cached queries which were dependent on any of the updated tables.

Using the sample code

Here’s a step-by-step guide to downloading and using the sample code in your project:

1. Download the sample project from MSDN Code Gallery and built it.
2. Take EFCachingProvider.dll, EFTracingProvider.dll and EFProviderWrapperToolkit.dll and place them in a common directory for easy referencing.
3. Add reference to those three assemblies in your application.
4. Register the providers – either:
   1. Locally for your application (recommended) – put registration entries in App.config (see Provider Registration section below) and make sure that provider DLLs are in your application directory (adding reference and building should take care of that).
   2. Globally: GAC the providers and add required registration entries (see Provider Registration section below) to machine.config
5. Copy EFProviderWrapperDemo\ExtendedNorthwindEntities.cs from the sample and put it in your project – rename the class as appropriate – you will need to use this class instead of a regular strongly typed object context class.
6. Modify the base class by replacing NorthwindEntities with the name of your strongly typed object context class.
7. Modify the default constructor by providing your own connection string.
8. (optional) You can also modify the second constructor by specifying which wrapper providers to use.

That’s it.

Caching and Tracing APIs

By using ExtendedNorthwindEntities which was created in previous step, instead of NorthwindEntities you get access to new APIs which control caching and tracing:

public TextWriter Log { get; set; }

Specifies the text writer where log output should be written - same as in LINQ to SQL

public ICache Cache { get; set; }

Specifies which cache should be used for the context (typically a global one). The sample comes with 3 implementations of ICache interface which you can be used in your applications:

• AspNetCache – cache which uses ASP.NET caching mechanism
• InMemoryCache – simple, in-memory cache with basic LRU expiration policy
• VelocityCache – implementation of caching which uses Microsoft Distributed Cache codename "Velocity" CTP3.

public CachingPolicy CachingPolicy { get; set; }

Specifies caching policy. There are 3 policies included in the package:

• CachingPolicy.CacheAll – caches all queries regardless of their results size or affected tables
• CachingPolicy.NoCaching – disables caching
• CustomCachingPolicy – includes user-configurable list of tables that should and should not be cached, as well as expiration times and result size limits.

It is also possible to write your own caching policy by creating a class which derives from CachingPolicy and overriding a bunch of methods.

For more advanced logging scenarios there are also 3 events, which provide access to raw DbCommand objects and some additional information:

public event EventHandler<CommandExecutionEventArgs> CommandExecuting
public event EventHandler<CommandExecutionEventArgs> CommandFinished
public event EventHandler<CommandExecutionEventArgs> CommandFailed

The events are raised before and after each command is executed.

Global configuration

You can also configure logging defaults through static properties of EFTracingProviderConfiguration class and they will apply to all new contexts:

public static bool LogToConsole { get; set; }

Specifies whether every SQL command should be logged to the console.

public static string LogToFile { get; set; }

Specifies global log file.

public static Action<CommandExecutionEventArgs> LogAction { get; set; }

Specifies global custom logging action – a delegate that will be invoked before and after each command is executed.

Tracing Example

In order to write all SQL commands to a file, you must create a text writer object to write to and assign it to context.Log:

using (TextWriter logFile = File.CreateText("sqllogfile.txt"))
{
  using (var context = new ExtendedNorthwindEntities())
  {
    context.Log = logFile;     // ... 
  }
}

Logging to the console is even easier:

using (var context = new ExtendedNorthwindEntities())
{
  context.Log = Console.Out;   // ... 
}

More advanced logging can be achieved by hooking up Command*events:

using (var context = new ExtendedNorthwindEntities())
{
  context.CommandExecuting += (sender, e) =>
  {
   Console.WriteLine("Command is executing: {0}", e.ToTraceString());
  };
  context.CommandFinished += (sender, e) =>
  {
    Console.WriteLine("Command has finished: {0}", e.ToTraceString());
  };   // ... 
}

To enable tracing globally for all connections (to both console and a log file):

EFTracingProviderConfiguration.LogToConsole = true;
EFTracingProviderConfiguration.LogToFile = "MyLogFile.txt"

Caching Example

In order to use caching using InMemoryCache implementation, you must create global instances of your cache and caching policy objects:

ICache cache = new InMemoryCache(); 
CachingPolicy cachingPolicy = CachingPolicy.CacheAll;

In order to use caching with Velocity CTP3, you must create DataCache object and pass it to VelocityCache constructor.

private static ICache CreateVelocityCache(bool useLocalCache)
{
  DataCacheServerEndpoint endpoint = new DataCacheServerEndpoint("localhost", 22233, "DistributedCacheService");
  DataCacheFactory fac = new DataCacheFactory(new DataCacheServerEndpoint[] { endpoint }, useLocalCache, useLocalCache);

  return new VelocityCache(fac.GetCache("Velocity"));
}

Now in order to use either of the caches we need to set up Cache and CachingPolicy properties on the context:

using (var context = new ExtendedNorthwindEntities())
{
  // set up caching
  context.Cache = cache;
  context.CachingPolicy = cachingPolicy;   // ... 
}

Configuring Providers

Provider Registration

Before a provider can work with Entity Framework it must be registered, either in machine.config file or in application configuration file. Configuration for each provider specifies the factory class and gives it three names, two of which are human-readable name and one - provider invariant name is used to refer to the provider in the connection string and SSDL.

Configuration for the providers included in the sample looks like this:

<system.data>
  <DbProviderFactories>
    <add name="EF Caching Data Provider"
         invariant="EFCachingProvider"
         description="Caching Provider Wrapper"
         type="EFCachingProvider.EFCachingProviderFactory, EFCachingProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b" />
    <add name="EF Tracing Data Provider"
         invariant="EFTracingProvider"
         description="Tracing Provider Wrapper"
         type="EFTracingProvider.EFTracingProviderFactory, EFTracingProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b" />
    <add name="EF Generic Provider Wrapper"
         invariant="EFProviderWrapper"
         description="Generic Provider Wrapper"
         type="EFProviderWrapperToolkit.EFProviderWrapperFactory, EFProviderWrapperToolkit, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b" />
  </DbProviderFactories>
</system.data>

This XML fragment can be copied/pasted into any project which needs to use EF provider wrappers or put in the machine.config to be shared by all applications.

Injecting into provider chain

In order to inject the provider into the provider chain, you have to modify SSDL files for your project as well as the connection string. There are two ways to do so: automated (which requires application code changes) or manual which can be done externally just by changing configuration file and SSDL file.

In order to create an EntityConnection with injected wrapped providers, you can use the provided helper method:

connection = EntityConnectionWrapperUtils.CreateEntityConnectionWithWrappers(
  connectionString, "EFTracingProvider", "EFCachingProvider")

You can then pass connection to ObjectContext constructor or use the connection to run ESQL queries as usual.

Alternative injection method

If you cannot change the application code, you have to make the modifications manually, which involves changing SSDL file and the connection string. Let’s take a quick look to see what SSDL and connection string look like today: The provider name is specified in the Provider attribute of the <Schema/> element:

<Schema Namespace="NorthwindEFModel.Store"
  Alias="Self"
  Provider="System.Data.SqlClient"
  ProviderManifestToken="2005"
  xmlns="http://schemas.microsoft.com/ado/2006/04/edm/ssdl">

Provider invariant name is also be specified in the connection string:

<connectionStrings>
  <add name="NorthwindEntities"
       connectionString="metadata=NorthwindEFModel.csdl | NorthwindEFModel.msl | NorthwindEFModel.ssdl;
                         provider=System.Data.SqlClient; 
                         provider connection string=&quot;Data Source=.\sqlexpress;
Initial Catalog=NorthwindEF;Integrated Security=True;MultipleActiveResultSets=True&quot;"
       providerName="System.Data.EntityClient" />
</connectionStrings>

In order to inject our own provider we need to override those to point to our provider. In SSDL, we put the name of the new provider in the Provider attribute and concatenate the previous provider with its provider manifest token in the ProviderManifestToken field, like this:

<?xml version="1.0" encoding="utf-8"?>
<Schema Namespace="NorthwindEFModel.Store"
  Alias="Self"
  Provider="EFCachingProvider"
  ProviderManifestToken="System.Data.SqlClient;2005"
  xmlns="http://schemas.microsoft.com/ado/2006/04/edm/ssdl">

Modifying connection string is a bit different – we need to put the pointer to the new *.ssdl, change provider name and add new keyword to provider connection string:

<connectionStrings>
  <add name="NorthwindEntities"
       connectionString="metadata=NorthwindEFModel.csdl | NorthwindEFModel.msl | NorthwindEFModel.Modified.ssdl;
                         provider=EFCachingProvider; 
                         provider connection string=&quot;wrappedProvider=System.Data.SqlClient;Data Source=.\sqlexpress;
Initial Catalog=NorthwindEF;Integrated Security=True;MultipleActiveResultSets=True&quot;"
       providerName="System.Data.EntityClient" />
</connectionStrings>

Specifying tracing configuration in the configuration file:

It is also possible to specify tracing configuration in App.config file. The following parameters are available:

<appSettings>
  <!-- write log messages to the console. -->
  <add key="EFTracingProvider.logToConsole" value="true" />
    
  <!-- append log messages to the specified file -->
  <add key="EFTracingProvider.logToFile" value="sqllog.txt" />
</appSettings>

Limitations and Disclaimers

The providers have not been extensively tested beyond what’s included in the sample code, so you should use tem at your own risk.

As with any other sample, Microsoft is not offering any kind of support for it, but if you find bugs or have feature suggestions, please use this blog’s contact form and let me know about them.

Entity Framework samples converted to Visual Basic

We have recently published Visual Basic versions of our MSDN Code Gallery samples for Entity Framework v3.5SP1.

The following translated samples are currently available in two languages (C# and VB)

ADO.NET Entity Framework Query Samples
http://code.msdn.com/EFQuerySamples

Persistence Ignorance (POCO) Adapter for Entity Framework
http://code.msdn.com/EFPocoAdapter

Transparent Lazy Loading for Entity Framework
http://code.msdn.com/EFLazyLoading

The samples have been fully converted to use and generate Visual Basic code instead of C#. More samples are in the pipeline and will be published soon - stay tuned.

Full list of current EF samples published by the Entity Framework team is always available at:
http://code.msdn.microsoft.com/adonetefx

New posts on EF Design Blog

I’d like to turn your attention to two blog posts on EF Design Blog about exciting new features planned for Entity Framework:

• Foreign Key Support – we’re introducing new type of association, which can be manipulated through references OR foreign key values available on entities themselves. We believe that this will greatly improve scenarios such as data binding, disconnected entities (N-Tier) and many others.
• Self-Tracking-Entities – it’s a simple way to have your entities track their state across tiers (based on custom code generation) and simple mechanism to save those changes to the database.

Your feedback is very important to us, please tell us what you think about those new features!

N-Tier Improvements for Entity Framework

We have recently published an article on API changes that we want to make in next version of Entity Framework in order to enable more N-Tier scenarios. Getting this right is very important to us and we're interested in getting as much of your feedback on the proposed design as possible.

Please read and comment on the article at: http://blogs.msdn.com/efdesign/archive/2008/11/20/n-tier-improvements-for-entity-framework.aspx

Using EF Oracle Sample Provider with EDM Designer

Many people are asking if it is possible to use EFOracleProvider with EDM Designer in Visual Studio 2008 SP1. The answer is yes, but because the sample doesn't include a DDEX provider required for VS integration, there are certain steps that have to be run manually.

I've compiled a step-by-step guide for those interested in trying this out (this assumes NorthwindEF sample database installed according to instructions included with the sample, but it should be straightforward to adjust it to your own setup)

PART 1 : INSTALLING ORACLE SAMPLE PROVIDER

1. Download and unzip EFOracleSampleProvider.zip from http://code.msdn.com/EFOracleProvider

2. Follow instructions in the README.txt to set up a sample database.

3. Open elevated Visual Studio instance. Build the sample project.

4. Open elevated command prompt and open machine.config using notepad: 
notepad %WINDIR%\Microsoft.NET\Framework\v2.0.50727\config\machine.config

5. Find <DbProviderFactories> section and add EFOracleProvider entry:

<add name="EF Oracle Data Provider" invariant="EFOracleProvider" 
     description="EF Provider for Oracle testing" 
     type="EFOracleProvider.EFOracleProviderFactory, EFOracleProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b"/>

6. The completed system.data section has to look similar to this:

<system.data>
  <DbProviderFactories>
    <add name="Odbc Data Provider" invariant="System.Data.Odbc" description=".Net Framework Data Provider for Odbc" type="System.Data.Odbc.OdbcFactory, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
    <add name="OleDb Data Provider" invariant="System.Data.OleDb" description=".Net Framework Data Provider for OleDb" type="System.Data.OleDb.OleDbFactory, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
    <add name="OracleClient Data Provider" invariant="System.Data.OracleClient" description=".Net Framework Data Provider for Oracle" type="System.Data.OracleClient.OracleClientFactory, System.Data.OracleClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
    <add name="SqlClient Data Provider" invariant="System.Data.SqlClient" description=".Net Framework Data Provider for SqlServer" type="System.Data.SqlClient.SqlClientFactory, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
    <add name="EF Oracle Data Provider" invariant="EFOracleProvider" description="EF Provider for Oracle testing" type="EFOracleProvider.EFOracleProviderFactory, EFOracleProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=def642f226e0e59b"/>
    <add name="Microsoft SQL Server Compact Data Provider" invariant="System.Data.SqlServerCe.3.5" description=".NET Framework Data Provider for Microsoft SQL Server Compact" type="System.Data.SqlServerCe.SqlCeProviderFactory, System.Data.SqlServerCe, Version=3.5.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91"/>
  </DbProviderFactories>
</system.data>

PART 2: GENERATING MODEL FROM ORACLE DATABASE

7. Create a new project in VS. For simplicity let's create a console application

8. Open elevated command prompt. Enter the directory that contains the newly created project and run the following command:

%WINDIR%\Microsoft.NET\Framework\v3.5\edmgen.exe /provider:EFOracleProvider /mode:fullgeneration 
      /connectionstring:"data source=XE;user id=edmuser;password=123456" /project:NorthwindEFModel

The output should be:

Microsoft (R) EdmGen version 3.5.0.0
Copyright (C) 2008 Microsoft Corporation. All rights reserved.

Loading database information...
warning 6005: The data type 'timestamp(9)' is not supported, the column 'OrderDate' in table 'dbo.Orders' was excluded.
warning 6005: The data type 'timestamp(3)' is not supported, the column 'RequiredDate' in table 'dbo.Orders' was excluded.
Writing ssdl file...
Creating conceptual layer from storage layer...
Writing msl file...
Writing csdl file...
Writing object layer file...
Writing views file...

Generation Complete -- 0 errors, 2 warnings

9. This will create a bunch of NorthwindEFModel.* files for you.

10. Open Northwind.ssdl file in a text editor and replace all instances of Schema="dbo" with empty string (this is needed because tables in the sample Oracle database don't use a schema)

11. In order to use generated model in the EF Ddesigner, we have to create NorthwindEFModel.edmx file. This can be done manually (just copy/paste contents of individual files into an empty EDMX as indicated by the comments) or by using EdmGen2 tool from Code Gallery:

C:\Path\To\EdmGen2.exe /toedmx NorthwindEFModel.csdl NorthwindEFModel.ssdl NorthwindEFModel.msl

12. This will create NorthwindEFModel.edmx, which we can add to the project in VS.

13. At this point you can now delete the following files generated by EdmGen.exe, which won't be necessary:

• NorthwindEFModel.csdl
• NorthwindEFModel.ssdl
• NorthwindEFModel.msl
• NorthwindEFModel.ObjectLayer.cs
• NorthwindEFModel.Views.cs

PART 3: TESTING GENERATED MODEL

14. The only remaining thing to do is to add App.config file with connection string for our Oracle database:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <connectionStrings>
    <add name="NorthwindEFModelContext" 
         connectionString="provider=EFOracleProvider;
                           metadata=res://*/NorthwindEFModel.csdl|res://*/NorthwindEFModel.ssdl|res://*/NorthwindEFModel.msl;
                           Provider Connection String='data source=XE;user id=edmuser;password=123456'" 
         providerName="System.Data.EntityClient" />
  </connectionStrings>
</configuration>

15. We can now try out our model by running a sample LINQ to Entities query:

using (NorthwindEFModelContext context = new NorthwindEFModelContext())
{
    foreach (var c in context.Customers.Where(c=>c.City == "Seattle"))
    {
        Console.WriteLine(c.CompanyName);
    }
}

EF POCO Adapter updated (v1.03)

I've just posted another round of updates to EF POCO Adapter based on feedback from users. This set of changes is focused on N-tier scenarios and detached entities (it adds missing wrappers for EF APIs and fixes behavior of existing ones).

Change log:

• Added wrapper for ObjectContext.ApplyPropertyChanges() to IEntitySet<T>
• Added wrapper for ObjectContext.Detach() to IEntitySet<T>
• Fixed handling of detached/out-of-scope entities w.r.t lazy loading and attaching to other contexts. Lazy loading outside of scope will do nothing if the context had DeferredLoadingEnabled=false, otherwise will throw.
• Added partial OnZZZCreated() methods on generated contexts, adapters and proxies to enable integration with 3rd-party extensions
• Fixed code generation from EDMX.
• EFPocoContext.EnableLazyLoading has been renamed to EFPocoContext.DeferredLoadingEnabled to better align with Linq to SQL and our plans for EF v2
• Fixed adapter-POCO synchronization for complex types

The new release is here: http://code.msdn.microsoft.com/EFPocoAdapter/Release/ProjectReleases.aspx?ReleaseId=1580

Big thanks to everyone who sent me comments, bug reports and improvement suggestions for this release. This is very valuable feedback as it allows us to better understand user scenarios around POCOs for EF v2.

EF POCO Adapter updated (v1.02)

I've just posted updates to EF POCO Adapter (v1.02) which resolves a couple of issues spotted by users:

1. Detached proxies (created by EFPocoContxt.CreateObject<T>) as well as proxies in the Added state were failing on property navigation. This has been fixed, and the property navigation doesn't attempt the load in one of these states. Thanks to Corey Gaudin for reporting this.
2. GetAdapterObject() was failing for proxies created with CreateObject<T>. This has been fixed by ensuring that adapters are always cached whenever proxies are present. As a side effect of this change, DetectChanges() should be a bit faster, as we don't have to walk object state manager anymore. Thanks to Corey Gaudin for reporting this.
3. Fixed code generator to eliminate compilation warning of unused property.

The updated version is available here.

Persistence Ignorance (POCO) Adapter for Entity Framework V1

One of the limitations of Entity Framework V1 released as part of .NET Framework 3.5 SP1 and Visual Studio 2008 SP1 is lack of support for POCO entity objects. POCO stands for Plain Old CLR Object and refers to an object that does not have any persistence concerns and is not aware of persistence framework. Modern development approaches such as DDD (Domain-Driven Design) and TDD (Test-Driven Development) prefer clear separation of concerns which implies persistence ignorance.

In order to be consumable by EF, entity classes have to:

• Derive from EntityObject class or implement at least one mandatory IPOCO interface: IEntityWithChangeTracker. If the entity participates in relationships it also has to implement IEntityWithRelationships interface. Implementing those interfaces is not hard, but causes entity objects to have a dependency on EF assembly, which is sometimes not desirable
• Provide assembly, class and property-level attributes to define mapping from CLR space to model space (we call those O-C mapping attributes)
• Use Entity-Framework-provided relationship classes: EntityReference<T>, EntityCollection<T> and RelationshipManager instead of CLR collections (List<T>, ICollection<T>)

Clearly those requirements make EF’s entity objects persistence-aware.

It is possible to write (generate) an adapter layer that will translate between POCO objects and Entity Framework-aware objects and provide services on top of POCO objects, such as:

• Change tracking (snapshot-based and proxy-based when possible)
• Transparent lazy loading
• Immutable Value Objects
• Queries (LINQ and Entity SQL)
• Shadow state (maintaining certain persistence-related fields outside of entity class)

We have recently published a sample called EFPocoAdapter that includes code generator and a supporting library that implements POCO object tracking on top of Entity Framework V1. This serves two purposes:

• To demonstrate that it is possible to track POCO objects using the current version of EF
• To gather feedback on POCO-specific features (such as transparent lazy loading, immutable objects, change detection, etc) that customers would like to see implemented natively in future versions of Entity Framework.

The source code is released under Microsoft Public License on Code Gallery. There is no formal support for this sample, but we are interested in improving and evolving it based on user’s feedback. For legal reasons we cannot accept source code contributions, but you are more than welcome to share feature ideas and POCO usage scenarios with us.

Adapter Design

The main idea used when implementing EFPocoAdapter was the use of adapter objects. Adapters are classes that are EF-friendly (in terms of O/C mapping, relationship manager and so on), but internally manage and materialize POCO objects. Materialization is done transparently by delegating property setters and association change notifications to the POCO object. This way, whenever Entity Framework materializes (hydrates) an adapter object, a corresponding POCO object is also created. Note that property setters not only delegate values to POCO objects, but they also store a copy for themselves – this helps later in the change detection process.

Let’s take a look at a simple adapter class on top of a Region POCO that has 2 properties: RegionID and RegionDescription. It has [EdmEntityType] attribute and [EdmScalarProperty] attributes on every property. Every property setter delegates property write to corresponding property on _pocoEntity object along with firing events necessary for change tracking. A copy of object state is stored in _RegionID and _RegionDescription properties.

[EdmEntityType(NamespaceName="NorthwindEFModel", Name="Region")]
public class RegionAdapter : PocoAdapterBase<NorthwindEF.Region>, IPocoAdapter<NorthwindEF.Region>
{
    private String _RegionDescription;
    public RegionAdapter() { }

    public RegionAdapter(NorthwindEF.Region pocoObject) : base(pocoObject) { }
    // some methods removed...

    public override void DetectChanges()
    {
        // change detection – see below
    }

    private Int32 _RegionID;
 
    [EdmScalarProperty(EntityKeyProperty=true, IsNullable=false)]
    public Int32 RegionID
    {
        get
        {
            return _RegionID;
        }
        set
        {
            if (_pocoEntity != null)
            {
               // pass the property value to POCO object
                PocoEntity.RegionID = value;
            }
            ReportPropertyChanging("RegionID");
            _RegionID = value;
            ReportPropertyChanged("RegionID");
        }
    }

    [EdmScalarProperty(EntityKeyProperty=false, IsNullable=false)]
    public String RegionDescription
    {
        get
        {
            return _RegionDescription;
        }
        set
        {
            if (_pocoEntity != null)
            {
                PocoEntity.RegionDescription = value;
            }

            ReportPropertyChanging("RegionDescription");
            _RegionDescription = value;
            ReportPropertyChanged("RegionDescription");
        }
    }
}

Change detection is a step that occurs just before changes are saved to the database. Adapter objects compare their stored values and relationship information against information in POCO objects (that users may have changed). If the values differ, adapter sets its own property to the value from POCO object which essentially triggers EF change tracker and ensures that SaveChanges() will persist the object correctly. Change detection looks similar to (in reality more corner cases are covered):

public override void DetectChanges()
{
  // change detection - simplified for explanation purposes
  base.DetectChanges();
  if (PocoEntity.RegionID != _RegionID)
    this.RegionID = PocoEntity.RegionID;
  if (PocoEntity.RegionDescription != _RegionDescription)
    this.RegionDescription = PocoEntity.RegionDescription;
}

DetectChanges() method seems heavy and it really is – it has to store and compare all property values of all objects. Fortunately we can optimize the cost of change detection by introducing Proxies. Proxies are objects that implement POCO interface (by deriving from a POCO class), but they have a hidden dependency on an adapter object. They override all property getters and setters that can be overridden and provide just-in-time change tracking. See the following example of a proxy for Territory class that has a single virtual property called TerritoryDescription:

public partial class TerritoryProxy : NorthwindEF.Territories.Territory
{
    NorthwindEF.Territories.PocoAdapters.TerritoryAdapter _adapter;

    object IEntityProxy.Adapter
    {
        get { return _adapter; }
    }

    public TerritoryProxy(TerritoryAdapter adapter) { _adapter = adapter; }

    public override String TerritoryDescription
    {
        get { return this._adapter.TerritoryDescription; }
        set
        {
           // pass the value to the POCO property
            base.TerritoryDescription = value;
            if (this._adapter.Context != null 
               && value != this._adapter.TerritoryDescription)
            {
               // raise change detection event
              _adapter.Context.RaiseChangeDetected(this, 
                     "TerritoryDescription",
                     this._adapter.TerritoryDescription, value);
            }
           // pass the value to the Adapter which triggers change tracker
            this._adapter.TerritoryDescription = value;
        }
    }
}

Adapters are not meant to be seen in any scenarios and users should not be aware of their existence. To do this, we need to expose a programmable layer that hides adapters completely. We need to be able to specify LINQ to Entities queries in terms of POCO objects (similar to ObjectQuery<T>) and we want inserts/attaches/deletes to work too. We are introducing a new object called EntitySet<T>, that combines function of ObjectQuery<T> and object-management methods (AddToX(), AttachTo(), DeleteObject()). We are also generating a replacement for ObjectContext, which makes the end-to-end programming experience with POCOs more natural. See the "Working with POCO classes” section below for more usage information.

Components

The sample includes two main components:

·         EFPocoAdapter - library which you reference in your applications

·         EFPocoClassGen - code generator which generates the adapter layer

To use the sample you will need five additional pieces of the puzzle:

·         POCO classes (you typically write them by hand, the sample includes NorthwindEF project)

·         POCO container – root object used for querying POCO (equivalent of ObjectContext for persistence-aware objects).

·         POCO adapter – generated

·         Entity Data Model (CSDL,SSDL & MSL files)

·         Connection string (in your App.config or in code)

In a typical case you will generate POCO classes or write them by hand, but EFPocoClassGen has a mode to generate them from an existing model.

POCO container can be written by hand or generated from a CSDL file.

POCO Adapter for your model is generated at compile time or at runtime. In the former case, you will be able to debug the adapter code as the source code will be stored on your hard drive. In the latter case, your application will include just classes written by hand and no artificial artifacts.

Compile-Time Adapter Generation

EFPocoClassGen has three modes and can generate:

·         POCO Adapter (default)

·         POCO container

·         POCO classes

Assuming you want to generate a POCO adapter for classes that are stored in NorthwindEF.dll and your model is in NorthwindEFModel.csdl, you should run the following command:

EFPocoClassGen.exe /incsdl:NorthwindEFModel.csdl /outputfile:PocoAdapter.cs 
                   /ref:NorthwindEF.dll /map:NorthwindEFModel=NorthwindEF

This will generate PocoAdapter.cs file based on NorthwindEFModel.csdl (specified in /incsdl option). By specifying /ref argument, you allow class generator to scan the assembly and generate certain optimizations such as proxy-based change tracking for virtual properties. This is optional but if you don’t provide the assembly references you will have to put optimization hints in the CSDL file manually. /map allows you to use a different namespace in CSDL than in your classes. Code generator has some more options; to see them all, run it without any parameters.

It is usually best to automate code generation for a project by putting the appropriate commands in Pre-Build Step of VS project or in BeforeCompile target in MSBuild script. The generator will not regenerate the output file if the input hasn’t changed, so you will not get unnecessary recompiles because of that.

POCO Container

The POCO Container is a simple class that derives from EFPocoContext. It has a bunch of properties representing top-level entity sets. All methods that developers interact with are defined in the base class or in entity set methods.

public partial class NorthwindEntities : EFPocoContext<NorthwindEF.PocoAdapters.NorthwindEntitiesAdapter>
{
    public NorthwindEntities()
     : base(new NorthwindEntitiesAdapter()) { }
    public NorthwindEntities(string connectionString)
     : base(new NorthwindEntitiesAdapter(connectionString)) { }
    public NorthwindEntities(EntityConnection connection)
     : base(new NorthwindEntitiesAdapter(connection)) { }

    public IEntitySet<NorthwindEF.Employee> Employees
    {
        get { return GetEntitySet<NorthwindEF.Employee>("Employees"); }
    }
 
    public IEntitySet<NorthwindEF.Order> Orders
    {
        get { return GetEntitySet<NorthwindEF.Order>("Orders"); }
    }
 
    // additional entity sets removed for brevity
}

To generate POCO container, run the same command as above with /mode:PocoContainer – just be sure to specify a different output file:

EFPocoClassGen.exe /mode:PocoContainer /incsdl: NorthwindEFModel.csdl /outputfile:NorthwindEntities.cs 
                   /ref:NorthwindEF.dll /map:NorthwindEFModel=NorthwindEF 

If you prefer, you can also write the class by hand – the class needs to have as many entity set properties as you require (not all EntitySets from CSDL have to exposed) and has to have one or more of the constructors as seen above.

Setting Up the Solution

The recommended way of setting up your solution when using compile-time code generation is:

1.       Put all POCO classes in a separate assembly, say NorthwindEF.dll. The assembly doesn’t need to have any references (other than mscorlib.dll and maybe System.Core.dll)

2.       Create a separate library project for your adapter, name it for example NorthwindEF.PocoAdapter.dll Add references to System.Data.Entity.dll, EFPocoAdapter.dll and NorthwindEF.dll

3.       Set up pre-build scripts that will generate PocoAdapter.cs and NorthwindEntities.cs. Build the project once and add generated files to the project (you don’t have to add them to source control as they will be regenerated at build time)

4.       In your code that uses the POCO adapter, add references to: EFPocoAdapter.dll, NorthwindEF.dll, NorthwindEF.PocoAdapter.dll and System.Data.Entity.dll

POCO Adapter can be also generated at runtime using EFPocoAdapterGenerator class:

// set up generator
var generator = new EFPocoAdapterGenerator<MyHandWrittenNorthwindEntities>();
generator.EntityConnectionString = "name=NorthwindEntities";
generator.EdmToClrNamespaceMapping["NorthwindEFModel"] = "NorthwindEF";
generator.GenerateProxies = true;
generator.ObjectAssemblies.Add(typeof(Customer).Assembly);
generator.ObjectAssemblies.Add(typeof(Territory).Assembly);

// generate adapter and create context factory
var factory = generator.CreateContextFactory();

// use generated context
using (var context = factory.CreateContext())
{
    Assert.IsInstanceOfType(context, typeof(MyHandWrittenNorthwindEntities));
    var cust = context.Customers.First(c => c.CustomerID == "ALFKI");
}

See EFPocoAdapterGeneratorTests class for a full working example.

Working with POCO classes

Usage patterns of the generated POCO adapter layer are very similar to regular EF, but because objects are now persistence-ignorant, some APIs have been changed. This section describes important differences:

Managing POCO Container

POCO Container should be managed in the same way as a regular ObjectContext. Same rules and recommendations apply. It is usually best to have short-living contexts and ensure proper disposal of all resources, preferably with the use of the using statement:

using (NorthwindEntities context = new NorthwindEntities())
{
    // use context
}

Queries

EFPocoContext exposes CreateQuery<T> which internally wraps ObjectQuery<T>. POCO context also exposes entity sets that can be used to query objects using LINQ to Entities, ESQL builder methods and Entity SQL. Of course, you can still use EntityClient and EntityCommand to do queries in the value layer.

In short: queries look exactly the same as in EF, just entity types and query object types are different.

using (NorthwindEntities context = new NorthwindEntities())
{
    Customer k = (from c in context.Customers 
                  where c.CustomerID == "ALFKI"
                  select c).First();
    Console.WriteLine(k.Address.City);
}

See QueryTests.cs and BuilderMethodsTests.cs for more examples of usage.

Inserting, Removing and Attaching Objects

POCO context code exposes IEntitySet<T> properties that can be used to query (because they derive from IQueryable<T>) but also insert, remove and attach objects to the context. This is a change from EF where you used to use AddObject(), AttachTo() and DeleteObject() on ObjectContext class. The idea of exposing EntitySet<T> object to aggregate all operations to entity sets was originally introduced in the EFExtensions project:

using (NorthwindEntities context = new NorthwindEntities(conn))
{
  // add new object
  context.Customers.InsertOnSaveChanges(new Customer { 
      CustomerID = "YYYY", 
      CompanyName = "Una Firma" 
  });

  // attach existing object for tracking
  var product = new Product(new Supplier()) { ProductID = 1 };
  context.Products.Attach(product);

  // locate and delete object
  var cust = context.Customers.First(c => c.CustomerID == "XXXX");
  context.Customers.DeleteOnSaveChanges(cust);

  // persist changes
  context.SaveChanges();
}

See EntitySetTests.cs and CRUDTests.cs for more examples.

Transparent Lazy Loading

Lazy loading happens automatically when you reference a virtual property on an entity object that was retrieved from a query. Because the property is marked as virtual, EFPocoClassGen generates a derived class called a proxy, which overrides the property and provides lazy-loading behavior on it.

Similar thing happens for collections – they get loaded on first access, but in this case the collection accessor doesn’t have to be declared virtual but its type must be an interface (ICollection<T>, IList<T>) instead of a concrete class (such as List<T>). In this case POCO adapter initializes the collection with a tracked collection that provides transparent lazy loading on enumeration and other types of access.

You can disable this lazy loading behavior at runtime, by setting EFPocoContext.EnableLazyLoading property to false or at compile time, by passing /enableLazyLoading:false to EFPocoClassGen. Note that if compile your adapter classes without lazy loading support, you won’t be able to enable lazy loading at runtime.

If the lazy loading has been disabled, you can still load a reference or collection you want to navigate to by using new methods on EFPocoContext:

·         IsPropertyLoaded(o,p) – determines whether object or collection represented by property p on object o has been loaded. Property can be specified as a string or as a strongly typed lambda.

·         LoadProperty(o,p) – loads the related end of the navigation property p on object o. Property can be specified as a string or as a strongly typed lambda.

See LazyLoadingTests.cs for example usage.

Object-to-Conceptual Mapping

You can specify the namespace mapping (from conceptual space to CLR space) when generating POCO adapter using /map parameter to EFPocoClassGen. If you need to fine-tune this (for example to materialize a class from another assembly or namespace), you can attach an attribute called ClrType to <EntityType> or <ComplexType> in your CSDL schema.

<EntityType Name="CurrentEmployee" BaseType ="NorthwindEFModel.Employee" 
            objectmapping:ClrType="Some.Clr.Namespace.CurrentEmployee" />

The sample comes with NorthwindEF schema split into two assemblies: NorthwindEF and NorthwindEF.Territories.dll and classes live in three namespaces to demonstrate this concept. See NorthwindEFModel.csdl for details.

Read-Only Value Objects

You can map your entities to classes that are read-only and all they do is a public constructor that initializes all properties. You can also have some properties read-only and others settable. All you have to do to get this to work is to use a constructor parameter naming convention. If the property is read-only, the adapter will pass its initial value as the corresponding constructor argument:

public struct CommonAddress
{
    private string _address;
    private string _city;
    private string _region;
    private string _postalCode;
    private string _country;

    public CommonAddress(string address, string city, string region, 
                         string postalCode, string country)
    {
        _address = address;
        _city = city;
        _region = region;
        _postalCode = postalCode;
        _country = country;
    }
 
    public String Address
    {
        get { return _address; }
    }

    public String City
    {
        get { return _city; }
    }
 
    public String Region
    {
        get { return _region; }
    }
 
    public String PostalCode
    {
        get { return _postalCode; }
    }

    public String Country
    {
        get { return _country; }
    }
}

See CommonAddress.cs, Product.cs and Category.cs for more examples of usage.

Change Tracking

Change tracking for scalar properties and complex types is implemented in one of two modes: snapshot-based or proxy-based. Snapshot-based change tracking means that when EF materializes an object, the adapter stores a copy of its field values for further comparison. This way it knows which fields have changes and can notify object state manager accordingly.

Proxy-based change tracking is used when you declare your POCO object property to be virtual. EFPocoClassGen generates a proxy class that overrides all virtual properties in your POCO classes and injects notifications that update object state manager just-in-time.

Change tracking for collections can also use snapshots or proxies – in the first case EntityCollection<T> managed by ObjectStateManager is used as original values, if proxies are used (only possible when collection formal type is ICollection<T> or IList<T>) the Object State Manager gets updated just-in-time.

Change detection happens just before changes are saved to the database. You can force it by calling DetectChanges() on ObjectStateManager. Forcing change detection can be useful to do relationship fixup before changes are saved. It also raises change tracking events.

There are 2 change tracking events on EFPocoContext:

·         ChangeDetected that gets raised whenever a change is detected on a scalar, complex type property or a relationship where related end is of single cardinality

·         CollectionChangeDetected that gets raised whenever a change in collection contents is detected

You can hook up those events for debugging purposes, but there may be other possible uses, such as auditing or tracing.

Compiled Queries

Compiled queries work in the same way as in EF, except that you use EFPocoCompiledQuery.Compile() instead of CompiledQuery.Compile()

var getCustomerByID = EFPocoCompiledQuery.Compile(
   (NorthwindEntities context, string customerID) =>
       context.Customers.Where(c => c.CustomerID == customerID).First());

using (NorthwindEntities context = new NorthwindEntities())
{
    var alfki = getCustomerByID(context, "ALFKI");
}

See CompiledQueryTests.cs for more information.

Working With Proxies

Proxies are injected automatically when EFPocoAdapter materializes new POCO objects. This applies to all objects being results of queries and does NOT apply to objects passed by users.

Users can force a proxy to be created by calling EFPocoContext.CreateObject<T> instead of new T().

using (NorthwindEntities context = new NorthwindEntities())
{
    var cust = context.CreateObject<Customer>();
    Assert.IsTrue(context.IsProxy(cust));
}

Users can also enable and disable proxy use (which effectively disables transparend lazy loading) by setting EFPocoContext.EnableChangeTrackingUsingProxies property to false. Note that this applies to newly materialized objects, not to objects that have already been created and managed by the context. You can convert your existing POCO objects to Proxies with and back with ConvertPocoToProxies() and ConvertProxiesToPoco() methods accordingly.

Shadow State

Sometimes we want to hide certain properties from POCO objects and let the framework managed them. For example, we typically don’t need to see GUIDs, timestamp or version values used for optimistic concurrency checks as long as the framework can manage them automatically. The state for those properties can still be maintained by the adapter, but the property doesn’t have to be present on the POCO object.

In the similar manner we may want to hide primary keys if they don’t have business value, provided that there you have defined your column as IDENTITY.

For example, the following POCO class:

public class Territory
{
    public virtual string TerritoryDescription { get; set; }
}

Is declared in CSDL as:

<EntityType Name="Territory" objectmapping:ClrType="NorthwindEF.Territories.Territory">
  <Key>
      <PropertyRef Name="TerritoryID" />
  </Key>

  <Property Name="TerritoryID" Type="Int32" Nullable="false"
            objectmapping:ChangeTracking="Hidden" />
  <Property Name="TerritoryDescription" Type="String" Nullable="false" />
</EntityType>

Note the use of objectmapping:ChangeTracking="Hidden" to notify the code generator that the property is not present on the POCO class. This is optional if you use /ref: as in this case the code generator can detect that the property is missing and automatically assume “shadow” property.

Other Properties of EFPocoContext

EFPocoContext exposes several helper properties, similar to ObjectContext:

·         MetadataWorkspace – gives you access to metadata workspace for the model

·         QueryProvider – exposes IQueryProvider used to translate LINQ queries to the form that Linq To Entities understands

·         Connection – gives you access to underlying EntityConnection

Limitations

The first version of the sample has certain limitations. Not all APIs of EF are wrapped and exposed in a POCO-friendly manner. You can still access the following APIs through adapter objects:

·         ObjectStateManager/ObjectStateEntry

·         Query modes other than AppendOnly

·         ObjectResult and other databinding-specific classes

It should be relatively straightforward to add the wrappers as needed using techniques described in the article.

Transparent Caching Support for Entity Framework one-pager

We've just posted a one-pager on transparent caching support in Entity Framework on our EFDesign blog.

http://blogs.msdn.com/efdesign/archive/2008/07/09/transparent-caching-support-in-the-entity-framework.aspx

We want to make sure that you will be able to interface EF caching with various caching solutions, so we would be very grateful for comments on proposed ICache interface.

Software Development Meme

I was recently tagged by Danny to answer questions about my programming experience as part of a longer chain letter, so here they are:

How old were you when you first started programming

I was 8 (some 0x17 years ago) when I got my first computer: Timex 2048 which was ZX Spectrum clone sold in Poland. Originally I was planning to use it for gaming. Unfortunately my computer didn’t come with an essential component (a tape recorder J), so I really had no choice but to learn programming.

How did you get started in programming?

Fortunately Timex had a built-in BASIC language interpreter and it came with a nice language manual. I started exploring that and found that I could do cool things with programs. First things I tried were graphics-related and used PLOT/DRAW combo extensively to produce some nice geometric patterns.

Later I realized that there was another language that my computer understood which was much faster: Z80 assembly language (the computer manual even included an introduction to Z80 assembly  - compare it with today’s manuals!) so I learned about it (with the help of GENS assembler and MONS debugger and “The Complete Spectrum ROM Disassembly” book) and quickly fell in love with it.

Side-note: because programs for ZX Spectrum were mostly distributed on audio tapes (with a buzzing sound on it kind of similar to a modem negotiating connection), some people in Poland came up with a super clever idea of distributing programs during radio shows in the middle of the night. The show was called “RadioKomputer” and that’s how I got some educational programs.

What was your first language?

My first languages were Sinclair Basic and Z80 assembly, many years later I moved to a PC (386 SX with 2MB of RAM) and then I started playing with Turbo Pascal & Turbo C from Borland.

What was the first real program you wrote?

I don’t remember what the first real program was, there were many of them, most were graphics related. I wrote some flashy graphics demos in assembly language which included some scrolling text, nice graphics, explosions, music (such as “Axel F" theme or Jean Michel Jarre because it was the only thing that sounded ok on Timex beeper). We demoed those on computer fairs which were organized every week in a nearby high school.

Later, on PC I wrote my first Norton Commander clone. Everyone else I knew had also written their own clones, too – I guess it was kind of a sport in the nineties. I was proud of my Jarek Commander, because it only took <500 bytes of resident memory while original NC consumed some 4 KB (in the world of 640K this was a huge difference).

My first big program (sometime around 1994-95) was a graphics program called “Canaletto” named after Bernardo Belotto who was a 18th century painter. It was a fully featured graphics package for DOS which included drawing and image processing capabilities (like MS Paint on steroids with some basic image processing effects which you can see in Adobe Photoshop today). I wrote everything from scratch in Turbo C & 8086 assembly, including my own windowing library, printing support, image processing algorithms and even some graphics drivers for non-VESA video cards. We wanted to sell Canaletto to Get Rich Quickly(tm). We prepared the first “batch” in my friend’s kitchen using his color printer and even shrink-wrapped the boxes manually using his gas stove. We sold whopping 3 copies and 33% of buyers have registered their copies by mailing in registration cards.

What languages have you used since you started programming?

Sinclair Basic, Z80 assembly, Turbo Pascal, C/C++, 80386 assembly, Java, JavaScript, Perl, Python, C#, Awk, Unix shell scripts, SQL, HTML, XML/XSLT/XPath/XQuery. I have also experimented with countless other languages such as Smalltalk, Boo, functional languages and more.

What was your first professional programming gig?

My first professional software development project was a PC game called “RoboRumble” which was produced in 1998 and released worldwide. It was a real-time strategy in 3D with some very cool accelerated graphics. I was responsible for gameplay, AI, multiplayer and various putting-it-all-together tasks.

After that experience I moved on to do some more “serious” development. I spent next 7 years building large workflow systems for enterprises. I implemented at least three O/R Mappers along the way. I just didn’t know that those things were called O/RM back then.

If you knew then what you know now, would you have started programming? 

Absolutely!

If there is one thing you learned along the way that you would tell new developers, what would it be?

It’s all about requirements. No matter how cool the code is, it is useless if it doesn’t do what the customer wants. Customers often don’t know what they want, so it takes a lot of communication skills to “extract” that information. Also, be prepared for ever-changing requirements and incorporate them in your development process. BDUF rarely wins against agile practices. Usability is also extremely important; even if the customer does not realize the importance of it, you should do all you possibly can to make common user scenarios very easy to use and other scenarios possible.

What's the most fun you've ever had ... programming?

Definitely working on RoboRumble was a lot of fun. In the end-game we spent like 20 hours a day working on finishing the product (we literally left @4 AM just to get back at 8AM) but despite some local frustrations we absolutely loved it.

Who’s next

Because of the exponential nature of this chain letter it is increasingly complex to find more people with blogs, but here are the next nominees:

Zlatko Michailov
David Sceppa
Ju-Yi Kuo
Maurycy Markowski

Sample Entity Framework Provider for Oracle

I wanted to let you know that we have just released Sample Entity Framework Provider for Oracle on MSDN Code Gallery.

http://code.msdn.com/EFOracleProvider

The provider is implemented using essentially the same technique as EFSampleProvider, but targets System.Data.OracleClient instead of System.Data.SqlClient. The provider is compatible with Visual Studio SP1 Beta and the code is released under Microsoft Public License (Ms-PL).

There are three notable features of this provider:

• Non-default type mappings. In EFOracleProviderManifest we are supporting non-default type mappings:
  • EFOracle.number(1,0) maps to Edm.Boolean
  • EFOracle.number(5,0) maps to Edm.Int16
  • EFOracle.number(11,0) maps to Edm.Int32
  • EFOracle.raw(16) maps to Edm.Guid

Here's the interesting piece of code taken from EFOracleProviderManifest.GetEdmType() that does it. It looks at facets attached to the given store type and translates certain combinations of them into mentioned EDM types.

case "number":
    {
        byte precision;
        byte scale;

        if (MetadataHelpers.TryGetPrecision(storeType, out precision) &&
            MetadataHelpers.TryGetScale(storeType, out scale))
        {
            if (precision == 1 && scale == 0)
                return TypeUsage.CreateDefaultTypeUsage(PrimitiveType.GetEdmPrimitiveType(PrimitiveTypeKind.Boolean));

            if (precision == 5 && scale == 0)
                return TypeUsage.CreateDefaultTypeUsage(PrimitiveType.GetEdmPrimitiveType(PrimitiveTypeKind.Int16));

            if (precision == 11 && scale == 0)
                return TypeUsage.CreateDefaultTypeUsage(PrimitiveType.GetEdmPrimitiveType(PrimitiveTypeKind.Int32));

            return TypeUsage.CreateDecimalTypeUsage(edmPrimitiveType, precision, scale);
        }
        else
        {
            return TypeUsage.CreateDecimalTypeUsage(edmPrimitiveType);
        }
    }

• Data type coercion in the data reader. Data provider analyzes Canonical Query Tree (CQT) and remembers expected CLR types for each result column ordinal. Whenever this data type is different from the data returned by wrapped OracleDataReader, the value is coerced using Convert.ChangeType().

The code below extracts ExpectedColumnTypes from the canonical query tree:

// Set expected column types for DbQueryCommandTree
DbQueryCommandTree queryTree = commandTree as DbQueryCommandTree;
if (queryTree != null)
{
    DbProjectExpression projectExpression = queryTree.Query as DbProjectExpression;
    if (projectExpression != null)
    {
        EdmType resultsType = projectExpression.Projection.ResultType.EdmType;

        StructuralType resultsAsStructuralType = resultsType as StructuralType;
        if (resultsAsStructuralType != null)
        {
            command.ExpectedColumnTypes = new PrimitiveType[resultsAsStructuralType.Members.Count];

            for (int ordinal = 0; ordinal < resultsAsStructuralType.Members.Count; ordinal++)
            {
                EdmMember member = resultsAsStructuralType.Members[ordinal];
                PrimitiveType primitiveType = member.TypeUsage.EdmType as PrimitiveType;
                command.ExpectedColumnTypes[ordinal] = primitiveType;
            }
        }
    }
}

// Set expected column types for DbFunctionCommandTree
DbFunctionCommandTree functionTree = commandTree as DbFunctionCommandTree;
if (functionTree != null)
{
    if (functionTree.ResultType != null)
    {
        Debug.Assert(MetadataHelpers.IsCollectionType(functionTree.ResultType.EdmType),
            "Result type of a function is expected to be a collection of RowType or PrimitiveType");

        EdmType elementType = MetadataHelpers.GetElementTypeUsage(functionTree.ResultType).EdmType;

        if (MetadataHelpers.IsRowType(elementType))
        {
            ReadOnlyMetadataCollection<EdmMember> members = ((RowType)elementType).Members;
            command.ExpectedColumnTypes = new PrimitiveType[members.Count];

            for (int ordinal = 0; ordinal < members.Count; ordinal++)
            {
                EdmMember member = members[ordinal];
                PrimitiveType primitiveType = (PrimitiveType)member.TypeUsage.EdmType;
                command.ExpectedColumnTypes[ordinal] = primitiveType;
            }

        }
        else if (MetadataHelpers.IsPrimitiveType(elementType))
        {
            command.ExpectedColumnTypes = new PrimitiveType[1];
            command.ExpectedColumnTypes[0] = (PrimitiveType)elementType;
        }
        else
        {
            Debug.Fail("Result type of a function is expected to be a collection of RowType or PrimitiveType");
        }
    }
}

The provider needs to store this information in the DbCommand and pass down to DbDataReader. In the reader implementation, actual data coercion is done in the following manner:

public partial class EFOracleDataReader : DbDataReader 
{
    private PrimitiveType[] _expectedColumnTypes;

    // ...

    public override DateTime GetDateTime(int ordinal)
    {
        return (DateTime)GetValue(ordinal);
    }

    public override decimal GetDecimal(int ordinal)
    {
        return (decimal)GetValue(ordinal);
    }

    public override object GetValue(int ordinal)
    {
        object rawValue = _oracleReader.GetValue(ordinal);

        if (_expectedColumnTypes != null)
        {
            if (!(rawValue is DBNull) && rawValue.GetType()
                != _expectedColumnTypes[ordinal].ClrEquivalentType)
            {
                rawValue = ChangeType(rawValue, _expectedColumnTypes[ordinal].ClrEquivalentType);
            }
        }
        return rawValue;
    }

    private object ChangeType(object sourceValue, Type targetType)
    {
        if (sourceValue is byte[] && targetType == typeof(Guid))
        {
            return new Guid((byte[])sourceValue);
        }
   
        if (sourceValue is DateTime && targetType == typeof(DateTimeOffset))
        {
            return new DateTimeOffset((DateTime)sourceValue);
        }

        return Convert.ChangeType(sourceValue, targetType, CultureInfo.InvariantCulture);
    }

    // ...
}

• Canonical Functions. Most Canonical Functions required by Entity Framework have been implemented using Oracle's built-in functions. In some cases it required some query rewriting, such as when handling Right() canonical function which has no direct Oracle translation and SUBSTR() must be used instead.

private static ISqlFragment HandleCanonicalFunctionRight(SqlGenerator sqlgen, DbFunctionExpression e)
{
    SqlBuilder result = new SqlBuilder();
    result.Append("(CASE WHEN LENGTH(");
    result.Append(e.Arguments[0].Accept(sqlgen));
    result.Append(") >= (");
    result.Append(e.Arguments[1].Accept(sqlgen));
    result.Append(") THEN ");
    result.Append("SUBSTR (");
    result.Append(e.Arguments[0].Accept(sqlgen));
    result.Append(",-(");
    result.Append(e.Arguments[1].Accept(sqlgen));
    result.Append("),");
    result.Append(e.Arguments[1].Accept(sqlgen));
    result.Append(")");
    result.Append(" ELSE ");
    result.Append(e.Arguments[0].Accept(sqlgen));
    result.Append(" END)");
    return result;
}

Let me know if you have any questions about the EFOracleProvider release and EF provider interface in general.

EF Mapping Advisor

Ju-Yi Kuo from our team has started blogging about advanced mapping scenarios supported by Entity Framework.

Part 1 of his EF Mapping Advisor series is online at http://blogs.msdn.com/juyik/archive/2008/05/21/ef-mapping-advisor-1-the-language-of-mapping-part-a.aspx and it explains TPT (table-per-type) mapping strategy.

Entity Framework Sample Provider for Visual Studio 2008 SP1 Beta is Online

We have just published an updated Sample ADO.NET Provider for Entity Framework. 

Changes in the provider APIs since Beta3 release:

• removed DbProviderServices.GetProviderManifest(DbConnection)
• added DbProviderServices.GetProviderManifestToken(DbConnection)
• completely redesigned CSDL/SSDL/MSL for SchemaInformation
• changed the format of the provider manifest XML:
   - DefaultValue vs Default
   - removed facets from functions
• added DbProviderManifest.Token/DbProviderManifest.Provider properties

Provider writers can get the source code from MSDN Code Gallery: http://code.msdn.microsoft.com/EFSampleProvider

Transparent Lazy Loading for Entity Framework – part 3 – Anatomy of a Stub

This post is a part of the series that describes EFLazyLoading library.

• Part 1 - Strategies for implementing lazy loading
• Part 2 - Implementation of EFLazyLoading
• Part 3 - Anatomy of a Stub

In two previous articles I have introduced EFLazyLoading – a framework for lazy loading of entities on top of Entity Framework. In this post I will explain what stubs are and how they work.

Let’s establish some terminology first:

• Shell object is a public object that the users interact with. It has the properties of an entity, but no backing fields except for the primary key.
• Data object is an internal data structure that has backing fields for the object. It implements ILazyEntityDataObject interface.
• Stub object is a shell object that has no data object attached to it.
• Fully loaded object is a shell that has a data object attached and populated.

Here is a typical pair of shell and data objects – NorthwindEF.Category. Note a few things:

• _CategoryID is the only field in the shell class (disregard the base class for a while). All other fields are declared in the data class
• The only public properties on the shell class are properties that correspond to EntityType definition.
• Data class is a nested type inside the shell class.
• Data class is a backing store for all non-key properties.
• Data objects must be able to deep-clone themselves. This is the purpose of ILazyEntityDataObject.Copy() method.
• Data properties (see previous article) are also declared in the Data class. This is because of pure convenience as it enables fields to be private.
• There are no public methods – only protected CreateDataObject() which takes care of creating a private data object.

Shell objects implement ILazyEntityObject interface in addition to three IPOCO interfaces: IEntityWithKey, IEntityWithChangeTracking and IEntityWithRelationships. In current implementation those interfaces are implemented in the base class called LazyEntityObject.

Data object in current implementation it is implemented as a class with fields, but in theory it could be implemented as a hash table (to allow for types with huge number of nullable columns that are often nulls) or in some other way.

How stubs are born

Stubs can come to life in four possible ways:

1. Relationship navigation – when navigating a many-to-one relationship, a stub object is created to represent the related end (if the entity is not already being tracked by the ObjectStateManager).
2. IQueryable<T>.AsStubs(). It is possible to construct a sequence of stubs (IEnumerable<T>) by calling AsStubs() on IQueryable<T>. This will convert a query to a query that only projects primary keys (thus saving on database connection bandwidth).
3. IQueryable<T>.GetStub() that returns a single result. This is a stub equivalent of calling First().
4. It is also possible to populate LazyEntityCollection with stub objects (instead of fully loaded objects) by calling LoadStubs() method.

There is also a way for unmodified fully loaded object to become stub again. All you have to do is to discard their data object by calling LazyObjectContext.Reset() or by calling LazyObjectContext.ResetAllUnchangedObjects() which does the same thing for all unmodified objects in the context. This can help reduce memory footprint of your unit of work, when you are dealing with large objects and you are done processing them. Instead of detaching an object from the context, you simply discard its data – object identity is preserved and it can still be found in all relationships it belongs to, but most of objects memory can be reclaimed by GC.

Examples

// instantiate a fully loaded entity
var prod = entities.Products.First();

// stub gets created because of relationship navigation - no load from the database here
var cat = prod.Category; 

// category object gets fully loaded on first property access
Console.WriteLine("name: {0}", cat.CategoryName);

// once it is loaded we can access all properties - no database access here
Console.WriteLine("desc: {0}", cat.Description);

// iterate through details
// note that collection is populated with LoadStubs which only brings keys
// into memory
foreach (OrderDetail det in prod.OrderDetails.LoadStubs())
{
    // order can be Order or InternationalOrder so it will be eagerly loaded
    // because we don't know the concrete type (see below)
    // next time (even in a different ObjectContext) we'll use cached type information
    // so there's no server roundtrip
    var order = det.Order;

    Console.WriteLine("{0} {1}", det.Product.ProductName, order.OrderDate);
}

// execute a query and return collection of stub objects
var stubs = entities.Suppliers.Where(c => c.Products.Any(d=>d.Category.CategoryID == cat.CategoryID)).AsStubs();

// iterate over stubs - as we go through the collection, individual suppliers are loaded on-demand
// note how LoadStubs() is used to count Products without fully loading them
foreach (var p in stubs)
{
    Console.WriteLine("Shipper {0} - {1} - {2} products", p.CompanyName, p.Phone, p.Products.LoadStubs().Count);
}

// execute a query that returns a single stub object
var singleStub = entities.Suppliers.GetStub(c=>c.SupplierID == 4);
Console.WriteLine("Stub: {0}", singleStub.Phone);

Problem with polymorphic types

Despite our intention, we sometimes get fully loaded objects instead of stubs when calling one of the above methods – that is because of polymorphic types. For example, when your schema has a Customer base type and InternationalCustomer type derived from and there is an association from Order to Customer, you can get either a customer or international customer when you navigate the association:

We cannot possibly know the concrete type up front by examining its EntityKey. Unfortunately to create a stub/shell object we need to know the CLR type. When doing eager load, Entity Framework materializer takes care of determining concrete type by sending a specially crafted SQL query down to the server. The query includes a special discriminator column down which is used to resolve back to concrete type. Unfortunately in this case we don’t want to send any query. Even if we wanted to do that, neither Entity SQL nor LINQ have a way to project object type without loading full object, so store cannot really help us here.

Enter IObjectTypeCache

IObjectTypeCache is one proposed solution to this problem. It exploits the fact, that (using normal methods) objects never change their type – there is no way to change the class of an entity stored in a database table, because Entity Framework does not allow inheritance discriminator columns (in TPH mapping) to be written to and there is no way to achieve the same thing in case of TPT or TPC mappings.

IObjectTypeCache s a cache whose keys are EntityKey objects and values are CLR types (in fact they are factory methods that return objects). This gives us amortized low cost of determining the type given a CLR type.

Every time we create a stub (of type T), we check whether the EntityType has subclasses (defined in CSDL). If the type is known to not to be polymorphic, we just create a new instance of T.

If the type can have subclasses, we check whether the mapping from EntityKey to type is found in cache – if it is there – we just call the factory method and the stub is ready.

If the mapping is not found in the cache (which typically happens the first time a particular EntityKey is materialized in an application), we don’t try to create stubs at all – we fall back to running the fully materialized query which resolves the type for us. After this is done, we add newly discovered key-to-type mappings to our cache, so that the mappings are known next time.

There is a singleton object that holds a reference to IObjectTypeCache that all LazyObjectContexts will use – it is currently held in a static property of LazyObjectContext called ObjectTypeCache.

WARNING: The default implementation of IObjectTypeCache (as of EFLazyLoading v0.5) does not do any cache eviction. This is typically not a problem for databases that have about one million of polymorphic objects (cache can grow up to consume 30-50MB of RAM which is usually not a problem nowadays). If your application has to scale to support more polymorphic objects than that, the sample has to be modified to add some automatic eviction (based on LRU, LFU or other strategy).