meek : expressions

InvocationExpression and LINQ to Entities

cmeek — Sun, 07 Dec 2008 17:02:00 GMT

I talked a little bit about patterns using InvocationExpression in a previous post (you might want to review the post for some context). It turns out my advice wasn’t that easy to implement… Fortunately, the EFExtensions helper library now includes an ExpandInvocations method that implements the same functionality (and a little bit more). I’ll return to the same example I used in my previous post, “all cars that are red or cheap”:

Expression<Func<Car, bool>> theCarIsRed = c1 => c1.Color == "Red";

Expression<Func<Car, bool>> theCarIsCheap = c2 => c2.Price < 10.0;

ParameterExpression p = theCarIsRed.Parameters.Single();

Expression<Func<Car, bool>> theCarIsRedOrCheap = Expression.Lambda<Func<Car, bool>>(

Expression.Or(theCarIsRed.Body, Expression.Invoke(theCarIsCheap, p)), p);

var query = carQuery.Where(theCarIsRedOrCheap);

Unfortunately, the Entity Framework does not support InvocationExpressions, so the query fails with a NotSupportedException. Our new helper method does the work of rewriting the query to avoid the unsupported invocations:

// Rewrite query to remove InvocationExpression

query = query.ExpandInvocations();

A lot easier! If you’re interested in the implementation, read on…

Remember math class? Let’s define a couple of functions:

f(x) = x + 1, g(y) = f(y + 2) * f(y + 3)

To evaluate g(4), we just expand the values inline:

g(4) = f(4 + 2) * f(4 + 3) = ((4 + 2) + 1) * ((4 + 3) + 1)

We just need to teach EFExtensions how to do the same trick. Whenever an invocation is encountered (e.g. “f(y + 2)” in our example), we want to replace the parameter in the body of the function with the corresponding argument. If we’re expanding an expression tree, we just need to remember which argument values correspond to which parameters:

private sealed class InvocationExpander : ExpressionVisitor {

private readonly ParameterExpression _parameter;

private readonly Expression _expansion;

private readonly InvocationExpander _previous;

…

protected override Expression VisitInvocation(InvocationExpression iv) {

if (iv.Expression.NodeType == ExpressionType.Lambda) {

LambdaExpression lambda = (LambdaExpression)iv.Expression;

return lambda

.Parameters

// zip together parameters and the corresponding argument values

.Zip(iv.Arguments, (p, e) => new { Parameter = p, Expansion = e })

// add to the stack of available parameters bindings (this class doubles as an immutable stack)

.Aggregate(this, (previous, pair) => new InvocationExpander(pair.Parameter, pair.Expansion, previous))

// visit the body of the lambda using an expander including the new parameter bindings

.Visit(lambda.Body);

}

return base.VisitInvocation(iv);

}

…

A couple of details of the VisitInvocation method might need explanation. First of all, I’m using the (very handy) Zip method, which allows me to consider the elements of two sequences pair-wise, like the teeth of a zipper coming together. (Good news: Zip is in .NET 4.0!) In this case, the lambda expression’s Parameters are aligned pair-wise with the invocation expression’s Arguments. Second of all, I’m using the call stack to keep track of which parameters are currently in scope. If you squint, InvocationExpander looks a lot like an immutable stack implementation… The _previous field is the ‘tail’ and the _parameter and _expansion fields are the ‘head’. The use of the Aggregate method gives me implicit the Push behavior I’m looking for, adding parameter bindings like layers in an onion. When control returns to the caller of VisitInvocation, the parameters are Popped out of scope, or peeled away, as a side effect of ceding control to the _previous visitor. At every position in the call stack, the InvocationVisitor instance that is in scope also captures the relevant parameter stack. When we run into a parameter expression that potentially requires expansion, we just iterate over the stack until we find a match:

protected override Expression VisitParameter(ParameterExpression p) {

InvocationExpander expander = this;

while (null != expander) {

if (expander._parameter == p) {

return base.Visit(expander._expansion);

}

expander = expander._previous;

}

return base.VisitParameter(p);

}

OK… There might be easier ways to write this solution, but I like immutable types in general. Why? I’m not smart enough to figure out state changes. I hate reviewing code and having to ask questions like “have I initialized field x at this point?”, “what happens if someone modifies collection y?” or “do I need to use a lock here?” I’m sure you can find plenty of other good reasons to like them.

Using LINQ Expressions to Generate Dynamic Methods II

cmeek — Wed, 14 May 2008 00:28:00 GMT

A beta of Visual Studio 2008 SP1 was released on Monday and the ADO.NET Entity Framework (EF) is now in the box! You can download and install the Beta here. The EF Extensions library has been updated to work with the beta and includes several public and internal changes. Source code is available at http://code.msdn.com/EFExtensions. The latest release introduces some performance improvements in the materializer (you can read about the library here). These improvements illustrate another powerful expression pattern.

To improve code clarity, the EF Extensions API encourages you to write:

var products = command.Materialize<Product>(r => new Product {

ProductID = r.Field<int>("ProductID"),

Name = r.Field<string>("Name"),

…

}).ToList();

instead of:

List<Product> products = new List<Product>();

using (SqlDataReader reader = command.ExecuteReader()) {

int idOrdinal = reader.GetOrdinal("ProductID");

int nameOrdinal = reader.GetOrdinal("Name");

…

while (reader.Read()) {

Product product = new Product {

ProductID = (int)reader.GetValue(idOrdinal),

Name = reader.IsDBNull(nameOrdinal) ? (string)null : (string)reader.GetValue(nameOrdinal),

…

};

products.Add(product);

}

There’s usually a tradeoff. This is no exception... While the code in the first example is easier to write, read and maintain, looking up column ordinals on each call to Field<T> is expensive: for every row in every column, I’m incurring the cost of the lookup. Field<T> also verifies arguments on every call and checks for DBNull whether or not the requested type accepts nulls. Most of this work is redundant or unnecessary for materialization.

Fortunately, there’s a simple solution to these problems. If we represent the “shaper” delegate as a LINQ expression, we can rewrite it (using the technique described here) for efficiency. Basically, we can rewrite calls to Field<T> to calls to the underlying reader, caching column ordinals for efficiency. In the above example, the expression:

r => new Product() {ProductID = r.Field("ProductID"), Name = r.Field("Name")}

now becomes

r => new Product() {ProductID = Convert(r.GetValue(0)), Name = Convert(IIF(r.IsDBNull(1), null, r.GetValue(1)))}

The rewritten version is identical to the more performant version we wrote by hand.

The EFExtensions library uses an extensible pattern to perform these optimizations. Methods that can be optimized or rewritten are flagged with an attribute indicating a handler, in this example FieldMethodOptimizer:

[MaterializerOptimizedMethod(typeof(FieldMethodOptimizer))]

public static T Field<T>(this IDataRecord record, string name);

When materialization begins, field names from the reader are immediately retrieved. Whenever a method with this attribute is encountered in the shaper expression, the corresponding optimizer is called to rewrite the expression:

protected override Expression VisitMethodCall(MethodCallExpression m) {

Expression result = base.VisitMethodCall(m);

if (result.NodeType == ExpressionType.Call) {

m = (MethodCallExpression)result;

MaterializerOptimizedMethodAttribute attribute = m.Method.GetCustomAttributes(typeof(MaterializerOptimizedMethodAttribute), false)

.Cast<MaterializerOptimizedMethodAttribute>()

.SingleOrDefault(); // multiple attributes not permitted; not inherited

if (null != attribute) {

return attribute.Optimizer.OptimizeMethodCall(this.fieldNames, this.recordParameter, m);

}

return result;

}

As in my previous post, I’m leveraging the ExpressionVisitor to do the rewrite. In this case, I’m intercepting and replacing MethodCallExpressions only.

End result: we can now use a more concise coding pattern without sacrificing performance. Unfortunately, we still need to pay the cost of compiling the materializer delegate, but this can be offset by reusing the delegate. To facilitate reuse, the Materializer class in EFExtensions is thread-safe and stores the optimized delegate on first use.

Using LINQ Expressions to Generate Dynamic Methods

cmeek — Sat, 26 Apr 2008 03:31:00 GMT

This week at DevConnections in Orlando, I gave a “deep-dive” talk on LINQ. I wanted to give people a feel for what's possible with the new language features and core APIs in .NET 3.5. I spent most of the talk discussing a single example: take an ADO.NET 2.0 code sample and simplify. Instead of using an existing library like LINQ to SQL or LINQ to Entities, I built a (limited) LINQ provider from scratch. As boilerplate code is moved into helper methods and a proto-LINQ-to-SQL API evolves, the sample is boiled down to something much more compact:

static List<Customer> GetCustomers()

{

using (Table<Customer> table = new Table<Customer>(GetConnectionString(), "Customers"))

{

IEnumerable<Customer> query = from customer in table

where customer.City == "London"

select customer;

return query.ToList();

}

In this post, I’ll drill down on one component developed in the talk, which was also included in the EFExtensions library. This component takes rows from a data reader and transforms them (or “shapes” them) into typed results. For instance, I may want to transform records into customers. Using a couple of extension methods (Field<T> and Materialize<T>), we can leverage a “shaper” delegate to do just this:

SqlCommand command = …;

return command.Materialize<Category>(r =>

new Category

{

CategoryID = r.Field<int>("CategoryID"),

CategoryName = r.Field<string>("CategoryName"),

…

});

The shaper delegate code (highlighted above) is still annoying though: for every property of the customer, I’m retrieving a column of the same name. The code is mechanical and repetitive, the sort of thing you want a machine to do rather than a programmer.

Here are three strategies you can use to automate this pattern in .NET 2.0:

1. Reflection: Write a general purpose delegate that uses reflection to construct an instance of T and then dynamically invokes property setters. While this code is relatively easy to write, the performance is sub-optimal. For information on the performance of various method dispatch patterns, take a look at this great talk by Joel Pobar and Joe Duffy.

2. Automatically generate the code: You can automatically generate wrapper classes encapsulating shaping logic. Code generation has its challenges however, in particular integration with build systems and the development environment. It’s also a lot of code to maintain.

3. Create a DynamicMethod implementing the pattern: .NET allows you to compile a delegate at runtime using dynamic methods. This resolves the performance and maintenance problems of solutions 1 and 2. MSIL generation is hard to get right unfortunately and the code does not reflect the intent of the generated delegate.

Digression: random thoughts on APIs

I recently heard the expression “Swiss Army Knife API”. These are interfaces that handle all kinds of little (possibly unrelated) problems. They can be useful but they are also hard to package and discover. The Zip method in EFExtensions illustrates these problems. It’s a handy method, but what is it doing in an EF library? It has nothing to do with the EF or with the scenarios addressed by the library (it’s poorly packaged), and no one trying to pair the elements of two iterators would think to look in that particular library (it’s not discoverable). If you need to install a PCI card, fillet a fish and hand-stitch a saddle, you might need a Swiss Army API.

At the other extreme, there are narrowly targeted APIs that can solve complex problems but most often require specialized knowledge or training. To make matters worse, once you’ve mastered them, you can rarely apply your knowledge to different domains. You can probably think of a few examples of this pattern.

LINQ achieves a useful balance. While the LINQ project was motivated by a specific requirement – seamless support for non-object data within .NET applications – all components of the solution are generically useful. Consider…

The System.Linq.Expressions API serves a specific need for integrated queries: it allows the compiler to describe the user’s code as a data structure that can then be translated to targets other than MSIL at runtime, like SQL, Web Services, etc. Expressions can also be compiled into delegates at runtime, which brings me to a .NET 3.5 solution to the default shaper problem… If the compiler can use expressions to describe code, so can we!

Here’s the code pattern we want to generate:

r => new T

{

Property₁ = r.Field<Type[Property₁]>("Property₁"),

Property₂ = r.Field<Type[Property₂]>("Property₂"),

…

}

Shortcut: learning how to build an expression programmatically

If you want to figure out how to build expressions programmatically, a simple trick will probably save you some time. Just follow the compiler’s lead. First, write an example of the pattern, e.g.:

Expression<Func<IDataRecord, Customer>> example =

r => new Customer { City = r.Field<string>("City"), };

After compiling your program, use a tool like .NET Reflector to figure out how the compiler builds the expression tree. In its default mode, this tool is actually a little bit too smart, disassembling the code into precisely what we wrote to begin with. If you change the disassembler settings (View à Options à Disassembler) to use “.NET 2.0” optimizations, it will give more useful output, e.g.:

ParameterExpression CS$0$0000;

Expression<Func<IDataRecord, Customer>> example =

Expression.Lambda<Func<IDataRecord, Customer>>(

Expression.MemberInit(

Expression.New((ConstructorInfo) methodof(Customer..ctor), new Expression[0]),

new MemberBinding[] {

Expression.Bind(

(MethodInfo) methodof(Customer.set_City),

Expression.Call(null, (MethodInfo) methodof(Utility.Field), new Expression[] { CS$0$0000 = Expression.Parameter(typeof(IDataRecord), "r"), Expression.Constant("City", typeof(string)) })) }),

new ParameterExpression[] { CS$0$0000 });

Once you’ve figured out the expression pattern, you can implement a general solution, e.g.:

private static Expression<Func<IDataRecord, T>> CreateDefaultShaper<T>()

{

// Compiles a delegate of the form (IDataRecord r) => new T { Prop1 = r.Field<Prop1Type>("Prop1"), ... }

ParameterExpression r = Expression.Parameter(typeof(IDataRecord), "r");

// Create property bindings for all writable properties

List<MemberBinding> bindings = new List<MemberBinding>();

foreach (PropertyInfo property in GetWritableProperties<T>())

{

// Create expression representing r.Field<property.PropertyType>(property.Name)

MethodCallExpression propertyValue = Expression.Call(

typeof(Utility).GetMethod("Field").MakeGenericMethod(property.PropertyType),

r, Expression.Constant(property.Name));

// Assign the property value to property through a member binding

MemberBinding binding = Expression.Bind(property, propertyValue);

bindings.Add(binding);

}

// Create the initializer, which instantiates an instance of T and sets property values

// using the member bindings we just created

Expression initializer = Expression.MemberInit(Expression.New(typeof(T)), bindings);

// Create the lambda expression, which represents the complete delegate (r => initializer)

Expression<Func<IDataRecord, T>> lambda = Expression.Lambda<Func<IDataRecord, T>>(

initializer, r);

return lambda;

}

Expression<T> has a Compile method which produces an instance of T. In this case, T is our delegate type, Func<IDataRecord, T>, so we’re done! We build up a description of the logic using LINQ expressions, and then compile the logic to produce a delegate. This code is an order of magnitude easier to write using the expressions API than using custom IL generation (if you don’t believe me, give it a try!)

I can’t resist offering another implementation of the CreateDefaultShaper method that uses query expressions. This example nicely closes the loop, using LINQ to create LINQ expressions:

private static Expression<Func<IDataRecord, T>> CreateDefaultShaper<T>()

{

ParameterExpression r = Expression.Parameter(typeof(IDataRecord), "r");

return

Expression.Lambda<Func<IDataRecord, T>>(

Expression.MemberInit(

Expression.New(typeof(T)),

from property in GetWritableProperties<T>()

let fieldMethod = typeof(Utility).GetMethod("Field")

.MakeGenericMethod(property.PropertyType)

let propertyValue = Expression.Call(

fieldMethod, r, Expression.Constant(property.Name))

select (MemberBinding)Expression.Bind(property, propertyValue)),

r);

}

Problem:

Before signing off, I’d like to leave you with a simple problem: the CreateDefaultShaper method currently assumes that record instances have columns exactly corresponding to every property. Try adding support for column renames (e.g. the “customer_id” column maps to the “CustomerID” property). It’s probably easiest to implement this pattern with custom attributes on the materialized type properties, but it’s probably best to decouple the declaration of the mapping from its usage. This allows the mapping to be retrieved from an arbitrary location (e.g. either from CLR custom type attributes or from a separate configuration file).