Entity Framework Design : Linq to Entities

Model Defined Functions

efdesign — Wed, 07 Jan 2009 23:32:00 GMT

Today the Entity Framework, and more specifically the Entity Data Model, have a limited notion of Functions.

We are currently restricted to Function Imports that allow stored procedures to be invoked, and Canonical / Store Functions for database independent and database specific functions respectively.

Now however we want to support functions defined, not just declared, in the EDM (aka. the CSDL).

An example would be:

<Function Name="GetAge" ReturnType="Edm.Int32">
      <Parameter Name="Person" Type="Model.Person" />
      <DefiningExpression>
            Edm.DiffYears(Edm.CurrentDateTime(), Person.Birthday)
      </DefiningExpression>
</Function>

Here are some things to notice:

The DefiningExpression is eSQL.
The function can have zero or more parameters.
The Function must have a return type.
Function Parameters are referenced directly by Name in the DefiningExpression: meaning there is no parameter denoting prefix like @. This means you must be careful to choose parameter names that don't collide with identifiers you need to use in the rest of eSQL expression.
Unlike functions in SSDL, functions in CSDL only support In bound Parameters (i.e Mode="In") because otherwise they become non-composable.
For this reason the Mode of a parameter cannot be set in CSDL (it is always Mode="In").
Functions are declared as Global Items and are declared within the <Schema> element. As such there identity is made up of the Schema's namespace and the function name.
The function parameters and return type can be any of the following:
- A scalar type or collection of scalar types.
- An entity type or collection of entity types.
- A complex type or collection of complex types.
- A row type or collection of row types (See below).
- A ref type or collection of ref types.
Functions with a DefiningExpression do not require mapping, since the eSQL expression is composed out of eSQL fragments that are already mapped.
Functions without a DefiningExpression are simply declarations. Today the Entity Framework doesn't complain when loading a CSDL with such a function, but you can't invoke it. In the future these functions might be used to support mapping Table Value Functions in the store to functions in the Conceptual Model.
Since it is trivial in eSQL to create arbitrary un-named types, imagine a projection that projects 3 of the properties from an Entity, we now need a mechanism for defining these "RowTypes" inline, so that they can be used when defining Function Parameters and ReturnTypes. For example:

<Parameter Name="Coordinate">
   <RowType>
      <Property Name="X" Type="int" Nullable="false"/>
      <Property Name="Y" Type="int" Nullable="false"/>
      <Property Name="Z" Type="int" Nullable="false"/>
   </RowType>
</Parameter>
Since eSQL is primarily set based, we also need a way of defining parameters and return types that are collections of RowTypes:

<Parameter Name="Coordinates">
   <CollectionType>
      <RowType>
         <Property Name="X" Type="int" Nullable="false"/>
         <Property Name="Y" Type="int" Nullable="false"/>
         <Property Name="Z" Type="int" Nullable="false"/>
      </RowType>
   </CollectionType>
</Parameter>

Using the Function via eSQL:

It is trivial to use the function via eSQL. For example:

SELECT Namespace.GetAge(p)
FROM Container.People AS P
WHERE P.Firstname = ‘Jim’

Here we get Jim's age, assuming of course there is only one Jim!

It is also possible to compose functions together, you must simply ensure return types and target parameter types are the same (in the case of named types) or structurally equivalent (in the case of row types or collections of row types).

Things get a little trickier when you are dealing with functions that return sets, for example imagine a function that returns someone's friends, used in conjunction with the GetAge function:

SELECT VALUE (F)
FROM Container.People AS P
CROSS APPLY Namespace.GetFriends(P) AS F
WHERE Namespace.GetAge(P) > 21

Here we get all the friends of people older than 21.

As you can see to do this sort of thing you need a crash course in eSQL.

Using the Function via LINQ:

It is also possible to use these functions in LINQ, but this does require a extra step to create an appropriate stub function in the CLR.

This solution is based on the techniques described here, and involves creating a Stub function in the CLR language of your choice, and annotating it something like this:

[EdmFunction("Namespace", "GetAge")]
public static int GetAge(Person p)
{
throw new NotSupportedException(…);
}

The Entity Framework uses the signature of the function and the EdmFunction attribute to map calls to this function when encountered to the appropriate Model Defined Function.

Once you have this stub it is then trivial to use it in a LINQ query like this:

var peopleOver21 =
     from p in ctx.People
     where GetAge(p) < 21
     select p;

Indeed if you are familiar with LINQ you will probably find composing functions together a lot easier too:

var friendOfPeopleOver21 =
     from p in ctx.People
     from f in GetFriends(p)
     where GetAge(p) < 21
     select f;

Notice that the CLR functions don't need to be directly callable, in the example above the CLR stub throws an exception if called directly.

However the existence of the stub allows you to create LINQ expressions that compile correctly, and then at runtime, when used in a LINQ to Entities query, the function call is simply translated by the entity framework into a query that runs in the database.

Summary

As you can see Model Defined Function's are very powerful, and this post has barely scratched the surface of possibilities they open up.

The Entity Framework team would love to hear your comments.

Alex James
Program Manager,
Entity Framework Team

This post is part of the transparent design exercise in the Entity Framework Team. To understand how it works and how your feedback will be used please look at this post.

EDM and Store functions exposed in LINQ

efdesign — Wed, 08 Oct 2008 11:37:00 GMT

In this post Colin Meek and Diego Vega delve into some enhancements we are planning for LINQ to Entities, anyway over to them...

Entity Framework v1 customers preferring to write their queries using LINQ often hit a limitation on the range of functions and query patterns supported in LINQ to Entities. For some of those customers, having to resort to Entity SQL, or even to Entity SQL builder methods, feels awkward and reduces the appeal of Entity Framework.

There are two things we want to do in order to address this in future versions:

Expand the range of patterns and standard BCL methods we recognize in LINQ expressions.
Provide an extensibility mechanism that people can use to map arbitrary CLR methods to appropriate server and EDM functions.

This blog post expands on the second approach:

It is actually possible for us to improve our LINQ implementation so that all functions defined in the EDM and in the store, and even user defined functions, can be mapped to CLR methods with homologous signatures.

Design

Problem space

There are multiple dimensions to the problem space we want to address:

Functions can be defined in either the conceptual or the storage space
Functions can be defined in either the manifest, or just declared in the model
Functions can be mapped to either static CLR methods or to instance methods on the ObjectContext
This feature specifically targets composable functions

How it looks like: EdmFunctionAttribute

The basis of the extensibility mechanism is a new method-level attribute that carry function mapping information. Here is the basic signature of the attribute’s constructor:

public EdmFunctionAttribute(string namespaceName, string functionName)

The namespaceName parameter indicates the namespace for the function in metadata (i.e. “EDM” or “SQLSERVER”, or other store provider namespace). The functionName parameter takes the name of the function itself.

The following example could be product code or customer code applying the attribute on an extension method (it could be a regular static function) in order to map it to the standard deviation SQL Server function:

public static class SqlFunctions       
{       
    [EdmFunction("SqlServer", "stdev")]       
    public static double? StandardDeviation(this IEnumerable<int?> source)       
    {     
        throw EntityUtil.NotSupported(       
            System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);       
    }       
} 

Notice that while this method can’t be called directly it can be used in a query like this:

var query =  
    from p in context.Products       
    where !p.Discontinued       
    group p by p.Category into g       
    select g.Select(each => each.ReorderLevel).StandardDeviation(); 

The following example shows how the canonical DiffYear function is mapped:

public static class EntityFunctions        
{       
    [EdmFunction("EDM", "DiffYears")]       
    public static Int32? DiffYears(DateTime? arg1, DateTime? arg2)       
    {       
        throw EntityUtil.NotSupported(System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);  
    }       
} 

Usage is:

var query =     
    from p in context.Products     
    where EntityFunctions.DiffYears(DateTime.Today, p.CreationDate) < 5     
    select p; 

The following example shows how a user defined function defined in SQL Server can be mapped:

public static class MyCustomFunctions      
{     
    [EdmFunction("SqlServer", "MyFunction")]     
    public static Int32? MyFunction(string myArg)     
    {     
        throw new NotSupportedException("Direct calls not supported");  
    }     
} 

Convention based function name

We can establish that by convention the name of the CLR function defines the value of the functionName parameter. That makes the functionName parameter in the EdmFunctionAttribute optional.

EdmFunctionNamespaceAttribute

To avoid having to always specify the namespaceName for each function, we define a new class-level attribute named EdmFunctionNamespaceAttribute that would define the namespace mapping globally for a given class:

public EdmFunctionNamespaceAttribute(string namespaceName)

Using EdmFunctionNamespaceAttribute and the convention based constructor:

[EdmFunctionNamespace("EDM")]     
public static class EdmMethods      
{     
    [EdmFunction]     
    public static Int32? DiffYears(DateTime? arg1, DateTime? arg2)     
    {     
        throw EntityUtil.NotSupported(     
            System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);  
    }     
}

How it works

When a method with the EdmFunction attribute is detected within a LINQ query expression, its treatment is identical to that of a function within an Entity-SQL query. Overload resolution is performed with respect to the EDM types (not CLR types) of the function arguments. Ambiguous overloads, missing functions or lack of overloads result in an exception. In addition, the return type of the method must be validated. If the CLR return type does not have an implicit cast to the appropriate EDM type, the translation will fail.

Instance methods on the ObjectContext will be supported as well. This allows the method to bootstrap itself and trigger direct evaluation, as in the following example (definition of the method and sample query):

public static class MyObjectContext : ObjectContext     
{     
    // Method definition     
    [EdmFunction("edm", "floor")]     
    public double? Floor(double? value)     
    {  
        return this.QueryProvider.Execute<double?>(Expression.Call(     
            Expression.Constant(this),     
            (MethodInfo)MethodInfo.GetCurrentMethod(),     
            Expression.Constant(value, typeof(double?))));     
    }     
}     
    
…     
    
// evaluated in the store!     
context.Floor(0.1);     

Without the ObjectContext, the function cannot reach the store! To support this style of bootstrapping, the context needs to expose the LINQ query provider. For this reason, we now expose a “QueryProvider” property on the ObjectContext. This provider includes the necessary surface to construct or execute a query given a LINQ expression.

public class ObjectContext     
{     
    public IQueryProvider QueryProvider { get; }     
} 

If such a method is encountered inline in another query, then we must validate that the instance argument (MethodCallExpression.Object) is the correct context, but the instance is otherwise ignored:

// positive     
var q1 = from p in context.Products select context.Floor(p.Price);     
    
// negative     
var q2 = from p in context.Products select context2.Floor(p.Price); 

A function proxy can sometimes bootstrap itself without an explicit context, e.g. when an input argument is itself an IQueryable:

public static class SqlFunctions     
{     
    [EdmFunction("SqlServer", "stdev")]     
    public static double? StandardDeviation(this IQueryable<int?> source)     
    {     
        return source.Provider.Execute<double?>(Expression.Call(     
            (MethodInfo)MethodInfo.GetCurrentMethod(),     
            Expression.Constant(source)));     
    }     
} 

Nullability considerations

Particularly for functions taking collections, we will need to provide overloads for nullable and non-nullable elements. We don’t want to require awkward constructions like:

var query = (from p in products select (int?)p.ReorderLevel).StandardDeviation(); 

Tool for Generating the Functions

We created a simple internal tool that generates the classes that represent all the EDM canonical function and the SQL Server store functions. The tool will take the function definitions from Metadata and generate the appropriate function stubs/implementations.

The tool will be outside the product and will be run on demand. We expect to make a version of this tool available for provider writers together with the provider samples.

Naming

The methods will be in the following classes:

Namespace	Class name
System.Data.Objects	EntityFunctions
System.Data.Objects.SqlClient	SqlFunctions

Note: The equivalent class in LINQ to SQL is System.Data.Linq.SqlClient.SqlMethods.

The method names will correspond to the name of the EDM/SQL function they represent. The argument names will correspond to the argument names of the EDM/SQL functions as retrieved by the metadata.

The recommendation for provider writers will be to include a similar static class in a namespace of the following form:

System.Data.Objects.[Standard provider namespace].[Standard provider prefix]Functions

Overloads and Implementation

Non-aggregate Functions

For each non-aggregate function we create an overload with all inputs type as nullable of the CLR equivalent of their EDM primitive type, and the return type nullable of the CLR equivalent of their EDM primitive type.

The implementation of the functions (what gets executed if the function is invoked outside an expression tree) will be to throw a NotSupportedException.

Example:

public static class EntityFunctions      
{     
    [EdmFunction("EDM", "DiffYears")]     
    public static Int32? DiffYears(DateTime? arg1, DateTime? arg2)     
    {     
        throw EntityUtil.NotSupported(System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);  
    }     
}      
      
public static class SqlFunctions      
{     
    [EdmFunction("SqlServer", "DiffYears")]     
    public static Int32? DiffYears(DateTime? arg1, DateTime? arg2)     
    {     
        throw EntityUtil.NotSupported(System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);  
    }     
} 

Aggregate Functions

For each aggregate function we will provide two overloads, one with IEnumerable<Nullable<T>> and another one with IEnumerable<T>, where T is the CLR equivalent of the EDM primitive type of the input. The implementations of these will check whether the input is IQueryable in which case it will implement the self-bootstrapping.

Example:

[EdmFunction("EDM", "VARP")]     
public static double? VarP(IEnumerable<int> arg1)     
{     
    ObjectQuery<int> objectQuerySource = source as ObjectQuery<int>;     
    if (objectQuerySource != null)     
    {     
        return ((IQueryable)objectQuerySource).Provider.Execute<double?>(Expression.Call(     
            (MethodInfo)MethodInfo.GetCurrentMethod(),     
        Expression.Constant(source)));     
    }     
    throw EntityUtil.NotSupported(System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);  
}     
    
[EdmFunction("EDM", "VARP")]     
public static double? VarP(IEnumerable<int?> arg1)     
{     
    ObjectQuery<int?> objectQuerySource = source as ObjectQuery<int?>;     
    if (objectQuerySource != null)     
    {     
        return ((IQueryable)objectQuerySource).Provider.Execute<double?>(Expression.Call(     
            (MethodInfo)MethodInfo.GetCurrentMethod(),     
        Expression.Constant(source)));     
    }     
    throw EntityUtil.NotSupported(System.Data.Entity.Strings.ELinq_EdmFunctionDirectCall);  
}     

The Entity Framework team would love to hear your comments.

Alex James
Program Manager,
Entity Framework Team

This post is part of the transparent design exercise in the Entity Framework Team. To understand how it works and how your feedback will be used please look at this post.