Digital Ruminations

EF v2 Goodness + RIA Services

2009-11-19T18:44:17Z

If you’ve spent any time on the RIA Services forums or attempting to use our LinqToEntitiesDomainService base class yourself, you’ll know that it wasn’t very hard to run into issues :) For example, it was very easy to run into server side issues when processing more complicated graph updates. As I have explained many times in the forums, all of these issues were caused by the fact that EF v1 really didn’t support N-Tier so in our domain layer, we were doing our best to mask that via some non-trivial workarounds. Our team has been in close contact with the EF team over the last year discussing these limitations, and the great news is that in EF v2, they’ve vastly improved the N-Tier support (see the MSDN topic N-Tier Applications With Entity Framework for details) in .NET 4.0. With this new support, we’ve been able to completely overhaul our EF provider. These changes are transparent to you, their main impact being that you should no longer run into any of the previous issues that have plagued the provider. Our breaking changes document describes two changes that were required for us to make these improvements, but I’ll summarize those here:

Only EF models that include FK info are supported – This is the new 4.0 default when creating an EF model, so we’re just lining up our story with that. What this allowed us to do was remove all the dynamic TypeDescriptor stuff we were required to do before to manufacture FK members dynamically from the EF model. This was a bug factory.

Only EF models that embed their mapping info as a resource are supported – This is important for partial trust scenarios where we shouldn’t be attempting to load EF mapping files from disk. Again this is the EF default.

As I mentioned, the changes to our provider were mainly behind the scenes, save for one change to wizard codegen. The Insert method we generate now uses the new ObjectStateManager.ChangeObjectState API:

   1: public void InsertProduct(Product product)

   2: {

   3:     if ((product.EntityState != EntityState.Added))

   4:     {

   5:         if ((product.EntityState != EntityState.Detached))

   6:         {

   7:             this.ObjectContext.ObjectStateManager

   8:                 .ChangeObjectState(product, EntityState.Added);

   9:         }

  10:         else

  11:         {

  12:             this.ObjectContext.AddToProducts(product);

  13:         }

  14:     }

  15: }

In summary, what all this means for me, is a big sigh of relief, since I was the author and maintainer of our EF provider, and had the unfortunate task of answering for all the issues people encountered. It also means I don’t have to spend time in the forums explaining esoteric EF issues and providing workarounds :) What this means for you is a vastly improved EF experience in RIA Services :)

Composition Support in RIA Services

2009-11-18T08:27:00Z

I’m having a great time at PDC 2009 talking to customers about Silverlight and RIA Services. The second keynote just finished, Scott unveiled SL4 which means now our team can talk about our PDC Beta of RIA Services for VS 2008 / SL3 as well as our preview for VS 2010 / SL4! You can download the new bits from our WCF RIA Services landing site.

Between sessions, I've found time to make the first of a series of posts on the new features and work we’ve done in this release. In this post I’ll discuss the support we’ve added to RIA Services for compositional relationships. Lets jump right in :) A compositional relationship has the following characteristics:

the lifetime of the child of the association is governed by the parent
the parent and its children are treated as a single unit
children have no identity/existence independent of their parent

The classic PurchaseOrder/OrderDetails association has these characteristics and is often modeled as a composition. A detail cannot exist independent of its parent and generally when operating on an order it and all its details are viewed together and operated upon / validated together. In contrast, the Employee / Reports association is not a compositional – reporting employees exist independent of their manager, meaning if the manager is fired, the reports are hopefully re-parented :)

To facilitate application models that include compositional relationships, we’ve introduced CompositionAttribute which can be used to mark an association as a composition. Of course you can have compositional hierarchies of arbitrary depth. Compositions in RIA Services gain the following behaviors:

Hierarchical change tracking – when a child entity is modified, it’s parent also transitions to the Modified state.
When a parent is in the Modified state, all of its children are included in the change-set sent to the server, including any Unmodified children.
Operation ordering – Often only CRUD operations for the parent or root type in a compositional hierarchy will be exposed by a DomainService. This allows you to write all your business logic for a hierarchy in a single method. However writing explicit methods for child Types is supported, in which case parent operations are always called before child operations. For example, if a new OrderDetail was added to an existing PurchaseOrder, the Update method for PurchaseOrder would be called before the Insert for the OrderDetail.
Public EntitySets for child Types are not generated on the code-genned DomainContext. Children are only accessible via their parent relationship.

Simply marking an association with the attribute enables all this functionality automatically for the association:

   1: public class PurchaseOrderMetadata

   2: {

   3:     [Include]

   4:     [Composition]

   5:     public EntitySet<OrderDetail> OrderDetails;

   6: }

In addition to this new attribute there are a few new server side APIs to help with compositions. These new APIs are shown below in a hierarchical update method that takes a PurchaseOrder / OrderDetail compositional hierarchy and controls the update for the entire hierarchy:

   1: public void UpdateOrder(PurchaseOrder order)

   2: {

   3:     PurchaseOrder origOrder = this.ChangeSet.GetOriginal(order);

   4:     if (origOrder != null)

   5:     {

   6:         this.DataContext.PurchaseOrders.Attach(order, origOrder);

   7:     }

   8:     else

   9:     {

  10:         this.DataContext.PurchaseOrders.Attach(order);

  11:     }

12:

  13:     foreach (OrderDetail detail in

  14:       this.ChangeSet.GetAssociatedChanges(order, p => p.OrderDetails))

  15:     {

  16:         ChangeOperation op = this.ChangeSet.GetChangeOperation(detail);

  17:         switch (op)

  18:         {

  19:             case ChangeOperation.Insert:

  20:                 this.DataContext.OrderDetails.InsertOnSubmit(detail);

  21:                 break;

  22:             case ChangeOperation.Update:

  23:                 this.DataContext.OrderDetails.Attach(detail,

  24:             this.ChangeSet.GetOriginal(detail));

  25:                 break;

  26:             case ChangeOperation.Delete:

  27:                 this.DataContext.OrderDetails.Attach(detail);

  28:                 this.DataContext.OrderDetails.DeleteOnSubmit(detail);

  29:                 break;

  30:             default:

  31:                 break;

  32:         }

  33:     }

  34: }

The new APIs shown above are:

ChangeSet.GetAssociatedChanges – given a parent object and an association, this method returns all child objects in the change-set for that association. You could simply enumerate PurchaseOrder.OrderDetails directly, but of course that wouldn’t include deleted Details. The helper method returns those as well.
ChangeSet.GetChangeOperation – for a specified object, returns a ChangeOperation enum value indicating whether the object operation is Insert, Update, Delete, or None (unmodified).

This post has been a quick introduction to the feature. Attached is a zipped project containing a fully functional composition sample (VS 2010). The solution includes a detailed readme which walks you through the details of the sample and demonstrates the feature in more depth. Now I’m going to go down to the Silverlight booth to chat with more users :)

.NET RIA Services July CTP

2009-07-10T20:51:00Z

After much work, our team just released the July CTP of .NET RIA Services. We're releasing this CTP in step with the big Silverlight 3 release, as trumpeted on ScottGu's SL3 blog post here. As mentioned in Scott's post, "The new .NET RIA Services framework (which now has a go-live license) can be used to easily build multi-tier data applications that span the client and server." If you're interested in building rich multi-tier Silverlight applications you should check out the framework we're building. Some relevant links:

Install the .NET RIA Services July CTP - From this link you can download the overview doc which describes the framework, as well as install the latest bits.
.NET RIA Services project forum - an active forum where our current CTP users discuss issues
.NET RIA Services samples - a place where our team uploads sample applications, and other types of samples for the framework

My intention going forward is to use this blog to explain framework features, discuss design points for new and existing features, as well as to share sample code.

Devs Writing Haiku?

2007-08-30T23:11:00Z

A while back Jomo Fisher on our team started a rather interesting tradition. It all began one day after he had sent out a code-review to the team but after some time had heard no response. Being the impatient guy that he is, he decided to prod his colleagues with a little poetry, resending the request with the following haiku added:

Spring’s warmth melts snow while
nearby code awaits review.
Sweet check-in beckons.

This beautiful haiku had the desired effect - his code was reviewed promptly! With this success, a tradition was born of threatening with haiku if code review requests were ignored for too long. Other devs on the team began crafting their own haiku, hoping to expedite their reviews as well. Here's a few I've sent out:

Code review ignored
Changeset languishing unseen
Evening approaches

Unreviewed change-set
Like an unharvested fruit
Dying on the vine

Devs writing haiku
Idly waiting to check in
Orcas slowly slips

Here's one from Matt Warren after a completed code review:

Code reviewed okay
Some shenanigans undone
Quality improves

From haiku, things have started snowballing into other verse forms. Here's a great one by Keith Farmer in response to an email set out by Luca Bolognese after the C# PM team hit their ZBB (zero bug bounce) milestone:

To Xen PU did LucaBol
An email send, full of glee:
Tests finally ran,
Past bugs measureless to man,
Down to a ZBB.

I'll have to hunt around my mail folders for the rest, I know there are some I've missed. Recently Jomo suggested that haiku are going out of style and that he preferred limmericks. We'll have to see where that goes ...

Linq To Sql POCO Support

2007-08-28T21:22:00Z

A colleague and I were recently discussing POCO support in Linq To Sql (LTS), and he pointed me to a great blog post by Ian Cooper on Domain v.s. Data Centric design methodologies and how Linq To Sql can be used for domain centric development. As Ian points out, LTS places very little requirements on domain classes, allowing them to remain "persistence ignorant". For example, we don't require any particular base class or interface, no particular constructors are required (except the default), no special types are required for association members, etc. I don't believe the last point is widely known which is why I decided to make this post. The EntitySet<T>/EntityRef<T> classes that we codegen by default for associations are not required. In POCO scenarios you can map your association members to either List<T> or T[] like so (see the attached project for complete sources):

    public class Customer
    {
        private List<Order> _Orders;

        public List<Order> Orders {
            get {
                return this._Orders;
            }
            set {
                this._Orders = value;
            }
        }
    }

Assuming this member is mapped as an association in your external mapping file, LTS will treat it as such. Likewise, instead of EntityRef<T> for singleton associations, you will just type your member as T. Of course now that you're not using our association types, one of the things you lose is support for deferred loading. Normally during materialization Linq To Sql assigns deferred sources to your EntitySets/EntityRefs which upon enumeration result in the deferred source query being compiled and executed. In the case of POCO associations you'll have to explicitly specify the span to eager load - by default no associations will be loaded:

     Northwind nw = new Northwind("Northwind.sdf");

     // Load Customer, specifying span
     DataLoadOptions ds = new DataLoadOptions();
     ds.LoadWith<Customer>(p => p.Orders);
     ds.LoadWith<Order>(p => p.OrderDetails);
     ds.AssociateWith<Customer>(p => p.Orders.Where(o => o.Freight > 20));
     nw.LoadOptions = ds;
     Customer cust = nw.Customers.Where(p => p.City == "London").First();

With the DataLoadOptions set for preloading, the materializer compiles the appropriate member query, executing it with the correct FK arguments for each row materialized. In this example, the returned customer has all of it's Orders with a Freight > 20 preloaded, and all OrderDetails for each Order are also preloaded.

There are some other Linq To Sql features that are useful from a domain driven design perspective. For example, DataContext exposes CreateDatabase functionality allowing you to generate a database from your domain model (expressed as a DataContext of POCO entities), as opposed to the normal method of using SqlMetal or the LTS Designer to generate full featured entities from your database (Data Centric). When coupled with our support for SqlCE, this is a very convenient feature:

     Northwind nw = new Northwind("NewNorthwind.sdf");
     if (!nw.DatabaseExists()) {
          nw.CreateDatabase();
     }
     Customer cust = new Customer { CustomerID = "MATHC", CompanyName = "Microsoft" };
     nw.Customers.Add(cust);
     nw.SubmitChanges();

One other final note on POCO - as you'll notice, entities code genned by SqlMetal implement INotifyPropertyChanging. As stated above this is not required, but it does allow our change tracker to be more efficient in it's copying of original values. For classes implementing this interface the change tracker subscribes to the PropertyChanging event, allowing us to put off copying original values until the object has actually changed. Otherwise we have no choice but to copy the values immediately on materialization/attach.

In summary, Linq To Sql offers very good support for POCO classes, providing framework services to your objects in a non-invasive, loosely coupled manner.

C# LINQ to SQL Chat

2006-12-14T00:23:00Z

We just had another community chat yesterday on Linq to Sql (aka DLinq) - you can access a copy of the transcript here: http://blogs.msdn.com/charlie/archive/2006/12/13/c-linq-to-sql-chat-dec-12-2006.aspx.

We had a good turnout and got a lot of great questions and feedback - it's great to see that people are excited about this exciting technology :) On the Linq to Sql team we just finished up a several month coding milestone - we've implemented some new cool new features and also polished up the existing code, fixed bugs, etc. The fruits of this labor will be available in the February 2007 Orcas CTP.

DLinq User-Defined Function support

2006-06-13T08:19:00Z

[Updated on 8/30/2007: Code samples below updated for Beta2. The main thing we did was simplify the code required by providing framework methods ExecuteMethodCall/CreateMethodCallQuery that create the required expression trees (where you had to create them before). You'll also notice that there is a single mapping attribute for both sprocs and functions, with IsComposible being used to distinguish between the two.]

In the last post, we looked at DLinq Stored Procedure support. Another cool new DLinq feature in the latest LINQ preview is support for user-defined functions. We’ve added support for mapping methods to user-defined functions (both scalar and table-valued) allowing the UDFs to be used in queries naturally.

You can run SqlMetal from a VS command prompt against the Northwind database specifying the /functions option to generate mapped methods for user-defined functions:

C:\Program Files\Microsoft Visual Studio 9.0\VC>sqlmetal /server:localhost

/database:"Northwind" /namespace:nwind /code:northwind.cs /language:csharp /functions

If you take a look at the generated code you’ll see that methods have been added to the Northwind data context that map to the user-defined functions in the database. An excerpt is shown below:

public partial class Northwind : DataContext {
    . . .
    [Function(Name = "dbo.TotalProductUnitPriceByCategory", IsComposable = true)]
    [return: Parameter(DbType = "Money")]
    public System.Nullable<decimal> TotalProductUnitPriceByCategory(
                   [Parameter(DbType = "Int")] System.Nullable<int> categoryID) {
        return ((System.Nullable<decimal>)
               (this.ExecuteMethodCall(this, 
               ((MethodInfo)(MethodInfo.GetCurrentMethod())), 
               categoryID).ReturnValue));
    }

    [Function(Name="dbo.ProductsUnderThisUnitPrice", IsComposable=true)]
    public IQueryable<ProductsUnderThisUnitPriceResult> ProductsUnderThisUnitPrice(
                   [Parameter(DbType="Money")] System.Nullable<decimal> price) {
        return this.CreateMethodCallQuery<ProductsUnderThisUnitPriceResult>(this, 
               ((MethodInfo)(MethodInfo.GetCurrentMethod())), price);
    }
    . . .
}

In the method bodies for these UDF mapped methods, you see that we call the ExecuteMethodCall context method to create the required expression tree, execute the query and return the result. When the method is used inline in a query this code is not executed – the query translator knows how to translate the method based on metadata (attribute or external). However if the method called outside of a query then the method body is executed to create the method call expression, turn it into a query and execute it.

The first UDF ‘TotalProductUnitPriceByCategory’ takes a category ID and returns the total of all unit prices for products in that category. Here is an example using this function in a query:

       var q = from c in db.Categories
               select new {
                 c.CategoryName,
                 Total = db.TotalProductUnitPriceByCategory(c.CategoryID)
               };

The function is used inline in a query just like you use a UDF inline in a SQL query. Below is the SQL that is generated – the UDF call is translated as you would expect:

SELECT [t0].[CategoryID],

[dbo].[TotalProductUnitPriceByCategory]([t0].[CategoryID]) AS [value]

FROM [Categories] AS [t0]

If the same UDF is executed outside of a query the mapped method builds the expression and creates and executes a query:

      decimal total = db.TotalProductUnitPriceByCategory(8);

SELECT [dbo].[TotalProductUnitPriceByCategory](@p0) AS [value]

The ‘ProductsUnderThisUnitPrice’ method is mapped to a table valued UDF, and returns an IQueryable. As such, it can be used in a query wherever a table or sub-query can be. Here is an example joining a table to the results of a TVF (and the resulting SQL):

      var q = from c in db.Categories
              from p in db.ProductsUnderThisUnitPrice(8.50M)
              where c.CategoryID == p.CategoryID
              select new {
                    c.CategoryID,
                    c.CategoryName,
                    p.ProductName,
                    p.UnitPrice
              };

SELECT [t0].[CategoryID], [t0].[CategoryName],

[t1].[ProductName], [t1].[UnitPrice]

FROM [Categories] AS [t0]

CROSS JOIN [dbo].[ProductsUnderThisUnitPrice](@p0) AS [t1]

WHERE [t0].[CategoryID] = [t1].[CategoryID]

As a final example let’s look at how to map system functions. While DLinq automatically maps many .NET methods (like string/DateTime methods) to corresponding system functions, there may be some system functions that have no .NET counterpart that you’d like to use. In such cases, you can map a method yourself. For example, if you’re just dying to be able to use the SqlServer Soundex function directly in your DLinq queries, you can map it as follows:

     [Function(Name = "Soundex", IsComposable = true)]
     public string SoundEx([Parameter(Name = "Expression")] string expression) {
         return (string)this.ExecuteMethodCall(this, 
                ((MethodInfo)(MethodInfo.GetCurrentMethod())), 
                 expression).ReturnValue;
     }

This function can now be used as follows:

     var q = from e in db.Employees
             orderby db.Soundex(e.FirstName)
             select new {
                 e.FirstName, 
                 Soundex=db.Soundex(e.FirstName)
             };
     string s = db.Soundex("Mathew Charles");

If you use UDFs in your database and are playing around with the new CTP, give them a try in your DLinq queries and please let us know what you think!

Spring Linq CTP finally hits the shelves!

2006-05-11T02:14:00Z

[Updated on 8/30/2007: Code samples below updated for Beta2. The main thing we did was simplify the code required by providing framework methods ExecuteMethodCall that creates the required expression trees (where you had to create them before). You'll also notice that there is a single mapping attribute for both sprocs and functions, with IsComposible being used to distinguish between the two.]

The latest and greatest LINQ preview has finally been released! (you can download here: http://msdn.microsoft.com/netframework/future/) In addition to fixing a lot of bugs since the last CTP and making additions and modifications based on community feedback, we’ve also added some cool new features. I’d like to use the next few posts to explore a sampling of some of these new features (you can read about more of them in the C# readme here). The topic for today: Stored-Procedure Support.

We’ve added support for mapping methods to stored procedures allowing you to invoke them in a strongly typed manner as methods on the DataContext. To see this in action, you can run SqlMetal against the Northwind sample database specifying the /sprocs option to generate mapped methods for stored-procedures:

C:\Program Files\Microsoft Visual Studio 9.0\VC>sqlmetal /server:localhost
/database:"Northwind" /namespace:nwind /code:northwind.cs /language:csharp /sprocs

If you take a look at the generated code, you’ll see that methods have been added to the Northwind data context that map to the stored procedures in the database. An excerpt is shown below. The [Parameter] attributes aren't strictly necessary if your method parameters have the same names as the sproc params in the database, and explicit type mapping is not needed, but I've included them for completeness.

    public partial class Northwind : DataContext {
        . . .
        [Function(Name="dbo.CustOrderTotal")]
        [return: Parameter(DbType="Int")]
        public int CustOrderTotal(
            [Parameter(Name="CustomerID", DbType="NChar(5)")] string customerID, 
            [Parameter(Name="TotalSales", DbType="Money")] ref Nullable<decimal> totalSales)
        {
            IExecuteResult result = this.ExecuteMethodCall(this, 
                ((MethodInfo)(MethodInfo.GetCurrentMethod())), customerID, totalSales);
            totalSales = ((System.Nullable<decimal>)(result.GetParameterValue(1)));
            return ((int)(result.ReturnValue));
        }

        [Function(Name="dbo.Sales by Year")]
        public ISingleResult<SalesByYearResult> SalesByYear(
            [Parameter(Name="Beginning_Date", DbType="DateTime")] 
                Nullable<DateTime> beginning_Date, 
            [Parameter(Name="Ending_Date", DbType="DateTime")] 
                Nullable<DateTime> ending_Date)
        {
            IExecuteResult result = this.ExecuteMethodCall(this, 
                ((MethodInfo)(MethodInfo.GetCurrentMethod())), beginning_Date, ending_Date);
            return ((ISingleResult<SalesByYearResult>)(result.ReturnValue));
        }

        [Function(Name="dbo.Get Customer And Orders")]
        [ResultType(typeof(GetCustomerAndOrdersResult1))]
        [ResultType(typeof(GetCustomerAndOrdersResult2))]
        public IMultipleResults GetCustomerAndOrders(
            [Parameter(Name="CustomerID", DbType="NChar(5)")] string customerID)
        {
            IExecuteResult result = this.ExecuteMethodCall(this, 
                ((MethodInfo)(MethodInfo.GetCurrentMethod())), customerID);
            return ((IMultipleResults)(result.ReturnValue));
        }
        . . .
    }

The ‘CustOrderTotal’ method is mapped to a simple scalar sproc that takes an input parameter and returns an output parameter. Here’s how you would call it (all samples below assume a Northwind context instance ‘db’):

    decimal? totalSales = 0;
    db.CustOrderTotal("ALFKI", ref totalSales);

The ‘SalesByYear’ method demonstrates how a rowset returning sproc is mapped. For sprocs returning rowsets, a class is automatically generated for the rowset. Here is how you would call this sproc and access its result set:

    DateTime begin = new DateTime(1996, 11, 5);
    DateTime end = new DateTime(1997, 3, 15);
    ISingleResult<SalesByYearResult> sales = db.SalesByYear(begin, end);
    var bigSales = sales.Where(s => s.Subtotal > 5000.00m);

Important points to note here:

The return type ISingleResult<SalesByYearResult> implements IEnumerable allowing the returned rows to be enumerated, and also allows the sproc return value to be accessed if needed (via the ReturnValue property).
In cases where a sproc returns both a result set AND an out parameter, all results are read and buffered automatically (triggered by the first out parameter access) prior to the mapped method returning, since reading all the results off the wire is the only way to get the out parameter values.

Finally the ‘GetCustomerAndOrders’ method shows how sprocs that return multiple result-sets are mapped. From the database metadata, we can determine what possible rowset shapes can be returned from the sproc, but not in what order. Thus it is up to you to request the results in the correct order, as shown below:

    IMultipleResults results = db.GetCustomerAndOrders("ALFKI");
    var cust = results.GetResult<GetCustomerAndOrdersResult1>().First();
    var orders = results.GetResult<GetCustomerAndOrdersResult2>().ToArray();

If you are hand mapping sproc methods you'll likely want to use names more appropriate than Result1, Result2, etc. Using the DLinq designer, you can map your sprocs manually and rename things appropriately. You might also want to map the results to existing types (i.e. use Customers and Orders in the last example).

Note there are some limitations to the types of sprocs we can automatically generate code for. For example, the following categories of sprocs cannot be auto generated:

Sprocs using temporary tables – Since using FMTONLY true when executing a sproc (the mechanism we use to determine result shapes) doesn’t execute any statements, any CREATE TABLE statements used in a sproc to create a temporary table will not be executed. This means that if the sproc returns a result set formed by querying such a temporary table, any attempt to get the metadata will fail, since the table doesn’t exist
Sprocs using dynamic SQL to return result sets – If a sproc contains conditional logic that is used to build a dynamic SQL statement, then execute it to return a result set, there is no way to get metadata for the result set, since the query used to generate it can be variable, and is only known at runtime.

If you have sprocs that fall into these categories, SqlMetal will skip generation for them and log an error message. Of course you have the option of hand mapping them yourself.

As you can see such method mapping makes stored procedures convenient to call, by encapsulating the invocation as a simple method call, and providing data binding. The SampleQueries project (LINQ Preview\Samples\C#\SampleQueries) includes some additional sproc samples. Give them a try, and please let us know what you think! Next post we'll take a look at DLinq support for User-Defined Functions (UDFs), and see how you can use them inline in your queries.

Genesis...

2006-02-13T22:11:00Z

As a first post, I'll introduce myself, and state briefly my intention for this blog going forward. I've been a software engineer at Microsoft for four years. I spent the last 3+ years working on a project named MBF (Microsoft Business Framework), but sadly that project was recently disbanded. I'm now on the CSharp team working on the exciting new Linq/DLinq project. In the future, expect this blog to contain fascinating, action packed posts related to Linq/DLinq specifically, as well as interesting posts on the wonderful art of Software Engineering in general.