The Visual Basic Team : IQueryable

Spot the Bug! – The Key to using Anonymous Types (Jonathan Aneja)

VBTeam — Tue, 17 Nov 2009 07:54:00 GMT

This one’s going to be long, but for those of you who’ve felt the first 3 in this series were too easy I promise this one’s tougher J.

Let’s say you want to list all the customers from a table in a ComboBox, and update the UI based on which one is selected. To do this we’ll need to bring back two fields from the database – the customer’s name and the customer’s ID. When a customer is selected we want the ComboBox’s SelectedValue property to equal the customer’s ID.

Here’s some quick code that gets us up and running (using Northwind and LINQ to SQL):

Dim db As New NorthwindDataContext

Dim query = From row In db.Customers _

Select row.CompanyName, row.CustomerID

ComboBox1.DataSource = query.ToList()

ComboBox1.DisplayMember = "CompanyName"

ComboBox1.ValueMember = "CustomerID"

But now let’s say your boss looks at the app and says “I want you to add an ‘All Customers’ option as the first item in the list.” How would you do that? You’d need to insert an anonymous type into an existing sequence of anonymous types, which is tricky given that you can never actually use the type’s name.

Thankfully there’s a trick that uses generic parameter inference that allows us to do this (it relies on the fact that the compiler will share (or “unify”) the definition of multiple anonymous types when they have the same number of members, in the same order, of the same type, with the same names, and the same mutability):

Dim db As New NorthwindDataContext

Dim query = From row In db.Customers _

Select row.CompanyName, row.CustomerID

Dim allOption = New With {.CompanyName = "All Customers", _

.CustomerID = "-1"}

ComboBox1.DataSource = AddOptionForAll(query, allOption).ToList()

ComboBox1.DisplayMember = "CompanyName"

ComboBox1.ValueMember = "CustomerID"

...

Function AddOptionForAll(Of T)(ByVal sequence As IEnumerable(Of T), _

ByVal allOption As T) As IEnumerable(Of T)

'wrap individual element in an array and then union the two sequences

Return (New T() {allOption}).Union(sequence)

End Function

As long as we have an anonymous type that’s compatible with the anonymous type that the query generated, the compiler will determine that sequence and allOption are actually the same type and this should work fine.

Ok so let’s get to the bug – well in this case spotting it is pretty simple (it doesn’t compile J), but fixing it is tricky (though I’ve already given two pretty big hints). Here’s the text of the compiler error (it’s on the line that calls AddOptionForAll):

Data type(s) of the type parameter(s) in method 'Public Function AddOptionForAll(Of T)(sequence As System.Collections.Generic.IEnumerable(Of T), allOption As T) As System.Collections.Generic.IEnumerable(Of T)' cannot be inferred from these arguments because they do not convert to the same type. Specifying the data type(s) explicitly might correct this error.

What’s wrong and how do we fix it?

Answer: We actually told the compiler to generate two different anonymous type definitions, and thus it can’t unify them because the types really are different.

From earlier: “…the compiler will share (or “unify”) the definition of multiple anonymous types when they have the same number of members, in the same order, with the same names, and the same mutability”

The anonymous type that the query generates will have ReadOnly properties, whereas the anonymous type that we generated (allOption) will have Read/Write properties. The fix is to allOption immutable so that its structure will match the result of the query:

Dim allOption = New With {Key .CompanyName = "All Customers", _

Key .CustomerID = "-1"}

The “Key” modifier tells the compiler to make those properties ReadOnly and to override Equals and GetHashCode such that they only consider “Key” properties when deciding if two instances of the same anonymous type are equal. (the other hint was in the title J). For the query above, the compiler automatically inserts the “Key” modifier - i.e. what we wrote is exactly equivalent to this:

Dim query = From row In db.Customers _

Select New With {Key row.CompanyName, Key row.CustomerID}

So with a simple fix to the allOption line we’re now using the same anonymous type definition and everything works fine.

VB’s anonymous type syntax is very flexible and provides three different options: immutable, fully mutable, or partially mutable (i.e. some fields are ReadOnly while others are not). Even for a simple scenario like adding an option to a list, the better you understand how things work under the covers the easier it’ll be to debug problems later J.

Dynamic Searching using LINQ - Dataset + Joins

VBTeam — Sat, 11 Jul 2009 03:22:00 GMT

About two years ago I posted some code that shows how to dynamically construct LINQ queries at runtime. On average there's a couple questions per month about trying it in some edge case, and usually the support already exists within the API. Recently though someone posted a question that has to do with Datasets and Joins that raised some interesting issues:

I have a dataset with two datatables (One Loaded From A CSV file) and (the other loaded from a query) what I would like to be able to do is set the criteria (i.e dtCSV.Field1=dtQuery.Field7 AND (dtCSV.Field5 != dtQuery.Field4 or dtCSV.Field5 != dtQuery.Field9)

Returning dtQuery.Field1, dtCSV.Field5

Previously the API supported conditions of the form "object.property=value", but here we have something of the form "object1.property = object2.property". It's still a completely valid expression, but it means the API will need to generate a different expression tree.

Let's walk through how you can add this support to the API:

1. At the end of Module DynamicQuery add the following extension method:

'Extension method that can be called off any type that implements IEnumerable(Of T),

'which constructs a Condition with T as the element type and compares one property (or field) with another

<Extension(), EditorBrowsable(EditorBrowsableState.Always)> _

Public Function CreateMutliColumnCondition(Of T)(ByVal dataSource As IEnumerable(Of T), _

ByVal propName As String, _

ByVal condType As condition.Compare, _

ByVal propName2 As String) As Condition(Of T)

Return Condition.CreateMutliColumn(dataSource, propName, condType, propName2)

End Function

2. Add this method to the Condition class:

'Constructs a Condition with T as the element type that compares one field/property with another

Public Shared Function CreateMutliColumn(Of T)(ByVal dataSource As IEnumerable(Of T), _

ByVal propertyName As String, _

ByVal condType As Compare, _

ByVal propertyName2 As String) As Condition(Of T)

Return New Condition(Of T)(propertyName, condType, propertyName2)

End Function

3. Add the following code to the Condition(Of T) class:

'Returns a MemberExpression for a property or field access. Also handles nesting scenarios

Private Function GetPropertyExpression(ByVal propName As String, ByVal paramExpr As ParameterExpression) As MemberExpression

'Split the string to handle nested property access

Dim s = propName.Split("."c)

'Get the PropertyInfo instance for propName

Dim pInfo = GetType(T).GetProperty(s(0))

Dim callExpr = Expression.MakeMemberAccess(paramExpr, pInfo)

'For each member specified, construct the additional MemberAccessExpression

'For example, if the user says "myCustomer.Order.OrderID = 4" we need an

'additional MemberAccessExpression for "Order.OrderID = 4"

For i = 1 To UBound(s)

pInfo = pInfo.PropertyType.GetProperty(s(i))

callExpr = Expression.MakeMemberAccess(callExpr, pInfo)

Return callExpr

End Function

Friend Sub New(ByVal propName As String, ByVal condType As Compare, ByVal propName2 As String)

Dim paramExpr = GetParamInstance(GetType(T))

Dim b As Expression = CombineExpression(GetPropertyExpression(propName, paramExpr), _

condType, _

GetPropertyExpression(propName2, paramExpr))

LambdaExpr = Expression.Lambda(Of Func(Of T, Boolean))(b, New ParameterExpression() {paramExpr})

'Compile the lambda expression into a delegate

del = DirectCast(LambdaExpr.Compile(), Func(Of T, Boolean))

End Sub

4. (optionally) the old constructor in this method should be refactored to use the helper method we introduced in #3.

Friend Sub New(ByVal propName As String, ByVal condType As Compare, ByVal value As Object, ByVal valueType As Type)

Dim paramExpr = GetParamInstance(GetType(T))

Dim callExpr = GetPropertyExpression(propName, paramExpr)

'ConstantExpression representing the value on the left side of the operator

Dim valueExpr = Expression.Constant(value, valueType)

Dim b As Expression = CombineExpression(callExpr, condType, valueExpr)

LambdaExpr = Expression.Lambda(Of Func(Of T, Boolean))(b, New ParameterExpression() {paramExpr})

'Compile the lambda expression into a delegate

del = DirectCast(LambdaExpr.Compile(), Func(Of T, Boolean))

End Sub

Now that the API supports it, we can do joins against Datasets:

'The code in this method is equivalent to the following query, but constructed dynamically

'Dim filteredQuery = From csvRow In dtCSV _

' Join orderRow In dtOrders On csvRow.CustomerID Equals orderRow.CustomerID

Dim dtCSV As New myDataset.dtCSVDataTable

dtCSV.Rows.Add(1, "Redmond")

dtCSV.Rows.Add(2, "Toronto")

dtCSV.Rows.Add(3, "Seattle")

Dim dtOrders As New myDataset.dtOrdersDataTable

dtOrders.Rows.Add(10001, 1)

dtOrders.Rows.Add(10002, 3)

dtOrders.Rows.Add(10003, 4)

dtOrders.Rows.Add(10004, 2)

Dim query = From csvRow In dtCSV _

From orderRow In dtOrders

Dim c = query.CreateMutliColumnCondition("csvRow.CustomerID", Compare.Equal, "orderRow.CustomerID")

Dim filteredQuery = query.Where(c)

DataGridView1.DataSource = (From row In filteredQuery _

Select row.orderRow.OrderID, row.csvRow.CustomerID, row.csvRow.City).ToList()

The key line here is this one:

Dim c = query.CreateMutliColumnCondition("csvRow.CustomerID", Compare.Equal, "orderRow.CustomerID")

Instead of comparing a column to a value we can now compare one column to another (either in the same table or in a different table).

<geeky overload resolution details>

Why did I name the method CreateMultiColumnCondition instead of just adding another overload called CreateCondition? To avoid introducing a breaking change! If you had code that called CreateCondition with the last parameter as a String, the compiler would infer the type parameter at that position to be String. But now with a new overload that also has a parameter of type String in that position, the compiler would now choose the new overload because that method is "less generic" (i.e. a closer match).

I don't like the idea of having to think through advanced overload resolution details to figure out which method will actually get called, so it seemed easier to just rename the method :)

</geeky>

A better (and likely more performant) implementation would actually introduce an overload of Join similar to this one for Where:

'Filters an IEnumerable(Of T) according to the specified condition

<Extension()> _

Public Function Where(Of T)(ByVal source As IEnumerable(Of T), _

ByVal condition As Condition(Of T)) As IEnumerable(Of T)

Return source.Where(condition.del)

End Function

This would involve a bit more work in the API though as the Join method takes 3 delegates rather than 1. (so Condition(Of T) would have to generate 3 smaller trees rather than one big one). It's already almost 6 on a Friday though and I'm lazy, so this is left as an exercise to the reader :)

Full VS2008 .sln is attached.

Jonathan

VSLive San Francisco Content (Amanda Silver)

VBTeam — Tue, 01 Apr 2008 00:19:26 GMT

All - I'm about to go on for my LINQ Deep Dive and Tips &Tricks session at VSLive in San Francisco. I posted all demos & slides here:

https://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=LaunchVB&ReleaseId=805

Hidden Gems in Visual Basic 2008 (Amanda Silver)

VBTeam — Wed, 21 Nov 2007 09:24:16 GMT

Yesterday I promised to post about the hidden gems in Visual Basic and Visual Studio 2008 that you haven’t read about in blog posts or seen at conferences. I forgot that I mentioned a few of my favorite features in the Beta1 announcement where I showed off the improvements in the Intellisense experience – statement completion for keywords, local variables, and expressions. In that post, I also cover a new language feature called Relaxed Delegates which allows you to provide alternate signatures for events. Both of those are pretty great features – but I promised to tell you about some new stuff.

I polled the team and we came up with the following top ten that weren’t mentioned in my previous post. There’s so much I could talk about – but you’re going to have to check back. But first, I have to start with a keynote feature…

0) Multi-targetting

In short, multi-targetting allows you to use Visual Studio 2008 (and VB 9!) to target the .NET 2.0 frameworks. All of the following features I’ll show you can be leveraged even if you’re targeting .NET 2.0! (With the exception of items #5 & #7 as they depend on the LINQ to Objects and LINQ to XML APIs which reside in .NET 3.5) To do this, when you start up Visual Studio and create a new project, just make sure that the target framework is set to .NET Framework 2.0.

Note that projects you’re upgrading, will automatically target .NET Framework 2.0 until you add references to .NET 3.0 or 3.5 assemblies.

1) Type Inference

In Visual Basic 9, in the following bit of code and NOTHING is late-bound – everything is bound at compile-time which means you get Intellisense and type checking.

        Dim dialog = New OpenFileDialog()
        Dim result = dialog.ShowDialog()
        Dim printStr = "C:\"
        If result = Windows.Forms.DialogResult.OK Then
            printStr = dialog.FileName
        End If
        MsgBox(printStr)

This makes prototyping very quick, easy, and type-safe! This feature and the mechanisms that control it are comprehensively covered in this article by a member of our QA team, Bill Horst.

2) If Operator

Ever notice that the IIF function returns something of type Object? This means that you don’t get Intellisense or type-checking by default on that result. For those of you that insist on type-safe, early-bound code, you have to cast the result. The code then looks something like this:

        Dim intC As Integer = CInt(IIf(intA = intB, intA, intB - 1))

With the If operator, you can now write that line as:

        Dim intD As Integer = If(intA = intB, intA, intB)

And with type inference it gets even easier on the eyes:

I'm a fan of any feature that improves readability.

3) Object Initializers

Generally, in the .NET frameworks, we try to design APIs that use the create-set-call pattern as our usability studies have indicated they are the easiest to use by the average programmer. Nevertheless, objects that don’t have a parameterless constructor exist in the .NET framework. When that happens, it’s really nice to be able to leverage Object Initializers which are kind of like a Dim and a With statement combined into an expression. That makes parameterized constructors somewhat easier to tolerate:

        Dim strm As New StreamWriter( _
                      New FileStream("C:\out.txt", FileMode.OpenOrCreate) _
                        With {.Position = 10})

It also makes creating arrays of objects much easier.

        Dim Capitals() As City = {New City With {.Name = "Antanarivo", .Country = "Madagascar"}, _
          New City With {.Name = "Belmopan", .Country = "Belize"}, _
          New City With {.Name = "Monaco", .Country = "Monaco"}, _
          New City With {.Country = "Palau", .Name = "Koror"}}

4) Nullable

Nullable is the feature you’ll notice but rarely have to think about. It’s basically the .NET representation for a Nullable value type (Integer, Date, etc.) Using the designer for LINQ to SQL, the Object-Relational Mapping layer introduced in Visual Studio 2008, nullable columns in your database are mapped to this type. The result is that you can write the following expression in VB and the right thing happens – null valued rows propagate null. In the example below, the Independence property on the Country type is a nullable date, denoted as Date?

        Dim virginIslands As New Country With {.Independence = Nothing}
        Dim palau As New Country With {.Independence = #10/1/1994#}

        Dim vILength = #8/24/2005# - virginIslands.Independence ' Nothing
        Dim pLength = #8/24/2005# - palau.Independence ' 3980.00:00:00

5) LINQ to DataSet

I love this feature because it means never having to say you’re sorry… No, really, it means that you don’t need to migrate to a new data access technology to reap the benefits of LINQ. I fill my DataSet just as before, and then I can query against the DataSet:

        Me.EmployeesTableAdapter.Fill(Me.NORTHWNDDataSet.Employees)
        Dim query = From emp In Me.NORTHWNDDataSet.Employees _
                    Where emp.Country = "USA" _
                    Order By emp.HireDate _
                    Select emp
        Me.EmployeesBindingSource.DataSource = query

6) Syntax Tooltips

How cool is this?

How about this?

What do you think of this?

7) XML Namespace support in Intellisense

We’ve blogged about XML Intellisense before. But there’s a tidbit we’ve overlooked.

When namespaces are used in the XML document you’re reading from, the Intellisense engine matches the namespace prefix and the local name. This little productivity boost can yield a significant savings in keystrokes. Instead of typing the prefix then the colon and then the local name, you just type a little bit of the local name and hit enter. Here is a small example of how it works, starting with an input document and the applicable intellisense:

If we just type the letter “t” the “tomato” entry will be selected:

If we type the letter “e” the “exa:eggplant” will be selected and the completion list will also contain the prefix “exa” and the “elephant” entry:

The next character that will be typed will determine which single entry will get selected. Now, how easy was that?

8) GoTo Type Definition

Often, when you define a variable, you want to view the type definition in code or in the Object Browser. There’s a new option in the context menu which allows you to go directly to the type definition instead of just the variable declaration. This is great when combined with type inference as it allows you confirm that the type of the variable is actually what you believe it to be.

9) Type inference for loop variables

Check out the following code:

And this code:

Without having to specify the type of the control variable, it’s inferred from right-hand-side expression or the collection we’re iterating over.

10) Performance improvements and… Non-blocking operations!

The background compiler is an awesome feature that gives you immediate feedback on the correctness of the code you write. We’ve made some tremendous strides in the performance of the background compiler in this release. We believe that the background compiler is 3 times faster uses 3 times less memory. Anyone who has migrated their project to 2008 should notice an improvement after a couple minutes of use.

While we’ve made huge improvements on the throughput, certain operations like changing the declaration of a base class that’s used many times in a large project is computationally expensive. If you attempted to invoke a feature that relied on compiled information before the background compiler has a chance to do its magic, like Intellisense or the Drop Downs, previous versions of Visual Studio would grind to a halt until the compilation was complete. Those operations are no-longer blocking. So, while you might occasionally get the drop downs to look like this:

The IDE will still be responsive.

Now do you have to download it? Remember, you can always grab the Express Edition!

Visual Basic 2008 Ships!!! (Amanda Silver)

VBTeam — Mon, 19 Nov 2007 20:27:00 GMT

Hooray! Today we’ve finally shipped Visual Studio 2008, previously known as “Orcas”, previously known as some symbol which doesn’t have an ascii representation. We’ve come a long way, but it’s been a great ride. In his blog, Soma mentions many of the flagship features at a high-level that will undoubtedly make the every developer and head of IT salivate, but this blog is about Visual Basic. :)

This is a tremendous release for Visual Basic users everywhere as it’s the first to introduce query expressions and XML as a first class data-type. Not only does this make Visual Basic the most productive choice for programming data-centric applications, it also makes us VB developers the envy of many an alpha-geek. If you'd like to just try it out -- be sure to check out Visual Basic Express which comes with all the features I talk about below.

With query the entire .NET frameworks become a queryable data source. You can now query against anything available in the .NET Frameworks – the registry, file system, live processes, etc. For example, the code below queries against the running processes and returns those with a thread count greater than 10, orders them by the thread count:

        Dim query = From proc In Process.GetProcesses _
            Let ThreadCount = proc.Threads.Count _
            Where ThreadCount > 10 _
            Order By ThreadCount _
            Select proc.ProcessName, proc.Id, ThreadCount

Using the LINQ to SQL Object-Relational Mapping framework, you can query against a SQL database as though it was just another collection of objects exposed by the frameworks. The following query finds all the products that have been discontinued, groups them into categories, and finds the total number of units in stock for each category:

        Dim db As New NorthwindDataContext(My.Settings.NORTHWNDConnectionString)
        Dim query = From prod In db.Products _
                    Where prod.Discontinued = True _
                    Group By prod.CategoryID _
                    Into Count(), NumInStock = Sum(prod.UnitsInStock) _
                    Join cat In db.Categories On cat.CategoryID Equals CategoryID _
                    Select cat.CategoryName, Count, NumInStock

Unlike writing huge SQL query strings, I get immediate feedback on the query I write.

Intellisense:

Syntax checking:

Schema checking:

I can use the same query language (VB!) to query against relational databases, XML, and collections of objects – I don’t need to learn any domain specific language, VB gives me access to all! It’s really easy for me to join across domains (I’ll take that up in a later blog post.)

With XML as a first class data-type, Visual Basic becomes a full XML transform language. But it’s also really easy for me to project to XML from my SQL database:

        Dim query = From prod In db.Products _
            Where prod.Discontinued = True _
            Group By prod.CategoryID _
            Into prodGroup = Group, NumInStock = Sum(prod.UnitsInStock) _
            Join cat In db.Categories On cat.CategoryID Equals CategoryID _
            Select <Products>
                       <Category CategoryID=<%= CategoryID %> CategoryName=<%= cat.CategoryName %>>
                           <%= From prod In prodGroup _
                               Select <Product Price=<%= prod.UnitPrice %>><%= prod.ProductName %></Product> %>
                       </Category>
                   </Products>

Last but not least, for those of you who aren’t so excited about the LINQ (Language INtegrated Query) project, Visual Basic 2008 comes with a tremendous improvement in the Intellisense experience overall. One of our MVPs said that he literally types two characters and hits tab, two characters, tab – it’s as though the code writes itself!

Obviously, we’re very excited about this release and hope it meets you with open arms eager to embrace the productivity! If you're ready to go to the next level and dig in, be sure to check out our learning pages and How-Do-I videos which walk you through these features and so much more.

Tomorrow I’ll blog about the least often mentioned goodies hidden in Visual Basic 2008 so that you can become the guru master of VB9. ‘Til then…

Lambda Expressions and Expression Trees

VBTeam — Wed, 12 Sep 2007 05:10:30 GMT

Hi there!

My name is Tim, and I'm the dev lead for the VB compiler team. Recently Amanda convinced me to blog on the VB team blog rather then my own blog for a variety of reasons, and so here I am. My current plan is that I'll post VB related postings here, and more compiler implementation/technology/fun facts on my own blog (ie, things that don't relate to the VB language/features).

Recently my article on lambda expressions got published on MSDN magazine. For this blog post, I'm going to expand on some of the ideas in that article, so feel free to check out the article if some of the ideas here don't make sense yet.

Lambda expressions are really cool not only because of the reasons stated in the article, but also because it's possible to capture the contents of a lambda expression in a readable form and treat it as data rather than executable code.

For example, the following line of code turns a lambda expression into a callable delegate:

Dim f As Func(Of Integer, Integer) = function(x) x * 2
Dim r = f(10)

In this code sample, the first line creates a lambda expression and does the magic to assign it to a callable delegate f, and the second line invokes f and returns 20.

This is all cool, and I hope my article highlights some of the reasons why I think this is cool. What's even more cool is that you can actually assign a lambda expression into a special variable type and the compiler does some more magic for you:

Dim e As Expression(Of Func(Of Integer, Integer)) = function(x) x * 2

In this code sample, e is not a callable delegate. Rather, it's an expression tree. An expression tree is a data representation of the lambda expression in a form that is easy for you to read and reason about the lambda expression in.

If you look through the value of e in the debugger, you'll see that you can easily determine that the expression was a multiplication operation of variable called x of type integer with a constant integer 2.

By default, query operators that extend IEnumerable take lambda expressions. Query operators that extend IQueryable take expression trees. Therefore, when you issue a query over a type that is queryable, the compiler will generate an expression tree rather than a lambda expression.

Scott, one of the developers on my team, has written an excellent article as well on some of these ideas. He has a great diagram there showing the expression tree representation of a few lambda expressions.

What can you do with expression trees?

In general, you can do 2 things:

Reason about the data in the expression tree, and do custom processing over it. You can also convert the data in the expression tree into another domain (such as XML).
Use the .Compile() method to turn the expression tree into a delegate so that you can execute it.
Build queries with dynamic conditions: see Jonathan's (our PM) excellent post. He has a great example of how you can manipulate the expression trees and build custom conditions in your lambda expressions.

Item 2 is interesting, because it means that you can write a lambda expression into an expression tree, reason about it, etc, and then turn it into a callable delegate.

Dim e As Expression(Of Func(Of Integer, Integer)) = function(x) x * 2
Dim f = e.Compile
Dim r = f(10)

In this example, the value of r is 20, just like the previous example where we assigned the lambda directly to a delegate type.

Why is this interesting?

Without expression trees, its impossible to describe expressions in a descriptive format; the only description of them is IL, which is too low level to re-create the user's intent for the expression. Rather, expression trees provide a high level tree representation of an expression, so that you can easily understand the user's intent for the expression.

Here are some uses of expression trees that may be interesting:

Reason about expressions, then convert them to delegates and call them.
Convert expressions into SQL so that they can be executed by a server (this is what LINQ to SQL does).
Convert expressions into XML and write them to disk.
Convert expressions into a custom format to send over a network protocol to a server that can then rebuild the expression tree from the format, recreate the expression, and reason about it and/or execute it.

Over the next several posts, I'm going to blog specifically about the expression trees that the VB compiler generates, and some differences between C# trees and VB trees in order to maintain VB semantics. I will also provide tips on things you can do to be sure that any library that you write that consumes expression trees will provide the best experience for VB developers.

Tim

Implementing Dynamic Searching Using LINQ

VBTeam — Thu, 30 Aug 2007 01:51:00 GMT

A common requirement in forms-over-data applications is that users be able to search by any combination of fields to construct a dynamic query at run time. For example, the search feature in this application allows the user to find all records that meet criteria defined on multiple columns:

LINQ makes it easy to write powerful queries like this over various data sources; for instance, we can use the following query to find all Orders shipped to a given country within a user-specified timeframe:

Dim query = From order In db.Orders _

Where order.ShipCountry = txtCountry.Text _

And order.ShippedDate >= dtpStartDate.Value _

And order.ShippedDate <= dtpEndDate.Value

This is easy at compile-time, but what if we want to check the date the order was entered instead of the date it was shipped? In this case we’d have to write a separate query using order.OrderDate. Doing this dynamically at runtime isn’t all that difficult if you’re just building up a SQL string, but how would we do this with LINQ? Doesn’t LINQ require me to specify the criteria at compile time in order to construct the query? Fortunately the answer’s no, LINQ supports constructing dynamic queries at runtime through the Expression Tree API and the Expression Compiler.

In Visual Studio 2008, any valid VB expression can be represented as an expression tree. What we need to do is create an Expression Tree to represent the user’s criteria, and then pass it to the LINQ to SQL runtime to do the SQL translation. So for the first part of our Where clause above it’d look something like this:

Dim p = Expression.Parameter(GetType(Order), "")

Dim order = GetType(Order).GetProperty("ShipCountry")

Dim expr = Expression.Equal(Expression.PropertyOrField(p, order.Name), Expression.Constant("Germany"))

Dim predicate = Expression.Lambda(Of Func(Of Order, Boolean))(expr, New ParameterExpression() {p})

>Yikes, that’s a mouthful…and we’ve only done 1/3 of the Where clause so far! I definitely don’t want to write 12 lines just to construct a Where clause at runtime. What I’d like to be able to do is write an extension method called CreateCondition that would allow me to construct the expression tree in one easy line, something like this:

Dim condition1 = db.Orders.CreateCondition("ShipCountry", Compare.Equal, "Germany")

And then for ShippedDate we could make these two conditions:

Dim startDate? = #1/1/1997#

Dim endDate? = #1/31/1997#

Dim condition2 = db.Orders.CreateCondition("ShippedDate", Compare.GreaterThanOrEqual, startDate)

Dim condition3 = db.Orders.CreateCondition("ShippedDate", Compare.LessThanOrEqual, endDate)

(Note: There’s a similar sample in Beta2 called DynamicQueries which does a lot more than what we’re looking at here; think of this as an easier-to-write subset of DynamicQueries).

Notice that we passed in a String the first time, and a Nullable Date literal for the next two; we can do this because the CreateCondition method is generic and infers the type based on the parameter passed in. We now need to combine the conditions into one big condition:

Dim c = Condition.Combine(condition1, Compare.And, condition2, condition3)

Or we could take advantage of Operator Overloading and do it this way (equivalent to the line above):

Dim c = condition1 And condition2 And condition3

Ok so now that we’ve constructed our Condition object, let’s use it to filter the data:

'Filter out all Orders that don't match the Condition

'Note that the query isn't executed yet to due to deferred execution

Dim filteredQuery = db.Orders.Where(c)

'We can now perform other operations (such as Order By) on filteredQuery

Dim query = From row In filteredQuery _

Order By row.OrderDate, row.OrderID _

Select row

'Executes the query and displays the results in DataGridView1

DataGridView1.DataSource = query

So far so good, we’re using LINQ over a dynamically-constructed condition and everything’s strongly-typed. We’re still only hitting the database once since deferred execution ensures the query doesn’t run until we actually enumerate the results (through databinding). The Condition API has made it a lot easier to construct and compile the expression trees, but we’d still have to write a fair bit of code to construct these conditions based on user input. That’s where the ConditionBuilder control comes in:

This allows the user to dynamically specify the criteria at runtime:

Notice that for Date fields we automatically get a DateTimePicker instead of a TextBox, and Booleans would result in a CheckBox.

Ok so we’ve covered what the user experience is at runtime, but how do we actually create the Condition API? There’s a fair bit of documentation in the code below so I won’t go through all of it, but here’s the basics:

1. There are three main classes: Condition, Condition(Of T), and Condition(Of T, S)

a. Condition is an abstract class that is used to construct the generic versions. By structuring it this way we get the benefits of generic type parameter inference – i.e. we don’t have to worry about passing the generic type parameters to the method; the factory method figures it out for us.

b. Condition(Of T) is used to join multiple conditions together. T is the element type (i.e. Order in the example above).

c. Condition(Of T, S) is the simplest type; it represents an “object.propery <comparison> value” expression. The type parameter S will be inferred to be the type of the value passed in (i.e. String, Date, Boolean etc…).

2. For local execution of a query we compile the LambdaExpression to a delegate so that it can be executed in-memory. The user can invoke this delegate by calling the Matches method.

'Compile the lambda expression into a delegate

del = DirectCast(LambdaExpr.Compile(), Func(Of T, Boolean))

3. The extension methods at the bottom are defined on IQueryable(Of T) for remote execution, and IEnumerable(Of T) for local execution.

The complete solution is attached in the file DynamicCondition.zip below, note that you’ll need to update the ConnectionString in app.config to point to your version of Northwind.

Imports System.ComponentModel

Imports System.Linq.Expressions

Imports System.Runtime.CompilerServices

Public Module DynamicQuery

Public MustInherit Class Condition

'Used to ensure we get the same instance of a particular ParameterExpression

'across multiple queries

Private Shared ParamTable As New Dictionary(Of String, ParameterExpression)

'The expression tree which will be passed to the LINQ to SQL runtime

Protected Friend LambdaExpr As LambdaExpression

'Enumerates all the different comparisons which can be performed

Public Enum Compare

[Or] = ExpressionType.Or

[And] = ExpressionType.And

[Xor] = ExpressionType.ExclusiveOr

[Not] = ExpressionType.Not

Equal = ExpressionType.Equal

[Like] = ExpressionType.TypeIs + 1

NotEqual = ExpressionType.NotEqual

[OrElse] = ExpressionType.OrElse

[AndAlso] = ExpressionType.AndAlso

LessThan = ExpressionType.LessThan

GreaterThan = ExpressionType.GreaterThan

LessThanOrEqual = ExpressionType.LessThanOrEqual

GreaterThanOrEqual = ExpressionType.GreaterThanOrEqual

End Enum

'Constructs a Condition with T as the element type and S as the value's type

Public Shared Function Create(Of T, S)(ByVal dataSource As IEnumerable(Of T), _

ByVal propertyName As String, _

ByVal condType As Compare, _

ByVal value As S) As Condition(Of T, S)

Return New Condition(Of T, S)(propertyName, condType, value)

End Function

'Constructs a Condition with T as the element type and valueType as the value's type

'This is useful for situations where you won't know the value's type until runtime.

Public Shared Function Create(Of T)(ByVal dataSource As IEnumerable(Of T), _

ByVal propertyName As String, _

ByVal condType As Compare, _

ByVal value As Object, _

ByVal valueType As Type) As Condition(Of T)

Return New Condition(Of T)(propertyName, condType, value, valueType)

End Function

''' <summary>

''' Creates a Condition which combines two other Conditions

''' </summary>

''' <typeparam name="T">The type the condition will execute against</typeparam>

''' <param name="cond1">The first Condition</param>

''' <param name="condType">The operator to use on the conditions</param>

''' <param name="cond2">The second Condition</param>

''' <returns>A new Condition which combines two Conditions into one according to the specified operator</returns>

''' <remarks></remarks>

Public Shared Function Combine(Of T)(ByVal cond1 As Condition(Of T), _

ByVal condType As Compare, _

ByVal cond2 As Condition(Of T)) As Condition(Of T)

Return Condition(Of T).Combine(cond1, condType, cond2)

End Function

'Combines multiple conditions according to the specified operator

Public Shared Function Combine(Of T)(ByVal cond1 As Condition(Of T), _

ByVal condType As Compare, _

ByVal ParamArray conditions() As Condition(Of T)) As Condition(Of T)

Return Condition(Of T).Combine(cond1, condType, conditions)

End Function

'Combines two Expressions according to the specified operator (condType)

Protected Shared Function CombineExpression(ByVal left As Expression, _

ByVal condType As Compare, _

ByVal right As Expression) As Expression

'Join the Expressions based on the operator

Select Case condType

Case Compare.Or : Return Expression.Or(left, right)

Case Compare.And : Return Expression.And(left, right)

Case Compare.Xor : Return Expression.ExclusiveOr(left, right)

Case Compare.Equal : Return Expression.Equal(left, right)

Case Compare.OrElse : Return Expression.OrElse(left, right)

Case Compare.AndAlso : Return Expression.AndAlso(left, right)

Case Compare.NotEqual : Return Expression.NotEqual(left, right)

Case Compare.LessThan : Return Expression.LessThan(left, right)

Case Compare.GreaterThan : Return Expression.GreaterThan(left, right)

Case Compare.LessThanOrEqual : Return Expression.LessThanOrEqual(left, right)

Case Compare.GreaterThanOrEqual : Return Expression.GreaterThanOrEqual(left, right)

Case Compare.Like

'For the Like operator we encode a call to the LikeString method in the VB runtime

Dim m = GetType(CompilerServices.Operators).GetMethod("LikeString")

Return Expression.Call(m, left, right, Expression.Constant(CompareMethod.Binary))

Case Else

Throw New ArgumentException("Not a valid Condition Type", "condType", Nothing)

End Select

End Function

'Since both type parameters must be the same, we can turn what would normally

'be a Func(Of T, T, Boolean) into a Func(Of T, Boolean)

Protected Shared Function CombineFunc(Of T)(ByVal d1 As Func(Of T, Boolean), _

ByVal condType As Compare, _

ByVal d2 As Func(Of T, Boolean)) As Func(Of T, Boolean)

'Return a delegate which combines delegates d1 and d2

Select Case condType

Case Compare.Or : Return Function(x) d1(x) Or d2(x)

Case Compare.And : Return Function(x) d1(x) And d2(x)

Case Compare.Xor : Return Function(x) d1(x) Xor d2(x)

Case Compare.Equal : Return Function(x) d1(x) = d2(x)

Case Compare.OrElse : Return Function(x) d1(x) OrElse d2(x)

Case Compare.AndAlso : Return Function(x) d1(x) AndAlso d2(x)

Case Compare.NotEqual : Return Function(x) d1(x) <> d2(x)

Case Compare.LessThan : Return Function(x) d1(x) < d2(x)

Case Compare.GreaterThan : Return Function(x) d1(x) > d2(x)

Case Compare.LessThanOrEqual : Return Function(x) d1(x) <= d2(x)

Case Compare.GreaterThanOrEqual : Return Function(x) d1(x) >= d2(x)

Case Else

Throw New ArgumentException("Not a valid Condition Type", "condType")

End Select

End Function

'Guarantees that we get the same instance of a ParameterExpression for a given type t.

Protected Shared Function GetParamInstance(ByVal dataType As Type) As ParameterExpression

'Parameters are matched by reference, not by name, so we cache the instances in a Dictionary.

If Not ParamTable.ContainsKey(dataType.Name) Then

ParamTable.Add(dataType.Name, Expression.Parameter(dataType, dataType.Name))

End If

Return ParamTable.Item(dataType.Name)

End Function

End Class

Public Class Condition(Of T) : Inherits Condition

'Delegate that contains a compiled expression tree which can be run locally

Friend del As Func(Of T, Boolean)

Friend Sub New()

End Sub

Friend Sub New(ByVal propName As String, ByVal condType As Compare, ByVal value As Object, ByVal valueType As Type)

'Split the string to handle nested property access

Dim s = propName.Split("."c)

'Get the PropertyInfo instance for propName

Dim pInfo = GetType(T).GetProperty(s(0))

Dim paramExpr = GetParamInstance(GetType(T))

Dim callExpr = Expression.MakeMemberAccess(paramExpr, pInfo)

'For each member specified, construct the additional MemberAccessExpression

'For example, if the user says "myCustomer.Order.OrderID = 4" we need an

'additional MemberAccessExpression for "Order.OrderID = 4"

For i = 1 To UBound(s)

pInfo = pInfo.PropertyType.GetProperty(s(i))

callExpr = Expression.MakeMemberAccess(callExpr, pInfo)

'ConstantExpression representing the value on the left side of the operator

Dim valueExpr = Expression.Constant(value, valueType)

Dim b As Expression = CombineExpression(callExpr, condType, valueExpr)

LambdaExpr = Expression.Lambda(Of Func(Of T, Boolean))(b, New ParameterExpression() {paramExpr})

'Compile the lambda expression into a delegate

del = DirectCast(LambdaExpr.Compile(), Func(Of T, Boolean))

End Sub

'Combines two conditions according to the specified operator

Friend Overloads Shared Function Combine(ByVal cond1 As Condition(Of T), _

ByVal condType As Compare, _

ByVal cond2 As Condition(Of T)) As Condition(Of T)

Dim c As New Condition(Of T)

Dim b As Expression = CombineExpression(cond1.LambdaExpr.Body, _

condType, _

cond2.LambdaExpr.Body)

Dim paramExpr() = New ParameterExpression() {GetParamInstance(GetType(T))}

'Create the LambdaExpression and compile the delegate

c.LambdaExpr = Expression.Lambda(Of Func(Of T, Boolean))(b, paramExpr)

c.del = Condition.CombineFunc(cond1.del, condType, cond2.del)

Return c

End Function

'Combines multiple conditions according to the specified operator

Friend Overloads Shared Function Combine(ByVal cond1 As Condition(Of T), _

ByVal condType As Compare, _

ByVal ParamArray conditions() As Condition(Of T)) As Condition(Of T)

Dim finalCond = cond1

For Each c In conditions

finalCond = Condition.Combine(finalCond, condType, c)

Return finalCond

End Function

'Run query locally instead of remotely

Public Function Matches(ByVal row As T) As Boolean

Return del(row) 'passes the row into the delegate to see if it's a match

End Function

'Overloaded operators - allows syntax like "(condition1 Or condition2) And condition3"

Public Shared Operator And(ByVal c1 As Condition(Of T), ByVal c2 As Condition(Of T)) As Condition(Of T)

Return Condition.Combine(c1, Compare.And, c2)

End Operator

Public Shared Operator Or(ByVal c1 As Condition(Of T), ByVal c2 As Condition(Of T)) As Condition(Of T)

Return Condition.Combine(c1, Compare.Or, c2)

End Operator

Public Shared Operator Xor(ByVal c1 As Condition(Of T), ByVal c2 As Condition(Of T)) As Condition(Of T)

Return Condition.Combine(c1, Compare.Xor, c2)

End Operator

End Class

'Represents a condition like "object.Property = value"

'In this case object is of type T, and value is of type S

'Even though most of the logic for this is already in the base class,

'defining a second generic parameter means the user doesn't have to

'pass in a System.Type - it can just be inferred.

Public Class Condition(Of T, S) : Inherits Condition(Of T)

Friend Sub New(ByVal propName As String, ByVal condType As Compare, ByVal value As S)

MyBase.New(propName, condType, value, GetType(S))

End Sub

End Class

#End Region

#Region "Extension Methods"

'Filters an IQueryable(Of T) according to the specified condition

<Extension()> _

Public Function Where(Of T)(ByVal source As IQueryable(Of T), _

ByVal condition As Condition(Of T)) As IQueryable(Of T)

Dim callExpr = Expression.Call(GetType(Queryable), "Where", _

New Type() {source.ElementType}, source.Expression, _

Expression.Quote(condition.LambdaExpr))

Return CType(source.Provider.CreateQuery(callExpr), IQueryable(Of T))

End Function

'Filters an IEnumerable(Of T) according to the specified condition

<Extension()> _

Public Function Where(Of T)(ByVal source As IEnumerable(Of T), _

ByVal condition As Condition(Of T)) As IEnumerable(Of T)

Return source.Where(condition.del)

End Function

'Extension method that can be called off any type that implements IEnumerable(Of T),

'which constructs a Condition with T as the element type and S as the value's type

<Extension(), EditorBrowsable(EditorBrowsableState.Always)> _

Public Function CreateCondition(Of T, S)(ByVal dataSource As IEnumerable(Of T), _

ByVal propName As String, _

ByVal condType As condition.Compare, _

ByVal value As S) As Condition(Of T, S)

Return Condition.Create(dataSource, propName, condType, value)

End Function

#End Region

End Module

The complete sample (including the code for the ConditionBuilder control) is available in the attached .zip file below. While we’ve got the basics going pretty well, there’s definitely a lot of things you can extend the sample to do, such as:

1. Add support for Grouping conditions into the ConditionBuilder – right now you’d have to drop down to the API to construct something like (condition1 Or condition2) And condition3.

2. Add a .Select extension method to dynamically control which fields you bring back from the database. This will improve performance in situations where you don’t want to return all fields, but you’ll lose strong-typing of your results since the method will have to return IQueryable instead of IQueryable(Of T).

3. Add extension methods for OrderBy, Take, Skip, Distinct etc… Each of these methods can still return strongly-typed results since they don’t change the element type; if you pass in T they return T.

4. The Condition API supports nested property access (i.e. Where customer.Orders.Salesman.Name = “Bob”), but it’d take a bit more work to surface this in the ConditionBuilder control.

That’s everything, now all you have to do is download Beta2 and try it out!

Jonathan

How to implement IQueryable (by Kevin Halverson)

VBTeam — Thu, 12 Jul 2007 01:15:00 GMT

Here’s a link to a post I just finished on ‘How to implement IQueryable’ (create a custom Linq provider).

http://blogs.msdn.com/kevin_halverson/archive/2007/07/10/how-to-implement-iqueryable.aspx

My example project allows you to query the Windows Vista (Desktop) Search engine for files on your local machine. For example:

Dim index As New WDSQueryObject

Dim cutoffDate = #6/28/2007#

Dim r = From file In index _

Where file.CreationTime > cutoffDate And _

file.Name Like "%.exe" _

Select file.FullName, file.CreationTime