Yet Another Language Geek : Compilers

Having Trouble with Queries

wesdyer — Thu, 04 Jan 2007 19:55:25 GMT

A Funny Joke and a Sad Joke

There is a joke that goes something like this:

Two men are hopelessly lost in hot air balloon. Their condition is aggravated by the fact that they are enshrouded by a thicket of fog. When it seems all is lost, suddenly the fog parts and miraculously a man is standing on the ground below.

They shout, "Hey sir!" The man looks up a little bewildered by the sudden appearance of a hot air balloon.

Their hearts take courage and they ask, "Do you know where we are?"

The man on the ground thinks for a moment as the two men above wait in desperation.

Finally, the man looks up as the fog begins to engulf him again and says, "You are in a hot air balloon."

The two men slump into the hot air balloon as their last hope slips away. One of them says to the other, "He must have been a professor."

The other says, "Why do you say that?"

The first responds, "Because what he said was totally correct but absolutely useless."

The joke has several variants with lawyer, teacher, engineer, etc. substituted for professor. This joke's punch line also is a very good moral: Don't annoy users by telling them something that is correct but useless. By useless, I mean of no practical value which can include things that are incomprehensible to the average user.

Lets change the story a little bit:

Two programmers are hopelessly behind on an important software project. Their condition is aggravated by the fact that they are having trouble learning to use the new query feature in C# 3.0. They write the following query:

var q = from c in Customers
where c.State == "WA"
select c.ToStrin(); // c.ToStrin() should be c.ToString()
It doesn't seem quite right but they don't know what they did wrong.

They say to each other, "Maybe the compiler's error message will be helpful and give a hint or two at how to use the language feature correctly."

Their hearts take courage and they look at the error message pane in Visual Studio in anticipation thinking to themselves, "What mistake did we make when writing our query?"

The compiler computes for a moment as the two programmers wait in desperation.

Finally, the compiler spits out an error message which says, "The type arguments for method 'System.Linq.Queryable.Select<TSource,TResult>(System.Linq.IQueryable<TSource>, System.Linq.Expressions.Expression<System.Linq.Func<TSource,TResult>>)' cannot be inferred from the usage. Try specifying the type arguments explicitly."

The two programmers slump into their chairs as their last hope slips away. One of them says to the other, "Those C# compiler guys must be studying the C++ template error messages and the ML type inference error messages."

The other says, "Why do you say that?"

The first responds, "Because what it said was totally correct but absolutely useless."

Error messages are tricky to get right and for some reason compilers often have some of the most cryptic error messages. To be fair, the primary purpose of a compiler is to correctly translate a correct program and to reject incorrect programs. But to paraphrase a common saying, computer programming languages are primarily for humans and only incidentally for machines. Maybe the error messages are cryptic because it is a tool written by programmers for programmers or possibly because the push to have more language features or better optimizations outweighs the demand for quality error messages. Whatever the reason is, compilers tend not to be very helpful in providing good error messages.

The C# compiler team devotes a lot of thought and work to getting error messages right. Eric Lippert, one of my teammates, has a great post about error messages (apparently GrantRi made one too). Early on in C# 3.0, we focused on getting the language features complete (100% spec compliant) and sometimes we sacrificed performance or the quality of error messages to achieve this. Now later in the product cycle we are focusing on things like the quality of the error messages that we generate, the performance of the C# compiler, and the performance of the generated IL.

The Challenges of Getting It Right

C# 3.0 presents a bunch of new challenges to our goal of getting our users the right product. The language features are much further from the IL that is generated than they ever have been (try writing a small app with a handful of queries then open the compiled assembly in ILDASM or Reflector and you will see what I mean). What all of this amounts to is that the distance from when an error introduced in compilation and when it detected has increased making it more difficult to produce very good error messages. Furthermore, C# 3.0 uses type inference quite a bit which makes it even more difficult to produce good error messages. A glance at a few languages that use type inference extensively will show you that there currently seems to be a tradeoff between the power of a type inference algorithm and its ability to produce good error messages.

Consider when an error occurs in a Linq to Objects query (c.Lastname should be c.LastName):

var q = from c in Customers
        join o in Orders on c.ID equals o.CustomerID
        where c.FirstName == "Wes"
        orderby o.Date
        select new { c.Lastname, o };

When the query is compiled using an early production compiler, an error message like the following would occur:

The type arguments for method 'System.Linq.Queryable.Select<TSource,TResult>(System.Linq.IQueryable<TSource>, System.Linq.Expressions.Expression<System.Linq.Func<TSource,TResult>>)' cannot be inferred from the usage. Try specifying the type arguments explicitly.

Which probably will only serve to evoke a scratch of the head and an indefinite, "Hmm...". This error message is bad for a number of reasons. First what is this talk about type arguments, I don't see any generics in sight. Second, why is the error referring to Queryable when the query is clearly an Enumerable query. Third, what is this Select method that is being referred to? The query doesn't make a single method call. Fourth, why does the error talk about expression trees when the query didn't even have a lambda in it? It gets even worse if the span of the query is considered. The span for this particular query is the "select" query contextual keyword. Is that where the error occurred? That doesn't seem likely.

It would be so much better if the error message was just

'Customer' does not have a member named 'Lastname'.

where the span for the error was "c.Lastname" in the select clause. So why not just produce this error message and why produce the first error message? Believe it or not, the first error message actually makes sense if you think about it really hard (no really, it's true!). The error message refers to type arguments and the Select method because the select clause is translated into a call to a Select method which in this case is a generic method. The reason why it refers to Queryable is because the methods that are candidates are all extension methods and one of those extension methods has a Queryable receiver. Finally, the expression in the select clause becomes the body for a generated lambda that is passed to the Select method, but this lambda does not explicitly specify the types of its arguments so they must be inferred. The problem is that no types can be inferred for the arguments such that the body does not have errors and hence we get the type inference errors. I know, I know this still doesn't excuse the error message. While the error message is correct, it is not very helpful. Especially considering that the user should not need to know how queries are translated or how lambdas work or how type inference works and so on.

A Deeper Look

During our third milestone, we did work to make this error message better. It was the worst of all of our error messages. I can't find words adequate to describe how painful it was to fix very large queries that had simple errors.

The root of the problem lies in producing good error messages for lambdas even though the lambda may be bound several different ways through type inference. Consider the following program:

using System.Linq;
class Program
{
static void Main(string[] args)
{
    M(x => x.FooM());            // case 1
    M(x => x.BarM());            // case 2
    M(x => x.BazM());            // case 3
    M(x => x.FooM() + x.BarM()); // case 4
}
static void M(Func<Foo, Foo> f)
{
}
static void M(Func<Bar, Bar> f)
{
}
}
class Foo
{
public Foo FooM() { return this; }
}
class Bar
{
public Bar BarM() { return this; }
}

In each call to M, a lambda is passed as the only parameter. In each lambda, x could have the type Foo or the type Bar since there are two methods named M which take a parameter type of Foo and a parameter type of Bar respectively. In case 1, if x has type Foo then the lambda binds correctly but if it has type Bar then an error occurs so x must have type Foo. In case 2, we can use the same logic to show that x must have type Bar. In case 3, the lambda has an error whether x is bound having type Foo or type Bar so case 3 is an error. In case 4, we have the same as case 3. So both case 3 and case 4 produce cryptic error messages.

Let's look at cases 3 and 4 closely. In case 3, when we bind the lambda with type Foo we get one error:

test.cs(9,18): error CS1061: 'Foo' does not contain a definition for 'BazM' and
no extension method 'BazM' accepting a first argument of type 'Foo'
could be found (are you missing a using directive or an assembly
reference?)

When we bind the same lambda with type Bar we also get one error:

test.cs(9,18): error CS1061: 'Bar' does not contain a definition for 'BazM' and
no extension method 'BazM' accepting a first argument of type 'Bar'
could be found (are you missing a using directive or an assembly
reference?)

Notice the similarity between the two error message? They both have the same error number and the same span. Essentially, the same program text caused the error and both are the same error class. They only differ in the parameters that are used to produce the error text ("Foo" vs "Bar").

In case 4, when we bind the lambda with type Foo we get one error:

test.cs(10,29): error CS1061: 'Foo' does not contain a definition for 'BarM' and
no extension method 'BarM' accepting a first argument of type 'Foo'
could be found (are you missing a using directive or an assembly
reference?)

And when we bind the lambda with type Bar we get one error:

test.cs(10,18): error CS1061: 'Bar' does not contain a definition for 'FooM' and
no extension method 'FooM' accepting a first argument of type 'Bar'
could be found (are you missing a using directive or an assembly
reference?)

This time we have two errors with the same error number but different spans. The difficulty of dealing with the possibly multiple bindings of lambdas and the potentially (very) large number of error messages led to the incomprehensible error message. The sad part is that those more helpful error messages are being hidden (maybe swallowed is a better term) by the more cryptic error. But how can we let them shine through?

Fixing the Problem

We can solve this by introducing two sets of error messages: the union of error messages and the intersection of error messages. Each binding of a lambda produces a set of error messages. In the first such binding of error messages we initialize both the union and intersection sets to be the set of errors produced by the binding. On each subsequent binding we refine the sets by either taking the union of the binding error message set with the union set or the intersection of the binding error message set with the intersection set. After trying all the error messages, if the intersection set is not empty then we report only the error messages from that set. Otherwise, we report all of the errors from the union set. For the purposes of set semantics, we define two error messages to be the same iff they have both the same span and the same error number (not necessarily that they have the same text).

The intuition behind this is that the intersection set of errors are those errors that we are sure need to be fixed for the lambda to bind since they are errors under every binding of the lambda. The union errors are possible errors depending on which binding of the lambda is under consideration. So we don't report the union unless the intersection is empty since if they fix all errors in the intersection that may make the program compile correctly (and most often does). In practice, the compiler rarely reports the union set since the intersection set is almost always not empty. In both of the example queries above the intersection set is not empty and so it can be reported.

We are still working on getting the error messages right and would love to have your feedback on other problems that you notice. Next time you see a good error message inside of a query you can now understand how it got there. Or maybe not. Maybe it will be helpful enough that you won't even think of it. If that is the case then we have done our job right.

Transparent Identifiers

wesdyer — Fri, 22 Dec 2006 22:53:41 GMT

My last post described in detail how query expressions in C# are translated, but I have a confession to make. I did a little bit of hand waving through one part of the translation rules. The astute reader (whomever that is, whenever I read such things I always wonder, "Am I that astute reader?") may have noticed that four of the rules introduced a decidedly foreign language feature into rewrite rules. For example, examine rule #14 closely:

14. From followed by let

from x1 in e1 let x2 = e2 ...
from * in e1.Select(x1 => new { x1, x2 = e2 }) ...

What is that asterisk in the resultant transformation and what does it mean? It isn't a pointer deferencing operator and it isn't a multiplication operator, hmmm...What could it be? It looks like it is taking the place of an identifier in the from clause but that is a strange name for an identifier.

In fact, the asterisk denotes a transparent identifier and it is the name of this identifier. Transparent identifiers are one of the strangest additions to C# 3.0. The spec has precious little to say about them.

1. They are introduced to a program only by query rewriting

2. Each transparent identifier has exactly one associated anonymous type (the anonymous type introduced during the same rewrite step as the transparent identifier)

3. When a transparent identifier is in scope, its members are in scope as well (transivitely)

Rule 3 is kind of strange because it mentions scope in an otherwise syntactic rewrite. Scope has to do with semantics and not syntax so we have to be very careful how we apply this rule. Now consider the following query:

from x in foo

let y = f(x)

let z = g(x, y)

select h(x, y, z)

Applying rule 14 the first time the query is reduced to:

from * in foo.Select(x => new { x, y = f(x) })

let z = g(x, y)

select h(x, y, z)

Here we introduced a transparent identifier and the anonymous type that is associated with it is new { x, y = f(x) }. Now we apply rule 14 again.

from * in foo.Select(x => new { x, y = f(x) }).Select(* => new { *, z = g(x, y) })

select h(x, y, z)

We have now introduced another transparent identifier but this one is associated with a different anonymous type. It is associated with new { *, z = g(x, y) }. In order to note the difference they will labeled them *1 and *2.

from *2 in foo.Select(x => new { x, y = f(x) }).Select(*1 => new { *1, z = g(x, y) })

select h(x, y, z)

Finally, we can apply rule 15.

foo.Select(x => new { x, y = f(x) }).Select(*1 => new { *1, z = g(x, y) }).Select(*2 => h(x, y, z))

Where *1 and *2 are associated with the following anonymous types:

*1 = new { x, y = f(x) }

*2 = new { *1, z = g(x, y) }

You may notice that in the final select call that there is a lambda with only one parameter (*2) but in the body of this lambda we reference x, y, and z but none of these variables are in scope. Using rule 3 about transparent identifiers we see that when *2 is in scope so are *1 and z but since *1 is now in scope so are x and y (the transitive closure of the members of *2). So really when we refer to x in h(x, y, z) we are really referring to *2.*1.x. Thus we have the following:

foo.Select(x => new { x, y = f(x) }).Select(*1 => new { *1, z = g(*1.x, *1.y) }).Select(*2 => h(*2.*1.x, *2.*1.y, *2.z))

Finally, once we realize that the compiler treats these transparent identifiers just as unspeakable compiler generated names then we see that all the magic has been removed:

foo.Select(x => new { x, y = f(x) }).Select(t0 => new { t0, z = g(t0.x, t0.y) }).Select(t1 => h(t1.t0.x, t1.t0.y, t1.z))

Voila! No more transparent identifiers. Transparent identifiers are used in query rewriting to package up the intermediate results and pass them onto the next clause. They are means of essentially creating a little scope and passing it around. It is a fantastic idea that allows variables to flow through the queries providing the kind of behavior that users expect to see.

Comprehending Comprehensions

wesdyer — Fri, 22 Dec 2006 01:35:00 GMT

Not long ago, I was reading through some articles posted on programming.reddit.com when I came across an article claiming that C# is trying to be a dynamic language. One user posted a comment that mentioned that C# 3.0 included among other things "embedded SQL". Unfortunately, it seems that there is still some confusion about what query expressions are. Let's be clear: query expressions are not embedded SQL. In fact, they are closer to list comprehensions in languages like Haskell and Python than they are to embedded SQL.

At a high level, query expressions provide nine basic operations (from, join, join...into, group...by, orderby, where, let, select, into). Each of these basic operations is expressed as a clause. The clauses are then chained together to provide a declaration of query; however, if you only view a query then you can't be sure what the behavior at runtime will be. The reason is that queries are translated into a series of method calls but the methods that are invoked are not predefined methods from the compiler's point of view. The compiler just creates the calls and binds the calls to whatever methods are applicable given the arguments and the receiver of the call. This is both the source of the beauty and the confusion of query expressions.

Syntactic Sugar

Query expressions are an interesting language feature because they are pure syntactic sugar. But to really understand queries we need to see what lies beneath. Let's break through the syntactic sugar.

Consider the following query:

   1:  var q = from x in foo

   2:          where x > 0

   3:          select x + 1;

This query is transformed into:

   1:  var q = foo.Where(x => x > 0)

   2:             .Select(x => x + 1);

But what "Where" and "Select" methods are being called? It is impossible to tell from this code snippet; it depends on what the type of "foo" is and which extension methods are in scope. This is the beauty of query expressions and why we can do neat things like LINQ to Objects, DLINQ, and LINQ to Entities and so on. Each of these providers can author their own "Where" and "Select" methods which would be called if foo was a DLINQ object or a LINQ to entities object. If you wanted the queries to behave differently all you need to do is define your own methods such that the compiler will bind to them when you write a query. The set of methods that needs to be implemented is called the "Query Pattern" and implementers of this pattern are called "Providers".

Transformation Rules

Their are (only ;) eighteen rules that define how queries are transformed. Here is how the transformations can be performed given that you have a query (each transformation lists the pattern that to match first and then the resultant transformation on the next line):

First remove query continuations.

1. If the query has a continuation rewrite it (only rewrite the first one right now)

from x1 in e1 ... into x2 ...

from x2 in (from x1 in e1 ...) ...

Now transform the query into a list of clauses and for each clause if it is explicit then make it implicit like rules 2-4 state:

2. Explicit from

from T x1 in e1

from x1 in e1.Cast<T>()

3. Explicit join

join T x1 in e1 on k1 equals k2

join x1 in e1.Cast<T>() on k1 equals k2

4. Explicit join into

join T x1 in e1 on k1 equals k2 into g

join x1 in e1.Cast<T>() on k1 equals k2 into g

Now we check for the degenerate query and rewrite it in a special way.

5. Now if the query is degenerate then do a special transform

from x1 in e1 select x1

e1.Select(x1 => x1)

Finally, now that we have a list of clauses (without explicit types) we simply apply rules 6-18 repeatedly until we don't have a query any more. Think of it as pattern matching. If the list of clauses matches the given pattern then reduce the list given the transformation rule. Rinse and Repeat.

6. Two froms followed by a select

from x1 in e1 from x2 in e2 select e3

e1.SelectMany(x1 => e2, (x1, x2) => e3)

7. Two froms followed by a non-select

from x1 in e1 from x2 in e2 ...

from * in e1.SelectMany(x1 => e2, (x1, x2) => new { x1, x2 }) ...

8. From followed by a join followed by a select

from x1 in e1 join x2 in e2 on k1 equals k2 select e3

e1.Join(e2, x1 => k1, x2 => k2, (x1, x2) => e3)

9. From followed by a join followed by a non-select

from x1 in e1 join x2 in e2 on k1 equals k2 ...

from * in e1.Join(e2, x1 => k1, x2 => k2, (x1, x2) => new { x1, x2 }) ...

10. From followed by a join with into followed by a select

from x1 in e1 join x2 in e2 on k1 equals k2 into g select e3

e1.GroupJoin(e2, x1 => k1, x2 => k2, (x1, g) => e3)

11. From followed by a join with into followed by a non-select

from x1 in e1 join x2 in e2 on k1 equals k2 into g ...

from * in e1.GroupJoin(e2, x1 => k1, x2 => k2, (x1, g) => new { x1, g }) ...

12. From followed by orderby (for some number of keys)

from x1 in e1 orderby k1, k2, k3 ...

from x1 in e1.OrderBy(x1 => k1).ThenBy(x1 => k2).ThenBy(x1 => k3) ...

13. From followed by where

from x1 in e1 where e2 ...

from x1 in e1.Where(x1 => e2) ...

14. From followed by let

from x1 in e1 let x2 = e2 ...

from * in e1.Select(x1 => new { x1, x2 = e2 }) ...

15. From followed by non-identity select

from x1 in e1 select e2

e1.Select(x1 => e2)

16. From followed by identity select

from x1 in e1 select x1

e1

17. From followed by non-identity group by

from x1 in e1 group e2 by e3

e1.GroupBy(x1 => e3, x1 => e2)

18. From followed by identity group by

from x1 in e1 group x1 by e2

e1.GroupBy(x1 => e2)

Example Translation

Consider the following query:

   1:  from x in foo

   2:  where x > 0

   3:  from y in bar

   4:  select x + y

To translate this query, we first remove check to see if rule 1 is applicable (no) and then we translate the query into a list of clauses while applying rules 2-4 (which don't apply to this query). After rules 1-4 we have the following list of clauses.

[from x in foo, where x > 0, from y in bar, select x + y]

Rule 5 does not apply because it is not the degenerate identity select. Now we will apply rules 6-18 repeatedly until the query is fully reduced. The rules are always applied to the front of the list and so the first rule that is applicable is rule 13. Which states:

from x1 in e1 where e2 ...

from x1 in e1.Where(x1 => e2) ...

Applying this rule the list becomes:

[from x in foo.Where(x > 0), from y in bar, select x + y]

Now the next rule that is applicable is rule 6 which states:

from x1 in e1 from x2 in e2 select e3

e1.SelectMany(x1 => e2, (x1, x2) => e3)

The list now becomes:

[foo.Where(x > 0).SelectMany(x => bar, (x, y) => x + y)]

We no longer have a query so we are done and the final rewrite looks like:

   1:  foo.Where(x => x > 0)

   2:     .SelectMany(x => bar, (x, y) => x + y)

Other queries are translated in a similar fashion. As you translate more of these, you will notice that the rules are in fact very simple and can be easily mechanically applied. With some practice, you should be able to translate them quickly in your head much like doing math. It is simply a process of applying the rules in the right order and reaching a fixed point.

You may have noticed that rule 1 actually creates a nested query. The rewrite rules as I explained them did not remove this nested query. After rewriting a query, recurse and rewrite queries found in its subexpressions. And that is it!

Implications

Interestingly, the C# spec does not specify much more than this about the semantics of query expressions; however, the rewrite rules do imply some significant details that may not be apparent. Some of these details include scoping rules, error message and debugging behavior, and even hidden new language features like transparent identifiers.