Fabulous Adventures In Coding : C#

Query transformations are syntactic

Eric Lippert — Mon, 07 Dec 2009 14:20:00 GMT

As you probably know, there are two ways to write a LINQ query in C#. The way I personally prefer is to use the “query comprehension” syntax:

from customer in customerList
where customer.City == "London"
select customer.Name

Or you can, equivalently, use the “fluent method call” syntax:

customerList
.Where(customer=>customer.City == "London")
.Select(customer=>customer.Name)

These are guaranteed to be equivalent because the compiler simply transforms the former syntax into the latter syntax before it compiles it. An interesting aspect of this transformation is that it is (almost) entirely syntactic. (The "transparent identifiers" generated for certain queries require a small amount of semantic analysis to determine the corresponding anonymous type, but for the most part, all we do is just pull the raw hunks of code out of each clause and reorganize the program into the method call form.) Once it is in a form that the rest of the compiler can understand, then semantic analysis proceeds as usual.

This means that it is perfectly legal to do dumb things. For example, suppose we decide that by “where” we mean don’t mean “filter”, we actually mean “multiply”. And by “select” we actually mean “add”, not “project”. No problem!

static class ExtInt
{
 public static int Where(this int c, Func<int, int> f)
 {
 return f(0) * c;
 }
 public static int Select(this int c, Func<int, int> f)
 {
 return f(0) + c;
 }
}
…
int ten = 10;
int twenty = 20;
int thirty = 30;
int result = from c in ten where twenty select thirty;
Console.WriteLine(result);

And sure enough, ten where/times twenty select/plus thirty is 230. This is a deeply strange way to write a mathematical expression, but legal.

The semantics of the Where method are supposed to be “takes a collection of T and a predicate mapping T to bool, and returns a filtered collection of those T items which match the predicate”. This method does not take a collection or a predicate, it does not return a collection, and it certainly does not have the semantics of filtering, but the compiler neither knows nor cares about any of those facts. All the compiler does is syntactically turn that line into

int result = ten.Where(c=>twenty).Select(c=>thirty);

and compile it; that line compiles just fine, so, no problem as far as the compiler is concerned.

The C# specification has a section which describes the pattern we expect a query provider to implement: that Where takes a predicate, and so on. But we perform no checks whatsoever that you successfully implemented a query provider that implements either the form or the semantics of our recommended pattern. If you do something crazy, like redefine “where” to take something other than a predicate and you get crazy results, then, well, what can I tell you? If it hurts when you do that then don’t do that!

Exact rules for variance validity

Eric Lippert — Thu, 03 Dec 2009 14:32:00 GMT

I thought it might be interesting for you all to get a precise description of how exactly it is that we determine when it is legal to put "in" and "out" on a type parameter declaration in C# 4. I'm doing this here because (1) it's of general interest, and (2) our attempt to make a more human-readable version of this algorithm in the draft C# 4.0 specification accidentally introduced some subtle errors. We're working on correcting those errors for the final release of the specification; until then, these definitions are the definitions I actually worked from to do the implementation, so they're accurate.

These definitions are pretty much stolen straight from the CLI spec section on variance; I am indebted to its authors for their careful and precise definitions.

The first things we need to define are three kinds of "validity" for types. I want to define "valid covariantly", "valid contravariantly" and "valid invariantly", but only as applied to types. We'll talk about what makes an interface declaration a valid one later; we need these definitions first.

Before we get into a precise definition, I want to talk about what "valid covariantly" means logically. The idea that we are attempting to capture here boils down to "the type we're talking about is not contravariant". Whether it is covariant, we don't really care. All we care about it is that if it is valid covariantly, then it is not contravariant. Similarly, by "valid contravariantly", we mean "not covariant".

This leads us to our first brain-hurting definition: A type is "valid invariantly" if it is both valid covariantly and valid contravariantly. That sounds a bit crazy -- how can it be valid invariantly if it is both covariant and contravariant? But "valid covariantly" does not mean "is covariant", it means "it's guaranteed to not be contravariant". So if a type is valid covariantly and valid contravariantly, then it is guaranteed to be neither contravariant nor covariant, and therefore must be invariant.

Anyway. We want to take any specific type and classify it as valid covariantly, valid contravariantly, or both, a.k.a., valid invariantly. (We don't need to worry about coming up with a term for the "or none of the above" case because in practice that case is never relevant.)

A type is valid covariantly if it is:

1) a class, struct, nullable, enum, pointer, non-generic delegate or non-generic interface type.

This should make sense. Remember, what we're getting at is "not contravariant". Classes and structs are either non-generic, in which case they are clearly not contravariant, or they are generic, in which case again they are not contravariant because we do not support variance on classes or structs. Nullable types are just aliases for Nullable<T> which is a struct type, and therefore not variant. Enums and pointers are non-generic so they are not variant. And non-generic delegates and interfaces are by definition not generic and therefore not variant either. These are the easy cases.

2) An array type T[] where T is valid covariantly.

Because we have array covariance in C#, but not array contravariance, we can make arrays valid covariantly. Again, remember, "valid covariantly" means "not contravariant".

3) A generic type parameter type, if it was not declared as being contravariant.

Generic type parameter types are of course types as far as the compiler is concerned. If you're inside the declaration of a generic interface then you can use its type parameters as types. In that context, such types are valid covariantly if they were not declared as contravariant (in).

4) A constructed interface or delegate type X<T1, ... Tk> might be valid covariantly. To determine if it is, we examine each type argument differently, depending on whether the corresponding type parameter was declared as covariant (out), contravariant (in), or invariant (neither). If the ith type parameter was declared as covariant, then Ti must be valid covariantly. If it was declared as contravariant, then Ti must be valid contravariantly. If it was declared as invariant, then Ti must be valid invariantly.

As one would expect, covariant validity preserves the direction of validity; covariant parameters must be valid covariantly, contravariant parameters must be valid contravariantly.

OK, I hope that was relatively painless. The rules for contravariant validity are similar; as one would expect from the "backwards" nature of contravariance, the directions are reversed in the complicated case:

A type is valid contravariantly if it is:

1) a class, struct, nullable, enum, pointer, non-generic delegate or non-generic interface type.

2) An array type T[] where T is valid contravariantly.

Arrays are covariant in their element type. Covariance preserves the direction of variance. Therefore, to be valid contravariantly, an array (covariant in its element type) must be contravariantly valid in its element type.

3) A generic type parameter type, if it was not declared as being covariant.

Remember, "valid contravariantly" means "not covariant". It does not mean "contravariant".

4) A constructed interface or delegate type X<T1, ... Tk> might be valid contravariantly. If the ith type parameter was declared as contravariant, then Ti must be valid covariantly. If it was declared as covariant, then Ti must be valid contravariantly. If it was declared as invariant, then Ti must be valid invariantly.

As one might expect, contravariant validity reverses the direction of validity.

Now perhaps you see why we wanted to rewrite these definitions into something more human-readable for the spec. And perhaps you also see why we accidentally introduced errors in doing so; bending your brain around all this logic is not always easy.

OK, now that we've got that, we can make a definition of what it means for an interface to be valid. An interface must meet the following conditions:

* The return types of all non-void interface methods must be valid covariantly.

* Every formal parameter type of all interface methods must be valid contravariantly. (Invariantly if it is an out or ref parameter.)

* For all generic methods on an interface, every constraint on the generic method type parameters must be valid contravariantly.

* All its base interface types must be valid covariantly.

* The type of a property or indexer must be valid covariantly if it has a "getter" and valid contravariantly if it has a "setter".

* Any formal parameter types of an indexer must be valid contravariantly.

* The delegate types of all its events must be valid contravariantly.

The first two are pretty straightforward; return types "go out" so they have to be valid covariantly, formal parameter types "go in", so they have to be valid contravariantly. But what's up with the third one? What do constraints on generic method type parameters have to do with interface validity?

Well, let's suspend the third rule and see what goes wrong.

interface I<out T>
{
 void M() where U : T;
 // illegal; this has to be valid contravariantly but it is a covariant type parameter constraint.
 // Let it ride for now and demonstrate an error.
}

class C<T> : I<T> { public void M() {} }
// the constraint is inherited implicitly and not re-stated.

I<Giraffe> igiraffe = new C<Giraffe>(); // C<T> implements I<T>
I<Animal> ianimal = igiraffe; // interface is covariant in T
ianimal.M<Turtle>(); // satisifies the constraint that U must be an Animal.

Uh oh. ianimal is really an instance of C<Giraffe>. The M method on C<Giraffe> inherits a requirement that U inherit from Giraffe. Turtle does not inherit from Giraffe. Therefore we've just violated the constraint on M. The only places where we can catch this is in the declaration; every other step is perfectly legal. Therefore, a constraint cannot be covariant. But if we make it contravariant (or invariant) then it all works out. For example, let's make a contravariant type parameter constraint:

interface I2<in T> // contravariant this time
{
void M() where U : T;
}

class C2<T> : I2<T> { public void M() {} }
I2<Animal> i2animal = new C2<Animal>(); // C2<T> implements I2<T>
I2<Mammal> i2mammal = i2animal; // interface is contravariant in T
i2mammal.M<Giraffe>(); // satisifies the constraint that U must be an Animal.

And now everything is fine; the compile-time constraint checker verifies that Giraffe is Mammal; at runtime it must be Animal, and so the compiler has verified that by verifying that it is Mammal.

The rules for delegate declarations are a straightforward simplification of the rules for interface declarations. To be a valid delegate declaration, the return type must be valid covariantly (or void), the formal parameter types must be valid contravariantly (or invariantly if they are out/ref), and any type parameter constraints must be valid contravariantly.

What's the difference between covariance and assignment compatibility?

Eric Lippert — Mon, 30 Nov 2009 14:51:00 GMT

I've written a lot about this already, but I think one particular point bears repeating.

As we're getting closer to shipping C# 4.0, I'm seeing a lot of documents, blogs, and so on, attempting to explain what "covariant" means. This is a tricky word to define in a way that is actually meaningful to people who haven't already got degrees in category theory, but it can be done. And I think it's important to avoid defining a word to mean something other than its actual meaning.

A number of those documents have led with something like:

"Covariance is the ability to assign an expression of a more specific type to a variable of a less specific type. For example, consider a method M that returns a Giraffe. You can assign the result of M to a variable of type Animal, because Animal is a less specific type that is compatible with Giraffe. Methods in C# are 'covariant' in their return types, which is why when you create a covariant interface, it is indicated with the keyword 'out' -- the returned value comes 'out' of the method."

But that's not at all what covariance means. That's describing "assignment compatibility" -- the ability to assign a value of a more specific type to a storage of a compatible, less specific type is called "assignment compatibility" because the two types are compatible for the purposes of verifying legality of assignments.

So what does covariance mean then?

First off, we need to work out precisely what the adjective "covariant" applies to. I'm going to get more formal for a bit here, but try to keep it understandable.

Let's start by not even considering types. Let's think about integers. (And here I am speaking of actual mathematical integers, not of the weird behaviour of 32-bit integers in unchecked contexts.) Specifically, we're going to think about the ≤ relation on integers, the "less than or equal to" relation. (Recall that of course a "relation" is a function which takes two things and returns a bool which indicates whether the given relationship holds or does not hold.)

Now let's think about a projection on integers. What is a projection? A projection is a function which takes a single integer and returns a new integer. So, for example, z → z + z is a projection; call it D for "double". So are z → 0 - z, N for "negate" and z → z * z, S for "square".

Now, here's an interesting question. Is it always the case that (x ≤ y) = (D(x) ≤ D(y))? Yes, it is. If x is less than y, then twice x is less than twice y. If x is equal to y then twice x is equal to twice y. And if x is greater than y, then twice x is greater than twice y. The projection D preserves the direction of size.

What about N? Is it always the case that (x ≤ y) = (N(x) ≤ N(y))? Clearly not. 1 ≤ 2 is true, but -1 ≤ -2 is false. But we notice that the reverse is always true! (x ≤ y) = (N(y) ≤ N(x)). The projection N reverses the direction of size.

What about S? Is it always the case that (x ≤ y) = (S(x) ≤ S(y))? No. -1 ≤ 0 is true, but S(-1) ≤ S(0) is false. What about the opposite? Is it always the case that (x ≤ y) = (S(y) ≤ S(x)) ? Again, no. 1 ≤ 2 is true, but S(2) ≤ S(1) is false. The projection S does not preserve the direction of size, and nor does it reverse it.

The projection D is "covariant" -- it preserves the ordering relationship on integers. The projection N is "contravariant". It reverses the ordering relationship on integers. The projection S does neither; it is "invariant".

Now I hope it is more clear exactly what is covariant or contravariant. The integers themselves are not variant, and the "less than" relationship is not variant. It's the projection that is covariant or contravariant -- the rule for taking an old integer and making a new one out of it.

So now lets abandon integers and think about reference types. Instead of the ≤ relation on integers, we have the ≤ relation on reference types. A reference type X is smaller than (or equal to) a reference type Y if a value of type X can be stored in a variable of type Y. That is, if X is "assignment compatible" with Y.

Now consider a projection from types to types. Say, the projection "T goes to IEnumerable<T>". That is, we have a projection that takes a type, say, Giraffe, and gives you back a new type, IEnumerable<Giraffe>. Is that projection covariant in C# 4? Yes. It preserves the direction of ordering. A Giraffe may be assigned to a variable of type Animal, and therefore an sequence of Giraffes may be assigned to a variable that can hold a sequence of Animals.

We can think of generic types as "blueprints" that produce constructed types. Let's take the projection that takes a type T and produces IEnumerable<T> and simply call that projection "IEnumerable<T>". We can understand from context when we say "IEnumerable<T> is covariant" what we mean is "the projection which takes a reference type T and produces a reference type IEnumerable<T> is a covariant projection". And since IEnumerable<T> only has one type parameter, it is clear from the context that we mean that the parameter to the projection is T. After all, it is a lot shorter to say "IEnumerable<T> is covariant" than that other mouthful.

So now we can define covariance, contravariance and invariance. A generic type I<T> is covariant (in T) if construction with reference type arguments preserves the direction of assignment compatibility. It is contravariant (in T) if it reverses the direction of assignment compatibility. And it is invariant if it does neither. And by that, we simply are saying in a concise way that the projection which takes a T and produces I<T> is a covariant/contravariant/invariant projection.

UPDATE: My close personal friend (and fellow computer geek) Jen notes that in the Twilight series of novels, the so-called "werewolves" (who are not transformed by the full moon and therefore not actually werewolves) maintain their rigid social ordering in both wolf and human form; the projection from human to wolf is covariant in the social-ordering relation. She also notes that in high school, programming language geeks are at the bottom of the social order, but the projection to adulthood catapults them to the top of the social order, and therefore, growing up is contravariant. I am somewhat skeptical of the latter claim; the former, I'll take your word for it. I suppose the question of how social order works amongst teenage werewolves who are computer geeks is a subject for additional research. Thanks Jen!

The Purpose, Revealed

Eric Lippert — Wed, 25 Nov 2009 17:03:00 GMT

Interestingly enough, no one correctly guessed why I needed code in the compiler to transform a method argument list into a form that batched up the side effecting expressions, stuck their values in variables, and then called the method with the side-effect-free variable references. There certainly were some fascinating ideas in the comments though!

The reason is quite mundane. We had a bug in the code that performs semantic analysis of calls with named arguments. One of our rules in C# that I've talked about a lot on this blog is that sub-expression evaluation occurs in left-to-right order. If you have a method void M(int x, int y) {} and then a call site M(y : F(), x : G()); we cannot simply generate the code as though you'd written M(G(), F()). The side effects of F have to happen before the side effects of G, so we generate code as though you'd said temp1 = F(); temp2 = G(); M(y : temp1, x : temp2); and now we can generate the call as M(temp2, temp1); without worrying about re-ordering a side effect.

The bug was that the rewriter was not getting the reordering correct if the parameters were ref/out or the arguments were locals without side effects. As I often do when fixing a bug, I reasoned that the initial bug had been caused because the rewriter was under-specified.

The fix will get into the initial release of C# 4, but the problems that Pavel identified with it -- that certain obscure side effects such as the order in which bad array accesses throw exceptions, or the order in which static class constructors execute, are not correctly preserved by my algorithm, will not be fixed for the initial release. Doing these correctly requires us to be able to generate temporaries of "reference to argument" types, which we've never done before and which our IL generator was not built to handle. Had we discovered the bug much earlier in the cycle, we probably could have applied a more solid fix, but it is too late now.

Thanks again to Pavel for the great analysis, and thanks to our old friend nikov for reporting the bug in the first place.

And happy thanksgiving to my readers in the United States! As I do every year, I'm roasting a turkey to feed fourteen people. I highly recommend the "brine the bird then roast it upside-down" method described in The Joy Of Cooking. It's worked perfectly for me for the last ten years straight.

Always write a spec, Part Two

Eric Lippert — Mon, 23 Nov 2009 15:01:00 GMT

Upon submitting that specification for review, even before seeing my code, Chris found a bug and an omission.

The omission is that I neglected to say what happens when the ref variable is an index into a fixed-size array buffer. As it turns out, that case is also rewritten as a pointer dereference by the time we get to this point in the code, so missing that case turned out to not be a big deal.

The bug is that this line is wrong:

if x is ref/out instance.field then add var temp=instance to L and ref/out temp.field to A2.

because it does not meet the stated goal of the method, namely, to produce the same results after the transformation:

struct S { public int q; }
static void M(ref int r) { r = r + 1; }
static int Zero() { Console.WriteLine("hello!"); return 0; }
...
S[] arr = new[] { new S() };
M(ref arr[Zero()].q);
Console.WriteLine(arr[0].q); // 1

Here we have a variable expression of the form instance.field which has a side effect; it prints "Hello". So according to the spec, we rewrite that as

S temp = arr[Zero()];
M(ref temp.q);
Console.WriteLine(arr[0].q); // 0 !

Once more, mutable value types are revealed to be pure concentrated evil. We just mutated temp.q, which is a copy of arr[0].

The interesting thing here is that Chris found the bug by reading the spec and thinking about whether I'd missed any interesting cases. Bugs are always cheaper to fix the earlier you find them; a bug you fix as a result of reading the spec is a bug that you don't have to pay QA to find, it's a bug that your customers never see, it's a bug that never causes a backwards-compatibility breaking change, and so on.

I thought hard about this and realized that the painful case only happens when we have ref/out instance.field with a side effect and instance is a variable of value type. That's the missing case. Furthermore, we can solve this by recursing!

if x is ref/out instance.field and instance is a variable of value type, then recurse. Compute T("ref instance"), as if we were invoking a method that takes a single parameter of this ref type. That will give us a resulting declaration list, which we merge onto the end of L, and a resulting argument list with one element on it, r. Add "ref r.field" to A2.
if x is a ref/out instance.field but instance is not a variable of value type, then generate var temp=instance on L, ref temp.field on A2.

So if we have, say, M(ref arr[index].q), that ultimately is generated as var t1 = arr, var t2 = index, M(ref t1[t2].q), which is correct.

I fixed up my implementation, wrote some test cases to test the spec, and sent it off to testing for further analysis.

There's still a bug in this spec, which our tester David found during his analysis of my spec plus proposed implementation. Hint: the problem has nothing to do with ref/out semantics; it's a more fundamental error in reasoning.

Feel free to pause and reflect for a moment if you want a chance to figure it out for yourself.

...

The error is in the default case:

First, if x has no side effects, then simply keep x in the appropriate place in A2.

Whether x has side effects or not is completely irrelevant; what is relevant is whether the computation of x depends on the order of every other side effect!

For example, our spec would have that

int k = 0;
M( k, arr[k=1] );

is the same as

int k = 0;
var t1 = arr;
var t2 = (k = 1);
M(k, t1[t2]);

which is clearly not correct; the first argument is supposed to be zero, not one. The fact that the first argument has no side effects is irrelevant; it depends on the second argument's side effect taking effect after the first argument's value is computed.

We have to fix up the spec (and implementation) to say that if the argument corresponds to a value parameter, you always save away the value in a temporary, regardless of whether it has side effects or not. If it corresponds to, say, a ref parameter, and the argument is a ref to a local then we can still simply generate the ref to the local normally; regardless of whether the contents of the local change, the managed address of the local is always the same, and computing the address of a local causes no side effects.

I note that we could also make an exception in the case that the value is, say, a compile-time constant. Clearly the computation of its value does not have any dependency on the ordering of other side effects. But for the sake of not further complexifying my spec and implementation, I glossed over this detail. The optimizer will probably take care of it.

Commenter Pavel Minaev found several more corner cases that none of us found. The big one is that we should be clear about what happens with multidimensional arrays, and in fact, I had a bug in my implementation for multidimensional arrays, so thanks Pavel! He also found some obscure ones. One is that accessing a static field does have a side effect; if this is the first access to the class then that causes the static class constructor to run, which could have observable side effects. Another is that the exact semantics of when bad arguments cause exceptions is slightly changed by this transformation.

At this point I now believe that I have a correct specification of my function T (modulo Pavel's weird issues, which I'm still mulling over. But those are bizarre corner cases that might require some language spec work to sort out.) Since the implementation is a straightforward transformation of the specification into C++, I also believe I have a correct implementation. I've written test cases that verify every case in the spec, and handed it off to testing to look for practical cases I missed.

Anyone see any other problems with this spec? Anyone?

And anyone have any guesses as to why I needed this function implemented?

Always write a spec, part one

Eric Lippert — Thu, 19 Nov 2009 14:45:00 GMT

Joel had a great series of articles many years ago about the benefits of writing functional specifications, that is, specifications of how the product looks to its users. I want to talk a bit about technical specifications, that is, a specification of how something actually works behind the scenes. A while back, I described how I wrote a seemingly-trivial piece of code by first writing the technical spec, and then writing the code to exactly implement the spec, and the test cases to test each statement of the spec. I use this technique frequently; it almost always saves me time and effort. If you don't write a spec first, it's very easy to get bogged down in a morass of bugs, and hard to know when you're actually done writing the code.

Here's an example of another seemingly-trivial function that smart people gave a good half-dozen completely wrong answers to because they didn't write a spec. Notice how once I state what the spec is, the code follows very naturally.

I thought I might go through a recent example of a slightly less trivial problem. I needed a function in the compiler which could take a valid list of arguments A1 to an applicable method M, and give me back two lists. The first result list is a list of local variable declarations, L, the second, another argument list, A2. Here's the kicker: evaluating each member of A2 must have no side effects. And, the combination of executing L followed by M(A2) must give exactly the same results as executing M(A1). (I also knew that the members of A1 had already had all the conversions generated to make them match the types of M's parameter list, and so on. Overload resolution was completely done at this point.)

Why I needed such a function is not particularly important; perhaps you can speculate as to why I needed it.

So I started writing a spec, starting with the description above: the function T takes a valid list... blah blah blah.

That's not enough of a spec to actually write code.

I realized that some expressions in A1 might have side effects, and some might not. If an expression had side effects, then I could declare a temp local, execute the side effects, assign the result to the local, and then use the local as the expression in the same place in A2.

Super. Now I can write more sentences in my spec.

For each argument x in A1, we potentially generate a temporary value as follows: First, if x has no side effects, then simply keep x in the appropriate place in A2. Otherwise, if x does have side effects, then generate a temporary variable declaration in L, var temp = x. Then put an expression that evaluates the temporary variable in the appropriate place in A2.

Then I thought "can I correctly implement this spec?"

I started thinking about the possible cases for the input, A1. The members of A1 could be passed by value, out, or ref; I already knew they were correct, since the particular method M was an applicable method of argument list A1. In this particular point in the compilation process, there was no need to worry about whether there were "params" arguments, whether there were missing arguments with default values, and so on; that had already been taken care of. I also noted that whether they were "ref" or "out" was irrelevant; ref and out are the same thing behind the scenes. As far as the compiler is concerned, the only difference between them is that the definite assignment rules are different for each. So we had two cases for each argument: it's being passed by value, or by ref/out.

If it's being passed by ref/out, then what we actually do is we pass the managed address of a variable to the method. In M(ref int x), the type of x is actually a special type int&, "managed reference to variable of type int", which is not a legal type in C#. That's why we hide it from you by requiring a goofy "ref" syntax any time you want to talk about a managed reference to a variable.

Unfortunately, our IL generation pass was written with the assumption that all local variables are never of these magic managed-reference-to-variable types. I was then faced with a choice. Either come up with a technique for working with this limitation, or rewrite the most complicated code in the IL gen to support this scenario. (The code which figures out how to deal with managed references is quite complex.) I decided to opt for the former strategy. Which meant more spec work, since our spec now doesn't handle these cases!

For each argument x in A1, we potentially generate a temporary value as follows:

First, if x has no side effects, then simply keep x in the appropriate place in A2.

Otherwise, if x does have side effects and corresponds to a value parameter, then generate ... etc.

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then a miracle happens.

Clearly that last bit needed some work.

So I wrote down all the possible cases I could think of. Clearly x has to be a variable if its type is "reference to variable", so that makes for a small number of cases:

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then the possible cases are:

* x is a local variable
* x is a value parameter
* x is an out parameter
* x is a ref parameter
* x is a field of an instance
* x is a static field
* x is an array element
* x is a pointer dereferenced with *
* x is a pointer dereferenced with [ ]

and for each, a miracle happens.

Again, clearly this is not good enough. I re-examined my list to see if I could eliminate any of these cases. Locals, parameters and static fields never have side effects, so we can eliminate them.

Also, at this point in our analysis, I knew that pointer dereferences of the form "pointer[index]" had already been rewritten into the form "*(pointer + index)", which meant that in practice, we would never actually hit that last case in this algorithm; the second-last case would take care of both.

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then the possible cases are:

* x is a field of an instance
* x is an array element
* x is a pointer dereferenced with *

and for each, a miracle happens.

I then started to think of what side effects could happen for each. We could have "ref instance.field", "ref array[index]", or "ref *pointer", and "instance", "array", "index" and "pointer" can all be expressions that have a side effect. ("field" cannot, it merely names a field.) So now we can use the same specification as before:

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then the possible cases are:

x is ref/out instance.field: in this case, add var temp=instance to L and ref/out temp.field to A2.
x is ref/out array[index]: in this case, add var t1 = array and var t2 = index to L and ref/out t1[t2] to A2.
x is ref/out *pointer: in this case, add var temp = pointer to L and ref/out *temp to A2.

And now we have something I can implement. So I sent this proposed spec around for review, while I started plugging away at the implementation.

This spec is wrong. Can you spot the bugs in my implementation that my coworkers found by reading my spec?

Next time, the thrilling conclusion.

Closing over the loop variable, part two

Eric Lippert — Mon, 16 Nov 2009 16:18:00 GMT

Thanks to everyone who left thoughtful and insightful comments on last week's post.

More countries really ought to implement Instant Runoff Voting; it would certainly appeal to the geek crowd. Many people left complex opinions of the form "I'd prefer to make the change, but if you can't do that then make it a warning". Or "don't make the change, do make it a warning", and so on. But what I can deduce from reading the comments is that there is a general lack of consensus on what the right thing to do here is. In fact, I just did a quick tally:

Commenters who expressed support for a warning: 26
Commenters who expressed the sentiment "it's better to not make the change": 24
Commenters who expressed the sentiment "it's better to make the change": 25

Wow. I guess we'll flip a coin. :-) (*)

Four people suggested to actually make it an error to do this. That's a pretty big breaking change, particularly since we would be breaking not just "already broken" code, but plenty of code that works perfectly well today -- see below. That's not likely to happen.

People also left a number of interesting suggestions. I thought I'd discuss some of those a little bit.

First off, I want to emphasize that what we're attempting to address here is the problem that the language encourages people to write code that has different semantics than they think it has. The problem is NOT that the language has no way to express the desired semantics; clearly it does. Just introduce a new variable explicitly into the loop.

A number of suggestions were for ways that the language could more elegantly express that notion. Some of the suggestions:

foreach(var x in c) inner
foreachnew(var x in c)
foreach(new var x in c)
foreach(var x from c)
foreach(var x inside c)

Though we could do any of those, none of them by themselves solve the problem at hand. Today, you have to know to use a particular pattern with foreach to get the semantics you want: declare a variable inside the loop. With one of these changes, you still have to know to use a particular keyword to get the semantics you want, and it is still easy to accidentally do the wrong thing.

Furthermore, a change so small and so targetted at such a narrow scenario probably does not provide enough benefit to justify the large cost of creating a new syntax, particularly one which is still easily confused with an existing syntax.

C++ luminary Herb Sutter happened to be in town and was kind enough to stop by my office to describe to me how they are solving a related problem in C++. Apparently the next version of the C++ standard will include lambdas, and they're doing this:

[q, &r] (int x) -> int { return M(x, q, r); }

This means that the lambda captures outer variable q by value, captures r by reference, takes an int and returns an int. Whether the lambda captures values or references is controllable! An interesting approach but one that doesn't immediately solve our problem here; we cannot make lambdas capture by value by default without a huge breaking change. Capturing by value would have to require new syntax, and then we're in the same boat again: the user has to know to use the new syntax when in a foreach loop.

A number of people also asked what the down sides of adding a warning are. The down side is that a warning which warns about correct behaviour is a very bad warning; it makes people change working code, and frequently they break working code in order to eliminate a warning that shouldn't have been present in the first place. Consider:

foreach(var insect in insects)
{
var query = frogs.Where(frog=>frog.Eats(insect));
Console.WriteLine("{0} is eaten by {1} frogs.", insect, query.Count());
}

This makes a lambda closed over insect; the lambda never escapes the loop, so there's no problem here. But the compiler doesn't know that. The compiler sees that the lambda is being passed to a method called Where, and Where is allowed to do anything with that delegate, including storing it away to be called later. Which is exactly what Where does! Where stores away the lambda into a monad that represents the execution of the query. The fact that the query object doesn't survive the loop is what keeps this safe. But how is the compiler supposed to suss out that tortuous chain of reasoning? We'd have to give a warning for this case, even though it is perfectly safe.

It gets worse. A lot of people are required by their organizations to compile with "warnings are errors" turned on. Therefore, any time we introduce a new warning for a pattern that is often actually safe and frequently used, we are effectively causing an enormous breaking change. A vaccine which kills more healthy people than the disease would have is probably not a good bet. (**)

This is not to say that a warning is a bad idea, but that it is not the obvious slam dunk good idea that it initially appears to be.

A number of people suggested that the problem was in the training of the developers, not in the design of the language. I disagree. Obviously modern languages are complex tools that require training to use, but we are working hard to make a language where people's natural intuitions about how things work lead them to write correct code. I have myself made this error a number of times, usually in the form of writing code like the code above, and then refactoring it in such a manner that suddenly some part of it escapes the loop and the bug is introduced. It is very easy to make this mistake, even for experienced developers who thoroughly understand closure semantics. That's a flaw in the design of the language.

And finally, a number of people made suggestions of the form "make it a warning in C# 4, and an error in C# 5", or some such thing. FYI, C# 4 is DONE. We are only making a few last-minute "user is electrocuted"-grade bug fixes, mostly based on your excellent feedback from the betas. (If you have bug reports from the beta, please keep sending them, but odds are good they won't get fixed for the initial release.) We are certainly not capable of introducing any sort of major design change or new feature at this point. And we try to not introduce semantic changes or new features in service packs. We're going to have to live with this problem for at least another cycle, unfortunately.

********

(*) Mr. Smiley Face indicates that Eric is indulging in humourous japery.

(**) I wish to emphasize that I am 100% in favour of vaccinations for deadly infectious diseases, even vaccines that are potentially dangerous. The number of people made ill or killed by the smallpox vaccine was tiny compared to the number of people who did not contract this deadly, contagious (and now effectively extinct) disease as a result of mass vaccination. I am a strong supporter of vaccine research. I'm just making an analogy here people.

Closing over the loop variable considered harmful

Eric Lippert — Thu, 12 Nov 2009 14:50:00 GMT

I don't know why I haven't blogged about this one before; this is the single most common incorrect bug report we get. That is, someone thinks they have found a bug in the compiler, but in fact the compiler is correct and their code is wrong. That's a terrible situation for everyone; we very much wish to design a language which does not have "gotcha" features like this.

But I'm getting ahead of myself. What's the output of this fragment?

var values = new List<int>() { 100, 110, 120 };
var funcs = new List<Func<int>>();
foreach(var v in values)
funcs.Add( ()=>v );
foreach(var f in funcs)
Console.WriteLine(f());

Most people expect it to be 100 / 110 / 120. It is in fact 120 / 120 / 120. Why?

Because ()=>v means "return the current value of variable v", not "return the value v was back when the delegate was created". Closures close over variables, not over values. And when the methods run, clearly the last value that was assigned to v was 120, so it still has that value.

This is very confusing. The correct way to write the code is:

foreach(var v in values)
{
var v2 = v;
funcs.Add( ()=>v2 );
}

Now what happens? Every time we re-start the loop body, we logically create a fresh new variable v2. Each closure is closed over a different v2, which is only assigned to once, so it always keeps the correct value.

Basically, the problem arises because we specify that the foreach loop is a syntactic sugar for

{
 IEnumerator<int> e = ((IEnumerable<int>)values).GetEnumerator();
 try
 {
 int m; // OUTSIDE THE ACTUAL LOOP
 while(e.MoveNext())
 {
 m = (int)(int)e.Current;
 funcs.Add(()=>m);
 }
 }
 finally
 {
 if (e != null) ((IDisposable)e).Dispose();
 }
}

If we specified that the expansion was

    try
    {
      while(e.MoveNext())
      {
      int m; // INSIDE
        m = (int)(int)e.Current;
        funcs.Add(()=>m);
      }

then the code would behave as expected.

It's compelling to consider fixing this for a hypothetical future version of C#, and I'd like to hear your feedback on whether we should do so or not. The reasons FOR making the change are clear; this is a big confusing "gotcha" that real people constantly run into, and LINQ, unfortunately, only makes it worse, because it is likely to increase the number of times a customer is going to use a closure in a loop. Also, it seems reasonable that the user of the foreach loop might think of there being a "fresh" loop variable every time, not just a fresh value in the same old variable. Since the foreach loop variable is not mutable by user code, this reinforces the idea that it is a succession of values, one per loop iteration, and not "really" the same variable over and over again. And finally, the change has no effect whatsoever on non-closure semantics. (In fact, in C# 1 the spec was not clear about whether the loop variable went inside or outside, since in a world without closures, it makes no difference.)

But that said, there are some very good reasons for not making this change.

The first reason is that obviously this would be a breaking change, and we hates them, my precious. Any developers who depend on this feature, who require the closed-over variable to contain the last value of the loop variable, would be broken. I can only hope that the number of such people is vanishingly small; this is a strange thing to depend on. Most of the time, people do not expect or depend on this behaviour.

Second, it makes the foreach syntax lexically inconsistent. Consider foreach(int x in M()) The header of the loop has two parts, a declaration int x and a collection expression, M(). The int x is to the left of the M(). Clearly the M() is not inside the body of the loop; that thing only executes once, before the loop starts. So why should something to the collection expression's left be inside the loop? This seems inconsistent with our general rule that stuff to the left logically "happens before" stuff to the right. The declaration is lexically NOT in the body of the loop, so why should we treat it as though it were?

Third, it would make the "foreach" semantics inconsistent with "for" semantics. We have this same problem in "for" blocks, but "for" blocks are much looser about what "the loop variable" is; there can be more than one variable declared in the for loop header, it can be incremented in odd ways, and it seems implausible that people would consider each iteration of the "for" loop to contain a fresh crop of variables. When you say for(int i; i < 10; i += 1) it seems dead obvious that the "i += 1" means "increment the loop variable" and that there is one loop variable for the whole loop, not a new fresh variable "i" every time through! We certainly would not make this proposed change apply to "for" loops.

And fourth, though this is a nasty gotcha, there is an easy workaround, and tools like ReSharper detect this pattern and suggest how to fix it. We could take a page from that playbook and simply issue a compiler warning on this pattern. (Though adding new warnings brings up a whole raft of issues of its own, which I might get into in another post.) Though this is vexing, it really doesn't bite that many people that hard, and it's not a big deal to fix, so why go to the trouble and expense of taking a breaking change for something with an easy fix?

Design is, of course, the art of compromise in the face of many competing principles. "Eliminate gotchas" in this case directly opposes other principles like "no breaking changes", and "be consistent with other language features". Any thoughts you have on pros or cons of us taking this breaking change in a hypothetical future version of C# would be greatly appreciated.

Three Umpires

Eric Lippert — Mon, 09 Nov 2009 15:01:00 GMT

Three baseball umpires are having lunch together. The first umpire says "Well, a lot of them are balls, and a lot of them are strikes, but I always calls 'em as I sees 'em."

The second umpire says "Hmph. I calls 'em as they are."

The third umpire slowly looks at his two colleagues and declares "They ain't nothin' until I calls 'em."

Those of you unfamiliar with the bizarre rules of baseball might need a brief primer. Suppose the pitcher throws a pitch and the batter swings and misses. Such a failure to strike the ball is, bizarrely enough, called a "strike", and counts against the batter; three strikes and the batter is out. But what if the batter fails to swing at all? If the umpire decides that the pitch was "inside the strike zone" then the pitch counts as a strike against the batter. If the pitch was outside the strike zone then the pitch counts as a "ball" (another bizarre and confusing name; obviously the object pitched is also called a ball). If the pitcher pitches four "balls" before the batter accumulates three "strikes" then the batter gets to "walk" to first base for free. (At least the walk is sensibly named.)

Formally, the strike zone is reasonably well-defined by the rules (though as the Wikipedia article linked to above indicates, there are some subtle points left out of the definition.) But the formal definition is actually irrelevant; the rules of baseball also state that a strike is any pitch that the umpire says is a strike. Umpires are given wide lattitude to declare what is a strike, and there are no appeals allowed.

And hence the fundamental disagreement between the three umpires. The first umpire believes that whether a pitch was in the strike zone or not is a fact about an objective reality, and that the call is a sometimes-imperfect subjective judgment about that reality. The second umpire seems to be basically agreeing with the objective, materialist stance of the first umpire, and simply bragging about having 100% accuracy in judgment. The third umpire's position is radically different from the first two: that the rules of baseball say that regardless of the objective reality of the path of the baseball, what makes a pitch into a ball or a strike is the umpire's call, no more, no less.

I think about the three umpires a lot. The C# language has a clear and mostly unambiguous definition of "the strike zone"; the specification should in theory allow us to classify any finite set of finite strings of text into either "a legal C# program" or "not a legal C# program". As a spec author, I take the third umpire's position: what the spec says is the definition of what is legal, end of story. But as an all-too fallible compiler writer, I take the first umpire's position: the compiler calls 'em as it sees 'em. Sometimes an illegal program accidentally (or deliberately; we implement a small number of extensions to the formal C# language) makes it through the compiler. And sometimes a legal program is incorrectly flagged as an error, or cannot be successfully compiled because it causes the compiler to run out of stack space or some other resource. (Also, though I believe that the compiler always in theory terminates, there are ways to build short C# programs that take exponentially long to analyze, making the compiler a sometimes impractical tool for deciding correctness.) But we calls 'em as we sees 'em, and if we get it wrong, then that's a bug.

But as a practical matter for our customers, the compiler is more like the third umpire: the arbiter of correctness, with no appeal. And of course I haven't even begun to consider the runtime aspects of correctness! Not only should the compiler decide what programs are legal, it should also generate correct code for every legal program. And again, the code generator plus the CLR's verifier and jitter act like our third umpire; the de facto arbiter of what "the right behaviour" actually is.

Simple names are not so simple, Part Two, plus, volcanoes and fried foods

Eric Lippert — Thu, 05 Nov 2009 14:52:00 GMT

I've returned from a brief vacation, visiting friends on the island of Maui. I'd never been to that part of the world before. Turns out, it's a small island in the middle of the Pacific Ocean, entirely made out of volcanoes. Weird! But delightful.

The most impressive thing about the Hawaiian Islands for me was just how obvious were -- even to my completely untrained eyes -- the geomechanical and fluvial processes which shaped the landscape. The mountains and craters and river valleys and red sand beaches and easily-fractured rocks were very different from the (also somewhat volcanic) much older mountainous landscape I've lived in for the past decade.

Also quite amusing to me was learning to read and pronounce Hawaiian place names. It is all very logical once you know the system; before long I could easily pronounce signs like WAINAPANAPA STATE PARK -- wa-ee-napa-napa -- or PUUNENE AVENUE -- pu-oo-nay-nay -- or MAILIBEHANAMONOTANA STREET -- "Miley-Stewart-is-really-Hannah-Montana".

Many thanks to K and R and D for putting me and Leah up for a week; if you're going to Hawai'i and can stay with locals, I highly recommend it, particularly if they are awesome people. Everyone in Maui was awesome, with the exception of the rangers at (stunningly beautiful, even for Maui) Wa'inapanapa State Park, who are apparently consistently grumpy. As one Hawaiian, himself in the camping sector of the economy put it to me, "They do not have the big aloha".

The most amusing encounter was on the Hana Highway. There are numerous little stops along the way, where someone has erected a hut or parked a trailer and is selling coconuts, smoothies, banana bread, and so on. Hand-lettered signs, stunning natural beauty, middle of nowhere, you get the picture I'm sure. At one of the larger such stops there was a young fellow, probably in his late twenties, serving a variety of fried foods. It was mostly traditional American-style Chinese food, but also he had french fries, fish'n'chips, and so on. He was clearly not a native speaker of English, but spoke understandably with a strong accent. We were waiting behind a middled-aged woman with a typically midwestern American accent. Their conversation went something like this:

Her: I'm not very hungry, can I just get the fish without the chips?

Him, not quite following her: Half order?

Her, louder: How much without the fries?

This went back and forth for some time, both sides becoming increasingly frustrated by the communication breakdown, until:

Her, even louder: Can I speak to your manager?

Leah and K and I silently boggled -- there is no other word for it -- at each other for a moment. Where on earth did she imagine that a manager was going to emerge from? There was a counter, behind that, a trailer with a wok in it, behind that, jungle, and behind that, huge jagged lava rocks followed immediately by the Pacific Ocean. And what sort of management structure does she think one really needs to manage a single guy selling pineapple fried rice at the side of a highway? My conclusion: people have strange beliefs. Sometimes their beliefs cause them to leave in a huff with neither fish nor chips, even when fish and chips are both plentiful and reasonably priced. Hopefully she had better luck in Hana.

Anyway, enough travelogue. Regarding the puzzle from last time: the code is correct, and compiles without issue. I was quite surprised when I first learned that; it certainly looks like it violates our rule about not using the same simple name to mean two different things in one block.

The key is to understanding why this is legal is that the query comprehensions and foreach loops are specified as syntactic sugars for another program, and it is that program which is actually analyzed for correctness. Our original program:

static void Main()
{
int[] data = { 1, 2, 3, 1, 2, 1 };
foreach (var m in from m in data orderby m select m)
System.Console.Write(m);
}

is transformed into

static void Main()
{
int[] data = { 1, 2, 3, 1, 2, 1 };
{
 IEnumerator<int> e = ((IEnumerable<int>)(data.OrderBy(m=>m)).GetEnumerator();
 try
 {
 int m;
 while(e.MoveNext())
 {
 m = (int)(int)e.Current;
 Console.Write(m);
 }
 }
 finally
 {
 if (e != null) ((IDisposable)e).Dispose();
 }
}
}

There are five usages of m in this transformed program; it is:

1) declared as the formal parameter of a lambda.
2) used in the body of the lambda; here it refers to the formal parameter.
3) declared as a local variable
4) written to in the loop; here it refers to the local variable
5) read from in the loop; here it refers to the local variable

Is there any usage of a local variable before its declaration? No.

Are there any two declarations that have the same name in the same declaration space? It would appear so. The body of Main defines a local variable declaration space, and clearly the body of Main contains, indirectly, two declarations for m, one as a formal lambda parameter and one as a local. But I said last time that local variable declaration spaces have special rules for determining overlaps. It is illegal for a local variable declaration space to directly contain a declaration such that another nested local variable declaration space contains a declaration of the same name. But an outer declaration space which indirectly contains two such declarations is not an error. So in this case, no, there are no local variable declarations spaces which directly contain a declaration for m, such that a nested local variable declaration space also directly contains a declaration for m. Our two local variable declarations spaces which directly contain a declaration for m do not overlap anywhere.

Is there any declaration space which contains two inconsistent usages of the simple name m? Yes, again, the outer block of Main contains two inconsistent usages of m. But again, this is not relevant. The question is whether any declaration space directly containing m has an inconsistent usage. Again, we have two declaration spaces but they do not overlap each other, so there's no problem here either.

The thing which makes this legal, interestingly enough, is the generation of the loop variable declaration logically within the try block. Were it to be generated outside the try block then this would be a violation of the rule about inconsistent usage of a simple name throughout a declaration space.

Simple names are not so simple

Eric Lippert — Mon, 02 Nov 2009 14:23:00 GMT

C# has many rules that are designed to prevent some common sources of bugs and encourage good programming practices. So many, in fact, that it is often quite confusing to sort out exactly which rule has been violated. I thought I might spend some time talking about what the different rules are. We'll finish up with a puzzle.

To begin with, it will be vital to understand the difference between scope and declaration space. To refresh your memory of my earlier article: the scope of an entity is the region of text in which that entity may be referred to by its unqualified name. A declaration space is a region of text in which no two things may have the same name (with an exception for methods which differ by signature.) A "local variable declaration space" is a particular kind of declaration space used for declaring local variables; local variable declaration spaces have special rules for determining when they overlap.

The next thing that you have to understand to make any sense o this is what a "simple name" is. A simple name is always either just a plain identifier, like "x", or, in some cases, a plain identifier followed by a type argument list, like "Frob<int, string>".

Lots of things are treated as "simple names" by the compiler: local variable declarations, lambda parameters, and so on, always have the first form of simple name in their declarations. When you say "Console.WriteLine(x);" the "Console" and "x" are simple names but the "WriteLine" is not. Confusingly, there are some textual entities which have the form of simple names, but are not treated as simple names by the compiler. We might talk about some of those situations in later fabulous adventures.

So, without further ado, here are some relevant rules which are frequently confused. It's rules 3 and 4 that people find particularly confusing.

1) It is illegal to refer to a local variable before its declaration. (This seems reasonable I hope.)
2) It is illegal to have two local variables of the same name in the same local variable declaration space or nested local variable declaration spaces.
3) Local variables are in scope throughout the entire block in which the declaration occurs. This is in contrast with C++, in which local variables are in scope in their block only at points after the declaration.
4) For every occurrence of a simple name, whether in a declaration or as part of an expression, all uses of that simple name within the immediately enclosing local variable declaration space must refer to the same entity.

The purpose of all of these rules is to prevent the class of bugs in which the reader/maintainer of the code is tricked into believing they are referring to one entity with a simple name, but are in fact accidentally referring to another entity entirely. These rules are in particular designed to prevent nasty surprises when performing what ought to be safe refactorings.

Consider a world in which we did not have rules 3 and 4. In that world, this code would be legal:

class C
{
int x;
void M()
{
    // 100 lines of code
    x = 20; // means "this.x";
    Console.WriteLine(x); // means "this.x"
    // 100 lines of code
    int x = 10;
    Console.WriteLine(x); // means "local x"
}
}

This is hard on the person reading the code, who has a reasonable expectation that the two "Console.WriteLine(x)" lines do in fact both print out the contents of the same variable. But it is particularly nasty for the maintenance programmer who wishes to impose a reasonable coding standard upon this body of code. "Local variables are declared at the top of the block where they're used" is a reasonable coding standard in a lot of shops. But changing the code to:

class C
{
int x;
void M()
{
    int x;
    // 100 lines of code
    x = 20; // no longer means "this.x";
    Console.WriteLine(x); // no longer means "this.x"
    // 100 lines of code
    x = 10;
    Console.WriteLine(x); // means "local x"
}
}

changes the meaning of the code! We wish to discourage authoring of multi-hundred-line methods, but making it harder and more error-prone to refactor them into something cleaner is not a good way to achieve that goal.

Notice that the original version of this program, rule 3 means that the program violates rule 1 -- the first usage of "x" is treated as a reference to the local before it is declared. The fact that it violates rule 1 because of rule 3 is precisely what prevents it from being a violation of rule 4! The meaning of "x" is consistent throughout the block; it always means the local, and therefore is sometimes used before it is declared. If we scrapped rule 3 then this would be a violation of rule 4, because we would then have two inconsistent meanings for the simple name "x" within one block.

Now, these rules do not mean that you can refactor willy-nilly. We can still construct situations in which similar refactorings fail. For example:

class C
{
int x;
void M()
{
    {
      // 100 lines of code
      x = 20; // means "this.x";
      Console.WriteLine(x); // means "this.x"
    }
    {
      // 100 lines of code
      int x = 10;
      Console.WriteLine(x); // means "local x"
    }
}
}

This is perfectly legal. We have the same simple name being used two different ways in two different blocks, but the immediately enclosing block of each usage does not overlap that of any other usage. The local variable is in scope throughout its immediately enclosing block, but that block does not overlap the block above. In this case, it is safe to refactor the declaration of the local to the top of its block, but not safe to refactor the declaration to the top of the outermost block; that would change the meaning of "x" in the first block. Moving a declaration up is almost always a safe thing to do; moving it out is not necessarily safe.

Now that you know all that, here's a puzzle for you, a puzzle that I got completely wrong the first time I saw it:

using System.Linq;
class Program
{
static void Main()
{
    int[] data = { 1, 2, 3, 1, 2, 1 };
    foreach (var m in from m in data orderby m select m)
      System.Console.Write(m);
}
}

It certainly looks like name "m" is being used multiple times to mean different things. Is this program legal? If yes, why do the rules for not re-using simple names not apply? If no, precisely what rule has been violated?

[Eric is on vacation; this posting was pre-recorded.]

I have a Fit, but a lack of Focus.

Eric Lippert — Thu, 29 Oct 2009 13:54:00 GMT

Here's a statement I read the other day about making comparisons between objects of reference type in C#:

Object.ReferenceEquals(x,y) returns true if and only if x and y refer to the same object.

True or false?

My wife Leah recently acquired a Honda Fit, thanks to the imminant failure of the automatic transmission solenoids in her aged Honda Civic. The back seats in the Fit fold down flat. You can fit a llama or a whole pile of hula hoops or whatever into that thing. It's quite handy. Not what I would call a powerful engine by any means, but for quick trips around town, it certainly gets the job done.

Since we were married when she bought the car, and we continue to be married, what's mine is hers and what's hers is mine. So if x = Eric's Honda Fit, and y = Leah's Honda Fit, then x and y are "reference equals". Those two things refer to the same object, viz, the shiny black object full of llamas and hula hoops in my driveway.

Now, we could have bought a different car. Say, a Ford Focus. But we did not. We own a total of zero Ford Foci. Suppose I said that x = Eric's Ford Focus, and y = Leah's Ford Focus. What's the sensible way to characterize the nature of x and y? Do we say that x and y refer to the same Ford Focus, namely that they refer to the Ford Focus that does not exist? The mind boggles at the repugnant and paradoxical implication that there exists a Ford Focus that is the Ford Focus that does not exist! (*) Rather, the right way to characterize this is to say that neither x nor y refer to any object. They're "null references" -- references that do not have any referent, but rather, capture the notion of "a lack of referent".

And that's why it's incorrect to say that Object.ReferenceEquals(x,y) returns true if and only if x and y refer to the same object.If x and y both do not refer to any object, then clearly they do not refer to the same object, because neither refers to an object in the first place. The correct way to characterize the behaviour of reference equality is

Object.ReferenceEquals(x,y) returns true if and only if either x and y refer to the same object, or x and y are both null references.

***********

(*) And yet I am a fan of the "null object pattern". Life is just full of these little contradictions.

[Eric is on vacation this week; this posting is pre-recorded.]

Some new videos

Eric Lippert — Mon, 26 Oct 2009 13:17:00 GMT

Somehow it has happened again; people just keep on recording videos of me and putting them on the internet.

In these videos you find out what I look like when lit from above and behind. Kinda spooky. We should have made the room entirely dark and held a flashlight underneath my face. That would be, like, ten times scarier. Anyway, if you're interested in me blathering on about my favourite feature in C# 4, covariance and contravariance of interface and delegate types, here are two little demo videos: Part One, Part Two. (There seems to be some minor sound sync issues here and there, but it's not really a problem; most of the audio is voice-over.)

Charlie has been crazy busy getting these little videos together; here are some more of his recent efforts, including some good ones from my colleagues Chris and Sam talking about all the other far more awesome features of C# 4.0: dynamic interop, improved interop with Office, named and optional parameters, and so on. Links to all of our recent videos are here: http://blogs.msdn.com/charlie/archive/2009/10/19/community-convergence-lvi.aspx.

Have an amusing and safe Hallowe'en -- I'll be going to Hallowe'en parties on a small island this year, just for a change of pace.

[Eric is on vacation this week; this posting was pre-recorded]

What is this thing you call "thread safe"?

Eric Lippert — Mon, 19 Oct 2009 13:50:00 GMT

Caveat: I am not an expert on multi-threading programming. In fact, I wouldn't even say that I am competent at it. My whole career, I've needed to write code to spin up a secondary worker thread probably less than half a dozen times. So take everything I say on the subject with some skepticism.

A question I'm frequently asked: "is this code thread safe?" To answer the question, clearly we need to know what "thread safe" means.

But before we get into that, there's something I want to clear up first. A question I am less frequently asked is "Eric, why does Michelle Pfeiffer always look so good in photographs?" To help answer this pressing question, I consulted Wikipedia:

"A photogenic subject is a subject that usually appears physically attractive or striking in photographs."

Why does Michelle Pfeiffer always look so good in photographs? Because she's photogenic. Obviously.

Well, I'm glad we've cleared up that mystery, but I seem to have wandered somehwat from the subject at hand. Wikipedia is just as helpful in defining thread safety:

"A piece of code is thread-safe if it functions correctly during simultaneous execution by multiple threads."

As with photogenicity, this is obvious question-begging. When we ask "is this code thread safe?" all we are really asking is "is this code correct when called in a particular manner?" So how do we determine if the code is correct? We haven't actually explained anything here.

Wikipedia goes on:

"In particular, it must satisfy the need for multiple threads to access the same shared data, ..."

This seems fair; this scenario is almost always what people mean when they talk about thread safety. But then:

"...and the need for a shared piece of data to be accessed by only one thread at any given time."

Now we're talking about techniques for creating thread safety, not defining what thread safety means. Locking data so that it can only be accessed by one thread at a time is just one possible technique for creating thread safety; it is not itself the definition of thread safety.

My point is not that the definition is wrong; as informal definitions of thread safety go, this one is not terrible. Rather, my point is that the definition indicates that the concept itself is completely vague and essentially means nothing more than "behaves correctly in some situations". Therefore, when I'm asked "is this code thread safe?" I always have to push back and ask "what are the exact threading scenarios you are concerned about?" and "exactly what is correct behaviour of the object in every one of those scenarios?"

Communication problems arise when people with different answers to those questions try to communicate about thread safety. For example, suppose I told you that I have a "threadsafe mutable queue" that you can use in your program. You then cheerfully write the following code that runs on one thread while another thread is busy adding and removing items from the mutable queue:

if (!queue.IsEmpty) Console.WriteLine(queue.Peek());

Your code then crashes when the Peek throws a QueueEmptyException. What is going on here? I said this thing was thread safe, and yet your code is crashing in a multi-threaded scenario.

When I said "the queue is threadsafe" I meant that the queue maintains its internal state consistently no matter what sequence of individual operations are happening on other threads. But I did not mean that you can use my queue in any scenario that requires logical consistency maintained across multiple operations in a sequence. In short, my opinion of "correct behaviour" and your opinion of the same differed because what we thought of as the relevant scenario was completely different. I care only about not crashing, but you care about being able to reason logically about the information returned from each method call.

In this example, you and I are probably talking about different kinds of thread safety. Thread safety of mutable data structures is usually all about ensuring that the operations on the shared data always operate on the most up-to-date state of the shared data as it mutates, even if that means that a particular combination of operations appears to be logically inconsistent, as in our example above. Thread safety of immutable data structures is all about ensuring that use of the data across all operations is logically consistent, at the expense of the fact that you're looking at an immutable snapshot that might be out-of-date.

The problem here is that the choice about whether to access the first element or not is based on "stale" data. Designing a truly thread-safe mutable data structure in a world where nothing is allowed to be stale can be very difficult. Consider what you'd have to do in order to make the "Peek" operation above actually threadsafe. You'd need a new method:

if (!queue.Peek(out first)) Console.WriteLine(first);

Is this "thread safe"? It certainly seems better. But what if after the Peek, a different thread dequeues the queue? Now you're not crashing, but you've changed the behaviour of the previous program considerably. In the previous program, if, after the test there was a dequeue on another thread that changed what the first element was, then you'd either crash or print out the up-to-date first element in the queue. Now you're printing out a stale first element. Is that correct? Not if we always want to operate on up-to-date data!

But wait a moment -- actually, the previous version of the code had this problem as well. What if the dequeue on the other thread happened after the call to Peek succeeded but before the Console.WriteLine call executed? Again, you could be printing out stale data.

What if you want to ensure that you are always printing out up-to-date data? What you really need to make this threadsafe is:

queue.DoSomethingToHead(first=>{Console.WriteLine(first);});

Now the queue author and the queue user agree on what the relevant scenarios are, so this is truly threadsafe. Right?

Except... there could be something super-complicated in that delegate. What if whatever is in the delegate happens to cause an event that triggers code to run on another thread, which in turn causes some queue operation to run, which in turn blocks in such a manner that we've produced a deadlock? Is a deadlock "correct behaviour"? And if not, is this method truly "safe"?

Yuck.

By now you take my point I'm sure. As I pointed out earlier, it is unhelpful to say that a building or a hunk of code is "secure" without somehow communicating which threats the utilized security mechanism are and are not proof against. Similarly, it is unhelpful to say that code is "thread safe" without somehow communicating what undesirable behaviors the utilized thread safety mechanisms do and do not prevent. "Thread safety" is nothing more nor less than a code contract, like any other code contract. You agree to talk to an object in a particular manner, and it agrees to give you correct results if you do so; working out exactly what that manner is, and what the correct responses are, is a potentially tough problem.

************

(*) Yes, I'm aware that if I think something on Wikipedia is wrong, I can change it. There are two reasons why I should not do so. First, as I've already stated I'm not an expert in this area; I leave it to the experts to sort out amongst themselves what the right thing to say here is. And second, my point is not that the Wikipedia page is wrong, but rather that it illustrates that the term itself is vague by nature.

As Timeless As Infinity

Eric Lippert — Thu, 15 Oct 2009 13:25:00 GMT

User: Recently I found out about a peculiar behaviour concerning division by zero in floating point numbers in C#. It does not throw an exception, as with integer division, but rather returns an "infinity". Why is that?

Eric: As I've often said, "why" questions are difficult for me to answer. My first attempt at an answer to a "why" question is usually "because that's what the specification says to do"; this time is no different. The C# specification says to do that in section 4.1.6. But we're only doing that because that's what the IEEE standard for floating point arithmetic says to do. We wish to be compliant with the established industry standard. See IEEE standard 754-1985 for details. Most floating point arithmetic is done in hardware these days, and most hardware is compliant with this specification.

User: It seems to me that division by zero is a bug no matter how you look at it!

Eric: Well, since clearly that is not how the members of the IEEE standardization committee looked at it in 1985, your statement that it must be a bug "no matter how you look at it" must be incorrect. Some industry experts do not look at it that way.

User: Good point. What motivated this design decision?

Eric: I wasn't there; I was busy playing Jumpman on my Commodore 64 at the time. But my educated guess is that it is desirable for all possible operations on all floats to produce a well-defined float result. Mathematicians would call this a "closure" property; that is, the set of floating point numbers is "closed" over all operations.

Positive infinity seems like a reasonable choice for dividing a positive number by zero. It seems plausible because of course the limit of 1 / x as x goes to zero (from above) is "positive infinity", so why shouldn't 1/0 be the number "positive infinity"?

Now, speaking as a mathematician, I find that argument specious. A thing and its limit need not have any particular property in common; it is fallacious to reason that just because, say, a sequence has a particular limit that a fact about the limit is also a fact about the sequence. Mathematically, "positive infinity" (in the sense of a limit of a real-valued function; let's leave transfinite ordinals, hyperbolic geometry, and all of that other stuff out of this discussion) is not a number at all and should not be treated as one; rather, it's a terse way of saying "the limit does not exist because the sequence diverges upwards".

When we divide by zero, essentially what we are saying is "solve the equation x * 0 = 1"; the solution to that equation is not "positive infinity", it is "I cannot because there is no solution to that equation". It's just the same as asking to solve the equation "x + 1 = x" -- saying "x is positive infinity" is not a solution; there is no solution.

But speaking as a practical engineer who uses floating point numbers to do an imprecise approximation of ideal arithmetic, this seems like a perfectly reasonable choice.

User: But surely it is impossible for the hardware to represent "infinity".

Eric: It certainly is possible. You've got 32 bits in a single-precision float; that's over four billion possible floats. All bit patterns of the form

?11111111???????????????????????

are reserved for "not-a-number" values. That's over sixteen million possible NaN combinations. Two of those sixteen million NaN bit patterns are reserved to mean positive and negative infinity. Positive infinity is the bit pattern 01111111100000000000000000000000 and negative infinity is 11111111100000000000000000000000.

User: Do all languages and applications use this convention of division-by-zero-becomes-infinity?

Eric: No. For example, C# and JScript do but VBScript does not. VBScript gives an error if you do that.

User: Then how do language implementors get the desired behaviour for each language if these semantics are implemented by the hardware?

Eric: There are two basic techniques. First, many chips which implement this standard allow the programmer to make float division by zero an exception rather than an infinity. On the 80x87 chip, for example, you can use bit two of the precision control register to determine whether division by zero returns an infinity or throws a hardware exception.

Second, if you don't want it to be a hardware exception but do want it to be a software exception, then you can check bit two of the status register after each division; it records whether there was a recent divide-by-zero event.

The latter strategy is used by VBScript; after we perform a division operation we check to see whether the status register recorded a divide-by-zero operation; if it did, then the VBScript runtime creates a divide-by-zero error and the usual VBScript error management process takes over, same as any other error.

Similar bits exist for other operations that seem like they might be better treated as exceptions, like numeric overflow.

The existence of the "hardware exception" bits creates problems for the modern language implementor, because we are now often in a world where code written in multiple languages from multiple vendors is running in the same process. Control bits on hardware are the ultimate "global state", and we all know how irksome it is to have global, public state that random code can stomp on.

For example: I might be misremembering some details, but I seem to recall that Delphi-authored controls set the "overflows cause exceptions" bit. That is, the Delphi implementors did not use the VBScript strategy of "try it, allow it to succeed, and check to see whether the overflow bit was set in the status register". Rather, they used the "make the hardware throw an exception and then catch the exception" strategy. This is deeply unfortunate. When a VBScript script calls a Delphi-authored control, the control flips the bit to force exceptions but it never "unflips" it. If, later on in the script, the VBScript program does an overflow, then we get an unhandled hardware exception because the bit is still set, even though the Delphi control might be long gone! I fixed that by saving away the state of the control register before calling into a component and restoring it when control returns. That's not ideal, but there's not much else we can do.

User: Very enlightening! I will be sure to pass this information along to my coworkers. I would be delighted to see a blog post on this.

Eric: And here you go!