Fabulous Adventures In Coding : Language Design

Exact rules for variance validity

Eric Lippert — Thu, 03 Dec 2009 14:32:00 GMT

I thought it might be interesting for you all to get a precise description of how exactly it is that we determine when it is legal to put "in" and "out" on a type parameter declaration in C# 4. I'm doing this here because (1) it's of general interest, and (2) our attempt to make a more human-readable version of this algorithm in the draft C# 4.0 specification accidentally introduced some subtle errors. We're working on correcting those errors for the final release of the specification; until then, these definitions are the definitions I actually worked from to do the implementation, so they're accurate.

These definitions are pretty much stolen straight from the CLI spec section on variance; I am indebted to its authors for their careful and precise definitions.

The first things we need to define are three kinds of "validity" for types. I want to define "valid covariantly", "valid contravariantly" and "valid invariantly", but only as applied to types. We'll talk about what makes an interface declaration a valid one later; we need these definitions first.

Before we get into a precise definition, I want to talk about what "valid covariantly" means logically. The idea that we are attempting to capture here boils down to "the type we're talking about is not contravariant". Whether it is covariant, we don't really care. All we care about it is that if it is valid covariantly, then it is not contravariant. Similarly, by "valid contravariantly", we mean "not covariant".

This leads us to our first brain-hurting definition: A type is "valid invariantly" if it is both valid covariantly and valid contravariantly. That sounds a bit crazy -- how can it be valid invariantly if it is both covariant and contravariant? But "valid covariantly" does not mean "is covariant", it means "it's guaranteed to not be contravariant". So if a type is valid covariantly and valid contravariantly, then it is guaranteed to be neither contravariant nor covariant, and therefore must be invariant.

Anyway. We want to take any specific type and classify it as valid covariantly, valid contravariantly, or both, a.k.a., valid invariantly. (We don't need to worry about coming up with a term for the "or none of the above" case because in practice that case is never relevant.)

A type is valid covariantly if it is:

1) a class, struct, nullable, enum, pointer, non-generic delegate or non-generic interface type.

This should make sense. Remember, what we're getting at is "not contravariant". Classes and structs are either non-generic, in which case they are clearly not contravariant, or they are generic, in which case again they are not contravariant because we do not support variance on classes or structs. Nullable types are just aliases for Nullable<T> which is a struct type, and therefore not variant. Enums and pointers are non-generic so they are not variant. And non-generic delegates and interfaces are by definition not generic and therefore not variant either. These are the easy cases.

2) An array type T[] where T is valid covariantly.

Because we have array covariance in C#, but not array contravariance, we can make arrays valid covariantly. Again, remember, "valid covariantly" means "not contravariant".

3) A generic type parameter type, if it was not declared as being contravariant.

Generic type parameter types are of course types as far as the compiler is concerned. If you're inside the declaration of a generic interface then you can use its type parameters as types. In that context, such types are valid covariantly if they were not declared as contravariant (in).

4) A constructed interface or delegate type X<T1, ... Tk> might be valid covariantly. To determine if it is, we examine each type argument differently, depending on whether the corresponding type parameter was declared as covariant (out), contravariant (in), or invariant (neither). If the ith type parameter was declared as covariant, then Ti must be valid covariantly. If it was declared as contravariant, then Ti must be valid contravariantly. If it was declared as invariant, then Ti must be valid invariantly.

As one would expect, covariant validity preserves the direction of validity; covariant parameters must be valid covariantly, contravariant parameters must be valid contravariantly.

OK, I hope that was relatively painless. The rules for contravariant validity are similar; as one would expect from the "backwards" nature of contravariance, the directions are reversed in the complicated case:

A type is valid contravariantly if it is:

1) a class, struct, nullable, enum, pointer, non-generic delegate or non-generic interface type.

2) An array type T[] where T is valid contravariantly.

Arrays are covariant in their element type. Covariance preserves the direction of variance. Therefore, to be valid contravariantly, an array (covariant in its element type) must be contravariantly valid in its element type.

3) A generic type parameter type, if it was not declared as being covariant.

Remember, "valid contravariantly" means "not covariant". It does not mean "contravariant".

4) A constructed interface or delegate type X<T1, ... Tk> might be valid contravariantly. If the ith type parameter was declared as contravariant, then Ti must be valid covariantly. If it was declared as covariant, then Ti must be valid contravariantly. If it was declared as invariant, then Ti must be valid invariantly.

As one might expect, contravariant validity reverses the direction of validity.

Now perhaps you see why we wanted to rewrite these definitions into something more human-readable for the spec. And perhaps you also see why we accidentally introduced errors in doing so; bending your brain around all this logic is not always easy.

OK, now that we've got that, we can make a definition of what it means for an interface to be valid. An interface must meet the following conditions:

* The return types of all non-void interface methods must be valid covariantly.

* Every formal parameter type of all interface methods must be valid contravariantly. (Invariantly if it is an out or ref parameter.)

* For all generic methods on an interface, every constraint on the generic method type parameters must be valid contravariantly.

* All its base interface types must be valid covariantly.

* The type of a property or indexer must be valid covariantly if it has a "getter" and valid contravariantly if it has a "setter".

* Any formal parameter types of an indexer must be valid contravariantly.

* The delegate types of all its events must be valid contravariantly.

The first two are pretty straightforward; return types "go out" so they have to be valid covariantly, formal parameter types "go in", so they have to be valid contravariantly. But what's up with the third one? What do constraints on generic method type parameters have to do with interface validity?

Well, let's suspend the third rule and see what goes wrong.

interface I<out T>
{
 void M() where U : T;
 // illegal; this has to be valid contravariantly but it is a covariant type parameter constraint.
 // Let it ride for now and demonstrate an error.
}

class C<T> : I<T> { public void M() {} }
// the constraint is inherited implicitly and not re-stated.

I<Giraffe> igiraffe = new C<Giraffe>(); // C<T> implements I<T>
I<Animal> ianimal = igiraffe; // interface is covariant in T
ianimal.M<Turtle>(); // satisifies the constraint that U must be an Animal.

Uh oh. ianimal is really an instance of C<Giraffe>. The M method on C<Giraffe> inherits a requirement that U inherit from Giraffe. Turtle does not inherit from Giraffe. Therefore we've just violated the constraint on M. The only places where we can catch this is in the declaration; every other step is perfectly legal. Therefore, a constraint cannot be covariant. But if we make it contravariant (or invariant) then it all works out. For example, let's make a contravariant type parameter constraint:

interface I2<in T> // contravariant this time
{
void M() where U : T;
}

class C2<T> : I2<T> { public void M() {} }
I2<Animal> i2animal = new C2<Animal>(); // C2<T> implements I2<T>
I2<Mammal> i2mammal = i2animal; // interface is contravariant in T
i2mammal.M<Giraffe>(); // satisifies the constraint that U must be an Animal.

And now everything is fine; the compile-time constraint checker verifies that Giraffe is Mammal; at runtime it must be Animal, and so the compiler has verified that by verifying that it is Mammal.

The rules for delegate declarations are a straightforward simplification of the rules for interface declarations. To be a valid delegate declaration, the return type must be valid covariantly (or void), the formal parameter types must be valid contravariantly (or invariantly if they are out/ref), and any type parameter constraints must be valid contravariantly.

The Purpose, Revealed

Eric Lippert — Wed, 25 Nov 2009 17:03:00 GMT

Interestingly enough, no one correctly guessed why I needed code in the compiler to transform a method argument list into a form that batched up the side effecting expressions, stuck their values in variables, and then called the method with the side-effect-free variable references. There certainly were some fascinating ideas in the comments though!

The reason is quite mundane. We had a bug in the code that performs semantic analysis of calls with named arguments. One of our rules in C# that I've talked about a lot on this blog is that sub-expression evaluation occurs in left-to-right order. If you have a method void M(int x, int y) {} and then a call site M(y : F(), x : G()); we cannot simply generate the code as though you'd written M(G(), F()). The side effects of F have to happen before the side effects of G, so we generate code as though you'd said temp1 = F(); temp2 = G(); M(y : temp1, x : temp2); and now we can generate the call as M(temp2, temp1); without worrying about re-ordering a side effect.

The bug was that the rewriter was not getting the reordering correct if the parameters were ref/out or the arguments were locals without side effects. As I often do when fixing a bug, I reasoned that the initial bug had been caused because the rewriter was under-specified.

The fix will get into the initial release of C# 4, but the problems that Pavel identified with it -- that certain obscure side effects such as the order in which bad array accesses throw exceptions, or the order in which static class constructors execute, are not correctly preserved by my algorithm, will not be fixed for the initial release. Doing these correctly requires us to be able to generate temporaries of "reference to argument" types, which we've never done before and which our IL generator was not built to handle. Had we discovered the bug much earlier in the cycle, we probably could have applied a more solid fix, but it is too late now.

Thanks again to Pavel for the great analysis, and thanks to our old friend nikov for reporting the bug in the first place.

And happy thanksgiving to my readers in the United States! As I do every year, I'm roasting a turkey to feed fourteen people. I highly recommend the "brine the bird then roast it upside-down" method described in The Joy Of Cooking. It's worked perfectly for me for the last ten years straight.

Closing over the loop variable considered harmful

Eric Lippert — Thu, 12 Nov 2009 14:50:00 GMT

I don't know why I haven't blogged about this one before; this is the single most common incorrect bug report we get. That is, someone thinks they have found a bug in the compiler, but in fact the compiler is correct and their code is wrong. That's a terrible situation for everyone; we very much wish to design a language which does not have "gotcha" features like this.

But I'm getting ahead of myself. What's the output of this fragment?

var values = new List<int>() { 100, 110, 120 };
var funcs = new List<Func<int>>();
foreach(var v in values)
funcs.Add( ()=>v );
foreach(var f in funcs)
Console.WriteLine(f());

Most people expect it to be 100 / 110 / 120. It is in fact 120 / 120 / 120. Why?

Because ()=>v means "return the current value of variable v", not "return the value v was back when the delegate was created". Closures close over variables, not over values. And when the methods run, clearly the last value that was assigned to v was 120, so it still has that value.

This is very confusing. The correct way to write the code is:

foreach(var v in values)
{
var v2 = v;
funcs.Add( ()=>v2 );
}

Now what happens? Every time we re-start the loop body, we logically create a fresh new variable v2. Each closure is closed over a different v2, which is only assigned to once, so it always keeps the correct value.

Basically, the problem arises because we specify that the foreach loop is a syntactic sugar for

{
 IEnumerator<int> e = ((IEnumerable<int>)values).GetEnumerator();
 try
 {
 int m; // OUTSIDE THE ACTUAL LOOP
 while(e.MoveNext())
 {
 m = (int)(int)e.Current;
 funcs.Add(()=>m);
 }
 }
 finally
 {
 if (e != null) ((IDisposable)e).Dispose();
 }
}

If we specified that the expansion was

    try
    {
      while(e.MoveNext())
      {
      int m; // INSIDE
        m = (int)(int)e.Current;
        funcs.Add(()=>m);
      }

then the code would behave as expected.

It's compelling to consider fixing this for a hypothetical future version of C#, and I'd like to hear your feedback on whether we should do so or not. The reasons FOR making the change are clear; this is a big confusing "gotcha" that real people constantly run into, and LINQ, unfortunately, only makes it worse, because it is likely to increase the number of times a customer is going to use a closure in a loop. Also, it seems reasonable that the user of the foreach loop might think of there being a "fresh" loop variable every time, not just a fresh value in the same old variable. Since the foreach loop variable is not mutable by user code, this reinforces the idea that it is a succession of values, one per loop iteration, and not "really" the same variable over and over again. And finally, the change has no effect whatsoever on non-closure semantics. (In fact, in C# 1 the spec was not clear about whether the loop variable went inside or outside, since in a world without closures, it makes no difference.)

But that said, there are some very good reasons for not making this change.

The first reason is that obviously this would be a breaking change, and we hates them, my precious. Any developers who depend on this feature, who require the closed-over variable to contain the last value of the loop variable, would be broken. I can only hope that the number of such people is vanishingly small; this is a strange thing to depend on. Most of the time, people do not expect or depend on this behaviour.

Second, it makes the foreach syntax lexically inconsistent. Consider foreach(int x in M()) The header of the loop has two parts, a declaration int x and a collection expression, M(). The int x is to the left of the M(). Clearly the M() is not inside the body of the loop; that thing only executes once, before the loop starts. So why should something to the collection expression's left be inside the loop? This seems inconsistent with our general rule that stuff to the left logically "happens before" stuff to the right. The declaration is lexically NOT in the body of the loop, so why should we treat it as though it were?

Third, it would make the "foreach" semantics inconsistent with "for" semantics. We have this same problem in "for" blocks, but "for" blocks are much looser about what "the loop variable" is; there can be more than one variable declared in the for loop header, it can be incremented in odd ways, and it seems implausible that people would consider each iteration of the "for" loop to contain a fresh crop of variables. When you say for(int i; i < 10; i += 1) it seems dead obvious that the "i += 1" means "increment the loop variable" and that there is one loop variable for the whole loop, not a new fresh variable "i" every time through! We certainly would not make this proposed change apply to "for" loops.

And fourth, though this is a nasty gotcha, there is an easy workaround, and tools like ReSharper detect this pattern and suggest how to fix it. We could take a page from that playbook and simply issue a compiler warning on this pattern. (Though adding new warnings brings up a whole raft of issues of its own, which I might get into in another post.) Though this is vexing, it really doesn't bite that many people that hard, and it's not a big deal to fix, so why go to the trouble and expense of taking a breaking change for something with an easy fix?

Design is, of course, the art of compromise in the face of many competing principles. "Eliminate gotchas" in this case directly opposes other principles like "no breaking changes", and "be consistent with other language features". Any thoughts you have on pros or cons of us taking this breaking change in a hypothetical future version of C# would be greatly appreciated.

Simple names are not so simple

Eric Lippert — Mon, 02 Nov 2009 14:23:00 GMT

C# has many rules that are designed to prevent some common sources of bugs and encourage good programming practices. So many, in fact, that it is often quite confusing to sort out exactly which rule has been violated. I thought I might spend some time talking about what the different rules are. We'll finish up with a puzzle.

To begin with, it will be vital to understand the difference between scope and declaration space. To refresh your memory of my earlier article: the scope of an entity is the region of text in which that entity may be referred to by its unqualified name. A declaration space is a region of text in which no two things may have the same name (with an exception for methods which differ by signature.) A "local variable declaration space" is a particular kind of declaration space used for declaring local variables; local variable declaration spaces have special rules for determining when they overlap.

The next thing that you have to understand to make any sense o this is what a "simple name" is. A simple name is always either just a plain identifier, like "x", or, in some cases, a plain identifier followed by a type argument list, like "Frob<int, string>".

Lots of things are treated as "simple names" by the compiler: local variable declarations, lambda parameters, and so on, always have the first form of simple name in their declarations. When you say "Console.WriteLine(x);" the "Console" and "x" are simple names but the "WriteLine" is not. Confusingly, there are some textual entities which have the form of simple names, but are not treated as simple names by the compiler. We might talk about some of those situations in later fabulous adventures.

So, without further ado, here are some relevant rules which are frequently confused. It's rules 3 and 4 that people find particularly confusing.

1) It is illegal to refer to a local variable before its declaration. (This seems reasonable I hope.)
2) It is illegal to have two local variables of the same name in the same local variable declaration space or nested local variable declaration spaces.
3) Local variables are in scope throughout the entire block in which the declaration occurs. This is in contrast with C++, in which local variables are in scope in their block only at points after the declaration.
4) For every occurrence of a simple name, whether in a declaration or as part of an expression, all uses of that simple name within the immediately enclosing local variable declaration space must refer to the same entity.

The purpose of all of these rules is to prevent the class of bugs in which the reader/maintainer of the code is tricked into believing they are referring to one entity with a simple name, but are in fact accidentally referring to another entity entirely. These rules are in particular designed to prevent nasty surprises when performing what ought to be safe refactorings.

Consider a world in which we did not have rules 3 and 4. In that world, this code would be legal:

class C
{
int x;
void M()
{
    // 100 lines of code
    x = 20; // means "this.x";
    Console.WriteLine(x); // means "this.x"
    // 100 lines of code
    int x = 10;
    Console.WriteLine(x); // means "local x"
}
}

This is hard on the person reading the code, who has a reasonable expectation that the two "Console.WriteLine(x)" lines do in fact both print out the contents of the same variable. But it is particularly nasty for the maintenance programmer who wishes to impose a reasonable coding standard upon this body of code. "Local variables are declared at the top of the block where they're used" is a reasonable coding standard in a lot of shops. But changing the code to:

class C
{
int x;
void M()
{
    int x;
    // 100 lines of code
    x = 20; // no longer means "this.x";
    Console.WriteLine(x); // no longer means "this.x"
    // 100 lines of code
    x = 10;
    Console.WriteLine(x); // means "local x"
}
}

changes the meaning of the code! We wish to discourage authoring of multi-hundred-line methods, but making it harder and more error-prone to refactor them into something cleaner is not a good way to achieve that goal.

Notice that the original version of this program, rule 3 means that the program violates rule 1 -- the first usage of "x" is treated as a reference to the local before it is declared. The fact that it violates rule 1 because of rule 3 is precisely what prevents it from being a violation of rule 4! The meaning of "x" is consistent throughout the block; it always means the local, and therefore is sometimes used before it is declared. If we scrapped rule 3 then this would be a violation of rule 4, because we would then have two inconsistent meanings for the simple name "x" within one block.

Now, these rules do not mean that you can refactor willy-nilly. We can still construct situations in which similar refactorings fail. For example:

class C
{
int x;
void M()
{
    {
      // 100 lines of code
      x = 20; // means "this.x";
      Console.WriteLine(x); // means "this.x"
    }
    {
      // 100 lines of code
      int x = 10;
      Console.WriteLine(x); // means "local x"
    }
}
}

This is perfectly legal. We have the same simple name being used two different ways in two different blocks, but the immediately enclosing block of each usage does not overlap that of any other usage. The local variable is in scope throughout its immediately enclosing block, but that block does not overlap the block above. In this case, it is safe to refactor the declaration of the local to the top of its block, but not safe to refactor the declaration to the top of the outermost block; that would change the meaning of "x" in the first block. Moving a declaration up is almost always a safe thing to do; moving it out is not necessarily safe.

Now that you know all that, here's a puzzle for you, a puzzle that I got completely wrong the first time I saw it:

using System.Linq;
class Program
{
static void Main()
{
    int[] data = { 1, 2, 3, 1, 2, 1 };
    foreach (var m in from m in data orderby m select m)
      System.Console.Write(m);
}
}

It certainly looks like name "m" is being used multiple times to mean different things. Is this program legal? If yes, why do the rules for not re-using simple names not apply? If no, precisely what rule has been violated?

[Eric is on vacation; this posting was pre-recorded.]

As Timeless As Infinity

Eric Lippert — Thu, 15 Oct 2009 13:25:00 GMT

User: Recently I found out about a peculiar behaviour concerning division by zero in floating point numbers in C#. It does not throw an exception, as with integer division, but rather returns an "infinity". Why is that?

Eric: As I've often said, "why" questions are difficult for me to answer. My first attempt at an answer to a "why" question is usually "because that's what the specification says to do"; this time is no different. The C# specification says to do that in section 4.1.6. But we're only doing that because that's what the IEEE standard for floating point arithmetic says to do. We wish to be compliant with the established industry standard. See IEEE standard 754-1985 for details. Most floating point arithmetic is done in hardware these days, and most hardware is compliant with this specification.

User: It seems to me that division by zero is a bug no matter how you look at it!

Eric: Well, since clearly that is not how the members of the IEEE standardization committee looked at it in 1985, your statement that it must be a bug "no matter how you look at it" must be incorrect. Some industry experts do not look at it that way.

User: Good point. What motivated this design decision?

Eric: I wasn't there; I was busy playing Jumpman on my Commodore 64 at the time. But my educated guess is that it is desirable for all possible operations on all floats to produce a well-defined float result. Mathematicians would call this a "closure" property; that is, the set of floating point numbers is "closed" over all operations.

Positive infinity seems like a reasonable choice for dividing a positive number by zero. It seems plausible because of course the limit of 1 / x as x goes to zero (from above) is "positive infinity", so why shouldn't 1/0 be the number "positive infinity"?

Now, speaking as a mathematician, I find that argument specious. A thing and its limit need not have any particular property in common; it is fallacious to reason that just because, say, a sequence has a particular limit that a fact about the limit is also a fact about the sequence. Mathematically, "positive infinity" (in the sense of a limit of a real-valued function; let's leave transfinite ordinals, hyperbolic geometry, and all of that other stuff out of this discussion) is not a number at all and should not be treated as one; rather, it's a terse way of saying "the limit does not exist because the sequence diverges upwards".

When we divide by zero, essentially what we are saying is "solve the equation x * 0 = 1"; the solution to that equation is not "positive infinity", it is "I cannot because there is no solution to that equation". It's just the same as asking to solve the equation "x + 1 = x" -- saying "x is positive infinity" is not a solution; there is no solution.

But speaking as a practical engineer who uses floating point numbers to do an imprecise approximation of ideal arithmetic, this seems like a perfectly reasonable choice.

User: But surely it is impossible for the hardware to represent "infinity".

Eric: It certainly is possible. You've got 32 bits in a single-precision float; that's over four billion possible floats. All bit patterns of the form

?11111111???????????????????????

are reserved for "not-a-number" values. That's over sixteen million possible NaN combinations. Two of those sixteen million NaN bit patterns are reserved to mean positive and negative infinity. Positive infinity is the bit pattern 01111111100000000000000000000000 and negative infinity is 11111111100000000000000000000000.

User: Do all languages and applications use this convention of division-by-zero-becomes-infinity?

Eric: No. For example, C# and JScript do but VBScript does not. VBScript gives an error if you do that.

User: Then how do language implementors get the desired behaviour for each language if these semantics are implemented by the hardware?

Eric: There are two basic techniques. First, many chips which implement this standard allow the programmer to make float division by zero an exception rather than an infinity. On the 80x87 chip, for example, you can use bit two of the precision control register to determine whether division by zero returns an infinity or throws a hardware exception.

Second, if you don't want it to be a hardware exception but do want it to be a software exception, then you can check bit two of the status register after each division; it records whether there was a recent divide-by-zero event.

The latter strategy is used by VBScript; after we perform a division operation we check to see whether the status register recorded a divide-by-zero operation; if it did, then the VBScript runtime creates a divide-by-zero error and the usual VBScript error management process takes over, same as any other error.

Similar bits exist for other operations that seem like they might be better treated as exceptions, like numeric overflow.

The existence of the "hardware exception" bits creates problems for the modern language implementor, because we are now often in a world where code written in multiple languages from multiple vendors is running in the same process. Control bits on hardware are the ultimate "global state", and we all know how irksome it is to have global, public state that random code can stomp on.

For example: I might be misremembering some details, but I seem to recall that Delphi-authored controls set the "overflows cause exceptions" bit. That is, the Delphi implementors did not use the VBScript strategy of "try it, allow it to succeed, and check to see whether the overflow bit was set in the status register". Rather, they used the "make the hardware throw an exception and then catch the exception" strategy. This is deeply unfortunate. When a VBScript script calls a Delphi-authored control, the control flips the bit to force exceptions but it never "unflips" it. If, later on in the script, the VBScript program does an overflow, then we get an unhandled hardware exception because the bit is still set, even though the Delphi control might be long gone! I fixed that by saving away the state of the control register before calling into a component and restoring it when control returns. That's not ideal, but there's not much else we can do.

User: Very enlightening! I will be sure to pass this information along to my coworkers. I would be delighted to see a blog post on this.

Eric: And here you go!

Why No Extension Properties?

Eric Lippert — Mon, 05 Oct 2009 16:29:00 GMT

I'm frequently asked "you guys added extension methods to C# 3, so why not add extension properties as well?"

Good question.

First, let me talk a bit about C# 3. Clearly the big feature in C# 3 was LINQ. In a sense we had only three features in C# 3:

everything necessary for LINQ -- implicitly typed locals, anonymous types, lambda expressions, extension methods, object and collection initializers, query comprehensions, expression trees, improved method type inference
partial methods
automatically implemented properties

The latter two were tiny compared to the items in that first bucket. On the design side, the syntax and semantics of both features are straightforward. On the implementation side, we already had the mechanisms in place to remove a call site; partial methods and conditional methods are pretty much the same thing behind the scenes. Auto props were also straightforward to analyze and generate code for. The testing burden was not particularly large for these features either.

The C# team was "the long pole" for the 2008 release of Visual Studio and the .NET Framework. By that I mean that if you took the amount of time required to do the work each team signed up for, given our level of staffing, blah blah blah, and made a pole proportionally long for each team in Developer Division, the C# pole would have been the longest pole. Which means that every other team in devdiv had slack in their schedule, but if we slipped our schedule a day, the new VS/CLR would also ship a day late. If any other team slipped a day, well, as long as that didn't make their pole longer than ours, they were still OK. Within the team, dividing up the work amongs the various development team members, the "long pole" work was the lambda binding and method type inference work, which was mine. So in a sense, every day that I was late, we'd slip the whole product that many days. (No pressure!)

Fortunately we have an excellent team here, we all supported each other very well through that release, picked up each other's slack when necessary, and delivered a quality release in plenty of time. My point is simply that unless a feature was either necessary for LINQ, or small and orthogonal and easy to cut if necessary (like the other two), it got cut immediately. There was no way we were going to risk slipping the entire product for any feature that was both complex and unnecessary.

It was of course immediately obvious that the natural companion to extension methods is extension properties. It's less obvious, for some reason, that extension events, extension operators, extension constructors (also known as "the factory pattern"), and so on, are also natural companions. But we didn't even consider designing extension properties for C# 3; we knew that they were not necessary and would add risk to an already-risky schedule for no compelling gain.

So now we come to C# 4.

As I'm fond of pointing out, the answer to every question of the form "why doesn't product X have feature Y?" is the same. It's because in order for a product to have a feature, that feature must be:

thought of in the first place
desired
designed
specified
implemented
tested
documented
shipped to customers

You've got to hit every single one of those things, otherwise, no feature.

When we started working on C# 4, we made a list of every feature request we'd heard of. It had hundreds of features on it. As I described last year, we categorized that list into "gotta have / nice to have / bad idea" buckets. Extension properties were in the "gotta have" bucket. We then looked at our available budget -- which was not so much measured in dollars as in available designers, developers, testers, writers and management multiplied by available time -- and determined that we did not have the resources to do more than about half the things in the "gotta have" bucket. So we cut half that stuff. Extension properties made it past that cut.

We then designed the feature. We had many hours of debate about the proposed syntax on the declaration side, how to call an extension property getter or setter "directly" as a static method, and so on. We came up with a syntax we could agree was acceptable, designed the semantics, wrote up a draft specification, and started writing code and test plans.

By the time the code was in reasonable shape -- not yet seriously tested, but usable and compliant with the spec -- we'd been having meetings with the WPF team. WPF developers were the assumptive primary consumers of extension properties. WPF already has a mechanism that resembles extension properties; it would be nice to unify that mechanism with our mechanism. Unfortunately, after taking a deep look at their real-world scenarios, we came to the disappointing conclusion that we had designed the wrong thing; this was not actually a feature that would solve their problems.

This is, admittedly, in a sense a failure of the design process. In retrospect, we should have gotten input and feedback from the primary customer much earlier. Had we done that then they could have either influenced the design to make something that would work for them, or we could have cut the feature without taking on the expense of writing spec, code and test plans.

But when you're in an unfortunate situation, you've got to decide to stop throwing good money after bad. Rather than take on the additional costs -- testing, documentation, shipping, and then maintenance of the feature for the rest of the life of the language -- in exchange for a feature that did not serve the needs of our customers, we cut the feature. At that point we did not feel confident that we had enough cycles remaining to redesign the feature the right way.

Therefore, sadly, no extension properties in C# 4. Perhaps in a hypothetical future version of C#.

What's the difference between a partial method and a partial class?

Eric Lippert — Mon, 14 Sep 2009 18:16:00 GMT

Like "fixed" and "into", "partial" is also used in two confusingly similar-yet-different ways in C#.

The purpose of a partial class is to allow you to textually break up a class declaration into multiple parts, usually parts found in separate files. The motivation for this feature was machine-generated code that is to be extended by the user by adding to it directly. When you draw a form in the forms designer, the designer generates a class for you representing that form. You can then further customize that class by adding more code to it. If you can edit the machine-generated code then any number of problems arise. What if it is re-generated? What if the machine is using the code itself as a persisted state for design-time information about the class, and your edit messes up the machine's parser? It's much better to simply put the machine-generated half in its own file, generate a comment that says "can't touch this", and put the user code in its own location.

There are other uses of partial classes that do not involve machine-generated code but they are relatively rare. Some cases where I see partial classes used are:

If a class is really large and implements a bunch of interfaces, sometimes one interface implementation per file makes sense. More often though this is a bad code smell which indicates a class that is trying to do too much; it might be better to split this thing up into multiple classes.
It's somtimes nice to put nested classes in their own files; making the containing class partial is the only good way to do that.
And so on.

Partial methods are a subtly different story. Like partial classes, partial methods are about combining multiple declarations of the same method to make machine-generated code scenarios better. But though the high-level purpose of the feature is the same, the details are rather different.

The way a partial method works is there are two declarations, the "latent" declaration and the "actual" declaration. The latent declaration does not have a body, like it was an abstract method. The actual declaration does. The latent declaration goes in the machine-generated side of a partial class, the actual declaration goes in the human-authored side. If there is an actual declaration, then the latent declaration is completely ignored. But if there is no actual declaration then all calls to the method are removed, just as it the method were compiled with the conditional attribute! And in fact, the latent declaration is also logically removed when we spit out the metadata for the generated class; it's as if it never was.

The reason for this behaviour is because we wanted to enable this scenario:

// Machine generated code:
partial class MyFoo
{
void ButtonClickEventHandler(/*whatever*/)
{
      // call user code to see if they want to do anything at this time
      OnBeforeButtonClick(whatever);
      blah blah blah
      // call user code again
      OnAfterButtonClick(whatever);
}
partial void OnBeforeButtonClick(/*whatever*/);
partial void OnAfterButtonClick(/*whatever*/);
...

The user is going to be customizing the partial class; by putting in partial methods, the machine-generated side can create simple customization points all over the show that the user can then implement. But consider the down sides of this in a world without partial methods. The user is forced to implement empty methods. If they do not, they get errors. There are potentially hundreds of these empty methods to implement, which is vexing. And each of those methods generates a non-trivial amount of metadata, making the final binary larger than it needs to be. Disk space is cheap but network latency has replaced disk space as the factor that discourages large assemblies. If we can eliminate that metadata burden, that would be great.

Partial methods fit the bill. The user can provide actual implementations for as many methods as they choose, and now this becomes a "pay to play" model; you get as much implementation expense as the number of actual methods you implement.

The name of the feature during the design process was "latent and actual methods"; we strongly considered adding new keywords "latent" and "actual". But since the feature only makes sense in partial class scenarios, we decided to re-use the existing contextual keyword "partial" and renamed the feature. Hopefully the consonance between the two uses of "partial" helps more than the subtle differences hurt.

SUPER EXTRA BONUS: Yet more partiality

We considered adding a third kind of "partiality" to C# 4; this feature made it through the design phase but got cut before implementation. (If there is high demand, we'll consider adding it to hypothetical future versions of C#.)

Sometimes you're in a machine-generated code scenario like this:

// Machine generated
partial class C
{
int blah;
...

and then in the user-generated side of things, you want to do something like:

// User generated
partial class C : ISerializable
{

And oh heck, I need to put the NotSerialized attribute on blah, but I cannot edit the text of blah because when it gets re-generated, that will be lost.

The idea of this new kind of partiality is to re-state the declaration of a member -- a field, method, nested type, whatever -- with metadata attributes. It's like a latent/actual method but in reverse; the "actual" thing is in the machine-generated side, the "latent" thing in the user-generated side is just there to add metadata:

[NotSerialized] partial int blah; // not actually a declaration of a field

I like this feature but during the design process I pushed back hard on using the "partial" keyword to have a third meaning subtly different from the other two. Adding this confusion once seemed justifiable, but twice? That's pushing it. We therefore settled on adding another contextual keyword:

[NotSerialized] existing int blah; // not actually a declaration of a field

Decorating a declaration with "existing" would mean "this is not a real declaration, this is a mention of an existing declaration; please verify that such a declaration exists somewhere else in this partial class, and add this metadata to that member".

Like I said, this handy feature got cut because of resource constraints. If you have really awesome scenarios that this would make easier, I'd love to hear about them; obviously I cannot make any promises about possible future features of unannounced, entirely hypothetical products. But real user scenarios are a highly motivating factor in getting budget for features.

What's the Difference? Part Three: fixed vs. fixed

Eric Lippert — Thu, 27 Aug 2009 16:43:00 GMT

I got an email the other day that began:

I have a question about fixed sized buffers in C#:

unsafe struct FixedBuffer { public fixed int buffer[100]; }

Now by declaring buffer as fixed it is not movable...

And my heart sank. This is one of those deeply unfortunate times when subtle choices made in the details of language design encourage misunderstandings.

When doing pointer arithmetic in unsafe code on a managed object, you need to make sure that the garbage collector does not move the memory you're looking at. If a collection on another thread happens while you're doing pointer arithmetic on an object, the pointer can get all messed up. Therefore, C# classifies all variables as "fixed" or "movable". If you want to do pointer arithmetic on a movable object, you can use the "fixed" keyword to say "this local variable contains data which the garbage collector should not move." When a collection happens, the garbage collector needs to look at all the local variables for in-flight calls (because of course, stuff that is in local variables needs to stay alive); if it sees a "fixed" local variable then it makes a note to itself to not move that memory, even if that fragments the managed heap. (This is why it is important to keep stuff fixed for as little time as possible.) So typically, we use "fixed' to mean "fixed in place".

But that's not what "fixed" means in this context; this means "the buffer in question is fixed in size to be one hundred ints" -- basically, it's the same as generating one hundred int fields in this structure.

Obviously we often use the same keyword to mean conceptually the same thing. For example, we use the keyword "internal" in many ways in C#, but all of them are conceptually the same. It is only ever used to mean "accessibility to some entity is unrestricted to all code in this assembly".

Sometimes we use the same keyword to mean two completely different things, and rely upon context for the user to figure out which meaning is intended. For example:

var results = from c in customers where c.City == "London" select c;

class C<T> where T : IComparable<T>

It should be clear that "where" is being used in two completely different ways: to build a filter clause in a query, and to declare a type constraint on a generic type parameter.

We cause people to run into trouble when one keyword is used in two different ways but the difference is subtle, like our example above. The user's email went on to ask a whole bunch of questions which were predicated on the incorrect assumption that a fixed-in-size buffer is automatically fixed in place in memory.

Now, one could say that this is just an unfortunate confluence of terms; that "fixed in size" and "fixed in place" just happen to both use the word "fixed" in two different ways, how vexing. But the connection is deeper than that: you cannot safely access the data stored in a fixed-in-size buffer unless the container of the buffer has been fixed in place. The two concepts are actually quite strongly related in this case, but not at all the same.

On the one hand it might have been less confusing to use two keywords, say "pinned" and "fixed". But on the other hand, both usages of "fixed" are only valid in unsafe code. A key assumption of all unsafe code features is that if you are willing to use unsafe code in C#, then you are already an expert programmer who fully understands memory management in the CLR. That's why we make you write "unsafe" on the code; it indicates that you're turning off the safety system and you know what you're doing.

A considerable fraction of the keywords of C# are used in two or more ways: fixed, into, partial, out, in, new, delegate, where, using, class, struct, true, false, base, this, event, return and void all have at least two different meanings. Most of those are clear from the context, but at least the first three -- fixed, into and partial -- have caused enough confusion that I've gotten questions about the differences from perplexed users. I'll take a look at "into" and "partial" next.

Iterator Blocks Part Seven: Why no anonymous iterators?

Eric Lippert — Mon, 24 Aug 2009 16:23:00 GMT

This annotation to a comment in part five I think deserves to be promoted to a post of its own.

Why do we disallow anonymous iterators? I would love to have anonymous iterator blocks. I want to say something like:

IEnumerable<int> twoints = ()=>{ yield return x; yield return x*10; };
foreach(int i in twoints) ...

It would be totally awesome to be able to build yourself a little sequence generator in-place that closed over local variables. The reason why not is straightforward: the benefits don't outweigh the costs. The awesomeness of making sequence generators in-place is actually pretty small in the grand scheme of things and nominal methods do the job well enough in most scenarios. So the benefits are not that compelling.

The costs are large. Iterator rewriting is the most complicated transformation in the compiler, and anonymous method rewriting is the second most complicated. Anonymous methods can be inside other anonymous methods, and anonymous methods can be inside iterator blocks. Therefore, what we do is first we rewrite all anonymous methods so that they become methods of a closure class. This is the second-last thing the compiler does before emitting IL for a method. Once that step is done, the iterator rewriter can assume that there are no anonymous methods in the iterator block; they've all be rewritten already. Therefore the iterator rewriter can just concentrate on rewriting the iterator, without worrying that there might be an unrealized anonymous method in there.

Also, iterator blocks never "nest", unlike anonymous methods. The iterator rewriter can assume that all iterator blocks are "top level".

If anonymous methods are allowed to contain iterator blocks, then both those assumptions go out the window. You can have an iterator block that contains an anonymous method that contains an anonymous method that contains an iterator block that contains an anonymous method, and... yuck. Now we have to write a rewriting pass that can handle nested iterator blocks and nested anonymous methods at the same time, merging our two most complicated algorithms into one far more complicated algorithm. It would be really hard to design, implement, and test. We are smart enough to do so, I'm sure. We've got a smart team here. But we don't want to take on that large burden for a "nice to have but not necessary" feature.

Arrays of arrays

Eric Lippert — Mon, 17 Aug 2009 17:25:00 GMT

Most people understand that there’s a difference between a “rectangular” and a “ragged” two-dimensional array.

int[,] rectangle = {
{10, 20},
{30, 40},
{50, 60} };
int[][] ragged = {
new[] {10},
new[] {20, 30},
new[] {40, 50, 60} };

Here we have a two-dimensional array with six elements, arranged in three rows of two elements each. And we have a one-dimensional array with three elements, where each element is itself a one-dimensional array with one, two or three elements.

That’s all very straightforward. Where things get brain-destroying is when you try to make a ragged array of two-dimensional arrays. Quick, don’t think, just answer.

int[,][] crazy;

What is the type of crazy?

Option one: It’s a one-dimensional array, each element is a two-dimensional array of ints.
Option two: It’s a two-dimensional array, each element is a one-dimensional array of ints.

OK, now that you have your snap answer, think about it carefully. Does your answer change?

I’m not going to tell you the answer just yet. Instead let’s explore the consequences of each possibility.

Consequence One

Surely the way you make any type into an array is to append [] to the type specification, right? But in option two, you stick the [,] into the middle.

Option two is weird. Option one is sensible.

But wait. If [,][] means "a 1-d of 2-d's", then the order you read it off the page opposes the order you say it -- it looks like two-d-thing-followed-by-one-d-thing, so why shouldn't it read "a 2-d of 1-d's"?

But then why does the "int" come first? By that logic it should come last.

Argh. Maybe option one isn't entirely sensible. Clearly something is not quite perfect with both options. Oh well. Let's move on.

Consequence Two

Now suppose that you wanted to obtain a value in that array, assuming that it had been initialized correctly to have plenty of elements everywhere we need them. How would you do it?

Suppose we’re in option one. It’s a one-d array. Therefore crazy[10] is a two-d array. Therefore crazy[10][1, 2] is an int.

Suppose we’re in option two. It’s is a two-d array. Therefore crazy[1,2] is a one-d array. Therefore crazy[1,2][10] is an int.

Option one is weird -- crazy is of type int[,][] but you dereference it as [10][1,2]! Whereas option two is sensible; the dereferencing syntax exactly matches the ordering of the type name syntax.

Consequence Three

Now suppose you want to initialize the “outer” array but are going to fill in the “ragged” interior arrays later. You’ll just keep them set to null at first. What’s the appropriate syntax to initialize the outer array?

Suppose we’re in option one. It’s a one-d array. Therefore it should be initialized as crazy = new int[,][20]; right?

Suppose we’re in option two. It’s a two-d array. Therefore it should be initialized as crazy = new int[][4,5]; right?

Option two is weird. We said int[,][] but initialized it as [][4,5]. Option one is sensible.

What C# actually does

It’s a mess. No matter which option we choose, something ends up not matching our intuition. Here’s what we actually chose in C#.

First off: option two is correct. We force you to live with the weirdness entailed by Consequence One; you do not actually make an element type into an array of that type by appending an array specifier. You make it into an array type by prepending the specifier to the list of existing array specifiers. Crazy but true.

This prevents you from having to live with any weirdness from Consequence Two; in this option, the dereferencing happens with the same lexical structure as the declaration.

What about Consequence Three? This one is the real mind-bender. Neither choice I offered you is correct; apparently I’m a sneaky guy. The correct way to initialize such an array in C# is:

crazy = new int[4,5][];

This is very surprising to people!

The design principle here is that users expect the lexical structure of the brackets and commas to be consistent across declaration, initialization and dereferencing. Option two is the best way to ensure that if declaration has the shape [,][] then the initialization also has that shape, and so does the dereferencing.

That all said, multidimensional ragged arrays are almost certainly a bad code smell. Hopefully you will never, ever have to use your new knowledge of these rules in a production environment.

Life is much better if you can instead use generic collections. It is completely clear what List<int[,]> means – that’s a list of two-dimensional arrays. Whereas List<int>[,] means a two-d array of lists of int.

Four switch oddities

Eric Lippert — Thu, 13 Aug 2009 16:33:00 GMT

The C# switch statement is a bit weird. Today, four quick takes on things you probably didn't know about the switch statement.

Case 1:

You probably know that it is illegal to "fall through" from one switch section to another:

switch(attitude)
{
case Attitude.HighAndMighty:
    Console.WriteLine("High");
    // we want to fall through, but this is an error
case Attitude.JustMighty:
    Console.WriteLine("Mighty");
    break;
}

But perhaps you did not know that it is legal to force a fall-through with a goto:

switch(attitude)
{
case Attitude.HighAndMighty:
    Console.WriteLine("High");
    goto case Attitude.JustMighty;
case Attitude.JustMighty:
    Console.WriteLine("Mighty");
    break;
}

Pretty neat! Cases are semantically just labels to which the switch does a conditional branch; we let you do an explicit branch if you want to.

Case 2:

A common and confusing error that C programmers using C# (like me!) make all the time:

switch(attitude)
{
case Attitude.HighAndMighty:
    Console.WriteLine("High and Mighty");
    break;
case Attitude.JustMighty:
    Console.WriteLine("Just Mighty");
    break;
default:
    // Do nothing
}

That's an error because in the default case, you "fall through". Admittedly, there is nothing to "fall through" to, but the compiler is picky about this one. It requires that every switch section, including the last one, have an unreachable end point. The purpose of this rule, and of the no-fall-through rule in general, is that we want you to be able to arbitrarily re-order your switch sections without accidentally introducing a breaking change. Fix it by making the "do nothing" explicit with a break statement.

This is particularly confusing because some people interpret the error message as saying that the problem is falling into the default case, when actually the problem is that you're falling out of the default case.

Case 3:

As I discussed earlier, a declaration space is a region of code in which two declared things may not have the same name. The foreach loop implicitly defines its own declaration space, so this is legal:

foreach(var blah in blahs) { ... }
foreach(var blah in otherblahs) { ... }

Switch blocks also define their own declaration spaces, but switch sections do not:

switch(x)
{
case OneWay:
    int y = 123;
    FindYou(ref y);
    break;
case TheOther:
    double y = 456.7; // illegal!
    GetchaGetcha(ref y);
    break;
}

You can solve this problem in a number of ways; the easiest is probably to wrap the body of the switch section in curly braces:

switch(x)
{
case OneWay:
  {
    int y = 123;
    FindYou(ref y);
    break;
}
case TheOther:
{
    double y = 456.7; // legal!
    GetchaGetcha(ref y);
    break;
}
}

which tells the compiler "no, really, I want these to be different declaration spaces".

If you have a variable that you want to declare once and use it in a bunch of different places, that's legal, but a bit strange:

switch(x)
{
case OneDay:
    string s;
    SeeYa(out s);
    break;
case NextWeek:
    s = "hello"; // legal, we use the declaration above.
    Meetcha(ref s);
    break;
}
}

That looks a bit weird, I agree, but it also looks a bit weird to have one switch block with two unbraced competing declarations in it. There are pros and cons of each, the design team had to pick one way or the other, and they chose to have switch cases not define a declaration space.

Case 4:

A funny consequence of the "reachability analysis" rules in the spec is that this program fragment is not legal:

int M(bool b)
{
switch(b)
{
case true: return 1;
case false: return 0;
}
}

Of course in reality you would probably write this as the far more concise "return b ? 1 : 0;" but shouldn't that program be legal? It is not, because the reachability analyzer reasons as follows: since there is no "default" case, the switch might choose some option that is not either case. Therefore the end point of the switch is reachable, and therefore we have an int-returning method with a reachable code path that falls off the end of the method without returning.

Yeah, the reachability analyzer is not very smart. It does not realize that there are only two possible control flows and that we've covered all of them with returns. And of course, if you switch on a byte and have cases for each of the 256 possibilities, again, we do not detect that the switch is exhaustive.

This shortcoming of the language design is silly, but frankly, we have higher priorities than fixing this silly case. If you find yourself in this unfortunate case, just stick a "default:" label onto one of the sections and you'll be fine.

-- Eric is on vacation; this posting was prerecorded. --

Iterator Blocks, Part Six: Why no unsafe code?

Eric Lippert — Mon, 27 Jul 2009 17:07:00 GMT

There are three good reasons to disallow unsafe blocks inside an iterator block.

First, it is an incredibly unlikely scenario. The purpose of iterator blocks is to make it easy to write an iterator that walks over some abstract data type. This is highly likely to be fully managed code; it's simply not a by-design scenario.

Second, the scenario is a violent mixing of "levels." You think of the level of abstraction of a programming language feature as how "far from the machine" the feature is. Unsafe pointer manipulation is as close to the machine as it gets in C#. You are running without a safety net, poking and peeking at the raw bytes in the virtual memory space of the process. All the protections of verifiable type safety are completely gone and you have total responsibility over every bit in the user-mode address space.

Iterator blocks, by contrast, are the highest level of abstraction in C# 2.0. There is no immediately obvious mapping from the source code to the generated code; it's a magic box that does the right thing because the compiler is doing some pretty massive transformations on the code, generating classes and implementing interfaces for you.

Mixing those two levels of abstraction in one method seems like a really bad idea.

Third, we would have to do some extra work to make "fixed" blocks work correctly, work that would be directly opposed to our guidelines for the correct and efficient use of the "fixed" block.

I suppose I should briefly explain what the "fixed" block does. When you call unmanaged code that needs to, say, write into the unmanaged address that is the beginning of an array of 16 bit chars, somehow you need to tell the garbage collector "hey, you, GC, running on that other thread over there -- if you get the urge to reorganize memory to be more efficient, I need you to please not move this array for a moment, because unmanaged code on this thread is going to be writing into that address". This operation is called "fixing" or "pinning" the object.

Clearly it is a bad idea to pin lots of stuff and leave it pinned for a long time. The GC works as well as it does because it is able to reorganize memory to be more efficient; pinning blocks of memory across collections impedes the GC's ability to get work done. We therefore recommend that you pin as little as possible for as little time as possible.

The CLR provides us two ways to pin something. The hard, flexible, expensive way is to explicitly get a GCHandle on the object, tell the garbage collector "this object is pinned", take the address of the object, do what you need to do, and then unpin it.

The easy way is to mark a local variable as a "pinned local" using the "fixed" statement. The C# compiler will generate special code that marks the local as "pinned". When the garbage collector does a collection, clearly the GC needs to look at all the locals on the current stack frame, because those locals are all "alive". When the GC sees that a local is pinned, it not only notes that it is alive, it also makes a note to itself that the referent of the contents of this variable are not to be moved during the collection. This is a cheap and easy alternative to explicitly getting a GC handle, but it depends on the variable in question being a local.

In an iterator block, locals are all realized by hoisting them to fields. Because we cannot pin fields, we'd have to change the code generation for the fixed block so that it took out a handle to the object, pinned it, and ensured that the object was unpinned at some point.

That's bad enough; now consider what happens when you yield while an object is pinned. Arbitrarily much time could pass between the yield and the unpin! This directly violates our guidance that things should be pinned for as little time as possible.

For all those reasons, no unsafe code is allowed in iterator blocks. In the unlikely event that you need to dereference pointers in an iterator block, you can always extract the unsafe code to its own helper method and call it from the iterator block.

***********

I hope you've enjoyed this little trip into the weird corner cases of iterator blocks. This is a complicated feature with lots of interesting design choice points.

I'm flying south to Canada to spend some time with my family on the beaver-shark infested shores of the great inland sea known as Lake Huron, so the next few fabulous adventures will be pre-recorded. See you when I'm back.

Iterator Blocks, Part Five: Push vs Pull

Eric Lippert — Thu, 23 Jul 2009 17:05:00 GMT

A while back I posted some commentary I did for the Scripting Summer Games where I noted that there is an isomorphism between "pull" sequences and "push" events. Normally you think of events as something that "calls you", pushing the event arguments at you. And normally you think of a sequence as something that you "pull from", asking it for the next value until you're done. But you can treat a stream of event firings as being a sequence of event argument objects. And similarly, you could implement a sequence iterator so that it called a method for every object in the collection. Heck, you could implement all of LINQ on this model if you really wanted.

The implementation of iterator blocks is clearly on the "pull" paradigm. It didn't have to be. We could have gone with a sort of "inversion of control" approach. "Pull" iterators have to simulate coroutines with a little state machine that knows how to get back to the state where the code left off. But we already have a mechanism for code to "get back to the state where it left off" -- that's what you do every time you call a method! You remember what you were doing, call the method, and then pick up where you left off. We could have done the same thing with iterators. We could say that this:

void Integers(int length, IObserver<int> observer)
{
for (int i = 0; i < length; ++i) yield return i;
}

is a syntactic sugar for

void Integers(int length, IObserver<int> observer)
{
for (int i = 0; i < length; ++i) observer.Yield(i);
observer.Break();
}

That is, we "raise an event" by calling the observer every time something yields, and raise another event when we're done.

This would be a much more trivial transformation to make than our current state machine approach, but since most people want sequence iteration on the "pull" model, we do the harder work to make that happen.

I said this had something to do with exception handling. What's the connection?

Did you notice something weird about how we handle finally blocks? Consider what happens with the "push model":

TextReader reader = File.Open(file);
try
{
blah blah blah yield the lines
}
finally
{
reader.Close();
}

If something in the "blah blah blah" is realized as a call to observer.Yield(line), what happens if the code consuming the result throws? Easy. It is just a method call like any other. The call stack unwinds, we find the finally, the file gets closed, everything is good.

Now suppose this is realized as the MoveNext of a "pull" iterator. What happens when the code that is consuming the result throws? If we're consuming the result then we've returned from the call to MoveNext! There is no "try", there is no "finally". And yet usually the finally gets executed anyways! If the code that is consuming the results throws, odds are good that it was in a foreach; when foreach throws, the enumerator is disposed, and when the enumerator is disposed, we figure out which "finallys" were around when we last left MoveNext, and we run a special method that does whatever was in the pending finally blocks.

This is pretty weird. What's going on here is that for finally blocks, "pull" iterators have the same semantics as "push" iterators. When you yield from a try containing a finally, it looks like the finally is still "on the stack" and runs when the consumer throws.

So what if the try block has a catch?

The original designers of C# 2.0 -- and remember, this is long before my time on the team -- had a huge debate over this. A debate which was repeated in miniature when I sent them all an email asking for the justification for this decision. There are three basic positions:

1) Do not allow yield returns in try blocks at all. (Or blocks that are sugars for try blocks, like using blocks.) Use some other mechanism other than "finally" to express the idea of "this is the code that you run when the caller shuts down the enumeration."

2) Allow yield returns in all try blocks.

3) Allow yield returns in try blocks that have finally blocks, but not if they have catch blocks.

The down sides of (1) are that you have to come up with some syntax for representing the "finally" logic that isn't actually a "finally", and that it becomes more difficult to iterate over stuff that requires cleanup, like iterating over the contents of a file. This makes the feature both confusing and weak; we should avoid this option if at all possible.

The down side of (2) is that the exception handling behaviour beomes deeply, weirdly inconsistent between finally blocks and catch blocks. If you have a yield in a try block with a finally, and the consumer of the iteration throws, then the finally runs, as though you were on the "push" model. But if you have a yield in a try block with a catch, and the consumer of the iteration throws, then the catch does not run, because its not on the call stack. Users have a reasonable expectation that finally and catch work more or less the same way when an exception happens; that is, that control will be consistently transferred to a matching catch or the finally.

The down side of (3) is that the rule seems arbitrary and weird -- until you read five unnecessarily prolix blog entries that explain what on earth the design team was thinking.

Obviously, they picked (3), and now you know why.

Next time, we'll finish up this series with a look at unsafe code. Now that you know all this stuff, seeing why there's no unsafe code allowed in an iterator block is pretty straightforward by comparison.

Iterator blocks Part Four: Why no yield in catch?

Eric Lippert — Mon, 20 Jul 2009 16:44:00 GMT

Now that you know why we disallow yielding in a finally, it’s pretty straightforward to see why we also disallow yielding in a catch.

First off, we still have the problem that it is illegal to “goto” into the middle of the handler of a try-protected region. The only way to enter a catch block is via the “non-local goto” that is catching an exception. So once you yielded out of the catch block, the next time MoveNext was called, we’d have no way to get back into the catch block where we left off.

Second, the exception that was thrown and caught is an intrinsic part of the execution of the catch block because it can be re-thrown using the “empty throw” syntax. We have no way of preserving that state across calls to MoveNext.

I suppose that we could in theory allow yields inside catch blocks that do not have an “empty throw”. We could turn this:

try
{
M();
}
catch(Exception x)
{
yield return x.ToString().Length;
yield return 123;
}

into

switch(this.state)
{
case 0: goto LABEL0;
case 1: goto LABEL1;
case 2: goto LABEL2;
}
LABEL0:
try
{
M();
this.state = 2;
goto LABEL2;
}
catch(Exception x)
{
this.x = x;
}
this.current = this.x.ToString().Length;
this.state = 1;
return true;
LABEL1:
this.current = 123;
this.state = 2;
return true;
LABEL2:
return false;

But frankly, that would be pretty weird. It would feel like a very odd restriction to disallow a fundamental part of the “catch” syntax.

And really, what’s the usage case that motivates this situation in the first place? Do people really want to try something and then yield a bunch of results if it fails? I suppose you could do something like:

IEnumerable<string> GetFooStrings(Foo foo)
{
try
{
yield return Foo.Bar();
}
catch(InvalidFooException x)
{
yield return "";
}
… etc.

but there are lots of ways to easily rewrite this so that you don’t have to yield in a catch:

IEnumerable<string> GetFooStrings(Foo foo)
{
string result;
try
{
result = Foo.Bar();
}
catch(InvalidFooException x)
{
result = "";
}
yield return result;
… etc.

Since there are usually easy workarounds, and we would have to put crazy-seeming restrictions on the use of re-throws, it’s easier to simply say “no yields inside a catch”.

We still haven’t explained why it is illegal to yield inside a try that has a catch, nor why it is legal to yield inside a try that has a finally. To understand why, next time I’ll need to muse about the duality between sequences and events.

Iterator Blocks, Part Three: Why no yield in finally?

Eric Lippert — Thu, 16 Jul 2009 16:18:00 GMT

There are three scenarios in which code could be executing in a finally in an iterator block. In none of them is it a good idea to yield a value from inside the finally, so this is illegal across the board. The three scenarios are (1) normal cleanup, (2) exception cleanup, and (3) iterator disposal.

For the first scenario, suppose we have something like

try
{
Setup();
yield return M();
}
finally
{
yield return N();
Cleanup();
}

How should we transform this into an iterator state machine? Naively, we want to do something like:

switch (this.state)
{
case 0: goto LABEL0;
case 1: goto LABEL1:
case 2: goto LABEL2:
case 3: goto LABEL3:
}
LABEL0:
try
{
Setup();
this.current = M();
this.state = 1;
return true; // BUT DON'T RUN THE FINALLY!
LABEL1:
}
finally
{
this.current = N();
this.state = 2;
return true;
LABEL2:
Cleanup();
}
LABEL3:
this.state = 3;
return false;

There's an immediate problem with this: we both "goto" into a finally block and "return" out of one. Neither is legal.

Leaving a finally block via "return" is illegal because it is weird. Suppose the try block returns, causing the finally to execute. What happens when the finally block returns? The original return value would be lost and some other return value would be substituted for it. This seems bizarre and bug-prone, so its not legal; you cannot exit a finally via return. And of course, you don't want the finally to run after the return in the try in this case!

Furthermore, in the CLR model of exception handling it is illegal to branch via a goto into a try block or its "handler" (that is, the catch or finally clause). Nor can you branch out of the handler. These special regions have special code that needs to be run when the region is entered and exited; you cannot skip either with a goto.

So we have no immediately obvious way to generate verifiable code for this scenario. Right off the bat, we have a huge number of points against this feature; we'd have to either convince the CLR guys to allow spaghetti code involving protected regions, or come up with some clever technique for generating this code.

And that's just the scenario where nothing is going wrong yet! Suppose a miracle happens and we manage to successfully generate code for scenario one. Now consider our second scenario, where M throws exception X. Remember, the finally block "catches" the exception and processes the cleanup code. If the cleanup code throws, then the original exception is discarded and the new processing of the new exception takes over. If the cleanup code succeeds, then the original exception continues to be "thrown" up the call stack, looking for more finally blocks or catch blocks.

Suppose the cleanup code does not throw. What should the control flow look like? The caller calls MoveNext for the first time. M() throws X. The finally block takes over. The finally block calls N(), and returns control to the caller with the results of N() !? What happened to X, the exception? Is it just waiting there, in limbo? When the second call to MoveNext happens, should the cleanup code get run then, and X suddenly pops back into existence and continues to be thrown up the stack? That makes no sense at all; the two calls to MoveNext could have had completely different stacks! This is crazy, plus we have no mechanism at all in the CLR for these kind of shenanigans.

Third, suppose we manage to solve all these problems. Now consider what happens when the caller calls MoveNext, M() succeeds, control returns to the caller, and the caller calls an early Dispose() on the enumerator. (I guess they wanted only one item.) We generate a Dispose method that checks the current state and executes any pending finally blocks. What the heck is the Dispose method supposed to do when it encounters the yield return in the pending finally block? We're not even in a call to MoveNext anymore! Should we call N() and ignore the result? Should we return from the Dispose() after the call to N(), or do the cleanup and then return? What exactly should the control flow do here? We're in a context where we might not even be iterating anymore, and yet we're still yielding.

It really doesn't make any logical sense to do a yield return in this scenario; we're possibly not in a position where the caller is expecting things to continue to be yielded to it.

So in short, we'd have to do at least two impossible things in order to enable a scenario that makes no sense in the first place. If ever a feature called out to be cut at the design phase, this is it. Therefore: no yields inside finally blocks. Thank goodness for that.

Next time: now that you know all that, figuring out "why no yields inside catch blocks" is pretty straightforward.