Fabulous Adventures In Coding : Puzzles

Always write a spec, part one

Eric Lippert — Thu, 19 Nov 2009 14:45:00 GMT

Joel had a great series of articles many years ago about the benefits of writing functional specifications, that is, specifications of how the product looks to its users. I want to talk a bit about technical specifications, that is, a specification of how something actually works behind the scenes. A while back, I described how I wrote a seemingly-trivial piece of code by first writing the technical spec, and then writing the code to exactly implement the spec, and the test cases to test each statement of the spec. I use this technique frequently; it almost always saves me time and effort. If you don't write a spec first, it's very easy to get bogged down in a morass of bugs, and hard to know when you're actually done writing the code.

Here's an example of another seemingly-trivial function that smart people gave a good half-dozen completely wrong answers to because they didn't write a spec. Notice how once I state what the spec is, the code follows very naturally.

I thought I might go through a recent example of a slightly less trivial problem. I needed a function in the compiler which could take a valid list of arguments A1 to an applicable method M, and give me back two lists. The first result list is a list of local variable declarations, L, the second, another argument list, A2. Here's the kicker: evaluating each member of A2 must have no side effects. And, the combination of executing L followed by M(A2) must give exactly the same results as executing M(A1). (I also knew that the members of A1 had already had all the conversions generated to make them match the types of M's parameter list, and so on. Overload resolution was completely done at this point.)

Why I needed such a function is not particularly important; perhaps you can speculate as to why I needed it.

So I started writing a spec, starting with the description above: the function T takes a valid list... blah blah blah.

That's not enough of a spec to actually write code.

I realized that some expressions in A1 might have side effects, and some might not. If an expression had side effects, then I could declare a temp local, execute the side effects, assign the result to the local, and then use the local as the expression in the same place in A2.

Super. Now I can write more sentences in my spec.

For each argument x in A1, we potentially generate a temporary value as follows: First, if x has no side effects, then simply keep x in the appropriate place in A2. Otherwise, if x does have side effects, then generate a temporary variable declaration in L, var temp = x. Then put an expression that evaluates the temporary variable in the appropriate place in A2.

Then I thought "can I correctly implement this spec?"

I started thinking about the possible cases for the input, A1. The members of A1 could be passed by value, out, or ref; I already knew they were correct, since the particular method M was an applicable method of argument list A1. In this particular point in the compilation process, there was no need to worry about whether there were "params" arguments, whether there were missing arguments with default values, and so on; that had already been taken care of. I also noted that whether they were "ref" or "out" was irrelevant; ref and out are the same thing behind the scenes. As far as the compiler is concerned, the only difference between them is that the definite assignment rules are different for each. So we had two cases for each argument: it's being passed by value, or by ref/out.

If it's being passed by ref/out, then what we actually do is we pass the managed address of a variable to the method. In M(ref int x), the type of x is actually a special type int&, "managed reference to variable of type int", which is not a legal type in C#. That's why we hide it from you by requiring a goofy "ref" syntax any time you want to talk about a managed reference to a variable.

Unfortunately, our IL generation pass was written with the assumption that all local variables are never of these magic managed-reference-to-variable types. I was then faced with a choice. Either come up with a technique for working with this limitation, or rewrite the most complicated code in the IL gen to support this scenario. (The code which figures out how to deal with managed references is quite complex.) I decided to opt for the former strategy. Which meant more spec work, since our spec now doesn't handle these cases!

For each argument x in A1, we potentially generate a temporary value as follows:

First, if x has no side effects, then simply keep x in the appropriate place in A2.

Otherwise, if x does have side effects and corresponds to a value parameter, then generate ... etc.

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then a miracle happens.

Clearly that last bit needed some work.

So I wrote down all the possible cases I could think of. Clearly x has to be a variable if its type is "reference to variable", so that makes for a small number of cases:

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then the possible cases are:

* x is a local variable
* x is a value parameter
* x is an out parameter
* x is a ref parameter
* x is a field of an instance
* x is a static field
* x is an array element
* x is a pointer dereferenced with *
* x is a pointer dereferenced with [ ]

and for each, a miracle happens.

Again, clearly this is not good enough. I re-examined my list to see if I could eliminate any of these cases. Locals, parameters and static fields never have side effects, so we can eliminate them.

Also, at this point in our analysis, I knew that pointer dereferences of the form "pointer[index]" had already been rewritten into the form "*(pointer + index)", which meant that in practice, we would never actually hit that last case in this algorithm; the second-last case would take care of both.

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then the possible cases are:

* x is a field of an instance
* x is an array element
* x is a pointer dereferenced with *

and for each, a miracle happens.

I then started to think of what side effects could happen for each. We could have "ref instance.field", "ref array[index]", or "ref *pointer", and "instance", "array", "index" and "pointer" can all be expressions that have a side effect. ("field" cannot, it merely names a field.) So now we can use the same specification as before:

Otherwise, if x does have side effects and corresponds to a ref/out parameter, then the possible cases are:

x is ref/out instance.field: in this case, add var temp=instance to L and ref/out temp.field to A2.
x is ref/out array[index]: in this case, add var t1 = array and var t2 = index to L and ref/out t1[t2] to A2.
x is ref/out *pointer: in this case, add var temp = pointer to L and ref/out *temp to A2.

And now we have something I can implement. So I sent this proposed spec around for review, while I started plugging away at the implementation.

This spec is wrong. Can you spot the bugs in my implementation that my coworkers found by reading my spec?

Next time, the thrilling conclusion.

Simple names are not so simple, Part Two, plus, volcanoes and fried foods

Eric Lippert — Thu, 05 Nov 2009 14:52:00 GMT

I've returned from a brief vacation, visiting friends on the island of Maui. I'd never been to that part of the world before. Turns out, it's a small island in the middle of the Pacific Ocean, entirely made out of volcanoes. Weird! But delightful.

The most impressive thing about the Hawaiian Islands for me was just how obvious were -- even to my completely untrained eyes -- the geomechanical and fluvial processes which shaped the landscape. The mountains and craters and river valleys and red sand beaches and easily-fractured rocks were very different from the (also somewhat volcanic) much older mountainous landscape I've lived in for the past decade.

Also quite amusing to me was learning to read and pronounce Hawaiian place names. It is all very logical once you know the system; before long I could easily pronounce signs like WAINAPANAPA STATE PARK -- wa-ee-napa-napa -- or PUUNENE AVENUE -- pu-oo-nay-nay -- or MAILIBEHANAMONOTANA STREET -- "Miley-Stewart-is-really-Hannah-Montana".

Many thanks to K and R and D for putting me and Leah up for a week; if you're going to Hawai'i and can stay with locals, I highly recommend it, particularly if they are awesome people. Everyone in Maui was awesome, with the exception of the rangers at (stunningly beautiful, even for Maui) Wa'inapanapa State Park, who are apparently consistently grumpy. As one Hawaiian, himself in the camping sector of the economy put it to me, "They do not have the big aloha".

The most amusing encounter was on the Hana Highway. There are numerous little stops along the way, where someone has erected a hut or parked a trailer and is selling coconuts, smoothies, banana bread, and so on. Hand-lettered signs, stunning natural beauty, middle of nowhere, you get the picture I'm sure. At one of the larger such stops there was a young fellow, probably in his late twenties, serving a variety of fried foods. It was mostly traditional American-style Chinese food, but also he had french fries, fish'n'chips, and so on. He was clearly not a native speaker of English, but spoke understandably with a strong accent. We were waiting behind a middled-aged woman with a typically midwestern American accent. Their conversation went something like this:

Her: I'm not very hungry, can I just get the fish without the chips?

Him, not quite following her: Half order?

Her, louder: How much without the fries?

This went back and forth for some time, both sides becoming increasingly frustrated by the communication breakdown, until:

Her, even louder: Can I speak to your manager?

Leah and K and I silently boggled -- there is no other word for it -- at each other for a moment. Where on earth did she imagine that a manager was going to emerge from? There was a counter, behind that, a trailer with a wok in it, behind that, jungle, and behind that, huge jagged lava rocks followed immediately by the Pacific Ocean. And what sort of management structure does she think one really needs to manage a single guy selling pineapple fried rice at the side of a highway? My conclusion: people have strange beliefs. Sometimes their beliefs cause them to leave in a huff with neither fish nor chips, even when fish and chips are both plentiful and reasonably priced. Hopefully she had better luck in Hana.

Anyway, enough travelogue. Regarding the puzzle from last time: the code is correct, and compiles without issue. I was quite surprised when I first learned that; it certainly looks like it violates our rule about not using the same simple name to mean two different things in one block.

The key is to understanding why this is legal is that the query comprehensions and foreach loops are specified as syntactic sugars for another program, and it is that program which is actually analyzed for correctness. Our original program:

static void Main()
{
int[] data = { 1, 2, 3, 1, 2, 1 };
foreach (var m in from m in data orderby m select m)
System.Console.Write(m);
}

is transformed into

static void Main()
{
int[] data = { 1, 2, 3, 1, 2, 1 };
{
    IEnumerator<int> e = ((IEnumerable<int>)(data.OrderBy(m=>m)).GetEnumerator();
    try
    {
      int m;
      while(e.MoveNext())
      {
        m = (int)(int)e.Current;
        Console.Write(m);
      }
    }
    finally
    {
      if (e != null) ((IDisposable)e).Dispose();
    }
}
}

There are five usages of m in this transformed program; it is:

1) declared as the formal parameter of a lambda.
2) used in the body of the lambda; here it refers to the formal parameter.
3) declared as a local variable
4) written to in the loop; here it refers to the local variable
5) read from in the loop; here it refers to the local variable

Is there any usage of a local variable before its declaration? No.

Are there any two declarations that have the same name in the same declaration space? It would appear so. The body of Main defines a local variable declaration space, and clearly the body of Main contains, indirectly, two declarations for m, one as a formal lambda parameter and one as a local. But I said last time that local variable declaration spaces have special rules for determining overlaps. It is illegal for a local variable declaration space to directly contain a declaration such that another nested local variable declaration space contains a declaration of the same name. But an outer declaration space which indirectly contains two such declarations is not an error. So in this case, no, there are no local variable declarations spaces which directly contain a declaration for m, such that a nested local variable declaration space also directly contains a declaration for m. Our two local variable declarations spaces which directly contain a declaration for m do not overlap anywhere.

Is there any declaration space which contains two inconsistent usages of the simple name m? Yes, again, the outer block of Main contains two inconsistent usages of m. But again, this is not relevant. The question is whether any declaration space directly containing m has an inconsistent usage. Again, we have two declaration spaces but they do not overlap each other, so there's no problem here either.

The thing which makes this legal, interestingly enough, is the generation of the loop variable declaration logically within the try block. Were it to be generated outside the try block then this would be a violation of the rule about inconsistent usage of a simple name throughout a declaration space.

Simple names are not so simple

Eric Lippert — Mon, 02 Nov 2009 14:23:00 GMT

C# has many rules that are designed to prevent some common sources of bugs and encourage good programming practices. So many, in fact, that it is often quite confusing to sort out exactly which rule has been violated. I thought I might spend some time talking about what the different rules are. We'll finish up with a puzzle.

To begin with, it will be vital to understand the difference between scope and declaration space. To refresh your memory of my earlier article: the scope of an entity is the region of text in which that entity may be referred to by its unqualified name. A declaration space is a region of text in which no two things may have the same name (with an exception for methods which differ by signature.) A "local variable declaration space" is a particular kind of declaration space used for declaring local variables; local variable declaration spaces have special rules for determining when they overlap.

The next thing that you have to understand to make any sense o this is what a "simple name" is. A simple name is always either just a plain identifier, like "x", or, in some cases, a plain identifier followed by a type argument list, like "Frob<int, string>".

Lots of things are treated as "simple names" by the compiler: local variable declarations, lambda parameters, and so on, always have the first form of simple name in their declarations. When you say "Console.WriteLine(x);" the "Console" and "x" are simple names but the "WriteLine" is not. Confusingly, there are some textual entities which have the form of simple names, but are not treated as simple names by the compiler. We might talk about some of those situations in later fabulous adventures.

So, without further ado, here are some relevant rules which are frequently confused. It's rules 3 and 4 that people find particularly confusing.

1) It is illegal to refer to a local variable before its declaration. (This seems reasonable I hope.)
2) It is illegal to have two local variables of the same name in the same local variable declaration space or nested local variable declaration spaces.
3) Local variables are in scope throughout the entire block in which the declaration occurs. This is in contrast with C++, in which local variables are in scope in their block only at points after the declaration.
4) For every occurrence of a simple name, whether in a declaration or as part of an expression, all uses of that simple name within the immediately enclosing local variable declaration space must refer to the same entity.

The purpose of all of these rules is to prevent the class of bugs in which the reader/maintainer of the code is tricked into believing they are referring to one entity with a simple name, but are in fact accidentally referring to another entity entirely. These rules are in particular designed to prevent nasty surprises when performing what ought to be safe refactorings.

Consider a world in which we did not have rules 3 and 4. In that world, this code would be legal:

class C
{
int x;
void M()
{
    // 100 lines of code
    x = 20; // means "this.x";
    Console.WriteLine(x); // means "this.x"
    // 100 lines of code
    int x = 10;
    Console.WriteLine(x); // means "local x"
}
}

This is hard on the person reading the code, who has a reasonable expectation that the two "Console.WriteLine(x)" lines do in fact both print out the contents of the same variable. But it is particularly nasty for the maintenance programmer who wishes to impose a reasonable coding standard upon this body of code. "Local variables are declared at the top of the block where they're used" is a reasonable coding standard in a lot of shops. But changing the code to:

class C
{
int x;
void M()
{
    int x;
    // 100 lines of code
    x = 20; // no longer means "this.x";
    Console.WriteLine(x); // no longer means "this.x"
    // 100 lines of code
    x = 10;
    Console.WriteLine(x); // means "local x"
}
}

changes the meaning of the code! We wish to discourage authoring of multi-hundred-line methods, but making it harder and more error-prone to refactor them into something cleaner is not a good way to achieve that goal.

Notice that the original version of this program, rule 3 means that the program violates rule 1 -- the first usage of "x" is treated as a reference to the local before it is declared. The fact that it violates rule 1 because of rule 3 is precisely what prevents it from being a violation of rule 4! The meaning of "x" is consistent throughout the block; it always means the local, and therefore is sometimes used before it is declared. If we scrapped rule 3 then this would be a violation of rule 4, because we would then have two inconsistent meanings for the simple name "x" within one block.

Now, these rules do not mean that you can refactor willy-nilly. We can still construct situations in which similar refactorings fail. For example:

class C
{
int x;
void M()
{
    {
      // 100 lines of code
      x = 20; // means "this.x";
      Console.WriteLine(x); // means "this.x"
    }
    {
      // 100 lines of code
      int x = 10;
      Console.WriteLine(x); // means "local x"
    }
}
}

This is perfectly legal. We have the same simple name being used two different ways in two different blocks, but the immediately enclosing block of each usage does not overlap that of any other usage. The local variable is in scope throughout its immediately enclosing block, but that block does not overlap the block above. In this case, it is safe to refactor the declaration of the local to the top of its block, but not safe to refactor the declaration to the top of the outermost block; that would change the meaning of "x" in the first block. Moving a declaration up is almost always a safe thing to do; moving it out is not necessarily safe.

Now that you know all that, here's a puzzle for you, a puzzle that I got completely wrong the first time I saw it:

using System.Linq;
class Program
{
static void Main()
{
    int[] data = { 1, 2, 3, 1, 2, 1 };
    foreach (var m in from m in data orderby m select m)
      System.Console.Write(m);
}
}

It certainly looks like name "m" is being used multiple times to mean different things. Is this program legal? If yes, why do the rules for not re-using simple names not apply? If no, precisely what rule has been violated?

[Eric is on vacation; this posting was pre-recorded.]

Color Color

Eric Lippert — Mon, 06 Jul 2009 16:29:00 GMT

Pop quiz: What does the following code do when compiled and run?

class C
{
    public static void M(string x)
    {
        System.Console.WriteLine("static M(string)");
    }
    public void M(object s)
    {
        System.Console.WriteLine("M(object)");
    }
}
class Program
{
    static void Main()
    {
        C c = new C();
        c.M("hello");
    }
}

(1) writes static M(string)
(2) writes M(object)
(3) uh, dude, this code doesn’t even compile much less run
(4) something else

Think about that for a bit and then try it and see if you were right.

In option (1), the compiler could decide that the best match is the static one and let you call it through the instance, ignoring the instance value. In option (2), the compiler could decide that the object version is the best match and ignore the static one, even though the argument match is better. But neither of those actually happens; the rules of C# say that the compiler should pick the best match based on the arguments, and then disallow static calls that are through an instance! The actual result here is therefore (3):

error CS0176: Member 'C.M(string)' cannot be accessed with an instance reference; qualify it with a type name instead

What is up with this craziness? If you believe that the rule “static methods should never be accessed through instances” is a good rule – and it seems reasonable – then why doesn’t overload resolution remove the static methods from the candidate set when called through an instance? Why does it even allow the “string” version to be an applicable candidate in the first place, if the compiler knows that it can never possibly work?

I agree that this seems really goofy at first.

To explain why this is not quite as dumb as it seems, consider the following variation on the problem. Class C stays the same.

class B
{
public C C = new C();
public void N()
{
C.M("hello");
}
}

What does a call to N on an instance of B do?

(1) writes static M(string)
(2) writes M(object)
(3) compilation error
(4) something else

A bit trickier now, isn’t it? Does C.M mean “call instance method M on the instance stored in this.C?” or does it mean “call static method M on type C”? Both are applicable!

Because of our goofy rule we do the right thing in this case. We first resolve the call based solely on the arguments and determine that the static method is the best match. Then we check to see if the “receiver” at the call site can be interpreted as being through the type. It can. So we make the static call and write “static M(string)”. If the instance version had been the best match then we would successfully call the instance method through the property.

So the reason that the compiler does not remove static methods when calling through an instance is because the compiler does not necessarily know that you are calling through an instance. Because there are situations where it is ambiguous whether you’re calling through an instance or a type, we defer deciding which you meant until the best method has been selected.

Now, one could make the argument that in our first case, the receiver cannot possibly be a type. We could have further complicated the language semantics by having three overload resolution rules, one for when the receiver is known to be a type, one for when the receiver is known to not be a type, and one for when the receiver might or might not be a type. But rather than complicate the language further, we made the pragmatic choice of simply deferring the staticness check until after the bestness check. That’s not perfect but it is good enough for most cases and it solves the common problem.

The common problem is the situation that arises when you have a property of a type with the same name, and then try to call either a static method on the type, or an instance method on the property. (This can also happen with locals and fields, but local and field names typically differ in case from the names of their types.) The canonical motivating example is a public property named Color of type Color, so we call this “the Color Color problem”.

In real situations the instance and static methods typically have different names, so in the typical scenario you never end up calling an instance method when you expect to call a static method, or vice versa. The case I presented today with a name collision between a static and an instance method is relatively rare.

If you’re interested in reading the exact rules for how the compiler deals with the Color Color problem, see section 7.5.4.1 of the C# 3.0 specification.

What Would Tufte Do?

Eric Lippert — Wed, 27 May 2009 17:00:00 GMT

What is this a chart of? I'll post the answer tomorrow.

UPDATE: Someone has already correctly deduced the answer. (And man, that was fast!) So don't read the comments if you don't want spoilers.

Comma Quibbling

Eric Lippert — Wed, 15 Apr 2009 17:06:00 GMT

[UPDATE: Holy goodness. Apparently this was a more popular pasttime than I anticipated. There's like a hundred solutions in there. Who knew there were that many ways to stick commas in a string? It will take me some time to go through them all, so don't be surprised if it's a couple of weeks until I get them all sorted out.]

The point of Monday’s post about comma-separated lists was not so much about the actual problem; it’s a rather trivial problem. Rather, I wanted to make two points. First, stating the actual problem rather than a much harder and more general version of the problem is likely to get you a realistic solution to your actual problem much faster. And second, reworking the statement of the problem into an equivalent but structurally different statement is a great way to see solutions that you might have otherwise missed.

But whenever I make a post illustrating such points with a specific example, lots of people pipe up with their ideas for how to solve the specific example. Which is awesome; I encourage this behaviour.

So in that spirit, here’s a slightly harder version of the string concatenation problem, just for the fun of it. Write me a function that takes a non-null IEnumerable<string> and returns a string with the following characteristics:

(1) If the sequence is empty then the resulting string is "{}".
(2) If the sequence is a single item "ABC" then the resulting string is "{ABC}".
(3) If the sequence is the two item sequence "ABC", "DEF" then the resulting string is "{ABC and DEF}".
(4) If the sequence has more than two items, say, "ABC", "DEF", "G", "H" then the resulting string is "{ABC, DEF, G and H}". (Note: no Oxford comma!)

I think you get the idea. You can post your solution in the comments or use the link on the blog page to email your solution to me.

The strings in the sequence can be assumed to be non-null but can otherwise be any string value, including empty strings or strings containing commas, braces and "and".

There’s no size limit on the sequence; it could be tiny, it could be thousands of strings. But it will be finite.

All you get are the methods of IEnumerable<string>; if you want to make that thing into a list or an array, you’re going to need to do that explicitly rather than casting it and hoping for the best.

I am particularly interested in solutions which make the semantics of the code very clear to the code maintainer.

Of course, C# is most interesting to me, but if there are neat ways to express this in other languages, I’d love to see them too.

If there are any particularly amusing or interesting implementations I’ll dissect them on the blog in a future episode, probably in a week or so. I’m not going to have time to do a detailed analysis of every one.

And… go!

Mutating Readonly Structs

Eric Lippert — Wed, 14 May 2008 17:10:00 GMT

Consider this program which attempts to mutate a readonly mutable struct. What happens?

struct Mutable {
    private int x;
    public int Mutate() {
        this.x = this.x + 1;
        return this.x;
    }
}

class Test {
    public readonly Mutable m = new Mutable();
    static void Main(string[] args) {
        Test t = new Test();
        System.Console.WriteLine(t.m.Mutate());
        System.Console.WriteLine(t.m.Mutate());
        System.Console.WriteLine(t.m.Mutate());
    }
}

There are a number of things this program could do. Does it:

1) Print 1, 2, 3 -- because m is readonly, but the "readonly" only applies to m, not to its contents.
2) Print 0, 0, 0 -- because m is readonly, x cannot be changed. It always has its default value of zero.
3) Throw an exception at runtime, when the attempt is made to mutate the contents of a readonly field.
4) Do something else

People are frequently surprised to learn that the answer is (4). In fact, this prints 1, 1, 1.

Why?

Because, remember, accessing a value type gives you a COPY of the value. When you say t.m, you get a copy of whatever is presently stored in m. m is immutable, but the copy is not. The copy is then mutated, and the value of x in the copy is returned. But m remains untouched.

The relevant section of the specification is 7.5.4, which states that when resolving "E.I" where E is an object and I is a field...

...if the field is readonly and the reference occurs outside an instance constructor of the class in which the field is declared, then the result is a value, namely the value of the field I in the object referenced by E.

The important word here is that the result is the value of the field, not the variable associated with the field. Readonly fields are not variables outside of the constructor. (The initializer here is considered to be inside the constructor; see my earlier post on that subject.)

Great. What about that second dot, as in ".Mutate()"? We look at section 7.4.4 to find out how to invoke E.M():

If E is not classified as a variable, then a temporary local variable of E's type is created and the value of E is assigned to that variable. E is then reclassified as a reference to that temporary local variable. The temporary variable is accessible as this within M, but not in any other way. Thus, only when E is a true variable is it possible for the caller to observe the changes that M makes to this.

And there you go. Value semantics are tricky!

This is yet another reason why mutable value types are evil. Try to always make value types immutable.

Why Do Initializers Run In The Opposite Order As Constructors? Part Two

Eric Lippert — Mon, 18 Feb 2008 18:25:12 GMT

As you might have figured out, the answer to last week's puzzle is "if the constructors and initializers run in their actual order then an initialized readonly field of reference type is guaranteed to be non null in any possible call. That guarantee cannot be met if the initializers run in the expected order."

Suppose counterfactually that initializers ran in the expected order, that is, derived class initializers run after the base class constructor body. Consider the following pathological cases:

class Base
{
    public static ThreadsafeCollection t = new ThreadsafeCollection();
    public Base()
    {
        Console.WriteLine("Base constructor");
        if (this is Derived) (this as Derived).DoIt();
        // would deref null if we are constructing an instance of Derived
        Blah();
        // would deref null if we are constructing an instance of MoreDerived
        t.Add(this);
        // would deref null if another thread calls Base.t.GetLatest().Blah();
        // before derived constructor runs
    }
    public virtual void Blah() { }
}
class Derived : Base
{
    readonly Foo derivedFoo = new Foo("Derived initializer");
    public DoIt()
    {
        derivedFoo.Bar();
    }
}
class MoreDerived : Derived
{
    public override void Blah() { DoIt(); }
}

Calling methods on derived types from constructors is dirty pool, but it is not illegal. And stuffing not-quite-constructed objects into global state is risky, but not illegal. I'm not recommending that you do any of these things -- please, do not, for the good of us all. I'm saying that it would be really nice if we could give you an ironclad guarantee that an initialized readonly field is always observed in its initialized state, and we cannot make that guarantee unless we run all the initializers first, and then all of the constructor bodies.

Note that of course, if you initialize your readonly fields in the constructor, then all bets are off. We make no guarantees as to the fields not being accessed before the constructor bodies run.

Next time on FAIC: how to get a question not answered.

Why Do Initializers Run In The Opposite Order As Constructors? Part One

Eric Lippert — Sat, 16 Feb 2008 02:01:00 GMT

Pop quiz!

What do you expect the output of this program to be?

using System;

class Foo
{
    public Foo(string s)
    {
        Console.WriteLine("Foo constructor: {0}", s);
    }
    public void Bar() { }
}
class Base
{
    readonly Foo baseFoo = new Foo("Base initializer");
    public Base()
    {
        Console.WriteLine("Base constructor");
    }
}
class Derived : Base
{
    readonly Foo derivedFoo = new Foo("Derived initializer");
    public Derived()
    {
        Console.WriteLine("Derived constructor");
    }
}
static class Program
{
    static void Main()
    {
        new Derived();
    }
}

I got a question from a user recently noting that the order was not as he expected. One naively expects that the order will go "base initializers, base constructor body, derived initializers, derived constructor body". In fact the order actually is that first all the initializers run in order from derived to base, and then all the constructor bodies run in order from base to derived.

The latter bit makes perfect sense; the more derived constructors may rely upon state initialized by the less derived constructors, so the constructors should run in order from base to derived. But most people assume that the call sequence of the code above is equivalent to this pseudocode:

// Expected
BaseConstructor()
{
    ObjectConstructor();
    baseFoo = new Foo("Base initializer");
    Console.WriteLine("Base constructor");
}
DerivedConstructor()
{
    BaseConstructor();
    derivedFoo = new Foo("Derived initializer");
    Console.WriteLine("Derived constructor");
}

When in point of fact it is equivalent to this:

// Actual
BaseConstructor()
{
    baseFoo = new Foo("Base initializer");
    ObjectConstructor();
    Console.WriteLine("Base constructor");
}
DerivedConstructor()
{
     derivedFoo = new Foo("Derived initializer");
    BaseConstructor();
    Console.WriteLine("Derived constructor");
}

That explains the mechanism whereby the initializers run in order from derived to base and the constructor bodies run in the opposite order, but why did we choose to implement that mechanism instead of the more intuitively obvious former way?

Puzzle that one over for a bit, and then read on for a hint.

...

The "readonly" modifiers in there were no accident. The code gives the appearance that any call to derivedFoo.Bar() and baseFoo.Bar() should never fail with a null dereference exception because both are readonly fields initialized to non-null values.

Is that appearance accurate, or misleading?
Now suppose initializers ran in the "expected" order and answer question (1) again.

I'll post the answers and analysis next week. Have a fabulous weekend!

Psychic Debugging, Part Two

Eric Lippert — Thu, 06 Sep 2007 18:31:00 GMT

A number of readers have the mysterious fifth sense which gives them the ability to deduce that the GetBars method from yesterday's post contains a yield return and is therefore an iterator. Remember, as the standard states (in section 10.14.4):

[...] execution of the code in the iterator block occurs when the enumerator object's MoveNext method is invoked.

Since the test program does not invoke MoveNext, the check for null is never executed, and therefore the exception is never thrown.

Since most of the interesting new sequence operators available in C# 3.0 are implemented with iterators it probably will be increasingly important for developers to understand a bit more about how iterators work behind the scenes. I may do some blog posts on that over the next little while.

Psychic Debugging, Part One

Eric Lippert — Wed, 05 Sep 2007 20:30:00 GMT

Here is a compiler bug report I got the other day. The user is trying to write a unit test for a method which takes a Foo and returns a collection of Bars. The test is supposed to confirm that GetBars throws an exception if the argument is null. The test was failing with “got no exception”. The user was wondering if somehow the compiler had optimized away the call.

    static void Test() {
        bool gotException = false;
        try {
            IEnumerable<Bar> bars = Foo.GetBars(null);
        }
        catch (ArgumentNullException ex) {
            Console.WriteLine("SUCCESS: Got expected exception");
            gotException = true;
        }
        catch (Exception ex) {
            Console.WriteLine("FAILURE: Got unexpected exception");
            gotException = true;
        }
        finally {
            if (!gotException)
                Console.WriteLine("FAILURE: Got no exception");
        }
    }

Several people (including myself and Cyrus) psychically debugged this one independently. The user did not include the source code of GetBars, which is what necessitated the use of our psychic powers. Because I am a nice guy, I will give you the beginning:

    static public IEnumerable<Bar> GetBars(Foo foo) {
        if (foo == null)
            throw new ArgumentNullException("foo");

Our psychic powers correctly told us that the behaviour of the test program is correct and expected. What do your psychic powers tell you about the rest of the implementation of GetBars? What is going on here?

An Inheritance Puzzle, Part Two

Eric Lippert — Mon, 30 Jul 2007 17:00:00 GMT

Today, the answer to Friday's puzzle. It prints "Int32". But why?

Some readers hypothesized that M would print out "Int32" because the declaration B : A<int> somehow tells B that T is to be treated as int, now and forever. Though the answer is right, the explanation is not quite right. One can illustrate this by taking C out of the picture. If you say (new A<string>.B()).M(), you'll get "String". The fact that B is an A<int> doesn't make T always int inside B!

The always-keen Stuart Ballard was the first to put his finger upon the real crux of the problem -- but he's seen this problem before, so he had an advantage. Eamon Nerbonne was the first to post a complete and correct explanation.

The really thorny issue here is that the declaration class C : B is upon close inspection, somewhat ambiguous. Is that equivalent to class C : A<T>.B or class C : A<int>.B?

Clearly it matters which we choose. A method group can only be treated as a member if the method is on a base class. Merely being on an outer class doesn't cut it:

public class X { public void M(){} }
public class Y {
public void N() {}
public class Z : X {}
}
...
(new Y.Z()).M(); // legal, from base class
(new Y.Z()).N(); // illegal -- outer class members are not members of inner classes.

In our example, when we called M on an instance of A<string>.B.C, it was calling a method of the base class of C. If the base class of C is A<T>.B, then that should call A<T>.B.M, and it should print out whatever the current value of T is -- in this case, "String". If the base class of C is A<int>.B, then that should call A<int>.B.M, so it should print out "Int32".

We choose the latter as the base class. That was certainly a surprise to me. And Stuart Ballard. And, amusingly enough, when I sprang this one upon Anders and didn't give him time to think about it carefully, it was a surprise to him as well. When I sprang it on Cyrus, he cheerfully pointed out that he already posted a harder version of this problem back in 2005, the solution of which Stuart characterized back then as "Insanely complex, but it makes perfect sense." I couldn't agree more, though at least my version of the puzzle is somewhat simpler.

Anyway, why on earth ought that to be the case? Surely the B in class C : B means the immediately containing class, which is A<T>.B, not A<int>.B! And yet it does not.

These generics are screwing up our intuitions. Let's look at an example which has no generics at all:

public class D {
    public class E {}
}
public class F {
    public class E { }
    public class G {
        public E e; // clearly F.E
    }
}
public class H : D {
    public E e; // clearly D.E
}

This is all legal so far, and should be pretty clear. When we are binding a name to a type, the type we get is allowed to be a member of any base class or a member of any outer class. But what if we have both to choose from?

public class J {
    public class E {}
    public class K : D {
        public E e; // Is this J.E or D.E?
    }
}

We could just throw up our hands and say that this is ambiguous and therefore illegal, but we'd rather not do that if we can avoid it. We have to prefer one of them, and we've decided that we will give priority to base classes over outer classes. Derived classes have an "is a kind of" relationship with their base classes, and that is logically a "tighter" binding than the "is contained in" relationship that inner classes have with outer classes.

Another way to think about it is that all the members you get from your base class are all "in the current scope"; therefore all the members you get from outer scopes are given lower priority, since stuff inside inner scopes takes priority over stuff in outer scopes.

The algorithm we use to search for a name used in the context of a type S is as follows:

search S's type parameters
search S's accessible inner classes
search accessible inner classes of all of S's base classes, going in order from most to least derived
S←S's outer class, start over

(And if that fails then we invoke the whole mechanism of searching the namespaces that are in scope, checking alias clauses, etc.)

With that in mind, now the solution should finally make some sense. At the point where we are resolving the base class of C we know that C has no type parameters. We do not know what the base class or inner classes of C are -- that's what we're trying to figure out -- so we skip checking them.

The next thing we check is the outer class, which is A<T>.B, but we do NOT say, aha, the outer class is called B, we're done. That is not at all what the algorithm above says. Instead, it says check A<T>.B to see if it has a type variable called B or an inner type called B. It does not, so we keep searching.

The base type of A<T>.B is A<int>. The outer type of A<T>.B is A<T>. Both have an accessible inner class called B. Which do we pick? The base type gets searched first, so B resolves to A<int>.B. Obviously.

Having members of base classes bind tighter than members from outer scopes can lead to bizarre situations but they are generally pretty contrived. For example:

public class K { public class L {} }
public class L : K {
L myL; // this is K.L!
}

And of course, you can always get around these problems by eliminating the ambiguity:

public class A<T> {
public class B : A<int> {
public void M() { ... }
public class C : A<T>.B { } // no longer ambiguous which B is picked.

Finally, why did I say that you couldn't appeal to the standard? When I wrote that I actually thought that the standard was wrong, but upon a closer reading, I see that it is correct. It's just really hard to read:

Otherwise, if T contains a nested accessible type having name I and K type parameters, then the namespace-or-type-name refers to that type constructed with the given type arguments. If there is more than one such type, the type declared within the more derived type is selected.

By "if T contains a nested accessible type", it means "if T, or any of its base classes, contains a nested accessible type". I completely failed to comprehend that the first n times I read that section. I'll see if I can get that clarified in the next version of the standard.

An Inheritance Puzzle, Part One

Eric Lippert — Fri, 27 Jul 2007 19:36:00 GMT

Once more I have returned from my ancestral homeland, after some weeks of sun, rain, storms, wind, calm, friends and family. I could certainly use another few weeks, but it is good to be back too.

Well, enough chit-chat; back to programming language design. Here's an interesting combination of subclassing with nesting. Before trying it, what do you think this program should output?

public class A<T> {
    public class B : A<int> {
        public void M() {
            System.Console.WriteLine(typeof(T).ToString());
        }
        public class C : B { }
    }
}
class MainClass {
    static void Main() {
        A<string>.B.C c = new A<string>.B.C();
        c.M();
    }
}

Should this say that T is int, string or something else? Or should this program not compile in the first place?

It turned out that the actual result is not what I was expecting at least. I learn something new about this language every day.

Can you justify either the actual behaviour or what you believe the behaviour should be? Note that appealing to the C# standard will not help you here because the standard gets it wrong.

I'll discuss how exactly the standard gets it wrong and what the justification is for the actual behaviour next time on FAIC.

[UPDATE: I have reviewed the standard more carefully. The standard is actually correct, but really hard to read. I'll post a full analysis on Monday.]

Even More Conversion Trivia, Part Two

Eric Lippert — Tue, 01 May 2007 17:24:00 GMT

Reader Barry Kelly came up with the solution I was thinking of for my trivia question yesterday. If the value in question is, say, a nullable integer which has no value then casting it explicitly to object results in the null object. Calling a virtual method on a null object results in an exception at runtime. However, using it implicitly as an object boxes up the nullable int as a valid ValueType, which overrides ToString. So that's the difference: the one without the cast produces an empty string, the one with the cast throws an exception.

Honourable mention to reader Bill Wagner, who points out that if the object implements an interface with a ToString method then there are two ToString slots, one from the base class and one from the interface. Casting to object can cause overload resolution to pick the base version over the interface version, which might otherwise be preferred. (This reminds me of another fun fact about C# overload resolution which I will blog about at a later date.)

As Barry Kelly points out, the CLR took a design change with Visual Studio 2005 Beta 2 to implement the special case that a nullable without a value becomes a null ValueType rather than a boxed ValueType when boxed. How then do we implement the desired behaviour, ie, that calling ToString without a cast creates a boxed ValueType regardless of whether the nullable has a value?

Fortunately the CLR implemented a special kind of virtual call, called the "constrained" virtual call. Without the ability to constrain the transformation of a value to a particular type when making a virtual call, certain operations become difficult to codegen. (In particular, it is hard to do virtual calls on type parameters of generic types, because you might not know whether the type parameter is going to be a value type or a reference type under construction.) If you take a look at the IL we generate for something like foo.ToString() on a nullable int, it looks like:

ldsflda valuetype [mscorlib]System.Nullable`1<int32> Test::foo
constrained. valuetype [mscorlib]System.Nullable`1<int32>
callvirt instance string [mscorlib]System.Object::ToString()

Which tells the CLR to box up that thing as is, rather than doing a boxing operation. Had we inserted a cast in there we would have generated

ldsfld     valuetype [mscorlib]System.Nullable`1<int32> Test::foo
box        valuetype [mscorlib]System.Nullable`1<int32>
callvirt   instance string [mscorlib]System.Object::ToString()

instead.

Even More Conversion Trivia

Eric Lippert — Tue, 01 May 2007 00:10:00 GMT

I learn something new about C# every day. This is a subtle and tricky language.

Pop quiz: foo is of a type which has a base class which has made a normal, everyday virtual override of System.Object.ToString. There are no additional overrides, hides, etc of ToString anywhere in foo's inheritance hierarchy. No ToString extension methods, etc. Under what circumstances do foo.ToString() and ((object)foo).ToString() produce different results? Remember, it is a virtual function, so the cast should not make a difference in what function is actually called.

As it turns out, my mistaken belief that the two of these have the same semantics caused a bug in the part of the compiler where we translate an expression tree lambda into a call to the Lambda<T> factory. That in turn was masking an even more subtle bug in the dynamic expression tree compiler.

Anyone out there have a guess as to what type and value of foo produces different behaviour here? I'll post the answer next time along with some discussion of the codegen issue.