Fabulous Adventures In Coding : Breaking Changes

Closing over the loop variable, part two

Eric Lippert — Mon, 16 Nov 2009 16:18:00 GMT

Thanks to everyone who left thoughtful and insightful comments on last week's post.

More countries really ought to implement Instant Runoff Voting; it would certainly appeal to the geek crowd. Many people left complex opinions of the form "I'd prefer to make the change, but if you can't do that then make it a warning". Or "don't make the change, do make it a warning", and so on. But what I can deduce from reading the comments is that there is a general lack of consensus on what the right thing to do here is. In fact, I just did a quick tally:

Commenters who expressed support for a warning: 26
Commenters who expressed the sentiment "it's better to not make the change": 24
Commenters who expressed the sentiment "it's better to make the change": 25

Wow. I guess we'll flip a coin. :-) (*)

Four people suggested to actually make it an error to do this. That's a pretty big breaking change, particularly since we would be breaking not just "already broken" code, but plenty of code that works perfectly well today -- see below. That's not likely to happen.

People also left a number of interesting suggestions. I thought I'd discuss some of those a little bit.

First off, I want to emphasize that what we're attempting to address here is the problem that the language encourages people to write code that has different semantics than they think it has. The problem is NOT that the language has no way to express the desired semantics; clearly it does. Just introduce a new variable explicitly into the loop.

A number of suggestions were for ways that the language could more elegantly express that notion. Some of the suggestions:

foreach(var x in c) inner
foreachnew(var x in c)
foreach(new var x in c)
foreach(var x from c)
foreach(var x inside c)

Though we could do any of those, none of them by themselves solve the problem at hand. Today, you have to know to use a particular pattern with foreach to get the semantics you want: declare a variable inside the loop. With one of these changes, you still have to know to use a particular keyword to get the semantics you want, and it is still easy to accidentally do the wrong thing.

Furthermore, a change so small and so targetted at such a narrow scenario probably does not provide enough benefit to justify the large cost of creating a new syntax, particularly one which is still easily confused with an existing syntax.

C++ luminary Herb Sutter happened to be in town and was kind enough to stop by my office to describe to me how they are solving a related problem in C++. Apparently the next version of the C++ standard will include lambdas, and they're doing this:

[q, &r] (int x) -> int { return M(x, q, r); }

This means that the lambda captures outer variable q by value, captures r by reference, takes an int and returns an int. Whether the lambda captures values or references is controllable! An interesting approach but one that doesn't immediately solve our problem here; we cannot make lambdas capture by value by default without a huge breaking change. Capturing by value would have to require new syntax, and then we're in the same boat again: the user has to know to use the new syntax when in a foreach loop.

A number of people also asked what the down sides of adding a warning are. The down side is that a warning which warns about correct behaviour is a very bad warning; it makes people change working code, and frequently they break working code in order to eliminate a warning that shouldn't have been present in the first place. Consider:

foreach(var insect in insects)
{
var query = frogs.Where(frog=>frog.Eats(insect));
Console.WriteLine("{0} is eaten by {1} frogs.", insect, query.Count());
}

This makes a lambda closed over insect; the lambda never escapes the loop, so there's no problem here. But the compiler doesn't know that. The compiler sees that the lambda is being passed to a method called Where, and Where is allowed to do anything with that delegate, including storing it away to be called later. Which is exactly what Where does! Where stores away the lambda into a monad that represents the execution of the query. The fact that the query object doesn't survive the loop is what keeps this safe. But how is the compiler supposed to suss out that tortuous chain of reasoning? We'd have to give a warning for this case, even though it is perfectly safe.

It gets worse. A lot of people are required by their organizations to compile with "warnings are errors" turned on. Therefore, any time we introduce a new warning for a pattern that is often actually safe and frequently used, we are effectively causing an enormous breaking change. A vaccine which kills more healthy people than the disease would have is probably not a good bet. (**)

This is not to say that a warning is a bad idea, but that it is not the obvious slam dunk good idea that it initially appears to be.

A number of people suggested that the problem was in the training of the developers, not in the design of the language. I disagree. Obviously modern languages are complex tools that require training to use, but we are working hard to make a language where people's natural intuitions about how things work lead them to write correct code. I have myself made this error a number of times, usually in the form of writing code like the code above, and then refactoring it in such a manner that suddenly some part of it escapes the loop and the bug is introduced. It is very easy to make this mistake, even for experienced developers who thoroughly understand closure semantics. That's a flaw in the design of the language.

And finally, a number of people made suggestions of the form "make it a warning in C# 4, and an error in C# 5", or some such thing. FYI, C# 4 is DONE. We are only making a few last-minute "user is electrocuted"-grade bug fixes, mostly based on your excellent feedback from the betas. (If you have bug reports from the beta, please keep sending them, but odds are good they won't get fixed for the initial release.) We are certainly not capable of introducing any sort of major design change or new feature at this point. And we try to not introduce semantic changes or new features in service packs. We're going to have to live with this problem for at least another cycle, unfortunately.

********

(*) Mr. Smiley Face indicates that Eric is indulging in humourous japery.

(**) I wish to emphasize that I am 100% in favour of vaccinations for deadly infectious diseases, even vaccines that are potentially dangerous. The number of people made ill or killed by the smallpox vaccine was tiny compared to the number of people who did not contract this deadly, contagious (and now effectively extinct) disease as a result of mass vaccination. I am a strong supporter of vaccine research. I'm just making an analogy here people.

Closing over the loop variable considered harmful

Eric Lippert — Thu, 12 Nov 2009 14:50:00 GMT

I don't know why I haven't blogged about this one before; this is the single most common incorrect bug report we get. That is, someone thinks they have found a bug in the compiler, but in fact the compiler is correct and their code is wrong. That's a terrible situation for everyone; we very much wish to design a language which does not have "gotcha" features like this.

But I'm getting ahead of myself. What's the output of this fragment?

var values = new List<int>() { 100, 110, 120 };
var funcs = new List<Func<int>>();
foreach(var v in values)
funcs.Add( ()=>v );
foreach(var f in funcs)
Console.WriteLine(f());

Most people expect it to be 100 / 110 / 120. It is in fact 120 / 120 / 120. Why?

Because ()=>v means "return the current value of variable v", not "return the value v was back when the delegate was created". Closures close over variables, not over values. And when the methods run, clearly the last value that was assigned to v was 120, so it still has that value.

This is very confusing. The correct way to write the code is:

foreach(var v in values)
{
var v2 = v;
funcs.Add( ()=>v2 );
}

Now what happens? Every time we re-start the loop body, we logically create a fresh new variable v2. Each closure is closed over a different v2, which is only assigned to once, so it always keeps the correct value.

Basically, the problem arises because we specify that the foreach loop is a syntactic sugar for

{
 IEnumerator<int> e = ((IEnumerable<int>)values).GetEnumerator();
 try
 {
 int m; // OUTSIDE THE ACTUAL LOOP
 while(e.MoveNext())
 {
 m = (int)(int)e.Current;
 funcs.Add(()=>m);
 }
 }
 finally
 {
 if (e != null) ((IDisposable)e).Dispose();
 }
}

If we specified that the expansion was

    try
    {
      while(e.MoveNext())
      {
      int m; // INSIDE
        m = (int)(int)e.Current;
        funcs.Add(()=>m);
      }

then the code would behave as expected.

It's compelling to consider fixing this for a hypothetical future version of C#, and I'd like to hear your feedback on whether we should do so or not. The reasons FOR making the change are clear; this is a big confusing "gotcha" that real people constantly run into, and LINQ, unfortunately, only makes it worse, because it is likely to increase the number of times a customer is going to use a closure in a loop. Also, it seems reasonable that the user of the foreach loop might think of there being a "fresh" loop variable every time, not just a fresh value in the same old variable. Since the foreach loop variable is not mutable by user code, this reinforces the idea that it is a succession of values, one per loop iteration, and not "really" the same variable over and over again. And finally, the change has no effect whatsoever on non-closure semantics. (In fact, in C# 1 the spec was not clear about whether the loop variable went inside or outside, since in a world without closures, it makes no difference.)

But that said, there are some very good reasons for not making this change.

The first reason is that obviously this would be a breaking change, and we hates them, my precious. Any developers who depend on this feature, who require the closed-over variable to contain the last value of the loop variable, would be broken. I can only hope that the number of such people is vanishingly small; this is a strange thing to depend on. Most of the time, people do not expect or depend on this behaviour.

Second, it makes the foreach syntax lexically inconsistent. Consider foreach(int x in M()) The header of the loop has two parts, a declaration int x and a collection expression, M(). The int x is to the left of the M(). Clearly the M() is not inside the body of the loop; that thing only executes once, before the loop starts. So why should something to the collection expression's left be inside the loop? This seems inconsistent with our general rule that stuff to the left logically "happens before" stuff to the right. The declaration is lexically NOT in the body of the loop, so why should we treat it as though it were?

Third, it would make the "foreach" semantics inconsistent with "for" semantics. We have this same problem in "for" blocks, but "for" blocks are much looser about what "the loop variable" is; there can be more than one variable declared in the for loop header, it can be incremented in odd ways, and it seems implausible that people would consider each iteration of the "for" loop to contain a fresh crop of variables. When you say for(int i; i < 10; i += 1) it seems dead obvious that the "i += 1" means "increment the loop variable" and that there is one loop variable for the whole loop, not a new fresh variable "i" every time through! We certainly would not make this proposed change apply to "for" loops.

And fourth, though this is a nasty gotcha, there is an easy workaround, and tools like ReSharper detect this pattern and suggest how to fix it. We could take a page from that playbook and simply issue a compiler warning on this pattern. (Though adding new warnings brings up a whole raft of issues of its own, which I might get into in another post.) Though this is vexing, it really doesn't bite that many people that hard, and it's not a big deal to fix, so why go to the trouble and expense of taking a breaking change for something with an easy fix?

Design is, of course, the art of compromise in the face of many competing principles. "Eliminate gotchas" in this case directly opposes other principles like "no breaking changes", and "be consistent with other language features". Any thoughts you have on pros or cons of us taking this breaking change in a hypothetical future version of C# would be greatly appreciated.

The void is invariant

Eric Lippert — Mon, 29 Jun 2009 17:30:00 GMT

[UPDATES below]

A while back I described a kind of variance that we’ve supported since C# 2.0. When assigning a method group to a delegate type, such that both the selected method and the delegate target agree that their return type is a reference type, then the conversion is allowed to be covariant. That is, you can say:

Giraffe GetGiraffe() { … }
…
Func<Animal> f = GetGiraffe;

This works logically because anyone who calls f must be able to handle any animal that comes back. The actual method claims to only return animals, and in fact, makes the stronger claim to only return giraffes.

This works out in the CLR because the bits that make up a reference to an instance of Giraffe are exactly the same bits that make up a reference to that Giraffe interpreted as an instance of Animal. We can allow this magical conversion to happen because the CLR guarantees that it will all just work out without going in there and having to futz around with the bits.

This is why this trick only works with reference types. A method that returns, say, a double cannot be converted via a covariant conversion to a delegate type that expects the method to return an object. Somewhere there would have to be code emitted that takes the returned double and boxes it to object; the bits of a double and the bits of a reference to an object boxing a double are completely different.

But why doesn’t this trick work with void types? Here we have a method that returns some sort of success or failure code. Maybe we don’t care what it returns.

static bool DoSomething(bool b)
{
if (b) return DoTheThing();
else return DoTheOtherThing();
}

Action<bool> action = DoSomething;

This doesn’t work. Why not? The caller of the action is not even going to use the returned value, so it doesn’t matter one bit what it is! Shouldn’t “void” be considered a supertype of all possible types for the purposes of covariant return type conversions from method groups to delegate types?

No, and I’ll tell you why.

Consider what happens when you do this:

bool x = DoSomething(true);

We spit out IL that does the following:

(1) put true on the IL stack – the stack gets one deeper
(2) call DoSomething – the argument is removed from the stack and the return value is placed on the stack. Net, the stack stays the same size as before
(3) stuff whatever on top of the stack into local variable x – the stack now returns to its original depth.

Now consider what happens when you do this:

DoSomething(true);

We spit out IL that does the first two steps as before. But we cannot stop there! There is now a bool on the IL stack which needs to be removed. We generate a pop instruction to represent the fact that the returned bool has been discarded.

Now consider what happens when you do this:

action(true);

The compiler believes that action is a void-returning method, so it does not generate a pop instruction. If we allowed you to stuff DoSomething into the action, then we would be allowing you to misalign the IL stack!

But didn’t I say “the stack is an implementation detail?” Yes, but that’s a different stack. The CLI specification describes a “virtual machine” which passes around arguments and returned values on a stack. An implementation of the CLI is required to make something that behaves like the specified machine, but it is not required to do so in any particular manner. It is not required to use the million-bytes-per-thread stack supplied to each thread by the operating system as its implementation of the IL stack; that’s a convenient structure to use, of course, but it’s an implementation detail that it does so.

(As an aside: when we implemented the script engines, we also first specified our own private stack-based virtual machine. When we implemented it, we decided to put the information about “return addresses” – that is, “what code do I run next?” on the system stack, but we put arguments and return values of script functions in a stack-shaped block of memory that we allocated on our own. This made building the JScript garbage collector easier.)

In practice, the jitter uses the system stack for some things and registers for other things. Return values are actually often sent back in a register, not on the stack. But that implementation detail doesn’t help us out when deciding what the conversion rules are; we have to assume that the implementation can do no more than what the CLI specification says. Had the CLI specification said “the returned value of any function is passed back in a ‘virtual register’” rather than having it pushed onto the stack, then we could have made void-returning delegates compatible with functions that returned anything. You can always just ignore the value in the register. But that’s not what the CLI specified, so that’s not what we can do.

[UPDATE]

A number of people have asked in the comments why we do not simply generate a helper method that does what you want. That is, when you say

Action<bool> action = DoSomething;

realize that as

static void DoSomethingHelper(bool b)
{
bool result = DoSomething(b); // result is ignored
}
...
Action<bool> action = DoSomethingHelper;

We could do that. But where would you like the line to be drawn? Should you be able to assign a reference to a method that returns an int to a Func<Nullable<int>>? We could spit a helper method that converts the int to a nullable int. What about Func<double>? We could spit a helper method that converts the int to a double. What about Func<object>? We could spit a helper method that boxes the int, unexpectedly allocating memory off the heap every time you call it. What about a Func<Foo> where there is a user-defined implicit conversion from int to Foo?

We could be spitting arbitrarily complex fixer-upper methods that would seamlessly "do what you meant to say", and we have to stop somewhere. The exact semantics of what we do and do not fix up would have to be designed, specified, implemented, tested, documented, shipped to customers and maintained forever. Those are costs. Plus, every time we add a new conversion rule to the language we add breaking changes. The costs of those breaking changes to our customers have to be factored in.

But more fundamentally, one of the design principles of C# is "if you say something wrong then we tell you rather than trying to guess what you meant". JScript is deliberately a "muddle on through and do the best you can" language; C# is not. If what you want to do is make a delegate to a helper method then you express that intention by going right ahead and making that method.

Reserved and Contextual Keywords

Eric Lippert — Mon, 11 May 2009 18:07:00 GMT

Many programming languages, C# included, treat certain sequences of letters as “special”.

Some sequences are so special that they cannot be used as identifiers. Let’s call those the “reserved keywords” and the remaining special sequences we’ll call the “contextual keywords”. They are “contextual” because the character sequence might one meaning in a context where the keyword is expected and another in a context where an identifier is expected.*

The C# specification defines the following reserved keywords:

abstract as base bool break byte case catch char checked class const continue decimal default delegate do double else enum event explicit extern false finally fixed float for foreach goto if implicit in int interface internal is lock long namespace new null object operator out override params private protected public readonly ref return sbyte sealed short sizeof stackalloc static string struct switch this throw true try typeof uint ulong unchecked unsafe ushort using virtual void volatile while

The implementation also reserves the magic keywords __arglist __makeref __reftype __refvalue which are for obscure scenarios that I might blog about in the future.

Those are the keywords that we reserved in C# 1.0; no new reserved keywords have been added since. It is tempting to do so, but we always resist. Were we to add a new reserved keyword then any program that used that keyword as an identifier would break upon recompilation. Yes, you can always use a keyword as an identifier if you really want: @typeof @goto = @for.@switch(@throw); is perfectly legal, though more than a little weird. But we prefer to avoid as many breaking changes as possible.

We also have a whole bunch of contextual keywords.

The “preprocessor” † uses all the directives (#define, and so on) which of course were never valid identifiers in the first place. But it also uses contextual keywords hidden default disable restore checksum.

C# 1.0 had contextual keywords get set value add remove.

C# 2.0 added where partial global yield alias.

C# 3.0 added from join on equals into orderby ascending descending group by select let var.

C# 4.0 will add dynamic.

Every time we add one of these we need to carefully design the grammar so that if possible, the use of the new contextual keyword does not possibly change the meaning of an existing program which used it.

For example, when defining a partial class, the partial must go immediately before the class. Since there was never a legal C# 1.0 program where partial appeared immediately before class, we knew that adding this new feature to the grammar would not possibly break any existing programs.

Or, another example. Consider var x = 1; – that could have been a legal C# 2.0 program if there was a type called var with a user-defined implicit conversion from int. The semantic analyzer for declaration statements checks to see whether there is a type called var that is accessible at the declaration; if there is then the normal declaration rules are used. Only if there is not such a type can we do the analysis as an implicitly typed local declaration.

One might wonder why on earth we added five contextual keywords to C# 1.0, when there was no chance of breaking backwards compatibility. Why not just make get set value add remove into “real” keywords?

Because we could easily get away with making them contextual keywords, and it seemed likely that real people would want to name variables or methods things like get, set, value, add or remove. So we left them unreserved as a courtesy.

Those were easy to make contextual, unlike, say, return. That’s a lot harder to make a contextual keyword because then return (10); would be ambiguous; is that calling the method named “return” or returning ten? So we didn’t make any of the other reserved keywords into contextual keywords.

*******

(*) An unfortunate consequence of this definition is that using is said to be a reserved keyword even though its meaning depends on its context; whether the using begins a directive or a statement determines its meaning.

(†) An unfortunate name, since “preprocessing” is not done before regular language processing. In C#, the so-called “preprocessing” happens during lexical analysis.

Future-Proofing A Design

Eric Lippert — Fri, 16 Jan 2009 20:21:00 GMT

Last time on FAIC a user asked for guidance on the potential pitfalls of refactoring an automatic property into a regular explicit property. This is just an example of a far more general problem: how can we design programs so that they are easy to get right when things inevitably change in the future?

This is an incredibly difficult question to answer, one which whole books could be written on. Today, I'll just give three general points to think about.

First: Premature generality is expensive.

Designing code to be future-proof without a clear understanding of what the future holds often leads to excess generality; generality has very real costs, and they might be higher than the benefit accrued.

If you do a good job of solving the problem at hand with the tools at hand, if the code is clean and organized and has few moving parts, then it will be easier to generalize it in the future if you need to in order to solve a problem then.

Design your code to solve the well-understood problem. If you design it to solve an unknown future problem, odds are good that you're going to solve it poorly, making even more work for the future.

Second: Represent in your model only those things which are always in the problem domain and whose class relationships are unchanging.

The relationships between the classes TrafficLight, TrafficSign, Vehicle, Car, Truck, Pedestrian, Roadway, Intersection, SignalTiming and RightOfWayLaw are likely to be eternal and unchanging.

Things change. The semantics of a particular RightOfWayLaw might change subtly or grossly as the city council does its job. There might be new subclasses of Vehicle created in the future. A specific traffic light might have a faulty implementation that the rest of the system needs to handle gracefully. But each of these situations is about changing the implementation details of a class, never changing its relationships with other classes.

If the class relationships are future-proofed then it is a lot easier to edit the implementations of those classes without having to worry that the whole system will thereby get messed up.

The easiest way to keep the relationships straight is to base them as much as possible on concepts directly from the problem domain.

Third: Keep your policies away from your mechanisms.

The mechanism of a traffic light works the same way no matter what policies like light timings at rush hour are. If mechanisms (TrafficLight) are separated from policies (SignalTiming) then you can change the mechanism to a more efficient one without worrying that you’re going to inadvertently change policy, and can change policy without worrying that you’re going to break mechanism.

My earlier example of a bank balance was deliberately an example of what goes wrong when mechanism and policy become conflated. The “Balance” getter was originally a mechanism, but after the edit, it became a security policy enforcement tool.

You must then ask yourself, “where in this code do I care about enforcing the policy, and where do I care about executing the mechanism?” and make the appropriate edits. After the edit, everywhere that needs policy enforcement needs to use “Balance”, everywhere that needs to effect the action of the mechanism needs to use “balance”.

What are the odds that there is going to be a bug introduced, given that we now have a difference which is important and almost invisible? Pretty high! This difference should probably be made more visible by renaming the backing store to something that calls it out as semantically different from the property accessor.

Let's bring this back to the question at hand. What guidance do we propose for future-proofing turning automatic properties into regular properties? The guidance that I am proposing here is:

First, think about whether making the design general now to solve an unknown problem that you might not even have in the future is worthwhile.
Second, make sure that the design in general makes the relationships between classes unchanging, even if the specifics of each class change.
Third, consider whether that hypothetical future change to the property will be splitting apart its role as a mechanism from its role as an enforcer of policy; if this is going to be an important distinction, then consider getting that distinction into the class implementation early so that you don’t have to retrofit it in a tedious and error-prone manner later.

Automatic vs Explicit Properties

Eric Lippert — Wed, 14 Jan 2009 20:11:25 GMT

Here's a question I got from a C# user last year, a question I get fairly frequently:

User: With “regular” explicit properties, I tend to use the private backing field directly from within the class. Of course, with an automatic property, you can’t do this. My concern is that in the future, if I decide I need an explicit property for whatever reason, I’m left with the choice of changing the class implementation to use the new private field, or continue going through the property. I’m not sure what the right thing to do in this case is.

You say “for whatever reason”, and that's key. The answer to your question will depend entirely upon what the reason was that motivated the change.

If the reason that motivated the change from automatically implemented property to explicitly implemented property was to change the semantics of the property then you should evaluate whether the desired semantics when accessing the property from within the class are identical to or different from the desired semantics when accessing the property from outside the class.

If the result of that investigation is “from within the class, the desired semantics of accessing this property are different from the desired semantics of accessing the property from the outside”, then your edit has introduced a bug. You should fix the bug. If they are the same, then your edit has not introduced a bug; keep the implementation the same.

That is a bit abstract. Let’s get more concrete. Suppose you have

sealed class BankAccount
{
public decimal Balance { get; set; }

and somewhere in the class you have a calculation:

if (Balance > 0) …

Then one day you decide to change the semantics of Balance:

sealed class BankAccount
{
private decimal balance;
public decimal Balance
{
    get
    {
      if (!CurrentUser.HasReadAccess(this))
        throw new SecurityException();
      return balance;
…

OK, you’ve changed the semantics of Balance. Now, everywhere inside your class, do you intend all calculations that at present use Balance to have these security semantics, or do you intend them to not have these security semantics?

If the former, keep the calling code the way it is; it is correct; changing it will introduce a bug.

If the latter, change it; it is now wrong; changing it will fix a bug.

If the reason that you changed the property from automatic to explicit was NOT to modify the semantics then… then… then why on Earth did you change it? And why are you contemplating making further inessential changes?

I presume that in this case there must be some motivation for changing correct working code; if you have some motivation that is causing you to change correct, working code, I suppose it can be applied equally well to all the other correct, working code in the class that calls that property.

User: Thanks. This is on my mind because I’m starting my first C# 3.0 project, and am experimenting with the new features. The scenario that tripped me up is exactly as you state; someone making a future semantic change.

Then your question is actually an instance of a more general question: “how do I future-proof a design?” That is, how do I design and implement a class hierarchy now so that inevitable changes in the future are easier and less bug-prone?

That's a hard question, one which someone more knowledgeable than I am could write a whole book on. Next time on FAIC I'll give a few musings on this topic.

Why Can't I Access A Protected Member From A Derived Class, Part Two: Why Can I?

Eric Lippert — Sat, 29 Mar 2008 02:30:43 GMT

This is a follow-up to my 2005 post on the same subject which I believe sets a personal record for the longest time between parts of a series. (Of course, I didn't know it was a series when I started it.) Please read the previous article in this series, as this post assumes knowledge of part one.

.......

OK, now that you've read that, it's clear why you can only access a protected member from an instance of an object known to be of a type at least as derived as the current context. You can therefore deduce the answer to the question asked to me by a (very polite) reader this morning: Why did this code compile in C# 2.0 but give an error in C# 3.0?

public abstract class Item{
private Item _parent;
public Item Parent {
 get { return _parent; }
 protected set { _parent = value; }
}
}
public class Bag:Item{
private List<Item> list = new List<Item>();
public void Add(Item item)
{
 item.Parent = this; // Error in C# 3.0
 list.Add(item);
}
}
public class Torch : Item { }

It compiled in C# 2.0 because the compiler had a bug. We forgot to enforce the semantics of "protected" access for the property setter. Though the compiler did not generate an error, it did generate code which would not pass the CLR verification check! The code would run if it happened to be fully trusted, but it was not safe to do so. We fixed the bug in C# 3.0 and took the breaking change.

This raises a more interesting point though. Now that we correctly prevent you from setting the "parent" reference in this manner, how would you implement the desired pattern? The reader wanted the following conditions to be met:

There are two kinds of items -- "leaf" items, and "container" items. Container items may contain either kind of item, so you could have a box containing a bag, which in turn contains a torch.
An item may be in a container, and if it is, its parent reference refers to that container.
These classes must be extensible by arbitrary third parties.
"Unauthorized" code must not be able to muck around with the parenting invariants.

The reader was attempting to enforce these invariants by making the parent setter protected. But even in a world where it is legal to access a protected member from an arbitrary derived class, that does not ensure that the invariants are maintained! If you allow any derived class to muck with the parenting, then you are relying upon every third-party derived class to "play nicely" and maintain your invariant. If any of them are buggy or hostile, then who knows what can happen?

When I'm faced with this kind of problem, I try to go back to first principles. Considering each design constraint leads us to an implementation decision which implements that constraint:

There are two kinds of abstract items -- items and containers. Therefore, there should be two abstract base classes, not just one.
Containers are items. Therefore, the container class should derive from the item class.
The parenting invariant must be maintained across all containers. Therefore the invariant implementation must be inside the abstract container base class.
Every possible container requires "write" access to the parent state of every possible Item. We want that access to be as restricted as possible. Ideally, we want it to be private. The only by-design way to get access to a private is to be inside the class.

And now a full solution becomes very straightforward:

using System;
using System.Collections.Generic;

public abstract class Item {
public Item Parent { get; private set; }
public abstract class Container : Item {
 private HashSet<Item> items = new HashSet<Item>();
 public void Add(Item item) {
 if (item.Parent != null)
 throw new Exception("Item has inconsistent containment.");
 item.Parent = this;
 items.Add(item);
 }
 public void Remove(Item item) {
 if (!Contains(item))
 throw new Exception("Container does not contain that item.");
 items.Remove(item);
 item.Parent = null;
 }
 public bool Contains(Item item) {
 return items.Contains(item);
 }
 public IEnumerable<Item> Items {
 // Do not just return items. Then the caller could cast it
 // to HashSet<Item> and then make modifications to your
 // internal state! Return a read-only sequence:
 get {
 foreach(Item item in items) yield return item;
 }
 }
}
}

// These can be in third-party assemblies:

public class Bag : Item.Container { }
public class Box : Item.Container { }
public class Torch : Item { }
public class TreasureMap : Item { }

public class Program {
public static void Main() {
    var map = new TreasureMap();
    var box = new Box();
    box.Add(map);
    var bag = new Bag();
    bag.Add(box);
    foreach(Item item in bag.Items)
      Console.WriteLine(item);
}
}

Pretty slick eh?

A couple questions for you to ponder:

1) Suppose you were a hostile third party and you wanted to mess up the parenting invariant. Clearly, if you are sufficiently trusted, you can always use private reflection or unsafe code to muck around with the state directly, so that's not a very interesting attack. Any other bright ideas come to mind for ways that this code is vulnerable to tampering?

2) Suppose you wanted to make this hierarchy an immutable collection, where "Add" and "Remove" returned new collections rather than mutating the existing collection. How would you represent the parenting relationship?

Next time, another oddity involving "protected" semantics. Have a good weekend!

Covariance and Contravariance in C#, Part Nine: Breaking Changes

Eric Lippert — Fri, 02 Nov 2007 17:33:00 GMT

Today in the last entry in my ongoing saga of covariance and contravariance I’ll discuss what breaking changes adding this feature might cause.

Simply adding variance awareness to the conversion rules should never cause any breaking change. However, the combination of adding variance to the conversion rules and making some types have variant parameters causes potential breaking changes.

People are generally smart enough to not write:

if (x is Animal)
DoSomething();
else if (x is Giraffe)
DoSomethingElse(); // never runs

because the second condition is entirely subsumed by the first. But today in C# 3.0 it is entirely sensible to write

if (x is IEnumerable<Animal>)
DoSomething();
else if (x is IEnumerable<Giraffe>)
DoSomethingElse();

because there did not used to be any conversion between IEnumerable<Animal> and IEnumerable<Giraffe>. If we turn on covariance in IEnumerable<T> and the compiled program containing the fragment uses the new library then its behaviour when given an IEnumerable<Giraffe> will change. The object will be assignable to IEnumerable<Animal>, and therefore the “is” will report “true”.

There is also the issue of existing source code changing semantics or turning compiling programs into erroneous programs. For example, overload resolution may now fail where it used to succeed. If we have:

interface IBar<T>{} // From some other assembly
...
void M(IBar<Tiger> x){}
void M(IBar<Giraffe> x){}
void M(object x) {}
...
IBar<Animal> y = whatever;
M(y);

Then overload resolution picks the object version today because it is the sole applicable choice. If we change the definition of IBar to

interface IBar<-T>{}

and recompile then we get an ambiguity error because now all three are applicable and there is no unique best choice.

We always want to avoid breaking changes if possible, but sometimes new features are sufficiently compelling and the breaks are sufficiently rare that it’s worth it. My intuition is that by turning on interface and delegate variance we would enable many more interesting scenarios than we would break.

What are your thoughts? Keep in mind that we expect that the vast majority of developers will never have to define the variance of a given type argument, but they may take advantage of variance frequently. Is it worth our while to invest time and energy in this sort of thing for a hypothetical future version of the language?

Future Breaking Changes, Part Three

Eric Lippert — Tue, 04 Sep 2007 17:30:00 GMT

As I said earlier, we hate causing breaking changes in our product, the C# compiler, because they cause our customers pain.

Said customers are also software developers, and presumably they hate causing breaking changes for their customers as much as we do. We want to throw our customers into the Pit of Success and give them tools which encourage them where possible to prevent breaking changes. This leads to some subtle issues in language design.

Pop quiz. What does this program do?

// Alpha.DLL
namespace Alpha {
public class Charlie {
public void Frob(int i) { System.Console.WriteLine("int"); }
// etc.
}
}

// Bravo.EXE, references Alpha.DLL.
namespace Bravo {
public class Delta : Alpha.Charlie {
    public void Frob(float f) { System.Console.WriteLine("float"); }
    // etc.
    public static void Main() {
      Delta d = new Delta();
      d.Frob(1);
    }
}
}

Most people look at this program and say “clearly Charlie.Frob(int) is the best possible match for the call, so that is called.” A compelling argument, but wrong. As the standard says, “methods in a base class are not candidates if any method in a derived class is applicable".

In other words, the overload resolution algorithm starts by searching the class for an applicable method. If it finds one then all the other applicable methods in deeper base classes are removed from the candidate set for overload resolution. Since Delta.Frob(float) is applicable, Charlie.Frob(int) is never even considered as a candidate. Only if no applicable candidates are found in the most derived type do we start looking at its base class.

Why on earth would we do that? Clearly in this example the base class member is the far better match, so why wouldn’t we even consider it?

It is instructive to consider what happens in a world where we do implement the rule “pick the best applicable candidate from any base”. Suppose we did that.

In the previous version of Alpha.DLL, Charlie did not have a method Frob(int).; When Bravo Corporation wrote Bravo.EXE, every call inside class Delta to method Frob was a call to Delta.Frob(float). Then one day Alpha corporation did customer research and discovered that a lot of their customers like to frob integers. They added this feature in their latest version. Delta corporation gets the new version of Alpha.DLL, recompiles Bravo.EXE, and suddenly their carefully developed code is sometimes calling a method that they didn’t write, which does something subtly incompatible with their implementation.

Alpha corporation has just pushed a breaking change onto Bravo corporation, which, if they don’t catch it in time, may now be pushing a subtly broken version onto their customers in turn, and hey! we’re in the Pit of Despair again!

This particular family of breaking changes is called the "brittle base class problem"; there are many versions of it and different languages deal with it in different ways. Lots of work went into the design of C# to try and make it harder for people to accidentally cause brittle base class problems. That is why we make you distinguish between the original definition of a virtual method and an overriding method. That is why we make you put “new” on methods which shadow other methods. All these semantics are in part to help prevent, mitigate or diagnose brittle base class issues and thereby prevent accidental breaking changes in C# code.

Next time on FAIC: some psychic debugging. Then a bit later I want to talk more about breaking changes, this time in the context of thinking about covariance and contravariance.

Future Breaking Changes, Part Two

Eric Lippert — Fri, 31 Aug 2007 17:00:00 GMT

Last time I mentioned that one of the subtleties of programming language design is weighing the benefit of adding a feature against the pain it will cause you in the future. This is a specific subset of a more general set of problems. Languages run into the same problem that other large, multiply-versioned software products run into; if you have n existing features and want to add m new features, then you have at least m x n possible interactions to consider. In an ideal world the features would all be orthogonal to each other, but in reality of course new features modify the behaviour of existing features.

Just as a random example, what on earth do partial methods have to do with expression trees? Both are new features for C# 3.0, but other than that, they seem to have nothing to do with each other. But what happens when you have an expression tree lambda which contains a call to a partial method which is going to be removed? This interaction has to be carefully defined. Every interaction between every feature has to be carefully defined, and the larger the area of the feature interaction is, the more likely it is that there will be unfortunate consequences of this nonorthogonality, such as breaking changes.

I said last time that implicitly typed lambdas are just such a feature -- big surface area of nonorthogonal interactions which will lead to future breaking changes. I'd like to explore that in more detail, because it might not be immediately obvious why that is the case.

The rule for determining whether an implicitly typed lambda is convertible to a given delegate type is (somewhat simplified) “If possible, infer the return type and the types of the lambda parameters from the delegate type. Try to bind the lambda with those types. If the inference and binding succeeds with no errors then the lambda expression is convertible to that delegate type.”

For example, if we had a lambda expression like x=>x - 123.4, that would be convertible to Func<double, double> but not to Func<string, double> (because the subtraction cannot be legally bound) or Func<double, int> (because the conversion to the return type cannot be bound).

Notice how the overlap between this new feature – implicitly typed lambda convertibility to a delegate type – overlaps with almost every other feature in the entire C# language. Consider the impact of this design decision on the potential for breaking changes in the future. Every single time we make a change to the rules for how the body of a method is bound, we are potentially changing whether a given lambda expression is convertible to a given type.

Now, is that really so bad? Generally when we make changes to the method body binding rules, we do so in as non-breaking a manner as possible. We should never be making an existing lambda that does convert suddenly stop converting, because then we would also be breaking compilation of the equivalent nominal method body. Really we should only be causing presently-erroneous code to suddenly start being non-erroneous, and as we’ve already discussed, that’s not a breaking change.

Or is it?

Actually, now in a lot of cases it could be.

Consider a silly example. Suppose we decide that in some hypothetical C# 4.0(‡) language it should be legal to subtract a double from a string. That’s not entirely farfetched – that’s perfectly legal in JScript. You just convert the string to a number and subtract. We might reason that this new feature is not a breaking change, because no program that ever subtracted a double from a string ever compiled before. But it is a breaking change, because now this program stops compiling:

using System;
class Program {
void M(Func<double, int> f){}
void M(Func<string, double> f){}
void M(Func<double, double> f){}
static void Main() {
M(x=>x-123.4);
}
}

Before the change, the program unambiguously chooses the third overload. But with our change to addition semantics, now the second and third overloads would both work. Neither would be clearly better than the other. The compiler would then give an ambiguity error. Thus, this is a breaking change.

It gets even worse; with some cleverness we could come up with more subtle breaking changes, where the compiler would not produce an error but instead would choose a different overload than before. At least a program which fails to compile calls attention to the problem; recompiling and having the behaviour change subtly might go unnoticed for a long time.

This is a general problem with overload resolution: it depends upon conversion semantics. If we make the conversion semantics more strict then it is possible to go from having exactly one overload which works to zero, which is a breaking change. If we make the conversion semantics less strict then it is possible to go from having exactly one overload which works to two, which is also (usually) a breaking change.

Since lambda convertibility depends on every other language rule for binding an expression, any change to any of those rules is a potential change in convertibility, and hence a potential breaking change for overload resolution. Now, hopefully few real-world codebases will end up in the above situation, where a method is overloaded solely on a bunch of different delegate types, but still, I worry. We think implicitly typed lambdas are worth it, but this was a tough call that we agonized over for a long time.

Next time on FAIC: More on subtle language design issues involving breaking changes. Have a pleasant Labour Day weekend, Canadian and American readers!

(‡) I call out that this is the hypothetical C# 4.0 compiler because of course I do not discuss the feature set of unannounced and non-existing products on public blogs.

Future Breaking Changes, Part One

Eric Lippert — Thu, 30 Aug 2007 17:00:00 GMT

We on the C# team hate making breaking changes.

As my colleague Neal called out in his article on the subject from a couple years ago, by “breaking change” we mean a change in the compiler behaviour which causes an existing program to either stop compiling entirely, or start behaving differently when recompiled. We hate them because they can cause intense customer pain, and are therefore barriers to upgrading. We work hard on improving the quality of the C# toolset every day and want our customers to gain the benefits of upgrading; they cannot do that if doing so causes more pain than the new features are worth.

Which is not to say that we do not make breaking changes occasionally. We do. But in those circumstances where we have to, we try to mitigate the pain as much as possible. For example:

We try to ensure that breaking changes only affect unusual corners of the language which real production code is unlikely to hit.
When we do cause breaking changes, we sometimes introduce heuristics and warnings which detect the situation and warn the developer.
If possible, breaking changes should move the implementation into compliance with the published standard. (Often breaking changes are a result of fixing a compliance bug.)
We try to communicate the reasoning behind a breaking change crisply and succinctly.
And so on.

I could write a whole series of blog articles about specific breaking changes – and in fact, I have done so many times over the years. (For example, here, here, here, here and here.)

Given that we hate breaking changes, clearly we want to ensure that as we add new features to the C# language we do not break existing features. New features start out with points against them and have to justify their benefits. If the feature is a breaking change, that is hugely more points against it.

For example, adding generics to C# 2.0 was a breaking change. This program, legal in C# 1.0, no longer compiles:

class Program {
 static void M(bool x, bool y) {}
 static void Main() {
 int N = 1, T = 2, U = 3, a = 4, b = 5;
 M(N < T, U > (a+b));
 }
}

because we now think that N is a generic method of one argument. But the compelling benefit of generics greatly outweighed the pain of this rather contrived example, so we took the breaking change. (And the error message now produced diagnoses the problem.)

But what I want to talk about in this set of articles is something a bit more subtle than these specific breaking changes.

Given that we hate breaking changes, we want to ensure that as we add new features to the C# language we are not setting ourselves up to have to make breaking changes in the future. If we are, then the feature needs to be justified against not only any breaking changes it is presently introducing, but also against the potential for breaking changes in the future. We do not want to introduce a new feature that makes it harder for us to introduce entirely different features in the future unless the proposed new feature is really great.

Implicitly typed lambdas are an example of a feature which will cause us innumerable breaking change headaches in the future, but we believe that the compelling user benefit of them is so high (and the breaks are sufficiently isolated to corner cases) that we are willing to take that pain.

Next time on FAIC, I’ll describe in detail how it is that implicitly typed lambdas are going to make it harder to get new features into the C# type system because of potential breaking changes.

Chained user-defined explicit conversions in C#

Eric Lippert — Mon, 16 Apr 2007 17:00:00 GMT

Reader Niall asked me why the following code compiles but produces an exception at runtime:

class Base {}
class Derived : Base {}
class Castable {
  public static explicit operator Base() {
    return new Base();
  }
}
// ...
Derived d = (Derived)(new Castable());

It should be clear why this produces an exception at runtime; the user-defined operator returns a Base and that is not assignable to a variable of type Derived. But why does the compiler allow it in the first place?

First off, let’s define the difference between an implicit and an explicit conversion. An implicit conversion is one which the compiler knows can always be done without incurring the risk of a runtime exception. When you have a method int Foo(int i){...} and call it with long l = Foo(myshort);, the compiler inserts implicit conversions from int to long on the return side and from short to int on the call side. There is no int which doesn’t fit into a long and there is no short which doesn’t fit into an int, so we know that the conversions will always succeed, so we just up and do them for you.

There are also conversions which we know at compile time will never succeed. If there is no user-defined conversion from Giraffe to int, then Foo(new Giraffe()) is always going to fail at runtime, so this fails at compile time.

An explicit conversion is a conversion which might succeed sometimes but might also fail. We cannot disallow it, because it might succeed, but we can’t go silently inserting one either, since it might fail unexpectedly. We need to force the developer to acknowledge that risk explicitly. If you called ulong ul = Foo(mynullableint); then that might fail, so the compiler requires you to spell out that the conversions are explicit. The assignment could be written ulong ul = (ulong)Foo((int)mynullableint);.

There are two times that the compiler will insert an explicit cast for you without producing a warning or error. The first is the case above. When a user-defined explicit cast requires an explicit conversion on either the call side or the return side, the compiler will insert the explicit conversions as needed. The compiler figures that if the developer put the explicit cast in the code in the first place then the developer knew what they were doing and took the risk that any of the conversions might fail. That’s what the cast means: this conversion might fail, I will deal with it.

I understand that this puts a burden upon the developer to fully understand the implications of a cast, but the alternative is to make you spell it out even further, and it just gets to be too much. The logical extreme of this would be a case such as

public struct S{
public static explicit operator decimal?(S s) {return 1.0m;}
}
//...
S? s = new S();
int i = (int) s;

Here we do first an explicit conversion from S? to S, then a user-defined explicit conversion from S to decimal?, then an explicit conversion from decimal? to decimal, and then an explicit conversion from decimal to int. That’s four explicit conversions for the price of one cast, which I think is pretty good value for your money.

I want to note at this point that this is as long as the chain gets. A user-defined conversion can have built-in conversions inserted automatically on the call and return sides, but we never automatically insert other user-defined conversions. We never say that there’s a user-defined conversion from Alpha to Bravo, and a user-defined conversion from Bravo to Charlie, and therefore casting an Alpha to a Charlie is legal. That doesn’t fly.

And a built-in conversion can be lifted to nullable, which may introduce additional conversions as in the case above. But again, these are never user-defined.

The second is in the foreach loop. If you have foreach(Giraffe g in myAnimals) then we generate code which fetches each member of the collection and does an “explicit” conversion to Giraffe. If there happens to be a Snail or a WaterBuffalo in myAnimals, that’s a runtime exception. I considered adding a warning to the compiler for this case to say hey, be aware that your collection is of a more general type, this could fail at runtime. It turns out that there are so many programs which use this programming style, and so many of them have “compile with warnings as errors” turned on in their builds, that this would be a huge breaking change. So we opted to not do it.

Odious ambiguous overloads, part two

Eric Lippert — Thu, 06 Apr 2006 20:13:00 GMT

There were a number of ideas in the comments for what we should do about the unfortunate situation I described yesterday. We considered all of these options and more. I thought I might talk a bit about the pros and cons of each.

I suggested do nothing. We shipped with this behaviour in C# 2.0, the world didn't come to an end, we can keep on shipping with it. The pro side is that this is the cheapest and easiest thing to do. The con side is that as a reader pointed out, this is a gross "gotcha" just waiting for someone to stumble upon it.

DrPizza suggested that we come up with a syntax that disambiguates between the two ambiguous cases. On the pro side, the problem would be clearly solved. On the con side, this might be a breaking change (because suddenly one of the methods that used to bind one way might start binding the other way, depending on what the choice of syntax was.)

Also on the con side, adding new language syntax is something that we do not do lightly. There is considerable work involved in changing the specification, and this is kind of an edge case.

Ayende Rahien suggested that we add an attribute that hints to the compiler which is the right binding. This has the nice property of not introducing new language syntax, but is kind of ugly.

These are both good ideas but unfortunately, as it turns out, neither of these work for technical reasons which I will go into in detail below.

Chris D suggested that the constructed interface should be treated as an interface with only one method. That's a really nice idea, but unfortunately it doesn't work with the CLR definition of generics. An interface with n methods has n vtable slots that need filling, period, and n is invariant under construction. We'd have to change both the semantics of constructed interfaces and their implementation in the CLR, which means changing both the C# spec and the CLR spec, and that's way too much work for this edge case.

Kyle Lahnakoski suggested that it simply be an error. There are several ways this could be an error.

It could be an error to have an unconstructed generic type which could possibly be constructed to produce a duplicated signature. This was how generics originally worked in C#, but we decided that we could allow these cases and we can't really go back on that decision now – that would be a huge breaking change. There are many serializable generic objects which use a pattern of having a method that takes a type parameter to overload a method that takes a serialization object parameter, and all of those would suddenly be errors.

It could be an error to construct a generic type so as to actually produce a duplicate signature, but again, we decided to allow that in C# 2.0 and it would be a large breaking change to go back now.

It could be an error to have any explicit implementation of an interface method which matches ambiguously. This would be a smaller breaking change, but still breaking.

What do any of these breaking changes buy the user though? Not more functionality, but less. Basically we'd be saying "you could do this before, and now you can't, too bad." That's a hard sell.

Carlos suggested unification – that is, have an explicit implementation map to every method that it matches, not just the first one. This is a breaking change, but it seems like the most natural thing to do. Since an implicit implementation maps to every interface slot that it matches, why shouldn't an explicit implementation do the same? This means adding new language to the spec, but no new syntax.

This is what we decided to do, and I tried to implement it, only to discover that, uh, actually that stuff I told you yesterday about the implicit implementation matching the first one in source code order is a bit of a mistruth. That's what I thought when I first looked at the problem, but upon closer examination, something deeper was wrong.

In fact what is going on here is that when the CLR loads the class for the first time, it does a verification check to ensure that the class does in fact implement every method of every interface that it says it does. It is the CLR, not C#, which matches the explicit implementation to the first matching slot in the constructed interface type! It just so happens that since metadata for the interface is emitted in source-code order, that the CLR appears to do the match in source code order. It actually does it in a CLR-implementation-defined order that just happens to be source code order.

To understand why we can't fix this, you need to understand how explicit implementations are represented in metadata. In the CLR metadata for a class there is a table called the MethodImpl table which says "method blah of class foo explicitly implements method bar of interface IFred". The "method bar of interface IFred" portion of that mapping may be either a token to a method of the unconstructed interface (a "MethodDef") , or it may be a token representing a method on the constructed interface (a "MemberRef").

We can't use a MethodDef because there is a signature mismatch between the class's implementation and the method on the unconstructed interface. One takes an int, the other takes a type parameter, and the CLR flags that as illegal. So that's out.

But we can't use the latter because the MemberRef identifies the constructed interface method by – you guessed it – its signature! And now we're in the same hideous bind all over again. We have no way to tell the CLR "no, really, use the generic-substituted version of the method", because it can't tell the two signatures apart either.

To fix this problem by either unifying, or adding a syntax/attribute to disambiguate, we need some way to represent that unification/disambiguation in metadata. But the CLR simply lacks the concept of "method token for constructed method identified by something more unique than signature".

Though we can certainly suggest that this is a weakness in the CLR metadata design, waiting around for that to be addressed to fix this edge-case problem seems like a non-starter. Given that we can't fix it in the current CLR the way we'd like, and making it an error seems like an egregious overreaction that buys the user nothing, but at the same time doing nothing is leaving a gotcha in the code waiting to happen, I took the only course of action left:

Warning: Explicit interface implementation 'I1<int>.M(int)' matches more than one interface member. Which interface member is actually chosen is implementation-dependent. Consider using a non-explicit implementation instead."

Unfortunately, sometimes that's the best you can do.

Odious ambiguous overloads, part one

Eric Lippert — Wed, 05 Apr 2006 18:58:00 GMT

As you might have gathered, a lot of the decisions we have to make day-to-day here involve potential breaking changes on odd edge cases. Here's another one.

Consider the following terrible but legal code:

public interface I1 {
void M(U i);
void M(int i);
}

My intense pain begins when the user writes:

public class C1 : I1<int> {

In the early version of the C# specification it was actually illegal to have an ambiguity like this, but the spec was changed so that when doing overload resolution on a call to M we can choose one, according to section 14.4.2.2:

If one [argument type] is non-generic, but the other is generic, then the non-generic is better.

But still, as you will soon see, we're in a world of hurt for another reason, namely that this class must now implement both M(int i) and, uh, M(int i). Fortunately, a nonexplicit implementation binds to both methods, so this works just fine:

public class C1 : I1<int> {
 public void M(int i) {
 Console.WriteLine("class " + i);
 }
}

The method implements both versions of M and the contract is satisfied. But we have problems if we try to do an explicit interface implementation:

public class C2 : I1<int> {
 void I1<int>.M(int i) {
 Console.WriteLine("explicit " + i);
 }
}

Does this explicitly implement both members of I1? Or just one? If so, which one?

In the current compiler this code produces a terrible, terrible error:

error CS0535: 'C2' does not implement interface member 'I1<int>.M(int)'

Is that so? It sure looks like it implements it!

What happens when we have both an explicit implementation and a class implementation? The spec does not actually say what to do. It turns out that we end up in a situation where runtime behaviour depends on source code order of the interface! Check this out:

public interface I1 {
void M(U i); // generic first
void M(int i);
}

public interface I2 {
void M(int i);
void M(U i); // generic second
}

public class C3: I1<int>, I2<int> {
 void I1<int>.M(int i) {
 Console.WriteLine("c3 explicit I1 " + i);
 }
 void I2<int>.M(int i) {
 Console.WriteLine("c3 explicit I2 " + i);
 }
 public void M(int i) {
 Console.WriteLine("c3 class " + i);
 }
}

class Test {
 static void Main() {
 C3 c3 = new C3();
 I1<int> i1_c3 = c3;
 I2<int> i2_c3 = c3;
 i1_c3.M(101);
 i2_c3.M(102);
 }
}

What happens here is that the explicit interface implementation mappings in the class match the methods in the interfaces in a first-come-first-served manner:

void I1<int>.M(U) maps to explicit implementation void I1<int>.M(int i)
void I1<int>.M(int) maps to implicit implementation public void M(int i)
void I2<int>.M(int) maps to explicit implementation void I2<int>.M(int i)
void I2<int>.M(U) maps to implicit implementation public void M(int i)

Then (because of the aforementioned section 14.4.2.2) when we see

i1_c3.M(101);
i2_c3.M(102);

we prefer the typed-as-int versions to the generic substitution versions, so this program calls the two non-generic versions and produces the output:

c3 class 101
c3 explicit I2 102

And as you'd expect, if we force the compiler to pick the generic versions then we get similar behaviour:

static void Main() {
 C3 c3 = new C3();
 Thunk1<int>(c3,103);
 Thunk2<int>(c3, 104);
}
static void Thunk1(I1 i1, U u) {
 i1.M(u);
}
static void Thunk2(I2 i2, U u) {
 i2.M(u);
}

The binding of the overload resolution in the thunk bodies happens before the substitution of the type parameters, so these always bind to the generic versions of the methods. As you would expect from the mappings above, this outputs

c3 explicit I1 103
c3 class 104

Again, this shows that source code order has an unfortunate semantic import.

Given this unfortunate situation -- no spec guidance and an existing implementation that behaves strangely -- what would you do? (Of course "do nothing" is an option.) I'm interested to hear your ideas, and I'll describe what we actually did next time.

The Root of All Evil, Part Two

Eric Lippert — Thu, 30 Mar 2006 00:45:00 GMT

There were a number of interesting comments to yesterday's posting about premature optimization.

Several readers pointed out that there are various options we could take:

Do nothing. Maintain backwards compatibility and spec incompatibility.
Change the spec to exactly match the current implementation.
Change the spec and the implementation.
Make the weird behaviour a warning now and an error later.
Fix the implementation to match the specification, but add another compiler switch to go back to the previous behaviour.

These all have various pros and cons, and even for a relatively trivial spec incompatibility such as this one, we think very hard about what the right thing to do is.

Option #2 sounded good, and I actually was an advocate of it for a few hours yesterday, right up until I dug into the optimization and discovered exactly what it was doing. If we were to change the specification to exactly match the behaviour, then the specification would read like this:

Implicit Enumeration Conversions

There is an implicit conversion from any "magic zero" to any enumerated type. "Magic zero" is recursively defined as follows:

the integer literal zero, or
any magic zero + - | ^ any compile-time integer constant expression equal to zero, or
any compile-time integer expression * & any magic zero, or
any magic zero / % any non-zero compile-time constant integer expression, or
any magic zero * & any non-zero compile-time constant integer expression

This means for example, that (8*(0+(7-7))) is a magic zero, but (8*((7-7)+0) is not.

I think we can all agree that we don't want this nonsense in the specification. Changing the spec to exactly match the buggy behaviour leads to a crazy spec.

We could change both the specification and the implementation. We could, for instance, say that any compile-time constant integer zero is implicitly convertible to any enumerated type. This means that nonsense like (8*(0+(7-7))) is assignable to enum, but at least it also means that (8*((7-7)+0) is too.

Adding a new warning is a tempting idea, and we might end up doing it. Doing so means that it is easier to turn it into an error in a future release. New warnings, however, are expensive. They add new code paths to test, we have to test all the scenarios to make sure the warning always appears when it should, warnings have to be translated into a dozen languages, they have to be documented, the documentation has to be translated... it's not cheap.

Adding a compiler switch is also a possibility, but also expensive. The more switches there are, again, more code paths, more tests, bigger testing matrix, the switch has to be documented, and so on.

So now you see why premature optimization is the root of all evil. I've spent the better part of two days wrestling with this thing. Why? Because suppose one of these guys is in a lambda expression that needs to be translated into an expression tree. I want to make for darn sure that the translation is correct, and the more "optimizations" that happen before that transformation, the more the transformation is likely to be not as the user intended! But I can't remove the optimizations without breaking the type system!

UPDATE: The thrilling conclusion is that we have made all constant-according-to-the-spec zero integers convertible to any enum. This is a spec violation, but at least it is a consistent spec violation. In doing so I also introduced a bug, which I believe did ship to customers but which we have since fixed, whereby some "non-integer zeros" (like default instances of structs, or null references) were convertible to any enum. I apologize for the error.

I also challenged you folks to come up with a situation where this premature optimization can screw up definite assignment checking. Reader Carlos found two.

int aa; int bb = 12; int cc;
int dd = (aa * 0) + 12;
if (bb * 0 == 0) cc = 123;
Console.WriteLine(cc);

According to the spec, aa should be flagged as "used before definitely assigned" (even though really it doesn't matter), but because the premature optimization throws away the aa, the flow checker never sees it. Also according to the spec, cc should be flagged as "used before definitely assigned" (even though it always is assigned), but because the optimization throws away the bb, the flow checker does not realize that the conditional is not a compile-time-constant.

The important thing here is not that the code gets it wrong. Indeed, one could make the argument that it gets it right. The problem is that it gets it right for the wrong reasons – it gets it right by accident, in violation of the specification. The definite assignment specification isn't supposed to be perfect, it's supposed to be practical and understandable and predictable and implementable. Undocumented extensions to it make it hard to know whether any two implementations of the spec will agree.