Fabulous Adventures In Coding : C# 4.0

Exact rules for variance validity

Eric Lippert — Thu, 03 Dec 2009 14:32:00 GMT

I thought it might be interesting for you all to get a precise description of how exactly it is that we determine when it is legal to put "in" and "out" on a type parameter declaration in C# 4. I'm doing this here because (1) it's of general interest, and (2) our attempt to make a more human-readable version of this algorithm in the draft C# 4.0 specification accidentally introduced some subtle errors. We're working on correcting those errors for the final release of the specification; until then, these definitions are the definitions I actually worked from to do the implementation, so they're accurate.

These definitions are pretty much stolen straight from the CLI spec section on variance; I am indebted to its authors for their careful and precise definitions.

The first things we need to define are three kinds of "validity" for types. I want to define "valid covariantly", "valid contravariantly" and "valid invariantly", but only as applied to types. We'll talk about what makes an interface declaration a valid one later; we need these definitions first.

Before we get into a precise definition, I want to talk about what "valid covariantly" means logically. The idea that we are attempting to capture here boils down to "the type we're talking about is not contravariant". Whether it is covariant, we don't really care. All we care about it is that if it is valid covariantly, then it is not contravariant. Similarly, by "valid contravariantly", we mean "not covariant".

This leads us to our first brain-hurting definition: A type is "valid invariantly" if it is both valid covariantly and valid contravariantly. That sounds a bit crazy -- how can it be valid invariantly if it is both covariant and contravariant? But "valid covariantly" does not mean "is covariant", it means "it's guaranteed to not be contravariant". So if a type is valid covariantly and valid contravariantly, then it is guaranteed to be neither contravariant nor covariant, and therefore must be invariant.

Anyway. We want to take any specific type and classify it as valid covariantly, valid contravariantly, or both, a.k.a., valid invariantly. (We don't need to worry about coming up with a term for the "or none of the above" case because in practice that case is never relevant.)

A type is valid covariantly if it is:

1) a class, struct, nullable, enum, pointer, non-generic delegate or non-generic interface type.

This should make sense. Remember, what we're getting at is "not contravariant". Classes and structs are either non-generic, in which case they are clearly not contravariant, or they are generic, in which case again they are not contravariant because we do not support variance on classes or structs. Nullable types are just aliases for Nullable<T> which is a struct type, and therefore not variant. Enums and pointers are non-generic so they are not variant. And non-generic delegates and interfaces are by definition not generic and therefore not variant either. These are the easy cases.

2) An array type T[] where T is valid covariantly.

Because we have array covariance in C#, but not array contravariance, we can make arrays valid covariantly. Again, remember, "valid covariantly" means "not contravariant".

3) A generic type parameter type, if it was not declared as being contravariant.

Generic type parameter types are of course types as far as the compiler is concerned. If you're inside the declaration of a generic interface then you can use its type parameters as types. In that context, such types are valid covariantly if they were not declared as contravariant (in).

4) A constructed interface or delegate type X<T1, ... Tk> might be valid covariantly. To determine if it is, we examine each type argument differently, depending on whether the corresponding type parameter was declared as covariant (out), contravariant (in), or invariant (neither). If the ith type parameter was declared as covariant, then Ti must be valid covariantly. If it was declared as contravariant, then Ti must be valid contravariantly. If it was declared as invariant, then Ti must be valid invariantly.

As one would expect, covariant validity preserves the direction of validity; covariant parameters must be valid covariantly, contravariant parameters must be valid contravariantly.

OK, I hope that was relatively painless. The rules for contravariant validity are similar; as one would expect from the "backwards" nature of contravariance, the directions are reversed in the complicated case:

A type is valid contravariantly if it is:

1) a class, struct, nullable, enum, pointer, non-generic delegate or non-generic interface type.

2) An array type T[] where T is valid contravariantly.

Arrays are covariant in their element type. Covariance preserves the direction of variance. Therefore, to be valid contravariantly, an array (covariant in its element type) must be contravariantly valid in its element type.

3) A generic type parameter type, if it was not declared as being covariant.

Remember, "valid contravariantly" means "not covariant". It does not mean "contravariant".

4) A constructed interface or delegate type X<T1, ... Tk> might be valid contravariantly. If the ith type parameter was declared as contravariant, then Ti must be valid covariantly. If it was declared as covariant, then Ti must be valid contravariantly. If it was declared as invariant, then Ti must be valid invariantly.

As one might expect, contravariant validity reverses the direction of validity.

Now perhaps you see why we wanted to rewrite these definitions into something more human-readable for the spec. And perhaps you also see why we accidentally introduced errors in doing so; bending your brain around all this logic is not always easy.

OK, now that we've got that, we can make a definition of what it means for an interface to be valid. An interface must meet the following conditions:

* The return types of all non-void interface methods must be valid covariantly.

* Every formal parameter type of all interface methods must be valid contravariantly. (Invariantly if it is an out or ref parameter.)

* For all generic methods on an interface, every constraint on the generic method type parameters must be valid contravariantly.

* All its base interface types must be valid covariantly.

* The type of a property or indexer must be valid covariantly if it has a "getter" and valid contravariantly if it has a "setter".

* Any formal parameter types of an indexer must be valid contravariantly.

* The delegate types of all its events must be valid contravariantly.

The first two are pretty straightforward; return types "go out" so they have to be valid covariantly, formal parameter types "go in", so they have to be valid contravariantly. But what's up with the third one? What do constraints on generic method type parameters have to do with interface validity?

Well, let's suspend the third rule and see what goes wrong.

interface I<out T>
{
 void M() where U : T;
 // illegal; this has to be valid contravariantly but it is a covariant type parameter constraint.
 // Let it ride for now and demonstrate an error.
}

class C<T> : I<T> { public void M() {} }
// the constraint is inherited implicitly and not re-stated.

I<Giraffe> igiraffe = new C<Giraffe>(); // C<T> implements I<T>
I<Animal> ianimal = igiraffe; // interface is covariant in T
ianimal.M<Turtle>(); // satisifies the constraint that U must be an Animal.

Uh oh. ianimal is really an instance of C<Giraffe>. The M method on C<Giraffe> inherits a requirement that U inherit from Giraffe. Turtle does not inherit from Giraffe. Therefore we've just violated the constraint on M. The only places where we can catch this is in the declaration; every other step is perfectly legal. Therefore, a constraint cannot be covariant. But if we make it contravariant (or invariant) then it all works out. For example, let's make a contravariant type parameter constraint:

interface I2<in T> // contravariant this time
{
void M() where U : T;
}

class C2<T> : I2<T> { public void M() {} }
I2<Animal> i2animal = new C2<Animal>(); // C2<T> implements I2<T>
I2<Mammal> i2mammal = i2animal; // interface is contravariant in T
i2mammal.M<Giraffe>(); // satisifies the constraint that U must be an Animal.

And now everything is fine; the compile-time constraint checker verifies that Giraffe is Mammal; at runtime it must be Animal, and so the compiler has verified that by verifying that it is Mammal.

The rules for delegate declarations are a straightforward simplification of the rules for interface declarations. To be a valid delegate declaration, the return type must be valid covariantly (or void), the formal parameter types must be valid contravariantly (or invariantly if they are out/ref), and any type parameter constraints must be valid contravariantly.

What's the difference between covariance and assignment compatibility?

Eric Lippert — Mon, 30 Nov 2009 14:51:00 GMT

I've written a lot about this already, but I think one particular point bears repeating.

As we're getting closer to shipping C# 4.0, I'm seeing a lot of documents, blogs, and so on, attempting to explain what "covariant" means. This is a tricky word to define in a way that is actually meaningful to people who haven't already got degrees in category theory, but it can be done. And I think it's important to avoid defining a word to mean something other than its actual meaning.

A number of those documents have led with something like:

"Covariance is the ability to assign an expression of a more specific type to a variable of a less specific type. For example, consider a method M that returns a Giraffe. You can assign the result of M to a variable of type Animal, because Animal is a less specific type that is compatible with Giraffe. Methods in C# are 'covariant' in their return types, which is why when you create a covariant interface, it is indicated with the keyword 'out' -- the returned value comes 'out' of the method."

But that's not at all what covariance means. That's describing "assignment compatibility" -- the ability to assign a value of a more specific type to a storage of a compatible, less specific type is called "assignment compatibility" because the two types are compatible for the purposes of verifying legality of assignments.

So what does covariance mean then?

First off, we need to work out precisely what the adjective "covariant" applies to. I'm going to get more formal for a bit here, but try to keep it understandable.

Let's start by not even considering types. Let's think about integers. (And here I am speaking of actual mathematical integers, not of the weird behaviour of 32-bit integers in unchecked contexts.) Specifically, we're going to think about the ≤ relation on integers, the "less than or equal to" relation. (Recall that of course a "relation" is a function which takes two things and returns a bool which indicates whether the given relationship holds or does not hold.)

Now let's think about a projection on integers. What is a projection? A projection is a function which takes a single integer and returns a new integer. So, for example, z → z + z is a projection; call it D for "double". So are z → 0 - z, N for "negate" and z → z * z, S for "square".

Now, here's an interesting question. Is it always the case that (x ≤ y) = (D(x) ≤ D(y))? Yes, it is. If x is less than y, then twice x is less than twice y. If x is equal to y then twice x is equal to twice y. And if x is greater than y, then twice x is greater than twice y. The projection D preserves the direction of size.

What about N? Is it always the case that (x ≤ y) = (N(x) ≤ N(y))? Clearly not. 1 ≤ 2 is true, but -1 ≤ -2 is false. But we notice that the reverse is always true! (x ≤ y) = (N(y) ≤ N(x)). The projection N reverses the direction of size.

What about S? Is it always the case that (x ≤ y) = (S(x) ≤ S(y))? No. -1 ≤ 0 is true, but S(-1) ≤ S(0) is false. What about the opposite? Is it always the case that (x ≤ y) = (S(y) ≤ S(x)) ? Again, no. 1 ≤ 2 is true, but S(2) ≤ S(1) is false. The projection S does not preserve the direction of size, and nor does it reverse it.

The projection D is "covariant" -- it preserves the ordering relationship on integers. The projection N is "contravariant". It reverses the ordering relationship on integers. The projection S does neither; it is "invariant".

Now I hope it is more clear exactly what is covariant or contravariant. The integers themselves are not variant, and the "less than" relationship is not variant. It's the projection that is covariant or contravariant -- the rule for taking an old integer and making a new one out of it.

So now lets abandon integers and think about reference types. Instead of the ≤ relation on integers, we have the ≤ relation on reference types. A reference type X is smaller than (or equal to) a reference type Y if a value of type X can be stored in a variable of type Y. That is, if X is "assignment compatible" with Y.

Now consider a projection from types to types. Say, the projection "T goes to IEnumerable<T>". That is, we have a projection that takes a type, say, Giraffe, and gives you back a new type, IEnumerable<Giraffe>. Is that projection covariant in C# 4? Yes. It preserves the direction of ordering. A Giraffe may be assigned to a variable of type Animal, and therefore an sequence of Giraffes may be assigned to a variable that can hold a sequence of Animals.

We can think of generic types as "blueprints" that produce constructed types. Let's take the projection that takes a type T and produces IEnumerable<T> and simply call that projection "IEnumerable<T>". We can understand from context when we say "IEnumerable<T> is covariant" what we mean is "the projection which takes a reference type T and produces a reference type IEnumerable<T> is a covariant projection". And since IEnumerable<T> only has one type parameter, it is clear from the context that we mean that the parameter to the projection is T. After all, it is a lot shorter to say "IEnumerable<T> is covariant" than that other mouthful.

So now we can define covariance, contravariance and invariance. A generic type I<T> is covariant (in T) if construction with reference type arguments preserves the direction of assignment compatibility. It is contravariant (in T) if it reverses the direction of assignment compatibility. And it is invariant if it does neither. And by that, we simply are saying in a concise way that the projection which takes a T and produces I<T> is a covariant/contravariant/invariant projection.

UPDATE: My close personal friend (and fellow computer geek) Jen notes that in the Twilight series of novels, the so-called "werewolves" (who are not transformed by the full moon and therefore not actually werewolves) maintain their rigid social ordering in both wolf and human form; the projection from human to wolf is covariant in the social-ordering relation. She also notes that in high school, programming language geeks are at the bottom of the social order, but the projection to adulthood catapults them to the top of the social order, and therefore, growing up is contravariant. I am somewhat skeptical of the latter claim; the former, I'll take your word for it. I suppose the question of how social order works amongst teenage werewolves who are computer geeks is a subject for additional research. Thanks Jen!

The Purpose, Revealed

Eric Lippert — Wed, 25 Nov 2009 17:03:00 GMT

Interestingly enough, no one correctly guessed why I needed code in the compiler to transform a method argument list into a form that batched up the side effecting expressions, stuck their values in variables, and then called the method with the side-effect-free variable references. There certainly were some fascinating ideas in the comments though!

The reason is quite mundane. We had a bug in the code that performs semantic analysis of calls with named arguments. One of our rules in C# that I've talked about a lot on this blog is that sub-expression evaluation occurs in left-to-right order. If you have a method void M(int x, int y) {} and then a call site M(y : F(), x : G()); we cannot simply generate the code as though you'd written M(G(), F()). The side effects of F have to happen before the side effects of G, so we generate code as though you'd said temp1 = F(); temp2 = G(); M(y : temp1, x : temp2); and now we can generate the call as M(temp2, temp1); without worrying about re-ordering a side effect.

The bug was that the rewriter was not getting the reordering correct if the parameters were ref/out or the arguments were locals without side effects. As I often do when fixing a bug, I reasoned that the initial bug had been caused because the rewriter was under-specified.

The fix will get into the initial release of C# 4, but the problems that Pavel identified with it -- that certain obscure side effects such as the order in which bad array accesses throw exceptions, or the order in which static class constructors execute, are not correctly preserved by my algorithm, will not be fixed for the initial release. Doing these correctly requires us to be able to generate temporaries of "reference to argument" types, which we've never done before and which our IL generator was not built to handle. Had we discovered the bug much earlier in the cycle, we probably could have applied a more solid fix, but it is too late now.

Thanks again to Pavel for the great analysis, and thanks to our old friend nikov for reporting the bug in the first place.

And happy thanksgiving to my readers in the United States! As I do every year, I'm roasting a turkey to feed fourteen people. I highly recommend the "brine the bird then roast it upside-down" method described in The Joy Of Cooking. It's worked perfectly for me for the last ten years straight.

Some new videos

Eric Lippert — Mon, 26 Oct 2009 13:17:00 GMT

Somehow it has happened again; people just keep on recording videos of me and putting them on the internet.

In these videos you find out what I look like when lit from above and behind. Kinda spooky. We should have made the room entirely dark and held a flashlight underneath my face. That would be, like, ten times scarier. Anyway, if you're interested in me blathering on about my favourite feature in C# 4, covariance and contravariance of interface and delegate types, here are two little demo videos: Part One, Part Two. (There seems to be some minor sound sync issues here and there, but it's not really a problem; most of the audio is voice-over.)

Charlie has been crazy busy getting these little videos together; here are some more of his recent efforts, including some good ones from my colleagues Chris and Sam talking about all the other far more awesome features of C# 4.0: dynamic interop, improved interop with Office, named and optional parameters, and so on. Links to all of our recent videos are here: http://blogs.msdn.com/charlie/archive/2009/10/19/community-convergence-lvi.aspx.

Have an amusing and safe Hallowe'en -- I'll be going to Hallowe'en parties on a small island this year, just for a change of pace.

[Eric is on vacation this week; this posting was pre-recorded]

Why No Extension Properties?

Eric Lippert — Mon, 05 Oct 2009 16:29:00 GMT

I'm frequently asked "you guys added extension methods to C# 3, so why not add extension properties as well?"

Good question.

First, let me talk a bit about C# 3. Clearly the big feature in C# 3 was LINQ. In a sense we had only three features in C# 3:

everything necessary for LINQ -- implicitly typed locals, anonymous types, lambda expressions, extension methods, object and collection initializers, query comprehensions, expression trees, improved method type inference
partial methods
automatically implemented properties

The latter two were tiny compared to the items in that first bucket. On the design side, the syntax and semantics of both features are straightforward. On the implementation side, we already had the mechanisms in place to remove a call site; partial methods and conditional methods are pretty much the same thing behind the scenes. Auto props were also straightforward to analyze and generate code for. The testing burden was not particularly large for these features either.

The C# team was "the long pole" for the 2008 release of Visual Studio and the .NET Framework. By that I mean that if you took the amount of time required to do the work each team signed up for, given our level of staffing, blah blah blah, and made a pole proportionally long for each team in Developer Division, the C# pole would have been the longest pole. Which means that every other team in devdiv had slack in their schedule, but if we slipped our schedule a day, the new VS/CLR would also ship a day late. If any other team slipped a day, well, as long as that didn't make their pole longer than ours, they were still OK. Within the team, dividing up the work amongs the various development team members, the "long pole" work was the lambda binding and method type inference work, which was mine. So in a sense, every day that I was late, we'd slip the whole product that many days. (No pressure!)

Fortunately we have an excellent team here, we all supported each other very well through that release, picked up each other's slack when necessary, and delivered a quality release in plenty of time. My point is simply that unless a feature was either necessary for LINQ, or small and orthogonal and easy to cut if necessary (like the other two), it got cut immediately. There was no way we were going to risk slipping the entire product for any feature that was both complex and unnecessary.

It was of course immediately obvious that the natural companion to extension methods is extension properties. It's less obvious, for some reason, that extension events, extension operators, extension constructors (also known as "the factory pattern"), and so on, are also natural companions. But we didn't even consider designing extension properties for C# 3; we knew that they were not necessary and would add risk to an already-risky schedule for no compelling gain.

So now we come to C# 4.

As I'm fond of pointing out, the answer to every question of the form "why doesn't product X have feature Y?" is the same. It's because in order for a product to have a feature, that feature must be:

thought of in the first place
desired
designed
specified
implemented
tested
documented
shipped to customers

You've got to hit every single one of those things, otherwise, no feature.

When we started working on C# 4, we made a list of every feature request we'd heard of. It had hundreds of features on it. As I described last year, we categorized that list into "gotta have / nice to have / bad idea" buckets. Extension properties were in the "gotta have" bucket. We then looked at our available budget -- which was not so much measured in dollars as in available designers, developers, testers, writers and management multiplied by available time -- and determined that we did not have the resources to do more than about half the things in the "gotta have" bucket. So we cut half that stuff. Extension properties made it past that cut.

We then designed the feature. We had many hours of debate about the proposed syntax on the declaration side, how to call an extension property getter or setter "directly" as a static method, and so on. We came up with a syntax we could agree was acceptable, designed the semantics, wrote up a draft specification, and started writing code and test plans.

By the time the code was in reasonable shape -- not yet seriously tested, but usable and compliant with the spec -- we'd been having meetings with the WPF team. WPF developers were the assumptive primary consumers of extension properties. WPF already has a mechanism that resembles extension properties; it would be nice to unify that mechanism with our mechanism. Unfortunately, after taking a deep look at their real-world scenarios, we came to the disappointing conclusion that we had designed the wrong thing; this was not actually a feature that would solve their problems.

This is, admittedly, in a sense a failure of the design process. In retrospect, we should have gotten input and feedback from the primary customer much earlier. Had we done that then they could have either influenced the design to make something that would work for them, or we could have cut the feature without taking on the expense of writing spec, code and test plans.

But when you're in an unfortunate situation, you've got to decide to stop throwing good money after bad. Rather than take on the additional costs -- testing, documentation, shipping, and then maintenance of the feature for the rest of the life of the language -- in exchange for a feature that did not serve the needs of our customers, we cut the feature. At that point we did not feel confident that we had enough cycles remaining to redesign the feature the right way.

Therefore, sadly, no extension properties in C# 4. Perhaps in a hypothetical future version of C#.

What's the difference, part one: Generics are not templates

Eric Lippert — Thu, 30 Jul 2009 16:58:00 GMT

Because I'm a geek, I enjoy learning about the sometimes-subtle differences between easily-confused things. For example:

I'm still not super-clear in my head on the differences between a hub, router and switch and how it relates to the gnomes that live inside of each.
Hunks of minerals found in nature are rocks; as soon as you put them in a garden or build a bridge out of them, suddenly they become stones.
When a pig hits 120 pounds, it's a hog.

I thought I might do an occasional series on easily confounded concepts in programming language design.

Here’s a question I get fairly often:

public class C
{
public static void DoIt<T>(T t)
{
 ReallyDoIt(t);
}
private static void ReallyDoIt(string s)
{
 System.Console.WriteLine("string");
}
private static void ReallyDoIt<T>(T t)
{
 System.Console.WriteLine("everything else");
}
}

What happens when you call C.DoIt<string>? Many people expected that “string” is printed, when in fact “everything else” is always printed, no matter what T is.

The C# specification says that when you have a choice between calling ReallyDoIt<string>(string) and ReallyDoIt(string) – that is, when the choice is between two methods that have identical signatures, but one gets that signature via generic substitution – then we pick the “natural” signature over the “substituted” signature. Why don’t we do that in this case?

Because that’s not the choice that is presented. If you had said

ReallyDoIt("hello world");

then we would pick the “natural” version. But you didn’t pass something known to the compiler to be a string. You passed something known to be a T, an unconstrained type parameter, and hence it could be anything. So, the overload resolution algorithm reasons, is there a method that can always take anything? Yes, there is.

This illustrates that generics in C# are not like templates in C++. You can think of templates as a fancy-pants search-and-replace mechanism. When you say DoIt<string> in a template, the compiler conceptually searches out all uses of “T”, replaces them with “string”, and then compiles the resulting source code. Overload resolution proceeds with the substituted type arguments known, and the generated code then reflects the results of that overload resolution.

That’s not how generic types work; generic types are, well, generic. We do the overload resolution once and bake in the result. We do not change it at runtime when someone, possibly in an entirely different assembly, uses string as a type argument to the method. The IL we’ve generated for the generic type already has the method its going to call picked out. The jitter does not say “well, I happen to know that if we asked the C# compiler to execute right now with this additional information then it would have picked a different overload. Let me rewrite the generated code to ignore the code that the C# compiler originally generated...” The jitter knows nothing about the rules of C#.

Essentially, the case above is no different from this:

public class C
{
public static void DoIt(object t)
{
    ReallyDoIt(t);
}
private static void ReallyDoIt(string s)
{
    System.Console.WriteLine("string");
}
private static void ReallyDoIt(object t)
{
    System.Console.WriteLine("everything else");
}
}

When the compiler generates the code for the call to ReallyDoIt, it picks the object version because that’s the best it can do. If someone calls this with a string, then it still goes to the object version.

Now, if you do want overload resolution to be re-executed at runtime based on the runtime types of the arguments, we can do that for you; that’s what the new “dynamic” feature does in C# 4.0. Just replace “object” with “dynamic” and when you make a call involving that object, we’ll run the overload resolution algorithm at runtime and dynamically spit code that calls the method that the compiler would have picked, had it known all the runtime types at compile time.

Reserved and Contextual Keywords

Eric Lippert — Mon, 11 May 2009 18:07:00 GMT

Many programming languages, C# included, treat certain sequences of letters as “special”.

Some sequences are so special that they cannot be used as identifiers. Let’s call those the “reserved keywords” and the remaining special sequences we’ll call the “contextual keywords”. They are “contextual” because the character sequence might one meaning in a context where the keyword is expected and another in a context where an identifier is expected.*

The C# specification defines the following reserved keywords:

abstract as base bool break byte case catch char checked class const continue decimal default delegate do double else enum event explicit extern false finally fixed float for foreach goto if implicit in int interface internal is lock long namespace new null object operator out override params private protected public readonly ref return sbyte sealed short sizeof stackalloc static string struct switch this throw true try typeof uint ulong unchecked unsafe ushort using virtual void volatile while

The implementation also reserves the magic keywords __arglist __makeref __reftype __refvalue which are for obscure scenarios that I might blog about in the future.

Those are the keywords that we reserved in C# 1.0; no new reserved keywords have been added since. It is tempting to do so, but we always resist. Were we to add a new reserved keyword then any program that used that keyword as an identifier would break upon recompilation. Yes, you can always use a keyword as an identifier if you really want: @typeof @goto = @for.@switch(@throw); is perfectly legal, though more than a little weird. But we prefer to avoid as many breaking changes as possible.

We also have a whole bunch of contextual keywords.

The “preprocessor” † uses all the directives (#define, and so on) which of course were never valid identifiers in the first place. But it also uses contextual keywords hidden default disable restore checksum.

C# 1.0 had contextual keywords get set value add remove.

C# 2.0 added where partial global yield alias.

C# 3.0 added from join on equals into orderby ascending descending group by select let var.

C# 4.0 will add dynamic.

Every time we add one of these we need to carefully design the grammar so that if possible, the use of the new contextual keyword does not possibly change the meaning of an existing program which used it.

For example, when defining a partial class, the partial must go immediately before the class. Since there was never a legal C# 1.0 program where partial appeared immediately before class, we knew that adding this new feature to the grammar would not possibly break any existing programs.

Or, another example. Consider var x = 1; – that could have been a legal C# 2.0 program if there was a type called var with a user-defined implicit conversion from int. The semantic analyzer for declaration statements checks to see whether there is a type called var that is accessible at the declaration; if there is then the normal declaration rules are used. Only if there is not such a type can we do the analysis as an implicitly typed local declaration.

One might wonder why on earth we added five contextual keywords to C# 1.0, when there was no chance of breaking backwards compatibility. Why not just make get set value add remove into “real” keywords?

Because we could easily get away with making them contextual keywords, and it seemed likely that real people would want to name variables or methods things like get, set, value, add or remove. So we left them unreserved as a courtesy.

Those were easy to make contextual, unlike, say, return. That’s a lot harder to make a contextual keyword because then return (10); would be ambiguous; is that calling the method named “return” or returning ten? So we didn’t make any of the other reserved keywords into contextual keywords.

*******

(*) An unfortunate consequence of this definition is that using is said to be a reserved keyword even though its meaning depends on its context; whether the using begins a directive or a statement determines its meaning.

(†) An unfortunate name, since “preprocessing” is not done before regular language processing. In C#, the so-called “preprocessing” happens during lexical analysis.

Zip Me Up

Eric Lippert — Thu, 07 May 2009 17:19:00 GMT

Suppose you’ve got a sequence of Foos and you want to project from that a sequences of Bars. That’s straightforward using LINQ:

IEnumerable<Bars> bars = from foo in foos select MakeBar(foo);

or, without the query sugar:

IEnumerable<Bars> bars = foos.Select(foo=>MakeBar(foo));

But what if you have two sequences that you want to project from? Say you’ve got two sequences of doubles that are the same length, and you want to project a sequences of points.

The operation of “project from two sequences of the same length” is called a “zip join”, because its like doing up a zipper – you need each member of the left sequence to correspond to a member of the right sequence. In the version of the base class library that will ship with C# 4.0, we’ll be adding a Zip sequence operator to the standard extension methods.

The code to do so is pretty trivial; if you happen to need this in C# 3.0, I’ve put the source code below.

We considered adding a query syntax for zip joins. Something like

from foo in foos, bar in bars select MakeBlah(foo, bar)

from (foo, bar) in (foos, bars) select MakeBlah(foo, bar)

or some such thing, which would be syntactically transformed into

foos.Zip(bars, (foo, bar)=>MakeBlah(foo, bar))

However, we decided that adding a query syntax for a relatively obscure operation that is not actually supported by most relational databases was not worth the cost of complicating the grammar and the syntactic translation rules.

Here’s the source code:

public static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>
 (this IEnumerable<TFirst> first,
 IEnumerable<TSecond> second,
 Func<TFirst, TSecond, TResult> resultSelector)
{
 if (first == null) throw new ArgumentNullException("first");
 if (second == null) throw new ArgumentNullException("second");
 if (resultSelector == null) throw new ArgumentNullException("resultSelector");
 return ZipIterator(first, second, resultSelector);
}

private static IEnumerable<TResult> ZipIterator<TFirst, TSecond, TResult>
 (IEnumerable<TFirst> first,
 IEnumerable<TSecond> second,
 Func<TFirst, TSecond, TResult> resultSelector)
{
 using (IEnumerator<TFirst> e1 = first.GetEnumerator())
 using (IEnumerator<TSecond> e2 = second.GetEnumerator())
 while (e1.MoveNext() && e2.MoveNext())
 yield return resultSelector(e1.Current, e2.Current);
}

Locks and exceptions do not mix

Eric Lippert — Fri, 06 Mar 2009 19:41:00 GMT

A couple years ago I wrote a bit about how our codegen for the lock statement could sometimes lead to situations in which an unoptimized build had different potential deadlocks than an optimized build of the same source code. This is unfortunate, so we've fixed that for C# 4.0. However, all is still not rainbows, unicorns and Obama, as we'll see.

Recall that lock(obj){body} was a syntactic sugar for

var temp = obj;
Monitor.Enter(temp);
try { body }
finally { Monitor.Exit(temp); }

The problem here is that if the compiler generates a no-op instruction between the monitor enter and the try-protected region then it is possible for the runtime to throw a thread abort exception after the monitor enter but before the try. In that scenario, the finally never runs so the lock leaks, probably eventually deadlocking the program. It would be nice if this were impossible in unoptimized and optimized builds.

In C# 4.0 we've changed lock so that it now generates code as if it were

bool lockWasTaken = false;
var temp = obj;
try { Monitor.Enter(temp, ref lockWasTaken); { body } }
finally { if (lockWasTaken) Monitor.Exit(temp); }

The problem now becomes someone else's problem; the implementation of Monitor.Enter takes on responsibility for atomically setting the flag in a manner that is immune to thread abort exceptions messing it up.

So everything is good now, right?

Sadly, no. It's consistently bad now, which is better than being inconsistently bad. But there's an enormous problem with this approach. By choosing these semantics for "lock" we have made a potentially unwarranted and dangerous choice on your behalf; implicit in this codegen is the belief that a deadlocked program is the worst thing that can happen. That's not necessarily true! Sometimes deadlocking the program is the better thing to do -- the lesser of two evils.

The purpose of the lock statement is to help you protect the integrity of a mutable resource that is shared by multiple threads. But suppose an exception is thrown halfway through a mutation of the locked resource. Our implementation of lock does not magically roll back the mutation to its pristine state, and it does not complete the mutation. Rather, control immediately branches to the finally, releasing the lock and allowing every other thread that is patiently waiting to immediately view the messed-up partially mutated state! If that state has privacy, security, or human life and safety implications, the result could be very bad indeed. In that case it is possibly better to deadlock the program and protect the messed-up resource by denying access to it entirely. But that's obviously not good either.

Basically, we all have a difficult choice to make whenever doing multi-threaded programming. We can (1) automatically release locks upon exceptions, exposing inconsistent state and living with the resulting bugs (bad) (2) maintain locks upon exceptions, deadlocking the program (arguably often worse) or (3) carefully implement the bodies of locks that do mutations so that in the event of an exception, the mutated resource is rolled back to a pristine state before the lock is released. (Good, but hard.)

This is yet another reason why the body of a lock should do as little as possible. Usually the rationale for having small lock bodies is to get in and get out quickly, so that anyone waiting on the lock does not have to wait long. But an even better reason is because small, simple lock bodies minimize the chance that the thing in there is going to throw an exception. It's also easier to rewrite mutating lock bodies to have rollback behaviour if they don't do very much to begin with.

And of course, this is yet another reason why aborting a thread is pure evil. Try to never do so!

The Future Of C#, Part Five

Eric Lippert — Fri, 07 Nov 2008 19:18:00 GMT

When we're designing a new version of the language, something we think about a lot is whether the proposed new features "hang together". Having a consistent "theme" to a release makes it easier for us to prioritize and organize proposals, it makes it easier for our marketing and user education people to effectively communicate with customers, it's just all-around goodness.

If you look at C# 2.0, it was a bit of a grab-bag. The big features were clustered around the notion of enabling rich, typesafe programming with abstract data types that represent collections of data -- and thus generic types and iterator blocks. But there was a whole lot of other stuff in there as well: implementing anonymous methods was a major feature that doesn't fit well with this theme. And there were other more minor features as well: partial classes, improvements to properties, and so on.

With C# 3.0, the theme was very clear: language-integrated query. Anything that did not directly support LINQ was immediately made lower priority. It is rather amazing to me that partial methods and auto-implemented properties got in at all; that they were relatively easy features to design, implement, test and document was what saved them.

What then is the theme of C# 4.0? Again, it seems like rather a grab-bag: covariance and contravariance, improved interop with dynamic languages, improved interop with legacy COM object models, named and optional parameters. It also seems like a pretty small set of new features compared to generics or query comprehensions.

That was deliberate. Some feedback that we received loud and clear throught the C# 3.0 ship cycle was "this is awesome, we need these language features immediately!" and, somewhat contradictorily, "please stop fundamentally changing the way I think about programming every couple years!" Rather than trying to find some way to yet again radically increase the expressiveness and power of the language, we decided to spend a cycle on making what we already have work better with the other stuff in our programming platform infrastructure.

"Now actually works the way you'd expect it to" is not really a theme that gets people excited, but sometimes you've got to stop running forward at full speed and take some time to fix the existing stuff that is annoying a lot of people. (When I was on the VSTO team I petitioned the C# team to please, please make ref parameters optional on calls to legacy COM object models, but they were too busy with designing LINQ; I'm delighted that we've finally gotten that in.)

We also want to make sure that we are anticipating the problems that people are about to face and mitigate them now. We know that dynamic languages and object models designed with dynamism in mind are becoming increasingly popular. Given that there will be stronger demand for statically typed C# to interoperate with them in the future, let's get dynamic programming interoperability in there proactively, rather than be reactive about it later.

Looking forward, it's not clear what exactly the theme of future (hypothetical!) versions of the language will be. The expected onslaught of cheap hardware parallelism looms large in our minds, so that's a possible theme. Enabling metaprogramming is another possible theme on our minds, thought it is not at all clear how that would happen. (Make C# its own metalanguage? Extend expression trees to statement trees, declaration trees, and so on? Open up the internals of the compiler and provide an object model that lets people generate programs directly? It is hard to say what direction is the right one to go in here.) Fortunately, people way smarter than I am are thinking about these things.

The Future of C#, Part Four: Yet More Video

Eric Lippert — Sun, 02 Nov 2008 18:07:00 GMT

Mads, Chris, Sam and me on Channel Nine. Enjoy.

The Future of C#, Part Three: On video, plus the dead-trees edition

Eric Lippert — Wed, 29 Oct 2008 19:48:00 GMT

Amazingly enough, it has happened again. Why does this keep happening to me?

Anders, Mads and Eric on C# 4.0:

C# 4.0 with Anders Hejlsberg, Mads Torgersen, and Eric Lippert - Part 1 of 2

C# 4.0 with Anders Hejlsberg, Mads Torgersen, and Eric Lippert - Part 2 of 2

Anders, Mads and Scott on the new dead-trees edition of the C# 3.0 specification:

The C# Programming Language, 3rd Edition

Enjoy!

The Future of C#, Part Two

Eric Lippert — Tue, 28 Oct 2008 18:01:00 GMT

Well, I intended to spend the last three weeks blogging about C# design process in anticipation of our announcements at the PDC, and then I got crazy busy at work and never managed to do so!

As I'm sure you know by now, we have announced the existence and feature set of that hitherto hypothetical language C# 4.0. Of course I'll be blogging extensively about that over the next year; today though I want to pick up where I left off last time and talk about the tragedy of Object Happiness Disease.

OHD is a disorder believed to be related to Thread Happiness Disease, the belief that threads are awesome and that you should therefore make as many as you possibly can. OHD is an even more common condition which afflicts developers at all levels of experience. Symptoms of OHD in C# developers include:

making utterances like "I don't like extension methods because they're not object-oriented"
requesting that multiple class inheritance be added to the language
writing interfaces only intended to be implemented by one class
designing every class to enable derivation, whether it needs it or not
and so on.

In short, the Object Happy believe that object-oriented programming is a good in of itself and that OOP principles are objectively (ha ha) good principles.

I don't share these beliefs.

Now, don't get me wrong. I believe that OOP is frequently useful. In fact, it is frequently the best available tool for a great many real-world jobs. OOP works particularly well when you have:

multi-person teams of software enginneers,
working on solving practical problems,
by logically associating the problem data with the functions performed upon them,
in a type hierarchy that sensibly models the "is a" and "can do" relationships between the various parts.

And that is where its goodness comes from: OOP techniques make real people more productive in solving real problems and thereby creating value. The techniques are not good in of themselves, they're good because they're practical.

Perhaps surprisingly, our goal in making C# is not to make an object-oriented language. Our goal is to make a compelling and practical language for general-purpose application development on our platforms, and thereby enable our customers to be successful. Making an object-oriented language called C# is just a means to that end, not an end in of itself.

So, yes, the oft-heard criticism that "extension methods are not object-oriented" is entirely correct, but also rather irrelevant. Extension methods certainly are not object-oriented. They put the code that manipulates the data far away from the code that declares the data, they cannot break encapsulation and talk to the private state of the objects they appear to be methods on, they do not play well with inheritance, and so on. They're procedural programming in a convenient object-oriented dress.

They're also incredibly convenient and make LINQ possible, which is why we added them. The fact that they do not conform to some philosophical ideal of what makes an object-oriented language was not really much of a factor in that decision.

Here's another way to think about it.

I had a discussion with Jim Miller a while back about the meaning of the term "functional language" in the context of Scheme. To paraphrase somewhat, Jim's position was along the lines that a "functional language" is one in which the design of the language makes it easy to program in a functional style and difficult (or impossible!) to program in a non-functional style. For example, Scheme does allow mutation of shared state -- a very non-functional-style feature -- but the convention in Scheme is to call attention to the fact that you are doing so by marking mutating functions with a "!". It is possible to program in an object-oriented style in Scheme, but the language resists you at every turn.

I certainly see the merits of Jim's position, but I take a weaker stance. When I say "functional language", I mean a language in which it is possible without undue difficulty to program in a functional style. I do not require the language to work against you if you try to program in a non-functional style in it. For example, I would call JScript a functional language. It is certainly possible to program in a non-functional style in JScript, but you can also treat JScript as Scheme with a slightly goofy syntax if you really want to.

I think about "object-oriented language" the same way; C# is an object-oriented language not because every feature of the language strictly adheres to OO philosophy. Rather, because as a practical matter it is possible to program in an OO style in C# if that's what you want. Adding features that enable other styles of programming does not make C# less of an OO language.

Next time, I'll talk a bit about the "theme" of C# 4.0 and how that influenced the design process. (And if you absolutely positively cannot wait to hear about some of the new language feature in detail, try my colleague Chris Burrows' blog.)

The Future of C#, Part One

Eric Lippert — Wed, 08 Oct 2008 18:06:00 GMT

An attentive reader pointed me at this long thread on a third-party forum where some people are musing about possible future features of C#. There is far, far more here than I possibly have time to respond to in any kind of detail.

Also, I am not going to spill the beans. Anders is giving a talk at the PDC at the end of the month. His talk is titled "The Future of C#" and I'm not going to steal Anders' thunder (or give our marketing department conniptions) by giving specific details at this time.

Of course, I note also that we have not announced that there even will be a language called C# 4. We have not announced that there will be new versions of Visual Studio or the CLR either. But we also know that you people who read blogs about programming languages are not idiots. I posted a link to a video a while back where Charles interviews the C# design team; the fact that there is a design team is evidence that we're designing something. I've been posting about adding more covariance and contravariance to the type system, Charlie has been collecting comments about what kinds of interactions people would like to see between C# and dynamic languages. Clearly we're up to something here, and it is not hard to figure out the broad outlines.

This puts me in a bit of a tight spot -- I want to respond to all these great ideas, I want to inform the community about our plans and get feedback early, but at the same time I want to make sure that we don't overpromise and underdeliver, and I want to ensure that we provide information that is fully-baked and professionally presented.

Therefore, I want to talk about these ideas in the context of the design process we go through here, rather than in the context of specific implementation decisions of specific features.

I made a new friend at a birthday party I attended the other day. We got to making small talk about our jobs. I mentioned that I designed programming languages as part of my job. Now, let me tell you, when I say that at a party I get one of three responses: (1) that's really cool, (2) excuse me, I need to refill this drink, or (3) a puzzled look, followed by "you mean, someone actually designs programming languages? I thought they just, you know, sprang into existence... somehow." I informed my new friend that no, in fact there is a huge amount of design that goes into a language -- six hours a week of meetings for almost ten years, in fact, for C# alone.

I've already linked several times to Eric Gunnerson's great post on the C# design process. The two most important points in Eric's post are: (1) this is not a subtractive process; we don't start with C++ or Java or Haskell and then decide whether to leave some feature of them out. And (2) just being a good feature is not enough. Features have to be so compelling that they are worth the enormous dollar costs of designing, implementing, testing, documenting and shipping the feature. They have to be worth the cost of complicating the language and making it more difficult to design other features in the future.

After we finished the last-minute minor redesigns of various parts of C# 3.0, we made a list of every feature we could think of that could possibly go into a future version of C#. We spent many, many hours going through each feature on that list, trying to "bucket" it. Each feature got put into a unique bucket. The buckets were labelled:

Pri 1: Must have in the next version
Pri 2: Should have in the next version
Pri 3: Nice to have in the next version
Pri 4: Likely requires deep study for many years before we can do it
Pri 5: Bad idea

Obviously we immediately stopped considering the fours and fives in the context of the next version. We then added up the costs of the features in the first three buckets, compared them against the design, implementation, testing and documenting resources we had available. The costs were massively higher than the resources available, so we cut everything in bucket 2 and 3, and about half of what was in bucket 1. Turns out that some of those "must haves" were actually "should haves".

Understanding this bucketing process will help when I talk about some of the features suggested in that long forum topic. Many of the features suggested were perfectly good, but fell into bucket 3. They didn't make up the 100 point deficit, they just weren't compelling enough.

Next time, I think I'll talk a bit about the design process in the context of OOP.

A Face Made For Email, Part Four

Eric Lippert — Mon, 14 Jul 2008 20:32:00 GMT

Good heavens, this just keeps on happening to me.

If you're interested in what we're musing about for future versions of C#, check out this video of Anders Hejlsberg, Scott Wiltamuth, Paul Vick, Mads Torgersen, Matt Warren, Jim Hugunin and, off in one corner, me.

Working with this caliber of people every day is a huge part of what makes this the best job ever.

http://channel9.msdn.com/posts/Charles/C-40-Meet-the-Design-Team/