Fabulous Adventures In Coding

Making the code read like the spec

Eric Lippert — Mon, 08 Feb 2010 14:58:00 GMT

As I mentioned a while back, there are some bugs in the compiler code which analyzes whether a set of classes violates the “no cycles” rules for base classes. (That is, a class is not allowed to inherit from itself, directly or indirectly, and not allowed to inherit from one of its own nested classes, directly or indirectly.) The bugs are almost all of the form where we accidentally detect a cycle in suspicious-looking generic code but in fact there is no cycle; the bugs are the result of an attempt to modify the cycle detector from C# 1 rather than simply rewriting it from scratch to handle generics. Unfortunately we were unable to get the fixes into C# 4; these are obscure corner-case scenarios and the risk of doing the fix was considered too high.

There are a number of tricky issues here, mostly around the fact that obviously we cannot know whether a set of base types are circular until we know what the base types are, but resolving the program text to determine what type the base type string “C.D<E.F>” requires us to know the base types of C, because D might be a nested type of C’s base class, not of C, so we have a bit of a chicken-and-egg problem. The code which turns strings into types has to be robust in the face of circular base types because the base type cycle detector depends on its output!

So like I said, I’ve come up with a new algorithm that implements the spec more exactly, and I wanted to test it out. Rather than modifying the existing compiler to use it, I mocked it up in C# quickly first, just to give me something to play with. One of the problems that we have with the existing compiler is that it is not at all clear which parts of the code are responsible for implementing any given line in the spec. In my “maquette” of the compiler I wanted to make sure that I really was exactly implementing the spec; that might show up logical problems with either the implementation or the spec. I therefore wanted the code to read much like the spec.

This little hunk of code that I wrote made me inordinately happy. Here’s the spec:

A class directly depends on its direct base class (if any) and directly depends on the class within which it is immediately nested (if any). The complete set of classes upon which a class depends is the reflexive and transitive closure of the directly-depends-upon relationship.

First off, what is this thing called the “reflexive and transitive closure”?

Consider a “relation” – a function that takes two things and returns a Boolean that tells you whether the relation holds. A relation, call it ~>, is reflexive if X~>X is true for every X. It is symmetric if A~>B necessarily implies that B~>A. And it is transitive if A~>B and B~>C necessarily implies that A~>C. (*)

For example, the relation “less than or equal to” on integers is reflexive: X≤X is true for all X. It is not symmetric: 1≤2 is true, but 2≤1 is false. And it is transitive: if A≤B and B≤C then it is necessarily true that A≤C.

The relation “is equal to” is reflexive, symmetric and transitive; a relation with all three properties is said to be an “equivalence relation” because it allows you to partition a set into mutually-exclusive “equivalence classes”.

The relation “is the parent of” on people is not reflexive: no one is their own parent. It is not symmetric: if A is the parent of B, then B is not the parent of A. And it is not transitive: if A is the parent of B and B is the parent of C, then A is not the parent of C. (Rather, A is the grandparent of C.)

It is possible to take a nontransitive relation like “is the parent of” and from it produce a transitive relation. Basically, we simply make up a new relation that is exactly the same as the parent relation, except that we enforce that it be transitive. This is the “is the ancestor of” relation: if A is the ancestor of B, and B is the ancestor of C, then A is necessarily the ancestor of C. The “ancestor” relation is said to be the transitive closure of the “parent” relation.

Similarly we can define the reflexive closure, and so on.

When we’re talking about closures, we’re often interested not so much in the relation itself as the set of things which satisfy the relation with a given item. That’s what we mean in the spec when we say “The complete set of classes upon which a class depends is the reflexive and transitive closure of the directly-depends-upon relationship.” Given a class, we want to compute the set that contains the class itself (because the closure is reflexive), its base class, its outer class, the base class of the base class, the outer class of the base class, the base class of the outer class, the outer class of the outer class… and so on.

So the first thing I did was I wrote up a helper method that takes an item and a function which identifies all the items that have the non-transitive relation with that item, and computes from that the set of all items that satisfy the transitive closure relation with the item:

static HashSet<T> TransitiveClosure<T>(
    this Func<T, IEnumerable<T>> relation,
    T item)
{
    var closure = new HashSet<T>();
    var stack = new Stack<T>();
    stack.Push(item);
    while(stack.Count > 0)
    {
        T current = stack.Pop();
        foreach(T newItem in relation(current))
        {
            if (!closure.Contains(newItem))
            {
                closure.Add(newItem);
                stack.Push(newItem);
            }
        }
    }
    return closure;
}
static HashSet<T> TransitiveAndReflexiveClosure<T>(
    this Func<T, IEnumerable<T>> relation,
    T item)
{
var closure = TransitiveClosure(relation, item);
closure.Add(item);
return closure;
}

Notice that essentially what we’re doing here is a depth-first traversal of the graph defined by the transitive closure relation, avoiding descent into regions we’ve visited before.

Now I have a mechanism which I can use to write code that implements the policies described in the specification.

static IEnumerable<TypeSymbol> DirectDependencies(TypeSymbol symbol)
{
    // SPEC: A class directly depends on its direct base class (if any) ...
    if (symbol.BaseTypeSymbol != null)
        yield return symbol.BaseTypeSymbol;
    // SPEC: ...and directly depends on the class within which it
    // SPEC: is immediately nested (if any).
    if (symbol.OuterTypeSymbol!= null)
        yield return symbol.OuterTypeSymbol;
}

Great, we now have a method that exactly implements one sentence of the spec – given a type symbol, we can determine what its direct dependencies are. Now we need another method to implement the next sentence of the spec:

static HashSet<TypeSymbol> Dependencies(TypeSymbol classSymbol)
{
    // SPEC: The complete set of classes upon which a class
    // SPEC: depends is the reflexive and transitive closure of
    // SPEC: the directly-depends-upon relationship.
    return TransitiveAndReflexiveClosure(DirectDependencies, classSymbol);
}

That’s what I like to see: code that reads almost exactly like the spec.

Note also that I’ve made the design choice here to make these methods static methods of a policy class, rather than methods on the TypeSymbol class. I want to keep my policies logically and textually separated from my mechanisms. For example, I could be using the same classes that represent classes, structs, namespaces, and so on, to implement VB instead of C#. I want the policies of the C# language to be in a class whose sole responsibility is implementing these policies.

Another nice aspect of this approach is that I can now re-use my transitive closure computing mechanism when I come across this bit of the spec that talks about base interfaces of an interface:

The set of base interfaces is the transitive closure of the explicit base interfaces.

Unsurprisingly, the code in my prototype that computes this looks like:

static HashSet<TypeSymbol> BaseInterfaces(TypeSymbol interfaceSymbol)
{
    // SPEC: The set of base interfaces is the transitive closure
    // SPEC: of the explicit base interfaces.
    return TransitiveClosure(ExplicitBaseInterfaces, interfaceSymbol);
}

In fact, there are transitive closures in a number of places in the C# spec, and now I have a mechanism that I can use to implement all of them, should I need to for my prototype.

One final note: notice that I am returning a mutable collection here. Were I designing these methods for public consumption, I would probably choose to return an immutable set, rather than a mutable one, so that I could cache the result safely; these methods could be memoized since the set of dependencies does not change over time. But for the purposes of my quick little prototype, I’m just being lazy and returning a mutable set and not caching the result.

Hopefully we'll get this new algorithm into the hypothetical next compiler release.

*************

(*) Incidentally, we are holding on to the wavy arrow ~> as a potential operator for future hypothetical versions of C#. We recently considered it to have the meaning a~>b() means ((dynamic)a).b(). The operator is known as “the naj operator” after Cyrus Najmabadi, who advocated the use of this operator to the C# design team. Anyone who has a great idea for what the naj should mean, feel free to leave a comment.

How Many Passes?

Eric Lippert — Thu, 04 Feb 2010 14:52:00 GMT

Large bodies of code written in the C/C++ languages typically divide up the code into “header” files, which are just declarations of methods and types (and definitions of macros). The actual bodies of those methods and types are in completely different files.

People sometimes ask me why doesn’t C# need header files? which is a bit of an odd way to phrase the question; I would have asked the equivalent question why does C++ need header files? Header files seem like a huge potential point of failure; all the time I edit C++ code and change the signature of a method; if I forget to update the header file, then the code doesn’t compile and often gives some cryptic error message. Hopefully this large cost actually buys you something.

It buys the compiler writer one thing, and the user one thing.

What it buys the user is that you can compile each individual “cpp” file into a “obj” file independently, provided that you have all the necessary headers. All the information necessary to generate the bodies that are in a given cpp file can be gleaned from the set of headers. This means that the build system can recompile just those cpp files that changed, provided that no header changed.

What it buys the compiler writer is that every file can be compiled in “one pass”. Because every type and method declaration is processed before its first usage, the compiler can simply start from the top of the file, pull in all the included headers, and proceed from the top down, spitting out the obj file as it goes, never having to go back and revisit something its seen already.

The C# language does not require that declarations occur before usages, which has two impacts, again, on the user and on the compiler writer. The impact on the user is that you don’t get to recompile just the IL that changed when you change a file; the whole assembly is recompiled. Fortunately the C# compiler is sufficiently fast that this is seldom a huge issue. (Another way to look at this is that the “granularity” of recompilation in C# is at the project level, not the file level.)

The impact on the compiler writer is that we have to have a “two pass” compiler. In the first pass, we look for declarations and ignore bodies. Once we have gleaned all the information from the declarations that we would have got from the headers in C++, we take a second pass over the code and generate the IL for the bodies.

A good way to think about this is that the first pass computes metadata: the “top level” stuff, like namespaces, classes, structs, enums, interfaces, delegates, methods, type parameters, formal parameters, constructors, events, attributes, and so on. The second pass computes the IL: the code that goes in the method bodies, constructor bodies, and so on.

In practice, though we only do “two passes” over the raw text, we do many, many passes over the resulting data structures. I thought I might lay out for you just what passes we do in order to implement the C# language.

The first “metadata” phase goes like this:

The first thing we do is take the text of the sources and break it up into a stream of tokens. That is, we do lexical analysis to determine that "class c : b { }" is class, identifier, colon, identifier, left curly, right curly.

We then do a "top level parse" where we verify that the token streams define a grammatically-correct C# program. However, we skip parsing method bodies. When we hit a method body, we just blaze through the tokens until we get to the matching close curly. We'll come back to it later; we only care about getting enough information to generate metadata at this point.

We then do a "declaration" pass where we make notes about the location of every namespace and type declaration in the program. At this point we're done looking at the source code for the first phase; every subsequent pass is over the set of "symbols" deduced from the declarations.

We then do a pass where we verify that all the types declared have no cycles in their base types. We need to do this first because in every subsequent pass we need to be able to walk up type hierarchies without having to deal with cycles.

We then do a pass where we verify that all generic parameter constraints on generic types are also acyclic.

We then do a pass where we check whether every member of every type -- methods of classes, fields of structs, enum values, and so on -- is consistent. No cycles in enums, every overriding method overrides something that is actually virtual, and so on. At this point we can compute the "vtable" layouts of all interfaces, classes with virtual methods, and so on.

We then do a pass where we work out the values of all "const" fields.

At this point we have enough information to emit almost all the metadata for this assembly. We still do not have information about the metadata for iterator/anonymous function closures or anonymous types; we do those late.

We can now start generating IL. For each method body (and properties, indexers, constructors, and so on), we rewind the lexer to the point where the method body began and parse the method body.

[UPDATE: A number of people have asked me about how the passes below are optimized. During the initial binding pass we make notes on things like "there's nullable arithmetic in here", or "there's an object initializer in here" and so on. When we go to do object initializer rewriting, say, we first check to see if we even have to, and skip the pass entirely if we know it will be a no-op. (And testing runs a special version of the compiler that does the pass anyway, and reports back whether it had any effect or not, so that we can test that our pass-skipping logic is correct.) ]

Once the method body is parsed, we do an initial "binding" pass, where we attempt to determine the types of every expression in every statement. We then do a whole pile of passes over each method body. Again, at this point we are done looking at the source code; we're now passing over the "annotated parse tree" many times, and rewriting it as we go.

We first run a pass to transform loops into gotos and labels.

(The next few passes look for bad stuff.)

Then we run a pass to look for use of deprecated types, for warnings.

Then we run a pass that searches for uses of anonymous types that we haven't emitted metadata for yet, and emit those.

Then we run a pass that searches for bad uses of expression trees. For example, using a ++ operator in an expression tree.

Then we run a pass that looks for all local variables in the body that are defined, but not used, to report warnings.

Then we run a pass that looks for illegal patterns inside iterator blocks.

Then we run the reachability checker, to give warnings about unreachable code, and tell you when you've done something like forgotten the return at the end of a non-void method.

Then we run a pass that verifies that every goto targets a sensible label, and that every label is targeted by a reachable goto.

Then we run a pass that checks that all locals are definitely assigned before use, notes which local variables are closed-over outer variables of an anonymous function or iterator, and which anonymous functions are in reachable code. (This pass does too much. I have been meaning to refactor it for some time now.)

At this point we're done looking for bad stuff, but we still have way more passes to go before we sleep.

Next we run a pass that detects missing ref arguments to calls on COM objects and fixes them. (This is a new feature in C# 4.)

Then we run a pass that looks for stuff of the form "new MyDelegate(Foo)" and rewrites it into a call to CreateDelegate.

Then we run a pass that transforms expression trees into the sequence of factory method calls necessary to create the expression trees at runtime.

Then we run a pass that rewrites all nullable arithmetic into code that tests for HasValue, and so on.

Then we run a pass that finds all references of the form base.Blah() and rewrites them into code which does the non-virtual call to the base class method.

Then we run a pass which looks for object and collection initializers and turns them into the appropriate property sets, and so on.

Then we run a pass which looks for dynamic calls (in C# 4) and rewrites them into dynamic call sites that use the DLR.

Then we run a pass that looks for calls to removed methods. (That is, partial methods with no actual implementation, or conditional methods that don't have their conditional compilation symbol defined.) Those are turned into no-ops.

Then we look for unreachable code and remove it from the tree. No point in codegenning IL for it.

Then we run an optimization pass that rewrites trivial "is" and "as" operators.

Then we run an optimization pass that looks for switch(constant) and rewrites it as a branch directly to the correct case.

Then we run a pass which turns string concatenations into calls to the correct overload of String.Concat.

(Ah, memories. These last two passes were the first things I worked on when I joined the compiler team.)

Then we run a pass which rewrites uses of named and optional parameters into calls where the side effects all happen in the correct order.

Then we run a pass which optimizes arithmetic; for example, if we know that M() returns an int, and we have 1 * M(), then we just turn it into M().

Then we do generation of the code for anonymous types first used by this method.

Then we transform anonymous functions in this body into methods of closure classes.

Finally, we transform iterator blocks into switch-based state machines.

Then we emit the IL for the transformed tree that we've just computed.

UPDATE: I've glossed over a number of additional passes that happen during IL generation. Basically, the IL generator turns the program tree into a set of "basic blocks" -- that is, sequences of code where nothing "in the middle" of the block leaves the sequence, and no other sequences branch to "the middle" of any other sequence. Once we have the code in basic block form we can start doing additional analysis on it: rewriting branches to be more efficient, removing basic blocks that the arithmetic optimizer has proven to be dead code, and so on. See my article on the optimize switch for more details of what we do during this pass.

Easy as pie!

Style follows semantics

Eric Lippert — Mon, 01 Feb 2010 16:29:00 GMT

Which is better style?

bool abc;
if (Foo())
abc = Bar();
else
abc = false;

bool abc = Foo() && Bar();

To me, this comes down to the question “is Bar useful solely for obtaining its value, or also for its side effects?” The stylistic choices should typically be driven by a desire to clearly communicate the semantics of the program fragment.

The metasyntatic names are therefore making this harder to answer, not easier. Suppose the choice were in fact between:

bool loginSuccessful;
if (NetworkAvailable())
loginSuccessful= LogUserOn();
else
loginSuccessful= false;

and

bool loginSuccessful= NetworkAvailable() && LogUserOn();

I would always choose the former, because I want LogUserOn to be in a statement of its own. Statements emphasize “I am useful for my side effects”. Statements emphasize control flow and provide convenient places to put breakpoints.

If however the choice were between

bool canUseCloud;
if (NetworkAvailable())
canUseCloud = UserHasFreeSpaceInCloud();
else
canUseCloud = false;

and

bool canUseCloud = NetworkAvailable() && UserHasFreeSpaceInCloud();

I would always choose the latter; here we’re using && idiomatically to compute a value safely.

Calling constructors in arbitrary places

Eric Lippert — Thu, 28 Jan 2010 15:10:00 GMT

C# lets you call another constructor from a given constructor, but only before the body of the calling constructor runs:

public C(int x) : this(x, null)
{
// …
}
public C(int x, string y)
{
// …
}

Why can you call another constructor at the beginning of a constructor block, but not at the end of the block, or in the middle of the block?

Well, let's break it down into two cases. (1) You're calling a "base" constructor, and (2) you're calling a "this" constructor.

For the "base" scenario, it's quite straightforward. You almost never want to call a base constructor after a derived constructor. That's an inversion of the normal dependency rules. Derived code should be able to depend on the base constructor having set up the "base" state of the object; the base constructor should never depend on the derived constructor having set up the derived state.

Suppose you're in the second case. The typical usage pattern for this scenario is to have a bunch of constructors that take different arguments and then all "feed" into one master constructor (often private) that does all the real work. Typically the public constructors have no bodies of their own, so there's no difference between calling the other constructor "before" or "after" the empty block.

Suppose you're in the second case and you are doing work in each constructor, and you want to call other constructors at some point other than the start of the current constructor.

In that scenario you can easily accomplish this by extracting the work done by the different constructors into methods, and then calling the methods in the constructors in whatever order you like. That is superior to inventing a syntax that allows you to call other constructors at arbitrary locations. There are a number of design principles that support this decision. Two are:

1) Having two ways to do the same thing creates confusion; it adds mental cost. We often have two ways of doing the same thing in C#, but in those situations we want the situation to "pay for itself" by having the two different ways of doing the thing each be compelling, interesting and powerful features that have clear pros and cons. (For example, "query comprehensions" vs "fluent queries" are two very different-looking ways of building a query.) Having a way to call a constructor the way you'd call any other method seems like having two ways of doing something -- calling an initialization method -- but without a compelling or interesting "payoff".

2) We'd have to add new language syntax to do it. New syntax comes at a very high cost; it's got to be designed, implemented, tested, documented -- those are our costs. But it comes at a higher cost to you because you have to learn what the syntax means, otherwise you cannot read or maintain other people's code. That's another cost; again, we only take the huge expense of adding syntax if we feel that there is a clear, compelling, large benefit for our customers. I don't see a huge benefit here.

In short, achieving the desired construction control flow is easy to do without adding the feature, and there's no compelling benefit to adding the feature. No new interesting representational power is added to the language.

Why are unused using directives not a warning?

Eric Lippert — Mon, 25 Jan 2010 15:00:00 GMT

As I’ve discussed before, we try to reserve warnings for only those situations where we can say with almost certainty that the code is broken, misleading or useless. One reason for trying to ensure that warnings are not “false positives” is that we don’t ever want to encourage someone to take working, correct code and break it so as to remove the warning. Another is that since many people compile with “warnings are errors” turned on, we do not want to introduce a whole lot of unnecessary build breakages should the warning turn out to be unwarranted.

An unused using directive is almost certainly not “broken”, and it’s probably not misleading. It certainly is useless though. Why not produce a warning for this, so that you can remove the useless directive from your program?

Suppose we make an unused using into a warnings. Suppose you do have “warnings are errors” turned on. And suppose you create a new console project in Visual Studio. We generate you this code:

using System;
using System.Text;
using System.Linq;

class Program
{
static void Main(string[] arguments)
{
}
}

… which doesn’t compile. Which is better, generating code that doesn't compile, or omitting the using directives and making you type "using System;" the first time you try to use Console.WriteLine?

Either way is a really bad user experience. So, no such feature.

What’s the difference between a destructor and a finalizer?

Eric Lippert — Thu, 21 Jan 2010 14:20:00 GMT

Today, another dialogue, and another episode of my ongoing series "what's the difference?"

What’s the difference, if any, between a “destructor” and a “finalizer”?

Both are mechanisms for cleaning up a resource when it is no longer in use. When I was asked this, at first I didn’t think there was a difference. But some Wikipedia searches turned up a difference; the term “destructor” is typically used to mean a deterministically-invoked cleanup, whereas a “finalizer” runs when the garbage collector says to run it.

Doesn’t that mean that the C# spec uses the term “destructor” incorrectly?

Yes, by these definitions, the C# spec gets it wrong. What we call a “destructor” in the spec is actually a finalizer, and what we call the “Dispose()” method invoked by a “using” statement is in fact a “destructor”.

The CLI spec calls the finalizer by its right name.

Why did the authors of the C# spec get it wrong?

I don't know, but I can guess. I have two guesses.

Guess #1 is that on May 12th, 1999 there was not a Wikipedia article clearly describing the subtle difference between these two concepts. That's because there wasn't a Wikipedia. Remember back when there wasn't a Wikipedia? Dark ages, man. The error might simply have been an honest mistake, believing that the two terms were identical.

Heck, for all I know, the two terms were identical on May 12th, 1999, and the difference in definitions only evolved later, as it became obvious that there was a need to disambiguate between eager/deterministic and lazy/nondeterministic cleanup methods. Anyone who has more historical perspective on this than I do, feel free to chime in here.

Guess #2 is that on May 12th, 1999, the language design committee wished to leave open the possibility that a C# "destructor" could be implemented as something other than a CLR finalizer. That is, the "destructor" was designed to be a C# language concept that did not necessarily map one-to-one with the CLR’s "finalizer" concept.

When designing a language at the same time as the framework it sits atop is also being designed, sometimes you want to insulate yourself against late-breaking design changes in your subsystems. Deliberately preventing name conflation is one way to do that.

What’s your sudden obsession with May 12th, 1999 about?

The language committee's notes for May 12th 1999 read in part:

We're going to use the term "destructor" for the member which executes when an instance is reclaimed. Classes can have destructors; structs can't. Unlike in C++, a destructor cannot be called explicitly. Destruction is non-deterministic – you can't reliably know when the destructor will execute, except to say that it executes at some point after all references to the object have been released. The destructors in an inheritance chain are called in order, from most descendant to least descendant. There is no need (and no way) for the derived class to explicitly call the base destructor. The C# compiler compiles destructors to the appropriate CLR representation. For this version that probably means an instance finalizer that is distinguished in metadata.

Notice that this supports my hypothesis that the language design team was attempting to insulate themselves from becoming tied to a particular CLR term.

A Definite Assignment Anomaly

Eric Lippert — Mon, 18 Jan 2010 14:28:00 GMT

UPDATE: I have discovered that this issue is considerably weirder than the initial bug report led me to believe. I've rewritten the examples in this article; the previous ones did not actually demonstrate the bug.

Consider the following code:

struct S
{
private string blah;
public S(string blah)
{
this.blah = blah;
}
public void Frob()
{ // whatever
}
}

This method body code fragment is legal (though probably ill-advised):

S s1 = new S();
s1.Frob();

Every struct has a default constructor which initializes its fields to their default values; this frobs the zero handle.

What about this?

S s2;
s2.Frob();

Looks like a definite assignment error, doesn’t it?

An interesting and little-known fact about the C# compiler is that this is reported as a definite assignment error if the struct is compiled from source code, but not if the struct is in a referenced assembly! What is the reasoning behind this seemingly anomalous behaviour?

Well, consider the following related situation:

struct SS
{
public int x;
public int y;
public void Bar() {…}
}

Here we have the pure unadulterated evil that is a mutable value type. Is this legal?

SS ss;
ss.x = 123;
ss.y = 456;
ss.Bar();

Yes, that is legal. The local variable s essentially has two sub-variables, x and y. If x and y are both definitely assigned, then s is also considered to be definitely assigned. When checking definite assignment on a struct, the compiler actually checks to see whether we know that every field of the struct has been initialized. A constructor call is assumed to automatically initialize every field, but if there is not a constructor call, then we simply track every field independently and then pool the results. (And if the fields are themselves of value type, we do so recursively of course.)

The anomaly happens because when the compiler sees that the type in question is in source code, it tracks the definite assignment of all the fields, detects that “s1.blah” is an unassigned field, and gives a definite assignment error on s1. But when the type in question is from a referenced assembly, the compiler ignores inaccessible fields of reference type. We see that there were zero fields initialized out of the zero accessible fields that needed to be initialized, and declare that s1 must be fully assigned!

UPDATE: Initially I thought this behaviour was odd and undesirable but plausibly justifiable. However I have since learned that in fact, for reasons still obscure to me, the compiler only ignores inaccessible fields of reference type. Inaccessible fields of value type are still tracked for definite assignment. This weird inconsistency points strongly in the direction of this being a bug, not a feature. Continuing with my original text...

This behaviour is undoubtedly weird; whether it is desirable or not is a judgment call. There certainly is an argument to be made that the user has no ability to do anything about a private field on an imported type; a type they almost certainly did not author themselves, do not know the details of, and cannot change. Whereas if the type is in source code, then the user running the compiler has personal knowledge of its internals. So, the argument goes, we should give the user a pass on having to prove that the fields they cannot access and know nothing about are assigned; just let them be assigned to their default values and don’t worry about them.

There is also the (more compelling to me) argument that we ought to be consistent here and either consider all the fields all the time, or consider only the accessible fields all the time, rather than swapping back and forth between the two choices depending on the details of the compilation process. My choice would be the former; force the user to call the default constructor or use the “default(T)” operator if they really want the all-zero default version of the struct. That makes the intention very clear in the code.

Unfortunately, though I think I’ve argued that this behaviour is plausibly undesirable, we’re stuck with it. There are numerous BCL structs which have no public fields, and lots of existing code which uses them without calling the default constructor. Changing now to make this an error is a breaking change with no compelling benefit, and we’re not going to do that. Particularly since the anomaly is essentially harmless; whether we force you to say “s1 = new S()” or not, the handle field is in practice always initialized to its default value.

Nor do I think we should change this by making the code legal in all cases, even if the type is in source code; that’s making a bad situation worse merely in order to achieve consistency. This seems like the very definition of “foolish consistency”.

Why Can't I Access A Protected Member From A Derived Class? Part Six

Eric Lippert — Thu, 14 Jan 2010 14:42:00 GMT

Reader Jesse McGrew asks an excellent follow-up question to my 2005 post about why you cannot access a protected member from a derived class. (You probably want to re-read that post in order to make sense of this one.)

I want to be clear in my terminology, so I’m going to define some terms. Suppose we have a call foo.Bar() inside class C. The value of foo is the “receiver” of the call. The compile-time type of foo is the “compile time type of the receiver”. The “runtime type of the receiver” could potentially be more derived. The “call site type” is C.

The rule in C# is that if Bar is protected, then the compile-time type of the receiver must be the call site type, or a type derived from the call site type. It cannot be a base type of the call site type, because then the runtime type of the receiver might have a “cousin” (or sibling, or uncle… but let’s not split hairs, let’s just call them all cousins) relationship to the call site type, rather than an “ancestor/descendant” relationship.

Jesse quite rightly points out that my original answer to the question was not really complete. There are two questions unanswered:

1) Why would it be a bad thing to allow calling a protected method from a receiver whose runtime type is a “cousin” class?

2) Supposing for the sake of argument that there is a good answer to (1) – why doesn’t that argument apply equally well to calling a protected method via a receiver whose compile-time type is the same as the call site type? The runtime type of the receiver still could be more derived.

I’ll answer the first question first. In this answer I’m going to use a humorous and exaggerated example to illustrate the problem; I want to emphasize that this is not how you should actually design applications that need “big S” Security, like bank accounts. What I want to illustrate here is simply that allowing protected methods to be called via “cousins” makes it difficult to maintain invariants and therefore difficult to write correct, robust code.

// Good.dll:

    public abstract class BankAccount
    {
      abstract protected void DoTransfer(
        BankAccount destinationAccount,
        User authorizedUser,
        decimal amount);
    }
    public abstract class SecureBankAccount : BankAccount
    {
      protected readonly int accountNumber;
      public SecureBankAccount(int accountNumber)
      {
        this.accountNumber = accountNumber;
      }
      public void Transfer(
        BankAccount destinationAccount,
        User authorizedUser,
        decimal amount)
      {
        if (!Authorized(user, accountNumber)) throw something;
        this.DoTransfer(destinationAccount, user, amount);
      }
    }
    public sealed class SwissBankAccount : SecureBankAccount
    {
      public SwissBankAccount(int accountNumber) : base(accountNumber) {}
      override protected void DoTransfer(
        BankAccount destinationAccount,
        User authorizedUser,
        decimal amount)
      {
        // Code to transfer money from a Swiss bank account here.
        // This code can assume that authorizedUser is authorized.
        // We are guaranteed this because SwissBankAccount is sealed, and
        // all callers must go through public version of Transfer from base
        // class SecureBankAccount.
      }
    }
    // Evil.exe:
    class HostileBankAccount : BankAccount
    {
      override protected void Transfer(
        BankAccount destinationAccount,
        User authorizedUser,
        decimal amount) { }
      public static void Main()
      {
        User drEvil = new User("Dr. Evil");
        BankAccount yours = new SwissBankAccount(1234567);
        BankAccount mine = new SwissBankAccount(66666666);
        yours.DoTransfer(mine, drEvil, 1000000.00m); // compilation error
        // You don't have the right to access the protected member of
        // SwissBankAccount just because you are in a
        // type derived from BankAccount.
      }
    }

Dr. Evil's attempt to steal ONE... MILLION... DOLLARS... from your Swiss bank account has been foiled by the C# compiler.

Obviously this is a silly example, and obviously, fully-trusted code could do anything it wants to your types -- fully-trusted code can start up a debugger and change the code as its running. Full trust means full trust. Again, do not actually design a real security system this way!

But my point is simply that the "attack" that is foiled here is someone attempting to do an end-run around the invariants set up by SecureBankAccount in order to access the code in SwissBankAccount directly.

The second question is "Why doesn't SecureBankAccount also have this restriction?" In my example, SecureBankAccount says “this.DoTransfer(destinationAccount, user, amount);”

Clearly "this" is of type SecureBankAccount or something more derived. It could be any value of a more derived type, including a new SwissBankAccount. Doesn’t the same concern apply? Couldn't SecureBankAccount be doing an end-run around SwissBankAccount's invariants?

Yes, absolutely! And because of that, the authors of SwissBankAccount are required to understand and approve of everything that their base class does! You can't just go deriving from some class willy-nilly and hope for the best. The implementation of your base class is allowed to call the set of protected methods exposed by the base class. If you want to derive from it then you are required to read the documentation for that class, or the code, and understand under what circumstances your protected methods will be called, and write your code accordingly. Derivation is a way of sharing implementation details; if you don't understand the implementation details of the thing you are deriving from then don't derive from it.

And besides, the base class is always written before the derived class. The base class isn't up and changing on you, and presumably you trust the author of the class to not attempt to break you sneakily with a future version. (Of course, a change to a base class can always cause problems; this is yet another version of the brittle base class problem.)

The difference between the two cases is that when you derive from a base class, you have the behaviour of one class of your choice to understand and approve of. That is a tractable amount of work. The authors of SwissBankAccount are required to precisely understand what SecureBankAccount guarantees to be invariant before the protected method is called. But they should not have to understand and trust every possible behaviour of every possible cousin class that just happens to be derived from the same base class. Those guys could be implemented by anyone and do anything. You would have no ability whatsoever to understand any of their pre-call invariants, and therefore you would have no ability to successfully write a working protected method. Therefore, we save you that bother and disallow that scenario.

And besides, we have to allow you to call protected methods on receivers of potentially more-derived classes. Suppose we didn't allow that and deduce something absurd. Under what circumstances could a protected method ever be called, if we disallowed calling protected methods on receivers of potentially-more-derived classes? The only time you could ever call a protected method in that world is if you were calling your own protected method from a sealed class! Effectively, protected methods could almost never be called, and the implementation that was called would always be the most derived one. What's the point of "protected" in that case? Your "protected" means the same thing as "private, and can only be called from a sealed class". That would make them rather less useful.

So, the short answer to both questions is "because if we didn't do that, it would be impossible to use protected methods at all." We restrict calls through less-derived receiver compile-time types because if we don't, it's impossible to safely implement any protected method that depends on an invariant. We allow calls through potential subtypes because if we do not allow this, then we don't allow hardly any calls at all.

Finally, an unasked question: what if you are the person writing the base class with a protected method? Essentially you are in the same boat as the person writing any public or virtual method; then you have to accept that anyone can call your public method in any way it chooses, or that derived class can override your virtual method and make it do something completely different, at their discretion. If you write a protected method, you have to accept that any derived class can call that method in any way it chooses and write the base class code accordingly.

When you write a public method, you have to consider the consequences of bad callers; if there are ways that bad callers can misuse your public method, then you need to consider the cost to the user vs the cost of hardening the method against the potentially bad caller. The same is true of virtual methods; you have to consider the consequences of bad overriders. And the same is true of protected methods; you have to consider the consequence of bad derived classes calling your code.

Designing robust code that has public methods is hard. Designing code that is robust and has virtual or protected methods is even harder; designing for extendability is a difficult problem in general, and shouldn’t be taken on lightly. Consider sealing classes that aren’t designed for extension.

Continuing to an outer loop

Eric Lippert — Mon, 11 Jan 2010 14:54:00 GMT

When you have a nested loop, sometimes you want to “continue” the outer loop, not the inner loop. For example, here we have a sequence of criteria and a sequence of items, and we wish to determine if there is any item which matches every criterion:

match = null;
foreach(var item in items)
{
foreach(var criterion in criteria)
{
    if (!criterion.IsMetBy(item))
    {
      // No point in checking anything further; this is not
      // a match. We want to “continue” the outer loop. How?
    }
}
match = item;
break;
}

There are many ways to achieve this. Here are a few, in order of increasing awesomeness:

Technique #1 (ugly): When you have a loop, the “continue” statement is essentially a “goto” which branches to the bottom of the loop, just before it does the loop condition test and pops back up to the top of the loop. (Apparently when you spell “goto” as “continue”, the “all gotos are all evil all the time” crowd doesn’t bother you as much.) You can of course simply make this explicit:

match = null;
foreach(var item in items)
{
foreach(var criterion in criteria)
{
    if (!criterion.IsMetBy(item))
    {
      goto OUTERCONTINUE;
    }
}
match = item;
break;
OUTERCONTINUE:
;
}

Technique #2 (better): When I see a nested loop, I almost always consider refactoring the interior loop into a method of its own.

match = null;
foreach(var item in items)
{
if (MeetsAllCriteria(item, critiera))
{
match = item;
break;
}
}

where the body of MeetsAllCriteria is obviously

foreach(var criterion in criteria)
if (!criterion.IsMetBy(item))
return false;
return true;

Technique #3 (awesome): The “mechanism” code here is emphasizing how the code works and hiding the meaning of the code. The purpose of the code is to answer the question “what is the first item, if any, that meets all the criteria?” If that’s the meaning of the code, then write code that reads more like that meaning:

var matches = from item in items
              where criteria.All(
                criterion=>criterion.IsMetBy(item))
              select item;
match = matches.FirstOrDefault();

That is, search the items for items that meet every criterion, and take the first of them, if there is one. The code now reads like what it is supposed to be implementing, and of course, if there are no loops in the first place then there’s no need to “continue” at all!

The biggest thing I love about LINQ is that it enables a whole new way to think about problem solving in C#. It’s gotten to the point now that whenever I write a loop, I stop and think about two things before I actually type “for”:

* Have I described anywhere what this code is doing semantically, or does the code emphasize more what mechanisms I am using at the expense of obscuring the semantics?

* Is there any way to characterize the solution to my problem as the result of a query against a data source, rather than as the result of a set of steps to be followed procedurally?

Is there such a thing as too much precision?

Eric Lippert — Thu, 07 Jan 2010 15:02:00 GMT

Well, enough chit-chat, back to programming language design.

Suppose you’re building electronic piano software. As we’ve discussed before, the “equal temperament” tuning for a piano goes like this: the 49th note from the left on a standard 88 key piano is A, and its frequency is 440 Hz. Each octave above or below that doubles or halves the frequency. Why? Because humans perceive the ratio between two frequencies as the relevant factor, not the (subtractive) difference. There are twelve semitones in the chromatic scale, so to make each one sound like it has the same relationship with its previous note, each one multiplies the frequency of the previous one by the twelfth root of two.

In short, the frequency of the nth key is 440 x 2 ^{(n-49) / 12}

Now, on a real piano, its a bit more complicated than that. The high notes have some interesting problems associated with them. For the long strings in the middle of the piano, the ratio of the diameter of the string to its length is really small; zero, for practical purposes. But the short strings at the high end are short enough that the ratio is no longer practically zero. The thickness of the string causes its harmonic vibrations to be flatter than they ought to be; this is called “inharmonicity”. In addition, humans tend to perceive high notes as being flatter than they really are; our ability to determine relative pitches is somewhat skewed at the high end. For both these reasons, piano tuners often “stretch” the octaves at the high end, making the fundamental frequencies of the high notes slightly sharper than the equal temperament would suggest. Obviously electronic pianos do not have inharmonicity, and we’ll ignore the psychoacoustic problems for our purposes. Because I actually want to talk about floating point arithmetic.

Suppose you say “our expression above is equivalent to 440 x R^(n-49)where R is the twelfth root of two. You compute the 12th root of 2 using calc.exe, and write some code:

static double Frequency(int key)
{
const double TwelfthRootOfTwo = 1.0594630943592952645618252949463;
const double A4Frequency = 440.0;
return A4Frequency * Math.Pow(TwelfthRootOfTwo, key-49);
}

Now, this code is fine, it works, and there’s no need to mess with it. It might be nice to validate the parameter to make sure that its between 1 and 88, but, whatever. However, does it strike you as odd how much precision is in the constant? There are about 20 more decimal digits of precision in there than can be represented in a double! A double is only accurate to around 14 or 15 digits of precision.

One thing that we can deduce from this is that calc.exe does not do its calculations in doubles, which I suppose is interesting. It must use some much higher-precision internal format.

Another thing we can note is that for this application, even fifteen digits is way more precision than we need. The highest value we’re going to get out of this is less than 4200 Hz; a rounding error after the fifteenth digit is going to be far, far too small for a human to hear the difference. If you divide an octave up into 1200 notes instead of 12, each note is called a “cent” (because 100 of them fit into a semitone). Most humans cannot hear a difference of less than a handful of cents, and this is plenty of precision to be accurate to within a few cents.

But if the compiler is simply throwing away all that precision, then that portion of the program is essentially meaningless. Shouldn’t the compiler be warning you that it is throwing away your precision?

Well, no, not really. And here’s why.

Suppose you say

const double d = 108.595;

We wish to represent the exact mathematical value 108595 / 1000 in a double. But doubles use binary arithmetic; we need a power of two on the denominator, not a power of ten. The closest we can get using the 52 precision bits of a double is 1101100.1001100001010001111010111000010100011110101110 or, if you prefer fractions, that’s 3820846886986711 / 35184372088832. That fraction has the exact value in decimal of 108.594999999999998863131622783839702606201171875, which you must admit is awfully close to 108.595, though a touch lower than it.

So here’s the thing. You want to represent the exact value of 108.595, but we don’t let you; we make you accrue some error. And we don’t warn about that; we assume that you know that some error will be accrued. (If you wanted an exact result then you should have used decimal, not double.) But what if the exact value you actually wanted to represent was in fact 108.594999999999998863131622783839702606201171875 ? That number can be represented in binary exactly! We certainly should not give a warning that you have too much precision if you say

const double d = 108.594999999999998863131622783839702606201171875;

because you have exactly the right amount of precision in order to have zero representation error. It seems wacky to say “please round that off to fifteen decimal digits so that we can give you this exact value right back again.” Why should we punish you for being more accurate?

The best we could do is give a warning when there was more precision than could be represented, *and* the representation error was non-zero, *and* losing the extra precision makes the representation error smaller, *and*… now we’ve written a whole lot of complicated math code, the result of which makes the user scratch their head and say “how the heck am I supposed to make the compiler happy with this constant?” Warnings that are nothing but a source of pain are bad warnings, and we try to avoid them.

First Cousins Once Removed

Eric Lippert — Mon, 04 Jan 2010 15:04:00 GMT

Happy New Year all, and welcome to 2010, or, as my friend Professor Orbifold prefers it, MMX. I hope your festive holiday season was as festive and enjoyable as mine.

The extended Lippert family continues to grow; this year at the annual Boxing Day party we needed two overflow tables for dinner instead of the usual one. The older cousins are now all married, some of them are bringing new babies, and some of the younger cousins are starting to bring boyfriends and girlfriends. I pointed out that the youngest person at the table was my brand-new “first cousin once removed”, which touched off a long discussion of how exactly one computes degree of cousin-hood and removed-ness. As a public service, here’s how it works.

Recursive explanation for computer programmers and mathematicians:

Base case: if X and Y are zeroth cousins no times removed then X and Y are siblings.

Recursive case 1: if n > 0 and X and Y are nth cousins no times removed then X.Parent and Y.Parent are (n-1)th cousins no times removed.

Recursive case 2: if m > 0 and X and Y are nth cousins m times removed then WOLOG assume that X is of a later generation than Y. In this case, X.Parent and Y are nth cousins (m-1) times removed.

Explanation for normal people:

Take two different people who have a common ancestor but who are not related by direct succession (that is, neither is the mother, father, grandmother, and so on, of the other):

Degree of cousin-hood is the minimum of the numbers of generations back that you have to go to find the nearest common ancestor, minus one.

Removed-ness is the absolute difference between the numbers of generations back you have to go to find the nearest common ancestor.

For example, consider this fragment of a family tree:

        Mary
       /     \
   Laura     Bob
     |        |
   John     Helen
     |        |
Xerxes   Melinda

Take Helen and Xerxes for example. Their nearest common ancestor is Mary. Helen has to go back two generations to get to Mary. Xerxes has to go back three. The minimum of two and three, minus one, is one. The absolute difference of two and three is one. So Helen and Xerxes are first cousins, once removed. So are John and Melinda.

“Zeroth cousins” are not called cousins, but have special names for the relationship. “Zeroth cousins no times removed” are of course brothers and sisters. “Zeroth cousins once removed” are uncles, aunts, nieces and nephews. “Zeroth cousins twice removed” are great-uncles, great-aunts, great-nieces and great-nephews. (Or sometimes grand-uncles, and so on.)

Summing up this family tree:

Mary is Laura and Bob’s mother, John and Helen’s grandmother, and Xerxes and Melinda’s great-grandmother.

Laura is Mary’s daughter, Bob’s sister, John’s mother, Helen’s aunt, Xerxes’ grandmother, and Melinda’s great-aunt.

Bob is Mary’s son, Laura’s brother, John’s uncle, Helen’s father, Xerxes’ great-uncle, and Melinda’s grandfather.

John is Mary’s grandson, Laura’s son, Bob’s nephew, Helen’s first cousin (no times removed), Xerxes’ father, and Melinda’s first cousin once removed.

Helen is Mary’s granddaughter, Laura’s niece, Bob’s daughter, John’s first cousin (no times removed), Xerxes’ first cousin once removed, and Melinda’s mother.

Xerxes is Mary’s great-grandson, Laura’s grandson, Bob’s great-nephew, John’s son, Helen’s first cousin once removed, and Melinda’s second cousin (no times removed).

Melinda is Mary’s great-granddaughter, Laura’s great-niece, Bob’s granddaughter, John’s first cousin once removed, Helen’s daughter, and Xerxes’ second cousin (no times removed).

Of course, this is all perfectly straightforward. If, say, your mother dies and your father marries her sister and has more children, then working out who is whose cousin can get rather trickier. And heaven only knows what happens if you are your own grandfather.

Incidentally, it is illegal in the state of Washington for a man to marry his widow’s sister. Anyone care to guess why?

It's the most wonderful time of the year

Eric Lippert — Sat, 26 Dec 2009 16:01:00 GMT

Here's a little holiday cheer for you all. Or, at least for you all in Commonwealth countries.

static object M<T>(T t) where T : struct
{
return t;
}

int ii = 10;
int? jj = 20;
object xx = ii;
object yy = jj;
System.ValueType zz = ii;
IComparable aa = ii;
System.Enum bb = MidpointRounding.ToEven;
object cc = M(ii);

I hope you're having a festive holiday season. Happy Boxing Day, and we'll see you in the New Year for more fabulous adventures!

[Eric is on vacation; this posting is pre-recorded.]

Use the right tool for the job

Eric Lippert — Mon, 14 Dec 2009 14:02:00 GMT

Consider the following scheme:

I have some client software which I sell. When the client software starts up for the first time, it obtains a “token” from the user. The token string can be anything; the user can choose their name, their cat’s name, their password, the contents of some disk file, whatever. How the client software obtains the token from the user is not particularly relevant; perhaps they type it in, perhaps it is set in a registry key, but whatever it is, it is the user’s choice what information they use to identify themselves.

Anyway, the first time the client software starts up it obtains the token, and then phones home over the internet to my server. My server does some unspecified expensive and painful process by which I satisfy myself that the user is authorized to use my client software. (Assume for the sake of argument that such a process exists.)

I encrypt the token message with my RSA private key and send it back to the client. The client writes the encrypted message out to disk.

I do not want to go through the expensive and painful process of contacting the server again every subsequent time the software starts up. Instead, the client software decrypts the stored encrypted token using my RSA public key, and then compares it byte-by-byte to the user’s token. (Again, perhaps they type it in, or perhaps it is in a file or registry key, or whatever.) If the encrypted token matches the decrypted token then I know that I’ve authorized this user at least once already, so I don’t need to do it again.

If the token cannot be authenticated then I go into some error path and deny access to the software.

Does any of this seem like a good idea to you? Because none of this seems like a good idea to me.

The fundamental problem here is that I am attempting to build my own security system instead of purchasing a solution off the shelf that was written by security experts. There are plenty of third-party software licensing solutions available; just buy one.

But it gets worse. Consider the attacks that are possible against this system.

First off, clearly there is nothing stopping any hostile client from simply decompiling my client software, removing the security checks, recompiling, and partying on. The premise of the scenario is that the client is hostile; doing the security check on the client and trusting the result seems ill-advised.

Second, suppose the attacker simply obtains a valid encrypted token once, and then puts the pair up on the internet. Any hostile user can now use the unmodified software without contacting the server. If I notice, I suppose I can check my records to see who put their token up on the internet and sue that person. Or can I? As we’ll see later, a clever attacker can get around this.

Third, suppose there is a hostile eavesdropper Eve, listening in on the connection between a benign client (Alice) and the server (Bob). Alice’s token goes out, the encrypted token comes back, and the eavesdropper has now captured enough information to use the software.

One could mitigate this attack by encrypting the token with the public key first. Then only the server can determine the contents of the token, right? As we’ll see, that’s maybe not true either…

Fourth, there’s no mechanism proposed whereby encrypted tokens expire or can otherwise be treated as “known to be compromised” in some way. There’s no way to identify hostile clients and revoke their ability to run the software.

All of these are pretty fundamental problems with this licensing scheme. Those aren’t the specific problem I want to get at today though. There’s also an extremely dangerous mistake in the design of the crypto mechanism.

The expression “use the right tool for the job” never applies more than when considering crypto solutions to real-world problems. Cryptosystems are designed to solve very specific problems very well, but as soon as you deviate even slightly from the by-design scenario of a cryptosystem, you are off the trail and skiing downhill through the trees; you hit a tree, that’s your fault – you shouldn’t have gone off the trail.

The RSA cryptosystem is extremely good at solving two problems:

1) Alice (again, the benign client) wishes to send a message of her choice to Bob (the server), encrypted with Bob’s public key. The message can be decrypted by Bob, but not by Eve, the hostile eavesdropper.

2) Bob wishes to publish a message of his choice that could only have come from Bob. The message is encrypted with Bob’s private key and can be decrypted by anyone with his public key. The fact that the message can be decrypted by Bob’s public key is evidence that the message came from Bob.

(And obviously these scenarios can be combined; Alice’s message to Bob could be encrypted with her private key, so that Bob is the only one who can read it, and Alice is the only one who could have sent it. Provided, of course, that Alice and Bob have found some secure mechanism by which to reliably exchange public keys. We assume that such a mechanism exists and has been used. This might not be a valid assumption in the real world!)

Does the scheme outlined above actually fall into either of these scenarios? No, absolutely not. It might appear that we are in scenario two, but we’re not. Scenario two is that Bob is encrypting a message of Bob’s choice with Bob’s private key. But in the scheme outlined here, the server is encrypting a message of the presumed-hostile client’s choice! The RSA cryptosystem was not designed to be secure against attacks wherein you allow an attacker to encrypt an arbitrary message with your private key. Such attacks are called “chosen ciphertext” attacks because the attacker chooses a text which is then “decrypted” -- that is, encrypted with the key that Bob would normally use to decrypt a message from Alice.

And in fact, RSA is not secure against such attacks. If you allow an arbitrary attacker to send you an arbitrary message, which you then encrypt with your private key, then the attacker can carefully craft messages which allow her to do all kinds of crazy things:

* Most obviously, an attacker can simply make a token which is a document that says something that Bob would never willingly say. The existence of such a service means that no one can use the fact that a message is decryptable with Bob's public key to conclude that this message is vouched for by Bob. But it is far worse than that.

* Suppose Eve has captured a token message from Alice to Bob, encrypted with Bob's public key. Since encrypting a message with the private key decrypts messages encrypted with the public key, Eve simply sends that as her token to Bob. Bob decrypts Alice's token for Eve. Now Eve knows Alice's token. Of course, Bob can detect that Eve has submitted a token which looks suspiciously like it decrypts to Alice's token, so this might not be the smartest thing for Eve to do. But...

* An attacker can contrive a novel message which, when encrypted with the private key, results in enough information to decrypt a previously-captured message that was encrypted with the public key. That is: Alice encrypts her token with Bob's public key and sends it to the server (Bob), who decrypts it with the private key, encrypts the result with the private key, and sends it back. Eve intercepts both messages. Eve then crafts a very special message based on a transformation of the captured content and asks Bob to encrypt it with Bob’s private key. The contents of the result can be used by Eve to determine what Alice’s token is. Bob has no easy way of knowing that Eve’s special message is in fact an attempt to decrypt Alice’s token.

* A hostile authenticated client can choose a token she would like to be encrypted with Bob’s private key, but she does not want Bob to actually see the token. It is possible to contrive a random-seeming token such that you can figure out what a chosen token’s encryption with Bob's private key would be. The hostile client can use this attack in order to generate a token/encrypted token pair that can be distributed to other attackers, but which does not point back to the authenticated client.

* And so on. We haven’t even gotten into issues like correct padding of the message to prevent replay attacks. There are potentially many more issues here.

I don’t know much about cryptography; the most important thing that I do know on the subject is that I don’t know nearly enough about cryptography to safely design or implement a crypto-based security system.

So, what have we learned?

0) If you possibly can, simply don’t go there. Encryption is extremely difficult to get right and is frequently the wrong solution in the first place. Use other techniques to solve your security problems.

1) If the problem is an untrustworthy client then don’t build a security solution which requires trusting the client.

2) If you can use off-the-shelf parts then do so.

3) If you cannot use off-the-shelf-parts and do have to use a cryptosystem then don’t use a cryptosystem that you don’t fully understand.

4) If you have to use a cryptosystem that you don’t fully understand, then at least don’t use it to solve problems it was not designed to solve.

5) If you have to use a cryptosystem to ski through the trees, then at least don’t allow the presumably hostile client to choose the message which is encrypted. Choose the token yourself. If the token must include information from the client then sanitize it in some way; require it to be only straight ASCII text, insert random whitespace, and so on.

6) If you have to allow the client to choose the token, then don’t encrypt the token itself. Sign a cryptographically-secure hash of the token. Its much harder for the attacker to choose a token which produces a desired hash.

7) Don’t use the same key pair for encrypting outgoing messages as you do for protecting incoming messages. Get a key pair for each logically different operation you're going to perform.

8) Encrypt the communications both ways.

9) Consider having a revocation mechanism, so that once you know that Eve is attacking you, you can at least revoke her license. (Or you can revoke a known-to-be compromised license, and so on.)

*********

And with those cheerful thoughts, I'm off on my annual flight south to Canada to spend a whirlwind Christmas vacation with family and friends back in Waterloo. I hope you all have a festive holiday season!

Constraints are not part of the signature

Eric Lippert — Thu, 10 Dec 2009 17:31:00 GMT

What happens here?

class Animal { }
class Mammal : Animal { }
class Giraffe : Mammal { }
class Reptile : Animal { }
…
static void Foo<T>(T t) where T : Reptile { }
static void Foo(Animal animal) { }
static void Main()
{
Foo(new Giraffe());
}

Most people assume that overload resolution will choose the second overload. In fact, this program produces a compile error saying that T cannot be Giraffe. Is this a compiler bug?

No, this behaviour is correct according to the spec. First we attempt to determine the candidate set. Clearly the second overload is a member of the candidate set. The first overload is a member of the candidate set if type inference succeeds.

The method type inference algorithm considers only whether the method type arguments can be consistently inferred from the types of the arguments. Method type inference cares not a bit about whether the resulting method is malformed in some other way. Its only job is to work out the best possible type arguments given the arguments to the method. Clearly the best type for T is Giraffe, so that’s what we infer.

So we now have two methods in the candidate set, Foo<Giraffe>, and the second overload. Which is better?

Again, overload resolution looks only at the arguments you passed in, and compares them to the types of all the candidates. The argument is of type Giraffe. We have a choice: argument of type Giraffe goes to parameter of type Giraffe, or argument of type Giraffe goes to parameter of type Animal. Clearly the former is better; it’s an exact match.

Therefore we discard the second overload because it is worse than another candidate. That leaves one candidate left, which is an exact match. Only then, after overload resolution, do we check to see whether the generic constraints are violated.

When I try to explain this to people, they often bring up this portion of the specification as evidence that I am wrong, wrong, wrong:

If F is generic and M has no type argument list, F is a candidate when:
1) type inference succeeds, and
2) once the inferred type arguments are substituted for the corresponding method type parameters, all constructed types in the parameter list of F satisfy their constraints, and the parameter list of F is applicable with respect to A. [emphasis added]

This appears at first glance to say that Foo<Giraffe> cannot be a candidate because the constraints are not satisfied. That is a misreading of the specification; the bit “in the parameter list” is referring to the formal parameter list, not the type parameter list.

Let me give you an example of where this rule comes into play, so that it’s clear. Suppose we have

class C<T> where T : Mammal {}
…
static void Bar<T>(T t, C<T> c) where T : Mammal {}
static void Bar(Animal animal, string s) { }
…
Bar(new Iguana(), null);

Type inference infers that Bar<Iguana> might be a candidate. But that would mean that we are calling a method that converts null to C<Iguana>, which violates the constraint in the declaration of C<T>. Therefore the results of type inference are discarded, and Bar<Iguana> is not added to the candidate set.

So if that’s not the relevant bit of the spec, what is? The relevant bit happens after the best method has been determined, not before.

If the best method is a generic method, the type arguments (supplied or inferred) are checked against the constraints declared on the generic method. If any type argument does not satisfy the corresponding constraints on the type parameter, a compile-time error occurs.

It is often surprising to people that an invalid method can be chosen as the best method, chosen over a less good method that would be valid. (*) The principle here is overload resolution (and method type inference) find the best possible match between a list of arguments and each candidate method’s list of formal parameters. That is, they look at the signature of the candidate method. If the best possible match between the arguments and the signature of the method identify a method that is for whatever reason not possible to call, then you need to choose your arguments more carefully so that the bad thing is no longer the best match. We figure that you want to be told that there's a problem, rather than silently falling back to a less-good choice.

UPDATE: My Twilight-reading friend Jen from a couple episodes back points out that this is not just good language design, it's good dating advice. If the best possible match between your criteria and an available guy on Match.com identifies a guy who, for whatever reason, is impossible to call, then you need to choose your criteria more carefully so that the bad guy is no longer the best match. Words to live by.

(*) I already discussed the related situation of how it is the case that we can choose a static method over an instance method even when there’s no hope of the static method being the right one.

Query transformations are syntactic

Eric Lippert — Mon, 07 Dec 2009 14:20:00 GMT

As you probably know, there are two ways to write a LINQ query in C#. The way I personally prefer is to use the “query comprehension” syntax:

from customer in customerList
where customer.City == "London"
select customer.Name

Or you can, equivalently, use the “fluent method call” syntax:

customerList
.Where(customer=>customer.City == "London")
.Select(customer=>customer.Name)

These are guaranteed to be equivalent because the compiler simply transforms the former syntax into the latter syntax before it compiles it. An interesting aspect of this transformation is that it is (almost) entirely syntactic. (The "transparent identifiers" generated for certain queries require a small amount of semantic analysis to determine the corresponding anonymous type, but for the most part, all we do is just pull the raw hunks of code out of each clause and reorganize the program into the method call form.) Once it is in a form that the rest of the compiler can understand, then semantic analysis proceeds as usual.

This means that it is perfectly legal to do dumb things. For example, suppose we decide that by “where” we mean don’t mean “filter”, we actually mean “multiply”. And by “select” we actually mean “add”, not “project”. No problem!

static class ExtInt
{
    public static int Where(this int c, Func<int, int> f)
    {
        return f(0) * c;
    }
    public static int Select(this int c, Func<int, int> f)
    {
        return f(0) + c;
    }
}
…
int ten = 10;
int twenty = 20;
int thirty = 30;
int result = from c in ten where twenty select thirty;
Console.WriteLine(result);

And sure enough, ten where/times twenty select/plus thirty is 230. This is a deeply strange way to write a mathematical expression, but legal.

The semantics of the Where method are supposed to be “takes a collection of T and a predicate mapping T to bool, and returns a filtered collection of those T items which match the predicate”. This method does not take a collection or a predicate, it does not return a collection, and it certainly does not have the semantics of filtering, but the compiler neither knows nor cares about any of those facts. All the compiler does is syntactically turn that line into

int result = ten.Where(c=>twenty).Select(c=>thirty);

and compile it; that line compiles just fine, so, no problem as far as the compiler is concerned.

The C# specification has a section which describes the pattern we expect a query provider to implement: that Where takes a predicate, and so on. But we perform no checks whatsoever that you successfully implemented a query provider that implements either the form or the semantics of our recommended pattern. If you do something crazy, like redefine “where” to take something other than a predicate and you get crazy results, then, well, what can I tell you? If it hurts when you do that then don’t do that!