Fabulous Adventures In Coding : Rarefied Heights

What's the difference between covariance and assignment compatibility?

Eric Lippert — Mon, 30 Nov 2009 14:51:00 GMT

I've written a lot about this already, but I think one particular point bears repeating.

As we're getting closer to shipping C# 4.0, I'm seeing a lot of documents, blogs, and so on, attempting to explain what "covariant" means. This is a tricky word to define in a way that is actually meaningful to people who haven't already got degrees in category theory, but it can be done. And I think it's important to avoid defining a word to mean something other than its actual meaning.

A number of those documents have led with something like:

"Covariance is the ability to assign an expression of a more specific type to a variable of a less specific type. For example, consider a method M that returns a Giraffe. You can assign the result of M to a variable of type Animal, because Animal is a less specific type that is compatible with Giraffe. Methods in C# are 'covariant' in their return types, which is why when you create a covariant interface, it is indicated with the keyword 'out' -- the returned value comes 'out' of the method."

But that's not at all what covariance means. That's describing "assignment compatibility" -- the ability to assign a value of a more specific type to a storage of a compatible, less specific type is called "assignment compatibility" because the two types are compatible for the purposes of verifying legality of assignments.

So what does covariance mean then?

First off, we need to work out precisely what the adjective "covariant" applies to. I'm going to get more formal for a bit here, but try to keep it understandable.

Let's start by not even considering types. Let's think about integers. (And here I am speaking of actual mathematical integers, not of the weird behaviour of 32-bit integers in unchecked contexts.) Specifically, we're going to think about the ≤ relation on integers, the "less than or equal to" relation. (Recall that of course a "relation" is a function which takes two things and returns a bool which indicates whether the given relationship holds or does not hold.)

Now let's think about a projection on integers. What is a projection? A projection is a function which takes a single integer and returns a new integer. So, for example, z → z + z is a projection; call it D for "double". So are z → 0 - z, N for "negate" and z → z * z, S for "square".

Now, here's an interesting question. Is it always the case that (x ≤ y) = (D(x) ≤ D(y))? Yes, it is. If x is less than y, then twice x is less than twice y. If x is equal to y then twice x is equal to twice y. And if x is greater than y, then twice x is greater than twice y. The projection D preserves the direction of size.

What about N? Is it always the case that (x ≤ y) = (N(x) ≤ N(y))? Clearly not. 1 ≤ 2 is true, but -1 ≤ -2 is false. But we notice that the reverse is always true! (x ≤ y) = (N(y) ≤ N(x)). The projection N reverses the direction of size.

What about S? Is it always the case that (x ≤ y) = (S(x) ≤ S(y))? No. -1 ≤ 0 is true, but S(-1) ≤ S(0) is false. What about the opposite? Is it always the case that (x ≤ y) = (S(y) ≤ S(x)) ? Again, no. 1 ≤ 2 is true, but S(2) ≤ S(1) is false. The projection S does not preserve the direction of size, and nor does it reverse it.

The projection D is "covariant" -- it preserves the ordering relationship on integers. The projection N is "contravariant". It reverses the ordering relationship on integers. The projection S does neither; it is "invariant".

Now I hope it is more clear exactly what is covariant or contravariant. The integers themselves are not variant, and the "less than" relationship is not variant. It's the projection that is covariant or contravariant -- the rule for taking an old integer and making a new one out of it.

So now lets abandon integers and think about reference types. Instead of the ≤ relation on integers, we have the ≤ relation on reference types. A reference type X is smaller than (or equal to) a reference type Y if a value of type X can be stored in a variable of type Y. That is, if X is "assignment compatible" with Y.

Now consider a projection from types to types. Say, the projection "T goes to IEnumerable<T>". That is, we have a projection that takes a type, say, Giraffe, and gives you back a new type, IEnumerable<Giraffe>. Is that projection covariant in C# 4? Yes. It preserves the direction of ordering. A Giraffe may be assigned to a variable of type Animal, and therefore an sequence of Giraffes may be assigned to a variable that can hold a sequence of Animals.

We can think of generic types as "blueprints" that produce constructed types. Let's take the projection that takes a type T and produces IEnumerable<T> and simply call that projection "IEnumerable<T>". We can understand from context when we say "IEnumerable<T> is covariant" what we mean is "the projection which takes a reference type T and produces a reference type IEnumerable<T> is a covariant projection". And since IEnumerable<T> only has one type parameter, it is clear from the context that we mean that the parameter to the projection is T. After all, it is a lot shorter to say "IEnumerable<T> is covariant" than that other mouthful.

So now we can define covariance, contravariance and invariance. A generic type I<T> is covariant (in T) if construction with reference type arguments preserves the direction of assignment compatibility. It is contravariant (in T) if it reverses the direction of assignment compatibility. And it is invariant if it does neither. And by that, we simply are saying in a concise way that the projection which takes a T and produces I<T> is a covariant/contravariant/invariant projection.

UPDATE: My close personal friend (and fellow computer geek) Jen notes that in the Twilight series of novels, the so-called "werewolves" (who are not transformed by the full moon and therefore not actually werewolves) maintain their rigid social ordering in both wolf and human form; the projection from human to wolf is covariant in the social-ordering relation. She also notes that in high school, programming language geeks are at the bottom of the social order, but the projection to adulthood catapults them to the top of the social order, and therefore, growing up is contravariant. I am somewhat skeptical of the latter claim; the former, I'll take your word for it. I suppose the question of how social order works amongst teenage werewolves who are computer geeks is a subject for additional research. Thanks Jen!

Five-Dollar Words For Programmers, Part Three: Homoiconic

Eric Lippert — Mon, 23 Mar 2009 23:33:42 GMT

Jeff Atwood was kind enough to once more give me the shout-out in his blog the other day. Thanks Jeff!

This inspires me to continue my series on five-dollar words for programmers. Here’s one that I only learned relatively recently, when I helped write the code that translates a lambda expression into an expression tree which represents the content of the lambda: homoiconic.

A language is said to be homoiconic if the representation of the program can be seen as a data structure expressible in that language. With expression lambdas being convertible to expression trees (which can then be compiled into code at runtime), C# 3.0 is somewhat homoiconic. But it pales in comparison to, say, LISP, where pretty much everything you can do in the language you can also represent as structurally isomophic data.

Something I personally would like to see more of in C# in the future is greater homoiconicity. We could extend expression trees to statement trees, declaration trees, program trees, and so on. This series of steps would enable increasingly powerful and interesting metaprogramming scenarios.

C# suffers from the lack of a metalanguage. We absolutely do not want to go towards the horrible and primitive metaprogramming language exemplified by the C preprocessor language. We already have a great language, C#, so why not use C# as a metalanguage? Wouldn’t it be nice to make C# its own metalanguage? And once we do that, then we get into some truly strange loops, where we could use those data structures in the compiler implementation itself!

This is a long way off and might never happen, but a guy can dream.

Path Finding Using A* in C# 3.0, Part Four

Eric Lippert — Wed, 10 Oct 2007 17:17:00 GMT

Finally we are ready to translate our pseudocode into C# 3.0.

What do we need to make the algorithm run? We need a start node, a destination node, a function which tells us the exact distance between two neighbours, and a function which tells us the estimated distance between the last node on a proposed path and the destination node.

We also need the ability to take any node and get a list of its neighbours. We'll represent that by an interface:

interface IHasNeighbours<N>
{
IEnumerable<N> Neighbours { get; }
}

Recall that the pseudocode looked like this:

closed = {}
q = emptyqueue;
q.enqueue(0.0, makepath(start))
while q is not empty
    p = q.dequeueCheapest
    if closed contains p.last then continue;
    if p.last == destination then return p
    closed.add(p.last)
    foreach n in p.last.neighbours
        newpath = p.continuepath(n)
        q.enqueue(newpath.TotalCost + estimateCost(n, destination), newpath)
return null

And now we're all set. The translation from pseudocode to real code is quite straightforward:

static public Path<Node> FindPath<Node>(
    Node start,
    Node destination,
    Func<Node, Node, double> distance,
    Func<Node, double> estimate)
    where Node : IHasNeighbours<Node>
{
    var closed = new HashSet<Node>();
    var queue = new PriorityQueue<double, Path<Node>>();
    queue.Enqueue(0, new Path<Node>(start));
    while (!queue.IsEmpty)
    {
        var path = queue.Dequeue();
        if (closed.Contains(path.LastStep))
            continue;
        if (path.LastStep.Equals(destination))
            return path;
        closed.Add(path.LastStep);
        foreach(Node n in path.LastStep.Neighbours)
        {
            double d = distance(path.LastStep, n);
            var newPath = path.AddStep(n, d);
            queue.Enqueue(newPath.TotalCost + estimate(n), newPath);
        }
    }
    return null;
}

You might wonder why we check to see if the current-best-path ends in a "closed" node after the dequeue, rather than testing that before we do the enqueue. Think about it this way: suppose we take the best path off the queue, note that its end node is "closed" so that we don't come back to it, and put all of its extensions on the queue. There might be a number of other paths which end in that node presently on the queue. All of them must be, by definition, worse than the path we just took off the priority queue. We should therefore not even consider them as possbilities anymore. If we happen to put more paths with that property on the queue, that's unfortunate. But when we eventually get to them we'll discard them automatically so its not a big deal.

A nice property of the A* algorithm is that it finds the optimal path in a reasonable amount of time provided that:

the estimating function never overestimates the distance to the goal. (Think about what happens if the estimating function sometimes overestimates the distance; if the optimal path is one of the ones overestimated then it will possibly not make it to the front of the priority queue before a nonoptimal solution is found.)
calling the estimating function does not take very long.

One way to ensure that the estimating function never overestimates the distance is to always estimate zero. If you do so then this becomes Dijkstra's algorithm. However, the better your estimating function can get without going over (this really should have been called the "The Price Is Right" algorithm...) the faster this will identify the truly optimal path.

Unfortunately, the space complexity of this algorithm can be really quite high on complicated graphs. There are more complex versions of this algorithm which can deal with the space complexity, but I'm not going to go there.

I have a nice example of using A* to find the shortest path around a lake (but not the longest shortest path!) but I'm not going to post it here until Orcas ships. I'll just put up the whole project file at that time and you can check it out all at once.

Next time: at long last, Eric talks about variance in C#.

Path Finding Using A* in C# 3.0, Part Three

Eric Lippert — Mon, 08 Oct 2007 17:03:00 GMT

In order to make the A* algorithm work we need to get the lowest-estimated-cost-path-discovered-so-far out of the list of paths under consideration. The standard data structure for doing so is called a “priority queue”. Priority queues are so-called because they are typically used to store a list of jobs where each job has an associated priority.

Now, there’s a bit of a semantic problem with priority queues in that we typically think of “priority 1” as being higher priority than “priority 2”, even though 1 is a smaller number than 2. Fortunately for us, that is exactly what we want in this case; the “highest priority” path is the one with the least estimated cost.

There are lots of ways to implement priority queues – I have a nice example of an immutable priority queue that I’ll probably blog about at some point, but for now, let’s go with an old-fashioned mutable priority queue.

A priority queue can be implemented as list of sub-queues with the list sorted in priority order. The implementation is pretty straightforward:

class PriorityQueue<P, V>
{
    private SortedDictionary<P, Queue<V>> list = new SortedDictionary<P, Queue<V>>();
    public void Enqueue(P priority, V value)
    {
        Queue<V> q;
        if (!list.TryGetValue(priority, out q))
        {
            q = new Queue<V>();
            list.Add(priority, q);
        }
        q.Enqueue(value);
    }
    public V Dequeue()
    {
        // will throw if there isn’t any first element!
        var pair = list.First();
        var v = pair.Value.Dequeue();
        if (pair.Value.Count == 0) // nothing left of the top priority.
            list.Remove(pair.Key);
        return v;
    }
    public bool IsEmpty
    {
        get { return !list.Any(); }
    }
}

We could implement IEnumerable<V>, a Peek operation, etc, but this is all we need for A*, so let’s not go crazy here.

Next time: now we have all the parts we need to implement A*, so let's do it already.

Path Finding Using A* in C# 3.0, Part Two

Eric Lippert — Thu, 04 Oct 2007 17:33:00 GMT

In order to implement the A* algorithm in C# 3.0 I am going to need to implement some custom data structures. Today we’ll consider how to implement the “path”.

You’ll notice in my description of the algorithm that the only operations we perform on paths are:

Make a new path consisting of a single node.
Make a new path by adding a new node onto the end of an existing path.
Fetch the last element on the path.
Get the total cost of a path.

This cries out to me “Eric! A path is an immutable stack!”

A mutable stack like System.Collections.Generic.Stack<T> is clearly not suitable. We want to be able to take an existing path and create new paths from it for all of the neighbours of its last element, but pushing a new node onto the standard stack modifies the stack. We’d have to make copies of the stack before pushing it, which is silly because then we’d be duplicating all of its contents unnecessarily.

Immutable stacks do not have this problem. Pushing onto an immutable stack merely creates a brand-new stack which links to the old one as its tail. Since the stack is immutable, there is no danger of some other code coming along and messing with the tail contents. You can keep on using the old stack to your heart’s content.

ASIDE: Immutable data structures are the way of the future in C#. It is much easier to reason about a data structure if you know that it will never change. Since they cannot be modified, they are automatically threadsafe. Since they cannot be modified, you can maintain a stack of past “snapshots” of the structure, and suddenly undo-redo implementations become trivial. On the down side, they do tend to chew up memory, but hey, that’s what garbage collection was invented for, so don’t sweat it. I’ll be talking more about programming using immutable data structures in this space over the next few months.

Let’s make an immutable stack of nodes which tracks the total cost of the whole path. Later on, we’ll see that we need to enumerate the nodes of this thing, so we’ll do a trivial implementation of IEnumerable while we’re at it. And since we don’t know exactly what a node will look like, let’s make the thing generic:

class Path<Node> : IEnumerable<Node>
{
    public Node LastStep { get; private set; }
    public Path<Node> PreviousSteps { get; private set; }
    public double TotalCost { get; private set; }
    private Path(Node lastStep, Path<Node> previousSteps, double totalCost)
    {
        LastStep = lastStep;
        PreviousSteps = previousSteps;
        TotalCost = totalCost;
    }
    public Path(Node start) : this(start, null, 0) {}
    public Path<Node> AddStep(Node step, double stepCost)
    {
        return new Path<Node>(step, this, TotalCost + stepCost);
    }
    public IEnumerator<Node> GetEnumerator()
    {
        for (Path<Node> p = this; p != null; p = p.PreviousSteps)
            yield return p.LastStep;
    }
    IEnumerator IEnumerable.GetEnumerator()
    {
        return this.GetEnumerator();
    }
}

Well, that was painless. Next time: implementing the priority queue.

Path Finding Using A* in C# 3.0, Part One

Eric Lippert — Wed, 03 Oct 2007 02:31:00 GMT

As we get into the home stretch for the Orcas release I’ve been spending a little time lately just playing around implementing some “classic” algorithms and data structures in C#. The one I implemented today is the famous A* algorithm.

Surely you’ve played games like Age of Empires. You know the kind of games I mean. There are a bunch of little dudes walking around a map containing barriers (mountains, walls, etc) and somehow the game’s pseudo-intelligence finds the shortest possible path for the guys to walk from point A to point B. Most games do this with a surprisingly straightforward algorithm.

Suppose you’ve got a bunch of “nodes”. Say, intersections in a city. Each node has a bunch of “neighbours” – intersections that can be immediately reached by locomoting down a particular street involved in the intersection. Each “edge” (ie, street) between nodes has a “cost” – the amount of time it takes to travel from one node to the neighbour on that street. Perhaps the cost is based on its length, traffic density, etc.

A “path from A to B” consists of a list of nodes starting with A, ending with B, where every step along the path is from neighbour to neighbour. The total cost of a path is obviously the sum of all the costs of every transition from neighbour to neighbour on the path.

Let’s start by considering a really stupid, awful, buggy path finding algorithm, and then we’ll make it better.

q = emptyqueue
q.enqueue(makepath(start))
while q is not empty
p = q.dequeue
if p.last == destination then return p
foreach n in p.last.neighbours
q.enqueue(p.continuepath(n))
// failed to find path.
return null

What’s wrong with this picture?

Well, suppose we have four nodes,arranged like this: A <--> C <--> D <--> B. Does this algorithm find the obvious path from A to B?

Run the code in your head: it starts by putting the path “A” on the queue. Then it takes it off and puts “A-C” on the queue. Then it takes it off and puts “A-C-A” and “A-C-D” on the queue. Then it takes “A-C-A” off the queue and puts “A-C-A-C” on the queue... OK, eventually we get there, but we seem to be spending a lot of time going in circles. And if D does not actually connect to B, if you can't get there from here, then we really do go into an infinite loop.

Clearly this is all screwed up. We need to disallow paths which loop back on themselves, because those are clearly not going to be the best possible path. We're at best wasting time looking at them, and at worst going into an infinite loop that chews up all the memory in the system.

Let’s fix that up. We’ll create a “closed” set, ie, the set of “nodes we have already considered, so do not loop back to them.”

closed = {}
q = emptyqueue;
q.enqueue(makepath(start))
while q is not empty
    p = q.dequeue
    if closed contains p.last then continue
    if p.last == destination then return p
    closed.add(p.last)
    foreach n in p.last.neighbours
        q.enqueue(p.continuepath(n))
return null

This is better, in that it actually terminates in a finite amount of time. But my goodness, is this algorithm ever slow. Essentially the algorithm is “At every intersection, check to see if you're at your destination. If not, pick a direction – any direction! - and walk as far as you can like that without revisiting your steps. If you ever run out of options, backtrack until you get to an intersection that goes somewhere you haven’t tried yet. Keep doing that until either you get to your destination or you have tried every possible intersection.”

I guarantee you that if you try that in a reasonably sized city, you will not get from point A to point B in any kind of reasonable time. And when you do succeed, the path which you come up with may well be absolutely crazy. Obviously this is not how people navigate between points in cities.

The problem is that we are picking which neighbour to try next essentially at random. We can do better than that. We want to direct our search so that the path we spend most of our time concentrating on is the path which is most likely to lead us to the goal.

Now, that sounds like a bit of a chicken-and-egg problem. Finding the best path is precisely what we are trying to do, so how can we determine which path is the one we want to be concentrating on?

We shall have to guess. We will rank all potential paths based on the sum of two factors:

First, what is the total cost of each potential path so far? This is known.
Second, what is the estimated cost of getting from the end of each potential path to the goal? This is a guess.

And now our algorithm becomes:

which is the famous A* algorithm! Surprisingly straightforward, eh?

Next time: actually implementing this thing in C# 3.0.

What exactly does 'lifted' mean?

Eric Lippert — Wed, 27 Jun 2007 17:00:00 GMT

(Note: all type placeholders S, T, U that I mention in this article are non-nullable value types.)

I got a question the other day from a reader who had been taking a close look at the C# 2.0 standard. He noticed that when we talk about nullables in the standard we talk about "lifting", but we do so inconsistently. To summarize what the standard says:

1) For every equality and relational operator that compares an S to a T and returns a bool there is a corresponding lifted operator that compares an S? to a T? and returns a bool. It returns false if either argument is null. (With the additional exception that if one argument to an equality operator is the null literal then the result may be determined by checking if the nullable term has a value.)

2) For the unary operators + ++ - -- ! ~ where the operator takes an S and returns a T there is a corresponding lifted operator that takes an S? and returns a T?. It returns null if its argument is null.

3) For the binary operators + - * / % & | ^ << >>, where the operator takes an S and a T and returns a U there is a corresponding lifted operator that takes an S? and a T? and returns a U?. It returns null if either argument is null.

4) For user-defined conversions from S to T, there is a corresponding lifted user-defined conversion from S? to T?. It returns null if the argument is null.

5) For built-in conversions from S to T, there are corresponding nullable conversions:
5.1) from S? to T?, which returns null if the argument is null.
5.2) from S to T?, which never returns null, and
5.3) an explicit-only conversion from S? to T which throws if the argument is null.

Why does the standard call out that the conversions in (5) are "nullable" but not "lifted"? Everything else is "lifted"!

This is a bit of a mess I'm afraid.

I talked to Dr. T (AKA Mads Torgersen, the C# language PM) about this. He defined for me precisely what mathematicians mean by “lifted”.

Suppose we've got a function f which maps values from a set A into a set B. That is f:A→B.

Suppose further that null is not a member of either A or B.

Now consider the sets A' = A ∪ { null } and B' = B ∪ { null }

We define the "lifted function" f' as

f':A'→B' such that f'(x) = f(x) for all x ∈ A and f'(null) = null

Similarly, if we had a two-argument function f: A × B → C, we would define f': A' × B' → C' as f'(x,y) = f(x,y) for all (x,y) ∈ A × B and null if either x or y is null.

What we’re getting at here is that “lifted” means “takes nulls, always agrees with the unlifted version when arguments are not null, maps everything else onto null”.

Now you probably see why I said this was a bit of a mess. By the mathematician's definition, (1) is incorrectly called "lifted", (2), (3), and (4) are correctly called "lifted", (5.1) is incorrectly NOT called "lifted", and (5.2) and (5.3) are correctly not called "lifted". Of course we can choose to define what "lifted" means in C# any way we like, but it would be nice if our definition agreed with convention and was used consistently!

I regret the confusion. I do not believe there is any particular sensible reason for these inconsistencies. Rather, the details were changed so many times over the years as the nullable feature was developed that these sorts of subtle problems crept into the spec and were never expunged. Though of course all of us have as a goal that the standard be a model of clarity and permanence, it is fundamentally a working, evolving, imperfect document; these kinds of things will happen. Hopefully in the next version of the standard some of these sorts of details will be tidied up.

I hope that answers the question!

Lambda Expressions vs. Anonymous Methods, Part Five

Eric Lippert — Wed, 28 Mar 2007 17:45:00 GMT

Last time I demonstrated that the compiler could have to do an exponential number of bindings in order to determine whether there was a unique best overload resolution for a function call that takes a lambda. Some of you may have wondered whether we simply were not being clever enough in the compiler. Perhaps there is some way to optimize this problem so that the unique solution is found in less than exponential time.

As it turns out, the question of whether there is a clever way to do this in C# 3.0 is equivalent to solving the most famous unsolved problem in computer science. That is a problem which has stumped generations of the finest minds in academia, and is widely believed to be unsolvable, so I don't feel particularly bad about not solving this one myself.

Consider the following slight revision of the set of overloads I sketched out last time:

class MainClass
{
    class T{}
    class F{}
    delegate void DT(T t);
    delegate void DF(F f);
    static void M(DT dt)
    {
        System.Console.WriteLine("true");
        dt(new T());
    }
    static void M(DF df)
    {
        System.Console.WriteLine("false");
        df(new F());
    }
    static T Or(T a1, T a2){return new T();}
    static T Or(T a1, F a2){return new T();}
    static T Or(F a1, T a2){return new T();}
    static F Or(F a1, F a2){return new F();}
    static T And(T a1, T a2){return new T();}
    static F And(T a1, F a2){return new F();}
    static F And(F a1, T a2){return new F();}
    static F And(F a1, F a2){return new F();}
    static F Not(T a){return new F();}
    static T Not(F a){return new T();}
    static void MustBeT(T t){}
    static void Main()
    {
        // Introduce enough variables and then encode any Boolean predicate:
        // eg, here we encode (!x3) & ((!x1) & ((x1 | x2) & (x2 | x3)))
        M(x1=>M(x2=>M(x3=>MustBeT(
          And(
            Not(x3),
            And(
              Not(x1),
              And(
                Or(x1, x2),
                Or(x2, x3))))))));
    }
}

This expression makes the compiler solve the Boolean satisfiability problem (aka "SAT"). If SAT has a unique solution then the program compiles and produces the unique solution. If it has more than one solution then compilation fails with an ambiguity error. If it has no solution then the program fails with the error that MustBeT has an F argument. But no matter how you slice it, the compiler must solve SAT.

Determining whether a given predicate has a set of valuations for its variables which make it true is an NP-complete problem; actually finding the set of valuations is an NP-hard problem. Therefore, overload resolution in C# 3.0 is at the very least NP-hard. There is no known polynomial-time algorithm for any NP-complete or NP-hard problem and it is widely believed that there is none to be found, though that conjecture has yet to be proven. So our dream of fast lambda type analysis in C# 3.0 is almost certainly doomed, at least as long as we have the current rules for overload resolution.

Does this really matter in practice? As I mentioned last time, hopefully not. Other languages also have this issue. When I ran an early draft of this post past him, Erik Meijer immeditely pointed out that the ML type inference system has similar worst cases, and the Haskell type inference system is even worse. Apparently in Haskell you can encode a Turing machine into the type system and make the compiler run it! In practice these sorts of problems do not arise in real-world code in any of these languages, so I am not too worried about it for C# 3.0.

Every Number Is Special In Its Own Special Way

Eric Lippert — Tue, 28 Nov 2006 21:21:00 GMT

I got a question recently about where in the .NET framework the "special numbers" were defined. The questioner was actually asking about the Double.NaN, Double.PositiveInfinity, etc, special values for floating point numbers. Of course there are other "special numbers" defined by the framework, such as Math.PI.

The question was easily answered but it got me thinking, which, as we know, is usually trouble.

Clearly zero is a very special number, being the first natural number.

One is pretty special too, being the multiplicative identity.

Two is the only even prime.

Three is the lowest odd prime...

Clearly lots of numbers are special. This led me to propose the following theorem:

Theorem: Every natural number (0, 1, 2, ...) is a special number.

Proof:

Let’s posit that the set of nonspecial natural numbers is nonempty, and deduce a contradiction.

If there exists a nonspecial natural number then there must be a lowest nonspecial natural number.

What an unusual property! The lowest nonspecial natural number!

Whatever number has that unusual property must be kinda... special.

Therefore the lowest nonspecial natural number is special.

Therefore, if the set of nonspecial natural numbers is nonempty then it contains a special number.

That is clearly nonsensical, therefore the set of nonspecial natural numbers is empty.

Therefore all natural numbers are special, QED.

And yet I can't find anything special about 7920687935872092847630945767548023. But it must be special somehow!

Extending the proof show that all real numbers are special is left as an exercise for the reader.

Regular Expressions From Scratch, Part Twelve: Superposition of States

Eric Lippert — Fri, 20 Jan 2006 22:47:00 GMT

Happy New Year everyone. Over the break I had a wonderful time reconnecting with my friends and family. And of course I came back to a huge pile of work! We're going through the flaws that were discovered in C# 2.0 too late in the cycle to risk fixing, and some of them illustrate interesting corner cases in the language. But those will have to wait, as we've still got a lot of ground to cover before I'm done this crazy long series on regular expressions.

To summarize the story so far: we've defined alphabets and languages over those alphabets. We've shown how to create a special "regular expression" alphabet and language for any given alphabet. We've also come up with a rule that associates a language with a regular expression. This language is the set of strings which "matches" the regular expression.

We've determined that there are deterministic finite state computers which consume one character of a string at a time that can "recognize" strings from some regular languages. We've also determined that there are "nondeterministic" finite automata that somehow "magically" figure out which rules to apply at any time to match a string.

We've also shown that any NFA that has rules that act on more than one character can be turned into an equivalent NFA which only has single-character or no-character transitions.

What we would like to do is show the following three facts: first, that every NFA is equivalent to a DFA. Second, that every DFA/NFA recognizes a regular language. And third, that every regular language has a DFA/NFA that recognizes it.

We've still got a ways to go to show that first result. Rather than prove the general result, which will be tedious, I'll just go through an example and hope that it is illustrative enough to convince you that we could give this treatment to any NFA.

Let M be an NFA with alphabet {a, b}, states {1, 2, 3, 4}, start state is 1, and the set of acceptable states is {4}. We'll assume that we've already eliminated all the multi-character rules and have a set of rules as follows:

(1, e) → 2
(1, b) → 3
(2, a) → 1
(2, e) → 3
(2, a) → 4
(3, b) → 4

This NFA accepts the language a*( bb ∪ b ∪ a ), and as you can see, we've got lots of empty and ambiguous rules.

Here's the trick: given an NFA with 4 states like this one, we can find a DFA which is equivalent to this guy that has 2⁴ = 16 states or fewer. We can think of the NFA as at any time living in a "superstate" that consists of all of the states that it COULD be in right now.

Let me try to explain that a little better. We start in state 1, right? But since (1, e) → 2 and (2, e) → 3, we could also be starting in states 2 or 3 before we process any input. Let's create a new automaton with a start state that represents the concept "right now M could be in states 1, 2, or 3." We'll call that state 123x, which is one big symbol, not a string of four symbols. In our new automaton, the start state is 123x.

We're trying to write a DFA here, so there needs to be a rule for every input and every state:

(123x, a) → ?
(123x, b) → ?

Look at the original automaton. When we were in states 1, 2 or 3, what were the possible state transitions for a? The only ones were (2, a) → 1 and (2, a) → 4. But we're not finished! Again, we need to consider what could happen from states 1 and 4 if we processed the "empty string" transition rules. Since (1, e) → 2 and (2, e) → 3, we could end up in state 1, 2, 3 or 4. Let's create a new state called 1234 for our DFA and finish off that rule:

(123x, a) → 1234

Now do the same analysis for (123x, b) and we'll see that in the original automaton, the only possible resulting states starting in 1, 2 or 3, and processing a b are 3 and 4. Add another new state:

(123x, b) → xx34

Are we done? No. We've added two more states, 1234 and xx34, so we need to figure out the state transitions for them too, which we do by the same process. We discover that

(1234, a) → 1234
(1234, b) → xx34
(xx34, a) → ?
(xx34, b) → xxx4

Uh oh. There are no state transitions in the original NFA that start in states 3 or 4 and take an a. Remember that in that case we assume that the NFA goes into a special "reject" state. Let's call the reject state xxxx.

We now have two more new states to work out the transitions for. If we're in the reject state, we stay in the reject state, and we see from the original NFA that there are no transitions out of state 4, so we can round out our list with:

(xx34, a) → xxxx
(xxxx, a) → xxxx
(xxxx, b) → xxxx
(xxx4, a) → xxxx
(xxx4, b) → xxxx

Since 4 was the acceptable state in the original NFA, any "superstate" that contains 4 must be an acceptable state. So our new DFA is a machine M₂ with alphabet {a}, b}, states {1234, 123x, xx34, xxx4, xxxx}, start state is 123x, and the set of acceptable states is {1234, xx34, xxx4}. The rules are

(1234, a) → 1234
(1234, b) → xx34
(123x, a) → 1234
(123x, b) → xx34
(xx34, a) → xxxx
(xx34, b) → xxx4
(xxx4, a) → xxxx
(xxx4, b) → xxxx
(xxxx, a) → xxxx
(xxxx, b) → xxxx

This DFA accepts the language a*( bb ∪ b ∪ a ), so we've found an equivalent DFA to our NFA. We've turned an NFA with four states and five rules into an equivalent DFA with five states and ten rules. We did well -- you can find NFAs that require 2ⁿ new states, where n is the number of NFA states. An NFA with a thousand states could require 2¹⁰⁰⁰ states to represent as a DFA! 2¹⁰⁰⁰ is a finite, albeit rather large number. But it's not that big. Clearly we could build a 2¹⁰⁰⁰ state machine with only 1000 bits to store the state information. It's the state transition rules that are the pain to work out!

Using the techniques from this and the previous entry we can turn any NFA into an equivalent DFA, so NFAs are really not magic at all. They're just a convenient way to talk about DFAs.

Now, obviously what I've done here isn't a proof; one example does not a proof make. But the proof is both tedious and complicated, so I'm going to skip it. The key result here is that we can stop saying "deterministic finite automaton" and "nondeterministic finite automaton" and just start saying "finite automaton", because they're essentially the same thing.

Now that we know that we can use NFAs with impunity, we can try to answer questions such as:

Is every regular language recognizable by a finite automaton?

Are there any FAs that recognize irregular languages?

Tune in next time to find out!

Regular Expressions From Scratch, Part Eleven: Eliminating Multi-Symbol Rules

Eric Lippert — Thu, 22 Dec 2005 18:00:00 GMT

The story so far: we have deterministic and nondeterministic finite automata. DFAs follow a rigid, fully specified set of rules which, on any string, yield either an accept or reject state after exactly as many moves as the string has characters. NFAs, on the other hand, are poorly specified. They somehow magically are able to choose which rules to apply given the input to find a path that yields an accept state, if one exists.

NFA magic buys us a lot, but at the high price that we can now no longer easily see a clear relationship between this magic box and a computer we could actually build. We need to get back to reality. Here is the amazing result that I'm going to try to justify in your minds:

Every NFA is equivalent to some DFA. Even better: given a description of an NFA, we can construct a description of an equivalent DFA.

In other words, NFA magic doesn't buy us any extra power - no NFA can recognize a language that some DFA cannot also recognize.

This is excellent news, because it means that we can describe and reason about machines using short, convenient, vague nondeterministic notation, but still have confidence that we could build a fully deterministic machine that did exactly the same job.

But how the heck are we going to motivate that amazing result?

One step at a time. We're going to take some NFAs and make simple transformations that gradually turn them into different but equivalent NFAs, and eventually one of those NFAs will actually be a DFA. Rather than giving a full formal proof, which would be tedious and boring, I'm going to rely on the sketch being convincing enough that you believe me that we can take any NFA and turn it into a DFA. Maybe later in this series we'll actually write some C# code that implements the transformation.

The magical bits that we need to remove are: rules can have multiple characters in state transitions, rules can have "empty" state transitions, rules can have ambiguous state transitions, there can be situations for which no stated rule applies, and there's a magical 'crash' state. Once we eliminate all those weirdnesses, we'll be left with a DFA.

Step one: Any NFA that has rules that have multiple-character state transitions is equivalent to some NFA that has only single-character or empty-string transitions.

We can get rid of the multiple-character transitions by adding one new state for each extra character. Recall our previous example:

Alphabet: S = {a, b}
States: K = {0}
Start: s = 0
Acceptable: F = {0}

Rules:

(0,ab) → 0
(0,aba) → 0

Suppose we added three new (non-acceptable) states, 1, 2 and 3. You agree I hope that the first rule is equivalent to these two rules:

(0,a) → 1
(1,b) → 0

And the second rule is equivalent to these three rules:

(0,a) → 2
(2,b) → 3
(3,a) → 0

We never need to add more than a finite number of new states or rules, so the new machine is always still an NFA.

Throw away the original two rules and hey presto, we have an equivalent NFA where every rule only has a zero or one-symbol string. Multi-symbol transition rules are no longer magical; we can eliminate them easily.

This example above has no empty-string transitions. Next time we'll pick another example that has empty string transitions and one-character transitions, and then show that we can turn something that looks like that into a DFA.

"Next time" will likely be in the new year, as I am heading back to my ancestral home for the next week to visit friends and relatives during this festive holiday season. I hope all of you have a fun and safe holiday; thanks for reading and we'll see you in 2006.

Regular Expressions From Scratch, Part Ten: Magic!

Eric Lippert — Mon, 19 Dec 2005 18:00:00 GMT

Let's recap the story so far.

Starting only with basic set theory, sequences, symbols and numbers, we've defined alphabets, languages, regular expressions, and the mapping between regular expressions and regular languages. We've also defined deterministic finite automata as machines that take in strings one character at a time, change their internal state to one of a finite set according to strict rules, and end up in either an accept or reject state as output.

Where we're going with this is towards unification of these two concepts. We want to show that for every DFA there's a regexp, and vice versa. But to get there is going to take some magic.

The trouble with DFAs is that they're kind of a pain to specify. You have an alphabet with, say, a hundred symbols, and you have, say, fifty states. You need to come up with a state transition rule for 100 x 50 = 5000 possible combinations. Since most of those will likely be transitions to rejection states anyway, that's majorly boring.

I'm hereby declaring that we have nondeterministic finite automata. An NFA works just like a DFA, except that the "rules" for determining the state transitions can be ambiguous and weird. We'll write an NFA like this:

Let M₁ be an NFA such that:

Alphabet: S = {a, b}
States: K = {0, 1, 2}
Start: s = 0
Acceptable: F = {0}

Rules:

(0,a) → 1
(1,b) → 2
(2,a) → 0
(2,e) → 0

Notice that we're not specifying the six rules we ought to be, and one of those rules is a transition on an "empty input"! This last rule means that if we are in state 2, we can go to state 0 "for free", without consuming any characters.

We say that M₁ accepts a string if there is any way to make it eventually yield an acceptable state with no more input. For example, suppose we start with ababa:

(0, ababa) → (1, baba) → (2, aba) → (0, aba) → (1, ba) → (2, a) → (0, e)

is one path to an acceptable state. Even though

(0, ababa) → (1, baba) → (2, aba) → (0, ba) gets us stuck in a state that we can go no further in, that's OK. NFAs are magic. If there exists any path such that (0, ababa) ⶒ (0, e) then the NFA will find it and end in an accepting state. The fact that some nonsensical or rejecting paths exist doesn't matter if an accepting path exists. Only if every possible path either ends in a rejecting state, or leaves us unable to find any applicable rule, does the NFA reject the string.

In fact, this machine accepts the regular language (ab∪aba)*

(Recall that I am being sloppy about parenthesizing. This isn't a "real" regular expression but you get the point that when I say that I mean L(((ab)∪((ab)a))*) I'm sure.)

We've got lots of magic already, so let's add more. We'll say that a rule can consume any number of matching symbols that it wants.

Let M₂ be an NFA such that:

Alphabet: S = {a, b}
States: K = {0}
Start: s = 0
Acceptable: F = {0}

Rules:

(0,ab) → 0
(0,aba) → 0

This machine clearly accepts (ab∪aba)* except for the fact that it is now totally unclear what we mean by "accepts" vs. "rejects" in a machine with only one state! How does it reject anything?

Let's assume that NFAs have a magic implicit "crash" state which means "For this input I was unable to find any path that did not end in a state where I lacked a rule telling me what to do next." The crash state is always a rejecting state.

Clearly M₁ and M₂ accept the same language. Let's make up a word for that relationship.

Definition 15: Two machines are equivalent if they both accept exactly the same language.

Next time: Can we in fact build a device which does magic?

Regular Expressions From Scratch, Part Nine: A Dream of a Machine

Eric Lippert — Thu, 15 Dec 2005 18:28:00 GMT

I want to come up with the simplest possible device that can identify whether a given string is a member of a given regular language. We need some kind of computer, but to make it easy to analyze, I want to put as many restrictions upon it as possible. For example, I want there to be very limited memory storage, I want to process only one character at a time, and so on.

Definition 14: A deterministic finite automaton (DFA) is an idealized machine. It has an input tape with a finite string written down on it in a given alphabet. The tape is exactly as long as the string. The DFA has a finite number of distinct states that it can be in. A DFA can be in only one state at a time, called the current state. The DFA always starts in a particular state called the start state. The DFA reads the string from left to right, one symbol at a time. There is no backtracking allowed. Each new symbol on the input tape may cause the machine to change the current state, but this choice can only be based on the current symbol and current state, and the rules for how that choice is made must be specified in advance and cannot change. Every state is either an acceptable state or an unacceptable state. When the machine is done reading the string, if it is in an acceptable state then the machine accepts the string, otherwise it rejects it.

I think you'll agree that this is a very limiting set of restrictions to place upon a computer.

We'll write down DFAs like this example:

Let M be a DFA such that:
Alphabet: S = {a, b}
States: K = {0, 1}
Start: s = 0
Acceptable: F = {1}

Every DFA has rules for determining what the state transitions are, one rule for every possible combination of states and symbols. For example, M might have rules:

current state	current input	new state
0	a	1
0	b	1
1	a	0
1	b	1

Note that there MUST be a rule for every combination of K and S. Since it will get cumbersome to write out these tables constantly I'm going to declare a new notation:

(0, a) → 1
(0, b) → 1
(1, a) → 0
(1, b) → 1

The arrow is read as "yields".

Consider the action of M on a tape containing aab. The machine uses these rules in this order:

(0, a) → 1
(1, a) → 0
(0, b) → 1

The string is accepted because we've run out of string and we're in an acceptable state.

Since any set of strings is a language, the set of strings which a machine M accepts forms a language. We can construct a finite sequence of sets (alphabet, states, transition rules, etc) that exactly characterizes M, so we can build a function L which maps M onto the language which M accepts. Call the language L(M).

Convince yourself that in the case above, L(M) = { (a(aa)*) ∪ (((a ∪ b)*b) (aa)*) } - that is, the regular language where every string is either an odd number of as or any string ending in a b followed by an even number of as.

This idea of listing the rules used by the machine is pretty good, but it requires us to mentally keep track of how far along the input tape the machine is. Also, there's some redundancy there, since obviously the state of the next line is going to be the same as the new state of the current line. I'm therefore going to change the notation I use to describe how a machine is working. For the example above we'd say in our new notation

(0, aab) → (1, ab) → (0, b) → (1, e)

Clearly this is the very definition of a mechanical process. It's going to get tedious to write out all of the steps in more complex machines. Let me declare a new "super arrow", which means "yields eventually".

(0, aab) ⇒ (1, e)

I'll use the super arrow to mean "yields in zero or more steps". Maybe one, maybe a thousand, but some finite number of steps.

What's the point of a DFA? It's pretty much the simplest thing we can possibly call a computer. It's got input and output and storage, but is very limited. The input, sure, it can be as long or as short as you want. But the output and the storage consists of a single "register" which can only hold one of a finite number of states. In our example today, we've got only a single bit of storage and four rules, and we can already accept a fairly complex language.

If we can reason about the limits of a DFA then we can determine whether a machine with finite storage is buff enough to recognize any interesting languages. Today's machine recognizes a regular language, and that's a start.

Next time, we'll add a little magic to a DFA.

Regular Expressions From Scratch, Part Eight: The Diagonal Argument

Eric Lippert — Mon, 12 Dec 2005 18:00:00 GMT

As we know, each regular expression is associated with a language by our function L. We also determined last time that we could list all members of S* and R in order first by length and then by alphabetical order. Here's a weird question: is the nth string in S*'s alphabetical ordering a member of the language associated with the nth alphabetical regular expression?

Let's make a table and see if we can discern any pattern.

s	r	s in L(r)?
e	a	no
a	b	no
b	Ø	no
aa	a*	yes
ab	b*	no
ba	Ø*	no
bb	a**	no
aaa	b**	no
aab	Ø**	no
aba	(aa)	no

We could continue this table forever, of course, and it seems that sometimes, pretty much at random, we're going to get a match. Matches might not be very often, but they will happen from time to time. As it turns out, it doesn't really matter whether we can come up with some pattern here. All that matters is that we can answer the question definitively for every possible entry.

Now define the language D such that D = { w in S* such that w has "no" in the column above }

D = {e, a, b, ab, ba, bb, aaa, aab, aba, abb, … }

This is a weird but perfectly well-defined property. Given any string we can determine very quickly whether it is a member of D or not. Just figure out where on the table it is, compute the nth regular expression, and see if it is a member of that regular language. If it is not, then it is in D.

Is there any regular expression that specifies the language D?

Which one would it be? We've got a complete alphabetical list of regular expressions, so let's just go down the list. By our definition of D, clearly it can't be any of them. If the third column is "yes" then the nth regexp matches a string not in D. If it is "no" then the nth regexp fails to match a string in D. Therefore there is no such regular expression that matches everything in D. Therefore D is not a regular language.

By a similar argument we can show that for every nonempty alphabet there exists a language which is not regular.

That's a pretty unlikely example of an irregular language though. Later we'll see that many perfectly normal languages are not regular, including, ironically enough, the regular expression language itself.

Remember that the reason why we came up with regular expressions in the first place is because we wanted a concise way to characterize languages using short, simple expressions. We of course could come up with other mechanical means to cleverly map between strings in one alphabet and languages in another, but ultimately it wouldn't do us much good if our aim is to capture all possible languages. The diagonal argument above doesn't depend upon any special features of regular expressions; it applies to any function that maps strings onto languages.

No language description system which maps finite strings onto languages can describe every language.

This is a stunning result; there are languages which we cannot characterize in any finite number of symbols, no matter how clever we are with our symbolic manipulations!

It's simply a fundamental fact that the definition of "language" we've chosen - an arbitrary set of finite strings - affords an immense number of possible results, more immense than the number of strings in any description language you'd care to name. Some - infinitely many - will be indefinable.

You might have noticed a bit of a hand-wave in today's entry: I've made the claim that we can easily take a regular expression and determine if a string is in its regular language. Over the next few entries we'll justify that claim by building some simple abstract machines that can make this determination.

Regular Expressions From Scratch, Part Seven: Listing All Members Of A Language In Order

Eric Lippert — Thu, 08 Dec 2005 18:00:00 GMT

Regular languages are by definition those languages which can be described by a regular expression. It should be clear from the definition that the union of any finite number of regular languages is a regular language, the concatenation of any finite number of regular languages is a regular language, and the Kleene Closure of any regular language is a regular language.

That's rather a lot of languages. Are all languages regular languages?

We should have a strong intuition that regular languages are a very limited subset of all languages. It seems like it would be hard to come up with a finite regular expression that, say, matches the language {w is in 1* such that there are a prime number of 1s in w}.

But we can do better than intuition. There are many interesting ways to prove that there exists at least one non-regular language. Over the next couple entries we'll find a non-regular language by using a clever trick. The first thing we'll need to do is enumerate every member of a language.

It doesn't really matter how we enumerate the language, as long as we guarantee that we eventually hit every member of the language. Here's one way to do such an enumeration.

Consider an alphabet, say S = {a, b}. We can enumerate all the strings of S* first by length and then by alphabetical order. Of course, we'll need to define an order for our alphabet, but since alphabets are by definition finite sets, we can choose any old order. In general, for alphabets consisting of Roman alphabet characters and numbers we'll use the standard alphabetical ordering we're all used to. We'll start with the one zero-length string, then the two one-symbol strings, then the four two-symbol strings, and so on:

e, a, b, aa, ab, ba, bb, aaa, aab, aba, abb, …

(A commenter correctly pointed out that there are languages which cannot be easily enumerated in this way, but my coming argument does not rely upon an alphabetical ordering. All we need is some schema for enumerating a language which guarantees that we eventually get all of them.)

Consider R, the regular expression language of S. We can enumerate it too, first by length and then by alphabetical order. Let's say that alphabetical order of R is * ( ) a b Ø ∪ just to pick an arbitrary ordering for the symbols. We can then enumerate R in the same way like this:

a, b, Ø, a*, b*, Ø*, a**, b**, Ø**, (aa), (ab), (aØ), (ba), (bb), (bØ), (Øa), (Øb), (ØØ), …

Of course we are leaving out any strings which are not in R, such as * or ))(.

Next time we'll use the fact that we can make both these lists to show that a non-regular language exists.