Fabulous Adventures In Coding : Recursion

Immutability in C# Part Seven: More on Binary Trees

Eric Lippert — Wed, 19 Dec 2007 23:01:00 GMT

Lots of good comments on my previous post. To briefly follow up:

One of the downsides of immutable tree implementations is that usually the tree must be built from the leaves up, which is not always convenient. We'll look at implementations which hide this fact from the user in future posts.

One smartypants pointed out that sure, this tree can have cycles -- if you have another binary tree implementation that has cycles and then paste such a cyclic tree in. True; what I meant was that using only the stated implementation, it's impossible to create a tree with cycles. If you are hell bent on making a bad tree, you can surely do that. Again, in a future post we'll have a tree which is built up by calling tree "mutating" operations, and those really will be guaranteed acyclic even in a world where there are other bad implementations hanging around.

And finally, one reader correctly identified the problem with my recursive implementation of in-order iteration. A tree of maximum height h ends up allocating O(h) iterator objects. The recursive calls get O(h) deep. If h is very large, this could blow the call stack. And if the tree has n nodes, each with an average height of O(h), then iterating each node will require O(h) recursive calls apiece. Therefore the total time cost in calls for iterating the entire tree is O(n h).

In a binary tree with n nodes, h is always between log n and n, so that means that this iterator has best case asymptotic performance of O(n log n) and worst case of O(n²) in time, and uses between O(log n) and O(n) in both call stack and heap space. That's pretty lousy.

A better implementation would be to write a tree traversal algorithm which does not use recursion. Use an explicit stack rather than the call stack:

public static IEnumerable<T> InOrder<T>(this IBinaryTree<T> tree)
{
    IStack<IBinaryTree<T>> stack = Stack<IBinaryTree<T>>.Empty;
    for (IBinaryTree<T> current = tree; !current.IsEmpty || !stack.IsEmpty; current = current.Right)
    {
        while (!current.IsEmpty)
        {
            stack = stack.Push(current);
            current = current.Left;
        }
        current = stack.Peek();
        stack = stack.Pop();
        yield return current.Value;
    }
}

This consumes O(n) time, O(h) heap space, and O(1) call stack space. It is also painful to read and analyze compared to the slow naive implementation.

I would very much like to add new syntax to a hypothetical future version of C# which would be a syntactic sugar for the code above. If, hypothetically, we were prioritizing features for unannounced future versions of C#, unfortunately that one would not be our highest priority, so I wouldn't expect it any time soon. My colleague Wes has written a great article that goes into more details about the problem above and ways we might solve it in the future, so check that out if you want the details.

I have the code for the next few blog posts written, but odds are good that I'm not going to have time to write the surrounding text until after the festive holiday season is over. I hope you have a wonderful time during the remainder of 2007 and we'll see you bright and early in 2008 for more fabulous adventures!

Immutability in C# Part Six: A Simple Binary Tree

Eric Lippert — Tue, 18 Dec 2007 22:30:00 GMT

OK, we've gotten pretty good at this by now. A straightforward implementation of an immutable generic binary tree requires little comment on the basics. A binary tree is either empty, or a value, left tree and right tree:

public interface IBinaryTree<V>
{
    bool IsEmpty { get; }
    V Value { get; }
    IBinaryTree<V> Left { get; }
    IBinaryTree<V> Right { get; }
}

And our implementation has, as always, a singleton empty tree. One minor point is that unlike previous data structures, where the structure itself knew how to build more of the same, this one does not. There's no obvious "insert" operation in a binary tree, so we'll just leave it up to the implementer. (A binary tree is essentially defined by its shape, not by the operations one can perform on it.) In this case, we'll just make a constructor:

public sealed class BinaryTree<V> : IBinaryTree<V>
{
    private sealed class EmptyBinaryTree : IBinaryTree<V>
    {
        public bool IsEmpty { get { return true; } }
        public IBinaryTree<V> Left { get { throw new Exception("Empty tree"); } }
        public IBinaryTree<V> Right { get { throw new Exception("Empty tree"); } }
        public V Value { get { throw new Exception("Empty tree"); } }
    }
    private static readonly EmptyBinaryTree empty = new EmptyBinaryTree();
    public static IBinaryTree<V> Empty { get { return empty; } }

    private readonly V value;
    private readonly IBinaryTree<V> left;
    private readonly IBinaryTree<V> right;

    public bool IsEmpty { get { return false; } }
    public V Value { get { return value; } }
    public IBinaryTree<V> Left { get { return left; } }
    public IBinaryTree<V> Right { get { return right; } }
    public BinaryTree(V value, IBinaryTree<V> left, IBinaryTree<V> right)
    {
        this.value = value;
        this.left = left ?? Empty;
        this.right = right ?? Empty;
    }
}

Note that another nice feature of immutable data structures is that it is impossible to accidentally (or deliberately!) create a tree which contains a cycle. In a mutable tree you could do something stupid like making the root a child of one of the leaves (or, for that matter, a child of itself!) Then you have to either live with the possibility that a user will create a cyclic "tree" and hope for the best (or, in other words, crash and/or hang) or you have to build yourself a cycle detector. Cycle detectors have potentially serious performance implications. It may be best to simply prevent the situation from arising in the first place.

A feature conspicuous by its absence from the above implementation is enumeration. Unlike a stack or a queue, there's no obvious best order in which to enumerate the members a binary tree. First off, there's the question of whether to enumerate in depth-first or breadth-first order, and then the matter of whether the parent comes before, after or between the children.

We could create extension methods which did various different enumerations on a binary tree. Here is a bad implementation of an "in order" traversal (depth first, parent comes between the children -- so-called because in a binary search tree, this enumerates the nodes in sorted order.)

public static IEnumerable<V> InOrder<V>(this IBinaryTree<V> tree)
{
    // BAD IMPLEMENTATION, DO NOT DO THIS
    if (tree.IsEmpty) yield break;
    foreach(V v in tree.Left.InOrder()) yield return v;
    yield return tree.Value;
    foreach(V v in tree.Right.InOrder()) yield return v;
}

This seems perfectly straightforward and sensible, but this implementation is actually quite flawed. We can do a lot better. What is wrong with this implementation of enumeration, and how would you fix it? (Hint: start your analysis with the assumption that the tree has maximum height h and total number of nodes n.)

Next time: a better implementation of traversal, and then after that, some more complex kinds of immutable binary tree.

Why does a recursive lambda cause a definite assignment error?

Eric Lippert — Fri, 18 Aug 2006 22:09:00 GMT

Hey fabulous readers, sorry for not much blogging lately. Between implementing LINQ and making plans to attend my first Burning Man, there's been no time for anything. We've had lots of new ideas generated here for the type inferencing algorithm which I will discuss in detail when I get back.

For now, here's a question I got recently about definite assignment analysis and recursive functions. Clearly this is legal:

delegate void Action<A>(A a);
delegate void Traversal<T>(Tree<T> t, Action<T> a);

static void DepthFirstTraversal<T>(Tree<T> t, Action<T> a){
if (t == null) return;
a(t.Value);
DepthFirstTraversal(t.Left, a);
DepthFirstTraversal(t.Right, a);
}
static void Traverse<T>(Tree<T> t, Traversal<T> f, Action<T> a){
f(t, a);
}
static void Print<T>(T t){
System.Console.WriteLine(t.ToString());
}
static void PrintTree<T>(Tree<T> t){
Traverse(t, DepthFirstTraversal, Print);
}

What if we wanted that last one to use lambdas?

static void PrintTree<T>(Tree<T> t){
Traversal<T> df = (t, a)=>{
if(t==null) return;
a(t.Value);
df(t.Left, a);
df(t.Right, a);
};
Traverse(t, df, Print);
}

That doesn't work because it's a definite assignment error. The compiler says that local variable df is being used before it is assigned. You'd have to say

Traversal<T> df = null;
df = (t, a)=>{

And suddenly it would start working.

What the heck? I mean, the original code is a syntactic sugar for:

privateclass Locals{
public Traversal<T> df;
public void M<T>(Tree<T> t, Action<T> a){
    if(t==null) return;
    a(t.Value);
    this.df(t.Left, a);
    this.df(t.Right, a);
}
}
static void PrintTree<T>(Tree<T> t){
Locals locals = new Locals();
locals.df = new Traversal<T>(locals.M<T>);
Traverse(t, locals.df, Print);
}

And clearly there is no use-before-initialization problem there.

The reason this is an error is because though there is actually no definite assignment problem here, it is not too hard to find a similar situation which does create a problem. Here's a silly example that nevertheless demonstrates a real problem:

delegate D D(D d);
//...
D d = ((D)(c=>c(d))(e=>e);

If you trace out the logic of that thing you'll find that in fact the local variable delegate d is potentially used before it is initialized, and therefore this must be an error.

We could immensely complicate the definite assignment rules by adding rules to discover which lambda and anonymous method bodies are guaranteed to be never called during the initialization, and therefore suppress definite assignment errors on some locals in their bodies. But one of the goals of the standardization process is to make rules that are clear and unambiguous and comprehensible by mortals; enabling recursive anonymous methods isn't a big enough win for the amount of complexity, particularly when you consider that there is the simple workaround of assigning to null and then reassigning.

Recursion, Part Six: Making CPS Work

Eric Lippert — Mon, 15 Aug 2005 13:00:00 GMT

JScript doesn't support CPS natively but we can write another dispatch engine that makes it work. There's only ever one active continuation, so lets have a new rule: JScript CPS functions are allowed to return, but the last thing that they do must be to tell our dispatch engine what the continuation was.

To keep things simple, we'll also say that every CPS-style function takes exactly one argument. Of course, that can be an object that contains multiple fields should the function require multiple arguments.

Let's go through our CPS treeDepth program and replace all the continuation calls with calls that tell the runtime engine what continuation to use next. We'll then return normally and let the runtime engine call the continuation for us.

The dispatch engine is very simple:

var continuation = null;
function cont(newfunc, newargs)
{
continuation = { func: newfunc, args : newargs };
}

function run()
{
while(continuation != null)
{
    var curfunc = continuation.func;
    var curargs = continuation.args;
    continuation = null;
    curfunc(curargs);
}
}

No stack at all this time. Just a single global variable saying what to call next. If there is nothing to call next, we're done. To determine the depth of a tree we simply tell the continuation engine what the next thing to do is, and start it running:

cont(treeDepth, {curtree: mytree, afterDepth: print});
run();

and hey presto, we've just implemented a recursive function with no stack that prints the depth of an arbitrary binary tree.

Nice trick! Of course, really what we've done is we've just hidden the call stack in another harder-to-see data structure. Where is it? (Hint: draw some diagrams that show what all the closure name table bindings are during the recursion.)

That's all I managed to get done here before my vacation. When I get back, maybe I'll pop the stack and pick up where we left off, or maybe I'll continue with something completely different and forget all about my current state. (Or maybe I'm actually interrupt driven, which seems most likely.) Who knows? Whatever happens though, it'll probably be another fabulous adventure in coding.

***********************************

Recursion, Part Five: More on CPS

Eric Lippert — Thu, 11 Aug 2005 13:00:00 GMT

Suppose we wanted to write this by-now-familiar little function in continuation passing style:

function treeDepth(curtree)
{
if (curtree == null)
    return 0;
else
{
    var leftDepth = treeDepth(curtree.left);
    var rightDepth = treeDepth(curtree.right);
    return 1 + Math.max(leftDepth, rightDepth);
}
}

Let's start by getting rid of the returns, since there are no returns in CPS. Rather, we "return" by passing the return value to the continuation. Remember, the continuation is "what are we supposed to do after this function is done?"

function treeDepth(curtree, afterDepth)
{
if (curtree == null)
    afterDepth(0);
else
{
    //UNDONE: var leftDepth = treeDepth(curtree.left);
    //UNDONE: var rightDepth = treeDepth(curtree.right);
    afterDepth(1 + Math.max(leftDepth, rightDepth));
}
}

We'll presume that the addition and maximum guys are not written in CPS, just to keep this simple.

OK, this is a start, but now we need to fix up those recursive calls. Consider the second recursive call. What is its continuation? That is, what needs to happen after the right tree depth is determined? Two things. First, the maximum of left and right must be calculated and incremented. Second, we've promised our caller that when we're done, we'll call its continuation. So let's write a closure that does those two things.

We're making progress. What do we need to do after we determine the depth of the left tree? Pass the left depth to a continuation that recurses on the right tree to determine the right depth, does the max, and calls the afterDepth continuation.

And we're done. We've got a CPS version of treeDepth which never returns and therefore we never need to keep track of what is on the stack, because we're never coming back.

Of course, like I said last time, in reality JScript does not detect that we're in this situation and optimize away the stack frames. This program still pushes interpreters on the system stack and still runs out of stack space if you have a 1500-deep branch in your tree.

But we can fake it out. Next time, we'll finish up with a CPS program that really doesn't run out of stack space, even in JScript.

(Eric is on vacation; this message was pre-recorded.)

Recursion, Part Four: Continuation Passing Style

Eric Lippert — Mon, 08 Aug 2005 13:00:00 GMT

We're getting hung up on the stack management aspects of recursive programming. Why do we need a stack at all? What purpose does it serve?

If you're like most developers, you probably learned about subroutines and functions at an early age. The idea is pretty straightforward. You stop what you're doing right now, go perform some other task, and then pick up where you left off, possibly using data computed by the other task. But this idea of "pick up where you left off" means that you have to have some mechanism for storing information about what you were doing and where to pick it up. A stack is a natural data structure for that, so we've been thinking a lot about stacks lately when we consider how to operate on recursive data structures.

But what if we simply denied the whole premise of functions? What if we said that no function was ever allowed to return? It either terminates the program, or itself calls another function.

You might be thinking that clearly the cure is worse than the disease. I mean, sure, now you don't have to keep track of what you were doing, but that's because you're never going to get back there! How is it possible to get any work done at all if every time you try to call a procedure for a little help, it takes over control and never comes back to you?

This sounds like crazy talk, but let's think about the consequences of removing all returns from the language. Three things are clear.

First, if you yourself write a function then calling another function has to be the last thing you do. Control is never coming back, so there's no point in having any code after a function call.

Second, suppose you have more work to do after you call a function foo. Foo is in the same position as you -- the last thing it does will either be a function call or terminating the program. Therefore, you need to put the work that needs to be done after foo into function bar, and make for darn sure that foo calls bar rather than terminating!

But if you didn't write foo, how do you know that foo is going to call bar when its done? You need to somehow tell foo to call bar, and hope that foo cooperates.

Third, if someone calls your function then they're in the same boat -- it's the last thing they're going to do, ever, because you're not coming back. You need to find some way to allow your caller to tell you what to do when you're done your work.

This idea of functions never returning, and callers passing their functions information about what needs to be done next, is a bizarre but powerful style of programming called Continuation Passing Style. The "information about what to do next" is called a "continuation", which is passed from function to function, hence the name.

Many languages support CPS natively -- Scheme, Ruby and Rhino, for example, all support CPS. To get it working in JScript will take some doing, but when we're done there will be no explicit stack at all in our formerly recursive program.

In the next episode I'm going to write our tree depth program in a mix of CPS and regular procedural programming. (To be purely CPS we'd write CPS versions of the addition operators, the Math.max function, and so on, but I'm not going to go there -- that's overkill.) But before we get into the tree depth program let's get a little more familiar with CPS.

Suppose we had this fragment of a JScript program:

function foo(x)
{
var y = bar();
blah(x,y);
}
function bar()
{
return 123;
}
function blah(a, b)
{
print(a+b);
}
foo(1);

I hope you agree that this is a very straightforward program. How would we do this in CPS? Well, first of all, every function would have to take an extra argument for the continuation. Our program calls foo and then terminates, so the continuation of the call to foo is "terminate the program". Let's presume that we have a magic function that does that.

function foo(x, cfoo)
{
// UNDONE: rewrite foo in CPS
}
foo(1, terminate);

Let's reason about foo from end to start. foo wants to ensure that three things happen in this order:

bar runs
blah runs
foo's continuation from its caller runs

But blah is never going to return. Therefore we'll make it blah's responsibility to call foo's continuation when it is done. We'll rewrite blah so that it does CPS, so that we can say:

function foo(x, cfoo)
{
var y = bar();
blah(x,y,cfoo);
}

OK, super, we've taken care of our responsibility to our caller by foisting it off onto blah. But now we have an additional problem. bar is never going to return, so we're never going to call blah! We've both failed in our responsibility to our caller and haven't done the work we need to do.

We have an additional problem. bar was going to return a value that we were going to use. But we've just removed returns from the language! Let's solve both of these problems. We'll say that bar takes a continuation and passes the value that it was going to return as an argument to its continuation. That's functionally the same thing, right? Before, you return the value to "whatever you were going to do next", now you pass the value to "whatever you were going to do next", same thing.

But what continuation should foo pass to bar? Well, what does foo want done when bar calls its continuation with its "return" value? That's easy. It wants to call blah with that value and foo's original arguments. We can write a helper function that does that.

foo(1, terminate);

foocontinuation is a closure, so it keeps track of what the values of x and afterfoo were when it was passed to bar. So we're all set here.

The global code passes 1, terminate to foo
foo passes foocontinuation to bar
bar passes 123 to foocontinuation
foocontinuation passes 1, 123, terminate to blah
blah adds together its arguments and passes 124, terminate to print
print presumably prints out 124 and calls terminate, and we're done

Not once did any function return in there, and we did everything in the right order.

Now of course in reality, JScript does not know that none of these functions are going to return. Nor is JScript smart enough to realize that even if they did return, none of these functions now does anything after the subroutine call, and therefore keeping track of the old frames on the stack is unnecessary, since they're never going to be read from again. If it did, we could totally write programs in this style and never worry about running out of stack space, but unfortunately, it doesn't.

But just as we helped JScript along by writing our own explicit frame stack, we can help it along by writing our own CPS system. Stay tuned!

(Eric is on vacation; this message was prerecorded.)

Recursion, Part Three: Building a Dispatch Engine

Eric Lippert — Thu, 04 Aug 2005 13:00:00 GMT

There's a particular technique that I like to use to solve problems -- it doesn't always work, but when it does, it can produce programs of surprising elegance and power. The technique is this: if you have a specific problem to solve, solve a more general problem, and then your problem is just a special case. It's like that joke someone mentioned during my series on high-dimensional arithmetic -- it's easy to imagine a 9-dimensional space, just imagine an n-dimensional space and set n=9.

That sounds like more work, doesn't it? Often it is, but paradoxically, sometimes it's less work.

Our solution last time for de-recursivizing a three-line recursive algorithm was an unreadable mess in part because the explicit stack logic was all mixed up with the bits of the function actually doing work. Rather than messing around trying to implement JScript's call logic in JScript itself, complete with frames and return values and activation objects, let's solve a more general problem. Let's build a little dispatch engine, and build some functions that use that dispatch engine to solve problems. Solving tree depth will then just be a special case.

Our dispatch engine will have two stacks: the functions-I-still-need-to-call stack, and the values-to-pass-to-them stack. But to keep it simple, the functions themselves will be responsible for taking the values on and off the value stack. With those design characteristics, our "dispatch engine" becomes very, very simple. It just pops a function off the function stack and calls it. When there are no more functions on the stack, we're done.

while(functionstack.length != 0)
functionstack.pop()();

We'll need to do a little stack magic to make it all work, but we can do it. We'll call the functions D, R and M:

D:
action: either computes the depth of a null node or pushes a program that "recursively" computes the depth of a node.
value stack precondition: value stack top contains tree node
value stack postcondition: if tree node on stack is null, replace it with zero. Otherwise push right and left trees
function stack postcondition: if tree node is not null, push MDRD onto function stack

R:
action: reorganizes the value stack so that the depth of the right tree can be determined
value stack precondition: top of stack is left tree depth, followed by tree node
value stack postcondition: top of stack is tree node, followed by left tree depth

M:
action: takes the max + 1 of two numbers
value stack precondition: two numbers on top of stack
value stack postcondition: one number on top of stack

Does that make sense? Let's think about how that would play out for a tree with root B, left child A, right child C. I'll use X to represent an empty (null) tree. We put D on the function stack and B on the value stack and start the engine:

functions values notes
D         B      B is not null, so we push its children on the value stack and MDRD on the function stack
MDRD      CA     Again, D is the top of the function stack, so we do it again, this time on A
MDRMDRD   CXX    Now D is at the top of the function stack and we have a null. Replace it with zero.
MDRMDR    CX0    Swap the top two
MDRMD     C0X    Replace null with zero
MDRM      C00    M pops both numbers, finds the larger, adds one, pushes it
MDR       C1     Swap the top two, replace the tree with its right child
MD        1C     and so on...
MMDRD     1XX
MMDR      1X0
MMD       10X
MM        100
M         11
          2

And if we've done it right, when we're done the value stack contains the depth of the tree that we started with, and the function stack is empty.

Looks promising. Let's write the code.

var functionstack = new Array();
var valuestack = new Array();

function treeDepth()
{
var curtree = valuestack.pop();
if (curtree == null)
    valuestack.push(0);
else
{
    functionstack.push(max);
    functionstack.push(treeDepth);
    functionstack.push(reorder);
    functionstack.push(treeDepth);
    valuestack.push(curtree.right);
    valuestack.push(curtree.left);
}
}

function reorder()
{
var leftdepth = valuestack.pop();
var righttree = valuestack.pop();
valuestack.push(leftdepth);
valuestack.push(righttree);
}

function max()
{
valuestack.push(1 + Math.max(valuestack.pop(), valuestack.pop()));
}

functionstack.push(treeDepth);
valuestack.push(mytree);
while(functionstack.length != 0)
functionstack.pop()();
print(valuestack.pop());

Each function is a lot cleaner now that the execution management has been pushed to an external engine. But we've paid a pretty high price in that it's now non-obvious that this program actually works on all trees. I mean, it sure looks like it works, but we haven't proven it. But at least its not directly recursive, so we're pretty sure that we don't have to worry about running out of system stack.

This time we attacked the problem of eliminating recursion by getting rid of the notion of explicit frames on the stack. Next time we'll take a very different tack and try to eliminate stacks altogether.

(Eric is on vacation; this message was pre-recorded.)

Recursion, Part Two: Unrolling a Recursive Function With an Explicit Stack

Eric Lippert — Mon, 01 Aug 2005 13:00:00 GMT

That recursive solution is pretty cool, to be sure. But there is one big problem with it. Consider this Jscript program that puts a few more nodes into our tree from last time:

function tree(value, left, right)
{
this.value = value;
  this.left = left;
  this.right = right;
}
for (var i = 1 ; i < 1500 ; ++i)
  mytree = new tree(i, mytree, null)
print(treeDepth(mytree));

This is not a very well-behaved binary tree – it’s actually mostly a linked list, hardly a tree at all. Recursing down 1500 branches to get to the bottom blows the stack.

Each time you enter a JScript function, that pushes a new JScript "activation" frame on the JScript stack. But the JScript stack is actually allocated off of the system heap, so this isn't the problem. Unfortunately, doing so also pushes a new copy of the interpreter on the system stack, which consumes a few hundred bytes of stack space. Since by default the system stack only has a million bytes available, this quickly consumes the entire system stack.

Recursive functions are sometimes the most elegant solution, but not always the most practical.

How are we going to solve this problem?

One way to do it is to not get into the situation in the first place. There are various algorithms for guarateeing that when you construct a binary tree that it is a "balanced" binary tree. That is, there are no long, deep paths. A perfectly balanced binary tree with 1500 nodes in it is guaranteed to require no more than 11 recursive steps, and doubling the number of nodes only adds on one more step. A perfect tree with a million nodes requires no more than 20 recursive steps, so odds are pretty good that you'll never blow the stack if you know you have a balanced tree. There are algorithms such as the famous red-black algorithm that guarantee that a binary tree is always within a factor of two of being perfectly balanced, so maybe a million-node tree is 40 deep, which seems doable.

Another approach is to keep track of how deep you are on the stack and bail out:

function treeDepth(curtree, stackdepth)
{
if (curtree == null)
    return 0;
if (stackdepth > 1000)
    throw "tree is too deep, bailing out!";
return 1 + Math.max(
    treeDepth(curtree.left, stackdepth + 1),
    treeDepth(curtree.right, stackdepth + 1));
}

But that's pretty hacked up. What if you're wrong about how much stack you have left? Maybe 1000 is too deep. What if you can't guarantee that the tree is balanced? What if you want to have an algorithm that works for any binary tree without danger of blowing the stack or getting a goofy exception?

Some languages, such as Scheme, have what’s called “tail recursion”, where they detect when they can optimize away pushing another frame on the stack, and therefore do not have this problem. But JScript does not have this optimization, and even if it did, this tree algorithm is a poor candidate for tail recursion. Tail recursion works really well when the recursive step is last, but this algorithm has two recursive steps, so one of them is going to be not last, obviously. (Though as we'll see in a few episodes, there are things we can do about that.)

Hmm.

I just said that the JScript stack is actually allocated off of the system heap -- the interpreter simulates a stack for the script engines rather than re-using the system stack. We can do the same thing with our recursive program: make the stack explicit rather than implicit.

That's going to be complicated. Think about what has to happen every time you call a function in JScript. Somehow the interpreter has to remember a whole bunch of things, like:

values of local variables in the current activation frame -- variables and arguments in the callee's frame must not change the variables in the caller's frame.
where to pick up execution when the callee returns
what to do with the return value of the callee
values of "anonymous locals" -- for example, when you have x = 1 + 1 + foo(); somehow the engine has to remember that there's a 2 sitting there in temporary storage waiting to be added when foo returns.

Let's take our recursive depth algorithm and refactor it a little bit so that the temporary variables are spelled out and each recursion happens on its own line:

function treeDepth(curtree)
{
if (curtree == null)
return 0;
var leftdepth = treeDepth(curtree.left);
var rightdepth = treeDepth(curtree.right);
return 1 + Math.max(leftdepth, rightdepth);
}

For each recursive step we'll need a frame that keeps track of what the three local variables are. Before each recursion we need to remember where we were, so that when we start executing, we know whether to pick up on the "var leftdepth..." line or "var rightdepth..." line or the "return 1 + ..." line. We'll remember that by keeping track of whether leftdepth and/or rightdepth are initialized on the frame. When we complete a calculation we'll pop off the current frame and fill in the "calling" frame's variables with the appropriate "return value" so that our execution loop knows where to go next.

Since there is no recursion here at all now, there's no danger of running out of stack. But good heavens, that's a freakin' mess compared to the three-line recursive solution.

This is probably more trouble than it's worth. Next time we'll look at another explicit-stack based approach that is a whole lot cleaner than this dog's breakfast.

(Eric is on vacation; this message was prerecorded.)

Recursion, Part One: Recursive Data Structures and Functions

Eric Lippert — Wed, 27 Jul 2005 13:00:00 GMT

The first thing to wrap your head around is recursively defined data structures. Let's start with something simple. Think about the abstract idea of “list”. Most people think of a “list” as an ordered collection of “items”, one after the other, with a beginning and an end. That’s a very non-recursive way to think about a list, but it’s the usual way of doing so.

The recursive way to think about a list emphasizes the similarity of a fragment of the list to the whole list. We might recursively define a list as either

the empty list with zero items on it, or
an item followed by a list.

This certainly appears to be a circular definition, but it’s really not. It’s more sort of a “spiral” definition. You start off with something extremely simple; it’s harder to get more simple than a list with no elements. Then you provide a rule for turning something simple into something very slightly more complicated. The recursive part of the definition means that you can take a list of any size, starting with zero, and construct a list one bigger than it, by sticking an item on to the beginning. It’s not a truly circular definition because there is always a “base case”. There is always some point from which you begin, and upon which you can build.

But the true power of the recursive definition comes from going the other way: if you have something that meets the definition, then there is usually a way to move back towards the base case, getting simpler and simpler as you go.

That’s what recursive programming is all about. All recursive functions follow the same basic pattern:

Am I in the base case? If so, return the easy solution.
Otherwise, figure out some way to make the problem closer to the base case and solve that problem.

That’s why you can get away with calling a function from itself – because you know that the recursion will eventually bottom out when you’ve found the base case.

Now, for list processing, a recursive solution is usually overkill. The iterative solution of “run down the list in a loop” is usually good enough. But many other problems are naturally solved by other recursively defined data structures. For example, consider the highly useful binary tree. A binary tree is either:

an empty tree, or
a value associated with two binary search trees, called the left and right child trees

For example, this is a binary tree:

          E
         / \
        /   \
       B     G
      / \     \
     A   C     H

The children of A, C and H, and the left child of G, are empty trees. (Sharp eyed readers will have noted that this particular example is a binary search tree, because every parent is between its children. We might discuss properties of search trees later.)

Pretty picture, but how are we going to represent this thing in code? Suppose you’ve got a binary tree: in JScript

var mytree =
{
left:
{
    left:
    {
      left: null,
      value : "A",
      right: null
    },
    value: "B",
    right:
    {
      left: null,
      value: "C",
      right: null
    }
},
value:"E",
right :
{
    left: null,
    value: "G",
    right:
    {
      left: null,
      value: "H",
      right: null
    }
}
};

Now we can use this recursive definition to write recursive functions that answer questions about this tree. For the rest of this series I'll use as my motivating example one of the simplest questions you can ask about a binary tree: what is the deepest you can go in a given tree?

We have a recursive definition, so come up with a recursive solution.

What’s the base case? An empty tree. Clearly the deepest you can go in an empty tree is zero steps.

What’s the recursive case? If you haven’t got an empty tree then you’ve got a value with two children. Each child is itself a tree. The deepest you can go from this point is one (counting yourself), plus the depth of the deepest child. And we've got a function that gives you the depth of a tree, so let's call it on each child:

function treeDepth(curtree) {
if (curtree == null)
return 0;
return 1 + Math.max(treeDepth(curtree.left), treeDepth(curtree.right));
}

It's magically delicious. It's like having n dominoes lined up. If you knock over the last one, and you know that every domino knocked over also knocks over the previous, then you know that they'll all go down eventually.

Formally, we could use complete induction to show that this works for all trees. We know that it works for trees of height zero. As an inductive hypothesis, assume that it works for trees of height <= k. Since a tree of height k+1 has one subtree of height k and another of height <=k, by hypothesis the recursion correctly calculates their heights and computes the max as k+1, and we're done. (Of course, how do we know that complete induction works? That question takes us into Peano's Axioms and other topics in the foundations of mathematical logic. Perhaps another time...)

Next time: Trouble in paradise: the out-of-stack error rears its ugly head.

Recursion, Part Zero

Eric Lippert — Mon, 25 Jul 2005 13:00:00 GMT

I’ve mentioned recursive programming techniques several times over the years in this blog (Topological Sort, How Not To Teach Recursion, Fibonacci Challenge, Recursion and Dynamic Programming, Sometimes Breadth Is Better Than Depth, and probably a few others). A lot of developers, particularly those without a formal education in computer science, are still pretty vague on this weird idea of functions that call themselves.

If you’re interviewing at Microsoft for a development position you can expect that at least one person will ask you a whiteboarding question that involves some kind of recursive problem. This isn’t because we’re a bunch of architecture astronauts or theory wonks, it’s because if you want to be a dev here, recursion is simply one of the tools that you have to have in your toolbox. We use recursive data structures and algorithms quite a bit, and people are expected to know how they work.

I’d therefore like to take this opportunity to do a series on some of the theory and practice of writing recursive and recursive-like functions. I want to show that (a) they're not as mysterious as they seem, but (b) that they are potentially dangerous, and (c) there are some absolutely crazy-but-fun techniques for turning a recursive program into a nonrecursive program. Hopefully by the time we're done here your head will hurt, but hey, it builds character.

I'm going to get as much of this done as possible and queued up, because I'm going to go dark shortly. I'm getting married in less than two weeks! Between getting ready for that and finishing up stuff at work, I'm going to be swamped. (And you'd better believe I'm not blogging about recursive programming techniques from my honeymoon.) I'll be back in late August and things should pick up then.

Bad Recursion Revisited

Eric Lippert — Thu, 28 Apr 2005 21:12:00 GMT

We have internal email lists for questions about programming languages. Here's one that came across recently that I thought illustrated a good point about language design.

An interview candidate gave the following awful implementation of the factorial function. (Recall that factorial is notated "n!", and is defined as the product of all the integers from 1 to n. 0! is defined as 1. So 4! = 4 x 3 x 2 x 1 = 24.)

If you note that n! = n x ((n-1)!) then a recursive solution comes to mind. When asked to implement the factorial function in C, an interview candidate came up with this:

int F(int x){
return (x > 1) ? (x * F(--x)) : x;
}

Now, leaving aside for the moment the fact that this badly-named function does no bounds checking on the inputs, potentially consumes the entire stack, returns a wrong or nonsensical answer for inputs less than one, and is a recursive solution to a problem that can easily be solved with a simple lookup table, it has another big problem -- it doesn't even return the correct answer for any input greater than one either! F(4) will return 6, not 24, in Microsoft C.

And yet the seemingly equivalent C#, JScript and VBScript programs return 24.

The question was "What the heck is up with that?"

This looks perfectly straightforward. If x is 4, then we evaluate the consequence of the trinary operator. 4 * F(3) = 4 * 3 * F(2) = 4 * 3 * 2 * F(1) = 4 * 3 * 2 * 1 = 24, right? What is broken with C?

Page 52 of K&R has the answer.

"C, like most languages, does not specify the order in which the operands of an operator are evaluated. (The exceptions are &&, ||, ?: and ','.) For example, in a statement like x = f()+g(); f may be evaluated before g or vice versa; thus if either f or g alters a variable on which the other depends, x can depend on the order of evaluation."

The Microsoft C compiler actually evaluates the function call first, which causes x to be decremented before the multiplication. So this is actually the same as 3 * F(3) = 3 * 2 * F(1) = 3 * 2 * 1 = 6.

Why does the specification call out that the compiler can choose any order for subexpression evaluation? Because then the compiler can choose to optimize the order so that it consumes a small number of stack or register slots for the results.

Of course, C was written back in the dark ages -- the 1970's -- when indeed most languages were pretty ill-specified and full of terrible "gotchas" like this. JScript, VBScript, C#, and most modern languages do guarantee that functions will be evaluated in left-to-right order. In all these languages,

x = f() + g() * h();

will evaluate f, then g, then h.

As K&R notes "The moral is that writing code that depends on order of evaluations is a bad programming practice in any language." Amen to that!

Breadth is sometimes better than depth

Eric Lippert — Mon, 27 Sep 2004 17:58:00 GMT

A while back the Scripting Guys blogged about using recursion to list all the files in a directory and all its subdirectories. Something that you'll notice about this very standard solution to this problem is that it is "depth first". That is, it lists the files like this:

c:\xyz.txt
c:\foo\bbb.txt
c:\foo\abc\def.txt
c:\foo\bar\baz.txt
c:\foo\bar\blah.txt
c:\qaz\zaq.txt
c:\qaz\lll\ggg.txt

It picks a branch and follows it down the tree as far as it can, listing directories as it goes, retreats back up as little as possible, picks another branch to go down, goes as far down it as possible, and so on, until the entire tree has been explored. You can see why this is called "depth first".

You don't need to use recursion to do this; recursion is just an elegant way to solve the problem. The essence of the algorithm is that information about the next node to process is stored in a stack. The algorithm is simple: pop the stack, process the directory, push all the subdirectories onto the stack, repeat until the stack is empty. Recursion just lets you do this algorithm without making the stack explicit; the stack is the call stack and doing the recursive call pushes a new frame on the call stack for you. We can easily do this without recursion by making the stack explicit:

Notice that I've wrapped the file manipulation code in try-catches. As I've mentioned before, in this script if there is a problem reading a file -- like, I don't own it and therefore I get a security violation -- I want to skip the problem and continue on. I do not want to write robust error handling for this trivial one-off script.

The other day I had a problem -- a few weeks earlier I had written a little test program and stuck it in a directory somewhere very near the top of my directory hierarchy, but I couldn't remember where exactly. I already had the depth-first program above, and could easily modify it to add a regular expression search of each file looking for a pattern. But there are some very deep directories on that drive where I knew there would be no matches, but would take a lot of time to search. I could have written code to omit those directories, but there was another way to solve this problem: since I knew that the file was going to be somewhere shallow, don't do a depth-first search in the first place. Do a breadth-first search. That is, search all the files in the top level, then all the files that are one deep, then all the files that are two deep, then three deep, and so on.

The nice thing is that it's basically the same algorithm, just with a minor change. Instead of doing a last-in-first-out stack, we'll just do a first-in-first-out queue and magically get a breadth-first traversal:

This then puts out the files in the order

c:\xyz.txt
c:\qaz\zaq.txt
c:\foo\bbb.txt
c:\qaz\lll\ggg.txt
c:\foo\abc\def.txt
c:\foo\bar\baz.txt
c:\foo\bar\blah.txt

going from shallowest to deepest gradually rather than continually diving way down and then coming back up.

Recursion and Dynamic Programming

Eric Lippert — Wed, 21 Jul 2004 16:43:00 GMT

Back in May we were discussing the merits and drawbacks of recursive programming techniques -- that is, writing functions which break down problems into sub-problems, and then call themselves.

The drawback of such an approach is that in some cases, you end up doing way more work than necessary. The example I used was the naïve Fibonacci algorithm:

function fib(n) {
      if (n == 1 || n == 2)
            return 1;
      return fib(n-1) + fib(n-2);
}

This is an exponential algorithm -- calculating the first few numbers is very cheap, but once we get up into the tens and twenties, you very quickly end up doing millions of recursive steps. Something that you might notice is that since fib(20) cannot possibly need to calculate more than 19 distinct previous fib values, that most of those millions of calculations are the same ones being done over and over again. (ASIDE: This seemingly obvious argument has a name: the Pigeonhole Principle.)

Therefore, there's a way to keep this recursive AND make it efficient: don't recompute a result you already know! How about this?

var fibarr = new Array(0, 1, 1);
function fib(n) {
      if (fibarr[n] == null)
            fibarr[n] = fib(n-1) + fib(n-2);
      return fibarr[n];
}

All of a sudden our formerly exponential algorithm has become linear if n has not already been calculated, and constant if it has. The time performance gets a huge win, but does so at a cost -- it chews up memory to store the results. It's a pretty small cost though compared to the win.

This technique of taking an inefficient algorithm and improving its performance by storing and reusing earlier results is called memoization or dynamic programming. Dynamic programming isn't hard, but a lot of script programmers don't know about this powerful technique, so I thought I might talk about it a bit today.

(ASIDE: Some would draw the subtle distinction that "dynamic programming" specifically means “using memoization to solve optimization problems” -- but I wouldn't. Note also that this term predates its use in the context of computer programming. The mathematician Richard Bellman originated the term in the 1950's -- he was studying the behaviour of Markovian systems. However, for my purposes I'm going to ignore the historical link between dynamic programming and the probability problems it was invented to solve, and talk just about the recursive programming applications of the technique.)

I started thinking about this because recently someone asked me how "diff" algorithms work -- you know, those programs that take two text files and show you what the differences and commonalities are. There are lots of different "diff" algorithms, and some of them use dynamic programming techniques.

Here's a related problem: determine the longest common subsequence of two arrays. Let me clarify what exactly I mean by that. Consider the following lists:

var original = new Array("jackdaws", "love", "my", "big", "sphinx", "of", "quartz");
var modified = new Array("some", "big", "jackdaws", "love", "my", "sphinx", "of", "quartz");

We want the longest sequence of items that are found in both sets in the same order. Here the longest common subsequence would be "jackdaws", "love", "my", "sphinx", "of", "quartz". The "big" isn't in the same order with respect to the rest of the subsequence, so it does not appear in the longest common subsequence.

I'm sure you can see how this pertains to the "diff" problem -- if you know the longest common subsequence then it makes sense that it would be the "in common" part, and everything that is not in the longest common subsequence is in the "things that have changed" part. (Of course, diff algorithms usually operate on lines of text, not on individual words, but the difference is irrelevant, so we'll stick with words for the sake of the example. Also, there are more sophisticated diff algorithms, which I won't go into today.)

How do we solve this problem? We could start by noticing that if the two sequences start with the same word, then the longest common subsequence always contains that word. We can automatically put that word on our list, and we've just reduced the problem to finding the longest common subset of the rest of the two lists. We've made the problem smaller, which is goodness.

But if the two lists do not begin with the same word, then one or both of them is not on the longest list. But one of them might be. How do we determine which one, if any, to add? Try it both ways and see! Either way, the two sub-problems are manipulating smaller lists, so we know that the recursion will eventually terminate. Whichever trial results in the longer common subsequence is the winner.

Instead of "throwing it away" by deleting the item from the array, instead let's just write a routine that finds the longest common subsequence that exists after a particular starting point in both arrays.

Here's a solution without memoization:

function LongestCommonSubsequence(arr1, arr2)
{
      return LCS(0, 0);

function LCS(start1, start2)
{
      var result;
      var remainder1;
      var remainder2;

// If we're at the end of either list, then the longest subsequence is empty

      if (start1 == arr1.length || start2 == arr2.length)
            result = new Array();
      else if (arr1[start1] == arr2[start2])
      {

// If the start element is the same in both, then it is on the LCS, so
// we'll just recurse on the remainder of both lists.

            result = new Array();
            result[0] = arr1[start1];
            result = result.concat(LCS(start1 + 1, start2 + 1));
      }
      else
      {

// We don't know which list we should discard from.
// Try both ways, pick whichever is better.

            remainder1 = LCS(start1 + 1, start2);
            remainder2 = LCS(start1, start2 + 1);
            if (remainder1.length > remainder2.length)
                  result = remainder1;
            else
                  result = remainder2;
      }

      return result;
}
}

This algorithm works, but holy cow is it ever inefficient! If the two arrays are both around n items long then this turns out to be exponential, for the same reason that the naïve fib implementation was exponential. It recalculates the same subsequences over and over again. In this particular example there are 2410 recursions to find the LCS of a seven and an eight item array! Clearly there are only 56 possible "start1" and "start2" combinations to try, so by the Pigeonhole Principle there must be massive re-calculation going on here. If the arrays get longer, this algorithm blows up enormously.

We can do a lot better, using the same technique as we used above. Every time we solve a subproblem, we make a note of the solution.

Which only does 82 recursions.

If you were really being clever, you might notice that the recursive solution eventually fills in almost the entire solution array, and only ever looks "up" from the current position, so you might as well simply write an iterative program that fills in the solution array backwards.

A downside of this particular implementation is that it chews up rather a lot of memory -- on the order of n^2 extra arrays in the solutions table, many of which are potentially a sizable fraction of the inputs. But in fact there are ways to use dynamic programming more aggressively to solve this problem using much less memory. There are ways to solve this problem that only require storing the lengths of the subsequences, rather than the subsequences themselves, and there are ways to "throw away" some of the stored results as you no longer need them. That would eliminate a lot of work -- constructing the subsequence arrays isn't cheap.

However, I don't want to get into that level of detail on this particular problem. Clearly, dynamic programming is a huge topic, and this just gives a brief taste. My point is simply that caching the solutions of problems you've already solved gives a potentially huge performance win, particularly when doing recursive calculations.

Results Of The Fibonacci Challenge Are In

Eric Lippert — Fri, 21 May 2004 00:11:00 GMT

Another bunch of good replies to my challenge of yesterday. And again, a bunch of answers that I didn't expect, and some of the points that I was thinking of weren't mentioned.

People seem to like this more conversational format; I'll probably use it more in the future.

Three readers had conjectures as to the running time of fib_1, the recursive algorithm. Deriving the asymptotic order is pretty straightforward. There must be some function f(n) such that fib_1 is O(f(n)). Well, what are the costs? It's trivially true that fib_1(1) is O(1), and so is fib_1(2). Let's therefore say that f(1) = 1 and f(2) = 1.

What's the cost of the algorithm for large n? Well, there's a constant bit at the beginning with the comparison, and a constant cost in the addition, but we can neglect those. The real work is done in the recursion. Clearly O(f(n)) = O(f(n-2) + O(f(n-1))

That looks familiar! f(n) = fib(n), so fib_1 is O(fib(n)).

We’ve established the truth of Frederik Slijkerman's comment that recursive fib has a running time proportional to the size of its output. Steven Bone conjectured that recursive fib was O(2ⁿ). He was also right -- almost. How is that possible?

The definition of the Fibonacci numbers that I gave is called a "recurrence relation". Let me define two more recurrences:

s(1) = s(2) = 1, s(n) = s(n-2) + s(n-2)
f(1) = f(2) = 1, f(n) = f(n-2) + f(n-1)
b(1) = b(2) = 1, b(n) = b(n-1) + b(n-1)

You agree that it is clear that s(n) <= f(n) <= b(n), right? The first sequence is 1, 1, 2, 2, 4, 4, 8, 8, 16, 16... which grows at O(SQRT(2)ⁿ). The second sequence is 1, 1, 2, 4, 8, 16, 32, ... which clearly grows at O(2ⁿ). f is sandwiched between two functions that grow at O(somethingⁿ) so it must also be O(somethingⁿ)!

The exact constant doesn't really matter for our purposes. It's actually about O(1.6ⁿ). Yes, I'm handwaving here. A more precise definition of order notation would make it clear when two functions are asymptotically equivalent, but that's a subject for another day. There are different notations for functions with asymptotic upper, lower, and strict approximations, but I don't want to get into that now. The point is that this thing grows way faster than any polynomial in the long run and, more important, grows extremely quickly even in the short run.

Clearly fib_1 consumes O(n) memory off the stack. Equally clearly, fib_2 is O(n) and consumes O(1) memory. I hinted that there was an O(1) solution. Indeed there is, provided that we more carefully define the problem. Before I get to it, I mentioned that I often ask about robustness in interviews.

Clearly it's a good idea to make sure that n is one or larger; fib_2 goes into an infinite loop if it isn't. But that's certainly not all! What's the actual vs. desired output of

fib_2();
fib_2(1, 2, 3)
fib_2("2B||!2B");
fib_2("hello, world!");
fib_2(null);
fib_2(3.14159);
fib_2(new Date());

and so on? None of those make much sense and some of them go into infinite loops. Remember, JScript isn't C++ in fancy dress. It's not a statically typed language, so if you have type restrictions, you're going to have to write a bunch of runtime code, or use a different language.

Furthermore, clearly there is a sensible lower bound of one. Is there a sensible upper bound? I note that

fib_2(77) --> 5527939700884757
fib_2(78) --> 8944394323791464
fib_2(79) --> 14472334024676220

How do we add together a number ending in 7 and a number ending in 4 and end up with an even number? We've run out of precision. A floating point number only has 53 bits of precision; as soon as the value exceeds 2⁵³, we start dropping bits off the end and therefore start returning incorrect results. I asked for a function that produced the nth Fibonacci number, not one that produced a close approximation of it! That's a point which a good interview candidate would clarify.

Assuming that we really do need exact numbers, and assuming further that the caller is willing to accept the limitations on precision that an IEEE float imposes, this method only has 78 sensible inputs -- the first 78 positive integers. Therefore it only has 78 possible outputs. When I hear that a method only has 78 outputs and the inputs are integers, I like table driven solutions. (If we were using some other language that had 64 bit or 32 bit integers, the limits would be different, but they'd still be small enough to make a table-driven solution feasible.)

var fibarr = new Array(null, 1, 1, 2, 3, 5, /*[...]*/, 8944394323791464);
function fib_3(n)
{
// [appropriate checking here.]
return fibarr[n];
}

That's certainly O(1).

If those aren't sensible restrictions, then we have waaaaay more work to do. We need to figure out what the restrictions are, then write a numeric computation library that handles numbers up to the precision we need, and so on. That introduces a whole other set of costs which we'd have to analyze on a case by case basis. If we really did have to handle arbitrarily big numbers then a well-implemented algorithm would probably end up being O(log n). Several people mentioned Binet's Formula, which we could use. However, the mathematics of all that is more than I want to tackle in today's blog entry; I have work to do.

If the caller really, really wanted this to be as fast as possible in raw speed terms, not algorithmic complexity terms, then we have to start talking about what we can change. Language? Seems to me that a C++ or assembly language implementation is going to be faster all around. What hardware are we stuck with? How big is the processor cache? Will the table fit into it? Can we hint to the compiler that the table is read-only after it is constructed and eke out more speed somehow? What are these numbers being used for anyway, and how do we know that this method is even the bottleneck?

Finally, Dan Shappir mentioned tail recursion and lazy evaluation using a zipper function on an infinite list. That's quite coincidental; a couple days ago coworker of mine showed me a problem from an old ACM contest he'd been working on for fun. The problem was to implement a particular subset of Haskell's infinite list and zipper function capabilities. Which got me thinking that there are lots of interesting programming ideas that I could blog about. Perhaps I'll take up some of that stuff in the coming weeks.

How Not To Teach Recursion

Eric Lippert — Thu, 20 May 2004 01:12:00 GMT

A Joel On Software reader asked the other day for examples of recursive functions other than old chestnuts like Fibonacci or factorial. Excellent question. I suggested topological sort, of course, but there are plenty of other examples that are way better than the Fibonacci numbers. Why do I think Fibonacci is bad? Read on.

In case it's slipped your mind, the Fibonacci numbers are defined as follows:

fib(1) = 1
fib(2) = 1
fib(n) = fib(n-1) + fib(n-2)

thus, they go 1, 1, 2, 3, 5, 8, 13, 21, …

The Fibonacci numbers have many interesting properties. For example, suppose you have a pair of rabbits that every year gives birth to a pair of rabbits, rabbits take one year to reach maturity, and rabbits never die. In that (admittedly unrealistic) situation, the number of pairs alive at any given time is a Fibonacci number. Clearly the number alive this year is the number alive the year before (who all live) plus the number alive two years before (who are now mature enough to breed.) Fibonacci numbers also have a relationship to the Golden Mean, logarithmic spirals, and many other interesting nooks of mathematics. But that's not what I want to talk about today.

Because they have a straightforward recursive definition, it's natural to introduce recursive programming to students by simply translating the definition into code:

function fib_1(n)
{
    if (n == 1 || n == 2)
        return 1;
    else
      return fib_1(n-1) + fib_1(n-2);
}

I am in principle very much for this pedagogic approach. Recursive programming can be difficult to get your head around. Starting with recursive definitions, getting your head around those, and then using those definitions in code is a pretty good way to approach the material. If you think of lists as being "zero or more items in a row", it is conceptually hard to think of anything other than an iterative approach to processing the list. If you think of a list as "a list may be empty, or may be an item followed by a list", then the definition of the data leads itself to a recursive programming approach very naturally.

Practically however, this is probably the worst possible simple way to solve the Fibonacci problem. When you introduce recursion by using it to solve a problem which iteration solves much, much better, students come away thinking that recursion is dumb -- that it is both harder to understand and produces worse solutions. Better to pick an example where the iterative solution is not better!

In fact, this is a good whiteboarding question for interviews: implement me a recursive Fibonacci algorithm, tell me why it is terrible, and write me a better algorithm. A good candidate should be able to come up with the above and something like this:

function fib_2(n)
{
    if (n == 1 || n == 2)
        return 1;
    var fibprev = 1;
    var fib = 1;
    for (var cur = 2 ; cur < n ; ++cur)
    {
        var temp = fib;
        fib += fibprev;
        fibprev = temp;
    }
    return fib;
}

Typical questions that I might ask at this point in an interview are

What are the asymptotic running time and memory usage of the two algorithms you've written?

Can you write me an algorithm with even better asymptotic running time and memory usage? Alternatively, can you convince me that this is as good as it gets?

Suppose I asked you to make this algorithm absolutely as fast as possible. What would you do?

Now suppose that I do not care much about raw speed but I do want the method to be robust in the face of bad input. Now what do you do?

and so on. (Anyone who cares to propose answers to these can leave comments; consider this a spoiler warning if you want to work them out for yourself. Working out the asymptotic running time of recursive fib is quite easy and produces a somewhat surprising result.)

Another old chestnut that's often used as an introduction to recursion is the famous Tower Of Hanoi problem. Briefly stated, you have a number of disks with a hole in the middle, and three spikes upon which you can place disks. The disks are all of different sizes and are arranged from biggest to smallest in a pyramid on one spike. You must move one disk at a time from spike to spike, you must never place a larger disk on top of a smaller disk, and you must end up with the entire pyramid on a different spike than you started with.

The recursive solution is straightforward. Moving a "sub-pyramid" of size one is obviously trivial. To move a sub-pyramid of size n, first move a sub-pyramid of size n-1 to an extra spike. Then move the bottommost disk of the size n pyramid to the other extra spike. Then move the sub-pyramid of size n-1 onto the disk on the second extra spike. A nice recursive solution!

Many recursive implementations for this old chestnut exist. But I'm not a big fan of this problem as a pedagogic approach to recursion for two reasons. First, it's totally contrived (as is “fib“). Second, there is a very simple iterative solution for this problem, which is almost never mentioned when it is used as an example of the power of recursive programming! Again, we should pick examples where recursion is clearly preferable.

In case you're curious, the iterative solution can be produced by writing a program that implements these rules in a loop:

Number the disks 1 to n starting from the top.

Never move an odd disk onto an odd disk.

Never move an even disk onto an even disk.

Never move a larger disk onto a smaller disk.

Never move the same disk twice in a row.

Never move a disk onto an empty spike unless you have to.

Move disks until there are no legal moves.

If you don't believe me, try it yourself with ace through ten of a deck of cards sometime. Or, perhaps start with ace through four -- ace through ten will take you a while. Both the recursive and iterative solutions are optimal in the number of moves. The recursive solution is O(n) in terms of memory usage, the iterative solution is O(1).