Yet Another Language Geek : Functional Programming

The Marvels of Monads

wesdyer — Fri, 11 Jan 2008 03:06:13 GMT

If the word "continuation" causes eyes to glaze over, then the word "monad" induces mental paralysis. Perhaps, this is why some have begun inventing more benign names for monads.

These days, monads are the celebrities of programming language theory. They gloss the cover of blogs and have been compared to everything from boxes of fruit to love affairs. Nerds everywhere are exclaiming that the experience of understanding monads causes a pleasantly painful mental sensation.

Like continuations, monads are simpler than they sound and are very useful in many situations. In fact, programmers write code in a variety of languages that implicitly use common monads without even breaking a sweat.

With all of the attention that monads get, why am I writing yet another explanation of monads? Not to compare them to some everyday occurrence or to chronicle my journey to understanding. I explain monads because I need monads. They elegantly solve programming problems in a number of languages and contexts.

Introducing Monads

Monads come from category theory. Moggi introduced them to computer scientists to aid in the analysis of the semantics of computations. In an excellent paper, The Essence of Functional Programming, Wadler showed that monads are generally useful in computer programs to compose together functions which operate on amplified values rather than values. Monads became an important part of the programming language Haskell where they tackle the awkward squad: IO, concurrency, exceptions, and foreign-function calls.

Monads enjoy tremendous success in Haskell, but like an actor who does well in a particular role, monads are now stereotyped in the minds of most programmers as useful only in pure lazy functional languages. This is unfortunate, because monads are more broadly applicable.

Controlling Complexity

Composition is the key to controlling complexity in software. In The Structure and Interpretation of Computer Programs, Abelson and Sussman argue that composition beautifully expresses complex systems from simple patterns.

In our study of program design, we have seen that expert programmers control the complexity of their designs with the same general techniques used by designers of all complex systems. They combine primitive elements to form compound objects, they abstract compound objects to form higher-level building blocks, and they preserve modularity by adopting appropriate large-scale views of system structure.

One form of composition, function composition, succinctly describes the dependencies between function calls. Function composition takes two functions and plumbs the result from the second function into the input of the first function, thereby forming one function.

public static Func<T, V> Compose<T, U, V>(this Func<U, V> f, Func<T, U> g)
{
    return x => f(g(x));
}

For example, instead of applying g to the value x and then applying f to the result, compose f with g and then apply the result to the value x. The key difference is the abstraction of the dependency between f and g.

var r = f(g(x));         // without function composition
var r = f.Compose(g)(x); // with function composition

Given the function Identity, function composition must obey three laws.

public static T Identity<T>(this T value)
{
    return value;
}

1. Left identity

     Identity.Compose(f) = f

2. Right identity

     f.Compose(Identity) = f

3. Associative

     f.Compose(g.Compose(h)) = (f.Compose(g)).Compose(h)

Often, values are not enough. Constructed types amplify values. The type IEnumerable<T> represents a lazily computed list of values of type T. The type Nullable<T> represents a possibly missing value of type T. The type Func<Func<T, Answer>, Answer> represents a function, which returns an Answer given a continuation, which takes a T and returns an Answer. Each of these types amplifies the type T.

Suppose that instead of composing functions which return values, we compose functions which take values and return amplified values. Let M<T> denote the type of the amplified values.

public static Func<T, M<V>> Compose<T, U, V>(this Func<U, M<V>> f, Func<T, M<U>> g)
{
    return x => f(g(x)); // error, g(x) returns M<U> and f takes U
}

Function composition fails, because the return and input types do not match. Composition with amplified values requires a function that accesses the underlying value and feeds it to the next function. Call that function "Bind" and use it to define function composition.

public static Func<T, M<V>> Compose<T, U, V>(this Func<U, M<V>> f, Func<T, M<U>> g)
{
    return x => Bind(g(x), f);
}

The input and output types determine the signature of Bind. Therefore, Bind takes an amplified value, M<U>, and a function from U to M<V>, and returns an amplified value, M<V>.

public static M<V> Bind<U, V>(this M<U> m, Func<U, M<V>> k)

The body of Bind depends on the mechanics of the amplified values, M<T>. Each amplified type will need a distinct definition of Bind.

In addition to Bind, define a function which takes an unamplified value and amplifies it. Call this function "Unit".

public static M<T> Unit<T>(this T value)

Together the amplified type, M<T>, the function Bind, and the function Unit enable function composition with amplified values.

Meet the Monads

Viola, we have invented monads.

Monads are a triple consisting of a type, a Unit function, and a Bind function. Furthermore, to be a monad, the triple must satisfy three laws:

1. Left Identity

     Bind(Unit(e), k) = k(e)

2. Right Identity

     Bind(m, Unit) = m

3. Associative

     Bind(m, x => Bind(k(x), y => h(y)) = Bind(Bind(m, x => k(x)), y => h(y))

The laws are similar to those of function composition. This is not a coincidence. They guarantee that the monad is well behaved and composition works properly.

To define a particular monad, the writer supplies the triple, thereby specifying the mechanics of the amplified values.

The Identity Monad

The simplest monad is the Identity monad. The type represents a wrapper containing a value.

class Identity<T>
{
    public T Value { get; private set; }
    public Identity(T value) { this.Value = value; }
}

The Unit function takes a value and returns a new instance of Identity, which wraps the value.

static Identity<T> Unit<T>(T value)
{
    return new Identity<T>(value);
}

The bind function takes an instance of Identity, unwraps the value, and invokes the delegate, k, with the unwrapped value. The result is a new instance of Identity.

static Identity<U> Bind<T,U>(Identity<T> id, Func<T,Identity<U>> k)
{
    return k(id.Value);
}

Consider a simple program that creates two Identity wrappers and performs an operation on the wrapped values. First, bind x to the value within the wrapper containing the value five. Then, bind y to the value within the wrapper containing the value six. Finally, add the values, x and y, together. The result is an instance of Identity wrapping the value eleven.

var r = Bind(Unit(5), x =>
Bind(Unit(6), y =>
Unit(x + y)));

Console.WriteLine(r.Value);

While this works, it is rather clumsy. It would be nice to have syntax for dealing with the monad. Fortunately, we do.

C# 3.0 introduced query comprehensions which are actually monad comprehensions in disguise. We can rewrite the identity monad to use LINQ. Perhaps, it should have been called LINM (Language INtegrated Monads), but it just doesn't have the same ring to it.

Rename the method Unit to ToIdentity and Bind to SelectMany. Then, make them both extension methods.

public static Identity<T> ToIdentity<T>(this T value)
{
    return new Identity<T>(value);
}

public static Identity<U> SelectMany<T, U>(this Identity<T> id, Func<T, Identity<U>> k)
{
    return k(id.Value);
}

The changes impact the calling code.

var r = 5.ToIdentity().SelectMany(
            x => 6.ToIdentity().SelectMany(
                y => (x + y).ToIdentity()));

Console.WriteLine(r.Value);

Equivalent methods are part of the standard query operators defined for LINQ. However, the standard query operators also include a slightly different version of SelectMany for performance reasons. It combines Bind with Unit, so that lambdas are not deeply nested. The signature is the same except for an extra argument that is a delegate which takes two arguments and returns a value. The delegate combines the two values together. This version of SelectMany binds x to the wrapped value, applies k to x, binds the result to y, and then applies the combining function, s, to x and y. The resultant value is wrapped and returned.

public static Identity<V> SelectMany<T, U, V>(this Identity<T> id, Func<T, Identity<U>> k, Func<T,U,V> s)
{
    return id.SelectMany(x => k(x).SelectMany(y => s(x, y).ToIdentity()));
}

Of course, we can remove some of the code from the generalized solution by using our knowledge of the Identity monad.

public static Identity<V> SelectMany<T, U, V>(this Identity<T> id, Func<T, Identity<U>> k, Func<T,U,V> s)
{
    return s(id.Value, k(id.Value).Value).ToIdentity();
}

The call-site does not need to nest lambdas.

var r = 5.ToIdentity()
         .SelectMany(x => 6.ToIdentity(), (x, y) => x + y);

Console.WriteLine(r.Value);

With the new definition of SelectMany, programmers can use C#'s query comprehension syntax. The from notation binds the introduced variable to the value wrapped by the expression on the right. This allows subsequent expressions to use the wrapped values without directly calling SelectMany.

var r = from x in 5.ToIdentity()
        from y in 6.ToIdentity()
        select x + y;

Since the original SelectMany definition corresponds directly to the monadic Bind function and because the existence of a generalized transformation has been demonstrated, the remainder of the post will use the original signature. But, keep in mind that the second definition is the one used by the query syntax.

The Maybe Monad

The Identity monad is an example of a monadic container type where the Identity monad wrapped a value. If we change the definition to contain either a value or a missing value then we have the Maybe monad.

Again, we need a type definition. The Maybe type is similar to the Identity type but adds a property denoting whether a value is missing. It also has a predefined instance, Nothing, representing all instances lacking a value.

class Maybe<T>
{
    public readonly static Maybe<T> Nothing = new Maybe<T>();
    public T Value { get; private set; }
    public bool HasValue { get; private set; }
    Maybe()
    {
        HasValue = false;
    }
    public Maybe(T value)
    {
        Value = value;
        HasValue = true;
    }
}

The Unit function takes a value and constructs a Maybe instance, which wraps the value.

public static Maybe<T> ToMaybe<T>(this T value)
{
    return new Maybe<T>(value);
}

The Bind function takes a Maybe instance and if there is a value then it applies the delegate to the contained value. Otherwise, it returns Nothing.

public static Maybe<U> SelectMany<T, U>(this Maybe<T> m, Func<T, Maybe<U>> k)
{
    if (!m.HasValue)
        return Maybe<U>.Nothing;
    return k(m.Value);
}

The programmer can use the comprehension syntax to work with the Maybe monad. For example, create an instance of Maybe containing the value five and add it to Nothing.

var r = from x in 5.ToMaybe()
        from y in Maybe<int>.Nothing
        select x + y;

Console.WriteLine(r.HasValue ? r.Value.ToString() : "Nothing");

The result is "Nothing". We have implemented the null propagation of nullables without explicit language support.

The List Monad

Another important container type is the list type. In fact, the list monad is at the heart of LINQ. The type IEnumerable<T> denotes a lazily computed list.

The Unit function takes a value and returns a list, which contains only that value.

public static IEnumerable<T> ToList<T>(this T value)
{
    yield return value;
}

The Bind function takes an IEnumerable<T>, a delegate, which takes a T and returns an IEnumerable, and returns an IEnumerable.

public static IEnumerable<U> SelectMany<T, U>(this IEnumerable<T> m, Func<T, IEnumerable<U>> k)
{
    foreach (var x in m)
        foreach (var y in k(x))
            yield return y;
}

Now, the programmer can write the familiar query expressions with IEnumerable<T>.

var r = from x in new[] { 0, 1, 2 }
        from y in new[] { 0, 1, 2 }
        select x + y;

foreach (var i in r)
    Console.WriteLine(i);

Remember that it is the monad that enables the magic.

The Continuation Monad

The continuation monad answers the question that was posed at the end of the last post: how can a programmer write CPS code in a more palatable way?

The type of the continuation monad, K, is a delegate which when given a continuation, which takes an argument and returns an answer, will return an answer.

delegate Answer K<T,Answer>(Func<T,Answer> k);

The type K fundamentally differs from types Identity<T>, Maybe<T>, and IEnumerable<T>. All the other monads represent container types and allow computations to be specified in terms of the values rather than the containers, but the continuation monad contains nothing. Rather, it composes together continuations the user writes.

To be a monad, there must be a Unit function which takes a T and returns a K<T,Answer> for some answer type.

public static K<T, Answer> ToContinuation<T, Answer>(this T value)

What should it do? When in doubt, look to the types. The method takes a T and returns a function, which takes a function from T to Answer, and returns an Answer. Therefore, the method must return a function and the only argument of that function must be a function from T to Answer. Call the argument c.

return (Func<T, Answer> c) => ...

The body of the lambda must return a value of type Answer. Values of type Func<T,Answer> and a T are available. Apply c to value and the result is of type Answer.

return (Func<T, Answer> c) => c(value);

To be a monad, Bind must take a K<T,Answer> and a function from T to K<U, Answer> and return a K<U, Answer>.

public static K<U, Answer> SelectMany<T, U, Answer>(this K<T, Answer> m, Func<T, K<U, Answer>> k)

But what about the body? The result must be of type K<U, Answer>, but how is a result of the correct type formed?

Expand K's definition to gain some insight.

return type

 Func<Func<U, Answer>, Answer>

m's type

 Func<Func<T, Answer>, Answer>

k's type

 Func<T, Func<Func<U, Answer>, Answer>>

Applying k to a value of type T results in a value of type K<U,Answer>, but no value of type T is available. Build the return type directly by constructing a function, which takes a function from U to Answer. Call the parameter c.

return (Func<U,Answer> c) => ...

The body must be type of Answer so that the return type of Bind is K<U,Answer>. Perhaps, m could be applied to a function from T to Answer. The result is a value of type Answer.

return (Func<U,Answer> c) => m(...)

The expression inside the invocation of m must be of type Func<T,Answer>. Since there is nothing of that type, construct the function by creating a lambda with one parameter, x, of type T.

return (Func<U,Answer> c) => m((T x) => ...)

The body of this lambda must be of type Answer. Values of type T, Func<U,Answer>, and Func<T,Func<Func<U,Answer>, Answer>> haven't been used yet. Apply k to x.

return (Func<U,Answer> c) => m((T x) => k(x)...)

The result is a value of type Func<Func<U,Answer>,Answer>. Apply the result to c.

return (Func<U,Answer> c) => m((T x) => k(x)(c));

The continuation monad turns the computation inside out. The comprehension syntax can be used to construct continuations.

Construct a computation, which invokes a continuation with the value seven. Pass this computation to another computation, which invokes a continuation with the value six. Pass this computation to another computation, which invokes a continuation with the result of adding the results of the first two continuations together. Finally, pass a continuation, which replaces "1"s with "a"s, to the result.

var r = from x in 7.ToContinuation<int,string>()
        from y in 6.ToContinuation<int,string>()
        select x + y;

Console.WriteLine(r(z => z.ToString().Replace('1', 'a'))); // displays a3

The continuation monad does the heavy-lifting of constructing the continuations.

Monadic Magic

Beautiful composition of amplified values requires monads. The Identity, Maybe, and IEnumerable monads demonstrate the power of monads as container types. The continuation monad, K, shows how monads can readily express complex computation.

Stay tuned for more with monads. Until then, see what monads can do for you.

Continuation-Passing Style

wesdyer — Sat, 22 Dec 2007 09:51:58 GMT

There are some technical words that cause quite a stir even amongst geeks. When someone says the word "continuation", people's eyes glaze over and they seek the first opportunity to change the subject. The stir is caused because most people don't understand what a continuation is or why someone would want to use one. This is unfortunate, because they really are rather more simple than they sound.

Defining Continuations

A continuation represents the remainder of a computation given a point in that computation. For example, suppose that two methods are defined: M and Main. The method Main invokes M and then writes "Done" to the console. The method M assigns x the value five, invokes F, increments x, and writes x to the console.

static void M()
{
    var x = 5;
    F();  // <----------- Point in the computation
    ++x;
    Console.WriteLine(x);
}

static void Main()
{
    M();
    Console.WriteLine("Done");
}

The continuation of the computation at the invocation of F is the remainder of the program execution beginning with incrementing x in the method M. In this case, the continuation includes incrementing x, writing x, returning to Main, and writing "Done" to the console.

Some languages give programmers explicit access to continuations. For example, Scheme has a function which calls a function passing a function representing the current continuation. The function is aptly named call with current continuation or call/cc. If such a function existed in .NET it might return a T given a delegate that returns a T given a delegate from T to T.

T CallCC<T>(Func<Func<T,T>, T> f)

Continuations are the functional counterparts of GOTOs both in power and inscrutability. They can express arbitrary control flow like coroutines and exceptions while bewildering some of the brightest programmers.

Returning Values with Continuations

The simplest use of a continuation is to simulate returning out of a function. Suppose that .NET had the function CallCC. If Main calls a function Foo passing the value four and if Foo immediately invokes CallCC with a lambda that binds Return to the continuation at the point of the call to CallCC, then when Return is invoked inside of the lambda with the value four, computation will immediately jump out of the lambda to the point after CallCC but before the return with the value four on the stack. This happens regardless what of occurs after the invocation of Return inside the lambda.

static int Foo(int n)
{
    return CallCC<int>(Return =>
        {
            Return(n);
            return n + 1;
        });
}

static void Main()
{
    Foo(4);
}

The result of running the program is therefore four and not five.

All of this demonstrates that returning from a function is equivalent to invoking the continuation defined at the function's call-site. When a function "returns", it implicitly invokes the continuation of its call-site.

When people explain continuations, they usually discuss stack frames and instruction pointers. It is easy to see why. The implicit invocation of the "return" continuation restores the stack frame at the call-site and sets the instruction pointer to the instruction immediately after the call-site. This is what invoking a continuation does: restore the appropriate stack frame and set the instruction pointer.

Continuation-Passing Style

It is still possible to use continuations if a language does not support a function like call/cc. A programmer can explicitly construct the continuation of a computation and pass it directly to a function.

To illustrate this transformation, suppose that a function, Identity, is defined that returns the value it is given.

static T Identity<T>(T value)
{
    return value;
}

The return statement in Identity implicitly invokes the continuation of the call-site causing computation to leave Identity and resume at the point immediately after the invocation of Identity at the call-site. If Main contains only a call to WriteLine passing the result of the call to Identity, then computation resumes with the call to WriteLine.

static void Main()
{
    Console.WriteLine(Identity("foo"));
}

However, instead of implicitly invoking Identity's continuation, the programmer can pass the continuation to Identity explicitly. An extra argument is added to Identity which is a void-returning delegate with one parameter that is the same type as the return type of the former Identity function.

static void Main()
{
    Identity("foo", s => Console.WriteLine(s));
}

static void Identity<T>(T value, Action<T> k)
{
    k(value);
}

At the call-site, the remainder of the computation has moved into the lambda passed to Identity. The continuation is explicitly passed. On the callee's side, the return statement is replaced by the invocation of the continuation parameter, k, with the return value.

This pattern is called continuation-passing style or CPS.

Converting to CPS

Now that we know enough to be dangerous, let's see what we can do. Consider the typical Max function.

static int Max(int n, int m)
{
    if (n > m)
        return n;
    else
        return m;
}

To convert this function to CPS, follow these steps:

1. Change the return type to void

2. Add an extra argument of type Action<T> where T is the original return type

3. Replace all return statements with invocations of the new continuation argument passing the expression used in the return statement

Applying these steps to the integer version of Max, the function now returns void, takes an extra parameter, k, of type Action<int>, and invokes k everywhere a return statement appeared.

static void Max(int n, int m, Action<int> k)
{
    if (n > m)
        k(n);
    else
        k(m);
}

To convert the call-site to CPS:

1. Remove all of the remaining computation after the call-site

2. Put the remaining computation in the body of the lambda corresponding to the continuation argument

For example, suppose the user defined a method Main which invokes WriteLine with the result of applying Max to three and four.

static void Main()
{
Console.WriteLine(Max(3, 4));
}

The remaining computation after the invocation of Max is the call to WriteLine. Therefore, the call to WriteLine is moved into the lambda representing the continuation passed to Max.

static void Main()
{
Max(3, 4, x => Console.WriteLine(x));
}

The steps for converting the call-site skirted the issue of what to do with return statements in the continuation. For example, suppose there are three methods Main, F, and G where Main calls F and F calls G. To convert F and G to CPS, follow the same transformation steps as with Max.

static void Main()
{
    Console.WriteLine(F(1) + 1);
}

static int F(int n)
{
    return G(n + 1) + 1;
}

static int G(int n)
{
    return n + 1;
}

However, what should the transformation do with the return statement in F? The continuation passed to G should definitely not attempt to return a result because the continuation is a void-returning delegate.

static void F(int n, Action<int> k)
{
    G(n + 1, x => { return x + 1; });  // error, Action<int> cannot return a value
}

That wouldn't make any sense. Delegates with void return-types cannot return values. Furthermore, the code completely ignores k.

Instead, the return statement should be transformed into an invocation of k.

static void Main()
{
    F(1, x => Console.WriteLine(x + 1));
}

static void F(int n, Action<int> k)
{
    G(n + 1, x => k(x + 1));
}

static void G(int n, Action<int> k)
{
    k(n + 1);
}

This is how functions that use explicit continuations are composed together.

What about recursive functions like Factorial?

static void Main()
{
    Console.WriteLine(Factorial(5));
}

static int Factorial(int n)
{
    if (n == 0)
        return 1;
    else
        return n * Factorial(n - 1);
}

Recursive functions can be transformed just as easily following the same steps.

static void Main()
{
    Factorial(5, x => Console.WriteLine(x));
}

static void Factorial(int n, Action<int> k)
{
    if (n == 0)
        k(1);
    else
        Factorial(n - 1, x => k(n * x));
}

CPS turns a program inside-out. In the process, the programmer may feel that his brain has been turned inside-out as well.

Why CPS?

What exactly can a programmer do with CPS besides showoff at parties?

Compilers use a more thorough CPS transformation to produce an intermediate form amenable to many analyses. UI frameworks use CPS to keep the UI responsive while allowing nonlinear program interaction. Web servers use CPS to allow computation to flow asynchronously across pages.

Most programmers have used functions which take a callback. Often, the callback is the code that is invoked upon completion of the function. In these cases, the callback is an explicitly-passed continuation.

Asynchronous Calls

Hiding network latency requires asynchronous calls. In the first technology preview, Volta allows programmers to add asynchronous versions of methods on tier boundaries.

[RunAtOrigin]
static class C
{
    public static int F(string s)
    {
        return s != null ? s.Length : 0;
    }
}

To make a method asynchronous, define the CPS-equivalent method signature and annotate it with the Async attribute. Volta will generate the body and modify the call-sites accordingly.

[Async]
public static void F(string s, Action<int> k);

If the programmer invokes an asynchronous method F, then Volta will launch the invocation on another thread and invoke the continuation upon completion of the call to F.

C.F("foo", x => Console.WriteLine(x));

Wrapping it Up

Continuations are powerful. CPS gives programmers a way to use continuations in their day-to-day work. Perhaps, someday soon we'll discuss how to make CPS more palatable, but we will need to discuss the "M" word first.

Immutability, Purity, and Referential Transparency

wesdyer — Thu, 01 Mar 2007 10:44:12 GMT

How often do you write code that just works? It seems to happen so rarely that I find myself suspicious if code does just work. When was the last time that you wrote a non-trivial program that had no compile errors on the first try and then ran fine as well? If it happened recently then congratulations. If it hasn't then join the club. Admittedly, most programmers don't just write a whole program and then submit it to a compiler and a test harness to see if it works. Typically it is built incrementally: a function here, a class there, testing all the time. Code-Compile-Test...rinse and repeat.

I recently read a post where the author describes why Haskell programs just seem to work. It got me thinking about my own experience in Haskell. One of the most interesting things about Haskell is that many programmers don't use a debugger (GHCi does have one). It seems that the impulse to break out the debugger to better understand code occurs far less than in other languages.

Can you imagine working in C# without a debugger? So why is it that many Haskell users seem to get along without a debugger?

A Tale of Two Sorts

Consider writing a function that sorts an IEnumerable<T> using the quicksort algorithm. Here is one possible implementation:

static IEnumerable<T> QuickSort<T>(this IEnumerable<T> sequence) where T : IComparable<T>
{
var array = sequence.ToArray();
QuickSort(array, 0, array.Length - 1);
return array;
}
static void QuickSort<T>(this T[] array, int start, int end) where T : IComparable<T>
{
if (end - start < 1)
 return;
var pivot = array[start];
var j = start;
for (var i = start + 1; i <= end; ++i)
{
 if (array[i].CompareTo(pivot) < 0)
 {
 ++j;
 var temp = array[j];
 array[j] = array[i];
 array[i] = temp;
 }
}
array[start] = array[j];
array[j] = pivot;
QuickSort(array, start, j - 1);
QuickSort(array, j + 1, end);
}

Now imagine that ++j was in the wrong place (at the end of the block). The output of sorting

4, 3, 6, 9, 1, 0, 2, 7, 5, 8

would be

4, 4, 4, 4, 4, 9, 6, 7, 7, 8

Clearly this isn't right. So after a quick scan of the code, most programmers would attach a debugger to see what is going on.

The programmer might then step through the code until some state transition ends in a corrupt state. Hopefully the programmer understands the intent of the code so that a fix can be issued to bring the implementation in line with the intent.

Contrast the previous definition of quicksort with the following definition:

static IEnumerable<T> QuickSort<T>(this IEnumerable<T> sequence) where T : IComparable<T>
{
if (!sequence.Any())
 return sequence;
var pivot = sequence.First();
var remaining = sequence.Skip(1);
 return ((from x in remaining where x.CompareTo(pivot) < 0 select x).QuickSort())
 .Concat(pivot)
 .Concat((from x in remaining where x.CompareTo(pivot) >= 0 select x).QuickSort());
}

What do you notice that is different besides the number of lines of code?

The first definition creates an array and then does in an place quicksort on the array, destructively updating the contents. It creates several temporary values (i, j) needed to store intermediate results. If a programmer hasn't written the algorithm recently then chances are that the code will contain at least an one off by one error or some other similar error. I've seen too many people do poorly on whiteboards on similar problems (where incidentally there is no debugger, syntax highlighting, or intellisense) to believe that most people would just get it right the first time.

The second definition does not modify any data. Sure it makes assignments but each of these assignments are to a newly introduced variable. These variables are really just aliases for the expressions that are assigned to them because the expressions are side-effect free. Furthermore, the second definition is essentially just a brief description of what the quicksort is rather than a step by step description of how it is accomplished.

At this point, some people will probably be thinking, "But isn't the first approach so much faster than the second approach?"

Maybe, maybe not. Perhaps it doesn't even matter.

I think it is important to use the following principle: Make it correct, make it clear, make it concise, make it fast. In that order.

If I find that the code in question is not performant enough given the user scenarios that we need to support or if the code is overly complex for the sake of eliminating side-effects then I consider adding state. But note that if I wanted to introduce state to our quicksort, I could do it in two different ways: contained within the quicksort function or exposed to the caller of quicksort. This is somewhat analogous to calling either the first function or the second function in the first definition of quicksort. If the state is exposed then quicksort could do an in-place sort on the collection passed to the quicksort function. Alternatively, quicksort could simply return a new collection that is sorted and not update the original collection. This leads to the second principle:

Write side-effect free code. If side-effects are required then confine them to the smallest scope possible.

Most programmers have been following this to some extent for years. We all know that generally it is not a good idea to use global variables. This is basically the extreme of exposing side-effects (the global scope). Unfortunately, many of the programmers who don't use global variables don't realize that the same principles apply to fields, properties, parameters, and variables on a more limited scale: don't mutate them unless you have a good reason.

A Little Story about State

Some time ago, I wrote a post about a way to save the state of an IEnumerator<T>. What I didn't tell is the story behind the post.

I was writing a managed lexer and parser generator. The system is pretty neat and includes things like GLR parsing, automatic inference of the AST types, automatic error recovery, and whole slew of other features. The parser takes an IEnumerable<IToken> in order to decouple the lexer from the parser. However, for many of the features of the parser, it needs a way to save the state the of the underlying token stream and possibly restore it at a later time after more of the stream has been consumed.

Originally, I wrote a MarkableEnumerator<T> which wrapped an IEnumerator<T> and consumed the input. Clients could request a mark which they could then restore at a later point. If they did this then the MarkableEnumerator<T> would continue at the restored point when MoveNext was called again. This little piece of software (about 100 lines of code) caused Cyrus and me lots of trouble. It was super complicated because it was heavily optimized to keep its internal storage as small as possible while preserving requested past state (and possibly restoring it). It used a cyclical buffer and a number of opaque handles. One of the worst problems is that it was a somewhat leaky abstraction since the complexity leaked into the code consuming it.

When the code was first introduced it was reasonably contained but over time its inherent complexity began to pervade the whole parser. Eventually, it was hard to make improvements to the parser without breaking the invariants of the MarkableEnumerator. It was an absolute horror. I remember one night well past midnight, Cyrus and I were talking about how much we hated the thing, but we needed it to provide good performance.

Then it occurred to us that we could replace the whole mess of opaque handles, circular buffers, mark and release code, and MarkableEnumerators within MarkableEnumerators with just a linked list that wrapped the enumerator. If someone wanted to save the state of the enumerator and resume it then all they need to do is keep a reference to the desired node. To all of the clients, it looked just like an immutable linked list. That is it. So much simpler and guess what? It was almost as fast. But even more importantly, because of its simplicity we were able to make even more important code faster. Our development time for new features rocketed again.

There are several lessons from this story, but one of them is perhaps the most important considering our current discussion. Reducing the number and scope of side-effects simplifies the code.

Immutable Binary Trees

Mutation often seems to just cause problems.

Suppose that we want to create a BinaryTree class. One straightforward implementation is

class BinaryTree<T>
{
public T Value { get; set; } // yes, we have autoimplemented
public BinaryTree<T> Parent { get; set; } // properties in case you haven't
public BinaryTree<T> Left { get; set; } // heard
public BinaryTree<T> Right { get; set; }
}

I don't like it very much. Those sets give me an uneasy feeling. Of course it depends on how the objects are used, but suppose that we intend to use the binary tree in a multithreaded environment. If someone changes any of the members of a node there is a chance that some other thread already has a reference to a node in the tree and will then have a tree in an inconsistent state. Yes, we could use a lock to fix the problem but I tend to like a different approach to this problem.

class BinaryTree<T>
{
T value;
BinaryTree<T> left;
BinaryTree<T> right;
public BinaryTree(T value, BinaryTree<T> left, BinaryTree<T> right)
{
this.value = value;
this.left = left;
this.right = right;
}
public T Value { get { return value; } }
public BinaryTree<T> Left { get { return left; } }
public BinaryTree<T> Right { get { return right; } }
}

This is essentially the same as the first BinaryTree except that it is immutable (no sets and no methods causing mutation). This requires that we have a constructor that takes in the appropriate values. Also, you may have noticed that there is no Parent property. The reason is that if the node is immutable then it is hard to know what the parent is at the time of creation. With an immutable tree, you generally can either create parents before children or children before parents. I prefer the later (despite the biological conundrum posed by this). Note that this doesn't mean that we can't know what the parent of a given node is. Indeed we can, but we need to introduce a new concept before addressing this.

Nodes can really be parented by n nodes. Since any node can be constructed with a given node as either the left or the right child. But given a tree (a root), we can enforce (assuming that we don't want DAGs) that a node has at most one parent. We will call this a BinaryTreeVersion.

class BinaryTreeVersion<T>
{
BinaryTree<T> tree;
Dictionary<BinaryTree<T>, BinaryTree<T>> parentMap;
public BinaryTreeVersion(BinaryTree<T> tree)
{
 this.tree = tree;
}
public BinaryTree<T> Tree { get { return tree; } }
public BinaryTree<T> GetParentOf(BinaryTree<T> node)
{
 if (parentMap == null)
 {
 parentMap = new Dictionary<BinaryTree<T>, BinaryTree<T>>();
 FillParentMap(tree, null);
 }
 return parentMap[node];
}
void FillParentMap(BinaryTree<T> tree, BinaryTree<T> parent)
{
 if (tree == null)
 return;
 parentMap.Add(tree, parent);
 FillParentMap(tree.Left, tree);
 FillParentMap(tree.Right, tree);
}
}

The parent operation is exposed on the BinaryTreeVersion. The parent of a given node is stored in the parentMap which is filled with node-parent pairs. So now we have a tree that is immutable and the ability to access a node's parent.

But we can take it one step further, we can create functions that will modify the tree. Note that we will not actually modify the tree but we will return a new BinaryTreeVersion. The ChangeNodeValues function takes two functions: a function that determines whether to modify the given node's value and a function to compute the node's new value (there are better ways to do this for specific modifications and yes I could have used a visitor).

class BinaryTreeVersion<T>
{

...
public BinaryTreeVersion<T> ChangeNodeValues(Func<BinaryTree<T>, bool> predicate, Func<BinaryTree<T>, T> newValue)
{
 return new BinaryTreeVersion<T>(ChangeNodeValue(tree, predicate, newValue));
}
BinaryTree<T> ChangeNodeValue(BinaryTree<T> node, Func<BinaryTree<T>, bool> predicate, Func<BinaryTree<T>, T> newValue)
{
 if (node == null)
 return null;
 var left = ChangeNodeValue(node.Left, predicate, newValue);
 var right = ChangeNodeValue(node.Right, predicate, newValue);
 var value = predicate(node) ? newValue(node) : node.Value;
 if (value.Equals(node.Value) && left == node.Left && right == node.Right)
 return node;
 return new BinaryTree<T>(value, left, right);
}
}

The neat thing is that since we use immutable trees, we can share subtrees that are not changed. Even so, if another thread has a reference to the first BinaryTreeVersion or a node in the first tree then things will just work because we are mucking around with the data.

The Benefits of Purity

One of the most painful things in software development is integrating two or more things (objects, assemblies, functions, ...) together. Invariably something will go wrong and a programmer that worked on one or the other things in question will protest, "It worked on my machine!" Yes, indeed it probably did and it probably even works by itself but when put together with other things it doesn't play so nice.

One way to increase the reliability of a unit is to eliminate the side-effects. This makes composing and integrating units together much easier and more robust. Since they are side-effect free, they always work the same no matter the environment. This is called referential transparency.

Conversely, things that are side-effect free are also easier to divide. For a good example of this take a look at PLinq which farms out querying work to the available threads. It divides up the work and then integrates the results: map-reduce.

But that is a discussion for another day...

The Virtues of Laziness

wesdyer — Wed, 14 Feb 2007 00:11:57 GMT

It seems that I riled some people up with my blog post yesterday. After some thought, I think the primary reason that there was some backlash is because some people feel that I violated one of the sacred principles of FP: lists are *the* data structure. Well, let me set the matter straight. I love lists. Especially the singlely linked immutable kind that are found in many functional languages. Furthermore, Microsoft is not trying to launch some massive marketing campaign to sell IEnumerable<T> as the *new* list. So to be clear, I talked about IEnumerable<T> yesterday because people who have used functional languages will wonder where are the lists in C# 3.0. IEnumerable<T> is not the same as a list, but there are many similarities between IEnumerable<T> and lazy lists that I pointed out yesterday when I showed they are isomorphic by describing the bijective mapping between them.

Furthermore, it is the case that some data structures are generally better than others. But there are tradeoffs between using the various data structures. Also, *not* all of the tradeoffs are captured by looking exclusively at their time complexity for some operation. There is of course the space complexity and there is the complexity of implementation itself. And don't forget that often they have different characteristics for different operations. The key point here is that a given problem will have a number of constraints and each of the competing designs has a number of trade-offs. As an interviewer, when I notice a candidate is not aware of the trade-offs that he is making then I start to worry that they were not considered at all. Furthermore, when I notice that a candidate seems to favor one data structure to the exclusion of others, I start to wonder how many tools are in the toolbox. But after observing this behavior many times, I am convinced that it often is not the coding problem that is driving usage of some data structure but the mode of thinking that the candidate employs.

One more thing before I continue on. At least one reader wondered why I said the following:

"Where most programmers who are accustomed to imperative style would naturally use an array, a variable, or a mutable object, a functional programmer will often use a list."

Yes, I meant to say exactly what I said. In fact, when I wrote it, I paused before deciding to include it because I thought it might be misunderstood. When I first read SICP, the most mind bending and rewarding topic was at the end of chapter 3. The section on streams. One of the motivations that was given for using a stream (infinite list) was that variables that change their value over time cause problems. One way to address this was to have streams represent the state of the variable over time where each element of the stream represents a state of the variable at some point in time.

So without further ado, let's take a look at streams...

What we want is an infinite list. The problem is that infinite list can never actually be fully realized because of their infinite nature. So if we attempt to realize an infinite list either the computer will enter an infinite loop or it will run out of resources (stack overflow, out of memory, etc.).

We can overcome this problem by having lazy lists. Lists where the next element in the list is not realized until it is needed. Yesterday, I presented one such lazy list which uses an enumerator to realize the next element. Today, I present another which has a more intriguing definition.

class Stream<T> : IList<T>
{
Func<IList<T>> next;
T value;
public Stream(T value, Func<IList<T>> next)
{
this.value = value;
this.next = next;
}
public IList<T> Next { get { return next(); } }
public T Value { get { return value; } }
}

This lazy list is very similar to a normal list. The only difference is that instead of taking a list as the value of the next node in the list, it takes a function which will evaluate to the next node in the list. But this difference is critical.

The first difference can easily be seen by imaging a list type, ErrorList, that when constructed throws an exception.

class ErrorList<T> : IList<T>
{
public ErrorList() { throw new Exception(); }
public IList<T> Next { get { throw new Exception(); } }
public T Value { get { throw new Exception(); } }
}

Now consider the following code:

var list = new List<int>(1, new ErrorList<int>()); // error, exception thrown
var stream = new Stream<int>(1, () => new ErrorList<int>()); // no error

An exception is thrown when the list is constructed but when the stream is constructed no exception is thrown. Unless the Next property is evaluated on the stream, there will never be an exception.

The second difference can be seen in the following code:

IList<BigInteger> onesList = null;
onesList = new List<BigInteger>(1, onesList);
IList<BigInteger> onesStream = null;
onesStream = new Stream<BigInteger>(1, () => onesStream);

If you try to list all of the values in onesList, then you will notice that it only contains one value. Whereas onesStream contains an infinite number of values. The reason is that when we constructed onesList, we passed in onesList but onesList had the value null at the time so the next node was set to null. But in the stream case we passed in a function that would be evaluated sometime in the future. By the time that we evaluate it, it will return the proper value of onesStream.

A third difference is found in the performance of the two lists. With the lazy lists, parts of the list that are never realized are never paid for. So it is a pay as you go model as opposed to pay everything up front. Furthermore, less space can be required since the whole list is not necessarily held in memory at once but only a process decription that can compute each element as it is required.

So now we have this infinite stream of ones, but can we do anything more interesting? Sure. First, let's define a zip function between lists.

public static IList<V> Zip<T,U,V>(Func<T,U,V> f, IList<T> list1, IList list2)
{
 if (list1 == null || list2 == null)
 return null;
 return new Stream<V>(f(list1.Value, list2.Value), () => Zip<T,U,V>(f, list1.Next, list2.Next));
}

Yes, that is somewhat of a mouthful. Here is what it does. It takes two lists where the lists may differ from each other in what kind of elements they contain. It also takes a function from the type of the first list's elements and the type of the second list's elements and returns possibly a new type. Then if either of the lists is empty we return an empty list (null). But if both lists are non-empty then we return a stream where the first element is the application of the given function to the first element of both of the lists and the rest of the list will be the evaluation of Zip on the rest of the two lists. It is important that Zip uses a stream because these lists may be infinite and if we try to immediately evaluate the entire list we will run out of resources.

Now that we have Zip, let's put it to use to define the natural numbers.

IList<BigInteger> ones = null;
ones = new Stream<BigInteger>(1, () => ones);
IList<BigInteger> natural = null;
natural = new Stream<BigInteger>(0, () => Zip((x, y) => x + y, natural, ones));

So what we are saying is that the natural numbers begin with zero and then are followed by the sum of the first natural number with the first element of ones (0 + 1). The second natural number is the sum of the second element of natural numbers with the second element of ones (1 + 1) and so on. This works because each element of natural numbers is defined only in terms of the elements of natural numbers that occur previous to it.

So now we can easily define the odd numbers (2k + 1) and even numbers (2k). But first we need a map function for lists.

public static IList Map<T, U>(Func<T, U> f, IList<T> list)
{
if (list == null)
return null;
return new Stream(f(list.Value), () => Map(f, list.Next));
}

Now here are the definitions of odd and even numbers.

IList<BigInteger> odds = Map(x => 2 * x + 1, natural);
IList<BigInteger> evens = Map(x => 2 * x, natural);

We can also define the fibonacci sequence as a stream.

IList<BigInteger> fibs = null;
fibs = new List<BigInteger>(0, new Stream<BigInteger>(1, () => Zip(add, fibs, fibs.Next)));

What we are saying here is that the first fibonacci number is zero and the second is one but then the next one is the sum of the first number and the second number and so on. This is similar to the natural numbers definition which used itself to compute itself.

If you try it out, you will also notice that it isn't very efficient. This is because we are back to our exponential time complexity. But this can easily be remedied by memoizing the next function in the constructor to the stream:

public Stream(T value, Func<IList<T>> next)
{
this.value = value;
this.next = next.Memoize();
}

Now it has linear time complexity by trading off some space (linear as well).

Let's finish by solving the famous problem of producing the Hamming numbers. The problem is to list all of the positive integers who have no prime factors other than 2, 3, or 5 in ascending order. The first ten Hamming numbers are:

1, 2, 3, 4, 5, 6, 8, 9, 10, 12

This problem is notoriously difficult without lazy evaluation and is used to demonstrate the power of laziness.

To solve this problem first note that the first hamming number is 1. Then if h is a hamming number so is 2h, 3h, and 5h. So we can define three streams which map the hamming numbers to 2h, 3h, and 5h respectively. The only remaining requirement is that they must be in order. We can maintain this invariant by defining a function named Merge:

public static IList<T> Merge<T>(IList<T> list1, IList<T> list2) where T : IComparable<T>
{
if (list1 == null)
 return list2;
else if (list2 == null)
 return list1;
int c = list1.Value.CompareTo(list2.Value);
if (c < 0)
 return new Stream<T>(list1.Value, () => Merge(list1.Next, list2));
else if (c > 0)
 return new Stream<T>(list2.Value, () => Merge(list1, list2.Next));
else
 return new Stream<T>(list1.Value, () => Merge(list1.Next, list2.Next));
}

Now we are ready to define the Hamming numbers. Notice how close the definition in the code is to our description:

IList<BigInteger> hamming = null;
hamming = new Stream<BigInteger>(1, () => Merge(Map(x => x * 2, hamming), Merge(Map(x => x * 3, hamming), Map(x => x * 5, hamming))));

Now for fun, try to think of how to do it without lazy evaluation.

Why all the love for lists?

wesdyer — Mon, 12 Feb 2007 22:41:00 GMT

One of the things that I have noticed when participating in interviews with potential candidates is that most candidates have a favorite data structure. Presumably this is the data structure that they feel the most comfortable with. The problem is that no matter what coding question I ask them, they see the answer through the lenses of their favorite data structure. I have met programmers who are in love with hash tables, others who have a certain fondness for B-trees, and quite a few who adore arrays. My feeling is that this happens not because they understand the data structure itself better than other data structures (honestly, I can't believe that the guy who loves b-trees doesn't understand what arrays are), but it is because each data structure encourages a certain way of thinking about problems. So really the candidates are just thinking about the question in a certain way that is expressed on the outside as using some particular data structure.

One of the first things that a newcomer to functional programming will notice is that lists are all the rage. Where most programmers who are accustomed to imperative style would naturally use an array, a variable, or a mutable object, a functional programmer will often use a list. Why is this?

If we consider the insight that it is not the data structure itself but rather the mode of thinking that encourages the usage of the data structure then we can make some progress on understanding this. Lists are the simplest recursive data structure. A list is just a node consisting of a value and the next node in the list. The definition of a list is defined in terms of itself.

interface IList<T>
{
IList<T> Next { get; }
T Value { get; }
}

But notice how similar this way of thinking is to recursive functions. For example a very common function defined over lists is Length. This function computes the length of a given list.

public static int Length<T>(this IList<T> list)
{
if (list == null)
return 0;
return 1 + Length(list.Next);
}

Length is a function that is defined in terms of itself. Like the definition of list, Length has a value (either the 0 or the 1 +) and it has a next (the recursive call to Length(list.Next)). So one reason why lists are so popular in functional programming is that they encourage a recursive way of thinking about problems.

In C# 3.0, lists are not so commonly used as IEnumerable<T>. In fact, IEnumerable<T> is used so much that it sticks out in the same way the usage of lists in functional programming languages sticks out. The primary Linq provider that will be shipped is Linq to Objects which requires that the source of a query implement IEnumerable<T>.

In some senses, IEnumerable<T> is not so very different from the definition of IList<T> shown above. Both of them represent sequences of elements where only the current value and the next element are available at a given time. In fact, it is relatively easy to define one in terms of the other.

Here is a definition of IEnumerable<T> in terms of IList<T>.

public static IEnumerable<T> GetEnumerator<T>(this IList<T> list)
{
for (var current = list; current != null; current = current.Next)
yield return current.Value;
}

And here is a corresponding definition of IList<T> in terms of IEnumerable<T> (take from this post).

class LazyList<T> : IList<T>
{
IList<T> next;
T value;
IEnumerator<T> enumerator;
 LazyList(T value, IEnumerator<T> enumerator)
{
 this.value = value;
 this.enumerator = enumerator;
}
public static IList<T> Create(IEnumerable<T> enumerable)
{
 return Create(enumerable.GetEnumerator());
}
public static IList<T> Create(IEnumerator<T> enumerator)
{
 if (enumerator.MoveNext())
 return new LazyList<T>(enumerator.Current, enumerator);
 return null;
}
public IList<T> Next
{
 get
 {
 if (enumerator != null)
 {
 next = Create(enumerator);
 enumerator = null;
 }
 return next;
 }
}
public T Value { get { return value; } }
}

But it should be noted that there is a critical difference between both of these definitions and the following definition of List<T>.

class List<T> : IList<T>
{
T value;
IList<T> next;
public List(T value, IList<T> next)
{
this.value = value;
this.next = next;
}
public IList<T> Next { get { return next; } }
public T Value { get { return value; } }
public static IList<T> Empty { get { return null; } }
}

Did you notice what it is? The previous definitions were lazy but this definition is not. The iterator did not describe the data itself but rather a process for sequencing data that will be executed possibly sometime in the future. The LazyList also does not actually realize an entire list. It describes a way to construct the next element of a list on demand. But the definition of List does not defer anything to the future. Now the question is, does it matter?

Memoization and Anonymous Recursion

wesdyer — Mon, 05 Feb 2007 21:12:19 GMT

Keith Farmer brought it to my attention that there is at least a little confusion about how closures work. Hopefully, I can help shed a little light on the subject. The question is why doesn't the following code actually memoize fib in the call to Test?

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;
Test(fib.Memoize());

Before I explain why this code doesn't work, I want to return to the example that I used in my last post.

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;
Func<int, int> fibCopy = fib;
Console.WriteLine(fib(6)); // displays 8
Console.WriteLine(fibCopy(6)); // displays 8
fib = n => n * 2;
Console.WriteLine(fib(6)); // displays 12
Console.WriteLine(fibCopy(6)); // displays 18

Probably the easiest way to see why this code behaves so strangely is to show what code the C# compiler generates. This can easily be done with ildasm.exe or reflector. There are two lambdas defined in the code. The first one captures the local variable named fib and the second does not capture any locals. Because fib is capture by a lambda, it must be hoisted on to the heap. So a class is declared that contains the captured local.

class DisplayClass
{
public Func<int, int> fib;
}

Now the compiler must emit a method that corresponds to the first lambda. This method will need access to the members of DisplayClass and must also be convertible to a Func<int,int> delegate. Today the compiler achieves this by emitting the method on the display class as an instance method (we could also have used currying but that is a story for another day).

class DisplayClass
{
public Func<int, int> fib;
public int M1(int n)
{
return n > 1 ? fib(n - 1) + fib(n - 2) : n; // <-- 1st lambda body
}
}

The second lambda does not capture any local variables. So this method can be emitted as a static method.

static int M2(int n)
{
return n * 2;
}

Finally, we are ready to show the emitted code for the original code fragment.

DisplayClass locals = new DisplayClass();
locals.fib = null;
locals.fib = locals.M1;
Func<int, int> fibCopy = locals.fib;
Console.WriteLine(locals.fib(6)); // displays 8
Console.WriteLine(fibCopy(6)); // displays 8
locals.fib = M2;
Console.WriteLine(locals.fib(6)); // displays 12
Console.WriteLine(fibCopy(6)); // displays 18

Notice how the first call to fib is really just a call to M1 and when M1 "recurses" on fib it just ends up calling M1 again because that is what is assigned to fib. The call to fibCopy is a little trickier because the original call is really a call to M1 as well but when it "recurses" it invokes fib instead of fibCopy which also happens to be M1 at the time. So the first two calls behave as expected.

Now it starts to get a little strange. First we assign M2 to fib. Then when we invoke fib on the next line it doesn't invoke our original "fib" function M1 anymore but it not invokes M2. This of course displays the result of multiplying 6 by 2.

Now for the strangest part, we invoke fibCopy. But fibCopy actually still references M1 and not M2 and since 6 > 1 then it "recurses" by invoking fib twice with 5 and 4 respectively and then summing the results. But the calls to fib actually invoke M2 now. So the results of the calls to fib are 10 and 8 which then are summed to produce 18.

Now let's return to our original problem.

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;
Test(fib.Memoize());

Notice that the function passed to Test is memoized but the body of the function still calls the unmemoized fib. The function itself is memoized by all of the "recursive" calls are not. So it probably will not behave as intended. This can be corrected by doing the following.

Func<int, int> fib = null;
fib = Memoize(n => n > 1 ? fib(n - 1) + fib(n - 2) : n);
Test(fib);

Now the calls to fib when n > 1 are made to the memoized fib delegate.

If you read my last post on anonymous recursion in C# which introduces the Y combinator then you may have tried the following.

Func<int, int> fib = Y<int, int>(f => n => n > 1 ? f(n - 1) + f(n - 2) : n).Memoize();

This behaves very similarly to the incorrectly written fib above. Calls to the function itself are memoized but all of the recursive calls are not. This is because fib is memoized but f is not. Erik Meijer pointed me to a fantastic paper that discusses how to memoize functions that are the fixed point of functionals. In this paper, Cook and Launchbury present a function written in Haskell that enables memoized anonymous recursive functions.

memoFix f = let g = f h
h = memo g
in h

Using the MemoizeFix function in C# looks very similar to using the Y combinator, but instead of just returning a recursive function, it would also memoize the function and all the recursive calls.

Func<int, int> memoFib = MemoizeFix<int, int>(f => n => n > 1 ? f(n - 1) + f(n - 2) : n);

But there are two problems with implementing the function in C#. First, Haskell lets programmers write mutually recursive definitions (g is defined in terms of h and h is defined in terms g). Second, Haskell is a lazy language. A straightforward implementation in C# will either produce a null reference exception or a stack-overflow because of the eager evaluation and mutually recursive definitions. Fortunately, lambdas can be used to solve both problems.

static Func<A, R> MemoizeFix<A, R>(this Func<Func<A, R>, Func<A, R>> f)
{
Func<A, R> g = null;
Func<A, R> h = null;
g = a => f(h)(a);
h = g.Memoize();
return h;
}

Here we define both g and h before they are assigned their final values. Then instead of assigning f(h) to g which would immediately invoke f causing a Null Reference exception, we assign a lambda to g which will be evaluated sometime in the future when h has a non-null value. The rest of the function is very similar to the Haskell version.

Notice what happens when memoFib is invoked. This will actually invoke h which was defined in MemoizeFix, but h is really just a memoized version of g. Since this is the first invocation of h and there will be no precomputed value and so g will be invoked. But when g is invoked then it actually invokes f passing h (memoized g) as the function to use for recursion. This is exactly what we want. Beautiful isn't it?

Anonymous Recursion in C#

wesdyer — Fri, 02 Feb 2007 22:17:00 GMT

Recursion is beautiful and lambdas are the ultimate abstraction. But how can they be used together? Lambdas are anonymous functions and recursion requires names. Let's try to define a lambda that computes the nth fibonacci number.

Func<int, int> fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;

But this doesn't work. The compiler will complain about the use of fib in the lambda.

Use of unassigned local variable 'fib'

Eric Lippert has a great blog post on this issue. The problem is that the right hand side is evaluated before fib is definitely assigned. In this case, the compiler could potentially deduce (if the language spec allowed it) that fib is not used before it is definitely assigned, but in other cases it might need to be used before fib is assigned.

A quick workaround is to assign the value null to fib and then assign the lambda to fib. This causes fib to be definitely assigned before it is used.

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;
Console.WriteLine(fib(6)); // displays 8

In fact, letrec in Scheme does something very similar.

But our C# workaround doesn't really use recursion. Recursion requires that a function calls itself. The fib function really just invokes the delegate that the local variable fib references. It may seem that this is just nit picking about words, but there is a difference. For example, consider the following code:

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;
Func<int, int> fibCopy = fib;
Console.WriteLine(fib(6)); // displays 8
Console.WriteLine(fibCopy(6)); // displays 8
fib = n => n * 2;
Console.WriteLine(fib(6)); // displays 12
Console.WriteLine(fibCopy(6)); // displays 18

Huh!? Notice how the result of calling fib changes and that the result of calling fibCopy differs even from the result of calling fib! (See if you can figure out why)

We can stop this kind of craziness by passing in the function that will be used for the recursive call. So the lambda looks the same except that it takes an extra parameter f that is called instead of fib. When a call to f is made, we need to pass f to itself as the first argument.

(f, n) => n > 1 ? f(f,n - 1) + f(f,n - 2) : n

Unfortunately, not all is done yet. In order to convert the lambda to a delegate, we need a delegate type. Although it is clear what type fib should be, it may not be apparent what should be the type of our new delegate. So let's start with the type of fib, Func<int,int>. We do know that the return type of the delegate should be the same, so it must be int. We also know that the second argument's type should be the same which is also int. As for the type of the first argument, we will be passing in a delegate which will then be called with the same arguments as the delegate we are defining. But wait! That is recursion. We are defining the delegate type in terms of itself. Therefore, the first parameter will be of the same type that the delegate we are defining is.

delegate int Recursive(Recursive r, int n);

The delegate can be generalized by parameterizing the type of the first argument and the return type.

delegate R Recursive<A,R>(Recursive<A,R> r, A a);

Now we can use the lambda that we defined.

Recursive<int, int> fib = (f, n) => n > 1 ? f(f,n - 1) + f(f,n - 2) : n;
Console.WriteLine(fib(fib,6)); // displays 8

While this is an improvement, neither the lambda nor its application look as nice as the original code. The fact that the first argument should be the delegate itself seems a little strange. The lambda can be curried in order to separate the passing of f to f from the mechanics of the fib function. The innermost lambda will have the same signature as the original fib function.

f => n => n > 1 ? f(f)(n - 1) + f(f)(n - 2) : n

Now that the lambda has been curried, a new definition for the Recursive delegate is required. It now only takes one argument which is the first argument of the previous definition, but the return type changes. Recursive returns a function (the effect of currying it) which takes the second parameter of the original definition and returns the original return type.

delegate Func<A,R> Recursive<A,R>(Recursive<A,R> r);

Once the lambda is assigned to a Recursive delegate then we can apply the delegate to itself to get the underlying function. Furthermore, if we made a copy of the delegate and then changed the original then none of the strange effects that happened early would occur.

Recursive<int, int> fibRec = f => n => n > 1 ? f(f)(n - 1) + f(f)(n - 2) : n;
Func<int, int> fib = fibRec(fibRec);
Console.WriteLine(fib(6)); // displays 8

It is now possible to clean-up the duplicated self-application of f. We can get rid of it by creating a lambda inside of the fibRec lambda which contains the essential fibonacci definition but which takes an argument that references what function should be called for recursion. Then, inside of fibRec, we can do the self-application of f.

Recursive<int, int> fibRec = f => n =>
{
Func<Func<int, int>, int, int> g = (h, m) => m > 1 ? h(m - 1) + h(m - 2) : m;
return g(f(f), n);
};
Func<int, int> fib = fibRec(fibRec);
Console.WriteLine(fib(6)); // displays 8

This inner lambda looks very similar to our original lambda which took two parameters. In fact, it appears that the whole process of construction and self-application that initially happened at the top level is now happening inside of the lambda. The only difference is that the inner lambda does not pass a reference to itself around. This has now be abstracted out in the call to g.

We can simplify the fibRec definition by currying the inner lambda.

Recursive<int, int> fibRec = f => n =>
{
Func<Func<int, int>, Func<int, int>> g = h => m => m > 1 ? h(m - 1) + h(m - 2) : m;
return g(f(f))(n);
};
Func<int, int> fib = fibRec(fibRec);
Console.WriteLine(fib(6)); // displays 8

Note that the definition of g does not depend on f or n at all. Therefore, we can move it outside of the outer lambda.

Func<Func<int, int>, Func<int, int>> g = h => m => m > 1 ? h(m - 1) + h(m - 2) : m;
Recursive<int, int> fibRec = f => n => g(f(f))(n);
Func<int, int> fib = fibRec(fibRec);
Console.WriteLine(fib(6)); // displays 8

Notice in the above code that g now represents our original concept of the fibonacci function while fibRec does all of the handy work to enable anonymous recursion. The whole process of building of fibRec and then applying it to itself only requires a reference to g. So let's move the definition of fibRec and fib to a different function named CreateFib.

static Func<int, int> CreateFib(Func<Func<int, int>, Func<int, int>> g)
{
Recursive<A, R> fibRec = f => n => g(f(f))(n);
return fibRec(fibRec);
}

We can now call CreateFib instead of creating fibRec.

Func<Func<int, int>, Func<int, int>> g = h => m => m > 1 ? h(m - 1) + h(m - 2) : m;
Func<int, int> fib =CreateFib(g);
Console.WriteLine(fib(6)); // displays 8

But really our CreateFib function is useful for more than just creating fibonacci functions. It can create any recursive function which takes one argument. Parameterizing the types used by CreateFib leads to what is known as the Y fixed-point combinator.

static Func<A, R> Y<A, R>(Func<Func<A, R>, Func<A, R>> f)
{
Recursive<A, R> rec = r => a => f(r(r))(a);
return rec(rec);
}

We can now create any recursive lambda of one parameter that we want.

Func<int,int> fib = Y<int,int>(f => n => n > 1 ? f(n - 1) + f(n - 2) : n);
Func<int, int> fact = Y<int, int>(f => n => n > 1 ? n * f(n - 1) : 1);
Console.WriteLine(fib(6)); // displays 8
Console.WriteLine(fact(6)); // displays 720

Finally, the goal of anonymous recursion has been reached.

Currying and Partial Function Application

wesdyer — Mon, 29 Jan 2007 23:36:53 GMT

When I first heard the term Currying, I thought immediately of tasty Thai and Indian food. To my dismay, I found that the conversation was not about wonderful spices but rather about transforming a function that takes n arguments into a function that takes only one argument and returns a curried function of n - 1 arguments. Why in the world would that be useful?

From a theoretical standpoint, it is interesting because it simplifies the lambda calculus to include only those functions which have at most one argument. From a practical perspective, it allows a programmer to generate families of functions from a base function by fixing the first k arguments. It is akin to pinning up something on the wall that requires two pins. Before being pinned, the object is free to move anywhere on the surface; however, when when first pin is put in then the movement is constrained. Finally, when the second pin is put in then there is no longer any freedom of movement. Similarly, when a programmer curries a function of two arguments and applies it to the first argument then the functionality is limited by one dimension. Finally, when he applies the new function to the second argument then a particular value is computed.

What this means in C# is if I have a delegate which is of type Func<A,B,R> (delegate with two arguments of type A and B respectively and returns type R) then I can create a delegate which is of type Func<A,Func<B,R>>. Notice how the curried delegate only has one argument but it returns a delegate which takes the original function's second argument and finally returns a value.

Consider generating functions from the add function.

Func<int,int,int> add = (x,y) => x + y;

We can curry add by applying the Curry function to add.

Func<int,Func<int,int>> curriedAdd = add.Curry();

This curried add function is really a function that creates functions that add n where n is the argument to the curried add function. For example, we can create an increment function by applying the curried add function to the value one.

Func<int,int> inc = curriedAdd(1);

The increment function will now return one plus the value of its argument when invoked. We can now use our three functions to do various forms of addition.

Console.WriteLine(add(3,4));            // 7
Console.WriteLine(curriedAdd(3)(5));    // 8
Console.WriteLine(inc(2));              // 3

So how would this function Curry look? It's really pretty simple.

public static Func<A, Func<B, R>> Curry<A, B, R>(this Func<A, B, R> f)
{
return a => b => f(a, b);
}

It just takes a function of two arguments and then returns a lambda which fixes the first argument and then the second argument. Once both arguments have been provided it evaluates the original function with the arguments. It is easy to follow the same pattern and create a function Curry which curries a functions of other arities.

Let's examine what went on when we created each of the functions. First we created a function called add which looked like:

(x, y) => x + y

Once we curried add, the function became:

x => y => x + y

We created inc by calling curried add with the value 1. This essentially created the following function:

y => 1 + y

The idea of currying a function and then fixing the first n arguments of the original function can be generalized into an concept called partial function application. For instance, if we consider our add function from the previous example then we can directly create the increment from add without having to create curriedAdd first.

Func<int,int> inc = add.Partial(1);

Where the Partial function is written as:

public static Func<B, R> Partial<A, B, R>(this Func<A, B, R> f, A a)
{
return b => f(a, b);
}

Notice how the function takes a function and a value which has the same type as the first argument of the function. It then returns a function which takes the remaining arguments and then applies all of the arguments to the original function. This can be generalized into a set of functions that produce partially applied functions.

Function Memoization

wesdyer — Fri, 26 Jan 2007 04:44:00 GMT

One of my favorite pastimes is playing games. No not XBox 360, PS3, or Wii games nor other computer games, but board games, card games, and other such games. It's probably because I'm from a large family - I have 8 siblings - and we would often spend time together playing games. It is a good way to accommodate a wide variety of interests, skills, and aptitudes. Some of my favorite games are Settlers of Catan, Carcassonne, Spades, Rook, Mafia, Take Two, Stratego, Ticket to Ride, and of course Axis and Allies.

For my birthday this past year, I received an interesting board game titled RoboRally. The premise of the game is there are a number of robots in a factory and they are bored, so they begin to indulge themselves by competing in races around the factory floor. Each player takes command of a robot and seeks to win these competitions by controlling the robot. The players are dealt a number of instruction cards whereupon each player elects five cards to load into his program registers. After all the players have programmed their robots, their programs are executed somewhat simultaneously. The robots can interact with each other by shooting, pushing, or otherwise hampering each other's progress. But the real catch is that the board also interacts with the robots by way of conveyer belts, lasers, pushers, gears, pits, etc. The uncertainty caused by the sheer number of interactions coupled with the deterministic behavior of programmed robots creates pandemonium.

One day I was just itching to write some code so I thought why not implement a computer version of RoboRally. I thought that it would be particularly fun because of the number of interacting objects and the amount of state that is involved. So I began to code away on a board editor. This took a little bit of time mostly because I'm not very skilled at graphic art. Here is a screenshot from an early version of the board editor.

The code for the editor consists of three assemblies: the windows application, a game engine, and a functional programming library. This was one of the first non-trivial apps that I wrote using C# 3.0 and it was very fun to use the new language features. In fact, it was incredible because I found that many design patterns that I commonly used changed drastically or were no longer needed. Furthermore, I found that entirely new designs were possible. Today, I will touch on just one of these design patterns and the underlying principle used in it.

The game engine has an class called Board and these boards are composed of many Tiles which are each associated with a Location. Tiles can be simple or rather complex. For instance, there are blank tiles and there are move left express conveyer belt tiles. But there are also gear tiles with a normal wall on the top side and a laser wall on the left side which also have a flag on the tile. I made a design decision early on to have basic tiles and decorator tiles. So complex tiles are formed from the composition of a basic tile and any number of decorator tiles. I wrote a factory which creates either some basic tile or some decorator tile. For a first cut, I didn't worry that I created an overabundance of blank tiles even if they are all identical. Nor did I worry about the fact that all move right conveyer tiles are the same. However, later as I was refining the design I came back to the factory and wanted to improve it by reducing the number of tiles that were created to the bare minimum. A typical solution to this problem for the blank tiles would look like this:

IBlankTile blankTile;
public IBlankTile CreateBlankTile()
{
if (blankTile == null)
blankTile = new BlankTile();
return blankTile;
}

But how do we do this for conveyer tiles?

Dictionary<Direction, IConveyerTile> conveyerTiles = new Dictionary<Direction, IConveyerTile>();
public IConveyerTile CreateConveyerTile(Direction direction)
{
IConveyerTile result;
if (!conveyerTiles.TryGetValue(direction, out result))
{
result = new ConveyerTile(direction);
conveyerTiles.Add(direction, result);
}
return result;
}

Ick! It only gets even more complicated for tiles that are parameterized on more items. Furthermore, it seems that there is a lot of repetition going on. Specifically, all of the machinery to track instance existence and creation is being duplicated for each tile type. To some extent the Singleton and Multiton patterns also suffer from the same problem. There must be a better way.

In fact, there is a better way. What we really want here is a function that returns the same value each time it is applied to the same parameter values. Functional programmers will instantly recognize this as function memoization.

Memoization can be implemented as a function which takes a function as a parameter and returns a new function which when invoked the first time on some set of parameters will compute the result and when invoked with the same values will not recompute the result but will instead return the previously computed result.

Now let's see how our code would have looked if we had a static method called Memoize is a functional library called Fun (why not? functional programming is fun isn't it?).

public readonly Func<IBlankTile> CreateBlankTile = Fun.Memoize(() => new BlankTile());
public readonly Func<Direction,IConveyerTile> CreateConveyerTile = Fun.Memoize(dir => new ConveyerTile(dir));

The new code is beautiful and doesn't have all of the redundancy and complexity of the old code. This is a very practical example of the modularity of functional style code that I referred to previously.

"But wait", you say, "how exactly does the magic happen?" Well, let's take a look. First, let's examine a memoize function which takes a function of no arguments and returns a memoized version of that function.

public static Func<R> Memoize<R>(this Func<R> f)
{
R value = default(R);
bool hasValue = false;
return () =>
 {
 if (!hasValue)
 {
 hasValue = true;
 value = f();
 }
 return value;
 };
}

Memoize takes a function which has no arguments and a return type of R and returns a function with the same signature. When Memoize is called it creates two local variables value and hasValue. Memoize then returns a new function that returns value if hasValue is true otherwise it computes value by evaluating the parameter f, sets hasValue to true, and then returns value. Notice that the function that is returned from Memoize accesses hasValue, value, and f. These three variables are local to Memoize. The three variables together with a reference to the function that is created by Memoize form a closure.

So what happens when we invoke the function returned from Memoize for the first time? The variable hasValue will be set to false, so we will set hasValue to true and then apply f saving the result to value. Finally, we will return the value which is the result of f. So the memoized function will return the same thing that f would have returned on its first invocation.

But what about on subsequent invocations? When we call memoized function again, hasValue will be true so we will simply return value without recomputing it. So the memoized function will always return the same value as it did on the first invocation without recomputing the result through f.

This has subtle implications such as if f has side effects then those side effects will only occur when we apply the memoized function the first time. So we need to be careful about using it. But consider how CreateBlankTile used Memoize. When CreateBlankTile is run the first time, it will create a blank tile and then return the result, but on subsequent invocations it will continue to return the same blank tile. This is exactly the behavior that we want.

Can we do the same thing for functions that have arguments? Absolutely, let's take a look at how to Memoize functions of one argument.

public static Func<A, R> Memoize<A, R>(this Func<A, R> f)
{
var map = new Dictionary<A, R>();
return a =>
 {
 R value;
 if (map.TryGetValue(a, out value))
 return value;
 value = f(a);
 map.Add(a, value);
 return value;
 };
}

This time instead of storing the computed values in a local variable directly, we store them in a dictionary and instead of marking that we have computed the value we just check for existence in the dictionary. This can be generalized to functions of n arguments by creating composite keys of n fields that are used in the dictionary.

Consider how this Memoize function works with CreateConveyerTile. When we invoke it with some direction for the first time then it will not find that direction in the dictionary and so it will create a new conveyer tile parameterized on that direction and then store it in the dictionary. Subsequent calls with the same direction will not create new tiles but instead grab out the existing one from the dictionary. Again this is the behavior that we want. Furthermore, all of the mechanics of storing and fetching tiles are located in Memoize instead of CreateBlankTile or CreateConveyerTile.

Memoize is also useful for the improving performance of recursive functions. Eric Lippert has a great post about how not to teach recursion where he specifically mentions not to use fibonacci numbers. However, I'm going to use fibonacci numbers not to teach recursion but to show how to improve the performance of recursive functions. Recall, that the definition of the nth fibonacci number where n >= 0 is:

fib(0) = 0
fib(1) = 1
fib(n) = fib(n - 1) + fib(n - 2) if n > 1

We can easily write a little function to do this:

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;

The problem is this function is very inefficient. Consider calculating the 5th fibonacci number.

fib(5)
= fib(4) + fib(3)
= fib(3) + fib(2) + fib(2) + fib(1)
= fib(2) + fib(1) + fib(1) + fib(0) + fib(1) + fib(0) + 1
= fib(1) + fib(0) + 1 + 1 + 0 + 1 + 0 + 1
= 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1
= 5

Notice how the computation branches out forming a tree of expansions. In fact, the number of leaves in a tree for the computation of the nth fibonacci number is fib(n + 1) where the number of ones is fib(n) and the number of zeros is fib(n - 1).

So calculating fib(5) will cause fib to evaluated twelve times (the sum of the fibonacci numbers from 0 to 5) but only six distinct calls will be made (fib(0), fib(1), ..., fib(5)). The problem only gets worse because the fibonacci numbers grow very quickly. Here are the first 20 fibonacci numbers:

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181

So our straightforward recursive function for computing fibonacci numbers will be exponential in n. Ouch! But wait, we can Memoize fib so that if it calls fib again with the same argument then it will not recalculate but instead use the previous result. This will move our exponential function to a linear one at the cost of linear space to store the previous results (we could possibly use a weak reference hash map instead to solve problems with space if they existed). In fact, subsequent calls to fib with previously computed values will be evaluated in constant time.

Func<int, int> fib = null;
fib = n => n > 1 ? fib(n - 1) + fib(n - 2) : n;
fib = fib.Memoize();

If you want, you can add a Console.WriteLine to show when the evaluations happen to both versions and compare. It is very enlightening.

Memoize is turning into quite the useful function. We have used it to solve parameterized identity creation problems and performance issues. Later, we will see other ways we can use Memoize to solve design problems.

Why Functional Programming is Important in a Mixed Environment

wesdyer — Thu, 18 Jan 2007 13:37:00 GMT

While my last post addressed how to go about learning to think functionally, it did not address why a programmer should embark on the journey in the first place. Why is learning to think functionally important, especially in a mixed environment like C#, Python, or Ruby? Why is it not good enough to stick with the tried and true object-oriented and structured programming styles?

Learning to program functionally will lead to more modular, expression-oriented, and conceptually simple code.

Modular Code

Think about a programmer who is introduced to structured programming for the first time. He may remark that it is all about while, if, and for statements. A programmer who encounters object-oriented programming may opine in turn that it is all about inheritance, encapsulation, and polymorphism. In our consideration of functional programming, let's not also make the mistake of confusing the tools we use with the goals we are working towards accomplishing. All of these programming styles are designed to increase the beauty, maintainability, and modularity of code.

Programmers are trained to notice bad smells. One particularly potent stench is duplicated code. A programmer tends to notice and remove the smell when the culprit is a block of contiguous statements that is repeated throughout a codebase, but sometimes the smell is more subtle and the cure is less obvious.

Consider a function that computes the sum of the squares of a list of numbers. A typical approach might look like this:

static int SumOfSquares(IEnumerable<int> nums)
{
int result = 0;
foreach (var item in nums)
result += item * item;
return result;
}

It seems good enough, but look closely. How many times do programmers compute the sum of a list of numbers? All of the time. So let's write a function to compute the sum.

static int Sum(this IEnumerable<int> nums)
{
int result = 0;
foreach (var item in nums)
result += item;
return result;
}

Ok, now we can simplify our SumOfSquares function by reusing the code to compute the sum.

static int SumOfSquares(IEnumerable<int> nums)
{
List<int> squares = new List<int>();
foreach (var item in nums)
squares.Add(item * item);
return squares.Sum();
}

Hmm. Not really the effect we wanted. If anything the code only got more complicated. Maybe we should extract out the part that takes a list of numbers and returns a list of squares. No, that really isn't a reusable piece of code so it would only be moving the complexity around. The problem is that the code is tightly coupled to the expression item * item. The expression is really local to the problem that we are solving, but the act of taking a list of things and performing some operation on each element to produce a new list is a common reusable operation. Functional programmers call this function Map.

static IEnumerable Map<T, U>(this IEnumerable<T> s, Func<T, U> f)
{
foreach (var item in s)
yield return f(item);
}

Now we can use Map to reduce the complexity of SumOfSquares.

static int SumOfSquares(IEnumerable<int> nums)
{
return nums.Map(x => x * x).Sum();
}

Now that is something to write home about. It does exactly what we want without specifying state-transition by state-transition how to go about doing it. Although this is a small example, it is very compelling. Functional programming took a tightly coupled section of code (item * item) and removed the dependency by use of higher-order functions.

Expression-Oriented vs Statement-Oriented

As a compiler guy, I often work with parsers and abstract syntax trees. Most people are familiar enough with parsing to know there are top-down and bottom-up approaches. What is sometimes overlooked is that there are top-down and bottom-up ways to create parse trees as well. If parent nodes are required to be created before child nodes then the tree is created top-down but if the child nodes must be created before the parent node then the tree is created bottom-up. There are several reasons why a particular system may prefer one approach or the other but one reason deserves closer examination for our purposes: top-down creation of trees is statement-oriented whereas bottom-up creation is expression-oriented. Take a look at the following code to create a parse tree for an if statement.

IfNode ifNode = CreateIfNode(parent);
ifNode.Condition = ParseCondition(ifNode);
ifNode.Consequence = ParseBlock(ifNode);
ifNode.Alternative = ParseBlock(ifNode);

Contrast this statement-oriented creation with an expression-oriented approach:

CreateIfNode(ParseCondition(), ParseBlock(), ParseBlock());

The statement-oriented approach is characterized by series of state transitions caused by assignments where some syntactic elements are L-values and others are R-values. The expression-oriented approach is just a combination of values which is itself a value. Expression-oriented code is more succinct and easier to grasp since it is not cluttered. Each piece is either a primitive value or a combination of values. In contrast, statement-oriented code is marred by lack of conceptual unity.

As the C# language has added more functional features, it has become a more expression-oriented language. Similarly, as a programmer learns to follow a functional style, he will naturally write code that is more expression-oriented in nature.

Conceptually Simpler

Imperative programming is sometimes reminiscent of a Rube Goldberg machine. Both require meticulous thought to ensure that a process works correctly despite a myriad of state transitions and interdependencies. It is amazing these complicated programs work at all.

Dijkstra pointed out that too many programmers rely on executing a program in order to understand it. The reason is imperative programs lack sufficient underlying formalisms to make guarantees about any but the most trivial of programs. As much as I love a debugger, it is disheartening to need to use it to understand my code.

Contrast this with functional programs which have solid theoretical foundations and therefore can more easily make runtime guarantees. In fact, many functional programming books dedicate a significant portion of their pages to showing the reader how to prove various runtime properties of programs. While most programmers wouldn't take the time to write proofs like Euclid's about their programs, it is compelling to think about creating sections of a program which are written so well that their properties are provable and understandable. In our own codebase, those sections of the code that approach this ideal are the most understandable and solid pieces of code that we have.

The Journey to Understanding

While reaching a point where we naturally think functionally is the goal, do not think that there are not ample rewards on the journey to understanding. Pause to appreciate the beauty of higher-order functions. Bask in the joy of type inferencing. Reflect on the infinite. Over the next couple of months, I will be writing about various functional gems that pertain to C# 3.0. And if I haven't done a good enough job explaining why functional programming is important then here are some fantastic papers which detail the subject.

Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs - John Backus

Why Functional Programming Matters - John Hughes

Thinking Functionally

wesdyer — Tue, 16 Jan 2007 00:45:00 GMT

Every programmer has a story about how he got his start. My own journey began at age 14 when I was bored to death while completing a second-year algebra assignment. My father who was working on the couch with a brand new laptop showed me that I could write a program to do the work for me. I instantly fell in love with the idea and have been rediscovering that love ever since.

I first began programming in BASIC. It didn't take long to learn BASIC. Within a matter of days, I had written a few adventure games, a prime number sieve, and a menu system. I had come to embrace GOTO statements. Hopefully, my brain hasn't sustained irreparable damage.

Within a few months of learning BASIC, I wanted to learn something else a little more lean and mean so I started to learn C. The process was extremely painful. I still remember feeling at times the strong desire to just give up and go back to BASIC. I felt this when the concepts just didn't sink in or when they did sink in but I wasn't as proficient at "doing it in C" as I was at "doing it in BASIC". When confronted with a new programming task, my instinct was to run off and do it in BASIC but I forced myself to learn to do it in C and within a month or two I had not only met my previous ability in BASIC but far surpassed it. New opportunities were open to me after I made the change.

As silly as this sounds now, when I first started the transition from BASIC to C, I wasn't even sure that it was possible. It was an act of faith to proceed to learn something that I wasn't even sure would payoff. After seeing the tremendous benefit at the end of the proverbial rainbow, I vowed that I would never let pain and difficultly get in the way of accomplishing something new.

The lesson has been repeated throughout my life. I remember another time when I first began riding seriously (mountain biking). It was painful. I would approach the bottom of a seemingly endless hill and whimper inside at the prospect of climbing it. After the ascent began it was only worse. My whole body screamed at me to drop the bike and walk up the hill, but I persevered. In the end, it was the constant perseverance that strengthened my muscles, my lungs, and my confidence such that I wasn't daunted by the hills anymore. In fact, I soon came to love the hills that put my strength, endurance, and determination to the test.

There are a number of lessons that can be gleaned from both experiences:

1. Pain is part of the learning process.

2. Do not allow previous knowledge to act as a crutch in the learning process.

3. The best way to succeed is to embrace the challenge.

Continuing the story of my programming language education, a few years after I learned C, I desired to learn about object-oriented programming so I began reading books about C++ and writing OO code. The learning process began anew but this time it was coupled with a new problem: when, where, and how should I apply OO techniques to the problems that I was solving. This proved much more difficult than just absorbing OO syntax or semantics. In fact, it took a number of years to get a good grasp on this problem. The solution involved several important parts: learn from the best minds (read good code and books, discuss with experts), practice, and experiment.

Of course, I was not alone in my quest to understand OO. The mainstream programming community was introduced to it in the 80s and really bought into it in the 90s. Quite a few people get it somewhat, some people have no clue (think thinly veiled procedural programs in OO clothing or a clutter of design patterns where there is barely a need), and a few people really get it right. I personally think that OO is especially hard to get since there is a dearth of good literature on the subject. Only a few books stand out as generally accepted, Design Patterns and Refactoring come to mind, but even these books are criticized by many.

More recently there has been a trend towards programming languages incorporating functional language features in the language. This has been happening for some time (procedures, recursion, and more recently garbage collection), but it really has been hitting its stride in recent years (closures, lambdas, continuations, type inference, monad like patterns). The problem is this is the first time that the mainstream programmer has been exposed to these concepts. How is she supposed to learn how to use these features? How will it impact her work? Are we in for the same process of maturation that OO programming experienced? I think not.

Fortunately, there is a wealth of information about functional programming and a long history of experience. This doesn't entirely free the would be functional programmer from the cost of learning a new way of thinking but it does drastically reduce the cost. So how should a programmer learn to think functionally? I recommend that a programmer read the best books, immerse himself in a functional language, and then practice, practice, practice.

The Best Books

Functional programming has been around since LISP which was the second programming language (following FORTRAN). Whereas OO programming had its genesis in common sense modeling of the real world (simulation software), functional programming was born through theory (lambda calculus). Academics have really embraced functional programming in research and so there are a number of very good books on functional programming. Here are two of the best books on the subject.

1. Structure and Interpretation of Computer Programs - Abelson and Sussman

2. Introduction to Functional Programming 1st Edition - Bird and Wadler (Out of Print - maybe on Ebay?)

Read these books and enjoy drinking from the firehose.

Learning by Immersion

My second recommendation is to meet the challenge head on. Learn a functional language like Lisp, Scheme, ML, Haskell, or Erlang. It doesn't matter which language it is as long as it doesn't allow a regression to previous ways of thinking. There will be a temptation at first to attempt a problem in ways that are comfortable and familiar, but working in a functional language will require a functional approach to the problem. It is very analogous to my story of learning to climb hills on a bike. I wanted to walk and at first it might well have been faster to walk but I forced myself to stay in saddle and in the end I could climb hills much faster on the bike than I ever could walking. So stay in the saddle and learn to think functionally.

Practice

Once good literature and immersion have been used in the learning process, it will be natural to apply functional programming techniques to problems in a language that allows both functional and non-functional approaches to problem solving. The final step is learning when to use the new tool in a mixed environment and the key to overcoming this challenge is to practice. Experience is vital in any art because it tunes the senses to appreciate beautiful solutions.

C# 3.0 offers some features which can drastically improve the programming experience but only by learning to think functionally can they be fully appreciated.

Thus Quoth the Humble Programmer

wesdyer — Sat, 23 Dec 2006 02:20:01 GMT

I love Scheme! It is such a beautiful language. I was first introduced to it during college. At that time, I thought it was an interesting language but I didn't see the power of the language until later. That moment came when I experienced reading The Structure and Interpretation of Computer Programs. It was a great joy to read the text and work through the exercises. But the most fulfilling part was thinking about what it all meant. I definitely encourage anyone who wants to expand their mind to read this wonderful text.

Quoting in Scheme

Scheme (or any other LISP dialect) allows programmers to define variables, define lambdas, and evaluate expressions (I know that I am greatly simplifying here). But one feature that is particularly interesting is the ability to quote. Quoting an expression causes the expression not be evaluated but instead to return the structure of the expression. For example (line numbers added for discussion purposes):

1:    (define f1 (lambda (x y) (+ x y)))
2:    f1
3:    (define e (quote (lambda (x y) (+ x y))))
4:    e
5:    (define f2 (eval e))
6:    (f1 3 4)
7:    (f2 3 4)

This program produces the following results.

#<procedure:f1>
(lambda (x y) (+ x y))
7
7

Line number 1 defines a variable f1 to be the result of evaluating a lambda expression, thus f1 is a function. Line 2 evaluates f1 which simply returns the function. Line 3 defines e to be the result of evaluating the quote of the same lambda expression that appeared in line 1. But now e is not a function but rather a structure describing the lambda. Line 4 evaluates e, which displays the structure it references. Line 5 defines a variable f2 to be the result of apply eval to e. Eval runs the Scheme interpreter on its arguments. Recall, that e is a structure describing a lambda. So applying eval to e we get a lambda and so f2 is now a function described by this lambda. Finally, on lines 6 and 7 we apply f1 and f2 respectively to arguments 3 and 4. Since they are both functions that correspond to the same lambda structure we expect to get the same results and indeed we do.

Quoting in C#

With C# 3.0 we can do essentially the same thing (again line numbers added for discussion purposes).

1: Func<int, int, int> f1 = (x, y) => x + y;
2: Console.WriteLine(f1);
3: Expression<Func<int, int, int>> e = (x, y) => x + y;
4: Console.WriteLine(e);
5: Func<int, int, int> f2 = e.Compile();
6: Console.WriteLine(f1(3, 4));
7: Console.WriteLine(f2(3, 4));

This program displays the following:

System.Linq.Func`3[System.Int32,System.Int32,System.Int32]
(x, y) => (x + y)
7
7

Notice how remarkably similar both the programs and that outputs are. Again, on line 1 we define a delegate to be the result of evaluating a lambda expression. On line 2, we display this delegate. On line 3, we assign e the result of evaluating a lambda expression. Notice that in C# both assignments (lines 1 and 3) look the same except for the type of the variable to which the lambda is assigned. Lambdas can be converted either to delegates or to expression trees depending on the usage. Delegates can be invoked while expression trees preserve the structure of the lambda. On line 4, we display the expression tree for e which corresponds to the structure of the lambda. On line 5, we do the equivalent of the eval in Scheme by compiling the expression tree structure to a delegate. We can now invoke f2 like we can invoke f1. On lines 6 and 7, we invoke these two delegates and display their results.

So What?

While this is all very remarkable, it may seem a bit esoteric. Why take the trouble to have a variable with a value that corresponds to the structure of a lambda and then convert that to an invocable form? Why not just always create the delegate immediately? The reason is with the expression tree form of the lambda, we can reason about the structure and then take action on it. Consider queries. When a query is translated it creates a number of lambdas, but one of the key questions is: Are those lambdas then converted to delegates or expression trees? For example:

var q = from x in foo
where x > 1
select x + 1;

This is translated to:

var q = foo.Where(x => x > 1)
.Select(x => x + 1);

We have already discussed that which "Where" and "Select" methods are called depends upon what the type of foo is and which extension methods are in scope. If the "Where" and "Select" methods take a delegate then the lambdas will be converted to delegates; however, if they take an expression tree then the lambdas are converted to expression trees instead. So a LINQ provider (or someone who has implemented the query pattern methods like "Where" and "Select") can choose whether they would like the lambdas in delegate or expression tree form. Now lets look at a few providers and see what they do. LINQ to objects (or the in-memory query operators that apply to IEnumerable<T>) take delegates. This is because they will invoke the delegates to do things like evaluate predicates or projections. However, LINQ to SQL on the other hand takes expression trees. It does not "invoke" the lambdas but instead reasons over the structure of the lambdas and then translates the expression trees to equivalent SQL code given a schema for a database. This is the beauty of the LINQ pattern.

Adding the ability to quote lambdas enables all sorts of possiblities. You might easily imagine other uses for expression trees: remoting evaluation of expressions, query planning, executing them on a GPU, ....

What can you think of doing?