Yet Another Language Geek : Programming

Musings on Software Testing

wesdyer — Fri, 07 Dec 2007 22:42:24 GMT

It was spring 2003, I had just finished a weekend camping in the southern Arizona desert. I was dusty and physically exhausted from hours of playing paintball. For those who have never been in those parts, imagine long straight roads with dry sage brush, painful cactus, and jagged mountains. I needed a book to pass the time during the drive. Fortunately, I had brought along "Extreme Programming Explained" by Kent Beck. After having spent too much time in projects that either completely lacked process or had way too much of it, Beck's position seemed like a revelation. Over the next few months, I became acquainted with test-driven development, design patterns, refactoring and many of the other topics typically associated with agile programming.

Test Driven Development

In the spring, I had a course that ended up just teaching agile programming concepts. One of the things that the professor emphasized was following the test-driven development (TDD) process to the letter.

The TDD Process:

1. Add a test

2. Run all tests and see the new one fail

3. Write some code

4. Run the automated tests and see them succeed

5. Refactor code

Repeat

When I say refactor, I mean it in the strictest sense. I mean changing how a program is arranged internally without changing the semantics of the program at all. This of course ignores the fact that things like timing, working set, or stack size change which can be material to the run-time behavior of the program.

During step three a programmer should remember, "It is important that the code written is only designed to pass the test; no further (and therefore untested) functionality should be predicted and 'allowed for' at any stage".

Machine Learning

I fully bought into this rigorous approach to TDD until one summer day when I was reading through Tom Mitchell's marvelous book "Machine Learning". The book begins by defining machine learning:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

Intuitively, the learning process is trying to learn some unknown concept. The learner has access to its past experiences (the training data) and uses them to generate a hypothesis of the unknown concept. This hypothesis is then evaluated and the new experiences are added to the training data. The learning process then repeats itself as the learner forms a better approximation of the underlying concept.

TDD is a learning process. Where the training experience E is the automated tests, the task T is improving the quality of the program, and the performance measure P is the percentage of tests passed. In this view of TDD, the programmer continues to hack away at his program generating hypotheses, where each hypothesis is a version of the code, until one satisfies the data. Note that unless the tests exhaustively specify the desired behavior of a program, then the tests are not the target concept that needs to be learned but rather a few data points where the target concept is manifest. Furthermore, the program should not only do well against the tests but also against real world scenarios which may not be covered by the tests.

Bias and Learning

A learner is characterized by its hypothesis space and how it searches through that space. Notice that since each learner has a hypothesis space, a given concept may have not be in that space. One solution to this problem is to make a learner's hypothesis space include all hypotheses, but this leads to a fundamental problem: "a learner that makes no a priori assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances." The only thing that an unbiased learner can do is regurgitate the training data because it has no ability to make the assumptions and logical leaps necessary to deal with unseen data.

For example, suppose that I am trying to learn to classify animals as either mammals or non-mammals and suppose that I make no assumptions whatsoever in the process of learning this task. If I am told that giraffes, lions, mice, bats, and whales are mammals but snakes, butterflies, frogs, lobsters, and penguins are not, then I only have the ability to answer questions regarding those animals for which I already know the answer. I could have done much better by trying to guess at what makes an animal a mammal even if I got it wrong!

Returning to TDD, note that at no time is a programmer supposed to write code that generalizes to the target concept. The programmer should only write either the minimal code to make a test pass or refactor the code and not change the semantics. So according to the pure TDD approach, the program would only learn to regurgitate the training data and do who knows what against new examples. A critic might respond that more test cases are needed. In some cases where the input space is finite and small, this might well help, but in general this is impractical.

Sufficiently Large Projects

A similar situation arises in projects that are sufficiently large. These projects are characterized by the fact that they cannot be held inside any one person's head. This means that when someone is making a change, they cannot know all of the consequences of their change. Often in projects of this size, when people ask if a particular change is good, the response is, "It passed the tests." This leads to a state where the development process as a whole is a learning process that uses the test cases as the training data and generates programs as hypotheses that are evaluated against the training data. Most likely, the target concept is not normally fully determined by the tests.

You know the projects that I am talking about, and chances are that you have seen some code that was added in just to satisfy some test but didn't really capture the essence of what the test was originally trying to make the program do. While tests are invaluable to programmers for debugging purposes, it would be better if programmers returned to the specification in order to understand what they are supposed to develop instead of returning to the tests.

Testing practices like clean-room testing and various forms of white box testing address address the programmer bias deficiency, but it can also be addressed automatically.

Testing an Infinite Space

I had a development lead who liked to say that testing is just sampling. In a sense he is right, since a test is a particular input sequence given to a program from the set of all possible input sequences. Think of a program as a state machine encoded in your favorite process calculus. A test then verifies that a particular path of edges corresponding to the transitions based on the input ends in a particular state. Most interesting programs have an infinite number of possible input sequences because of their recursive nature and so the set of tests must be sampled from an infinite input space.

Testing creates tests by drawing input sequences out of the input sequence space. It is easy to imagine that tests could be generated randomly by selecting random input sequences. Random selection has several advantages. First, it enables statistical measures of quality. For example, if we sample uniformly then the pass/fail rate should be representative of the whole given a large enough sample. We could also keep generating more and more tests to improve our confidence that the program is indeed correct. Second, random tests have the added benefit of not allowing the programmers to just "pass the tests" since the programmers cannot anticipate what tests will be run. This forces the programmers to generate a hypothesis that more closely matches the target concept instead of the training data.

There are however some disadvantages. The first apparent problem is repeatability of the tests. This can be remedied by storing the seed that generates a set of tests. A second more serious problem is that a uniform random sample doesn't necessarily deliver the best results.

On Bugs

Not all bugs are created equal. Practices such as testing the boundary conditions and integration testing are based on that fact. Various testing practices explore parts of the input space that are richer in bugs or are more likely to contain serious bugs.

The cost of a bug can be quantified as the total monetary loss due to the presence of that bug in the software. This includes all sorts of things like lost sales through negative customer perception, cost of software patches, legal action, etc. While this is an accurate measure of cost, it is not very practical because it is very hard to estimate.

A more practical measure for estimating the relative cost of a bug might be the probability that users will find it multiplied by the severity of the bug. The first part of the equation is interesting because it indicates that those bugs that customers never find are not important for testing to find.

Back to Training

The testing ideal is to minimize the total cost of bugs. There are many good methods for writing test cases. In addition to these, we could also select input sequences based upon the probability that a user would select such an input sequence.

Imagine if it were possible to collect all of the input sequences that would ever be given to a program including duplicates. Then our problem would be reduced to uniformly selecting input sequences out of this collection. Obviously, this collection is unavailable but if we use the same machine learning principles which we have been discussing then we could develop a program that could learn to generate test cases according to some distribution by using a representative set of user input sequences as training data.

Markov Chains

One way that this could be done is by returning to the formulation of a computer program as a state machine. The object is to learn the probability that a user would take some transition given a particular state and a sequence of previous states and transitions. This can be formulated as a Markov chain.

Consider generating random test cases for a parser. A straightforward approach is to randomly derive a string in the language from the root non-terminal; however, it will quickly become apparent that the least used parts of your language are getting undue attention. Some more sophisticated tools allow productions in the grammar to be attributed with weights. This works better but forces the tester to learn the proper weights manually and it doesn't accurately describe the weights since the probability that a particular production for a non-terminal will be used is also based on the history of previously expanded productions.

A better approach is to learn weights for the Markov chain corresponding to the pushdown automata. Since a parser for an arbitrary context-free grammar can be expressed by a pushdown automaton, we can instrument the automaton to collect data on the probability of some transition given a state and a history of states and transitions. The instrumented automaton can then be fed a large number of user-written programs. Finally, a generator can be created that uses the data to generate programs that syntactically look like user-written programs.

Another approach might be to instrument branches and event sequences in a program and then train against user input. The learner could then build a Markov chain that can be used to generate input sequences roughly like user input.

Conclusion

Next time you find yourself in a project that is devolving into an unbiased learning process, restore the sanity by using the specification and not the tests to decide what to implement.

All About Iterators

wesdyer — Fri, 23 Mar 2007 11:26:00 GMT

Design patterns have been all of the rage for a number of years now. We have design patterns for concurrency, user interfaces, data access, object creation, and so many other things. The seminal work on the topic is the Gang of Four's book, Design Patterns. When used appropriately they are a fantastic way to codify the wisdom gleaned from the battles we have fought building software systems.

One of the criticisms leveled at design patterns is that they are simply formalisms to address weaknesses in programming languages. They require the human compiler to generate code whenever a specific recurring problem is encountered that cannot be solved directly with language support. Now this might sound like heresy to some, but there is some truth to the criticism. Programmers adapt to the shortcomings in the languages they use by generating pattern like code either by hand or with metaprogramming (generics, dynamic code generation, reflection, expression trees, macros).

Let's take a look at one such example.

The Iterator Design Pattern

Among the many patterns in the literature is the iterator pattern.

In .NET this pattern is embodied by IEnumerable/IEnumerable<T> and IEnumerator/IEnumerator<T>.

An IEnumerable<T> is something that can be enumerated (iterated) by calling GetEnumerator which will return an IEnumerator<T> (iterator). The IEnumerator<T> is used to move a virtual cursor over the items that are iterated.

Implementing the iterator pattern is a bit onerous. For example, here is the suggested iterator implementation for List<T> from the Design Pattern book.

class ListIterator<T> : IEnumerator<T>
{
List<T> list;
int current;
public ListIterator(List<T> list)
{
    this.list = list;
    current = -1;
}
public T Current { get { return list[current]; } }
public bool MoveNext()
{
    return ++current < list.Count;
}
public void Reset()
{
    current = -1;
}
}

Since we can't change the list itself, we can introduce a ListIterable<T> that wraps a list.

class ListIterable<T> : IEnumerable<T>
{
List<T> list;
public ListIterable(List<T> list)
{
this.list = list;
}
public IEnumerator<T> GetEnumerator()
{
return new ListIterator<T>(list);
}
}

And finally, we can write a method called GetElements which returns an IEnumerable<T> over the elements of a List<T>.

static IEnumerable<T> GetElements<T>(this List<T> list)
{
return new ListIterable<T>(list);
}

Iterators in C#

Fortunately in most cases programmers don't need to deal with the iterator design pattern directly since the introduction of iterators in C# 2.0. Instead of writing the iterator above, we can simply write the following:

static IEnumerable<T> GetElements<T>(this List<T> list)
{
for (int index = 0; index < list.Count; ++index)
yield return list[index];
}

When the C# compiler sees this method, it translates it into something very similar to the ListIterator<T> and ListIterable<T> above. Using Reflector or ILDasm we can see that the GetElements method is rewritten as (the names have been changed for clarity as the compiler generates unspeakable names;):

private static IEnumerable<T> GetElements<T>(this List<T> list)
{
GetElementsIterator<T> temp = new GetElementsIterator<T>(-2);
temp.listParameter = list;
return temp;
}

This is remarkably closer to the first GetElements we wrote rather than the second. Like in the first GetElements, we create an object that implements IEnumerable<T> and parameterize this object with the list that was passed in. The only other thing that happens in this implementation that doesn't in the first GetElements method is a -2 is passed into the object. I'll come back to this later, but first let's take a look at the GetElementsIterator<T> class (I omitted a few details and changed names for clarity).

private sealed class GetElementsIterator<T> : IEnumerable<T>, IEnumerator<T>
{
int state;
T current;
public List<T> listParameter;
int initialThreadId;
public int index;
public List<T> list;
public GetElementsIterator(int state);
private bool MoveNext();
IEnumerator<T> IEnumerable<T>.GetEnumerator();
void IEnumerator.Reset();
T IEnumerator<T>.Current { get; }
}

The most important thing to note at this time is that GetElementsIterator implements both IEnumerable<T> and IEnumerator<T>. A strange combination, but we will see why in a little while. Now let's look at what actually happened when we ran the constructor.

public GetElementsIterator(int state)
{
this.state = state;
this.initialThreadId = Thread.CurrentThread.ManagedThreadId;
}

Did the constructor run the for loop? No, it didn't. It took some variable called state in and marked what thread created the iterator (if you look at Whidbey code the initialThreadId stuff won't be there...I'll get to that). The state variable marks what state the GetElementsIterator is in. The number -2 is the "I'm an IEnumerable<T>" state. Now, let's look at the GetEnumerator method.

IEnumerator<T> IEnumerable<T>.GetEnumerator()
{
GetElementsIterator<T> temp;
if ((Thread.CurrentThread.ManagedThreadId == initialThreadId) && (state == -2))
{
    state = 0;
    temp = this;
}
else
{
    temp = new GetElementsIterator<T>(0);
}
temp.list = this.listParameter;
return temp;
}

The method first checks to see if the thread that is calling GetEnumerator is the same thread that created the GetElementsIterator object. If it is the same thread and if the GetElementsIterator is in the "I'm an IEnumerable<T>" state then the state is changed to the "I'm an initialized IEnumerator<T>" state. Otherwise, a new object of the same type is created but it is immediately put in the "I'm an initialized IEnumerator<T>" state so that thread safety is maintained. Finally, in either case the list that was passed from the GetElements method is copied into the list field for consumption by the MoveNext method.

In Whidbey code, you will see that the if statement is different because an Interlock.Exchange was used to perform the same task. The change was made to improve performance (especially for iterators that have 0 or 1 items to iterate over).

Now we have seen why GetElementsIterator implements both IEnumerable<T> and IEnumerator<T>, because it can morph from the IEnumerable<T> role into the IEnumerator<T> role without creating any new objects in most cases.

The Current property is very simple.

T IEnumerator<T>.Current
{
get
{
return current;
}
}

The Reset method simply throws a NotSupportedException.

This leaves us with only the MoveNext to examine. We still haven't seen where the for loop went. Hopefully, it is in the MoveNext. The following code is produced with the /o+ compiler option (optimizations turned on).

private bool MoveNext()
{
switch (state)
{
case 0:
    state = -1;
    index = 0;
    break;
case 1:
    state = -1;
    index++;
    break;
default:
    goto Done;
}
if (index < list.Count)
{
    current = list[index];
    state = 1;
    return true;
}
Done:
return false;
}

Hm...a switch statement over a variable called state with a number integer case labels. It looks like a state machine and indeed it is. When MoveNext is first called, state is equal to 0, so the state is set to -1 (the "I'm finished" state) and the index is set to 0. Then we check to see if the index is less than the number of things in the list. If so then we set current to the current list element based on the index and set the state to 1. We return true indicating that there is something to consume.

The second call to MoveNext will run case 1 since the state is equal to 1. It will again set the state to -1 and increment the index. If we still have elements in the list then the current will be set appropriately and the state will be set to 1 (the "we need to check again" state) and true will be returned.

This continues until there nothing left to consume and false is returned.

Finally, we found our for loop. It is encoded in the MoveNext. But note that on each call to MoveNext is only computes the part of the for loop that is relevant to realize the next element. It never computes more than it needs to. This is why iterators are an example of deferred execution. When the actual GetElements method was called, no elements were realized at all! Later as MoveNext is called, one element at a time is realized. This of course enables all of sorts of great scenarios such as the pay-as-you-go model and the ability to have infinite lists.

The Cost of Iterators

Iterators are very performant. In almost all situations that I have encountered they are more than performant enough and they simplify the code drastically as we have seen. But sometimes you can get into trouble.

Consider the definition of the Concat sequence operator in Linq to Objects. It looks something like this:

static IEnumerable<T> Concat<T>(this IEnumerable<T> sequence1, IEnumerable<T> sequence2)
{
foreach (var item in sequence1)
yield return item;
foreach (var item in sequence2)
yield return item;
}

Let's write a little benchmark to evaluate the performance of concat.

var stopWatch = new Stopwatch();
for (int length = 0; length <= 10000; length += 1000)
{
var list = new[] { 1 };
IEnumerable<int> ones = list;
for (int i = 0; i < length; ++i)
ones = ones.Concat(list);
stopWatch.Reset();
stopWatch.Start();
foreach (var item in ones) ;
stopWatch.Stop();
Console.WriteLine("Length: {0} Time: {1}", length, stopWatch.ElapsedMilliseconds);
}

The results may be perhaps surprising. The time to evalute the foreach statement is not linearly proportional to the number of concats that are composed together. In fact it is proportional to the square of the number of concats composed together.

Upon closer inspection the reason why is obvious. The time complexity of Concat is O(m+n) where m is the number of items in the first sequence and n is the number of items in the second sequence. But note that in this example, n is always 1. The outermost call is O(m+1). The next call has O((m-1)+1), then O((m-2)+1), ... O(1+1). There are m of these calls so the running time should be O(m^2). Essentially, composing concats together like this causes O(m^2) yield returns to be executed.

Of course, using a List<T> here and adding on the sequences would have been much more performant because it eliminates the redundant calculations but it would not have been evaluated lazily.

Iterators are even more fun if the data structure that is being enumerated is more complicated. For example, consider iterating over n-ary trees. Here is a quick definition of a n-ary tree.

class Tree<T>
{
public T Value { get; private set; }
public Tree<T> NextSibling { get; private set; }
public Tree<T> FirstChild { get; private set; }
public Tree(T value, Tree<T> nextSibling, Tree<T> firstChild)
{
    Value = value;
    NextSibling = nextSibling;
    FirstChild = firstChild;
}
public IEnumerable<Tree<T>> GetChildren()
{
    for (var current = FirstChild; current != null; current = current.NextSibling)
      yield return current;
}
}

Now it is easy to define an iterator that performs a preorder traversal of a n-ary tree.

static IEnumerable<T> PreOrderWalk<T>(this Tree<T> tree)
{
if (tree == null)
    yield break;
yield return tree.Value;
foreach (var subTree in tree.GetChildren())
    foreach (var item in subTree.PreOrderWalk())
      yield return item;
}

Just the way that I like code: clear and concise. The only problem is that the iterator could be more efficient. This may or may not be a problem. In a library it will almost certainly be a problem.

We can improve the efficiency somewhat by changing the code:

static IEnumerable<T> PreOrderWalk<T>(this Tree<T> tree)
{
var stack = new Stack<Tree<T>>();
if (tree != null)
    stack.Push(tree);
while (stack.Count > 0)
{
    for (var current = stack.Pop(); current != null; current = current.FirstChild)
{
    yield return current.Value;
    if (current.NextSibling != null)
      stack.Push(current.NextSibling);
    }
}
}

This second iterator doesn't recursively call iterators thus avoiding both the recursive call and the extra allocations. Instead, it maintains a stack of work to do after the leftmost path has been exhausted. Once the leftmost path has been exhausted then a node is popped off and the leftmost traversal is resumed at that node.

When we measure the difference, we see that the improvement is noticeable but that the number of nodes, O(b^d) where b is the branching factor and d is depth, dominates the cost of the traversal. In the graph below, the green line indicates the total number of nodes in the tree. The trees have a branching factor of 2.

So the key takeaway here is that iterators have great performance but as always measure the performance of your code. If you find that performance is suffering, use a combination of profiling and analysis to find the problem. If the problem is an iterator, you might be able to increase the performance by reworking the iterator as in the n-ary tree case. In other cases, it might be the usage of the iterators as with pathological Concats.

One Possibility for Language Improvement (not in Orcas)

The Concat sequence operator is interesting because there is a lot of code that is seemingly redundant. It takes two sequences and then has to iterate over them and yield their elements. It's like it needs to expand their insides just to package them up together again. As we have seen this doesn't lead to the best performance and the code is overly verbose.

Bart Jacobs, Erik Meijer, Frank Piessens, and Wolfram Shulte wrote a very interesting paper on a possible language improvement that would improve both the usability and the performance of iterators. The second half of the paper details what they call nested iterators which avoid the multiple evaluation problem of the composed Concats and implicitly keep an internal stack like the modified n-ary tree iterator.

For example with this language feature, the Concat sequence operator would look something like this:

static IEnumerable<T> Concat<T>(this IEnumerable<T> sequence1, IEnumerable<T> sequence2)
{
yield foreach sequence1;
yield foreach sequence2;
}

Notice that the code is simplier and it is also more performant.

A programmer could use yield return, yield break, and yield foreach in the same iterator. An iterator FromTo can be defined recursively as follows:

static IEnumerable<int> FromTo(int b, int e)
{
if (b > e)
yield break;
yield return b;
yield foreach FromTo(b + 1, e);
}

If instead of the yield foreach, there was the foreach expansion that yielded each result then the FromTo method would suffer from quadratic performance; however, with nested iterators the performance would be linear.

The next post will pick up on understanding the performance of Linq to objects queries.

Performance Engineering

wesdyer — Tue, 20 Mar 2007 12:52:56 GMT

Recently, many people have asked me about the performance of Linq. The questions have ranged from the broad, "How can I analyze the performance of code using Linq?", to the very specific, "Why is the performance of this code sample not what I expected?"

I want to address these questions. So before I dive into some of them individually, I want to set up a framework for discussions about performance. The framework consists of ideas that are well known and understood and yet often when people talk or think about performance they forget about them.

Performance is critical to the success of any application and engineering performance is a fundamental part of what developers should be doing on a day to day basis. Great performance doesn't just happen. It must have requirements, be designed, implemented, and tested just like any other feature. Individuals and teams must foster a culture of performance.

Performance Requirements

Rico Mariani wrote a great post about why performance must be measured in context. That means that performance goals should be made in relation to what the user sees and cares about. Then when possible these goals should be translated into more low-level goals that are easier to quantify and directly measure, but never take your eye of the original goals. Many times when performance is tuned in one dimension it can degrade in other dimensions. It is like trying to cram a whole bunch of stuff into a tiny space, you may push it in here but it pops out over there. So how does a programmer know which dimension is most important? Consider the performance in context. Furthermore, the performance tuning that a programmer makes might not even matter when considered in context. So spend your time where in counts. Finally, it is much easier to consider trade-offs between performance requirements and other requirements when the performance requirements are in context. Set up a performance budget in terms that are material and important to the customer.

Designing for Performance

Designing for performance is important. Many performance problems can be fixed as they are identified later, but some are deeply rooted in poor design choices that were made early on that are difficult to resolve later. However, care must be taken because often these poor design choices were made for the sake of "better performance". For example, in an app where there is a lot of data parallelism, it doesn't make a lot of sense to make it so complex with stateful operations that it precludes concurrency.

Most of these poor design choices are caused by the design not solving the right problem (performance requirements) or because the requirements changed over time and the earlier design made it difficult to adapt to the new requirements.

Understanding Performance

There are at least two basic tools for understanding performance.

1. Asymptotic Analysis

I am sure that my readers know and love asymptotic analysis. It is so important that developers know what they are doing when they call a System.* function, which has an O(n) time complexity, n times -- O(n^2) behavior. Know what is going on in terms of both space and time.

2. Benchmarks / Profilers

So often when a performance problem exists, developers will convince themselves that they know what the problem is without accurately measuring the behavior of the system. They will proceed to "fix" the problem only to discover that what they thought would be of great benefit is in fact unimportant.

I've done that. When C# 1.0 was in beta back in 2000, I was working at a small company that decided to try out this new .NET stuff. My first application was a good sized application that including a custom scripting engine inside of it. The performance wasn't great and so I set about fixing it. I was sure that I knew what the problem was and so I began improving some algorithms in various complex parts of the system. But I found that it didn't help matters at all.

At this point, I broke out a profiler and began measuring the actual behavior of the system. The suprising result (at the time) was that pretty much all of the performance problems came from a small loop that was building up the text buffer for the scripting engine. It was using the + operator on strings. Since this was my first .NET application, I had no idea that concatenating hundreds of thousands of strings was a bad idea.

When I changed the code to use a StringBuilder instead, it sailed. I made a few other improvements (all of which were targeted and very small), and the application was running fine.

Now the point of all of this is not that the + operator on strings should not be used. If that were true then we would make the C# compiler issue a warning when you used it. The point is that a programmer should be aware of the costs and tradeoffs involved with a programming decision and act accordingly. Measurement is a powerful tool. Knowing where the problems lie is the key to success. Measure early and often.

The result of profiling is a large set of various statistics about an application. If the measurements are taken in context then they are a powerful tool. As with any statistics, what is being measured is as important as the actual measurement.

Analyzing an application is very rewarding. You are able to see the material benefits of any improvement in a very quantitative way. Apply the scientific method in the process of analyzing a problem. Once you suspect you know what the problem is then consider it the hypothesis and then prove it by observing the data (profiling data) from an operation (the suspect code) across various trials.

Try to think big and understand not only what is the problem but why it is a problem. The best solutions attack not only the symptoms but the fundamental causes of the problem. Look to understand the nature of the problem.

Some Good Performance Resources

Rico Mariani's (Visual Studio Performance Architect) Blog

Vance Morrison's (CLR Performance Architect) Blog

Linq to ASCII Art

wesdyer — Fri, 23 Feb 2007 09:02:00 GMT

Last night I was searching for an audio version of Painters and Hackers by Paul Graham. Pretty soon I had completely forgotten about the book and found myself reading the Wikipedia article about Hackers. Isn't Internet search great?

Of all of the things in the article, the one thing that captured my attention was the ASCII picture of Adrian Lamo. I immediately thought, "How cool is that!?" So instead of popping off my current stack frame and returning to searching for Painters and Hackers, I began thinking about how to create such a picture. I certainly did not want to create it by hand. I didn't even want to pick which characters or colors would be printed to the screen. So accordingly, I began thinking of writing a program to do it for me.

I grabbed the latest bits and began writing code. A few hours later, I had a working program with a few bells and whistles too.

ASCII Art

Here is an example of converting Gandalf from pixels to ASCII:

How to Convert and Image to ASCII Art

The process works like this:

Determine how big of an area will be used for displaying the ASCII art. By default, the width of the image will be the width of the console's buffer and the height will be computed so as the preserve the ratio of width to height from the original image.

Now divide the original image into as many rectangles as will be used to display the ASCII. For each rectangle in the original picture determine its grayscale value by averaging the grayscale values of each pixel. The grayscale value of a pixel is:

GrayScale = .3 * Red + .59 * Green + .11 * Blue

We then increase the contrast of each region by some amount (d is the luminosity of a region, contrast is any double but usually around 1.5):

public static double Contrast(this double d, double contrast)
{
return (((d - .5) * contrast) + .5).Bound(0, 1);
}

Finally, for each region in the ASCII art we choose from an array of possible ASCII figures (sorted) by the grayscale value for that region. The ASCII figure with the closest luminosity is picked.

The only point that I haven't covered is how the array of ASCII figures was generated. I could have done this by hand if I wanted to, but I didn't want to. Instead, I generate the grayscale figures each time. Each figure consists of a character and a console color either ConsoleColor.DarkGray, ConsoleColor.Gray, or ConsoleColor.White. The characters are one of the alphanumeric characters or the special symbols (things like %, $, @, ...). The program generates all of the combinations and then measures their grayscale value by drawing them to a hidden bitmap and then computing their average grayscale value. The figures are sorted by this value and a final grayscale is built.

A Few Snippets

If you take a look at the source, you will notice that it is written rather different than most C# code. A lot of the code resides in extension methods and there are a number of lambdas and queries. In fact, loops are kind of rare in the code. For example, here is a function that returns all of the pixels in a rectangular region of a bitmap.

public static IEnumerable<Color> GetPixels(this Bitmap image, Rectangle rect)
{
return from y in rect.Top.To(rect.Bottom)
from x in rect.Left.To(rect.Right)
select image.GetPixel(x, y);
}

The To function is a helpful little function that I wrote.

public static IEnumerable<int> To(this int start, int end)
{
var diff = end - start > 0 ? 1 : -1;
for (var current = start; current != end; current += diff)
yield return current;
}

Of course, I also could have written Iterate and then written To in terms of that.

public static IEnumerable<T> Iterate<T>(T initialValue, Func<T, bool> predicate, Func<T, T> next)
{
for (var current = initialValue; predicate(current); current = next(current))
yield return current;
}

public static IEnumerable<int> To(this int start, int end)
{
var diff = end - start > 0 ? 1 : -1;
return Iterate(start, x => x != end, x => x + diff);
}

Another interesting part is setting up the display and then restoring the original display properties of the console.

using (SetupConsole())
...

Where SetupConsole is defined as:

static ActionDisposable SetupConsole()
{
var originalBackgroundColor = Console.BackgroundColor;
var originalForegroundColor = Console.ForegroundColor;
return new ActionDisposable(() =>
    {
      Console.BackgroundColor = originalBackgroundColor;
      Console.ForegroundColor = originalForegroundColor;
    });
}

Notice that the original state of the console is captured only in the SetupConsole method and does not clutter the rest of the code. The closure that is created contains all of the necessary information to restore the console to its original state. This way we hide data even from private members of the same class. If they don't need to know, then they don't need to know. Instead only a means for restoring the original state is provided.

I'm sure that the program can be improved a lot. So if you have any comments, suggestions, or bugs then just post a comment. Enjoy!

Update:

I took the comments as well as some of my own ideas and improved the ASCII art generator. I also modified the How to Convert an Image to ASCII Art section to reflect some changes. Thank you to everyone that contributed.

Download the Source

AsciiPictureSource.zip

AsciiPictureSourcev1.1.zip

How Linq to Objects Queries Work

wesdyer — Wed, 03 Jan 2007 04:40:26 GMT

If you have ever tried to step through a Linq to Objects query in the debugger, you may have been mildly surprised at the results. It may have seemed as if the program had a mind of its own and ran certain expressions when it wanted to and not when it was supposed to. Be assured, it is doing the right thing and the order of evaluation is giving you a glimpse of what actually happens when a query is evaluated.

Consider this query:

using System;
using System.Linq;
class Program
{
static void Main()
{
    var foo = new[] { 4, 2, 1, 3, 0 };
    var q = from x in foo
            let square = x * x
            where square > x
            select new { x, square };
    foreach (var item in q) Console.WriteLine(item);
}
}

Now consider where breakpoints can be set within this program:

Pop Quiz

Which order are the breakpoint spans visited in. Try to guess the order of evaluation. Of course you can verify the order in a debugger.

The order may of evaluation may surprise you. Why does it behave like this? Well, to really understand we need to translate it. First, lets translate the query:

var q = foo.Select(x => new { x, square = x * x })
.Where(t0 => t0.square > t0.x)
.Select(t0 => new { t0.x, t0.square });

Now, lets change the extension methods to static method calls.

var q = Enumerable.Select(
          Enumerable.Where(
            Enumerable.Select(foo, x => new { x, square = x * x }),
            t0 => t0.square > t0.x),
          t0 => new { t0.x, t0.square });

Now, lets look at how Enumerable.Select and Enumerable.Where are implemented. We can not include the "using System.Linq" and instead implement Select, Where, and Func ourselves.

using System;
using System.Collections.Generic;
delegate R Func<A,R>(A arg0);
static class Program
{
static void Main()
{
    var foo = new [] { 4, 2, 1, 3, 0 };
    var q = Select(
              Where(
                Select(foo, x => new { x, square = x * x }),
                t0 => t0.square > t0.x),
              t0 => new { t0.x, t0.square });
    foreach (var item in q) Console.WriteLine(item);
}
static IEnumerable<U> Select<T,U>(IEnumerable<T> sequence, Func<T, U> selector)
{
    foreach (var item in sequence)
      yield return selector(item);
}
static IEnumerable<T> Where<T>(IEnumerable<T> sequence, Func<T, bool> predicate)
  {
    foreach (var item in sequence)
      if (predicate(item))
        yield return item;
}
}

Now consider what happens when we run this program which is essentially identical to our original program. First, foo is assigned a reference to an array of ints. Next since C# uses pass-by-value parameter passing by default, the arguments to the methods must be evaluated before the functions can be applied to their arguments. So the innermost Select call is called first which creates a Select Iterator using foo as the source but note that it does not run the iterator. This behavior is called lazy evaluation. In some other world, in perhaps a different language, the iterator could immediately evaluate all of its results, cache them, and then return them when demanded; but, this is not what happens with iterators in C#. When you call a method that is an iterator, the iterator is only constructed but not run. Remember that the iterator object that was constructed implements IEnumerable and so in this case it can be used as the first parameter to the Where method. Finally, the result of the Where method is passed to the final Select method. The whole result is assigned to q. Note, that q is not actually a realized result set. In fact, q is object that can compute the result set when required. Since, q is made up of iterators it will only do the minimum work necessary to return the next element when asked.

So in the foreach statement when the first element of q is requested, the outermost Select iterator begins running but it requires its source to return the first element which in turn runs the Where iterator which needs its source which in turn calls the innermost Select iterator which requires its source to be realized but that source is foo which simply returns 4 as the first element. The innermost Select iterator then produces a tuple of x and the square of x and returns this result. The Where iterator then checks the predicate against this first element and it passes so it returns the result. The outermost Select iterator then produces a projection using this result. Note that the minimum possible work was done to compute the first result of the query.

This works precisely as before for the second query result but when we ask for the third result things start to get a bit more interesting. Again the outermost Select iterator calls the Where iterator which calls the innermost Select iterator which asks foo for its next element which is 1. The innermost Select iterator makes a tuple with 1 and its square. The Where iterator then checks the condition and it fails so it does not return the item but requests the innermost Select iterator to produce another result which is does and this second result then passes the Where predicate. The outermost Select iterator then gets 3 and its square which are projected and returned as the third result of q. It is easy to see that queries essentially become a pipeline where each computation is fed to the next and all are independent of the others as has been previously described.

There are several important implications based on how Linq to Objects queries actually work.

1. Laziness enables the ability to work with infinite sequences

2. Operations which require the entire source to be examined change the order of evaluation

1. Laziness and the Infinite

Here is a program which displays the first 10 positive odd numbers.

using System;
using System.Collections.Generic;
using System.Linq;
static class Program
{
static void Main()
{
    var odds = from x in NaturalNumbers()
               select 2 * x + 1;
    foreach (var item in odds.Take(10)) Console.WriteLine(item);
}
static IEnumerable<int> NaturalNumbers()
{
    for (var n = 0; ; n++)
      yield return n;
}
}

Notice that the NaturalNumbers iterator never ends (Yes, it would overflow but lets ignore that for the sake of this discussion). The query then says for each natural number project twice the number and add one which is the definition of odd. If this query immediately evaluated the results then this program would not terminate since NaturalNumbers never terminates; however, the program does terminate. When query is created, it only constructs the information necessary to evaluate the query later. Then when we foreach over the query, the query returns one result at a time as they are required. In fact, in this case we also add another computation step onto q in the foreach statement by saying that we will only take the first 10 results so only the first ten natural numbers were even created. This is beautiful. Lazy evaluation enables the ability to work with the infinite. I will have a lot more to say about this topic another time.

2. Forced Evaluation

There are some operations that require whole source examination before a single result can be passed on to the next operation. One such operation is orderby. In order to determine the position of any element, the whole sequence must be examined. Thus, if we add an orderby to our original query then the order of evaluation of the query will dramatically change.

var q = from x in foo
        let square = x * x
        where square > x
        orderby x
        select new { x, square };

If you run the query in the debugger now, you will notice that the let, where, and orderby operations are all performed before the first select operation is performed. Subsequent elements will skip the let, where, and orderby and go straight to the select statement. This is because the orderby operation forced the evaluation of all of the previous operations. Subsequent results do not need to evaluate the let, where, or orderby operations because they have already been evaluated.

Other common operations that cause this behavior are join, join...into, and group...by clauses.

Corollary to (1) and (2) - Forced Evaluation and the Infinite

Now consider what would happen if we tried to use orderby with NaturalNumbers. As you might expect, our program would never terminate. To use the two together we would need to make sure that we only use a section of NaturalNumbers like NaturalNumbers.Take(10). This is because even though queries are lazy, if operations are used which force the evaluation of all of the results then the benefits of being lazy are effectively removed. So we must be careful when using infinite sequences.

Reading and Writing Queries

wesdyer — Tue, 26 Dec 2006 05:01:22 GMT

Now that we have discussed some of the underpinnings of query expressions, we can turn our attention to how to use them. As has already discussed, most people who see query expressions immediately think of SQL and often even mistake query expressions for embedded SQL. While it is true that there is a similarity in the syntax, it has already been pointed out that they are fundamentally different in important ways. Because of their semantic difference and syntactic similarity they seem to cause somewhat of a cognitive dissonance with respect to SQL. In other words, people see query expressions and assume that they behave exactly like SQL. With some providers (LINQ to SQL) this may be largely true, but with others they are very different (LINQ to objects). In this post, we will examine the syntax and scoping rules of query expressions.

Query Syntax

First, let's turn our attention to the matter of syntax. A query is simply a sequence of clauses. There are only nine types of clauses: from, join, join...into, let, where, orderby, group...by, select, into. A query must begin with a from clause and end with either a select or a group by clause. In between the start and the end there may be any number of from, join, join...into, let, where, and orderby clauses. After a query has been ended it may be "continued" by using an into clause followed by any number of from, join, join...into, let, where, and orderby clauses and then followed by either a select or group...by clause. Here is a description of the syntactic structure of each clause (modified EBNF where more descriptive labels are given to the expression and identifier non-terminals).

fromClause -> from variable in source

joinClause -> join variable in source on outerKey equals innerKey

joinIntoClause -> join variable in source on outerKey equals innerKey into groupingVariable

letClause -> let variable = expression

whereClause -> where condition

orderbyClause -> orderby key [ascending | descending] {, key [ascending | descending ]}

selectClause -> select projection

groupByExpression -> group groupingResult by groupKey

intoClause -> into variable

Here is an example:

from c in customers

where c.State == "WA"

orderby c.Name

join o in orders on c.ID equals o.CustomerID

let OrderTotal = o.Price * o.Quantity

select new { c.Name, OrderTotal }

Unlike SQL, C# queries do not begin with a select clause. The primary motivation for having the from clause come first is because in C# variables are always declared before they are used. So when c is used in the where clause, it is obvious that this c was introduced before it (in the from clause). One benefit of having a strongly typed language that declares variables before they are used is that IDEs can provide intellisense for these variables when they are used. Other reasons for the differing order of clauses will be discussed in the next post.

Scoping Out Queries

Based on the syntax and translation rules of query expressions and those of transparent identifiers, it is possible to derive the scoping rules for queries. Mostly these scoping rules are somewhat intuitive, but there are a few places where you might not get what you expect. To discuss the scoping of rules effectively, it is helpful to introduce a few formalisms. We can think of each query clause as a function that takes a scope as input and produces a scope as output. So for example, if s represents the set of variables that were introduced in this query's previous clauses and are still in scope and o represents the set of variables that are in scope from outside of the query then after the from clause all the variables in s and o are in scope as well as the variable that the from clause introduced. Also, in the source expression in the from clause both the variables from o and from s are in scope but x is not in scope. We will denote this as:

[o,s] -> from x in [o,s] -> [o,s,x]

First we see in this notation that variables in o and s flow into the query. Then we see that in the source expression in the from clause, all of the variables in o and s are in scope. Finally, the notation indicates that after the from clause all of the variables in o and s as well as the variable x are in scope. The following are the scoping rules for each clause (based on the rewrite rules).

[o,s] -> from x in [o,s] -> [o,s,x]

[o,s] -> join x in [o] on [o,s] equals [o,x] -> [o,s,x]

[o,s] -> join x in [o] on [o,s] equals [o,x] into g -> [o,s,g]

[o,s] -> let x = [o,s] -> [o,s,x]

[o,s] -> where [o,s] -> [o,s]

[o,s] -> orderby [o,s] -> [o,s]

[o,s] -> group [o,s] by [o,s] -> [o,s]

[o,s] -> select [o,s] -> [o,s]

[o,s] -> into x -> [o,x]

Again, the spec does not call out these scoping rules. They are an emergent behavior given the rewrite rules and the rules for transparent identifiers.

Occasionally, I talk to someone who wonders why the two key expressions in a join clause are not commutable. One reason becomes apparent after examining the scoping rules: the expressions are not commutable because they don't have the same variables in scope. The join query variable is not in scope in the outer key expression while the previous query variables are not in scope in the inner key expression.

Using these scoping rules we can understand now which variables are in scope in each expression in a query. Try this query out. See if you can figure out which variables are in scope at each point (the answer is at the end of this post).

var src = ...;

var q = from x in [???]

        where [???]

        from y in [???]

        join z in [???] on [???] equals [???] into g

        orderby [???]

        let a = [???]

        select [???]

        into b

        join c in [???] on [???] equals [???]

        group [???] by [???];

Thinking About Query Continuations

In the previous section you may have noticed that query continuations (into variable ...) erase all of the variables that were in scope in the query. This is because query continuations take the result of a query and pipe it in to the next query. Looking at the translation rules, we see that a query like:

from x in foo

select f(x)

into y

select f(y)

Becomes:

from y in from x in foo

select f(x)

select f(y)

The key here is that we can divide a query into a series of sections where each section ends in either a select or a group by. Each section can be considered independently. For example, instead of thinking of the query as the rewrite indicates, we could think of it as:

var q1 = from x in foo

select f(x);

var q2 = from y in q1

select f(y);

Notice how each section of the query becomes one query and then is used as the source for the next query section. The difference between this "rewrite" and the specified rewrite is that the specified rewrite produces a single expression while this "rewrite" produces several statements. So while this is not the actual translation, it is helpful in understanding how query continuations work.

Answer to Scoping Question

var src = ...;

var q = from x in [src]

        where [src, x]

        from y in [src, x]

        join z in [src] on [src,x,y] equals [src,z] into g

        orderby [src,x,y,g]

        let a = [src,x,y,g]

        select [src,x,y,g,a]

        into b

        join c in [src] on [src,b] equals [src,c]

        group [src,b,c] by [src,b,c];

Thus Quoth the Humble Programmer

wesdyer — Sat, 23 Dec 2006 02:20:01 GMT

I love Scheme! It is such a beautiful language. I was first introduced to it during college. At that time, I thought it was an interesting language but I didn't see the power of the language until later. That moment came when I experienced reading The Structure and Interpretation of Computer Programs. It was a great joy to read the text and work through the exercises. But the most fulfilling part was thinking about what it all meant. I definitely encourage anyone who wants to expand their mind to read this wonderful text.

Quoting in Scheme

Scheme (or any other LISP dialect) allows programmers to define variables, define lambdas, and evaluate expressions (I know that I am greatly simplifying here). But one feature that is particularly interesting is the ability to quote. Quoting an expression causes the expression not be evaluated but instead to return the structure of the expression. For example (line numbers added for discussion purposes):

1:    (define f1 (lambda (x y) (+ x y)))
2:    f1
3:    (define e (quote (lambda (x y) (+ x y))))
4:    e
5:    (define f2 (eval e))
6:    (f1 3 4)
7:    (f2 3 4)

This program produces the following results.

#<procedure:f1>
(lambda (x y) (+ x y))
7
7

Line number 1 defines a variable f1 to be the result of evaluating a lambda expression, thus f1 is a function. Line 2 evaluates f1 which simply returns the function. Line 3 defines e to be the result of evaluating the quote of the same lambda expression that appeared in line 1. But now e is not a function but rather a structure describing the lambda. Line 4 evaluates e, which displays the structure it references. Line 5 defines a variable f2 to be the result of apply eval to e. Eval runs the Scheme interpreter on its arguments. Recall, that e is a structure describing a lambda. So applying eval to e we get a lambda and so f2 is now a function described by this lambda. Finally, on lines 6 and 7 we apply f1 and f2 respectively to arguments 3 and 4. Since they are both functions that correspond to the same lambda structure we expect to get the same results and indeed we do.

Quoting in C#

With C# 3.0 we can do essentially the same thing (again line numbers added for discussion purposes).

1:    Func<int, int, int> f1 = (x, y) => x + y;
2:    Console.WriteLine(f1);
3:    Expression<Func<int, int, int>> e = (x, y) => x + y;
4:    Console.WriteLine(e);
5:    Func<int, int, int> f2 = e.Compile();
6:    Console.WriteLine(f1(3, 4));
7:    Console.WriteLine(f2(3, 4));

This program displays the following:

System.Linq.Func`3[System.Int32,System.Int32,System.Int32]
(x, y) => (x + y)
7
7

Notice how remarkably similar both the programs and that outputs are. Again, on line 1 we define a delegate to be the result of evaluating a lambda expression. On line 2, we display this delegate. On line 3, we assign e the result of evaluating a lambda expression. Notice that in C# both assignments (lines 1 and 3) look the same except for the type of the variable to which the lambda is assigned. Lambdas can be converted either to delegates or to expression trees depending on the usage. Delegates can be invoked while expression trees preserve the structure of the lambda. On line 4, we display the expression tree for e which corresponds to the structure of the lambda. On line 5, we do the equivalent of the eval in Scheme by compiling the expression tree structure to a delegate. We can now invoke f2 like we can invoke f1. On lines 6 and 7, we invoke these two delegates and display their results.

So What?

While this is all very remarkable, it may seem a bit esoteric. Why take the trouble to have a variable with a value that corresponds to the structure of a lambda and then convert that to an invocable form? Why not just always create the delegate immediately? The reason is with the expression tree form of the lambda, we can reason about the structure and then take action on it. Consider queries. When a query is translated it creates a number of lambdas, but one of the key questions is: Are those lambdas then converted to delegates or expression trees? For example:

var q = from x in foo
where x > 1
select x + 1;

This is translated to:

var q = foo.Where(x => x > 1)
.Select(x => x + 1);

We have already discussed that which "Where" and "Select" methods are called depends upon what the type of foo is and which extension methods are in scope. If the "Where" and "Select" methods take a delegate then the lambdas will be converted to delegates; however, if they take an expression tree then the lambdas are converted to expression trees instead. So a LINQ provider (or someone who has implemented the query pattern methods like "Where" and "Select") can choose whether they would like the lambdas in delegate or expression tree form. Now lets look at a few providers and see what they do. LINQ to objects (or the in-memory query operators that apply to IEnumerable<T>) take delegates. This is because they will invoke the delegates to do things like evaluate predicates or projections. However, LINQ to SQL on the other hand takes expression trees. It does not "invoke" the lambdas but instead reasons over the structure of the lambdas and then translates the expression trees to equivalent SQL code given a schema for a database. This is the beauty of the LINQ pattern.

Adding the ability to quote lambdas enables all sorts of possiblities. You might easily imagine other uses for expression trees: remoting evaluation of expressions, query planning, executing them on a GPU, ....

What can you think of doing?

Transparent Identifiers

wesdyer — Fri, 22 Dec 2006 22:53:41 GMT

My last post described in detail how query expressions in C# are translated, but I have a confession to make. I did a little bit of hand waving through one part of the translation rules. The astute reader (whomever that is, whenever I read such things I always wonder, "Am I that astute reader?") may have noticed that four of the rules introduced a decidedly foreign language feature into rewrite rules. For example, examine rule #14 closely:

14. From followed by let

from x1 in e1 let x2 = e2 ...
from * in e1.Select(x1 => new { x1, x2 = e2 }) ...

What is that asterisk in the resultant transformation and what does it mean? It isn't a pointer deferencing operator and it isn't a multiplication operator, hmmm...What could it be? It looks like it is taking the place of an identifier in the from clause but that is a strange name for an identifier.

In fact, the asterisk denotes a transparent identifier and it is the name of this identifier. Transparent identifiers are one of the strangest additions to C# 3.0. The spec has precious little to say about them.

1. They are introduced to a program only by query rewriting

2. Each transparent identifier has exactly one associated anonymous type (the anonymous type introduced during the same rewrite step as the transparent identifier)

3. When a transparent identifier is in scope, its members are in scope as well (transivitely)

Rule 3 is kind of strange because it mentions scope in an otherwise syntactic rewrite. Scope has to do with semantics and not syntax so we have to be very careful how we apply this rule. Now consider the following query:

from x in foo

let y = f(x)

let z = g(x, y)

select h(x, y, z)

Applying rule 14 the first time the query is reduced to:

from * in foo.Select(x => new { x, y = f(x) })

let z = g(x, y)

select h(x, y, z)

Here we introduced a transparent identifier and the anonymous type that is associated with it is new { x, y = f(x) }. Now we apply rule 14 again.

from * in foo.Select(x => new { x, y = f(x) }).Select(* => new { *, z = g(x, y) })

select h(x, y, z)

We have now introduced another transparent identifier but this one is associated with a different anonymous type. It is associated with new { *, z = g(x, y) }. In order to note the difference they will labeled them *1 and *2.

from *2 in foo.Select(x => new { x, y = f(x) }).Select(*1 => new { *1, z = g(x, y) })

select h(x, y, z)

Finally, we can apply rule 15.

foo.Select(x => new { x, y = f(x) }).Select(*1 => new { *1, z = g(x, y) }).Select(*2 => h(x, y, z))

Where *1 and *2 are associated with the following anonymous types:

*1 = new { x, y = f(x) }

*2 = new { *1, z = g(x, y) }

You may notice that in the final select call that there is a lambda with only one parameter (*2) but in the body of this lambda we reference x, y, and z but none of these variables are in scope. Using rule 3 about transparent identifiers we see that when *2 is in scope so are *1 and z but since *1 is now in scope so are x and y (the transitive closure of the members of *2). So really when we refer to x in h(x, y, z) we are really referring to *2.*1.x. Thus we have the following:

foo.Select(x => new { x, y = f(x) }).Select(*1 => new { *1, z = g(*1.x, *1.y) }).Select(*2 => h(*2.*1.x, *2.*1.y, *2.z))

Finally, once we realize that the compiler treats these transparent identifiers just as unspeakable compiler generated names then we see that all the magic has been removed:

foo.Select(x => new { x, y = f(x) }).Select(t0 => new { t0, z = g(t0.x, t0.y) }).Select(t1 => h(t1.t0.x, t1.t0.y, t1.z))

Voila! No more transparent identifiers. Transparent identifiers are used in query rewriting to package up the intermediate results and pass them onto the next clause. They are means of essentially creating a little scope and passing it around. It is a fantastic idea that allows variables to flow through the queries providing the kind of behavior that users expect to see.

Comprehending Comprehensions

wesdyer — Fri, 22 Dec 2006 01:35:00 GMT

Not long ago, I was reading through some articles posted on programming.reddit.com when I came across an article claiming that C# is trying to be a dynamic language. One user posted a comment that mentioned that C# 3.0 included among other things "embedded SQL". Unfortunately, it seems that there is still some confusion about what query expressions are. Let's be clear: query expressions are not embedded SQL. In fact, they are closer to list comprehensions in languages like Haskell and Python than they are to embedded SQL.

At a high level, query expressions provide nine basic operations (from, join, join...into, group...by, orderby, where, let, select, into). Each of these basic operations is expressed as a clause. The clauses are then chained together to provide a declaration of query; however, if you only view a query then you can't be sure what the behavior at runtime will be. The reason is that queries are translated into a series of method calls but the methods that are invoked are not predefined methods from the compiler's point of view. The compiler just creates the calls and binds the calls to whatever methods are applicable given the arguments and the receiver of the call. This is both the source of the beauty and the confusion of query expressions.

Syntactic Sugar

Query expressions are an interesting language feature because they are pure syntactic sugar. But to really understand queries we need to see what lies beneath. Let's break through the syntactic sugar.

Consider the following query:

   1:  var q = from x in foo

   2:          where x > 0

   3:          select x + 1;

This query is transformed into:

   1:  var q = foo.Where(x => x > 0)

   2:             .Select(x => x + 1);

But what "Where" and "Select" methods are being called? It is impossible to tell from this code snippet; it depends on what the type of "foo" is and which extension methods are in scope. This is the beauty of query expressions and why we can do neat things like LINQ to Objects, DLINQ, and LINQ to Entities and so on. Each of these providers can author their own "Where" and "Select" methods which would be called if foo was a DLINQ object or a LINQ to entities object. If you wanted the queries to behave differently all you need to do is define your own methods such that the compiler will bind to them when you write a query. The set of methods that needs to be implemented is called the "Query Pattern" and implementers of this pattern are called "Providers".

Transformation Rules

Their are (only ;) eighteen rules that define how queries are transformed. Here is how the transformations can be performed given that you have a query (each transformation lists the pattern that to match first and then the resultant transformation on the next line):

First remove query continuations.

1. If the query has a continuation rewrite it (only rewrite the first one right now)

from x1 in e1 ... into x2 ...

from x2 in (from x1 in e1 ...) ...

Now transform the query into a list of clauses and for each clause if it is explicit then make it implicit like rules 2-4 state:

2. Explicit from

from T x1 in e1

from x1 in e1.Cast<T>()

3. Explicit join

join T x1 in e1 on k1 equals k2

join x1 in e1.Cast<T>() on k1 equals k2

4. Explicit join into

join T x1 in e1 on k1 equals k2 into g

join x1 in e1.Cast<T>() on k1 equals k2 into g

Now we check for the degenerate query and rewrite it in a special way.

5. Now if the query is degenerate then do a special transform

from x1 in e1 select x1

e1.Select(x1 => x1)

Finally, now that we have a list of clauses (without explicit types) we simply apply rules 6-18 repeatedly until we don't have a query any more. Think of it as pattern matching. If the list of clauses matches the given pattern then reduce the list given the transformation rule. Rinse and Repeat.

6. Two froms followed by a select

from x1 in e1 from x2 in e2 select e3

e1.SelectMany(x1 => e2, (x1, x2) => e3)

7. Two froms followed by a non-select

from x1 in e1 from x2 in e2 ...

from * in e1.SelectMany(x1 => e2, (x1, x2) => new { x1, x2 }) ...

8. From followed by a join followed by a select

from x1 in e1 join x2 in e2 on k1 equals k2 select e3

e1.Join(e2, x1 => k1, x2 => k2, (x1, x2) => e3)

9. From followed by a join followed by a non-select

from x1 in e1 join x2 in e2 on k1 equals k2 ...

from * in e1.Join(e2, x1 => k1, x2 => k2, (x1, x2) => new { x1, x2 }) ...

10. From followed by a join with into followed by a select

from x1 in e1 join x2 in e2 on k1 equals k2 into g select e3

e1.GroupJoin(e2, x1 => k1, x2 => k2, (x1, g) => e3)

11. From followed by a join with into followed by a non-select

from x1 in e1 join x2 in e2 on k1 equals k2 into g ...

from * in e1.GroupJoin(e2, x1 => k1, x2 => k2, (x1, g) => new { x1, g }) ...

12. From followed by orderby (for some number of keys)

from x1 in e1 orderby k1, k2, k3 ...

from x1 in e1.OrderBy(x1 => k1).ThenBy(x1 => k2).ThenBy(x1 => k3) ...

13. From followed by where

from x1 in e1 where e2 ...

from x1 in e1.Where(x1 => e2) ...

14. From followed by let

from x1 in e1 let x2 = e2 ...

from * in e1.Select(x1 => new { x1, x2 = e2 }) ...

15. From followed by non-identity select

from x1 in e1 select e2

e1.Select(x1 => e2)

16. From followed by identity select

from x1 in e1 select x1

e1

17. From followed by non-identity group by

from x1 in e1 group e2 by e3

e1.GroupBy(x1 => e3, x1 => e2)

18. From followed by identity group by

from x1 in e1 group x1 by e2

e1.GroupBy(x1 => e2)

Example Translation

Consider the following query:

   1:  from x in foo

   2:  where x > 0

   3:  from y in bar

   4:  select x + y

To translate this query, we first remove check to see if rule 1 is applicable (no) and then we translate the query into a list of clauses while applying rules 2-4 (which don't apply to this query). After rules 1-4 we have the following list of clauses.

[from x in foo, where x > 0, from y in bar, select x + y]

Rule 5 does not apply because it is not the degenerate identity select. Now we will apply rules 6-18 repeatedly until the query is fully reduced. The rules are always applied to the front of the list and so the first rule that is applicable is rule 13. Which states:

from x1 in e1 where e2 ...

from x1 in e1.Where(x1 => e2) ...

Applying this rule the list becomes:

[from x in foo.Where(x > 0), from y in bar, select x + y]

Now the next rule that is applicable is rule 6 which states:

from x1 in e1 from x2 in e2 select e3

e1.SelectMany(x1 => e2, (x1, x2) => e3)

The list now becomes:

[foo.Where(x > 0).SelectMany(x => bar, (x, y) => x + y)]

We no longer have a query so we are done and the final rewrite looks like:

   1:  foo.Where(x => x > 0)

   2:     .SelectMany(x => bar, (x, y) => x + y)

Other queries are translated in a similar fashion. As you translate more of these, you will notice that the rules are in fact very simple and can be easily mechanically applied. With some practice, you should be able to translate them quickly in your head much like doing math. It is simply a process of applying the rules in the right order and reaching a fixed point.

You may have noticed that rule 1 actually creates a nested query. The rewrite rules as I explained them did not remove this nested query. After rewriting a query, recurse and rewrite queries found in its subexpressions. And that is it!

Implications

Interestingly, the C# spec does not specify much more than this about the semantics of query expressions; however, the rewrite rules do imply some significant details that may not be apparent. Some of these details include scoping rules, error message and debugging behavior, and even hidden new language features like transparent identifiers.

Happy Querying!

Types of confusion

wesdyer — Thu, 21 Dec 2006 00:06:00 GMT

When I began programming in my early teens, I was very excited to learn about programming and also excited to become a "real" programmer. I remember picking up a book from the library that purported to teach what real programmers were like. I absorbed the material and enjoyed every page. I can only remember a handful of things from that book which I now realize was not very serious material, but I do remember one quote: "Strong typing is for people with weak minds." I thought to myself, "I don't want to have a weak mind" and I didn't want people to think that I have a weak mind so I thought that I should definitely not embrace strong typing. But what was "strong typing" anyway, I wondered. After much reflection on my (brief) programming experience, I decided that strong typing was using the keyboard a lot. So I decided that people with weak minds wrote overly verbose code. I then applied my new found principle.

Later I had a similar experience when I began perusing large numbers of articles, blogs, and comments about programming and programming languages. I would routinely come across some fellow programmer who would make a statement like: "dynamic languages are better because you need to write so much less code" or "python is no good because it is weakly typed". Furthermore, in many cases it was claimed that some language was strongly typed by one advocate and another person would claim that the same language was weakly typed. It was obvious that my earlier misunderstanding was shared by many of my peers.

C# 3.0 extends and adds some wonderful language features focused on type inference. The same confusion about what makes a language "dynamic" or "strongly typed" surrounds language features like the "var" contextual keyword.

On Definitions

Benjamin Pierce, the author of Types and Programming Languages, wrote in an email, "I spent a few weeks... trying to sort out the terminology of "strongly typed," "statically typed," "safe," etc., and found it amazingly difficult.... The usage of these terms is so various as to render them almost useless."

Based on the apparent confusion, I think it is best to clarify what I mean by each of the following terms.

Type Checking - Verifying that code respects type constraints.
Statically Typed - Type checking occurs at compile time.
Dynamically Typed - Type checking occurs at run time.
Type Safe Language - A language which protects its own abstractions.
Type Unsafe Language - A language which is not type safe.
Strongly Typed and Weakly Typed - Depends on the author; The definitions are so many and so varied that the terms are practically useless. It seems that anyone can claim that language X is either strongly typed or weakly typed based on sound reasoning derived from one of the various definitions. Here are a few sources: Wikipedia, Reddit, Google Groups, and another one.
Dynamic Language - A language which enables runtime inspection or modification of a program; most languages can do this but dynamic languages make it easy. It is common for people to refer to "dynamic languages" and mean "dynamically typed languages" as the term is defined here.
Type Inference - Deducing the type of an expression based upon the eventual evalution of that expression. Type inference enables the compiler to deduce types without type annotations.

Based on these definitions we can tackle some common misconceptions.

#1 - Dynamic languages are dynamically typed

Depends. Admittedly, most people seem to use the two terms interchangeably. But given the definitions used above then they are not the same. It is true that many dynamic languages are dynamically typed but it is not a requirement. It also seems to be the case that dynamic language features tend to be more easily adapted to dynamically typed languages. However, C# can be claimed to be a dynamic language (or at least that it has dynamic language features) since it has reflection, runtime code generation (from libraries), EnC (tool based), and now it has expression tree (which can be created, type checked, and evaluated at runtime).

#2 - Languages that allow programming without type annotations are dynamically typed

False. Whether a language is dynamically typed or not depends on whether type checking happens at runtime or compile time. Languages with type inference for example do not require type annotations but the compiler infers them (as if a programmer had typed them) and then performs type checking at compile time for consistent usage.

#3 - Dynamically typed languages are more terse

False. Maybe, a claim could be made that dynamically typed languages tend to be more terse given languages like Python, Perl, and Lisp; however, the statically typed languages have some very terse representatives as well (Haskell, ML dialects).

#4 - Statically/Dynamically typed languages are better than Dynamically/Statically typed languages

False. It is hard to make this claim objectively since "better" is usually ill-defined. There are some very good languages that are statically typed and some very good languages that are dynamically typed. Define what "better" means here and then we can talk. Of course, your definition of "better" may be disputed (and often is).

#5 - A language is either dynamically or statically typed but not both

Depends. In fact, most statically typed languages have a few features that are dynamically typed. In C# there are several examples: array bounds checking and casting. If you access an element outside of the bounds of an array then the error occurs at runtime because the type checking for the validity of this operation occurrsat runtime. If you cast an object to a string when in fact it contained an int then the type system will complain at runtime. Despite this, a language is usually referred to as being either statically or dynamically typed.

The "var" contextual keyword

Now we are ready to look the new contextual keyword "var". It is a very handy device that enables the programmer to omit type declarations for variables that are initialized in their declaration. However, it should be carefully noted that this feature still uses static type checking. For example:

var x = true; // same as bool x = "foo"
var y = 5; // same as int y = 5
y += 1; // ok
var z = x + y; // error! cannot add bool and int, won't compile

Some people mistakenly confuse the semantics of the "var" contextual keyword with the "var" declarations used in javascript or the variant datatype in VB. They are very different. Javascript is dynamically typed so when an assignment is made to a variable there is no complaint about mismatched types. This is because each variable does not have a type per se but a value that is assigned to a variable has a type. So values of different types may be assigned to the same variable as long as usage of that value at runtime is consistent with the type. So in javascript we could do the following:

var x = "foo";
x = 5; // no error here, but if this were C# code then a compile error (not a runtime error) would occur

I love using the new C# 3.0 "var" syntax for declaring local variables. In fact, it is the C# 3.0 feature that I use most frequently. Here are several reasons to use "var":

1. In my opinion, good programming style dictates that variables should be initialized when they are declared anyway and the type can usually be deduced from the expression used to assign the variable
2. Specifying the type is often redundant
Foo foo = new Foo();
3. The programmer may not care what type a variable is as long as her usage of that variable type checks
SomeFileType file = GetFileAttributes(filename);
if (file.IsReadOnly) ...
4. The programmer may not be able to specify the name of the type
??? foo = new { X = "bar", Y = 3.14159 }; // anonymous type (type which has no name)
5. The type name may be very very very long
Dictionary<Pair<int,List<Set<double>>>, Pair<double,int>> map = new Dictionary<Pair<int,List<Set<double>>>, Pair<double,int>>();

Only #4 is a mandated use for "var", the other reasons are more asthetic. A natural follow-up question is when can't I or when shouldn't I use "var".

1. When the type of the initializing expression is not the type that you want the variable to be
var x = new Bar();
if (someCondition) x = new Foo(); // where Bar derives from Foo
var y = null; // this doesn't work because null doesn' thave a type, although you could do var y = default(string); instead
var z = (a, b) => a + b; // this doesn't work because lambdas do not have a type, they must be converted to a delegate type first
var y = 1;
y += 1.1; // error y is int, should have been declared double or possibly initialized with 1.0
2. When you want to explicitly state the type of a variable for human readability

I personally don't think #2 is very compelling especially since with IDE support you can easily hover over "var" and see the type for each variable. Furthermore, I think that some of the arguments used by the proponents of dynamically typed languages apply here (not that this is a dynamically typed language feature). The number of type annotations in a program can distract from the intent of the program. From this perspective, the "var" contextual keyword actually improves readability. But unlike a dynamically typed language, the compiler can still type check the program at compile time and provide feedback on possible errors.

The "var" contextual keyword is just one of several features in C# 3.0 that uses type inference. But those features will be discussed another day.

Saving the State of Enumerators

wesdyer — Fri, 13 Jan 2006 23:51:00 GMT

Cyrus and I were writing some code together the other day and we used an interesting data structure that I wanted to share with you. This data structure I will call a chain which is essentially a immutable singlely linked list.

interface IChain<T> : IEnumerable<T>
{
IChain<T> Next { get; }
T Value { get; }
}

It is possible to provide a lot of the necessary functionality through an abstract class.

abstract class AbstractChain<T> : IChain<T>
{
   public abstract IChain<T> Next { get; }
   public abstract T Value { get; }

   public IEnumerator<T> GetEnumerator()
   {
      IChain<T> current = this;
      while (current != null)
      {
         yield return current.Value;
         current = current.Next;
      }
   }

   System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
   {
      return this.GetEnumerator();
   }
}

Of course, a simple concrete Chain class has a very straightforward implementation.

class Chain<T> : AbstractChain<T>
{
   IChain<T> next;
   T value;

   public Chain(IChain<T> next, T value)
   {
      this.next = next;
      this.value = value;
   }

   public Chain(T value)
      : this(null, value)
   {
   }

   public override T Value
   {
      get { return value; }
   }

   public override IChain<T> Next
   {
      get { return next; }
   }
}

In this implementation, each node contains a value and every node but the last node has a link to the next node. The last node has a null instead of a link to the next node. But what if I don't want to create the whole list upfront? What if I know how to create it, but I don't want the overhead of creation and then traversal. The solution of course is to lazily create the linked list or create it on demand.

class LazyChain<T> : AbstractChain<T>
{
   IChain<T> next;
   T value;
   IEnumerator<T> enumerator;

   LazyChain(T value, IEnumerator<T> enumerator)
   {
      this.value = value;
      this.enumerator = enumerator;
   }

   public static IChain<T> Create(IEnumerable<T> enumerable)
   {
      return Create(enumerable.GetEnumerator());
   }

   static IChain<T> Create(IEnumerator<T> enumerator)
   {
      if (enumerator.MoveNext())
      {
         return new LazyChain<T>(enumerator.Current, enumerator);
      }
      return null;
   }

   public override IChain<T> Next
   {
      get
      {
         if (enumerator != null)
         {
            next = Create(enumerator);
            enumerator = null;
         }
         return next;
      }
   }

   public override T Value
   {
      get { return value; }
   }
}

Notice how the constructor to a lazy chain does not take a link to the next chain, it takes an enumerator. It uses this enumerator to create the next link when it is first accessed. One great application where this is useful is in rollbacking an enumerator. It is sometimes useful to be able to mark a place in the execution of an enumerator and then reset to that point and iterate again. The lazy chain makes such behavior possible and efficient. For an example, consider the following program.

class Program
{
   static IEnumerable<int> GetNumbers()
   {
      for (int i = 0; i < 10; ++i)
      {
         yield return i;
      }
   }
   static void Main(string[] args)
   {
      IChain<int> intChain = LazyChain<int>.Create(GetNumbers());
      IChain<int> savedChain = null;
      for (int i = 0; intChain != null; ++i)
      {
         if (intChain.Value == 5)
         {
            savedChain = intChain;
         }
         intChain = intChain.Next;
      }
      if (savedChain != null)
      {
         foreach (int i in savedChain)
         {
            Console.WriteLine(i);
         }
      }
   }
}

Although the iterator iterates through all of the numbers, we pick execution up again where the value was five and then proceed to print out the numbers. Furthermore, the code to iterate over the chains is ignorant of the fact of what kind of chain it is dealing with. While this example is a bit contrived, it does illustrate the ability of the data structure to mark and continue over enumerators while preserving immutability. For Cyrus and I, it greatly simplified the design of a component that we were working on while preserving performance.

captivated foreach statements

wesdyer — Thu, 06 Oct 2005 21:10:00 GMT

I am sure that most people who use C# regularly use the foreach construct. Intimately understanding this construct is not usually required for proper usage. The syntax is straightforward and it would seem that the semantics would also be straightforward; however, beginning in C# 2.0 there is a subtle gotcha that can cause seemingly odd behavior. Take this code for example:

using System;

class Program {
   private delegate void Proc();

   static void Main(string[] args) {
      int[] numbers = { 0, 1, 2, 3, 4, 5 };
      Proc[] actions = new Proc[numbers.Length];
      int current = 0;

      foreach (int number in numbers) {
         actions[current] = delegate { Console.WriteLine(number); };
         actions[current++]();
      }
   }
}

You would expect that this program would just list the numbers zero through five on sequential lines. This is exactly what the program does; however, what will this slight modification do to the output:

using System;

class Program {
   private delegate void Proc();

   static void Main(string[] args) {
      int[] numbers = { 0, 1, 2, 3, 4, 5 };
      Proc[] actions = new Proc[numbers.Length];
      int current = 0;

      foreach (int number in numbers) {
         actions[current++] = delegate { Console.WriteLine(number); };
      }

      foreach (Proc action in actions) {
         action();
      }
   }
}

So what is the output to this program. It just repeats the number five on six lines. Why? Well, there are two important concepts to grasp. First, how the foreach statement works and second how anonymous methods "capture" local variables. A quick glance at the ECMA spec tells us that the foreach statement is rewritten like this:

foreach (T item in collection) {
...statements...
}

becomes (if collection implements IEnumerable<T> other situations are similar)...

IEnumerator<T> enumerator = ((IEnumerable<T>)(collection)).GetEnumerator();
try {
   T element;
   while (enumerator.MoveNext()) {
      element = enumerator.Current;
      ...statements...
   }
} finally {
   enumerator.Dispose();
}

Note that the declaration of element is outside of the loop. It could have been in the loop and in most cases the placement does not matter, but the spec says that it is outside of the loop. Now we move onto anonymous methods.

Anonymous methods capture local variables by putting them on the heap instead of on the stack. A private nested class is generated by the compiler that contains the local variables that are captured. This class is then instantiated when the local variables come into scope. Since the CLR does not provide native support for anonymous methods, these methods are rewritten in constructs that do exist. Here is what the code looks like when the anonymous method and foreach statements are rewritten.

using System;
using System.Collections;

class Program {
   private delegate void Proc();

   static void Main(string[] args) {
      int[] numbers = { 0, 1, 2, 3, 4, 5 };
      Proc[] actions = new Proc[numbers.Length];
      int current = 0;

      IEnumerator enumerator1 = numbers.GetEnumerator();
      try {
         Locals1 locals1 = new Locals1();
         while (enumerator1.MoveNext()) {
            locals1.number = (int)enumerator1.Current;
            actions[current++] = new Proc(locals1.Method1);
         }
      } finally {
         IDisposable disposable = enumerator1 as IDisposable;
         if (disposable != null) disposable.Dispose();
      }

      IEnumerator enumerator2 = actions.GetEnumerator();
      try {
         Proc action;
         while (enumerator2.MoveNext()) {
            action = (Proc)enumerator2.Current;
            action();
         }
      } finally {
         IDisposable disposable = enumerator2 as IDisposable;
         if (disposable != null) disposable.Dispose();
      }
   }

   class Locals1 {
      public int number;

      public void Method1() {
         Console.WriteLine(number);
      }
   }
}

Since the locals are created outside of the loop, the same instance of locals1 is being used by all of the delegates. This leads to them all displaying the same value. How can we fix this? Here is how:

using System;

class Program {
   private delegate void Proc();

   static void Main(string[] args) {
      int[] numbers = { 0, 1, 2, 3, 4, 5 };
      Proc[] actions = new Proc[numbers.Length];
      int current = 0;

      foreach (int number in numbers) {
         int temp = number;
         actions[current++] = delegate { Console.WriteLine(temp); };
      }

      foreach (Proc action in actions) {
         action();
      }
   }
}

With our newfound knowledge, we can implement a parallelized version of the Map and ForEach methods. First, lets start with the serial versions.

using System;
using System.Collections.Generic;

class Program {
   delegate T Func<A0,T>(A0 arg0);
   delegate void Proc<A0>(A0 arg0);

   static IEnumerable<U> Map<T,U>(IEnumerable<T> list, Func<T,U> func) {
      foreach (T item in list) {
         yield return func(item);
      }
   }

   static void ForEach<T>(IEnumerable<T> list, Proc<T> proc) {
      foreach (T item in list) {
         proc(item);
      }
   }

   static void Main() {
      int[] numbers = { 0, 1, 2, 3, 4, 5 };
      Func<int,int> squareInt = delegate(int x) { return x * x; };
      Proc<int> displayInt = delegate(int x) { Console.WriteLine(x); };

      ForEach(Map(numbers, squareInt), displayInt);
   }
}

This displays the squares of the integer values zero through five inclusively. Now we can parallelize them.

using System;
using System.Threading;
using System.Collections.Generic;

class Program {
   delegate T Func<A0,T>(A0 arg0);
   delegate void Proc<A0>(A0 arg0);

   class ThreadAdapter<T,U> {
      Thread thread;
      T input;
      U output;
      Func<T,U> func;

      public ThreadAdapter(Func<T,U> func, T input) {
         this.input = input;
         this.func = func;
         thread = new Thread(new ThreadStart(Execute));
      }

      void Execute() {
         output = func(input);
      }

      public Thread Thread {
         get { return thread; }
      }

      public U Output {
         get { return output; }
      }
   }

   static IEnumerable<U> Map<T,U>(IEnumerable<T> list, Func<T,U> func) {
      List<ThreadAdapter<T,U>> threads = new List<ThreadAdapter<T,U>>();

      foreach (T item in list) {
         ThreadAdapter<T,U> thread = new ThreadAdapter<T,U>(func, item);
         threads.Add(thread);
         thread.Thread.Start();
      }

      foreach (ThreadAdapter<T,U> thread in threads) {
         thread.Thread.Join();
         yield return thread.Output;
      }
   }

   static void ForEach<T>(IEnumerable<T> list, Proc<T> proc) {
      List<Thread> threads = new List<Thread>();

      foreach (T item in list) {
         T temp = item;
         Thread thread = new Thread(new ThreadStart(delegate { proc(temp); }));
         threads.Add(thread);
         thread.Start();
      }

      foreach (Thread thread in threads) {
         thread.Join();
      }
   }

   static void Main() {
      int[] numbers = { 0, 1, 2, 3, 4, 5 };
      Func<int,int> squareInt = delegate(int x) { return x * x; };
      Proc<int> displayInt = delegate(int x) { Console.WriteLine(x); };

      ForEach(Map(numbers, squareInt), displayInt);
   }
}

Just for kicks, we might now write this code with the new C# 3.0 syntax.

using System;
using System.Query;
using System.Threading;
using System.Collections.Generic;

static class Extensions {
   class ThreadAdapter<T,U> {
      Thread thread;
      T input;
      U output;
      Func<T,U> func;

      public ThreadAdapter(Func<T,U> func, T input) {
         this.input = input;
         this.func = func;
         thread = new Thread(new ThreadStart(Execute));
      }

      void Execute() {
         output = func(input);
      }

      public Thread Thread {
         get { return thread; }
      }

      public U Output {
         get { return output; }
      }
   }

   public static IEnumerable<U> Map<T,U>(this IEnumerable<T> list, Func<T,U> func) {
      var threads = new List<ThreadAdapter<T,U>>();

      foreach (var item in list) {
         var thread = new ThreadAdapter<T,U>(func, item);
         threads.Add(thread);
         thread.Thread.Start();
      }

      foreach (var thread in threads) {
         thread.Thread.Join();
         yield return thread.Output;
      }
   }

   public static void ForEach<T>(this IEnumerable<T> list, Action<T> action) {
      var threads = new List<Thread>();

      foreach (var item in list) {
         var temp = item;
         var thread = new Thread(new ThreadStart(() => { action(temp); }));
         threads.Add(thread);
         thread.Start();
      }

      foreach (var thread in threads) {
         thread.Join();
      }
   }
}

class Program {
   static void Main() {
      var numbers = new [] { 0, 1, 2, 3, 4, 5 };

      numbers.Map(x => x * x).ForEach(x => { Console.WriteLine(x); });
   }
}

Enjoy!