Parallel Programming with .NET : ThreadPool

PDC09 Parallelism Session Videos Now Available

toub — Sat, 21 Nov 2009 09:05:20 GMT

Attendees at PDC09 this past week were privy to quite a few sessions on parallel computing. Now that the videos of these sessions are online, you can view them as well from the comfort of your own home. Here are some of the key parallelism-related sessions from this past week:

Overview

The State of Parallel Programming

Managed code in Visual Studio 2010

Native code in Visual Studio 2010

HPC Server

Accelerating Applications Using Windows HPC Server 2008

Research and Incubation

Enjoy!

Slides from Parallelism Tour

toub — Mon, 02 Nov 2009 02:58:00 GMT

Last week, I had the privilege of touring around Tennessee, Kentucky, Ohio, and Michigan, speaking about the new parallel computing support in Visual Studio 2010 and the .NET Framework 4. Many folks I spoke with were interested in getting a copy of the slide deck I used, so I’ve attached it to this blog post. Enjoy!

Task.Wait and “Inlining”

toub — Thu, 15 Oct 2009 15:45:59 GMT

“What does Task.Wait do?”

Simple question, right? At a high-level, yes, the method achieves what its name implies, preventing the current thread from making forward progress past the call to Wait until the target Task has completed, one way or another. If the Task ran to completion, Wait will return successfully. If the Task completed due to faulting or being canceled, it will throw an exception indicating the relevant details (since the waiting code likely expects that all work to be completed by the task ran successfully if Wait returns successfully).

The details are a bit more interesting. Wait could simply block on some synchronization primitive until the target Task completed, and in some cases that’s exactly what it does. But blocking threads is an expensive venture, in that a thread ties up a good chunk of system resources, and a blocked thread is dead weight until it’s able to continue executing useful work. Instead, Wait prefers to execute useful work rather than blocking, and it has useful work at its finger tips: the Task being waited on. If the Task being Wait’d on has already started execution, Wait has to block. However, if it hasn’t started executing, Wait may be able to pull the target task out of the scheduler to which it was queued and execute it inline on the current thread.

To do that, Wait asks for assistance from the target scheduler, by making a call to the TaskScheduler’s TryExecuteTaskInline method. TryExecuteTaskInline can do whatever logic the scheduler needs in order to validate that the current context is acceptable for inlining the Task. If it is, the scheduler tries to execute the task then and there as part of the call to TryExecuteTaskInline (using the base TaskScheduler’s TryExecuteTask method) and returns whether the Task could be executed. If it couldn’t be executed in TryExecuteTaskInline, the scheduler returns false, and Wait will need to block until the Task completes. A Task may not be able to be executed if it’s already been or is currently being executed elsewhere. As an example of this, the default scheduler (based on the ThreadPool) is aggressive about inlining, but only if the task can be efficiently removed from the data structures that hold tasks internally in the ThreadPool (such as if the task is living in the local queue associated with the thread attempting to inline it).

This is also very similar to what happens when Task.RunSynchronously is used to execute a Task (RunSynchronously may be used instead of Task.Start to run a Task currently in the TaskStatus.Created state). RunSynchronously results in the Framework calling the target scheduler’s TryExecuteTaskInline, allowing the scheduler to decide whether to inline or not. If the scheduler chooses not to inline, the Task will instead be queued to the scheduler, and the system will block until the Task is completed.

Of course, schedulers may want to behave differently depending on whether the code is explicitly choosing to inline (e.g. RunSynchronously) or whether the inlining is happening implicitly (e.g. Wait). To support this difference, in addition to accepting as a parameter the Task to be inlined, TryExecuteTaskInline also accepts as a parameter a Boolean taskWasPreviouslyQueued. This value will be false if it is known that the specified task has not yet been submitted to the scheduler, and true otherwise. If RunSynchronously is being used, taskWasPreviouslyQueued will be false; if Wait, it will be true. The default scheduler does, in fact, differentiate between these two cases, in that it always supports explicit inlining through RunSynchronously.

TaskScheduler.FromCurrentSynchronizationContext

toub — Tue, 22 Sep 2009 19:46:13 GMT

The Task abstractions in .NET 4 run on instances of the TaskScheduler class. Two implementations of TaskScheduler ship as part of the .NET Framework 4. The first is the default scheduler, which is integrated with the .NET 4 ThreadPool and takes advantage of its work-stealing queues. The second is the type of TaskScheduler returned from the static method TaskScheduler.FromCurrentSynchronizationContext.

According to MSDN, SynchronizationContext “provides the basic functionality for propagating a synchronization context in various synchronization models.” What does that really mean? At its core, SynchronizationContext provides two methods, Send and Post, both of which accept a delegate to be executed. Send synchronously invokes the delegate, and Post asynchronously invokes the delegate. That’s it, and the base implementation of SynchronizationContext doesn’t do anything fancier than that:

public virtual void Send(SendOrPostCallback d, object state)
{
d(state);
}

public virtual void Post(SendOrPostCallback d, object state)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(d.Invoke), state);
}

Where things get interesting is when new types are derived from SynchronizationContext, something typically done by a UI framework (though there are other non-UI framework implementations). For folks familiar with Windows Forms and Windows Presentation Framework development, you’re likely aware that UI controls should only be accessed by the thread that created them, almost always the main UI thread. Thus, if a thread doing work in the background wants to update something in the UI, it needs to marshal that work back to the GUI thread so that the controls may be accessed safely. Different UI frameworks expose different ways for accomplishing this marshaling. For example, in Windows Forms, one uses the Invoke or BeginInvoke method of the target Control (or at least a Control created on the same thread as the target Control). In WPF, one uses the target thread’s Dispatcher and corresponding Invoke/BeginInvoke methods. With every UI framework having its own model for marshaling work to a particular “synchronization context”, it becomes difficult to write code that supports this marshaling concept but which is agnostic to the particular environment that it’s in. Enter SynchronizationContext. A new type may be derived from SynchronizationContext such that its Send method synchronously marshals a delegate to the right thread for execution, and Post does the same but asynchronously. If you look at the implementations of the SynchronizationContexts provided by Windows Forms and WPF, that’s exactly what they do, delegating to the relevant Invoke/BeginInvoke methods from Send and Post to marshal the work correctly.

To make it easy to get at the right SynchronizationContext, a UI framework like Windows Forms will publish an instance of its SynchronizationContext-derived class to SynchronizationContext.Current. Code can then grab SynchronizationContext.Current and use it to marshal work, without having to know whether it’s being used from Windows Forms or Windows Presentation Foundation or another similar model.

“TaskScheduler.FromCurrentSynchronizationContext” should now make more sense. This method creates a TaskScheduler that wraps the SynchronizationContext returned from SynchronizationContext.Current. Thus, this gives you a TaskScheduler that will execute Tasks on the current SynchronizationContext. Why is that useful? It means you can create Tasks that are able to access UI controls safely, simply by running them on the right scheduler.

Let’s say that I wanted to load three images from some data source. When those images have been loaded, I want to blend them all together, and then I want to display the result into a PictureBox on my UI. Using Tasks and TaskScheduler.FromCurrentSynchronizationContext, I could write code like the following:

private void Button1_Click(…)
{
    var ui = TaskScheduler.FromCurrentSynchronizationContext();
    var tf = Task.Factory;

    // Load the three images asynchronously
    var imageOne = tf.StartNew(() => LoadFirstImage());
    var imageTwo = tf.StartNew(() => LoadSecondImage());
    var imageThree = tf.StartNew(() => LoadThirdImage());

    // When they’ve been loaded, blend them
    var blendedImage = tf.ContinueWhenAll(
        new [] { imageOne, imageTwo, imageThree }, _ =>
        BlendImages(imageOne.Result, imageTwo.Result, imageThree.Result));

    // When we’re done blending, display the blended image
    blendedImage.ContinueWith(_ =>
    {
        pictureBox1.Image = blendedImage.Result;
    }, ui);
}

This code runs three tasks to load the three input images asynchronously. When all of those have been loaded, again asynchronously some BlendImages method is used to blend the images, taking the three inputs and returning the blended image. Finally, once that’s done, another task is used to render the blended image by storing it into a PictureBox on the UI. Since this modifies a UI control, we need to do it from the UI thread. Thus, we pass a TaskScheduler to the ContinueWith method; this scheduler targets the UI’s SynchronizationContext, and will cause the Task to execute on the UI thread.

TaskScheduler.FromCurrentSynchronizationContext is provided for convenience and because this is a very common need. However, due to TaskScheduler’s extensibility, it’s actually possible to implement this behavior yourself, and in doing so you could modify it to suit your own needs however you see fit.

Let’s say you did want to develop a new SynchronizationContextTaskScheduler. It might look something like this:

public class SynchronizationContextTaskScheduler : TaskScheduler
{
    private ConcurrentQueue<Task> _tasks = new ConcurrentQueue<Task>();
    private SynchronizationContext _context;

    public SynchronizationContextTaskScheduler() :
        this(SynchronizationContext.Current) { }

    public SynchronizationContextTaskScheduler(
        SynchronizationContext context)
    {
        if (context == null) throw new ArgumentNullException("context");
        _context = context;
    }

    protected override void QueueTask(Task task)
    {
        // Add the task to the collection
        _tasks.Enqueue(task);

        // Queue up a delegate that will dequeue and execute a task
        _context.Post(delegate
        {
            Task toExecute;
            if (_tasks.TryDequeue(out toExecute)) TryExecuteTask(toExecute);
        }, null);
    }

    protected override bool TryExecuteTaskInline(
        Task task, bool taskWasPreviouslyQueued)
    {
        return SynchronizationContext.Current == _context &&
                  TryExecuteTask(task);
    }

    public override int MaximumConcurrencyLevel { get { return 1; } }

    protected override IEnumerable<Task> GetScheduledTasks()
    {
        return _tasks.ToArray();
    }
}

Not a lot of code for a fairly powerful thing.

The constructors simply accept the target SynchronizationContext and store it, also initializing a thread-safe queue that will store the tasks to be executed.
The QueueTask method is called whenever the system is providing a Task for this scheduler to execute: this scheduler handles it by storing that Task into a queue, and then Post’ing to the SynchronizationContext a delegate that will pull the next Task from the queue and execute it.
The TryExecuteTaskInline is invoked any time the system wants to run a Task inline on the current thread (either from a call to RunSynchronously or from a Wait attempt): we need to make sure that the call is coming from the same SynchronizationContext as the target, otherwise we may end up running in the wrong context.
We only intended to support SynchronizationContexts that represent a single thread of execution (that’s most common), so we return 1 from MaximumConcurrencyLevel.
And we want the debugger to be able to display tasks scheduled to this scheduler, so we override GetScheduledTasks to return an array of the tasks queued. (This is why we need to explicitly store the queue of tasks; otherwise, we could have simply relied on lambda closures to capture each task to be executed.)

TaskCreationOptions.PreferFairness

toub — Tue, 07 Jul 2009 19:08:38 GMT

One of the ways in which the Task Parallel Library achieves good performance is through “work-stealing”. Work-stealing is supported in the .NET 4 ThreadPool for access through the Task Parallel Library and its default scheduler. This manifests as every thread in the ThreadPool having its own queue for work; when that thread creates Tasks, by default these tasks are enqueued into the thread’s local queue, rather than into the global queue that a call to ThreadPool.QueueUserWorkItem would typically target. When a thread goes in search for work to be executed, it starts with its local queue, an operation which enable some extra efficiencies due to improved cache locality, minimized contention, and so forth. However, this logic also affects fairness.

A typical thread pool will have a single queue that maintains all of the work to be executed. When threads in the pool are ready to process another work item, they’ll dequeue the work from the head of the queue, and when new work arrives to be executed by the pool, it’ll be enqueued onto the queue’s tail. This provides a level of fairness between work items, in that work items that arrive first are more likely to be picked off and start executing first.

Work-stealing perturbs that fairness. Threads external to the pool could be queuing up work, but if the threads in the pool are also generating work, the work generated by the pool will be preferred over the other work items, based on the search order employed by the threads in the pool (which start searching for work first with their local queues, only proceeding on to the global queue and then on to other threads’ queues if work wasn’t available locally). This behavior is typically expected and even desired, in that if a work item being executed is generating more work, that generated work is typically considered part of the overall operation being processed, and thus it makes sense it would be preferred over other unrelated work. For example, imagine a quicksort operation, where each recursive sort invocation potentially results in several further recursive calls; those calls, which in a parallel implementation are likely individual tasks, are part of the all-up sort operation.

Still, there are some scenarios where this default behavior is inappropriate, where fairness should be maintained between particular work items generated by threads in the pool and work items generated by other threads. This is often the case for long chains of continuations, where the work generated isn’t considered part of the current work but rather a follow-on to the current work. In those cases, you may want that follow-on work to be prioritized in a fair manner with other work in the system. This is where TaskCreationOptions.PreferFairness can prove useful.

When a Task is scheduled to the default scheduler, the scheduler looks to see whether the current thread from which the Task is being queued is a ThreadPool thread that has its own local queue. If it isn’t, the work item will be queued to the global queue. If it is, the scheduler will also check whether the TaskCreationOptions value for the Task includes the PreferFairness flag, which is not on by default. If the flag is set, even if the thread does have its own local queue, the scheduler will still enqueue the Task to the global queue rather than to the local queue. In this manner, that Task will be considered fairly along with all other work items queued globally.

What was just described is the current implementation of the PreferFairness flag in the default scheduler. The implementation could of course change, but what won’t change is the purpose of the flag: by specifying PreferFairness, you’re telling the system that this Task shouldn’t be prioritized just because it’s coming from a local queue. You’re telling the system that you want the system to do its best to ensure that this Task is prioritized in a first-come, first-serve nature.

One other thing to note is that Task itself knows nothing of this flag; it’s just a flag set as an option on the Task. It’s the scheduler that decides how it wants to handle this particular option, just as with TaskCreationOptions.LongRunning. The default scheduler handles it as described above, but another scheduler (such as one you write) may do with this flag whatever it wants, including ignoring it. Hence the naming “Prefer” rather than something stricter like “Guarantee”.

Tasks and Unhandled Exceptions

toub — Mon, 01 Jun 2009 07:34:11 GMT

Prior to the .NET Framework 2.0, unhandled exceptions were largely ignored by the runtime. For example, if a work item queued to the ThreadPool threw an exception that went unhandled by that work item, the ThreadPool would eat that exception and continue on its merry way. Similarly, if a finalizer running on the finalizer thread threw an exception, the system would eat the exception and continue on executing other finalizers.

That changed for .NET 2.0. Exceptions are meant to indicate a problem, and automatically eating unhandled exceptions often hides significant errors and reliability problems in an application. As such, for .NET 2.0, this unhandled exception behavior was changed by default to more closely match how exceptions are handled by Windows in general: if an exception goes unhandled, the process comes crashing down. While this may seem harsh, allowing the process to crash typically makes it easier to catch an underlying problem, but more importantly, it prevents the application from continuing to hobble along in a potentially very bad state, with corrupted data, etc. Thus, since .NET 2.0, if a work item running on the ThreadPool throws an unhandled exception, by default the process crashes.

There’s a notable exemption to this behavior, even after 2.0: the Asynchronous Programming Model (APM) pattern. With the APM pattern, work is started asynchronously with a BeginXx method, and at some point later the results of the work are retrieved with a corresponding EndXx method. If the asynchronous work throws an exception, that exception is then propagated out of the call to the EndXx method when the EndXx method is invoked. This, of course, counts on the EndXx method being invoked. If the developer makes a mistake such that End is never invoked, any exception that occurred as part of the asynchronous invocation will likely go unnoticed, as would have happened in more situations in .NET 1.x.

With Tasks in .NET 4.0, we face a similar situation as does the APM pattern. A Task represents an asynchronous operation that may throw an unhandled exception, but unlike work items generated by ThreadPool.QueueUserWorkItem, a Task instance is used to later join with the asynchronous work. As such, any unhandled exceptions will be stored into the relevant Task instance, later thrown any time that Task is waited on (and also available through the Task’s Exception property). In fire-and-forget scenarios, where the developer has no intention of joining with the Task, or if the developer simply forgets to join with the Task, the exception may never be observed, and thus if we did nothing special, we’d be in a bad situation very much like that which the APM pattern faces with unhandled exceptions.

To address this, Tasks keep track of whether an unhandled exception has been “observed.” In this context, “observed” means that code has joined with the Task in some fashion in order to at least be made aware of the exception. This could be calling Wait/WaitAll on the Task. It could be checking the Task’s Exception property after the Task has completed. Or it could be using a Task<TResult>’s Result property. If a Task sees that its exception has been observed in some manner, life is good. If, however, all references to a Task are removed (making the Task available for garbage collection), and if its exception hasn’t yet been observed, the Task knows that its exception will never be observed. In such a case, the Task takes advantage of finalization, and uses a helper object to propagate the unhandled exception on the finalizer thread. With the behavior described earlier, that exception on the finalizer thread will go unhandled and invoke the default unhandled exception logic, which is to log the issue and crash the process.

In this manner, we can have our cake and eat it, too. As with the APM, we make any exceptions that occurred asynchronously available later for the app to retrieve. And as with work items on the ThreadPool, if an exception goes unhandled, it will cause the process to be torn down.

Of course, there may still be situations where you do want a fire-and-forget task, but where you want to automatically “observe” any unhandled exception, either to log it or something similar, or just to prevent the process from crashing in a situation where you know the exception will be benign. In that case, you can take advantage of continuations to address your needs.

Consider a Task t. After creating the Task, I can run code like the following:

t.ContinueWith(c => { var ignored = c.Exception; },
    TaskContinuationOptions.OnlyOnFaulted |
    TaskContinuationOptions.ExecuteSynchronously |
    TaskContinuationOptions.DetachedFromParent);

This code creates a continuation off of Task t that will only be scheduled if Task t completes in a Faulted state, meaning that it completed due to an exception going unhandled. When this continuation runs, it will observe the Task’s exception, preventing it from getting finalized. You could even wrap this kind of logic up into an extension method, such as:

public static Task IgnoreExceptions(this Task task)
{
    task.ContinueWith(c => { var ignored = c.Exception; },
        TaskContinuationOptions.OnlyOnFaulted |
        TaskContinuationOptions.ExecuteSynchronously |
        TaskContinuationOptions.DetachedFromParent);
    return task;
}

With this in hand, any time I create a fire-and-forget Task where I want an unhandled exception to be ignored, I can simply tag on IgnoreExceptions, e.g. instead of:

var t = Task.Factory.StartNew(…);

I could write:

var t = Task.Factory.StartNew(…).IgnoreExceptions();

Of course, the exception handling logic in Task exists for good reasons, to prevent unhandled exceptions from going unnoticed, and as such it’s not a great idea to liberally sprinkle use of an IgnoreExceptions extension method like this. But in some situations, it can be quite useful.

There’s a corollary to IgnoreExceptions which may also be useful. One of the downsides to the logic we use to tear down the process is that it relies on finalization. Finalization isn’t guaranteed to occur in a timely fashion, so an exception may go unhandled and it may be some time before the app then crashes; some time is better than never, but it’s still not ideal. If you want a more timely crash, you could use an extension method like the following:

public static Task FailFastOnException(this Task task)
{
    task.ContinueWith(c => Environment.FailFast(“Task faulted”, c.Exception),
        TaskContinuationOptions.OnlyOnFaulted |
        TaskContinuationOptions.ExecuteSynchronously |
        TaskContinuationOptions.DetachedFromParent);
    return task;
}

Rather than ignoring the exception, this uses Environment.FailFast to immediately crash the process. In this manner, I could write code like the following:

var t = Task.Factory.StartNew(…).FailFastOnException();

and as soon as Task t faults, the process will crash. This could be beneficial when debugging, as you’ll know immediately when something goes wrong. There are lots of variations on this as well. Debugger.Break could be used to immediately break into an attached debugger, you could raise an event of your choosing, the exception could be logged to an application log file, and so forth. In this manner, continuations are quite powerful.

(For anyone interested in extending Parallel Extensions with extension methods like this, the Parallel Extensions Extras project in the Beta 1 samples available at http://code.msdn.microsoft.com/ParExtSamples includes a plethora of interesting and useful examples. Enjoy!)

Concurrent, Multi-core Programming on Windows and .NET

toub — Tue, 28 Oct 2008 08:51:00 GMT

Thanks to everyone who attended our PDC pre-conference session yesterday on parallelism and concurrency! We had a wonderful turnout at the event, and David, Joe, and I all had a terrific time. Attached to this post are the slides we presented.

(It turns out that the PDC site does allow you to submit an evaluation for a precon. If you attended, we'd love your feedback.)

Q&As from the 2008 Financial Services Developer Conference

toub — Thu, 03 Apr 2008 18:27:00 GMT

A few weeks ago, I presented on Parallel Extensions to the .NET Framework at the 6th annual Microsoft Financial Services Developer Conference (the decks from the conference are now available online). I had a great time and a great audience, and during the presentation on Thursday I received some good questions. Here are some of them along with answers.

Question: This stuff looks really cool, but why do we need to modify our code to use Parallel Extensions; why can't you just automatically parallelize it for me?

Igor Ostrovsky, a developer on our team, has a nice set of explanations in his responses to some questions on the Parallel Extensions MSDN Forums:

"The way we traditionally express programs makes it very difficult for the compiler to understand the code at a high level, and parallelize the work. (A part of the problem is that understanding code at a high level is in some sense impossible - see the halting problem.) However, the user can help the parallel engine in various ways:

The user expresses the goal of the computation instead of the explicit steps to perform it (e.g. Parallel LINQ).

The user tells the parallel engine how to parallelize the computation (e.g. Task Parallel Library)

The user specifies certain invariants of the code (e.g. which parts are atomic - transactional memory)"

Question: Parallel Extensions looks like a great way to express concurrency in my applications, but how will I then debug my apps? Debugging parallel applications today is really hard... what are you doing about that?

It most certainly is hard, and we fully realize that developing software doesn’t begin and end with writing code. Tools are incredibly important in the life cycle of applications, especially when it comes to debugging and improving performance. This is never more true than when writing parallel applications, as concurrency introduces a whole host of issues that can be very difficult to find and fix. Alongside Parallel Extensions, the Parallel Computing Platform team is investing in the development of debugging and profiling tools that will increase developer productivity and improve the correctness and maintainability of parallel code. We haven't released previews of these tools yet, but you can expect to see them in the future.

Question: Do you need multiple cores or multiple processors to do multithreading? Is it possible to use Parallel Extensions on a single CPU machine, and if so, is there overhead in doing so?

Even with a single CPU, there are many places where multithreading can provide significant value. For example, in GUI programming on Windows, doing any significant work on the main GUI thread will cause user interaction with the application to suffer. Instead, it's better to offload work to a background thread, which can do processing while allowing the application to stay responsive. But when it comes to computationally-intensive tasks, multithreading on a single CPU can lead to decreased performance. Parallel Extensions seeks to minimize that decrease in performance. It can be used on a machine with a single CPU; the idea is that you write your code using Parallel Extensions correctly, and it will then help your application to scale from one to many cores. There is overhead to using Parallel Extensions (as there will be for pretty much any parallel framework), but we're working hard to minimize that overhead, and for many problems on multi-core machines, the overhead introduced will be greatly dwarfed by the parallel benefits received. On a single CPU, we can typically reduce the overhead to the point where for any problem that would have benefited from parallelism on a multi-core machine, the overhead will be negligible on a single core.

Question: Can one mix and match Parallel Extensions with existing concurrency mechanisms? For example, can I use the .NET ThreadPool in the same application as Parallel Extensions? Can I use monitors and mutexes and sempahores and reader-writer locks with Parallel Extensions just as I do with threads today?

Yes, to all of your questions. Parallel Extensions relies on threads, so you can continue to use existing synchronization primitives with Parallel Extensions just as you would with threads you spin up manually or with threads from the ThreadPool; we're also introducing some new synchronization and coordination primitives in upcoming releases of Parallel Extensions, so stay tuned for those. (Note, however, that there are some corner-case rules to be aware of. As an example, don’t evaluate futures or wait on tasks while holding a reentrant lock. Doing so isn't implicitly a problem, but it could lead to some issues that you'll need to be aware of; I'll dive into this further in a future blog post.)

Question: What is the relationship between HPC Server and Parallel Extensions? Does one replace the other?

Neither replaces the other, and in fact they're useful in combination. HPC Server is all about scaling out to nodes in a cluster of machines, and Parallel Extensions is all about scaling up on an individual machine. Just as you can use Parallel Extensions to parallelize your desktop applications, you can use Parallel Extensions to parallelize the .NET applications running on nodes in your cluster. This can be quite powerful when used in conjunction with HPC Server functionality like the new WCF-based SOA Service Broker.

Question: Is there any way to hook into the scheduler used by Parallel Extensions to automatically fan out the work to a cluster?

The scheduler in Parallel Extensions is not currently extensible in this way.

Question: Will the thread pool used by Parallel Extensions allow additional threads to be introduced when the number of cores increases?

Absolutely. When the threads in the pool are all being used to do useful work, the scheduler tries to maintain a ratio of one thread per core, so more cores typically translates to more threads. But in a variety of situations that ratio will not always be 1:1. In some cases, more threads may be used, such as if threads block (e.g. synchronization, I/O, etc.) or the user explicitly requests more threads through a TaskManagerPolicy. Similarly, a TaskManagerPolicy can be used to dial down the amount of available concurrency.

Question: Could you run out of memory if you have a large number of tasks?

Tasks are objects just like any other in .NET, and are subject to the same memory constraints as any other managed code. In the same vein, they're also managed by the CLR's garbage collector, such that completed tasks with no references left to them may be collected.

Question: Why will this be faster than using the ThreadPool?

There are a variety of reasons a work-stealing scheduler can perform better than a typical thread pool. For one, the scheduler in Parallel Extensions enables more scalable queue management in that it's not bottlenecked on a single queue storing all work items. Work items are distributed amongst multiple queues, ideally limiting contention on a single queue for all work; this becomes more and more important as the size of the work items decreases, as the number of threads contending for work increases, and as the number of cores running that work increases. These processor-local queues also enable lock-freedom, which contributes to the efficiency of retrieving work items and executing them. There are other significant benefits related to work-stealing for many types of problems, such as very efficient queueing mechanisms when creating tasks recursively. Moreover, the design can provide significant improvements to things like memory locality, which can greatly improve an algorithm's performance.

Question: If I'm testing my code on a single-core machine, can I simulate more cores?

To some extent. With the PLINQ programming model exposed in the CTP, you can provide a degree of parallelism value to the AsParallel extension method, which controls the number of threads that can be used to execute the query; by default on a single-core machine, this value will be 1, but you can explicitly make it larger. And for the Task Parallel Library, you can create new TaskManagers with specific TaskManagerPolicy instances that increase the number of threads that can be used beyond the number of cores on the machine. However, while it is possible to simulate it this way, your application won't be subject to some of the complexities of having multiple cores, and as a result some bugs in the parallel implementation may remain hidden or get hit with much less frequently.

Question: Does Parallel Extensions use threads or fibers?

Threads. Avoid using fibers in managed code: see http://www.bluebytesoftware.com/blog/PermaLink,guid,2d0038b5-7ba5-421f-860b-d9282a1211d3.aspx for more details.

Question: If I spin up a task, and it throws an exception, where will that exception bubble up?

Just as with the Asynchronous Programming Model pattern, where exceptions thrown during asynchronous execution are reraised when the EndXx method is called, exceptions thrown during the execution of a task will be reraised when the Task is waited on (e.g. task.Wait()).

Wrapping an APM implementation with Future

toub — Sun, 16 Mar 2008 22:59:32 GMT

In a previous post, I talked about implementing the Asynchronous Programming Model pattern using Future<T> from Parallel Extensions to the .NET Framework. It's also possible to go in the opposite direction, to create a Future<T> from an existing APM implementation.

As has been shown in previous examples, in this example we'll take advantage of the "promise" capabilities provided by Future<T>. This allows a Future<T> to be created without a Func<T> delegate, such that the future's Value and Exception properties can be set explicitly, but only once; any thread waiting on the Future<T> will block until one of these properties is set.

static Future<T> Create<T>(
    Action<AsyncCallback> beginFunc,
    Func<IAsyncResult, T> endFunc)
{
    var f = Future<T>.Create();
    beginFunc(iar => {
        try { f.Value = endFunc(iar); }
        catch (Exception e) { f.Exception = e; }
    });
    return f;
}

This Create<T> method first creates a Future<T> as a promise. It then calls a delegate provided by the caller to start the asynchronous action using the relevant BeginXx method; as an argument to that action, this code passes a delegate that will be called back by the asynchronous operation when it completes; the APM method will pass an IAsyncResult, which will then be passed in this callback to the endFunc delegate provided by the caller. That endFunc will call the EndXx method, and will use the resulting value or exception to set the Future<T>.

As an example of using this, consider a FileStream. FileStream exposes a BeginRead method:

IAsyncResult BeginRead(
byte[] array, int offset, int numBytes,
AsyncCallback userCallback, object stateObject)

as well as a corresponding EndRead method:

int EndRead(IAsyncResult asyncResult);

If we want to create a Future<T> that represents an asynchronous read operation on a FileStream, we could do so with our new Create method as follows:

var readFuture = Create<int>(
ac => fs.BeginRead(buffer, 0, buffer.Length, ac, null),
fs.EndRead);

I can now go off and do something else, and when I later get readFuture's Value, I'll block until the asynchronous operation completes (or return immediately with the result if it's already completed). If the FileStream was created to support asynchronous I/O operations and if the underlying version of Windows supports asynchronous I/O, while the read request is happening, no threads will be used, and yet I'll still have a valid Future<T> to represent the operation.

There are of course other possible designs for and implementations of such a Create method. An alternate implementation might look like this:

static Future<T> Create<T>(
    IAsyncResult iar, Func<IAsyncResult, T> endFunc)
{
    var f = Future<T>.Create();
    ThreadPool.RegisterWaitForSingleObject(
            iar.AsyncWaitHandle, delegate {
        try { f.Value = endFunc(iar); }
        catch (Exception e) { f.Exception = e; }
    }, null, -1, true);
    return f;
}

Here's I'm using the same promise functionality of Future<T>, but instead of using the callback functionality of the APM, I'm relying on ThreadPool's RegisterWaitForSingleObject functionality. When the provided IAsyncResult's AsyncWaitHandle is signaled (which will happen when the asynchronous operation completes), my delegate will be called to call endFunc and set the Future<T>'s Value or Exception property as shown previously. This version can be used as follows:

var readFuture = Create<int>(
fs.BeginRead(buffer, 0, buffer.Length, null, null),
fs.EndRead);

Now comes the interesting question...

We don't currently have a Create method like this on Future<T>. You can write your own as has been shown here, but is this important enough for us to include one on Future<T> itself? Do you have such a need in your applications? Should we provide deeper support to further enable this kind of integration? Let us know :)

Ordering the output of parallel computations

toub — Wed, 12 Mar 2008 07:52:00 GMT

Frequently when attempting to do multiple operations in parallel, ordering becomes an issue. Consider an application where I'm rendering and writing out to a video file frames of a movie:

for (int i = 0; i < numberOfFrames; i++)
{
var frame = GenerateFrame(i);
WriteToMovie(frame);
}

For a bit of pizzazz, I'll show the same thing with LINQ:

var frames = from i in Enumerable.Range(0, numberOfFrames)
select GenerateFrame(i);
foreach (var frame in frames) WriteToMovie(frame);

Now, my GenerateFrame method is expensive and computationally intensive, so I'd like to generate frames in parallel. For this example, we'll assume that my GenerateFrame method is thread-safe and can be called concurrently (rendering one frame doesn't modify any state used to render another frame), but access to my WriteToMovie method must be serialized:

using (ManualResetEvent mre = new ManualResetEvent(false))
{
    int count = numberOfFrames;
    object obj = new object();
    for (int i = 0; i < numberOfFrames; i++)
    {
        ThreadPool.QueueUserWorkItem(state =>
        {
            var frame = GenerateFrame((int)state);
            lock(obj) WriteToMovie(frame);

            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();

        }, state);
    }
    mre.WaitOne();
}

Unfortunately, that's quite a bit of code overhead, and it also suffers from a fundamental ordering problem: the frames may end up being written to the output movie file in an arbitrary order, based on when the frame generation completes. A version with this ordering issue fixed might look like the following:

var frames = new Bitmap[numberOfFrames];
var events = (from i in Enumerable.Range(0, numberOfFrames)
              select new ManualResetEvent(false)).ToArray();

for (int i = 0; i < numberOfFrames; i++)
{
    ThreadPool.QueueUserWorkItem(state =>
    {
        int frameNum = (int)state;
        frames[frameNum] = GenerateFrame(frameNum);
        events[frameNum].Set();
    }, i);
}

for (int i = 0; i < numberOfFrames; i++)
{
    events[i].WaitOne();
    WriteToMovie(frames[i]);
}

This has the general effect I want, but there are still some unfortunately overheads here (for example, I'm creating, setting, and waiting on a ManualResetEvent per frame). It's also verbose. Instead, I can use Future<T> from Parallel Extensions to solve the same problem:

var frames = (from i in Enumerable.Range(0, numberOfFrames)
select Future.Create(() => GenerateFrame(i))).
ToArray();
foreach (var frame in frames) WriteToMovie(frame.Value);

I love how close this is to the original LINQ version (and how much less code it is the than my previous ThreadPool sample). Rather than selecting GenerateFrame(i), I'm selecting Future.Create(() => GenerateFrame(i)), and rather than calling WriteToMovie(frame), I'm calling WriteToMovie(frame.Value). With those changes, the frames are now being computed in parallel (note I'm also using ToArray to calls the entire query to be enumerated), and they're being written out to the movie file in the correct order without having to do any explicit locking or coordination. Sweet!

If I wanted to, I could do the same thing without LINQ, tracking the collection of futures explicitly:

var frames = new Queue<Future<Bitmap>>();
for (int i = 0; i < numberOfFrames; i++)
{
var num = i;
frames.Enqueue(Future.Create(() => GenerateFrame(num)));
}
while (frames.Count > 0) WriteToMovie(frames.Dequeue().Value);

I could also use PLINQ:

var frames = from i in Enumerable.Range(0, numberOfFrames).
AsParallel(ParallelQueryOptions.PreserveOrdering)
select GenerateFrame(i);
foreach (var frame in frames) WriteToMovie(frame);

Three different approaches to solving the same problem with Parallel Extensions, and all of them requiring less code (and less complicated code) than my ThreadPool example. Just makes you smile, doesn't it? :)

Custom parallel looping constructs

toub — Wed, 27 Feb 2008 19:25:35 GMT

For those of you that have examined the internals of the Task Parallel Library in our December '07 CTP release, you've likely noticed that the methods on the System.Threading.Parallel type are implemented on top of System.Threading.Tasks.Task type, and that they do so taking advantage of Task's self-replicating functionality. The idea behind replication is that the Task will make just enough copies of itself as is necessary to ensure that all available processing cores are saturated with work. This is useful not only for the Parallel constructs we implement, but also for custom loop constructs that you might want to implement for your own needs. We're providing what we believe to be the most commonly desired constructs, but we're also quite aware that there will be plenty more than we can provide that will be useful.

As an example, we don't currently provide a parallel while construct, but one could easily be built on top of Task. Consider a typical while loop:

while (condition()) body();

Here, body is executed over and over while condition is true; when condition is false, the loop exits. We can create a parallel loop that behaves very much like this, but running on all available processing cores. How might we do so? First, let's try with the ThreadPool:

public static void ParallelWhile(
    Func<bool> condition, Action body)
{
    int n = Environment.ProcessorCount;
    int count = n;

    using (ManualResetEvent mre =
        new ManualResetEvent(false))
    {
        WaitCallback wc = delegate
        {
            while (condition()) body();
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        };

        for (int i = 0; i < n; i++)
        {
            if (i == n - 1) wc(null);
            else ThreadPool.QueueUserWorkItem(wc);
        }

        mre.WaitOne();
    }
}

Certainly not an infeasible amount of code to write, but there are some difficulties here, and the implementation is not as efficient as one might like. We can implement a similar construct with the Task Parallel Library:

public static void ParallelWhile(
    Func<bool> condition, Action body)
{
    Task.Create(delegate
    {
        while (condition()) body();
    },
    TaskCreationOptions.SelfReplicating).Wait();
}

Here, there's significantly less boilerplate necessary, and many of the potential performance issues with the previous example are addressed by the underlying library (or at least will be by the final release; there are performance issues with the current CTP, but that's all part of the beauty of a technology preview ;).

Now, neither of these completely duplicates the semantics of the sequential while loop, which might be fine, but you might also have special requirements for how you want the loop to function. As an example, in the sequential implementation, once condition returns false, no additional invocations of body will occur, even if another call to condition would return true; that's not the case with this parallel implementation. Imagine this were running on four threads: one thread could see condition return false and exit the loop, but other threads could see condition return true and continue on (e.g. the condition delegate randomly returns true or false). If those semantics were important to us, we could allow such information to be communicated between multiple iterations of the loop, at the expense of some volatile reads and writes:

private class VolatileBool { public volatile bool Value; }

public static void ParallelWhile(
    Func<bool> condition, Action body)
{
    VolatileBool shouldExit = new VolatileBool();
    Task.Create(delegate
    {
        while (!shouldExit.Value && condition()) body();
        shouldExit.Value = true;
    },
    TaskCreationOptions.SelfReplicating).Wait();
}

Now when one iteration sees condition return false, it'll signal to all other threads that they should stop as soon as possible, such that even if condition returned true, they should exit before starting their next iteration. As another example, if one iteration in the sequential implementation throws an exception, no additional iterations will start; while we can't mimic that entirely in the parallel implementation, as with the previous example we can at least signal to other threads that they should bail as soon as possible:

private class VolatileBool { public volatile bool Value; }

public static void ParallelWhile(
    Func<bool> condition, Action body)
{
    VolatileBool shouldExit = new VolatileBool();
    Task.Create(delegate
    {
        try
        {
            while (!shouldExit.Value && condition()) body();
        }
        finally { shouldExit.Value = true; }
    },
    TaskCreationOptions.SelfReplicating).Wait();
}

Again, not a lot of code, but a fairly complete implementation (a few things remain, like validating that the arguments to the method are not null).

When would a parallel while loop like this be useful? Consider a problem where you might use a heuristic such as simulated annealing to approximate the best result. You might have, say, five seconds to find the best result, and thus you might run the heuristic over and over as many times as you can within that five seconds looking for the best possible answer. With multiple cores, you could run this over and over in parallel, and for that a ParallelWhile construct like the one we just created would be very handy:

object monitor = new object();
double bestResult = 0;
DateTime endTime = DateTime.UtcNow.AddSeconds(5);
ParallelWhile(() => DateTime.UtcNow < endTime, delegate
{
double result = SimulatedAnnealing();
lock (monitor) if (result > bestResult) bestResult = result;
});

We can't provide all possible variations of looping constructs that all customers will be interested in. We do, however, want to provide the lower-level mechanisms that will making building custom parallel loop implementations easy. If you have comments or suggestions, we'd really love to hear them.

Task Parallel Library on Channel 9

toub — Wed, 20 Feb 2008 01:38:00 GMT

Charles from Channel 9 sat down with several of us from the Parallel Computing Platform team to discuss the Task Parallel Library component of Parallel Extensions. A video of the conversation is now available on Channel9: http://channel9.msdn.com/Showpost.aspx?postid=384229. We hope you like it, and as always, feedback is welcome and appreciated!

Recursion and Concurrency

toub — Thu, 31 Jan 2008 22:55:27 GMT

When teaching recursion in an introductory computer science course, one of the most common examples used involves a tree data structure. Trees are useful in this regard as they are simple and recursive in nature, with a tree's children also being trees, and allow for teaching different kinds of traversals (in-order, pre-order, post-order, level-order, and so forth). But when these introductory concepts meet multithreading, the concepts are no longer so simple, at least not with the tools available in mainstream languages today like C#, C++, and Java. Consider a simple Tree data structure:

class Tree<T>
{
public Tree<T> Left, Right; // children
public T Data; // data for this node
}

Let's say we want to execute an Action<T> for each datum stored in the tree, and let's assume I don't care about order (introducing parallelism could be questionable if I did). That's straightforward to do sequentially and recursively:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Process the current node, then the left, then the right
    action(tree.Data);
    Process(tree.Left, action);
    Process(tree.Right, action);
}

I could also do so without recursion by maintaining an explicit stack (or queue, or some other data structure with different ordering guarantees):

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;
    var toExplore = new Stack<Tree<T>>();

    // Start with the root node
    toExplore.Push(tree);
    while (toExplore.Count > 0)
    {
        // Grab the next node, process it, and push its children
        var current = toExplore.Pop();
        action(current.Data);
        if (current.Left != null)
            toExplore.Push(current.Left);
        if (current.Right != null)
            toExplore.Push(current.Right);
    }
}

Now, let's assume the action we're performing on each node of the tree is independent, relatively expensive and/or that the tree is relatively large, and as such we want to process the tree in parallel (we're of course also assuming that the action delegate is thread-safe), meaning that we want multiple threads each running the action delegate on distinct tree nodes. How do we do this with what we have in .NET today?

There are multiple approaches, some more valid than others. The first thing someone might try is to follow the original recursive implementation but using the ThreadPool, which could look something like this:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to prevent this method from
    // returning until its children have completed
    using (var mre = new ManualResetEvent(false))
    {
        // Process the left child asynchronously
        ThreadPool.QueueUserWorkItem(delegate
        {
            Process(tree.Left, action);
            mre.Set();
        });

        // Process current node and right child synchronously
        action(tree.Data);
        Process(tree.Right, action);

        // Wait for the left child
        mre.WaitOne();
    }
}

The idea behind this implementation is to, given a node, spin up a work item to process that node's left child in parallel with the current node, and then process the current node's data as well as its right child. Of course, I could be losing out on some parallelism here as I delay processing of the right child until I'm done processing the current data. So we modify it slightly:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to wait for the children
    using (var mre = new ManualResetEvent(false))
    {
        int count = 2;

        // Process the left child asynchronously
        ThreadPool.QueueUserWorkItem(delegate
        {
            Process(tree.Left, action);
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        });

        // Process the right child asynchronously
        ThreadPool.QueueUserWorkItem(delegate
        {
            Process(tree.Right, action);
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        });

        // Process the current node synchronously
        action(tree.Data);

        // Wait for the children
        mre.WaitOne();
    }
}

I've now fixed that issue, such that both the left and the right children could potentially be processed in parallel with the current node, but that was by far not the worst problem. For starters, I'm creating a ManualResetEvent for every node in the tree... that's expensive. ManualResetEvent is a thin wrapper around a Win32 kernel event primitive, so creating one of these things requires kernel transitions, as does setting and waiting on one. Next, every time I process a node, I block waiting for its children to complete. And as the processing of a node (all but the root) is happening on a thread from the ThreadPool, I'm blocking ThreadPool threads. If a ThreadPool thread gets blocked, the ThreadPool will need to inject additional threads in order to process the remaining work items, and thus this implementation will require approximately one thread from the pool per node in the tree. That's a lot of threads! And that carries with it some serious problems. By default, a thread in .NET has a megabyte of stack space committed for it, so each thread burns a megabyte of (virtual) memory. The ThreadPool also throttles the creation of additional threads, such that introducing a new thread (once the number of pool threads equals the number of processors) will take 500 ms. For a tree of 250 nodes, that means its processing will take close to 2 minutes, purely for the overhead of creating threads, nevermind the actual processing of the nodes. And worse, there is a maximum number of threads in the pool: in .NET, the ThreadPool has a limited number of threads, by default 25 per processor in .NET 1.x/2.0 and 250 per processor in .NET 2.0 SP1. If the pool reaches the maximum, no new threads will be created, and thus this implementation could deadlock. Parent nodes will be waiting for their child nodes to complete, but the child nodes can't be processed until their parent nodes complete and relinquish a thread from the pool to execute.

Obviously, we need a better implementation. Next up, we can walk the tree sequentially, queuing up a work item in the pool for each node in the tree. Here we take that approach based on the recursive implementation of a tree walk:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to wait for all of the nodes to complete
    using (var mre = new ManualResetEvent(false))
    {
        int count = 1;

        // Recursive delegate to walk the tree
        Action<Tree<T>> processNode = null;
        processNode = node =>
        {
            if (node == null) return;

            // Asynchronously run the action on the current node
            Interlocked.Increment(ref count);
            ThreadPool.QueueUserWorkItem(delegate
            {
                action(node.Data);
                if (Interlocked.Decrement(ref count) == 0)
                    mre.Set();
            });

            // Process the children
            processNode(node.Left);
            processNode(node.Right);
        };

        // Start off with the root node
        processNode(tree);

        // Signal that no more work items will be created
        if (Interlocked.Decrement(ref count) == 0) mre.Set();

        // Wait for all of the work to complete
        mre.WaitOne();
    }
}

This implementation is better from a resources perspective and won't lead to the same kind of deadlocks as in the previous example. We're still creating a work item for each node, but now the work item is simply to asynchronously execute the action on the node's data, whereas the tree walk itself is still happening sequentially on the main thread (taking advantage of a recursive delegate). To maintain blocking behavior similar to the previous examples (where the call to Process won't return until all of the work has completed) though not identical (the processing of children nodes don't block the parent), a counter is incremented before each work item is created and is decremented when each work item finishes; only when that counter reaches 0 will the main thread continue (note how we need to start the counter at 1 and then decrement it after all of the work items have been created... this is to ensure that a work item completing before all of the work has been kicked off doesn't set the event prematurely). We can take the same approach with a variant of the iterative implementation shown earlier:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to wait for all of the nodes
    // to complete
    using (var mre = new ManualResetEvent(false))
    {
        int count = 1;

        // Start with the root node
        var toExplore = new Stack<Tree<T>>();
        toExplore.Push(tree);

        // Process all of the nodes
        while (toExplore.Count > 0)
        {
            // Get the current node and and push its children
            var current = toExplore.Pop();
            if (current.Left != null)
                toExplore.Push(current.Left);
            if (current.Right != null)
                toExplore.Push(current.Right);

            // Asynchronously process the data
            Interlocked.Increment(ref count);
            ThreadPool.QueueUserWorkItem(delegate
            {
                action(current.Data);
                if (Interlocked.Decrement(ref count) == 0)
                    mre.Set();
            });
        }

        // Signal that no more work items will be created
        if (Interlocked.Decrement(ref count) == 0) mre.Set();

        // Wait for all work items to be completed
        mre.WaitOne();
    }
}

These approaches work by queueing a work item per node in the tree. There are, of course, other approaches. Rather than adding a work item per node, we can create N work items, and have each work item process approximately 1/Nth of the nodes in the tree, where N is the number of threads. For example, we could store all of the nodes into a list and then split that list into N pieces and have a work item process each piece:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Create a list of all nodes in the tree
    var nodes = new List<Tree<T>>();
    var toExplore = new Stack<Tree<T>>();
    toExplore.Push(tree);
    while (toExplore.Count > 0)
    {
        var current = toExplore.Pop();
        nodes.Add(current);
        if (current.Left != null)
            toExplore.Push(current.Left);
        if (current.Right != null)
            toExplore.Push(current.Right);
    }

    // Divide the list up into chunks
    int workItems = Environment.ProcessorCount;
    int chunkSize = Math.Max(
        nodes.Count / workItems, 1);
    int count = workItems;

    // Use an event to wait for all work items
    using (var mre = new ManualResetEvent(false))
    {
        // Each work item processes appx 1/Nth of the data items
        WaitCallback callback = state =>
        {
            int iteration = (int)state;
            int from = chunkSize * iteration;
            int to = iteration == workItems - 1 ?
                nodes.Count : chunkSize * (iteration + 1);

            while (from < to) action(nodes[from++].Data);

            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        };

        // The ThreadPool is used to process all but one of the
        // chunks; the current thread is used for that chunk,
        // rather than just blocking.
        for (int i = 0; i<workItems; i++)
        {
            if (i < workItems-1)
                ThreadPool.QueueUserWorkItem(callback, i);
            else
                callback(i);
        }

        // Wait for all work to complete
        mre.WaitOne();
    }
}

Rather than statically dividing the work up, I could also rewrite the code such that each of the work items themselves is responsible for pulling work to handle:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Get an enumerator for the tree
    IEnumerator<T> enumerator =
        GetNodes(tree).GetEnumerator();
    int workItems = Environment.ProcessorCount;
    int count = workItems;

    // Use an event to wait for all work items to complete
    using (var mre = new ManualResetEvent(false))
    {
        // Each work item will continually pull data from the
        // enumerator and process it until there is no more data
        // to process
        WaitCallback callback = delegate
        {
            while (true)
            {
                T data;
                lock (enumerator)
                {
                    if (!enumerator.MoveNext()) break;
                    data = enumerator.Current;
                }
                action(data);
            }
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        };

              // The ThreadPool is used to process all but one of the
        // chunks; the current thread is used for that chunk,
        // rather than just blocking.
        for (int i = 0; i < workItems; i++)
        {
            if (i < workItems-1)
                ThreadPool.QueueUserWorkItem(callback, i);
            else
                callback(i);
        }

        // Wait for all work to complete
        mre.WaitOne();
    }
}

// An enumerator for a tree
public static IEnumerable<T> GetNodes<T>(Tree<T> tree)
{
    if (tree != null)
    {
        yield return tree.Data;
        foreach (var data in GetNodes(tree.Left))
            yield return data;
        foreach (var data in GetNodes(tree.Right))
            yield return data;
    }
}

Here I use a C# iterator to create an enumerator for all of the nodes in the tree. A work item is queued for each processor, and each of those work items retrieves an item from the enumerator, processes it, and goes back for more. Since multiple threads are accessing the enumerator (which isn't thread-safe), retrieving an item from it must be done under lock. Unfortunately, that's expensive; a better implementation would shift to grabbing more items from the enumerator so that the lock could be taken fewer times.

There are still further possible approaches. For example, we could walk the tree until we get to nodes of a certain depth (such as the log of the number of cores) and then process all of the subtrees at that depth in parallel with each other.

All of these implementations also have potential locality issues. It's reasonable to assume (though certainly not guaranteed) that nodes in the tree near to each other were created near to each other in time and are likely near to each other in memory (this assumption may be more applicable to other recursive problems, such as ones to be discussed shortly). For best cache performance, we'd like to try to have nodes near each other processed by the same thread and near to each other in time so that cache hits are optimized. But all of these solutions have been, in effect, randomly assigning nodes to threads.

Once of the nice things about the tree walk example is that it's very simple; there are no pre- or post- processing steps necessary for a node, which means I don't actually need to block at the end of a parent in order to wait for its children as I did in some of the previous examples. The problem also allows me to start processing children of a node before I've finished processing its parent. Many recursive problems are not like that, however. Consider a typical recursive sort, like quick sort:

public static void Quicksort<T>(T[] arr, int left, int right)
    where T : IComparable<T>
{
    if (right > left)
    {
        int pivot = Partition(arr, left, right);
        Quicksort(arr, left, pivot - 1);
        Quicksort(arr, pivot + 1, right);
    }
}

Order here definitely makes a difference. If we think of the Partition step like the execution of the Action in the tree walk example, I'm not able to do the Partition in parallel with the processing of the children, since the processing of the children is dependent on the outcome of the call to Partition. That's a pretty easy problem to workaround of course; I just wouldn't start the children executing asynchronously until after the Partition step is complete. But now consider a recursive sort like merge sort:

public static void Mergesort<T>(T[] arr, int left, int right)
    where T : IComparable<T>
{
    if (right > left)
    {
        int mid = (right + left) / 2;
        Mergesort(arr, left, mid);
        Mergesort(arr, mid + 1, right);
        Merge(arr, left, mid + 1, right);
    }
}

The merge step at a particular node must occur after the recursive calls, which means in a parallel implementation either we'd need to block waiting for the two recursive calls to complete before starting the call to Merge, or we'd have to use a form of continuation-passing style to join on the two merge calls and have their completion start the execution of the merge operation. Either way, this kind of processing with the ThreadPool is incredibly tricky to implement efficiently.

One of our goals when designing Parallel Extensions to the .NET Framework was to make such recursive parallel operations much easier to implement, and much easier to implement efficiently.

Let's start with PLINQ and our tree walk. Taking the Tree<T> iterator we already implemented, processing the tree in parallel is now a cakewalk:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
if (tree == null) return;
GetNodes(tree).AsParallel().ForAll(action);
}

Under the covers, this is very much like our implementation where we created several work items, and each of those work items pulled items (under lock) from the enumerator. PLINQ is more efficient, however, and uses varying chunk sizes (rather than always pulling just one element at a time) in an attempt to minimize the number of times the lock must be taken, hopefully reducing contention on the lock. This, of course, as with the similar previous implementation is doing a sequential tree walk but running the actions in parallel. If we actually want to walk the tree in parallel, we can use the Task Parallel Library:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;
    Parallel.Do(
        () => action(tree.Data),
        () => Process(tree.Left, action),
        () => Process(tree.Right, action));
}

It's almost scary how simple that is. The Parallel.Do statement potentially runs the three provided operations in parallel, blocking until all three have completed. A really cool thing about this though is that it will attempt to reuse the current thread as much as possible, so we don't run into the same kinds of issues that we had with the ThreadPool where we were blocking a whole slew of ThreadPool threads. The underlying work-stealing nature of the implementation also goes a long way towards helping to address the cache locality issues we previously mentioned. If we want more control, we can dive down below the Parallel class and explicitly use Tasks:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;
    var t1 = Task.Create(
        delegate { Process(tree.Left, action); });
    var t2 = Task.Create(
        delegate { Process(tree.Right, action); });
    action(tree.Data);
    Task.WaitAll(new Task[] { t1, t2 });
}

The same techniques can apply to the quicksort implementation:

public static void Quicksort<T>(T[] arr, int left, int right)
    where T : IComparable<T>
{
    if (right > left)
    {
        int pivot = Partition(arr, left, right);
        Parallel.Do(
            () => Quicksort(arr, left, pivot - 1),
            () => Quicksort(arr, pivot + 1, right));
    }
}

as well as to the mergesort implementation:

public static void Mergesort<T>(
    T[] arr, int left, int right) where T : IComparable<T>
{
    if (right > left)
    {
        int mid = (right + left) / 2;
        Parallel.Do(
            () => Mergesort(arr, left, mid),
            () => Mergesort(arr, mid + 1, right));
        Merge(arr, left, mid + 1, right);
    }
}

I love how simple the parallelization of this code becomes. However, this code still has some issues. There is necessary overhead to an operation like Parallel.Do, which under the covers creates Tasks for the individual operations and waits on those Tasks; that overhead may not be a big deal if the work being done is significant, but for fine-grain concurrency that overhead can end up dominating the computation, potentially causing the parallel implementation to run slower than the sequential implementation. For now, the best way to work around that is to complicate the code slightly with thresholds. The idea behind a threshold is that you introduce enough parallelism to keep the system saturated, but once enough has been expressed, you switch over to a sequential implementation so as to avoid the extra overhead. With the tree walk, using a depth as a threshold might look like this:

private static void Process<T>(
    Tree<T> tree, Action<T> action, int depth)
{
    if (tree == null) return;
    if (depth > 5)
    {
        action(tree.Data);
        Process(tree.Left, action, depth + 1);
        Process(tree.Right, action, depth + 1);
    }
    else
    {
        Parallel.Do(
            () => action(tree.Data),
            () => Process(tree.Left, action, depth + 1),
            () => Process(tree.Right, action, depth + 1));
    }
}

I have both a sequential and a parallel implementation here, and I switch from one to the other based on how deep in the tree I am. Depth may not always be the right gating factor to use as a threshold, especially if the tree isn't balanced. We're working through ways to reduce the overhead we currently see for these operations in the early CTP we released in December, and we're testing out some approaches that may eliminate the need to use thresholds in certain cases. For now, though, as you're playing around with the bits, keep thresholds in mind. The same technique could be used with both sorting examples, as well.

All in all, parallelism is a very interesting problem when it comes to recursion, and we hope the Parallel Extensions to the .NET Framework go a long way towards making the merging of these two concepts significantly easier.