Parallel Programming with .NET : .NET 4.0

Updated Beta 2 samples for parallel programming

toub — Mon, 07 Dec 2009 03:59:00 GMT

We've refreshed our Beta 2 samples for parallel programming with the .NET Framework 4. Thanks to the gracious assistance of the fabulous Lisa Feigenbaum and others on the Visual Basic team, in this refresh the majority of the samples are now available not only in C# but also in Visual Basic. The samples are available for download at http://code.msdn.microsoft.com/ParExtSamples.

Enjoy!

What’s new in Beta 2 for the Task Parallel Library? (Part 2/3)

dashih — Tue, 27 Oct 2009 18:02:00 GMT

Last week, we talked about how TPL adopted a new, better cancellation model. Today, we’ll cover a change that makes Tasks Detached by Default, some ContinueWhenAll/Any Refactoring, and the handy UnobservedTaskException event.

Tasks are Detached by Default

In Beta 2, we have changed an important default. Tasks are now created as detached (instead of attached) if no options specify otherwise.

Let’s consider the following code to review the difference between attached and detached Tasks.

Task p = Task.Factory.StartNew(() =>

{

Task c = Task.Factory.StartNew(() =>

{

DoWork();

});

p.Wait();

In Beta 1 and before, since the default options are used, ‘c’ is created as a child Task of Task ‘p’, the parent Task; we refer to this as Task ‘c’ being “attached” to Task ‘p’. This means that the p.Wait() statement will not return until the call to DoWork completes, because parent Tasks do not complete until all of their child Tasks complete. To opt out of this behavior, a user needs to create ‘c’ with the DetachedFromParent option:

Task c = Task.Factory.StartNew(() =>

{

DoWork();

}, TaskCreationOptions.DetachedFromParent);

The original code shown behaves differently in Beta 2. Now, by default, ‘c’ is not related to ‘p’ (it’s “detached” by default), and the p.Wait() statement will return as soon as ‘p’ completes, regardless of the status of Task ‘c’ and thus regardless of when DoWork returns. To opt in to the parent/child relationship, a user needs to create ‘c’ with the AttachedToParent option:

Task c = Task.Factory.StartNew(() =>

{

DoWork();

}, TaskCreationOptions.AttachedToParent);

Here is a summary of the changes:

· Removed the DetachedFromParent option

· Added the AttachedToParent option

· Changed the default behavior so that Tasks do not enlist in parent/child relationships when no options are specified.

There were a number of reasons why we decided that detached is the correct default and to move forward with this change, including:

· Many users were using attached Tasks unknowingly. The vast majority of the time, users create Tasks for simple, asynchronous work. In such scenarios, parent/child relationships (and the implicit waiting) are not needed. We found through many interactions that folks were just going with the default options and were accidentally opting in to this behavior. In the best case, this would only result in a slight performance cost. In the worst case, this would bring with it incorrect behavior that would lead to difficult to diagnose errors.

· Easier migration from ThreadPool.QueueUserWorkItem. Tasks are now the recommended way to queue work to the ThreadPool, but the easiest way to create Tasks resulted in different behavior from QueueUserWorkItem (where there’s no concept of parent/child work items). This change makes Task.Factory.StartNew (with no options) a true replacement for QueueUserWorkItem.

· Additional behavior should be opt-in and pay-for-play. Almost everything in TPL that results in additional behavior is opt-in, e.g. cancellation, LongRunning, PreferFairness. With the Beta 1 default, users opt-out of parent/child relationships. In Beta 2, users opt-in, making it consistent. This makes the extra functionality provided by parent/child relationships pay-for-play, such that you don’t pay the cost for parents implicitly waiting for their children or for exceptions propagating from children to parents unless you need that functionality.

ContinueWhenAny/All Refactoring

We have refactored the set of ContinueWhenAny and ContinueWhenAll overloads to make things more intuitive, consistent, and complete.

To demonstrate the main issue, let’s consider the following overload that was provided in Beta 1.

public class TaskFactory<TResult>

{

public Task<TNewResult> ContinueWhenAny(

Task<TResult>[] tasks,

Func<Task<TResult>, TNewResult> continuationFunction);

}

This confused the meaning of TaskFactory<TResult>, which is meant to create tasks of type Task<TResult>. However, with these overloads, TaskFactory<TResult> could be used to create tasks of type Task<TNewResult>. As an example, consider the code:

Task<int>[] taskOfInts = ...;
Task<string> t = Task<int>.Factory.ContinueWhenAll(taskOfInts, _ => “”);

This compiles and works just fine, but the type parameter mismatch (shown in bold) is certainly odd. To address this, we changed a bunch of overloads, so that instead of taking Task<TResult>s and returning a Task<TNewResult>, they take Task<TAntecedentResult>s and return Task<TResult>s. For example, the overload that replaced the above is:

public Task<TResult> ContinueWhenAny<TAntecedentResult>(

Task<TAntecedentResult>[] tasks,

Func<Task<TAntecedentResult>, TResult> continuationFunction);

And the above example becomes:

Task<int>[] taskOfInts = ...;
Task<string> t = Task<string>.Factory.ContinueWhenAll(taskOfInts, _ => “”);

In addition to this change, we also added, removed, or modified a number of other overloads to make the set consistent and complete. Now, the entire set of ContinueWhenAll and ContinueWhenAny overloads follow these clear rules:

· A TaskFactory creates Tasks, but also provides overloads to create Task<TResult>s.

· A TaskFactory<TResult> only ever creates Task<TResult>s (never Tasks or Task<TNewResult>s).

UnobservedTaskException event

We’ve added an event that fires for every Task exception that goes unobserved. Recall that to “observe” a Task’s exceptions, you must either Wait on the Task or access its Exception property after it has completed. At least one of these actions must be done before the Task object is garbage collected, or its exceptions will propagate (currently this occurs on the finalizer thread).

The new static event resides on the TaskScheduler class, and subscribing to it is straightforward. Here’s an example to log all unobserved exceptions and mark them as observed (preventing them from being propagated).

TaskScheduler.UnobservedTaskException +=

(object sender, UnobservedTaskExceptionEventArgs exceptionArgs) =>

{

exceptionArgs.SetObserved();

LogException(exceptionArgs.Exception);

};

Some customers have complained that TPL’s exception policy is too strict. The UnobservedTaskException event provides an easy way out by allowing you to simply squash all Task exceptions in an application (though using it in this manner is not recommended). The primary reason that we made the addition was to support host-plugin scenarios where a host application can still be perfectly useful in the presence of some truly harmless exceptions (thrown by buggy plugins). These scenarios may be achieved using the UnobservedTaskException event in conjunction with AppDomains to sandbox plugins. Look for a future post that describes this in more detail!

We’re done for now! The 3^rd and final post of this series will cover the new Unwrap APIs, a Parallel namespace change, and some changes under the covers.

What’s new in Beta 2 for the Task Parallel Library? (Part 1/3)

dashih — Mon, 19 Oct 2009 19:42:00 GMT

Visual Studio 2010 and .NET 4 Beta 2 is here! In terms of completeness and readiness for production coding, Beta 2 promises to be much better than Beta 1, and TPL is one component that delivers significant improvements over what was previously available. To get you excited about it, this series of posts details key additions and changes for Beta 2. Enjoy!

In this post, we’re talking about cancellation. A few months ago, our flurry of “What’s new in Beta 1” posts hinted at a new cancellation model and promised more information about it. We made good on that promise with .NET 4 Cancellation Framework, and this post goes further to explain how TPL, specifically, has fully adopted the new model.

The Old Way (Beta 1 and before)

Let’s consider the following code to review the old TPL cancellation model.

Task myTask = Task.Factory.StartNew(() =>

{

for (; ; )

{

if (Task.Current.IsCancellationRequested)

{

Task.Current.AcknowledgeCancellation();

return;

}

...

}

});

// Elsewhere.

myTask.Cancel();

myTask just loops infinitely, checking its IsCancellationRequested property to see if it has been canceled. Elsewhere, cancellation is requested on myTask using the Cancel method. At that point, myTask agrees to get canceled by calling its AcknowledgeCancellation method and returning. This is all necessary, because Task cancellation is cooperative; to enter the Canceled state, outside logic must request cancellation and the Task must acknowledge that cancellation request. However, note that cooperative cancellation is only relevant for already running Tasks. If a Task’s Cancel method is called before it is in the Running state, it will transition directly into the Canceled state.

Waiting on a canceled Task results in a TaskCanceledException wrapped in an AggregateException, hence the try/catch block. Executing this code prints “Canceled” to the console, indicating that myTask was successfully canceled.

This approach works, but we identified a number of problems with it including the following:

1. The cancellation model required exposing a Task.Current static property. This leaks implementation details from libraries that utilize Tasks internally; any code called from the Task can muck with the current Task or take dependencies on its existence. Imagine calling into 3^rd party code and having that code cancel your Tasks, schedule continuations off of them, etc.

2. Anyone with a Task’s reference can request cancellation on it. In many scenarios, it is valuable to separate the ability to check for cancellation and the ability to actually request cancellation.

3. From a cancellation perspective, Tasks didn’t compose well with other APIs that were cancelable, such as executing a cancelable PLINQ query inside of a Task.

4. Tasks were often more expensive than they needed to be, due to needing to track extra cancellation state per Task.

The New Way (Beta 2 and beyond)

The new TPL cancellation model is centered around two types: CancellationTokenSource and CancellationToken. You can read up about them in the post linked to above, but here is a simplified overview of their APIs:

namespace System.Threading
{
    public sealed class CancellationTokenSource
    {
        public void Cancel();
        public CancellationToken Token { get; }
        ...
    }

    public struct CancellationToken
    {
        public Boolean IsCancellationRequested { get; }
        public void ThrowIfCancellationRequested();
        ...
    }
}

Here’s the gist. A CancellationTokenSource contains a CancellationToken, and it can request cancellation on that token using a Cancel method. A CancellationToken can only check if cancellation has been requested on it. Ignore the ThrowIfCancellationRequested method for now; we’ll see why it’s handy later.

Adopting the new model involved not only adding support for these two types, but also ripping out the old model. Here’s a summary of the changes:

· All cancellation-related APIs on the Task class were removed (Cancel, AcknowledgeCancellation, IsCancellationRequested, etc)

· Other APIs that were no longer relevant were removed (Task.Current, Task.Parent, TaskCreationOptions.RespectParentCancellation, etc)

· Overloads that accept CancellationToken were added to many methods (StartNew, ContinueWith, etc)

And now, here’s a table that outlines how achieving cancellation in TPL has changed:

Action	Old Model	New Model

To set up cancellation	Just create a Task	Create a CancellationTokenSource and pass its Token to an API that creates a Task.
To check if cancellation has been requested	Check the IsCancellationRequested property on the relevant Task	Check the IsCancellationRequested property on the CancellationToken that was passed to the API that created the Task
To acknowledge cancellation	Check to ensure that IsCancellationRequested is true, then call the AcknowledgeCancellation() method on the relevant Task	Throw an OperationCanceledException with the task’s CancellationToken
To cancel a tree of tasks	Create all of the tasks as attached tasks and with the RespectParentCancellation flag set, then cancel the root task.	Pass the same CancellationToken to all Tasks, then cancel the associated CancellationTokenSource

To see how this all works, let’s rewrite the code above for Beta 2.

CancellationTokenSource cts = new CancellationTokenSource();

CancellationToken token = cts.Token;

Task myTask = Task.Factory.StartNew(() =>

{

for (; ; )

{

token.ThrowIfCancellationRequested();

...

}

}, token);

// Elsewhere.

cts.Cancel();

A CancellationTokenSource (cts) is initialized, and a CancellationToken (token) is initialized to cts’s Token. myTask still loops indefinitely, but now it calls ThrowIfCancellationRequested. This method just checks the IsCancellationRequested property on a CancellationToken. If true, it throws an OperationCanceledException(token), which is the way to acknowledge cancellation in the new model. Elsewhere, Cancel is called on cts to request cancellation on myTask.

The new model addresses all of the problems with the old model listed above:

1. No more Task.Current (Task.Parent was removed too, as it could be used to get at the current Task).

2. The ability to check for cancellation requests may now be separated from the ability to request cancellation. If only the former is desired, grant access to the CancellationToken only.

3. TPL now uses the new .NET 4 cancellation types. You can use the same CancellationToken{Source} to effect cancellation on multiple Tasks, Parallel.For calls, PLINQ queries, and more, all at once. This provides both better composition and better performance.

Finally, there are other advantages to this model. One is that it is much easier to organize cancellation sets within a large nest of Tasks; just create the right number of tokens and pass them around accordingly. Before, this scenario would have required tedious bookkeeping of many Task references and/or careful architecting of cancellation chains using TaskCreationOptions.RespectParentCancellation.

That’s it for now. The next post in the series will discuss a change regarding Tasks and parent/child relationships.

.NET 4 Beta 2 is here!

toub — Mon, 19 Oct 2009 18:50:05 GMT

The .NET Framework 4 Beta 2 is now available!

MSDN Subscribers can download it today, and it will be generally available for download on Wednesday. More information is available at http://msdn.microsoft.com/en-us/vstudio/dd582936.aspx. Additionally, one of the really exciting things about this Beta release is that it’s “go-live”; more information on that is available at “Going live” with Visual Studio 2010 Beta 2.

As part of .NET 4 Beta 2, of course, comes improved parallelism support. In the coming days and weeks, we’ll be blogging here about some of the exciting updates and additions in this release. Stay tuned…

In the meantime, the MSDN documentation on .NET 4’s parallelism support has been updated. It’s a great place to start learning about all of the functionality available in this release.

You can also learn more about the tools in Visual Studio 2010 for parallel development in the documentation at:

Enjoy!

Parallel Computing Presentations in Michigan, Ohio, Kentucky, and Tennessee

toub — Fri, 16 Oct 2009 15:49:29 GMT

In a week, I’m going to be traveling through Michigan, Ohio, Kentucky, and Tennessee, speaking about parallel computing, Visual Studio 2010, and .NET 4, primarily at corporations during the day and at user groups in the evenings.

If you’re in the area and interested, please do attend, and I look forward to meeting you! A list of events is available at http://blogs.msdn.com/jennifer/archive/2009/10/14/stephen-toub-parallel-computing-tour.aspx.

See you in a week!

Stephen

Parallelized Map and Filter Operations

toub — Mon, 12 Oct 2009 16:03:38 GMT

Common operations like map and filter are available in parallelized form through PLINQ, though the names differ. A map can be achieved with PLINQ’s Select operator, and a filter with PLINQ’s Where operator.

For example, I could implement a ParallelMap operation that takes in one array and returns another as follows:

public static TOutput [] ParallelMap<TInput,TOutput>(
TInput [] input, Func<TInput,TOutput> map)
{
return input.AsParallel().Select(map).ToArray();
}

and a ParallelFilter operation could be implemented in a similar fashion, based on PLINQ’s Where operator:

public static TInput [] ParallelFilter<TInput>(
TInput [] input, Func<TInput,Boolean> filter)
{
return input.AsParallel().Where(filter).ToArray();
}

In general, PLINQ is the recommended way to achieve such parallelization. Not only is it useful for individual operations like this, but it’s also a great way to compose operations. For example, if I wanted to write a ParallelFilterAndMap, I could achieve that as follows:

public static TOutput [] ParallelFilterAndMap<TInput,TOutput>(
TInput [] input, Func<TInput,Boolean> filter, Func<TInput,TOutput> map)
{
return input.AsParallel().Where(filter).Select(map).ToArray();
}

It is also possible to build such operations on top of Parallel.*. For example, here’s ParallelMap re-implemented in terms of Parallel.For:

public static TOutput [] ParallelMap<TInput,TOutput>(
    TInput [] input, Func<TInput,TOutput> map)
{
    var output = new TOutput[input.Length];
    Parallel.For(0, input.Length, i => output[i] = map(input[i]));
    return output;
}

and ParallelFilterAndMap re-implemented in terms of Parallel.ForEach and ConcurrentBag<T>:

public static TInput [] ParallelFilterAndMap<TInput,TOutput>(
    TInput [] input, Func<TInput,Boolean> filter, Func<TInput,TOutput> map)
{
    var output = new ConcurrentBag<T>();
    Parallel.ForEach(input, item =>
    {
        if (filter(item)) output.Add(map(item));
    });
    return output.ToArray();
}

As mentioned, in most cases you’re better off using PLINQ for these kinds of operations. However, there may be cases where a Parallel.*-based version is best. For example, let’s say that you wanted to implement an in-place parallel map operation, e.g. a map operation that maps from TData to TData, and that stores the new output into the input array. PLINQ doesn’t modify the input data source, so if you wanted that functionality, you could build it on top of Parallel.For:

public static void ParallelMap<TData>(
TData [] data, Func<TData,TData> map)
{
Parallel.For(0, data.Length, i => data[i] = map(data[i]));
}

There are a few other subtle differences that you could take advantage of by building on Parallel.For/ForEach instead of on PLINQ. For example, PLINQ uses a static number of threads for the processing of a given query (which defaults to Environment.ProcessorCount), whereas the number of threads used by Parallel.* may vary over time as influenced by the ThreadPool’s thread injection and retirement logic. Parallel.* can also be targeted to a specific and custom TaskScheduler, something not supported by PLINQ in .NET 4 (PLINQ always runs on TaskScheduler.Default, which is based on the .NET 4 ThreadPool).

Happy coding.

TaskScheduler.FromCurrentSynchronizationContext

toub — Tue, 22 Sep 2009 19:46:13 GMT

The Task abstractions in .NET 4 run on instances of the TaskScheduler class. Two implementations of TaskScheduler ship as part of the .NET Framework 4. The first is the default scheduler, which is integrated with the .NET 4 ThreadPool and takes advantage of its work-stealing queues. The second is the type of TaskScheduler returned from the static method TaskScheduler.FromCurrentSynchronizationContext.

According to MSDN, SynchronizationContext “provides the basic functionality for propagating a synchronization context in various synchronization models.” What does that really mean? At its core, SynchronizationContext provides two methods, Send and Post, both of which accept a delegate to be executed. Send synchronously invokes the delegate, and Post asynchronously invokes the delegate. That’s it, and the base implementation of SynchronizationContext doesn’t do anything fancier than that:

public virtual void Send(SendOrPostCallback d, object state)
{
d(state);
}

public virtual void Post(SendOrPostCallback d, object state)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(d.Invoke), state);
}

Where things get interesting is when new types are derived from SynchronizationContext, something typically done by a UI framework (though there are other non-UI framework implementations). For folks familiar with Windows Forms and Windows Presentation Framework development, you’re likely aware that UI controls should only be accessed by the thread that created them, almost always the main UI thread. Thus, if a thread doing work in the background wants to update something in the UI, it needs to marshal that work back to the GUI thread so that the controls may be accessed safely. Different UI frameworks expose different ways for accomplishing this marshaling. For example, in Windows Forms, one uses the Invoke or BeginInvoke method of the target Control (or at least a Control created on the same thread as the target Control). In WPF, one uses the target thread’s Dispatcher and corresponding Invoke/BeginInvoke methods. With every UI framework having its own model for marshaling work to a particular “synchronization context”, it becomes difficult to write code that supports this marshaling concept but which is agnostic to the particular environment that it’s in. Enter SynchronizationContext. A new type may be derived from SynchronizationContext such that its Send method synchronously marshals a delegate to the right thread for execution, and Post does the same but asynchronously. If you look at the implementations of the SynchronizationContexts provided by Windows Forms and WPF, that’s exactly what they do, delegating to the relevant Invoke/BeginInvoke methods from Send and Post to marshal the work correctly.

To make it easy to get at the right SynchronizationContext, a UI framework like Windows Forms will publish an instance of its SynchronizationContext-derived class to SynchronizationContext.Current. Code can then grab SynchronizationContext.Current and use it to marshal work, without having to know whether it’s being used from Windows Forms or Windows Presentation Foundation or another similar model.

“TaskScheduler.FromCurrentSynchronizationContext” should now make more sense. This method creates a TaskScheduler that wraps the SynchronizationContext returned from SynchronizationContext.Current. Thus, this gives you a TaskScheduler that will execute Tasks on the current SynchronizationContext. Why is that useful? It means you can create Tasks that are able to access UI controls safely, simply by running them on the right scheduler.

Let’s say that I wanted to load three images from some data source. When those images have been loaded, I want to blend them all together, and then I want to display the result into a PictureBox on my UI. Using Tasks and TaskScheduler.FromCurrentSynchronizationContext, I could write code like the following:

private void Button1_Click(…)
{
    var ui = TaskScheduler.FromCurrentSynchronizationContext();
    var tf = Task.Factory;

    // Load the three images asynchronously
    var imageOne = tf.StartNew(() => LoadFirstImage());
    var imageTwo = tf.StartNew(() => LoadSecondImage());
    var imageThree = tf.StartNew(() => LoadThirdImage());

    // When they’ve been loaded, blend them
    var blendedImage = tf.ContinueWhenAll(
        new [] { imageOne, imageTwo, imageThree }, _ =>
        BlendImages(imageOne.Result, imageTwo.Result, imageThree.Result));

    // When we’re done blending, display the blended image
    blendedImage.ContinueWith(_ =>
    {
        pictureBox1.Image = blendedImage.Result;
    }, ui);
}

This code runs three tasks to load the three input images asynchronously. When all of those have been loaded, again asynchronously some BlendImages method is used to blend the images, taking the three inputs and returning the blended image. Finally, once that’s done, another task is used to render the blended image by storing it into a PictureBox on the UI. Since this modifies a UI control, we need to do it from the UI thread. Thus, we pass a TaskScheduler to the ContinueWith method; this scheduler targets the UI’s SynchronizationContext, and will cause the Task to execute on the UI thread.

TaskScheduler.FromCurrentSynchronizationContext is provided for convenience and because this is a very common need. However, due to TaskScheduler’s extensibility, it’s actually possible to implement this behavior yourself, and in doing so you could modify it to suit your own needs however you see fit.

Let’s say you did want to develop a new SynchronizationContextTaskScheduler. It might look something like this:

public class SynchronizationContextTaskScheduler : TaskScheduler
{
    private ConcurrentQueue<Task> _tasks = new ConcurrentQueue<Task>();
    private SynchronizationContext _context;

    public SynchronizationContextTaskScheduler() :
        this(SynchronizationContext.Current) { }

    public SynchronizationContextTaskScheduler(
        SynchronizationContext context)
    {
        if (context == null) throw new ArgumentNullException("context");
        _context = context;
    }

    protected override void QueueTask(Task task)
    {
        // Add the task to the collection
        _tasks.Enqueue(task);

        // Queue up a delegate that will dequeue and execute a task
        _context.Post(delegate
        {
            Task toExecute;
            if (_tasks.TryDequeue(out toExecute)) TryExecuteTask(toExecute);
        }, null);
    }

    protected override bool TryExecuteTaskInline(
        Task task, bool taskWasPreviouslyQueued)
    {
        return SynchronizationContext.Current == _context &&
                  TryExecuteTask(task);
    }

    public override int MaximumConcurrencyLevel { get { return 1; } }

    protected override IEnumerable<Task> GetScheduledTasks()
    {
        return _tasks.ToArray();
    }
}

Not a lot of code for a fairly powerful thing.

The constructors simply accept the target SynchronizationContext and store it, also initializing a thread-safe queue that will store the tasks to be executed.
The QueueTask method is called whenever the system is providing a Task for this scheduler to execute: this scheduler handles it by storing that Task into a queue, and then Post’ing to the SynchronizationContext a delegate that will pull the next Task from the queue and execute it.
The TryExecuteTaskInline is invoked any time the system wants to run a Task inline on the current thread (either from a call to RunSynchronously or from a Wait attempt): we need to make sure that the call is coming from the same SynchronizationContext as the target, otherwise we may end up running in the wrong context.
We only intended to support SynchronizationContexts that represent a single thread of execution (that’s most common), so we return 1 from MaximumConcurrencyLevel.
And we want the debugger to be able to display tasks scheduled to this scheduler, so we override GetScheduledTasks to return an array of the tasks queued. (This is why we need to explicitly store the queue of tasks; otherwise, we could have simply relied on lambda closures to capture each task to be executed.)

The meaning of TaskStatus

toub — Sun, 30 Aug 2009 10:35:40 GMT

Every System.Threading.Tasks.Task instance goes through a lifecycle, and it only makes this journey once. To provide insight into where in that lifecycle a given Task is, the Task class provides an instance Status property. That property returns a value from the TaskStatus enumeration that reflects the current point in the lifecycle.

“Current” is the operative word here. Many of the states are transient, meaning that we’re dealing with a concurrent system, and by the time you’ve received the value of the status, the status may have changed. However, the lifecycle does not contain cycles (as ironic as that may sound), meaning that there’s a directionality to the various states, and once a given state has been reached, the Task won’t return back to that state again or to any that came before it. Additionally, there are three “final” states, where once in those states, the Task is no longer executing and will no longer change state.

Here are the states and an informal description of what they imply:

TaskStatus Enumeration Value	Informal Description
Created	This is the starting state for tasks created through a Task constructor. Tasks in this state will not leave the state until Start or RunSynchronously is called on the instance or until the Task is canceled.
WaitingForActivation	This is the starting state for tasks created through methods like ContinueWith, ContinueWhenAll, ContinueWhenAny, and FromAsync, as well as from a TaskCompletionSource<TResult>. The task isn’t currently scheduled, and won’t be until some dependent operation has completed (some tasks may never be scheduled, such as those created by a TaskCompletionSource<TResult> that have nothing relevant to be scheduled). For example, a task created by ContinueWith won’t be scheduled until the antecedent task (the one off of which ContinueWith was called) completes execution.
WaitingToRun	The task has been scheduled to a TaskScheduler and is waiting to be picked up by that scheduler and run. This is the starting state for tasks created through TaskFactory.StartNew; by the time the Task is returned from StartNew, it will already have been scheduled, and thus the state will be at least WaitingToRun (“at least”, since by the time StartNew returns, the Task could of course have already started or even completed executing).
Running	The Task is currently executing.
WaitingForChildrenToComplete	A Task isn’t considered complete until its attached children have completed. If a Task has finished executing its code body, it will leave the Running state, and if it’s then implicitly waiting for its children to complete, it will enter this state.
RanToCompletion	One of the three final states. A Task in this state has successfully completed execution, running to the end of its body without cancellation and without throwing an unhandled exception.
Canceled	One of the three final states. A Task in this state completed execution, but it did so through cancellation. To end in the Canceled state, a Task must either have cancellation requested prior to starting execution, or it must acknowledge a cancellation request during its execution.
Faulted	One of the three final states. A Task in this state completed execution due to an unhandled exception in its body or due to one of its attached children completing in this state.

Several of these status values are also communicated through helper properties on the Task class. For a Task t:

t.IsCanceled returns true if and only if t.Status equals TaskStatus.Canceled.
t.IsFaulted returns true if and only if t.Status equals TaskStatus.Faulted. Additionally, t.Exception will be non-null if and only if t.IsFaulted is true.
t.IsCompleted returns true if and only if t.Status equals TaskStatus.RanToCompletion, TaskStatus.Canceled, or TaskStatus.Faulted (i.e. one of the three final states). Note that IsCompleted does not just correspond to RanToCompletion, but rather to all final states.

Several operations in the Task Parallel Library are tied to the concept of a final state, or more generally to IsCompleted. For example, a call to the Wait method on a Task will not return until that Task’s IsCompleted is true. And the continuations for a Task will not fire until that same condition is met.

Parallel Extensions and I/O

dashih — Wed, 05 Aug 2009 00:55:00 GMT

In this post, we’ll investigate some ways that Parallel Extensions can be used to introduce parallelism and asynchrony to I/O scenarios.

Here’s a simple scenario. I want to retrieve data from a number of web resources.

static string[] Resources = new string[]

{

"http://www.microsoft.com", "http://www.msdn.com",

"http://www.msn.com", "http://www.bing.com"

};

Using the WebClient class, I might end up with the following.

var data = new List<byte[]>();

var wc = new WebClient();

foreach (string resource in Resources)

{

data.Add(wc.DownloadData(resource));

}

// Use the data.

However, these days, downloading data from the web usually utilizes only a small fraction of my available bandwidth. So there are potential performance gains here, and with TPL’s parallel ForEach loop, they are easily had.

var data = new ConcurrentBag<byte[]>();

Parallel.ForEach(Resources, resource =>

{

data.Add((new WebClient()).DownloadData(resource));

});

// Use the data.

Note that WebClient instances do not support multiple pending asynchronous operations (and the class is not thread-safe), so I need a separate instance for each operation. Also, since the normal BCL collections (List<T>, etc.) are not thread-safe, I need something like ConcurrentBag<T> to store the results. Of course, storing all the data in a collection assumes the scenario requires that all retrieval operations complete before processing. If this was not the case, I could start processing each data chunk right after obtaining it right in the loop, exploiting more parallelism. However, for the purposes of this investigation, I wanted to determine the possible performance gains in the absence of CPU-intensive work.

As it turns out, the above often yields linear speedup against sequential, with some variation due to the inconsistent nature of web site response times. And it was pretty straightforward. However, things would have been even easier had I started out with a “LINQ” frame of mind. First, I can convert my original sequential code to a LINQ query. Then, I can turn it into PLINQ using the AsParallel method and use WithDegreeOfParallelism to control the number of concurrent retrievals.

var data =

from resource in Resources

.AsParallel()

.WithDegreeOfParallelism(numConcurrentRetrievals)

select (new WebClient()).DownloadData(resource);

// Sometime later...

foreach (byte[] result in data) { }

(As an aside, it’s worth noting that WithDegreeOfParallelism causes PLINQ to use exactly numConcurrentRetrivals Tasks. This differs from the MaxDegreeOfParallelism option that I could have used with my previous Parallel.ForEach code, because that option sets the maximum; the actual number of threads still depends on the ThreadPool’s thread-adjusting logic.)

This code offers enhanced readability and makes storing the data easier. In addition, I can continue on the main thread, as PLINQ queries do not execute until the data they represent is accessed – that is, when MoveNext is called on the relevant enumerator. However, in this particular case, I don’t want to delay my query’s execution until I need the data; I actually want to execute my query while continuing on the main thread. To do so, I can wrap my query in a Task and force its immediate execution using ToArray.

var t = Task.Factory.StartNew(() =>

{

return

from resource in Resources

.AsParallel()

.WithDegreeOfParallelism(numConcurrentRetrievals)

select (new WebClient()).DownloadData(resource).ToArray();

});

// Sometime later...

foreach (byte[] result in t.Result) { }

// OR, use a continuation

t.ContinueWith(dataTask =>

{

foreach (byte[] result in dataTask.Result) { }

});

Now, I’ve got asynchrony, and I still get similar speedup. However, there’s still something about this code that is not ideal. The work (sending off download requests and blocking) requires almost no CPU, but it is being done by ThreadPool threads since I’m using the default scheduler. Ideally, threads should only be used for CPU-bound work (when there’s actually work to do). Of course, this probably won’t matter much for most typical client applications, but in scenarios where resources are tight, it could be a serious issue. Therefore, it’s worth investigating how we might reduce the number of blocked threads, perhaps by not using threads at all where possible.

To achieve this, I’ll be using ideas from a previous post: Tasks and the Event-based Asynchronous Pattern. That article explained how to create a Task<TResult> from any type that implements the EAP, and it presented an extension method for WebClient (available along with many others in the ParallelExtensionsExtras):

public static Task<byte[]> DownloadDataTask(

this WebClient webClient, Uri address);

The key point is that this method produces a Task<TResult> by integrating WebClient’s EAP implementation with a TaskCompletionSource<TResult>, and I can use it to rewrite my scenario.

var tasks = new Queue<Task<byte[]>>();

foreach (string resource in Resources)

{

WebClient wc = new WebClient();

tasks.Enqueue(wc.DownloadDataTask(new Uri(resource)));

}

// Sometime later...

while (tasks.Count > 0)

{

byte[] result = tasks.Dequeue().Result;

}

// OR, use a continuation

Task<byte[]>.Factory.ContinueWhenAll(tasks.ToArray(), dataTasks =>

{

foreach (var dataTask in dataTasks)

{

byte[] result = dataTask.Result;

}

});

With this, I’ve got a solution that uses parallelism for speed-up, is asynchronous, and does not burn more threads than necessary!

To recap, in this post, we considered a typical I/O scenario. First, we saw how easy it was to arrive at solutions that are better than the sequential one. Then, we delved deeper to discover a more complex solution (integrating EAP with Tasks) that offers even more benefits.

TaskCreationOptions.PreferFairness

toub — Tue, 07 Jul 2009 19:08:38 GMT

One of the ways in which the Task Parallel Library achieves good performance is through “work-stealing”. Work-stealing is supported in the .NET 4 ThreadPool for access through the Task Parallel Library and its default scheduler. This manifests as every thread in the ThreadPool having its own queue for work; when that thread creates Tasks, by default these tasks are enqueued into the thread’s local queue, rather than into the global queue that a call to ThreadPool.QueueUserWorkItem would typically target. When a thread goes in search for work to be executed, it starts with its local queue, an operation which enable some extra efficiencies due to improved cache locality, minimized contention, and so forth. However, this logic also affects fairness.

A typical thread pool will have a single queue that maintains all of the work to be executed. When threads in the pool are ready to process another work item, they’ll dequeue the work from the head of the queue, and when new work arrives to be executed by the pool, it’ll be enqueued onto the queue’s tail. This provides a level of fairness between work items, in that work items that arrive first are more likely to be picked off and start executing first.

Work-stealing perturbs that fairness. Threads external to the pool could be queuing up work, but if the threads in the pool are also generating work, the work generated by the pool will be preferred over the other work items, based on the search order employed by the threads in the pool (which start searching for work first with their local queues, only proceeding on to the global queue and then on to other threads’ queues if work wasn’t available locally). This behavior is typically expected and even desired, in that if a work item being executed is generating more work, that generated work is typically considered part of the overall operation being processed, and thus it makes sense it would be preferred over other unrelated work. For example, imagine a quicksort operation, where each recursive sort invocation potentially results in several further recursive calls; those calls, which in a parallel implementation are likely individual tasks, are part of the all-up sort operation.

Still, there are some scenarios where this default behavior is inappropriate, where fairness should be maintained between particular work items generated by threads in the pool and work items generated by other threads. This is often the case for long chains of continuations, where the work generated isn’t considered part of the current work but rather a follow-on to the current work. In those cases, you may want that follow-on work to be prioritized in a fair manner with other work in the system. This is where TaskCreationOptions.PreferFairness can prove useful.

When a Task is scheduled to the default scheduler, the scheduler looks to see whether the current thread from which the Task is being queued is a ThreadPool thread that has its own local queue. If it isn’t, the work item will be queued to the global queue. If it is, the scheduler will also check whether the TaskCreationOptions value for the Task includes the PreferFairness flag, which is not on by default. If the flag is set, even if the thread does have its own local queue, the scheduler will still enqueue the Task to the global queue rather than to the local queue. In this manner, that Task will be considered fairly along with all other work items queued globally.

What was just described is the current implementation of the PreferFairness flag in the default scheduler. The implementation could of course change, but what won’t change is the purpose of the flag: by specifying PreferFairness, you’re telling the system that this Task shouldn’t be prioritized just because it’s coming from a local queue. You’re telling the system that you want the system to do its best to ensure that this Task is prioritized in a first-come, first-serve nature.

One other thing to note is that Task itself knows nothing of this flag; it’s just a flag set as an option on the Task. It’s the scheduler that decides how it wants to handle this particular option, just as with TaskCreationOptions.LongRunning. The default scheduler handles it as described above, but another scheduler (such as one you write) may do with this flag whatever it wants, including ignoring it. Hence the naming “Prefer” rather than something stricter like “Guarantee”.

Asynchronous methods, C# iterators, and Tasks

toub — Tue, 30 Jun 2009 19:09:32 GMT

More and more, developers are realizing the significant scalability advantages that asynchronous programming can provide, especially as it relates to I/O.

Consider an application that needs to copy data from one stream to another stream, such as is being done in the following synchronous implementation:

static void CopyStreamToStream(Stream input, Stream output)
{
    // Buffer space for the data to be read and written
    byte [] buffer = new byte[0x2000];

    // While there’s data to be read and written
    while(true)
    {
        // Read data. If we weren’t able to read any, bail.
        // Otherwise, write it out and start over again.
        int numRead = input.Read(buffer, 0, buffer.Length);
        if (numRead == 0) break;
        output.Write(buffer, 0, numRead);
    }
}

In many cases like this, such as if the Streams are FileStream instances representing files on disk, or NetworkStream instances representing remote data, the operations being performed require little-to-no computational power, as the majority of the time is spent waiting on the I/O subsystems and devices. If one such operation is being performed at a time, that waiting isn’t such a big deal. But in a synchronous implementation like the one above, that waiting ends up blocking a thread, rendering the thread useless to do anything else while waiting. Threads by default take up a sizeable chunk of memory as well as kernel resources. Thus, if multiple concurrent calls to CopyStreamToStream are executed, multiple threads may be wasted. Since threads consume a non-negligable amount of resources, we try to limit the number of threads in an application at any one time, such as by using a thread pool, and this synchronous style of programming I/O can lead to scalability bottlenecks, especially in server components where we desire to process as many user requests concurrently as the machine’s resources will possibly allow.

One solution to this problem is through compiler support. A compiler could recognize this synchronous pattern and translate it into an asynchronous one. Such a transformation of that same snippet might look like the following (note that this is hand-generated and is not the actual output of any particular compiler, and rather is my attempt to write this out concisely while still human-understandable):

static void CopyStreamToStreamAsync(
    Stream input, Stream output, Action<Exception> completed)
{
    // Buffer space for the data to be read and written
    byte[] buffer = new byte[0x2000];

    // The read/write loop. The parameter is the IAsyncResult of the
    // last read operation that still need to be completed with a call
    // to EndRead. If the parameter is null, that means a new read needs
    // to be started.
    Action<IAsyncResult> readWriteLoop = null;
    readWriteLoop = iar =>
    {
        try
        {
            // Determine whether to start with a BeginRead or
            // an EndRead/BeginWrite, based on whether iar is null.
            // Then, as long as the loop continues, alternate between
            // reading and writing.
            for (bool isRead = iar == null; ; isRead = !isRead)
            {
                switch (isRead)
                {
                    // Do BeginRead(...)
                    case true:
                        // Start the asynchronous read
                        iar = input.BeginRead(buffer, 0, buffer.Length, readResult =>
                        {
                            // If the read completed synchronously, immediately
                            // return from the callback, as the processing of the
                            // read will be handled synchronously
                            // from the same thread that called BeginRead
                            if (readResult.CompletedSynchronously) return;

                            // The read completed asynchronously, so we need
                            // to run the processing loop,
                            // starting with EndRead/BeginWrite.
                            readWriteLoop(readResult);
                        }, null);

                        // If the read is completing asynchronously, bail, as
                        // there's nothing more to do.
                        // If it completed synchronously, loop around to do
                        // the EndRead/BeginWrite synchronously.
                        if (!iar.CompletedSynchronously) return;
                        break;

                   // Do BeginWrite(...)
                   case false:
                       // Complete the previous read. If there's no more data
                       // to be read/written, bail.
                       int numRead = input.EndRead(iar);
                       if (numRead == 0)
                       {
                           completed(null);
                           return;
                       }

                       // Now that we know how much data was read, write it
                       // out asynchronously
                       iar = output.BeginWrite(buffer, 0, numRead, writeResult =>
                       {
                           // If the write completed synchronously, allow
                           // the thread that called BeginWrite
                           // to handle it.
                           if (writeResult.CompletedSynchronously) return;

                           // Otherwise, complete the asynchronous write
                           // and launch the read/write loop
                           // to continue all over again.
                           output.EndWrite(writeResult);
                           readWriteLoop(null);
                       }, null);

                       // If the write is completing asynchronously, bail,
                       // as there's nothing more to do.
                       // Otherwise, complete the write synchronously and
                       // loop around.
                       if (!iar.CompletedSynchronously) return;
                       output.EndWrite(iar);
                       break;
                   }
               }
           } catch(Exception e) { completed(e); }
    };

    // Start the whole process off with a read.
    readWriteLoop(null);
}

The Axum compiler, available on DevLabs, is actually capable of these kinds of transformations for asynchronous programming, and you could imagine such functionality being baked into a mainstream language like C#. Here’s how the code could be written asynchronously with Axum:

static asynchronous void CopyStreamToStream(Stream input, Stream output)
{
    byte [] buffer = new byte[0x2000];
    while(true)
    {
        int numRead = input.Read(buffer, 0, buffer.Length);
        if (numRead == 0) break;
        output.Write(buffer, 0, numRead);
    }
}

That’s quite lovely. Comparatively, the hand-written code is mind-numbing, and it’s not something you want to have to write each and every time you need to perform some kind of repeating operation like this asynchronously.

As such, some developers have started to take advantage of C# iterators for writing asynchronous code. While not originally designed for this purpose, the compiler transformations employed for C# iterators are similar to what’s necessary for writing asynchronous code, and thus with a bit of library-based support, it’s possible to write an iterator that looks sequential but that takes advantage of asynchrony. Several libraries have been based on this approach, including the Concurrency & Coordination Runtime (CCR) from Microsoft Robotics, Jeffrey Richter’s AsyncEnumerator, and others.

The key to taking advantage of this pattern is yielding something from an iterator that can invoke a callback when an operation completes. The pattern then becomes:

IEnumerable<ThingThatHasCallbackWhenCompletes> AsyncMethod()
{
    ...
    yield return SomethingThatReturnsThingThatHasCallbackWhenCompletes();
    ... // code here executes when the
        // yielded ThingThatHasCallbackWhenCompletes completes
}

The idea is that the iterator method returns an IEnumerable of instances that represent a piece of an asynchronous operation. A utility function is used to invoke the iterator method, and iterates over the resulting enumerator. Each time the utility function gets the next instance from the enumerator, it registers some code to monitor the operation for completion, and when the operation completes, it moves next on the enumerator. Moving next on the enumerator results in re-entering the iterator method at the code location after the last yield point, thus allowing another asynchronous operation to be yielded. In this fashion, the iterator method can in effect yield asynchronous operations, and by using “yield return” to instrument the code with those async points, you as the developer can write an asynchronous method in a manner that looks largely sequential.

Now, think about the description above for the kind of object that needs to be yielded: “something that can invoke a callback when an operation completes.” Sound familiar? The System.Threading.Tasks.Task class in .NET 4 provides this exact functionality. A Task represents an asynchronous operation, and it has a ContinueWith method that enables a callback to be invoked when that asynchronous operation completes. Thus, we should be able to yield Task instances from an iterator in order to write an asynchronous method. Here’s the same CopyStreamToStream example implemented asynchronously in this fashion:

static IEnumerable<Task> CopyStreamToStreamAsync(
    Stream input, Stream output)
{
    // Buffer space for the data to be read and written
    byte [] buffer = new byte[0x2000];

    // While there’s data to be read and written
    while(true)
    {
        // Read data asynchronously. When the operation completes,
        // if no data could be read, we’re done.
        var read = Task<int>.Factory.FromAsync(
            input.BeginRead, input.EndRead, buffer, 0, buffer.Length, null,
            TaskCreationOptions.DetachedFromParent);
        yield return read;
        if (read.Result == 0) break;

        // Write the data asynchronously
        yield return Task.Factory.FromAsync(
            output.BeginWrite, output.EndWrite, buffer, 0, read.Result, null,
            TaskCreationOptions.DetachedFromParent);
    }
}

Much simpler. Where we were previously doing synchronous reads and writes, now we’re yielding the result of calling the built-in Task.Factory.FromAsync method, which creates Tasks that represent asynchronous reads and writes following the APM pattern. We could of course simplify this code further by using a few helper extension methods to hide some of the asynchronous details (these helpers are part of the Beta 1 samples at http://code.msdn.microsoft.com/ParExtSamples):

public static Task<int> ReadTask(this Stream stream,
    byte [] buffer, int offset, int count)
{
    return Task<int>.Factory.FromAsync(stream.BeginRead, stream.EndRead,
        buffer, offset, count, null, TaskCreationOptions.DetachedFromParent);
}

public static Task WriteTask(this Stream stream,
    byte [] buffer, int offset, int count)
{
    return Task.Factory.FromAsync(stream.BeginWrite, stream.EndWrite,
        buffer, offset, count, null, TaskCreationOptions.DetachedFromParent);
}

This then enables the previous code to be simplified to:

static IEnumerable<Task> CopyStreamToStreamAsync(
    Stream input, Stream output)
{
    byte [] buffer = new byte[0x2000];
    while(true)
    {
        var read = input.ReadTask(buffer, 0, buffer.Length);
        yield return read;
        if (read.Result == 0) break;
        yield return output.WriteTask(buffer, 0, read.Result);
    }
}

That looks a lot like the synchronous version, and is *much* easier and less error-prone to write than the manual version shown earlier.

Of course, now we need a mechanism for iterating over the asynchronous iterator. As mentioned, we can take advantage of ContinueWith for the main body of the operation (as with the earlier helpers, the following method and several variants of it are available in the Beta 1 samples):

public static Task Iterate(this TaskFactory factory,
    IEnumerable<Task> asyncIterator)
{
    // Validate parameters
    if (factory == null) throw new ArgumentNullException("factory");
    if (asyncIterator == null)
        throw new ArgumentNullException("asyncIterator");

    // Get the scheduler to use, either the one provided by the factory
    // or the current one if the factory didn’t have one specified.
    var scheduler = factory.TaskScheduler ?? TaskScheduler.Current;

    // Get an enumerator from the enumerable
    var enumerator = asyncIterator.GetEnumerator();
    if (enumerator == null) throw new InvalidOperationException();

    // Create the task to be returned to the caller. And ensure
    // that when everything is done, the enumerator is cleaned up.
    var trs = new TaskCompletionSource<object>(factory.CreationOptions);
    trs.Task.ContinueWith(_ => enumerator.Dispose(),
        TaskContinuationOptions.DetachedFromParent, scheduler);

    // This will be called every time more work can be done.
    Action<Task> recursiveBody = null;
    recursiveBody = antecedent =>
    {
        try
        {
            // If the previous task completed with any exceptions, bail
            if (antecedent != null && antecedent.IsFaulted)
                trs.TrySetException(antecedent.Exception);

            // If the user requested cancellation, bail.
            else if (trs.Task.IsCancellationRequested) trs.TrySetCanceled();

            // If we should continue iterating and there's more to iterate
            // over, create a continuation to continue processing. We only
            // want to continue processing once the current Task (as yielded
            // from the enumerator) is complete.
            else if (enumerator.MoveNext())
                enumerator.Current.ContinueWith(recursiveBody,
                    TaskContinuationOptions.DetachedFromParent, scheduler).
                        IgnoreExceptions();

            // Otherwise, we're done!
            else trs.TrySetResult(null);
        }
        // If MoveNext throws an exception, propagate that to the user
        catch (Exception exc) { trs.TrySetException(exc); }
    };

    // Get things started by launching the first task
    factory.StartNew(() => recursiveBody(null),
        TaskCreationOptions.DetachedFromParent, scheduler).
            IgnoreExceptions();

    // Return the representative task to the user
    return trs.Task;
}

Using this implementation, we can now run “asynchronous methods” that return IEnumerable<Task>, as did our CopyStreamToStreamAsync method:

var asyncOperation = Task.Factory.Iterate(
CopyStreamToStreamAsync(input, output));

Note that the Iterate implementation shown includes a few handy additions on top of the previous hand-coded solution. First, the Iterate method returns a Task, which can be used to track the entire asynchronous operation. Second, it supports cancellation, meaning a caller can request that the asynchronous iteration to shutdown early, even if it hasn’t completed yet. Third, we’ve now separated out the run logic into a separate method, which means that we no longer need all of that goop in the actual target asynchronous method.

On top of all that, this implementation is now based on TaskFactory, which means we can do things like ensure that the code runs on a certain scheduler, such as a scheduler that targets the UI. As an example of where that is handy, consider a method that asynchronously reads from a long, remote stream and stores the resulting data into a TextBox as it’s available:

static IEnumerable<Task> ReadStreamIntoTextBox(Stream stream)
{
    byte [] buffer = new byte[0x2000];
    Encoding enc = new UTF8Encoder();
    while(true)
    {
        var read = stream.ReadTask(buffer, 0, buffer.Length);
        yield return read;
        if (read.Result == 0) break;
        myTextBox.Text += enc.GetString(buffer, 0, read.Result)
    }
}

As previously shown, I could invoke this method as follows:

public void button1_Click(object sender, EventArgs e)
{
Task.Factory.Iterate(ReadStreamIntoTextBox(inputStream)); // buggy
}

However, accessing myTextBox.Text from a thread other than the thread that created myTextBox is a no-no, and yet that’s potentially what will happen in the above. To address that, I want to ensure that the actual code from the iterator is executed on the UI thread (but I still don’t want the asynchronous operations to block the UI thread). To accomplish that, I can create a TaskFactory that will run tasks on the UI thread, as I do in the following code:

public void button1_Click(object sender, EventArgs e)
{
    var uiFactory = new TaskFactory(
        TaskScheduler.FromCurrentSynchronizationContext());
    uiFactory.Iterate(ReadStreamIntoTextBox(inputStream));
}

Of course, while just being able to yield individual operations is useful, things get more interesting when you start considering multi-task continuations, as exposed through ContinueWhenAny and ContinueWhenAll. For example, in our previous copy stream example, we’re reading, then writing, then reading, then writing, and so forth. But we should be able to write the previously read bits while reading the next chunk, thereby achieving better speeds by overlapping latencies. Writing the code to do that using manual asynchrony would be a nightmare… with iterators and tasks, it’s manageable, almost fun:

static IEnumerable<Task> CopyStreamToStreamAsync(
    Stream input, Stream output)
{
    byte[][] buffers = new byte[2][] {
        new byte[BUFFER_SIZE], new byte[BUFFER_SIZE] };
    int filledBufferNum = 0;
    Task writeTask = null;

    while (true)
    {
        var readTask = input.ReadTask(
            buffers[filledBufferNum], 0, buffers[filledBufferNum].Length);
        yield return writeTask == null ?
            readTask :
            Task.Factory.ContinueWhenAll(new[] { readTask, writeTask },
                tasks => Task.WaitAll(tasks), // to propagate exceptions
                TaskContinuationOptions.DetachedFromParent);
        if (readTask.Result == 0) break;

        writeTask = output.WriteTask(
            buffers[filledBufferNum], 0, readTask.Result);
        filledBufferNum = filledBufferNum == 0 ? 1 : 0;
    }
}

Here, ContinueWhenAll is used to ensure that the iterator isn’t re-entered until both any pending writes and reads have completed.

There are of course many variations to this code that you could implement. At the end of the day, I find it quite interesting to see how the Task primitive can be used to enable such scenarios.

Parallel For Loops over Non-Integral Types

toub — Wed, 24 Jun 2009 23:26:00 GMT

In a previous post, it was demonstrated how for loops with very small loop bodies could be parallelized by creating an iterator over ranges, and then using Parallel.ForEach over those ranges. A similar technique can be used to write parallel loops over iteration spaces of non-integers. For example, let’s say I wanted to parallelize the following loop, where the iteration range is based on doubles:

for(double d = 0.0; d < 1.0; d += .001)
{
Process(d);
}

Parallel.For only contains overloads for where the iteration variable is an Int32 or an Int64. To accomodate doubles, one approach would be to translate the range into an integer-based range in order to use Parallel.For, and then within the body of the loop translate it into a double. As an example, the previously shown loop could be rewritten as:

Parallel.For(0, 1000, i =>
{
double d = i / 1000.0;
Process(d);
});

Due to floating point arithmetic, this may not be exactly the same, but it may be close enough. Another approach is to implement an iterator like the following:

private static IEnumerable<double> Iterate(
double fromInclusive, double toExclusive, double step)
{
for(double d = fromInclusive; d < toExclusive; d += step) yield return d;
}

With that Iterate method, now I can parallelize the sequential loop using Parallel.ForEach:

Parallel.ForEach(Iterate(0.0, 1.0, .001), d =>
{
Process(d);
});

This same technique can be applied to a wide variety of scenarios. Keep in mind, however, that the IEnumerator<T> interface isn’t thread-safe, which means that Parallel.ForEach needs to take locks when accessing the data source. While ForEach internally uses some smarts to try to ammortize the cost of such locks over the processing, this is still overhead that needs to be overcome by more work in the body of the ForEach in order for good speedups to be achieved.

Parallel.ForEach has optimizations used when working on indexible data sources, such as lists and arrays, and in those cases the need for locking is decreased. Thus, performance may actually be improved in some cases by transforming the iteration space into a list or an array, which can be done using LINQ, even though there is both time and memory cost associated with creating an array from an enumerable. For example:

Parallel.ForEach(Iterate(0.0, 1.0, .001).ToArray(), d =>
{
Process(d);
});

Happy coding.

Cancellation in Parallel Extensions

mikelid — Mon, 22 Jun 2009 12:00:00 GMT

One of the great features that crosses all of Parallel Extensions types is a consistent approach to cancellation (see http://blogs.msdn.com/pfxteam/archive/2009/05/22/9635790.aspx). In this post we explore some of the ways cancellation is used in Parallel Extensions and explain the guidance we developed.

The new cancellation system is a cooperative approach based on two new types: CancellationTokenSource, which initiates cancellation requests, and CancellationToken which communicates a cancellation request to asynchronous operations and to long-running and blocking method calls.

If you are experimenting with Parallel Extensions, you might like to keep an eye out for the methods that accept a CancellationToken and test them out.

Blocking calls

Some of the Parallel Extension types introduce blocking methods, for example BlockingCollection.Take() and Task.Wait(). The default overloads for these methods will block indefinitely if the condition they are waiting for never occurs. One solution used by these APIs, and many others like them, is to provide a timeout overload so that they may return after some duration and report that the condition did not occur. However, a timeout isn't particularly convenient if you only want to stop waiting if a specific activity occurs, such as a user clicking a 'cancel' button. One possibility is to wait for some amount of time such as 100 milliseconds, check if the button was pressed, and if not then go back to waiting, but this puts the burden on the call-site and must be re-implemented repeatedly; and depending on the frequency of the polling, it may also add unnecessary cost. The types in Parallel Extensions use the new cancellation system and provide overloads that accept a CancellationToken which can cause early termination of blocking methods in response to a specific cancellation request.

Here are some examples of the blocking calls in Parallel Extensions that accept a CancellationToken:

BlockingCollection:
void Add(T item, CancellationToken cancellationToken)
T Take(CancellationToken cancellationToken)

ManualResetEventSlim:
void Wait(CancellationToken cancellationToken)

Task:
void Wait(CancellationToken cancellationToken)
void WaitAll(Task[] tasks, CancellationToken cancellationToken)

In each case, you supply a CancellationToken to the method, and if the CancellationToken is signaled via a call to the associated CancellationTokenSource.Cancel(), then the method will wake up and throw an OperationCanceledException. This frees you from the need to use the timeouts if you are happy to wait until the condition is true or a specific cancellation request occurs. Many of these methods also have overloads that accept both a CancellationToken and a timeout, so that both needs may be accommodated.

Long-running calls

There are also a variety of methods that may simply take a while to complete, particularly if large amounts of data or complex processing is taking place. For example, the following call is likely to take some time, even if individual calls to DoFunc(x) are relatively fast:

Parallel.For(1, 1000000000, (x) => DoFunc(x));

To assist with this, the Parallel APIs have overloads that accept a ParallelOptions class instance that holds a CancellationToken. The machinery inside Parallel.For(...)observes the supplied CancellationToken and will exit early if it sees that the token has be signaled (for more information on exiting loops early, see http://blogs.msdn.com/pfxteam/archive/2009/05/27/9645023.aspx). For example,

CancellationTokenSource cts = new CancellationTokenSource();
ParallelOptions options = new ParallelOptions
{CancellationToken = cts.Token};
Parallel.For(1, 100, options, (x) => DoSlowFunc(x));
// from another thread, call cts.Cancel() to request cancellation.

As you can see, the approach for canceling blocking calls and long-running calls is identical. In both cases you supply a CancellationToken to a specific method call and use the associated CancellationTokenSource to signal a cancellation request.

Digression: Internal cancellation

In certain situations in Parallel Extensions and in other systems, it is necessary to wake up a blocked method for reasons that aren't due to explicit cancellation by a user. For example, if one thread is blocked on blockingCollection.Take() due to the collection being empty and another thread subsequently calls blockingCollection.CompleteAdding(), then the first call should wake up and throw an InvalidOperationException to represent an incorrect usage. A great way to implement this is to use an internal CancellationTokenSource that can wake things up due to internal concerns, and to link it to a second, external CancellationTokenSource that has been supplied by the user. For example, the following captures the essential details of how CompleteAdding() and Take(CancellationToken) are implemented:

class BlockingCollection<T>
{
CancellationTokenSource internalCTS = new CancellationTokenSource();

   public void FinishAdding()
   {
      //...
      internalCTS.Cancel();
   }

   public T Take(CancellationToken externalToken)
   {
      using(CancellationTokenSource linkedTokenSource =
      CancellationTokenSource.CreateLinkedTokenSource(
      internalCTS.Token, externalToken)){
      try
      {
         return InternalTake(linkedTokenSource.Token);
      }
      catch (OperationCanceledException oce)
      {
         if(internalCTS.Token.IsCancellationRequested)
            throw new InvalidOperationException("..msg..");
         else if(externalToken.IsCancellationRequested)
            throw new OperationCanceledException(externalToken);
         else throw;
      }
   }
}

In this fragment, a linked CancellationTokenSource is created and its token passed to the InternalTake() method. This simplifies the implementation for InternalTake() but still allows it to be woken up due to either the external token being signaled or an incorrect concurrent call to CompleteAdding(). If the InternalTake() method throws an OperationCanceledException, we can test the individual tokens to determine what the cause was and take appropriate action. Note that oce.CancellationToken will be equal to linkedTokenSource.Token as this is the token that was actually being observed, but we can look at the original tokens to determine the cause for the linked token being signaled. There is a minor risk of confusion if both sources are signaled simultaneously, but this confusion is benign as it is reasonable to behave as though either source was responsible, and a particular implementation may choose to prioritize one mechanism over the other.

Cooperative cancellation with user-code

Both TPL and PLINQ provide infrastructure that calls back to user code. If the user code is long running or blocking, then it is very useful for the user code to be able to respond to cancellation in a cooperative fashion. TPL and PLINQ achieve this by tracking a supplied CancellationToken, and if they see some user code throw an OperationCanceledException that mentions this specific token, then it is treated as cooperative acknowledgement of cancellation. This means that both TPL and PLINQ understand that the user code did not suffer some catastrophic exceptional conditional, but rather that it is simply responding to cancellation. As a result, TPL and PLINQ constructs will simply exit with an OperationCanceledException of their own, rather than aggregating exceptions from all the participating worker threads and throwing an AggregateException.

PLINQ follows this basic plan; for example, the following query will throw a single OperationCanceledException if the CancellationTokenSource becomes signaled. It doesn't matter whether PLINQ or the user delegate sees the cancellation request first, as the user-delegate exception will be correctly understood by the PLINQ execution engine.

CancellationTokenSource cts = new CancellationTokenSource();
var resultArray =
    data.AsParallel()
        .WithCancellation(cts.Token)
        .Select((x) =>
           {
              if (cts.Token.IsCancellationRequested)
                 throw new OperationCanceledException(cts.Token);
              // ...
           })
        .ToArray();

In both PLINQ and TPL, it is interesting to note that the external cancellation token is not supplied back to the user-delegate via a parameter as may be expected. This would have bloated the number of method overloads considerably and, as shown above, the external token can be conveniently accessed via closures or equivalent techniques.

If the user code throws an OperationCanceledException but one or more "normal" exceptions occur on other threads, then all of the exceptions will be collated into an AggregateException, including the OperationCanceledException. Hence, if only cancellation occurred then a single OperationCanceledException is thrown, but an actual failure will always result in an AggregateException.

PLINQ Queries

PLINQ offers deep support for cancellation and endeavors to ensure that cancellation will be enacted swiftly for any query. For queries that involve only simple and fast user delegates, simply supply a CancellationToken via the WithCancellation() method and PLINQ takes care of the rest. If your query involves long-running or blocking user code, then you should use the cooperative cancellation pattern described in the preceding section. For reasons that are discussed in the next section, PLINQ may not check for cancellation every time it calls a user-delegate, and so long running delegates should check cancellation frequently (as a working rule of thumb, PLINQ will check for cancellation approximately once per one hundred calls to a user delegate). For example, if a delegate call runs for 50ms but does not perform any cancellation checks of its own, then PLINQ may not detect the cancellation request itself for up to 5 seconds, and this could significantly affect a user’s experience. For this reason, we recommend that long-running delegates called from PLINQ queries should check for cancellation as frequently as possible. A good lower bound is to check cancellation once per few thousand IL instructions executed, and an upper bound is to check no less frequently than once per 10ms, but once per 1ms is preferred to ensure snappy response.

The following example demonstrates a user delegate that may run for a while, and so checks cancellation itself every few thousand IL instructions. This provides for timely cancellation support with a negligible cost to performance.

// Assume an external CancellationToken 'token' has been supplied.
int[] data = ...;
var query =
   data.AsParallel()
       .WithCancellation(token)
       .Select((x) =>
          {
             int result = 0;
             for ( int i = 0; i < x; i++ ) {
                for (int j = 0; j < 1000 * x; j++)
                {
                   result += SimpleFunc(i,j);
                }
                token.ThrowIfCancellationRequested();
             }
             return result;})
       .ToArray();

[Update: the call to token.ThrowIfCancellationRequested shown above is not available in Beta1 (but will appear in subsequent releases). This behavior can be acheived via a manual test-and-throw as shown in earlier examples, or via the extension-method approach described in the comments. -Mike]

Summary

The Parallel Extension types use the new cancellation system to provide consistent and rich cancellation features. All of the long-running and blocking methods in Parallel Extensions provide overloads that take a CancellationToken, and any callbacks made to user-code can participate in the process. Application code can also make use of the same ideas and facilities to ensure that cancellation can be a fully supported feature in any situation.

Tasks and the Event-based Asynchronous Pattern

toub — Fri, 19 Jun 2009 18:43:00 GMT

As has been discussed previously, one of the new features in the Task Parallel Library is TaskCompletionSource<TResult>, which enables the creation of a Task<TResult> that represents any other asynchronous operation.

There are a wide variety of sources in the .NET Framework for asynchronous work. One comes from components that implement the Asynchronous Programming Model (APM) pattern, which we discussed here. Another includes types that implement the Event-based Asynchronous Pattern (EAP). For some synchronous method Xyz, the EAP provides an asynchronous counterpart XyzAsync. Calling this method launches the asynchronous work, and when the work completes, a corresponding XyzCompleted event is raised. (That’s an oversimplification of the pattern, but it provides a grounding.)

For many situations, the EAP is quite straightforward and simple to use. However, there are cases where you’d like to be able to do things like join across multiple EAP asynchronous invocations, such as to download three different web pages asynchronously, and only when all three have completed do something else. Tasks in .NET 4.0 make this kind of operation easy, through Task.Factory.ContinueWhenAll (or ContinueWhenAny if you want to do something when any one of the items completes rather than when all of them do). However, in order to use these methods, you need Tasks, thus to use them with components that implement the EAP, you need to create Tasks from EAP. TaskCompletionSource<TResult> can be used to do exactly that.

First, let’s create two small helper functions that we can reuse over and over for multiple EAP-to-Task implementations:

private static TaskCompletionSource<T> CreateSource<T>(object state)
{
    return new TaskCompletionSource<T>(
        state, TaskCreationOptions.DetachedFromParent);
}

private static void TransferCompletion<T>(
    TaskCompletionSource<T> tcs, AsyncCompletedEventArgs e,
    Func<T> getResult, Action unregisterHandler)
{
    if (e.UserState == tcs)
    {
        if (e.Cancelled) tcs.TrySetCanceled();
        else if (e.Error != null) tcs.TrySetException(e.Error);
        else tcs.TrySetResult(getResult());
        if (unregisterHandler != null) unregisterHandler();
    }
}

The first helper, CreateSource, simply creates an instance of TaskCompletionSource<T> with the right type and settings. I’ve included TaskCreationOptions.DetachedFromParent under the assumption that tasks created for the EAP pattern aren’t necessarily meant to participate in parent/child relationships, but you could certainly change this if your scenarios needed that functionality.

The second helper, TransferCompletion, will be used whenever an EAP event signals completion of an asynchronous operation. It takes the AsyncCompletedEventArgs provided through the XyzCompleted event and uses it to determine whether the operation completed due to cancellation or due to an unhandled exception. Both of those pieces of data are standard to the base AsyncCompletedEventArgs class. However, each EAP operation typically comes with its own type derived from AsyncCompletedEventArgs: to be able to use this one helper function with any of those to mine the result of the operation, TransferCompletion also accepts a Func<T> that will return the results from the derived instance.

With those two helpers, writing a Task wrapper for an EAP operation is a cinch. Consider System.Net.WebClient, which provides support for downloading and uploading to and from URIs, with methods like DownloadData. We can write a Task-based wrapper for DownloadData in just a few lines (in this case, as an extension method):

public static Task<byte[]> DownloadDataTask(
    this WebClient webClient, Uri address)
{
    var tcs = CreateSource<byte[]>(address);
    webClient.DownloadDataCompleted +=
        (sender, e) => TransferCompletion(tcs, e, () => e.Result, null);
    webClient.DownloadDataAsync(address, tcs);
    return tcs.Task;
}

We first create a TaskCompletionSource<byte[]> whose Task<byte[]> will be returned from the method. We then register with the DownloadDataCompleted event handler, such that when the event is raised, the downloaded data (or exception or cancellation information) will be transferred to the returned Task. And then we start the operation. Piece of cake.

One interesting thing to note about the EAP is that some implementations support multiple asynchronous operations concurrently on the same instance. In such cases, a user-supplied token is needed to correlate the asynchronous invocations to the asynchronous completions (something that’s provided in Task implicitly by having a Task reference returned from and serving as a reference for an asynchronous operation). For that purpose, we pass the created TaskCompletionSource<TResult> as the user-supplied token, and in TransferCompletion, we only transfer the results if the token received matches the target completion source. If it doesn’t match, this is an event completion for another operation and should be ignored.

One potential issue with the previously shown implementation (and with the EAP pattern in general) is that, if they’re used over and over, the XyzCompleted event handlers will start to build up. Delegates are being registered with the event but not released. To fix that, we can also remove the event handler in the delegate handling the event. For example, DownloadDataTask can be rewritten as:

public static Task<byte[]> DownloadDataTask(
    this WebClient webClient, Uri address)
{
    var tcs = CreateSource<byte[]>(address);

    DownloadDataCompletedEventHandler handler = null;
    handler = (sender, e) => TransferCompletionToTask(tcs, e, () => e.Result, () => webClient.DownloadDataCompleted -= handler);

    webClient.DownloadDataCompleted += handler;
    try { webClient.DownloadDataAsync(address, tcs); }
    catch {
        webClient.DownloadDataCompleted -= handler;
        tcs.TrySetCanceled();
        throw;
    }

    return tcs.Task;
}

The Beta 1 samples available at http://code.msdn.microsoft.com/ParExtSamples already include Task-based extensions for WebClient, as well as extensions for other EAP implementations like SmtpClient and Ping. Download and enjoy!

How PLINQ processes an IEnumerable on multiple cores

igoro — Sat, 13 Jun 2009 12:39:00 GMT

As Ed Essey explained in Partitioning in PLINQ, partitioning is an important step in PLINQ execution. Partitioning splits up a single input sequence into multiple sequences that can be processed in parallel. This post further explains chunk partitioning, the most general partitioning scheme that works on any IEnumerable<T>.

Chunk partitioning appears in two places in Parallel Extensions. First, it is one of the algorithms that PLINQ uses under the hood to execute queries in parallel. Second, chunk partitioning is available as a standalone algorithm through the Partitioner.Create() method.

To explain the design of the chunk partitioning algorithm, let's walk through the possible ways of processing an IEnumerable<T> with multiple worker threads, finally arriving at the solution used in PLINQ (approach 4).

Approach 1: Load the input sequence into an intermediate array

As a simple solution, we could walk over the input sequence and store all elements into an array. Then, we can split up the array into ranges, and assign each range to a different worker.

The disadvantage of this approach is that we need to allocate an array large enough to store all input elements. If the input sequence is long, this will algorithm leads to unnecessarily large memory consumption. Also, we need to wait until the entire input sequence is ready before the workers can start executing.

Approach 2: Hand out elements to threads on demand

An entirely different approach is to have all worker threads share one input enumerator. When a worker is ready to process the next input element, it takes a shared lock, gets the next element from the input enumerator, and releases the lock.

This algorithm has a fairly large overhead because processing every element requires locking. Also, handing out elements individually is prone to poor cache behavior.

This approach does have an interesting advantage over Approach 1, though: since workers receive data on demand, the workers that finish faster will come back to request more work. In contrast, Approach 1 splits up all work ahead of time, and a worker that is done early simply goes away.

Approach 3: Hand out elements in chunks

To mitigate the two drawbacks of Approach 2 (synchronization cost and cache behavior), we can hand out elements to threads in "chunks". When a thread is ready to process more inputs, it will take say 64 elements from the input enumerator.

Unfortunately, while this approach nicely amortizes the synchronization cost over multiple elements, it does not work well for short inputs. For example, if the input contains 50 elements and the chunk size is 64, all inputs will go into a single partition. Even if the work per element is large, we will not be able to benefit from parallelism, since one worker gets all the work.

And since IEnumerable<T> in general does not declare its length, we cannot simply tune the chunk size based on the input sequence length.

Approach 4: Hand out elements in chunks of increasing size

A solution to the problem with small inputs is to use chunks of a growing size. The first chunk assigned to each thread is of size 1 and subsequent chunks are gradually larger, until a specific threshold is reached.

Our solution doubles the chunk size every few chunks. So, each thread first receives a few chunks of size 1, then a few chunks of size 2, then 4, and so forth. Once the chunk size reaches a certain threshold, it remains constant.

This chunking strategy ensures that if the input is short, it will still get split up fairly among the cores. But, the chunk size also grows fairly quickly, and the per-chunk overheads are small for large inputs. Also, the algorithm is quite good at load-balancing, so if one worker is taking longer to process its inputs, other workers will process more elements to decrease the overall processing time.

One interesting consequence of the chunk partitioning algorithm is that multiple threads will call MoveNext() on the input enumerator. The worker threads will use a lock to ensure mutual exclusion, but the enumerator must not assume that MoveNext() will be called from a particular thread (e.g., it should not use thread-local storage, manipulate UI, etc).

The current implementation of both PLINQ chunk partitioning and Partitioner.Create() follows approach 4 fairly closely. Now you know how it behaves and why!