Parallel Programming with .NET : MSDN

Parallelism in October 2008 MSDN Magazine

toub — Thu, 02 Oct 2008 20:57:49 GMT

The October 2008 issue of MSDN Magazine just went online yesterday, and it's chock full of content on parallelism. Definitely worth a cover to cover read!

Paradigm Shift: Design Considerations For Parallel Programming

Coding Tools: Improved Support For Parallelism In The Next Version Of Visual Studio

Concurrency Hazards: Solving 11 Likely Problems In Your Multithreaded Code

Easy Async: Build Concurrent Apps From Simple F# Expressions

Editor's Note:
new Thread(ReadEditorsNote).Start(); yourAttention.WaitOne();

Windows With C++: Exploring High-Performance Algorithms

.NET Matters: False Sharing

{ End Bracket }: Don’t Be Afraid To Throw Away Your Work

Another Parallel Extensions screencast

toub — Thu, 13 Dec 2007 02:16:00 GMT

Daniel Moth is on a roll. Two weeks ago he created three great overview screencasts for Parallel Extensions to the .NET Framework. Today, he released a fourth video, covering the Task and TaskManager classes. Thanks, Daniel! We'll get this added to the MSDN Parallel Computing developer center soon, but in the meantime, head on over to Channel 9 and enjoy.

Give Feedback Through the Connect site

essey — Tue, 04 Dec 2007 20:52:00 GMT

Have you found any bugs, unknown issues (vs. known issues), or any wish list items? Go to our Connect site, sign up for the Parallel Extensions to the .NET Framework connection, and start giving us direct feedback.

While the blog comments and forums are a great way to raise concerns, ask how-to questions, and engage in a dialogue, the Connect site is the best way for you to report an issue or log a feature request--you can file bugs directly into our work items database. And, the Connect site is great because we can respond to the issue, you can see our progress on it, and others have a chance to see, discuss, and rate what is there.

MSDN Forums

toub — Tue, 04 Dec 2007 09:19:00 GMT

There are some great discussions going on over at the Parallel Extensions forum on MSDN. Download the CTP, try out the bits, head on over to the forums, and get involved! We'd love to hear from you.

New "Parallel Computing" dev center on MSDN

toub — Fri, 30 Nov 2007 04:37:00 GMT

There's a new Parallel Computing developer center on MSDN:

"Microsoft’s Parallel Computing developer center is dedicated to providing information, ideas, community, and technology to developers to make it easier to write programs that perform and scale well on parallel hardware."

Check it out! There are already a plethora of resources available at the center, including several great videos from Daniel Moth on using Parallel Extensions. We'll be populating the site with more in the coming weeks and months (if there are any resources you feel should be added, please let us know).

In addition to the dev center, there are forums for discussing parallel computing in general as well as Parallel Extensions, and there's a Microsoft Connect site for submitting bugs and feature requests.

We'd love to hear from you.

PLINQ changes since the MSDN Magazine article

toub — Thu, 29 Nov 2007 18:20:00 GMT

I posted about changes we've made to the Task Parallel Library since we published the MSDN Magazine article outlining its design. In this post, I'll do the same thing for PLINQ.

Most of the October 2007 article on PLINQ is still accurate. After all, PLINQ is largely an implementation of the .NET Standard Query Operators, and thus its API is predominantly dictated by the API shipped in the .NET Framework 3.5. However, there are APIs exposed by PLINQ that are not part of the .NET Standard Query Operators (such as the most visible member of PLINQ, AsParallel), and there are also behavioral differences that manifest due to its very nature as a parallel implementation versus the sequential implementation that exists today. Since the MSDN article was published, there have been a few changes to both of these, and I'll outline those differences here.

First, as with the Task Parallel Library, PLINQ is not contained in System.Concurrency.dll as is claimed by the article. We've since moved all of this functionality to the System.Threading.dll assembly, which is what you'll need to reference to access PLINQ functionality in your .NET projects as of this CTP.

Next, the article shows the AsParallel extension method as living on the System.Linq.ParallelEnumerable class. This method has migrated to the new System.Linq.ParallelQuery class. Of course, given its primary use as an extension method for IEnumerable<T>, and given that ParallelQuery is in the same namespace and assembly as ParallelEnumerable, the change in declaring type doesn't really affect the programming model, as the change isn't at all visible in code using AsParallel as an extension method.

Moving on, the article states that "parallelism doesn’t get introduced until you start processing the output of the query. If you’re familiar with IEnumerable<T>, this equates to calling the GetEnumerator method." Of course, if you're very familiar with LINQ, you'll know that the query isn't actually evaluated until MoveNext() is called on the IEnumerator<T> returned from GetEnumerator. PLINQ behaves in the same way... none of the data in the query will be processed until MoveNext() is called.

The next change is important to keep in mind. Here's the original MSDN article text:

"...pipelined processing, in which case the thread doing
the enumeration is separate from the threads devoted to running
the query. PLINQ will use many threads for query execution, but will reduce the degree of parallelism by one so that the enumerating thread is not interfered with. For instance, if you have eight processors available, seven of them will run the PLINQ query while the remaining processor runs the foreach loop on the output of the PLINQ query as elements become available. This carries the benefit of allowing more incremental processing of output, thus reducing the memory requirements necessary to hold the results; however, having many producer threads and only a single consumer thread can often lead to uneven work distribution, resulting in processor inefficiency."

This was true at the time the article was written, but for a variety of reasons we've changed the behavior since. Now, when using pipelining (the default when iterating over a query with foreach) we don't decrease the degree of parallelism. If you still want such behavior, you can take advantage of the AsParallel overload that accepts the degree of parallelism as an argument, and provide to it Environment.ProcessorCount-1. Why did we change this? The reasoning described in the article is sound, however there were other competing factors that, in the end, we decided were more important. As an example of one of these, consider the computer you're currently reading this on... is it a dual-core? A quad-core? One of those is likely. And if you were to iterate over a PLINQ query with the aforementioned pipelining behavior on a dual-core machine, guess how many threads would be used to process the query. One. That's very likely not what you'd expect from a parallel LINQ implementation.

Next (and very minor), the article describes an overload of IParallelEnumerable.GetEnumerator that accepts a Boolean parameter named pipelined that determines whether pipelining will be used when executing the query. This parameter has been renamed to usePipelining. Note, however, that we've already received feedback about this design (e.g. using a Boolean argument to enable/disable pipelining), and we'll be reevaluating it in the future.

In the section of the article covering concurrent exceptions, System.Concurrency.MultipleFailuresException is mentioned. This exception type still exists and behaves as is described in the article, however it's been renamed. The exception type is now System.Threading.AggregateException and is the same exception type used for this purpose across all of the Parallel Extensions to the .NET Framework. Additionally, the article mentions that the InnerExceptions property is an array of Exception instances; the AggregateException.InnerExceptions property is now a ReadOnlyCollection<Exception>.

When discussing order preservation in PLINQ queries, the article mentions that QueryOptions.PreserveOrdering can be provided as a parameter to the AsParallel method. This capability still exists, but the name has been changed slightly to ParallelQueryOptions.PreserveOrdering.

I believe those are the primary API/behavioral differences between the article and the current CTP release of PLINQ. If you find any other significant differences, please us know, and we'll augment the information provided here.

Thanks!

Task Parallel Library changes since the MSDN Magazine article

toub — Thu, 29 Nov 2007 18:19:00 GMT

Back in the October 2007 issue of MSDN Magazine, we published an article on the beginning stages of what has become the Task Parallel Library (TPL) that's part of the Parallel Extensions to the .NET Framework. While the core of the library and the principles behind it have remained the same, as with any piece of software in the early stages of its lifecycle, the design changes frequently. In fact, one of the reasons we've put out an early community technology preview (CTP) of Parallel Extensions is to solicit your feedback on the APIs so that we know if we're on the right track, how we may need to change them further, and so forth. Aspects of the API have already changed since the article was published, and here we'll go through the differences.

First, the article refers to System.Concurrency.dll and the System.Concurrency namespace. We've since changed the name of the DLL to System.Threading.dll, and TPL is now contained in two different namespaces, System.Threading and System.Threading.Tasks. AggregateException, the higher-level Parallel class, and Parallel's supporting types (ParallelState and ParallelState<TLocal>) are contained in System.Threading, while all of the lower-level task parallelism types are in System.Threading.Tasks.

Next, the second page of the article states "if any exception is thrown in any of the iterations, ... the first thrown exception is rethrown in the calling thread." The semantics here have changed such that we now have a common exception handling model across all of the Parallel Extensions, including PLINQ. If an exception is thrown, we still cancel all unstarted iterations, but rather than just rethrowing one exception, we bundle all thrown exceptions into an AggregateException container exception and throw that new exception instance. This allows developers to see all errors that occurred, which can be important for reliability. It also preserves the stack traces of the original exceptions.

In the section on aggregation, the article talks about the Parallel.Aggregate API. Since the article was published, we've dropped this method from the Parallel class. Why? Because a) PLINQ already supports parallel aggregation through the ParallelEnumerable.Aggregate extension method, and b) because aggregation can be implemented with little additional effort on top of Parallel.For if PLINQ's support for aggregation isn't enough. Consider the example shown in the article:

int sum = Parallel.Aggregate(0, 100, 0,
delegate(int i) { return isPrime(i) ? i : 0; },
delegate(int x, int y) { return x + y; });

We can implement this with PLINQ as follows:

int sum = (from i in ParallelEnumerable.Range(0, 99)
where isPrime(i)
select i).Sum();

Of course, this benefits from LINQ and PLINQ already supporting a Sum method. We can do the same thing using the Aggregate method for general reduction support:

int sum = (from i in ParallelEnumerable.Range(0, 99)
where isPrime(i)
select i).Aggregate((x,y) => x+y);

If we prefer not to use PLINQ, we can do a similar operation using Parallel.For (in fact, this is very similar to how Parallel.Aggregate was implemented internally):

int sum = 0;
Parallel.For(0, 100, () => 0, (i,state)=>
{
if (isPrime(i)) state.ThreadLocalState += i;
},
partialSum => Interlocked.Add(ref sum, partialSum));

Here, we're taking advantage of the overload of Parallel.For that supports thread-local state. On each thread involved in the Parallel.For loop, the thread-local state is initialized to 0 and is then incremented by the value of every prime number processed by that thread. After the thread has completed processing all iterations it's assigned, it uses an Interlocked.Add call to store the partial sum into the total sum. In fact, if you found yourself needing Aggregate functionality a lot, you could generalize this into your own ParallelAggregate method, something like the following:

static T ParallelAggregate<T>(
    int fromInclusive, int toExclusive, T seed,
    Action<int> selector, Action<T,T> aggregator)
{
    T result = seed;
    object aggLock = new object();
    Parallel.For(fromInclusive, toExclusive, () => initialValue, (i,state) =>
    {
        state.ThreadLocalState =
            aggregator(state.ThreadLocalState, selector(i));
    },
    partial => { lock(aggLock) result = aggregator(partial, result); } );
    return result;
}

With this, you can use ParallelAggregate just as is done with Parallel.Aggregate in the article:

int sum = ParallelAggregate(0, 100, 0,
delegate(int i) { return isPrime(i) ? i : 0; },
delegate(int x, int y) { return x + y; });

Moving on to the Task class, we've made some fairly substantial changes to the public facing API. Here's a summary of the differences from what's described in the article:

As shown in the article, Task does not have a base class. In the CTP, Task now derives from TaskCoordinator, a base class which provides all of the waiting and cancelation functionality.
The Task class described in the article exposes a constructor that accepts an Action delegate. During construction, the Task is queued for execution. Instead, we changed this so that Task doesn't expose any constructors. Tasks are now created using the Task.Create static method; we felt this factory approach was clearer than the constructor approach... please let us know what you think regarding this change. Were we right? Is one approach better than the other? Would it be a positive or a negative if both approaches were supported?
The type of the delegate used by Task.Create is Action<Object> rather than just Action, and overloads of Task.Create support providing a state argument that will be passed to the Task's action.
The article states that calling Cancel on a Task will cancel that task and all of the tasks that task created. We've modified this approach so that the model is now opt-in, meaning that by default tasks created by a task will not be canceled when the parent task is canceled. That behavior can be opted into, however, by specifying the RespectParentCancelation flag when the child Task instances are created.
The article describes a Task<T> type that derives from Task. This has been renamed Future<T>. As with Task, it no longer exposes a constructor that accepts a Func<T> delegate. Instead, Future<T> provides a static Create factory method. In addition, to better support type inference, we provide a non-generic Future class that exposes generic static Create<T> methods. This makes it possible to create a Future<T> where T is an anonymous type returned from the provided Func<T>.
The article describes the ReplicableTask class as deriving from Task. For the CTP, we've gotten rid of ReplicableTask, but we've kept the concept. To create a Task as a replicable task, provide the TaskCreationOptions.SelfReplicating value to the Task.Create method when creating the Task.
The article describes the exception handling model for self-replicating tasks being one where only one exception is rethrown even if multiple exceptions were thrown from multiple invocations of the replicable task. As with the loop exception handling model described earlier, we now collect all thrown exceptions into one new AggregateException and throw that exception instead of picking one of the thrown exceptions at random.

In addition to changes to the Task class, there have also been changes to the TaskManager class described in the article:

In the article, TaskManager exposed two constructors, one that's parameterless and one that accepts an integer representing the maximum concurrency level for the TaskManager. We've modified that second constructor to instead accept a TaskManagerPolicy, a type which represents the allowed behavior of the TaskManager, providing information such as desired concurrency level and execution context flow behavior.
Similarly, rather than exposing a MaxConcurrentThreads property, TaskManager now exposes a TaskManagerPolicy property, which allows the current policy to be retrieved and a new policy to be provided dynamically.
The article shows Task providing a constructor that accepts a TaskManager, in order to associate that Task with a specific TaskManager. Such behavior is still possible, but the TaskManager is provided to overloads of Task.Create.

That sums up the changes we've made. Even with these changes, the article should still provide you with a good overview of the library and its intended usage. For more information, check out the documentation that's included with the CTP, and stay tuned to this blog! As mentioned before, we're very interested in your feedback on the API, so please let us know what you think (the good and the bad).