Parallel Programming with .NET : Parallelism Blockers

Achieving Speedups with Small Parallel Loop Bodies

toub — Sat, 06 Jun 2009 23:27:00 GMT

The Parallel class represents a significant advancement in parallelizing managed loops. For many common scenarios, it just works, resulting in terrific speedups. However, while ideally Parallel.For could be all things to all people, such things rarely work out, and we’ve had to prioritize certain scenarios over others.

One area Parallel.For may fall a bit short is in attempts to use it with very small loop bodies, such as:

int [] array = new int[100000000];
Parallel.For(0, array.Length, i=>
{
array[i] = i*i*i;
});

Such an operation should be readily parallelizable; after all, every iteration is completely independent of every other iteration, and we’re dealing with a big data parallel problem. The devil is in the details, however. To handle typical scenarios where the time it takes to complete each iteration ends up being non-uniform, Parallel.For’s implementation takes a lot of care to load balance across all threads participating in the loop, and that load balancing comes at a small performance cost. This cost is typically trumped by the performance benefits that come from doing the load balancing; however, when the body of the loop is as tiny as it is in the example above, even small overheads add up. Another overhead that also contributes is the delegate invocation required to invoke the loop body. It can be easy to forget when looking at a Parallel.For call that Parallel.For is really just a method, accepting as a parameter a delegate to be invoked for every iteration. That invocation isn’t free, and in the above case may even be more expensive than the body of the loop itself.

Fear not, however, as there exist ways to still achieve good speedups on such cases. One way is based on creating larger chunks of work for Parallel to operate on: as the chunk size increases, the overhead costs start to pale in comparison, and speedups are realized.

Consider a new ForRange method you could implement:

public static ParallelLoopResult ForRange(
int fromInclusive, int toExclusive, Action<int, int> body);

Unlike For, which invokes the body once per iteration, ForRange will invoke the body with a start and end of a range. Thus, given an initial sequential loop like the following:

for(int i=0; i<N; i++)
{
DoWork(i);
}

with For it would be parallelized by replacing the for loop with a Parallel.For:

Parallel.For(0, N, i=>
{
DoWork(i);
});

and with ForRange, it would be parallelized by wrapping the for loop with a ForRange:

ForRange(0, N, (from,to) =>
{
    for(int i=from; i<to; i++)
    {
        DoWork(i);
    }
});

There are several ways we can now implement ForRange. The first is simply by doing a little math. We can calculate the boundaries of each range and use a Parallel.For to run the user-supplied body action for each range, e.g.

public static ParallelLoopResult ForRange(
    int fromInclusive, int toExclusive, Action<int, int> body)
{
    int numberOfRanges = Environment.ProcessorCount;
    int range = toExclusive - fromInclusive;
    int stride = range / numberOfRanges;
    if (range <= 0) numberOfRanges = 0;
    return Parallel.For(0, numberOfRanges, i => {
        int start = i * stride;
        int end = (i == numberOfRanges - 1) ? toExclusive : start + stride;
        body(start, end);
    });
}

Another way is actually by using Parallel.ForEach under the covers. Rather than doing the math as was done above, we can write an iterator in C# that yields the ranges, and then Parallel.ForEach over those ranges, e.g.

public static ParallelLoopResult ForRange(
    int fromInclusive, int toExclusive, Action<int, int> body)
{
    int rangeSize = (toExclusive - fromInclusive) / Environment.ProcessorCount;
    if (rangeSize == 0) rangeSize = 1;
    return Parallel.ForEach(
        CreateRanges(fromInclusive, toExclusive, rangeSize), range =>
    {
        body(range.Item1, range.Item2);
    });
}

private static IEnumerable<Tuple<int,int>> CreateRanges(
    int fromInclusive, int toExclusive, int rangeSize)
{
    // Enumerate all of the ranges
    for (int i = fromInclusive; i < toExclusive; i += rangeSize)
    {
        int from = i, to = i + rangeSize;
        if (to > toExclusive) to = toExclusive;
        yield return Tuple.Create(from, to);
    }
}

(You can download an implementation of ForRange as part of the Beta 1 samples at http://code.msdn.microsoft.com/ParExtSamples.)

In general, we expect the design and implementation of Parallel.For will be right for the vast majority of scenarios. However, solutions like those above can be used to accommodate cases that don’t quite fit the mold.

Getting random numbers in a thread-safe way

toub — Thu, 19 Feb 2009 17:40:00 GMT

It’s very common in a parallel application to need random numbers for this or that operation. For situations where random numbers don’t need to be cryptographically-strong, the System.Random class is typically a fast-enough mechanism for generating values that are good-enough. However, effectively utilizing Random in a parallel app can be challenging in a high-performance manner.

There are several common mistakes I’ve seen developers make. The first is to assume that Random is thread-safe and is ok to be used concurrently from multiple threads. This is a bad idea, and can have some drastic consequences on the quality of the random numbers (degrading them well below the “good enough” bar). For an example of this, consider the following program:

using System;
using System.Threading;

class Program
{
    static void Main(string[] args)
    {
        Random rand = new Random();

        Parallel.For(0, 1000000, (i, loop) =>
        {
            if (rand.Next() == 0) loop.Stop();
        });

        while (true)
        {
            Console.WriteLine(rand.Next());
            Console.ReadLine();
        }
    }
}

While it won’t happen every time you run the app, on a multi-core machine it’s likely that once the Parallel.For in this example exits, rand.Next() will always return 0. Not very random, is it? This is due to an implementation detail of Random that does not tolerate multithreaded access (it wasn’t designed for it, and the docs explicitly call out that Random should not be used from multiple threads, as they do for most types in the .NET Framework).

Another common mistake is to fix a problem like that in the previous example by instantiating a new Random instance each time a random number is needed. This can also affect the quality of the random numbers. When a seed isn’t explicitly provided to Random through its constructor, it internally uses Environment.TickCount (at least in the current release) as the seed. TickCount returns a value that’s heavily influenced by the resolution of the system timer, and thus multiple calls to TickCount in a row are likely to yield the same value (on the system on which I’m writing this post, TickCount changes in value approximately once every 15 milliseconds). This means that Random will suffer a similar fate, as is exemplified in the following program:

using System;

class Program
{
    static void Main(string[] args)
    {
        int lastNumber = 0, count = 0;
        while (true)
        {
            // Get the next number
            int curNumber = new Random().Next();

            // If it's the same as the previous number,
            // note the increased count
            if (curNumber == lastNumber) count++;

            // Otherwise, we have a different number than
            // the previous. Write out the previous number
            // and its count, then start
            // over with the new number.
            else
            {
                if (count > 0)
                    Console.WriteLine(
                        count + ": " + lastNumber);
                lastNumber = curNumber;
                count = 1;
            }
        }
    }
}

This program continually creates new Random instances and gets a random value from each, counting the number of “random” values in a row that are the same. When I run this app, I consistently see each number showing up thousands of times before the number changes; again, not very random. The point of this example is that, in a multithreaded context, you may be creating a Random instance for each iteration, but many iterations across threads are likely to end up with the same value.

There are several approaches then one could take in using Random from multiple threads to avoid these kinds of issues.

One approach is to use a lock. A shared Random instance is created, and every access to the Random instance is protected by a lock, e.g.

public static class RandomGen1
{
    private static Random _inst = new Random();

    public static int Next()
    {
        lock (_inst) return _inst.Next();
    }
}

This has the benefit of being easy to write, simple to explain, and obviously does the right thing. Unfortunately, there’s an expensive lock on every access, and the more threads that want random numbers, the more contention there will be for this lock. Sharing hurts.

Another approach is to use one Random instance per thread, rather than one per AppDomain (as is the case with the RandomGen1 approach shown above). Here’s one way that might be implemented:

public static class RandomGen2
{
    private static Random _global = new Random();
    [ThreadStatic]
    private static Random _local;

    public static int Next()
    {
        Random inst = _local;
        if (inst == null)
        {
            int seed;
            lock (_global) seed = _global.Next();
            _local = inst = new Random(seed);
        }
        return inst.Next();
    }
}

An AppDomain-wide Random instance is maintained in order to provide seeds for new Random instances created for any new threads that come along wanting random numbers. Each thread maintains its own Random instance in a ThreadStatic field, such that once initialized, calls to Next need only retrieve the ThreadStatic Random instance and use its Next method; no locks are necessary, and sharing is minimized. I tested this in a loop like the following:

Stopwatch sw;
const int NUM = 1000000;

while (true)
{
    sw = Stopwatch.StartNew();
    Parallel.For(0, NUM, i =>
    {
        RandomGen1.Next();
    });
    Console.WriteLine(sw.Elapsed);

    sw = Stopwatch.StartNew();
    Parallel.For(0, NUM, i =>
    {
        RandomGen2.Next();
    });
    Console.WriteLine(sw.Elapsed);

    Console.ReadLine();
}

On my dual-core laptop using .NET 3.5 and the June 2008 CTP of Parallel Extensions, the ThreadStatic approach ended up running twice as fast as the lock-based version. Moreover, as the number of cores increases, so too will the amount of contention on the lock in the lock-based approach, thus increasing the gap between the two approaches. Additionally, a lot of work has been done in .NET 4.0 to improve the performance of things such as accessing [ThreadStatic]. When I run this same code on current builds of .NET 4.0 (again on a dual-core), I see the ThreadStatic version running more than four times faster than the lock-based version. Even when there’s no contention on the lock in the RandomGen1 solution (simulated by switching the parallel loops to sequential loops), the ThreadStatic version in .NET 4.0 still runs significantly faster than the lock-based version.

Of course, if you really care about the quality of the random numbers, you should be using RNGCryptoServiceProvider, which generates cryptographically-strong random numbers (Addendum: davidacoder makes a good point in his comments on this post that while Random has certain statistical properties, using multiple Random instances as part of the same algorithm may change the statistical properties in unknown or undesirable ways). For a look at how to get a Random-based facade for RNGCryptoServiceProvider, see .NET Matters: Tales from the CryptoRandom in the September 2007 issue of MSDN Magazine. You could also settle on an intermediate solution, such as using an RNGCryptoServiceProvider to provide the seed values for the ThreadStatic Random instances in a solution like that in RandomGen2 (which would help to avoid another issue here, that of two threads starting with the same seed value due to accessing the global Random instance in the same time quantum):

public static class RandomGen3
{
    private static RNGCryptoServiceProvider _global =
        new RNGCryptoServiceProvider();
    [ThreadStatic]
    private static Random _local;

    public static int Next()
    {
        Random inst = _local;
        if (inst == null)
        {
            byte[] buffer = new byte[4];
            _global.GetBytes(buffer);
            _local = inst = new Random(
                BitConverter.ToInt32(buffer, 0));
        }
        return inst.Next();
    }
}

Concurrent, Multi-core Programming on Windows and .NET

toub — Tue, 28 Oct 2008 08:51:00 GMT

Thanks to everyone who attended our PDC pre-conference session yesterday on parallelism and concurrency! We had a wonderful turnout at the event, and David, Joe, and I all had a terrific time. Attached to this post are the slides we presented.

(It turns out that the PDC site does allow you to submit an evaluation for a precon. If you attended, we'd love your feedback.)

Feedback requested: TaskManager shutdown, Fair scheduling

toub — Fri, 01 Aug 2008 17:54:29 GMT

One of the primary reasons we've released CTPs of Parallel Extensions is to solicit feedback on the design and functionality it provides. Does it provide all of the APIs you need to get your job done? Are there scenarios you wished the APIs supported and that you need to work around in klunky ways? And so forth. We've received some terrific feedback thus far, we've already made changes based on it, we're currently making changes based on it, and we'll continue making changes based on it moving forward.

We continually have discussions internally about additional support we could provide. Frequently these discussions result in our needing to know more about our customers' needs. There's a practically unlimited amount of functionality we could bake into Parallel Extensions, but each additional piece not only requires design, development, testing, and support, but it also can complicate other aspects of the design, potentially slow down other primary scenarios, and so forth. In order to focus our efforts, we need feedback.

Two such discussions occurred recently, and any feedback you provide would be useful in our deciding how to move forward.

The first is about shutting down a TaskManager instance. In the current implementation, we only support one kind of shutdown, which is triggered through a call to Dispose on TaskManager. This implementation is a synchronous invocation that blocks until all of the Tasks previously scheduled to the TaskManager have completed. However, there are other semantics we could potentially implement. For example, we could provide an asynchronous shutdown option, that allowed you to asynchronously call Shutdown, and the TaskManager would only be disposed of when all of the tasks scheduled to it completed. Or with a bit more internal reworking, we could support automatically canceling all of the tasks scheduled to the TaskManager. And again that could be done synchronously (canceling all and then waiting for all) or asynchronously (canceling all and then only cleaning up after the TaskManager when any that were currently executing completed). How useful would such capabilities be to you? In what scenarios would you find them useful or, more importantly, necessary? Would such capabilities be dangerous at all in your scenarios?

The second is about scheduling order. One of the benefits that a work-stealing scheduler (the kind of scheduler employed by the Task Parallel Library) provides is a distribution of work across all cores, where each core prefers to schedule new work to and pull work from its local queue(s). This scheduling can be made extremely efficient and can be made to improve locality and the like by using LIFO ordering, meaning that the task most recently scheduled is the one that will execute first. This is separate from work scheduled to the scheduler from other threads (such as an application's main thread), which will typically still be scheduled in a generally FIFO order. The question, then, is if there are scenarios you might have where you always want that FIFO-ish order, regardless of where the work is scheduled from. Such an option would likely decrease performance in some key scenarios, but it would also be more fair in terms of the order in which work gets executed. Do you have any scenarios that would require such a PreferFairness option? We'd love to hear about them if you do.

PLINQ Ordering

essey — Thu, 12 Jun 2008 01:43:44 GMT

There is a natural tension between ordering and performance in a parallel partitioning system such as PLINQ, which we addressed as guidance in the Dec07 CTP documentation: “Although you can opt into ordering, this does come at a cost to performance because it constrains the options which PLINQ can use for executing a query, so it is better to avoid it in general if performance is your main concern. A better approach is to avoid assumptions on ordering in your programs where possible, or to explicitly sort elements in your query. This is consistent with the approach that most relational databases take.”

In the June 2008 CTP, we did a major overall of the ordering model in PLINQ. Here's an overview of our thinking, our rationale, and the design decisions that we made along the way. As always, your feedback is very important. We are especially looking for input regarding the default choice that we have made - to be unordered unless otherwise specified.

Importance of PERFORMANCE

PLINQ’s raison d’etre is to provide improved performance to developers on multi-core systems. As such, it is as the forefront of our mind when creating these systems. Customers in many spaces that could make excellent use out of data parallelism do not have requirements on ordering, yet they would benefit from large performance gains.

This is certainly the case in the finance industry. Financial institutions have told us that they could easily absorb 1000x increase in performance, and performance alone is what would drive them to adopt PLINQ (they don’t even look to LINQ, because ease of programming is secondary in importance). There are many scenarios that are pure number crunching and do not have expectations on ordering that need to be met.

There are other precedents in LINQ for having an unordered system: LINQ-to-SQL and other data providers do not provide a default ordering contract. They require the end user to sort if this is necessary.

Importance of ORDER

On the other side, there are also reasons to support ordering in the system.

1. LINQ-to-Objects sets the precedent – the original recipe, predecessor to PLINQ, does maintain ordering by default. While some data providers no not require ordering, it still remains that PLINQ, for many scenarios, can be a drop-in replacement for LINQ-to-Objects. I bet you didn’t miss that caveat “for many scenarios”, which draws attention to what we have dubbed the Parallelism Blockers. Again from the CTP documentation, note #4:

A very common question is: Why is an explicit programming model needed for concurrency instead of having the system automatically decide to provide parallelism?

There are several reasons why there needs to be some programmer involvement in the parallelism process:

1.    Mutable data structures and impurity
2.    Concurrent exceptions
3.    Thread affinity
4.    Ordering expectations
5.    Performance - Problems with < 1.0 speedup

2. Some code expects ordering - As an illustration, this can provide unexpected behavior in code such as:

Array.Sort(a);

var q = from x in a ...;

The initial Array.Sort's demand on order would not be kept by the system, and it would be shuffled in the query execution by default. PLINQ provides mechanisms to preserve ordering, but usually at a cost to performance. So, you can opt-in to order preservation in PLINQ explicitly:

Array.Sort(a);

from x in a.AsParallel().AsOrdered() ...

3. Some operators require or insist on ordering – The OrderBy and ThenBy operators specify ordering. If these were disregarded and ignored after being declared, many scenarios would be cut off. For this reason, these turn on ordering that flows forward in the system.

Further, operators such as Reverse, SequenceEquals, TakeWhile, SkipWhile, and Zip lose practical purposes when there is not a defined ordering. Since these operators are useful in many cases, it is important to keep them.

Our Goals

These goals are the list of requirements that must be achieved to successfully address this tension. These are listed in no particular order and are not ranked. They must all be achieved.

Transparent, clear mental model for developers – The developer should be able to understand the system through a very clear and succinctly stated design statement. There should not be exceptions to this. It should be straight-forward and simple to write code against.
Good default – There should be a default model that works for the important customer cases as determined by the group. For Dev10 this is for Finance and Multimedia scenarios. A good default also means that there is nothing additional that needs to be specified in the opt-in mechanism (i.e. AsParallel() without any parameters should be sufficient).
Flexibility to address both sides of the performance/ordering tension – While the model does pick a default, scenarios that favor the other side of the tension should be able to toggle in all or parts of the query.

Non-Goals

These are the things that we are not explicitly trying to accomplish nor avoid; they are not the main purpose of this design. If they happen along the way, they happen.

Map completely to LINQ-to-Objects or another provider – There is no specific ordering or full set of operators contract required for the LINQ-specification, and it is not a goal to strictly adhere to another provider. It is more important to adhere to a customer scenario.

The New Mental Model

The mental model is the suggested way for the developer to think of the system. How it is actually implemented under the covers may differ from this, as long as the end results align with user expectations, consistent with the mental model.

The mental model is succinctly:

Ordering operators re-establish order, shuffle points shuffle the order.

And more detailed:

a.    The query is shuffled at the start
b.    unless the default is overridden to keep the order at the start
c.    ordering operators establish a new ordering that flows afterwards in the query,
d.    a shuffle point can be inserted anywhere in the query, reshuffling the query,
e.    until there is another (optional) ordering operator…

Default: unordered

In favor of performance, PLINQ’s raison d’etre, the default for queries is to be unordered.

Rationale for this is covered above. This is a decision that we will validate through performance work and feedback from you. This is not a clear-cut decision and the right data could persuade us. If you have good reasons to keep or change this default, please let us know. What do you think the default should be and why?

Programming Model Implications

To provide a richer programming model for working with ordering, there are mechanisms to override the default and turn ordering on and off in the query.

This specification announces the addition of AsOrdered and AsUnordered operators to the PLINQ syntax and the removal of the ParallelQueryOptions enum and all overloads of AsParallel that accept this enum.

Overriding the default

Call AsOrdered on an IEnumerable or IList to opt into PLINQ preserving the input ordering.

AsOrdered can only be validly invoked on AsParallel, Range, and Repeat operators. Any other operators will throw a runtime InvalidOperationException.

New to the ParallelEnumerable static class is:

public static IParallelEnumerable<TSource> AsOrdered<TSource>(IParallelEnumerable<TSource> source);

Usage examples:

src.AsParallel().AsOrdered().Take(1000)

ParallelEnumerable.Range(1,1000000000).AsOrdered().Take(1000)

ParallelEnumerable.Repeat(2,1000000000).AsOrdered().Take(1000)

Shuffling midquery

If, mid-query, a point is reach where the developer knows that ordering is no longer important, the developer can add a shuffle point to tell the query engine that ordering can now be arbitrary. A new operator is added to PLINQ for this purpose.

New to the ParallelEnumerable static class is:

public static IParallelEnumerable<TSource> AsUnordered<TSource>(IParallelEnumerable<TSource> source);

It can be used as follows:

pages.AsParallel().AsOrdered().Take(1000000).AsUnordered().Join(…).Select(…);

In this case, the query no longer requires ordering for the following Join and Select, which are the heavy lifting of the query.

Turning on ordering midquery

Ordering can be turned on, then off, then on again midquery:

On: Using original LINQ operators OrderBy or OrderByDescending and then optionally ThenBy or ThenByDescending operators will establish ordering in the query that will continue to flow throughout. Take for example:

pages.AsParallel().Where(…).Select(…).OrderBy(i => i.rank).Take(10);

In the above query, the Where and the Select operators can function in arbitrary order, then the pages are ordered by rank for the Take operator to return the top 10 pages by rank.

Then Off: This ordering requirement is lifted at the next shuffle point if there is one.

pages.AsParallel().Where(…).Select(…).OrderBy(i => i.rank).Take(10000)

.AsUnordered().Join(…);

In the above example, the initial parts of the query remain the same as above, but the query no longer needs to maintain the ordering when performing the Join operator.

Then On again: This ordering requirement is reset at a subsequent ordering operator.

pages.AsParallel().Where(…).Select(…).OrderBy(i => i.rank).Take(10000)

.AsUnordered().Join(…)

.OrderBy(i => i.domain);

In the above example, ordering is turned on again at the end to reorder by domain for the output.

Parallel loop performance

toub — Thu, 13 Mar 2008 08:57:49 GMT

We've received several questions on the MSDN Forums for Parallel Extensions about the performance of the Parallel class, and specifically of the loop constructs we provided in the CTP. We're very much aware that the performance of Parallel.For/ForEach in the CTP is not optimal, and that for some situations, the overhead for these constructs will be too much for a variety of situations. That's ok: we released the preview of the Task Parallel Library in the December CTP to solicit early feedback on the APIs we're providing (e.g. Are they useful? What's missing? etc.), and to do so and to get the preview out early, we did so at the expense of performance and reliability (it is a technology preview, after all). We'll be spending a whole lot of time getting the performance of these loops to be stellar; for now, please let us know whether the APIs are right so that we can adjust if they're not. And if you have interesting situations where the performance isn't great, we'd love to know about that, too, so that we factor that in when determining the best implementations to ship.

That said, there will always be some cases that we won't handle perfectly, due simply to the sheer number of variations and use cases that exist. And for those situations, we provide the lower-level Task construct with its replication capabilities to allow you to write custom loops that are best suited to your specific needs (as an example, we previously posted about how to write a parallel while loop).

Consider another situation: you want a for loop, but the body of the for loop contains so little work that even the cost of a delegate invocation (to execute the body) is too much. For example, consider this implementation of a very simple ParallelFor loop:

static void ParallelFor(
    int fromInclusive, int toExclusive, Action<int> body)
{
    int index = fromInclusive;
    Task.Create(delegate
    {
        int i;
        while ((i = Interlocked.Increment(
            ref index) - 1) < toExclusive)
        {
            body(i);
        }
    },
    TaskCreationOptions.SelfReplicating).Wait();
}

For loop bodies with a significant amount of work, this implementation isn't all that bad. But for small bodies, there is a fairly large overhead here per iteration, in that each iteration incurs both an interlocked operation (to advance the index variable) and a delegate invocation (to run the body). Following this same approach, I could cut down on the number of interlocked operations by incrementing each time by an amount larger than 1:

static void ParallelFor(
    int fromInclusive, int toExclusive, Action<int> body)
{
    const int STRIDE = 8;
    int index = fromInclusive;
    Task.Create(delegate
    {
        int i;
        while ((i = Interlocked.Add(
            ref index, STRIDE) - STRIDE) < toExclusive)
        {
            int endExclusive =
                Math.Min(i + STRIDE, toExclusive);
            for(; i<endExclusive; i++)
            {
                body(i);
            }
        }
    },
    TaskCreationOptions.SelfReplicating).Wait();
}

In spirit, this is actually very similar to the implementation of Parallel.For in the CTP (which, based on my previous comments about CTP performance, should hint at the performance quality of this example ;). Each CPU grabs 8 elements using an interlocked addition. This decreases the number of interlocked operations per iteration from 1 to approximately 1/8, but it does so at the expense of load balancing. And still, for each iteration we have the cost of a delegate invocation in order to execute the body of the loop as provided by the caller. In fact, no matter what we do here, with this overall design of ParallelFor, we won't be able to avoid that delegate invocation, which could be prohibitive for certain problems with very, very small bodies.

To fix that, we could go with a different design:

static void ParallelForRange(
int fromInclusive, int toExclusive,
Action<int, int> body);

The idea here is that rather than providing a loop body that accepts the current iteration, the loop body accepts two integers, one representing the start of a range and the other representing the end (just as with fromInclusive and toExclusive, but on a smaller scale):

static void ParallelForRange(
    int fromInclusive, int toExclusive,
    Action<int, int> body)
{
    const int STRIDE = 8;
    int index = fromInclusive;
    Task.Create(delegate
    {
        int i;
        while ((i = Interlocked.Add(
            ref index, STRIDE) - STRIDE) < toExclusive)
        {
            body(i, Math.Min(i + STRIDE, toExclusive));
       }
    },
    TaskCreationOptions.SelfReplicating).Wait();
}

To use this, rather than doing the following as you would with the ParallelFor method:

ParallelFor(0, N, i =>
{
// original loop body here
});

you could do the following:

ParallelForRange(0, N, (fromInclusive, toExclusive) =>
{
    for (int i = fromInclusive; i < toExclusive; i++)
    {
        // original loop body here
    }
});

Just as with the previous ParallelFor implementation, where we used a stride to decrease the number of interlocked operations, here in ParallelForRange we've used that same stride to decrease the number of delegate invocations, at the expense of forcing the user to implement their own loop inside of the parallel loop.

Now, as previously mentioned, this striding approach has a downside of decreasing the possibility for load balancing. The implementation of Parallel.For that we eventually ship will do its best to both decrease overhead per iteration and maximize load balancing. But there may still be cases where the cost of the delegate invocation per iteration will be too much. Forgetting the implementation details for a moment, would it be useful for you if in addition to Parallel.For, we also provided a Parallel.ForRange with a design similar to the method I just showed? We'd love your feedback on issues like this, as we want to make sure we're providing the right solutions for your problem domain. And are there other common looping constructs that you'd like to see us provide? Now is a great time to download the CTP, try it out, and send us your feedback... we're listening.

Ordering the output of parallel computations

toub — Wed, 12 Mar 2008 07:52:00 GMT

Frequently when attempting to do multiple operations in parallel, ordering becomes an issue. Consider an application where I'm rendering and writing out to a video file frames of a movie:

for (int i = 0; i < numberOfFrames; i++)
{
var frame = GenerateFrame(i);
WriteToMovie(frame);
}

For a bit of pizzazz, I'll show the same thing with LINQ:

var frames = from i in Enumerable.Range(0, numberOfFrames)
select GenerateFrame(i);
foreach (var frame in frames) WriteToMovie(frame);

Now, my GenerateFrame method is expensive and computationally intensive, so I'd like to generate frames in parallel. For this example, we'll assume that my GenerateFrame method is thread-safe and can be called concurrently (rendering one frame doesn't modify any state used to render another frame), but access to my WriteToMovie method must be serialized:

using (ManualResetEvent mre = new ManualResetEvent(false))
{
    int count = numberOfFrames;
    object obj = new object();
    for (int i = 0; i < numberOfFrames; i++)
    {
        ThreadPool.QueueUserWorkItem(state =>
        {
            var frame = GenerateFrame((int)state);
            lock(obj) WriteToMovie(frame);

            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();

        }, state);
    }
    mre.WaitOne();
}

Unfortunately, that's quite a bit of code overhead, and it also suffers from a fundamental ordering problem: the frames may end up being written to the output movie file in an arbitrary order, based on when the frame generation completes. A version with this ordering issue fixed might look like the following:

var frames = new Bitmap[numberOfFrames];
var events = (from i in Enumerable.Range(0, numberOfFrames)
              select new ManualResetEvent(false)).ToArray();

for (int i = 0; i < numberOfFrames; i++)
{
    ThreadPool.QueueUserWorkItem(state =>
    {
        int frameNum = (int)state;
        frames[frameNum] = GenerateFrame(frameNum);
        events[frameNum].Set();
    }, i);
}

for (int i = 0; i < numberOfFrames; i++)
{
    events[i].WaitOne();
    WriteToMovie(frames[i]);
}

This has the general effect I want, but there are still some unfortunately overheads here (for example, I'm creating, setting, and waiting on a ManualResetEvent per frame). It's also verbose. Instead, I can use Future<T> from Parallel Extensions to solve the same problem:

var frames = (from i in Enumerable.Range(0, numberOfFrames)
select Future.Create(() => GenerateFrame(i))).
ToArray();
foreach (var frame in frames) WriteToMovie(frame.Value);

I love how close this is to the original LINQ version (and how much less code it is the than my previous ThreadPool sample). Rather than selecting GenerateFrame(i), I'm selecting Future.Create(() => GenerateFrame(i)), and rather than calling WriteToMovie(frame), I'm calling WriteToMovie(frame.Value). With those changes, the frames are now being computed in parallel (note I'm also using ToArray to calls the entire query to be enumerated), and they're being written out to the movie file in the correct order without having to do any explicit locking or coordination. Sweet!

If I wanted to, I could do the same thing without LINQ, tracking the collection of futures explicitly:

var frames = new Queue<Future<Bitmap>>();
for (int i = 0; i < numberOfFrames; i++)
{
var num = i;
frames.Enqueue(Future.Create(() => GenerateFrame(num)));
}
while (frames.Count > 0) WriteToMovie(frames.Dequeue().Value);

I could also use PLINQ:

var frames = from i in Enumerable.Range(0, numberOfFrames).
AsParallel(ParallelQueryOptions.PreserveOrdering)
select GenerateFrame(i);
foreach (var frame in frames) WriteToMovie(frame);

Three different approaches to solving the same problem with Parallel Extensions, and all of them requiring less code (and less complicated code) than my ThreadPool example. Just makes you smile, doesn't it? :)

Task Parallel Library on Channel 9

toub — Wed, 20 Feb 2008 01:38:00 GMT

Charles from Channel 9 sat down with several of us from the Parallel Computing Platform team to discuss the Task Parallel Library component of Parallel Extensions. A video of the conversation is now available on Channel9: http://channel9.msdn.com/Showpost.aspx?postid=384229. We hope you like it, and as always, feedback is welcome and appreciated!

Recursion and Concurrency

toub — Thu, 31 Jan 2008 22:55:27 GMT

When teaching recursion in an introductory computer science course, one of the most common examples used involves a tree data structure. Trees are useful in this regard as they are simple and recursive in nature, with a tree's children also being trees, and allow for teaching different kinds of traversals (in-order, pre-order, post-order, level-order, and so forth). But when these introductory concepts meet multithreading, the concepts are no longer so simple, at least not with the tools available in mainstream languages today like C#, C++, and Java. Consider a simple Tree data structure:

class Tree<T>
{
public Tree<T> Left, Right; // children
public T Data; // data for this node
}

Let's say we want to execute an Action<T> for each datum stored in the tree, and let's assume I don't care about order (introducing parallelism could be questionable if I did). That's straightforward to do sequentially and recursively:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Process the current node, then the left, then the right
    action(tree.Data);
    Process(tree.Left, action);
    Process(tree.Right, action);
}

I could also do so without recursion by maintaining an explicit stack (or queue, or some other data structure with different ordering guarantees):

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;
    var toExplore = new Stack<Tree<T>>();

    // Start with the root node
    toExplore.Push(tree);
    while (toExplore.Count > 0)
    {
        // Grab the next node, process it, and push its children
        var current = toExplore.Pop();
        action(current.Data);
        if (current.Left != null)
            toExplore.Push(current.Left);
        if (current.Right != null)
            toExplore.Push(current.Right);
    }
}

Now, let's assume the action we're performing on each node of the tree is independent, relatively expensive and/or that the tree is relatively large, and as such we want to process the tree in parallel (we're of course also assuming that the action delegate is thread-safe), meaning that we want multiple threads each running the action delegate on distinct tree nodes. How do we do this with what we have in .NET today?

There are multiple approaches, some more valid than others. The first thing someone might try is to follow the original recursive implementation but using the ThreadPool, which could look something like this:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to prevent this method from
    // returning until its children have completed
    using (var mre = new ManualResetEvent(false))
    {
        // Process the left child asynchronously
        ThreadPool.QueueUserWorkItem(delegate
        {
            Process(tree.Left, action);
            mre.Set();
        });

        // Process current node and right child synchronously
        action(tree.Data);
        Process(tree.Right, action);

        // Wait for the left child
        mre.WaitOne();
    }
}

The idea behind this implementation is to, given a node, spin up a work item to process that node's left child in parallel with the current node, and then process the current node's data as well as its right child. Of course, I could be losing out on some parallelism here as I delay processing of the right child until I'm done processing the current data. So we modify it slightly:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to wait for the children
    using (var mre = new ManualResetEvent(false))
    {
        int count = 2;

        // Process the left child asynchronously
        ThreadPool.QueueUserWorkItem(delegate
        {
            Process(tree.Left, action);
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        });

        // Process the right child asynchronously
        ThreadPool.QueueUserWorkItem(delegate
        {
            Process(tree.Right, action);
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        });

        // Process the current node synchronously
        action(tree.Data);

        // Wait for the children
        mre.WaitOne();
    }
}

I've now fixed that issue, such that both the left and the right children could potentially be processed in parallel with the current node, but that was by far not the worst problem. For starters, I'm creating a ManualResetEvent for every node in the tree... that's expensive. ManualResetEvent is a thin wrapper around a Win32 kernel event primitive, so creating one of these things requires kernel transitions, as does setting and waiting on one. Next, every time I process a node, I block waiting for its children to complete. And as the processing of a node (all but the root) is happening on a thread from the ThreadPool, I'm blocking ThreadPool threads. If a ThreadPool thread gets blocked, the ThreadPool will need to inject additional threads in order to process the remaining work items, and thus this implementation will require approximately one thread from the pool per node in the tree. That's a lot of threads! And that carries with it some serious problems. By default, a thread in .NET has a megabyte of stack space committed for it, so each thread burns a megabyte of (virtual) memory. The ThreadPool also throttles the creation of additional threads, such that introducing a new thread (once the number of pool threads equals the number of processors) will take 500 ms. For a tree of 250 nodes, that means its processing will take close to 2 minutes, purely for the overhead of creating threads, nevermind the actual processing of the nodes. And worse, there is a maximum number of threads in the pool: in .NET, the ThreadPool has a limited number of threads, by default 25 per processor in .NET 1.x/2.0 and 250 per processor in .NET 2.0 SP1. If the pool reaches the maximum, no new threads will be created, and thus this implementation could deadlock. Parent nodes will be waiting for their child nodes to complete, but the child nodes can't be processed until their parent nodes complete and relinquish a thread from the pool to execute.

Obviously, we need a better implementation. Next up, we can walk the tree sequentially, queuing up a work item in the pool for each node in the tree. Here we take that approach based on the recursive implementation of a tree walk:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to wait for all of the nodes to complete
    using (var mre = new ManualResetEvent(false))
    {
        int count = 1;

        // Recursive delegate to walk the tree
        Action<Tree<T>> processNode = null;
        processNode = node =>
        {
            if (node == null) return;

            // Asynchronously run the action on the current node
            Interlocked.Increment(ref count);
            ThreadPool.QueueUserWorkItem(delegate
            {
                action(node.Data);
                if (Interlocked.Decrement(ref count) == 0)
                    mre.Set();
            });

            // Process the children
            processNode(node.Left);
            processNode(node.Right);
        };

        // Start off with the root node
        processNode(tree);

        // Signal that no more work items will be created
        if (Interlocked.Decrement(ref count) == 0) mre.Set();

        // Wait for all of the work to complete
        mre.WaitOne();
    }
}

This implementation is better from a resources perspective and won't lead to the same kind of deadlocks as in the previous example. We're still creating a work item for each node, but now the work item is simply to asynchronously execute the action on the node's data, whereas the tree walk itself is still happening sequentially on the main thread (taking advantage of a recursive delegate). To maintain blocking behavior similar to the previous examples (where the call to Process won't return until all of the work has completed) though not identical (the processing of children nodes don't block the parent), a counter is incremented before each work item is created and is decremented when each work item finishes; only when that counter reaches 0 will the main thread continue (note how we need to start the counter at 1 and then decrement it after all of the work items have been created... this is to ensure that a work item completing before all of the work has been kicked off doesn't set the event prematurely). We can take the same approach with a variant of the iterative implementation shown earlier:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Use an event to wait for all of the nodes
    // to complete
    using (var mre = new ManualResetEvent(false))
    {
        int count = 1;

        // Start with the root node
        var toExplore = new Stack<Tree<T>>();
        toExplore.Push(tree);

        // Process all of the nodes
        while (toExplore.Count > 0)
        {
            // Get the current node and and push its children
            var current = toExplore.Pop();
            if (current.Left != null)
                toExplore.Push(current.Left);
            if (current.Right != null)
                toExplore.Push(current.Right);

            // Asynchronously process the data
            Interlocked.Increment(ref count);
            ThreadPool.QueueUserWorkItem(delegate
            {
                action(current.Data);
                if (Interlocked.Decrement(ref count) == 0)
                    mre.Set();
            });
        }

        // Signal that no more work items will be created
        if (Interlocked.Decrement(ref count) == 0) mre.Set();

        // Wait for all work items to be completed
        mre.WaitOne();
    }
}

These approaches work by queueing a work item per node in the tree. There are, of course, other approaches. Rather than adding a work item per node, we can create N work items, and have each work item process approximately 1/Nth of the nodes in the tree, where N is the number of threads. For example, we could store all of the nodes into a list and then split that list into N pieces and have a work item process each piece:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Create a list of all nodes in the tree
    var nodes = new List<Tree<T>>();
    var toExplore = new Stack<Tree<T>>();
    toExplore.Push(tree);
    while (toExplore.Count > 0)
    {
        var current = toExplore.Pop();
        nodes.Add(current);
        if (current.Left != null)
            toExplore.Push(current.Left);
        if (current.Right != null)
            toExplore.Push(current.Right);
    }

    // Divide the list up into chunks
    int workItems = Environment.ProcessorCount;
    int chunkSize = Math.Max(
        nodes.Count / workItems, 1);
    int count = workItems;

    // Use an event to wait for all work items
    using (var mre = new ManualResetEvent(false))
    {
        // Each work item processes appx 1/Nth of the data items
        WaitCallback callback = state =>
        {
            int iteration = (int)state;
            int from = chunkSize * iteration;
            int to = iteration == workItems - 1 ?
                nodes.Count : chunkSize * (iteration + 1);

            while (from < to) action(nodes[from++].Data);

            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        };

        // The ThreadPool is used to process all but one of the
        // chunks; the current thread is used for that chunk,
        // rather than just blocking.
        for (int i = 0; i<workItems; i++)
        {
            if (i < workItems-1)
                ThreadPool.QueueUserWorkItem(callback, i);
            else
                callback(i);
        }

        // Wait for all work to complete
        mre.WaitOne();
    }
}

Rather than statically dividing the work up, I could also rewrite the code such that each of the work items themselves is responsible for pulling work to handle:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;

    // Get an enumerator for the tree
    IEnumerator<T> enumerator =
        GetNodes(tree).GetEnumerator();
    int workItems = Environment.ProcessorCount;
    int count = workItems;

    // Use an event to wait for all work items to complete
    using (var mre = new ManualResetEvent(false))
    {
        // Each work item will continually pull data from the
        // enumerator and process it until there is no more data
        // to process
        WaitCallback callback = delegate
        {
            while (true)
            {
                T data;
                lock (enumerator)
                {
                    if (!enumerator.MoveNext()) break;
                    data = enumerator.Current;
                }
                action(data);
            }
            if (Interlocked.Decrement(ref count) == 0)
                mre.Set();
        };

              // The ThreadPool is used to process all but one of the
        // chunks; the current thread is used for that chunk,
        // rather than just blocking.
        for (int i = 0; i < workItems; i++)
        {
            if (i < workItems-1)
                ThreadPool.QueueUserWorkItem(callback, i);
            else
                callback(i);
        }

        // Wait for all work to complete
        mre.WaitOne();
    }
}

// An enumerator for a tree
public static IEnumerable<T> GetNodes<T>(Tree<T> tree)
{
    if (tree != null)
    {
        yield return tree.Data;
        foreach (var data in GetNodes(tree.Left))
            yield return data;
        foreach (var data in GetNodes(tree.Right))
            yield return data;
    }
}

Here I use a C# iterator to create an enumerator for all of the nodes in the tree. A work item is queued for each processor, and each of those work items retrieves an item from the enumerator, processes it, and goes back for more. Since multiple threads are accessing the enumerator (which isn't thread-safe), retrieving an item from it must be done under lock. Unfortunately, that's expensive; a better implementation would shift to grabbing more items from the enumerator so that the lock could be taken fewer times.

There are still further possible approaches. For example, we could walk the tree until we get to nodes of a certain depth (such as the log of the number of cores) and then process all of the subtrees at that depth in parallel with each other.

All of these implementations also have potential locality issues. It's reasonable to assume (though certainly not guaranteed) that nodes in the tree near to each other were created near to each other in time and are likely near to each other in memory (this assumption may be more applicable to other recursive problems, such as ones to be discussed shortly). For best cache performance, we'd like to try to have nodes near each other processed by the same thread and near to each other in time so that cache hits are optimized. But all of these solutions have been, in effect, randomly assigning nodes to threads.

Once of the nice things about the tree walk example is that it's very simple; there are no pre- or post- processing steps necessary for a node, which means I don't actually need to block at the end of a parent in order to wait for its children as I did in some of the previous examples. The problem also allows me to start processing children of a node before I've finished processing its parent. Many recursive problems are not like that, however. Consider a typical recursive sort, like quick sort:

public static void Quicksort<T>(T[] arr, int left, int right)
    where T : IComparable<T>
{
    if (right > left)
    {
        int pivot = Partition(arr, left, right);
        Quicksort(arr, left, pivot - 1);
        Quicksort(arr, pivot + 1, right);
    }
}

Order here definitely makes a difference. If we think of the Partition step like the execution of the Action in the tree walk example, I'm not able to do the Partition in parallel with the processing of the children, since the processing of the children is dependent on the outcome of the call to Partition. That's a pretty easy problem to workaround of course; I just wouldn't start the children executing asynchronously until after the Partition step is complete. But now consider a recursive sort like merge sort:

public static void Mergesort<T>(T[] arr, int left, int right)
    where T : IComparable<T>
{
    if (right > left)
    {
        int mid = (right + left) / 2;
        Mergesort(arr, left, mid);
        Mergesort(arr, mid + 1, right);
        Merge(arr, left, mid + 1, right);
    }
}

The merge step at a particular node must occur after the recursive calls, which means in a parallel implementation either we'd need to block waiting for the two recursive calls to complete before starting the call to Merge, or we'd have to use a form of continuation-passing style to join on the two merge calls and have their completion start the execution of the merge operation. Either way, this kind of processing with the ThreadPool is incredibly tricky to implement efficiently.

One of our goals when designing Parallel Extensions to the .NET Framework was to make such recursive parallel operations much easier to implement, and much easier to implement efficiently.

Let's start with PLINQ and our tree walk. Taking the Tree<T> iterator we already implemented, processing the tree in parallel is now a cakewalk:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
if (tree == null) return;
GetNodes(tree).AsParallel().ForAll(action);
}

Under the covers, this is very much like our implementation where we created several work items, and each of those work items pulled items (under lock) from the enumerator. PLINQ is more efficient, however, and uses varying chunk sizes (rather than always pulling just one element at a time) in an attempt to minimize the number of times the lock must be taken, hopefully reducing contention on the lock. This, of course, as with the similar previous implementation is doing a sequential tree walk but running the actions in parallel. If we actually want to walk the tree in parallel, we can use the Task Parallel Library:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;
    Parallel.Do(
        () => action(tree.Data),
        () => Process(tree.Left, action),
        () => Process(tree.Right, action));
}

It's almost scary how simple that is. The Parallel.Do statement potentially runs the three provided operations in parallel, blocking until all three have completed. A really cool thing about this though is that it will attempt to reuse the current thread as much as possible, so we don't run into the same kinds of issues that we had with the ThreadPool where we were blocking a whole slew of ThreadPool threads. The underlying work-stealing nature of the implementation also goes a long way towards helping to address the cache locality issues we previously mentioned. If we want more control, we can dive down below the Parallel class and explicitly use Tasks:

public static void Process<T>(Tree<T> tree, Action<T> action)
{
    if (tree == null) return;
    var t1 = Task.Create(
        delegate { Process(tree.Left, action); });
    var t2 = Task.Create(
        delegate { Process(tree.Right, action); });
    action(tree.Data);
    Task.WaitAll(new Task[] { t1, t2 });
}

The same techniques can apply to the quicksort implementation:

public static void Quicksort<T>(T[] arr, int left, int right)
    where T : IComparable<T>
{
    if (right > left)
    {
        int pivot = Partition(arr, left, right);
        Parallel.Do(
            () => Quicksort(arr, left, pivot - 1),
            () => Quicksort(arr, pivot + 1, right));
    }
}

as well as to the mergesort implementation:

public static void Mergesort<T>(
    T[] arr, int left, int right) where T : IComparable<T>
{
    if (right > left)
    {
        int mid = (right + left) / 2;
        Parallel.Do(
            () => Mergesort(arr, left, mid),
            () => Mergesort(arr, mid + 1, right));
        Merge(arr, left, mid + 1, right);
    }
}

I love how simple the parallelization of this code becomes. However, this code still has some issues. There is necessary overhead to an operation like Parallel.Do, which under the covers creates Tasks for the individual operations and waits on those Tasks; that overhead may not be a big deal if the work being done is significant, but for fine-grain concurrency that overhead can end up dominating the computation, potentially causing the parallel implementation to run slower than the sequential implementation. For now, the best way to work around that is to complicate the code slightly with thresholds. The idea behind a threshold is that you introduce enough parallelism to keep the system saturated, but once enough has been expressed, you switch over to a sequential implementation so as to avoid the extra overhead. With the tree walk, using a depth as a threshold might look like this:

private static void Process<T>(
    Tree<T> tree, Action<T> action, int depth)
{
    if (tree == null) return;
    if (depth > 5)
    {
        action(tree.Data);
        Process(tree.Left, action, depth + 1);
        Process(tree.Right, action, depth + 1);
    }
    else
    {
        Parallel.Do(
            () => action(tree.Data),
            () => Process(tree.Left, action, depth + 1),
            () => Process(tree.Right, action, depth + 1));
    }
}

I have both a sequential and a parallel implementation here, and I switch from one to the other based on how deep in the tree I am. Depth may not always be the right gating factor to use as a threshold, especially if the tree isn't balanced. We're working through ways to reduce the overhead we currently see for these operations in the early CTP we released in December, and we're testing out some approaches that may eliminate the need to use thresholds in certain cases. For now, though, as you're playing around with the bits, keep thresholds in mind. The same technique could be used with both sorting examples, as well.

All in all, parallelism is a very interesting problem when it comes to recursion, and we hope the Parallel Extensions to the .NET Framework go a long way towards making the merging of these two concepts significantly easier.

LINQ 101, "Parallelism Blockers," and PLINQ

toub — Tue, 18 Dec 2007 01:54:15 GMT

PLINQ is a very cool technology, and I believe it will prove useful for parallelizing operations in a wide range of important scenarios. Moreover, I believe that the programming model it provides will enable a wide-range of developers to easily take advantage of concurrency in their applications. However, one of the risks involved in our providing such a simple programming model is that we don't currently shield developers from all of the possible issues they may run into from using it to parallelize code.

As an example, there is a set of 101 LINQ samples available at http://msdn2.microsoft.com/aa336746.aspx. Unfortunately, many of these samples rely on implementation details and behaviors that don't necessarily hold when moving to a parallel model like the one employed by PLINQ. In fact, some of them are dangerous when it comes to PLINQ.

On our team, we've been referring to these kinds of dangers as "parallelism blockers," and they're one of the primary reasons we're not planning to automatically replace all LINQ-to-Objects queries with PLINQ queries, instead supporting the opt-in model available through the AsParallel extension method. We've talked about some of these issues before (see the CTP documentation for a more thorough look at some of these issues), but as a refresher, consider a few examples:

In your Windows Forms application, you're using LINQ to loop over a long list of UI controls, setting properties on each or searching for one control in particular. You decide to parallelize this with PLINQ. Boom! You get an InvalidOperationException, "Cross-thread operation not valid." In general, properties on UI controls can only be accessed by code running on the thread that created those controls; by using PLINQ, background threads were executing portions of the query, and those portions were accessing controls.
You're using LINQ to search a list containing just a few items for an element with a property set to a specific value. You switch over to PLINQ, and the search takes longer! Why? There is overhead in PLINQ to partitioning the work in order to execute on multiple threads, to merging that work back together in order to produce the output, and so forth. If you're only dealing with a little work, PLINQ likely won't help, and it may make execution slower. (Of course, if it's not taking much time to execute in the first place, one might question the value of parallelism for it.)
You're using LINQ to search a list, and in the process, you're updating a value, such as a count. You switch over to PLINQ, and now your count isn't reflecting the amount of work you've actually done. Why? LINQ was executing the updates sequentially, but PLINQ is executing them in parallel, and thus without synchronization, you have a bad race condition. (NOTE: please, please, please do not modify shared state in LINQ-to-Objects queries)

The 101 LINQ samples are not good exemplars for PLINQ. Consider the Query Execution category. These samples contain code like:

int i = 0;
var q = from n in numbers select ++i;

NOOOOOOO!!!!!!!!!!!!!! With PLINQ, those increments to i will potentially happen in parallel, and they won't be atomic. You could fix this by using Interlocked.Increment rather than ++, but then you'd likely destroy your performance.

Luckily, only a few of the 101 samples modify state. But that doesn't mean switching over to PLINQ will produce the same results as LINQ for all of these. Almost all of the 101 samples have the potential to produce different output orderings than the LINQ versions due to lack of order preservation by default (at least in the CTP; we're reconsidering whether that is a good default, so if you have any opinion on the issue, please let us know). For most of the samples, this doesn’t matter semantically (except for a hypothetical tool that does a straight output to output comparison between LINQ-to-Objects and PLINQ runs); for example, the first sample lists all of the numbers from the array { 5, 4, 1, 3, 9, 8, 6, 7, 2, 0 } where the numbers are less than 5, and the second sample lists all of the sold out products from a dynamically generated list of products. However, there are some where (depending on the intended usage) ordering may matter. For example, the eighth sample generates the name of a number in the array (e.g. 5 to “five”); the output ordering may not match up to the input array in this case, and thus while you’re successfully generating all of the relevant outputs, the correlation between input and output is lost (you’d either need to update the query to include in the results the original position, or you’d need to enable order preservation). There are other sample queries that will have this same issue.

Additionally, almost all of the samples deal with very small data sets, and do very little work on each item. It's likely that PLINQ's overhead will dominate in many of these cases, producing slower runs than if LINQ-to-Objects were used. We're spending a lot of time now on PLINQ performance, and we're hoping to see that overhead decrease as much as possible, but even with all of the optimizations we plan to throw at it, it's likely there will still be some queries for which the non-parallel LINQ-to-Objects will still be preferred. That's one place where we need your help. If you have queries that you believe should be executing much faster in parallel than they are, please send them our way; we'd love to analyze them to figure out where bottlenecks may be.