Parallel Programming with .NET : Beta

What's New in Beta 2 for PLINQ

essey — Thu, 05 Nov 2009 20:37:00 GMT

Included in the .NET 4 Framework Beta 2 is a more robust and faster version of PLINQ. Between B1 and B2, PLINQ changes have mainly been under the covers, so hopefully no need to rewrite any of your applications to see the improvements.

1. Many improvements to performance and scalability

2. GroupBy and GroupJoin now preserve ordering within the groups

3. Better integration with performance profiling, which now shows PLINQ markers

4. Cancellation consistently works across all PLINQ operators

5. Scoped some of the boundary conditions of PLINQ in .NET 4 in Beta2 of Parallel Extensions as follows:

a. Max Degree of Parallelism changed from 64 to 63

b. Greater consistency regarding maximum input length

i. PLINQ does not support inputs of length > Int32.MaxValue. Queries may throw OverflowException for many operators.

ii. LongCount no longer counts beyond Int32.MaxValue

6. Improved robustness

a. Reliability and concurrency focus

b. More consistent exception handling and error model

7. Other bug fixes

If any of these severely impact your applications, particularly the boundary conditions, please let us know so that we can include that in our planning efforts. For now, we found that it makes the most sense to scope some inputs and give a reliable and predictable experience in this release.

Happy Coding!

What’s new in Beta 2 for the Task Parallel Library? (Part 3/3)

dashih — Wed, 04 Nov 2009 21:52:00 GMT

Last time, we covered Tasks being detached by default and some refactorings in our multiple-Task continuation APIs. The final post of this series will discuss Nested Tasks and Unwrap, a Parallel namespace change, and some changes under the covers.

Nested Tasks and Unwrap

We’ve added the Unwrap APIs to address scenarios that deal with nested Tasks. Before jumping into Unwrap, let’s first talk about nested Tasks, e.g. a Task<Task> or Task<Task<TResult>>. For example:

var nestedTask = Task.Factory.StartNew(() =>
{
    return Task.Factory.StartNew(() =>
    {
        return 42;
    });
});

Nested Tasks commonly lead to unexpected behavior in applications. For example, consider an API that provides the following functionality for asynchronously logging into a web service (like one from Facebook), retrieving a list of friends, and sending an email to each friend.

// Given a user name and password, returns a Task whose

// result is a UserToken object.

public Task<UserToken> LogOn(string username, string password);

// Given a UserToken, returns a Task whose result

// is a collection of Friend objects.

public Task<FriendCollection> GetFriendList(UserToken userToken);

// Given a Friend, subject, and body, returns a Task that

// represents an email sending operation.

public Task SendEmail(Friend friend, string subject, string body);

A user would like to be able to write code like the following, utilizing these APIs and Task continuations:

var userToken = LogOn(user, pass);

var friends = userToken.ContinueWith(_ => GetFriendList(userToken.Result));

var emails = friends.ContinueWith(_ =>

{

var sentMails = new List<Task>();

foreach(var friend in friends.Result)

{

sentMails.Add(SendEmail(friend, subject, body));

}

return Task.Factory.ContinueWhenAll(

sentMails.ToArray(), tasks => Task.WaitAll(tasks));

});

emails.ContinueWith(_ => Console.WriteLine(“All emails sent”));

The bolded code is problematic. Because the GetFriendList method returns a Task<FriendCollection>, the ‘friends’ variable is actually going to be a Task<Task<FriendCollection>>. This will cause a compiler error at the foreach loop, because ‘friends.Result’ will return a Task<FriendCollection> instead of a FriendCollection. The compiler error in this case is a good thing, of course, highlighting a programming error. However, once a developer realizes the type mismatch, he still has to code additional logic to unwrap the ‘friends’ Task so that it returns an actual FriendCollection. This logic is nontrivial, especially if it is to correctly deal with exceptions, cancellation, etc. The last line is also problematic. The emails variable here is actually of type Task<Task>. The outer Task will complete once the inner Task is returned from its body, even if the inner Task hasn’t completed. The net result of this is that “All emails sent” could be written out prior to all of the email tasks actually completing.

Now, you may have noticed that, in Beta 1, we provided special ContinueWith overloads to deal with this type of scenario. For example:

public Task<TResult> ContinueWith<TResult>(

Func<Task, Task<TResult>> continuationFunction);

The Func returns a Task<TResult>, so normally, ContinueWith would return a Task<Task<TResult>>. But this ContinueWith overload does some magic under the covers to return a Task<TResult>. There were a number of reasons why we didn’t like this approach, including:

· Too much magic. It’s hard to explain why one set of ContinueWith overloads is “just different”.

· Only works for ContinueWith. What if user scenarios result in nested Tasks for other Task creation APIs like StartNew, ContinueWhenAll, etc.?

· What if nested Tasks are actually desired? If a user actually wants that Task<Task<TResult>>, he still might unknowingly bind to this magical overload.

Given these reasons, our solution for Beta 2 and beyond is two Unwrap extension methods.

namespace System.Threading.Tasks

{

public static class TaskExtensions

{

public static Task Unwrap(

this Task<Task> task);

public static Task<TResult> Unwrap<TResult>(

this Task<Task<TResult>> task);

}

These methods may be used to transform any Task<Task> or Task<Task<TResult>> into a Task or Task<TResult>, respectively. The transformation performed produces a Task or Task<TResult> that fully represents the original nested Task, including exceptions, cancellation state, etc.

With Unwrap, we can fix the above scenario (note the bolded).

var userToken = LogOn(user, pass);

var friends = userToken.ContinueWith(

_ => GetFriendList(userToken.Result)).Unwrap();

var emails = friends.ContinueWith(_ =>

{

var sentMails = new List<Task>();

foreach(var friend in friends.Result)

{

sentMails.Add(SendEmail(friend, subject, body));

}

return Task.Factory.ContinueWhenAll(

sentMails.ToArray(), tasks => Task.WaitAll(tasks));

}).Unwrap();

emails.ContinueWith(_ => Console.WriteLine(“All emails sent”));

Parallel Namespace Change

We’ve moved the Parallel class from the System.Threading namespace to the System.Threading.Tasks namespace. We found that most applications needed to bring in both namespaces when using TPL, so why not put everything into one namespace? Additionally, Parallel is such a common word (and will likely become more so in the future), and System.Threading such a common namespace, we wanted to reduce the chances of conflict with other .NET types as much as possible.

Here’s a useful IDE tip. Use “Ctrl + .” to automatically bring in the relevant namespace once you’ve typed a class/type name.

Under the Covers

Beta 2 contains quite a few bug fixes not done for Beta 1. All of them were important, but we’ll focus on only two here.

First, Parallel.For and ForEach have been tweaked for better load-balancing with other workloads in the current or other AppDomains. Essentially, the change was to service the parallel loops with Tasks that periodically retired and re-queued themselves, allowing other contenders to grab Threads and make progress.

Second, waiting for Tasks in parent/child relationships has become more efficient. In Beta 1 and before, parent Tasks waited for their children using explicit Waits, so even if a parent completed first, it would burn a thread until all of its children completed. In Beta 2, parent Tasks wait for their children using callbacks; the parent maintains a count for number of children it has, and each child decrements the count as it completes. This waiting logic significantly improves scalability.

That’s it folks! We hope you’ve enjoyed this series. Download Beta 2 and try it out!

Samples for Parallel Programming with the .NET Framework 4 Beta 2

toub — Mon, 02 Nov 2009 19:27:00 GMT

We've posted a whole bunch of samples on Code Gallery showcasing how to use the new parallelism support in the .NET Framework 4. You can find them at http://code.msdn.microsoft.com/ParExtSamples. Enjoy!

What’s new in Beta 2 for the Task Parallel Library? (Part 2/3)

dashih — Tue, 27 Oct 2009 18:02:00 GMT

Last week, we talked about how TPL adopted a new, better cancellation model. Today, we’ll cover a change that makes Tasks Detached by Default, some ContinueWhenAll/Any Refactoring, and the handy UnobservedTaskException event.

Tasks are Detached by Default

In Beta 2, we have changed an important default. Tasks are now created as detached (instead of attached) if no options specify otherwise.

Let’s consider the following code to review the difference between attached and detached Tasks.

Task p = Task.Factory.StartNew(() =>

{

Task c = Task.Factory.StartNew(() =>

{

DoWork();

});

p.Wait();

In Beta 1 and before, since the default options are used, ‘c’ is created as a child Task of Task ‘p’, the parent Task; we refer to this as Task ‘c’ being “attached” to Task ‘p’. This means that the p.Wait() statement will not return until the call to DoWork completes, because parent Tasks do not complete until all of their child Tasks complete. To opt out of this behavior, a user needs to create ‘c’ with the DetachedFromParent option:

Task c = Task.Factory.StartNew(() =>

{

DoWork();

}, TaskCreationOptions.DetachedFromParent);

The original code shown behaves differently in Beta 2. Now, by default, ‘c’ is not related to ‘p’ (it’s “detached” by default), and the p.Wait() statement will return as soon as ‘p’ completes, regardless of the status of Task ‘c’ and thus regardless of when DoWork returns. To opt in to the parent/child relationship, a user needs to create ‘c’ with the AttachedToParent option:

Task c = Task.Factory.StartNew(() =>

{

DoWork();

}, TaskCreationOptions.AttachedToParent);

Here is a summary of the changes:

· Removed the DetachedFromParent option

· Added the AttachedToParent option

· Changed the default behavior so that Tasks do not enlist in parent/child relationships when no options are specified.

There were a number of reasons why we decided that detached is the correct default and to move forward with this change, including:

· Many users were using attached Tasks unknowingly. The vast majority of the time, users create Tasks for simple, asynchronous work. In such scenarios, parent/child relationships (and the implicit waiting) are not needed. We found through many interactions that folks were just going with the default options and were accidentally opting in to this behavior. In the best case, this would only result in a slight performance cost. In the worst case, this would bring with it incorrect behavior that would lead to difficult to diagnose errors.

· Easier migration from ThreadPool.QueueUserWorkItem. Tasks are now the recommended way to queue work to the ThreadPool, but the easiest way to create Tasks resulted in different behavior from QueueUserWorkItem (where there’s no concept of parent/child work items). This change makes Task.Factory.StartNew (with no options) a true replacement for QueueUserWorkItem.

· Additional behavior should be opt-in and pay-for-play. Almost everything in TPL that results in additional behavior is opt-in, e.g. cancellation, LongRunning, PreferFairness. With the Beta 1 default, users opt-out of parent/child relationships. In Beta 2, users opt-in, making it consistent. This makes the extra functionality provided by parent/child relationships pay-for-play, such that you don’t pay the cost for parents implicitly waiting for their children or for exceptions propagating from children to parents unless you need that functionality.

ContinueWhenAny/All Refactoring

We have refactored the set of ContinueWhenAny and ContinueWhenAll overloads to make things more intuitive, consistent, and complete.

To demonstrate the main issue, let’s consider the following overload that was provided in Beta 1.

public class TaskFactory<TResult>

{

public Task<TNewResult> ContinueWhenAny(

Task<TResult>[] tasks,

Func<Task<TResult>, TNewResult> continuationFunction);

}

This confused the meaning of TaskFactory<TResult>, which is meant to create tasks of type Task<TResult>. However, with these overloads, TaskFactory<TResult> could be used to create tasks of type Task<TNewResult>. As an example, consider the code:

Task<int>[] taskOfInts = ...;
Task<string> t = Task<int>.Factory.ContinueWhenAll(taskOfInts, _ => “”);

This compiles and works just fine, but the type parameter mismatch (shown in bold) is certainly odd. To address this, we changed a bunch of overloads, so that instead of taking Task<TResult>s and returning a Task<TNewResult>, they take Task<TAntecedentResult>s and return Task<TResult>s. For example, the overload that replaced the above is:

public Task<TResult> ContinueWhenAny<TAntecedentResult>(

Task<TAntecedentResult>[] tasks,

Func<Task<TAntecedentResult>, TResult> continuationFunction);

And the above example becomes:

Task<int>[] taskOfInts = ...;
Task<string> t = Task<string>.Factory.ContinueWhenAll(taskOfInts, _ => “”);

In addition to this change, we also added, removed, or modified a number of other overloads to make the set consistent and complete. Now, the entire set of ContinueWhenAll and ContinueWhenAny overloads follow these clear rules:

· A TaskFactory creates Tasks, but also provides overloads to create Task<TResult>s.

· A TaskFactory<TResult> only ever creates Task<TResult>s (never Tasks or Task<TNewResult>s).

UnobservedTaskException event

We’ve added an event that fires for every Task exception that goes unobserved. Recall that to “observe” a Task’s exceptions, you must either Wait on the Task or access its Exception property after it has completed. At least one of these actions must be done before the Task object is garbage collected, or its exceptions will propagate (currently this occurs on the finalizer thread).

The new static event resides on the TaskScheduler class, and subscribing to it is straightforward. Here’s an example to log all unobserved exceptions and mark them as observed (preventing them from being propagated).

TaskScheduler.UnobservedTaskException +=

(object sender, UnobservedTaskExceptionEventArgs exceptionArgs) =>

{

exceptionArgs.SetObserved();

LogException(exceptionArgs.Exception);

};

Some customers have complained that TPL’s exception policy is too strict. The UnobservedTaskException event provides an easy way out by allowing you to simply squash all Task exceptions in an application (though using it in this manner is not recommended). The primary reason that we made the addition was to support host-plugin scenarios where a host application can still be perfectly useful in the presence of some truly harmless exceptions (thrown by buggy plugins). These scenarios may be achieved using the UnobservedTaskException event in conjunction with AppDomains to sandbox plugins. Look for a future post that describes this in more detail!

We’re done for now! The 3^rd and final post of this series will cover the new Unwrap APIs, a Parallel namespace change, and some changes under the covers.

What’s new in Beta 2 for the Task Parallel Library? (Part 1/3)

dashih — Mon, 19 Oct 2009 19:42:00 GMT

Visual Studio 2010 and .NET 4 Beta 2 is here! In terms of completeness and readiness for production coding, Beta 2 promises to be much better than Beta 1, and TPL is one component that delivers significant improvements over what was previously available. To get you excited about it, this series of posts details key additions and changes for Beta 2. Enjoy!

In this post, we’re talking about cancellation. A few months ago, our flurry of “What’s new in Beta 1” posts hinted at a new cancellation model and promised more information about it. We made good on that promise with .NET 4 Cancellation Framework, and this post goes further to explain how TPL, specifically, has fully adopted the new model.

The Old Way (Beta 1 and before)

Let’s consider the following code to review the old TPL cancellation model.

Task myTask = Task.Factory.StartNew(() =>

{

for (; ; )

{

if (Task.Current.IsCancellationRequested)

{

Task.Current.AcknowledgeCancellation();

return;

}

...

}

});

// Elsewhere.

myTask.Cancel();

myTask just loops infinitely, checking its IsCancellationRequested property to see if it has been canceled. Elsewhere, cancellation is requested on myTask using the Cancel method. At that point, myTask agrees to get canceled by calling its AcknowledgeCancellation method and returning. This is all necessary, because Task cancellation is cooperative; to enter the Canceled state, outside logic must request cancellation and the Task must acknowledge that cancellation request. However, note that cooperative cancellation is only relevant for already running Tasks. If a Task’s Cancel method is called before it is in the Running state, it will transition directly into the Canceled state.

Waiting on a canceled Task results in a TaskCanceledException wrapped in an AggregateException, hence the try/catch block. Executing this code prints “Canceled” to the console, indicating that myTask was successfully canceled.

This approach works, but we identified a number of problems with it including the following:

1. The cancellation model required exposing a Task.Current static property. This leaks implementation details from libraries that utilize Tasks internally; any code called from the Task can muck with the current Task or take dependencies on its existence. Imagine calling into 3^rd party code and having that code cancel your Tasks, schedule continuations off of them, etc.

2. Anyone with a Task’s reference can request cancellation on it. In many scenarios, it is valuable to separate the ability to check for cancellation and the ability to actually request cancellation.

3. From a cancellation perspective, Tasks didn’t compose well with other APIs that were cancelable, such as executing a cancelable PLINQ query inside of a Task.

4. Tasks were often more expensive than they needed to be, due to needing to track extra cancellation state per Task.

The New Way (Beta 2 and beyond)

The new TPL cancellation model is centered around two types: CancellationTokenSource and CancellationToken. You can read up about them in the post linked to above, but here is a simplified overview of their APIs:

namespace System.Threading
{
    public sealed class CancellationTokenSource
    {
        public void Cancel();
        public CancellationToken Token { get; }
        ...
    }

    public struct CancellationToken
    {
        public Boolean IsCancellationRequested { get; }
        public void ThrowIfCancellationRequested();
        ...
    }
}

Here’s the gist. A CancellationTokenSource contains a CancellationToken, and it can request cancellation on that token using a Cancel method. A CancellationToken can only check if cancellation has been requested on it. Ignore the ThrowIfCancellationRequested method for now; we’ll see why it’s handy later.

Adopting the new model involved not only adding support for these two types, but also ripping out the old model. Here’s a summary of the changes:

· All cancellation-related APIs on the Task class were removed (Cancel, AcknowledgeCancellation, IsCancellationRequested, etc)

· Other APIs that were no longer relevant were removed (Task.Current, Task.Parent, TaskCreationOptions.RespectParentCancellation, etc)

· Overloads that accept CancellationToken were added to many methods (StartNew, ContinueWith, etc)

And now, here’s a table that outlines how achieving cancellation in TPL has changed:

Action	Old Model	New Model

To set up cancellation	Just create a Task	Create a CancellationTokenSource and pass its Token to an API that creates a Task.
To check if cancellation has been requested	Check the IsCancellationRequested property on the relevant Task	Check the IsCancellationRequested property on the CancellationToken that was passed to the API that created the Task
To acknowledge cancellation	Check to ensure that IsCancellationRequested is true, then call the AcknowledgeCancellation() method on the relevant Task	Throw an OperationCanceledException with the task’s CancellationToken
To cancel a tree of tasks	Create all of the tasks as attached tasks and with the RespectParentCancellation flag set, then cancel the root task.	Pass the same CancellationToken to all Tasks, then cancel the associated CancellationTokenSource

To see how this all works, let’s rewrite the code above for Beta 2.

CancellationTokenSource cts = new CancellationTokenSource();

CancellationToken token = cts.Token;

Task myTask = Task.Factory.StartNew(() =>

{

for (; ; )

{

token.ThrowIfCancellationRequested();

...

}

}, token);

// Elsewhere.

cts.Cancel();

A CancellationTokenSource (cts) is initialized, and a CancellationToken (token) is initialized to cts’s Token. myTask still loops indefinitely, but now it calls ThrowIfCancellationRequested. This method just checks the IsCancellationRequested property on a CancellationToken. If true, it throws an OperationCanceledException(token), which is the way to acknowledge cancellation in the new model. Elsewhere, Cancel is called on cts to request cancellation on myTask.

The new model addresses all of the problems with the old model listed above:

1. No more Task.Current (Task.Parent was removed too, as it could be used to get at the current Task).

2. The ability to check for cancellation requests may now be separated from the ability to request cancellation. If only the former is desired, grant access to the CancellationToken only.

3. TPL now uses the new .NET 4 cancellation types. You can use the same CancellationToken{Source} to effect cancellation on multiple Tasks, Parallel.For calls, PLINQ queries, and more, all at once. This provides both better composition and better performance.

Finally, there are other advantages to this model. One is that it is much easier to organize cancellation sets within a large nest of Tasks; just create the right number of tokens and pass them around accordingly. Before, this scenario would have required tedious bookkeeping of many Task references and/or careful architecting of cancellation chains using TaskCreationOptions.RespectParentCancellation.

That’s it for now. The next post in the series will discuss a change regarding Tasks and parent/child relationships.

Does Parallel.For use one Task per iteration?

toub — Tue, 26 May 2009 18:31:35 GMT

In .NET 4, the new Parallel class provides For, ForEach, and Invoke methods for performing operations in parallel. One mental model that some folks use when thinking about Parallel.For is that it’s equivalent to running one System.Threading.Tasks.Task per iteration, e.g. that a loop like:

Parallel.For(0, N, i =>
{
DoWork(i);
});

is equivalent to:

var tasks = new List<Task>(N);
for(int i=0; i<N; i++)
{
tasks.Add(Task.Factory.StartNew(state => DoWork((int)state), i));
}
Task.WaitAll(tasks.ToArray());

From the perspective of every iteration potentially running in parallel with every other iteration, this is an ok mental model. However, it can also lead to some misunderstandings as to Parallel’s behavior. Parallel, in fact, does not necessarily use one Task per iteration, as that could add significantly more overhead than is necessary.

Under the covers, Parallel.For tries to use the minimum number of tasks necessary to complete the loop as fast as possible. A better mental model includes Parallel spinning up tasks as threads are available to process those tasks, and each of those tasks then participating in a range management scheme. A task asks for a chunk to be done (which may include multiple iterations), processes that work, and then goes back for more. The chunk sizes may vary based on a variety of factors, including the number of tasks participating, the load on the machine, the size of the iteration space, and so forth.

One of the most important takeaways from this revised mental model is that iterations are handed out in indivisible chunks, and only one thread is involved in the processing of a particular chunk. This has implications for interdependencies between iterations. If iteration i blocks waiting for iteration i+1 to be completed, and iterations i and i+1 are both allocated to the same chunk, the loop will likely deadlock. The thread processing those iterations will block processing iteration i, but as that thread is also responsible for processing iteration i+1, iteration i+1 will never get processed and iteration i will never unblock.

As such, if you find yourself in an obscure situation like this, you have several options. You can fall back to using a Task per iteration, as per the original model shown previously. You can also take advantage of custom partitioning support, which we’ll discuss more in a future post. Of course, Parallel provides significantly more functionality than is shown in that previous Tasks-based code snippet. Parallel tracks unhandled exceptions and prevents additional iterations from starting in the event of an exception. It supports cancellation, setting a maximum degree of parallelism, targeting a particular scheduler, breaking out of a loop early with ParallelLoopState.Stop/Break, and so forth. All of this functionality could be layered on top of Tasks with extra code and effort; after all, that’s how Parallel works (at least in the current bits; the implementation may of course change in the future).

Parallel.ForEach has similar implications as Parallel.For. A decent mental model for Parallel.ForEach when the source is indexible (like an array or a list) involves simply delegating to Parallel.For. For example, the following Parallel.ForEach loop:

T [] source = …;
Parallel.ForEach(source, item =>
{
DoWork(item);
});

could be thought of as being implemented like this:

T [] source = …;
Parallel.For(0, source.Length, i=>
{
var item = source[i];
DoWork(item);
});

Since our model for Parallel.For involves chunking, so too does our model for Parallel.ForEach. This is also true when Parallel.ForEach works with a non-indexable IEnumerable<T> as the data source. IEnumerable<T> isn’t thread-safe (even if the data source being enumerated is in some fashion), in that it requires two calls to retrieve the next item: one to MoveNext() to advance the enumerator, and one to Current to retrieve the item. Thus, when working with an enumerator, the threads involved in a Parallel.ForEach (as is also true with PLINQ) need to use a lock while accessing an enumerable data source. Taking that lock for each item in the data source proves typically to be prohibitively expensive, and thus Parallel.ForEach ammortizes the cost of the lock across multiple items by chunking the input source: acquire the lock, enumerate some number of elements from the data source into a private data structure, release the lock, and then process that privatized chunk (we’ll discuss this in more detail in a future post). As a result, this suffers from the same cross-iteration dependency issue as previously described. If the processing of element i in the enumerable blocks waiting for element i+1 to be processed, the processing may deadlock if those two elements happen to be in the same privatized chunk. As with Parallel.For, you can always fall back to using Tasks directly if you have these kind of unusual dependencies.

Parallel.Invoke is a more interesting creature. The signature of the primary Invoke overload is:

public static void Invoke(params Action[] actions);

This signature lends itself well to two different mental models. The first, as with Parallel.For/ForEach, is simply to use a Task to run each action:

var tasks = new List<Task>(actions.Length);
foreach(var action in actions)
{
tasks.Add(Task.Factory.StartNew(action));
}
Task.WaitAll(tasks.ToArray());

Another is based on observing that we’re really dealing with an array as an input data source, and we can use Parallel.ForEach:

Parallel.ForEach(actions, action => action());

In fact, the current (Beta 1) implementation of Parallel.Invoke uses approaches similar to both of these models, and it chooses which to use based on a variety of circumstances. This criteria includes the number of Actions provided (smaller numbers will favor the Task-per-Action approach, whereas larger will favor the ForEach approach) as well as whether ParallelOptions were provided to Invoke to control things like the degree of parallelism or cancellation.

At the end of the day, it’s important to remember that implementation details may change at any time. But having a good mental model for how something is implemented can also help to understand the constraints imposed by that functionality.

.NET 4 Cancellation Framework

mikelid — Sat, 23 May 2009 06:29:00 GMT

A very interesting addition to .NET 4 is a set of new types that specifically assist with building cancellation-aware applications and libraries. The new types enable rich scenarios for convenient and safe cancellation, and help simplify situations that used to be be difficult and error-prone and non-composable.

The details of the new types are described below, but lets begin with some of the motivating principles that the new types were designed to support:

When any unit of work is commenced, it should have a consistent means to support early termination in response to a cancellation request.
If some unit of work controls various other moving pieces, it should be able to conveniently chain cancellation requests to them.
Blocking calls should be able to support cancellation.
Calls to complex operations, such as MoveNext() on a PLINQ enumerator, should have simple yet comprehensive cancellation support.
When infrastructure makes calls back to potentially long-running user code, it should be possible for the user code to observe and respond to cancellation requests in a cooperative fashion.
Cancellation should be an obvious part of an API with clean and consistent semantics.
Cancellation should not be forceful but instead cooperative.

In many prevailing systems, cancellation has been a secondary feature that rarely gets treated in sufficient detail to enable all of the above principles in a comprehensive fashion. The new types introduced to .NET 4 raise cancellation to be a primary concept for .NET APIs and one that can be cleanly and easily incorporated into any system.

Previous approaches

Cancellation is a feature that finds its way into many APIs and applications either by design or by necessity. An interesting classification of common approaches is Herb Sutter's article "Interrupt Politely", which outlines four distinct techniques. Those four techniques can be distilled down to the two major techniques: The first is to use asynchronous, forced cancellation such as thread-abort, process-kill, or AppDomain-unload. Any asynchronous technique has the common trait that the target of cancellation is caught completely unaware and cannot easily ensure data safety. For many reasons, but particularly due to the risk of partial updates to shared data, the use of asynchronous techniques is strongly discouraged.

The second general technique is cooperative whereby a request for cancellation is communicated to the target in some manner and the target of cancellation observes and enacts the request itself. This is inherently safer, but it does require that all targets of cancellation are fully participating in the process. The most commonly implemented style for this approach involves every potential target object tracking some state that represents whether cancellation has been requested although as we shall see this has some inherent problems and can be improved.

.NET libraries in general follow the cooperative approach and various examples can be found; for example, the BackgroundWorker class in Windows Forms has a member called CancellationPending which is set to true by calling the method
BackgroundWorker.CancelAsync(). Similarly, the classes based on System.Net.WebRequest provide an Abort() method to cancel the current operation and prevent further operations from commencing. A BackgroundWorker has a clear notion of the work that should be canceled, but a WebRequest may have either an active GetResponse() or GetRequestStream() call, and so the target of the cancellation request is not as clear. More generally, if a single object can be used to run distinct operations or if it can run many operations simultaneously, then it becomes very awkward to track cancellation requests on the object itself. And if the object allows the cancellation request to be reset either before or after enactment of a cancellation request, then there are inherent race conditions that can lead to subtle problems.

So although cooperative cancellation is far superior to asynchronous cancellation, the common approach has the following issues:

Data is often imbued with a notion of cancellation, and this is a problematic abstraction. A better model is that specific occurrences of an operation may be cancelable, but data is just data.
Cancellation achieved by direct calls to Cancel() and Abort() methods on specific targets is not groupable or chainable.
It is often necessary to perform some work in addition to setting a Boolean flag when enacting cancellation, but this support is not standardized or ubiquitous.
If an operation on an object makes a callback to user code, it is often awkward to forward the cancellation request made on the parent object to the user code.
If a class tracks cancellation state but allows multiple concurrent operations, or if it allows operations to commence after an earlier operation was canceled , then race-conditions become an inherent issue that frequently leads to subtle bugs.
When cancellation is requested, identified and enacted, the resulting behaviors are not consistent. Some methods return as if nothing happened, some throw OperationCanceledException, some throw custom exceptions, and some return a custom error code.

It is to address these issues that .NET 4 introduces a new approach to cooperative cancellation. And as we will examine in a future blog post, this approach is perfectly suited to the needs of Parallel Extensions and has been adopted across all our types.

New cancellation types

The new types provide a cooperative cancellation framework that is rich and general purpose. Some specific design principles of the new approach are:

Only specific framework types should represent a request for cancellation.
Long-running or blocking methods should accept an explicit parameter that will inform the method of a cancellation request. This is in contrast to mechanisms whereby operations automatically attach themselves to an ambient cancellation-request object, such as one that has been stashed in thread-local-storage. Mechanisms that involve automatic discovery and attachment generally become awkward when the exact specification of the rules are fleshed out.
The response made when cancellation is enacted should be to throw a consistent exception type.

Two new types form the basis of the framework: A CancellationToken is a struct that represents a ‘potential request for cancellation’. This struct is passed into method calls as a parameter and the method can poll on it or register a callback to be fired when cancellation is requested. A CancellationTokenSource is a class that provides the mechanism for initiating a cancellation request and it has a Token property for obtaining an associated token. It would have been natural to combine these two classes into one, but this design allows the two key operations (initiating a cancellation request vs. observing and responding to cancellation) to be cleanly separated. In particular, methods that take only a CancellationToken can observe a cancellation request but cannot initiate one.

When a call is made to CancellationTokenSource.Cancel(), the IsCancellationRequested property of any associated CancellationToken will become true and methods can poll on this to observe the cancellation request. If polling is not possible or not convenient, a callback can be attached via CancellationToken.Register(). A CancellationToken can be copied and passed around and all the copies will simply point back to the original CancellationTokenSource which holds all the actual state regarding cancellation requests, callback lists, etc.

When methods see that CancellationToken.IsCancellationRequested == true, they should respond to this by throwing an OperationCanceledException(cancellationToken). Throwing this exception communicates that the method finished without fully completing its stated purpose and states which CancellationToken it observed. Note that the existing OperationCanceledException class has been augmented with constructors and a property for CancellationToken.

We can now look at some specific examples: the first example shows how to create a CancellationTokenSource, pass its associated CancellationToken to a method, and how to poll on that token and correctly throw an OperationCanceledException. Note that cancellation doesn't need to be checked after every single statement, but it might be checked once per few thousand instructions or so. A general guideline is that cancellation should be checked as frequently as possible subject only to avoiding significant performance impact. Because polling on ct.IsCancellationRequested is very cheap (a single volatile read and a handful of IL instructions), cancellation can be tested frequently without a significant performance impact.

EventHandler externalEvent;
void Example1()
{
   CancellationTokenSource cts = new CancellationTokenSource();
   externalEvent +=
      (sender, obj) => { cts.Cancel(); }; //wire up an external requester
   try
   {
      int val = LongRunningFunc(cts.Token);
   }
   catch (OperationCanceledException)
   {
      //cleanup after cancellation if required...
   }
}

private static int LongRunningFunc(CancellationToken token)
{
   int total = 0;
   for(int i=0; i<1000; i++){
      for (int j = 0; j < 1000; j++)
      {
         total++;
      }
      if(token.IsCancellationRequested){ // observe cancellation
         throw new OperationCanceledException(token); // acknowledge cancellation
      }
   }
   return total;
}

The second example demonstrates using the callback facility on CancellationToken for when polling is not an option.

void BlockingOperation(CancellationToken token)
{
   ManualResetEvent mre = new ManualResetEvent(false);
   //register a callback that will set the MRE
   CancellationTokenRegistration registration =
      token.Register(() => mre.Set());
   using (registration)
   {
      mre.WaitOne();
      if (token.IsCancellationRequested) //did cancellation wake us?
          throw new OperationCanceledException(token);
   } //dispose the registration, which performs the deregisteration.
}

A common pattern in these situations is to use the callback to force whatever condition the blocking call is waiting on, and then immediately check to see if cancellation was the reason the blocking call was woken. Although there is a race condition here that normal waking may occur just before a cancellation request is made, this race is benign as it is always safe to respond to the cancellation request regardless of why the wait finished.

The third example shows how to listen for cancellation via a regular WaitHandle. Under the covers, CancellationToken.WaitHandle is a lazily-allocated ManualResetEvent that becomes set when cancellation is requested.

void Wait(WaitHandle wh, CancellationToken token)
{
   WaitHandle.WaitAny(new [] {wh, token.WaitHandle});
   if (token.IsCancellationRequested) //did cancellation wake us?
     throw new OperationCanceledException(token);
}

And finally, the fourth example shows running multiple asynchronous operations which share a common CancellationToken.

void Example4()
{
   CancellationTokenSource cts = new CancellationTokenSource();
   StartAsyncFunc1(cts.Token);
   StartAsyncFunc2(cts.Token);
   StartAsyncFunc3(cts.Token);
   //...
   cts.Cancel(); // all listeners see the same cancellation request.
}

In particular it is interesting to note that each of the asynchronous functions might pass on the token to other methods and that everyone who is observing the token (or copies of the token) will see the cancellation request when cts.Cancel() is called.

Details

A few details are important for an understanding of the new approach.

A CancellationTokenSource may transition from non-canceled only once and cannot be reset. This prevents various race-conditions that arise deep inside method calls if reset were permitted. At an application level, however, reset functionality is often required and this is achieved by creating a new CancellationTokenSource to replace a used one.

A CancellationToken is a lightweight struct that includes only a single reference back to a CancellationTokenSource. As such, it is the same ‘weight’ as a normal object reference, but because it is a separate type, it can have its own API than provides only read operations.

When a callback is registered to a CancellationToken, the current thread's ExecutionContext is captured so that the callback will be run with the the exact same security context . The capturing of the current thread's synchronization context is optional can be requested via an overload of ct.Register() if required. Callbacks are normally stored and then run when cancellation is requested, but if a callback is registered after cancellation has been requested, the callback will run immediately on the current thread, or via Send() on the current SynchronizationContext if applicable.

When a callback is registered to a CancellationToken, the returned object is a CancellationTokenRegistration. This is a light struct type that is IDiposable, and disposing this registration object causes the callback to be deregistered. A guarantee is made that after the Dispose() method has returned, the registered callback is neither running nor will subsequently commence. A consequence of this is that CancellationTokenRegistration.Dispose()must block if the callback is currently executing. Hence, all registered callbacks should be fast and not block for any significant duration.

Advanced patterns

To finish this introduction to the new cancellation framework we can look briefly at two more advanced patterns of usage.

One situation that arises is that some methods wish to observe two separate tokens. The best approach here is to create a linked CancellationTokenSource that will be signaled if either of the source tokens has become signaled.

The method CancellationTokenSource.CreateLinkedTokenSource() is specifically designed to assist with this and registers the callbacks as required. For example:

void LinkingExample(CancellationToken ct1, CancellationToken ct2)
{
   CancellationTokenSource linkedCTS =
   CancellationTokenSource.CreateLinkedTokenSource(ct1, ct2);
   try
   {
      SlowFunc(linkedCTS.Token);
   }
   catch(OperationCanceledException oce)
   {
      if (ct1.IsCancellationRequested)
      {
         // ...
      }
      else if (ct2.IsCancellationRequested)
      {
         // ...
      }
   }
   linkedCTS.Dispose(); // clean up the linking. required.
}

By using cancellation linking, cancellation-aware methods need only take one CancellationToken parameter. Be aware however that because linking attaches callbacks onto the source tokens, it is very important to clean up properly, otherwise linkedCTS and the callback delegates cannot be garbage collected until the linked sources are also available for collection. This is achieve by explicitly disposing linkedCTS, or via the using() pattern.

Another scenario that arises is to allow library code and user code to cooperate when responding to cancellation. Consider a library method that accepts a CancellationToken, and then calls back to user code. If the user code responds to a cancellation request, it wants to communicate this cleanly to the library code. This is achieved by the user code throwing an OperationCanceledException that mentions the same CancellationToken that the library code was given. Typically the user code will know about the same CancellationToken by getting it through some side channel, e.g. via closure-capture, rather than the library method explicitly passing it on. For example:

private void RunQuery()
{
   int[] data = {1,2,3};
   CancellationTokenSource cts = new CancellationTokenSource();
   var query = data.AsParallel()
                .WithCancellation(cts.Token) // token given to library code
                .Select( (x) => SlowFunc(x, cts.Token) ); // token passed to user code
}

private int SlowFunc(int x, CancellationToken token)
{
   int result
   while(...)
   {
      if (token.IsCancellationRequested)
         throw new OperationCanceledException(token);
      ...
   }
   return result;
}

Notice that the library function (in this case a PLINQ query) and the user code are both observing the same token and that the user-code throws an OperationCanceledException that mentions this common token. This allows the library code to see that the user code has performed cooperative cancellation and has not simply thrown some arbitrary exception. The library may respond to this by behaving exactly as though it had seen the cancellation request itself, rather than thinking the user code failed unexpectedly. It also has the advantage that if the user code throws an OperationCanceledException for some unexpected reason (and mentions a different token or no token at all), it will not be mistakenly interpreted as cooperative cancellation. Because the token is tracked in the OperationCanceledException itself, there is no confusion about the reason for the exception being thrown.

Conclusion

The new cancellation types introduced to .NET 4 provide a new framework for building systems that have rich and consistent cancellation behaviors. This will specifically assist with GUIs, applications, and libraries that manage long running or blocking operations. Over time we expect to see more libraries migrate to this approach and reduce the variation and issues inherent in the current crop of cancellation solutions. Applications and 3rd-party libraries can use these new types to interoperate cleanly with Parallel Extensions and for their own purposes.

Samples for Parallel Programming with the .NET Framework 4

toub — Thu, 21 May 2009 03:36:00 GMT

Along with the release of the .NET Framework 4 Beta 1, we've just published a slew of samples that demonstrate using Parallel Extensions in a variety of ways. You can download these from Code Gallery at http://code.msdn.microsoft.com/ParExtSamples.

These samples include raytracers, a sudoku game, an image colorization algorithm, solvers for the n-queens problem, fractal image generators, and more. The samples include code written to use Parallel Extensions from C#, Visual Basic, F#, and C++/CLI. They include examples of using the Task Parallel Library, PLINQ, new coordination data structures, and even databinding a WPF UI to a collection that has data stored into it from a background thread.

On top of all of this, the samples include a project called "Parallel Extensions Extras", which features a healthy portion of extension methods for .NET 4 types, new data structures to augment what's being shipped in .NET 4, custom schedulers for TPL, and lots of other goodness. Definitely worth a perusal.

Enjoy!

.NET 4 MSDN documentation for parallelism

toub — Thu, 21 May 2009 01:59:00 GMT

In addition to the Betas of Visual Studio 2010 and the .NET Framework 4 being available for download today, the MSDN documentation for .NET 4 Beta 1 is also available. This includes quite a bit of useful information about the new parallelism constructs. Here's a summary of the relevant material, with links:

Parallel Programming in the .NET Framework

Task Parallel Library
Parallel LINQ
Data Structures for Parallel Programming
Parallel Diagnostic Tools

.NET Framework Class Library

Related to this and for more background information, you can also check out the documentation for our native concurrency runtime and programming models:

Concurrency Runtime

Reference

Concurrency Namespace

As with the actual bits, any and all feedback on the docs is welcome and encouraged. A great place to provide feedback is on the forums, for managed at http://social.msdn.microsoft.com/Forums/en-US/parallelextensions/threads and for native at http://social.msdn.microsoft.com/Forums/en-US/parallelcppnative/threads.

.NET 4 Beta 1 is now available, with parallelism!

toub — Thu, 21 May 2009 01:38:00 GMT

We’re very excited that the .NET Framework 4 Beta is now available for public download, as .NET 4 has Parallel Extensions built into its core. You can download the Betas for Visual Studio 2010 and .NET Framework 4 at http://msdn.microsoft.com/en-us/netframework/dd582936.aspx.

For more information on what’s new in Beta 1 regarding concurrency and parallelism, here are some useful links:

As you may have noticed, this blog has been a bit quiet as we’ve been getting ready for the Beta. Now that it’s here, you can expect us to be blogging quite frequently over the next days, weeks, and months. If there are specific topics you’d like to see discussed, please do let us know. And most importantly, we’re still extremely interested in your feedback, so any comments, questions, praises, criticisms, or suggestions you have should certainly be sent our way. The best place to do so is on the forums.

Enjoy!

What’s new in Beta 1 for Parallel LINQ (PLINQ)?

essey — Wed, 29 Apr 2009 21:54:00 GMT

A number of improvements have been made to Parallel Extensions since the Visual Studio 2010 CTP across the Task Parallel Library (TPL), Parallel LINQ (PLINQ), and our coordination data structures. You can find the latest on TPL (1 2 3) and the data structures (link) on this blog. Here are the big changes to PLINQ since that CTP:

With- Operators Pattern
Execution Mode
Cancellation
Custom Partitioning
Refactoring
Merge Options
AsMerged Renamed Back to AsSequential
Binary Operators Now Require AsParallel on Both Sides
Performance Improvements
Removed Seldom Used Operators

With- Operators Pattern

To parameterize PLINQ, we have added a set of controls beginning with With-. As we began to add controls to PLINQ, it became clear that pretty soon the number of overloads on AsParallel, Range, and Repeat would increase dramatically. This pattern resolves that and increases readability. Compare:

OLD: e.AsParallel(4)

NEW: e.AsParallel().WithDegreeOfParallelism(4)

The With- methods are currently:

WithDegreeOfParallelism
WithExecutionMode (see Execution Mode below)
WithCancellation (see Cancellation below)
WithMergeOptions (see Merge Options below)

The latter 3 have their own sections below.

Execution Mode

PLINQ does its best to ensure that it will never behave much worse than LINQ-to-Objects. For example, if a LINQ-to-Objects query needs a constant amount of memory to execute, PLINQ also aims to use only a constant amount of memory on the same query. Or, if a LINQ-to-Objects query has low overhead per element processed, the PLINQ query should also have low overhead per element. If PLINQ cannot meet these constraints by executing the query in parallel, it will execute the query sequentially.

Note that while the goal will be to never do much worse than LINQ to Objects, it is very unlikely that we will be able to achieve that goal for all queries. One suggestion that comes up regularly is to use a smart algorithm that does some sort of time measurements and switches back and forth between parallel and sequential execution. This is significantly more problematic in practice than it may sound, and we have no plans in this direction for V1.

PLINQ makes this decision solely based on query shape. It does not look into size of the data set or execution time of the delegates. If you believe that you will not see speedup in your query, consider comparing the versions of the query with and without AsParallel. Queries that will execute sequentially in the conservative mode include:

Queries contain indexed Select, indexed Where, indexed SelectMany, or ElementAt in a position where the indices are no longer in the original order. Index ordering is senstive to operators that change ordering (e.g. OrderBy) and operators that remove elements (e.g. Where).
Queries that contain operators Take, TakeWhile, Skip, SkipWhile when indices are not in the original order (see above bullet).
Queries that contain Zip, SequenceEquals, unless one of the data sources has an originally ordered index and the other data source is indexible (i.e. an array or IList<T>).
Queries that contain Concat, unless it is applied to indexible data sources.
Queries that contain Reverse, unless applied to an indexible data source.

A certain class of operators require the data to be partitioned with matching data, this is best achieved through hash-partitioning. Our other partitioning types (chunk, range, and striped) would not organize this data correctly for matching. These operators include Join, GroupBy, GroupJoin, Distinct, Union, Intersect, and Except. As for hash-partitioning operators From the data that we have, it seems that our hash-partitioning operators do not get much of a speedup, but do not inhibit it too drastically either. So, at least presently, hash-partitioning operators will not cause a query to execute sequentially (in the conservative mode). This allows for speedup when these operators combine with other operators.

The ParallelExecutionMode enum is added to the System.Linq namespace. It has Default and ForceParallelism values. If you believe that your query can in fact get speedup, but that we’re currently dropping back to sequential, write your query as

e.AsParallel().WithExecutionMode(ParallelExecutionMode.ForceParallelism)....

to override PLINQ’s conservative behavior.

Over time, we will likely be able to support more and more query shapes, so please let us know which ones are important to you so that we can consider them for perf tuning.

Cancellation

Just as Tasks, Parallel, BlockingCollection, Barrier, CountdownEvent, ManualResetEventSlim, and SemaphoreSlim are plumbed for cancellation, so is PLINQ. Cancellation patterns have been added to the .NET Framework 4.0, and our team has rebuilt our system on top of them to provide an elegant, common pattern to exit operations. A dedicated topic on cancellation is coming soon, though we’ll jump in right here with how PLINQ uses the types.

Preparie a query to allow cancellation by using WithCancellation on a query. Here’s an example usage pattern with Cancellation split between two query executions:

var cts = new CancellationTokenSource();

var q = a.AsParallel().WithCancellation(cts.Token).Where(x=>Filter(x)).Select(x=>DoWork(x);

-- separate thread --

foreach (var e in q) { … } // Statement 1

-- separate thread --

var l = q.ToList(); // Statement 2

-- separate thread --

cts.Cancel(); // this will attempt to cancel any in-flight queries,

// including both statements 1 and 2

Note on Dispose – Disposing the enumerator, as is done automatically by the foreach pattern, will also cancel the plinq work occurring in the background.

Custom Partitioning

There are some data types and data sources that PLINQ will not know the best way to partition. PLINQ optimizes for general cases and common cases, but sometimes you want control over how the data can best be partitioned in PLINQ.

So, the new Partitioner<TSource> abstract class, the new OrderablePartitioner<TSource> abstract class, and the new Partitioner factory classes have been created to allow you greater control over partitioning for increased performance. For PLINQ, there are overload extension methods of AsParallel that will hook these types into PLINQ.

This is a weighty topic and will have its own future post.

Renamings and Refactorings

Because there was never an intent or reason for others to extend IParallelEnumerable, IParallelEnumerable<TSource>, nor IParallelOrderedEnumerable<TSource>, these have been changed from interfaces to abstract classes which cannot be extended. This avoids the confusion on how they’re supposed to be used.

OLD –> NEW

IParallelEnumerable –> ParallelQuery

IParallelEnumerable<T> –> ParallelQuery<TSource>

IParallelOrderedEnumerable<T> –> OrderedParallelQuery<TSource>

While we were at it, we improved type names in generics, changing T –> TSource, and we ensured that all of the right extension methods were available for the non-generic ParallelQuery class.

ParallelMergeOptions

Handling of ParallelMergeOptions has been moved out of AsMerged. Merge buffering is now specified via the WithMergeOptions method.

AsMerged Renamed Back to AsSequential

In the last CTP we renamed AsSequential to AsMerged when we added an overload to take MergeOptions. Now, we renamed it back to AsSequential so that it would better pair with AsParallel.

Binary Operators Now Require AsParallel on Both Sides

We have changed the handling of the operators in LINQ which task 2 data sources to ensure that ordering expectations are more explicit. There are a number of operators in LINQ which take 2 data sources. Here are those operators with overload counts in parentheses (): Zip (1), Join (2), GroupJoin (2), Concat (1), SequenceEqual (2), Union (2), Intersect (2), Except (2).

Some of these, such as SequenceEqual make no sense without ordering. Others do make sense sense without ordering—and in fact, would be preferred this way—and some could go either way. Rather than make all kinds of mystical decisions without involving you, steps have been taken for developers to make a decision.

Previously, queries taking two input sequences could be parallelized as follows:

a.AsParallel().AsOrdered().Zip(b, (x, y) => x*y);

These operators, of the form Operator(this ParallelQuery<T>, Enumerable<T>, ...), now throw a NotSupportedException at runtime and are marked as obsolete*.

Now, to parallelize these queries, developers must choose:

a.AsParallel().AsOrdered().Zip(b.AsParallel(), (x, y) => x*y);

a.AsParallel().AsOrdered().Zip(b.AsParallel().AsOrdered(), (x, y) => x*y);

* You may wonder why these operators are marked as obsolete during their introductory release. We did this to maintain parity with LINQ-to-Objects and other LINQ providers as much as possible. One of our primary design goals is to “Just add AsParallel” and you can see speedup. In this case, we wanted to have the same methods as in LINQ-to-Objects, but use the obosolete methods to encourage the addition of AsParallel on the second source as well.

Performance Improvements

A number of performance improvements have been undertaken, including:

1. Order-preserving pipelining merges – Previously, just putting AsOrdered on a query forced the entire query to execute before a single element is yielded. This is now optimized so that elements from the query can be yielded as they are produced with Default (AutoBuffered) and NotBuffered values of MergeOptions.

2. Improved partitioning fairness for data sources that don't implement IList<T>

3. Better performance of some queries over IList<T> or an array

4. Chunk partitioning size tuning - Chunk partitioning is the most common partitioning scheme for queries over data sources other that IList<T> and arrays (i.e. non-indexible data sources). The sizes of these chunks now grow as more and more chunks are accessed. This is to balance between the cases of a) queries with small data sets but expensive delegates in the queries and b) queries with large data sets but inexpensive delegates in the queries.

5. Removal of likely false sharing cases, which have had up to 6x improvements in some cases

Removed Seldom Used Operators

A few PLINQ-specific operators that don’t really have use cases or existed simply for performance, yet did not show gains over existing LINQ operators, have been removed. Reduced surface area allows us to focus more on the important stuff for you and lowers the count of concepts you need to know. If you don’t notice which ones, good!

I hope that you enjoy using the Beta and send us lots of feedback!