Parallel Programming in Native Code : Concurrency Runtime

Writing a Custom Message Block – Part 1: Introduction

Mike Chu — Tue, 15 Dec 2009 00:14:34 GMT

The Asynchronous Agents Library within the Concurrency Runtime provides a set of basic message blocks which can be used to create a message passing network. In most cases, these blocks have sufficient enough flexibility and can be composed together to provide all the necessary functionality for your message-passing network. However, there are cases in which the provided blocks do not have the exact behavior you want, and a custom message block is desired. In this post, I will discuss the process for creating your own custom message block.

What kind of message block are you writing?

The first thing to decide when creating a message block is to determine what kind if block you desire. Is it a source block, in which case it only offers messages but does not receive them, or is it a target block, which can only receive messages for processing, but does not offer them out to anyone else? It could also be a combination of the two, a block that both receives from sources and offers to targets.

The Asynchronous Agents Library contains two interfaces ISource and ITarget, which provides the API interface for source and target blocks, respectively. While you are free to create your own block from these interfaces, we highly recommend you use one of the three base classes we provide: source_block, target_block, and propagator_block. These are explained in the table below:

source_block	A block that is only a source of messages. This block offers messages to other blocks, but cannot receive messages.
target_block	A block that is only a target of messages. This block receives messages from other blocks, but cannot offer them.
propagator_block	A block that is both a source and target of messages. This block can both receive messages from a source and propagate message out to other blocks.

We recommend using these base classes because they significantly ease the process of developing a custom message block. They take care of all the error checking, safe unlinking and the required locks so they free you up to focus purely on the desired behavior of your message block.

All three base classes are templated on their network link registry type. This defines how many source or target links a block can have. The Asynchronous Agents Library contains two such link registries: single_link_registry and multi_link_registry, which allow a single or multiple links, respectively. For example, if you wanted to create a message block that was purely a source block and can only accept one target, you would declare your message block as:

class example : public source_block<single_link_registry<ITarget<_Type>>>

Similarly, you could specify the block to use multi_link_registry to allow multiple links. This is only differentiated because a single_link_registry can be slightly optimized over multiple links.

I will now focus on creating source and target blocks, and how they can be customized for different behavior. A propagator block would have the combination of the two.

Writing a source_block

A source_block is a block that offers messages to its targets. Thus, it must deal with target blocks calling back to it to take messages. The following six methods must be defined in your derived class and can be customized to change the behavior of your block:

virtual void propagate_to_any_targets(message<_Type> * _PMessage)

This method is the main propagation routine for a source_block. This is where your block should be processing the received message and offering messages to its targets. This method is called automatically when calling one of two methods: async_send or sync_send, which are used to asynchronously or synchronously propagate a message out, respectively. These two methods are present on every block and serve to initiate either a asynchronous or synchronous propagation for the block.

virtual message<_Type> * accept_message(runtime_object_identity _MsgId)

This method is called from a target block after they are offered a message from this source_block. The call to accept must be made within the source’s call to target::propagate, thus this is part of the message passing handshake. The method is used to take ownership of the offered message. A common issue to decide in accept_message is if your block returns the actual offered message itself, or copies of messages. For example, our unbounded_buffer block passes the specific message offered to its targets, and only one will receive that message. Our overwrite_buffer, on the other hand, returns copies to each of its targets, so all targets can get a version of the message.

virtual bool reserve_message(runtime_object_identity _MsgId)

This method is called from a target block after they are offered a message from this source_block and want to reserve the message to consume it later. The function should return true if you want to allow this message to be reserved, false otherwise.

virtual message<_Type> * consume_message(runtime_object_identity _MsgId)

This method is called from a target block after they have already successfully reserved a message and now wish to consume it. Similar to accept, a common issue to decide here is whether to return the specific message or a copy of the message.

virtual void release_message(runtime_object_identity _MsgId)

This method is called from a target block after they have already successfully reserved a message and now do not wish to consume it. Mostly cleanup and possibly error checking should be done here.

virtual void resume_propagation()

This method is automatically called when a reserved message is consumed or a message is released. Here, you should decide how to proceed with propagation. In general an async_send(NULL) should be called, which will simply force the block to restart propagation without a new message.

Writing a target_block

A target_block is a block that can accept messages offered from its sources. Thus, it must deal with messages being passed to it from sources. There are two methods than can be overridden to handle the messages, propagate_message and send_message. Only propagate_message must be overridden in the derived class; send defaults to not allowing synchronous propagation.

virtual message_status propagate_message(message<_Type> * _PMessage,
ISource<_Type> * _PSource)

This method is called after a message, _PMessage, has been asynchronously offered from source _PSource and has successfully passed any filter that you may have added to your block. The message has not yet been accepted, so ownership still resides with the source _PSource. Thus, this target_block can now be customized to do whatever it wants with the message and source. Typically, you want to call
_PSource->accept(_PMessage->msg_id(), this);

in order to take ownership of this message from the source. However, the implementation of this method is totally dependent on the desired behavior for this block.

virtual message_status send_message(message<_Type> * _PMessage,
ISource<_Type> * _PSource)

This method is exactly the same as propagate_message except that the message _PMessage has been synchronously offered from source _PSource .

Combining the two: Writing a propagator_block

The main usage of a propagator_block is its ability to receive a message, like a target_block, then manipulate that message and send it out, like a source_block. Thus, creating a propagator_block is simply the combination of the methods defined above. The propagation of a received message can be done by using either of the two methods: async_send and sync_send.

The async_send(message * _PMessage) method will spin up a light-weight task in the Concurrency Runtime which will asynchronously call propagate_to_any_targets(_PMessage) to manipulate the messages as you wish according to your custom block, and offer the message out to its targets.

Similarly, the sync_send(message * _PMessage) method will also offer the message out to its targets, but does so synchronously, in line.

Example: Squaring block

As a simple example, let’s look at how I would write a simple message block that accepts any message from a source, squares the received message payload, then propagates out a message with the square to its targets.

First, since this block both receives in and sends out messages, I need to make it a propagator_block. I’ll define the class as:

    template<class _Type = int>
    class squaring_block :
          public propagator_block<multi_link_registry<ITarget<_Type>>,
                                  multi_link_registry<ISource<_Type>>>

This class declaration specifies that my block is a propagator_block that can link to any number of targets and be linked from any number of sources.

Next, I’ll need some place to store my messages as they arrive in this block. I’ll need to store them in case, for example, the input throughput exceeds the output throughput, or in case a target reserves one of the offered messages and we need to stall the network pipeline until it is consumed or released. To do this, let’s just include a simple queue that holds our messages:

std::queue<message<_Type> *> messageBuffer;

Depending on your block, you may not need to store a queue of messages, or any messages at all. It all depends on the functionality you desire.

First, I’ll specify the constructor and destructor for the block:

    squaring_block()
    {
        initialize_source_and_target();
    }

    ~squaring_block()
    {
        remove_network_links();
    }

The call to initialize_source_and_target is a base class method for propagator_block that initializes the block and allows you to specify which Scheduler or ScheduleGroup this block uses. For simplicity, and in most cases, we can just call this method with no arguments. (The source_block and target_block base classes have initialize_source and initialize_target methods, respectively.) The destructor must call remove_network_links, which is another base class method. This method is critical in that it ensures the links to and from this block are properly removed before destructing the object. Otherwise, asynchronous propagations could be continuing to deleted memory.

Now, let’s move on to the interesting aspects about customizing the block. As a target, this squaring_block is going to receive messages offered to it. A source block will do this by calling squaring_block::propagate. Since we are deriving from propagator_block, all the necessary error handling has already been done, and we just need to specify what happens after we get a message that we actually want in the propagate_message function (which is called by propagate):

    virtual message_status propagate_message(message<_Type> * _PMessage,
                                              ISource<_Type> * _PSource)
    {
        message_status result = accepted;

_PMessage = _PSource->accept(_PMessage->msg_id(), this);

        if (_PMessage != NULL)
        {
            async_send(_PMessage);
        }
        else
        {
            result = missed;
        }

return result;
}

The code here simply tries to accept a message from the source by calling _PSource->accept, then, if it properly returned a message, it would call async_send(_PMessage) on the returned message. As soon as the accept call has returned, the message has transferred ownership to our squaring_block from the source; however, our squaring_block has not yet processed or stored away the message and prepared it for transferring to any of our targets. That process is initiated by async_send.

Similar to propagate_message, there is a send_message function which can be specified. This function happens on a synchronous path, while propagate is for the asynchronous. For this purposes of this block, we only need to change the implementation to call sync_send instead of async_send and everything else is the same.

Propagation of messages

Now how about processing and offering the message out to the squaring_block’s targets? This is done via the source methods that were outlined earlier. First, let’s specify the propagate_to_any_targets method. This is the method that is called in a light-weight task created by the call to async_send during propagate_message and is where all the main processing should occur.

    virtual void propagate_to_any_targets(message<_Type> * _PMessage)
    {
        if (_PMessage != NULL)
        {
            // Square the input, delete the original message
            _Type outputPayload = _PMessage->payload * _PMessage->payload;
            message<_Type> * newMessage = new message<_Type>(outputPayload);
            delete _PMessage;

messageQueue.push(newMessage);

            if (messageQueue.size() > 1)
                return;
        }

        while (messageQueue.size() > 0)
        {
            message_status status = declined;

            for (target_iterator _Iter = _M_connectedTargets.begin();
                 *_Iter != NULL; ++_Iter)
            {
                ITarget<_Target_type> * _PTarget = *_Iter;
                status = _PTarget->propagate(messageQueue.front(), this);

                // Ownership of message changed. Do not propagate this
                // message to any other target.
                if (status == accepted)
                    break;
            }

            // If status is anything other than accepted, then the current
            // message was not propagated out. Nothing after it can be
            // propagated out.
            if (status != accepted)
                break;
        }
    }

Let’s step through that code slowly and investigate how I made this block behave how I wanted it to. The first part of the function is focused on the processing of the message:

    // Square the input, delete the original message
    _Type outputPayload = _PMessage->payload * _PMessage->payload;
    message<_Type> * newMessage = new message<_Type>(outputPayload);
    delete _PMessage;

messageQueue.push(newMessage);

if (messageQueue.size() > 1)
return;

First, upon receiving a message I squared the input, providing the desired functionality of this block. Next, I delete the old message, since this squaring_block is the current owner of it, but its lifetime will end with this block; i.e. there will be no further propagations of that specific message out of this block. Then, I checked if there was any current message being offered by this block. If so, I enqueue the new message to the queue of stored messages and return; otherwise, I can begin propagation.

One subtle note about message processing: while I technically could have done the processing of the message by squaring the input in propagate_message rather than in propagate_to_any_targets, it is better practice to do the latter. This is because propagate_message is called from your block’s source, thus it is running in a task initiated by your source. Doing processing in propagate_to_any_targets is better because it is your block’s propagation task.

At this point, everything else in the propagate_to_any_targets method has to do with the actual propagation to targets. The message has already been processed and is ready for offering. The main while loop takes each message, one at a time starting with the head, and offers it to each of the targets, also one at a time. By using our standard multi_link_registry and our base classes, I’m able to use our included iterators to search through the targets with a simple for loop and propagate out to my targets:

    for (target_iterator _Iter = _M_connectedTargets.begin();
         *_Iter != NULL; ++_Iter)
    {
        ITarget<_Target_type> * _PTarget = *_Iter;
        status = _PTarget->propagate(messageQueue.front(), this);

        // Ownership of message changed. Do not propagate this
        // message to any other target.
        if (status == accepted)
            break;
    }

This takes the currentMessage and propagates it out to the targets. After each propagate call returns, we check to see what the result is. If it was accepted, that means the ownership has changed and we can break out and offer the next message in our queue. Otherwise, we should propagate it to the next target we have. If no target accepts the message, we propagation should stall until that currentMessage is accepted or consumed. We cannot propagate other messages in the queue, otherwise messages will be propagated out of order.

Now that we’ve created a block that can offer messages out, we need to handle what happens when a target block calls back and tries to take the offered message. This is handled in the accept_message method:

    virtual message<_Type> * accept_message(runtime_object_identity _MsgId)
    {
        message<_Type> * msg = NULL;

        if (messageQueue.front()->msg_id() == _MsgId)
        {
            msg = messageQueue.front();
            messageQueue.pop();
        }

        return msg;
    }

This method just checks to see if the message the target is trying accept is the same as the current one that my squaring_block is holding and trying to propagate. If so, it dequeues it from my block, and returns it to the target. Note: as soon as it returns the message, ownership of the message has changed hands. We can no longer safely touch that message pointer.

At this point, we have a block that can participate in both synchronous and asynchronous message passing! For completeness, there are four more functions to handle reserving and consuming of messages. However, as this post is long enough already, I’ll simplify this block by disallowing reserving and consuming of message. In my next post, I’ll dive into the details of how to handle proper reserving of messages. Disallowing reservations can be done by:

    virtual bool reserve_message(runtime_object_identity _MsgId)
    {
       return false;
    }

    virtual message<_Type>* consume_message(runtime_object_identity _MsgId)
    {
       return NULL;
    }

    virtual void release_message(runtime_object_identity _MsgId)
    {
    }

    virtual void resume_propagation()
    {
    }

So that’s it! That’s a rather lengthy, but in-depth introduction to writing custom message blocks! Please let me know if there’s anything still confusing or there’s any interesting custom message blocks that you come up with! Look for my next post soon on allowing reservations.

Resource Management in Concurrency Runtime – Part 3

Atilla Gunal — Fri, 13 Nov 2009 20:13:00 GMT

In my previous blog post, I talked about the dynamic migration concept of the Concurrency Runtime’s (ConcRT) Resource Manager (RM). Today I will be demonstrating that concept in action and will focus on its performance characteristics.

A Demonstration of Dynamic Migration

The scenario that we are going to use to demonstrate the resource manager’s dynamic core migration concept involves two schedulers running different kinds of workloads in phases. The schedulers will be created with default policy on an 8 core machine (MinConcurrency=1, MaxConcurrency=8). The workload of schedulers in each phase will be characterized by one of the following:

i) Both schedulers have an equal amount of work

ii) One scheduler has work, the other one has no work

iii) One scheduler has more work than the other scheduler

Here are the phases and the recorded performance of the Resource Manager in detail:

Phase1: In this phase both schedulers have equal amount of work. No dynamic migration happens and schedulers keep their proportionally allocated resources. Note that both schedulers have 4 resources.

Phase2: Only Scheduler1 has work here. This phase starts at 2.246s; RM reacts at 2.309s by allocating all resources to Scheduler1. Note that Scheduler1 has 8 resources of which one is shared with Scheduler2. That is because resource manager cannot take away cores below the MinConcurrency value and has to have at least 1 core assigned to Scheduler1. However, Scheduler1 has no work and the resource is idle therefore Scheduler2 makes use of it without interruption.

Phase3: This is similar to Phase2, this time only Scheduler2 has work. The phase starts at 4.508s; RM reacts at 4.602s by allocating all resources to Scheduler2.

Phase4: Here both schedulers have work but Scheduler1’s work is more than Scheduler2’s. The phase starts at 6.833s; RM first balances resource usage among schedulers at 6.911s taking into account the workload of each. This continues until Scheduler2 completes all of its work. At 8.112s all resources are allocated back to the Scheduler1.

Focusing on Dynamic Migration Reaction Time

In order to give more details on the dynamic migration reaction time, a zoomed in view of Phase4 will be examined.

Here the first core migration happens after 78ms where RM balances the resources between the two schedulers. This reaction time depends on a number of factors:

1) The statistics polling interval in resource manager

All core migration decisions are done after resource manager polls the schedulers. With the current implementation poll interval is 100ms, therefore the reaction time can be anywhere between 0 and 100ms.

2) The statistics from the schedulers

Statistics is the representation of scheduler workload in resource manager including task completion rate, incoming task rate and total task queue size. The faster the growth in statistics the faster the scheduler will get extra resources. This also implies that with a slower growth resource manager may delay allocation to the upcoming polls.

3) The availability of resources

Resource manager will only migrate if there is an available resource. Resources are considered available if other schedulers have declared their resources idle or have declining statistics. In other words, resource manager’s reaction will be delayed until there is an available resource.

After rebalancing, the statistics don’t change until 8.112s therefore resource manager continues with the current allocation. Whenever Scheduler2 completes all work, all resources are allocated back to Scheduler1.

Next Steps and Feedback

I would love to hear feedback regarding the blog content or areas of interest with respect to Concurrency Runtime to have a guided focus on blogging.

What’s new in Beta 2 for the Concurrency Runtime, Parallel Pattern Library and Asynchronous Agents Library

rickmolloy — Thu, 29 Oct 2009 17:05:03 GMT

Last week Visual Studio 2010 Beta was released for download. Since Beta1, the team has been pretty busy adding enhanced functionality to make you more productive at expressing parallelism in your applications and improving the quality and performance of our runtime and programming models.

Here’s a guide to what’s new in Beta2: we’ve added 2 new concurrent containers, significantly enhanced our online documentation and improved our debug experience by adding better visualizations and more intuitive views in the parallel debug window. We’ve also modified a small number of APIs which you should be aware of if you are using Beta 1.

Concurrent containers – concurrent_queue and concurrent_vector

We’ve alluded more than once to concurrent_queue and concurrent_vector in our videos and live talks, these are finally in the box and with them two new header files concurrent_queue.h and concurrent_vector.h.

concurrent_queue<T> is very similar to std::queue<T> and it offers push, try_pop interfaces and ‘unsafe’ iterators and size accessors (these aren’t threadsafe during concurrent pushes and pops).

concurrent_vector<T> is most similar to a std::vector<T> and it offers a push_back method that is internally synchronized across threads and allows efficient thread safe growth of the vector. Like std::vector, concurrent_vector has random access iterators, but unlike std::vector, the guarantee of contiguous storage is removed and there are no insert and erase methods.

The interfaces to concurrent_queue and concurrent_vector will be incredibly familiar if you are a user of Intel’s Threading Building Blocks (the interfaces are identical), and a very big thank you goes out to their team for their assistance with this.

Here’s a brief example that uses both std::queue and std::vector in a parallel loop:

#include <ppl.h>
#include <concurrent_vector.h>
#include <concurrent_queue.h>
#include <iostream>
using namespace Concurrency;
using namespace std;
int main()
{
    concurrent_vector<int> odds;
    concurrent_queue<int> evens;
    parallel_for(0,100,1,[&odds,&evens](int i){
        if (i%2 == 0)
            evens.push(i);
        else
            odds.push_back(i);
    }); 
    cout << "We expect 100 items: " << evens.unsafe_size() + odds.size() << endl;
}

Debugging enhancements in Beta2

VS 2010 Beta 2 also now includes significant enhancements for parallel debugging. The locals window now includes visualizers for all first class objects in the PPL, the Agents Library so now when you a look at an instance of a concurrent_queue or an ubounded_buffer in the locals window they both look a lot like a std::queue instead of exposing implementation details. The Parallel Tasks and Parallel Stacks windows have also been enhanced; in addition to those videos, try the C++ code in the MSDN walkthrough and Daniel Moth’s blog. Finally, like the rest of the C Runtime, we’ve made our source code for the Concurrency Runtime available as part of the install, so if you need to debug deeper or really want to see how things work internally, you can now.

Documentation updated with more How To’s and walkthroughs

Our offline and online documentation has been significantly updated for Beta2. We’ve added multiple conceptual topics and expanded our How To topics significantly to include information not just on PPL and the Agents Library but on the underlying Concurrency Runtime and how to manage and use scheduler instances. Any feedback on these topics is greatly appreciated.

API updates to task_group and agent

For Beta 2, we’ve made a very small number of API updates to our task_group, structured_task_group and to our agent classes. The change to the agent is simple describe, we’ve simplified the state management and removed the parameter for the agent::done method, it’s sufficient to call this->done() instead of this->done(agent_done).

The change to task_group and structured_task_group is additive, we’ve added the method run_and_wait which takes a functor and runs it inline on the current thread or task. This offers the major benefit of being able to compose tasks and nest their cancellation. One of the easiest ways to see this in action is through implementing a parallel search algorithm, like a parallel version of the new C++0x library function ‘all_of’:

template<class InIt,class Pr>

inline bool parallel_all_of(InIt first, InIt last, Pr pred)
{

   typedef iterator_traits<InIt>::value_type Item_type;

   //create a structured task group
   structured_task_group tasks;

   auto for_each_predicate = [&pred,&tasks](const Item_type& cur){

   if (!pred(cur))
      tasks.cancel();
   };

   auto task = make_task([&](){
      parallel_for_each(first,last,for_each_predicate);
   });

   return tasks.run_and_wait(task) != canceled;
}

Here we are placing a call to parallel_for_each inside of a structured_task_group and cancelling the work when the predicate is not true. This has the effect of cancelling all nested tasks in the structured_task_group, potentially saving work if the predicate is long running and expensive to compute. We can use this in our example above like this (but don’t expect to see significant speedups over std::all_of for this example):

if (parallel_all_of(odds.begin(),odds.end(),[](int i){return i % 2;}))
    cout << "success!" << endl;

So try Beta 2 if you haven’t already

That’s about it for now, so download VS 2010 Beta 2 if you haven’t already and if you’re at PDC or TechEd Europe next month stop by our booth or come to one our talks.

-Rick

Resource Management in Concurrency Runtime – Part 2

Atilla Gunal — Tue, 21 Jul 2009 19:45:00 GMT

In my previous blog post, I talked about key concepts of the Concurrency Runtime’s (ConcRT) resource manager, starting with a definition of a resource. I then explained why an application might be composed of a number of scheduler instances. Eventually I mentioned how the resource manager helps in resource allocation to the schedulers in order to improve performance and increase utilization of hardware. Specifically, the ‘initial allocation’ algorithm was mentioned, which defines the resource allocation done at the time of scheduler initialization.

Today I will be talking about ‘dynamic migration’ which is another key concept of the resource manager. I will provide details on how the resource manager deals with resource utilization dynamically as the workload of schedulers change.

Dynamic Migration

Dynamic migration can be defined as moving resources from one scheduler to another in order to improve CPU utilization.

As an example, consider the scenario on an 8 way machine where there are two schedulers (let them be S1 and S2) created with MinConcurrency = 1 and MaxConcurrency = 8 (Please refer to this for a description of the scheduler policy values that involves resource management). As mentioned as part of the initial allocation algorithm, after S1 and S2 are created the resources will be shared proportional to the MaxConcurrency and they will both have 4 resources:

Until now this is all initial allocation. After this point, resource manager will start collecting information about the workload of the schedulers by polling periodically. At each poll the resource manager will capture the state of each scheduler instance with the IScheduler::Statistics() method. Let me note that the time a scheduler instance registers to the resource manager, it has to provide an implementation of IScheduler interface so that resource manager can call on that interface. The IScheduler::Statistics() method is expected to return three measurements calculated by the scheduler instance itself:

Number of completed tasks since last statistics call
Number of incoming tasks since last statistics call
The total size of all work queues

The resource manager will keep a history of these measurements and derive a metric for scheduler workload for each scheduler. A statistically significant change in that metric (roughly speaking a positive or negative change as a factor of the standard deviation) will signify to the resource manager that the scheduler either needs one or more additional resources or can give away one or more. The bigger the change the more resources will be added or removed.

After polling each scheduler, the resource manager will have a list of schedulers (L1) that can give away a number of resources and a list of schedulers (L2) that needs a number of resources. Next thing to do then for all schedulers in L1, starting from the one with the minimum resource allocation priority, is to remove extra resources and re-allocate them to the schedulers in L2 starting with the highest priority. It is important to note that if L2 is empty, then schedulers in L1 will still keep their resources.

Let’s return back to the example above and assume that the resource manager has decided S1 can give away 2 resources and S2 can make use of 2 resources. The resource manager will inform S1 to give away 2 resources, oversubscribe the hardware threads with 2 more resources and inform S2 that it can now use the extra resources. It is expected that S1 will obey the protocol and release the resources as soon as possible. After this protocol completes, the allocation will be as follows:

More Details on Dynamic Migration

We covered the general idea of dynamic migration. Let us walk through some what-if scenarios.

1. What happens if S1 has no work at all and S2 is still in need of resources, does S1 fall back to zero resources?

As a general rule of resource allocation, the resource manager will never allocate fewer resources to a scheduler than its MinConcurrency policy value. Therefore S1 will at least have 1 resource assigned. However since there is no work for S1, S1 will put that resource into inactive mode indicating to the resource manager that the resource is idle. Knowing this, the resource manager will share the underlying hardware thread by oversubscribing and providing another resource to S2.

2. What would happen if S1 shuts down and S2 is still in need of more?

All resources of S1 will become available. Resources will be allocated to S2 (not more than MaxConcurrency of S2) soon after.

3. On a machine with NUMA topology, is locality considered?

Locality information is used after the resource manager decides which scheduler is going to give away resources and which scheduler is going to take more (note that the schedulers are selected from the L1 and L2 list with respect to their allocation priority – therefore locality is less important compared to allocation priority). Let S1 (an element of L1) have R1 resources to give away and S2 (an element of L2) be in a need of R2 resources. If R1 is greater than R2 then the subset of resources to be allocated to S2 will be selected with respect to locality. Here is an example:

Assume this is the allocation layout before dynamic migration where Node1 and Node2 are NUMA nodes having 4 cores each. Let S1 be in need of 2 resources and S2 be able to give away 4. The resource manger will select two resources from S1 close to the ones in S2. Since resources of S2 (Core1 and Core3) are closer to Core2 and Core4 (as cores 1 to 4 are in the same node), the resource manager will migrate Core2 and Core4 to S2. After dynamic migration this would be the layout:

Next Steps and Feedback

Next I will demonstrate dynamic migration and focus on its performance characteristics. I would love to hear feedback regarding the blog content or areas of interest with respect to Concurrency Runtime to have a guided focus on blogging.

Synchronization with the Concurrency Runtime - Part 3

vinodsu — Thu, 04 Jun 2009 23:51:00 GMT

In my previous posts, I addressed the motivation behind using concurrency runtime’s synchronization primitives and also introduced Critical Section and reader writer lock. In this blog, I will cover concurrency runtime’s event.

Event

This is a bi-state type class which, unlike Critical Section or Reader Writer Lock, does not protect access to shared data. Events synchronize flow of execution and use concurrency runtime’s facilities to enable cooperative schedule of work. They behave similar to Win32 manual-reset event. The main difference between the concurrency runtime’s event and Win32 event is that the concurrency runtime’s event are designed to cooperatively yield to other cooperative tasks in the runtime when blocked in addition to preempting whereas Win32 events are, by design purely pre-emptive in nature.

Example:

This example enables the scheduler to create two threads and then calls DemoEvent function that takes event class and a Win32 manual-reset event class as template parameters. The Demo function first creates several tasks that simulate some work and then wait for a shared event to become signaled.

Special thanks to Rick Molloy for sharing this example.

// event.cpp : Defines the entry point for the console application.

// compile with: /EHsc

#include <windows.h>

#include <concrt.h>

#include <concrtrm.h>

#include <ppl.h>

using namespace Concurrency;

using namespace std;

class WindowsEvent

{

HANDLE m_event;

public:

WindowsEvent()

:m_event(CreateEvent(NULL,TRUE,FALSE,TEXT("WindowsEvent")))

{

}

~WindowsEvent()

{

CloseHandle(m_event);

}

void set()

{

SetEvent(m_event);

}

void wait(int count = INFINITE)

{

WaitForSingleObject(m_event,count);

}

};

template<class EventClass>

void DemoEvent()

{

EventClass e;

LONG volatile taskCtr = 0;

//create a taskgroup and schedule multiple copies of the task

task_group tg;

for(int i = 0;i < 8; ++i)

tg.run([&e, &taskCtr]{

//Simulate some work

Sleep(100);

printf_s("\tTask %d waiting for the event\n", InterlockedIncrement(&taskCtr));

e.wait();

});

Sleep(1000);

printf_s(" Setting the event\n");

//Set the event

e.set();

//wait for the tasks

tg.wait();

}

void main ()

{

// Create a scheduler that uses two threads.

CurrentScheduler::Create(SchedulerPolicy(2, MinConcurrency, 2, MaxConcurrency, 2));

printf_s("Cooperative Event\n");

DemoEvent<event>();

printf_s("Windows Event\n");

DemoEvent<WindowsEvent>();

}

Sample Output:

Cooperative Event

Task 1 waiting for the event

Task 2 waiting for the event

Task 3 waiting for the event

Task 4 waiting for the event

Task 5 waiting for the event

Setting the event

Windows Event

Task 1 waiting for the event

Task 2 waiting for the event

Setting the event

Task 3 waiting for the event

Task 4 waiting for the event

Task 5 waiting for the event

We observe that when using cooperative event, we execute all 5 tasks, each task when waiting on the event that is not set, cooperatively yields so that another task can be scheduled and run in the thread’s quantum. When using windows event, we observe that the 2 tasks scheduled block the thread until the event is set.

Samples posted for the Parallel Pattern Library and Concurrency Runtime

rickmolloy — Wed, 03 Jun 2009 00:40:00 GMT

Following the release of Visual Studio 2010, we've just published a set of sample applications for using the Parallel Pattern Library, the Agents Library and the Concurrency Runtime on code gallery. These supplement the documentation and samples provided on msdn for Beta1.

The samples include using task parallelism with PPL, a few based examples of using the Agents Library (Dining Philosophers, an example using choice) and a find in files example (which will be blogged about here in the next day or so). There are also some examples of using the Concurrency Runtime itself, a simple event based application demonstrating cooperative blocking and UMS threads and a task based example (fibonacci) which shows the Concurrent Suballocator.

As always your feedback here or in the forums is welcome.

-Rick

Implementing Dining Philosophers with the Agents Library

rickmolloy — Thu, 28 May 2009 22:34:00 GMT

The latest issue of msdn magazine includes an article that I wrote which illustrates implementing the Dining Philosophers purely in message passing and without using any explicit locking. If you have Visual Studio 2010 Beta1 installed, you can also download the code and build the project, even if you don't you can browse the code online.

For folks that are curious about how locks are avoided, I'll state that the chopsticks themselves are messages and rather than the classic solution with semaphores, the philosopher uses the join message block to receive both messages from the table when it is time to pick them up. i.e. the code looks like this:

vector<Chopstick*> PickupChopsticks()

{

//create the join

join<Chopstick*,non_greedy> j(2);

m_LeftChopstickProvider->link_target(&j);

m_RightChopstickProvider->link_target(&j);

//pickup the chopsticks

return receive(j);

}

I'd encourage you to read the article and provide feedback either here or in our forums.

-Rick

What's new in the Concurrency Runtime and the Parallel Patterns and Asynchronous Agents Libraries

rickmolloy — Thu, 21 May 2009 23:23:00 GMT

Visual Studio 2010 Beta1 has been released, and it is a full install version. The team has been busy, busy busy since the CTP release last fall to deliver most of the APIs and objects we’ve blogged about here into Beta1. So I wanted to take a moment to summarize what’s new in the Beta for the Parallel Pattern Library, the Asynchronous Agents Library and the Concurrency Runtime.

You can find the official reference documentation and walkthroughs in the MSDN Library here.

What’s new in the Parallel Patterns Library

The Parallel Patterns Library (PPL) provides an imperative programming model that promotes scalability and ease-of-use for developing concurrent applications. In general the library’s surface area remains largely unchanged, with a few high impact additions. You will also find it more stable, more robust, faster, and more scalable – lots of goodness!

Added support for exception handling in task_group.
The new is_current_task_group_canceling API helps query cancellation status.
The combinable<T> object has been added for helping avoid races on reductions and accumulations.
We’ve made significant performance improvements to our parallel loops: parallel_for, parallel_for_each and parallel_invoke
critical_section and reader_writer, the cooperative synchronization primitives that we blogged about recently, are now in.
We’ve also included a manual reset event called Concurrency::event.

What’s new in the Asynchronous Agents Library

The Asynchronous Agents Library (or just Agents Library) provides a programming model that enables you to increase the robustness of concurrency-enabled application development. The Agents Library is a C++ template library that promotes an actor-based programming model and in-process message passing for fine-grained dataflow and pipelining tasks. The Agents Library builds upon the scheduling and resource management components of the Concurrency Runtime.

Here’s a summary of what’s we’ve added and revised in the Agents library:

Added the choice, join and multitype_join message blocks to allow users to wait on a set of messages; choice waits for any message, join waits for all messages of a single type, and multitype_join waits for all messages and allows for multiple different types.
Renamed the transform message block to transformer (we didn’t want Concurrency::transformer to conflict with std::transform)
Added functor support to the pipeline message blocks call and transformer
Building custom message blocks is now easier due to some refactoring.

What’s new and improved in the Concurrency Runtime itself

The underlying Concurrency Runtime has undergone significant feature improvements and performance enhancements, many of which we have been discussing already on this blog. These changes will have improvements to everything built on top of the runtime, including the Parallel Patterns Library and the Agents Library.

Added support for User Mode Scheduled Threads on Windows 7 RC (see our blog posts on the Concurrency Runtime’s support and go deep on Win7 UMS)
Revised and simplified scheduler policies and APIs for constructing schedulers.
Added dynamic resource management for load balancing between multiple scheduler instances.
Further improved load balancing by applying the hill climbing technique to thread injection.
Added the concurrent sub-allocator for high performance allocations and frees on multiple threads

Call to Action

Download and install Microsoft Visual Studio 2010 Beta 1 today. Impress your friends and coworkers by being among the first to learn and use our libraries. Provide early and critical feedback on our forum that will shape the way that these libraries are ultimately released.

See this blog post on the value of your feedback: On Achieving Perfection –or– Why We Love Your Feedback (and Why You Can Love Giving It).

Happy coding!

Synchronization with the Concurrency Runtime - Part 2

vinodsu — Thu, 14 May 2009 21:46:00 GMT

In my previous post, I addressed the motivation behind using concurrency runtime’s synchronization primitives and also introduced Critical Section. In this blog, I will cover concurrency runtime’s reader writer lock.

Reader Writer Lock

This class enables multiple threads to read from a shared resource at the same time but only allows one thread to write to it at a time. They share many characteristics with concurrency runtime’s critical section, reader writer locks are non-reentrant and block cooperatively. The reader writer lock resembles the Win32 Slim reader/writer locks (SRWLock). Reader writer lock performs better than critical section in read-mostly environments.

Similarity to Win32 Slim reader/writer locks:

- Can be used only by threads of a single process.

- The reader writer lock can be owned by multiple reader-threads or only one writer thread at any time.

- Non-reentrant.

- Do not support upgrades or downgrades.

Differences with Win32 Slim reader/writer locks:

- The concurrency runtime’s reader writer lock object guarantees that the order of exclusive (writer) lock-ownership is on a first-come, first-serve basis.

- There is no need to explicitly call Initialization of resources before use of the concurrency runtime’s reader writer lock and release after the use of the reader writer lock.

- Cannot specify spin count for the concurrency runtime’s reader writer lock object.

- The concurrency runtime’s reader writer lock enforces cooperative blocking where they yield to other cooperative tasks in the runtime when blocked.

- The concurrency runtime’s reader writer locks give writer preference over readers; i.e. if there are readers and writer(s) simultaneously waiting for the lock, the lock would be handed over to the first writer in queue.

- Exceptions are thrown by the concurrency runtime’s reader writer lock object; on recursive calls, or if unlock is called when the lock is not held, or if a lock is destroyed when being held.

Example:

Given below is a code sample illustrating reader_writer_lock using 4 readers and 1 writer. The readers output the value of the shared data and the writer updates the value of the shared data and outputs it.

// reader_writer_lock.cpp

// compile with: /EHsc

#include <ppl.h>

#include <stdio.h>

#include <windows.h>

using namespace std;

using namespace Concurrency;

//number of iterations each thread performs

static const int NUM_ITERATIONS = 2;

//the shared data that needs protection from race/tearing

static unsigned int sharedData = 0;

//Demonstrates the use of the reader lock

void Reader(reader_writer_lock* pRWLock)

{

for( int i = 0; i < NUM_ITERATIONS; ++i)

{

//use the reader lock

pRWLock->lock_read();

printf_s("Reading %d\n", sharedData);

//Sleep for some time, this is to simulate potential work done while holding the lock

Sleep(100);

//release the lock

pRWLock->unlock();

}

//Demonstrates the use of the writer lock

void Writer(reader_writer_lock* pRWLock)

{

for( int i = 0; i < NUM_ITERATIONS; ++i)

{

//use the writer lock

pRWLock->lock();

printf_s("\tWriting %d\n", ++sharedData);

//Sleep for some time, this is to simulate potential work done while holding the lock

Sleep(100);

//release the lock

pRWLock->unlock();

}

int main()

{

reader_writer_lock rwlock;

//performs reader writer operations in parallel

parallel_invoke(

[&] { Reader(&rwlock); },

[&] { Writer(&rwlock); }

);

return 0;

}

Sample output:

Reading 0

Writing 1

Reading 1

Writing 2

Reading 2

An interesting point to note here is that even though we use 4 readers, we notice that only 3 readers output their value and the lock is taken by the writer. This happens because the lock prefers writers. We do not guarantee the order of task execution but if such a guarantee is required, you could consider using events, which I will cover in the next blog post.

Synchronization with the Concurrency Runtime - Part 1

vinodsu — Wed, 22 Apr 2009 23:21:00 GMT

In a concurrent world multiple entities work together to achieve a common goal. A common way to interact and coordinate is to use shared data. However, shared data must be accessed carefully. This can be achieved through synchronization, primarily using:

i) Blocking methods such as locks and mutexes

ii) Non-blocking methods such as lock-free programming techniques.

I will talk about the synchronization using blocking methods within a process, using constructs provided as part of the concurrency runtime and exposed through the Parallel Pattern Library (PPL). In this blog, I will address the concurrency runtime’s critical section and will cover reader writer lock and events in subsequent blog posts.

For a general picture of the native concurrency runtime, and high level roles of each of its components please refer to this post.

Motivation

Goals of the concurrency runtime’s synchronization primitives:

1. Simple APIs

Unlike their Win32 equivalent, concurrency runtime’s synchronization primitives don’t have C-style initialization and release/destroy type of resource management calls. The exposed interfaces are simple and conform to the C++0x standards and the synchronization objects throw meaningful exceptions on certain illegal operations.

2. Block in a cooperative manner

The synchronization objects are cooperative in nature, in that they yield to other cooperative tasks in the runtime in addition to preempting. For an illustration of this scenario, refer to this post.

Critical Section

This represents a non-reentrant, cooperative mutual exclusion object that uses concurrency runtime’s facilities to enable cooperative scheduling of work when blocked. This class satisfies all Mutex requirements specified in C++0x standards. The concurrency runtime’s critical section provides a C++ façade as compared to its C-styled Win32 equivalent: Windows CRITICAL_SECTION

Similarity to the Win32 CRITICAL_SECTION:

- Can be used only by threads of a single process.

- The critical section object can only be owned by one thread at a time.

Differences with Win32 CRITICAL_SECTION:

- The concurrency runtime’s critical sections are non-recursive. Exceptions are thrown upon recursive calls.

- The concurrency runtime’s critical section object guarantees that threads waiting on a critical section acquire it on a first-come, first-serve basis.

- There is no need to explicitly call Initialization/allocation of resources before use of the concurrency runtime’s critical section and release resources after the use of the critical section.

- Cannot specify spin count for the concurrency runtime’s critical section object.

- The concurrency runtime’s critical section enforces cooperative blocking where they yield to other cooperative tasks in the runtime when blocked.

- Exceptions are thrown by the concurrency runtime’s critical section object; on unlock calls when the lock is not held, or if a lock is destroyed when being held.

Example:

The sample below alternates between printing to standard output from FunctionA and FunctionB.

// critical_section.cpp

// compile with: /EHsc

#include <ppl.h>

#include <stdio.h>

#include <windows.h>

using namespace std;

using namespace Concurrency;

//number of iterations each thread performs

static const int NUM_ITERATIONS = 5;

//Demonstrates use of critical section

void FunctionA(critical_section* pMutex)

{

for( int i = 0; i < NUM_ITERATIONS; ++i)

{

//use exclusive lock

pMutex->lock();

printf_s("A %d\n", i);

//Sleep for some time, this is to simulate potential work done while holding the lock

Sleep(100);

//release exclusive lock

pMutex->unlock();

}

//Demonstrates use of critical section

void FunctionB(critical_section* pMutex)

{

for( int i = 0; i < NUM_ITERATIONS; ++i)

{

//use exclusive lock

pMutex->lock();

printf_s("\tB %d\n", i);

//Sleep for some time, this is to simulate potential work done while holding the lock

Sleep(100);

//release exclusive lock

pMutex->unlock();

}

int main()

{

critical_section mutex;

//call FunctionA and FunctionB in parallel

parallel_invoke(

[&] { FunctionA(&mutex); },

[&] { FunctionB(&mutex); }

);

return 0;

}

Sample output:

Note: There is a possibility that the order may be swapped, where B gets the lock before A; the locks are handed out on a first-come, first-serve basis, it’s a race to try and get the lock at the beginning. One way of guaranteeing consistency is using events.

Resource Management in the Concurrency Runtime – Part 1

Atilla Gunal — Tue, 10 Mar 2009 13:03:00 GMT

In my previous blog post, I gave an introduction to the native concurrency runtime in Visual Studio 2010. For a general picture of the native concurrency runtime, and high level roles of each of its components please refer to this post. Today, I will talk about the lowest level in that component stack: the Resource Manager (RM).

Resource Manager Responsibilities

In short, there are two main responsibilities of the Resource Manager:

1- Initial Allocation: Allocating resources to schedulers when schedulers are created.

2- Dynamic Migration: Constantly monitoring utilization of resources by schedulers, and dynamically migrating resources between schedulers.

This post will focus on the initial allocation and the dynamic migration will be detailed in my next blog post.

What is a Resource Anyway?

Before going into details about the “Resource Manager” for the Concurrency Runtime, let’s define what is meant by a resource in this context. A resource is the unit of the processing power of the machine which is owned by the RM and used by schedulers. A resource is abstracted in the form of a “Virtual Processor” in Concurrency Runtime. At a given time each virtual processor is associated with a thread running on a processor core (see below - Resource Boundary). The RM creates virtual processors and hands them to schedulers; meanwhile, scheduler instances use them to execute tasks in parallel.

Why do we need a Resource Manager?

Clearly, we wouldn’t need a Resource Manager if it was possible for a single scheduler to fit all the needs of parallel tasks. However, there are a couple of reasons why it is feasible for an app to use multiple schedulers:

· The hotter the cache, the faster the processing: Virtual processors have affinity to group of processor cores; the sockets or nodes. The more they do related work, the more likely they are to find that the data they need is already in the cache.

o Assume that your app uses a bunch of libraries. It is highly probable that work performed by a given library is related. Having a scheduler per library will keep the cache warm and enable faster processing.

· Not all schedulers are same: A scheduler instance is policy driven. This means different schedulers might have different characteristics.

o Assume that a game engine is to be implemented. As an example, it could have an image rendering component, a sound rendering component and an object state update component. All these will execute in parallel; however, we might want state update to have a higher thread priority and would like to decouple sound processing tasks from image rendering tasks. Having multiple schedulers will enable us to customize scheduler behavior and separate task execution.

Now given that there can be multiple schedulers in a process for the reasons mentions above, let’s talk about if a resource manager was not present what would happen:

1- Schedulers would be free to use any resource

2- Schedulers would implement a way of sharing resources

Apparently the first one would end up with a non-optimum utilization of resources; some processors cores would be over utilized and some would be underutilized due to no communication between schedulers. On the other hand, the second one would end up with a complex design contradicting with design patterns like ‘Low Coupling’ or ‘High Cohesion’ (http://en.wikipedia.org/wiki/GRASP_(Object_Oriented_Design) ). Thanks to RM that it helps resources to be utilized evenly meanwhile still keeping the design of a scheduler simpler. By the way it is important to keep the design of the scheduler simpler because users can also implement a custom scheduler on top of RM (see below for more on this).

Additionally, the scheduler implemented as part of the concurrency runtime is a general purpose scheduler; for specialty scenarios, it may be possible to implement a scheduler tuned for a particular purpose, and it’s important that such a scheduler be usable in conjunction with other work in the same process and without oversubscribing on resources. Thanks to RM again that by using its APIs custom schedulers can be built meanwhile respecting the resource usage of other schedulers in the process.

The Initial Allocation of Resources

Now we can describe initial allocation, one of the core functions of the Resource Manager.

As mentioned in the Resource Manager Responsibilities paragraph, initial allocation is about providing the resources to a scheduler at its creation time. A diagram of initial allocation steps can be found below.

A scheduler instance will place its request for resources by providing a set of policies to the resource manager. The resource manager will do the allocation depending on the policies and taking into account the other schedulers in the process. Eventually the resources will be provided to the scheduler in need.

The policy values given in step-1 play an important role in how many virtual processors the scheduler will have. Let’s go through those policies that effect initial allocation:

MinConcurrency and MaxConcurrency: These two policies specify the concurrency level of the scheduler. The number of virtual processors allocated to the scheduler by the RM will be bounded by these policy values. Depending on the existence and policy of the other schedulers, any number of resources can be allocated in between.

TargetOversubscriptionFactor: Oversubscription is defined as the number of threads associated with a particular processor core. RM will try to allocate resources up to MaxConcurrency by oversubscribing each core with the given TargetOversubscriptionFactor.

Here is an example where the scheduler actually asks for minimum of 1 processor core and maximum of 2 processor cores both oversubscribed by a factor of 2. If there are no other schedulers, on a 2 core machine the allocation will be as shown below.

ResourceAllocationPriority: RM will try to satisfy the requests of the schedulers with higher priority at the cost of lower priority schedulers.

Now let’s take a look at a general overview of the rules of initial allocation and some examples of it in order to understand the usage of the policy values.

Rules of Initial Allocation

Initial allocation has a set of rules that defines its behavior. I will mention those rules and then in the next paragraph will try to give examples to explain in more detail.

1) RM will never allocate less resources then MinConcurrency and more resources than MaxConcurrency

2) RM will share cores within schedulers as minimum as possible

3) The resources will be allocated as close as possible. The closeness criteria here is that the processor cores in a NUMA node / processor socket are closer with respect to processor cores in other NUMA nodes / processor sockets.

4) The resources will be proportionally distributed with respect to MaxConcurrency within the schedulers of equal priority if MaxConcurrency cannot be met.

5) Until MaxConcurrency resources are provided to a higher priority scheduler, lower priority schedulers will be reduced to their MinConcurrency

Examples of Initial Allocation

We have mentioned that multiple schedulers may reside in a process. What if there is an existing Scheduler (S1) when a Scheduler (S2) with the same ResourceAllocationPriority requests resources?

RM will try to avoid sharing of processor cores between S1 and S2 as much as possible.

Scenario1: If there are no resources available to satisfy S2 then a proportional allocation will be performed between all schedulers that have the same ResourceAllocationPriority (S1 and S2). Allocation is proportional to each scheduler’s policy value of MaxConcurrency.

Scenraio2: If there are N available resources such that MinConcurrency of S2 <= N <= MaxConcurrency of S2, N resources are allocated to S2, and S1 is left untouched.

Please note here that S1’s number of allocated cores is reduced from 4 to 2 when a scheduler with equal ResourceAllocationPriority is created.

Resources of S1 is not touched and S2 gets only 3 of its requested resources when there are at least MinConcurrency number of resources available for S2.

What if ResourceAllocationPriority of S2 is greater than S1?

In this case RM will lower S1 to MinConcurrency, and try to allocate N resources where MinConcurrency of S2 <= N <= MaxConcurrency to S2. If N is less than the minimum that S2 requires, RM will share processors cores with S1, to satisfy S2’s minimum request.

What if the machine supports NUMA architecture? Is locality taken into account?

Yes, RM will try to localize the allocation as much as possible. For example, on a machine with 2 sockets and 2 processor cores in each socket, the allocations will be made within socket boundaries.

What if both S1 and S2 has MinConcurrency=MaxConcurrency=Number of processor cores of the machine?

Since RM can’t allocate less than MinConcurrency and both schedulers have requested all cores, all the cores will be shared between S1 and S2. In general, RM will share cores in the final step of initial allocation algorithm, and share as little as possible.

What if one of S1 shutdowns? Are the resources made available to S2?

If S2 doesn’t have MaxConcurrency allocated at the time S1 shutdowns, dynamic migration will increase the number of resources of S2. However dynamic resource management is a whole topic unto itself, and I will write about it in my next blog.

Concurrency Runtime and Windows 7

dmccrady — Thu, 05 Feb 2009 05:30:32 GMT

Microsoft has recently released a beta for Windows 7, and a look at the official web site (http://www.microsoft.com/windows/windows-7/default.aspx) will show you a pretty impressive list of new features and usability enhancements. I encourage you to go look them over for yourself, but in this article I’m going to focus on a couple of Win7 features that are of particular importance to achieving the most performance of your parallel programs.

1. Support for more than 64 processors

2. User-Mode Scheduled Threads

Both of the new features I’m going to talk about will be supported in the Microsoft Concurrency Runtime, which will be delivered as part of Visual Studio 10.

An important note about each of these features is that they’re only supported on the 64-bit Windows 7 platform.

More Than 64 Processors

The most straightforward of these new features is Windows 7’s support for more than 64 processors. With earlier Windows OS’s, even high end servers could only schedule threads among a maximum of 64 processors. Windows 7 will allow threads to run on more than 64 processors by allowing threads to be affinitized to both a processor group, and a processor index within that group. Each group can have up to 64 processors, and Windows 7 supports a maximum of 4 processor groups. The mechanics of this new OS support are detailed in a white paper which is available at http://www.microsoft.com/whdc/system/Sysinternals/MoreThan64proc.mspx.

However, unless you actively modify your application to affinitize work amongst other processor groups, you’ll still be stuck with a maximum of 64 processors. The good news is that if you use the Microsoft Concurrency Runtime on Windows 7, you don’t need to be concerned at all with these gory details. As always, the runtime takes care of everything for you, and will automatically determine the total amount of available concurrency (e.g., total number of cores), and utilize as many as it can during any parallel computation. This is an example of what we call a “light-up” scenario. Compile your Concurrency Runtime-enabled application once, and you can run it on everything, from your Core2-Duo up to your monster Win7 256-core server.

User Mode Scheduling

User Mode Scheduled Threads (UMS Threads) is another Windows 7 feature that “lights up” in the Concurrency Runtime.

As the name implies, UMS Threads are threads that are scheduled by a user-mode scheduler (like Concurrency Runtime’s scheduler), instead of by the kernel. Scheduling threads in user mode has a couple of advantages:

1. A UMS Thread can be scheduled without a kernel transition, which can provide a performance boost.

2. Full use of the OS’s quantum can be achieved if a UMS Thread blocks for any reason.

To illustrate the 2^nd point, let’s assume a very simple scheduler with a single work queue. In this example, I’ll also assume that we have 100 tasks that can be run in parallel on 2 CPU’s.

Here we’ve started with 100 items in our task queue, and two threads have picked up Task 1 and Task 2 and are running them in parallel. Unfortunately, Task 2 is going to block on a critical section. Obviously, we would like the scheduler (i.e. the Concurrency Runtime) to use CPU 2 to run the other queued tasks while Task 2 is blocked. Alas, with ordinary Win32 threads, the scheduler cannot tell the difference between a task that is performing a very long computation and a task that is simply blocked in the kernel. The end result is that until Task 2 unblocks, the Concurrency Runtime will not schedule any more tasks on CPU 2. Our 2-core machine just became a 1-core machine, and in the worst case, all 99 remaining tasks will be executed serially on CPU 1.

This situation can be improved somewhat by using the Concurrency Runtime’s cooperative synchronization primitives (critical_section, reader_writer_lock, event) instead of Win32’s kernel primitives. These runtime-aware primitives will cooperatively block a thread, informing the Concurrency Runtime that other work can be run on the CPU. In the above example, Task 2 will cooperatively block, but Task 3 can be run on another thread on CPU 2. All this involves several trips through the kernel to block one thread and unblock another, but it’s certainly better than wasting the CPU.

The situation is improved even further on Windows 7 with UMS threads. When Task 2 blocks, the OS gives control back to Concurrency Runtime. It can now make a scheduling decision and create a new thread to run Task 3 from the task queue. The new thread is scheduled in user-mode by the Concurrency Runtime, not by the OS, so the switch is very fast. Now both CPU 1 and CPU 2 can now be kept busy with the remaining 99 non-blocking tasks in the queue. When Task 2 gets unblocked, Win7 places its host thread back on a runnable list so that the Concurrency Runtime can schedule it – again, from user-mode – and Task 2 can be continued on any available CPU.

You might say, “hey my task doesn’t do any kernel blocking, so does this still help me?” The answer is yes. First, it’s really difficult to know whether your task will block at all. If you call “new” or “malloc” you may block on a heap lock. Even if you didn’t block, the operation might page-fault. An I/O operation will also cause a kernel transition. All these occurrences can take significant time and can stall forward progress on the core upon which they occur. These are opportunities for the scheduler to execute additional work on the now-idle CPU. Windows 7 UMS Threads enables these opportunities, and the result is greater throughput of tasks and more efficient CPU utilization.

(Some readers may have noticed a superficial similarity between UMS Threads and Win32 Fibers. The key difference is that unlike a Fiber, a UMS Thread is a real bona-fide thread, with a backing kernel thread, its own thread-local state, and the ability to run code that can call arbitrary Win32 API’s. Fibers have severe restrictions on the kinds of Win32 API’s they can call.)

Obviously the above example is highly simplified. A UMS Thread scheduler is an extremely complex piece of software to write, and managing state when an arbitrary page fault can swap you out is challenging to say the least. However, once again users of the Concurrency Runtime don’t have to be concerned with any of these gory details. Write your programs once using PPL or Agents, and your code will run using Win32 threads or UMS Threads.

For more details about UMS Threads, check out Inside Windows 7 - User Mode Scheduler (UMS) with Windows Kernel architect Dave Probert on Channel9.

Better Together

Windows 7 by itself is a great enabler of fine grain parallelism, because of the above two specific features, as well as some other performance improvements made at the kernel level. The Concurrency Runtime helps bring the fullest potential of Windows 7 into the hands of programmers in a very simple and powerful way.

Dave Probert ‘goes deep’ on Win7 User Mode Scheduled Threads

rickmolloy — Mon, 02 Feb 2009 23:32:00 GMT

Charles Torre just posted a great Channel9 session with Windows kernel architect Dave Probert on functionality new to Windows7: User Mode Scheduled (UMS) Threads. Much like fibers, UMS threads enable a user mode scheduler (like the Concurrency Runtime) to have control over scheduling of threads in user mode; however unlike fibers, UMS threads keep the TEB and thread local information intact and provides notification of kernel blocking which enables more general use.

In the near future we'll be blogging here about how the Concurrency Runtime is taking advantage of UMS threads on Win7, but if you have some time to spare, check out what Dave has to say on this it's a wonderful session.

http://channel9.msdn.com/shows/Going+Deep/Dave-Probert-Inside-Windows-7-User-Mode-Scheduler-UMS/

-Rick

On Channel9: a chat with members from the Visual C++ and Concurrency Runtime teams

rickmolloy — Wed, 14 Jan 2009 04:36:00 GMT

Last week, Don McCrady, Damien Watkins from the C++ team and I sat down with Charles from Channel9 and chatted about the Concurrency Runtime, new C++ work in Visual Studio 2010 and how our teams are working together.

As Charles pointed out, we also touched briefly upon new features in the C++0x standard that haven’t been discussed before.

Here's the link:

http://channel9.msdn.com/shows/Going+Deep/Parallel-Computing-in-Native-Code-New-Trends-and-Old-Friends/

Enjoy and as always your feedback is very welcome,

Rick

An Introduction to Native Concurrency in Visual Studio 2010

Atilla Gunal — Mon, 12 Jan 2009 15:25:00 GMT

In this blog I will be giving an introduction to the native concurrency support in Visual Studio 2010. My motivation is that an architectural understanding of the features will enable the reader to make the most of the underlying infrastructure. In future blogs, I will go deeper into each sub topic.

The Goal

The ultimate goal of the native concurrency is to provide tools to the developers that will enable them to introduce scalable and maintainable parallelism into their applications.

One side of the goal is to inject efficient parallelism as easy as writing the following:

parallel_for(0, 10, [=](int i){ foo(i); } );

where foo() will be evaluated in parallel by the runtime. The other side is to enable productivity via the provided programming models.

The Architecture of Native Concurrency in Visual Studio 2010

Here’s a picture of the major components in the native concurrency development stack. I’d like to go through each of them and the interactions in between.

Resource Manager (RM)

As the name of the component suggests, the main responsibility of the RM is to manage the resources, where the resources here are the processor cores of the system. These resources are requested by the scheduler(s) to execute parallel work and are then distributed by the RM to the scheduler(s).

In order to accomplish the distribution, the RM captures the machine topology (i.e. number of processor cores and the closeness of the processor cores to each other) at its creation time. As the schedulers are created, it takes into account various factors to make the allocation including:

i- Minimum and maximum number of cores requested by each scheduler

ii- Scheduler priority

iii- Utilization of allocated processor cores per scheduler

iv- Closeness of the allocated cores of each scheduler

(i.e. on a NUMA Architecture RM will try to allocate cores as close as possible. In other words, RM will span the allocation to a minimum number of NUMA nodes)

RM will not only allocate resources at the scheduler's creation time but also continue to monitor all schedulers to maximize the utilization of resources by taking from the ones that don't utilize and by giving to the ones that are in need.

Users that are willing to implement their own scheduler will mostly use this interface of the runtime.

For more information on concurrency runtime's resource manager please refer to the following blog posts:

Scheduler

Each parallel application consists of multiple tasks to be executed. As an example for the code given above; foo(0), foo(1), ..., foo(9) can be considered as tasks to be executed in parallel. It is critical to have a component to have this set of work distributed to the processor cores available for execution in an efficient way. Within the Concurrency Runtime this is responsibility of the Scheduler.

The Scheduler, despite having a predefined default behavior, can be customized through a set of policies such as: the number of resources it will use, the number of OS threads mapped to its allocated resources, its resource allocation priority, and either it should give priority to execute tasks to increase cache hits or improve fairness across tasks.

An important aspect of the Scheduler is that it is cooperative in the sense that it will not preempt a running task until it finishes execution or that task cooperatively yields its execution on behalf of other tasks. This is particularly important since it avoids context switches and cache being thrashed due to randomization introduced by preemption.

Users that are willing to implement their own concurrent library will mostly target this interface of the runtime.

PPL

PPL stands for the Parallel Pattern Library and is meant to provide a convenient interface to execute work in parallel. By using PPL, you can introduce parallelism without even having to manage a scheduler. Here are the common patterns available:

Task execution patterns:

parallel_invoke: To execute from 2 to 10 tasks in parallel

parallel_for: To execute tasks in parallel over a range of integers

parallel_for_each: To execute tasks in parallel over a collection of items in an STL container

Synchronization patterns:

reader_writer_lock: A cooperative reader-writer lock that yields to other tasks instead of preempting.

critical_section: A cooperative mutual exclusion primitive that yields to other tasks instead of preempting.

For more information on synchronization patterns please refer to the following blog posts:

Data sharing pattern:

combinable: A scalable object that has a local copy for each thread where processing can be done lock free on the local copy and combined afterwards when parallel processing is done. For more info on combinable please refer to this.

Application developers will mostly use this interface to inject parallelism.

Agents

We all know that building software is not trivial. Building software that has to manage concurrent access to shared state is even harder. Agents are a model for decomposing parallel applications to make them easy to design, build and maintain. With Agents, you can implement the main components of the software in isolation, communicate between them by message passing (i.e. via messaging blocks), and in each Agent introduce concurrent computation using finer grain constructs like the PPL. For more info about using Agents and messaging bocks please refer to this blog or this Channel9 video.

Here are also other interesting blogs on agents and messaging blocks:

What next?

I hope this blog helped in a better understanding of what the Concurrency Runtime is about. I will continue to detail each component as I give more concrete examples while digging that particular feature. Your comments are valuable for me alot so please feel free to provide feedback.