jaredpar's WebLog : Threading

A more usable API for a mutable thread safe collection

Jared Parsons — Mon, 16 Feb 2009 16:00:00 GMT

In my last post we discussed the problems with designing a safer API for mutable thread safe collections that employ only an internal locking system. The result was an API that was more difficult to mess up, yet pretty much unusable. Lets take a look at this problem and see if we can come up with a usable API that still helps to eliminate mistakes.

One of the main issues I have with mutable thread safe collections is the use of decision procedures such as Count and Contains. Procedures such these only return information that pertains to the collection as it existed at a previous point in time. It can provide no relevant information to the collection in it’s current state and only encourages the user to write bad code. For example.

if (col.Count > 0) {
    // Collection can be modified before this next line executes leading to 
    // an error condition
    var first = col[0]; 
}

Therefore they have no place on a mutable thread safe collection. Yet, once you take away these procedures, you’re left with a collection that is virtually useless. It can only have a minimal API by which to access data. Here is the last example we were left with

public sealed class ThreadSafeList<T> {
    public void Add(T value) { ... }
    public bool TryRemove(T value) { ... }
    public bool TryGet(int index, out T value) { ... }
}

This is hardly a usable API. What’s worse, as wekempf point out, is that I inadvertently exposed a decision procedure in this API. It’s possible to infer state about a lower or equal index by a successful return result from TryGet(). For example, a user may say that “if I can access element 2, then surely element 1 must exist”. The result would still be evident in code (ignoring the return value of a TryGet method should be a red flag). But a better choice for this method would have been a TryGetFirst().

At the end of the day, users are going to want some level of determinism out of their collections. It’s possible to program against API’s like the above, but most people won’t do it. In order to be more used, the collection must be able to reliably implement procedures such as Count and Contains and allow the user to use the return to reason about the state of the collection.

One way to do this is to simply exposed the internal lock to the consumer of the collection. Consumers can take the lock and then query to their hearts content. Lets do a quick modification of the original sample to allow for this.

public sealed class ThreadSafeList<T> {
    private List<T> m_list = new List<T>();
    private object m_lock = new object();

    public object SyncLock { get { return m_lock; } }

    public void Add(T value) {
        lock (m_lock) {
            m_list.Add(value);
        }
    }
    public void Remove(T value) {
        lock (m_lock) {
            m_list.Remove(value);
        }
    }
    public bool Contains(T value) {
        lock (m_lock) {
            return m_list.Contains(value);
        }
    }
    public int Count { get { lock (m_lock) { return m_list.Count; } } }
    public T this[int index] {
        get { lock (m_lock) { return m_list[index]; } }
        set { lock (m_lock) { m_list[index] = value; } }
    }
}

Now we can go back to the original sample code and write a version which can use the decision procedures safely.

lock (col.SyncLock) {
    if (col.Count > 0) {
        var first = col[0];
        ...
    }
}

This code will function correctly. But the API leaves a lot to be desired. In particular …

It provides no guidance to the user as to which procedures must be accessed with the SyncLock object locked. They can just as easily write the original bad sample code.
All procedures used within the lock reacquire the lock recursively which is definitely not advisable. We could provide properties which do not acquire the lock such as CountNoLock that work around this problem. While ok in small doses, it's just a matter of time before you see this snippet in the middle of a huge mostly undocumented function
```
// Lock should be held at this point
int count = col.CountNoLock;
```
This code makes my eyes bleed
The API provides 0 information to the user on exactly what the rules are for this lock. It would be left as an artifact in documentation (which you simply cannot count on users reading).
There is really nothing telling the user that they ever have to unlock the collection. Surely, any user entering into the world of threading should know this but if they do a Monitor.Enter call without a corresponding Monitor.Exit, they will receive no indication this is a bad idea.
Overall this collection requires a lot of new knowledge about the collection to use

This design though is exactly how a “synchronized” collection in 1.0 version of the BCL worked. This code is essentially what you would get by passing an ArrayList instance to ArrayList.Synchronized (and most other BCL 1.0 collections). It was problematic enough that all of the new collections in 2.0 did not implement this feature. Here’s the BCL team’s explanation on this http://blogs.msdn.com/bclteam/archive/2005/03/15/396399.aspx

Overall this design poses several problems because it exposes internal implementation details directly to the consumer. An improved design should seek to hide the lock from direct access. What we really want is a way to not even provide API’s like Count and Contains unless the object is already in a locked state. This prevents them from being used at all in an incorrect scenario.

Lets run with this idea to design a more usable thread safe queue. First we’ll divide the interface for a queue into two parts.

All procedures that have 0 reliance on the internal state of the collection. Namely Enqueue, and Clear. No state is required to use these methods
All procedures that rely on the internal state of the collection to function correctly.

The ThreadSafeQueue class will contain all of the methods in category #1. It will also provide a method which returns an instance of an interface which has all of the methods in category #2.

public interface ILockedQueue<T> : IDisposable{
    int Count { get; }
    bool Contains(T value);
    T Dequeue();
}

The implementation of this interface object will acquire the internal lock of the original ThreadSafeQueue during construction and hold it for the duration if it’s lifetime. This effectively freezes the queue allowing for decision procedures to be used reliably. Implementing IDisposable and releasing the lock in the Dispose method provides a measure of lifetime management.

The rest of the code sample is below.

public sealed class ThreadSafeQueue<T> {

    #region LockedQueue
    private sealed class LockedQueue : ILockedQueue<T> {
        private ThreadSafeQueue<T> m_outer;
        internal LockedQueue(ThreadSafeQueue<T> outer) {
            m_outer = outer;
            Monitor.Enter(m_outer.m_lock);
        }

        #region ILockedQueue<T> Members
        public int Count { 
            get { return m_outer.m_queue.Count; }
        }
        public bool Contains(T value) {
            return m_outer.m_queue.Contains(value);
        }
        public T Dequeue() {
            return m_outer.m_queue.Dequeue();
        }
        #endregion
        #region IDisposable Members
        public void Dispose() {
            Dispose(true);
            GC.SuppressFinalize(this);
        }
        private void Dispose(bool disposing) {
            Debug.Assert(disposing, "ILockedQueue implementations must be explicitly disposed"); 
            if (disposing) {
                Monitor.Exit(m_outer.m_lock);
            }
        }
        ~LockedQueue() {
            Dispose(false);
        }
        #endregion
    }
    #endregion

    private Queue<T> m_queue = new Queue<T>();
    private object m_lock = new object();

    public ThreadSafeQueue() { }
    public void Enqueue(T value) {
        lock (m_lock) {
            m_queue.Enqueue(value);
        }
    }

    public void Clear() {
        lock (m_lock) {
            m_queue.Clear();
        }
    }

    public ILockedQueue<T> Lock() {
        return new LockedQueue(this);
    }
}

This design now cleanly separates out the two modes by which the collection can be asked. It completely hides the explicit synchronization aspects from the users and replaces it with design patterns (such as IDisposable) that they are likely already familiar with. Now our original bad sample can be rewritten as follows.

static void Example1(ThreadSafeQueue<int> queue) {
    using (var locked = queue.Lock()) {
        if (locked.Count > 0) {
            var first = locked.Dequeue();
        }
    }
}

No explicit synchronization code is needed by the user. This design makes it much harder for the user to make incorrect assumptions or misuses of the collection. The “decision procedures” are simply not available unless the collection is in a locked state.

As with most thread safe designs, there are ways in which this code can be used incorrectly

Using an instance of ILockedQueue<T> after it’s been disposed. This though is already considered taboo though and we can rely on existing user knowledge to help alleviate this problem. Additionally, static analysis tools, such as FxCop, will flag this as an error.

With a bit more rigor this can also be prevented. Simply add a disposed flag and check it on entry into every method.
It’s possible for the user to maintain values, such as Count, between calls to Lock and use it to make an incorrect assumption about the state of the list.
If the user fails to dispose the ILockedQueue<T> instance it will be forever locked. Luckily FxCop will also flag this as an error since it’s an IDisposable. It’s not a foolproof mechanism though.
There is nothing that explicitly says to the user “please only use ILockedQueue<T> for a very short time”. IDisposable conveys this message to a point but it’s certainly not perfect.
The actual ILockedQueue<T> implementation is not thread safe. Ideally users won’t pass instances of IDisposable between threads but it is something to think about.

The good news is that two of these flaws (1 and 3) are issues with existing types and tools are already designed to weed them out. FxCop will catch common cases for both of them.

Also, many of these cases are considered bad code in the absence of a thread safe collection. This allows users to rely on existing knowledge instead of forcing them to learn new design patterns for mutable thread safe collections.

Overall I feel like this design is a real win over the other versions. It provides an API which helps to limit the mistakes a user can make with a mutable thread safe collection without requiring a huge deal of new patterns in order to use.

Why are thread safe collections so hard?

Jared Parsons — Wed, 11 Feb 2009 16:00:00 GMT

Writing a collection which is mutable, thread safe and usable is an extremely difficult process. At least that’s what you’ve likely been told all through your schooling. But then you get out on the web and see a multitude of thread safe lists, maps and queues. If it’s so hard, why are there so many examples?

The problem is there are several levels of thread safe collections. I find that when most people say thread safe collection what they really mean “a collection that will not be corrupted when modified and accessed from multiple threads”. Lets call this “data thread safe” for brevity. This type of collection is rather easy to build. Virtually any collection that is not thread safe can be made “data thread safe” by synchronizing access via a simple locking mechanism.

For Example, lets build a data thread safe List<T>.

public sealed class ThreadSafeList<T> : IEnumerable<T> {
    private List<T> m_list = new List<T>();
    private object m_lock = new object();

    public void Add(T value) {
        lock (m_lock) {
            m_list.Add(value);
        }
    }
    public void Remove(T value) {
        lock (m_lock) {
            m_list.Remove(value);
        }
    }
    public bool Contains(T value) {
        lock (m_lock) {
            return m_list.Contains(value);
        }
    }
    public int Count { get { lock (m_lock) { return m_list.Count; } } }
    public T this[int index] {
        get { lock (m_lock) { return m_list[index]; } }
        set { lock (m_lock) { m_list[index] = value; } }
    }
    // IEnumerable<T> left out
}

And there you have it. The lock statement prevents concurrent access from multiple threads. So the actual m_list instance is only ever accessed by a single thread at a time which is all it’s designed to do. This means instances of ThreadSafeList<T> can be used from any thread without fear of corrupting the underlying data.

But if building a data thread safe list is so easy, why doesn’t Microsoft add these standard collections in the framework?

Answer: ThreadSafeList<T> is a virtually unusable class because the design leads you down the path to bad code.

The flaws in this design are not apparent until you examine how lists are commonly used. For example, take the following code which attempts to grab the first element out of the list if there is one.

static int GetFirstOrDefault(ThreadSafeList<int> list) {
    if (list.Count > 0) {
        return list[0];
    }
    return 0;
}

This code is a classic race condition. Consider the case where there is only one element in the list. If another thread removes that element in between the if statement and the return statement, the return statement will throw an exception because it’s trying to access an invalid index in the list. Even though ThreadSafeList<T> is data thread safe, there is nothing guaranteeing the validity of a return value of one call across the next call to the same object.

I refer to procedures like Count as decision procedures. They server only to allow you to make a decision about the underlying object. Decision procedures on a concurrent object are virtually useless. As soon as the decision is returned, you must assume the object has changed and hence you cannot use the result to take any action.

Decision procedures are one of the reasons why Immutable Collections are so attractive. They are both data thread safe and allow you to reason about there APIs. Immutable Collections don’t change ever. Hence it’s perfectly ok to have decision procedures on them because the result won’t get invalidated.

The fundamental issue with ThreadSafeList<T> is it is designed to act like a List<T>. Yet List<T> is not designed for concurrent access. When building a mutable concurrent collection, in addition to considering the validity of the data, you must consider how design the API to deal with the constantly changing nature of the collection.

When designing a concurrent collection you should follow different guidelines than for a normal collection class. For example.

Don’t add an decision procedures. They lead users down the path to bad code.
Methods which query the object can always fail and the API should reflect this.

Based on that, lets look at a refined design for ThreadSafeList<T>

public sealed class ThreadSafeList<T> {
    private List<T> m_list = new List<T>();
    private object m_lock = new object();

    public void Add(T value) {
        lock (m_lock) {
            m_list.Add(value);
        }
    }
    public bool TryRemove(T value) {
        lock (m_lock) {
            return m_list.Remove(value);
        }
    }
    public bool TryGet(int index, out T value) {
        lock ( m_lock ) {
            if( index < m_list.Count ) {
                value = m_list[index];
                return true;
            }
            value = default(T);
            return false;
        }
    }
}

Summary of the changes

Both the Contains and Count procedures were removed because they were decision procedures
Remove was converted to TryRemove to indicate it’s potential to fail
The TryGet property was added and is reflective of the fragile nature of the method. Sure it’s possible for users to simply ignore the return value and plow on with the invalid value. However the API is not lulling the user into a false sense of security
The collection no longer implements IEnumerable<T>. IEnumerable<T> is only valid when a collection is not changing under the hood. There is no way to easily make this guarantee with a collection built this way and hence it was removed.
The indexers were removed as well. I’m a bit wishy washy in this particular point as there is nothing in the API which gives a user a false sense of security. But at the same time mutable concurrent collections are dangerous and should be treated with a heightened sense of respect so the indexers were removed.

This version of ThreadSafeList is more resilient, but not immune to, accidental user failure. The design tends to lead users on the path to better code.

But is it really usable? Would you use it in your application?

Thread Local Storage template

Jared Parsons — Mon, 21 Apr 2008 20:52:03 GMT

Thread local storage is another method of synchronization between threads. It is different that most synchronization cases because instead of sharing state between threads it enables developers to have independent, thread specific pieces of data which have a similar or common purpose.

The uses of thread local storage (TLS) vary greatly but is a very powerful and lightweight method for storing data. TLS can easily be envisioned as a giant void* array for every thread. The entry point, TlsAlloc, provides an index into this array and allows the storage of arbitrary data [1].

TLS is particularly useful for storing state information. For example, one of my components lives in a highly multi-threaded environment. Each thread serves essentially the same purpose and has the same states and state transition semantics. Like any good paranoid programmer I wanted to add contracts to check my state transitions and semantics.

Contract.VerifyState(ExpectedState, ???CurrentState)

The question is where to store the state information for a thread? A global state variable won't suffice because there are N threads. A global array of state information also has it's share of problems: synchronization, determining an index, lifetime.

TLS is ideally suited to this scenario. Each thread has an independent but similar concept of state. In my initialization code I allocate an TLS index and now I have a place to store my state.

Contract.VerifyState(ExpectedState, *TlsGetValue(g_stateTlsIndex)

The next question is how to manage the lifetime? TLS provides a void* and the caller must manage the lifetime of the allocated memory. Since this is thread specific the ideal place is to manage the memory in the thread startup proc. However I don't own the creation of the thread, my component is called on a number of threads so this won't work.

The solution is to use the stack. The initial return for TlsGetValue is NULL. If this situation is detected then the current stack frame is set to own the memory for the slot. Further accesses to the value do not own the memory and simply access it. The semantics are straight forward but annoying to constantly rewrite, so naturally write a template :)

    template <typename T>
    class TlsValue
    {
    public:
        TlsValue(DWORD index, const T& defaultValue=T()) :
            m_pValue(NULL),
            m_index(index),
            m_owns(false)
        {
            m_pValue = reinterpret_cast<T*>(::TlsGetValue(m_index));
            if ( !m_pValue )
            {
                m_pValue = new T(defaultValue);
                m_owns = true;
                ::TlsSetValue(m_index, m_pValue);
            }
        }
        ~TlsValue()
        {
            if ( m_owns )
            {
                ::TlsSetValue(m_index, NULL);
                delete m_pValue;
            }
        }

        T* Value() const
        {
            return m_pValue;
        }

    private:
        // Do not auto generate
        TlsValue();
        TlsValue(const TlsValue<T>&);
        TlsValue& operator=(const TlsValue<T>&);

        T* m_pValue;
        DWORD m_index;
        bool m_owns;
    };

In addition to this blog post, I added a working sample to http://code.msdn.microsoft.com/TlsValue. This is my first attempt at posting a sample on http://code.msdn.com so please provide any and all feedback on the data.

[1] This is similar to data marked with the ThreadStatic attribute in managed code without all of the slot messiness and with the added benefit of strong typing.

ActiveObject

Jared Parsons — Sun, 30 Mar 2008 19:39:28 GMT

I've been busy lately and neglected my series on Active Objects. It's been a fairly busy time for me both in and out of work. Enough excuses, back to the fun.

With the basic PipeSingleReader class, we now have the last piece necessary to create an ActiveObject. This article will focus on building the base ActiveObject which will take care of scheduling, construction, destruction and error handling. The goal is to make implementing an ActiveObject that actually does work easy.

Lets break down the implementation of an ActiveObject into the three phases of any object; construction, destruction and running behavior.

Construction

Active Objects are associated with and have an affinity to a particular thread. Constructing an ActiveObject mainly consists of creating a thread and initializing the member variables of the object.

There are a couple of requirements that need to be met when initializing the object. The first is getting the thread into a known state before returning out of the constructor. It's possible for coders to create and then immediately destroy an object. Part of destructing an object is understanding the state you are destructing. Returning from an ActiveObject constructor before the thread is up and running means that we can be destructed while in an inconsistent state. Normally this isn't much an issue with objects because they are single threaded. We will fix this by doing a simple wait until the thread is finished initializing.

        protected ActiveObject() {
            m_thread = new Thread(() => InitializeAndRunBackgroundThread());
            m_thread.Start();
            while (0 == m_backgroundInitialized) { Thread.Sleep(0); }
        }

Next is providing implementers with a way to initialize member variables on the new thread. There are many reasons for wanting to initialize members on the ActiveObject thread. Besides general consistency concerns, there is also the issue that objects can have affinity to a particular thread and including forcing initialization to occur on that thread. To make this simple part of the thread initialization code will call a virtual method allowing base classes to initialize variables.

        private void InitializeAndRunBackgroundThread() {
            Interlocked.Exchange(ref m_affinity, new ThreadAffinity());
            Interlocked.Exchange(ref m_pipe, new PipeSingleReader<Future>());
            InitializeMembersInBackground();
            Interlocked.Exchange(ref m_backgroundInitialized, 1);
            RunBackgroundActions();
        }

        protected virtual void InitializeMembersInBackground() {
        }

Running Behavior

Active Objects exist for one reason, to run Futures. The main behavior is to loop over the set of Futures and run them. The PipeSingleReader class takes care of most of the scheduling and threading work. This leaves the ActiveObject free to make policy decisions.

One question that comes up is how to handle the case where a Future throws an exception? If we run the Future with no protection it will simple cause an unhandled exception and likely a process crash. We could catch and try to filter them but based on what criteria? IMHO there is no way to properly handle an exception in the Active Object base because we don't know what the purpose of that object is. Only the actual object implementer knows. Therefore we will make it their problem by passing unhandled exceptions into an abstract method.

        private void RunBackgroundActions() {
            do {
                RunFuture(m_pipe.GetNextOutput());
            } while (0 == m_backgroundFinished);
            Future future;
            while (m_pipe.TryGetOutput(out future)) {
                RunFuture(future);
            }
        }
        private void RunFuture(Future future) {
            try {
                m_affinity.Check();
                future.Run();
            }
            catch (Exception ex) {
                OnBackgroundUnhandledException(ex);
            }
        }

If the second loop looks a bit out of place, hopefully the destruction section will explain it's purpose.

All that is left is to provide helper methods to let base classes queue up Futures to run.

        protected Future RunInBackground(Action action) {
            var f = Future.CreateNoRun(action);
            m_pipe.AddInput(f);
            return f;
        }
        protected Future<T> RunInBackground<T>(Func<T> func) {
            var f = Future.CreateNoRun(func);
            m_pipe.AddInput(f);
            return f;
        }

Destruction

Destruction of an ActiveObject can be tricky with respect to handling pending actions. Should they be executed, aborted or just completely ignored? What happens if more input is added once we start the dispose process? If we don't allow more input where should we error?

IMHO, the simplest user and programming model is the following.

Dispose is synchronous. It will block until the background thread is destroyed. Dispose is the equivalent of destruction so it follows that all resources including the thread will be destroyed when destruction completes.
Once dispose starts input will be stopped. This prevents live-lock scenarios where one thread is disposing the ActiveObject and another thread is constantly adding data.
If another thread tries to add an operation during the middle of disposing they will be given an exception at that time.

In future posts, we'll explore how to create ActiveObjects with differing dispose semantics.

Now how can we signal the background thread that we are done processing? Just add a future to the queue to be running. Because this will run on the only thread reading the int there is no need for an Interlocked operation.

        private void Dispose(bool disposing) {
            if (disposing) {
                m_pipe.AddInput(Future.CreateNoRun(() => { m_backgroundFinished = 1; }));
                m_pipe.CloseInput();
                m_thread.Join();
            }
        }

Now that we've gone over the dispose code, hopefully the reason for the second loop in RunBackgroundActions is a little more apparent. Between the two calls to m_pipe in Dispose another thread can post a Future. Without the second loop the user will get no exception and the future will never run. Likely they would hopelessly deadlock. The second loop will run all Futures which get caught it this gap.

The Code

Here is the full version of the code.

    public abstract class ActiveObject :  IDisposable {
        private PipeSingleReader<Future> m_pipe;
        private ThreadAffinity m_affinity;
        private Thread m_thread;
        private int m_backgroundInitialized;
        private int m_backgroundFinished;

        protected ActiveObject() {
            m_thread = new Thread(() => InitializeAndRunBackgroundThread());
            m_thread.Start();
            while (0 == m_backgroundInitialized) { Thread.Sleep(0); }
        }
        #region Dispose
        public void Dispose() {
            Dispose(true);
            GC.SuppressFinalize(this);
        }
        ~ActiveObject() {
            Dispose(false);
        }
        private void Dispose(bool disposing) {
            if (disposing) {
                m_pipe.AddInput(Future.CreateNoRun(() => { m_backgroundFinished = 1; }));
                m_pipe.CloseInput();
                m_thread.Join();
            }
        }
        #endregion
        private void InitializeAndRunBackgroundThread() {
            Interlocked.Exchange(ref m_affinity, new ThreadAffinity());
            Interlocked.Exchange(ref m_pipe, new PipeSingleReader<Future>());
            InitializeMembersInBackground();
            Interlocked.Exchange(ref m_backgroundInitialized, 1);
            RunBackgroundActions();
        }
        private void RunBackgroundActions() {
            do {
                RunFuture(m_pipe.GetNextOutput());
            } while (0 == m_backgroundFinished);
            Future future;
            while (m_pipe.TryGetOutput(out future)) {
                RunFuture(future);
            }
        }
        private void RunFuture(Future future) {
            try {
                m_affinity.Check();
                future.Run();
            }
            catch (Exception ex) {
                OnBackgroundUnhandledException(ex);
            }
        }
        protected Future RunInBackground(Action action) {
            var f = Future.CreateNoRun(action);
            m_pipe.AddInput(f);
            return f;
        }
        protected Future<T> RunInBackground<T>(Func<T> func) {
            var f = Future.CreateNoRun(func);
            m_pipe.AddInput(f);
            return f;
        }
        protected virtual void InitializeMembersInBackground() {
        }
        protected abstract void OnBackgroundUnhandledException(Exception ex);
    }

PipeSingleReaderNoLock

Jared Parsons — Mon, 03 Mar 2008 08:46:00 GMT

Previously we discussed a multi-thread safe queue like data structure using locks as an internal synchronization mechanism. This time we'll look at a version requiring no locks.

In the previous version, locks were used to synchronize access to an underlying queue which stored the data. Removing the locks necessitates us moving away from Queue<T> to store the data and forces us onto another structure which is safe in the presence of multiple threads.

Immutable collections are safe in the presence of multiple threads and don't require any synchronization mechanisms. However they're immutable and we're trying to build a mutable data structure? No matter. Even though each instance is immutable they can be used to create new instances which represent a somewhat mutated state.

For this exercise we will be using a slight variant of Eric Lippert's Immutable Stack implementation [1]. The pipe will have two stacks; 1) for capturing input and 2) for storing output. Named m_writeStack and m_readStack respectively.

This makes the implementation of reading input straight forward. While m_readStack is not empty the reader thread can systematically pop off the values. This doesn't require any contention with writer threads and since their is only one reader thread and the stack is immutable the code is straight forward. Once the m_readStack is empty the reader thread will swap out the current state of m_writeStack with an empty stack. The original value will be reversed so we can maintain the FIFO ordering and set as the new m_readStack.

private bool CheckForInput() {
    if( m_readerStack.IsEmpty ) {
        var prev = Interlocked.Exchange(ref m_writerStack, ImmutableStack<T>.Empty);
        m_readerStack = prev.Reverse();
    }
    return !m_readerStack.IsEmpty;
}

Writing data is more complicated because it must deal with contention for updating the same data structure. It can be altered by other writers or the reader thread when it runs out of data. Basically it must push a value onto the stack and update the m_writerStack to point to the new value. In between the operations the value of m_writerStack could change and thus invalidate the push. To guard against this the writer must guarantee the current value of the m_writerStack is the same as it was before the push operation. Interlocked.CompareExchange will do the trick. If it's been changed then repeat the operation.

public void AddInput(T value) {
    bool done;
    do {
        if (m_inputClosed) {
            throw new InvalidOperationException("Input end of pipe is closed");
        }
        var originalStack = m_writerStack;
        var newStack = originalStack.Push(value);
        var currentStack = Interlocked.CompareExchange(ref m_writerStack, newStack, originalStack);
        done = object.ReferenceEquals(currentStack, originalStack);
    } while (!done);
    m_event.Set();
}

At a glance it may seem like this suffers from the ABA problem. This is not the case and it's easy to prove. In between the read and push operation only two other operations can modify the m_writeStack variable. The first is another write which will produce a new value of ImmutableStack<T>. In this case the CLR guarantees that the references will not be equal and the CAS operation won't succeed. The second is the reader thread swaps out the value and replaces it with Empty. It's possible to hit an ABA situation here (detailed below) but fundamentally if it was empty before and empty now the operation is still safe. No data is lost because we are still replacing an empty stack with a single value'd stack.

Here's a more elaborate version starting with an empty pipe of type int

Thread 1: Begins to write 5 and is stopped just before the CAS operation

originalStack points to ImmutableStack<int>.Empty

Thread 2: Starts and completes a write of the value 6
Thread 3: Reads a value from the pipe, having no data replaces m_writeStack with ImmutableStack<int>.Empty
Thread 1: Resumes and the CAS succeeds even though the m_writeStack technically changed in the middle of the operation

Even though this exhibits many characterstics of the ABA pattern it is not a problem because the behavior is still correct. No data was lost because it was transferred to the reader thread. The end result is m_writeStack pointing to a single vaue'd stack containing the data 5 which is valid. The pipe does not guarantee the ordering of data between writer threads (merely the ordering between a single writer thread).

Below is the implementation in it's entirety.

public class PipeSingleReaderNoLock<T> : IDisposable {
    private readonly ThreadAffinity m_affinity = new ThreadAffinity();
    private ImmutableStack<T> m_readerStack = ImmutableStack<T>.Empty;
    private ImmutableStack<T> m_writerStack = ImmutableStack<T>.Empty;
    private readonly AutoResetEvent m_event = new AutoResetEvent(false);
    private volatile bool m_inputClosed;

    public PipeSingleReaderNoLock() {
    }

    #region Dispose
    public void Dispose() {
        Dispose(true);
        GC.SuppressFinalize(this);
    }
    ~PipeSingleReaderNoLock() {
        Dispose(false);
    }
    private void Dispose(bool disposing) {
        if (disposing) {
            m_event.Close();
        }
    }
    #endregion
    public void WaitForOutput() {
        m_affinity.Check();
        while (!CheckForInput()) {
            m_event.WaitOne();
        }
    }
    public T GetNextOutput() {
        m_affinity.Check();
        T data;
        while (!TryGetOutput(out data)) {
            m_event.WaitOne();
        }
        return data;
    }
    public bool TryGetOutput(out T value) {
        m_affinity.Check();
        if (CheckForInput()) {
            value = m_readerStack.Peek();
            m_readerStack = m_readerStack.Pop();
            return true;
        }
        else {
            value = default(T);
            return false;
        }
    }
    private bool CheckForInput() {
        if (m_readerStack.IsEmpty) {
            var prev = Interlocked.Exchange(ref m_writerStack, ImmutableStack<T>.Empty);
            m_readerStack = prev.Reverse();
        }
        return !m_readerStack.IsEmpty;
    }
    public void AddInput(T value) {
        bool done;
        do {
            if (m_inputClosed) {
                throw new InvalidOperationException("Input end of pipe is closed");
            }
            var originalStack = m_writerStack;
            var newStack = originalStack.Push(value);
            var currentStack = Interlocked.CompareExchange(ref m_writerStack, newStack, originalStack);
            done = object.ReferenceEquals(currentStack, originalStack);
        } while (!done);
        m_event.Set();
    }
    public void CloseInput() {
        m_inputClosed = true;
    }

[1] If you haven't read this series you really should.

PipeSingleReader

Jared Parsons — Sun, 02 Mar 2008 20:53:00 GMT

Before we can get to building an Active Object implementation, there are some more primitive structures we need to define. Active Objects live on a separate thread where every call is executed in a serialized fashion on that thread. The next primitive will allow us to easily pass messages in the form of delegates from the caller to the background thread.

The structure must support adding messages from N callers but only needs to support 1 reader (the active object). Successive writes from the same thread should add the items in their respective order. The input end of the pipe can be closed while leaving the output end active. This allows for the consumer to deterministically reach an end state while not ignoring any any input.

The name of the structure is PipeSingleReader.

Designing a thread safe mutable structure is significantly different from a non-thread safe or immutable data structure. There is a significant temptation to apply existing patterns. We should question every pattern we apply to these collections. Many of these patterns lead us to design API's which encourage bad programming practices.

Our existing collection patterns are designed around structures which behave deterministically on any given thread because they are 1) Immutable or 2) designed for single thread use only. Many of these concepts do not apply to mutable thread safe collections because they are constantly being accessed and mutated by multiple threads. This gives them the appearance of behaving non-deterministically with respect to a given thread.

Take the member Count for instance. This is found on virtually every collection class in the BCL. Yet having it on a mutable thread safe collection class only leads to programming errors. The only reason to have a member such is Count is to use it to make a decision. However in a mutable thread-safe collection making a decision off of this value is wrong. The value can and will change between any two instructions in code.

Take the example below. Just because the Count is >0 in the if block has no dependable relevance to what the value will be inside the if block.

            var ThreadSafeList<int> col = GetList();
            if( col.Count > 0 )
            {
                //...
            }

To guard against this and have users of the collections avoid the pit of failure members such as Count should not appear on mutable thread-safe collections.

Instead we'll design API's around the functionality of this structure. The input end of the structure is straight forward. All writers want is to add data.

The output end is more interesting. The end goal is to read output from the pipe but how to deal with cases where there is no data. Some programs will want to check for data, others block until data is available. We can define three methods to satisfy most scenarios

WaitForOutput - void method which blocks until input is available
GetNextOutput - Blocks until input is available and returns it
TryGetOutput - Returns immediately. If output is available it will be returned.

    public class PipeSingleReader<T> : IDisposable {
        private readonly ThreadAffinity m_affinity = new ThreadAffinity();
        private readonly Queue<T> m_queue = new Queue<T>();
        private readonly AutoResetEvent m_event = new AutoResetEvent(false);
        private readonly object m_lock = new object();
        private bool m_inputClosed;

        public PipeSingleReader() {
        }

        #region Dispose
        public void Dispose() {
            Dispose(true);
            GC.SuppressFinalize(this);
        }
        ~PipeSingleReader() {
            Dispose(false);
        }
        private void Dispose(bool disposing) {
            if (disposing) {
                m_event.Close();
            }
        }
        #endregion
        public void WaitForOutput() {
            m_affinity.Check();
            do {
                lock (m_lock) {
                    if (m_queue.Count > 0) {
                        return;
                    }
                }
                m_event.WaitOne();
            } while (true);
        }
        public T GetNextOutput() {
            m_affinity.Check();
            T data;
            while (!TryGetOutput(out data))
            {
                m_event.WaitOne();
            } 
            return data;
        }
        public bool TryGetOutput(out T value) {
            m_affinity.Check();
            lock (m_lock) {
                if (m_queue.Count == 0) {
                    value = default(T);
                    return false;
                }

                value = m_queue.Dequeue();
                return true;
            }
        }
        public void AddInput(T value) {
            lock (m_lock) {
                if (m_inputClosed) {
                    throw new InvalidOperationException("Input end of pipe is closed");
                }
                m_queue.Enqueue(value);
            }
            m_event.Set();
        }
        public void CloseInput() {
            lock (m_lock) {
                m_inputClosed = true;
            }
        }
    }

At a quick glance it may seem odd in GetNextOutput that I loop around m_event being set and TryGetOutput. Why loop? Shouldn't a single check for the settness of m_event be enough? In this case no. The reason why is TryGetOutput will remove output from the queue without resetting the settness of m_event. Thus m_event can be set without actually having any data in m_queue. In general the implementation must treat m_event being set as the possibility of data rather than a guarantee.

This implementation uses locks to synchronize access to the data. In general I'm a fan of avoiding locks when possible since it's very easy to miss trivial cases. Next time we'll look at an implementation of PipeSingleReader which avoids the use of locks.

SynchronizationContext and Higher Order Functions

Jared Parsons — Sun, 24 Feb 2008 12:42:48 GMT

It's often useful to ensure that actions occur on specific threads, in particular event handlers. Take Windows Forms for instance where all operations on a Control must occur on the thread it was created on. Typically this is not a problem since WinForms respond to events such as Click, Move, etc... These events are sourced from the same thread so it's not an issue.

But there are cases where events are sourced from a separate thread and we need to Marshal it back onto the Control thread. One good example of this is FileSystemWatcher. If a SynchronizationObject is not provided it will raise the event on an unspecified thread. This event cannot directly touch a Control or an "Illegal cross thread call exception" will occur. Many examples use ISynchronizedInvoke to marshal the code back. There are a couple of downsides to this approach including

ISynchronizedInvoke on Controls won't work until the Handle is created or after it's destroyed. So if the event fires in either of these cases an unhandled exception will occur and typically crash the process.
Can't use an anonymous lambda because ISynchronizedInvoke is not typed to a specific delegate
Code is easy to get subtly wrong

Here is an example implementation.

        private void OnFileChanged(object sender, FileSystemEventArgs e)
        {
            if( this.InvokeRequired )
            {
                // If the handle is not created this will throw
                Invoke((MethodInvoker)(() => OnFileChanged(sender, e)));
                return;
            }

            textBox2.Text = String.Format("{0} {1}", e.ChangeType, e.Name);
        }

It would be easier if we could bind a delegate to a particular thread in such way that calls automatically marshal to the appropriate thread. Imagine for instance if we could type the following in such a way that all invocations of "del" below would automatically marshal to the thread for the Control. We could then freely pass this to any event source and not have to worry about what thread the event is raised on.

var del = SynchronizationContext.Current.BindDelegateAsPost(new FileSystemEventHandler(OnFileChanged));

Instead of ISynchronizedInvoke we'll use SynchronizationContext. IMHO this is a better approach for this type of work. It has the same functionality as ISynchronizedInvoke and helps with a few of the quirks. The Windows Forms Application Model (and if memory serves WPF) insert a SynchronizationContext for every thread running a WinForm application. It greatly reduces the chance your code will run into problem #1 above because the timespan for when it can be used to Marshal between threads is not dependent upon the internal workings of a particular Control. Instead it's tied to the lifetime of the Thread[1].

The basic strategy we'll take is to create a new delegate which wraps the original delegate. This will Marshal the call onto the appropriate thread and then call the original delegate. SynchronizationContext has two methods to Marshal calls between threads; Post and Send.

Creating a delegate instance on the fly is not straight forward. Unless we code all permutations of delegate signatures into a class we cannot use the Delegate.Create API because we cannot provide a method with the matching signature. Instead we need to go through Reflection.Emit. This allows us to build a method on the fly to match the delegate signature. In addition we can generate the IL to route the code through Post/Send before calling the delegate.

First up are extension methods for SynchronizationContext that call into a helper class.

public static T BindDelegateAsPost<T>(this SynchronizationContext context, T del)
{
    return DelegateFactory.CreateAsPost(context, del);
}

public static T BindDelegateAsSend<T>(this SynchronizationContext context, T del)
{
    return DelegateFactory.CreateAsSend(context, del);
}

Next is a class which injects the Send/Post call. We need this as a storage mechanism for holding the context and delegate. Essentially this is a hand generate closure.

    private class DelegateData
    {
        private SynchronizationContext m_context;
        private Delegate m_target;

        internal DelegateData(SynchronizationContext context, Delegate target)
        {
            m_target = target;
            m_context = context;
        }

        public void Send(object[] args)
        {
            m_context.Send(() => m_target.DynamicInvoke(args));
        }

        public void Post(object[] args)
        {
            m_context.Post(() => m_target.DynamicInvoke(args));
        }
    }

Now comes the actual delegate generation. The dynamic method will be bound to an instance of the DelegateData class. As such we must add an additional parameter to the delegate of type DelegateData in position 0. The rest of the method creates an object array with length equal to the number of parameters in the delegate. Each of the arguments are added to this array. Then it will call Post/Send in DelegateData passing the arguments along.

    private static T Create<T>(SynchronizationContext context, T target, string name)
    {
        Delegate del = (Delegate)(object)target;
        if (del.Method.ReturnType != typeof(void))
        {
            throw new ArgumentException("Only void return types currently supported");
        }

        var paramList = new List<Type>();
        paramList.Add(typeof(DelegateData));
        paramList.AddRange(del.Method.GetParameters().Project((x) => x.ParameterType));
        var method = new DynamicMethod(
            "AMethodName",
            del.Method.ReturnType,
            paramList.ToArray(),
            typeof(DelegateData));
        var gen = method.GetILGenerator();
        var localInfo = gen.DeclareLocal(typeof(object[]));
        gen.Emit(OpCodes.Ldc_I4, paramList.Count - 1);
        gen.Emit(OpCodes.Newarr, typeof(object));
        gen.Emit(OpCodes.Stloc, localInfo.LocalIndex);
        for (int i = 1; i < paramList.Count; ++i)
        {
            gen.Emit(OpCodes.Ldloc, localInfo.LocalIndex);
            gen.Emit(OpCodes.Ldc_I4, i - 1);
            gen.Emit(OpCodes.Ldarg, i);
            if (paramList[i].IsValueType)
            {
                gen.Emit(OpCodes.Box);
            }
            gen.Emit(OpCodes.Stelem_Ref);
        }

        gen.Emit(OpCodes.Ldarg_0);
        gen.Emit(OpCodes.Ldloc, localInfo.LocalIndex);
        gen.EmitCall(OpCodes.Call, typeof(DelegateData).GetMethod(name, BindingFlags.Instance | BindingFlags.Public), null);
        gen.Emit(OpCodes.Ret);
        return (T)(object)method.CreateDelegate(typeof(T), new DelegateData(context, del));
    }

    internal static T CreateAsSend<T>(SynchronizationContext context, T target)
    {
        return Create(context, target, "Send");
    }

    internal static T CreateAsPost<T>(SynchronizationContext context, T target)
    {
        return Create(context, target, "Post");
    }

The resulting delegate is now of the same type as the original delegate and invocations will occur on the targeted thread.

[1] Granted if you try to use a SynchronizationContext to Marshal between threads after the target thread has finished you will still get an error.

Thread Affinity

Jared Parsons — Sat, 23 Feb 2008 04:55:20 GMT

Part of creating a multithreading program is understanding which threads objects live on. Seems simple enough and typically is. However it's nice to insert guarantees to match the design.

One type of threading model is for objects or subsets of methods in an object to have an affinity to a particular thread. Meaning that those methods can only be validly called on a particular thread and no other. I create a small helper object to help validate this affinity.

Simple, but gets the job done.

    [Immutable]
    public sealed class ThreadAffinity
    {
        private readonly int m_threadId;

        public ThreadAffinity()
        {
            m_threadId = Thread.CurrentThread.ManagedThreadId;
        }

        public void Check()
        {
            if (Thread.CurrentThread.ManagedThreadId != m_threadId)
            {
                var msg = String.Format(
                    "Call to class with affinity to thread {0} detected from thread {1}.",
                    m_threadId,
                    Thread.CurrentThread.ManagedThreadId);
                throw new InvalidOperationException(msg);
            }
        }
    }

Building a Future which returns no Value

Jared Parsons — Tue, 19 Feb 2008 00:15:52 GMT

In addition to Future<T> there is also the concept of Futures that don't return any values. Instead the perform the operation and return. Because there is no additional data to pass between the threads building an Empty Future is fairly straight forward.

The biggest decision is how to declare it. EmptyFuture exposes no new accessible methods or properties above what Future already exposes. If a class doesn't provide any additional behavior is there any reason to expose it? In this case I think not. Therefore it will be declared as a private inner class of Future.

Yes this makes it impossible to directly create from a user perspective. You could also make a good argument that creation is new behavior and therefore EmptyFuture should be exposed. However for Future's, as with other generic classes, I prefer static factory methods for creation. It allows the user to take advantage of type inference as much as possible.

        private class EmptyFuture : Future
        {
            private Action m_action;

            internal EmptyFuture(Action action)
            {
                m_action = action;
            }

            protected override void RunCore()
            {
                m_action();
            }

        }

Next we'll go over the creation of the factory methods to create this and Future<T>.

Building Future

Jared Parsons — Wed, 13 Feb 2008 18:16:43 GMT

The last post dealt with building the base Future class. Now we'll build the child class used to run Func<TResult>'s.

The basic implementation is straight forward. The class will run a delegate typed to Func<TResult> in the override of RunCore. The trickiest part is how to store the value. The value is set on one thread and read off of another.

When a value is read and written on multiple threads there are a couple of options for synchronization between threads. One of them is to use the volatile keyword for the data. This forces the CLR to read the value from memory every time and prevents caching issues between threads. Unfortunately volatile cannot be applied to an unbounded generic.

To get around this I've declared the value to be of type object. Whenever the value is accessed by the user of Future<T> a cast is applied to the appropriate type. This incurs boxing overhead but it's minimal and in the typical case will be limited to one box and unbox per value type.

In addition Future<T> adds one new method; Wait; It's a combination of calling WaitEmpty followed by returning the value.

In a perfect world WaitEmpty in Future would really be called Wait and be virtual. Future<T> would override the method and alter the return type to be T. Unfortunately C#/VB don't support covariant return types on virtual method overrides so it's not possible. Truthfully I don't know if this is a C#/VB limitation or a CLR one.

    public class Future<T> : Future
    {
        private Func<T> m_function;
        private volatile object m_value;

        public T Value
        {
            get { return Wait(); }
        }

        public Future(Func<T> function)
        {
            m_function = function;
        }

        public T Wait()
        {
            base.WaitEmpty();
            return (T)m_value;
        }

        protected override void RunCore()
        {
            m_value = m_function();
        }

    }

Next time I'll go over the implementation of Futures which return no values.

Building the Base Future

Jared Parsons — Tue, 12 Feb 2008 18:48:53 GMT

In the end there are two basic types of Future implementations you can use.

Futures which return no values
Futures which return a value

The rest of the behavior and shape of the Future is the same and screams for a pattern of sorts. I've found the best way to implement this behavior is through an inheritance pattern. The base class is name of course Future. It's purpose is to provide a common way in which to schedule the invocation of a delegate, handle exceptions, wait for the completion of such delegate and enforce certain contracts such as not running the future more than once.

At the core it only needs a few members.

ActiveOperation m_operation which is used to implement the waiting portion
int m_run used to ensure a future is not run twice
Exception m_error to record any exceptions thrown by running the delegate

        private ActiveOperation m_operation = new ActiveOperation();
        private int m_run;
        private Exception m_error;

It has one public property to determine whether or not a Future has completed. It's a proxy into m_operation

        public bool HasCompleted
        {
            get { return m_operation.HasCompleted; }
        }

By using m_operation to deal with Waiting, the majority of WaitEmpty can be proxied to m_operation as well. The only additional work needed is to deal with Exceptions.

        public void WaitEmpty()
        {
            m_operation.Wait();
            if (m_error != null)
            {
                throw new FutureException("Error occurred running future", m_error);
            }
        }

The only behavior a child class needs is a place to invoke the delegate. A single abstract method is provided to allow implement this behavior.

        protected abstract void RunCore();

Before calling this method the base Future class must make sure that all of the contracts are met. This is implemented through a wrapper method around RunCore. It is the only method that calls RunCore directly.

        private void RunWrapper()
        {
            if (0 != Interlocked.CompareExchange(ref m_run, 1, 0))
            {
                throw new InvalidOperationException("Future is already run");
            }

            try
            {
                RunCore();
            }
            catch (Exception ex)
            {
                Interlocked.Exchange(ref m_error, ex);
            }
            finally
            {
                m_operation.Completed();
            }
        }

It's very important that m_operation is signalled as completed after exception handling occurs. The setting or not setting of m_error is the only way we know if an exception occurred when waiting. If we do this in the opposite order it's possible for WaitEmpty() to complete before the exception is set and hence miss the error.

Lastly is the code to actually run the Future. There are two ways to run the Future (more can be added).

Asynchronously through the ThreadPool. This is the more common case.
Synchronously on the same thread. This will mainly be used to implement such operations as methods in Active Objects.

        public void Run()
        {
            RunWrapper();
        }

        public void RunInThreadPool()
        {
            ThreadPool.QueueUserWorkItem((x) => RunWrapper());
        }

This leaves us with a base class implementation for Future's. Next we'll implement Future which return values.

Push Enumerators

Jared Parsons — Tue, 05 Feb 2008 20:26:13 GMT

If you read Jon Skeet's blog you'll notice he's been playing around lately with "push" style enumerators. Push enumerators are the concept of "we'll tell you when we're ready". This is different from IEnumerator<T> which is more of a pull; "ask me if I have more data model".

His latest idea is an interface based approach called IDataProducer<T>. The full post can be found here. http://msmvps.com/blogs/jon.skeet/archive/2007/11/29/group-pipelining-returns-new-and-improved-design.aspx.

After reading the post I decided to try and bridge the gap between the worlds of Push/Pull models by defining an enumerator over an IDataProducer<T>. Integration with the existing paradigms is extremely important for the adoption of a new technology. Eventually you'll eventually want to pass your asynchronous enumerator off to an API requiring IEnumerable<T>.

Since we are implementing a pull model it will revolve around asking for more data and blocking until such data is available. To implement the wait functionality I've used an AutoResetEvent to provide signaling between threads. One of the tricky aspects is that IDataProducer<T> can live on any thread and hence any thread can be raising the various events.

Below is my first attempt at getting it to work. It has a couple of defects.

I don't dispose of the AutoResetEvent. Reason being that it's possible for a consumer of IEnumerable<T> to only consume part of the data. If they only consume half of the data and then dispose of the EnumerableDataProducer and another event comes I'll be accessing a disposed WaitHandle. You can work around this but I wanted to keep it simpler for now.
It's not Reset-able. The generated IEnumerator<T> throws a NotSupportedException anyway but we'd have to do a bit of work to make this resetable.

    public class EnumerableDataProducer<T> : IEnumerable<T>
    {
        private object m_lock = new object();
        private bool m_finished;
        private AutoResetEvent m_event = new AutoResetEvent(false);
        private Queue<T> m_queue = new Queue<T>();

        public EnumerableDataProducer(IDataProducer<T> producer)
        {
            producer.DataProduced += new Action<T>(OnDataProdecuded);
            producer.EndOfData += new Action(OnEndOfData);
        }

        private void OnDataProdecuded(T obj)
        {
            lock (m_lock)
            {
                m_queue.Enqueue(obj);
            }

            m_event.Set();
        }

        private void OnEndOfData()
        {
            lock (m_lock)
            {
                m_finished = true;
            }
            m_event.Set();
        }

        #region IEnumerable<T> Members

        public IEnumerator<T> GetEnumerator()
        {
            while (true)
            {
                bool needWait = false;
                lock (m_lock)
                {
                    if (m_finished)
                    {
                        break;
                    }

                    if (m_queue.Count > 0)
                    {
                        yield return m_queue.Dequeue();
                    }
                    else
                    {
                        needWait = true;
                    }
                }

                if (needWait)
                {
                    m_event.WaitOne();
                }
            }
        }

        #endregion

        #region IEnumerable Members

        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
        {
            return GetEnumerator();
        }

        #endregion
    }

The first part of building a Future is ... Waiting

Jared Parsons — Mon, 04 Feb 2008 15:04:41 GMT

Future's are a great abstraction for asynchronous programming. One of the items making them so good is the easy manner in which you can declare one and wait for it to finish. The idea is to allow for many futures to be declared with as little overhead as possible. In order to do so you need to define an efficient way of waiting.

Normally when you need to wait on operations between two threads to complete you use a Thread.Join call or a form of a WaitHandle. I tend to prefer a ManualResetEvent.

Unfortunately a ManualResetEvent is not free and under the hood will allocate a kernel handle. There are a lot of handles to go around and while unlikely that a program purely using futures will allocate too many handles, you may be running with other code that is handle hungry.

In addition several cases do not need a handle. For instance in the below code, if the commented out section takes longer than the future to complete then why do you even need wait handle of any sort?

           var d = Future.Create(() => CallASimpleFunction);
            // ...
            d.Wait();

Therefore the first step is to define an efficient waiting mechanism. I call it an ActiveOperation. It provides 3 basic methods; HasCompleted, Completed and Wait. It optimizes for trying to not created a WaitEvent unless actually necessary. It has two member variables. An int for completion check and a ManualResetEvent to be used for shared waiting when necessary. Notice that m_hasCompleted is not volatile, instead all writes use a Interlocked operation to ensure it is propagated between threads.

        private int m_hasCompleted;
        private ManualResetEvent m_waitEvent;

HasCompleted is straightforward.

        public bool HasCompleted
        {
            get { return m_hasCompleted == 1; }
        }

Completed is a little bit trickier. It has to deal with a couple of cases.

Another thread already called Completed.
Completed called before another thread calls Wait.
Completed called while or after another thread calls Wait.

        public void Completed()
        {
            if (0 == Interlocked.CompareExchange(ref m_hasCompleted, 1, 0))
            {
                ManualResetEvent mre = m_waitEvent;
                if (mre != null)
                {
                    try
                    {
                        mre.Set();
                    }
                    catch (ObjectDisposedException)
                    {
                        // If another thread is in Wait at the same time and sees the completed flag
                        // it may dispose of the shared event.  In this case there is no need to signal
                        // just return.
                    }
                }
            }
        }

Wait is the trickiest one. It has the following cases to deal with

HasCompleted already set
Second or later thread to call Wait and needs to wait on m_waitEvent
While attempting to create m_waitEvent, another thread in Wait finishes first.
Thread successfully creates and owns the shared m_waitEvent variable before Completed() is called.
During the creation of m_waitEvent, another thread calls Completed() in which case there is no guarantee that m_waitEvent will be signaled.

        public void Wait()
        {
            // Case 1
            if (HasCompleted)
            {
                return;
            }

            // Case 2
            ManualResetEvent sharedEvent = m_waitEvent;
            if (sharedEvent != null)
            {
                WaitOnEvent(sharedEvent);
            }

            ManualResetEvent created = null;
            try
            {
                created = new ManualResetEvent(false);
                sharedEvent = Interlocked.CompareExchange(ref m_waitEvent, created, null);
                if (null != sharedEvent)
                {
                    // Case 3.  Another thread got here first and it's created is now the shared event.  Wait
                    // on that event
                    WaitOnEvent(sharedEvent);
                }
                else if (HasCompleted)
                {
                    // Case 5. In between the time we checked for completion and created the event a completion
                    // occurred.  Returning will dispose of m_waitEvent and force other threads Wait to complete 
                    return;
                }
                else
                {
                    // Case 4. 
                    WaitOnEvent(created);
                }
            }
            finally
            {
                if (created != null )
                {
                    if ( sharedEvent == null)
                    {
                        Interlocked.Exchange(ref m_waitEvent, null);
                    }

                    created.Close();
                }
            }
        }

        private void WaitOnEvent(ManualResetEvent mre)
        {
            try
            {
                mre.WaitOne();
            }
            catch (ObjectDisposedException)
            {

            }
        }

Now you have one of the basic building blocks of Futures.

Active Objects and Futures

Jared Parsons — Tue, 29 Jan 2008 07:57:08 GMT

Herb Sutter gave one of my favorite and inspiring presentations. It is called "The Free Lunch is Over". The original article can be found here. My first encounter though came from his PDC presentation and highly recommend viewing that as well.

The part that interested me the most about the talk was two new threading abstractions I hadn't encountered before. Future's and ActiveObjects. One of the basic premise is that concurrency should be grep`able and somewhat declarative. The act of calling a method on a background thread and later waiting for it to complete should be simple, not complicated.

Asynchronous programming is one of my favorite aspects of computing. What interests me the most is how asynchronous programming can be useful for UI. My biggest pet peeve is when UI hangs because an operation call, or network operation takes too long. Why not make multi-threading easy and give users a way to cancel out of these operations??? Or start loading on the background thread instead of waiting for the user to perform a specific. Hopefully over the next month or so I'll lay out some utilities and classes building on Futures and Active Objects that will do precisely this.

Future's are actions where work can be done now, but the result is not needed until a future time. Work occurs on a separate thread and the results can be easily joined once work is complete.

            var f = Future.Create(() => LongCalculation());
            // ...
            var result = f.Wait();

Future's are now exposed via the Parallel Extension team's work. You can download the CTP off of their web site and get to work.

ActiveObjects are objects which only expose Asynchronous functions where the return value is exposed as a Future. So instead of

        string GetName()

You would have

        Future<string> GetName()

An ActiveObject essentially lives on or owns a thread. All operations are queued up and processed one at a time. Since only one action at a time can be executing the object internals don't have to use locks or consider many types of race conditions. In fact if your return types are immutable a great many threading concerns go out the window. Yet all of the calls are inherently asynchronous so callers can get the result only when they are needed. The best of both worlds.

Both of these provide significant advantages over the "lock before use" patterns. In my experience I find these to be hard to maintain and lead to difficult to track down bugs. I can't tell you how many times I've gone through someone else's code, or even my own, and wondered ...

Did they forget to lock here or is this an optimization?
Is a join needed here or can these terminate at separate times?
OK I need to touch that variable, can it be accessed in multiple threads or is it safe?

Futures/Active Objects on the other hand are a bit more declarative and straight forward to understand than plain old locks. They allow you to do away with many uses of plain old locking. Don't confuse this with me saying they are a cure all for threading. They're not. But in my experiences I've found them to be a significant upgrade.

Over the next month or so I'll be laying out the design for a basic ActiveObject implementation. We will likely have to deviate off of the Parrallel Extension work to get certain behaviors but the concepts map well.

Tuples Part 7: Mutable Tuples

Jared Parsons — Wed, 23 Jan 2008 10:56:19 GMT

Part 6 left us with comparable tuples. At this point, the Tuple class is functionally complete. There will be a little more done with the debugability and overall fit into larger projects. But otherwise it is sound.

Now the focus shifts to generating mutable tuples. Immutability is nice for threading, memoization, etc ... However it's not always practical to use immutable objects. Often an algorithm does not benefit from immutability and lends itself to a more mutable type.

Mutable tuples behave like a Tuple in every other way except that they're ... mutable. This includes implementing interfaces as well as inheritance structure.

public sealed class MutableTuple<TA, TB> : 
    ITuple<TA, TB>, 
    IEquatable<MutableTuple<TA, TB>>, 
    IComparable<MutableTuple<TA, TB>>, 
    IComparable
{

As such the script already used will be sufficient to generate mutable classes in addition to the ones its already doing. The majority of the code difference is just in the naming of the classes. The only functional differences exist in the properties and indexer. Both of these add a setter method. Below is the modified code for generating the property and indexer.

function script:Gen-TupleAccess
{
    param ( [int] $count = $(throw "Need a count"), [bool]$mutable )
    "public int Count { get { return $count; } }"
    "public object this[int index] { get { switch (index){ "
    0..($count-1) | %{ "case $($_): return m_$($lowerList[$_]);" }
    "default: throw new InvalidOperationException(""Bad Index"");"
    "} }"

    if ( $mutable )
    {
        "set { switch (index) {"
        0..($count-1) | %{ "case $($_): m_$($lowerList[$_]) = (T$($upperList[$_]))value; break;" }
        "default: throw new InvalidOperationException(""Bad Index"");"
        "} } "
    }
    "}"
}

function script:Gen-Property
{
param ( [int] $index = $(throw "Need an index"), [bool]$mutable = $false)

    if (-not $mutable )
    {
@"
    private readonly T{0} m_{1};
    public T{0} {0} {{ get {{ return m_{1}; }} }}

"@ -f $upperList[$index],$lowerList[$index]
    }
    else
    {
@"
    private T{0} m_{1};
    public T{0} {0} {{ get {{ return m_{1}; }} set {{ m_{1} = value; }} }}

"@ -f $upperList[$index],$lowerList[$index]
}
}

Now creating a mutable tuple is the same as the immutable tuple with just a name tweak.

            var t1 = MutableTuple.Create("foo", 42);
            t1.A = "again";