jaredpar's WebLog : Futures

Factory Methods for Futures

Jared Parsons — Sun, 24 Feb 2008 04:35:06 GMT

Like most generic classes, I prefer to create Future instances through static factory methods which allows me to take maximum advantage of type inference.

In addition to the 2 straight forward declaration of Func<T> and Action, the methods will include overloads which take in Func<T> with varying numbers of arguments. The overloads will cury the arguments and create a Func<T> taking no arguments. This is a great convenience to the user and takes little extra code to implement.

In addition because we don't expose the EmptyFuture class directly we need to provide a factory method to create it in a non-run state. Otherwise we are forcing the EmptyFuture to always be created and run in the ThreadPool.

        public static Future Create(Action action)
        {
            var f = new EmptyFuture(action);
            f.RunInThreadPool();
            return f;
        }

        public static Future Create<TArg1>(Action<TArg1> action, TArg1 arg1)
        {
            return Create(() => action(arg1));
        }

        public static Future<T> Create<T>(Func<T> func)
        {
            var f = new Future<T>(func);
            f.RunInThreadPool();
            return f;
        }

        public static Future<TReturn> Create<TArg1,TReturn>(Func<TArg1, TReturn> func, TArg1 arg1)
        {
            return Create(() => func(arg1));
        }

        public static Future<T> CreateNoRun<T>(Func<T> func)
        {
            return new Future<T>(func);
        }

        public static Future CreateNoRun(Action action)
        {
            return new EmptyFuture(action);
        }

Building a Future which returns no Value

Jared Parsons — Tue, 19 Feb 2008 00:15:52 GMT

In addition to Future<T> there is also the concept of Futures that don't return any values. Instead the perform the operation and return. Because there is no additional data to pass between the threads building an Empty Future is fairly straight forward.

The biggest decision is how to declare it. EmptyFuture exposes no new accessible methods or properties above what Future already exposes. If a class doesn't provide any additional behavior is there any reason to expose it? In this case I think not. Therefore it will be declared as a private inner class of Future.

Yes this makes it impossible to directly create from a user perspective. You could also make a good argument that creation is new behavior and therefore EmptyFuture should be exposed. However for Future's, as with other generic classes, I prefer static factory methods for creation. It allows the user to take advantage of type inference as much as possible.

        private class EmptyFuture : Future
        {
            private Action m_action;

            internal EmptyFuture(Action action)
            {
                m_action = action;
            }

            protected override void RunCore()
            {
                m_action();
            }

        }

Next we'll go over the creation of the factory methods to create this and Future<T>.

Building Future

Jared Parsons — Wed, 13 Feb 2008 18:16:43 GMT

The last post dealt with building the base Future class. Now we'll build the child class used to run Func<TResult>'s.

The basic implementation is straight forward. The class will run a delegate typed to Func<TResult> in the override of RunCore. The trickiest part is how to store the value. The value is set on one thread and read off of another.

When a value is read and written on multiple threads there are a couple of options for synchronization between threads. One of them is to use the volatile keyword for the data. This forces the CLR to read the value from memory every time and prevents caching issues between threads. Unfortunately volatile cannot be applied to an unbounded generic.

To get around this I've declared the value to be of type object. Whenever the value is accessed by the user of Future<T> a cast is applied to the appropriate type. This incurs boxing overhead but it's minimal and in the typical case will be limited to one box and unbox per value type.

In addition Future<T> adds one new method; Wait; It's a combination of calling WaitEmpty followed by returning the value.

In a perfect world WaitEmpty in Future would really be called Wait and be virtual. Future<T> would override the method and alter the return type to be T. Unfortunately C#/VB don't support covariant return types on virtual method overrides so it's not possible. Truthfully I don't know if this is a C#/VB limitation or a CLR one.

    public class Future<T> : Future
    {
        private Func<T> m_function;
        private volatile object m_value;

        public T Value
        {
            get { return Wait(); }
        }

        public Future(Func<T> function)
        {
            m_function = function;
        }

        public T Wait()
        {
            base.WaitEmpty();
            return (T)m_value;
        }

        protected override void RunCore()
        {
            m_value = m_function();
        }

    }

Next time I'll go over the implementation of Futures which return no values.

Building the Base Future

Jared Parsons — Tue, 12 Feb 2008 18:48:53 GMT

In the end there are two basic types of Future implementations you can use.

Futures which return no values
Futures which return a value

The rest of the behavior and shape of the Future is the same and screams for a pattern of sorts. I've found the best way to implement this behavior is through an inheritance pattern. The base class is name of course Future. It's purpose is to provide a common way in which to schedule the invocation of a delegate, handle exceptions, wait for the completion of such delegate and enforce certain contracts such as not running the future more than once.

At the core it only needs a few members.

ActiveOperation m_operation which is used to implement the waiting portion
int m_run used to ensure a future is not run twice
Exception m_error to record any exceptions thrown by running the delegate

        private ActiveOperation m_operation = new ActiveOperation();
        private int m_run;
        private Exception m_error;

It has one public property to determine whether or not a Future has completed. It's a proxy into m_operation

        public bool HasCompleted
        {
            get { return m_operation.HasCompleted; }
        }

By using m_operation to deal with Waiting, the majority of WaitEmpty can be proxied to m_operation as well. The only additional work needed is to deal with Exceptions.

        public void WaitEmpty()
        {
            m_operation.Wait();
            if (m_error != null)
            {
                throw new FutureException("Error occurred running future", m_error);
            }
        }

The only behavior a child class needs is a place to invoke the delegate. A single abstract method is provided to allow implement this behavior.

        protected abstract void RunCore();

Before calling this method the base Future class must make sure that all of the contracts are met. This is implemented through a wrapper method around RunCore. It is the only method that calls RunCore directly.

        private void RunWrapper()
        {
            if (0 != Interlocked.CompareExchange(ref m_run, 1, 0))
            {
                throw new InvalidOperationException("Future is already run");
            }

            try
            {
                RunCore();
            }
            catch (Exception ex)
            {
                Interlocked.Exchange(ref m_error, ex);
            }
            finally
            {
                m_operation.Completed();
            }
        }

It's very important that m_operation is signalled as completed after exception handling occurs. The setting or not setting of m_error is the only way we know if an exception occurred when waiting. If we do this in the opposite order it's possible for WaitEmpty() to complete before the exception is set and hence miss the error.

Lastly is the code to actually run the Future. There are two ways to run the Future (more can be added).

Asynchronously through the ThreadPool. This is the more common case.
Synchronously on the same thread. This will mainly be used to implement such operations as methods in Active Objects.

        public void Run()
        {
            RunWrapper();
        }

        public void RunInThreadPool()
        {
            ThreadPool.QueueUserWorkItem((x) => RunWrapper());
        }

This leaves us with a base class implementation for Future's. Next we'll implement Future which return values.

Dealing with Exceptions in a Future

Jared Parsons — Mon, 11 Feb 2008 19:36:22 GMT

Besides waiting, the another important issue when dealing with Futures is how to deal with exceptions thrown by the user specified code.

Option 1: Ignore the Exception

Don't take any actions in the future code and force users to write exception free code. IMHO this is not the best way to approach the problem. The code will be running in the thread pool and unhandled exceptions in the thread pool result in the taking down of an appdomain/process. In addition Futures are designed to be simple. Adding a try/catch around every lambda is not practical and/or readable.

Option 2: Catch and Swallow

Catch the exception on the background thread and swallow it. Silently failing is in many cases worse than actually crashing. Behavior will become flaky and the user/developer won't have any indication there is an error.

Option 3: Re-throw the Exception when Wait is called

Catch and save the exception when it occurs on the background thread. Then when Wait() is called on a Future re-throw the exception. This makes exception handled deterministic.

It's also very similar to the exception handling semantics of calling a method. The only difference is that users must handle the exception at the point of method completion vs invocation. For synchronous methods this is just the same point.

The big downside to this approach is the stack trace information is lost from the exception. Re-throwing will instead add the stack trace at the point of the re-throw. Not having stack trace information makes it very difficult to actually track down the source of an error.

Option 4: Re-throw a new Exception when Wait is called

This is very similar to Option #3. The only difference is when the user calls Wait, throw a new exception and make the original exception an inner exception of the new one. We'll call this exception FutureException. This has the advantages of option 3 and in addition will preserve the stack trace information from the original exception.

There is a downside to this approach though. Users can no longer have different catch blocks to handle the different types of exceptions that can be thrown.

            try
            {
                Future.Create(() => SomeOperation());
            }
            catch (IOException ex)
            {
                // ...
            }
            catch (InvalidOperationException ex)
            {
                // ...
            }

Instead the user can only catch a Future exception and examine the inner result to take corrective action.

            try
            {
                Future.Create(() => SomeOperation());
            }
            catch (FutureException ex)
            {
                var type = ex.InnerException.GetType();
                if (type == typeof(IOException))
                {
                    // ...
                }
                else if (type == typeof(InvalidOperationException))
                {
                    // ...
                }
            }

This doesn't actually limit any functionality but users may find the syntax uncomfortable. VB users can still do exception filtering but this is not at option for C# users.

        Try
            Future.Create(Function() SomeOperation())
        Catch ex As Exception When ex.InnerException.GetType() Is GetType(IOException)

        End Try

The FutureException class is straight forward. A simple implementation of the exception snippet will do the trick.

    [global::System.Serializable]
    public class FutureException : Exception
    {


        public FutureException() { }
        public FutureException(string message) : base(message) { }
        public FutureException(string message, Exception inner) : base(message, inner) { }
        protected FutureException(
          System.Runtime.Serialization.SerializationInfo info,
          System.Runtime.Serialization.StreamingContext context)
            : base(info, context) { }
    }

The first part of building a Future is ... Waiting

Jared Parsons — Mon, 04 Feb 2008 15:04:41 GMT

Future's are a great abstraction for asynchronous programming. One of the items making them so good is the easy manner in which you can declare one and wait for it to finish. The idea is to allow for many futures to be declared with as little overhead as possible. In order to do so you need to define an efficient way of waiting.

Normally when you need to wait on operations between two threads to complete you use a Thread.Join call or a form of a WaitHandle. I tend to prefer a ManualResetEvent.

Unfortunately a ManualResetEvent is not free and under the hood will allocate a kernel handle. There are a lot of handles to go around and while unlikely that a program purely using futures will allocate too many handles, you may be running with other code that is handle hungry.

In addition several cases do not need a handle. For instance in the below code, if the commented out section takes longer than the future to complete then why do you even need wait handle of any sort?

           var d = Future.Create(() => CallASimpleFunction);
            // ...
            d.Wait();

Therefore the first step is to define an efficient waiting mechanism. I call it an ActiveOperation. It provides 3 basic methods; HasCompleted, Completed and Wait. It optimizes for trying to not created a WaitEvent unless actually necessary. It has two member variables. An int for completion check and a ManualResetEvent to be used for shared waiting when necessary. Notice that m_hasCompleted is not volatile, instead all writes use a Interlocked operation to ensure it is propagated between threads.

        private int m_hasCompleted;
        private ManualResetEvent m_waitEvent;

HasCompleted is straightforward.

        public bool HasCompleted
        {
            get { return m_hasCompleted == 1; }
        }

Completed is a little bit trickier. It has to deal with a couple of cases.

Another thread already called Completed.
Completed called before another thread calls Wait.
Completed called while or after another thread calls Wait.

        public void Completed()
        {
            if (0 == Interlocked.CompareExchange(ref m_hasCompleted, 1, 0))
            {
                ManualResetEvent mre = m_waitEvent;
                if (mre != null)
                {
                    try
                    {
                        mre.Set();
                    }
                    catch (ObjectDisposedException)
                    {
                        // If another thread is in Wait at the same time and sees the completed flag
                        // it may dispose of the shared event.  In this case there is no need to signal
                        // just return.
                    }
                }
            }
        }

Wait is the trickiest one. It has the following cases to deal with

HasCompleted already set
Second or later thread to call Wait and needs to wait on m_waitEvent
While attempting to create m_waitEvent, another thread in Wait finishes first.
Thread successfully creates and owns the shared m_waitEvent variable before Completed() is called.
During the creation of m_waitEvent, another thread calls Completed() in which case there is no guarantee that m_waitEvent will be signaled.

        public void Wait()
        {
            // Case 1
            if (HasCompleted)
            {
                return;
            }

            // Case 2
            ManualResetEvent sharedEvent = m_waitEvent;
            if (sharedEvent != null)
            {
                WaitOnEvent(sharedEvent);
            }

            ManualResetEvent created = null;
            try
            {
                created = new ManualResetEvent(false);
                sharedEvent = Interlocked.CompareExchange(ref m_waitEvent, created, null);
                if (null != sharedEvent)
                {
                    // Case 3.  Another thread got here first and it's created is now the shared event.  Wait
                    // on that event
                    WaitOnEvent(sharedEvent);
                }
                else if (HasCompleted)
                {
                    // Case 5. In between the time we checked for completion and created the event a completion
                    // occurred.  Returning will dispose of m_waitEvent and force other threads Wait to complete 
                    return;
                }
                else
                {
                    // Case 4. 
                    WaitOnEvent(created);
                }
            }
            finally
            {
                if (created != null )
                {
                    if ( sharedEvent == null)
                    {
                        Interlocked.Exchange(ref m_waitEvent, null);
                    }

                    created.Close();
                }
            }
        }

        private void WaitOnEvent(ManualResetEvent mre)
        {
            try
            {
                mre.WaitOne();
            }
            catch (ObjectDisposedException)
            {

            }
        }

Now you have one of the basic building blocks of Futures.

Active Objects and Futures

Jared Parsons — Tue, 29 Jan 2008 07:57:08 GMT

Herb Sutter gave one of my favorite and inspiring presentations. It is called "The Free Lunch is Over". The original article can be found here. My first encounter though came from his PDC presentation and highly recommend viewing that as well.

The part that interested me the most about the talk was two new threading abstractions I hadn't encountered before. Future's and ActiveObjects. One of the basic premise is that concurrency should be grep`able and somewhat declarative. The act of calling a method on a background thread and later waiting for it to complete should be simple, not complicated.

Asynchronous programming is one of my favorite aspects of computing. What interests me the most is how asynchronous programming can be useful for UI. My biggest pet peeve is when UI hangs because an operation call, or network operation takes too long. Why not make multi-threading easy and give users a way to cancel out of these operations??? Or start loading on the background thread instead of waiting for the user to perform a specific. Hopefully over the next month or so I'll lay out some utilities and classes building on Futures and Active Objects that will do precisely this.

Future's are actions where work can be done now, but the result is not needed until a future time. Work occurs on a separate thread and the results can be easily joined once work is complete.

            var f = Future.Create(() => LongCalculation());
            // ...
            var result = f.Wait();

Future's are now exposed via the Parallel Extension team's work. You can download the CTP off of their web site and get to work.

ActiveObjects are objects which only expose Asynchronous functions where the return value is exposed as a Future. So instead of

        string GetName()

You would have

        Future<string> GetName()

An ActiveObject essentially lives on or owns a thread. All operations are queued up and processed one at a time. Since only one action at a time can be executing the object internals don't have to use locks or consider many types of race conditions. In fact if your return types are immutable a great many threading concerns go out the window. Yet all of the calls are inherently asynchronous so callers can get the result only when they are needed. The best of both worlds.

Both of these provide significant advantages over the "lock before use" patterns. In my experience I find these to be hard to maintain and lead to difficult to track down bugs. I can't tell you how many times I've gone through someone else's code, or even my own, and wondered ...

Did they forget to lock here or is this an optimization?
Is a join needed here or can these terminate at separate times?
OK I need to touch that variable, can it be accessed in multiple threads or is it safe?

Futures/Active Objects on the other hand are a bit more declarative and straight forward to understand than plain old locks. They allow you to do away with many uses of plain old locking. Don't confuse this with me saying they are a cure all for threading. They're not. But in my experiences I've found them to be a significant upgrade.

Over the next month or so I'll be laying out the design for a basic ActiveObject implementation. We will likely have to deviate off of the Parrallel Extension work to get certain behaviors but the concepts map well.