When “ExecuteSynchronously” doesn’t execute synchronously

When “ExecuteSynchronously” doesn’t execute synchronously

  • Comments 13

When creating a task continuation with ContinueWith, developers have the opportunity to provide a TaskContinuationOptions enum value, which could include the TaskContinuationOptions.ExecuteSynchronously flag.  ExecuteSynchronously is a request for an optimization to run the continuation task on the same thread that completed the antecedent task off of which we continued, in effect running the continuation as part of the antecedent’s transition to a final state (the default behavior for ContinueWith if this flag isn’t specified is to run the continuation asynchronously, meaning that when the antecedent task completes, the continuation task will be queued rather than executed).  I explicitly used the word “request” in the previous sentence, because while TPL strives to honor this as much as possible, there are in fact a few cases where continuations created with TaskContinuationOptions.ExecuteSynchronously will still run asynchronously.  (What follows is a discussion of TPL’s current implementation in .NET 4 and the .NET 4.5 Developer Preview.)

One condition that will force a continuation to run asynchronously is if there is a thread abort pending on the thread completing the antecedent task.  It could be dangerous for TPL to continue running arbitrary amounts of work on a thread that the CLR is trying to tear down, so when a task completes and a thread abort is pending, all ContinueWith continuations will be queued rather than executed.  Here’s a code sample to demonstrate this behavior:

using System;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        var tcs = new TaskCompletionSource<bool>();
        var cont = tcs.Task.ContinueWith(delegate 
        { 
            Console.WriteLine(Thread.CurrentThread.ManagedThreadId); 
        }, TaskContinuationOptions.ExecuteSynchronously);

        new Thread(() =>
        {
            Console.WriteLine(Thread.CurrentThread.ManagedThreadId); 
            try
            {
                Thread.CurrentThread.Abort();
            }
            finally
            {
                tcs.SetResult(true);
            }
        }).Start();

        cont.Wait();
    }
}

If you try running this code, you’ll find that it outputs two different thread IDs, one for the thread that’s completing the antecedent task and one for the thread that’s running the continuation.  If you then comment out the line that’s issuing the Abort, you’ll find that it’ll output the same ID twice, since the continuation is then running synchronously on the same thread that completed the antecedent task.

Another condition that may force a continuation to run asynchronously has to do with stack overflows.  TPL has logic used in a few places (e.g. Wait inlining, ExecuteSynchronously) to determine whether it’s too deep on the stack to safely execute.  If TPL detects that it’s too close to the thread’s guard page for comfort, such that running the continuation synchronously has a high risk of overflowing, then TPL will force the continuation to run asynchronously.  We can see this behavior with the following code example:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        var tcs = new TaskCompletionSource<bool>();
        var t = (Task)tcs.Task;

        var counts = new List<int>();
        int lastThreadId = Thread.CurrentThread.ManagedThreadId;
        int curCount = 0;

        for (int i = 0; i < 1000000; i++)
        {
            t = t.ContinueWith(delegate
            {
                int threadId = Thread.CurrentThread.ManagedThreadId;
                if (threadId == lastThreadId)
                {
                    curCount++;
                }
                else
                {
                    lastThreadId = threadId;
                    counts.Add(curCount);
                    curCount = 0;
                }
            }, TaskContinuationOptions.ExecuteSynchronously);
        }
        tcs.SetResult(true);
        t.Wait();
        Console.WriteLine((int)counts.Average());
    }
}

Here, we’re creating a chain of a million ExecuteSynchronously continuations.  With a default stack size of 1MB, and with multiple stack frames require per continuation execution, this program would certainly overflow the stack if TPL didn’t account for it.  However, if we run this, we should see that it successfully executes without crashing.  Note that this code is keeping track of how many continuations contiguously run on the same thread so that we can see approximately how many continuations end up running synchronously before one is forced to run asynchronously in order to avoid an overflow (this isn’t a 100% accurate measure, as it’s theoretically possible the continuation could be queued to run asynchronously but then end up running on the same thread, if the thread unwound the whole stack and then picked up the continuation task to run it before any other thread did, but for our purposes it’s good enough).  When I run this, I see that on average we run approximately 2200 continuations synchronously before we force one to run asynchronously.

The third current condition where ExecuteSynchronously continuations won’t run synchronously is when the target scheduler doesn’t allow it.  A TaskScheduler has the ability to say whether tasks are able to run on the current thread or not.  For example, if you created a custom StaTaskScheduler whose job was to run tasks on STA threads, it shouldn’t allow a task to run on an MTA thread, so that scheduler should be able to override the ExecuteSynchronously behavior in the case where the antecedent isn’t completing on an STA thread.  When TPL goes to run a synchronous continuation, it does so via the target TaskScheduler’s TryExecuteTaskInline method.  The scheduler can either choose to execute the task in that call, or it can choose to return false, meaning that it refused to run the task, and in the latter case, TPL will then queue the continuation to the scheduler to run asynchronously.  Here’s an example:

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
        var tcs = new TaskCompletionSource<bool>();
        var cont = tcs.Task.ContinueWith(delegate
        {
            Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
        }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, new DummyTaskScheduler());
        tcs.SetResult(true);
        cont.Wait();
    }
}

class DummyTaskScheduler : TaskScheduler
{
    protected override void QueueTask(Task task)
    {
        ThreadPool.QueueUserWorkItem(delegate { TryExecuteTask(task); });
    }

    protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
    {
        //return TryExecuteTask(task);
        return false;
    }

    protected override IEnumerable<Task> GetScheduledTasks()
    {
        return null;
    }
}

Note that our dummy TaskScheduler’s TryExecuteTaskInline is returning false and not executing the task; thus, even though the continuation has ExecuteSynchronously specified, the continuation will still run asynchronously.  As with the previous examples, you can see this by running the code, which will output two different thread IDs.  If you then modify TryExecuteTaskInline by commenting out the existing “return false;” and uncommenting in the “return TryExecuteTask(task);” line, then the program should output the same thread ID twice.

Leave a Comment
  • Please add 7 and 1 and type the answer here:
  • Post
  • I hope no one gets the idea from your Abort() example that it is OK to call Abort() in tasks.

  • Thanks, Tanveer.  And yes, the post wasn't meant to promote Abort, rather it was meant to highlight that TPL actually needs to do special things in case a thread was being aborted so as to avoid blocking the thread's teardown.  This is of most importance for AppDomain unloads.

  • Should Wait() calls on the antecedent task wait until ExecuteSynchronously continuations have finished?  I wouldn't have thought so, but wasn't sure given the comment "in effect running the continuation as part of the antecedent’s transition to a final state"

    connect bug:

    connect.microsoft.com/.../tpl-wait-call-on-task-doesnt-return-until-all-continuations-scheduled-with-executesynchronously-also-complete

    SO thread:

    stackoverflow.com/.../bug-in-tpl-taskcontinuationoptions-executesynchronously

  • Hi James-

    There's no guaranteed behavior here, in that it's not specified exactly when Wait needs to return.  However, practically Wait shouldn't need to wait for continuations to run.  In .NET 4, this was implemented in a way where it would return immediately when the task completed, but at the cost of needing to have a wait handle associated with the Task.  In .NET 4.5, Task was optimized significantly in certain areas, one of which entailed reimplementing Wait in terms of a continuation; that's why in .NET 4.5 RC you see Wait not returning until after synchronous continuations registered before the Wait was issued.  For RTM, we've modified the implementation further to address this so that Wait is no longer delayed by such continuations.

  • It seems the logic to avoid stackoverflows does not apply when using async/await? Should the same logic not also be applied or is it a completely different issue?

    Stackoverflow question:

    stackoverflow.com/.../how-to-avoid-the-possible-stack-overflow-in-this-async-await-program

    Connect bug:

    connect.microsoft.com/.../stackoverflowexception-when-using-async-await

  • @Luke Horsley: Yes, the logic for checking for a stack overflow and forcing asynchronous continuations if it's a concern exists for ContinueWith but not for await.  It's feasible to add it (at the expense of some performance overhead), it just wasn't done.

  • Thanks for getting back to me Stephen, would you say it was likely we could see this added for async/await?

    I am currently getting this issue when using an async lock around a highly contested resource, the critical section usually completes synchronously which can result in this stack overflow if there are a lot of things await'ing the lock. My current solution is to use `await Task.Yield()` inside of the critical section, forcing an asynchronous continuation. It is not ideal but do you believe this is currently an acceptable approach to take?

    I believe the stack overflow detection should be added for async/await, I can understand the performance implications, however the big advantage of async/await is simplicity. Having a seemingly "random" StackOverflowException thrown due to async/await really doesn't fit this model and could provide a nightmare to debug for the unaware.

  • @Luke Horsley: I think it's a reasonable request, and I'll make sure the team sees it again, but you shouldn't expect it any time in the near future.  Can you share more details on your specific case?  e.g. some pseudo-code that highlights how you end up with such a long chain of awaits?

  • Hi  Stephen.

    I know you are quite a bit busy with Roslyn and CoreFx (and may be some other wonderful things).  

    But your blog is the place to find answers on .NET TPL :)

    So could you pls elaborate a bit in a separate post on a new option - RunContinuationsAsynchronously - in TaskCreationOptions and   TaskContinuationOptions.

  • @Omari_O: That's a good suggestion, thanks... I'll try to post something this week.

  • @StephenToub I have provided a gist here: gist.github.com/.../9a78212f432887e05d21 which demonstrates the problem. I am currently using your AsyncSemaphore implementation (AsyncLock results in the same issue, SemaphoreSlim doesn't, but that is because it invokes TrySetResult on the ThreadPool).

    In my production code, I am importing thousands of items which log once they are completed. When logging to the console, I change the console colors depending upon the severity/level of the log message (not shown in sample), the lock is to ensure nothing else attempts to use the console while these colors have been changed for the specific message.

    When using Console.Out, a TextWriter.SyncTextWriter is used, which has a synchronous implementation of WriteLineAsync; this means the Release is always called on the same thread that the continuation of WaitAsync was invoked upon, eventually resulting in a StackOverflowException.

    I hope this is a decent enough explanation and sample, let me know if you have any questions.

  • @Luke Horsley: Thanks for the explanation.  I figured it was something along those lines.  FYI, I just blogged about a new API in .NET 4.6 that can be used to help work around such issues (among others): blogs.msdn.com/.../new-task-apis-in-net-4-6.aspx.

  • @Omari_O: Here you go: blogs.msdn.com/.../new-task-apis-in-net-4-6.aspx

Page 1 of 1 (13 items)