The CPU is free, how come my server isn’t responding to requests?

You’ve written your web application using all the latest features of .NET.  Because some tasks should be done in the background, you invoke them by calling BeginInvoke on a delegate, asking the thread pool to run the task asynchronously.  That should have a minute effect on the rest of the system, right?

Not necessarily.  You may see, like we have, a different kind of contention.  It’s not threads competing for CPU time.  It’s requests competing for the thread pool.  The thread pool has a fixed maximum number of threads, to minimize the amount of time the system spends context switching.  The default maximum is 25 threads per CPU.

There is one shared thread pool per process.  ASP.NET uses threads from the pool to service web requests. Requests to the thread pool all have equal priority – they are granted in a first-come first-served way.  So if you have made 25 requests for background tasks, the system won’t begin the 26^th request until one of them completes.  It can’t – the threads are all occupied.  If that 26^th request is an ASP.NET web request, too bad, it will have to wait its turn.

“So why is the CPU only at 50%” you may ask.  Those threads won’t necessarily be using the CPU.   Most enterprise applications today are I/O-bound, spending most of their time waiting for the disk or the network.  It’s not uncommon to see 25 threads all waiting for I/O at the same time. 

So be on the watch for this situation if you use the thread pool in .NET.  Look to see if the number of threads in your process has maxed out. 

Later I’ll give some details on how we solved the problem for our system.