So you need a worker thread pool...

re: So you need a worker thread pool...

Shmuel Baron — Tue, 30 Mar 2004 11:01:00 GMT

Larry, what do you think about using the APC mechanism to implement this? Each thread has an APC queue, you could queue APCs to each thread in your pool in a round robbin fashion, or any other load balancing mechanism you wish, with the QueueUserAPC function. That should eliminate the lock convoys problem. Then have each thread use SleepEx to wait in an alertable wait state so it could execute the APCs when it gets some, and once it finished executing all the APCs go back to sleep. The OS manages the FIFO APC queue for you, so you don't have to worry about it either.

re: So you need a worker thread pool...

Larry Osterman — Tue, 30 Mar 2004 18:11:00 GMT

Grrgh. I had written an answer to you but somehow it got lost.

Lets try again.

That's a fascinating idea that I hadn't even considered before.

In my model, you have a pool of idempotent worker threads, which pick work up as needed and execute on it.

In your model, you have effectively two pools of threads - those that are idle and those that are doing work.

The only negative to your idea is that you need to know which thread is receiving the APC - you can't ever queue an APC to a thread that's actually doing work because (a) the thread might not wait interruptably for a long time, and (b) because the thread MIGHT wait interruptably and accidentally pick up your second APC and start executing it before it had finished its work - this could cause deadlocks or memory corruption.

To track this, you need to know which threads are processing work requests and which threads are idle, and only queue the requests to the idle threads.

It's more overhead than my solution, and the code to manage the thread state is also potentially subject to lock convoys, but...

I'm not saying it's a bad idea, it's not, but there is some subtlty in the solution that might bite you on the foot in the future if you're not careful.

re: So you need a worker thread pool...

Mike Dimmick — Thu, 01 Apr 2004 09:41:00 GMT

You don't mention that the completion port API also tries to limit the number of threads running simultaneously; if there's work to be done, but more than a given number of worker threads (associated with this port) are already running, NT won't release another thread. IIRC, this limit is 2x number of (virtual?) CPUs in the system.

Conversely, when a thread associated with a completion port blocks (e.g. to perform synchronous I/O), NT looks to see if it can release another thread to keep the CPU busy. If there are fewer than the above limit threads running, there are threads blocked on the completion port, and there's work queued, NT will release a thread to replace the one that blocked.

The net effect is to prevent context thrashing - where the scheduler spends more time trying to divide CPU time fairly among all the threads than the threads actually manage to get work done.

At least, that's my recollection from Richter (Programming Applications for Windows, 4th ed, or it might be in Programming Server-Side Applications).

re: So you need a worker thread pool...

Larry Osterman — Thu, 01 Apr 2004 15:03:00 GMT

You're totally right Mike, I didn't include that (mostly for lack of time).

See, there are LOTS of reasons to use completion ports for your work queues.

Things you shouldn't do, part 2: Dll's can't ever call CoInitialize* on the application's thread.

Larry Osterman's WebLog — Wed, 12 May 2004 19:22:00 GMT

re: So you need a worker thread pool...

Scott — Thu, 27 May 2004 20:26:00 GMT

So... is there a way to query the number of items currently waiting in the queue?

re: So you need a worker thread pool...

Larry Osterman — Thu, 27 May 2004 20:43:00 GMT

Nope.

The thing to keep in mind is that the number of items in the queue is meaningless. The microsecond after the value's loaded into the variable that holds the count, it's no longer valid.

You can't make decisions based on the value because by the time you get the value it's not valid. So the NT guys don't give you the ability to screw it up.

re: So you need a worker thread pool...

Scott — Fri, 28 May 2004 03:14:00 GMT

I'm actually trying to use the ATL:CThreadPool class (which uses I/O completion ports) to queue up a long list of tasks (~6000) that take quite some time to finish - the whole process takes around 24 hours. I am looking for a way to tell when the queue has emptied in order to move to the next phase of my application. Am I barking up the wrong tree?

re: So you need a worker thread pool...

Larry Osterman — Fri, 28 May 2004 05:25:00 GMT

Hmm... That's a good question...

Are you doing this on an MP machine? Are the tasks disk intensive? Since the CPU can only do one thing at a time, I'm wondering if queuing to the ATL::CThreadPool is the best idea.

But anyway, since I don't know what you're doing, I can't speak to that.

In your case, I'd just keep a counter - when you queue an item, increment it (with InterlockedIncrement), when you've processed a work item, you decrement it (InterlockedDecrement). When the count goes to 0, you're done.

The general case can't work because NT can't know why you want the count of items, but in your specific case, since you know you won't be queuing new items, you can make it work.

One gotcha - the count might go to 0 before you've finished queueing the items, especially on an MP machine, so you shouldn't check the count until you've queued all the tasks.

re: So you need a worker thread pool...

jbn — Mon, 07 Jun 2004 20:47:00 GMT

Another way would be to post a specific item that marks the end of the adding of items to the queue. Just create this special item before going MT, then in each worker thread, test the item you just retrieved from the queue against the special item, and if they're equal, just exit.

Why does Win32 even have Fibers?

Larry Osterman's WebLog — Thu, 06 Jan 2005 02:58:00 GMT

Why does Win32 even have Fibers?

Larry Osterman's WebLog — Mon, 10 Jan 2005 22:03:00 GMT

Concurrency, Part 9 - APIs that enable scalable programming

Larry Osterman's WebLog — Wed, 02 Mar 2005 20:37:00 GMT

Concurrency, Part 9 - APIs that enable scalable programming

Larry Osterman's WebLog — Wed, 02 Mar 2005 20:46:00 GMT

It's Getting Crowded in the ThreadPool

K. Scott Allen — Mon, 24 Jul 2006 05:35:21 GMT

It seems everybody loves the thread pool. The default scheduler in Windows Workflow uses the thread pool,...

A high waiter count on a free critical section may indicate a lock convoy

The Old New Thing — Tue, 12 Dec 2006 18:00:07 GMT

Nobody can make any progress because they keep getting stuck.

Threading deep dive – Day 8

My Adventures with Microsoft Technologies — Mon, 04 Aug 2008 18:31:36 GMT

Volatile Read / Writes MSDN says " The volatile modifier is usually used for a field that is accessed

Larry Osterman s WebLog So you need a worker thread pool | Insomnia Cure

Larry Osterman s WebLog So you need a worker thread pool | Insomnia Cure — Wed, 10 Jun 2009 03:40:00 GMT

PingBack from http://insomniacuresite.info/story.php?id=9259

Larry Osterman s WebLog So you need a worker thread pool | Cellulite Creams

Larry Osterman s WebLog So you need a worker thread pool | Cellulite Creams — Wed, 10 Jun 2009 04:45:09 GMT

PingBack from http://cellulitecreamsite.info/story.php?id=8889