MsgWaitForMultipleObjects and the queue state

re: MsgWaitForMultipleObjects and the queue state

Dave — Thu, 17 Feb 2005 15:33:00 GMT

Shouldn't that be:

while (PeekMessage ...

re: MsgWaitForMultipleObjects and the queue state

George — Thu, 17 Feb 2005 16:28:00 GMT

No, that was Raymond's whole point! If you peek, then wait, it fails as Raymond pointed out. While it is possible that you could write it as a PeekMessage loop, just writing while(PeekMessage()) isn't enough.

re: MsgWaitForMultipleObjects and the queue state

CN — Thu, 17 Feb 2005 16:38:00 GMT

If while's used to replace the current if, that should be enough, I think...

Raymond, in your example, is there actually any work done in the windows proc for WM_NEWRECORD? I currently don't understand why the falling behind would increase unless the counter was actually increased inside the window proc.

re: MsgWaitForMultipleObjects and the queue state

Raymond Chen — Thu, 17 Feb 2005 16:45:00 GMT

I guess I wasn't explicit enough. I assumed people would have figured out that the window procedure goes something like this:

case WM_NEWRECORD:
process the new record;
records++;
display new record count in window;
break;
case WM_GETRECORDCOUNT:
return records;

re: MsgWaitForMultipleObjects and the queue state

SS — Thu, 17 Feb 2005 17:03:00 GMT

When we have a message notification, we should process all available messages and check if the event is signalled .

case WAIT_OBJECT_1:
// we have message(s) - peek and dispatch
while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)) {
if(WaitForSingleObject(h,0) == WAIT_OBJECT_0)
DoSomethingWith(h); // event has been signalled

TranslateMessage(&msg);
DispatchMessage(&msg);
}
if (SendMessage(hwndNotify, WM_GETRECORDCOUNT,
0, 0) >= cRecordsExpected) {
return TRUE; // we got enough records
}
break;

re: MsgWaitForMultipleObjects and the queue state

asdf — Thu, 17 Feb 2005 17:15:00 GMT

Thanks for posting this entry. It, looks like I was doing it the wrong way in my code but never noticed any ill effects. There goes 95 and NT 4.0 support.

re: MsgWaitForMultipleObjects and the queue state

Rob Kennedy — Thu, 17 Feb 2005 17:22:00 GMT

The WaitMessage function works the same way. It only returns on NEW messages.

re: MsgWaitForMultipleObjects and the queue state

David Wright — Thu, 17 Feb 2005 18:41:00 GMT

Can somebody post what is the correct code? I've spent some time reading everything, but I don't want to be left guessing whether I understood it all correctly.
Thx!

Very helpful topic, BTW.

re: MsgWaitForMultipleObjects and the queue state

Ray Trent — Thu, 17 Feb 2005 18:52:00 GMT

Gads... this function is so florked up that several articles have appeared in Windows Developer/MSDN magazine about it (I wrote one of them, so it stuck in my mind).

A much more fundamental danger of this function is that it can break the synchronization of the mutex if you're not careful how you use it (recursive waits in particular are problematic).

Consider the case where you have a Lock() function that properly first calls MsgWaitForMultipleObjects, then processes the pending messages in a loop as Raymond suggests.

Now consider two different message handlers that need to access the protected resource:

case WM_PAINT:
Lock();
if (pProtected) {
// Call some function that calls some other
// function that does the following:
Lock(); // This recursive Lock is broken
pProtected->ProcessPaint();
Unlock();
}
Unlock();

case WM_KILLFOCUS:
Lock();
if (pProtected) {
delete pProtected;
pProtected = NULL;
}
Unlock();

The Lock above that's labelled "This recursive lock is broken" can process a WM_KILLFOCUS message which will successfully delete the protected resource (the thread already holds the mutex, so the wait will suceed). This, of course, causes a crash in the next line.

This all seems to be due to the fact that MsgWaitForMultipleObjects checks the message queue *first* rather than checking the state of the waited-for object first.

So, effectively, the necessity for processing the messages with a Peek() loop has generated a little preemptive operating system of its own, and since it's in the same thread that has the locked resource, the mutex doesn't protect.

The only solution I was able to come up with is to always put a "if (WaitForSingleObject(hMutex, 0) != WAIT_OBJECT_0)" before you try to use MsgWaitForMultipleObjects.

It should have been implemented this way in the OS, though.

re: MsgWaitForMultipleObjects and the queue state

Joshua Schaeffer — Thu, 17 Feb 2005 20:36:00 GMT

I would rather have had the ability to open a read-only handle to a live event object created by the message queue. It'd be way more flexible than trying to kludge MsgWaitForMultipleObjects into whatever scheme or framework I'm working with.

re: MsgWaitForMultipleObjects and the queue state

JDM — Thu, 17 Feb 2005 21:51:00 GMT

Can someone tell me if the following would be acceptable in the message loop for a window thread, or if I am still using the MsgWaitForMultipleObjects the wrong way. Sorry, in advance, if the code shows up incorrectly, I have never posted here before.
Thanks.

/* Start message loop for this thread. */
HANDLE[] hArray = { GetCurrentThread() };
while (TRUE)
{
MSG msg;
DWORD dwWait;
while (PeekMessage(&msg,NULL,0,0,PM_REMOVE))
{
if (msg.message == WM_QUIT)
{
ExitThread(0);
return 1;
}
TranslateMessage(&msg);
DispatchMessage(&msg);
}
dwWait = MsgWaitForMultipleObjectsEx (1, hArray, INFINITE, QS_ALLINPUT, 0);
if (dwWait == (WAIT_OBJECT_0 + 1))
{
continue;
}else{
// Do something..
}
}

re: MsgWaitForMultipleObjects and the queue state

Pavel Lebedinsky — Thu, 17 Feb 2005 22:11:00 GMT

Ray: reentrancy is a general problem in code that uses Windows message queues. This is just something you have to deal with - the system can't protect you from all possible scenarios.

Joshua: simply exposing an event would be very inefficient if you are only interested in a certain type of messages. You'd need some mechanism to tell the system when to signal the event. And then you'd have to deal with cases where two or more threads are waiting on the same event (should this be illegal? or should each thread get its own event object?)

re: MsgWaitForMultipleObjects and the queue state

Pavel Lebedinsky — Thu, 17 Feb 2005 22:19:00 GMT

re: MsgWaitForMultipleObjects and the queue state

Norman Diamond — Fri, 18 Feb 2005 00:13:00 GMT

Jiggling the mouse to get a response is a common occurence in the Windows 2000 Start menu and Windows XP Classic version Start menu. It is so common that it doesn't even take thinking. But this base note does make me wonder. You know, click on the Start button, move up to Programs, move to the right and locate the folder containing the link you really want to click on, but that folder doesn't expand. You have to move the mouse to hover over another folder and then move back to the one you really wanted to expand. So does Start menu processing contain the bug described here?

re: MsgWaitForMultipleObjects and the queue state

Raymond Chen — Fri, 18 Feb 2005 00:17:00 GMT

No that's caused by a global foreground idle hook. Remember? You complained about this last year and I debugged it for you.

re: MsgWaitForMultipleObjects and the queue state

Daniel Bowen — Fri, 18 Feb 2005 00:57:00 GMT

ATL includes a function "AtlWaitWithMessageLoop". Back a while ago on the ATL mailing lists, there was discussion of some deadlock possibilities with it (in part due to the issue you raise here), and how it might be improved. Here's the version that I came up with and currently use. I've often wondered if the approach I use has any possible problems. What improvements could be made to this version? (the formatting might get messed up)

BOOL WaitWithMessageLoop(
HANDLE hHandleToWaitOn,
DWORD dwInitialTimeOutMilliseconds = 0,
DWORD dwIterateTimeOutMilliseconds = 500)
{
DWORD dwRet=0;
MSG msg={0};

dwRet = ::WaitForSingleObject(hHandleToWaitOn, dwInitialTimeOutMilliseconds);
if(dwRet == WAIT_OBJECT_0)
{
// The object is already signalled.
return TRUE;
}

while(true)
{
// There are one or more window message available. Dispatch them.
while(::PeekMessage(&msg,NULL,NULL,NULL,PM_REMOVE))
{
::TranslateMessage(&msg);
::DispatchMessage(&msg);
dwRet = ::WaitForSingleObject(hHandleToWaitOn, 0);
if(dwRet == WAIT_OBJECT_0)
{
// The object is already signalled.
return TRUE;
}
}

// Now we've dispatched all the messages in the queue.
// Use MsgWaitForMultipleObjects to either tell us there are
// more messages to dispatch, or that our object has been signalled.

dwRet = ::MsgWaitForMultipleObjects(1, &hHandleToWaitOn, FALSE,
dwIterateTimeOutMilliseconds, QS_ALLINPUT);

if(dwRet == WAIT_OBJECT_0)
{
// The event was signaled.
return TRUE;
}
else if(dwRet == WAIT_OBJECT_0 + 1)
{
// New messages have come that need to be dispatched.
continue;
}
else if(dwRet == WAIT_TIMEOUT)
{
// We hit our time limit, continue with the loop.
continue;
}
else
{
// Something else happened.
return FALSE;
}
}

return FALSE;
}

re: MsgWaitForMultipleObjects and the queue state

Cheong — Fri, 18 Feb 2005 01:39:00 GMT

Ah... This is exactly one problem that I've encountered during my serial port application development. I was totally puzzled at the random lost of records. We (the development team) did tried to figure out the pattern of problem but never success, so the problem hang there until a new collegue told us about this...

re: MsgWaitForMultipleObjects and the queue state

CJP — Fri, 18 Feb 2005 03:03:00 GMT

What happens if a message is enqueued between the call to MsgWaitForMultipleObjects and the call to PeekMessage...

E.G:

<- (worker thread) Message 1 Added
(processing thread) MsgWaitForMultipleObjects
<- (worker thread) Message 2 Added
(processing thread) PeekMessage
(processing thread) Message 2 processed

Since PeekMessage is only looking at the top message on the queue would not the first message be 'lost' since it will be considered 'old' as far as MsgWaitForMultipleObjects is concerned?

Of course, I could be barking up the wrong tree, I don't have much experience with windows messaging and I suck at concurrency.

re: MsgWaitForMultipleObjects and the queue state

Norman Diamond — Fri, 18 Feb 2005 03:19:00 GMT

2/17/2005 4:17 PM Raymond Chen

> No that's caused by a global foreground idle
> hook. Remember? You complained about this

Sorry, I don't remember. Thank you for pointing it out. Perhaps I should keep notes :-)

> last year and I debugged it for you.

Thank you again. Do you know if it will be in Windows XP SP3 and Windows 2000 SP5? (I don't think I need to ask if you have the power to put it there, but can still ask if you know.)

re: MsgWaitForMultipleObjects and the queue state

Raymond Chen — Fri, 18 Feb 2005 06:05:00 GMT

It was some program you installed that was messing with foreground idle. A translation program or something as I recall. Perhaps you could complain to the vendor of that program.

re: MsgWaitForMultipleObjects and the queue state

Tom M — Fri, 18 Feb 2005 09:40:00 GMT

Don't you have to do something like while(::PeekMessage(&msg,NULL,NULL,NULL,PM_REMOVE))
{
::TranslateMessage(&msg);
::DispatchMessage(&msg);
}
on WAIT_OBJECT_0 too, or else you can forget about those messages as well?

re: MsgWaitForMultipleObjects and the queue state

Hallvard Vassbotn — Fri, 18 Feb 2005 12:15:00 GMT

> The MsgWaitForMultipleObjectsEx function lets you pass the MWMO_INPUTAVAILABLE flag to indicate that it should check for previously-ignored input.

But this flag is not supported on NT.

> You call MsgWaitForMultipleObjects and include the QS_ALLPOSTMESSAGE flag

What is the difference between

QS_ALLPOSTMESSAGE = $0100;

and

QS_POSTMESSAGE = $0008;

?

The MSDN docs says the same for both:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/msgwaitformultipleobjectsex.asp

"QS_POSTMESSAGE A posted message (other than those just listed) is in the queue."

"QS_ALLPOSTMESSAGE A posted message (other than those listed here) is in the queue"

IMO, this is a really messy API, but I think we are using it correctly - we use an object oriented wrapper (in Delphi) around the raw API calls, but it really boils down to calling

MsgWaitForMultipleObjects with the flag:

QS_ALLINPUT = (QS_INPUT or QS_POSTMESSAGE or QS_TIMER or QS_PAINT or QS_HOTKEY or QS_SENDMESSAGE);

and when it returns WAIT_OBJECT_0 + FObjCount we basically return to Delphi VCL's TApplication.ProcessMessages routine that will empty the message queue.

re: MsgWaitForMultipleObjects and the queue state

Dave — Fri, 18 Feb 2005 15:26:00 GMT

The very bottom of the MsgWaitForMultipleObjectsEx page says: "The QS_ALLPOSTMESSAGE and QS_POSTMESSAGE flags differ in when they are cleared" and some more detail after that.

I sure hope an answer key is forthcoming...

re: MsgWaitForMultipleObjects and the queue state

Alex Blekhman — Sat, 19 Feb 2005 09:35:00 GMT

In my opinion, situations that justify MsgWaitForMultipleObjects are quite rare nowadays. In the past thread were expensive and developers needed to scratch their heads how to cram in gazillion things in one thread. Today it’s just doesn’t worth it. Besides, this is annoying when you have message loops splattered here and there in the code. If I need to process some data which involves synchronization issues, then I don’t even want to think about messing it with GUI. Just stash it in worker thread, do the job, then notify GUI in some flexible and lazy way (say, with PostMessage).

re: MsgWaitForMultipleObjects and the queue state

Norman Diamond — Mon, 21 Feb 2005 03:42:00 GMT

2/17/2005 10:05 PM Raymond Chen

> It was some program you installed that was
> messing with foreground idle.

That is a completely unrelated issue. The matter of the Start menu, at random times not expanding folders unless the mouse is moved to a different folder and moved back, occurs even when all applications come from Microsoft. It is intermittently reproducible both on real PCs and virtual PCs. Now I resume thinking that your base note here looks like a likely explanation. The symptom really resembles the example you gave.

re: MsgWaitForMultipleObjects and the queue state

Raymond Chen — Mon, 21 Feb 2005 03:54:00 GMT

Hm well I've never experienced it. But I doubt it's this problem. Menus don't mix user and kernel waits.

Modality, part 2: Code-modality vs UI-modality

The Old New Thing — Mon, 21 Feb 2005 18:00:00 GMT

The dichotomy breaks in the other direction, too.

re: MsgWaitForMultipleObjects and the queue state

Ian Boyd — Mon, 21 Feb 2005 19:03:00 GMT

Is there a way to tell Windows to "from now on pretend i don't know about any messages in the queue" function? Basically undo the strangeness that PeekMessage does.

Does MsgWaitForMultipleObjects also tag queued messages as "read" like PeekMessage. i.e. Once i "realize" there is a message in the queue with MsgWaitForMultipleObjects, am i then forced to process all messages in the queue?

re: MsgWaitForMultipleObjects and the queue state

daev — Mon, 21 Feb 2005 21:35:00 GMT

"In my opinion, situations that justify MsgWaitForMultipleObjects are quite rare nowadays. In the past thread were expensive and developers needed to scratch their heads how to cram in gazillion things in one thread."

Actually MsgWaitForMultipleObjects turns out to be very useful in multithread situations.

It's the only way for you to add a user-defined window message that works like WM_PAINT -- a very low priority message, and there's only one of them in the queue no matter how many times you post it.

I used a message loop based on MsgWaitForMultipleObjects when I had a program with a worker thread that did computation and wrote output text. There was also a main UI thread which controlled the window, so the worker thread should not be allowed to change the window contents itself.

When the worker thread (which you can imagine as printing a million digits of Pi) has some text to output, it must post a message to the UI thread to get the text displayed. But if you used ordinary user-defined messages, you could get thousands of tiny WM_USERs clogging up the message queue.

The solution is to have a text buffer that holds the new output. Any time new text is added to the buffer, the worker thread posts a WM_PAINT-type message to the UI thread. There's only one WM_PAINT at any time, and it's always handled after more critical interactive messages (like scroll-bar manipulation). When the UI thread services the message, it outputs all the text in the buffer at once, just as a WM_PAINT handler updates the whole invalidated region of the window, regardless of how many smaller invalidations composed it.

But since you can't define your own WM_PAINT-type message, you can get the same effect by using MsgWaitForMultipleObjects and replacing "post a message" with "flip on the event object." If you write the loop correctly (as described by Raymond) you get the same behavior.

(Raymond and others: Are there any potential gotchas to this approach?)

re: MsgWaitForMultipleObjects and the queue state

Alex Blekhman — Tue, 22 Feb 2005 08:35:00 GMT

daev wrote:
"Actually MsgWaitForMultipleObjects turns out to be very useful in multithread situations.

It's the only way for you to add a user-defined window message that works like WM_PAINT -- a very low priority message, and there's only one of them in the queue no matter how many times you post it."

I agree. That's the example of MsgWaitForMultipleObjects where it's handy. However, with two reservations:

1. You use it to "improve" regular main message loop, not to implement bastard message loop somewhere in program.

2. While less elegant, simple WM_TIMER with polling of text buffer is much less confusing than MsgWaitForMultipleObjects. Unless you really need to update GUI ASAP, updating it 3-5 times in second will be indistinguishable from user's point of view. Consider how significantly less explanation requires WM_TIMER to maintenace programmer than esoteric gotchas of MsgWaitForMultipleObjects.

re: MsgWaitForMultipleObjects and the queue state

Alex Blekhman — Tue, 22 Feb 2005 08:36:00 GMT

re: MsgWaitForMultipleObjects and the queue state

Ray Trent — Thu, 24 Feb 2005 18:56:00 GMT

> Ray: reentrancy is a general problem in code
> that uses Windows message queues. This is just
> something you have to deal with - the system
> can't protect you from all possible scenarios.

This is, of course, true. And as I mentioned there are numerous ways you can screw yourself, by yourself, if you're not careful.

However, this function is the only one I know of where the action is entirely implicit. I.e. you're not calling something that you can know is going to be doing a SendMessage in the context of the current thread. You, in fact, can't know what it will do at all.

As a result, when you're talking about a mutex that's grabbed by message handlers (which is one of the main uses of this kind of function... to avoid the inevitable deadlock that will happen if you ever do anything that requires message processing in such a handler), the only safe thing to do is to always return without processing any pending messages if the mutex is already held.

Not only does this function not do that by default (which obviates almost its entire point), there's not even an option for it... you have to code that additional check by hand yourself or you're looking at a complete maintenance nightmare.

It's not the end of the world, but it is an extremely obscure pitfall.

re: MsgWaitForMultipleObjects and the queue state

Neil — Sat, 05 Mar 2005 23:54:00 GMT

>It's the only way for you to add a user-defined window message that works like WM_PAINT -- a very low priority message, and there's only one of them in the queue no matter how many times you post it.

I seem to remember WM_COALESCE_FIRST … WM_COALESCE_LAST were supposed to have had similar semantics in that there's only one of each in the queue.

Pumping messages while waiting for a period of time

The Old New Thing — Fri, 10 Feb 2006 19:54:32 GMT

Putting together pieces you already know.

COM, Re-entrancy, and Message Pumping

Tim Ng — Thu, 07 Sep 2006 07:51:14 GMT

Disclaimer: I hesitated posting this because this is a topic that is extremely complicated and deep,...

When and How should we use MsgWaitForMultipleObjects? « Sharing my thoughts…

Fri, 19 Sep 2008 20:23:03 GMT

PingBack from http://sarathc.wordpress.com/2008/09/19/when-and-how-should-we-use-msgwaitformultipleobjects/

From Zero To One » Active Sleep procedure - Aftermath

From Zero To One » Active Sleep procedure - Aftermath — Sat, 06 Dec 2008 17:17:25 GMT

PingBack from http://www.cromis.net/blog/2008/12/active-sleep-procedure-aftermath/