Holy cow, I wrote a book!
For some reason, the way values are returned from a dialog procedure confuses people,
so I'm going to try to explain it a different way.
The trick with dialog box procedures is realizing that they actually need to return two pieces
Since two pieces of information have to be returned, but a C function can have only
one return value, there needs to be some other way to return the second piece of information.
The return value of the dialog procedure is whether the message was handled. The second
piece of information - what the return value should be - is stashed in the DWLP_MSGRESULT
In other words, DefDlgProc goes something like this:
LRESULT CALLBACK DefDlgProc(
HWND hdlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
DLGPROC dp = (DLGPROC)GetWindowLongPtr(hdlg, DWLP_DLGPROC);
SetWindowLongPtr(hdlg, DWLP_MSGRESULT, 0);
BOOL_PTR fResult = dp(hdlg, uMsg, wParam, lParam);
if (fResult) return GetWindowLongPtr(hdlg, DWLP_MSGRESULT);
else ... do default behavior ...
If you return anything other than 0, then the value you set via SetWindowLongPtr(hdlg,
DWLP_MSGRESULT, value) is used as the message result.
For example, many WM_NOTIFY notifications allow you to override default behavior by
returning TRUE. To prevent a listview label from being edited, you can return TRUE
from the LVN_BEGINLABELEDIT notification. But if you are doing this from a dialog
procedure, you have to do this in two steps:
SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE);
The second line sets the return value for the dialog procedure, which tells DefDlgProc
that the message has been handled and default handling should be suppressed. The first
line tells DefDlgProc what value to return back to the sender of the message (the
listview control). If you forget either of these steps, the desired value will not
reach the listview control.
Notice that DefDlgProc sets the DWLP_MSGRESULT to zero before sending
the message. That way, if the dialog procedure neglects to set a message result explicitly,
the result will be zero.
This also highlights the importance of calling SetWindowLongPtr immediately before
returning from the dialog procedure and no sooner. If you do anything between setting
the return value and returning TRUE, that may trigger a message to be sent to the
dialog procedure, which would set the message result back to zero.
Caution: There are a small number of "special messages" which do not follow
this rule. The list is given in the
documentation for DialogProc. Why do these exceptions exist? Because when the
dialog manager was first designed, it was determined that special treatment for these
messages would make dialog box procedures easier to write, since you wouldn't have
to go through the extra step of setting the DWLP_MSGRESULT. Fortunately, since those
original days, nobody has added any new exceptions. The added mental complexity of
remembering the exceptions outweigh the mental savings of not having to write one
line of code ("SetWindowLongPtr(hdlg, DWLP_MSGRESULT, desiredResult)").