The other method of using a window-procedure-like dialog box is to change the rules of the game. Normally, the window procedure for a dialog box is the DefDlgProc function, which calls the dialog procedure and then takes action if the dialog procedure indicated that it desired the default action to take place.

The dialog procedure is subservient to DefDlgProc, providing advice when requested. The kernel of the idea for this technique is to "turn the tables". Make DefDlgProc be the one who gives advice and you be the one that asks for the advice when you want it.

We do this by making the window procedure be our own function which decides whether or not it wants the default action to happen. If so, it calls DefDlgProc to do it, after giving the dialog a dummy dialog procedure that always says "Just do the default".

Here's the flow diagram:

Message delivered
-> WLWndProc
   -> your WLDlgProc
      decide what to do
      want to do the default action
      -> DefDlgProc
         -> dummy dialog procedure
         <- always returns FALSE
         DefDlgProc does default action
      <- returns result of default behavior
      you do other stuff (perhaps modify
      default behavior after it occurred)
   <- returns result
<- returns result

To do this, we need to register a custom dialog class. You always wondered what that was for: Now you know.

BOOL
InitApp(void)
{
  WNDCLASS wc;

  wc.style = CS_DBLCLKS | CS_SAVEBITS | CS_BYTEALIGNWINDOW;
  wc.lpfnWndProc = WLWndProc;
  wc.cbClsExtra = 0;
  wc.cbWndExtra = DLGWINDOWEXTRA + sizeof(WLDLGPROC);
  wc.hInstance = g_hinst;
  wc.hIcon = NULL;
  wc.hCursor = LoadCursor(NULL, IDC_ARROW);
  wc.hbrBackground = NULL;
  wc.lpszMenuName = NULL;
  wc.lpszClassName = TEXT("WLDialog");

  if (!RegisterClass(&wc)) return FALSE;

  return TRUE;
}

This creates a new window class called "WLDialog" which we will use as our custom dialog class. When you create a custom dialog class, you must set the cbWndExtra to DLGWINDOWEXTRA bytes, plus any additional bytes you wish to use for yourself. We need to store an extra WLDLGPROC, so we add that in.

To use our custom dialog procedure, the dialog template must use the "CLASS" keyword to specify the custom dialog class:

1 DIALOGEX DISCARDABLE  0, 0, 200,200
STYLE DS_SHELLFONT | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CLASS "WLDialog"
CAPTION "sample"
FONT 8, "MS Shell Dlg"
BEGIN
    DEFPUSHBUTTON "&Bye", IDCANCEL, 7,4,50,14, WS_TABSTOP
END

This is exactly the same as a regular dialog box template, except that there is a "CLASS" entry which specifies that this dialog box should use our new class. Paralleling the DialogBoxParam function we have our own:

typedef LRESULT (CALLBACK* WLDLGPROC)(HWND, UINT, WPARAM, LPARAM);

struct WLDIALOGINFO {
  WLDLGPROC wldp;
  LPARAM lParam;
};

INT_PTR
WLDialogBoxParam(HINSTANCE hinst, LPCTSTR pszTemplate,
  HWND hwndParent, WLDLGPROC wldp, LPARAM lParam)
{
  WLDIALOGINFO wldi = { wldp, lParam };
  return DialogBoxParam(hinst, pszTemplate,
           hwndParent, WLDlgProc, (LPARAM)&wldi);
}

This packages up the WndProc-Like dialog procedure and its reference data so we can recover it in our window procedure:

LRESULT CALLBACK
WLWndProc(HWND hdlg, UINT uiMsg, WPARAM wParam, LPARAM lParam)
{
  if (uiMsg == WM_INITDIALOG) {
    WLDIALOGINFO *pwldi = (WLDIALOGINFO*)lParam;
    SetWindowLongPtr(hdlg, DLGWINDOWEXTRA, (LONG_PTR)pwldi->wldp);
    lParam = pwldi->lParam;
  }
  WLDLGPROC wldp = (WLDLGPROC)GetWindowLongPtr(hdlg, DLGWINDOWEXTRA);
  if (wldp) {
    return wldp(hdlg, uiMsg, wParam, lParam);
  } else {
    return DefDlgProc(hdlg, uiMsg, wParam, lParam);
  }
}

This is the window procedure for the custom dialog. When the WM_INITDIALOG message comes in, we recover the original parameters to WLDialogBoxParam. The WLDLGPROC we save in the extra bytes we reserved, and the original LPARAM becomes the lParam that we pass to the WLDLGPROC. Then for each message that comes in, we pass the message and its parameters directly to the WLDLGPROC and return the value directly. No DWLP_MSGRESULT necessary.

The last piece of the puzzle is the dialog procedure we actually hand to the dialog manager:

INT_PTR CALLBACK
WLDlgProc(HWND hdlg, UINT uiMsg, WPARAM wParam, LPARAM lParam)
{
  return FALSE;
}

All it says is, "Do the default thing."

Okay so let's write yet another version of our sample program, using this new architecture:

LRESULT CALLBACK SampleWLDialogProc(
HWND hdlg, UINT uiMsg, WPARAM wParam, LPARAM lParam)
{
  switch (uiMsg) {
  case WM_INITDIALOG:
    break;

  case WM_COMMAND:
    switch (GET_WM_COMMAND_ID(wParam, lParam)) {
    case IDCANCEL:
      MessageBox(hdlg, TEXT("Bye"), TEXT("Title"), MB_OK);
      EndDialog(hdlg, 1);
      break;
    }
    break;

  case WM_SETCURSOR:
    if (LOWORD(lParam) == HTCAPTION) {
      SetCursor(LoadCursor(NULL, IDC_SIZEALL));
      return TRUE;
    }
    break;
  }

  return DefDlgProc(hdlg, uiMsg, wParam, lParam);
}

int WINAPI WinMain(HINSTANCE hinst, HINSTANCE hinstPrev,
                   LPSTR lpCmdLine, int nShowCmd)
{
  InitApp();
  WLDialogBoxParam(hinst, MAKEINTRESOURCE(1),
            NULL, SampleWLDialogProc, 0);
  return 0;
}

In this style of WndProc-Like dialog, we just write our dialog procedure as if it were a window procedure, calling DefDlgProc() to perform default behavior. And to get this new behavior, we use WLDialogBoxParam instead of DialogBoxParam

So now I've developed two quite different ways you can write WndProc-Like dialog procedures. You may not like either one of them, so go ahead and write a third way if you prefer. But at least I hope you learned a little more about how Windows works.

Answer to previous exercise: The m_fRecursing flag does not need to be per-instance. It only needs to be valid long enough that the recursive call that comes immediately afterwards can be detected. However, a global variable would not work because two threads might be inside the recursive DefDlgProc call simultaneously. But a thread-local variable would work. (If you prefer to code in plain C or don't want to use classes, you may find this alternative easier to work with.)

In the discussion following my entry about dialog procedure return values, somebody suggested an alternate dialog design where you just call DefDlgProc to do default actions (the same way you write window procedures and DefWindowProc) rather than returning TRUE/FALSE.

So let's do that. In fact, we're going to do it twice. I'll cover one method today and cover an entirely different method later this week. Each method consists of a simple kernel of an idea; the rest is just scaffolding to make the kernel work.

The first way uses a recursive call from the dialog procedure back into DefDlgProc to trigger the default behavior. This technique requires that you have a flag that lets you detect (and therefore break) the recursion. Since you typically have instance data attached to your dialog box anyway, it's not too hard to add another member to it.

The kernel is to "subvert the recursive call". DefDlgProc calls your dialog procedure to see what you want to do. When you want to do the default action, just call DefDlgProc recursively. The inner DefDlgProc will call your dialog procedure to see if you want to override the default action. Detect this recursive call and return FALSE ("do the default"). The recursive DefDlgProc will then perform the default action and return its result. Now you have the result of the default action, and you can modify it or augment it before returning that as the result for the dialog box procedure, back to the outer DefDlgProc which returns that value back as the final message result.

Here's the flow diagram, for those who prefer pictures:

Message delivered
-> DefDlgProc
   -> your dialog procedure
      decide what to do
      want to do the default action
      -> DefDlgProc
         -> your dialog procedure
            detect recursion
         <- return FALSE
         DefDlgProc sees FALSE
         performs default behavior
      <- returns result of default behavior
      you do other stuff (perhaps modify
      default behavior after it occurred)
      set DWLP_MSGRESULT to desired result
   <- return TRUE
   retrieve DWLP_MSGRESULT
<- return it as message result

Given this sketch, you should be able to write it up yourself. Here's what I came up with. I call it a Wndproc-Like Dialog:

class WLDialogBox
{
public:
  virtual LRESULT WLDlgProc(
            HWND hdlg, UINT uMsg,
            WPARAM wParam, LPARAM lParam)
  {
    return DefDlgProcEx(hdlg, uMsg, wParam, lParam,
                        &m_fRecursing);
  }

  INT_PTR DoModal(HINSTANCE hinst, LPCTSTR pszTemplate,
                  HWND hwndParent)
  {
    m_fRecursing = FALSE;
    return DialogBoxParam(hinst, pszTemplate, hwndParent,
                          s_DlgProc, (LPARAM)this);
  }

private:
  static INT_PTR CALLBACK s_DlgProc(
            HWND hdlg, UINT uMsg,
            WPARAM wParam, LPARAM lParam)
  {
    if (uMsg == WM_INITDIALOG) {
      SetWindowLongPtr(hdlg, DWLP_USER, lParam);
    }

    WLDialogBox *self = (WLDialogBox*)GetWindowLongPtr(
                            hdlg, DWLP_USER);
    if (!self) {
      return FALSE;
    }

    CheckDefDlgRecursion(&self->m_fRecursing);

    return SetDlgMsgResult(hdlg, uMsg,
              self->WLDlgProc(
                hdlg, uMsg, wParam, lParam));
  }

private:
  BOOL m_fRecursing;
};

Let's walk through this class.

The WLDlgProc method is virtual because we expect derived classes to do custom actions in their dialog procedure that we invoke from our s_DlgProc. The default implementation in the base class uses the DefDlgProcEx macro from windowsx.h to do the dirty work. That's right, this technique has been published by Microsoft since 1992. If you look at DefDlgProcEx, it sets the recursion flag to TRUE and then calls DefDlgProc, which triggers the recursive call.

I could have had a separate WLDefDlgProc method which calls DefDlgProcEx and have WLDlgProc call WLDefDlgProc. (In fact, my first version did exactly that.) But I decided not to have a WLDefDlgProc to remove the temptation to bypass the base class's WLDefDlgProc. Instead, if you want default handling to take place, forward the call to your base class's WLDefDlgProc.

The s_DlgProc method is the dialog procedure used for all instances of Wndproc-Like dialogs. It initializes itself in the WM_INITDIALOG message so future messages can identify which instance of the dialog is handling the message. After short-circuiting messages that arrive before the dialog box has initialized, it uses the CheckDlgRecursion macro, also from windowsx.h. This macro checks the recursion flag; if set, then it resets the flag and just returns FALSE immediately. This is what stops the recursion. Otherwise, it calls the WLDlgProc method (which has probably been overriden in a derived class), then sets the dialog procedure return value and returns.

The SetDlgMsgResult macro also comes from windowsx.h: It stores the return value into the DWLP_MSGRESULT and returns TRUE. Well, unless the message is one of the special exceptions, in which case it returns the value directly. Note to 64-bit developers: There is a bug in this macro as currently written. The expression (BOOL)(result) should be changed to (INT_PTR)(result) so that the upper 32 bits of the return value is not truncated.

The last method is DoModal, which initializes the recursion flag and kicks off the dialog box.

Here's a sample program that illustrates the use of this class:

class SampleWLDlg : public WLDialogBox
{
  LRESULT WLDlgProc(HWND hdlg, UINT uMsg,
            WPARAM wParam, LPARAM lParam)
  {
    switch (uMsg) {
    HANDLE_MSG(hdlg, WM_COMMAND, OnCommand);
    HANDLE_MSG(hdlg, WM_SETCURSOR, OnSetCursor);
    }
    return __super::WLDlgProc(hdlg, uMsg, wParam, lParam);
  };

  void OnCommand(HWND hdlg, int id,
                 HWND hwndCtl, UINT codeNotify)
  {
    switch (id) {
    case IDCANCEL:
      MessageBox(hdlg, TEXT("Bye"), TEXT("Title"), MB_OK);
      EndDialog(hdlg, 1);
      break;
    }
  }

  BOOL OnSetCursor(HWND hdlg, HWND hwndCursor,
                   UINT codeHitTest, UINT msg)
  {
    if (codeHitTest == HTCAPTION) {
      SetCursor(LoadCursor(NULL, IDC_SIZEALL));
      return TRUE;
    }
    return FORWARD_WM_SETCURSOR(hdlg, hwndCursor,
                  codeHitTest, msg, __super::WLDlgProc);
  }
};

int WINAPI WinMain(HINSTANCE hinst, HINSTANCE hinstPrev,
                   LPSTR lpCmdLine, int nShowCmd)
{
    SampleWLDlg dlg;
    dlg.DoModal(hinst, MAKEINTRESOURCE(1), NULL);
    return 0;
}

1 DIALOGEX DISCARDABLE  0, 0, 200,200
STYLE DS_SHELLFONT | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CAPTION "sample"
FONT 8, "MS Shell Dlg"
BEGIN
DEFPUSHBUTTON "&Bye",IDCANCEL,"Button",WS_TABSTOP,7,4,50,14
END

To illustrate a custom return value, I override the WM_SETCURSOR message to display a custom cursor when the mouse is over the caption area. It's not exciting, but it gets the point across.

Observe that in two places, we invoked the default handler by calling __super::WLDlgProc. __super is a Visual C++ extension that resolves to the base class of your derived class. This is quite handy since it saves the reader the trouble of figure out "So which level in the class hierarchy are we forwarding this call to?" If you wanted to forward a call to your grandparent class, you would use __super::__super::WLDlgProc.

If your compiler doesn't support __super, you can fake it by adding this line to the definition of SampleWLDlg:

  typedef WLDialogBox super;

and using super::WLDlgProc without the underscores. In fact, this is the technique I use because I was doing it before the VC folks added the __super keyword and now it's just habit.

So what was wrong with our little subclassing sketch?

Most people figured this out.

Consider what would happen if somebody else had subclassed the window during the "... do stuff ..." section. When we unsubclassed the window, we would have removed two subclasses, the one we installed, and the one that was installed after us. If the other subclass allocated memory (which is very common), then that memory got leaked, in addition to the subclass failing to do whatever it was trying to do.

Do not assume that subclasses are added and removed in a purely stack-like manner. If you want to unsubclass and find that you are not the window procedure at the top of the chain you cannot safely unsubclass. You will have to leave your subclass attached until it becomes safe to unsubclass. Until that time, you just have to keep passing the messages through to the previous procedure.

This is quite a cumbersome process, so the shell team wrote some helper functions to do all this for you. The SetWindowSubclass function does all the grunt work of installing a subclass procedure, remembering the previous one, and passing reference data to the subclass procedure you provide. You use the DefSubclassProc function to forward the message to the previous subclass procedure, and when you're done, you use the RemoveWindowSubclass function to remove yourself from the chain. RemoveWindowSubclass does all the work to do the right thing if you are not the window procerure at the top of the chain.

One gotcha that isn't explained clearly in the documentation is that you must remove your window subclass before the window being subclassed is destroyed. This is typically done either by removing the subclass once your temporary need has passed, or if you are installing a permanent subclass, by inserting a call to RemoveWindowSubclass inside the subclass procedure itself:

...
case WM_NCDESTROY:
  RemoveWindowSubclass(hwnd, thisfunctionname, uIdSubclass);
  return DefSubclassProc(...);

One comment expressed concern that a message could be sent between the call to SubclassWindow and the store of the previous window procedure into the OldWndProc variable. This is actually a safe operation provided that you are doing the work from the thread that owns the window you are subclassing. Remember that message delivery occurs only when the thread is in a receiving state, such as when it calls GetMessage or PeekMessage. If somebody sends a message when the thread is not in a receiving state, the message merely waits until the thread finally calls GetMessage (for example) before being delivered. Since we don't make any message-receiving function calls between the SubclassWindow and the store into OldWndProc, there is no risk of an untimely message arriving before the store to OldWndProc has occurred.

Window subclassing is trickier than you think. Consider this code sketch:

// Subclass the window for a little while
WNDPROC OldWndProc = SubclassWindow(hwnd, NewWndProc);
... do stuff ...
// Okay all done, remove our subclass by
// restoring the previous window procedure
SubclassWindow(hwnd, OldWndProc);

If a framework exposes functionality provided by a lower layer, how hard should the framework try to insulate you from all the quirks and limitations of the lower layer?

Instinctively, of course, you would say, "The framework should insulate me completely." But be careful what you ask for. If a framework insulated you completely, then every limitation of the underlying layer needs to be worked around in some manner or other. This would mean writing a lot of code to emulate missing functionality or removing a limitation, just in case somebody using the framework actually runs into that limitation.

Let's take for example the ToolTip.AutoPopDelay property. The ToolTip class is a Windows Forms wrapper around the Common Controls ToolTip window class. If you look at the documentation for the TTM_SETDELAYTIME message, you'll see that the delay time (iTime) is passed in the low word of the lParam parameter. Consequently, it is limited to a 16-bit value, and in this case, it's a signed 16-bit value since negative values for iTime have special meaning (as noted in the documentation).

Since the maximum value for a signed 16-bit integer is 32767, the maximum value you can set for the delay time is a little over 32 seconds.

For some reason, the way values are returned from a dialog procedure confuses people, so I'm going to try to explain it a different way.

The trick with dialog box procedures is realizing that they actually need to return two pieces of information:

• Was the message handled?
• If so, what should the return value be?

Since two pieces of information have to be returned, but a C function can have only one return value, there needs to be some other way to return the second piece of information.

The return value of the dialog procedure is whether the message was handled. The second piece of information - what the return value should be - is stashed in the DWLP_MSGRESULT window long.

In other words, DefDlgProc goes something like this:

LRESULT CALLBACK DefDlgProc(
    HWND hdlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
{
    DLGPROC dp = (DLGPROC)GetWindowLongPtr(hdlg, DWLP_DLGPROC);
    SetWindowLongPtr(hdlg, DWLP_MSGRESULT, 0);
    BOOL_PTR fResult = dp(hdlg, uMsg, wParam, lParam);
    if (fResult) return GetWindowLongPtr(hdlg, DWLP_MSGRESULT);
    else ... do default behavior ...
}

If you return anything other than 0, then the value you set via SetWindowLongPtr(hdlg, DWLP_MSGRESULT, value) is used as the message result.

For example, many WM_NOTIFY notifications allow you to override default behavior by returning TRUE. To prevent a listview label from being edited, you can return TRUE from the LVN_BEGINLABELEDIT notification. But if you are doing this from a dialog procedure, you have to do this in two steps:

    SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE);
    return TRUE;

The second line sets the return value for the dialog procedure, which tells DefDlgProc that the message has been handled and default handling should be suppressed. The first line tells DefDlgProc what value to return back to the sender of the message (the listview control). If you forget either of these steps, the desired value will not reach the listview control.

Notice that DefDlgProc sets the DWLP_MSGRESULT to zero before sending the message. That way, if the dialog procedure neglects to set a message result explicitly, the result will be zero.

This also highlights the importance of calling SetWindowLongPtr immediately before returning from the dialog procedure and no sooner. If you do anything between setting the return value and returning TRUE, that may trigger a message to be sent to the dialog procedure, which would set the message result back to zero.

Caution: There are a small number of "special messages" which do not follow this rule. The list is given in the documentation for DialogProc. Why do these exceptions exist? Because when the dialog manager was first designed, it was determined that special treatment for these messages would make dialog box procedures easier to write, since you wouldn't have to go through the extra step of setting the DWLP_MSGRESULT. Fortunately, since those original days, nobody has added any new exceptions. The added mental complexity of remembering the exceptions outweigh the mental savings of not having to write one line of code ("SetWindowLongPtr(hdlg, DWLP_MSGRESULT, desiredResult)").