Holy cow, I wrote a book!
And then Windows is stuck having to support your buggy code forever.
So many people misdeclare their window procedures (usually by declaring them as __cdecl instead of __stdcall), that the function that dispatches messages to window procedures contains extra protection to detect incorrectly-declared window procedures and perform the appropriate fixup. This is the source of the mysterious 0xdcbaabcd on the stack. The function that dispatches messages to window procedures checks whether this value is on the stack in the correct place. If not, then it checks whether the window procedure popped one dword too much off the stack (if so, it fixes up the stack; I have no idea how this messed up a window procedure could have existed), or whether the window procedure was mistakenly declared as __cdecl instead of __stdcall (if so, it pops the parameters off the stack that the window procedure was supposed to do).
Many DirectX functions use callbacks, and people once again misdeclared their callbacks as __cdecl instead of __stdcall, so the DirectX enumerators have to do special stack cleanup for those bad functions.
I remember there was one program that decided to declare their CreateViewWindow function incorrectly, and somehow they managed to trick the compiler into accepting it!
class BuggyFolder : public IShellFolder ... { ... // wrong function signature! HRESULT CreateViewObject(HWND hwnd) { return S_OK; } }
Not only did they get the function signature wrong, they returned S_OK even though they failed to do anything! I had to add extra code to clean up the stack after calling this function, as well as verify that the return value wasn't a lie.
The function signature required for functions called by rundll32.exe is documented in this Knowledge Base article. That hasn't stopped people from using rundll32 to call random functions that weren't designed to be called by rundll32, like user32 LockWorkStation or user32 ExitWindowsEx.
Let's walk through what happens when you try to use rundll32.exe to call a function like ExitWindowsEx:
The rundll32.exe program parses its command line and calls the ExitWindowsEx function on the assumption that the function is written like this:
void CALLBACK ExitWindowsEx(HWND hwnd, HINSTANCE hinst, LPSTR pszCmdLine, int nCmdShow);
BOOL WINAPI ExitWindowsEx(UINT uFlags, DWORD dwReserved);
However, the function is expecting to see
What happens? The hwnd passed by rundll32.exe gets misinterpreted as uFlags and the hinst gets misinterpreted as dwReserved. Since window handles are pseudorandom, you end up passing random flags to ExitWindowsEx. Maybe today it's EWX_LOGOFF, tomorrow it's EWX_FORCE, the next time it might be EWX_POWEROFF.
Now suppose that the function manages to return. (For example, the exit fails.) The ExitWindowsEx function cleans two parameters off the stack, unaware that it was passed four. The resulting stack is
Okay, enough with the examples; I think you get the point. Here are some questions I'm sure you're asking:
It does. (Well, not the rundll32 one.) But people have gotten into the habit of just inserting the function cast to get the compiler to shut up.
Here's a random example I found:
LRESULT CALLBACK DlgProc(HWND hWnd, UINT Msg, WPARAM wParam, LPARAM lParam);
This is the incorrect function signature for a dialog procedure. The correct signature is
INT_PTR CALLBACK DialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
You start with
DialogBox(hInst, MAKEINTRESOURCE(IDD_CONTROLS_DLG), hWnd, DlgProc);
error C2664: 'DialogBoxParamA' : cannot convert parameter 4 from 'LRESULT (HWND,UINT,WPARAM,LPARAM)' to 'DLGPROC'
DialogBox(hInst, MAKEINTRESOURCE(IDD_CONTROLS_DLG), hWnd, reinterpret_cast<DLGPROC>(DlgProc));
"Aw, come on, who would be so stupid as to insert a cast to make an error go away without actually fixing the error?"
Apparently everyone.
I stumbled across this page that does exactly the same thing, and this one in German which gets not only the return value wrong, but also misdeclares the third and fourth parameters, and this one in Japanese. It's as easy to fix (incorrectly) as 1-2-3.
I'll answer this question tomorrow.