I've been talking at a bit of a higher level lately, and today I'm just in the mood to go into some of the deeper debugging aspect that you may run across when looking at Windows Vista application compatibility. (Well, perhaps not deep to some, but for most of us, we haven't spent much time in a low-level debugger lately so the skills have frequently atrophied somewhat.)

When I talk about debugging for application compatibility, I always emphasize this point: the application itself is the same - Windows is now different. So the entry and exit points from Windows take on an exaggerated importance in these debugging sessions. However, internally, we have a bit of an advantage: those entry points are more expressive. You see, we have private symbols to Windows, so the Locals window in WinDbg is quite helpful. Everyone else just gets public symbols. So, those entry points are a little bit more mysterious. Of course, that doesn't mean you are stuck, it just means you have to do a touch more work.

For this exercise, I'm just going to use an application that I know checks the version number in Windows, the MSDN library, and the Debugging Tools for Windows. We're going to get the same information internal folks get without requiring private symbols!

Going in, I know that the application has an error where it tells me that it's expecting an older version of the operating system (in this case, Windows XP). Let's dissect that entry point.

MSDN tells us that the function is implemented in Kernel32.dll, so let's set our breakpoint:

0:000> bp kernel32!GetVersionExW

Now, let's run it up to that breakpoint. While we're here, let's take a look at the function.

0:000> uf kernel32!getversionexw
kernel32!GetVersionExW:
769248d8 8bff         mov     edi,edi
769248da 55           push    ebp
769248db 8bec         mov     ebp,esp
769248dd 56           push    esi
769248de 8b7508       mov     esi,dword ptr [ebp+8]
769248e1 8b06         mov     eax,dword ptr [esi]
769248e3 57           push    edi
769248e4 bf1c010000   mov     edi,11Ch
769248e9 3bc7         cmp     eax,edi
769248eb 740b         je      kernel32!GetVersionExW+0x2b (769248f8)
769248ed 3d14010000   cmp     eax,114h
769248f2 0f858e8d0200 jne     kernel32!GetVersionExW+0x1c (7694d686)
769248f8 56           push    esi
769248f9 ff15f0139076 call    dword ptr [kernel32!_imp__RtlGetVersion (769013f0)]
769248ff 85c0         test    eax,eax
76924901 0f85868d0200 jne     kernel32!GetVersionExW+0x23 (7694d68d)
76924907 393e         cmp     dword ptr [esi],edi
76924909 7409         je      kernel32!GetVersionExW+0x3a (76924914)
7692490b 33c0         xor     eax,eax
7692490d 40           inc     eax
7692490e 5f           pop     edi
7692490f 5e           pop     esi
76924910 5d           pop     ebp
76924911 c20400       ret     4
76924914 a046ef9c76   mov     al,byte ptr [kernel32!BaseRCNumber (769cef46)]
76924919 88861b010000 mov     byte ptr [esi+11Bh],al
7692491f ebea         jmp     kernel32!GetVersionExW+0x45 (7692490b)
7694d686 6a7a         push    7Ah
7694d688 e8eeeeffff   call    kernel32!SetLastError (7694c57b)
7694d68d 33c0         xor     eax,eax
7694d68f e97a72fdff   jmp     kernel32!GetVersionExW+0x25 (7692490e)

We want to esablish the calling convention. If we were using _fastcall, we'd be passing arguments in registers (ECX and EDX), but we don't even use those registers here. We're calling ret 4 to unwind our own stack, so we can't be using a cdecl function (where the caller is responsible for unwinding the stack). We're looking at a stdcall function.

(While you're looking at it - the first instruction - mov edi,edi - is an instruction that does absolutely nothing. It's a 2-byte filler. Why? To enable hot patching. With two byes, we can add a short jump, from which we can long jump to a new implementation of the function.)

Now, we could set up our stack and begin looking at the arguments passed, but in this case, we don't really care much about what was passed. What's interesting is what we return. So, let's get out of this function and back into our main function and have a look.

First, let's look at our return value. Again, looking at MSDN, we see that it returns a BOOL, returning non-zero upon success. This appears in the EAX register. Let's see how we did.

0:000> r eax
eax=00000001

OK, so the function succeeded. However, you're probably far more interested in the LPOSVERSIONINFO argument that the function modified! How do we get at that?

Well, first we need to know where it sits. Because the arguments are passed on the stack, the stack pointer tells us where to look. We look at ESP+8 ESP+4, and we'll begin to find our arguments. And, fortunately, we can use the dt command to format it nicely for us.

0:000> dt OSVERSIONINFO esp+8
DWM_Compositing_Rendering_Dem!OSVERSIONINFO
   +0x000 dwOSVersionInfoSize : 6
   +0x004 dwMajorVersion   : 0
   +0x008 dwMinorVersion   : 0x1771
   +0x00c dwBuildNumber    : 2
   +0x010 dwPlatformId     : 0x650053
   +0x014 szCSDVersion     : [128]  "rvice Pack 1"

0:000> dt OSVERSIONINFO esp+4
DWM_Compositing_Rendering_Dem!OSVERSIONINFO
   +0x000 dwOSVersionInfoSize : 0x114
   +0x004 dwMajorVersion   : 6
   +0x008 dwMinorVersion   : 0
   +0x00c dwBuildNumber    : 0x1771
   +0x010 dwPlatformId     : 2
   +0x014 szCSDVersion     : [128]  "Service Pack 1"

So, this is how the function is coming back to us. And now we can see our first argument. Fortunately for us, this is our only argument, and our job is done.

(You may be scratching your head as to why szCSDVersion seems to be missing the letters Se - I'm doing the same thing. Actually, no more head scratching. I mixed up esp+4 outside with ebp+8 inside - fixed this.)

That wasn't so hard, was it? A little help from MSDN, and we are on our way.

Let's do one more, for good measure. This application happens to display a message box, so let's fast forward to that API call. MSDN tells us that this is found in user32.dll, so let's set our breakpoint.

0:000> bp user32!messageboxw

Now that we're in the function, this time we want to stay in it. The arguments here are interesting going in. Let's take a look at this function:

0:000> uf user32!messageboxw
USER32!MessageBoxW:
76d5d667 8bff            mov     edi,edi
76d5d669 55              push    ebp
76d5d66a 8bec            mov     ebp,esp
76d5d66c 833da89cd67600  cmp     dword ptr [USER32!gfEMIEnable (76d69ca8)],0
76d5d673 7424            je      USER32!MessageBoxW+0x32 (76d5d699)
76d5d675 64a118000000    mov     eax,dword ptr fs:[00000018h]
76d5d67b 6a00            push    0
76d5d67d ff7024          push    dword ptr [eax+24h]
76d5d680 6824a3d676      push    offset USER32!gdwEMIThreadID (76d6a324)
76d5d685 ff150412d076    call    dword ptr [USER32!_imp__InterlockedCompareExchange (76d01204)]
76d5d68b 85c0            test    eax,eax
76d5d68d 750a            jne     USER32!MessageBoxW+0x32 (76d5d699)
76d5d68f c70520a3d67601000000 mov dword ptr [USER32!gpReturnAddr (76d6a320)],1
76d5d699 6a00            push    0
76d5d69b ff7514          push    dword ptr [ebp+14h]
76d5d69e ff7510          push    dword ptr [ebp+10h]
76d5d6a1 ff750c          push    dword ptr [ebp+0Ch]
76d5d6a4 ff7508          push    dword ptr [ebp+8]
76d5d6a7 e849ffffff      call    USER32!MessageBoxExW (76d5d5f5)
76d5d6ac 5d              pop     ebp
76d5d6ad c21000          ret     10h

The first few lines set up our stack frame. We can step through the first 3 lines to set it up ourselves, or we can just use the existing stack pointer (ESP). Since it saves us some time, let's do that.

Looking at MSDN, the first argument is the HWND of the window that owns the message box. This will be a 4-byte HWND value. If we had stepped through our stack frame setup, we'd be starting at EBP+8, but since we haven't called PUSH EBP yet, we're going to be starting at ESP+4. So, our first argument is:

0:000> dc esp+4
0012fdd0  00000000

OK, so it's passing a NULL HWND. On to argument #2: a pointer to the text to display in the message box. We'll just walk 4 more bytes on the stack to grab that pointer:

0:000> ddu esp+8
0012fdd4  00402160 "This application requires Windows XP"

And what are they using as the caption in argument 3?

0:000> ddu esp+c
0012fdd8  00402134 "Unsupported Version"

And our last argument tells us what buttons it's going to support:

0:000> dc esp+10
0012fddc  00000010

Ah - this one is a bit harder to crack, because it's expressed as a UINT instead of a handy #define'd flag. Fortunately, MSDN also tells us where the API is defined (winuser.h), so we can just head there and search for MB_ - they're typically all grouped together. So, let's translate. 10 is MB_ICONHAND (which also happens to be MB_ICONERROR and MB_ICONSTOP if you continue reading the header), and, since we have to have buttons, we must have MB_OK (0x0). So, we must be calling this as MB_ICONHAND | MB_OK.

So, with a little help from the SDK header files, and a lot of help from MSDN, we were able to point our debugger at Windows APIs with public symbols and gather just as much information about arguments as the private symbols would give you.

Updated 2/23/2008: I'm so used to typing EBP+8 inside a method that I typed ESP+8 outside. I corrected this.