Chris Jackson's Semantic Consonance : Debugging

Windows 7 Debugging Demo: Troubleshooting the Broken Microsoft Time Zone Utility

Chris Jackson — Wed, 16 Dec 2009 20:12:11 GMT

Earlier today, I received this request from a customer:

This (http://www.microsoft.com/downloads/details.aspx?FamilyID=07fb0bd8-f390-458d-a629-6f0258ac7cdf) is a .NET 1.1 app and I have installed .NET 1.1 SP1 on Windows 7 32-bit and all .NET 1.1 SP1 security patches but it doesn't run. Gives me this error: An internal error has occurred: Object reference not set to an instance of an object" and then it quits.

Always being up for a challenge, I decided to have a look and see what wasn’t working. Reproducing the error is pretty straightforward. Just run the app – it’ll gladly crash for you. But the question is, why? And can you fix it?

If you hit this with a debugger, you’ll find the following stack when you hit an AV:

0:000> !dumpstack -ee
Current frame: (MethodDesc 0x2098d28 +0x1c SystemTimeZone.ConvertToTimeZoneInfoStructure)
ChildEBP RetAddr Caller,Callee
0018f4f4 0206d4ab (MethodDesc 0x2098d08 +0x43 SystemTimeZone.ToLocalTime)
0018f55c 0206cfc0 (MethodDesc 0x2098cd8 +0x38 SystemTimeZone.Convert)
0018f574 0206a484 (MethodDesc 0x2e5680 +0x15c frmMain.ShowCurrentTimes_Click)
0018f5d8 02067686 (MethodDesc 0x2e56d0 +0x446 frmMain.frmMain_Load)
0018f61c 02060365 (MethodDesc 0x2e5620 +0x2ed frmMain.Main)

Now, you could take a stab at this with SOS and start pulling out objects, but I didn’t believe that this would be the easiest way to solve the problem. Instead, I decided to reverse engineer the code and have a look. If you take a peek at the structure you’re trying to create from the SystemTimeZone class, you’ll find where the instance of the time zone came from – the code was rampaging through the registry looking for it. Here’s the method in question:

public TimeZoneList()
{
    m_TZHash = new Hashtable();
    m_TZList = new ArrayList();
    RegistryKey key = Registry.LocalMachine.OpenSubKey(@"SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones\");
    string[] subKeyNames = key.GetSubKeyNames();
    Type t = typeof(TZIStructure);
    int num2 = Marshal.SizeOf(t);
    int num3 = subKeyNames.Length - 1;
    for (int i = 0; i <= num3; i++)
    {
        RegistryKey key2 = key.OpenSubKey(subKeyNames[i]);
        SystemTimeZone zone = new SystemTimeZone();
        SystemTimeZone zone2 = zone;
        zone2.DaylightName = StringType.FromObject(key2.GetValue("Dlt"));
        zone2.DisplayName = StringType.FromObject(key2.GetValue("Display"));
        zone2.Index = IntegerType.FromObject(key2.GetValue("Index"));
        zone2.MapId = StringType.FromObject(key2.GetValue("MapID"));
        zone2.StandardName = StringType.FromObject(key2.GetValue("Std"));
        zone2.Name = key2.Name.Substring(key2.Name.LastIndexOf(@"\") + 1);
        zone2 = null;
        byte[] buffer = (byte[]) key2.GetValue("TZI");
        key2.Close();
        if ((buffer != null) && (buffer.Length >= num2))
        {
            GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
            IntPtr ptr = handle.AddrOfPinnedObject();
            zone.TZI = (TZIStructure) Marshal.PtrToStructure(ptr, t);
            handle.Free();
            if (!m_TZHash.Contains(zone.Index))
            {
                m_TZHash.Add(zone.Index, zone);
                m_TZList.Add(zone);
            }
        }
    }
    key.Close();
}

If you spend enough time with the !do command, you can eventually stumble upon the answer, but the easier approach is, again, to stay out of the debugger and leverage your favorite search engine. Here’s a KB article that points out the problem:

http://support.microsoft.com/kb/935369

Relevant information: “The Index registry value does not exist in Windows Vista and in Windows Server 2008.”

Aha. See the source code above? It wants that value, and it’s not there. Root cause found.

Now, on to solutions.

First of all, I wonder if you need a solution at all. If you click on the clock, and then click on Change date and time settings… you will get to the Date and Time control panel. Click on the Additional Clocks tab, and you can add 2 additional clocks for 2 new time zones. If you need more, you can always use Gadgets to fill the rest of your needs. So, for most people, this should provide a built-in solution to the problem – you don’t need the app any more because the OS provides 2 solutions already.

But, if you really really love that particular app, you could always copy that piece of the registry from Windows XP onto your Windows 7 computer, drop it in a new place inside of the registry, and then use the VirtualRegistry shim to redirect that app’s checks of the timezone portion of the registry to the copy you brought over.

Special thanks to my friend Gov for pointing me to the KB article, and for coming up with the VirtualRegistry workaround idea.

Debugging the Missing .NET Tab for Managed Applications in Process Explorer for Windows 7 x64

Chris Jackson — Fri, 28 Aug 2009 08:08:13 GMT

I was reading through some documentation on Sysinternals Process Explorer the other day, and one bit that caught my attention was some detail on the .NET tab. So, I pulled up my trusty managed StockViewer application, and went to go and look at that tab.

Only…

…it wasn’t there.

Fascinating.

So I found another managed code process on my system, and tried to look at it’s tab – the excellent Switcher application.

No joy.

OK, so after telling Mark he didn’t know how to write programs (don’t try this at home) it was time to pull out the debugger. With a little help, because I did ask Mark how he was checking to see whether he should be putting the .NET tab on there. He said he was using both perfmon APIs and the ICorPublish APIs.

So, off to the MSDN documentation to start brewing up a hypothesis, and to be honest it just seemed most likely that the ICorPublish APIs would be the culprit, since they were called per process and perfmon itself was working so those APIs were probably going to be working too. Just a hunch.

I browsed through the APIs, and fortunately there were only two. Of those two, one seemed the most likely to be called for this, so I set a breakpoint on it.

bp mscordbi!CorpubPublish::GetProcess.

Sure enough, it was hit. Good guess.

Of course, this bp was hit rather a lot (I had quite a few processes running at the time), so I didn’t want to have to stop and inspect every call. I wanted to stop and inspect the call when it was looking specifically at the process I expected was managed (Stock Viewer) but procexp didn’t think was managed. So, off to Task Manager (sorry, procexp, but you were frozen at a breakpoint in my debugger) to pull the PID for Stock Viewer, which I then translated to hex for the debugger. Voila – breakpoint set.

1:001> bl

1 e 00000642`ff91302c 0001 (0001) 1:**** mscordbi!CorpubPublish::GetProcess "j @rdx=0x16EC '' ; 'gc' "

I hit this breakpoint, and then stepped out of the function. I then inspected rax to see the return code, and found it was 0x80070057. the 8 already indicates bad news, so I was expecting an error message. It was COR_E_ARGUMENT. Now, that wasn’t particularly helpful this time, since there was only one argument (the pid), so the question is – why didn’t it like to answer this question about this process? It would take some additional stepping through.

So, I did some call tracing, and found that GetProcess was calling through some internal APIs, and one of the interfaces was returning 0x80131c30. It appeared to be another COM call, so it could be a failure hresult, but the utility I use to look up error messages wasn’t telling me anything. The error number was not found.

Along the way, Mark had reached out to another Technical Fellow, so at this point I’m thinking to myself, “holy crap, there had better actually be a bug here and not user error” – and this other Technical Fellow pointed me in the direction of Thomas, a Senior Development Lead for the CLR. And what a useful connection this was. He deciphered the error: it’s returned if there is a bitness difference between the caller and the target.

Aha.

Both of the managed apps happened to be specifically flagged as x86 apps. So, to do a quick check that this was the only thing wrong, I only needed to create an AnyCPU managed application, and see if ProcExp would display a .NET tab for it. Indeed it does!

So, mystery solved. ProcExp can’t (today) peer into .NET information for managed processes that are 32-bit on 64-bit Windows (the default compiler setting is to create AnyCPU binaries). The wheels are already churning to figure out how to close up this edge case. I just got unlucky with the two managed apps I happened to be running.

The Case of the SUA Missing Log File

Chris Jackson — Thu, 04 Jun 2009 07:16:31 GMT

Since I’m using Mark’s tools, I figure I may as well steal his blog title scheme…

I had a customer come to me with a question on Standard User Analyzer (SUA), looking for an explanation for why it was coming back with the following error:

Failed to load log file C:\Users\…\AppData\Local\Temp\sua
No (valid) log file is found.

I asked him to try opening it up in Application Verifier, which he was able to do successfully.

So, now we knew that the problem was somewhere between collecting the data and displaying it – we’d narrowed down the surface area rather significantly. What happens in that time? Well, SUA is kind enough to tell you. First, it clears existing logs:

Executing: cmd.exe /c "del /q "C:\Users\…\AppData\Local\Temp\sua""
Returned : 0

Then, we export the logs we just created:

Executing: "C:\Program Files (x86)\Microsoft Application Compatibility Toolkit 5\Standard User Analyzer\SUAnalyzerSrv.exe" exportlogs "C:\Users\…\AppData\Local\Temp\sua" "(symbol file directory)"
Returned : 0

Finally, we view it, but this was the step which was failing. We could browse to the temp directory it was using and discover that it was, indeed, clear, so now we know that the command to export logs was our most likely suspect. But it was returning 0, which presumably meant success, so we had very little to go on.

Fortunately, I was able to find a repro relatively quickly. And, when in doubt, use Process Monitor.

I didn’t even have to go any further than the process tree. I found the call to SUAnalyzerSrv.exe, and it made a call to appverif.exe – Application Verifier. Here was the arguments it was passing:

appverif.exe -export log -for AppVerifier Bug Generator.exe -with To=C:\Users\…\AppData\Local\Temp\sua\AppVerifier Bug Generator.exe.0.xml Log=0 Symbols=(symbol file directory)

It’s when you see this that you probably notice something: the name of the binary … has a space in it.

I renamed the binary to remove the spaces, and it worked. No problems importing. So, I filed a bug against SUA, and it’s going to be fixed for the next release. In the interim, if SUA isn’t working as well as you’d like, well, the EXTREMELY hacky workaround may be to rename the binary…

The Secret to Power App Compat Debugging

Chris Jackson — Thu, 16 Apr 2009 22:57:44 GMT

If you come to me for advice about how to become a debugger, chances are that I’m going to give you a couple of must-read reference books, an then tell you to start paying attention. Because, unless you’re drastically more lucky than I am, stuff is probably breaking on you all the time. While a lot of problems just go away, if you let it just go away, you’ve just squandered an opportunity to debug something.

The only way to become a master at debugging is to practice. A lot.

But there’s a level even above master debugger – being somebody able to get to the bottom of most every issue (eventually – hey, even for the best, it can take time, lots and lots of time). What’s that level? The Power Debugger. Somebody who dispenses with the need for time, and just fixes things quickly because there is no alternative.

How do you reach that level?

Simple.

Have a 4-year-old.

Four year olds don’t care about the challenges application compatibility. They just want their games to work, and they are quite vocal when they don’t. They look at you thinking, “why can’t you fix this? Aren’t you supposed to be able to do this? Can’t you see this is bothering me?” Oh, and then the cry. And yell. And cry. Great.

Fortunately, the solution for pre-school games is typically rather easy. For reasons that are completely beyond comprehension to me, it turns out that many developers of games for pre-schoolers assume that I want my 4-year-old to be an administrator on my computer.

RunAsAdmin, and we were on our way. Optimal solution? No. But you have to power debug with a 4-year-old.

So that leads me to this:

If you develop commercial games for pre-schoolers, I will help you debug your application so it works for standard users for free. But you have to promise to run your developer workstation either as a standard user, or as a protected administrator (that’s right, turn UAC back on) on Windows Vista or later.

Every application should run as a standard user. But games for children should have run for standard users even on XP.

Putting It All Together: Using My Ramblings to Solve Real Problems

Chris Jackson — Tue, 01 Jul 2008 21:35:59 GMT

I send out a lot of links to my articles in response to questions that come up, but the other day I had a chance to use a pile of them to solve a fairly complicated problem end to end. So, I figured I'd share how we can piece together all of this knowledge to solve a more sophisticated problem.

Let's begin at the beginning. The CorrectFilePaths shim wasn't working. We were trying to redirect c:\somedata.txt to %userappdata%\somedata.txt, but it wasn't working. Why not? We weren't sure.

So, we started out by turning on shim debug spew to see if the shim was being wired up. Indeed it was - we could see it picking up the shim when we launched the process. We could also look at the process in Process Monitor and see in the stack that the call to write c:\somedata.txt was passing through AcLayers!NS_CorrectFilePaths::APIHook_CreateFileA. So, we knew that the shim wasn't improperly applied.

Our next hypothesis, then, was that the shim's command line was not configured properly. I tried copying and pasting from Process Monitor in case there was a copy error, but no such luck. So, it was time to go in with a debugger and see what was happening.

Now, where do we set the debugger breakpoint? Well, why not when we enter the shim? So, I set a bp on AcLayers!NS_CorrectFilePaths::APIHook_CreateFileA and took a look at the arguments we were passing to this function. What did I see?

\somedata.txt

Indeed, it never specified the C drive - just the root of whatever drive you happened to be running on. And, since CorrectFilePaths just uses a literal string match, the c: was causing the match to fail.

Replace that argument with \somedata.txt;%userappdata\somedata.txt and everything worked fine!

So, using a bit of knowledge of Process Monitor, a bit of knowledge on the Debugging Tools for Windows, and knowledge from this blog, we could actually solve a real-world problem. And that's what I like to see. Let me know if you are finding gaps that nobody is talking about yet, and we'll see if we can get them filled here!

Debugging Without a Debugger: The Magic of wercon and a Web Browser

Chris Jackson — Tue, 17 Jun 2008 05:42:42 GMT

Last week, I was at TechEd 2008: IT Professionals in Orlando, FL. (More on that later.) I had the opportunity to talk to lots of people, and it is always a great learning experience for me.

Along the way, I even had a chance to solve an application compatibility problem, using nothing but a bit of string, 2 paper cups, and a paperclip.

Here's the story:

I was hanging around the Application Compatibility and Deployment booth, and a customer walks up frustrated that Adobe Photoshop Elements isn't working. I asked him which version, and he replied that it was the latest version. Which I know works with Windows Vista. Now, he just wants to know who to blame, because he's on a mission now. He promised his family he'd edit some photos that week, and now this crash was going to keep him from making good on his promise.

He doesn't have debugging tools on the system, so I figured I'd try to see what I could see without having to load one up (since he hadn't set up an Internet connection to the conference WiFi and was hoping to avoid that). So, I popped up wercon. This pointed out the Bucket ID (which is how we reference the crash in our error reporting system) and the Faulting Module. I didn't recognize the faulting module, so I hopped on to the corporate network to see if I could find out if we knew anything more, or had a stack trace (again, to avoid dropping debugging tools on his system). Alas, we didn't have any good data there.

So I popped open my web browser, and had a go with the faulting module name, to see what I could find out about it. capm3k.dll. A Canon DLL. I asked him if he had a Canon printer at home. He looked a bit surprised, and said yes. I asked him to uninstall the software for it.

A minute later, and he's back in business. Neither Adobe or Microsoft were bringing the app down, it was printer drivers! So, he was going to hunt down some updated ones when he got home, but for now he could keep his family happy!

It just goes to show - you don't need to be a master debugger to sniff out a problem. You can find and fix problems with nothing but wercon and a web browser. Nice.

Windows Vista SP1 Broke my Internet Explorer Compatibility Test Tool!

Chris Jackson — Thu, 20 Mar 2008 03:28:00 GMT

...but I fixed it. And I thought you might like to know how to fix it if you run into the issue.

So first, the problem. I was trying to launch IECTT and found this MessageBox:

---------------------------

Test Tool Error

---------------------------

The file size exceeds the limit allowed and cannot be saved

---------------------------

Hrm. This could be ... um ... more helpful. I have no idea what it's talking about.

Now, the fix:

Manually set the key HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\Internet Explorer\Main\FeatureControl\Feature_Enable_Compat_Logging\iexplore.exe = (DWORD) 1. Then, browse to some web sites which generate some issues (my trusty live.com homepage did just fine for me).

The problem exists when the Internet Explorer event log is empty, so we just need some way to make it non-empty.

Now, let's dig into the investigation, because it's always instructive to go back over what we found and try to understand what we can learn from it.

First, I did a quick stack trace. Here's what I saw:

ChildEBP RetAddr

002bedec 79f071ac KERNEL32!RaiseException
002bee4c 79f0a629 mscorwks!RaiseTheExceptionInternalOnly
002bef10 7a71713e mscorwks!JIT_Throw
002bef5c 7a659780 System_ni!System.Diagnostics.EventLog.get_OldestEntryNumber
002bef5c 7a6597ff System_ni!System.Diagnostics.EventLog.StartListening
002bef5c 7a655d33 System_ni!System.Diagnostics.EventLog.StartRaisingEvents
002bef5c 03cd6018 System_ni!System.Diagnostics.EventLog.set_EnableRaisingEvents
002befac 03cd52d0 TestTool!Microsoft.ApplicationExperience.ApplicationCompatibilityToolkit.TestTool.EventLogMonitor..ctor
002befac 03cd040c TestTool!Microsoft.ApplicationExperience.ApplicationCompatibilityToolkit.TestTool.MainForm.InitializeDataProviders
002befe0 03cd00b9 TestTool!Microsoft.ApplicationExperience.ApplicationCompatibilityToolkit.TestTool.MainForm..ctor
002befe0 79e7c74b TestTool!Microsoft.ApplicationExperience.ApplicationCompatibilityToolkit.TestTool.MainForm.Main
002beff0 79e7c6cc mscorwks!CallDescrWorker
002bf070 79e7c8e1 mscorwks!CallDescrWorkerWithHandler
002bf1b4 79e7c783 mscorwks!MethodDesc::CallDescr
002bf1d0 79e7c90d mscorwks!MethodDesc::CallTargetWorker
002bf1e4 79eefb9e mscorwks!MethodDescCallSite::CallWithValueTypes_RetArgSlot
002bf348 79eef830 mscorwks!ClassLoader::RunMain
002bf5b0 79ef01da mscorwks!Assembly::ExecuteMainMethod
002bfa80 79fb9793 mscorwks!SystemDomain::ExecuteMainMethod
002bfad0 79fb96df mscorwks!ExecuteEXE

OK, let's take a look at the OldestEntryNumber to see what it could be doing:

.method private hidebysig specialname instance int32 get_OldestEntryNumber() cil managed
{
    .maxstack 3
    .locals (
        [0] int32[] numArray,
        [1] bool flag,
        [2] int32 num)
    L_0000: ldarg.0
    L_0001: call instance bool System.Diagnostics.EventLog::get_IsOpenForRead()
    L_0006: brtrue.s L_000e
    L_0008: ldarg.0
    L_0009: call instance void System.Diagnostics.EventLog::OpenForRead()
    L_000e: ldc.i4.1
    L_000f: newarr int32
    L_0014: stloc.0
    L_0015: ldarg.0
    L_0016: ldarg.0
    L_0017: ldfld native int System.Diagnostics.EventLog::readHandle
    L_001c: newobj instance void [mscorlib]System.Runtime.InteropServices.HandleRef::.ctor(object, native int)
    L_0021: ldloc.0
    L_0022: call bool Microsoft.Win32.UnsafeNativeMethods::GetOldestEventLogRecord(valuetype [mscorlib]System.Runtime.InteropServices.HandleRef, int32[])
    L_0027: stloc.1
    L_0028: ldloc.1
    L_0029: brtrue.s L_0031
    L_002b: call class System.ComponentModel.Win32Exception System.Diagnostics.EventLog::CreateSafeWin32Exception()
    L_0030: throw
    L_0031: ldloc.0
    L_0032: ldc.i4.0
    L_0033: ldelem.i4
    L_0034: stloc.2
    L_0035: ldloc.2
    L_0036: brtrue.s L_003a
    L_0038: ldc.i4.1
    L_0039: stloc.2
    L_003a: ldloc.2
    L_003b: ret
}

Interesting - this is just a p/invoke into the native method advapi32!GetOldestEventLogRecord. If we debug into this guy, we see that this method calls advapi32!BaseSetLastNTError(-1073739516). This is, of course, STATUS_FILE_TOO_LARGE, which is what we saw in the first place. Now, we have an API returning a different result than it did before, that API has an owner, and we can contact that owner and have him take a look.

So, our debugging job is done. Remember, you want to look for opportunities to debug and investigate to sharpen your skills, but you also want to know when to stop. At this point, we have an owner, and we have a workaround, so we can wipe our hands and move on to the next problem.

Updated 20-Mar-2008

If you're having problems getting the per-user policies to take hold, you can try HKLM\Software\Microsoft\Internet Explorer\Main\FeatureControl\Feature_Enable_Compat_Logging instead of HKCU. I had to do this on one configuration.

Discovering the Arguments Passed to Windows API Functions with Public Symbols

Chris Jackson — Sat, 23 Feb 2008 08:21:37 GMT

I've been talking at a bit of a higher level lately, and today I'm just in the mood to go into some of the deeper debugging aspect that you may run across when looking at Windows Vista application compatibility. (Well, perhaps not deep to some, but for most of us, we haven't spent much time in a low-level debugger lately so the skills have frequently atrophied somewhat.)

When I talk about debugging for application compatibility, I always emphasize this point: the application itself is the same - Windows is now different. So the entry and exit points from Windows take on an exaggerated importance in these debugging sessions. However, internally, we have a bit of an advantage: those entry points are more expressive. You see, we have private symbols to Windows, so the Locals window in WinDbg is quite helpful. Everyone else just gets public symbols. So, those entry points are a little bit more mysterious. Of course, that doesn't mean you are stuck, it just means you have to do a touch more work.

For this exercise, I'm just going to use an application that I know checks the version number in Windows, the MSDN library, and the Debugging Tools for Windows. We're going to get the same information internal folks get without requiring private symbols!

Going in, I know that the application has an error where it tells me that it's expecting an older version of the operating system (in this case, Windows XP). Let's dissect that entry point.

MSDN tells us that the function is implemented in Kernel32.dll, so let's set our breakpoint:

0:000> bp kernel32!GetVersionExW

Now, let's run it up to that breakpoint. While we're here, let's take a look at the function.

0:000> uf kernel32!getversionexw
kernel32!GetVersionExW:
769248d8 8bff         mov     edi,edi
769248da 55           push    ebp
769248db 8bec         mov     ebp,esp
769248dd 56           push    esi
769248de 8b7508       mov     esi,dword ptr [ebp+8]
769248e1 8b06         mov     eax,dword ptr [esi]
769248e3 57           push    edi
769248e4 bf1c010000   mov     edi,11Ch
769248e9 3bc7         cmp     eax,edi
769248eb 740b         je      kernel32!GetVersionExW+0x2b (769248f8)
769248ed 3d14010000   cmp     eax,114h
769248f2 0f858e8d0200 jne     kernel32!GetVersionExW+0x1c (7694d686)
769248f8 56           push    esi
769248f9 ff15f0139076 call    dword ptr [kernel32!_imp__RtlGetVersion (769013f0)]
769248ff 85c0         test    eax,eax
76924901 0f85868d0200 jne     kernel32!GetVersionExW+0x23 (7694d68d)
76924907 393e         cmp     dword ptr [esi],edi
76924909 7409         je      kernel32!GetVersionExW+0x3a (76924914)
7692490b 33c0         xor     eax,eax
7692490d 40           inc     eax
7692490e 5f           pop     edi
7692490f 5e           pop     esi
76924910 5d           pop     ebp
76924911 c20400       ret     4
76924914 a046ef9c76   mov     al,byte ptr [kernel32!BaseRCNumber (769cef46)]
76924919 88861b010000 mov     byte ptr [esi+11Bh],al
7692491f ebea         jmp     kernel32!GetVersionExW+0x45 (7692490b)
7694d686 6a7a         push    7Ah
7694d688 e8eeeeffff   call    kernel32!SetLastError (7694c57b)
7694d68d 33c0         xor     eax,eax
7694d68f e97a72fdff   jmp     kernel32!GetVersionExW+0x25 (7692490e)

We want to esablish the calling convention. If we were using _fastcall, we'd be passing arguments in registers (ECX and EDX), but we don't even use those registers here. We're calling ret 4 to unwind our own stack, so we can't be using a cdecl function (where the caller is responsible for unwinding the stack). We're looking at a stdcall function.

(While you're looking at it - the first instruction - mov edi,edi - is an instruction that does absolutely nothing. It's a 2-byte filler. Why? To enable hot patching. With two byes, we can add a short jump, from which we can long jump to a new implementation of the function.)

Now, we could set up our stack and begin looking at the arguments passed, but in this case, we don't really care much about what was passed. What's interesting is what we return. So, let's get out of this function and back into our main function and have a look.

First, let's look at our return value. Again, looking at MSDN, we see that it returns a BOOL, returning non-zero upon success. This appears in the EAX register. Let's see how we did.

0:000> r eax
eax=00000001

OK, so the function succeeded. However, you're probably far more interested in the LPOSVERSIONINFO argument that the function modified! How do we get at that?

Well, first we need to know where it sits. Because the arguments are passed on the stack, the stack pointer tells us where to look. We look at ~~ESP+8~~ ESP+4, and we'll begin to find our arguments. And, fortunately, we can use the dt command to format it nicely for us.

0:000> dt OSVERSIONINFO esp+8
DWM_Compositing_Rendering_Dem!OSVERSIONINFO
   +0x000 dwOSVersionInfoSize : 6
   +0x004 dwMajorVersion   : 0
   +0x008 dwMinorVersion   : 0x1771
   +0x00c dwBuildNumber    : 2
   +0x010 dwPlatformId     : 0x650053
   +0x014 szCSDVersion     : [128] "rvice Pack 1"

0:000> dt OSVERSIONINFO esp+4
DWM_Compositing_Rendering_Dem!OSVERSIONINFO
   +0x000 dwOSVersionInfoSize : 0x114
   +0x004 dwMajorVersion   : 6
   +0x008 dwMinorVersion   : 0
   +0x00c dwBuildNumber    : 0x1771
   +0x010 dwPlatformId     : 2
   +0x014 szCSDVersion     : [128] "Service Pack 1"

So, this is how the function is coming back to us. And now we can see our first argument. Fortunately for us, this is our only argument, and our job is done.

(You may be scratching your head as to why szCSDVersion seems to be missing the letters Se - I'm doing the same thing. Actually, no more head scratching. I mixed up esp+4 outside with ebp+8 inside - fixed this.)

That wasn't so hard, was it? A little help from MSDN, and we are on our way.

Let's do one more, for good measure. This application happens to display a message box, so let's fast forward to that API call. MSDN tells us that this is found in user32.dll, so let's set our breakpoint.

0:000> bp user32!messageboxw

Now that we're in the function, this time we want to stay in it. The arguments here are interesting going in. Let's take a look at this function:

0:000> uf user32!messageboxw
USER32!MessageBoxW:
76d5d667 8bff            mov     edi,edi
76d5d669 55              push    ebp
76d5d66a 8bec            mov     ebp,esp
76d5d66c 833da89cd67600 cmp     dword ptr [USER32!gfEMIEnable (76d69ca8)],0
76d5d673 7424            je      USER32!MessageBoxW+0x32 (76d5d699)
76d5d675 64a118000000    mov     eax,dword ptr fs:[00000018h]
76d5d67b 6a00            push    0
76d5d67d ff7024          push    dword ptr [eax+24h]
76d5d680 6824a3d676      push    offset USER32!gdwEMIThreadID (76d6a324)
76d5d685 ff150412d076    call    dword ptr [USER32!_imp__InterlockedCompareExchange (76d01204)]
76d5d68b 85c0            test    eax,eax
76d5d68d 750a            jne     USER32!MessageBoxW+0x32 (76d5d699)
76d5d68f c70520a3d67601000000 mov dword ptr [USER32!gpReturnAddr (76d6a320)],1
76d5d699 6a00            push    0
76d5d69b ff7514          push    dword ptr [ebp+14h]
76d5d69e ff7510          push    dword ptr [ebp+10h]
76d5d6a1 ff750c          push    dword ptr [ebp+0Ch]
76d5d6a4 ff7508          push    dword ptr [ebp+8]
76d5d6a7 e849ffffff      call    USER32!MessageBoxExW (76d5d5f5)
76d5d6ac 5d              pop     ebp
76d5d6ad c21000          ret     10h

The first few lines set up our stack frame. We can step through the first 3 lines to set it up ourselves, or we can just use the existing stack pointer (ESP). Since it saves us some time, let's do that.

Looking at MSDN, the first argument is the HWND of the window that owns the message box. This will be a 4-byte HWND value. If we had stepped through our stack frame setup, we'd be starting at EBP+8, but since we haven't called PUSH EBP yet, we're going to be starting at ESP+4. So, our first argument is:

0:000> dc esp+4
0012fdd0 00000000

OK, so it's passing a NULL HWND. On to argument #2: a pointer to the text to display in the message box. We'll just walk 4 more bytes on the stack to grab that pointer:

0:000> ddu esp+8
0012fdd4 00402160 "This application requires Windows XP"

And what are they using as the caption in argument 3?

0:000> ddu esp+c
0012fdd8 00402134 "Unsupported Version"

And our last argument tells us what buttons it's going to support:

0:000> dc esp+10
0012fddc 00000010

Ah - this one is a bit harder to crack, because it's expressed as a UINT instead of a handy #define'd flag. Fortunately, MSDN also tells us where the API is defined (winuser.h), so we can just head there and search for MB_ - they're typically all grouped together. So, let's translate. 10 is MB_ICONHAND (which also happens to be MB_ICONERROR and MB_ICONSTOP if you continue reading the header), and, since we have to have buttons, we must have MB_OK (0x0). So, we must be calling this as MB_ICONHAND | MB_OK.

So, with a little help from the SDK header files, and a lot of help from MSDN, we were able to point our debugger at Windows APIs with public symbols and gather just as much information about arguments as the private symbols would give you.

Updated 2/23/2008: I'm so used to typing EBP+8 inside a method that I typed ESP+8 outside. I corrected this.

Channel 9 Video Now Live

Chris Jackson — Tue, 29 Jan 2008 23:22:06 GMT

A couple of weeks ago, I had a chance to sit down with Gov Maharaj (whose name rhymes with Orange), a developer in the Windows Fundamentals Dev team, and chat about debugging and mitigating application compatibility issues. You can view our discussion on Channel 9: http://channel9.msdn.com/Showpost.aspx?postid=377306.

Investigating Windows Vista Compatibility Issues by Decompiling VB Code: Don't Be Afraid to Cheat!

Chris Jackson — Fri, 07 Dec 2007 17:53:58 GMT

So, the other day I was investigating a visual issue in an application. On Windows 2000, the background of the form was white. On Windows Vista, in Windows Classic mode, the background of the form was white. But, for whatever reason, in either Windows Vista Aero or Windows Vista Basic, the background of the form was black. And it was a little hard to read that way.

The first thing I did was try to find something out about the application, but in this case is was an in-house application. The best I could do was to poke around the application and gather some facts so I could form my hypothesis on what could be going wrong, and how I could fix it.

A quick glance at the output of lm in the debugger revealed that msvbvm50.dll was loaded. OK, so we have a Windows 2000 application built on Visual Basic 5.0. This strongly suggested that the application wasn't leveraging visual styles and the functions in uxtheme.dll to pick up colors (which also meant that the FakeLunaTheme shim probably wouldn't do any good, and in fact it didn't - I tried anyway).

So then I started going down the road of setting breakpoints on functions such as GetSysColor to see what is going on, and then I was attempting to mentally translate the code from ASM to VB5. Oh, and I was doing this with public symbols too, which made it even more fun.

Then, I got a clever idea. What if I didn't have to do the translation in my head? I mean, Lutz Roeder's awesome Reflector for .NET is unbelievably useful for prying secrets from managed code, I wondered if anybody had ever done the same for VB5/6 code? It seemed like it could be possible...?

Some quick web searching later, and I came up with VBReFormer. Ah - and they have a free edition! Let's give that a go.

Literally 2 seconds later, I was at the form, looking at properties, and the answer was right in front of my face. The developer has set the form background color to be the system color for Title Bar Text. (Seriously.) And, alas, we don't have a shim for that one. But we could just change the title bar text, the easiest way being to switch to Windows Classic mode when using this app. Or, just put up with the fact that the black background didn't look as nice (at least the text was blue, so it wasn't unreadable as it would have been if it were black text).

And this is yet another example of what I am always preaching. There is a certain geek pride in going after a deep understanding of what's going on. But there are no prizes for spending the most time in a debugger. In fact, it's a great way to make time vanish if you aren't careful. Use the tool to get the answer to your question - prove or disprove a hypothesis - and then get out. If there is an easier tool, use it. If you think "wow, I bet this tool would be awesome if it existed" then poke around - it very well may.

Most of all, don't be afraid to cheat. Nobody is counting the number of times you typed bp in a debugger, or how quickly you can ferret out the calling convention to sort out arguments with public symbols. But they do tend to notice when you fix things.

Oh, and kudos to the VBReFormer team for saving me a boatload of time. I don't know if there is a better tool, I'm not an expert in that space and this is the first and only one I looked at. But I was thrilled that I didn't have to hand over that portion of my life to a debugger.

Debugging a Per-User Installation Error on Windows Vista

Chris Jackson — Thu, 25 Oct 2007 01:32:30 GMT

Yesterday, a friend asked for some assistance debugging an issue with Windows Installer. It was one of those situations where you've looked at the problem so closely, so many times, that you just needed another set of eyes. (They're close to shipping, so they've been doing nothing but look at bugs, probably for a while.)

Some background:

What they're doing is a rather interesting use of Windows Installer. They're using it to drop a setup.exe and then run it. So, they have a custom action of type 0xC02, which is a deferred custom action, promoted to not use impersonation, which runs as an exe. This custom action spawns another version of itself (with different arguments), and then the process exits so the installer can complete. The second version of this custom action then wants to uninstall the msi, as well as launch the setup.exe. So, the msi is just a way to package up the setup.exe.

And, with UAC, it was breaking.

Now, with UAC issues, you typically think of access denied. But we didn't expect that at all here. We had promoted the custom action, so it should be running as local system. How could local system be receiving an access denied error? Perhaps we weren't being promoted for some reason?

So, I took a peek with Process Explorer. The processes had the privileges I expected, Local System. Here's the chain of process creation and token inheritance:

So, I could pretty much rule out a permission denied issue - the custom action was being promoted as I expected it to be.

My next step was to turn on MSI logging and see what was going on. And, as it turned out, it wasn't getting access denied (which I already expected), but instead was getting a 1605 error:

MSI (s) (5C:F4) [20:42:05:761]: MainEngineThread is returning 1605

Now, 1605 is ERROR_UNKNOWN_PRODUCT. Basically, it means you're trying to uninstall something that isn't there. This seemed odd, since the time stamp indicated that the previous installation had completed, and the rules of Windows Installer tell us that we couldn't possibly have two concurrent instances of the installer running anyway. So, what happened?

Take a look at the colors above. I originally invoked msiexec.exe from a protected admin account. I then attempted to uninstall that product from the Local System account. Aha!

You see, this msi hadn't set the ALLUSERS property. That meant that we were running a per-user install. When Local System tried to uninstall, it couldn't, because it had never been installed for *that* user, it had been installed for the Protected Admin user!

When I added the ALLUSERS property to the property table, and then set the value to 1, then everything worked as expected. And this little repackaged installer was working as expected.

Close one more bug. And UAC gets a free pass on this one.

Improving Your Debugging Skills, and Knowing When to Stop Debugging

Chris Jackson — Tue, 16 Oct 2007 23:52:52 GMT

When I work with folks who are either new to debugging, or else want to sharpen up their chops (perhaps they have gotten rusty by not having to do so for a long time), one of the things I tell them to do is to watch out for things that don't do what you expect, and then figure out why. I mean, does everything on your computer doing exactly what you would want it to do, all of the time?

For example, I was working with a friend who doesn't have as many chances to debug things as he would like. Concomitantly, he was struggling with MS07-052, which (for the first couple of days) would cycle in an infinite loop for anybody who didn't install Crystal Reports as part of Visual Studio. It would install, succeed, and then detect that it needed to install again. He was about to ping a discussion alias about it. So I looked at him, and I said, "why don't you debug it?"

It's that kind of mindset that gets you practice debugging. Now, in this case, the cause was known, so he wasn't going to be helping anybody out. But in many cases, the cause may not be understood (you may be holding a repro that is really hard to get) - take advantage of that opportunity. And, even if you are doing some redundant debugging, if you learn something new and keep your skills sharp, would you begrudge yourself the time? If not, then perhaps...

One of the blogs I love to follow is Mark Russinovich's blog. He takes this exact approach over and over, outlining exactly how he uses tools to follow that investigation. And, obviously, learning how to better use the tools is one great takeaway.

But I think there's more to it than sharpening your skills with tools. Rather, it's a view into the mindset of somebody who is so natively curious. He sees a problem. It annoys him. So he figures out why it's happening, so he can get it to stop. By doing that alone, you can improve your skills exponentially, and Mark is kind enough to share his techniques for how to do that investigation.

One thing that I noticed that caught my attention even more than the great example of when to start debugging and how to do it was his awesome example of when to stop. In Mark's latest entry, The Case of the Frozen Clock Gadget, he proceeds through the investigation, and in the end determines the API that is causing a memory leak, after which he ... (drum roll) ... stops debugging it and files the bug. He doesn't dig into the implementation itself to see where the memory leak is happening. He never gets to the exact cause. But, what he has done is find somebody to assign the bug to, and then he moved on. There will be a single person who owns that API, and he can hand that person the symptoms and a repro. He can now go on to bigger and better things, and that person can fix up the code.

That, my friends, is a hard skill to learn. I mean, look at what I just said. You have to be curious enough to delve deeply into a problem at the first sign of trouble. You have to be creative enough to dig in and figure out what is going on. But then, before you see the actual culprit, you stop. You have to be able to turn off your curiosity, and know that your job is just to find an owner for the bug, and move on. (Don't worry, there are still plenty more bugs out there.) For those who are so natively curious as to get into the nitty-gritty of debugging, this can be a hard lesson to learn.

I see this all of the time working with enterprise customers who are testing application compatibility for a migration to Windows Vista, and one important step is knowing when to stop debugging. The answer, of course, can vary:

If the application is one that you will never run without full vendor support, stop debugging before you start. If you're not going to ever bother fixing it, then knowing what's wrong won't help you, will it? This one is a hard one to learn. Many times, I have said, "I can debug this, figure out if I can shim it, and then test it with the shims applied, but if I do, are you going to run an unsupported, shimmed application? If not, then we're just wasting our time."
If the application vendor isn't being responsive, or is playing the blame game ("It's the fault of xxx software that I depend on - if they'd just fix their bugs, then my software would run great - go talk to them.") then some debugging time can help you end the circle of blame and keep you talking to the people who can really help you. Remember,you're their customer, and if you come in saying, "I know it's you, you're doing this, this is illegal as documented here, and I need you to fix your code please" then you can circumvent the blame game and start getting some results. Note that, once you determine the culprit and can prove it, your debugging is done. Remember, you aren't going to change the code - the vendor is. Whichever one it turns out to be.
If you don't need support, and are willing to shim the third party application, then you need to debug until you can figure out which shim you should use. Once you know this, and you shim it and the tests pass, then you are done. If you prove that it can't be fixed using a shim, and you need to code changed, then you are done.
If the application is developed in-house, then you need to debug it until you know which person owns the component that is breaking. Then, assign the bug to them and let them finish debugging - your work is done.

Believe it or not, knowing when to stop is one of the hardest part of debugging applications. You need to encourage your innate curiosity to determine just enough, but then be able to turn it off. It's simply the only way to get everything done. And, believe me, by pointing somebody in the right direction, your help can be beyond invaluable. Happy debugging!