In Windows 7, we've added some techniques and tools that assist with troubleshooting hangs involving multiple processes (if you're interested in the evolutionary history, check out Ryan's "Let There Be Hangs" series, where he describes how hang reporting has changed from Windows XP on.) With these improvements come some new concepts, terms, and techniques, which I will attempt to convey so that, after reading this, you can fire up the debugger and understand a bit more about what’s going on.


Defined Terms and Concepts

XPROC cab

The “X” stand for cross and the “PROC” stands for process. So in other words, we’re talking about a cross process dump. And what is that exactly? Well, when an app is hung due to cross-process communication, we capture a dump of the hung process and, when possible, the process determined to be at the end of the “wait chain” in the inter-process communication. We include both dumps, as well as other analysis related information, in the cab submitted with the error report. XPROC dumps were introduced in Vista, but there were cases where the state of the hang would be lost before the heap dump was captured.

Process Reflection

At its core, Process Reflection is used to make a clone of an existing process. This clone, also referred to as the reflected process, contains a single thread and a copy of the original process’ address space. Mark Russinovich talks more about it here (31:00)

In the case of hang reporting, when a chain of processes is involved in the hang, we use Process Reflection to create a clone of the process determined to be at the end of the wait chain by virtue of the GetThreadWaitChain API. A heap dump is taken of the clone instead of the original. Cloning takes way less time than generating a heap dump file, so using this technique, we increase our chances of capturing the hung state, than if we were just create a dump of the original process. Also, it’s worth pointing out that our single thread in the clone does not execute during hang reporting, so it is simply parked and left idle. Here’s what its stack looks like today:

ChildEBP RetAddr
0028fdbc 77a97b8c ntdll!KiFastSystemCallRet
0028fdc0 77af02ef ntdll!ZwWaitForSingleObject+0xc
0028fe38 75ef0239 ntdll!RtlpProcessReflectionStartup+0x1bf
0028fe44 77ab1ec2 kernel32!BaseThreadInitThunk+0xe
0028fe84 77ab1e95 ntdll!__RtlUserThreadStart+0x70
0028fe9c 00000000 ntdll!_RtlUserThreadStart+0x1b

When it comes to debugging analysis, some things aren’t readily apparent with the reflected process. For instance, what about the original threads? Does the handle data conflict with that of the original process? After all, the reflected process is a process in its own right. The short answer is the debugger handles all of this for you. As they say, “the devil is in the details” so we’ll get into how the debugger does this shortly. But first, it’s important to understand the files contained in an AppHangXProcB1 bucket’s cab.


On to the Fun Stuff...

AppHangXProcB1 Cab Contents

So to recap:

A hang has occurred. We get a heap dump of the hung process. Using GetThreadWaitChain(), we discover the process at the end of the wait chain and clone it using ProcessRelection.
Next we grab a heap dump of the clone. But, we also have to deal with the fact that the thread and handle data from the original aren’t copied to the clone. So, in addition to this heap dump, we also capture a mini dump of the original process that has the thread and handle data. The debugger knows how to “stitch” both files together into a single, coherent target.

Let’s take a look at an example cab:

File name                      File size     Date      Time   Attrs
-----------------------------  ---------- ---------- -------- -----
   iexplore.exe.xml                  5496 2010/07/14 16:25:22  ----
   iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini      224 2010/07/14 16:25:22  ----
   WER57FB.tmp.ref                1596362 2010/07/14 16:25:22  ----
   iexplore.exe.6e79.wxhu.dmp    150380775 2010/07/14 16:25:22  ----
   memory.hdmp                   40343241 2010/07/14 16:26:18  ----
   WERInternalMetadata.xml           3226 2010/07/14 16:26:30  ----
   AppCompat.txt                    47354 2010/07/14 16:26:30  ----


Look at the contents of the dbgcfg.ini file and you'll see something like this:

[UserModeDump]
default=WER57FB.tmp.ref
memory=iexplore.exe.6e79.wxhu.dmp
module=iexplore.exe.6e79.wxhu.dmp


You may be able to discern what’s going on here. The *.tmp.ref file is the default target, which is the original process mini dump that I described earlier; the one with the thread and handle data. Using the debugger, you can view the flags passed to MiniDumpWriteDump used to create this file and thus the data contained within:

0:004> .dumpdebug
----- User Mini Dump Analysis

MINIDUMP_HEADER:
Version         A793 (61B0)
NumberOfStreams 11
Flags           1104
                0004 MiniDumpWithHandleData
                0100 MiniDumpWithProcessThreadData
                1000 MiniDumpWithThreadInfo

The *.wxhu.dmp file is the reflected process dump. Its naming convention hasn’t changed from Vista. So, those of you that have been working with XPROC dumps in Vista may have been a little confused when you analyzed this file from a Windows 7 machine, only to see one thread waiting in RtlpProcessReflectionStartup. Open the file directly in the debugger and you’ll see that it has full data, thread and memory info, etc…

0:000> .dumpdebug
----- User Mini Dump Analysis

MINIDUMP_HEADER:
Version         A793 (61B0)
NumberOfStreams 14
Flags           51B25
                0001 MiniDumpWithDataSegs
                0004 MiniDumpWithHandleData
                0020 MiniDumpWithUnloadedModules
                0100 MiniDumpWithProcessThreadData
                0200 MiniDumpWithPrivateReadWriteMemory
                0800 MiniDumpWithFullMemoryInfo
                1000 MiniDumpWithThreadInfo
                10000 MiniDumpWithPrivateWriteCopyMemory
                40000 MiniDumpWithTokenInformation


You’ll also probably notice that this dump also has handle data. But again, hang reporting does not duplicate the handle data from the original to the reflected process, so the only handle you should see is a process handle from the original process:

Stream 10: type HandleDataStream (12), size 00000038, RVA 08F35E0F
  1 descriptors, header size is 16, descriptor size is 40
    Handle(0000000000000008,"Process","")

Viewing the "Stitched" Target

There are at least two ways you can view the stitched target (using the same cab as an example):

  1. Extract the files from the cab and open the *.dbgcfg.ini file instead of the individual dump files

    C:\debuggers\cdb -z C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini

    Microsoft (R) Windows Debugger Version 6.13.0001.776 X86
    Copyright (c) Microsoft Corporation. All rights reserved.

    Loading Dump File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]
    Loading User-mode Dump Configuration File [C:\temp\
    xproc\iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini,C:,\temp\]
            default dump: [WER57FB.tmp.ref]
            memory dump: [iexplore.exe.6e79.wxhu.dmp]
            module list dump: [iexplore.exe.6e79.wxhu.dmp]

    Loading Dump File [C:\temp\xproc\WER57FB.tmp.ref]
    User Mini Dump File: Only registers, stack and portions of memory are available

    Loading Dump File [C:\temp\xproc\iexplore.exe.6e79.wxhu.dmp]
    User Mini Dump File: Only registers, stack and portions of memory are available


  2. Open the cab file directly in the debugger. Then open the dbgcfg.ini file from there

    C:\debuggers\cdb –z C:\temp\xproc\sample.cab

    Microsoft (R) Windows Debugger Version 6.13.0001.776 X86
    Copyright (c) Microsoft Corporation. All rights reserved.

    Extracted C:\temp\
    xproc\000011a00_memory.hdmp from C:\temp\xproc\sample.cab

    Loading Dump File [C:\temp\
    xproc\000011a00_memory.hdmp]
    User Mini Dump File: Only registers, stack and portions of memory are available

    0:016> .opendump /c "iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini";g
    Extracted C:\temp\xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini from C:\temp\
    xproc\sample.cab

    Loading Dump File [C:\temp\
    xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]

    Loading User-mode Dump Configuration File [C:\temp\
    xproc\000011a00_iexplore.exe.6e79.wxhu.dmp.dbgcfg.ini]
            default dump: [WER57FB.tmp.ref]
            memory dump: [iexplore.exe.6e79.wxhu.dmp]
            module list dump: [iexplore.exe.6e79.wxhu.dmp]

    Loading Dump File [C:\temp\
    xproc\000011a00_WER57FB.tmp.ref]

    Loading Dump File [C:\temp\
    xproc\000011a00_iexplore.exe.6e79.wxhu.dmp]


  3. Just run !analyze. It uses the same procedure outlined in option 2, and it leaves the targets opened once it’s finished.

 

Last But Not Least!

Make sure you have the latest debugger or else it may not recognize this type of target. You can download it from here.