Software Sleuthing : Debugging

MiniDumps and "Bad" Stacks

joshpoley — Mon, 10 Nov 2008 19:07:00 GMT

A fellow reader sent a comment outlining the following problem:

I'm using the DMPSTK example from the Debugging Tools SDK. If I generate a crash dump from within Visual Studio 2005, I can see the call stack perfectly. However, if I create my own crash dump (the same as you do in your blog), I get several LoadLibrary(ext), LoadLibrary(exts), ... failures. What's more important to notice is that the callstacks are different.

The LoadLibrary messages are benign and just an artifact of the debugger attempting to load additional plug-ins. But with regard to the stack traces, things get more interesting.

It turns out that the crash dump files created with Visual Studio are slightly different from the ones created via the API MiniDumpWriteDump. On top of that, WinDBG and the debugging engine exposed by WinDBG handle things a little bit differently with regard these dumps. The main difference stems from the use of embedded "contexts". A context contains the basic state of the system (CPU registers for example), and crash dumps can contain an embedded context which gets added in if you pass down the appropriate exception details to MiniDumpWriteDump.

If you were to load up one of these API generated crash dump files in WinDBG itself, the program will kindly tell you that there is an additional context and in order to use it, you need to issue the ".ecxr" command. Until you do this, the stack trace you will see possibly is not the stack you are interested in. This is where things are going wrong for our reader, who was displaying the default stack via the code in DMPSTK sample.

Luckily for us, we can do the equivalent to .ecxr in our code allowing us to get the "correct" stack.

To start with we replace IDebugControl for IDebugControl4. This then gives us access to the GetStoredEventInformation and GetContextStackTrace methods.

If we were to modify the initial code from this blog entry, we would end up with something like this which lets us try to get the embedded context, otherwise just fall back on the old standby (GetStackTrace).

HRESULT DumpStack(IDebugControl4 *control, IDebugSymbols *symbols)

{

#define MAX_FRAMES 1024

HRESULT hr = S_OK;

DEBUG_STACK_FRAME stackFrames[MAX_FRAMES] = {0};

ULONG numFrames = 0;

char context[1024] = {0};

ULONG type = 0;

ULONG procID = 0;

ULONG threadID = 0;

ULONG contextSize = 0;

char *contextData = NULL;

// look for an embedded event

hr = control->GetStoredEventInformation(&type, &procID, &threadID,

context, sizeof(context), &contextSize, NULL, 0, 0);

// get the stack trace

if(SUCCEEDED(hr))

{

contextData = new char[MAX_FRAMES*contextSize];

hr = control->GetContextStackTrace(context, contextSize,

stackFrames, ARRAYSIZE(stackFrames), contextData,

MAX_FRAMES*contextSize, contextSize, &numFrames);

}

else

{

hr = control->GetStackTrace(0, 0, 0, stackFrames,

ARRAYSIZE(stackFrames), &numFrames);

}

...

Just don't forget to delete the newly allocated "contextData" buffer at the end of the function.

Custom Debugger Auto-Expansion Tips

joshpoley — Thu, 24 Jan 2008 19:19:00 GMT

There is a fair amount of existing information (plus the official “EEAddIn” sample) on the subject of adding custom tool tips when hovering over various data types in the debugger. But lately I’ve run across a bunch of people who have never heard of it, so I figured I would post some details and a few of the extensions for standard Win32 types which I have in my collection.

For those new to the topic, there is some extensibility built into the VC debugger which allows you to supply custom text to the tool-tips which popup when you hover over a variable. For example, when you hover over a structure, the debugger will pop up a balloon which looks like:

Using a custom handler, we can query the object and provide our own output to make it look like:

The main idea is to turn something which might not contain very useful or actionable information into something with more context and meaning; thus making it easier to debug your application.

The above mentioned sample shows how to do this for the SYSTEMTIME and FILETIME structures, so here I will provide some functions which can be used to display custom text for the WIN32_FIND_DATAA, PROCESS_INFORMATION, CRITICAL_SECTION(¹), and OVERLAPPED structs.

Struct
CRITICAL_SECTION	Before
CRITICAL_SECTION	After
OVERLAPPED	Before
OVERLAPPED	After
PROCESS_INFORMATION	Before
PROCESS_INFORMATION	After

To start with, we will get the helper functions out of the way. Since I’m a nice guy, I support the “nBase” parameter which allows the debugger’s user to get back information in either base 10 or base 16 depending on their preference. The below function does the number to text formatting depending on the base. The return value is the number of bytes written to the string, which allows us too easily (and performantly(²)) concatenate strings. I have a handful of sprint_type functions, but I will just supply one of them as a sample, the others (which you will see used below) should be fairly self explanatory:

size_t sprint_int(char *buffer, size_t maxLen, int val, unsigned base)

{

if(base == 8)

return sprintf_s(buffer, maxLen, "%o", val);

else if(base == 10)

return sprintf_s(buffer, maxLen, "%d", val);

else if(base == 16)

return sprintf_s(buffer, maxLen, "0x%08x", val);

else

return sprintf_s(buffer, maxLen, "%d", val);

}

WIN32_FIND_DATAA

HRESULT __stdcall OS_WIN32_FIND_DATAA(DWORD dwAddress, DEBUGHELPER *pHelper, int nBase,

BOOL fReserved1, char *pResult, size_t max, DWORD dwReserved2)

{

WIN32_FIND_DATAA findData = {0};

DWORD bytesRead = 0;

HRESULT hr = S_OK;

hr = pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(findData), &findData, &bytesRead);

if(FAILED(hr))

{

return hr;

}

if(findData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)

{

sprintf_s(pResult, max, "\"%s\" (DIR)", findData.cFileName);

}

else

{

// package up the file size into a single 64bit value

LARGE_INTEGER size;

size.HighPart = findData.nFileSizeHigh;

size.LowPart = findData.nFileSizeLow;

double dblSize = (double)size.QuadPart;

// write out the filename + the file size using appropriate units

if(size.QuadPart > 1073741824)

sprintf_s(pResult, max, "\"%s\" (%.2lf GB)", findData.cFileName, dblSize / 1073741824.0);

else if(size.QuadPart > 1048576)

sprintf_s(pResult, max, "\"%s\" (%.2lf MB)", findData.cFileName, dblSize / 1048576.0);

else if(size.QuadPart > 1024)

sprintf_s(pResult, max, "\"%s\" (%.2lf KB)", findData.cFileName, dblSize / 1024.0);

else

sprintf_s(pResult, max, "\"%s\" (%I64u B)", findData.cFileName, size.QuadPart);

}

return S_OK;

}

Feel free (if needed) to expand on this to add in the file’s attributes and a timestamp.

CRITICAL_SECTION

HRESULT __stdcall OS_CRITICAL_SECTION(DWORD dwAddress, DEBUGHELPER *pHelper, int nBase,

BOOL fReserved1, char *pResult, size_t max, DWORD dwReserved2)

{

CRITICAL_SECTION cs = {0};

RTL_CRITICAL_SECTION_DEBUG csDebug = {0};

DWORD bytesRead = 0;

HRESULT hr = S_OK;

unsigned slen = 0;

hr = pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(cs), &cs, &bytesRead);

if(FAILED(hr))

{

return hr;

}

// write out the primary count values

slen += sprintf_s(pResult+slen, max-slen, "lock=");

slen += sprint_int(pResult+slen, max-slen, cs.LockCount, nBase);

slen += sprintf_s(pResult+slen, max-slen, " recursion=");

slen += sprint_int(pResult+slen, max-slen, cs.RecursionCount, nBase);

slen += sprintf_s(pResult+slen, max-slen, " spin=");

slen += sprint_unsigned(pResult+slen, max-slen, cs.SpinCount, nBase);

// if the debugging information looks valid, grab some information from

// the debug struct

if(cs.DebugInfo && !(cs.SpinCount&RTL_CRITICAL_SECTION_FLAG_NO_DEBUG_INFO))

{

hr = pHelper->ReadDebuggeeMemory(pHelper, (DWORD)cs.DebugInfo, sizeof(csDebug), &csDebug, &bytesRead);

if(SUCCEEDED(hr) && (bytesRead == sizeof(csDebug)))

{

slen += sprintf_s(pResult+slen, max-slen, " [entry=");

slen += sprint_unsigned(pResult+slen, max-slen, csDebug.EntryCount, nBase);

slen += sprintf_s(pResult+slen, max-slen, " contention=");

slen += sprint_unsigned(pResult+slen, max-slen, csDebug.ContentionCount, nBase);

slen += sprintf_s(pResult+slen, max-slen, "]");

}

return S_OK;

}

OVERLAPPED

HRESULT __stdcall OS_OVERLAPPED(DWORD dwAddress, DEBUGHELPER *pHelper, int nBase,

BOOL fReserved1, char *pResult, size_t max, DWORD dwReserved2)

{

OVERLAPPED over = {0};

DWORD bytesRead = 0;

HRESULT hr = S_OK;

unsigned slen = 0;

hr = pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(over), &over, &bytesRead);

if(FAILED(hr))

{

return hr;

}

// package up the offset values into a single 64bit value

LARGE_INTEGER offset = {0};

offset.LowPart = over.Offset;

offset.HighPart = over.OffsetHigh;

slen += sprintf_s(pResult+slen, max-slen, "offset=");

slen += sprint_u64(pResult+slen, max-slen, offset.QuadPart, nBase);

slen += sprintf_s(pResult+slen, max-slen, " event=0x%08x", over.hEvent);

return S_OK;

}

PROCESS_INFORMATION

HRESULT __stdcall OS_PROCESS_INFORMATION(DWORD dwAddress, DEBUGHELPER *pHelper, int nBase,

BOOL fReserved1, char *pResult, size_t max, DWORD dwReserved2)

{

PROCESS_INFORMATION procInfo = {0};

DWORD bytesRead = 0;

HRESULT hr = S_OK;

unsigned slen = 0;

hr = pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(procInfo), &procInfo, &bytesRead);

if(FAILED(hr))

{

return hr;

}

// first see if the process still exists

HANDLE hProc = OpenProcess(PROCESS_ALL_ACCESS, FALSE, procInfo.dwProcessId);

if(hProc == NULL)

{

sprintf_s(pResult, max, "handle=%p PID=%u (exited)", procInfo.hProcess, procInfo.dwProcessId);

return S_OK;

}

// attempt to query the filename of the process

char exeName[MAX_PATH+1] = {0};

if(GetProcessImageFileName(hProc, exeName, MAX_PATH))

{

char *filePart = strrchr(exeName, '\\');

if(filePart) // strip off the leading device\path information

slen += sprintf_s(pResult+slen, max-slen, "\"%s\"", filePart+1);

else

slen += sprintf_s(pResult+slen, max-slen, "\"%s\"", exeName);

}

else

{

// couldn't get the filename, so display the handle value instead

slen += sprintf_s(pResult+slen, max-slen, "handle=%p", procInfo.hProcess);

}

// if the process has exited, but there are still outstanding handles to

// it, then we can still get exit code status

DWORD exitCode = 0;

GetExitCodeProcess(hProc, &exitCode);

// write out the PID and the status

if(exitCode == STILL_ACTIVE)

slen += sprintf_s(pResult+slen, max-slen, " PID=%u (running)", procInfo.dwProcessId);

else

slen += sprintf_s(pResult+slen, max-slen, " PID=%u (exited: %lu)", procInfo.dwProcessId, exitCode);

CloseHandle(hProc);

return S_OK;

}

Setup

And as shown in the EEAddIn sample, you will want to use a .DEF file to specify the function exports; this allows you to use non-mangled names in the autoexp.dat file.

debugTypes.def

EXPORTS

OS_WIN32_FIND_DATAA

OS_PROCESS_INFORMATION

OS_CRITICAL_SECTION

OS_OVERLAPPED

Now just copy the built DLL into a directory where the IDE can find it (³) and add the following entries to the [AutoExpand] section of the autoexp.dat file, which should be located in the visual studio install path under "\Common7\Packages\Debugger". Since my DLL was named DebugTypes.dll, you will want to replace the filename below with your DLL’s name.

autoexp.dat

_WIN32_FIND_DATAA=$ADDIN(DebugTypes.dll,OS_WIN32_FIND_DATAA)

_PROCESS_INFORMATION=$ADDIN(DebugTypes.dll,OS_PROCESS_INFORMATION)

_RTL_CRITICAL_SECTION=$ADDIN(DebugTypes.dll,OS_CRITICAL_SECTION)

_OVERLAPPED=$ADDIN(DebugTypes.dll,OS_OVERLAPPED)

Footnotes

(1) A CRITICAL_SECTION is really just a typedef for the RTL_CRITICAL_SECTION struct. So you will see "_RTL_CRITICAL_SECTION" in the autoexp.dat file instead.

(2) By keeping track of the length as you go, you avoid the performance hit typically associated with a strcat() type of function call. And no, I don't think that "performantly" is a real word.

(3) I use a Post-Build event in my project to automatically copy the DLL into the Visual Studio install path:
copy "$(TargetPath)" "C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE"

Debugging a BugCheck 9C

joshpoley — Mon, 03 Dec 2007 21:38:00 GMT

Disclaimer: A BugCheck 9C is a generic catch-all error code which signifies that the hardware has encountered a catastrophic error. The steps I took below helped me diagnose and fix the issue with my system. If you are hitting a similar error, the following may or may not help you, as multiple (and diverse) failures are all represented with this one error code.

Random Crashes

My home PC started arbitrarily rebooting. Most of the time (at first) it would be while playing PC games which can stress the hardware pretty heavily. Although occasionally, the crash would happen during something fairly benign such as checking email in a web browser.

To get more information about the crash, I went looking for the mini-dump which I have configured to be written out when the OS hits a “BSOD”. Unfortunately, the output directory was completely empty, and no memory dumps were to be found anywhere on my hard drive. At this point I was thinking that this crash must be pretty catastrophic in order to completely bypass the kernel’s exception handler. After some mostly fruitless / generic troubleshooting and web browsing I found an obscure reference to the fact that if your Windows page file is not on the same partition as the system OS, then you won’t get crash dumps. I had my page file on its very own partition as a performance enhancement (which will prevent file fragmentation of the data). So (with some amount of bitterness) I moved my page file back to the C drive and waited around for the next crash to happen.

A little while later my box rebooted again, upon coming back up I happily saw I had a crash dump sitting there. Now I could get somewhere. I moved the generated .dmp file onto another computer and cranked up WinDbg (http://www.microsoft.com/whdc/devtools/debugging/default.mspx) so I could take a look at what happened (some spew has been omitted for brevity).

*******************************************************************************

* *

* Bugcheck Analysis *

* *

*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 9C, {0, f6232050, b2000000, 1040080f}

Probably caused by : Unknown_Image ( ANALYSIS_INCONCLUSIVE )

Followup: MachineOwner

---------

1: kd> !analyze -v

MACHINE_CHECK_EXCEPTION (9c)

A fatal Machine Check Exception has occurred.

Debugging Details:

------------------

NOTE: This is a hardware error. This error was reported by the CPU

via Interrupt 18. This analysis will provide more information about

the specific error. Please contact the manufacturer for additional

information about this error and troubleshooting assistance.

This error is documented in the following publication:

- IA-32 Intel(r) Architecture Software Developer's Manual

Volume 3: System Programming Guide

Bit Mask:

MA Model Specific MCA

O ID Other Information Error Code Error Code

VV SDP ___________|____________ _______|_______ _______|______

AEUECRC| | | |

LRCNVVC| | | |

^^^^^^^| | | |

6 5 4 3 2 1

3210987654321098765432109876543210987654321098765432109876543210

----------------------------------------------------------------

1011001000000000000000000000000000010000010000000000100000001111

VAL - MCi_STATUS register is valid

Indicates that the information contained within the IA32_MCi_STATUS

rules given for the OVER flag in the IA32_MCi_STATUS register when

overwriting previously valid entries. The processor sets the VAL

flag and software is responsible for clearing it.

UC - Error Uncorrected

Indicates that the processor did not or was not able to correct the

error condition. When clear, this flag indicates that the processor

was able to correct the error condition.

EN - Error Enabled

Indicates that the error was enabled by the associated EEj bit of the

IA32_MCi_CTL register.

PCC - Processor Context Corrupt

Indicates that the state of the processor might have been corrupted

by the error condition detected and that reliable restarting of the

processor may not be possible.

BUSCONNERR - Bus and Interconnect Error BUS{LL}_{PP}_{RRRR}_{II}_{T}_err

These errors match the format 0000 1PPT RRRR IILL

Concatenated Error Code:

--------------------------

_VAL_UC_EN_PCC_BUSCONNERR_F

This error code can be reported back to the manufacturer.

They may be able to provide additional information based upon

this error. All questions regarding STOP 0x9C should be

directed to the hardware manufacturer.

BUGCHECK_STR: 0x9C_GenuineIntel

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT

LAST_CONTROL_TRANSFER: from e0b89bff to e0bc7deb

STACK_TEXT:

f6232028 e0b89bff 0000009c 00000000 f6232050 nt!KeBugCheckEx+0x1b

f6232154 e0b84c52 f622ed70 00000000 00000000 hal!HalpMcaExceptionHandler+0xdd

f6232154 00000000 f622ed70 00000000 00000000 hal!HalpMcaExceptionHandlerWrapper+0x4a

STACK_COMMAND: kb

SYMBOL_NAME: ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME: Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP: 0

FAILURE_BUCKET_ID: 0x9C_GenuineIntel_ANALYSIS_INCONCLUSIVE

BUCKET_ID: 0x9C_GenuineIntel_ANALYSIS_INCONCLUSIVE

Followup: MachineOwner

---------

BugCheck 9C is the main error code, and the numbers immediately following it provide some additional information depending on your hardware configuration and the event that triggered the problem. Microsoft has the following generic KB article on the 9C errors: http://support.microsoft.com/?kbid=329284, which gives you some techno-speak for "your hardware did something horribly wrong and we have no idea of how to continue."

After doing some additional research on the web it looked like the most common causes of this issue is due to a faulty motherboard, CPU, power supply, or memory.

BIOS

The first thing I did was to turn on the full memory test that runs during boot up. This test ran successfully so I went onto the power supply. My BIOS allows me to monitor the voltage outputs being supplied to the components in the computer, so I let that screen run for a while and watched the values. But so far everything looked good: the 12 volt line was pretty steady; 5 volts looked good, etc.

Diagnostic Tools

The next step was to run some tools to test and stress the various components. I started with Microsoft's Memory Diagnostic (http://oca.microsoft.com/en/windiag.asp) tool which runs off of its own boot disk (so works well even if your current OS is in a bad state). I let this application run through several iterations without any errors detected, so then I went looking for CPU tests. I tried the Stress Prime 2004 test (http://sp2004.fre3.com/) with the same result: no errors. I also tried out SiSoftware's Sandra package (http://www.sisoftware.co.uk/) which will stress and monitor multiple system components. Unfortunately everything I used was pretty inconclusive: I'm still encountered crashes, but no actionable errors had been detected.

At this point the crash frequency had increased to the point where I could no longer keep Windows up and running long enough to do anything useful, so it was time to get a boot disk. Downloading the latest version of the "Ultimate Boot CD" (http://ubcd.sourceforge.net/) also gave me another set of diagnostic tools to try out. But the results were pretty much the same; some tests could run for extended periods of time without any problems, other tests would crash without giving me any useful information.

Opening the Case

Based on my initial tests I had a fairly high confidence level in the power supply and the memory, which left the CPU and the motherboard, so now it was time to open the case up. After disconnecting everything I pulled the side panel off and vacuumed out all the dust and cat hair that had collected over the years. I then went over the motherboard and components with a good light and looked for anything out of the ordinary. One thing that caught my eye was a bloated capacitor (http://blogs.msdn.com/photos/joshpoley/picture6646863.aspx), looking closer it even appeared to be leaking down into another component. While it wasn't as bad as what these guys experienced (http://www.pcstats.com/articleview.cfm?articleID=195), it definitely wasn't going to be making my computer any healthier. There is no direct evidence that the capacitor was the root cause of the BugCheck, but it is enough of a concern to warrant replacing the board.

If I hadn't noticed anything visually, the next step would have been to pull out half of my memory and run the box with one pair then the other to see if I could rule out (for certain) memory failure. But since I had a bad capacitor, the best thing to do was order a new replacement motherboard and test that first. After the time consuming process of completely disassembling my computer system, inserting a new board, and hooking everything back up, I apprehensively powered it all up.

After I was into the OS, it was back to running more tests to see if everything was hooked up correctly and see if I encounter another crash. I also monitored the component temperatures closely to ensure the CPU heat sink and system fans were all working optimally. Luckily I was able to run all weekend without any crashes, so it looks like the motherboard was the culprit of my STOP 9C errors.