Junfeng Zhang's Windows Programming Notes : Debugging

Managed Watson Dump

junfeng — Mon, 14 Jul 2008 18:00:00 GMT

From .Net Framework 2.0, Dr. Watson is able to generate dump compatible with .Net framework. This means, dumps with heap data generated by Dr. Watson contains information about managed heap so they can be analyzed with sos.dll.

Some update to Dr.Watson is necessary to make this happen. Dumps generated by older Dr.Watson is not compatible with .Net framework.

To see if a dump is compatible with .Net framework, make sure the mini dump's version is equal or higher than 6400.

0:003> .dumpdebug

----- User Mini Dump Analysis

MINIDUMP_HEADER:

Version A793 (6407)

NumberOfStreams 8

Flags 120

0020 MiniDumpWithUnloadedModules

0100 MiniDumpWithProcessThreadData

Inspect a 32 bit Process Dump Generated by a 64 bit Debugger

junfeng — Mon, 30 Jun 2008 18:00:00 GMT

64 bit Windows can run both 32 bit process and 64 bit process. For debugging though, you want to use 32 bit debugger to debug 32 bit process, and 64 bit debugger for 64 bit process. Otherwise it won’t be pretty.

Occasionally, I receive a 32 bit process dump generated by a 64 bit debugger.

When load some dump in the debugger, this is how it looks like:

0:020> .sympath SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

20 Id: 1994.15f4 Suspend: 1 Teb: 00000000`7efa4000 Unfrozen
RetAddr : Args to Child : Call Site
00000000`78b84191 : 00000023`7d61c918 00000000`00000023 00000000`00000202 00000000`0518fffc : wow64cpu!CpupSyscallStub+0x9
00000000`6b006a5a : 00000000`00000003 00000000`00000000 00000000`00000000 00000000`051cf7d0 : wow64cpu!Thunk2ArgNSpNSpReloadState+0x21
00000000`6b005e0d : 00000000`051cfd00 00000000`051cf1d0 00000000`051cf7d0 00000000`00000000 : wow64!RunCpuSimulation+0xa
00000000`77f109f0 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`7efdf000 : wow64!Wow64LdrpInitialize+0x2ed
00000000`77ef30a5 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrpInitialize+0x2aa
00000000`7d4d1504 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!KiUserApcDispatcher+0x15
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadStartThunk
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 0001002f`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 0001002f`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 0001002f`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 0001002f`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
0001002f`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x1002f`00000000
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0

I do not know if you can make sense of it. I cannot.

Fortunately, Windows debugger allows you to change the target processor type. Here is how it works:

0:020> .effmach x86
Effective machine: x86 compatible (x86)
0:020:x86> .load wow64exts

20 Id: 1994.15f4 Suspend: 1 Teb: 00000000`7efa4000 Unfrozen
ChildEBP RetAddr Args to Child
0518ff0c 7d4d0ec5 00000000 0518ff4c 7d4d8bfa ntdll_7d600000!ZwDelayExecution+0x15
0518ff74 7d4d14ef 0000ea60 00000000 0518ffac kernel32!SleepEx+0x68
0518ff84 776bbb0f 0000ea60 083e6ca8 776bbab4 kernel32!Sleep+0xf
0518ff90 776bbab4 00000000 00000000 083e6ca8 ole32!CROIDTable::WorkerThreadLoop+0x14
0518ffac 776b1704 00000000 0518ffec 7d4dfe21 ole32!CRpcThread::WorkerLoop+0x26
0518ffb8 7d4dfe21 083e6ca8 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
0518ffec 00000000 776b16e4 083e6ca8 00000000 kernel32!BaseThreadStart+0x34

Much better, isn’t it?

For best result though, please use 32 bit debugger to generate the dump for a 32 bit process.

Event Handles “leak”

junfeng — Mon, 28 Apr 2008 18:00:00 GMT

On our stress run, we saw our process’ handle count steadily increases until certain point, then it stabilizes. However the number of handles is high. Most of those handles are Event handles. We are concerned about it. So we went off and did some investigation.

Turns out the Event handles are coming from the use of Monitor.

When there is contention on the lock object, CLR internally creates an Event handle, presumably to facilitate the thread scheduling. The event handle is not cleaned up until the object is garbage collected.

It appears we were using Monitor in a lot of places, and we had lock contentions, which triggers CLR to allocate a lot of Event handles.

So if you have a lot of long lived objects, be careful about the usage of Monitor.

Use !htrace to debug handle leak

junfeng — Mon, 21 Apr 2008 18:00:00 GMT

Windbg Debugger’s !htrace extension is very handy to debug handle leak.

The process essentially boils down to the following simple steps:

1. Enable trace

2. Take a snapshot

3. Run scenario

4. Show the diff

On step 4, !htrace will show all the extra opened handles after the last snapshot, along with the callstack if available. This greatly helps to debug what handles are leak, and by whom.

Like any other resource leak detection tool, there will be false positives. You need to understand what is a real leak, and what is just a transient allocation.

!htrace

The !htrace extension displays stack trace information for one or more handles.

Syntax

User-Mode Syntax

!htrace [Handle [Max_Traces]]
!htrace -enable [Max_Traces]
!htrace -snapshot
!htrace -diff
!htrace -disable
!htrace -?

Kernel-Mode Syntax

!htrace [Handle [Process [Max_Traces]]]
!htrace -?

Parameters

Handle

Specifies the handle whose stack trace will be displayed. If Handle is 0 or omitted, stack traces for all handles in the process will be displayed.

Process

(Kernel mode only) Specifies the process whose handles will be displayed. If Process is 0 or omitted, then the current process is used. In user mode, the current process is always used.

Max_Traces

Specifies the maximum number of stack traces to display. In user mode, if this parameter is omitted, then all the stack traces for the target process will be displayed.

-enable

(User mode only) Enables handle tracing and takes the first snapshot of the handle information to use as the initial state by the -diff option.

-snapshot

(User mode only) Takes a snapshot of the current handle information to use as the initial state by the -diff option.

-diff

(User mode only) Compares current handle information with the last snapshot of handle information that was taken. Displays all handles that are still open.

-disable

(User mode only; Windows Server 2003 and later only) Disables handle tracing. In Windows XP, handle tracing can be disabled only by terminating the target process.

Displays some brief Help text for this extension in the Debugger Command window.

DLL

Windows NT 4.0	Unavailable
Windows 2000	Unavailable
Windows XP and later	Kdexts.dll Ntsdexts.dll

Comments

Before !htrace can be used, Application Verifier must be activated for the target process, and the Detect invalid handle usage option must be selected. By activating Application Verifier, stack trace information is saved each time the process opens a handle, closes a handle, or references an invalid handle. It is this stack trace information that !htrace displays. For more information, see Application Verifier.

The following example displays information about all handles in process 0x81400300:

kd> !htrace 0 81400300
Process 0x81400300
ObjectTable 0xE10CCF60

--------------------------------------
Handle 0x7CC - CLOSE:
0x8018FCB9: ntoskrnl!ExDestroyHandle+0x103
0x801E1D12: ntoskrnl!ObpCloseHandleTableEntry+0xE4
0x801E1DD9: ntoskrnl!ObpCloseHandle+0x85
0x801E1EDD: ntoskrnl!NtClose+0x19
0x010012C1: badhandle!mainCRTStartup+0xE3
0x77DE0B2F: KERNEL32!BaseProcessStart+0x3D
--------------------------------------
Handle 0x7CC - OPEN:
0x8018F44A: ntoskrnl!ExCreateHandle+0x94
0x801E3390: ntoskrnl!ObpCreateUnnamedHandle+0x10C
0x801E7317: ntoskrnl!ObInsertObject+0xC3
0x77DE23B2: KERNEL32!CreateSemaphoreA+0x66
0x010011C5: badhandle!main+0x45
0x010012C1: badhandle!mainCRTStartup+0xE3
0x77DE0B2F: KERNEL32!BaseProcessStart+0x3D
--------------------------------------
Handle 0x7DC - BAD REFERENCE:
0x8018F709: ntoskrnl!ExMapHandleToPointerEx+0xEA
0x801E10F2: ntoskrnl!ObReferenceObjectByHandle+0x12C
0x801902BE: ntoskrnl!NtSetEvent+0x6C
0x80154965: ntoskrnl!_KiSystemService+0xC4
0x010012C1: badhandle!mainCRTStartup+0xE3
0x77DE0B2F: KERNEL32!BaseProcessStart+0x3D
--------------------------------------
Handle 0x7DC - CLOSE:
0x8018FCB9: ntoskrnl!ExDestroyHandle+0x103
0x801E1D12: ntoskrnl!ObpCloseHandleTableEntry+0xE4
0x801E1DD9: ntoskrnl!ObpCloseHandle+0x85
0x801E1EDD: ntoskrnl!NtClose+0x19
0x010012C1: badhandle!mainCRTStartup+0xE3
0x77DE0B2F: KERNEL32!BaseProcessStart+0x3D
--------------------------------------
Handle 0x7DC - OPEN:
0x8018F44A: ntoskrnl!ExCreateHandle+0x94
0x801E3390: ntoskrnl!ObpCreateUnnamedHandle+0x10C
0x801E7317: ntoskrnl!ObInsertObject+0xC3
0x77DE265C: KERNEL32!CreateEventA+0x66
0x010011A0: badhandle!main+0x20
0x010012C1: badhandle!mainCRTStartup+0xE3
0x77DE0B2F: KERNEL32!BaseProcessStart+0x3D

--------------------------------------
Parsed 0x6 stack traces.
Dumped 0x5 stack traces.

Additional Information

For information about handles, see the Microsoft Windows SDK documentation and Microsoft Windows Internals by Mark Russinovich and David Solomon. To display further information about a specific handle, use the !handle extension.

A case study of a NullReferenceException

junfeng — Mon, 07 Apr 2008 18:00:00 GMT

We are seeing a NullReferenceException in our stress program. The investigation process may be helpful to some folks.

Thread 22 is showing a Watson dialog.

This is a register corruption.

0:022> kp

ChildEBP RetAddr

091bc9a0 76961220 ntdll!ZwWaitForSingleObject(void)+0x15

091bca10 76961188 kernel32!WaitForSingleObjectEx(void * hHandle = 0x00001034, unsigned long dwMilliseconds = 0x4e20, int bAlertable = 0)+0xbe

091bca24 7bb1fca4 kernel32!WaitForSingleObject(void * hHandle = 0x00001034, unsigned long dwMilliseconds = 0x4e20)+0x12

091bca54 7bb202b4 mscorwks!ClrWaitForSingleObject(void * handle = 0x00001034, unsigned long timeout = 0x4e20)+0x24

091bcf10 7bb23570 mscorwks!RunWatson(void * hWatsonSharedMemory = 0x000015f4, void * hEventAlive = 0x00001034, void * hEventDone = 0x00000d7c, void * hMutex = 0x00001378)+0x18f

091bd658 7bb23b43 mscorwks!DoFaultReportWorker(struct _EXCEPTION_POINTERS * pExceptionInfo = 0x091bd7ac, class TypeOfReportedError tore = class TypeOfReportedError, class Thread * pThread = 0x09b1d0b0, unsigned long dwThreadID = 0x300)+0xb52

091bd6a0 7bb317d3 mscorwks!DoFaultReport(struct _EXCEPTION_POINTERS * pExceptionInfo = 0x091bd7ac, class TypeOfReportedError tore = class TypeOfReportedError)+0x13d

091bd6c8 7bb361fb mscorwks!WatsonLastChance(class Thread * pThread = 0x09b1d0b0, struct _EXCEPTION_POINTERS * pExceptionInfo = 0x091bd7ac, class TypeOfReportedError tore = class TypeOfReportedError)+0x51

091bd728 7bb36387 mscorwks!InternalUnhandledExceptionFilter_Worker(struct _EXCEPTION_POINTERS * pExceptionInfo = 0x091bd7ac)+0x179

091bd734 7bb7578c mscorwks!ThreadBaseExceptionAppDomainFilter(struct _EXCEPTION_POINTERS * pExceptionInfo = 0x091bd7ac, void * pvParam = 0x091bf0f4)+0x1a

091bd76c 7baf167a mscorwks!ThreadBaseRedirectingFilter(struct _EXCEPTION_POINTERS * pExceptionInfo = 0x091bd7ac, void * _pCallState = 0x00000002)+0x7b

091bd774 7b9d2540 mscorwks!ManagedThreadBase_DispatchOuter(struct ManagedThreadCallState * pCallState = 0x091bf0f4)+0x38

091bd788 7ba6bf39 mscorwks!_EH4_CallFilterFunc(void)+0x12

091bd7b0 77e6b6d1 mscorwks!_except_handler4(struct _EXCEPTION_RECORD * ExceptionRecord = <Memory access error>, struct _EXCEPTION_REGISTRATION_RECORD * EstablisherFrame = <Memory access error>, struct _CONTEXT * ContextRecord = <Memory access error>, void * DispatcherContext = <Memory access error>)+0x8e

091bd7d4 77e6b6a3 ntdll!ExecuteHandler2(void)+0x26

091bd87c 77e4ee57 ntdll!ExecuteHandler(void)+0x24

091bd87c 06b2d7f3 ntdll!KiUserExceptionDispatcher(void)+0xf

091bea00 06b2d796 mscorlib!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(<HRESULT 0x80004001>)+0x43

091bea38 06b2d6b6 mscorlib!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Contains(<HRESULT 0x80004001>)+0x6

The first thing we should do is to find the right exception context. As pointed out in this article, we should look for _EXCEPTION_POINTERS, and the frame mscorwks!DoFaultReport gives us that (and along with many other frames). Let’s use it.

0:022> dc 0x091bd7ac

091bd7ac 091bd894 091bd8e4 77e6b6d1 091bd894 ...........w....

091bd7bc 091beed4 091bd8e4 091bd870 091bdc64 ........p...d...

091bd7cc 77e6b6e5 091beed4 091bd87c 77e6b6a3 ...w....|......w

091bd7dc 091bd894 091beed4 091bd8e4 091bd870 ............p...

091bd7ec 7ba6beb4 00000000 091bd894 091beed4 ...{............

091bd7fc 77e6b648 091bd894 091beed4 091bd8e4 H..w............

091bd80c 091bd870 7ba6beb4 00000002 091bd894 p......{........

091bd81c 091be6e8 7bb83bd0 035db834 00000004 .....;.{4.].....

The highlighted DWORD is the exception context. Let’s move to it.

0:022> .cxr 091bd8e4

eax=00000000 ebx=03339044 ecx=033310a8 edx=00000000 esi=00000001 edi=03312bc0

eip=06b2d7f3 esp=091be9f0 ebp=091bea00 iopl=0 nv up ei ng nz ac pe cy

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010297

mscorlib!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon)+0x43:

06b2d7f3 8b5610 mov edx,dword ptr [esi+10h] ds:002b:00000011=????????

So this is access invalid memory. This is very strange. LinkedList is pure managed code. I have never seen manage code accessing invalid memory.

Let’s look at the assembly code and see what happened.

0:022> !u 06b2d7f3

Normal JIT generated code

System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon)

Begin 06b2d7b0, size 83

06b2d7b0 55 push ebp

06b2d7b1 8bec mov ebp,esp

06b2d7b3 57 push edi

06b2d7b4 56 push esi

06b2d7b5 53 push ebx

06b2d7b6 50 push eax

06b2d7b7 8bf9 mov edi,ecx

06b2d7b9 8bda mov ebx,edx

06b2d7bb 8b7704 mov esi,dword ptr [edi+4]

06b2d7be 8b0f mov ecx,dword ptr [edi]

06b2d7c0 8b4120 mov eax,dword ptr [ecx+20h]

06b2d7c3 8b10 mov edx,dword ptr [eax]

06b2d7c5 83c208 add edx,8

06b2d7c8 8b02 mov eax,dword ptr [edx]

06b2d7ca 85c0 test eax,eax

06b2d7cc 7513 jne mscorlib!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon)+0x31 (06b2d7e1)

06b2d7ce 6a00 push 0

06b2d7d0 52 push edx

06b2d7d1 51 push ecx

06b2d7d2 bae900001b mov edx,1B0000E9h

06b2d7d7 b9f0047e02 mov ecx,27E04F0h (MD: System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon))

06b2d7dc e8e271fb74 call mscorwks!JIT_GenericHandle (7bae49c3)

06b2d7e1 8bc8 mov ecx,eax

06b2d7e3 e810a078f9 call mscorlib!System.Collections.Generic.EqualityComparer`1[[System.__Canon, mscorlib]].get_Default() (002b77f8)

06b2d7e8 8945f0 mov dword ptr [ebp-10h],eax

06b2d7eb 85f6 test esi,esi

06b2d7ed 743c je mscorlib!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon)+0x7b (06b2d82b)

06b2d7ef 85db test ebx,ebx

06b2d7f1 7422 je mscorlib!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon)+0x65 (06b2d815)

>>> 06b2d7f3 8b5610 mov edx,dword ptr [esi+10h]

06b2d7f6 53 push ebx

06b2d7f7 8b4df0 mov ecx,dword ptr [ebp-10h]

06b2d7fa 8b01 mov eax,dword ptr [ecx]

06b2d7fc ff5038 call dword ptr [eax+38h]

0:022> r

Last set context:

eax=00000000 ebx=03339044 ecx=033310a8 edx=00000000 esi=00000001 edi=03312bc0

eip=06b2d7f3 esp=091be9f0 ebp=091bea00 iopl=0 nv up ei ng nz ac pe cy

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010297

Microsoft_Live_Moe_Runtime!System.Collections.Generic.LinkedList`1[[System.__Canon, mscorlib]].Find(System.__Canon)+0x43:

06b2d7f3 8b5610 mov edx,dword ptr [esi+10h] ds:002b:00000011=????????

0:022> dc @edi+4

03312bc4 136deb74 00000000 00000008 00000000 t.m.............

03312bd4 00fe553c 03312be8 00000258 00000000 <U...+1.X.......

03312be4 00020000 00fe56b0 00000258 00000004 .....V..X.......

The highlighted code at 06b2d7f3 is where the exception happened. If we read the assembly code, register @esi is assigned to the value at address @edi+4 at offset 06b2d7bb, and not changed afterwards. Register @edi is assigned at offset 06b2d7b7, and not changed afterwards. This means @esi should be the same as the value at memory @edi+4.

However, @esi = 1, while the memory at @edi+4 is 136deb74. Apparently, one of them is wrong. Telling from the value, it looks like the register @esi is corrupted.

We still don’t know why the register is corrupted. But at least we are confident that this is not a bug in LinkedList.

Getting the right exception context from a memory dump

junfeng — Mon, 03 Mar 2008 19:00:00 GMT

When debug a memory dump, the dump may not in the right exception context when it is first loaded in the debugger. However, we can figure out the right context from the dump.

Let's load the dump.

c:\debuggers>cdb -z c:\temp\foo.dmp

Symbol search path is: *** Invalid ***
****************************************************************************
* Symbol loading may be unreliable without a symbol search path.           *
* Use .symfix to have the debugger choose a symbol path.                   *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************
Executable search path is:
Windows XP Version 2600 (Service Pack 2) MP (4 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Debug session time: Tue Feb 26 07:14:42.000 2008 (GMT-8)
System Uptime: not available
Process Uptime: 0 days 2:07:05.000
......................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(8c4.3d8): Wake debugger - code 80000007 (first/second chance not available)
eax=00000000 ebx=7c884700 ecx=0006ee6c edx=7c90eb94 esi=00000000 edi=0006f40c
eip=7c90eb94 esp=0006ee78 ebp=0006f528 iopl=0         nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000246
*** WARNING: Unable to verify timestamp for ntdll.dll
*** ERROR: Module load completed but symbols could not be loaded for ntdll.dll
ntdll+0xeb94:
7c90eb94 c3              ret

Let’s load symbols first.

0:000> .sympath SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
0:000> .reload
......................................

Unfortunately .ecxr did not give us the right context.

0:000> .ecxr
eax=00000000 ebx=7c884700 ecx=0006ee6c edx=7c90eb94 esi=00000000 edi=0006f40c
eip=7c90eb94 esp=0006ee78 ebp=0006f528 iopl=0         nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
7c90eb94 c3              ret
0:000> kb
ChildEBP RetAddr Args to Child
0006ee74 7c90e273 7c863487 d0000144 00000004 ntdll!KiFastSystemCallRet
0006ee78 7c863487 d0000144 00000004 00000000 ntdll!NtRaiseHardError+0xc
0006f528 77c32f0f 0006f578 00000000 00000000 kernel32!UnhandledExceptionFilter+0x653
0006f544 01006484 80000003 0006f578 01009818 msvcrt!_XcptFilter+0x161
WARNING: Stack unwind information not available. Following frames may be wrong.
0006ffc0 7c816fd7 00000016 03d7f12c 7ffd6000 foo+0x6484
0006fff0 00000000 0100660c 00000000 00000000 kernel32!BaseProcessStart+0x23

However, Kernel32!UnhandledExceptionFilter is on the stack. UnhandledExceptionFilter takes an EXCEPTION_POINTERS structure as its parameter, which contains the exception context.

0:000> dc 0006f578
0006f578 0006f69c 0006f6bc 01013ea4 00000001 .........>......
0006f588 fffffffe 0006f5b0 0006f5b0 01009c32 ............2...
0006f598 01015100 010066ff 0006f69c 0006ffb0 .Q...f..........
0006f5a8 0006f6bc 0006f670 0006f5d4 7c9037bf ....p........7.|
0006f5b8 0006f69c 0006ffb0 0006f6bc 0006f670 ............p...
0006f5c8 0006fd4c 7c9037d8 0006ffb0 0006f684 L....7.|........
0006f5d8 7c90378b 0006f69c 0006ffb0 0006f6bc .7.|............
0006f5e8 0006f670 01009c12 00000001 0006f69c p...............

We can change to the exception context.

0:000> .cxr 0006f6bc
eax=0006a1fa ebx=6a803bd8 ecx=00353ee8 edx=00080608 esi=6a8043d8 edi=6a8045c0
eip=7c901230 esp=0006f988 ebp=0006fca4 iopl=0         nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000246
ntdll!DbgBreakPoint:
7c901230 cc              int     3

Now we have the correct exception context.

0:000> k
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0006f984 6a870952 ntdll!DbgBreakPoint
WARNING: Stack unwind information not available. Following frames may be wrong.
0006fca4 6a8755f9 bar+0x70952
0006fcd4 6a8757ae bar+0x755f9
0006fcf0 6a86f49e bar+0x757ae
0006fcfc 6a86fc06 bar+0x6f49e
0006fd5c 7c9011a7 bar+0x6fc06
0006fd7c 7c923f31 ntdll!LdrpCallInitRoutine+0x14
0006fe00 7c81cd76 ntdll!LdrShutdownProcess+0x14f
0006fef4 7c81cdee kernel32!_ExitProcess+0x42
0006ff08 77c39d45 kernel32!ExitProcess+0x14
0006ff14 77c39e78 msvcrt!__crtExitProcess+0x32
0006ff24 77c39e90 msvcrt!_cinit+0xee
0006ff38 0100645e msvcrt!exit+0x12
0006ffc0 7c816fd7 foo+0x645e
0006fff0 00000000 kernel32!BaseProcessStart+0x23

Calculating the size of each GC generation

junfeng — Mon, 25 Feb 2008 19:00:00 GMT

Sos.dll does not have a command to tell the size of each GC generation. However, you can calculate the size based on the output of !eeheap -gc.

For example,

0:000> !eeheap -gc

Number of GC Heaps: 1

generation 0 starts at 0x1da6c430

generation 1 starts at 0x1da5c240

generation 2 starts at 0x01a41000

ephemeral segment allocation context: none

segment begin allocated size

01a40000 01a41000 02a1cc4c 0x00fdbc4c(16628812)

0c7f0000 0c7f1000 0d7e9fc8 0x00ff8fc8(16748488)

12170000 12171000 13164488 0x00ff3488(16725128)

13170000 13171000 14117640 0x00fa6640(16410176)

10170000 10171000 1115b134 0x00fea134(16687412)

14170000 14171000 1516fdec 0x00ffedec(16772588)

1e030000 1e031000 1eff9970 0x00fc8970(16550256)

0e6e0000 0e6e1000 0f6dce50 0x00ffbe50(16760400)

15170000 15171000 160f7a60 0x00f86a60(16280160)

16170000 16171000 170f7bc0 0x00f86bc0(16280512)

17170000 17171000 180ffefc 0x00f8eefc(16314108)

18170000 18171000 191144dc 0x00fa34dc(16397532)

19170000 19171000 1a167e84 0x00ff6e84(16739972)

1a170000 1a171000 1b166a68 0x00ff5a68(16734824)

1b170000 1b171000 1c115a9c 0x00fa4a9c(16403100)

1d030000 1d031000 1db35130 0x00b04130(11551024)

Large object heap starts at 0x02a41000

segment begin allocated size

02a40000 02a41000 03a07fc8 0x00fc6fc8(16543688)

072c0000 072c1000 082b6608 0x00ff5608(16733704)

11170000 11171000 11325aa0 0x001b4aa0(1788576)

Total Size 0x11961cdc(295050460)

------------------------------

GC Heap Size 0x11961cdc(295050460)

Gen2 size = 0x1da5c240 - 1d031000 + 16628812 + 16748488 + 16725128 + 16410176 + 16687412 + 16772588 + 16550256 + 16760400 + 16280160 + 16280512 + 16314108 + 16397532 + 16739972 + 16734824 + 16403100.

Gen1 size = 0x1da6c430 - 0x1da5c240.

Gen0 size = 0x1db35130 - 0x1da6c430

Figuring out object allocation graph from a managed memory dump

junfeng — Thu, 21 Feb 2008 19:00:00 GMT

when debugging a managed memory dump, we can use Sos.dll to inspect managed objects inside the dump.

for example, we can do !dumpheap to list the managed objects in GC Heap, we can also use !gcroot to figure out the reference path to a particular object.

However, if we want to understand the whole picture, !gcroot isn't going to cut it.

Then we can do !traverseheap. It writes the GC information to a format understood by CLR Profiler. We can then use CLR Profiler to inspect the object allocation graph.

http://msdn2.microsoft.com/en-us/library/bb190764(VS.80).aspx

TraverseHeap [-xml] <filename>

Writes heap information to the specified file in a format understood by the CLR Profiler. The -xml option causes the TraverseHeap command to format the file as XML.

You can download the CLR Profiler from: http://www.microsoft.com/downloads/details.aspx?FamilyID=a362781c-3870-43be-8926-862b40aa0cd0&DisplayLang=en

From Unhandled Exception to Debugger Attach

junfeng — Mon, 19 Nov 2007 19:00:45 GMT

I was always wondering what is happening between an application throws an unhandled exception, and a debugger attached to the process. Apparently the information is right under my nose, but I was blind.

Windows Debugger's documentation about post-mortem debugging has all the details you want to know.

Enabling Postmortem Debugging

The most common application errors are called exceptions. These include access violations, division-by-zero errors, numerical overflows, and many other kinds of errors.

Applications can also cause breakpoint interrupts. These occur when Windows is unable to run the application (for example, when a necessary module cannot be loaded) or when a breakpoint is encountered. Breakpoints can be inserted into the code by a debugger, or invoked through a function such as DbgBreakPoint. In assembly language, a breakpoint interrupt is generated by an int 3 instruction.

Windows can handle user-mode errors in a variety of ways. The following sequence shows the precedence used for error handling:

If a user-mode debugger is currently attached to the faulting process, all errors will cause the target to break into this debugger.
As long as the user-mode debugger is attached, no other error-handling methods will be used — even if the gn (Go With Exception Not Handled) command is used.
If no user-mode debugger is attached and the executing code has its own exception handling routines (for example, try - except), this exception handling routine will attempt to deal with the error.
If no user-mode debugger is attached, and Windows has an open kernel-debugging connection, and the error is a breakpoint interrupt, Windows will attempt to contact the kernel debugger.
Kernel debugging connections must be opened during Windows' boot process. If you are using Windows Server 2003 or a later version of Windows and wish to prevent a user-mode interrupt from breaking into the kernel debugger, you can use the KDbgCtrl utility with the -du parameter. For details on how to configure kernel-debugging connections and how to use KDbgCtrl, see Configuring Software on the Target Computer.

If Windows does attempt to contact a kernel debugger but there is no debugger running at the other end of the connection, Windows will freeze until kernel debugger is activated.

In the kernel debugger, you can use gh (Go With Exception Handled) to disregard the error and continue running the target. You can use gn (Go With Exception Not Handled) to bypass the kernel debugger and go on to step 4.
If the conditions in steps 1, 2, and 3 do not apply, Windows will activate a debugging tool. Any program can be selected in advance as the tool to use in this situation. The chosen program is referred to as the postmortem debugger. This is also known as the just-in-time debugger or the JIT debugger.
If the postmortem debugger is a standard user-mode debugger (such as CDB, WinDbg, or Microsoft Visual Studio), this debugger will start up and break into your application.

If the postmortem debugger is a tool for writing dump files (such as Dr. Watson), a memory dump file will be created, and then the application will be terminated.

Note If Dr. Watson is activated on Windows XP or a later version of Windows, a message box will appear. This window gives you the option of sending an error report to Microsoft. If you choose Don't Send, a dump file will created and stored on your hard disk. If you choose Send Error Report, a dump file will be created and stored on your hard disk, and will also be transmitted to Microsoft over the internet.

If you have not reconfigured Windows' postmortem settings, Dr. Watson is used as the default postmortem debugger. This setting can be changed programmatically or through the registry; any changes take effect immediately.

To change the postmortem debugger to WinDbg, run windbg -I. (The I must be capitalized.) This command will display a success or failure message after it is used. When WinDbg is the postmortem debugger, it will be activated whenever an application crashes.
To change the postmortem debugger to CDB, run cdb -iae or cdb -iaec KeyString.. When the -iaec switch is used, KeyString specifies a string to be added to the end of the AeDebug registry key. This command will display no message if it succeeds, but will display a failure message if it fails. When CDB is the postmortem debugger, it will be activated whenever an application crashes.
To change the postmortem debugger to NTSD, run ntsd -iae or ntsd -iaec KeyString.. When the -iaec switch is used, KeyString specifies a string to be added to the end of the AeDebug registry key. If KeyString contains spaces, it must be enclosed in quotation marks. This command will display no message if it succeeds, but will display a failure message if it fails. When NTSD is the postmortem debugger, it will be activated whenever an application crashes.
To change the postmortem debugger back to Dr. Watson, run drwtsn32 -i. When Dr. Watson is the postmortem debugger, a memory dump file will be written to disk if an application crashes. See Dr. Watson Command-Line Options for details.

Only a system administrator can alter the postmortem settings.

If a postmortem debugger has been installed, you can deliberately break into the debugger from a user-mode application by calling the DebugBreak function.

Editing the Registry

The postmortem debugging settings are stored in the registry. If you wish to control these settings, it is recommended that you use the WinDbg, CDB, NTSD, or Dr. Watson commands described above; these will automatically change the relevant registry keys. If you do need to manually edit the registy, do so very carefully, since improper changes to the registry may render Windows unusable.

On an x86 computer, the postmortem settings are stored in the \\HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug key.

On an Intel Itanium computer, there are two registry keys used for postmortem debugging:

A failing 64-bit application will be debugged according to the settings stored in the \\HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug key.
A failing 32-bit application will be debugged according to the settings stored in the \\HKEY_LOCAL_MACHINE\Software\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug key. However, if the Debugger value in this key specifies an application in the %windir%\system32 directory, Windows will look in %windir%\syswow64 instead.

When using an Itanium computer, it is often simplest to place identical values in each of these keys. In particular, if you are using WinDbg, CDB, or NTSD as your postmortem debugger, the same version will work for 32-bit and 64-bit user-mode applications on an Itanium computer. See Choosing a 32-bit or 64-bit Debugger Package for details.

There are two registry values that should appear in these keys:

Debugger: This REG_SZ value specifies the debugger that will handle postmortem debugging. The full path to the debugger must be listed, unless the debugger is located in a directory that is in the default path.
Auto: This REG_SZ value is always either 0 or 1. If Auto is set to 0, a message box will be displayed prior to postmortem debugging.

When an unhandled application error occurs, Windows checks to see if the Debugger and Auto registry values exist.

If the Auto value is 0, a message box will appear.

In Windows NT and Windows 2000, the message box will have one of the following formats:

If the Debugger value contains the name of a valid debugger or Dr. Watson, the message box will have two buttons: OK and Cancel. If the OK button is pressed, the application will be terminated. If the Cancel button is pressed, the tool specified in the Debugger value will be started.
If the Debugger value is empty, the message box will have only an OK button and no debugger will start.

In Windows XP and later versions of Windows, the message box will have one of the following formats:

If the Debugger value contains the name of a valid debugger or Dr. Watson, the message box will have three buttons: Send Error Report, Don't Send, and Debug. If the Don't Send button is pressed, the application will be terminated. If the Send Error Report button is pressed, a minidump file will be sent to Microsoft and the application will be terminated. If the Debug button is pressed, the tool specified in the Debugger value will be started. (Note that all of these buttons have different effects than the buttons on the message window for Dr. Watson when the Auto value does not equal 0.)
If the Debugger value is empty, the message box will have only the Send Error Report and Don't Send buttons.

If the Auto value equals 1, no message box appears. The debugger referred to in the Debugger value is automatically started.

Registry Examples

The following registry values can be used to set Dr. Watson as the postmortem debugger (this is the default):

Debugger = "drwtsn32 -p %ld -e %ld -g"
Auto = 1

The following values can be used to set WinDbg as the postmortem debugger:

Debugger = "Path\WinDbg -p %ld -e %ld"
Auto = 1

The following values can be used to set CDB as the postmortem debugger:

Debugger = "Path\CDB -p %ld -e %ld -g"
Auto = 1

In these examples, Path is the directory where the debugger is located, -p %ld specifies the process ID that will be debugged, -e %ld provides the event that caused the exception, and -g causes the debugger to skip the initial breakpoint. (Dr. Watson ignores the -g option.)

Security Vulnerabilities

If you are considering enabling postmortem debugging on a computer that you share with other people, see Security During Postmortem Debugging.

Mismatched PDBs --- Why?

junfeng — Mon, 05 Nov 2007 19:00:00 GMT

If you are developing software, most likely you have encountered the "mismatched pdbs" debugger error. It usually happens when you point the debugger to the wrong symbol path.

But there are times you are confident that the symbols you point to are the correct symbols, and it left you wondering why the debugger believes the symbols do not match.

"!sym noisy" will tell you how the windows debugger searches for symbols, and report mismatched pdbs when encountered, but it does not tell you why the symbols do not match.

The answer is in a debugger extension !itoldyouso.

0:000> !itoldyouso

!IToldYouSo <module> [symbol]

!IToldYouSo tests the validity of a module against a symbol file.
The module can be specified by either its name or base address.
If a symbol file is not specified, then the loaded symbol is tested.
Otherwise, if a pdb or dbg symbol file path is specified, it is tested
against the loaded module.

0:000> !itoldyouso foo c:\temp\foo.pdb

C:\temp\foo.dll
    Timestamp: 4549BCD2
SizeOfImage: BF000
          pdb: foo.pdb
      pdb sig: 18D24151-2AA3-4917-9980-58CB02F5E5F9
          age: 2

foo.pdb
pdb sig: 18D24151-2AA3-4917-9980-58CB02F5E5F9
age: 2

MATCH: foo.pdb and C:\temp\foo.dll

0:000> !itoldyouso foo c:\temp\foo1.pdb

C:\temp\foo.dll
    Timestamp: 4549BCD2
SizeOfImage: BF000
          pdb: foo.pdb
      pdb sig: 18D24151-2AA3-4917-9980-58CB02F5E5F9
          age: 2

foo.pdb
pdb sig: 4B81E62F-84A3-4864-8ADF-8AD0F9EFEEDB
age: 2

MISMATCH: foo.pdb and C:\temp\foo.dll

MiniDumpAuxiliaryDlls and KnownManagedDebuggingDlls

junfeng — Mon, 20 Aug 2007 18:00:00 GMT

I accidentally stumble on these two registry keys. And the expert in Windows Debugger team answers my question.

-----Original Message-----
From:
Subject: RE: managed minidump

Auxiliary DLLs are loaded inside of MiniDumpWriteDump when it finds a registered auxiliary DLL for a module in the target process. The lookup is to take the full path of the module and see if there's a registered auxiliary DLL. You can't have multiple aux DLLs for a single module path.

-----Original Message-----

From: Junfeng Zhang

Subject: RE: managed minidump

When are auxiliary dlls loaded?

What is the behavior when there are multiple entries under each key?

-----Original Message-----

From:

Subject: RE: managed minidump

Both are filled with string values of the form <full DLL path> = <full DLL path>. MiniDumpAuxiliaryDlls lists helper DLLs that the minidump code can use to get additional data during dump generation. For example, mscorwks.dll has a registered auxiliary of mscordacwks.dll, which provides extra CLR memory data for a minidump.

KnownManagedDebuggingDlls is a security measure so that a debugger, when attempting to load extra support DLLs for managed debugging, can know what DLLs are approved for use on the system. The CLR registers mscordacwks.dll here, for example.

Both are kept in HKLM so that they can only be written by an admin.

-----Original Message-----

From: Junfeng Zhang

Subject: managed minidump

How does OS use the following two registry keys?

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\MiniDumpAuxiliaryDlls

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\KnownManagedDebuggingDlls

Loading CLR DAC dll from a different path

junfeng — Mon, 13 Aug 2007 18:00:00 GMT

From .Net framework 2.0, CLR creates a debugger helper dll mscordacwks.dll. The dll is used by Windows Debugger to enumerate CLR data structures.

Mscordacwks.dll is part of .Net framework redist.

C:\Windows\Microsoft.NET\Framework\v2.0.50727>dir mscordacwks.dll

Volume in drive C is Vista

Volume Serial Number is C84C-2424

Directory of C:\Windows\Microsoft.NET\Framework\v2.0.50727

07/10/2007 04:38 PM 812,544 mscordacwks.dll

1 File(s) 812,544 bytes

When debugging a .Net framework application, Windows Debugger loads the CLR DAC dll based on where CLR (mscorwks.dll) is loaded.

You can use .cordll command to ask Windows Debugger to load the CLR DAC dll from a different path.

.cordll -lp c:\otherpath

Windows Debugger may have cached the CLR DAC dll. So you may have to unload it first.

.cordll -u -lp c:\otherpath

Full documentation on .cordll below:

.cordll (Control CLR Debugging)

The .cordll command controls managed code debugging and the Microsoft .NET common language runtime (CLR).

Syntax

.cordll [Options]

Parameters

Options

One or more of the following options:

-l

Loads the CLR debugging modules.

-u

Unloads the CLR debugging modules.

-e

Enables CLR debugging.

-d

Disables CLR debugging.

-D

Disables CLR debugging and unloads the CLR debugging modules.

-N

Reloads the CLR debugging modules.

-lp Path

Specifies the directory path of the CLR debugging modules.

-se

Enables using the short name of the CLR debugging module, Mscordacwks.dll.

-sd

Disables using the short name of the CLR debugging module, Mscordacwks.dll. Instead, the debugger uses the long name of the CLR debugging module, Mscordacwks_<spec>.dll. Turning off short name usage enables you to avoid having your local CLR used if you are concerned about mismatches.

-ve

Turns on verbose mode for CLR module loading.

-vd

Turns off verbose mode for CLR module loading.

Finding Memory leaks

junfeng — Mon, 23 Jul 2007 18:00:16 GMT

MSDN and Microsoft support web site have some great information on how to find memory leaks, both for managed application and native applications.

Unmanaged:

How to use Umdh.exe to find memory leaks

http://support.microsoft.com/kb/268343

Managed:

Debugging Memory Problems

http://msdn2.microsoft.com/en-us/library/ms954591.aspx

Unknown exception 0xe0434f4e

junfeng — Mon, 16 Jul 2007 18:00:00 GMT

If you attach windows debugger to a .Net framework application, occasionally you may see debug spew of unknown exception 0xe0434f4e.

(cfc.1244): Unknown exception - code e0434f4e (first chance)

0xe0434f4e is defined as EXCEPTION_HIJACK.

(Copied from Rotor 2.0 sources corexcep.h)

#define EXCEPTION_HIJACK 0xe0434f4e // 0xe0000000 | 'COM'+1

EXCEPTION_HIJACK is thrown by CLR as part of the process of suspending threads for garbage collection. It's raised to help resume execution after a suspension.

Break on exception in Windows Debugger

junfeng — Thu, 05 Jul 2007 18:00:00 GMT

In Windows Debugger, we can break on C++ exception by doing "sxe eh". We can also break on CLR exception by doing "sxe clr".

But sometimes you may see the following debug spew:

(94c.960): C++ EH exception - code e06d7363 (first chance)
(94c.960): Unknown exception - code e0434352 (first chance)
(94c.960): Unknown exception - code e0434352 (first chance)
(94c.960): Unknown exception - code e0434352 (first chance)
(94c.960): Unknown exception - code e0434352 (first chance)

To break on the unknown exception, we can use "sxe e0434352".

0:029> sxe e0434352
0:029> g
(94c.cf4): Unknown exception - code e0000001 (first chance)
(94c.f1c): C++ EH exception - code e06d7363 (first chance)
(94c.f1c): Unknown exception - code e0434352 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0584dfe8 ebx=00000001 ecx=00000000 edx=05813000 esi=0584e080 edi=00000005
eip=7c812a5b esp=0584dfe4 ebp=0584e038 iopl=0         nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000206
kernel32!RaiseException+0x53:
7c812a5b 5e              pop     esi

To remove the break point, use "sxd e0434352"