Decrypt my World : Debugging

CryptographicException: The data to be decrypted exceeds the maximum for this modulus of 0 bytes

alejacma — Fri, 09 Oct 2009 14:06:00 GMT

Hi all,

Some time ago a customer of mine was getting the following CryptographicException when calling EncryptedXml.DecryptDocument method to decrypt an XmlDocument in their .NET app:

[System.Security.Cryptography.CryptographicException] = {"The data to be decrypted exceeds the maximum for this modulus of 0 bytes"}
_HResult = -2146233296
_HResult = 0x80131430
No Inner Exception

This was only happening with a smart card certificate associated to a specific third-party CSP (non-MS).

In order to troubleshoot this, I used my CryptoAPI Tracer script to get some traces while reproducing the issue. Thanks to my script I could see these calls that .NET is making to the third-party CSP behind the scenes (I will omit key container and CSP names as they are not needed to understand the issue):

>>>>>>>>>>>>>>>>>>>>>>

CryptAcquireContextA (0x570)

IN
pszContainer
001bf1a0 "Cert Container Name"

pszProvider
001b4698 "Third-party CSP"

dwProvType
PROV_RSA_FULL

dwFlags
0

OUT
hProv
0x1bea28

RESULT
CryptAcquireContextA (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetUserKey (0x570)

IN
hProv
0x1bea28

dwKeySpec
AT_KEYEXCHANGE

OUT
hUserKey
0x1bf1a0

RESULT
CryptGetUserKey (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_ALGID

pbData
NULL

dwDataLen
0

dwFlags
0

OUT
dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_ALGID

pbData
0x1bd228

dwDataLen
4

dwFlags
0

OUT
bData
001bd228 0000a400

dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_KEYLEN

pbData
NULL

dwDataLen
0

dwFlags
0

OUT
dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_KEYLEN

pbData
0x1bd228

dwDataLen
4

dwFlags
0

OUT
bData
001bd228 00000000

dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

These calls mean the following: before .NET tries to decrypt the XML, it asks the CSP for some information on the key it will use to decrypt, like its length (CryptGetKeyParam API with KP_KEYLEN flag). For some unknown reason the third-party CSP is returning a length of 0 for the key (pbData points to 0), which is invalid, thus .NET won't continue with the decryption and will raise the exception we've seen. Why is the CSP returning an invalid length? Only the CSP developers can answer that question.

In my customer's particular case, the CSP developers quickly localized and fixed the issue.

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Locks

alejacma — Tue, 29 Sep 2009 11:19:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 2.

LOCKS

· We can see threads waiting for managed locks:

All managed objects have a syncblock (4 bytes) before their Method Table and properties. It can hold locking information for thread-safe operations. All syncblocks being used are stored in a table. A class like System.Threading.Monitor will use them to synchronize access to resources.

With this command we get the Index of the syncblock in the table, the address of the syncblock, the thread holding it & the object which syncblock we are waiting for. Let’s take a look to this sample:

0:006> !SyncBlk

Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner

19 00358ddc 5 2 0032c848 ca8 0 01a2d45c System.Object

-----------------------------

Total 23

CCW 0

RCW 0

ComClassFactory 0

Free 0

Thread 0 is holding another 2 threads (MonitorHeld = 1 for each owner and 2 for each waiter) with a synchronization object.

We can analyze the thread call stacks and see which threads may be held by thread 0:

0:006> ~*kpL

...

4 Id: 1868.ac Suspend: 1 Teb: 7ffdb000 Unfrozen

ChildEBP RetAddr

0461e89c 77550690 ntdll!KiFastSystemCallRet(void)

0461e8a0 76207e09 ntdll!ZwWaitForMultipleObjects(void)+0xc

0461e93c 79ed98fd KERNEL32!WaitForMultipleObjectsEx(unsigned long nCount = 0x461e8f0, void ** lpHandles = 0x00358df0, int bWaitAll = 0, unsigned long dwMilliseconds = 0xffffffff, int bAlertable = 1)+0x11d

0461e9a4 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT(unsigned long nCount = 1, void ** lpHandles = 0x00358df0, int bWaitAll = 0, unsigned long dwMilliseconds = 0xffffffff, int bAlertable = 1)+0x6f

0461e9c4 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait(int numWaiters = 1, void ** pHandles = 0x00358df0, int bWaitAll = 0, unsigned long timeout = 0xffffffff, WaitMode mode = WaitMode_Alertable (1))+0x3c

0461ea48 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker(int countHandles = 1, void ** handles = 0x00358df0, int waitAll = 0, unsigned long millis = 0xffffffff, WaitMode mode = WaitMode_Alertable (1))+0x13c

0461ea98 79ed9a62 mscorwks!Thread::DoAppropriateWait(int countHandles = 1, void ** handles = 0x00358df0, int waitAll = 0, unsigned long millis = 0xffffffff, WaitMode mode = WaitMode_Alertable (1), struct PendingSync * syncState = 0x00000000)+0x40

0461eaf4 79e78944 mscorwks!CLREvent::WaitEx(unsigned long dwMilliseconds = 0xffffffff, WaitMode mode = WaitMode_Alertable (1), struct PendingSync * syncState = 0x00000000)+0xf7

0461eb08 79ed7b37 mscorwks!CLREvent::Wait(unsigned long dwMilliseconds = 0xffffffff, int alertable = 1, struct PendingSync * syncState = 0x00000000)+0x17

0461eb94 79ed7a9e mscorwks!AwareLock::EnterEpilog(class Thread * pCurThread = 0x00386aa0, int timeOut = -1)+0x8c

0461ebb0 79ebd7e4 mscorwks!AwareLock::Enter(void)+0x61

0461ec50 0080201c mscorwks!JIT_MonEnterWorker_Portable(class Object * obj = 0x00386aa0)+0xb3

0461ec70 793b0d1f WindowsApplication1!WindowsApplication1.Form1.Run8(<HRESULT 0x80004001>)+0x24

...

5 Id: 1868.1fa8 Suspend: 1 Teb: 7ffd9000 Unfrozen

ChildEBP RetAddr

0446eac4 77550690 ntdll!KiFastSystemCallRet(void)

0446eac8 76207e09 ntdll!ZwWaitForMultipleObjects(void)+0xc

0446eb64 79ed98fd KERNEL32!WaitForMultipleObjectsEx(unsigned long nCount = 0x446eb18, void ** lpHandles = 0x00358df0, int bWaitAll = 0, unsigned long dwMilliseconds = 0xffffffff, int bAlertable = 1)+0x11d

0446ebcc 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT(unsigned long nCount = 1, void ** lpHandles = 0x00358df0, int bWaitAll = 0, unsigned long dwMilliseconds = 0xffffffff, int bAlertable = 1)+0x6f

0446ebec 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait(int numWaiters = 1, void ** pHandles = 0x00358df0, int bWaitAll = 0, unsigned long timeout = 0xffffffff, WaitMode mode = WaitMode_Alertable (1))+0x3c

0446ec70 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker(int countHandles = 1, void ** handles = 0x00358df0, int waitAll = 0, unsigned long millis = 0xffffffff, WaitMode mode = WaitMode_Alertable (1))+0x13c

0446ecc0 79ed9a62 mscorwks!Thread::DoAppropriateWait(int countHandles = 1, void ** handles = 0x00358df0, int waitAll = 0, unsigned long millis = 0xffffffff, WaitMode mode = WaitMode_Alertable (1), struct PendingSync * syncState = 0x00000000)+0x40

0446ed1c 79e78944 mscorwks!CLREvent::WaitEx(unsigned long dwMilliseconds = 0xffffffff, WaitMode mode = WaitMode_Alertable (1), struct PendingSync * syncState = 0x00000000)+0xf7

0446ed30 79ed7b37 mscorwks!CLREvent::Wait(unsigned long dwMilliseconds = 0xffffffff, int alertable = 1, struct PendingSync * syncState = 0x00000000)+0x17

0446edbc 79ed7a9e mscorwks!AwareLock::EnterEpilog(class Thread * pCurThread = 0x0038ec10, int timeOut = -1)+0x8c

0446edd8 79f59024 mscorwks!AwareLock::Enter(void)+0x61

0446ee40 79fc6352 mscorwks!AwareLock::Contention(int timeOut = -1)+0x199

0446eee0 0080201c mscorwks!JITutil_MonContention(class AwareLock * lock = 0x00358ddc)+0xa3

0446ef00 793b0d1f WindowsApplication1!WindowsApplication1.Form1.Run8(<HRESULT 0x80004001>)+0x24

...

0:006> ~4e !CLRStack -p

OS Thread Id: 0xac (4)

ESP EIP

0461eb30 77550f34 [GCFrame: 0461eb30]

0461ec00 77550f34 [HelperMethodFrame_1OBJ: 0461ec00] System.Threading.Monitor.Enter(System.Object)

0461ec58 0080201c WindowsApplication1.Form1.Run8()

...

So thread 5 is indeed waiting for our problematic syncblock, and I guess thread 4, too. But I couldn’t see any reference to the syncblock or the related object in the parameters of the call stack of thread 4. Well, the object we used with Monitor.Enter should be near the top of the stack of the thread, as we passed it as a parameter to the managed function at the top of the managed call stack:

0:006> ~4e !dso

OS Thread Id: 0xac (4)

ESP/REG Object Name

0461eb68 01a2d45c System.Object

0461ebc0 01a2d45c System.Object

0461ebd4 01a2d45c System.Object

0461ec30 01a669a4 System.Threading.ThreadStart

0461ec3c 01a2d45c System.Object

0461ec58 01a669fc System.Threading.ThreadHelper

...

And indeed it is, so both threads 4 & 5 are the ones being held by thread 0.

· We can see threads waiting for unmanaged locks:

A class like System.Threading.Mutex is not based on syncblocks, but on kernel mutexes instead. When not working with syncblocks, we can use the same unmanaged commands we’ll use with any unmanaged app:

0:006> !locks

0:006> !critsec

0:006> !SIEExtPub.critlist

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Note: this is the last post on the MANAGED DEBUGGING with WINDBG series. I hope my posts help you the same way they help me in my everyday work.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 2

alejacma — Tue, 29 Sep 2009 11:07:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 1.

BREAKING ON AN EXCEPTION. Part 2

What happens if !PrintException doesn’t return anything, or when we break (i.e. with a breakpoint or manually – Ctrl+Break) we can’t see mscorwks!RaiseTheExceptionInternalOnly in the call stack? What if we already missed the exception when we break?

We could still find the exception near the top of the managed stack of the current thread:

0:000> !DumpStackObjects

...

0:000> !dso

OS Thread Id: 0x15e4 (0)

ESP/REG Object Name

0020eb50 0195fe94 System.IndexOutOfRangeException

0020eb98 0195fe94 System.IndexOutOfRangeException

0020ebdc 01929a04 WindowsApplication1.Form1

0020ebe0 01941958 System.Int32[]

...

0:000> !pe 0x0195fe94

...

If not, we can try to find significant exceptions in the managed heap, as all recent exceptions are stored there:

0:000> !DumpHeap -type Exception

Address MT Size

01991024 790fe044 72

0199106c 790fe0e0 72

019910b4 790fe17c 72

019910fc 790fe284 72

01991144 790fe284 72

0199f848 79103c58 12

0199f860 79103ca4 12

019df490 79117f8c 72

019fc5c8 79117f8c 72

total 9 objects

Statistics:

MT Count TotalSize Class Name

79103ca4 1 12 System.Text.DecoderExceptionFallback

79103c58 1 12 System.Text.EncoderExceptionFallback

790fe17c 1 72 System.ExecutionEngineException

790fe0e0 1 72 System.StackOverflowException

790fe044 1 72 System.OutOfMemoryException

79117f8c 2 144 System.IndexOutOfRangeException

790fe284 2 144 System.Threading.ThreadAbortException

Total 9 objects

0:000> !DumpHeap -mt 79117f8c

Address MT Size

019df490 79117f8c 72

019fc5c8 79117f8c 72

total 2 objects

Statistics:

MT Count TotalSize Class Name

79117f8c 2 144 System.IndexOutOfRangeException

Total 2 objects

0:000> !pe 019df490

...

0:000> !pe 019fc5c8

...

We can also do the following to find all exceptions of a given type in the managed heap:

0:000> .foreach(ex {!dumpheap -type System.IndexOutOfRangeException -short}){!pe ex;.echo *************}

Exception object: 019df490

Exception type: System.IndexOutOfRangeException

Message: Index was outside the bounds of the array.

InnerException: <none>

StackTrace (generated):

SP IP Function

0017EE80 00861AAD WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55

StackTraceString: <none>

HResult: 80131508

*************

Exception object: 019fc5c8

Exception type: System.IndexOutOfRangeException

Message: Index was outside the bounds of the array.

InnerException: <none>

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131508

*************

Note that exception objects are normally created when they are needed. But there are a few exception objects which are created when process starts as they can’t be created when they need to be raised: OutOfMemoryException, ExecutionEngineException, StackOverflowException. So we will always see these exceptions in the heap. That doesn’t mean that they were ever raised.

· We can inspect the method where we got the exception:

We can get i.e. the source code line which failed:

0:000> !pe 019df490

Exception object: 019df490

Exception type: System.IndexOutOfRangeException

Message: Index was outside the bounds of the array.

InnerException: <none>

StackTrace (generated):

SP IP Function

0017EE80 00861AAD WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55

StackTraceString: <none>

HResult: 80131508

0:000> !u 00861AAD

Normal JIT generated code

WindowsApplication1.Form1.PlayWithArray(Int32[])

Begin 00861a58, size c0

00861a58 55 push ebp

...

00861aa3 3b5104 cmp edx,dword ptr [ecx+4]

00861aa6 7205 jb WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55 (00861aad)

00861aa8 e8d67e8c79 call mscorwks!JIT_RngChkFail (7a129983)

>>> 00861aad 03449108 add eax,dword ptr [ecx+edx*4+8]

00861ab1 7105 jno WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x60 (00861ab8)

00861ab3 e8837f8c79 call mscorwks!JIT_Overflow (7a129a3b)

...

00861b16 5d pop ebp

00861b17 c3 ret

0:000> u 00861aa8

WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x50 [C:\__WORKSHOP\Demos\BuggyNETApp\Form1.vb @ 135]:

00861aa8 e8d67e8c79 call mscorwks!JIT_RngChkFail (7a129983)

...

Note that not all exceptions we saw have the StackTrace field set. I’ve seen that when the debugger just breaks on an exception, the field is empty. Once we go beyond that point and we i.e. enter the “catch” section of the “try{}catch(){}” statement in the code, that field is set.

But if we just broke on an exception, we can still see the call stack with k, !CLRStack, !DumpStack and related commands as we saw in the Call Stack section before, and get the instruction pointer (IP) we need from there.

· We can stop breaking on CLR exceptions:

If we don’t want to break on any other CLR exception we do the following:

0:000> sxn clr

If we were breaking on specific exception types (!StopOnException) before, and we stop breaking on all CLR exceptions, we will still see those exceptions in the debugger:

0:000> g

(1728.1f58): C++ EH exception - code e06d7363 (first chance)

(1728.1f58): CLR exception - code e0434f4d (first chance)

'System.IndexOutOfRangeException hit'

Next post: MANAGED DEBUGGING with WINDBG. Locks.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 1

alejacma — Mon, 24 Aug 2009 12:57:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 5.

BREAKING ON AN EXCEPTION. Part 1

We can only break on exceptions when doing live debugging, but many of the commands explained here can be used when doing dump analysis, too.

· Exceptions we may get in .NET applications:

§ CLR exception (0xe0434f4d).

§ Stack Overflow exception (0x800703e9).

§ Access Violation (0xc0000005).

§ Integer divide-by-zero exception (0xc0000094).

§ ...

§ Fatal Execution Engine Error (<address of failure>). This normally indicates a bug in CLR or NT Heap corruption when doing P/Invoke with not big enough buffers. The address associated to this exception will tell us where in the code the issue happened.

The debugger will break by default when the process raises most of these exception types, but not on CLR exceptions, for instance.

In ASP.NET, if exceptions are raised in managed threads and they are unhandled, Global Error Handle catches them. But if they are raised in another thread like Timer or Finalizer threads, we will crash the process (it will be recycled).

· We can break on any CLR exception:

When our code raises a CLR exception, we see the following line in the debugger:

(1728.1f58): CLR exception - code e0434f4d (first chance)

Unmanaged world doesn’t know about specific CLR exception types, so all of them are raised with the same exception code: 0xe0434f4d.

As we’ve seen, WinDbg won’t break on any CLR exception by default, as they may be expected. We can break on all CLR exceptions with any of these commands:

0:000> sxe clr

0:000> sxe 0xe0434f4d

And when we break we’ll see something like this:

0:000> g

(1728.1f58): C++ EH exception - code e06d7363 (first chance)

(1728.1f58): CLR exception - code e0434f4d (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

eax=002de8e0 ebx=e0434f4d ecx=00000001 edx=00000000 esi=002de968 edi=0047c600

eip=77a1b09e esp=002de8e0 ebp=002de930 iopl=0 nv up ei pl nz ac po nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212

KERNEL32!RaiseException+0x58:

77a1b09e c9 leave

· We can break on a specific CLR exception type:

If we know the exact exception type, we can do the following:

0:004> !StopOnException -create System.IndexOutOfRangeException 1

Breakpoint set

0:004> !soe -create System.IndexOutOfRangeException 1

Breakpoint set

And when we break we’ll see something like this:

0:000> g

(1728.1f58): C++ EH exception - code e06d7363 (first chance)

(1728.1f58): CLR exception - code e0434f4d (first chance)

'System.IndexOutOfRangeException hit'

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

...

77a1b09e c9 leave

We can also stop only on those exceptions which type derives from a given one:

0:004> !soe -derived -create System.SystemException 1

Breakpoint set

· When we break, we can inspect the CLR exception:

Let’s take a look to the call stack of the thread which raised the exception when the debugger breaks:

0:000> kpL

ChildEBP RetAddr

002de930 79f071ac KERNEL32!RaiseException(unsigned long dwExceptionCode = 0xe0434f4d, unsigned long dwExceptionFlags = 1, unsigned long nNumberOfArguments = 1, unsigned long * lpArguments = 0x002de968)+0x58

002de990 79f9293a mscorwks!RaiseTheExceptionInternalOnly(class Object * throwable = 0x01940e4c, int rethrow = 0, int fForStackOverflow = 0)+0x2a8

002de9c8 7a129a34 mscorwks!UnwindAndContinueRethrowHelperAfterCatch(class Frame * pEntryFrame = 0x002dea20, class Exception * pException = 0x004d2aa8)+0x70

002dea68 00b81aad mscorwks!JIT_RngChkFail(void)+0xb0

002deab0 00b819fe WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(<HRESULT 0x80004001>)+0x55

KERNEL32!RaiseException raises the generic CLR exception (0xe0434f4d) to the unmanaged world, but the specific exception we are getting is the one that mscorwks!RaiseTheExceptionInternalOnly raises. We can inspect that specific exception as any other .NET object:

0:000> !DumpObj 0x0195fe94

...

0:000> !do 0x0195fe94

Name: System.IndexOutOfRangeException

MethodTable: 79117f8c

EEClass: 79117f1c

Size: 72(0x48) bytes

(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:

MT Field Offset Type VT Attr Value Name

790fd8c4 40000b5 4 System.String 0 instance 00000000 _className

7910ebc8 40000b6 8 ...ection.MethodBase 0 instance 00000000 _exceptionMethod

790fd8c4 40000b7 c System.String 0 instance 00000000 _exceptionMethodString

790fd8c4 40000b8 10 System.String 0 instance 01935108 _message

79116114 40000b9 14 ...tions.IDictionary 0 instance 00000000 _data

790fdf04 40000ba 18 System.Exception 0 instance 00000000 _innerException

790fd8c4 40000bb 1c System.String 0 instance 00000000 _helpURL

790fd0f0 40000bc 20 System.Object 0 instance 00000000 _stackTrace

790fd8c4 40000bd 24 System.String 0 instance 00000000 _stackTraceString

...

79102290 40000c1 38 System.Int32 0 instance -2146233080 _HResult

790fd8c4 40000c2 30 System.String 0 instance 00000000 _source

...

0:000> !do 01935108

Name: System.String

MethodTable: 790fd8c4

EEClass: 790fd824

Size: 102(0x66) bytes

(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

String: Index was outside the bounds of the array.

Fields:

...

Or we can inspect the exception in a more user friendly way:

0:000> !PrintException 0x0195fe94

...

0:000> !pe 0x0195fe94

Exception object: 0195fe94

Exception type: System.IndexOutOfRangeException

Message: Index was outside the bounds of the array.

InnerException: <none>

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131508

We can save some time if we directly inspect the last thrown exception on the current thread:

0:000> !pe

Exception object: 0195fe94

Exception type: System.IndexOutOfRangeException

Message: Index was outside the bounds of the array.

InnerException: <none>

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131508

Next post: MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 2.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 5

alejacma — Mon, 24 Aug 2009 12:50:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 4.

MANAGED HEAP. Part 5

· We can troubleshoot Loader Heap issues:

An assembly can’t be unloaded while the AppDomain where it was created is still alive. We will get a memory leak if we keep creating new assemblies continuously. The problem is that some .NET operations result on new assemblies being created behind the scenes:

§ Create a new XmlSerializer with any constructor but XmlSerializer(type) or XmlSerializer(type, defaultnamespace).

Memory usage is high when you create several XmlSerializer objects in ASP.NET

§ Make XSLT transformations where XSLT Transform contains a script.

PRB: Cannot unload assemblies that you create and load by using script in XSLT

§ Change ASPX, ASCX or ASMX pages which cause re-compilations.

This could be avoided by caching XMLSerializers and XSLT Transforms, and using maxNumRecompiles value which is set to 15 by default.

Imagine that we’ve detected a memory leak in our ASP.NET app, and PerfMon shows that during the time of the memory leak “.NET CLR Loading:Bytes in loader heap” is constantly increasing. The same thing is true for “.NET CLR Loading:Current Assemblies”. We check the number of assemblies in the heap:

0:000> !DumpHeap -stat -type Assembly

------------------------------

Heap 0

total 1141 objects

------------------------------

Heap 1

total 1220 objects

------------------------------

total 2361 objects

Statistics:

MT Count TotalSize Class Name

7912ad0c 1 24 System.Collections.Generic.List`1[[System.Reflection.Assembly, mscorlib]]

...

790fc79c 2297 55128 System.Reflection.Assembly

Total 2361 objects

And we try to get some info about those assemblies:

0:000> !DumpDomain

...

--------------------------------------

Domain 2: 001cb1d8

LowFrequencyHeap: 001cb1fc

HighFrequencyHeap: 001cb258

StubHeap: 001cb2b4

Stage: OPEN

SecurityDescriptor: 001c9ab0

Name: /LM/w3svc/1/ROOT/DebuggingWorkshop-1-128030642795871345

Assembly: 001b6660 [C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]

ClassLoader: 001773b8

SecurityDescriptor: 642a2e7e

Module Name

790c2000 C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

02272380 C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp

02272010 C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp

...

Assembly: 0020b220 [C:\WINDOWS\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll]

ClassLoader: 0020c808

SecurityDescriptor: 642a2e7e

Module Name

7ae72000 C:\WINDOWS\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll

Assembly: 0020e778 [3k2vmufw, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]

ClassLoader: 0020e800

SecurityDescriptor: 642a2e7e

Module Name

02837b58 3k2vmufw, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

Assembly: 0021c050 [zxsplww6, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]

ClassLoader: 0021c0d8

SecurityDescriptor: 642a2e7e

Module Name

02837f34 zxsplww6, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

...

Assembly: 14bdbf88 [oy_7tsbs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]

ClassLoader: 14bdc010

SecurityDescriptor: 642a2e7e

Module Name

19a273d8 oy_7tsbs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

0:000> !DumpModule 19a273d8

Name: oy_7tsbs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

Attributes: PEFile

Assembly: 14bdbf88

LoaderHeap: 00000000

TypeDefToMethodTableMap: 19bf6158

TypeRefToMethodTableMap: 19bf6170

MethodDefToDescMap: 19bf61dc

FieldDefToDescMap: 19bf6250

MemberRefToDescMap: 19bf62b8

FileReferencesMap: 19bf63fc

AssemblyReferencesMap: 19bf6400

MetaData start address: 19c312d8 (6176 bytes)

0:000> dc 19c312d8 19c312d8+6176

19c312d8 424a5342 00010001 00000000 0000000c BSJB............

...

19c319e8 00010000 0000039b 00000000 6f4d3c00 .............<Mo

19c319f8 656c7564 796f003e 7374375f 642e7362 dule>.oy_7tsbs.d

19c31a08 58006c6c 65536c6d 6c616972 74617a69 ll.XmlSerializat

19c31a18 576e6f69 65746972 72755072 73616863 ionWriterPurchas

19c31a28 64724f65 4d007265 6f726369 74666f73 eOrder.Microsoft

19c31a38 6c6d582e 7265532e 696c6169 6974617a .Xml.Serializati

19c31a48 472e6e6f 72656e65 64657461 65737341 on.GeneratedAsse

19c31a58 796c626d 6c6d5800 69726553 7a696c61 mbly.XmlSerializ

19c31a68 6f697461 6165526e 50726564 68637275 ationReaderPurch

19c31a78 4f657361 72656472 6c6d5800 69726553 aseOrder.XmlSeri

...

We check our code and we can see this:

protected void Page_Load(object sender, EventArgs e)

{

XmlSerializer mySerializer = new XmlSerializer(typeof(PurchaseOrder), new XmlRootAttribute("order"));

...

}

We can see how much memory is taken up by the Loader heap:

0:000> !EEHeap -loader

Loader Heap:

...

--------------------------------------

Module Thunk heaps:

Module 648e8000: Size: 0x0(0)bytes.

Module 68a06000: Size: 0x0(0)bytes.

...

Module 19a273d8: Size: 0x0(0)bytes.

Total size: 0x0(0)bytes

--------------------------------------

Total LoaderHeap size: 0xe33000(14888960)bytes

=======================================

0:000> !DumpModule 19a273d8

Name: oy_7tsbs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

...

Next post: MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 1.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 4

alejacma — Mon, 17 Aug 2009 09:32:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 3.

MANAGED HEAP. Part 4

· We can troubleshoot Heap issues:

These are some of the important performance counters we can check with Performance Monitor when having memory issues with .NET applications:

§ Process:Private Bytes: All the memory that the app is using (committed, could be paged). When this value reaches Virtual Bytes, Virtual Bytes doubles its size. If memory is shared, it won’t count as Private Bytes, but as Working Set.

§ Process:Virtual Bytes: Virtual Memory of a process. Memory can be reserved or used (committed, reflected as Private Bytes). This counter is useful when we get Out of Memory exceptions.

§ Process:Working Set: How much memory is paged in RAM.

§ .NET CLR Memory:#bytes in all heaps: .NET won’t use the NT Heap, but saves some virtual memory for its own heap. If this value doesn’t grow but Virtual Bytes do, the memory leak is in the unmanaged world.

§ .NET CLR Memory:#gen 0/1/2 collections.

§ .NET CLR Memory:Gen 0/1/2 Heap Size.

§ .NET CLR Memory:Large Object Heap Size.

§ .NET CLR Memory:# induced GC: Number of calls to GC.Collect. Calling this function in our code is not recommended, as GC auto-adjust for better performance.

§ .NET CLR Memory:# of pinned objects: Number of objects which reference has been blocked (“pinned”) in memory. GC won’t be able to move pinned objects around memory to optimize available space in heap.

§ .NET CLR Memory:finalization survivors: Number of objects which survived a GC because of having a Finalizer method.

§ .NET CLR Memory:Promoted Finalization Memory from Gen 0/1.

§ .NET CLR Memory:Promoted Memory from Gen 0/1.

§ .NET CLR Memory:% Time in GC.

§ .NET CLR Loading:Bytes in Loader Heap: A continuous increase in the size of the Loader Heap is typically an effect of extensive usage of dynamic modules (.NET CLR Loading:Current Assemblies) or having “debug=true” in ASP.NET apps.

§ .NET CLR Loading:Current Assemblies.

§ .NET CLR Loading:Current appdomains.

Additionally, we can generate a log file which can be used to troubleshoot managed memory usage issues. We can take a look to this log file with CLR Profiler:

0:004> !TraverseHeap c:\memory.log

Writing CLRProfiler format to file c:\memory.log

Gathering types...

tracing roots...

Scan Thread 0 OSTHread 1440

Scan Thread 2 OSTHread 194

Walking heap...

....................................................................................................................................................................................

file c:\memory.log saved

We can also check the heap for signs of corruption:

0:004> !VerifyHeap

-verify will only produce output if there are errors in the heap

· We can troubleshoot GCHandle leaks:

Sometimes the source of a memory leak is a GCHandle leak. If we discard a GCHandle which points to i.e. a big array without freeing it, the array will be kept alive.

We can see statistics about GCHandles in the process:

0:004> !GCHandles

GC Handle Statistics:

Strong Handles: 33

Pinned Handles: 8

Async Pinned Handles: 0

Ref Count Handles: 0

Weak Long Handles: 56

Weak Short Handles: 384

Other Handles: 0

Statistics:

MT Count TotalSize Class Name

790fd0f0 1 12 System.Object

...

7b485894 357 22848 System.Windows.Forms.Internal.DeviceContext

7912d8f8 8 28032 System.Object[]

Total 481 objects

We can also try to track down GCHandle leaks. We can search all of memory for any references to the Strong and Pinned GCHandles in the process:

0:004> !GCHandleLeaks

-------------------------------------------------------------------------------

GCHandleLeaks will report any GCHandles that couldn't be found in memory.

Strong and Pinned GCHandles are reported at this time. You can safely abort the

memory scan with Control-C or Control-Break.

-------------------------------------------------------------------------------

Found 41 handles:

001c1118 001c1120 001c1124 001c1148

...

001c13ec 001c13f0 001c13f4 001c13f8

001c13fc

Searching memory

Reference found in stress log will be ignored

Found 001c1188 at location 0030f23c

Found 001c11fc at location 00424fb8

...

Found 001c11d4 at location 7a3b9c88

------------------------------------------------------------------------------

All handles found.

A leak may still exist because in a general scan of process memory SOS can't

differentiate between garbage and valid structures, so you may have false

positives. If you still suspect a leak, use this function over time to

identify a possible trend.

------------------------------------------------------------------------------

Note that if a handle is found, we’ll see the address of the reference. This might be a stack address or a field within an object, for example.

We can see the object a handle references by dereferencing it:

0:004> !do poi(001c1118)

Name: WindowsApplication1.Form1

...

If a handle is not found in memory, we get a notification:

0:004> !GCHandleLeaks

-------------------------------------------------------------------------------

GCHandleLeaks will report any GCHandles that couldn't be found in memory.

Strong and Pinned GCHandles are reported at this time. You can safely abort the

memory scan with Control-C or Control-Break.

-------------------------------------------------------------------------------

Found 249 handles:

01dc1200 01dc1208 01dc1210 01dc1218

...

Searching memory

Found 01dc1ff8 at location 0012ce48

...

------------------------------------------------------------------------------

Some handles were not found. If the number of not-found handles grows over the

lifetime of your application, you may have a GCHandle leak. This will cause

the GC Heap to grow larger as objects are being kept alive, referenced only

by the orphaned handle. If the number doesn't grow over time, note that there

may be some noise in this output, as an unmanaged application may be storing

the handle in a non-standard way, perhaps with some bits flipped. The memory

scan wouldn't be able to find those.

------------------------------------------------------------------------------

Didn't find 232 handles:

01dc1200 01dc1208 01dc1210 01dc1218

...

0:004> !do poi(01dc1218)

...

If a serious leak is going on, we’ll get an ever-growing set of handle addresses that couldn't be found.

Next post: MANAGED DEBUGGING with WINDBG. Managed Heap. Part 5.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 3

alejacma — Mon, 17 Aug 2009 09:26:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 2.

MANAGED HEAP. Part 3

· We can take a look to which objects reference which objects in the heap:

We may see many objects in the heap that we think should have been cleaned up already. If they haven’t, it’s because they are still being referenced and GC can’t clean them.

We can look for references (“roots”) to an object. Let’s see this with a sample taken from an ASP.NET app:

0:000> !DumpHeap -stat

...

Statistics:

MT Count TotalSize Class Name

...

0ee10034 19218 7149096 ASP.lab3_1_aspx

...

Total 1748273 objects

0:000> !DumpHeap -mt 0ee10034

------------------------------

Heap 0

Address MT Size

...

0556b268 0ee10034 372

0556ccac 0ee10034 372

…

total 6524 objects

------------------------------

Heap 1

...

0:000> !GCRoot 0556b268

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Scan Thread 15 OSTHread 2c4

...

Scan Thread 13 OSTHread c4c

esi:Root:14f002dc(System.Web.HttpResponse)->

14f00194(System.Web.HttpContext)->

14f00a08(ASP.lab3_1_aspx)->

02b33d80(System.Web.Caching.Cache)->

02b33e60(System.Web.Caching.CacheMultiple)->

02b33e78(System.Object[])->

02b33e90(System.Web.Caching.CacheSingle)->

02b33f38(System.Web.Caching.CacheExpires)->

02b33f58(System.Object[])->

02b34878(System.Web.Caching.ExpiresBucket)->

10872ea0(System.Web.Caching.ExpiresPage[])->

094b5734(System.Web.Caching.ExpiresEntry[])->

0556bb94(System.Web.Caching.CacheEntry)->

0556bb74(System.Web.Caching.CacheItemRemovedCallback)->

0556b268(ASP.lab3_1_aspx)

Scan Thread 22 OSTHread 830

Scan Thread 23 OSTHread f7c

If we take a look to the source code of Lab3_1.aspx page, we can see why the page is being kept in memory by the Cache object:

public partial class Lab3_1 : System.Web.UI.Page

{

public void RemovedCallback(String k, Object v, System.Web.Caching.CacheItemRemovedReason r)

{

//do stuff when the item is removed

}

protected void Page_Load(object sender, EventArgs e)

{

UserInfo u = new UserInfo("James", "Dean", "MainStreet 21, NY");

Cache.Add(Guid.NewGuid().ToString(), u, null, System.Web.Caching.Cache.NoAbsoluteExpiration, new TimeSpan(0, 5, 0), System.Web.Caching.CacheItemPriority.NotRemovable, new System.Web.Caching.CacheItemRemovedCallback(this.RemovedCallback));

}

References can exist in several places:

1. In a registry of a thread:

0:022> !gcroot 0x14ef2a34

…

esi:Root: 14f002dc(System.Web.HttpResponse)->

…->

14ef2a34(ASP.lab3_1_aspx)

2. In the stack of a thread:

ESP:ef4f610:Root: 14f00194(System.Web.HttpContext)->

… ->

14ef2a34(ASP.lab3_1_aspx)

3. Within a GCHandle.

Different kind of handles can reference our objects:

- Strong Handles: they keep the object they point to alive until the handle is explicitly freed. Typically a static variable.

DOMAIN(001CAFF8):HANDLE(Strong):284118c:Root: 02b35d3c(System.Threading._TimerCallback)->

… ->

14ef2a34(ASP.lab3_1_aspx)

- Pinned Handles: they are used to prevent the GC from moving an object during collection. Pinning objects can cause heap fragmentation.

DOMAIN(001CCE68):HANDLE(Pinned) …

- Weak Short Handles: they are non null as long as the object they reference is alive because of another Strong reference. They are null otherwise.

DOMAIN(001CCE68):HANDLE(WeakSh) …

- Weak Long Handles: they are non null as long as the object they reference hasn’t been reclaimed by the GC. It can be resurrected.

DOMAIN(001CCE68):HANDLE(WeakLn) …

The distinction between Weak Short and Weak Long comes from the finalization semantics: a Weak Short reference is null if the object can be freed, but a Weak Long reference still refers to the freachable object.

4. In an object ready for finalization.

The object is no longer in use but nobody called its Dispose method.

Finalizer queue:Root: …

5. As a member of an object found in 1, 2 or 3 above.

If our object is referenced by another one, it can’t be garbage collected. We should get rid of that reference or dispose the object that references ours.

…

14ef3914(System.Web.Caching.CacheEntry)->

14ef38f4(System.Web.Caching.CacheItemRemovedCallback)->

14ef2a34(ASP.lab3_1_aspx)

If there is a live reference to an object, that object is said to be strongly rooted. .NET also introduces the notion of weakly rooted references. A weak reference provides a way for programmers to indicate to the GC that they want to be able to access an object, but they don't want to prevent the object from being collected. Such an object is available until it is collected by the GC. For example, you could allocate a large object, and rather than fully deleting and collecting the object, you could hold onto it for possible reuse, as long as there is no memory pressure to clean up the managed heap. Thus, weak references behave somewhat like a cache.

· We can calculate the size of an object:

We can see the size of an object including its child objects. For instance, to determine how much an ASP.NET app is caching, we can do the following:

0:028> !Name2EE * System.Web.Caching.Cache

...

--------------------------------------

Module: 6638c000 (System.Web.dll)

Token: 0x020000f7

MethodTable: 6639d878

EEClass: 6639d808

Name: System.Web.Caching.Cache

--------------------------------------

...

0:028> !DumpHeap -mt 6639d878

------------------------------

Heap 0

Address MT Size

046b9198 6639d878 12

total 1 objects

------------------------------

Heap 1

Address MT Size

total 0 objects

------------------------------

total 1 objects

Statistics:

MT Count TotalSize Class Name

6639d878 1 12 System.Web.Caching.Cache

Total 1 objects

0:028> !ObjSize 046b9198

sizeof(046b9198) = 319128 ( 0x4de98) bytes (System.Web.Caching.Cache)

Next post: MANAGED DEBUGGING with WINDBG. Managed Heap. Part 4.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 2

alejacma — Thu, 13 Aug 2009 10:47:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 1.

MANAGED HEAP. Part 2

· We can take a look to the objects in the heap:

We can see all objects in the heap:

0:004> !DumpHeap

Address MT Size

7b463c40 790fd0f0 12

7b463c4c 790fd8c4 36

7b463c70 790fd8c4 20

...

01905378 79119a1c 32

...

02907de0 003d6d48 16 Free

total 12400 objects

Statistics:

MT Count TotalSize Class Name

7b492254 1 12 System.Windows.Forms.VisualStyles.VisualStyleRenderer+ThemeHandle

...

79119a1c 3 96 System.Security.Policy.PermissionRequestEvidence

...

003d6d48 11 164 Free

...

790fd0f0 165 1980 System.Object

...

790fd8c4 7308 435252 System.String

Total 12400 objects

0:004> !do 01905378

Name: System.Security.Policy.PermissionRequestEvidence

...

0:004> !do 02907de0

Free Object

Size 16(0x10) bytes

Note the existence of Free objects in the heap. The Free entry represents objects which are not referenced and have been garbage collected, but whose memory hasn't been compacted and released to the operating system yet. These regions of space can be re-used by the Garbage Collector later. If 30% or more of the heap contains Free objects, the process may suffer from heap fragmentation.

I’m sure we noticed that there are too many objects in the heap, so it’s usually better to see only the statistics:

0:004> !DumpHeap -stat

total 12400 objects

Statistics:

MT Count TotalSize Class Name

7b492254 1 12 System.Windows.Forms.VisualStyles.VisualStyleRenderer+ThemeHandle

...

79119a1c 3 96 System.Security.Policy.PermissionRequestEvidence

...

003d6d48 11 164 Free

...

790fd0f0 165 1980 System.Object

...

790fd8c4 7308 435252 System.String

Total 12400 objects

We can also focus on the objects of a given type only:

0:004> !Name2EE * System.Security.Policy.PermissionRequestEvidence

Module: 790c2000 (mscorlib.dll)

Token: 0x0200048a

MethodTable: 79119a1c

...

0:004> !DumpHeap -mt 79119a1c

Address MT Size

01901ebc 79119a1c 32

01905378 79119a1c 32

019290c0 79119a1c 32

total 3 objects

Statistics:

MT Count TotalSize Class Name

79119a1c 3 96 System.Security.Policy.PermissionRequestEvidence

Total 3 objects

Or focus on objects of several types with part of the name in common:

0:004> !DumpHeap -type System.Security.Policy

Address MT Size

01901b78 7910ed68 20

...

01901ebc 79119a1c 32

...

0192971c 79119bf4 16

total 38 objects

Statistics:

MT Count TotalSize Class Name

791197b0 1 12 System.Security.PolicyManager

7911c924 1 24 System.Security.Policy.Hash

...

79119a1c 3 96 System.Security.Policy.PermissionRequestEvidence

...

790f97c4 3 156 System.Security.Policy.PolicyLevel

Total 38 objects

We can see the objects in the Large Object Heap, and we can do it in several ways:

0:004> !EEHeap -GC

Number of GC Heaps: 1

generation 0 starts at 0x01901018

generation 1 starts at 0x0190100c

generation 2 starts at 0x01901000

ephemeral segment allocation context: none

segment begin allocated size

00423878 7b463c40 7b47a744 0x00016b04(92932)

0041d178 7a733370 7a754b98 0x00021828(137256)

003d6600 790d8620 790f7d8c 0x0001f76c(128876)

01900000 01901000 01949ff4 0x00048ff4(298996)

Large object heap starts at 0x02901000

segment begin allocated size

02900000 02901000 02907df0 0x00006df0(28144)

Total Size 0xa787c(686204)

------------------------------

GC Heap Size 0xa787c(686204)

0:004> !DumpHeap 02901000 02907df0

Address MT Size

02901000 003d6d48 16 Free

02901010 7912d8f8 4096

...

total 15 objects

Statistics:

MT Count TotalSize Class Name

003d6d48 8 128 Free

7912d8f8 7 28016 System.Object[]

Total 15 objects

0:004> !SOS.DumpHeap -min 85000

...

Note that there are objects in LOH which size is smaller than 85,000 bytes (?).

We can also see the objects which are ready for finalization, and in general, which objects in our app have a Finalize method:

0:004> !FinalizeQueue

SyncBlocks to be cleaned up: 0

MTA Interfaces to be released: 0

STA Interfaces to be released: 0

----------------------------------

generation 0 has 703 finalizable objects (00457998->00458494)

generation 1 has 0 finalizable objects (00457998->00457998)

generation 2 has 0 finalizable objects (00457998->00457998)

Ready for finalization 0 objects (00458494->00458494)

Statistics:

MT Count TotalSize Class Name

7b492254 1 12 System.Windows.Forms.VisualStyles.VisualStyleRenderer+ThemeHandle

...

7ae77fa8 2 80 System.Drawing.Icon

...

7b485894 59 3776 System.Windows.Forms.Internal.DeviceContext

Total 703 objects

0:004> dd 00457998 00458494-4

00457998 01901f10 01905238 01906414 01906428

...

00457b28 019323f8 019324b0 0193296c 01936378

...

00458488 01949468 019494f0 0194959c

0:004> !DumpHeap -mt 7ae77fa8

Address MT Size

0193296c 7ae77fa8 40

01936378 7ae77fa8 40

total 2 objects

Statistics:

MT Count TotalSize Class Name

7ae77fa8 2 80 System.Drawing.Icon

Total 2 objects

0:004> !do 0193296c

Name: System.Drawing.Icon

...

0:004> !DumpMT -md 7ae77fa8

...

Name: System.Drawing.Icon

...

--------------------------------------

MethodDesc Table

Entry MethodDesc JIT Name

...

7ae2070c 7aea3c88 PreJIT System.Drawing.Icon.Finalize()

...

7ae201dc 7aea3c58 PreJIT System.Drawing.Icon.Dispose()

...

Look to the way Reflector.exe shows the Dispose and Finalize (destructor) methods of the “Icon” class we saw above, and how Dispose tells the GC to forget about the Finalize method:

public void Dispose()

{

this.Dispose(true);

GC.SuppressFinalize(this);

}

~Icon()

{

this.Dispose(false);

}

We can get extra information on the following objects registered for finalization: SyncBlocks and RuntimeCallableWrappers (RCWs). Both of these data structures are cached and cleaned up by the Finalizer thread.

0:000> !FinalizeQueue -detail

To be cleaned Com Data

0x4edc1b8 ComPlusWrapper

0x4edc590 ComPlusWrapper

...

0x4e227b8 ComPlusWrapper

SyncBlock to be cleaned up: 6371

MTA interfaces to be released: 0

STA interfaces to be released: 0

...

Every time a COM interface pointer enters the CLR, it is wrapped in a RCW. The RCW has a reference count that is incremented every time a COM interface pointer is mapped to it. The System.Runtime.InteropServices.Marshal.ReleaseComObject method decrements the reference count. If the same COM interface is passed more than once from unmanaged to managed code, the reference count on the wrapper is incremented every time and calling ReleaseComObject returns the number of remaining references. When the reference count reaches zero, the runtime releases all its references on the unmanaged COM object. We use this method to free the COM object and the resources it holds.

Next post: MANAGED DEBUGGING with WINDBG. Managed Heap. Part 3.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 1

alejacma — Thu, 13 Aug 2009 10:33:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 2.

MANAGED HEAP. Part 1

.NET won’t use the NT Heap. It saves some virtual memory for its own managed heap instead.

Objects in the managed heap are part of a Garbage Collector (GC) generation. We have Gen 0, 1 & 2.

All objects start in Gen 0 unless they are over 85,000 bytes in which case they end up in the Large Object Heap (LOH, also known as Gen 3). 85,000 bytes refers to the actual size of the object, not including child objects (member variables). If a member of an object is a reference to another object, the size of that reference is only 4 bytes (in 32 bit machines). Most commonly the items that we will see in the Large Object Heap are strings and large arrays.

When an object is allocated, the GC checks if we have reached one of the generational limits. If so, a GC is triggered. The generational limits are dynamically changed throughout the life of the process in order to better suit the allocation pattern of the application.

If an object is still referenced (“rooted”) during a garbage collection, it is promoted/moved into the next higher generation. The exception is objects in Gen 2 and LOH. Garbage collections on Gen 2 are less frequent than on Gen 1, and on Gen 1 than on Gen 0.

We use LOH for large objects for two main reasons: 1) it is assumed that if you allocate a large object, you are likely to keep it around for a while, 2) large objects are expensive to move. For these reasons LOH is collected less frequently, together with Gen2. Additionally, LOH is not compacted. Use LOH wisely!

Note that if an object is not referenced but has a Finalize (destructor) method, it will automatically get promoted to the next gen and clean up in that gen, unless we call its Dispose method. Dispose will call Finalize and instruct the GC to forget about it (by calling GC.SuppressFinalizer) and clean up the object in its current gen. Calling Dispose guarantees we clean up an object ASAP.

· We can take a look to the .NET heap:

We can see how much memory is taken up by the managed heap and its parts: generations, large object heap, etc.:

0:004> !EEVersion

2.0.50727.1433 retail

Workstation mode

SOS Version: 2.0.50727.42 retail build

0:004> !EEHeap -gc

Number of GC Heaps: 1

generation 0 starts at 0x01901018

generation 1 starts at 0x0190100c

generation 2 starts at 0x01901000

ephemeral segment allocation context: none

segment begin allocated size

00423878 7b463c40 7b47a744 0x00016b04(92932)

0041d178 7a733370 7a754b98 0x00021828(137256)

003d6600 790d8620 790f7d8c 0x0001f76c(128876)

01900000 01901000 01949ff4 0x00048ff4(298996)

Large object heap starts at 0x02901000

segment begin allocated size

02900000 02901000 02907df0 0x00006df0(28144)

Total Size 0xa787c(686204)

------------------------------

GC Heap Size 0xa787c(686204)

Note that workstation applications will only have 1 heap (see above), but server applications (ASP.NET) will have 1 heap per processor (HyperThreading aware).

0:028> !EEVersion

2.0.50727.1433 retail

Server mode with 2 gc heaps

SOS Version: 2.0.50727.42 retail build

0:028> !EEHeap -gc

Number of GC Heaps: 2

------------------------------

Heap 0 (02ec2c48)

generation 0 starts at 0x04722600

...

Heap Size 0xc8048(819272)

------------------------------

Heap 1 (02ec38e0)

generation 0 starts at 0x086f4264

...

Heap Size 0x4a248(303688)

------------------------------

GC Heap Size 0x112290(1122960)

In workstation apps, GC will take place in the thread that tries to allocate memory and can’t. In server apps there will be one dedicated GC thread per heap/processor.

· We can check if Preemptive GC mode is enabled on a thread:

0:004> !Threads

ThreadCount: 2

UnstartedThread: 0

BackgroundThread: 1

PendingThread: 0

DeadThread: 0

Hosted Runtime: no

PreEmptive GC Alloc Lock

ID OSID ThreadOBJ State GC Context Domain Count APT Exception

0 1 1d14 005a6e80 6020 Enabled 0299c3e8:0299dfe8 00571520 0 STA

2 2 1b34 005a8e00 b220 Enabled 00000000:00000000 00571520 0 MTA (Finalizer)

Preemptive GC indicates what GC mode the thread is in: "enabled" in the table means the thread is in preemptive mode where GC could preempt this thread at any time; "disabled" means the thread is in cooperative mode where GC has to wait the thread to give up its current work (the work is related to GC objects so it can't allow GC to move the objects around). When the thread is executing managed code (the current IP is in managed code), it is always in cooperative mode; when the thread is in Execution Engine (unmanaged code), EE code could choose to stay in either mode and could switch mode at any time; when a thread is outside of CLR (i.e. calling into native code using interop), it is always in preemptive mode.

Summing up, if a thread has PreEmptive GC disabled is because it does not want to be interrupted by the GC.

Next post: MANAGED DEBUGGING with WINDBG. Managed Heap. Part 2.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 2

alejacma — Tue, 11 Aug 2009 13:06:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 1.

THREAD STACKS. Part 2

· We can inspect the objects with unmanaged commands:

The reference to an object points to the method table (which is a pointer to the table with the method descriptors) followed by the data of the object. There are 4 other bytes before the pointer to the method table which are the syncblk.

The graph below will show the following info, fields and methods in a different way which will help us understand how an object looks in memory:

0:000> !do 0x01aef338

Name: WindowsApplication1.Form1

MethodTable: 001c6ecc

EEClass: 002d0a30

Size: 352(0x160) bytes

(C:\__WORKSHOP\Demos\BuggyNETApp\bin\Debug\WindowsApplication1.exe)

Fields:

MT Field Offset Type VT Attr Value Name

...

7ae75b24 40011e0 80 ...Drawing.Rectangle 1 instance 01aef3b8 displayRect

...

7b47f3dc 4000011 154 ...dows.Forms.Button 0 instance 01af0398 _Button6

...

0:000> !DumpVC 7ae75b24 01aef3b8

Name: System.Drawing.Rectangle

MethodTable 7ae75b24

EEClass: 7ae75aac

Size: 24(0x18) bytes

(C:\Windows\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll)

Fields:

MT Field Offset Type VT Attr Value Name

79102290 4000462 0 System.Int32 0 instance 0 x

79102290 4000463 4 System.Int32 0 instance 0 y

79102290 4000464 8 System.Int32 0 instance 384 width

79102290 4000465 c System.Int32 0 instance 352 height

...

0:000> !DumpMT -md 0x001c6ecc

EEClass: 002d0a30

Module: 001c2c3c

Name: WindowsApplication1.Form1

mdToken: 02000008 (C:\__WORKSHOP\Demos\BuggyNETApp\bin\Debug\WindowsApplication1.exe)

BaseSize: 0x160

ComponentSize: 0x0

Number of IFaces in IFaceMap: 15

Slots in VTable: 399

--------------------------------------

MethodDesc Table

Entry MethodDesc JIT Name

7b05e348 7b4a4668 PreJIT System.Windows.Forms.Form.ToString()

...

002c1828 001c6d80 JIT WindowsApplication1.Form1.get_Button1()

...

001cc221 001c6e08 NONE WindowsApplication1.Form1.Run()

...

Note: Click on the image to enlarge.

Next post: MANAGED DEBUGGING with WINDBG. Managed Heap. Part 1.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 1

alejacma — Tue, 11 Aug 2009 12:47:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Threads. Part 2.

THREAD STACKS. Part 1

· We can take a look to the objects in the stack:

The CLR is stack-based. All references to objects are in the stack, but objects themselves are in the managed heap. We also have to take into account that Structs that we instantiate go to the stack and Classes we instantiate go to the managed heap.

In any thread we can see a list of references to objects that are still on the thread’s stack (remember we have one stack per thread ;-):

0:000> !DumpStackObjects

...

0:000> !dso

OS Thread Id: 0x1334 (0)

ESP/REG Object Name

0018e710 01b3ed34 System.IndexOutOfRangeException

0018e758 01b3ed34 System.IndexOutOfRangeException

0018e79c 01aef338 WindowsApplication1.Form1

0018e7a0 01b3ed1c System.Int32[]

0018e7a4 01b3cf94 System.Windows.Forms.MouseEventArgs

0018e7ac 01aef338 WindowsApplication1.Form1

0018e7b4 01b3ed1c System.Int32[]

0018e7bc 01b3cf94 System.Windows.Forms.MouseEventArgs

0018e838 01b3ed1c System.Int32[]

0018e84c 01aef338 WindowsApplication1.Form1

0018e860 01af0398 System.Windows.Forms.Button

...

Note that the objects at the top of the stack correspond to the objects used by the methods at the top of the call stack:

0:000> kpL

ChildEBP RetAddr

0018e6f0 79f071ac KERNEL32!RaiseException(unsigned long dwExceptionCode = 0xe0434f4d, unsigned long dwExceptionFlags = 1, unsigned long nNumberOfArguments = 1, unsigned long * lpArguments = 0x0018e728)+0x58

0018e750 79f9293a mscorwks!RaiseTheExceptionInternalOnly(class Object * throwable = 0x01b3ed34, int rethrow = 0, int fForStackOverflow = 0)+0x2a8

0018e788 7a129a34 mscorwks!UnwindAndContinueRethrowHelperAfterCatch(class Frame * pEntryFrame = 0x0018e7e0, class Exception * pException = 0x003e65a8)+0x70

0018e828 002c1aad mscorwks!JIT_RngChkFail(void)+0xb0

0018e870 002c19fe WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(<HRESULT 0x80004001>)+0x55

...

0:000> !CLRStack -a

OS Thread Id: 0x1334 (0)

ESP EIP

0018e7e0 761db09e [HelperMethodFrame: 0018e7e0]

0018e830 002c1aad WindowsApplication1.Form1.PlayWithArray(Int32[])

PARAMETERS:

this = 0x01aef338

array = 0x01b3ed1c

LOCALS:

0x0018e848 = 0x00000000

0x0018e844 = 0x00000006

0x0018e840 = 0x00000003

0x0018e834 = 0x00000000

0x0018e83c = 0x00000003

0018e878 002c19fe WindowsApplication1.Form1.Button6_Click(System.Object, System.EventArgs)

PARAMETERS:

this = 0x01aef338

sender = 0x01af0398

e = 0x01b3cf94

LOCALS:

0x0018e87c = 0x00000000

<CLR reg> = 0x01b3ed1c

<CLR reg> = 0x00000000

...

· We can inspect the objects:

With a reference to an object we can inspect the object itself:

0:000> !DumpObj 0x01aef338

...

0:000> !do 0x01aef338

Name: WindowsApplication1.Form1

MethodTable: 001c6ecc

EEClass: 002d0a30

Size: 352(0x160) bytes

(C:\__WORKSHOP\Demos\BuggyNETApp\bin\Debug\WindowsApplication1.exe)

Fields:

MT Field Offset Type VT Attr Value Name

...

7ae75fa8 40010f2 2b8 System.Drawing.Font 0 static 01af1d18 defaultFont

...

7ae75b24 40011e0 80 ...Drawing.Rectangle 1 instance 01aef3b8 displayRect

...

7b47f3dc 4000011 154 ...dows.Forms.Button 0 instance 01af0398 _Button6

7912d7c0 4000013 158 System.Int32[] 0 instance 01b3ed1c myarray

...

0:000> !do 01af1d18

Name: System.Drawing.Font

...

Fields:

MT Field Offset Type VT Attr Value Name

...

790fd8c4 4000641 c System.String 0 instance 01af1544 systemFontName

...

0:000> !do -nofields 01af1544

Name: System.String

MethodTable: 790fd8c4

EEClass: 790fd824

Size: 40(0x28) bytes

(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

String: DefaultFont

0:000> !do 01aef3b8

<Note: this object has an invalid CLASS field>

Invalid object

0:000> !DumpVC 7ae75b24 01aef3b8

Name: System.Drawing.Rectangle

MethodTable 7ae75b24

EEClass: 7ae75aac

Size: 24(0x18) bytes

(C:\Windows\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll)

Fields:

MT Field Offset Type VT Attr Value Name

79102290 4000462 0 System.Int32 0 instance 0 x

79102290 4000463 4 System.Int32 0 instance 0 y

...

0:000> !DumpArray 01b3ed1c

...

0:000> !da 01b3ed1c

Name: System.Int32[]

MethodTable: 7912d7c0

EEClass: 7912d878

Size: 24(0x18) bytes

Array: Rank 1, Number of elements 3, Type Int32

Element Methodtable: 79102290

[0] 00000001

[1] 00000002

[2] 00000003

Note that the column VT contains the value 1 if the field is a valuetype structure, and 0 if the field contains a pointer to another object.

If we only need to see the static fields of the object, we can inspect the EEClass of the class (representation of a class created before any method invocations are made on it) of the object:

0:000> !DumpClass 002d0a30

Class Name: WindowsApplication1.Form1

mdToken: 02000008 (C:\__WORKSHOP\Demos\BuggyNETApp\bin\Debug\WindowsApplication1.exe)

Parent Class: 7b47b140

Module: 001c2c3c

Method Table: 001c6ecc

Vtable Slots: 180

Total Method Slots: 18f

Class Attributes: 100001

NumInstanceFields: 47

NumStaticFields: 2

MT Field Offset Type VT Attr Value Name

...

7ae75fa8 40010f2 2b8 System.Drawing.Font 0 static 01af1d18 defaultFont

...

7ae75b24 40011e0 80 ...Drawing.Rectangle 1 instance displayRect

...

7a7559dc 40011de 308 ...stics.TraceSwitch 0 static 00000000 AutoScrolling

...

7b47f3dc 4000011 154 ...dows.Forms.Button 0 instance _Button6

7912d7c0 4000013 158 System.Int32[] 0 instance myarray

...

But if we are looking for a way to display a static field of a class (and we don't have an instance of the class, so !DumpObj won't help us), we can inspect the EEClass of the class itself:

0:000> !Name2EE WindowsApplication1 WindowsApplication1.Form1

Module: 001c2c3c (WindowsApplication1.exe)

Token: 0x02000008

MethodTable: 001c6ecc

EEClass: 002d0a30

Name: WindowsApplication1.Form1

0:000> !DumpClass 002d0a30

Class Name: WindowsApplication1.Form1

...

Next post: MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 2.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Threads. Part 2

alejacma — Mon, 10 Aug 2009 13:25:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Threads. Part 1.

THREADS. Part 2

· We can see all .NET related threads in our process:

There are different types of .NET threads: finalizer, GC, debugger, timer, threadpool, primary and background threads. But not all threads related to .NET are managed. We can see all .NET threads like this:

0:004> !Threads -special

ThreadCount: 2

UnstartedThread: 0

BackgroundThread: 1

PendingThread: 0

DeadThread: 0

Hosted Runtime: no

PreEmptive GC Alloc Lock

ID OSID ThreadOBJ State GC Context Domain Count APT Exception

0 1 1f3c 003ec600 6020 Enabled 0198c744:0198dfe8 003b4bd8 0 STA

2 2 904 003f5990 b220 Enabled 00000000:00000000 003b4bd8 0 MTA (Finalizer)

OSID Special thread type

1 1b08 DbgHelper

2 904 Finalizer

· We can inspect the .NET Thread Pool of our process:

A thread pool is a collection of worker threads that efficiently execute asynchronous callbacks on behalf of the application. The thread pool is primarily used to reduce the number of application threads and provide management of the worker threads. Applications can queue work items, associate work with waitable handles, automatically queue based on a timer, and bind with I/O.

.NET implements the entire pooling and load-balancing mechanism using the native ThreadPoolMgr class. The ThreadPoolMgr class has five main thread types, and we can only use the first two to run our managed code:

1. Completion port thread : executes the callback function after an item is queued by QueueUserWorkItem(). If there are not enough completion port threads available, the work item is sent to the work request queue.

2. Worker thread: services the work request queue.

3. Gate thread: Monitors the health of the thread pool.

4. Wait thread: to wait on a synchronization object (i.e. System.Threading.WaitHandle).

5. Timer thread: timer callback functionality of the System.Threading.Timer class.

We can see basic information about the Thread Pool, including the number of work requests in the queue, timers and completion port threads. This info has been taken from an ASP.NET (w3wp.exe) process:

0:030> !ThreadPool

CPU utilization 4%

Worker Thread: Total: 5 Running: 5 Idle: 0 MaxLimit: 200 MinLimit: 2

Work Request in Queue: 0

--------------------------------------

Number of Timers: 8

--------------------------------------

Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 1 MaxLimit: 200 MinLimit: 2

Note that CPU utilization is the complete CPU utilization on the box, so not specifically for the ASP.NET process. In this sample, there are no work requests in the queue, and only a few timers waiting. We have a maximum of 200 worker threads to execute our requests, and a minimum of 2 alive at any time (this comes from our machine.config settings). And currently we are executing on 5 of these. There are no work requests in the queue, so the thread pool is not exhausted. All the completion port threads are free.

· The following list may help us identify some typical threads we may find in our applications as shown by calls to these specific functions:

- mscorwks!DebuggerRCThread: Debugger thread. All managed processes have a debugger thread, even if there is no debugger attached. There is only one debugger thread in a given process. This thread exists so that a managed debugger can attach and control the managed code.

- mscorwks!GCHeap::FinalizerThreadStart: Finalizer thread. Just like the debugger thread, there is always one finalizer thread per managed process.

- RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls: Lightweight Remote Procedure Call (RPC) thread. It’s used for local remote procedure call (LRPC) communication. Here it handles incoming RPC calls to this process.

- aspnet_wp!wmain: Main ASP.NET thread. It has initialized the runtime, connected the named pipes back to InetInfo.exe, and created the ping thread. Once it has completed the initialization, it loops, waiting for the process to exit.

- aspnet_wp!DoPingThread: ASP.NET ping thread. It responds to the pings it receives from the health monitoring thread in InetInfo.exe.

- comsvcs!CSTAThreadPool::KillThreadControlLoop: COM+ thread pool.

- comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop: COM+ thread pool.

- ole32!CRpcThread::WorkerLoop: COM worker threads. They may be waiting for COM to dispatch work.

- ole32!DLLHostThreadEntry: COM has created this to instantiate a COM component marked as Apartment.

- ole32!GetToSTA: You need to switch to an STA thread in the process for components marked as Aparment. This thread switch won't happen with ASPCompat page directive or if Apartment component is in COM+ (The component must be registered in COM+ as a library application, and Synchronization must be set to Required so that it runs on a COM+ STA worker thread), which forces the page's ProcessRequest method to execute on a COM+ STA worker thread. If component is marked as Free, we can run in MTA thread.

- comsvcs!CSTAThread::WorkerLoop: COM+ thread pool. STA threads used with ASPCompat or Aparment component in COM+. ASPCompat forces the ProcessRequest method to execute in a thread that has been initialized to the STA threading model.

Next post: MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 1.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Threads. Part 1

alejacma — Mon, 10 Aug 2009 13:10:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Call Stacks. Part 3.

THREADS. Part 1

· We can see all threads in our process:

If we reach a breakpoint or break on an exception, WinDbg command prompt shows the ID of the thread which reached the breakpoint or raised the exception. We can directly see the call stack, or inspect the exception we got, etc., for that thread.

If we break manually (i.e. Ctrl+Break), the command prompt shows the ID of the debugger thread (injected by the debugger in our process to interact with it). We can’t see or do anything useful there:

0:004> kL

ChildEBP RetAddr

04f1fcfc 77c0f0a9 ntdll!DbgBreakPoint

04f1fd2c 77a43833 ntdll!DbgUiRemoteBreakin+0x3c

04f1fd38 77bba9bd KERNEL32!BaseThreadInitThunk+0xe

04f1fd78 00000000 ntdll!_RtlUserThreadStart+0x23

So we need to change to another more useful thread. We can take a look to all threads as in any unmanaged application:

0:004> ~

0 Id: 1910.1f3c Suspend: 1 Teb: 7ffdf000 Unfrozen

1 Id: 1910.1b08 Suspend: 1 Teb: 7ffde000 Unfrozen

2 Id: 1910.904 Suspend: 1 Teb: 7ffdd000 Unfrozen

3 Id: 1910.1bbc Suspend: 1 Teb: 7ffdc000 Unfrozen

. 4 Id: 1910.1b60 Suspend: 1 Teb: 7ffdb000 Unfrozen

We can see for how long they’ve been running:

0:004> !runaway

User Mode Time

Thread Time

0:1f3c 0 days 0:00:00.078

4:1b60 0 days 0:00:00.000

3:1bbc 0 days 0:00:00.000

2:904 0 days 0:00:00.000

1:1b08 0 days 0:00:00.000

We can move to any other thread and see i.e. its call stack. Let’s take a look to thread 0 because it’s the one which apparently has been doing more stuff, and then go back to the thread where we broke:

0:004> ~0s

eax=00000000 ebx=00000000 ecx=003b008c edx=77be0f34 esi=00dd8c30 edi=00000000

eip=77be0f34 esp=0027e3a8 ebp=0027e3dc iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246

ntdll!KiFastSystemCallRet:

77be0f34 c3 ret

0:000> kL

ChildEBP RetAddr

0027e3a4 77afb5bc ntdll!KiFastSystemCallRet

0027e3a8 77af1598 USER32!NtUserWaitMessage+0xc

0027e3dc 77af1460 USER32!DialogBox2+0x202

…

0:000> ~

. 0 Id: 1910.1f3c Suspend: 1 Teb: 7ffdf000 Unfrozen

1 Id: 1910.1b08 Suspend: 1 Teb: 7ffde000 Unfrozen

2 Id: 1910.904 Suspend: 1 Teb: 7ffdd000 Unfrozen

3 Id: 1910.1bbc Suspend: 1 Teb: 7ffdc000 Unfrozen

# 4 Id: 1910.1b60 Suspend: 1 Teb: 7ffdb000 Unfrozen

0:000> ~#

# 4 Id: 1910.1b60 Suspend: 1 Teb: 7ffdb000 Unfrozen

Start: ntdll!RtlUserThreadStart (77be0f18)

Priority: 0 Priority class: 32 Affinity: 3

0:000> ~#s

…

0:004>

Thread 0 is the main GUI thread of a WinForms application, for instance.

Note that ‘~’ command will use a ‘.’ to indicate the active thread, and a ‘#’ to indicate the thread where the debugger broke in the first place.

Additionally, we may want to see all call stacks (kL100) for all threads (~*) to find the thread where we want to move:

0:000> ~*kL100

. 0 Id: 1910.1f3c Suspend: 1 Teb: 7ffdf000 Unfrozen

ChildEBP RetAddr

0027e3a4 77afb5bc ntdll!KiFastSystemCallRet

0027e3a8 77af1598 USER32!NtUserWaitMessage+0xc

0027e3dc 77af1460 USER32!DialogBox2+0x202

...

0027f894 00000000 ntdll!_RtlUserThreadStart+0x23

1 Id: 1910.1b08 Suspend: 1 Teb: 7ffde000 Unfrozen

ChildEBP RetAddr

00bef7b8 77be0690 ntdll!KiFastSystemCallRet

...

00bef980 00000000 ntdll!_RtlUserThreadStart+0x23

...

# 4 Id: 1910.1b60 Suspend: 1 Teb: 7ffdb000 Unfrozen

ChildEBP RetAddr

04f1fcfc 77c0f0a9 ntdll!DbgBreakPoint

...

04f1fd78 00000000 ntdll!_RtlUserThreadStart+0x23

Note that we will always see more threads than the ones we created. In a typical WinForms application we will see our main GUI thread, a Finalizer thread (Garbage Collector related), a GDI+ thread, a couple of debugger related threads, and any other thread we’ve created.

· We can see all managed threads in our process:

Some threads in our process will contain some managed calls (managed threads), but not all (unmanaged threads). We can only focus on managed threads:

0:004> !Threads

ThreadCount: 2

UnstartedThread: 0

BackgroundThread: 1

PendingThread: 0

DeadThread: 0

Hosted Runtime: no

PreEmptive GC Alloc Lock

ID OSID ThreadOBJ State GC Context Domain Count APT Exception

0 1 1f3c 003ec600 6020 Enabled 0198c744:0198dfe8 003b4bd8 0 STA

2 2 904 003f5990 b220 Enabled 00000000:00000000 003b4bd8 0 MTA (Finalizer)

0:004> ~0kL

ChildEBP RetAddr

0027e3a4 77afb5bc ntdll!KiFastSystemCallRet

0027e3a8 77af1598 USER32!NtUserWaitMessage+0xc

0027e3dc 77af1460 USER32!DialogBox2+0x202

…

00000001 00371a20 System_Windows_Forms_ni!System.Windows.Forms.MessageBox.Show(System.String)+0x26

00000001 7b062c9a WindowsApplication1!WindowsApplication1.Form1.Button6_Click(System.Object, System.EventArgs)+0xa8

...

0:004> ~2kL

ChildEBP RetAddr

03b4f5b0 77be0690 ntdll!KiFastSystemCallRet

03b4f5b4 77a47e09 ntdll!ZwWaitForMultipleObjects+0xc

03b4f650 77a48150 KERNEL32!WaitForMultipleObjectsEx+0x11d

03b4f66c 79ef224b KERNEL32!WaitForMultipleObjects+0x18

03b4f68c 79fb997b mscorwks!WKS::WaitForFinalizerEvent+0x77

03b4f6a0 79ef3207 mscorwks!WKS::GCHeap::FinalizerThreadWorker+0x79

03b4f6b4 79ef31a3 mscorwks!ManagedThreadBase_DispatchInner+0x4f

03b4f748 79ef30c3 mscorwks!ManagedThreadBase_DispatchMiddle+0xb1

03b4f784 79fb9643 mscorwks!ManagedThreadBase_DispatchOuter+0x6d

03b4f7ac 79fb960d mscorwks!ManagedThreadBase_NoADTransition+0x32

03b4f7bc 79fba09b mscorwks!ManagedThreadBase::FinalizerBase+0xd

03b4f7f4 79f95a2e mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb

03b4f88c 77a43833 mscorwks!Thread::intermediateThreadProc+0x49

03b4f898 77bba9bd KERNEL32!BaseThreadInitThunk+0xe

03b4f8d8 00000000 ntdll!_RtlUserThreadStart+0x23

We may want to see only managed calls (!CLRStack) in all threads (~*):

0:004> ~*e !CLRStack

OS Thread Id: 0x1f3c (0)

ESP EIP

0027e6c8 77be0f34 [NDirectMethodFrameStandalone: 0027e6c8] System.Windows.Forms.SafeNativeMethods.MessageBox(System.Runtime.InteropServices.HandleRef, System.String, System.String, Int32)

...

0027e780 7b28595e System.Windows.Forms.MessageBox.Show(System.String)

0027e788 00371a20 WindowsApplication1.Form1.Button6_Click(System.Object, System.EventArgs)

...

0027ecdc 00370117 WindowsApplication1.My.MyApplication.Main(System.String[])

0027ef24 79e7c74b [GCFrame: 0027ef24]

OS Thread Id: 0x1b08 (1)

Unable to walk the managed stack. The current thread is likely not a

managed thread. You can run !threads to get a list of managed threads in

the process

OS Thread Id: 0x904 (2)

Failed to start stack walk: 80004005

OS Thread Id: 0x1bbc (3)

Unable to walk the managed stack. The current thread is likely not a

managed thread. You can run !threads to get a list of managed threads in

the process

...

Note that we have to use ‘e’ with ‘~*’ to be able to execute a command from a debugger extension in all threads.

Note that thread 2 (Finalizer) is supposed to be a managed thread, but it doesn’t currently contain any managed calls in the call stack.

Next post: MANAGED DEBUGGING with WINDBG. Threads. Part 2.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Call Stacks. Part 3

alejacma — Wed, 05 Aug 2009 09:47:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Call Stacks. Part 2.

CALL STACKS. Part 3

· Let's review previous commands:

Let’s see a sample where the debugger broke when “PlayWithArray” raised an exception. We can use the return address of the function it called, “mscorwks!JIT_RngChkFail”, to see where in “PlayWithArray” we were when the exception happened:

0:000> kL

ChildEBP RetAddr

0027e600 79f071ac KERNEL32!RaiseException+0x58

0027e660 79f9293a mscorwks!RaiseTheExceptionInternalOnly+0x2a8

0027e698 7a129a34 mscorwks!UnwindAndContinueRethrowHelperAfterCatch+0x70

0027e738 00371aad mscorwks!JIT_RngChkFail+0xb0

0027e780 003719fe WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55

...

0:000> !u 00371aad

Normal JIT generated code

WindowsApplication1.Form1.PlayWithArray(Int32[])

Begin 00371a58, size c0

00371a58 55 push ebp

...

00371aa0 8b4dc8 mov ecx,dword ptr [ebp-38h]

00371aa3 3b5104 cmp edx,dword ptr [ecx+4]

00371aa6 7205 jb WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55 (00371aad)

00371aa8 e8d67edb79 call mscorwks!JIT_RngChkFail (7a129983)

>>> 00371aad 03449108 add eax,dword ptr [ecx+edx*4+8]

00371ab1 7105 jno WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x60 (00371ab8)

00371ab3 e8837fdb79 call mscorwks!JIT_Overflow (7a129a3b)

...

00371b16 5d pop ebp

00371b17 c3 ret

0:000> u 00371aa8

WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x50 [C:\__WORKSHOP\Demos\BuggyNETApp\Form1.vb @ 135]:

...

Note how we used the unmanaged version of this command, u, to find out the source code line.

That line corresponds to the following VB.NET statement in “PlayWithArray”:

result += array(i)

It seems we got some exception when accessing the array... Can we tell why? Let’s give it a try:

0:000> !CLRStack -a

OS Thread Id: 0x1f3c (0)

ESP EIP

0027e6f0 77a1b09e [HelperMethodFrame: 0027e6f0]

0027e740 00371aad WindowsApplication1.Form1.PlayWithArray(Int32[])

PARAMETERS:

this = 0x0192f338

array = 0x01983d38

LOCALS:

0x0027e758 = 0x00000000

0x0027e754 = 0x00000006

0x0027e750 = 0x00000003

0x0027e744 = 0x00000000

0x0027e74c = 0x00000003

...

0:000> !da 0x01983d38

Name: System.Int32[]

MethodTable: 7912d7c0

EEClass: 7912d878

Size: 24(0x18) bytes

Array: Rank 1, Number of elements 3, Type Int32

Element Methodtable: 79102290

[0] 00000001

[1] 00000002

[2] 00000003

It seems we tried to access the element at index 3 of the array (the 4^th element), but the array only has 3 elements.

How do I know which local corresponds to “i”? ILDASM may help here, because the locals it shows are in the same order as in !CLRStack. In our case, “i” is in the 3^rd position:

// Code size 49 (0x31)

.maxstack 3

.locals init ([0] int32 PlayWithArray,

[1] int32 result,

[2] int32 i,

[3] class [mscorlib]System.Exception ex,

[4] int32 VB$CG$t_i4$S0)

· We can use registers to check parameters of a method in the call stack:

When we are at the beginning of a function, we must take into consideration that JIT compiler uses _fastcall calling convention. This means that the first two parameters will be passed in the ecx and edx registers. Ecx is used to hold the “this” pointer. We can use those registers to inspect the parameters:

0:000> g

Breakpoint 0 hit

eax=001a6e28 ebx=01ad8880 ecx=01a9f338 edx=01ada608 esi=01ada608 edi=01a9f338

eip=008f1a58 esp=002fe8a4 ebp=00000000 iopl=0 nv up ei ng nz na pe cy

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000287

WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[]):

008f1a58 55 push ebp

0:000> !CLRStack -p

OS Thread Id: 0x17fc (0)

ESP EIP

002fe8a4 008f1a58 WindowsApplication1.Form1.PlayWithArray(Int32[])

PARAMETERS:

this = 0x01a9f338

array = 0x01ada608

...

002ff044 79e7c74b [GCFrame: 002ff044]

0:000> !do @ecx

Name: WindowsApplication1.Form1

...

0:000> !da @edx

Name: System.Int32[]

...

Array: Rank 1, Number of elements 3, Type Int32

...

Note that later in the function, esi is typically used to hold the “this” pointer (gotten from ecx).

Next post: MANAGED DEBUGGING with WINDBG. Threads. Part 1.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Call Stacks. Part 2

alejacma — Wed, 05 Aug 2009 09:32:00 GMT

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Call Stacks. Part 1.

CALL STACKS. Part 2

· We can see the source code of a method in the call stack:

First of all, Source Code mode has to be enabled. We’ll need symbols and source code files correctly configured.

We can choose the frame of the method we want to see:

0:000> knL100

# ChildEBP RetAddr

00 0027e600 79f071ac KERNEL32!RaiseException+0x58

01 0027e660 79f9293a mscorwks!RaiseTheExceptionInternalOnly+0x2a8

02 0027e698 7a129a34 mscorwks!UnwindAndContinueRethrowHelperAfterCatch+0x70

03 0027e738 00371aad mscorwks!JIT_RngChkFail+0xb0

04 0027e780 003719fe WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55

05 00000000 7b062c9a WindowsApplication1!WindowsApplication1.Form1.Button6_Click(System.Object, System.EventArgs)+0x86

06 0027e7e8 7b11cb29 System_Windows_Forms_ni!System.Windows.Forms.Control.OnClick(System.EventArgs)+0x6a

...

0:000> .frame 4

04 0027e780 003719fe WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0x55 [C:\__WORKSHOP\Demos\BuggyNETApp\Form1.vb @ 135]

WinDbg will automatically open the source code file and locate where in the function we are in this call stack:

Private Function PlayWithArray(ByVal array As Integer()) As Integer

Dim result As Integer

Try

For i As Integer = 0 To 3

result += array(i)

Catch ex As Exception

result = -1

End Try

Return result

End Function

· We can see the MSIL (Microsoft Intermediate Language) of a method in the call stack:

When compiling to managed code, the compiler translates our source code into Microsoft intermediate language (MSIL), which is a CPU-independent set of instructions that can be efficiently converted to native code. MSIL includes instructions for loading, storing, initializing, and calling methods on objects, as well as instructions for arithmetic and logical operations, control flow, direct memory access, exception handling, and other operations. Before code can be run, MSIL must be converted to CPU-specific code by a just-in-time (JIT) compiler.

To see the IL, we first need to get the method descriptor like this:

0:000> !CLRStack

OS Thread Id: 0x1f3c (0)

ESP EIP

0027e6f0 77a1b09e [HelperMethodFrame: 0027e6f0]

0027e740 00371aad WindowsApplication1.Form1.PlayWithArray(Int32[])

0027e788 003719fe WindowsApplication1.Form1.Button6_Click(System.Object, System.EventArgs)

0027e7a8 7b062c9a System.Windows.Forms.Control.OnClick(System.EventArgs)

...

0:000> !IP2MD 00371aad

MethodDesc: 00116e28

Method Name: WindowsApplication1.Form1.PlayWithArray(Int32[])

Class: 00380a30

MethodTable: 00116ecc

mdToken: 06000039

Module: 00112c3c

IsJitted: yes

m_CodeOrIL: 00371a58

Or directly with:

0:000> !DumpStack -EE

OS Thread Id: 0x1f3c (0)

Current frame:

ChildEBP RetAddr Caller,Callee

0027e738 00371aad (MethodDesc 0x116e28 +0x55 WindowsApplication1.Form1.PlayWithArray(Int32[]))

...

Additionally, if we just hit a breakpoint, we can also use the eip (Instruction Pointer) register for this purpose:

0:000> !IP2MD @eip

MethodDesc: 00116e28

Method Name: WindowsApplication1.Form1.PlayWithArray(Int32[])

...

Now we can take a look to IL code for that method:

0:000> !DumpIL 00116e28

ilAddr = 00d73298

IL_0000: nop

IL_0001: nop

.try

{

IL_0002: ldc.i4.0

IL_0003: stloc.2

IL_0004: ldloc.1

IL_0005: ldarg.1

IL_0006: ldloc.2

IL_0007: ldelem.i4

IL_0008: add.ovf

IL_0009: stloc.1

IL_000a: nop

IL_000b: ldloc.2

IL_000c: ldc.i4.1

IL_000d: add.ovf

IL_000e: stloc.2

IL_000f: ldloc.2

IL_0010: ldc.i4.3

IL_0011: stloc.s VAR OR ARG 4

IL_0013: ldloc.s VAR OR ARG 4

IL_0015: ble.s IL_0104

IL_0017: leave.s IL_002a

} // end .try

.catch

{

IL_0019: dup

IL_001a: call Microsoft.VisualBasic.CompilerServices.ProjectDat::SetProjectError

IL_001f: stloc.3

IL_0020: nop

IL_0021: ldc.i4.m1

IL_0022: stloc.1

IL_0023: call Microsoft.VisualBasic.CompilerServices.ProjectDat::ClearProjectError

IL_0028: leave.s IL_002a

} // end .catch

IL_002a: nop

IL_002b: ldloc.1

IL_002c: stloc.0

IL_002d: br.s IL_002f

IL_002f: ldloc.0

IL_0030: ret

Tip for Reading IL: The CLR is stack-based. A two-operand operation such as Add pops the two top values from the stack and adds them.

ILDASM.exe (Visual Studio.NET) will show the same IL code that we’ve just seen, plus the following declarations just before it:

// Code size 49 (0x31)

.maxstack 3

.locals init ([0] int32 PlayWithArray,

[1] int32 result,

[2] int32 i,

[3] class [mscorlib]System.Exception ex,

[4] int32 VB$CG$t_i4$S0)

And Reflector.exe will show our function like this:

Private Function PlayWithArray(ByVal array As Integer()) As Integer

Dim result As Integer

Try

Dim VB$CG$t_i4$S0 As Integer

Dim i As Integer = 0

result = (result + array(i))

i += 1

VB$CG$t_i4$S0 = 3

Loop While (i <= VB$CG$t_i4$S0)
Catch exception1 As Exception

ProjectData.SetProjectError(exception1)

Dim ex As Exception = exception1

result = -1

ProjectData.ClearProjectError

End Try

Return result

End Function

All this may help us to understand the correspondence between our code and its IL version.

· We can write an image loaded in memory to a file:

You can save a module after patching it on the fly when live debugging, or get the original DLLs or EXEs when debugging a dump. If we save a managed binary, we can use the file with ILDASM.exe or Reflector.exe, for instance. We save a module like this:

0:004> lm

start end module name

00a20000 00a2c000 WindowsApplication1 (deferred)

…

79e70000 7a3ff000 mscorwks (private pdb symbols) c:\symbols\mscorwks.pdb\62286AFFFC674D5198883B98518936FF2\mscorwks.pdb

...

7afd0000 7bc6c000 System_Windows_Forms_ni (deferred)

0:004> !SaveModule 79e70000 c:\mscorwks.dll

5 sections in file

section 0 - VA=1000, VASize=53abd2, FileAddr=400, FileSize=53ac00

section 1 - VA=53c000, VASize=9, FileAddr=53b000, FileSize=200

section 2 - VA=53d000, VASize=189d0, FileAddr=53b200, FileSize=16000

section 3 - VA=556000, VASize=670, FileAddr=551200, FileSize=800

section 4 - VA=557000, VASize=37bc4, FileAddr=551a00, FileSize=37c00

· We can see the assembly code of a method in the call stack:

If method is jitted, we can see its assembly code as we will do it with any other unmanaged method:

0:000> !DumpStack -EE

OS Thread Id: 0x1f3c (0)

Current frame:

ChildEBP RetAddr Caller,Callee

0027e738 00371aad (MethodDesc 0x116e28 +0x55 WindowsApplication1.Form1.PlayWithArray(Int32[]))

...

0:000> !DumpMD 0x116e28

Method Name: WindowsApplication1.Form1.PlayWithArray(Int32[])

Class: 00380a30

MethodTable: 00116ecc

mdToken: 06000039

Module: 00112c3c

IsJitted: yes

m_CodeOrIL: 00371a58

0:000> uf 00371a58

WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[]) [C:\__WORKSHOP\Demos\BuggyNETApp\Form1.vb @ 129]:

129 00371a58 55 push ebp

129 00371a59 8bec mov ebp,esp

129 00371a5b 57 push edi

129 00371a5c 56 push esi

129 00371a5d 53 push ebx

129 00371a5e 83ec34 sub esp,34h

129 00371a61 33c0 xor eax,eax

129 00371a63 8945e8 mov dword ptr [ebp-18h],eax

...

WindowsApplication1!WindowsApplication1.Form1.PlayWithArray(Int32[])+0xb5 [C:\__WORKSHOP\Demos\BuggyNETApp\Form1.vb @ 144]:

144 00371b0d 8b45d8 mov eax,dword ptr [ebp-28h]

144 00371b10 8d65f4 lea esp,[ebp-0Ch]

144 00371b13 5b pop ebx

144 00371b14 5e pop esi

144 00371b15 5f pop edi

144 00371b16 5d pop ebp

144 00371b17 c3 ret

Note that if we have symbols we can tell the line number in the source code file.

We have a .NET version of uf command which understands managed code:

0:000> !u 00371a58

Normal JIT generated code

WindowsApplication1.Form1.PlayWithArray(Int32[])

Begin 00371a58, size c0

>>> 00371a58 55 push ebp

00371a59 8bec mov ebp,esp

00371a5b 57 push edi

00371a5c 56 push esi

00371a5d 53 push ebx

00371a5e 83ec34 sub esp,34h

00371a61 33c0 xor eax,eax

00371a63 8945e8 mov dword ptr [ebp-18h],eax

...

00371b16 5d pop ebp

00371b17 c3 ret

You can pass to this command any address within that method and it will show the same assembly and where we are in it (>>>).

Note that we may run this command with an address within an unmanaged method and if we have private symbols and source code, it shows all the assembly in that method along with the source code itself!

Next post: MANAGED DEBUGGING with WINDBG. Call Stacks. Part 3.

Index: MANAGED DEBUGGING with WINDBG. Introduction and Index.

Regards,

Alex (Alejandro Campos Magencio)