.NET Debugging Demos Lab 2: Crash - Review

re: .NET Debugging Demos Lab 2: Crash - Review

Lucian Bargaoanu — Tue, 12 Feb 2008 11:01:42 GMT

I just wanted to say that the labs are great. And because of the hints is kind of easy to go through them and this is good because I'm pretty ignorant about this type of debugging. I'm sure they'll get more difficult later on.

re: .NET Debugging Demos Lab 2: Crash - Review

Essam — Tue, 12 Feb 2008 13:51:35 GMT

Tess,

What about a screen cast for the lab ?

Link Listing - February 11, 2008

Christopher Steen — Tue, 12 Feb 2008 16:23:43 GMT

Sharepoint Announcing the DoD 5015.2 Resource Kit for SharePoint Server 2007 [Via: sptblog ] WPF The...

Link Listing - February 11, 2008

Christopher Steen — Tue, 12 Feb 2008 16:23:43 GMT

Link Listing - February 11, 2008

re: .NET Debugging Demos Lab 2: Crash - Review

Tom — Thu, 14 Feb 2008 11:16:32 GMT

I aggree with Lucian Bargaoanu: "The labs are great." I am also pleased wit the hint because of my ignorance with this kind of debugging.

The lack of questions about the lab you notice, may be caused by the quality of the labs. So just keep on going as you are; I lik it.

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Thu, 14 Feb 2008 12:19:54 GMT

Great to hear:) I'll post a memory lab today or tomorrow depending on if when i can get an hour or two freed up.

As a teaser, the mem leak will be produced by running tinyget on 30 threads, 100 loops for links.aspx

re: .NET Debugging Demos Lab 2: Crash - Review

baal — Thu, 14 Feb 2008 12:50:00 GMT

“examine the disassembly ” parts is difficult for me

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Thu, 14 Feb 2008 13:01:38 GMT

agreed, I think that is a bit difficult overall... and not needed in most cases, which is why i just added it as an extra excercise.

do you have any specific questions around it?

re: .NET Debugging Demos Lab 2: Crash - Review

baal — Thu, 14 Feb 2008 13:11:39 GMT

Dear, questions will come, don't worry!

today i spent 8 hours to make 2 reviews articles to chinese, I have not run the code, just translate to catch up with you. i will investigate them, then ask you something.

-------

tired,over here

re: .NET Debugging Demos Lab 2: Crash - Review

Kevin — Fri, 15 Feb 2008 21:44:27 GMT

I'm curious, is there something you did to cause the disassembly to show that the memory contained an empty string? I do not see the same results when disassembling. My output does not show the ("") with the mov instruction.

01bc0f20 8bc8 mov ecx,eax

01bc0f22 8b158c46550b mov edx,dword ptr ds:[0B55468Ch]

01bc0f28 e8a3db7a77 call mscorlib_ni+0x2aead0 (7936ead0) (System.String.Equals(System.String, System.String), mdToken: 06000142)

Thanks,

Kevin

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Fri, 15 Feb 2008 22:31:05 GMT

interesting, no nothing special, it might be slightly different versions of sos.dll...

re: .NET Debugging Demos Lab 2: Crash - Review

charju — Sun, 24 Feb 2008 10:13:16 GMT

A littel comments for exception object:

0:018> kb

ChildEBP RetAddr Args to Child

01e9f9fc 79f97065 e0434f4d 00000001 00000001 kernel32!RaiseException+0x53

01e9fa5c 7a05b941 02678b1c 00000000 00000000 mscorwks!RaiseTheExceptionInternalOnly+0x226

Review the first args of the 1st line in callstack: kernel32!RaiseException+0x53, "e0434f3d", you can also found this strange number at eventvwr log just descripbed in tess's lab2. This is a common CLR exception number.

Second, the real CLR exception object is identified in the first arg of mscorwks!RaiseTheExceptionInternalOnly

so, you can dumpobject to examine this exception object by !do 02678b1c or directly by !pe 02678b1c . If you type command !do, then you can found below info:

790fa3e0 40000b7 c System.String 0 instance 02679cc4 _exceptionMethodString

we can continously !do 02679cc4 to see more information about this exception.

Thanks for tess's great post!!! I'd learned much more debugging skill and thoughts from you, thanks again!

re: .NET Debugging Demos Lab 2: Crash - Review

Raymond Hung — Mon, 25 Feb 2008 04:29:46 GMT

Thank you for all the posting. This is the best place to learn managed debugging. :)

I have one question on !dso.

I am getting the following

0:018> !dso

OS Thread Id: 0xb90 (18)

ESP/REG Object Name

01e9fa18 0266174c System.NullReferenceException

01e9fa1c 0266174c System.NullReferenceException

01e9fa64 0266174c System.NullReferenceException

01e9fa78 0266174c System.NullReferenceException

01e9facc 0266174c System.NullReferenceException

01e9fad8 0266174c System.NullReferenceException

01e9fb74 02659b0c System.NullReferenceException

01e9fb78 0266040c System.Byte[]

01e9fba4 0266174c System.NullReferenceException

01e9fba8 0266040c System.Byte[]

01e9fbb8 0266040c System.Byte[]

01e9fccc 025f75c0 System.CodeDom.Compiler.TempFileCollection

The difference between this and the one shows in the lab is that the bottom piece. Your output shows

0:020> !dso

OS Thread Id: 0x116c (20)

ESP/REG Object Name

02a0f9f4 02f0cb3c System.NullReferenceException

02a0f9f8 02f0cb3c System.NullReferenceException

02a0fa40 02f0cb3c System.NullReferenceException

02a0fa58 02f0cb3c System.NullReferenceException

02a0faac 02f0cb3c System.NullReferenceException

02a0fabc 02f0cb3c System.NullReferenceException

02a0fb58 06f1e5e4 System.NullReferenceException

02a0fb5c 02f0bd70 System.Byte[]

02a0fb78 02f0cb3c System.NullReferenceException

02a0fb7c 02f0bd70 System.Byte[]

02a0fb8c 02f0bd70 System.Byte[]

02a0fc58 06eb281c System.RuntimeType

02a0fcb0 02efd4b0 Review <<--

So in my case, I am not able to follow some piece of the lab. I am running IIS 6.0 with Win2K3 SP2.

I am interested on getting the offset like the one in the lab

"Dumping out the Review object that we found with !dso we can see that at offset 0x4 we have the member variable quote and looking at the value we can see that it is null."

Is there another way to get the offset?

re: .NET Debugging Demos Lab 2: Crash - Review

Raymond Hung — Mon, 25 Feb 2008 04:38:36 GMT

Found it. Sorry, should have tried a little harder before posting the question.

0:018> !dumpheap -mt 01f98ba4

Address MT Size

02644834 01f98ba4 16

02644a74 01f98ba4 16

total 2 objects

Statistics:

MT Count TotalSize Class Name

01f98ba4 2 32 Review

Total 2 objects

0:018> !do 02644834

Name: Review

MethodTable: 01f98ba4

EEClass: 024354ac

Size: 16(0x10) bytes

(C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\debug1\b1d4288d\b06837fb\App_Code.jbom0q7o.dll)

Fields:

MT Field Offset Type VT Attr Value Name

790fa3e0 4000008 4 System.String 0 instance 00000000 quote

790fa3e0 4000009 8 System.String 0 instance 00000000 source

re: .NET Debugging Demos Lab 2: Crash - Review

Eber Irigoyen — Thu, 06 Mar 2008 08:24:38 GMT

I have the same problem as Raymond, the Review object doesn't show when I run !dso

I can get to it if I run !dumpheap -stat, find it "manually", then use !dumpheap -mt [address]

but there's gotta be an easier way than that

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Thu, 06 Mar 2008 09:50:11 GMT

if you dont see the review object with !dso that probably means that you caught the dump when it was finalizing some other object, in which case you would need to run !dumpheap -stat to get any review objects on the heap. Can you post your !dso output?

re: .NET Debugging Demos Lab 2: Crash - Review

Eber Irigoyen — Fri, 07 Mar 2008 01:18:32 GMT

is pretty much the same (other than the addresses) as the one Raymond posted

0:018> !dso

OS Thread Id: 0xb90 (18)

ESP/REG Object Name

01e9fa18 0266174c System.NullReferenceException

01e9fa1c 0266174c System.NullReferenceException

01e9fa64 0266174c System.NullReferenceException

01e9fa78 0266174c System.NullReferenceException

01e9facc 0266174c System.NullReferenceException

01e9fad8 0266174c System.NullReferenceException

01e9fb74 02659b0c System.NullReferenceException

01e9fb78 0266040c System.Byte[]

01e9fba4 0266174c System.NullReferenceException

01e9fba8 0266040c System.Byte[]

01e9fbb8 0266040c System.Byte[]

01e9fccc 025f75c0 System.CodeDom.Compiler.TempFileCollection

is there an easier way to locate a specific object though?

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Fri, 07 Mar 2008 10:32:30 GMT

I'm not sure why the review object is not showing up, it is probably gone by the time you gathered the dump.

Looking at the review object is just an additional step to verify the issue but you can verify it through looking at the exception and through looking at the code as well so getting the review object is not crucial to the issue.

re: .NET Debugging Demos Lab 2: Crash - Review

KB — Sun, 16 Mar 2008 06:55:09 GMT

Hey Tess,

Another great one! Thanks! Is it beneficial to have a good understanding of assembly language to be proficient at this? I'm assuming it does but wanted to get your take.

And speaking of that, in the 0:020> !u 0FE90F24 command, I had a tough time seeing the lines in bold type. Tried on two different monitors. Screen res on both is cranked to the highest setting possible. Could be other settings I guess. Anyway, just thought I'd mention it.

Take care.

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Mon, 17 Mar 2008 09:27:32 GMT

Hi KB,

No it is not essential to have a good understanding of assembly language to be proficient at debugging .net. I just wanted to add it for completeness.

I am not sure why the bolded lines are not showing up online. The bolding is in the html but I can't see it online either now that you mention it... either way it is these two lines that were bolded

0fe90f23 8b01 mov eax,dword ptr [ecx]

0fe90f25 ff5028 call dword ptr [eax+28h]

Thanks

Tess

re: .NET Debugging Demos Lab 2: Crash - Review

Thyagu — Tue, 25 Mar 2008 09:32:58 GMT

Hi Tess,

This is the best blog which i could find for debugging .net code. I have a strange problem reported by a user. The problem is that a customer application crashes at a certain point of time. When we had a look at the event viewer and the dumps and saw a Null Reference exception in our code.

SP IP Function

1E57ED38 1B1F5023 com.uni.jellybeans.CommsManager.Run()

1E57ED70 793B0D1F System.Threading.ThreadHelper.ThreadStart_Context(System.Object)

1E57ED78 793740AB System.Threading.ExecutionContext.runTryCode(System.Object)

00000000 00000001 System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)

1E57F204 79373FF7 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

1E57F21C 79373EDE System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

1E57F234 793B0C68 System.Threading.ThreadHelper.ThreadStart()

I followed your instructions and was successful in diassembling the code. But i could not find the exact string which is causing the failure in our code. i would need your assistance on this. The following is the dso of the dump

!dso

OS Thread Id: 0x1dc0 (58)

ESP/REG Object Name

1e57f528 10ebe164 System.NullReferenceException

1e57f52c 10ebe164 System.NullReferenceException

1e57f574 10ebe164 System.NullReferenceException

1e57f58c 10ebe164 System.NullReferenceException

1e57f5e0 10ebe164 System.NullReferenceException

1e57f5f0 10ebe164 System.NullReferenceException

1e57f68c 04ed5b00 System.NullReferenceException

1e57f690 10ebd12c System.Byte[]

1e57f6ac 10ebe164 System.NullReferenceException

1e57f6b0 10ebd12c System.Byte[]

1e57f6c0 10ebd12c System.Byte[]

Please let me know if you need anything further to analyze

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Tue, 25 Mar 2008 10:20:05 GMT

Hi Thyagu,

From this we can see that the app is starting a new thread and runs com.uni.jellybeans.CommsManager.Run() on it. and in there there is a null reference exception. If that is not handled then that would definitely lead to a crash.

Its hard to say anything else about what is causing the null ref just based on the stack above but you can do

!u 1B1F5023 which will unassemble the code for com.uni.jellybeans.CommsManager.Run() and then locate where in the !u output that 1B1F5023 falls to get a bit more info about where you are getting the nullref. If that doesnt help then perhaps looking at the code and adding some debug.writelines would help

re: .NET Debugging Demos Lab 2: Crash - Review

Thyagu — Tue, 25 Mar 2008 11:57:39 GMT

Hi Tess,

Thanks a ton for your quick response. I would like to admit that i am an amateur in analyzing this dumps. I have pasted below the failing disassembly code.

1b1f5016 8b0d3cbd9612 mov ecx,dword ptr ds:[1296BD3Ch]

1b1f501c 8b45d8 mov eax,dword ptr [ebp-28h]

1b1f501f 8b5844 mov ebx,dword ptr [eax+44h]

1b1f5022 8b530c mov edx,dword ptr [ebx+0Ch]

1b1f5025 e8c685175e call mscorlib_ni+0x2ad5f0 (7936d5f0) (System.String.Concat(System.String, System.String), mdToken: 060001c7)

1b1f502a 50 push eax

1b1f502b 8bcf mov ecx,edi

1b1f502d 8bd6 mov edx,esi

1b1f502f 8b01 mov eax,dword ptr [ecx]

1b1f5031 ff505c call dword ptr [eax+5Ch]

1b1f5034 8d65f4 lea esp,[ebp-0Ch]

I would be pleased if you can help me or guide me on further action.

Thanks a lot for your help.

Cheers,

Thyagu

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Tue, 25 Mar 2008 12:32:57 GMT

Hi Thyagu

The NullRef happens between these lines.

1b1f5022 8b530c mov edx,dword ptr [ebx+0Ch]

1b1f5025 e8c685175e call mscorlib_ni+0x2ad5f0 (7936d5f0) (System.String.Concat(System.String, System.String), mdToken: 060001c7)

The top line is taking ebx+0xc (probably a membervariable of some object) and stores it in a temporary register

The second line calls string.concat so it is doing something along the lines of str1 = str2 + str3

I would look for the line in code that concatenates strings and see if any of the strings that it concatenates may be null. and also look at the line of code above it to see if could cause a null ref.

I am not exactly sure how the code looks but looking at the disassembly we have

mov ecx,dword ptr ds:[1296BD3Ch]

=> ecx = whatever is stored at 0x1296BD3C

(likely some static string)

mov eax,dword ptr [ebp-28h]

=> eax = what is stored at ebp-0x28

(some parameter or local) i'll call it myparam for now

ebx,dword ptr [eax+44h]

=> ebx = what is stored at eax+0x44

(some membervariable of myparam) i'll call it mymembervar

edx,dword ptr [ebx+0Ch]

=> edx = ebx + 0xc

(some membervar of mymembervar)

so the code should look something like this

str = somestring + myParam.myMemberVar.membervar

and one of these is probably null

re: .NET Debugging Demos Lab 2: Crash - Review

Thyagu — Tue, 25 Mar 2008 16:54:23 GMT

Hi Tess,

Thanks for your quick response.

I spent the whole day in figuring where the failure had happened, but i was not successful.

If there is any alternative way to figure out which object is responsible for the Null Reference exception, it would be helpful.

FYI : This problem is not reproducible in our environment, so we can't debug the code and find out the reason.

Your help will be very much appreciated.

Cheers,

Thyagu

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Tue, 25 Mar 2008 17:03:58 GMT

if the above didnt help then the best would be to add tracing statements as i mentioned before to print out what objects are null etc. inside that method...

re: .NET Debugging Demos Lab 2: Crash - Review

PaulG — Wed, 04 Jun 2008 00:49:37 GMT

Hello Tess,

I agree with the other folks ... nice work on these labs!

I am a little late to the party as this Lab2 was posted a few months ago, but I had a question about your explanation of the assembly code.

You made this comment:

2. Right before the bolded line we set EAX to ptr(ESI+0x4) and it is when we use EAX that we access violate/get the nullreference exception.

Did you mean to say ECX here instead of EAX?

Thanks again for sharing your knowledge.

Cheers,

PaulG

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Wed, 04 Jun 2008 08:47:45 GMT

0fe90f20 8b4e04 mov ecx,dword ptr [esi+4]

0fe90f23 8b01 mov eax,dword ptr [ecx]

Well actually both, first we set ecx to ptr[esi+4], and then we move the contents of this pointer to eax

re: .NET Debugging Demos Lab 2: Crash - Review

PaulG — Wed, 04 Jun 2008 18:18:50 GMT

Thanks Tess ... that makes sense.

I have another question ... I hope you don't mind.

I liked how you displayed the assembly code and then used the IP to show us which line of assembly code was causing the access violation / null reference.

Is there a way to point us to the line of IL that is causing the null reference?

I can find the IL by printing my exception:

0:016> !pe 03cdbb0c

Exception object: 03cdbb0c

Exception type: System.NullReferenceException

Message: Object reference not set to an instance of an object.

InnerException: <none>

StackTrace (generated):

SP IP Function

025BF520 027A0F1C App_Code_ie7jwbrp!Review.Finalize()+0x14

StackTraceString: <none>

HResult: 80004003

Then I use my IP to find the corresponding method desc:

0:016> !ip2md 027A0F1C

MethodDesc: 029b0ae0

Method Name: Review.Finalize()

Class: 025c54ac

MethodTable: 029b0b0c

mdToken: 06000013

Module: 00ee8670

IsJitted: yes

m_CodeOrIL: 027a0f08

And then I dump the IL for this method desc:

0:016> !DumpIL 029b0ae0

ilAddr = 025227f4

.try

{

IL_0000: ldarg.0

IL_0001: ldfld Review::quote

IL_0006: callvirt System.Object::ToString

IL_000b: ldsfld System.String::Empty

IL_0010: call System.String::op_Inequality

IL_0015: brfalse.s IL_001e

IL_0017: ldarg.0

IL_0018: ldnull

IL_0019: stfld Review::quote

IL_001e: leave.s IL_0027

} // end .try

.finally

{

IL_0020: ldarg.0

IL_0021: call System.Object::Finalize

IL_0026: endfinally

} // end .finally

IL_0027: ret

Because we know that we are calling ToString() on a null string we it is obvious that this line is the culprit.

IL_0006: callvirt System.Object::ToString

But is there some commnad that will point us exactly to this line of IL?

Cheers again,

PaulG

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Thu, 05 Jun 2008 09:26:48 GMT

unfortunately there is no way (other than matching it up to the assembly) to find out which line is causing the issue

re: .NET Debugging Demos Lab 2: Crash - Review

Michael L. — Wed, 23 Jul 2008 16:38:19 GMT

In step one, regarding the application event log entry for 2003, you stated: "The process crashed because there was an unhandled exception on the finalizer thread when finalizing an object of type Review."

I'm assuming that you just know it was in the finalizer thread because you wrote it, right? Just looking at the event log entry, we can obviously tell what the exception was, but I'm missing how one would know where it occured without walking through the debugging.

On that note, do you know of a link that breaks down what the fields are in the event ID 5000 entry? I've not had any luck finding that reference...

Thanks! These labs have been a big help!

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Mon, 04 Aug 2008 22:54:05 GMT

_______________________

Message: Object reference not set to an instance of an object.

StackTrace: at Review.Finalize()

______________________

The stack for the exception shows that it is on the finalizer i.e. in the finalize method for the review object, so that is how I know...

Regarding the breakdown of the 5000 entry, what specifically are you looking for?

re: .NET Debugging Demos Lab 2: Crash - Review

Michael L. — Tue, 09 Sep 2008 19:24:04 GMT

For the 5000 entry, I'm looking for what are the P fields.

So, for instance, for this entry:

EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 app_code.wbwztx_4, P5 0.0.0.0, P6 47a81aee, P7 13, P8 0, P9 system.nullreferenceexception, P10 NIL

P5 looks like an emtpy IP address. Is it? What about P10? What is Nil? What is the address that is being shows in P3?

I'm not really worried about what the answers to those questions are for this specific example.. just the generic answer such as "P1 is the executable that generated the exception". I just haven't had any luck finding a reference for that, and was curious about it.

re: .NET Debugging Demos Lab 2: Crash - Review

Robin Sanner — Mon, 01 Dec 2008 19:37:20 GMT

In the walkthrough you state "ESI is typically used to hold the this pointer". Is there a compilation somewhere of this type of information when debugging .NET applications?

Also with your help I found it very easy to identify which x86 assembler instruction the exception occurred on, however determining which .NET source line or as a minimum which IL instruction in the method the exception occurred on. I can use WinDBG to produce the assembler version of the JITed method or the IL version of the method but I need an IL version of the method interspersed with the assembler instructions so that I can correlate.

re: .NET Debugging Demos Lab 2: Crash - Review

Tess — Mon, 01 Dec 2008 22:38:09 GMT

Robin,

The fact that ESI usually holds the this pointer is not .net specific. I dont know off hand if there is a compilation of what registers are usually used for but the most common ones are that ESI holds this, and eax holds the return value of a function.

Regarding the IL, you can run !dumpil on the method descriptor to get the il of the method.