Getting started with windbg - part I

Getting started with windbg - part I — Thu, 15 Nov 2007 06:24:42 GMT

PingBack from http://54.musicnewsandviews.info/2007/11/13/getting-started-with-windbg-part-i/

Getting Started with Windbg

If broken it is, fix it you should — Mon, 19 Nov 2007 11:27:43 GMT

I'm back from Oredev which turned out to be a really cool conference. Meanwhile my colleague Johan wrote

re: Getting started with windbg - part I

Kent Boogaart — Mon, 19 Nov 2007 16:00:55 GMT

FYI, I have posted a windbg/sos cheat sheet here: http://kentb.blogspot.com/2007/11/windbg-sos-cheat-sheet.html

re: Getting started with windbg - part I

Aaron Lerch — Mon, 19 Nov 2007 17:18:08 GMT

Instead of copying SOS.dll and doing a ".load sos", you can just do ".loadby sos mscorwks" to load SOS from the relevant framework install location.

Thanks for the good intro!

re: Getting started with windbg - part I

Brian — Wed, 21 Nov 2007 17:37:22 GMT

Great article. I'm just getting into debugging for the first time and finding it quite frustrating :) I suppose that comes with the territory.

Perhaps someone can explain the confusion i'm having over something. I'm troubleshooting a problem with a .net app that will grow very large in mem usage and eat up high cpu until it eventually becomes unresponsive. This happens randomly after the app will be up and working fine for hours. I suspect that garbage collection might be an issue (would GC really be suspect of the hang if the problem was that it grew too large and then started too much GC? I think I'm barking up the wrong tree and should be looking for a mem leak)

Never the less, based on the output I get from doing a !threads, I want to take a look at thread 25, which appears like this:

25 2d0 164a2dc0 1800220 Enabled 00000000:00000000 000f13e0 2 MTA (GC) (Threadpool Worker)

so I do ~25s and then pull a !clrstack. Some data comes back with these being on top of the stack:

17d9ea04 7c8285ec [FRAME: ECallMethodFrame] [DEFAULT] Void System.GC.nativeCollectGeneration(I4)

17d9ea14 799dbec3 [DEFAULT] I8 System.GC.GetTotalMemory(Boolean)

After all this rambling, one of my main questions is this - when I do a !clrstack -p on this same thread, I get output identical as doing it without the -p option. What would the reason for this be? I want to delv into this thread further by using the !do but I cannot.

re: Getting started with windbg - part I

JohanS — Wed, 21 Nov 2007 18:00:56 GMT

Hi Brian,

It sounds like you're performing a lot of generation 2 garbage collections. This is not the GC's fault as such. I'd take a long, good look at your memory usage to see what can be minimized.

Here's what I'd do:

1. Remove ALL calls to GC.Collect() if any. They will mess up the "ecosystem" for the GC and should only be used for testing purposes.

2. Take a look at my previous post http://blogs.msdn.com/johan/archive/2007/01/11/i-am-getting-outofmemoryexceptions-how-can-i-troubleshoot-this.aspx The title mentions out of memory exceptions, but it will show you how to look at general memory usage for your application and hopefully pinpoint the culprit.

Regards / Johan

re: Getting started with windbg - part I

Cory Foy — Sat, 24 Nov 2007 21:15:19 GMT

@Aaron - Just to add - you can only do that if the framework has already been loaded. For example, you can't do that at the beginning of a process because mscorwks hasn't been loaded yet.

If you need to get SOS loaded as soon as mscorwks is loaded, you can do:

sxe ld mscorwks

.loadby sos mscorwks

Generally most of us do .loadby sos mscorwks, but there are circumstances where that won't work.

Cory

Debug Diagnostic Tool 1.1

IIS ve ASP.NET Notları — Fri, 30 Nov 2007 16:41:54 GMT

IIS üzerinde çalışan uygulamalarda yaşadığımız sorunları birkaç kategoride toplayabiliriz: "Crash" sorunları

re: Getting started with windbg - part I

Tess — Wed, 05 Dec 2007 15:00:15 GMT

A note for Brian here. GC.GetTotalMemory() will call GC.Collect if you pass it true. To get rid of that behaviour you should pass in false instead.

The reason you dont get any more data if you run !clrstack -p is probably because you are not building your dlls in debug mode (which btw is a very good practice in production).

With retail builds you may still get some more info if you run !clrstack -a

/Tess

Avoid Manipulating Passwords In Memory - It Is Easy To Reveal

Noticias externas — Sat, 08 Dec 2007 09:45:46 GMT

Revealing clear text passwords in memory seems to be a trivial task. This post describes how to reveal

re: Getting started with windbg - part I

MItch Wheat — Sun, 09 Dec 2007 03:10:51 GMT

Excellent introduction to Windbg. Nice one , Johan!

December 16th Links: ASP.NET, ASP.NET AJAX, ASP.NET MVC, VS, .NET, IIS7, WPF

ScottGu's Blog — Mon, 17 Dec 2007 10:29:21 GMT

Here is the latest in my link-listing series .  Also check out my ASP.NET Tips, Tricks and Tutorials

re: Getting started with windbg - part I

Paul — Thu, 20 Dec 2007 18:16:33 GMT

Johan, great stuff there, bookmarked for reference! ;o)

re: Getting started with windbg - part I

HS — Fri, 28 Dec 2007 23:13:30 GMT

Great article!

In your article you said "The threads with an ID of XXXX have ended and are waiting to be recycled.", but you didn't say what is considered "ended". I thought that IIS will continue using the same threads over & over again from its thread pool and thus would not end any threads in the threadpool. Is having "DeadThreads" normal or does it require some investigation? Thanks.

re: Getting started with windbg - part I

JohanS — Wed, 02 Jan 2008 12:36:54 GMT

Thanks,

The !threads-command only lists the managed threads. If you want the native threads you should use ~ instead.

Managed threads are occasionally ended when there's nothing left to do. One thread may handle a couple of requests and then end. Having ended threads on the heap, like in the sample above, is completely normal.

/ Johan

re: Getting started with windbg - part I

dannyR — Thu, 03 Jan 2008 13:02:22 GMT

where is the sos.dll for .NET 3.5? I can't find it anywhere.

re: Getting started with windbg - part I

JohanS — Fri, 04 Jan 2008 12:53:45 GMT

Hi Danny,

No problem. Use the 2.0 version.

/ Johan

re: Getting started with windbg - part I

Peter — Wed, 09 Jan 2008 22:03:01 GMT

Hi Johan,

Gr8 article. I came looking for help on debugging memory fragmentation for unmanaged applications. I have a simple C\C++ application running on WS2K3 SP1. I'm not sure, loading SOS in my WinDbg will do me much good. Could you give me some information on WinDbg commands for unmanaged applications.

Also, I've been using !heap -l to find out leaked heaps, but can I be really sure all the heaps displayed are really leaking memory.

Thanks !

re: Getting started with windbg - part I

Rene Pally — Fri, 11 Jan 2008 16:36:18 GMT

Nice Article,

Can you tell me please what it means? :

(a60.b5c): CLR exception - code e0434f4d (first chance)

Thank you very much

12月16日链接篇： ASP.NET, ASP.NET AJAX, ASP.NET MVC, VS, .NET, IIS7, WPF

ASP.NET Chinese Blogs — Fri, 18 Jan 2008 21:15:15 GMT

【原文地址】 December 16th Links: ASP.NET, ASP.NET AJAX, ASP.NET MVC, VS, .NET, IIS7, WPF 【原文发表日期】Sunday, December

re: Getting started with windbg - part I

lbq1221119@cnblogs — Tue, 22 Jan 2008 04:10:52 GMT

nice

i would like to know the detail of the listed results of the SOS extern Commands..

how and where can i find the detail documents??

3ks so much

re: Getting started with windbg - part I

JohanS — Tue, 22 Jan 2008 12:54:17 GMT

I'm not sure I follow. Could you please elaborate a little?

.NET Debugging Demos - Information and setup instructions

If broken it is, fix it you should — Mon, 04 Feb 2008 12:14:03 GMT

.NET Debugging Demos This is a series of debugging demos aimed to help you get some hands on experience

re: Getting started with windbg - part I

Matt G — Wed, 05 Mar 2008 19:52:17 GMT

What can be done to investigate threads that have ended and are waiting to be recycled?

I've got an intermittent ASP.NET worker process crash. Running the !threads command on the dump shows a managed thread with a StackOverflowException. Is there any way to get the stack trace on that thread?

If not, do you have any suggestions for setting up adplus to do a memory dump that does have the stack information?

Thanks

re: Getting started with windbg - part I

JohanS — Thu, 06 Mar 2008 18:43:37 GMT

Hi Matt,

Are you sure you've really experienced the StackOverflowException?

There is always a StackOverflowException created by default "just in case". The reason for this is that when a StackOverflow occurs the framework normally won't be in a state to create an exception, so it has to create it proactively. See my post on exception hunting for further details. http://blogs.msdn.com/johan/archive/2008/01/31/using-windbg-hunting-exceptions.aspx

/ Johan

re: Getting started with windbg - part I

Matt G — Sat, 08 Mar 2008 00:04:55 GMT

Based on your other blog post it appears I do not have a StackOverflowException.

I'm not sure where to go from here. I can get the exceptions from the heap per your post, but I don't know how to use that information to determine what killed w3wp.exe.

re: Getting started with windbg - part I

JohanS — Mon, 10 Mar 2008 12:26:14 GMT

Hi Matt,

Run ~*kb (this will show you the native stack for all threads) and look for kernel32!ExitProcess. This way you'll find the thread that ended the process.

re: Getting started with windbg - part I

Jun — Mon, 12 May 2008 23:24:16 GMT

Hi Johan,

about those threads with an ID of XXXX, should they go away after certain amount of idle time like 2 minutes? I am trouble shooting an application while the application should sit idle (because of no stimulation), however it is still using quite some CPU time and and threadpool threads, where it should not. So I created an adplus dump when it is sitting idle, and I found that there are many threads with ID XXXX, and I created a dump file again after 15 minutes, again, it still have the exact same XXXX threads. All those threads are completion port threads, and I wonder why they did not get recycled?

Also, when doing a '~*e !clrstack', most of the worker threads and completion port threads are showing "Failed to start stack walk: 80004005". Is there a way to show the stack for those threads, because those are the threads I am intereted in.

Thank you!

re: Getting started with windbg - part I

JohanS — Tue, 13 May 2008 16:40:53 GMT

The GC will be triggered when you’re allocating memory. If your application is inactive, then so is the Garbage Collector. Even if you’re making a few, random allocations they still may not be enough to trigger a GC, so this is not at all unexpected.

The “Failed to start stack walk: 80004005”-error is displayed when the thread did contain a managed stack, but no longer does.

.NET 调试实例 - 信息和安装说明 (原创翻译)

Justin — Mon, 07 Jul 2008 20:41:57 GMT

.NET调试实例

这是一个系列的调式实例，目的是为了帮助你在调式.NET应用程序中最常见的挂起(Hang)、性能(performance)、内存(memory)和系统崩溃(crash)方面获得一些...

re: Getting started with windbg - part I

Anish — Fri, 11 Jul 2008 15:04:10 GMT

Hi,

Great content.

I am new to debugging .net programs using windebug.

How can I set break point on Main method of a console application.

Following are the steps i have done

1.Open the executable in windbg.exe

2.Set a break poin in mscorwks using following command

bp mscorwks!EEStartup "g @$ra"

3. From here onwards i am able to use sos commands.

I wanted to set breakpoint in Main method of

ConsoleApplication1.Program.Main.

I have tried using !bpmd command as mentioned int eh sos help.

"bl" command lists the breakpoint also. But program is not breaking at the Main

what could be the reason

Thank You,

Anish

New to Debugging?

Adventures in IIS, Commerce and whatever else comes into my head. — Wed, 23 Jul 2008 13:34:03 GMT

I've just finished writing up an e-mail for some new people in my team about starting Debugging and the

Day of .Net Presentation Resources

David Truxall — Tue, 21 Oct 2008 15:02:00 GMT

Thanks to everyone who attended my talk! Here's some links related to the talk: My PowerPoint Presentation...

Debugging OutOfMemoryExceptions in managed code using Windbg

Amol Ravande's Performance blog — Tue, 16 Dec 2008 20:40:46 GMT

Before you start debugging make sure you have the symbols path set appropriately. Below link shows you

re: Getting started with windbg - part I

Kams — Mon, 30 Mar 2009 15:19:52 GMT

I have been using !clrstack to display stack trace, but what I don't see method names. In my case i get

0:028> !clrstack

OS Thread Id: 0xffc (28)

ESP EIP

04e9e1f4 7c90e4f4 [GCFrame: 04e9e1f4]

04e9e210 7c90e4f4 [GCFrame: 04e9e210]

04e9e3f4 7c90e4f4 [HelperMethodFrame_1OBJ: 04e9e3f4]

04e9e464 792d5348

04e9e4b4 792d514f

04e9e4f0 792d4fde

04e9e510 6384da66

04e9e560 6384d666

04e9e5bc 63a95573

04e9e5d0 07df1c91

04e9e600 064b6935

04e9e644 07df1b7c

04e9e684 07df15f1

04e9e6d8 0623e9c7

04e9e79c 056baf8c

04e9e7e4 056b89cf

04e9e864 056b7a1c

04e9e948 056b6b51

04e9ea94 056b5d0e

04e9ead4 056b5bd0

04e9ead8 056b123e

04e9eba0 05e9b38c

04e9ec10 062347a1

04e9ec90 062344b9

04e9f0c8 79e71b4c [HelperMethodFrame_PROTECTOBJ: 04e9f0c8]

04e9f20c 792795b3

04e9f230 7927038b

04e9f28c 7927023c

04e9f294 792701f5

04e9f2a8 792700e1

04e9f2dc 7926fe5f

04e9f318 7985f9ba

04e9f360 79864e7a

04e9f390 066d8553

04e9f484 066d7049

04e9f4f4 066d5ccf

04e9f550 066d5afc

04e9f580 066d59d1

04e9f588 066d58b7

04e9f5b8 066d57e8

04e9f5d0 7928cdc4

04e9f770 79e71b4c [GCFrame: 04e9f770]

Any clues on how I can get this right. I have set the symbol file path to the microsoft server as suggested above.

when I do a !dso, I can see objects on the stack, but If !clrstack would work it would be great!

Any help on this would be appreciated

re: Getting started with windbg - part I

JohanS — Mon, 30 Mar 2009 16:17:10 GMT

It seems likely that you're loading the wrong version of SOS. Try loading SOS using the following command:

.loadby sos mscorwks

/ Johan

[收藏]Getting started with windbg - part I

dudu — Sun, 26 Apr 2009 16:54:49 GMT

最近在看相关内容，作者写的相当不错，自己随手学习了一下，非常棒，顺便也锻炼一下自己的翻译能力（好久没有处理相关内容了，能力直线下降啊）原文地址：http://blogs.msdn.com/jo...

re: Getting started with windbg - part I

Dominic Zukiewicz — Mon, 01 Jun 2009 23:52:36 GMT

Thank you so much for an thorough introduction to WinDBG. It is something I have been looking for for months.

I am having some real trouble with the introduction. Funnily, I have been able to do the later labs (5-7), but came back to the beginning to check I hadn't missed anything.

I followed your above instructions with the "Crash" lab, to see the instructions generated, but I am finding that a lot of the commands are not yielding any good results.

For example

!threads

PreEmptive GC Alloc Lock

ID OSID ThreadOBJ State GC Context Domain Count APT Exception

1 1 c10 001a0a28 a808220 Enabled 00000000:00000000 001d04a8 1 MTA (Threadpool Completion Port)

7 2 664 001a4190 b220 Enabled 01056390:0105652c 0016e3a8 0 MTA (Finalizer) System.NullReferenceException (01055ad0)

8 3 12cc 001bd228 80a220 Enabled 00000000:00000000 0016e3a8 0 MTA (Threadpool Completion Port)

11 4 9a4 001cff88 1220 Enabled 00000000:00000000 0016e3a8 0 Ukn

Switch to Thread 2

0:011> ~2

2 Id: 158c.288 Suspend: 1 Teb: 7ffd9000 Unfrozen

Start: msvcr80!_endthread+0x71 (7813286e)

Priority: 0 Priority class: 32 Affinity: 3

!clrstack it

0:011> !clrstack

OS Thread Id: 0x9a4 (11)

Failed to start stack walk: 80004005

Nothing? I also tried !pe to get:

0:011> !pe

There is no current managed exception on this thread

But there must be? As it says on the !threads!

I'm not sure what I've been doing wrong. Can you give me guidance please?

Thanks again

Dominic

re: Getting started with windbg - part I

JohanS — Wed, 03 Jun 2009 15:03:14 GMT

Hi Dominic.

Here's where it went wrong:

~2 will give you info about the thread.

~2s will switch to the thread in question. This is the command you wanted.

As you can see you're still on thread 11:

0:011> !clrstack

OS Thread Id: 0x9a4 (11)

Failed to start stack walk: 80004005

/ Johan

re: Getting started with windbg - part I

Dominic Zukiewicz — Tue, 09 Jun 2009 11:58:54 GMT

Thanks Johan!

I always wonder what the different was between those 2 commands.

re: Getting started with windbg - part I

Caveman — Fri, 12 Jun 2009 06:07:48 GMT

Thank you for the cool intro. Johan!

re: Getting started with windbg - part I

JohnOpincar — Thu, 22 Oct 2009 12:46:42 GMT

Thanks for the great post. It's been invaluable to me as I've researched an IIS deadlock problem.