Maoni's WebLog : Debugging

Difference Between Perf Data Reported by Different Tools – 4

maoni — Fri, 12 Jan 2007 08:03:00 GMT

.NET CLR Memory\% Time in GC counter and !runaway on thread(s) doing GC.

The 2 common ways people use to look at the time spent in GC are the % Time in GC performance counter under .NET CLR Memory, and the CPU time displayed by the !runaway debugger command in cdb/windbg. What do they mean exactly? % Time in GC is calculated like this:

When the n^thGC starts (ie, after the managed threads are suspended and before the GC work starts), we record the timestamp at that time. Let’s call this TA(n);

When n^thGC ends (ie, after the GC work is done and before we resume the managed threads), we record another timestamp. Let’s call this TB(n);

So Time spent in this GC is TB(n) – TA(n). And the time since the last GC ended is TB(n) – TB(n-1). So % Time in GC is (TB(n) – TA(n)) / (TB(n) – TB(n-1)).

Since we only record the timestamps we don’t actually discount the time when the thread was switched out – for example, if you are on a single proc machine and another process has a thread of the same priority (as the thread that’s doing GC) that’s also ready to run it may take away some time from the thread that’s doing the GC. None the less it’s a good approximation.

One common scenario where it’s not a good approximation is when paging occurs. In this case you will see a very high % Time in GC but really the time that’s actually spent doing GC work is low ‘cause most of the time is spent doing IO. To verify if you hit this case you can look at the Memory\Pages/sec to see how much it’s paging.

!runaway is more accurate in the sense that it does record the actual time spent on the threads. However I did observe 2 common mistakes when using !runaway to look at time spent in GC.

1) mistake “time spent on the GC thread(s)” as “time spent in GC”.

Let’s take Server GC as an example. When a GC is needed the GC threads first perform the suspension work. Obviously this takes time. Sometimes it can take quite a bit time if you have many threads or some threads get stuck and it takes a long time to suspend them.

2) using the current !runaway output to judge how much time GC has taken without taking into count that some user threads have died. Besides looking at the User/Kernel Mode time you may want to also look at Elapsed Time. The following output is from one of the issues I looked at:

Elapsed Time

Thread Time

0:2094 0 days 11:57:10.406

1:2098 0 days 11:57:10.152

2:2044 0 days 11:57:09.898

3:27cc 0 days 11:57:09.882

4:20c0 0 days 11:57:09.597

13:810 0 days 11:57:09.565

12:21d4 0 days 11:57:09.565

11:2308 0 days 11:57:09.565

10:1d24 0 days 11:57:09.565

9:23e0 0 days 11:57:09.565

8:24d0 0 days 11:57:09.565

7:2168 0 days 11:57:09.565

6:2134 0 days 11:57:09.565

5:2124 0 days 11:57:09.565

14:570 0 days 11:57:08.582

15:10fc 0 days 11:57:00.000

16:1ac4 0 days 11:56:58.256

17:1900 0 days 11:56:58.176

18:1624 0 days 11:56:57.494

19:11f8 0 days 10:35:01.881

21:1620 0 days 0:41:45.017

22:1cc8 0 days 0:37:30.496

23:2de0 0 days 0:29:01.820

24:2e10 0 days 0:29:01.711

25:2d88 0 days 0:22:26.988

26:1668 0 days 0:19:26.175

27:2cb0 0 days 0:16:16.814

29:2d1c 0 days 0:11:53.779

28:1de8 0 days 0:11:53.779

30:2ed4 0 days 0:11:35.466

32:2030 0 days 0:11:22.544

31:2084 0 days 0:11:22.544

33:12b0 0 days 0:07:53.510

35:28e4 0 days 0:03:13.801

34:2654 0 days 0:03:13.801

36:2d68 0 days 0:02:31.988

The orange lines are GC threads. They were created about 12 hours ago. The green lines are user threads and they were created 2 mins to 40mins ago. So naturally at the current state these user threads couldn’t’ve spent much time.

Difference Between Perf Data Reported by Different Tools – 3

maoni — Wed, 20 Dec 2006 08:51:00 GMT

Both the !SOS.gchandles command (added in CLR 2.0) and the .NET CLR Memory\# GC Handles counter show you the number of GC handles you have in your process.

The # GC Handles counter is one of the rare counters in the .NET CLR Memory category that doesn’t get updated at the end of each GC. Rather we update it in the handle table code, for example, when some code in the CLR calls the function to create a GC handle (possibly because the user has requested to create a handle via managed code), we increase this counter value by one. For performance reasons we don’t use interlocked operations when we need to increase or decrease this value. This means the value can get changed concurrently by multiple threads. For this reason you should always trust the value returned by the !SOS.gchandles command if you ever doubt the counter value. The SOS command is accurate because we always walk the handle table when you issue the command so it returns the # of handles truthfully.

Difference Between Perf Data Reported by Different Tools - 1

maoni — Tue, 12 Dec 2006 10:16:00 GMT

So, there are many perf tools and some of them report either the same or the same type of data. I want to talk about various differences between the ones related to managed heap investigation. This is not supposed to cover everything..just the ones I think people use frequently.

Managed Heap Size

We have both .NET CLR Memory perf counters and SoS extensions that report manged heap size related data.

Difference 1

The values reported by the .NET CLR Memory perf counters are affected by

1) how often they are collected - meaning however often you tell the tool you use (most people use perfmon, internally I know many groups' test teams collect them as part of the automation). The most frequent interval perfmon gives you is once per second.

2) how often they are updated. Most .NET CLR Memory perf counters are updated only at the end of each GC.

So for example, assuming you are using perfmon to collect at one sample per second and if more than one GC happened in the past second, you'll be missing the intermediate values.

And since these perf counters are updated only every so often, before a value is refreshed it stays the same value as it was updated last time. For example, if you are looking at % Time in GC and say the last value was 80% and if no GC happens for 10 seconds, this counter will stay at 80% but really during the past 10 seconds no GCs were happening.

On the other hand, SOS data is only gotten when you request, which means you get the value at the time you did the SOS command (unless of course the command specifically tells you it reflects a value updated at some specific point in the past).

My application seems to hang. What do I do? – Part 2

maoni — Tue, 14 Nov 2006 11:58:00 GMT

Last time I talked about the hang scenario where your process is taking 0 CPU and the CPU is taking by other process(es) on the same machine.

The next scenario is your process is taking 0 CPU and the CPU is barely used by other processes.

As one of the readers correctly pointed out, this is very likely because you have a deadlock. Usually debugging deadlocks is relatively straightforward – you look at what the threads are waiting on and figure out which other threads are holding the lock(s). And there are plenty of online resources that talk about debugging deadlocks. If you use the Windows Debugger package there are built in debugger extension dlls that help you with this like !locks and etc. If you are debugging a managed app the SoS debugger extension has commands that will aid you - !SyncBlk shows you managed locks (for CLR 2.0 there’s also !Dumpheap –thinlock for objects locked with ThinLocks instead of SyncBlk’s).

Another possibility is your process is not doing any CPU related activities. A common activity is IO – for example if the process is heavily paging you will see almost 0 CPU usage but it appears hang because the memory it needs is getting loaded from the disk which is really slow. A very useful tool that shows you what processes are doing is Process Monitor. Yesterday a program on my machine paused periodically – very annoying. So I used process monitor which showed me that this program periodically checks if I am logged onto my account in another program and since I am not, it would log me on, does a little bit stuff then log me off. And the hang was due to waiting on network IO. So to make it happy I logged myself on then the annoying periodic hang disappeared.

Now if your process is indeed taking CPU it can also appear to hang – as I mentioned last time this means different things for different people. If you have a UI app this can mean the UI is not getting drawn; if you have a server app this can mean your app is not processing requests. So you’ll have to define what hang means to you. I will use server app not processing requests as an example. Usually server applications run on dedicated machines. So let’s assume that’s the case here – you run a server on a machine and the server could consist of multiple processes. You measure the server performance by throughput. One scenario is the CPU usage is high (perhaps even higher than usual) but the throughput is lower than usual.

The easiest case, as one of the readers pointed out, is an infinite loop – very easy to debug. You break into the debugger a few times and see a thread is taking all the CPU and that thread can not exit some function – so there goes your infinite loop. And if your process is pretty much the only process that uses CPU at the time this is super obvious. It gets a bit more complicated if you have multiple CPUs and other processes are also using CPUs. But still since it’s an infinite loop the nice thing is it will always be executing if you don’t interfere so it’s always available to you to investigate as soon as it happens.

It becomes hard if the hang only reproes sporadically and when it reproes it only lasts for a little while. Time to whip out a CPU profiler. As another reader pointed out, Process Explorer is a useful tool to get you started. It shows you which processes are “active” – meaning it’s using CPUs. Personally I start with collecting appropriate performance counters, partially because pretty much all test teams in the product groups at Microsoft have some sort of automated testing procedure that collects perf counters so requesting them is easy. And because of the low overhead you can collect them for a long period of time so you have a histogram.

These are the counters I usually request (comments in []’s):

Processor\% Processor Time for _Total and all processors

[This is so I have an idea what kind of CPU usage I am looking at and if there are paticular processors that get used more than the rest]

Process\% Processor Time for all processes or less the ones that you already know can not be the problem

Thread\% Processor Time for all processes or less the ones that you already know can not be the problem

[The above counters will tell you which threads are using the CPU so you know which threads to look at]

[Since I usually look at GC related issues I request all counters under .NET CLR Memory]

.NET CLR Memory counters for all managed processes or less the ones that you already know can not be the problem

[If you are looking at other things you should add appropriate counters – for example, ASP.NET counters for apps that use ASP.NET]

[If you know the kind of activities your processes do you can add appropriate counters for them. For me I often request memory related counters like:]

Memory\% Committed Bytes In Use

Memory\Available Bytes

Memory\Pages/sec

Process\Private Bytes for processes I am interested in

…

At this point I can look at the results and concentrate on the interesting parts – for example when the CPU is usually high. I will have an idea which threads in what processes are consuming the CPU and the aspects of them that interest me (usually GC and other memory activities). Then I can request more detailed data on those processes/threads. For example I can ask the person to use a sampling profiler so I can see what functions are executing in the part I am interested in (along with other info – this depends on what the profiler you are using is capable of).

Some people prefer to take memory dumps when the process hangs, sometimes this doesn’t necessarily work (when it works it’s great) because if the hang is related to timing/how threads are scheduled the threads can easily behave differently when you interrupt it in order to take memory dumps so the hang may not repro anymore. If you do have consecutive dumps from one hang then you can use the !runaway command to see which threads have been consuming CPU. One dump is hardly useful for debugging hangs because it only gives you info at one point in time how the process behaves.

Correlating the output of !eeheap -gc and !address

maoni — Thu, 09 Nov 2006 03:36:00 GMT

!address is a very powerful debugger command. It shows you exactly what your VM space looks like. If you already got the output from the !sos.eeheap -gc command (refer to this article for usage on sos), for example:

0:003> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x01245078
generation 1 starts at 0x0124100c
generation 2 starts at 0x01241000
ephemeral segment allocation context: (0x0125a900, 0x0125b39c)
segment    begin allocated     size
001908c0 793fe120 7941d8a8 0x0001f788(128904)
01240000 01241000 0125b39c 0x0001a39c(107420)
Large object heap starts at 0x02241000
segment    begin allocated     size
02240000 02241000 02243250 0x00002250(8784)
Total Size   0x3bd74(245108)
------------------------------
GC Heap Size   0x3bd74(245108)

you can correlate the segments with the output of !address to get a better view of them. For this specific case here's the excerpt of the output from the !address command:

0:003> !address
[omitted]
    01232000 : 01232000 - 0000e000
                    Type     00000000
                    Protect 00000001 PAGE_NOACCESS
                    State    00010000 MEM_FREE
                    Usage    RegionUsageFree
    01240000 : 01240000 - 00052000
                    Type     00020000 MEM_PRIVATE
                    Protect 00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageIsVAD
               01292000 - 00fae000
                    Type     00020000 MEM_PRIVATE
                    Protect 00000000
                    State    00002000 MEM_RESERVE
                    Usage    RegionUsageIsVAD
               02240000 - 00012000
                    Type     00020000 MEM_PRIVATE
                    Protect 00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageIsVAD
               02252000 - 00fee000
                    Type     00020000 MEM_PRIVATE
                    Protect 00000000
                    State    00002000 MEM_RESERVE
                    Usage    RegionUsageIsVAD
    03240000 : 03240000 - 73050000
                    Type     00000000
                    Protect 00000001 PAGE_NOACCESS
                    State    00010000 MEM_FREE
                    Usage    RegionUsageFree
    76290000 : 76290000 - 00001000
                    Type     01000000 MEM_IMAGE
                    Protect 00000002 PAGE_READONLY
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageImage
                    FullPath C:\WINDOWS\system32\IMM32.DLL
               76291000 - 00015000
                    Type     01000000 MEM_IMAGE
                    Protect 00000020 PAGE_EXECUTE_READ
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageImage
                    FullPath C:\WINDOWS\system32\IMM32.DLL
[omitted]

-------------------- Usage SUMMARY --------------------------
[omitted]

-------------------- State SUMMARY --------------------------
    TotSize   Pct(Tots) Usage
   0275c000 : 1.92%      : MEM_COMMIT
   7b20a000 : 96.20%      : MEM_FREE
   0268a000 : 1.88%      : MEM_RESERVE

Largest free region: Base 03240000 - Size 73050000

This says that the 2 segments (starting from 01240000 and 02240000) are adjacent to each other - part of them are committed, the rest is still reserved memory. Before and after the 2 segments we got some free space there. As I mentioned below it’s very unlikely that the managed heap is fragmenting the VM because we are good about requesting large chunks at a time and usually the OS is not bad at giving us addresses that are pretty contiguous if possible. One of the very few cases where you would see managed heap fragmenting VM is if you have temporary large object segments and GC needs to frequently acquire and release VM chunks. Those chunks could be scattered in the VM space especially considering there are other things that consume VM as well at the same time.

New MSDN Article - Investigating Memory Issues

maoni — Mon, 23 Oct 2006 08:35:00 GMT

We have a new MSDN article out in the November issue that talks about investigating managed memory issues.

http://msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut/default.aspx?loc=en

Take a look and let me know what you think.

Oh, and it's also in 6 other languages (German, Spanish, French, Russian, Portuguese and Chinese) for readers that prefer one of those languages.

Tools that help diagnose managed memory related issues

maoni — Tue, 09 Nov 2004 07:03:00 GMT

I was writing an internal wiki page on performance and thought this info is useful to many external readers as well so here it goes.

vadump is a good start. It’s an mstools tool – meaning you can find it on your NT CD under bin\mstools. You can take a snapshot of the process and see if the GC heap is an issue or not. It was created a long time ago before managed code was popular so the GC heap is part of "Other data" in the output of vadump (heh...:P). If it's a small portion of your working set it's better to look elsewhere. It shows a lot of useful info though and I've worked with many external customers that didn't know about it so I thought it'd be useful to mention it.

Type vadump -? for help. Below is an example using vadump:

E:\clr\ndp\clr\src\VM>tlist | findstr /i devenv

1812 devenv.exe Microsoft Development Environment [design] - gcpriv.h

E:\clr\ndp\clr\src\VM>vadump -so -p 1812

Catagory Total Private Shareable Shared

Pages KBytes KBytes KBytes KBytes

Page Table Pages 116 464 464 0 0

Other System 127 508 508 0 0

Code/StaticData 3470 13880 1000 7828 5052

Heap 6521 26084 26084 0 0

Stack 51 204 204 0 0

Teb 14 56 56 0 0

Mapped Data 943 3772 0 444 3328

Other Data 6595 26380 26376 4 0

Total Modules 3470 13880 1000 7828 5052

Total Dynamic Data 14124 56496 52720 448 3328

Total System 243 972 972 0 0

Grand Total Working Set 17837 71348 54692 8276 8380

Module Working Set Contributions in pages

Total Private Shareable Shared Module

10 4 6 0 devenv.exe

75 4 0 71 ntdll.dll

64 3 5 56 kernel32.dll

… [more modules omitted]

20 2 0 18 NETAPI32.dll

Heap Working Set Contributions

2564 pages from Process Heap (class 0x00000000)

0x00240000 - 0x00340000 242 pages

0x02A80000 - 0x02B80000 153 pages

0x02E10000 - 0x03010000 482 pages

0x1AF10000 - 0x1B310000 911 pages

0x207D0000 - 0x20FD0000 776 pages

13 pages from Private Heap 0 (class 0x00001000)

0x00350000 - 0x00360000 13 pages

… [more heap data omitted]

Stack Working Set Contributions

33 pages from stack for thread 00000DE8

2 pages from stack for thread 0000071C

0 pages from stack for thread 00000CEC

1 pages from stack for thread 00000B3C

… [more stack data omitted]

Usually when we investigate issues, a perfmon log is essential. It shows you the size of each generation in the GC heap, the number of collections done on each generation, the number of GC handles and etc. And most importantly it shows you a trend so you know how your app behaves over a period of time which is often extremely important for discovering problems. See my GC perf counter blog entry for an explanation of these counters.

When people have trouble finding out what's holding onto memory CLRProfiler is a great tool to help figure that out. The CLRProfiler is a visual tool that shows you a tremendous amount of info on the managed memory usage such as the types of the objects that were allocated and relocated, when GCs were triggered and how much memory is allocated by managed API calls. Since it comes with a great in-depth document I won’t repeat stuff here.

The SoS(Son of Strike) debugger extension is also a very useful and powerful tool. It gives you insights that are not easily found in other tools. It comes with the latest Windows Debugger Package (sos.dll in the clr10 directory). Michael Stanton has a nice blog entry about using it.

Abhi Khune has a blog entry that illustrates using some of the tools mentioned above to solve memory related problems. Please take a look at it here.