Rico Mariani's Performance Tidbits : using tools

Introducing Performance Console

ricom — Fri, 04 Aug 2006 02:03:00 GMT

Ever wonder how I get those nice looking HTML call trees with attributed costs like this one here in my last quiz?

Well, it turns out that Josh Williams wrote an excellent program for massaging the perf results that come out of our profiler. The way it works is that you get a profile as usual and then you use the tools (or the UI) to export the data in .csv format. Josh shows how in this article from the command line but the export function does the same job.

Once you have your .csv files you can use the newly released Performance Console to read those files and browse them. Now I love this tool because it includes some of my favorite features (yes I nag very well, Josh can attest *grin*) such as "show me the call tree with all functions at least 5% inclusive time." And the best part is you don't have to keep clicking open-open-open to browse deep call trees.

By the way, our own profiler team looks at this tool and others very closely in deciding what to do in their next iteration so hopefully we'll see more and more of this stuff being available without special downloads.

But why wait? You can have this goodness now.

One last great thing about the perfconsole tool -- you can save the results of a session as HTML and then mail them or blog them or whatever.

Josh will be posting some usage hints soon. But for now, go ahead and grab it and play. You'll love it! Heck, post me some of your favorite outputs.

Did I mention its extensible? I hope he posted the whole manual or I have to nag some more. :)

Try it, you won't be sorry.

UPDATE: Josh just wrote an introduction to Performance Console, find it here.

LogDump: CLRProfiler Log analysis tool

ricom — Tue, 09 Aug 2005 03:44:00 GMT

Ever wonder how I produce nice textual allocation summaries like this one?

This report shows allocations in dictread.log

Total Allocations 136301 Objects 6694232 Bytes

Top 10 Allocated Types

   Count    Bytes Type
108040 5836392 System.String
   26726   534520 DictionaryEntry
      73   275604 System.Object []
    1010    20200 System.Text.StringBuilder
      51    10466 System.Char []
      11     6358 System.Byte []
      53     1272 System.Collections.ArrayList
      52     1248 ArrayListEnumeratorSimple
       6     1008 bucket []
      18      504 System.Security.SecurityElement

Top 10 Allocating Stacks

Stack 1 allocates 3132656 bytes
NS.Test::Main static void (String[])
Dictionary::.ctor void ()
DictionaryEntry::Parse static DictionaryEntry (String)
System.String::Substring String (int32 int32)
   80178 3132656 System.String

Stack 2 allocates 2402768 bytes
NS.Test::Main static void (String[])
Dictionary::.ctor void ()
System.IO.StreamReader::ReadLine String ()
   25723 2382668 System.String
    1005    20100 System.Text.StringBuilder

Stack 3 allocates 534520 bytes
NS.Test::Main static void (String[])
Dictionary::.ctor void ()
DictionaryEntry::Parse static DictionaryEntry (String)
   26726   534520 DictionaryEntry

That's a report similar to the one that I generated for Performance Quiz #6 -- I get a ton of mileage out of these. Just use CLRProfiler for Beta2 or greater, save your .log files and run this little tool over it to get a summary of the total costs and their major sources. This little guy saves me a ton of time and I've used it to anlayze multi-gigabyte log files where the UI approach just isn't going to work. The algorithm is to just attribute each allocation to a callstack and then compute the callstacks with the biggest cost -- which is basically the same as going into the tree view in the UI and opening the biggest choice at each step, then 2nd biggest etc.

I've just posted the source code for this little dumper -- it's a quickie I wrote one afternoon and it shows -- but hopefully you can get some good usage out of it as well.

Legal Note: Like all other code samples this program is offered AS IS with no warranty implied and confers no rights.

Narrowing Down Performance Problems in Managed Code

ricom — Thu, 26 May 2005 03:10:00 GMT

My last entry was some generic advice about how to do a good performance investigation. I think actually it's too generic to be really useful -- in fact I think it fails my Peanut Butter Sandwich Test.

Digression to discuss the Peanut Butter Sandwich Test

I review a lot of documents and sometimes they say things that are so obvious as to be uninteresting. The little quip I have for this situation is, "Yes what you are saying is true of [the system] but it's also true of peanut butter sandwiches." Consider a snippet like this one, "Use a cache where it provides benefits," and compare with, "Use a peanut butter sandwich where it provides benefits." Both seem to work... that's a bad sign.

You certainly don't want to get an F on the Peanut Butter Sandwich Test but hopefully you won't settle for just a C-.

Back on topic

I thought it would be good to follow up the generic advice with some specific suggestions for things to look at. These are things I look at in step 2 or 3 of the investigation.

Under .NET CLR Memory, check "% Time in GC" if it's getting near 10% or higher you may have some memory issues, consider these secondary tests:

is the raw allocation rate "Allocated Bytes/sec" too high? -> reduce total allocations
is the promotion rate "Promoted Memory from Gen 1" too high? -> be careful about object lifetimes, avoid "mid-life crisis"
is the finalization rate "Finalization Survivors" too high? -> make sure you are disposing the key objects
is the heap growing when it shouldn't "# Bytes in all Heaps" -> check for reference leaks

Is the CPU not saturated when it should be? Look under .NET CLR LocksAndThreads

is the "Contention Rate / sec" counter high compared to your throughput rate? -> you should re-examine your locking strategy
is the "# of current physical Threads" too low for the problem? -> (ammended) more parallelism may be helpful, consider using the ThreadPool if not already in use, possibly adjust ThreadPool parameters to get more threads (not usually needed)
in the "Thread" category examine "Context Switches / sec", is this high compared to your throughput rate? -> perhaps the workitem you are giving threads in the thread pool is too small, consider something chunkier

Is the throughput rate low even though the CPU is saturated?

look under ".NET CLR Exceptions", is "# of Excepts Thrown / sec" high compared to your throughput? -> consider reducing use of exceptions in common paths
look under ".NET CLR Interop", is "# of marshalling" growing too fast? -> consider simplifying the arguments passed in interop cases so that marshalling is cheaper
look under ".NET CLR Security", is "% Time in RT checks" significant? -> consider simplying the demands being placed on the security system to lower the cost of security checks
look under ".NET CLR Jit", is "% Time in Jit" significant? This counter shouldn't stay high because jitting should settle out, if it remains high then perhaps there is dynamic code generation via reflection going on -> simply dynamic code cases

This just a taste of course, and each of these items would likely lead to further investigation with a profiling tool that is suitable to drilling into that particular kind of problem but these are examples of leading indicators that I use.

For more information on the GC Performance counters specifically see Maoni's blog entry on that subject. Her most recent article is on using the GC efficiently also very interesting, lots of good details there.

How To Do A Good Performance Investigation

ricom — Tue, 24 May 2005 02:54:00 GMT

I find that sometimes people have difficultly just getting started when doing a performance analysis – meaning they’re faced with a potentially big problem and don’t know where to begin. Over the years many people have come to me under those circumstances and asked me how I would approach the problem. So today I’m trying to distill those bits of advice – my Modus Operandi – into some simple steps in the hope that it might help others to get off to a good start.

So here it is, Rico’s advice on how to do a good performance investigation.

Preliminaries

The first thing to remember is not to try to do this in a rushed way. The more of a hurry you are in to get to the bottom of the problem the less you can afford to be hasty. Be deliberate and careful. Block out a good chunk of time to think about the problem. Make sure you have the resources you need to succeed – that means enough access to the right hardware and the right people. Prepare a log book – electronic if you like – so that you can keep notes and interim results at each step. This will be the basis of your final report and it will be an invaluable reference to anyone that follows in your footsteps – even if (especially if?) that someone is you.

Step 1 – Get the Lay of the Land

Before you look at any code, talk to the people involved. You’ll want to get a feel for what they are trying to accomplish. What are their key difficulties? What is inherently hard about this problem? What is the overall organization of the system? What are the inherent limitations of those choices? What is the chief complaint with the current system? What would a successful system look like? How do they measure success?

In addition to a basic understanding of how the system is intended to work the key question you want to answer is this: Which resource is the one that should be critical in this system? Is the problem fundamentally CPU bound, disk bound, network bound? If things were working perfectly what would be constraining the performance?

Step 2 – Identify the current bottlenecks

In this step we cast a broad net to see what resource is the current limiting factor. The tool I reach for first is PerfMon. Look at key counters like CPU Usage, Memory Usage, Disk and Network I/O. Examine these on all the various different machines involved if this is a server problem (i.e. check all the tiers). Check for high levels of lock contention.

At this point you should be able to identify which resource is currently the one that is limiting performance. Often it is not the resource identified in Step 1.

If it is the correct resource – the one that is supposed to be the limiting factor in this kind of computing – that’s a good preliminary sign that sensible algorithms have been selected. If it’s the wrong resource it means you are going to be looking for design problems where a supposedly non-critical resource has been overused to the point that it has become the critical resource. The design will have to be altered such that it does not have this (fundamentally unnecessary) dependence.

Step 3 – Drill down on the current bottleneck

A common mistake in a performance analysis is to try to do step 3 before step 2. This is going to lead to a good bit of waste because you could do a deep analysis of say CPU usage when CPU usage isn’t the problem. Instead, choose tools that are good at measuring the problem resource and don’t worry so much about the others for now. If it’s hard to measure the resource in question, add instrumentation for this resource if possible. The goal is to find out as much as you can about what is causing the (over) use of the bottleneck resource.

When doing this analysis, consider factors that control the resources at different system levels starting from the largest and going to the smallest. Is there something about the overall system architecture that is causing overuse of this resource? Is it something in the overall application design? Or is it a local problem with a module or subcomponent? Most significant problems are larger in scope, look at those first. They are the easiest to diagnose and sometimes the easiest to correct. E.g. if caching were disabled on a web server you could expect big problems in the back end servers. You’ll want to make sure caching is working properly before you decide to (e.g.) add more indexes to make some query faster.

Your approach will need to be tailored to the resource and the system. For CPU problems a good time profiler can be invaluable. For SQL problems there’s of course SQL Profiler (find the key queries) and Query Analyzer (view the plans). For memory issues there are abundant performance counters that can be helpful, including the raw memory counters and .NET memory management ones, and others. Tracking virtual memory use over time can be helpful – sampling with vadump is handy. Examination of key resources by breaking into the system with a debugger can also be useful.

Step 4 – Identify anomalies in the measurements

The most interesting performance problems are almost always highlighted by anomalous observed costs. Things are happening that shouldn’t be happening or need not happen. If the critical resource isn’t the “correct” one as identified in Step 1 it’s almost certainly the case that your root-cause analysis will find an undesired use of the resource. If the resource was the “correct” one then you’ll be looking for overuse to get an improvement.

In both cases it is almost always helpful to look at the resource costs per unit-of-work. What is the “transaction” in this system? Is it a mouse click event? Is it a business transaction of some kind? Is it an HTML page delivered to the user? A database query performed? Whatever it is look at your critical resource and consider the cost per unit of work. E.g. consider CPU cycles per transaction, network bytes per transaction, disk i/o’s per transaction, etc. These metrics will often show you the source of the mistake – consider in my recent analysis of Performance Quiz #6 where I looked at the number of string allocations per line of input and then bytes allocate per byte in the input file. Those calculations were both easy and revealing.

Expressing your costs in a per-unit-of work fashion will help you to see which costs are reasonable and which are problems.

Step 5 – Create a hypothesis and verify it

Based on the analysis in step 4 you should be able to find a root cause for the current bottleneck. Design an experiment to validate that this is the case. This can be a very simple test such as “if we change the settings [like so], it should make the problem much worse.” Take advantage of any kind of fairly quick validation that you can do… the worst thing to do is to plunge into some massive correction without being sure that you’ve really hit the problem. If there is nothing obvious, consider doing only a small fraction of the corrective work. Perhaps just enough for one test case to function – validate that before you proceed.

The trick to doing great performance work is to be able to try various things and yet spend comparatively little time on the losing strategies while quickly finding and acting on the winners.

Step 6 – Apply the finished corrections, verify, and repeat as needed

After Step 5 you should be highly confident that you have a winning change on your hands. Go ahead and finish it up to production quality and apply those changes. Make sure things went as you expected and only then move on. If your changes were not too sweeping you can probably resume at Step 2, or maybe even Step 3. If they were very big changes you might have to go back to Step 1.

Step 7 – Write a brief report

Summarize your method and findings for your teammates. It’s invaluable as a learning exercise for yourself and as a long-term resource for your team.

Post Script

I wrote a followup article which offers more prescriptive advice about managed code specifically -- see Narrowing Down Performance Problems in Managed Code

Performance Quiz #6 -- Analyzing the managed code

ricom — Wed, 18 May 2005 00:12:00 GMT

Raymond didn't post an update today so I think this is a good time for me to post the first analysis of the managed code. There's some things to notice right away and I've highlighted them in the table below. This table was generated in the same way as the others and again I've pruned the tree so that no functions with inclusive time less than 5% are shown, so that means some things won't add up because some small things are gone -- still it helps focus attention on the relevant bits and keeps this table reasonably sized. I've also removed functions which were basically just zero or near zero cost wrappers, so there are some internal "..." entries. That's what those are.

Name	Exclusive Percent	Inclusive Percent
__CorExeMain (from mscoree.dll)	0.00	92.31
__CorExeMain (from mscorwks.dll)	0.00	88.46
ExecuteEXE(struct HINSTANCE__ *)	0.00	69.23
SystemDomain::ExecuteMainMethod(...)	0.00	69.23
Assembly::ExecuteMainMethod(class PtrArray * *)	0.00	59.62
…	0.00	59.62
NS.Test.Main(string[])	0.00	59.62
NS.Test.Dictionary..ctor()	1.92	58.65
System.IO.StreamReader.ReadLine()	3.85	28.85
System.IO.StreamReader.ReadBuffer()	0.00	16.35
System.Text.DecoderNLS.GetChars(...)	0.00	15.39
…	0.00	15.39
…	0.00	15.39
System.Text.DBCSCodePageEncoding.GetChars(...)	15.39	15.39
System.String.CtorCharArrayStartLength(char[],int32,int32)	0.00	5.77
NS.Test.DictionaryEntry.Parse(string)	2.89	26.92
System.String.Substring(int32,int32)	0.96	14.42
System.String.InternalSubString(int32,int32)	0.00	13.46
...	0.00	7.69
…	0.00	6.73
Alloc(unsigned long,int,int)	6.73	6.73
String::IndexOfChar(...)	1.92	5.77
EnsureEEStarted(...)	0.00	18.27
...	0.00	18.27
...	0.00	18.27
InitializeProfiling	0.00	6.73

First thing to notice is that the time attributed to the overall "main" of the program (that would be __CorExeMain) is only 92%. Why isn't that 100%? Well there's a few reasons. The first is that the CLR has more than one thread so some of the time went to spinning up the finalizer thread for instance. A second reason is that some percentage of the run will be kernal time and we won't be able to attribute those stacks to anything so those are gone. And the third, and most subtle reason, is that some places during the run of a managed program have a stack frame that is so crazy (usually because we're in the middle of an exotic transition) that the profiler can't walk it. That causes it to drop some samples. The result of these factors is that main is usually close to but not 100% of the time. In this case 8% of the samples were elsewhere or not traceable.

But now lets get to some meatier conclusions: NS.Test.Main() which is the managed main function is only getting 59% of the time. Even allowing for the the fact that main was only 92.31% of the time, the managed main is only 64.6% of that. Why is that?

Well there were some assorted small things that were in the set of things pruned from this view (at less than 5%) but they added up... some of it was checking the security configuration only to discover that we are full trust, some of it was looking for possible administrative configurations, some of it was setting up the performance counters for this instance of the application.

Another chunk of the time still visible even in this pruned view -- EnsureEEStarted() for instance at 18.27% of the startup including 6.73% to get the profiling services working. That part of the cost would naturally not even be present if the program wasn't run under the profiler. All these startup tasks tend to require other DLLs, typically already loaded but we still have to get access to them in this process. It all adds up.

Am I happy with a number like 59.62%? Well no, it's my job to make that number bigger (i.e. reduce the overhead) but this is a fairly small application... the whole run will be just over 100ms so basically the (fairly) fixed cost of starting up the runtime is significant in this case. It would be less daunting if the program did more.

Note to reviewers of managed systems: If you want to make a framework -- any framework -- look bad, simply ask it to do nothing. It turns out pretty much all known frameworks are lousy at doing nothing. So "hello world" for instance is a great way to make a framework look like a pig.

Still there's room for improvement there which is why I have a job :)

Moving on, System.IO.StreamReader.ReadLine() is at 28% which may not look too high, after all this program is primarily a parser. However, looking closer we can see that the hefty work, decoding code page 950, is in DBCSCodePageEncoding.GetChars() and that's only at 15.39%. It's not unhealthy to see a number like 15% spent doing one of the nice meaty jobs at hand but how did we go from 15% to decode the text all the way to 28% to decode the text and break it into lines? That seems excessive... maybe we could do something about that.

Looking further down NS.Test.DictionaryEntry.Parse() is up at 26% compared to the 28% for System.IO.StreamReader.ReadLine(). Now what's up there... it's just some simple scanning and string allocation. The string allocation should be the main cost yet it's only 14.42% of the time in Substring of which a mere 6.73%. Plus the IndexOfChar worker, which should be doing most of the searching, is only at 5.77%. Well there is a good bit of control flow and function call setup in the Parse function. That's probably a good chunkof it right there.

Still, the overall picture is pretty good. No exotic critical sections, no weird interlocked operations, very little time in the allocator. You can see why we posted a great out-of-the-gate number like 124ms. Even with the CLR's fixed overhead we got great value out of the allocator and the input mapping services.

But we can make it better still :)

Three techniques for tracking down memory leaks due to undisposed objects

ricom — Fri, 15 Apr 2005 21:37:00 GMT

People often ask me for tips/tricks on how to find out which objects are not being properly disposed. But before I go any further I should talk briefly about why that's a problem.

If an object is Finalizable that means that the runtime must run the Finalize method on that object (in C# you use the ~Class syntax to define the finalizer) before the memory can be reclaimed. The mechanism for doing this is fairly straightforward -- whenever an object that is finalizable first becomes unreachable it is put on a queue of objects needing finalization. Naturally we only discover that the object is unreachable during a garbage collection, and since we need to keep the object alive to finalize it the object must survive that collection. When the finalizer runs the object will be again unreachable and on the next collection the memory will be recovered as usual. See Garbage Collector Basics and Performance Hints and Two Things to Avoid for Better Memory Usage for more details.

Now the problem with all this is that it means every Finalizable object must survive at least one collection and so they are more likely to be promoted into older generations where getting rid of them is more expensive. Furthermore, collections of those older generations is less frequent so the memory for the finalizable object may not be reclaimed as quickly as we might like even if the finalizer is run comparatively soon.

To avoid these problems, every finalizable object should also be IDisposable (and most are). If an object is IDisposable it means that when you are done with it you can call the Dispose() method which releases any unmanaged resources associated with the object and tells the runtime that the object no longer needs to be finalized. As a consequence the resources are returned to the system as quickly as possible and the object doesn't need to survive for finalization -- it dies nice and quick like a regular object.

With that background I can get back on topic: Sometimes despite our best efforts we forget to call the Dispose method on every Disposable object. When that happens those objects go through finalization. If that happens a lot we can get Mid-Life Crisis.

So, how do we detect that this is happening? And if it is happening how do we track down the source of the leak? I have some approaches for now and some hints as to how we'll do this in the future.

Technique #1

If you want to know if finalization is a problem any regular profiler can give you a good hint. Using CLR Profiler for instance you can run your application then use the Call Tree view and select the various tabs until you find the one for the finalizer thread. It's very obvious which one that is because all the method names will be things like "SomeClass::Finalize". A lot of times just knowing which classes are being finalized a lot is enough to fix the problem.

Technique #2

If you see a lot of finalization but the finalizer is associated with a class or classes for which there are many instances and you want to know which instance is the bad one there is more you can do. If the class is one you control you can call store a stacktrace in the object when it is created with "StackTrace(Thread targetThread, bool needFileInfo)" (note this is a heavyweight thing to do, not something you want to ship with) then when the object is finalized you can dump that stack trace so that you will see what the stacktrace created the object and you can target that trace for elimination. To keep the cost down you might take a stacktrace at random only 1/1000th of the time and you could still get enough samples of the bad pattern to identify it over a long run.

Technique #3

If you don't own the class then its harder but you still might be able to make some headway. The first thing you need is a durable way to identify a particular instance of a particular class -- you can't use an object pointer for this because the object can move over time but for many classes (almost all reference types) you can hash the class using GetHashCode() and the hash doesn't change over the life of the object. Hash the objects you care about as soon as you can and capture the best callstack you can to finger their creator (see above). Next use your favorite debugger to stop in the Finalizer for the classes in question -- once you are stopped there dump the object's hash code in the debugger -- you should be able to put a call to GetHashCode() in the debugger watch window for instance. You can use that hash code to find the callstack that created the object. If the callstack is uninteresting or too expensive you can use any other friendly description of the callsite that will help you diagnose the originating location.

Looking forward:

The profiling API (the one that CLR Profiler uses for instance) in V1.0 and V1.1 provides almost enough information to do this job more automatically. In V2.0 (Whidbey) we're doing more:

ICorProfilerCallback2 adds (among other things) this little gem:

        virtual HRESULT STDMETHODCALLTYPE FinalizeableObjectQueued(
            /* [in] */ DWORD finalizerFlags,
            /* [in] */ ObjectID objectID) = 0;

This fellow tells profilers directly when an object is queued for finalization. The object allocation notifications already tell whenever an object is allocated and of course you can get the stack (CLR Profiler does this today). Using the notifications for allocations and for object motion you can easily track the object pointer over its lifetime and when it becomes finalized you can identify the stack that created it. This one function was the missing piece.   But even without it, the three techniques above often give good results.

Tracking down managed memory leaks (how to find a GC leak)

ricom — Fri, 10 Dec 2004 18:02:00 GMT

If you think you've got memory leaks, or if you're just wondering what kind of stuff is on your heap you can follow the very same steps that I do and get fabulous results your friends will envy. OK, well maybe not, but they're handy anyway.

These steps will help you to go from a suspected memory leak to the specific object references that are keeping your objects alive. See the Resources at the end for the location of all these tools.

Step 1: Run your process and put it in the state you are curious about

Be sure to choose a scenario you can reproduce if that's at all possible, otherwise you'll never know if you're making headway in clearing out the memory leaks.

Step 2: Use tasklist to find its process ID

C:\>tasklist

Image Name                   PID Session Name     Session#    Mem Usage
========================= ====== ================ ======== ============
System Idle Process            0 RDP-Tcp#9               0         16 K
System                         4 RDP-Tcp#9               0        112 K
smss.exe                     624 RDP-Tcp#9               0        252 K
...etc...
ShowFormComplex.exe         4496 RDP-Tcp#9               0     20,708 K
tasklist.exe                3636 RDP-Tcp#9               0      4,180 K

From here we can see that my process is ID #4496

Step 3: Use VADump to get a summary of the process

C:\>vadump -sop 4496
Category                        Total        Private Shareable    Shared
                           Pages    KBytes    KBytes    KBytes    KBytes
      Page Table Pages        35       140       140         0         0
      Other System            15        60        60         0         0
      Code/StaticData       4596     18384      4020      3376     10988
      Heap                   215       860       860         0         0
      Stack                   30       120       120         0         0
      Teb                      4        16        16         0         0
      Mapped Data            129       516         0        24       492
      Other Data             157       628       624         4         0

      Total Modules         4596     18384      4020      3376     10988
      Total Dynamic Data     535      2140      1620        28       492
      Total System            50       200       200         0         0
Grand Total Working Set     5181     20724      5840      3404     11480

Here we can see that the process is mostly code (18384k)

The vast majority of the resources that the CLR uses are under "Other Data" -- this is because the GC Heap is directly allocated with VirtualAlloc -- it doesn't go through a regular windows heap. And same for the so-called "loader heaps" which hold type information and jitted code. Most of the conventional "Heap" allocations are from whatever unmanaged is running. In this case it's a winform application with piles of controls so there's storage associated with those things.

There isn't much "Other Data" here so the heap situation is probably pretty good but let's see where we stand on detailed CLR memory usage.

Step 4: Attach Windbg and load SOS

C:\> windbg -p 4496

Once the debugger loads use this command to load our extension DLL

0:004> .loadby sos mscorwks

This tells the debugger to load the extension "sos.dll" from the same place that mscorwks.dll was loaded. That ensures that you get the right version of SOS (it should be the one that matches the mscorwks you are using)

Step 5: Get the CLR memory summary

This command gives you a summary of what we've allocated. The output will be a little different depending on the version of the runtime you are using. But for a simple application you get the loader heaps for the two base domain structures (they just hold objects that can be shared in assorted ways) plus the storage for the first real appdomain (Domain 1). And of course the jitted code.

0:004> !EEHeap
Loader Heap:
--------------------------------------
System Domain: 5e093770
...etc...
Total size: 0x8000(32768)bytes
--------------------------------------
Shared Domain: 5e093fa8
...etc...
Total size: 0xa000(40960)bytes
--------------------------------------
Domain 1: 14f0d0
...etc...
Total size: 0x18000(98304)bytes
--------------------------------------
Jit code heap:
LoaderCodeHeap: 02ef0000(10000:7000) Size: 0x7000(28672)bytes.
Total size: 0x7000(28672)bytes
--------------------------------------
Module Thunk heaps:
...etc...
Total size: 0x0(0)bytes
--------------------------------------
Module Lookup Table heaps:
...etc...
Total size: 0x0(0)bytes
--------------------------------------
Total LoaderHeap size: 0x31000(200704)bytes

So here we've got 200k of stuff associated with what has been loaded, about 28k of which is jitted code.

Next in the output (same command) is the summary of the GC Heap.

=======================================
Number of GC Heaps: 1
generation 0 starts at 0x00a61018
generation 1 starts at 0x00a6100c
generation 2 starts at 0x00a61000
ephemeral segment allocation context: none
segment    begin allocated     size
001b8630 7a8d0bbc 7a8f08d8 0x0001fd1c(130332)
001b4ac8 7b4f77e0 7b50dcc8 0x000164e8(91368)
00157690 02c10004 02c10010 0x0000000c(12)
00157610 5ba35728 5ba7c4a0 0x00046d78(290168)
00a60000 00a61000 00aac000 0x0004b000(307200)
Large object heap starts at 0x01a61000
segment    begin allocated     size
01a60000 01a61000 01a66d90 0x00005d90(23952)
Total Size   0xcdd18(843032)
------------------------------
GC Heap Size   0xcdd18(843032)

You will likely have many fewer small-heap segments than I did because I did this test on an internal debug build so there's funny things like a 12 byte segment in the dump. But you'll see what segements there are and how big they are and you can see what the current boundaries are on the generations from which you can compute thier current exact size. (Note that this is likely different than what the performance counters report as you can see in Maoni's blog -- those counters are budgeted from the last GC not the instanteous value -- it would be too expensive to keep updating the instantaneous value)

So in this case we can see that there is about 843k of GC heap. Comparing that to the other data category there was about 2M total of other data. The CLR accounts for about 1M of that. The rest is likely bitmaps allocated from my winforms application's controls but whatever it is, it isn't CLR stuff...

Step 5: Dump the GC Heap statistics

Next we'll want to know, by type, what's on the heap at this exact instant

0:004> !DumpHeap -stat
... sorted from smallest to biggest ... etc. etc...

7b586c7c      436     10464 System.Internal.Gdi.WindowsGraphics
5ba867ac      208     11648 System.Reflection.RuntimeMethodInfo
7b586898      627     12540 System.Internal.Gdi.DeviceContext
5baa4954      677     39992 System.Object[]
5ba25c9c     8593    561496 System.String
Total 17427 objects

Note that this dump includes both reachable and unreachable objects so unless you know that the GC just ran before you did this command you'll see some dead stuff in this report as well. Sometimes its interesting and useful to force a GC to run before you do this so that you can get a summary of just the live stuff. Sometimes it's useful to do dumps before and after forcing a GC so that you can see what sort of things are dying. This may be a way to gather evidence that a forced GC is necessary. See my blog on When to Call GC.Collect().

So let's suppose that there weren't supposed to be 208 System.Reflection.RuntimeMethodInfo objects allocated here and that we thought that was a leak. One of the things we'll want to do is to use CLR Profiler to see where those objects are being allocated -- that will give us half the picture. But we can get the other half of the picture right here in the debugger.

Step 6: Dump Type Specific Information

We can dump each object whose type name includes a given string with a simple command

0:004> !DumpHeap -type System.Reflection.RuntimeMethodInfo
Address       MT     Size
00a63da4 5baa62c0       32
00a63e04 5baa6174       20
00a63e2c 5ba867ac       56
00a63e64 5baa5fa8       16
00a63e88 5baa5fa8       16
00a63f24 5baa6174       20
00a63f4c 5ba867ac       56
00a63f84 5baa5fa8       16
etc. etc. etc.
total 630 objects
Statistics:
      MT    Count TotalSize Class Name
5baa62c0        3        96 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeMethodInfo, mscorlib]]
5baa5fa8      211      3376 System.Reflection.CerArrayList`1[[System.Reflection.RuntimeMethodInfo, mscorlib]]
5baa6174      208      4160 System.Collections.Generic.List`1[[System.Reflection.RuntimeMethodInfo, mscorlib]]
5ba867ac      208     11648 System.Reflection.RuntimeMethodInfo
Total 630 objects

Note that the type we wanted was System.Reflection.RuntimeMethodInfo and we can see that it has a method table 5ba867ac. Those are the 56 byte objects. Now we can investigate some of these and see what is causing them to stay alive.

Step 7: Identify the roots of suspected leaks

One of the lines in the dump was

00a63e2c 5ba867ac 56

So that tells us there is an object of the type we want at address 00a63e2c. Let's see what's keeping it alive

0:004> !gcroot 00a63e2c
Scan Thread 0 OSTHread 1598
Scan Thread 2 OSTHread 103c

DOMAIN(0014F0D0):
HANDLE(WeakLn):3f10f0:
Root:00a63d20(System.RuntimeType+RuntimeTypeCache)
->00a63da4(System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeMethodInfo,mscorlib]])
->00a63e88(System.Reflection.CerArrayList`1[[System.Reflection.RuntimeMethodInfo, mscorlib]])
->00a63e98(System.Object[])
->00a63e2c(System.Reflection.RuntimeMethodInfo)

DOMAIN(0014F0D0):
HANDLE(Pinned):3f13ec:
Root:01a64b50(System.Object[])
->00a62f20(System.ComponentModel.WeakEventHandlerList)
->00a63fb4(System.ComponentModel.WeakEventHandlerList+ListEntry)
->00a63ec4(System.ComponentModel.WeakEventHandlerList+ListEntry)
->00aa5f6c(System.ComponentModel.WeakDelegateHolder)
->00a63e2c(System.Reflection.RuntimeMethodInfo)

I've added some extra line breaks to the output above to make it easier to read but otherwise it's the raw output.

The gcroot command is trying to tell you if the object is reachable and if so how it is reached from each root. The dump won't include all the ways the object is reachable but you do get at least one way to find the object -- usually that's enough. If multiple paths are dumped they often have a common tail. However the object is reachable (here it looks like maybe only weak references are left so this guy might go away on the next collect) that should give you a hint about (some of) the remaining references. From there you can decide what pointers to null so that the object is properly released.

Resources

You can get information on windbg at this location.
Vadump has usage information and a download from msdn and microsoft.com respectively.

If those links break, searching for windbg and vadump on the microsoft.com home page gave good results, that's how I got those links in the first place.

CLR Profiler is available here.

It comes with documentation but there is additional material available in the Performance PAG in Chapter 13.