Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

J. Marsch — Fri, 10 Dec 2004 19:39:00 GMT

Great post, Rico. I would love to know more about doing this type of low-level debugging. My current experience is limited to source code level debuggers.

Also, I'm a little bit curious about that last dump in your post -- the output of the gcroot command.

Does this imply that there will be events/delegates in .Net 2.0 that use weak refs?

re: Tracking down managed memory leaks (how to find a GC leak)

Rico Mariani — Fri, 10 Dec 2004 20:06:00 GMT

You know I'm often surprised by what I see when I dump the heap. This guy System.ComponentModel.WeakEventHandlerList seems like he could be quite interesting and I don't know thing one about him. It might be generally useful but it might also be a private type with an unfortunate name. Even internal types appear in the low level dumps like this.

I'll see if I can't find out something for my own curiosity if nothing else.

re: Tracking down managed memory leaks (how to find a GC leak)

Raymond Lewallen — Fri, 10 Dec 2004 20:26:00 GMT

Excellent post Rico. What I would like to see someone work on is dumping this information into a file directly from CLRProfiler. That way you can view the dump and the heap and object allocations knowing they all are generated under the same circumstances.

re: Tracking down managed memory leaks (how to find a GC leak)

Pavel Lebedinsky — Fri, 10 Dec 2004 20:31:00 GMT

For 1.1 it's probably better to use sos.dll that comes with the latest debuggers. It's loaded automatically when you attach to a process that has CLR dlls loaded, and it has some fixes and new commands/shortcuts that the original Everett version of sos doesn't.

re: Tracking down managed memory leaks (how to find a GC leak)

Rico Mariani — Fri, 10 Dec 2004 20:45:00 GMT

>That way you can view the dump and the heap and object allocations knowing they all are generated under the same circumstances.

It is often very useful to start the process under CLRProfiler and then also attach with the debugger so you can do both on the same process. This works just fine. In fact it's extra handy because CLR Profiler's "Dump Heap Now" forces a garbage collection so you can use it to see what dead things are going away in the debugger dump and visually in CLRProfiler.

>>For 1.1 it's probably better to use sos.dll that comes with the latest debuggers. It's loaded automatically when you attach to a process that has CLR dlls loaded, and it has some fixes and new commands/shortcuts that the original Everett version of sos doesn't.

Nothing is ever easy :)

It turns out there is this *other* thing that is also called SOS which isn't quite the same thing as the SOS that we build along with the runtime even though it has many of the same commands and common heritage. That one seems to be auto-loaded (you can find it in a subdirectory under wherever you install windbg) and I think it works on v1.0 and v1.1 of the runtime. I think it has a few features not present in the original SOS we deployed so it can be useful.

It may be that there will be an enhanced SOS for version 2.0 of the runtime some time after it ships. So basically you can try the commands without loading an SOS explicitly and just get whatever is there or you can go with the "golden" version and take what was originally shipped.

For myself, I always use .loadby sos mscorwks but of course I use the runtime build of the moment so anything else would be lunacy. For you, gentle reader, you may find that you like some of the enchanced features in the other SOS and it works fine on your runtime.

Either way the leak tracking instructions are the same.

re: Tracking down managed memory leaks (how to find a GC leak)

Ivan Peev — Sat, 11 Dec 2004 18:46:00 GMT

Rico,

Why the VS team doesn't include some functionality, which can automate or help tracing those kinds of managed memory leaks? Those steps can be automated, can't they ?

Tracking down managed memory leaks (how to find a GC leak) (from: Rico Mariani)

TOURNEY LOGIC LINK BLOG — Sun, 12 Dec 2004 07:50:00 GMT

Think the .Net Garbage Collector Isn't Doing It's Job?

protected virtual void jayBlog { — Sun, 12 Dec 2004 12:12:00 GMT

How to track down managed memory leak.

SmartWombat Blog — Mon, 13 Dec 2004 02:11:00 GMT

Blog link of the week 50

Daniel Moth — Mon, 13 Dec 2004 02:20:00 GMT

Blog link of the week 50

re: Tracking down managed memory leaks (how to find a GC leak)

Rico Mariani — Mon, 13 Dec 2004 22:18:00 GMT

Ivan Peev asks: "Why the VS team doesn't include some functionality, which can automate or help tracing those kinds of managed memory leaks? Those steps can be automated, can't they?"

There was very little in terms of memory analysis features in Visual Studio.NET -- I think that reflects two things: first that there were lots of problems to solve and we couldn't solve them all in one release and second is that some problems we didn't really know how to solve anyway. I think some of both is going on in this case.

Which brings me to the second point: Can this all be automated? Well, sort of, the tricky bits are in Step 5 -- how do you automatically know which types are the ones that *should* have gone away and which types were supposed to be living (because they are in a cache or something) -- and in Step 7 -- how do you automatically know which instances of the type are the problematic ones?

It's rather tricky.

Tracking down managed memory leaks (how to find a GC leak)

Eric Gunnerson's C# Compendium — Tue, 14 Dec 2004 22:53:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

rx — Tue, 14 Dec 2004 21:37:00 GMT

this is a good post.. thanks

Tracking down managed memory leaks (how to find a GC leak) in 7 steps

.Net Adventures — Wed, 15 Dec 2004 00:50:00 GMT

How to track down managed memory leaks - by Rico Mariani.

notgartner.com: Mitch Denny's Blog — Wed, 15 Dec 2004 12:42:00 GMT

Tracking down managed memory leaks (how to find a GC leak)

Anna Shchurova's Blog — Wed, 15 Dec 2004 21:39:00 GMT

Managed Memory Leaks

ThoughtChain — Thu, 16 Dec 2004 13:50:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Tim Bond — Thu, 16 Dec 2004 14:19:00 GMT

Great blog well worth bookmarking.

re: Tracking down managed memory leaks (how to find a GC leak)

Mark Levison — Thu, 16 Dec 2004 18:02:00 GMT

Great post - but why not use a tool do all this hardwork? During our release cycle this fall I found MemProfiler (www.scitech.se/memprofiler/ ) and blogged a bit about my experience (http://dotnetjunkies.com/WebLog/mlevison/archive/2004/09/30/27265.aspx).

For $100, I'm too lazy too work as hard as you.

.net Garbage Collector Tips and Tricks

Tony's Blog — Fri, 17 Dec 2004 06:40:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Ollie Riches — Fri, 17 Dec 2004 10:42:00 GMT

Great article will make a great set of interview questions :)

re: Tracking down managed memory leaks (how to find a GC leak)

Ollie Riches — Fri, 17 Dec 2004 10:48:00 GMT

Your blog rocks....

Fixing Managed Memory Leaks

Deep Thoughts... — Sat, 18 Dec 2004 02:39:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Corbin Dunn — Sat, 18 Dec 2004 00:42:00 GMT

I've found the CLRProfiler provides the same information, but in a much easier to use interface.

Just my 2 cents!

re: Tracking down managed memory leaks (how to find a GC leak)

Rico Mariani — Sat, 18 Dec 2004 00:48:00 GMT

You *can* get similar/related information from CLRProfiler and even better information from some 3rd party tools.

Advantages to the approach given above:

1) You can do this after the fact if you witness a problem, you don't have to start under the profiler as you can attach the debugger

2) You can get per-object information about objects and their reachability (!gcroot) which is much trickier to get from say the heap dump

3) You can get valuable summary information about the overall memory usage of the CLR (!EEHeap)

Plus it's all free :)

.NET User interface for Tracking down managed memory leaks

Wah On Terra .NET: Martin Granell's Blog — Mon, 20 Dec 2004 02:05:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Martin Granell — Sun, 19 Dec 2004 23:13:00 GMT

Hi Rico,
For all those who don't like to use command line tools...
I've written a small .NET application that drives the debugger, using the techniques in this article, and displays it in a user interface. The article and source (with screenshot) is on my blog. If you can think of any enhancements that would be useful, please drop me a comment.

.::Szőkelizer 113::.

RIO - Randektív Informatikai Oldal — Mon, 20 Dec 2004 17:54:00 GMT

<p><ul><li><a href="http://www.hriders.com/web_page.cfm?web_pageID=38" target="_blank">1 Tb mail account?</a></li><li><a href="http://mek.oszk.hu/01900/01984/html/" target="_blank&q

Tracking down managed memory leaks

Jackie Goldstein's Weblog — Thu, 23 Dec 2004 15:29:00 GMT

Tracking down managed memory leaks

Jackie Goldstein's Weblog — Thu, 23 Dec 2004 16:01:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Rico Mariani — Fri, 24 Dec 2004 01:02:00 GMT

The tool looks really neat. I'll give it a whirl when I'm back from vacation.

Happy Holidays everyone!

.net Garbage Collector Tips and Tricks

Tony's Blog — Tue, 28 Dec 2004 05:18:00 GMT

Tracking down managed memory leaks (ASP.NET)

James Alt — Tue, 28 Dec 2004 18:55:00 GMT

Will this work on the ASP.NET worker process? (aspnet_wp.exe) I've been having some memory issues where the size of the worker process just keeps growing and growing until I have to manually kill and restart the process before I can continue on with my work.

How to use WinDBG to track leaks in the managed heap

Teucer's Quiver — Wed, 29 Dec 2004 02:37:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Martin — Wed, 29 Dec 2004 09:42:00 GMT

http://www.scitech.se/memprofiler/

.Net Performance Management

ISkydive — Wed, 29 Dec 2004 18:22:00 GMT

OdeToCode Links For Dec 30

OdeToCode Link Blog — Thu, 30 Dec 2004 18:00:00 GMT

re: Tracking down managed memory leaks (how to find a GC leak)

Seth Hodgson — Wed, 05 Jan 2005 18:27:00 GMT

Thanks for the excellent article Rico. I have one (somewhat related) question about GC memory management/leaks and the CLR. Beyond the steps you outline to identify memory leaks, is there any way to control the min and max managed heap size used by the CLR? Most JVMs allow a min and max managed heap size to be specified as start-up params, and I haven't seen any mention of something similar for the CLR. I'd ideally like to be able to tell a .NET windows service app I'm developing a maximum allowed heap size, and then let the GC do it's thing within that constraint. Any thoughts?

re: Tracking down managed memory leaks (how to find a GC leak)

Rico Mariani — Wed, 05 Jan 2005 18:48:00 GMT

Some quick responses sort of in order:

Q: Will this work on the ASP.NET worker process?

A: I don't see why it wouldn't. It's not magic or anything, and you can attach to it with the debugger same as any other.

Re: http://www.scitech.se/memprofiler/

It looks pretty cool, I'll have to play with it some more. I wonder if it works on the daily build :)

Q: Is there any way to control the min and max managed heap size used by the CLR?

I don't think there are environment variables for that but you can do this and more with the hosting api (the CLR calls you to get memory and so forth so that it can be hosted in more exotic processes like say SQL Server where you don't want us to go and get memory directly)

http://www.gotdotnet.com/team/clr/about_clr_Hosting.aspx

re: Tracking down managed memory leaks (how to find a GC leak)

Seth Hodgson — Fri, 07 Jan 2005 00:56:00 GMT

Hi Rico - thanks for the pointer to the CLR Hosting articles. From what I can tell, the hosting APIs don't provide a way to limit the amount of memory used by the CLR beyond ICorRuntimeHost's Start() and Stop() methods. That seems like a strange way to manage CLR resource use - hard stopping it which unloads it from the current process and then restarting it in a new process. Is stopping the CLR the only way to release resources back to the system, and is the CLR team considering any enhancement to the CLR startup shim to allow the min/max managed heap size to be defined at application start?

Thanks again for all your excellent posts over this past year.

re: Tracking down managed memory leaks (how to find a GC leak)

Seth Hodgson — Fri, 07 Jan 2005 01:19:00 GMT

Scratch that - I had overlooked ICorConfiguration 's SetGCHostControl method:)

Tracking down managed memory leaks

Musings about Dot Net — Fri, 07 Jan 2005 16:33:00 GMT

Tracking down managed memory leaks

Musings about Dot Net — Fri, 07 Jan 2005 16:59:00 GMT

Tracking down managed memory leaks (how to find a GC leak) in 7 steps

.Net Adventures — Sun, 09 Jan 2005 10:49:00 GMT

Tracking down managed memory leaks (how to find a GC leak)

Petes Web Log — Thu, 24 Feb 2005 23:26:00 GMT

Tracking down managed memory leaks (how to find a GC leak)

A number of resource for locating GC leaks:
You might find this blog entry worth reading:
http://weblogs.asp.net/ricom/archive/2004/12/10/279612.aspx

SciTek's...

How to track down managed memory leak.

Smartwombat Blog — Mon, 28 Mar 2005 00:38:00 GMT

How to track managed memory leak, also how to use windbg and sos extension for managed debugging in...

It was the best of code. It was the worst of code. » Garbage collection

It was the best of code. It was the worst of code. » Garbage collection — Wed, 22 Feb 2006 13:19:35 GMT

PingBack from http://blog.actapps.com.au/2006-02-22/garbage-collection/

First rate posting on mem analysis

Rico Mariani's Performance Tidbits — Tue, 04 Apr 2006 04:50:37 GMT

I was just going through some memory leak information and I stumbled across a newish posting from Tess:...

The undetectebility of C++/CLI gcroot<> template references.

Cum Grano Salis — Thu, 13 Apr 2006 22:48:03 GMT

In recent builds, we have been having an awful memory leak in our system. Silvio was debugging it and...

The undetectebility of C++/CLI gcroot<> template references.

Cum Grano Salis — Thu, 13 Apr 2006 22:48:31 GMT

In recent builds, we have been having an awful memory leak in our system. Silvio was debugging it and...

Automated leak detection in managed code

Dan Crevier's Blog — Sun, 07 Jan 2007 07:44:47 GMT

Managed code makes memory management much easier, but it's still possible to have unintended memory leaks.

Memory leaks 101: Objects anchored by event generators

Rico Mariani's Performance Tidbits — Tue, 09 Jan 2007 22:20:43 GMT

This problem actually comes up pretty often so I thought I'd write a little article about it, and a couple

Track down managed memory leaks

DotNetKicks.com — Sun, 11 Feb 2007 04:08:56 GMT

You've been kicked (a good thing) - Trackback from DotNetKicks.com

File Open Performance -- Beware of "Extensions"

Rico Mariani's Performance Tidbits — Sun, 04 Nov 2007 07:07:49 GMT

Here is a little interchange I had a few days ago, "Nick From Chicago" graciously allowed me to share

MSDN Blog Postings » File Open Performance -- Beware of 'Extensions'

MSDN Blog Postings » File Open Performance -- Beware of 'Extensions' — Sun, 04 Nov 2007 07:20:53 GMT

PingBack from http://msdnrss.thecoderblogs.com/2007/11/04/file-open-performance-beware-of-extensions/

MSDN Blog Postings » File Open Performance -- Beware of 'Extensions'

MSDN Blog Postings » File Open Performance -- Beware of 'Extensions' — Sun, 04 Nov 2007 07:20:54 GMT

PingBack from http://msdnrss.thecoderblogs.com/2007/11/04/file-open-performance-beware-of-extensions/

File Open Performance -- Beware of 'Extensions'

Noticias externas — Sun, 04 Nov 2007 07:45:33 GMT

Here is a little interchange I had a few days ago; "Nick From Chicago" graciously allowed me

Finding Memory Leaks in WPF-based applications

WPF Performance — Tue, 05 Feb 2008 01:23:51 GMT

  There are numbers of blogs that folks wrote about memory leaks in Microsoft .Net Framework managed

Finding Memory Leaks in WPF-based applications

Noticias externas — Tue, 05 Feb 2008 01:57:26 GMT

There are numbers of blogs that folks wrote about memory leaks in Microsoft .Net Framework managed code

MSDN Blog Postings » Finding Memory Leaks in WPF-based applications

MSDN Blog Postings » Finding Memory Leaks in WPF-based applications — Tue, 05 Feb 2008 04:31:17 GMT

PingBack from http://msdnrss.thecoderblogs.com/2008/02/04/finding-memory-leaks-in-wpf-based-applications/

Debugging .NET exceptions without VS installed using WinDbg » Thursday Night

Sat, 24 May 2008 21:46:02 GMT

PingBack from http://blog.paulbetts.org/index.php/2008/05/24/debugging-net-exceptions-without-vs-installed-using-windbg/

Asp.net memory leak : case study - Asp.net fuite mémoire

Blog de Jérémie Clabaut (ak MinSou) — Wed, 04 Jun 2008 14:35:36 GMT

Last week one of my customers called me to help him resolve a big problem on an asp.net application: a memory leak. During the application stress test, the w3wp process memory increased abnormally as a result of an application crash (application pool

Tracking down managed memory leaks (how to find a GC leak) in 7 steps | devintelligence.com

Sun, 20 Jul 2008 16:46:15 GMT

PingBack from http://www19.a2hosting.com/~tarasn/devintelligence.com/?p=597

Tracking down managed memory leaks (how to find a GC leak)

IWebThereforeIAm — Sat, 11 Oct 2008 14:48:43 GMT

Tracking down managed memory leaks (how to find a GC leak)...

Tracking Down Managed Memory Leaks « Roman’s Blog

Tracking Down Managed Memory Leaks « Roman’s Blog — Sat, 18 Oct 2008 01:51:54 GMT

PingBack from http://rhnatiuk.wordpress.com/2008/10/18/tracking-down-managed-memory-leaks/

Memory Leak | keyongtech

Memory Leak | keyongtech — Sun, 18 Jan 2009 20:17:49 GMT

PingBack from http://www.keyongtech.com/683612-memory-leak

Speicherverbrauch von NET-Anwendungen | hilpers

Speicherverbrauch von NET-Anwendungen | hilpers — Tue, 20 Jan 2009 18:31:50 GMT

PingBack from http://www.hilpers.com/1040753-speicherverbrauch-von-net-anwendungen

rusanu.com » Blog Archive » CLR Memory Leak

rusanu.com » Blog Archive » CLR Memory Leak — Sun, 22 Feb 2009 12:49:58 GMT

PingBack from http://rusanu.com/2009/01/19/clr-memory-leak/

Where's your leak at? [Using WinDbg, SOS, and GCRoot to diagnose a .NET memory leak]

Delay's Blog — Thu, 12 Mar 2009 08:48:18 GMT

In my last post, I explained how it was possible for "hidden" event handlers to introduce memory leaks

Finding Managed Memory Leaks in WPF and Silverlight applications

WPF, Silverlight and .NET Musings — Sun, 22 Mar 2009 02:12:08 GMT

It's been a while since the last post was online. We have been very busy in working on one of the very

Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

Tracking down managed memory leaks (how to find a GC leak) (from: Rico Mariani)

Think the .Net Garbage Collector Isn't Doing It's Job?

How to track down managed memory leak.

Blog link of the week 50

re: Tracking down managed memory leaks (how to find a GC leak)

Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

Tracking down managed memory leaks (how to find a GC leak) in 7 steps

How to track down managed memory leaks - by Rico Mariani.

Tracking down managed memory leaks (how to find a GC leak)

Managed Memory Leaks

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

.net Garbage Collector Tips and Tricks

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

Fixing Managed Memory Leaks

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

.NET User interface for Tracking down managed memory leaks

re: Tracking down managed memory leaks (how to find a GC leak)

.::Szőkelizer 113::.

Tracking down managed memory leaks

Tracking down managed memory leaks

re: Tracking down managed memory leaks (how to find a GC leak)

.net Garbage Collector Tips and Tricks

Tracking down managed memory leaks (ASP.NET)

How to use WinDBG to track leaks in the managed heap

re: Tracking down managed memory leaks (how to find a GC leak)

.Net Performance Management

OdeToCode Links For Dec 30

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

re: Tracking down managed memory leaks (how to find a GC leak)

Tracking down managed memory leaks

Tracking down managed memory leaks

Tracking down managed memory leaks (how to find a GC leak) in 7 steps

Tracking down managed memory leaks (how to find a GC leak)

How to track down managed memory leak.

It was the best of code. It was the worst of code. » Garbage collection

First rate posting on mem analysis

The undetectebility of C++/CLI gcroot&lt;&gt; template references.

The undetectebility of C++/CLI gcroot&lt;&gt; template references.

Automated leak detection in managed code

Memory leaks 101: Objects anchored by event generators

Track down managed memory leaks

File Open Performance -- Beware of "Extensions"

MSDN Blog Postings » File Open Performance -- Beware of 'Extensions'

MSDN Blog Postings » File Open Performance -- Beware of 'Extensions'

File Open Performance -- Beware of 'Extensions'

Finding Memory Leaks in WPF-based applications

Finding Memory Leaks in WPF-based applications

MSDN Blog Postings » Finding Memory Leaks in WPF-based applications

Debugging .NET exceptions *without* VS installed using WinDbg » Thursday Night

Asp.net memory leak : case study - Asp.net fuite mémoire

Tracking down managed memory leaks (how to find a GC leak) in 7 steps | devintelligence.com

Tracking down managed memory leaks (how to find a GC leak)

Tracking Down Managed Memory Leaks « Roman’s Blog

Memory Leak | keyongtech

Speicherverbrauch von NET-Anwendungen | hilpers

rusanu.com » Blog Archive » CLR Memory Leak

Where's your leak at? [Using WinDbg, SOS, and GCRoot to diagnose a .NET memory leak]

Finding Managed Memory Leaks in WPF and Silverlight applications

The undetectebility of C++/CLI gcroot<> template references.

The undetectebility of C++/CLI gcroot<> template references.

Debugging .NET exceptions without VS installed using WinDbg » Thursday Night