.NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

snaveen — Mon, 23 Jan 2006 16:24:45 GMT

I know that the !dumpheap has a "–short " parameter that can be used within .foreach in the windbg. Is there any other command that can be used with in .foreach like !dumpheap?

Example

.foreach (obj { !dh -type System.String -gen 2 -short }) { !gcroot ${obj} }

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Mon, 23 Jan 2006 17:01:58 GMT

Many of the sos commands have a -short param. !dumpaspnetcache for example has a -short parameter.

Even if they dont you can often use the .foreach command like above by skipping tokens with /pS and /ps. I'm writing a blog post on automating debugging with .foreach now but your example above is excellent usage of the .foreach command.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

snaveen — Mon, 23 Jan 2006 17:19:54 GMT

Cool. How about some posts on debugging from managed code to unmanaged code? Like GCHandles, Pinning of memory, COM Callable Wrappers? I guess there aren’t many posts of this using windbg.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Mon, 23 Jan 2006 18:16:16 GMT

Sure thing, now on my todo list:) anything specific like memory issues or just how it works in general?

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

snaveen — Mon, 23 Jan 2006 18:45:56 GMT

I guess both.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Jason Haley — Mon, 23 Jan 2006 20:46:36 GMT

Excellent post!

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Orand — Tue, 24 Jan 2006 07:10:24 GMT

I used the exact technique you described a few months ago to track down a WinForms memory leak due to subscribing to the static Application.Idle event. They really need to clearly spell out in the documentation the consequences of not unsubscribing from a static event from the framework. I'm sure this bites the majority of users who subscribe to this event.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Tue, 24 Jan 2006 11:47:02 GMT

You're absolutely right, it does affect a lot of people... enough for me to think it was worthy of a post:)

There is a "send a comment on this topic" link at the bottom of each msdn entry. If you find something like this that you feel that there would be a benefit to have in the msdn docs. I urge you to use this comment feature. It's a great way to make sure that others do not fall into the same traps.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tom Harris — Wed, 25 Jan 2006 15:23:38 GMT

Hi there,

I just wanted to let you guys know that we have a dedicated .NET memory profiling tool that can really help you with these sort of problems. For the scenario described, you would simply point our tool at the web application and take 2 snapshots. The tool will then show you what new objects have been created. Finding the offending leak is straightforard as the tool shows which objects reference which other objects. It will even show you the exact source code for when the offending EventHandler object is created. I will write a technical article for Red Gate outlining the procedure in more detail. In the meantime, do try out ANTS Profiler next time you are concerned about memory issues in .NET applications.

Many thanks,

Tom Harris

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

dblack — Sat, 11 Mar 2006 20:40:34 GMT

Another great read!

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Paul — Tue, 14 Mar 2006 01:03:08 GMT

"The moral of the story here is: be very careful if you are hooking up event handlers to objects that have a longer lifespan than the page."

Is there anyway to tell a form to remove all of its event handlers from any objects it is referenced by on disposal?

Why wouldn't you want this to occur by default? (Why doesn't the .NET framework do this for us by default?)

Thanks,

Paul

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Tue, 14 Mar 2006 09:38:49 GMT

Because the objects are never disposed. As long as something is holding on to the eventhandlers and pages etc. they are not ready for garbage collection.

The framework can't do this because we have explicitly told it not to, by holding on to the static object. IMHO it would be a bug if the framework randomly removed event handlers.

ASP.NET Quiz Answers: Does Page.Cache leak memory?

If broken it is, fix it you should — Fri, 11 Aug 2006 12:36:35 GMT

A few days ago I posted a question I had gotten on email (look here for complete post): &nbsp; "We use...

Event Handlers Gone Wild

Todd Carter's WebLog — Wed, 06 Sep 2006 00:10:57 GMT

Today I ran across an issue that my good buddy Tess hit with one of her customers&nbsp;a while back....

Quick things to check if you are leaking your memory

A developer's strayings — Mon, 25 Sep 2006 13:00:46 GMT

After a couple of years working in InternetDev support I've seen many different kind of problems reported...

Quick things to check if you are leaking your memory

A developer's strayings — Sat, 07 Oct 2006 00:48:53 GMT

After a couple of years working in InternetDev support I've seen many different kind of problems reported

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Kamran Shahid — Thu, 09 Nov 2006 08:47:47 GMT

Hi Tess.It is the Wonderful article.

I just want's to ask what should I do to debug windows mobile 5.0 application.It have both type of modules that is managed and unmanaged.I want's to log unhandled exception.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Thu, 09 Nov 2006 10:48:32 GMT

Thank you,

I don't believe there are any debuggers that will work on a mobile device, but I believe you should be able to debug the emulator process. I haven't tried it yet, but you peaked my interest:)

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

ght — Wed, 15 Nov 2006 13:08:36 GMT

0x0213b483 0x0213b488 None [DEFAULT] [hasThis] Void MemoryIssues.WebForm1.Button1_Click

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

MengSung — Wed, 13 Dec 2006 11:11:23 GMT

Thank you ess. But do you really fix the problem by unhook ? I don't know why my code just can not unhook anyway. Could you tell me what's wrong with my VB.NET code.

...

' Here I Hook

AddHandler m_MailItem.Open, New Microsoft.Office.Interop.Outlook.ItemEvents_10_OpenEventHandler(AddressOf m_MailItem_Open)

AddHandler m_MailItem.Close, New Microsoft.Office.Interop.Outlook.ItemEvents_10_CloseEventHandler(AddressOf m_MailItem_Close)

...

' Here I do Unhook

RemoveHandler m_MailItem.Open, New Microsoft.Office.Interop.Outlook.ItemEvents_10_OpenEventHandler(AddressOf m_MailItem_Open)

RemoveHandler m_MailItem.Close, New Microsoft.Office.Interop.Outlook.ItemEvents_10_CloseEventHandler(AddressOf m_MailItem_Close)

=========================

Windbg result

0:036> !gcroot 06766a14

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Scan Thread 0 OSTHread a34

Scan Thread 17 OSTHread d58

Scan Thread 18 OSTHread b08

Scan Thread 14 OSTHread 118

Scan Thread 12 OSTHread f64

DOMAIN(061835F8):HANDLE(WeakSh):8c513c0:Root:06766a14(ItemWrapper.ItemWrapper)

DOMAIN(061835F8):HANDLE(RefCnt):8c51484:Root:06845650(Microsoft.Office.Interop.Outlook.ItemEvents_10_SinkHelper)->

06845630(Microsoft.Office.Interop.Outlook.ItemEvents_10_CloseEventHandler)->

06766a14(ItemWrapper.ItemWrapper)

DOMAIN(061835F8):HANDLE(RefCnt):8c5148c:Root:06766ecc(Microsoft.Office.Interop.Outlook.ItemEvents_10_SinkHelper)->

06766a90(Microsoft.Office.Interop.Outlook.ItemEvents_10_OpenEventHandler)->

06766a14(ItemWrapper.ItemWrapper)

==========================

It works fine to hook the event but it has memory leak.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Wed, 13 Dec 2006 11:21:10 GMT

Hi MengSung,

Where do you hook and unhook them? Are you sure that your unhook code has run at this point? If it has you shouldn't see this...

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Deepak Kumar — Thu, 28 Dec 2006 23:54:30 GMT

Do we need to remove the references which says

HANDLE(WeakSh) or HANDLE(WeakLn).

Please reply asap. it is very urgent. thanks

ASP.NET Quiz Answers: Does Page.Cache leak memory?

If broken it is, fix it you should — Thu, 04 Jan 2007 13:05:48 GMT

A few days ago I posted a question I had gotten on email (look here for complete post): " We use Page.Cache

Event Handlers Gone Wild

Todd Carter's WebLog — Thu, 22 Feb 2007 17:44:36 GMT

Today I ran across an issue that my good buddy Tess hit with one of her customers a while back. The application

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Allen Owen — Tue, 10 Jul 2007 19:33:35 GMT

Tess, your blog has just saved me! I've been fighting a serious memory leak for the last 2 weeks, and have been faithfully following the techniques that you propose.

After much soul searching, and memory exploration, I found this private member variable set in one of my classes:

HttpContext _context = HttpContext.Current

Nuff said? It looks like this object was the root cause of all my trouble.

Anyway, thanks very much, couldn't have done it without your insight. By the way, the site, from being a 1.5 Gb memory hog in about 2 hours is now running very smoothly at about 129 000Kb simulating 1 200 users doing large financial transactions!

Cheers

Allen

ASP.NET Memory Investigation

If broken it is, fix it you should — Mon, 13 Aug 2007 17:54:33 GMT

This is a bit of a continuation of ASP.NET Memory Issue: High memory usage in a 64bit w3wp.exe process

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

vikram nayak — Wed, 31 Oct 2007 06:26:25 GMT

excellent article. thanks

ASP.NET Memory: Thou shalt not store UI objects in cache or session scope

If broken it is, fix it you should — Wed, 28 May 2008 15:23:55 GMT

ASP.NET Memory: Thou shalt not store UI objects in cache or session scope Surprisingly the issue I wrote

ASP.NET Memory Issues - High Memory Usage with AjaxPro (fixed in current version)

If broken it is, fix it you should — Sun, 14 Sep 2008 18:05:16 GMT

I was helping a colleague out with an OOM (OutOfMemory) situation he was dealing with. Problem description:

Don't store ASP.NET controls in Session, Application or Cache

Readed By Wrocław NUG members — Sun, 21 Sep 2008 14:12:35 GMT

In some cases you may end up with the following exception when working with RadControls for ASP.NET Ajax

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Vince Fabro — Sat, 22 Nov 2008 01:06:29 GMT

Tess -- this (almost) 3-year old article really helped me troubleshoot a memory leak today. I was able to isolate the 2 classes that are not being disposed, although I was unable to track them down to their root references. When I ran !gcroot on them it just listed the threads and HandleTables, no actual classes. This is a multi-threaded app -- does that make it more difficult to trace the root reference? Is there a way to track down the root reference in this situation?

Thanks, Vince

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Sat, 22 Nov 2008 01:50:21 GMT

Hi Vince,

All .net apps are multi threaded so not sure what you mean there. Do you mean that the app spins up its own threads? in either case it won't make a difference.

To answer your question it would help if you posted some gcroot output.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Vince Fabro — Mon, 24 Nov 2008 21:57:46 GMT

Thanks for your quick reply. (BTW, I'm using .Net 1.1 and VS 2003.)

By multi-threaded I mean that I'm manually creating threads to run various jobs. I'm not familiar enough with WinDbg to know if my threads are somehow hiding the details I'm looking for.

I followed the steps you discussed above, and here is some of that output for you...

--------------------

!dumpheap -stat

...

e1d1dc 26894 430304 Oracle.DataAccess.Client.OracleParameterCollection

9826b0 1148 574548 System.Byte[]

79ba2f5c 29305 703320 System.Collections.ArrayList

79b980a4 45053 901060 System.Threading.ManualResetEvent

79bf4c3c 45239 904780 System.Threading.WaitHandle/__WaitHandleHandleProtector

79baaff0 91280 1095360 System.Int32

79c0d748 44884 1436288 System.Threading.Overlapped

79bc3c2c 80358 1928592 System.Decimal

79b946b0 61868 2125680 System.String

98209c 30237 2359624 System.Object[]

7b35fc9c 45256 3801504 System.Net.NestedSingleAsyncResult

7b367b0c 44884 5386080 System.Net.Sockets.OverlappedAsyncResult

1505c0 5686 12917492 Free

e1fb78 188197 21830852 Oracle.DataAccess.Client.OracleParameter

---------------------

There's an obvious leak with OracleParameter and OverlappedAsyncResult. I can manually dispose of the Oracle parameters (but I shouldn't have to as I'm disposing of all parent references) and I've found there's a known issue with leaking OverlappedAsyncResult instances, but I haven't found the solution to that.

Tracking down further, here's a sample gcroot for one of the Oracle parameter objects:

----------------------

Scan Thread 0 (1b84)

Scan Thread 2 (5f0)

Scan Thread 6 (2014)

Scan Thread 7 (1fa0)

Scan Thread 8 (1308)

Scan Thread 9 (2268)

Scan Thread 10 (235c)

Scan Thread 11 (2214)

Scan Thread 12 (1cd0)

Scan Thread 13 (1818)

Scan Thread 14 (1ed8)

Scan Thread 19 (de8)

Scan HandleTable 14a5b0

Scan HandleTable 14d330

----------------------

This is where I get stuck. Unlike your examples above, none of these point to a specific class reference. Is there something I can do to track down the root references to all these stranded instances?

Thanks,

Vince

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Tue, 25 Nov 2008 10:19:25 GMT

Hi Vince,

As you mention the !dumpheap -stat output shows that there seems to be something fishy with the amount of OracleParamenters in memory.

The gcroot output however shows that there are no roots for the object that you did !gcroot on which means one of 4 things

1. The object is not rooted so it is ready for garbage collection. (read http://blogs.msdn.com/tess/archive/2006/02/02/net-memory-my-object-is-not-rooted-why-wasn-t-it-garbage-collected.aspx)

2. You stopped the command you ran prior to running !gcroot so the sos cache was left in an inconsistent state and doesnt show roots. If you break a command, sometimes you have to run !gcroot twice to see some good results.

3. You are in the middle of a gc so the heap is in "movement" and thus gcroot doesn't give correct results (see if one of the threads is calling garbagecollectgeneration in the native stack)

4. You ran gcroot on the MT (method table) rather than an instance of an object.

If I were you, i would pick a couple of the OracleParameters and run !gcroot on them to see if maybe you just caught one that wasnt rooted while the others are. or better yet, try to root some of the OracleParameterCollections (since those will be holding the OracleParameters) or maybe even some OracleCommands or similar.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Tue, 25 Nov 2008 10:20:30 GMT

Also, if you can't find the roots, no matter how hard you try, then dump some of the oracleparameters out to see what they are, maybe that will give you a clue why there are so many on the heap.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Vince Fabro — Tue, 02 Dec 2008 22:42:43 GMT

It turns out that the finalizer thread was hung due to the STA thread issue that you blogged about elsewhere. I changed the "main" by adding [MTAThread] and that appears to have fixed it. I guess that also explains why they weren't rooted.

Thanks for your help!

Vince

The Event Handler That Cried Wolf

Infinities Loop — Thu, 15 Jan 2009 00:27:40 GMT

I ran into an interesting and unexpected behavior in ASP.NET AJAX event handling the other day. Once

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

jcentauri — Thu, 30 Apr 2009 13:16:29 GMT

Very useful and well structured blog here. Thanks for sharing all this information with us.

This post helped me to test a custom control which I designed.

Thanks

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

mling — Mon, 17 Aug 2009 19:49:02 GMT

Hi, Tess,

I ran into this article when I'm trying to find out some hint for my problem. May be you can give me some clue.

I have a WinForm application which has several working threads that receive data from IO and display these data on several forms. A form receives notification to display either from the input thread or the output thread. Mostly it works well, but sometimes we notice that there is no display for the input and we realize that the input thread stopped somewhere. From the logs we got to the point where the code notify the form by event. My suspect for the issue are:

1 - Some wrong code we implemented. We can't see any, but still looking for it. The error must be very subtle, since the application sometime runs well for a week under the same condition

2 - Some unknown behaviour by using thread, event and delegate

3 - Timing issue. My feeling is now I add many log into the code, it's harder to get the input stop.

3 - GC.

The application was implemented using .net 1.1.

Do you have any idea what is going on and what can I do to get close to it.

Thanks in advance,

Mling

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

alan — Wed, 02 Sep 2009 02:35:40 GMT

I am trying to find the method that an event handler is pointing to in a 64 bit process dump

I can't seem to find much info on this and was hoping you might be able to help???

0:007> !do 00000000116d70d8

Name: System.Windows.Forms.DataGridViewCellMouseEventHandler

MethodTable: 0000064276a10a38

EEClass: 0000064276ad3698

Size: 64(0x40) bytes

(M:\WINDOWS\assembly\GAC_MSIL\System.Windows.Forms\2.0.0.0__b77a5c561934e089\System.Windows.Forms.dll)

Fields:

MT Field Offset Type VT Attr Value Name

000006427881d280 40000f9 8 System.Object 0 instance 00000000113ff0d0 _target

0000064278852e60 40000fa 10 ...ection.MethodBase 0 instance 0000000000000000 _methodBase

000006427882b818 40000fb 18 System.IntPtr 0 instance -2145899464 _methodPtr

000006427882b818 40000fc 20 System.IntPtr 0 instance 0 _methodPtrAux

000006427881d280 4000106 28 System.Object 0 instance 0000000000000000 _invocationList

000006427882b818 4000107 30 System.IntPtr 0 instance 0 _invocationCount

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Tess — Wed, 02 Sep 2009 11:16:01 GMT

Its a bit tricky to find it. The easiest way is probably to dump out the target and look at any methods it has that might be handling it.

If you are worried about that eventhandler from a memory leak perspective, i do think you might be going down the wrong track though as that is a typical UI object that will go out of scope with the form so it wont keep the target alive an unneccessary amount of time (unless you are storing the datagridview in a static or something)...

If your datagrid has a lot of rows, you will have a lot of these eventhandlers but probably all pointing back to the same form. If you have a lot of them, i would look into the number of rows more than these eventhandlers.

I might be wrong though as i havent seen the rest of the data but that is just my first instinct seeing this info.

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

Chris — Tue, 03 Nov 2009 11:29:50 GMT

Still excellent 3 years later!

Thanks Tess,

Chris

re: .NET Memory Leak Case Study: The Event Handlers That Made The Memory Baloon

alejacma — Thu, 03 Dec 2009 12:18:01 GMT

Note that if you get "not in jit code range" when running "!ip2md", you may also do this directly: "u 0n34845811". In the dump I'm currently looking, I can see the name of the method in the very first line of the output of "u". No need to do all the "!dumpclass ; !dumpmt ; etc." thing to find the method of the event handler.

Cheers!