If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

Q&A Reader emails about .NET memory leaks and random questions

Q&A Reader emails about .NET memory leaks and random questions

  • Comments 9

Here are of some of the reader emails I got this week and my answers to them...

  • How do I troubleshoot this memory leak
  • Debugging techniques for capturing stacks of OOMS
  • Do you have a list of Debugging 101 links?
  • Debugging managed code in IE
  • Getting to dependency properties with sos
let me know if this is something you find useful so I should keep doing it from time to time, or if it's only useful for the selected few who's email I happened to pick this week:)

Note: some of the emails have been edited to fit better in a Q/A format, hopefully I haven't missed any important bits...

Q: How do I troubleshoot this memory leak

I started observing a slow leak in task manager of about 100 MB / 30 mins, in a web application consisting of WCF services.

In perfmon Gen 0, 1, and 2 collections steadily grew, and Gen 2 size kept growing but Gen 0 and Gen 1 just bounced around staying around the same level.  Private bytes and virtual bytes for the process didn't change much so this didn't allow me to go much further with the steps shown in the lab examples.

The threads (~* kb) were sitting in a state which you mentioned in one of your articles could be ignored, and !dumpheap -stat didn't show one object in particular taking up all the room.

Can you recommend a direction?

Supporting data:

0:000> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
    d866000 (  221592) : 10.57%    53.38%    : RegionUsageIsVAD
   66a90000 ( 1681984) : 80.21%    00.00%    : RegionUsageFree
    8fd6000 (  147288) : 07.02%    35.48%    : RegionUsageImage
     6fb000 (    7148) : 00.34%    01.72%    : RegionUsageStack
      1c000 (     112) : 00.01%    00.03%    : RegionUsageTeb
    260a000 (   38952) : 01.86%    09.38%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 19560000 (415104 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   66a90000 ( 1681984) : 80.21%   : <free>
    997f000 (  157180) : 07.50%   : MEM_IMAGE
     708000 (    7200) : 00.34%   : MEM_MAPPED
    f4d9000 (  250724) : 11.96%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   1266a000 (  301480) : 14.38%   : MEM_COMMIT
   66a90000 ( 1681984) : 80.21%   : MEM_FREE
    6ef6000 (  113624) : 05.42%   : MEM_RESERVE

Largest free region: Base 11810000 - Size 38f00000 (932864 KB)

!dumpheap -stat
...
7912dae8     2000      1217176 System.Byte[]
79131840     1213      1256668 System.DateTime[]
65410fd0    22839      1278984 System.Data.PropertyCollection
6540a734    29644      1422912 System.Data.Index+IndexTree
65409b14    29644      1660064 System.Data.Index
790fd8c4    37893      2876484 System.String
654088b4    23193      3432564 System.Data.DataColumn
65412bb4     3599      3728564 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
7912d9bc    27483      4171320 System.Collections.Hashtable+bucket[]
7912d7c0   119608      5075228 System.Int32[]
7912d8f8    98188     11996192 System.Object[]
654359c8    29650     30717400 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
Total 861370 objects

My answer: 

You will always see Gen 1 size and Gen 0 size stay pretty flat since there are limit to how big these can grow to be.  In fact the Gen 0 size is not really the size of Gen 0, rather it is the budget for Gen 0.  These counters are updated at the end of each garbage collection, and at that point Gen 0 will always contain 0 bytes, so the size wouldn't make much sense...

Since Gen 2 is continuously growing, the interesting question here would be, is it growing at approximately the same rate as the process? If so, you have a managed memory issue. You can also check out .net clr memory\#Bytes in all heaps

The dump you have taken is only at around 300 MB (1266a000 ( 301480) : 14.38% : MEM_COMMIT) which makes it a bit hard to say anything about memory usage since dlls and core parts of the framework will take up a large part of that. What I would recommend that you do is to take multiple memory dumps, preferably 2-3 taken about 100 MB apart, and then compare the size of the dumps or MEM_COMMIT, the GC heap sizes with !eeheap -gc to see if they grow at the same rate, and then finally, if they do, then compare the !dumpheap -stat output to see what objects increase the most.

Just judging by your !dumpheap -stat output you seem to have mostly datasets so it might be worthwhile checking the sizes with !objsize on dataset items and !gcroot them to see why they are sticking around. Having said that though, the dump is pretty small so the amount of dataset related items isn’t by any means alarming. Hopefully looking at the sizes etc. of the datasets you can see if they seem to be an issue, comparing it to what you think the application should use.

Q: Debugging techniques for capturing stacks of OOMS

I'd like know what software or technique you use (http://blogs.msdn.com/tess/archive/2008/09/02/outofmemoryexceptions-while-remoting-very-large-datasets.aspx) to capture full stack trace of execution?

Can this technique or software be used for ASP.NET?

My answer:

There are multiple ways to capture the stack trace of an OOM.  The technique I used in this case was to attach to the process with windbg (from the debugging tools for windows) and set a breakpoint on mscorwks!ThrowOutOfMemory

bp mscorwks!ThrowOutOfMemory

This is of course only feasible if you can break into the process, which you usually can't do on a live asp.net application.  In an ASP.NET application you could set up an adplus config file with this breakpoint, and have it take a dump when it encountered the breakpoint...

Creating dumps with Windbg and writing ADPlus Config files

or you can set up GCFailFastOnOOM per http://support.microsoft.com/kb/820745

This article on how to troubleshoot OOMs in ASP.NET with DebugDiag and Perfmon is also very useful in this scenario.

Q: Do you have a list of Debugging 101 links?

I have been reading your blogs and using your Lab Demos. So far they have been great but I was wondering if you had any recommendations about learning the debugging tool a little better. There are points where I get really confused and lost but I will admit that is because I don’t know or understand a command or two (or three) in the debugger. I kind of jumped into this in the middle so I would like to get “back to the basics” and try and understand the debugger a little better before I start debugging.

Any simple Debugging 101 links you could recommend would be greatly appreciated. In the mean time, looking forward to your posts!

My answer:

For debugging basics... have a look at the following posts

Back to Basics - How do I get the memory dumps in the first place? And what is SOS.dll?
Learning .NET debugging
.NET Hang Debugging Walkthrough
ASP.NET Memory Investigation
A big list of debugging resources

and of course the labs mentioned in the email

.NET Debugging Demos - Information and setup instructions

Q: Fragmentation of the LOH

I have been trying to reduce the memory foot print of my application recently and I found something interesting. My application takes advantage of the new Script Combining features in .NET 3.5 SP1 to reduce the number of HTTP requests as well as improve client load time.

After reviewing some memory dumps I am starting to see many large strings stored on the LOH. Many of these strings contain the output of the combined scripts. All of the combined scripts are embedded resources within the System.Web.Extensions.dll as well as our own embedded resources within our custom control library.

Will this cause heavy framentation within the LOH? Are you aware of anything I can do to help this?

Supporting data:

0:034> !dumpheap -stat -min 85000
Statistics:
      MT    Count    TotalSize Class Name
7933335c        2      1048608 System.Byte[]
000dbb08       10     13264408      Free
793308ec      432     45672312 System.String
Total 444 objects

My answer:

I haven't worked a whole lot with this, so I don't know how often these strings are generated and what the normal size of such a script is... I guess it depends on the content that is being combined...   More generally I can say that heavy usage of the LOH will cause pockets of Free memory like this, and the amount of fragmentation you see over time depends on how well new strings fit into these pockets.  If you are allocating and deallocating the same size objects all the time it should be fine.  If the get continously larger you will see more fragmentation. 

I would just observe it over time and see if it gets really large.  I don't know how long you have been running at this point, but depending on the circumstances, ~13 MB doesn't seem extreme.   If it gets too bad then the options would be to reduce the size of the combined scripts or finding a way such that you cache the scripts or something so they don't get generated as often.  Not sure how that would be done though without researching more...

Q: Debugging managed code in IE

I have a .NET component that I've written using the 1.1 version of the framework. This component gets invoked by Internet Explorer via COM, like an ActiveX control. One of our testers is experiencing some hanging - it takes 3-4 minutes to load something up that comes up instantly on just about every other machine.

I grabbed a memory dump of the process while it was hanging, but nothing in the dump is considered "managed"... so there's not really much I can do with it. I know that it has loaded up managed code though. Have you ever attempted to debug iexplore.exe like this, and if so, would you be able to explain how you were able to do it?

My answer:

Yes, I have debugged winforms user controls in IE (see http://blogs.msdn.com/tess/archive/2008/02/19/high-memory-usage-with-usercontrols-in-ie-using-debug-diag.aspx).  If by "noting in the dump is considered managed" you mean that all threads are reporting

Thread 27
Not a managed thread.

then I think the simple answer is that either

a) you didn't get the dump at the time the process hung or
b) you are not hanging in managed code,  check out ~* kb to see if any of the threads are doing anything interesting in the native part of the stacks...  or
c) the "hang" is outside of the process, i.e. some network latency or similar

Are you perhaps running multiple queries to the same server so that you are trying to load up multiple usercontrols at once.  If I recall correctly you can run into blocking in those situations, but I don't remember the details...

Q: Getting to dependency properties with sos

I’m trying to debug a WPF application and I want to reach through GCRoot which of my object remain in heap and what of those It’s doesn’t begin disposed, and for this task I want to know the name of the control, but name is a Dependency Property (DP) so it’s a shared property, I can access to the instance of class DepenedecyProperty of Name property, but not to his value, because values are stores in other side. The only interesting filed was GlobalIndexCount. My question is, how access to get a value for a control stored in a DP?, Thank you.

My answer:

Unfortunately I don't know enough about dependency properties to know how they are stored in memory.  I do remember it being tricky though, the few times I have debugged WPF applications.  My advice would be create a small sample and hook up visual studio and stop at a place where you have access to the dependency property and use the watch window there to get a better feel for how it is laid out, and then you can bring that knowledge back and use it with windbg and sos in the "real" scenario.

  

Contact disclaimer

As I have mentioned before, there isn't enough time in the day to answer all the emails I get through the blog, so I usually only answer emails where I feel that the answer is something that might benefit many people. 

If you have an issue, and you're considering emailing me about it I should mention that you have a better chance of getting the question answered if you write it as a comment on a post it relates to.  If you're in an extreme hurry and/or have a very specific situation, you should definitely call support (see my Contacting Tess... post for more info)

Laters,

Tess

  • Thanks for answering my question, Tess. I submitted the question about debugging managed code in IE. After digging through about two different memory dumps from the machine, I found that only 3-4 of them had any threads with managed code... and those threads looked like they were in the middle of performing a "SafeNativeMethod" call like this:

    0x07b7afe0  0x7c8285ec [FRAME: NDirectMethodFrameStandalone] [DEFAULT] Boolean System.Windows.Forms.SafeNativeMethods.SetWindowPos(ValueClass System.Runtime.InteropServices.HandleRef,ValueClass System.Runtime.InteropServices.HandleRef,I4,I4,I4,I4,I4)

    I'm at a total loss to figure out what the problem is, so I've gone ahead and opened up a ticket with Microsoft about it.

    I very much appreciate that you responded to my question on your blog!

    - Keith

  • Also, to answer your question...

    I've monitored the network traffic with Fiddler and used SQL Profiler to make sure my control wasn't doing some kind of processing when IE hung. The freeze always happens before or after every network\SQL call has completed... it never freezes up in the middle of processing.

    My thought right now is that the freeze is related to rendering. I feel like something is in a deadlock, but I just can't figure out what it could be. Syncblk doesn't show locks in any of my 22 memory dumps, and the !threads command shows a lock count of 0 for everything.

    I'll admit to being somewhat new to debugging via memory dumps, so hopefully the folks at Microsoft can help me figure this one out!

    Thanks again!

    - Keith

  • Regarding dependency properties....

    The dependency property itself is a static property of the class, and any instance property is a wrapper that calls the dependency property, making this trickier than it looks.

    However, you can only use a DependencyProperty on a class that derives from DependencyObject.  From Reflector, you can see that there's the following definition in the DependencyObject class:

    private IDictionary<DependencyProperty, object> dependencyPropertyValues;

    ... so I'd guess that's where the values are stored.  Remember though that these might not be the actual values - they might be a binding object that calculates the value.  Sadly I haven't had time to look any deeper yet...

  • Thanks for responding to my e-mail (about the memory leak).  I'm still working on it and will comment here when I figure it out.

    On another note, in reference to the "Fragmentation of the LOH" e-mail, I had read an article regarding the 1.0 framework that said the LOH is allocated but never really freed-up.  The objects might be removed and the space can be used again for the application, but the memory is not freed-up for the OS.

    Do you know if that has changed?  

    Does memory allocated for the LOH ever get released back to the OS?

    Thanks,

    James

  • yes, when the segment no longer has any objects... this was changed in 2.0 i believe

  • Though the explanation is rather tricky,it is good.

    I would request you to kindly explain it with the help of  examples .So that it is easier to understand this subject.

  • Though the explanation is rather tricky,it is good.

    I would request you to kindly explain it with the help of  examples .So that it is easier to understand this subject.

  • Other Don't Tell Me &quot;How&quot;, Tell me &quot;What&quot; Microsoft Network Monitor 3.2 .NET MSDN

  • I have a question about % time in GC and the 3 generations.  I have an app that has a very low % of time in the GC, but the ratio between the generations doesnt seem to be ideal.  The average % of time in the GC is 2.27%, but the ratio between gen0 and gen1 is .65, and the ratio between gen1 and gen2 is .19.  Should i be looking further, or since the % is so low, just ignore the ratio?

    Thanks,

    Nick

Page 1 of 1 (9 items)
Leave a Comment
  • Please add 6 and 2 and type the answer here:
  • Post