Walkthrough - Troubleshooting a native memory leak

re: Walkthrough - Troubleshooting a native memory leak

Guest — Thu, 21 Feb 2008 19:18:28 GMT

I just wanted to say that this is the first time I have seen someone take time out to actually explain the process of debugging a memory leak. It was very useful, thanks.

The only thing else I wanted to comment on was the section of code in "What causes the leak?", and the fact that it is not clear (to me at least, sorry if I am being dense!) what the trace is actually inferring there. What bit should I be looking at for?

Possible to highlight the offending bits please? That would be a great help on my journey.

Thanks again.

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Fri, 22 Feb 2008 11:52:57 GMT

Hi,

If you look at the top of the trace you see that we end up in the third party component. Before that we're using the System.Data namespace, but eventually we find ourselves in the Company.App.Server.DAL.DAO namespace. THIS is the part where we actually can make a change. The leak happens in the third party dll, but it's a fairly widely used component, so it could be that we've come across a known issue. Changing the way we call the data connector in our code might fix the situation.

For example: If there is a known possibility of a leak when you call a certain function with parameters X, Y & Z, then we might be able to resolve it by calling the function with parameters X & Y only.

If there is no such sollution, then we at least have sufficient information to prove to the third party that there is a problem in their product.

This is actually quite common, and I believe there is a value in showing an example where the problem is actually beyond our reach. Being able to say that you KNOW the problem lies within a certain component rather than you THINK definitively gives you the upper hand.

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

baal — Fri, 29 Feb 2008 05:15:06 GMT

i have the same problem of your customer,Memory usage would slowly increase and not be released,set the application pool to recycle at 500 MB,

this article is so great that i want kiss you. but when i finish read,i think this article is a big head and small end.

third-party problems!

in my app, no third-party, but i think the problem may be that two parts ,one is the connection to db with some resource not disposed, two is the file upload module with some resource not disposed.

re: Walkthrough - Troubleshooting a native memory leak

baal — Fri, 29 Feb 2008 09:47:19 GMT

when I got the dump file as your step, click "Start Analysis", when "loading leak track info. please wait", it looks like dead. does this really need so much time and I just wait?

re: Walkthrough - Troubleshooting a native memory leak

baal — Fri, 29 Feb 2008 11:45:36 GMT

can you give more ,write some more about native leak?

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Mon, 03 Mar 2008 14:53:30 GMT

Hi Baal,

Native leaks are always a lot more difficult to track than the managed ones.

To identify where in your code things go wrong I'd do the following:

1. Use leak track to identify the assemblies that are growing abnormally.

2. Investigate the memory usage report and look at the sample call stacks.

3. Cross-reference the callstacks with your code to identify what sections are allocating memory without releasing it properly.

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

David — Mon, 03 Mar 2008 15:52:18 GMT

Hi Johan,

I truely love your blog and find it very usefull.

How it is possible that dump size is 300MB less then size indicated by perfmon? Could it be caused by "memory holes" in the native memory?

What would you say about the following scenario:

I have a WinForms app running for 14 hours without user interaction.

Dump #1, at the beginning, size is 1020MB.

Dump #2, at the end, size is 1080MB.

GC Heap Size stays at ~200MB at both dumps.

Perfmon shows a linear increase in private bytes, from 600MB to 800MB, and increase in virtual bytes from 1300MB to 1400MB.

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Tue, 04 Mar 2008 10:17:40 GMT

Hi David,

I can't really say what the root cause is without investigating the dumps. If I had to guess I'd say that streaming is a likely cause of the inconsistency.

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

baal — Tue, 04 Mar 2008 12:57:30 GMT

Hi Johan,

Can you help me investigate my dump file?

the file is here: http://www.sula.cn/soft/w3wpdump.zip,

then write some more for the article?

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Tue, 04 Mar 2008 15:41:49 GMT

Hi Baal,

I'd recommend contacting your local Microsoft support representative and open up a regular support incident for this.

Thanks / Johan

re: Walkthrough - Troubleshooting a native memory leak

David — Wed, 05 Mar 2008 11:11:36 GMT

I want to analyze the dump myself. How would you start doing so in the described scenario ?

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Wed, 05 Mar 2008 12:48:49 GMT

Hi David,

Well, if you're not running the .NET framework I'd begin by running it through Debug Diag as described in the article above.

If it's managed I'd use windbg. For usefulo commands please refer to the "Debugging School"-section to the left as well as http://blogs.msdn.com/johan/archive/2007/01/11/i-am-getting-outofmemoryexceptions-how-can-i-troubleshoot-this.aspx

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

David — Wed, 05 Mar 2008 18:11:12 GMT

OK, but I am interested to know how streaming can cause such an inconsistency.. My application does use some kind of streaming.

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Wed, 05 Mar 2008 18:43:31 GMT

Like I said, It's not easy to guess without seeing it for myself.

I know Memory Mapped Files can cause this type of situaiton. (Sorry for the "streaming" confusion.) There might be other possible reasons that I just can't think of right now.

Does the dump look okay in all the other aspects or does it appear corrupt?

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

Martin Kulov — Thu, 15 May 2008 03:06:25 GMT

I would bet that there is some caching implementation in the thrid party component.

Did you get any result of the actual problem?

Thanks,

Martin

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Fri, 16 May 2008 17:39:06 GMT

I find that very likely as well. I was hoping to find out the root cause eventually, but unfortunately I still do not know what it was... :(

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

Nikhil — Sun, 02 Nov 2008 20:19:03 GMT

I have a native memory leak while running the ASP app on windows 2008 server under IIS6 management compatibility mode... But when I do the analysis using the steps mentioned by you, The memory size mentioned is in TBytes!! I have done this a Lot of times with a lot of options - any suggestions ?

re: Walkthrough - Troubleshooting a native memory leak

Ling — Tue, 18 Aug 2009 15:02:26 GMT

so how can we get the callstack of the leak function? I wrote a .net program, and allocate several large arrays without releasing them for test. I used the DebugDiag to monitor and analyze, finally i got the report. And it tells me that the leak function is mscorwks!EEVirtualAlloc+119, however, it doesnot show me any call stack sample. what's going on? could you please give me some suggestions?

re: Walkthrough - Troubleshooting a native memory leak

Denny Ferrassoli — Wed, 07 Oct 2009 21:05:28 GMT

Johan thank you for this post, it's been very helpful!

I have a similar issue but my callstack only shows memory addresses like below:

Function Source Destination

0x79E8C582 0x79E717B4

0x79F8C96F

kernel32!BaseThreadStart+34

I was expecting some .NET callstack info but as you can see there is none :) Do you know how I can further investigate this? Thanks

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Thu, 08 Oct 2009 10:49:57 GMT

Hi Denny!

It's not too much to work with, but if debug diag is reporting that this is the callstack hogging memory from your application I'd look into just how many threads you have running in the application. (Use ~* in windbg)

/ Johan

re: Walkthrough - Troubleshooting a native memory leak

JohanS — Thu, 08 Oct 2009 11:37:44 GMT

@Ling: (Sorry about the late reply)

Debug diag doesn't show you managed callstacks. You'll have to use windbg for that.

Open up the same dump in windbg, go to the thread identified by debug diag and use !clrstack to see the managed callstack.

/ Johan