ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

stephen patten — Fri, 23 Jun 2006 02:59:41 GMT

Glad you're back!

The evils of dynamically building strings with += and GC

Robert Seder — Fri, 23 Jun 2006 03:47:59 GMT

I ran across this post:
http://blogs.msdn.com/tess/archive/2006/06/22/643309.aspx
She went into painfully...

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Fernando Tubio — Fri, 23 Jun 2006 05:20:58 GMT

As always, a very enlightening article.

Just a minor quibble about the "Hello world" string concatenation example you give towards the end. In that case the compiler takes care of the concatenation of the string literals and only a single call to String.Concat( string, string ) is actually performed. Even if the concatenation involved variables, as in str += s1 + s2 + s3 + s4, the compiler first builds an array to hold the individual variables and then calls String.Concat( string [] ).

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

AlexP — Fri, 23 Jun 2006 09:56:16 GMT

Hi, Tess.
It's great article. Very interesting for me. Look forward for another new :)

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

James Bolles — Fri, 23 Jun 2006 16:42:23 GMT

Thanks for this post, Tess. I am in a similar situation where I have large amount of memory being allocated during web service processing. Have you ever experienced cases where a multi-threaded application running on a server with multiple CPUs continues to allocate memory even when profilers are stating the objects have been collected? We have some value types in our logic that are referenced by many threads and it seems that some of those value types are still in memory even after they should be deleted.

Keep posting your findings. Your blog has helped me to learn how powerful and useful WinDbg can be for production support.

Interesting Finds: June 23, 2006 AM edition

Jason Haley — Fri, 23 Jun 2006 17:07:27 GMT

Interesting Finds: June 23, 2006 AM edition

Jason Haley — Fri, 23 Jun 2006 17:09:56 GMT

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

snaveen — Fri, 23 Jun 2006 20:56:46 GMT

Hi,
Cool post. Another way to identify the cause of the objects in the LOH is to have a “bp on mscorwks!gc_heap::allocate_large_object” something like this bp 791c4fcd "!clrstack;.echo *********Allocation of large object heap***********;g". This should help identifying the objects that are getting allocated in the LOH.

Naveen

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Tess — Mon, 26 Jun 2006 09:13:36 GMT

Thanks for the note on the Hello world sample Fernando, that was news to me, but good news nevertheless:)

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Tess — Mon, 26 Jun 2006 09:21:04 GMT

HI James,

I am not sure exactly why this is happening in your case and how the particular profilers determine that it is deleted. I would look at if it is actually stating that it is deleted or just un-rooted, in which case it might not really be garbage collected yet? Also perhaps it is that you are pinning objects and leaving a lot of free space.

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Tess — Mon, 26 Jun 2006 09:22:26 GMT

Cool comment Naveen,

that is definitely worth running in a test environment if you are looking at high usage of the LOH.

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Howard — Thu, 28 Sep 2006 17:26:51 GMT

I've got an application which uses 2 app pools on the same machine, so I've got 2 w3wp processes. Ok, nothing wrong with that except then when I try to look at the .NET counters as per Tess's excellent (again) article they show up as w3wp and w3wp#1.

Does anybody know a way to correlate the PID, or App Pool or anything to the instance name shown in Perf Mon so I can tell which ones which?

Thanks

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Tess — Fri, 29 Sep 2006 09:35:45 GMT

If you look at the process counter it has an id correspoding to each process that you can correlate the other counters with.

.NET Hang Debugging Walkthrough

If broken it is, fix it you should — Mon, 16 Oct 2006 15:31:28 GMT

I have talked about a number of different hang/performance issues in my posts. This post is a generic

.NET Hang Debugging Walkthrough

If broken it is, fix it you should — Mon, 16 Oct 2006 15:32:24 GMT

I have talked about a number of different hang/performance issues in my posts. This post is a generic

ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

If broken it is, fix it you should — Fri, 24 Nov 2006 18:48:54 GMT

I get enough issues relating to bad perf caused by large viewstate that I felt like it is time to dedicate

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

philippe — Wed, 29 Nov 2006 13:35:32 GMT

Excellent article!

We are also in a similar situation. Our code utlise some wrappers to call some native C code and if we build the wrappers in C# ie in manaed code then the server hangs within a few minutes as they are too many alloc for the GC to keep up. If we build the wrappers using __pin and Marshall then the web servers are ok but %Time in GC over around 80% with a CPUI load at 75% in .NET2.0.

The exact same app in .NET1.1 does not experience any of this! Any clues here?

Your help would be greatly appreciated.

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Chris — Fri, 09 Mar 2007 00:17:50 GMT

Ah, to have your understanding... Currently our production app is experiencing almost exactly this issue. Our GC 0,1, and 2 are all in lock step after all allocation seems to be done. We're allocating a bunch (emphasis on bunch) of small things (around 400,000 a second) at different memory sizes but jumping to 30MB/sec at one point. Having the most impossible time breaking down what you have here to steps we can take because you're so much smarter than me and assume I know things I don't. I hope your enjoying your ambrosia.

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Tess — Fri, 09 Mar 2007 10:25:22 GMT

Hi Chris,

It's a bit hard to tell just from that description what is going on, but it does sound like one of 2 things if GC 0,1 and 2 increase by pretty much exactly the same numbers.

1. you do have some large objects that you are perhaps not aware of

2. someone is calling GC collect

I would suggest that you check out the large object heap size in perfmon (under .net clr memory) and see if it goes up, if it does, look at the dump and run !dumpheap -min 85000, perhaps all these small objects are stored in a large object array or similar that ends up on the LOH.

The other option with the GC.Collect you can verify or disspell by checking out the number of induced gc's (also under .net clr memory).

Having said this, GC perf issues like this are super tricky to figure out. I am working on one right now that I was planning on blogging on later (things have just been terribly busy lately), but my issue is caused by something really rare that i dont think applies to your situation. It is caused by a library used in the asp.net app calling SetProcessorAffinityMask limiting the process to run on one process:) and thus causing all the gc threads to spin on that one process in turn causing major cpu usage... a very ugly story...

High CPU in GC and other badness caused by SetProcessAffinityMask

If broken it is, fix it you should — Mon, 12 Mar 2007 18:31:23 GMT

I thought I'd share a support story with you from a very interesting case I have. My customer is running

ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState at Sanal Kiler

Sun, 15 Apr 2007 23:09:08 GMT

PingBack from http://sanal.org/?p=319

Top 20 .NET Garbage Collection (GC) Articles

roy ashbrook — Fri, 22 Jun 2007 08:36:32 GMT

Ah. Garbage Collection... how I love and hate thee. =P I think one sad thing about programming in .net

Top 10 WinDbg.exe Usage Articles

roy ashbrook — Tue, 26 Jun 2007 12:49:23 GMT

These are the articles (in no particular order) that I felt best showed a thorough use of the WinDbg

re: ASP.NET Case Study: High CPU in GC - Large objects and high allocation rates

Webmaster — Thu, 28 May 2009 12:05:50 GMT

Thanks, this really helped me find one of the problems leading to high cpu usage on our server!