Common Sources of Processor Performance Penalties: Five Issues

re: Common Sources of Processor Performance Penalties: Five Issues

Darren Oakey — Tue, 01 Mar 2005 23:44:00 GMT

Rico,
Gotta say - I'm loving all these performance articles. They're interesting for themselves, and also for what they incidentally tell us about the internals.

Particularly loved the previous ones where you did the GC tests.

re: Common Sources of Processor Performance Penalties: Five Issues

Rico Mariani — Wed, 02 Mar 2005 00:01:00 GMT

Thanks for the kind words Darren.

I had some thoughts as to what topics might have made a good #6, #7 etc. but maybe it's more useful to see what my readers think should have been #6.

re: Common Sources of Processor Performance Penalties: Five Issues

Kent Boogaart — Wed, 02 Mar 2005 05:58:00 GMT

Yep, excellent article as per usual Rico.

Cache misses due to internal waste must be very common indeed. How often do you have a collection of objects and want to perform an operation or access some data of each of those objects?

I don't really see any way to combat this problem, besides keeping types as small as possible and limiting loops over objects to where they're absolutely needed (perhaps even combining multiple loops into one). Would you agree?

re: Common Sources of Processor Performance Penalties: Five Issues

Tommy Carlier — Wed, 02 Mar 2005 07:35:00 GMT

Great article. I love it. But I don't really know how to translate it to 'best practices'. Are there practical techniques (programming in .NET) to avoid these issues as much as possible?

Minor Correction

Ryan Lamansky (Kardax) — Wed, 02 Mar 2005 15:51:00 GMT

"For data, that overhead tends to be dwarfed by locality benefits that come from using a garbage collected memory system."

This should say "a compacting garbage collected memory system". Emphasis on "compacting". A garbage collector that doesn't compact its heap does not get this benefit.

Great article :)

-Ryan

re: Common Sources of Processor Performance Penalties: Five Issues

Rico Mariani — Wed, 02 Mar 2005 18:13:00 GMT

That's a good point about "compacting" -- I'm too biased by our own but there are of course other ways to do garbage collection.

Several of the comments directly or indirectly ask how these thoughts might translate into best practices. I think that might make a good companion article. I'll try to get to that soon.

Processor Performance Top 5

Mike Taulty's Weblog — Thu, 03 Mar 2005 03:17:00 GMT

re: Common Sources of Processor Performance Penalties: Five Issues

Matt — Thu, 03 Mar 2005 10:01:00 GMT

Great article.
I'm not sure that i understand the rules that the processor uses to fill the cache. I can see that from one programs perspective locality and size would make a difference but as soon as you're running in a multi tasking/threading environment where many processes may be running, won't that kill the cache as the processor jumps all over the place to service each process?

Best practices and maybe some kind of test results to show just how much can be gained would be a fantastic article!

re: Common Sources of Processor Performance Penalties: Five Issues

Eric — Thu, 03 Mar 2005 16:33:00 GMT

"For data, that overhead tends to be dwarfed by locality benefits that come from using a garbage collected memory system."

Though practicaly, it doesn't happen and the linear allocation and compaction schemes actually end up maximizing stride when compared to classic allocation schemes or a segregated (unmanaged) allocator.
GC compaction has the tendency to trash the CPU cache, and GC delayed release also means that recently freed memory is unlikely to reused (while it has a high probability of being in CPU cache thanks to object cleanup logic).

The points made in this article are interesting, but the interpretation is somewhat biased towards overestimating GC locality, which in the real world is GC's effective Achille heel.

re: Common Sources of Processor Performance Penalties: Five Issues

Rico Mariani — Thu, 03 Mar 2005 17:11:00 GMT

Now I'm going to disagree with you but before I do that I want to thank you for that lucid comment because there's gems in there and this is a very slippery problem.

Eric writes: "[I]n the real world [locality] is GC's effective Achille [sic] heel"

That hasn't been my observation. If I had to pick the GC's weakness it's in patterns that put it in crisis (see "Mid-life crisis" http://weblogs.asp.net/ricom/archive/2003/12/04/41281.aspx). Customers can and do run into those -- it's something I see when people come to our labs.

In constrast I've yet to observe locality problems attributable to the GC.

On the other hand I have seen plenty of locality issues attributable to pointer spaghetti. That problem is universal though -- no allocator can save you from pointer spaghetti.

But I suspect there have been lots of papers written on this topic. I don't think there is a clear answer that's universally true. I can only speak to problems my customers tend to have or not have.

Finally, you *should* assume that what I write is biased. Not because I'm being nefarious but because I base my writings on the data that I have and I don't have all the data. It's always good to ask yourself "Are my scenarios similar to the ones that Rico is likely to have seen? Are my scenarios like those of large Microsoft customers?" If they are then it's more likely that my opinion will be relevant to you.

re: Common Sources of Processor Performance Penalties: Five Issues

Simonjit Dutta — Thu, 03 Mar 2005 21:50:00 GMT

Great article & Thanks for covering my personal hot button, issue#3. In code density (instruction locality) issues, it may be fair to say managed code is exerting unique presssue on todays TLB architectures. I found one of the best figurative explanation for this in your article: "instruction pointer dancing through various assemblies and visiting tiny helpers...". These tiny helpers are often heapalloc'ed and thus are way out in the address space from where the work is done. Will tuning of prejit large assemlies be able to address the locality issues of the tiny helpers reltive to where the work gets done?

re: Common Sources of Processor Performance Penalties: Five Issues

Eric — Fri, 04 Mar 2005 15:27:00 GMT

> It's always good to ask yourself "Are my scenarios similar to the ones that Rico is likely to have seen?
> Are my scenarios like those of large Microsoft customers?"
>If they are then it's more likely that my opinion will be relevant to you.

Well, a better question would be: in which scenario does my theory prove correct, ie. the GC does effectively deliver higher locality, as measured with CPU-level cache miss tools?

Don't take it personnally, you wouldn't be the first person believing in GC locality and then discovering how poor it can actually be (it may be higher than that of the Windows Heap Manager, but then, that's no great challenge to begin with).

re: Common Sources of Processor Performance Penalties: Five Issues

Keith Patrick — Sat, 05 Mar 2005 06:28:00 GMT

Related to #2: I could have sworn I read somewhere (maybe BradA...I'm too tired to look it up tonight) that due to the cost (assuming stack-related) of throwing an exception, that it is better to prevent one via a conditional than let an exception get thrown. Or maybe I'm misreading it, and you mean that you should use exceptions instead of return values to indicate error. So basically, I'm asking whether the savings of exceptions is in the throwing or in the catching.

re: Common Sources of Processor Performance Penalties: Five Issues

Rico Mariani — Sat, 05 Mar 2005 06:37:00 GMT

Be careful now. There is no savings in either the throwing OR the catching. Exceptions always cost you tons.

The savings you get are as follows and they're somewhat indirect.

Suppose you have exotic failure conditions. In regular code you have to check each one in each place it might happen: if (blah) if (blahblah) if (blahblahblah). Those tests are present even though the failure may never actually happen. You always pay the cost of the test. If you use exceptions you don't have those tests. So you win if the exception doesn't happen or happens very rarely.

If the exception happens fairly frequently you LOSE big time. Hence my general rule is, don't use an exception unless the failure happens less than about 1 time in 1000. And more exotic is better still. If the exception is too common then you will lose.

See http://weblogs.asp.net/ricom/archive/2003/12/19/44697.aspx Remove Comment 385718

re: Common Sources of Processor Performance Penalties: Five Issues

Shawn O'Keef — Wed, 09 Mar 2005 06:49:00 GMT

"In constrast I've yet to observe locality problems attributable to the GC."

This is less true if your heap is not in a contiguous space.