My last entry was some generic advice about how to do a good performance investigation. I think actually it's too generic to be really useful -- in fact I think it fails my Peanut Butter Sandwich Test.
Digression to discuss the Peanut Butter Sandwich Test
I review a lot of documents and sometimes they say things that are so obvious as to be uninteresting. The little quip I have for this situation is, "Yes what you are saying is true of [the system] but it's also true of peanut butter sandwiches." Consider a snippet like this one, "Use a cache where it provides benefits," and compare with, "Use a peanut butter sandwich where it provides benefits." Both seem to work... that's a bad sign.
You certainly don't want to get an F on the Peanut Butter Sandwich Test but hopefully you won't settle for just a C-.
Back on topic
I thought it would be good to follow up the generic advice with some specific suggestions for things to look at. These are things I look at in step 2 or 3 of the investigation.
Under .NET CLR Memory, check "% Time in GC" if it's getting near 10% or higher you may have some memory issues, consider these secondary tests:
Is the CPU not saturated when it should be? Look under .NET CLR LocksAndThreads
Is the throughput rate low even though the CPU is saturated?
This just a taste of course, and each of these items would likely lead to further investigation with a profiling tool that is suitable to drilling into that particular kind of problem but these are examples of leading indicators that I use.
For more information on the GC Performance counters specifically see Maoni's blog entry on that subject. Her most recent article is on using the GC efficiently also very interesting, lots of good details there.