I've been giving this advice for years internally, and the problem pops up way too often. I'm sure it will be addressed eventually, in a future version of the CLR, but for now, every released version of the CLR has this problem. If you allocate objects larger than 85000 bytes, they will be allocated in the Large Object Heap (LOH). The LOH is not compacted and collection only occurs after a full collection (so gen 2 must be collected in order to collect an object in the LOH). Short-lived and frequent LOH allocations will kill the performance of your application. As an example, if you allocate from the LOH each time an ASPX page is requested, your performance will be dismal.
There are a few ways to determine if you're allocating from the Large Object Heap (LOH). The easiest is via the performance counter ".NET CLR Memory\Large Object Heap size". You can more or less determine if your allocating from the LOH on a per-request basis by checking to see if the value of this performance counter fluctuates while you make requests.
You can also deterimine if you're allocating from the LOH by profiling a request with the CLR Profiler, which is discussed in an earlier blog post. The Histogram view shows you a histogram of the allocations broken down by size, and you can quickly check to see if there's anything larger than 85000 bytes. If you right click on columns in the histogram, you can select "Show who Allocated" to see a stack trace for the allocation(s).
Another way to determine if you’re allocating from the Large Object Heap (LOH) is by setting break points. For example, you can use this break point on v2.0 on a multi-proc box, where the server GC is enabled by default:
bp mscorwks!SVR::gc_heap::allocate_large_object ".echo size=; ? poi(@esp+4); k12; gc"
If you want to get fancy, the following will attach ntsd to w3wp.exe, disable break on first chance exceptions, and trace the calls to allocate_large_object to %SYSTEMDRIVE%\NTSD.log
ntsd.exe -y %WINDIR%\symbols;srv*%WINDIR%\Symbols*http://msdl.microsoft.com/download/symbols -xd * -c "bp mscorwks!SVR::gc_heap::allocate_large_object \".echo size=; ? poi(@esp+4); k12; gc\"; g" -G -logo %SYSTEMDRIVE%\NTSD.log -pn w3wp.exe
Aside from short-lived LOH allocations, the most common cause of memory related issues in ASP.NET apps is pinning managed memory during long async I/O operations. ASP.NET itself does not do this, but perhaps your application does?