In response to my first post, Scott asked a very topical question:

Not sure if this falls in your basket or not, but I would be interest in any tips or tricks for memory profiling ASP.NET applications.

Well, Scott, I'm pleased that you're interested in this subject, because it's a very good way to introduce how we're testing the performance of ASP.NET under stress.  For a grounding on the subject, I encourage anyone interested in this subject to start by reading this.  It's an article written by Thomas Marquardt, a developer (and former tester) on our team who did a lot of work creating the test infrastructure we're using today.  It should also handily answer your question.

The other reason why this I want to point you to this article is that the source code for one of the test tools used for stress automation, QQQ.exe, can be downloaded there.  QQQ is a tool that monitors performance counters for the ASP.NET worker process and automatically attaches a debugger if any of a few different critical thresholds are reached.  We use this with the Web Application Stress Tool (which, in the long tradition of giving things names that will freak out everybody in marketing, is also called “Homer”) to simulate the process of multiple users making large quantities of requests to the server and catching the problems that arise.  These tools, combined with some debugging tools, scripts for automatically re-imaging the test machines and launching stress, and (naturally) the test pages themselves, are the way in which we stress-test the product.

The tests themselves cover a pretty broad swath of scenarios.  As you'd expect, we have tests simulating high traffic in a production environment with all managed code, but we also have tests for pages that inter-operate with COM components and call data methods.  We also check how ASP.NET handles some more esoteric scenarios.  What happens if  the user changes machine.config every few minutes, causing the application to restart?  What if there's a memory leak in the user's applications?  What if the worker process crashes?  The process model, of course, should gracefully recover from these hiccups.  In all of these scenarios, the pages we use for stress testing are also functional tests that we use for regular verification of the product, so we can get as close to simulating real-life workloads and traffic as possible.

I used to debug stress, so a lot of this comes from memory (it's like riding a bicycle), but special thanks go to (fellow tester) Vladimir Kritchko for pointing me to the latest stress infrastructure documentation.