This article is targetted at the newbie who's trying to conduct some meaningful performance testing of a typical ASP.NET webservice. Typical, in the sense, having a multi-tiered architecture and maybe interfacing with an external application and/or Web Service.
 
I have presented the "things-to-do" in a bullted format so that it can be used as a checklist.
 
So here we go...
  1. Get the environment setup of the machines. In case of NLB farm, figure out how many nodes you'll need for a minimum. Eg: Two nodes for the Web Server and an accompanying NLB server IP. A couple of nodes for the Sql Server. A machine to simulate the interaction with an external system / app / WS. A machine to host and execute the test.
  2. Using VSTS, create a test project to be used for the performance testing. Next, follow the steps at the VSTS Load Test team blog @ http://blogs.msdn.com/vstsloadtestblog/archive/2006/09/22/765608.aspx to setup the Web Test for the Web Service. This project or solution needs to be used on the dedicated machine which has been setup to host the test rig. 
  3. Create a loadtest with the just created webtest as the executing test. Once you have set up the basic features from the UI while creating the LoadTest, right click on the "Run Settings" and click on Properties. In the properties window (which typically appears at the right side bottom) and change the Web Test Connection Model to ConnectionPool. Change the WebTest Connections Pool Size to a sufficiently large amount (maybe in the range of 2000-3000) if you are planning to test for thousands of concurrent users.
  4. In case you have logged onto the test rig machine with a credential not associated with a domain controller (aka Active Directory account), you will have to run this command at the test machine's command prompt to enable the test solution to access the performance counters of the physical nodes.
    net use \\<machinename>\ipc$ /user:<username> <password>
    Note: This command needs to be run for each and every node whose performance counters need to be accessed.
Now, start testing and figure out what all do the counters mean!
 
Some important notes:
Typically for a performance test, you would need to keep track of the following counters:
  • CPU utilization of each node - try to max this out with you test rig.
  • Requests / Sec executed - Found at the machine which hosts the rig.
  • ASP.NET request queue length - To find whether the processor is overloaded and requests keep pouring in.
  • Total Memory available. - Each node - to find if there are any memory leaks.
  • Disk usage - In case the web service acccesses some local files - there might be resource contention.
  • Failed requests and the exception generated.
  • SOAK Test: Make sure that you turn off the Worker Process Recycling for the Appliation Pool under which the Web Services is hosted. Else, you'll find that periodically your requests fail with either a 500 error or access denied error!
Do keep track of the CPU utilization of the machine which hosts the test rig and the machine in which the external system simulater has been installed. In case the test rig machine's CPU maxes out before the nodes' CPU, then you will have to think about acquiring another machine and running the tests parallely. In case the simulator's CPU maxes out, think about whether it makes sense at that point in time.
 
For each and every user load value, make use of the Step Load pattern and let the user count build up to the required number. Do not use the Constant Load Pattern as the Server will get flooded with requests suddenly and this might not capture proper statistics
 
In case you are stuck with some problem... well.. http://www.live.com :-)