<After initially posting this article, I got some feedback from a customer who said that his company's apps aren't using web protocols. Their apps use RDP. I realized that I was making an assumption, and I can't afford to do that in my job, so I am updating the article to cover "all" protocols>
With all of the testing we see in our labs, we often get asked how we are able to emulate all of the clients in order to really see the full picture of how the entire system will behave under load. This comes up with web based systems that are heavy on the client-side java script or use a lot of client plugins, and also with thick client apps. My standard reply often catches people off-guard: "Why in the world would you want to do that?" Yeah, I am being dramatic here. I am actually way more polite with the answer I speak out loud [and also usually way more sarcastic with the answer I would *LIKE* to say].
In order to perform proper end to end testing, there is no need to spin up actual clients or to simulate the client side functionality with all of the test users. All you need to do is provide a representative load to the server of the traffic it can expect and then add one or two full blown clients running a single user client end test. You can then get all of the data you need. Let's look at an example. Below is an infrastructure diagram from a lab engagement I performed a couple years ago:
When we create web tests to drive load against the server, these tests only worry about generating the proper HTTP(s) traffic that will flow through the BigIP. After all, that is the only part of the client that the server will see and be aware of. If a client has to crunch data locally for a while, this does not impact other clients or the SUT. Therefore, we do not need to simulate that work for every single client we are trying to emulate. We can use the environment above, driving hundreds or thousands of concurrent users against the SUT, and then simply add one more machine that houses a real client and get our client timings, thereby also getting our end to end timings.
So when I talk about testing with thick clients, this could refer to any type of client that uses a protocol other than HTTP(S). These clients must be simulated with a test type other than Visual Studio Webtests. In my world, that usually means Visual Studio Unit Tests and sometimes Visual Studio CodedUI tests. However, at the end of the day, the concept is identical. In order to simulate load, I need to generate whatever traffic will flow out of a client onto the network and into a server, and then back again. The key here is that I am focused ONLY on the traffic that goes across the network. If my app uses a .NET application to talk to SQL via ADO.NET, all I need the harness to drive is the actual ADO.NET traffic. If it is a WCF app that uses NET.TCP binding, all I need to drive is the WCF payload creation and transmission, etc.
This table covers a few objections that I have heard before, as well as my responses to them:
In order to decide what method I am going to use to test an application end to end, I consider the maintainability of the code and the cost of developing/executing the tests (including hardware costs and labor). Then I choose the method that will get me accurate results and still fit within my budget. 99.9% of the time, that will be to use web or unit tests that do NOT exercise the client code, but only send the necessary traffic across the wire. I then get client timing from a single client.