Troubleshooting Network Emulation
Recently we had a customer support issue on trouble shooting the Network Emulation driver in VS2010 Ultimate while doing load testing. I thought a blog on how we troubleshooted and isolated the problem would be helpful, so here it is. In this blog, I discuss the problem, symptoms and also explain how Network Emulation works in 2010. I also suggest specific steps to consider to isolate and narrow down the problem.
Scope
This applies to Visual Studio 2010 Ultimate
Customer Scenario
The trouble shooting in this document is applicable to situations where you are attempting to use the Network emulation capability newly available in VS 2010 Ultimate while creating a new Load Test and in the "Edit Network Mix" screen of the wizard you select any other network type other than LAN.
What is Network Emulation?
Microsoft Visual Studio 2010 uses software-based true network emulation for all test types. This includes load tests. True network emulation simulates network conditions by direct manipulation of the network packets. The true network emulator can emulate the behavior of both wired and wireless networks by using a reliable physical link, such as an Ethernet. The following network attributes are incorporated into true network emulation:
- Round-trip time across the network (latency)
- The amount of available bandwidth
- Queuing behavior
- Packet loss
- Reordering of packets
- Error propagations.
True network emulation also provides flexibility in filtering network packets based on IP addresses or protocols such as TCP, UDP, and ICMP. This can be used by network-based developers and testers to emulate a desired test environment, assess performance, predict the effect of change, or make decisions about technology optimization. When compared to hardware test beds, true network emulation is a much cheaper and more flexible solution.
How Network Emulation Works in VS2010
Network emulation in VS 2010 Ultimate uses a network device driver that was designed and built by the Microsoft Research Labs and is productized in Visual Studio 2010. The technology has been around since 2005 and is widely used within Microsoft across many server product teams.
To use Network Emulation, you will need to install the Visual Studio 2010 Ultimate SKU. Network Emulation is configured as part of Add and new Load Test Type in Visual Studio and following the wizard screens (see above). Once you have set up network emulation following instructions at https://msdn.microsoft.com/en-us/library/dd997557.aspx, you will run your load tests. When the load test starts, it allocates a range of available ports for each of the Network profiles (DSL, 56.K Modem etc.) that you have selected in your network mix. This port range is available to the Network Emulation Driver that is enabled at run time (by default the network emulation driver is disabled).
During load testing, when the load generator sends a request to the application under test it specifies a port from the port range. When the network emulation driver sees this port from the select port range, it is able to associate this port with the network profile that this request should follow. This enables the driver to throttle the load in software ensure it meets the network profile you have selected.
How to know Network Emulation is not working?
Often one of the symptoms you'll see is that load test records socket exceptions in the log such as the one below:
"The requested address is not valid in its context xx.xx.xx.xxx:80"
NOTE: There may be other conditions that maybe causing such socket exceptions as well. The load test may continue to work, but the socket exceptions get logged. The next section will help you isolate and trouble shoot where the problem lies.
How to trouble shoot Network Emulation
To troubleshoot and isolate problems effectively you must ensure that you have done the basic tests.
- 1. Ensure that you have full network connectivity across all the machines that are participating in your load test.
- 2. Ensure you have configured the Network Emulation correctly by following the instructions and making sure admin rights are available for the agent.
- 3. Ensure that any/all firewalls are dis-abled (at least for trouble shooting) to ensure that firewall is NOT blocking specific ports or traffic on the lab network.
- a. Run tcpview (available here) to ensure that any socket connections are actually visible during run time (check for "red" highlights). You may also run your favorite port monitoring tool (portmon is another example)
- 4. Ensure that there is no virus software on the load generator machine that is possibly obstructing this software.
- 5. To isolate whether the problem is with the Network Emulation Driver or the Load Test Components you should:
- a. Eliminating the network emulation driver as a cause
- Run the load test with network emulation configured correctly (even though you may be getting socket exceptions)
- Ping another host to see whether the output shows network show down and/or higher latency. Check if the delay value matches selected network profile. If the latency values match the profile you have selected, then the network driver is working well.
- From that agent machine where you are running the load test, attempt a connection to any host outside (like your favorite web page). This test verifies that while the load test is running and network driver is enabled, that external or lab connectivity is NOT a problem. This will isolate your network emulation driver from being a problem area.
- b. Eliminating the Load Test Components as cause
You should download and run this sample test program (available as is, not Microsoft supported) on the same machine as the load generator (agent machine). This sample program simulates the exact set of socket connection calls used in the load testing components. If this test program also displays Socket Exceptions (like in the image below) then this eliminates the Load Testing product as a cause for the socket exceptions and indicates the problem lies in the environment, machine, network or something external to the tooling. Please debug the external problem first before trying to run the load test again.
If this sample program is working correctly, you will see the output as below and this will confirm that there is a likely problem in the load test program and the environment is not the likely cause. Please contact support or post your query or situation in the forums for further help in this case.
- a. Eliminating the network emulation driver as a cause
Known Issues
There is a known issue with the Broadcom network cards where packets are dropped under heavy loads. We recommend if you run into this, try another network card until Broadcom addresses this problem.
Also, if IPSEC is enabled, the ports in the network packet are encrypted and as such the network emulation driver will not be able to determine that the packets are from the designated port range as set by the load test engine (described above in "How Network Emulation Works in VS2010"). You must disable IPSEC for network emulation to work.
Additional Resources:
https://msdn.microsoft.com/en-us/library/dd505008(VS.100).aspx