In the Beta 1 release of Visual Studio 10, we released “True Network Emulation.” With this release, you can simulate different types of networks when running tests (such as a unit test or load test). One thing that is missing, however, is the ability to easily emulate different networks when not running tests using the Visual Studio Team System framework. For example, I might want to just take an application and see how it performs under the restrictions of a low bandwidth connection. I might not want to create a test for this, I just want to play with it and see what happens. To accomplish this “out of the box”, I need to create a unit test that sleeps for a very long period of time and turn on network emulation. While this is fairly easy, this seems a bit awkward. This post will focus on creating a solution that will make this process as simple as pressing a start button when I want to start emulating and pressing a stop button when I want to quit.
Create a stub application
The first step in creating the network emulator is to open Visual Studio 2010 and create an Empty “Windows Forms Application.”
The next step is to copy the assembly that contains the Network Emulation API. This assembly is named “userapi.dll” and is located in %Program Files%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\DataCollectors\x86 (or x64 for 64 bit systems). Copy this file and place it into the solution directory, then add it to the solution and change the “Copy to Output Directory” to “Copy if newer”.
Create a class to hold the Simplified Network Emulation API
In this release of Visual Studio, the network emulation API is native code. Therefore we have a limited amount of ways that we can access the API from managed code. In this sample, we will make calls into the API using PInvoke. In order to do this, we will have to create some definitions for each of these methods before we can use them. So create a new class called NativeNetworkEmulationAPI and add the following declarations:
using System;
using System.Runtime.InteropServices;
namespace StandaloneNetworkEmulation
{
public class NativeNetworkEmulationAPI
{
[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEGetVersion(ref uint pdwVersion);
[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEInitialize(ref uint pdwVersion, ref IntPtr phEmulator);
[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NECleanup(IntPtr hEmulator);
[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEStartEmulation(IntPtr hEmulator);
[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEStopEmulation(IntPtr hEmulator, uint dwTimeout, int fForce);
[DllImport("userapi.dll", SetLastError = true, CharSet = System.Runtime.InteropServices.CharSet.Auto)]
public static extern int NELoadProfile(IntPtr hEmulator,
[MarshalAs(UnmanagedType.LPWStr)]
string profileName);
}
}
Copy and Modify the Predefined Network Profiles
The next step is to copy the predefined network profiles from VS2010 to our solution so that we can modify them for use in our stand-alone emulator. To do this, lets create a folder in the solution called “Profile” and copy all of the predefined network profiles (located in %Program Files%\Microsoft Visual Studio 10.0\Common7\IDE\Templates\LoadTest\Networks) to this folder and set the Build Action = “None” for each profile. Once this is done, we need to slightly modify each profile in order to get the emulator to read the profile properly. To do this, we need to load each profile and remove the two lines that contain the name “NetworkEmulationProfile” (This should be the first and last lines in the profiles). In the “3G.network” file, the two lines removed are:
After modifying this profile, the top level xml node should be <Emulation>. One final note is to just delete the LAN.network profile as it will not work with this emulator. It is a special case that we use in our framework to indicate that we want to use the full bandwidth of the available network (in other words, don’t simulate anything).
Complete the Main Form in the application
Next, lets add the start and stop buttons to the main form and a combobox that will allow us to select a profile to use for the emulation.
Add member variables to the form:
1: private IntPtr m_emulatorHandle;
Add a Load event handler to the form and add the following code:
1: NativeNetworkEmulationAPI.NELoadProfile(m_emulatorHandle, Path.Combine("Profiles", 2: m_networkProfiles.SelectedText) + ".Network");
3: if (0 == NativeNetworkEmulationAPI.NEStartEmulation(m_emulatorHandle))
4: { 5: startButton.Enabled = false;
6: stopButton.Enabled = true;
7: }
8: else
9: { 10: MessageBox.Show("There was an error starting the emulation."); 11: }
Add a FormClosed event handler to the form and add the following code:
1: if (0 != NativeNetworkEmulationAPI.NECleanup(m_emulatorHandle))
2: { 3: MessageBox.Show("There was an error cleaning up the emulator."); 4: }
Add a Click event handler to the start button and add the following code:
1: NativeNetworkEmulationAPI.NELoadProfile(m_emulatorHandle, Path.Combine("Profiles", 2: m_networkProfiles.SelectedText) + ".Network");
3: if (0 == NativeNetworkEmulationAPI.NEStartEmulation(m_emulatorHandle))
4: { 5: startButton.Enabled = false;
6: stopButton.Enabled = true;
7: }
8: else
9: { 10: MessageBox.Show("There was an error starting the emulation."); 11: }
Add a Click event handler to the stop button and add the following code:
1: if (0 == NativeNetworkEmulationAPI.NEStopEmulation(m_emulatorHandle, 2000, 1))
2: { 3: startButton.Enabled = false;
4: stopButton.Enabled = true;
5: }
6: else
7: { 8: MessageBox.Show("There was an error stopping the emulation."); 9: }
10: startButton.Enabled = true;
11: stopButton.Enabled = false;
Conclusion
In this article, we created a sample network emulator using the API provided in the VS2010 Network Emulation feature. Although this example was simple, there are a lot of things that can be done using this API, such as packet tracing, filtering, etc. Keep an eye on this blog and in the future, I will show you how to customize the profiles to perform some of the advanced features of network emulation.
Try it out and let me know
If you haven’t done so already, please download the Beta1 build of Visual Studio Team System 2010. Then try out this network emulator and let me know what you think. We would love to hear your feedback.
With the Beta 1 release of Visual Studio 2010, we have introduced true network emulation for load tests and pretty much every other test type we support.
What is True Network emulation?
My definition of true network emulation might be different than what others might think, but I characterize true network emulation as the ability to simulate network conditions by directly manipulating the network packets. The Network Emulation that we provide in Visual Studio 2010 is a software-based emulator which can emulate the behavior of both wired and wireless networks using a reliable physical link, such as an Ethernet. A variety of network attributes are incorporated into the VS2010 Network Emulation model, including round-trip time across the network (latency), the amount of available bandwidth, queuing behavior, packet loss, reordering of packets, and error propagations. VS2010 Network Emulation also provides flexibility in filtering network packets based on IP addresses or protocols such as TCP, UDP, and ICMP.
Why do I care about network emulation?
VS2010 Network Emulation can be used by network-based application developers and testers to emulate a desired test environment, assess performance, predict the impact of change, or make decisions about technology optimization. When compared to hardware test beds, the Network Emulation provided by Visual Studio Team System is a much cheaper and more flexible solution.
How do I use the Network Emulation that is provided in Visual Studio 2010?
The easiest way to use network emulation in Visual Studio 2010 is to create a test project and modify the test settings (formerly run configuration). On the “Execution Criteria” page, select and enable the Network Emulation collector.
Then click on the “Advanced…” button and select the profile of the network that you would like to use.
Next, save the test settings and have at it. When the test run starts, the emulation of the network you selected will start. When the run finishes, it will stop. Please note that these settings will not work on Load Test. Selecting the networks to emulate for load tests works the same way it did in VS2008.
Can I get a more detailed explanation?
Absolutely…. However, you will have to wait. In the coming weeks, I will be creating a few walkthroughs that better illustrate how to use network emulation during Unit Testing and Load Testing. I will also show you how to create a new profile that allows you to customize the networks that you want to simulate.
Try it out and let me know
If you haven’t done so already, please download the Beta1 build of Visual Studio Team System 2010. Then try out the network emulation and let me know what you think. We would love to hear your feedback.
The Visual Studio Team System test platform provides host processes for running tests. In Visual Studio 2008, the host process is only capable of running tests in 32 bit mode. As a 32 bit process, it was limited to running test assemblies on the 32 bit CLR and was limited to 2GB of memory. Now in Visual Studio 2010, we added support for executing tests on the 32 or 64 bit CLR from Visual Studio or on a remote Load Agent. This will allow you to use >2GB of memory (when executing on the 64 bit CLR) in your tests and most importantly test your assemblies on the 64 bit CLR.
Great, so how can I get my tests to execute on the 64 bit CLR?
To get your test assemblies to execute on the 64 bit CLR, you need to do three things:
1. Target your test assembly for the “Any CPU” platform.
- Open the properties of your test assembly, select the “Build” tab and then choose the “Any CPU” option for the Platform target:
2. Set the Host Process Platform flag to “MSIL”.
- Open the Test Settings and select the “Host” Page.
- Choose MSIL for the Host Process Platform.
3. Run the tests on an operating system that supports the 64 bit CLR.
What do I need to know before I try out 64 bit test execution?
Before you try to execute your tests on the 64 bit CLR, there are a few concepts that I need to explain. First, there are four different ways that you can compile your test assembly (“Any CPU”, “x64”, “x86”, and “Itanium”). Out of these four ways that you can compile your assembly, we really only support two (“Any CPU” and “x86”). Next, in order to support forcing a test to execute on the 32 bit CLR when the 64 bit CLR is available, we introduced a concept of a Host Process Platform flag. This flag has two settings (“MSIL” and “x86”) and allows you to specify how you would like your tests to be executed.
|
MSIL |
This means that if the host process (Visual Studio or Load Agent) is running on a 64 bit operating system and the 64 bit CLR is available and nothing else is preventing the assembly from running in 64 bit mode, the test assembly will execute on the 64 bit CLR. If any of these things are not in place, then the assembly will execute on the 32 bit CLR.
Note: This does not guarantee that it will execute on the 64 bit CLR. |
|
x86 |
This means that the test assembly will always execute on the 32 bit CLR regardless of whether or not the 64 bit CLR is available.
Note: This is the default setting. |
In Visual Studio 2008, tests were executed in the VSTestHost.exe process or the QTAgent.exe process. In Visual Studio 2010, this has changed. All test assemblies are now executed in either the QTAgent32.exe or the QTAgent.exe process. Which process is determined by a combination of the Host Process Platform flag and the “bitness” of the Machine OS. The following table describes which process will be launched and on which version of the CLR that the test assembly will be executed.
|
Machine OS |
Host Process Platform Flag |
Test Host Process |
CLR |
|
32 Bit |
x86 |
QTAgent32.exe |
32 Bit |
|
MSIL |
|
64 Bit |
x86 |
QTAgent32.exe |
32 Bit |
|
MSIL |
QTAgent.exe |
64 Bit if available, otherwise 32 Bit. |
There is a Gotcha!
As I stated earlier, we really only support test assemblies that target the “Any CPU” or “x86” platforms. Under certain conditions, you may be able to run test assemblies that specifically target the “x64” platform. However, for certain test types (unit tests in particular), MSTest.exe will need to load the test assembly into its own process. Since MSTest.exe is a 32 bit process, the test assembly needs to be able to load into a 32 bit process (in other words it must target the x86 or “Any CPU” architecture) to load properly. This means that if you specifically target the x64 platform and you are using one of these test types, regardless of the setting of the Host Process Platform flag, the test assembly will fail to load and the tests will be aborted. Additionally, because the JET database is and remains a 32 bit application, CSV (as well as any data sources that require the JET database) are not supported when executing test assemblies on the 64 bit CLR.
Try it out and let me know
If you haven’t done so already, please download the Beta1 build of Visual Studio Team System 2010. Then try out the 64 bit test execution and let me know what you think. We would love to hear your feedback.
Where is the 64 bit Load Agent Installer?
Although there is not a 64 bit installation for the Load Agent, fear not as the binaries that enable 64 bit test execution are installed with the standard 32 bit installer for the Team Test Load Agent.
Generic tests are a means to provide test authors with a mechanism to tests created using third party frameworks. However, generic tests can also be used to run scripts such as VBScript, JavaScript and even batch files.
After loading a test project with many tests in one class or an assembly, the IDE may respond slowly. This will only occur if there are quite a few tests in one assembly ( > 2000 unit tests) or if there are an unusually high number of tests inside of one class ( > 500). Depending upon the speed of your machine, these numbers can vary. The reason for this is that we attempt to discover all of the test methods in a class upon certain events that are generated in the IDE. For example: when you create a method inside of a class, the IDE generates an event to let us know the method and its signature. Unfortunately, in Visual Studio 2008, instead of notifying us that a particular method has changed, the IDE is signaling that the entire type has changed. Since we do not know which method has changed in this type, we are forced to rescan the entire type. When a class has a large number of methods, this can take more time than is acceptable and may sometimes make the IDE appear as if it is not responding.
Solutions
There are a few options that you can try to relieve these symptoms, please select the one that is most appropriate for your situation.
Reduce the number of unit tests in one single class by using Partial Classes:
When iterating through the methods of a class using reflection, the methods that show up are only the ones within the portion of the partial class that is being examined. For example: When a user changes a method within a class, the IDE tells us that the type has changed and passes us a reference to that type. When we iterate through that class, we only get the methods that appear in the definition of that type within the file that was changed. This means that if you had a class definition like the following:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace ClassLibrary1
{
public class Class1
{
public void Method1()
{
}
public void Method2()
{
}
public void Method3()
{
}
}
}
With minimal effort, you could separate it into two different files and reduce the effect of the background discovery of test methods by moving some of the methods into a separate file. The following class definitions show how the above referenced class was split into two partial class definitions.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace ClassLibrary1
{
public partial class Class1
{
public void Method1()
{
}
public void Method3()
{
}
}
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace ClassLibrary1
{
public partial class Class1
{
public void Method2()
{
}
}
}
Reduce the number of unit tests in one assembly:
This method reduces the effect of the background discovery of test methods by refactoring an assembly and separating test classes into smaller assemblies. Depending upon the complexity of the assembly, this can take some work and may be more than you wish to undertake.
Disable background discovery of test methods: (Available in VS2008 SP1 only)
1. Select the Tools | Options | Test Tools | Test Project menu option.
2. On the Test Project dialog (figure 1.1), select the “Disable background discover of test methods” checkbox option and press ok.
This will disable the background discovery of test methods and will dramatically speed up the IDE. However, since this is global option, this will disable the test method discovery on every project. If you wish to re-enable the discovery, simply load the Test Project dialog again and deselect the “Disable background discover of test methods” checkbox option.
Figure 1 - Test Project Dialog
