Lonny Kruger's Blog on Visual Studio Team System Load Testing

Switching Themes. Welcome to “It’s a Build Thing … “

lkruger — Fri, 03 Jun 2011 04:05:40 GMT

In the past, my focus has been on the Testing Tools provided in Visual Studio. However, since I am now working on the TFS build team, I think a switch is in order. I’m not sure about everyone else but I, personally, enjoy learning by looking at solutions to real world problems and applying those lessons to my future projects. So I think that I am going to try to incorporate this into my future posts. I have a few ideas in mind, but I am also looking for some topics that you would like to see covered.

Don’t be shy … Post Your Topic Suggestions and Questions. I will try my very best to answer all questions asked and if possible (and appropriate) provide an explanation of why it is that way.

Working around the AnyCPU Limitation with Visual Studio Unit Tests

lkruger — Fri, 03 Jun 2011 03:39:59 GMT

When running unit tests in Visual Studio 2010, you can run them in 64 bit mode, however, the assembly containing the tests MUST be compiled as AnyCPU. This is because the test discovery process happens inside of Visual Studio (or inside MSTest.exe if you are using the command line). Since both of these applications are 32 bit only, they cannot load the 64 bit assembly (unless it is compiled as AnyCPU) and consequently, you cannot run your unit tests.

Fortunately, thanks to the fact that both Visual Studio and MSTest.exe only need to discover these test and the way JIT’ing works, you can work around this problem. As a work-around, you can create a stub test assembly (ANYCPU) that will reference the 64 bit only assembly. This stub test assembly should contain all of the test methods that you want enumerated in the Visual Studio or MSTest.exe, but would not call any methods from a 64 bit only assembly directly. Instead, create a another method to call the methods that are inside of the 64 bit only assemblies. Because these methods will not be JIT’d until the test is actually run, this will allow the test methods to be enumerated by a 32 bit process, which in turn will allow you to run your 64 bit only tests.

For Example:

private void DoTest1()

{

SixtyFourBitClass.MethodToTest();

}

[TestMethod]

public void Test1()

{

DoTest1();

}

TFS Build Service Host and the Local System Account

lkruger — Fri, 03 Jun 2011 03:14:29 GMT

This is going to be a very short post because there is not a whole lot to say other than if you want to connect to a TFS AT that is on a different machine … “DON’T USE IT.”

OK … A Little More Information

There has been a lot of reports of the build service host not being able to connect to the TFS AT when the AT is on a different machine. This is because the “Local System” credentials do not have all of the appropriate permission needed to successfully connect and therefore fail. For the most part, we prevent you from using these credentials when they would otherwise fail. However, there is at least one way that I know of that the user will be able to use these credentials when they shouldn’t and unfortunately, each day more people are finding themselves using these credentials and being unable to have their build service host connect.

Unlocking multiple CPUs for load tests run from Visual Studio

lkruger — Fri, 03 Jun 2011 02:50:30 GMT

Before Visual Studio 2010, load tests run from Visual Studio were limited to run on only one CPU. For the most part, this is the same in Visual Studio 2010. However …, if you purchase the “Visual Studio Load Test Virtual User Pack 2010”, not only does it give you a license to run more virtual users, it also “unlocks” the limitation of running load tests on a single CPU. For more information about obtaining Virtual User licenses, please see this MSDN article.

Running Load Tests in Visual Studio and Virtual User Licenses

lkruger — Fri, 03 Jun 2011 02:41:40 GMT

When running load tests in Visual Studio 2010, you might come across something that mentions the need for “Virtual User” licenses. This is because, with Visual Studio 2010, each simultaneous “simulated user” needs a Virtual User License. must be licensed. If you try to run a load test that requires more virtual users than you have licenses, the load test will fail and return a message indicating that you do not have enough virtual user licenses to run the test.

Fortunately, each copy of Visual Studio 2010 Ultimate comes with an “implied” license for 250 virtual users. This will allow you to run load tests with up to 250 “simultaneous” virtual users. However …, this license is only good when you run load tests through Visual Studio using a “Local” test agent. The minute you connect the tests to a remote test agent, the “implied” licenses go away. In order to run a load test against a remote test agent, you must purchase a “Visual Studio Load Test Virtual User Pack 2010.” For each pack that you purchase, you will be licensed for an additional 1000 virtual users and can use them in local runs done in Visual Studio or remote test runs outside of Visual Studio.

Stand-Alone Network Emulation and CodePlex

lkruger — Wed, 07 Apr 2010 20:31:13 GMT

In my post “Creating a Stand-Alone Network Emulator using VS2010 – Beta 1, I showed you how to create a stand alone network emulator using the network emulation functionality introduced in the Beta 1 release of VS2010. Since then, the API for Network Emulation has gone through several changes and long story short, the API for Beta 1 will not work for RC or RTM. To make things a bit easier, I have created a new “Stand-Alone” Network Emulator UI (NEUI) that will allow you to take advantage of the Network Emulation features in VS2010 without having to fire up VS2010 and start a unit or load test. I have posted the source for this project on CodePlex for everyone to enjoy :). For now, it is in the project “Web and Load Test Plugins for Visual Studio Team Test”, but my hope is that it will gain enough community support and involvement that it will warrant going through the process of creating and maintaining it as a separate project.

Currently, the NEUI project is a simple UI that:

uses WPF
allows the user to select one Network Profile to emulate a specific network.
when minimized, displays in the system tray
when in the system tray, allows for the starting, stopping of network emulation and the selection of the network profile.

Please feel free to download and use the emulator. Also, if you feel strongly enough, feel free to suggest or contribute new features.

blogs.msdn.com Tags: VSTS2010, Visual Studio 2010, Network Emulation, NEWT, Load Test, Unit Test, Testing, Team System

Creating a Stand-Alone Network Emulator using VS2010 – Beta 1 Release

lkruger — Wed, 24 Jun 2009 23:22:00 GMT

In the Beta 1 release of Visual Studio 10, we released “True Network Emulation.” With this release, you can simulate different types of networks when running tests (such as a unit test or load test). One thing that is missing, however, is the ability to easily emulate different networks when not running tests using the Visual Studio Team System framework. For example, I might want to just take an application and see how it performs under the restrictions of a low bandwidth connection. I might not want to create a test for this, I just want to play with it and see what happens. To accomplish this “out of the box”, I need to create a unit test that sleeps for a very long period of time and turn on network emulation. While this is fairly easy, this seems a bit awkward. This post will focus on creating a solution that will make this process as simple as pressing a start button when I want to start emulating and pressing a stop button when I want to quit.

Create a stub application

The first step in creating the network emulator is to open Visual Studio 2010 and create an Empty “Windows Forms Application.”

The next step is to copy the assembly that contains the Network Emulation API. This assembly is named “userapi.dll” and is located in %Program Files%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\DataCollectors\x86 (or x64 for 64 bit systems). Copy this file and place it into the solution directory, then add it to the solution and change the “Copy to Output Directory” to “Copy if newer”.

Create a class to hold the Simplified Network Emulation API

In this release of Visual Studio, the network emulation API is native code. Therefore we have a limited amount of ways that we can access the API from managed code. In this sample, we will make calls into the API using PInvoke. In order to do this, we will have to create some definitions for each of these methods before we can use them. So create a new class called NativeNetworkEmulationAPI and add the following declarations:

using System;
using System.Runtime.InteropServices;

namespace StandaloneNetworkEmulation
{
    public class NativeNetworkEmulationAPI
    {
        [DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
        public static extern int NEGetVersion(ref uint pdwVersion);

[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEInitialize(ref uint pdwVersion, ref IntPtr phEmulator);

[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NECleanup(IntPtr hEmulator);

[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEStartEmulation(IntPtr hEmulator);

[DllImport("userapi.dll", SetLastError = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int NEStopEmulation(IntPtr hEmulator, uint dwTimeout, int fForce);

        [DllImport("userapi.dll", SetLastError = true, CharSet = System.Runtime.InteropServices.CharSet.Auto)]
        public static extern int NELoadProfile(IntPtr hEmulator,
            [MarshalAs(UnmanagedType.LPWStr)]
            string profileName);
    }
}

Copy and Modify the Predefined Network Profiles

The next step is to copy the predefined network profiles from VS2010 to our solution so that we can modify them for use in our stand-alone emulator. To do this, lets create a folder in the solution called “Profile” and copy all of the predefined network profiles (located in %Program Files%\Microsoft Visual Studio 10.0\Common7\IDE\Templates\LoadTest\Networks) to this folder and set the Build Action = “None” for each profile. Once this is done, we need to slightly modify each profile in order to get the emulator to read the profile properly. To do this, we need to load each profile and remove the two lines that contain the name “NetworkEmulationProfile” (This should be the first and last lines in the profiles). In the “3G.network” file, the two lines removed are:

<NetworkEmulationProfile name="3G" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
</NetworkEmulationProfile

After modifying this profile, the top level xml node should be <Emulation>. One final note is to just delete the LAN.network profile as it will not work with this emulator. It is a special case that we use in our framework to indicate that we want to use the full bandwidth of the available network (in other words, don’t simulate anything).

Complete the Main Form in the application

Next, lets add the start and stop buttons to the main form and a combobox that will allow us to select a profile to use for the emulation.

Add member variables to the form:

   1: private IntPtr m_emulatorHandle;

Add a Load event handler to the form and add the following code:

   1: NativeNetworkEmulationAPI.NELoadProfile(m_emulatorHandle, Path.Combine("Profiles",    2:     m_networkProfiles.SelectedText) + ".Network");   3: if (0 == NativeNetworkEmulationAPI.NEStartEmulation(m_emulatorHandle))   4: {   5:     startButton.Enabled = false;   6:     stopButton.Enabled = true;   7: }   8: else   9: {  10:     MessageBox.Show("There was an error starting the emulation.");  11: }

Add a FormClosed event handler to the form and add the following code:

   1: if (0 != NativeNetworkEmulationAPI.NECleanup(m_emulatorHandle))   2: {   3:     MessageBox.Show("There was an error cleaning up the emulator.");   4: }

Add a Click event handler to the start button and add the following code:

   1: NativeNetworkEmulationAPI.NELoadProfile(m_emulatorHandle, Path.Combine("Profiles",    2:     m_networkProfiles.SelectedText) + ".Network");   3: if (0 == NativeNetworkEmulationAPI.NEStartEmulation(m_emulatorHandle))   4: {   5:     startButton.Enabled = false;   6:     stopButton.Enabled = true;   7: }   8: else   9: {  10:     MessageBox.Show("There was an error starting the emulation.");  11: }

Add a Click event handler to the stop button and add the following code:

   1: if (0 == NativeNetworkEmulationAPI.NEStopEmulation(m_emulatorHandle, 2000, 1))   2: {   3:     startButton.Enabled = false;   4:     stopButton.Enabled = true;   5: }   6: else   7: {   8:     MessageBox.Show("There was an error stopping the emulation.");   9: }  10: startButton.Enabled = true;  11: stopButton.Enabled = false;

Conclusion

In this article, we created a sample network emulator using the API provided in the VS2010 Network Emulation feature. Although this example was simple, there are a lot of things that can be done using this API, such as packet tracing, filtering, etc. Keep an eye on this blog and in the future, I will show you how to customize the profiles to perform some of the advanced features of network emulation.

Try it out and let me know

If you haven’t done so already, please download the Beta1 build of Visual Studio Team System 2010. Then try out this network emulator and let me know what you think. We would love to hear your feedback.

Introducing True Network Emulation in Visual Studio 2010

lkruger — Mon, 08 Jun 2009 23:06:00 GMT

With the Beta 1 release of Visual Studio 2010, we have introduced true network emulation for load tests and pretty much every other test type we support.

What is True Network emulation?

My definition of true network emulation might be different than what others might think, but I characterize true network emulation as the ability to simulate network conditions by directly manipulating the network packets. The Network Emulation that we provide in Visual Studio 2010 is a software-based emulator which can emulate the behavior of both wired and wireless networks using a reliable physical link, such as an Ethernet. A variety of network attributes are incorporated into the VS2010 Network Emulation model, including round-trip time across the network (latency), the amount of available bandwidth, queuing behavior, packet loss, reordering of packets, and error propagations. VS2010 Network Emulation also provides flexibility in filtering network packets based on IP addresses or protocols such as TCP, UDP, and ICMP.

Why do I care about network emulation?

VS2010 Network Emulation can be used by network-based application developers and testers to emulate a desired test environment, assess performance, predict the impact of change, or make decisions about technology optimization. When compared to hardware test beds, the Network Emulation provided by Visual Studio Team System is a much cheaper and more flexible solution.

How do I use the Network Emulation that is provided in Visual Studio 2010?

The easiest way to use network emulation in Visual Studio 2010 is to create a test project and modify the test settings (formerly run configuration). On the “Execution Criteria” page, select and enable the Network Emulation collector.

Then click on the “Advanced…” button and select the profile of the network that you would like to use.

Next, save the test settings and have at it. When the test run starts, the emulation of the network you selected will start. When the run finishes, it will stop. Please note that these settings will not work on Load Test. Selecting the networks to emulate for load tests works the same way it did in VS2008.

Can I get a more detailed explanation?

Absolutely…. However, you will have to wait. In the coming weeks, I will be creating a few walkthroughs that better illustrate how to use network emulation during Unit Testing and Load Testing. I will also show you how to create a new profile that allows you to customize the networks that you want to simulate.

Try it out and let me know

If you haven’t done so already, please download the Beta1 build of Visual Studio Team System 2010. Then try out the network emulation and let me know what you think. We would love to hear your feedback.

Visual Studio Team Test Load Agent Goes 64 Bit!

lkruger — Mon, 08 Jun 2009 21:19:00 GMT

The Visual Studio Team System test platform provides host processes for running tests. In Visual Studio 2008, the host process is only capable of running tests in 32 bit mode. As a 32 bit process, it was limited to running test assemblies on the 32 bit CLR and was limited to 2GB of memory. Now in Visual Studio 2010, we added support for executing tests on the 32 or 64 bit CLR from Visual Studio or on a remote Load Agent. This will allow you to use >2GB of memory (when executing on the 64 bit CLR) in your tests and most importantly test your assemblies on the 64 bit CLR.

Great, so how can I get my tests to execute on the 64 bit CLR?

To get your test assemblies to execute on the 64 bit CLR, you need to do three things:

1. Target your test assembly for the “Any CPU” platform.

Open the properties of your test assembly, select the “Build” tab and then choose the “Any CPU” option for the Platform target:

2. Set the Host Process Platform flag to “MSIL”.

Open the Test Settings and select the “Host” Page.
Choose MSIL for the Host Process Platform.

3. Run the tests on an operating system that supports the 64 bit CLR.

What do I need to know before I try out 64 bit test execution?

Before you try to execute your tests on the 64 bit CLR, there are a few concepts that I need to explain. First, there are four different ways that you can compile your test assembly (“Any CPU”, “x64”, “x86”, and “Itanium”). Out of these four ways that you can compile your assembly, we really only support two (“Any CPU” and “x86”). Next, in order to support forcing a test to execute on the 32 bit CLR when the 64 bit CLR is available, we introduced a concept of a Host Process Platform flag. This flag has two settings (“MSIL” and “x86”) and allows you to specify how you would like your tests to be executed.

MSIL

This means that if the host process (Visual Studio or Load Agent) is running on a 64 bit operating system and the 64 bit CLR is available and nothing else is preventing the assembly from running in 64 bit mode, the test assembly will execute on the 64 bit CLR. If any of these things are not in place, then the assembly will execute on the 32 bit CLR.

Note: This does not guarantee that it will execute on the 64 bit CLR.

x86

This means that the test assembly will always execute on the 32 bit CLR regardless of whether or not the 64 bit CLR is available.

Note: This is the default setting.

In Visual Studio 2008, tests were executed in the VSTestHost.exe process or the QTAgent.exe process. In Visual Studio 2010, this has changed. All test assemblies are now executed in either the QTAgent32.exe or the QTAgent.exe process. Which process is determined by a combination of the Host Process Platform flag and the “bitness” of the Machine OS. The following table describes which process will be launched and on which version of the CLR that the test assembly will be executed.

Machine OS

Host Process Platform Flag

Test Host Process

CLR

32 Bit

x86

QTAgent32.exe

32 Bit

MSIL

64 Bit

x86

QTAgent32.exe

32 Bit

MSIL

QTAgent.exe

64 Bit if available, otherwise 32 Bit.

There is a Gotcha!

As I stated earlier, we really only support test assemblies that target the “Any CPU” or “x86” platforms. Under certain conditions, you may be able to run test assemblies that specifically target the “x64” platform. However, for certain test types (unit tests in particular), MSTest.exe will need to load the test assembly into its own process. Since MSTest.exe is a 32 bit process, the test assembly needs to be able to load into a 32 bit process (in other words it must target the x86 or “Any CPU” architecture) to load properly. This means that if you specifically target the x64 platform and you are using one of these test types, regardless of the setting of the Host Process Platform flag, the test assembly will fail to load and the tests will be aborted. Additionally, because the JET database is and remains a 32 bit application, CSV (as well as any data sources that require the JET database) are not supported when executing test assemblies on the 64 bit CLR.

Try it out and let me know

If you haven’t done so already, please download the Beta1 build of Visual Studio Team System 2010. Then try out the 64 bit test execution and let me know what you think. We would love to hear your feedback.

Where is the 64 bit Load Agent Installer?

Although there is not a 64 bit installation for the Load Agent, fear not as the binaries that enable 64 bit test execution are installed with the standard 32 bit installer for the Team Test Load Agent.

Did you know?

lkruger — Thu, 12 Jun 2008 02:02:00 GMT

Generic tests are a means to provide test authors with a mechanism to tests created using third party frameworks. However, generic tests can also be used to run scripts such as VBScript, JavaScript and even batch files.

Slow Performance in Visual Studio 2008 IDE after opening Test Project

lkruger — Wed, 04 Jun 2008 00:07:00 GMT

After loading a test project with many tests in one class or an assembly, the IDE may respond slowly. This will only occur if there are quite a few tests in one assembly ( > 2000 unit tests) or if there are an unusually high number of tests inside of one class ( > 500). Depending upon the speed of your machine, these numbers can vary. The reason for this is that we attempt to discover all of the test methods in a class upon certain events that are generated in the IDE. For example: when you create a method inside of a class, the IDE generates an event to let us know the method and its signature. Unfortunately, in Visual Studio 2008, instead of notifying us that a particular method has changed, the IDE is signaling that the entire type has changed. Since we do not know which method has changed in this type, we are forced to rescan the entire type. When a class has a large number of methods, this can take more time than is acceptable and may sometimes make the IDE appear as if it is not responding.

Solutions

There are a few options that you can try to relieve these symptoms, please select the one that is most appropriate for your situation.

Reduce the number of unit tests in one single class by using Partial Classes:

When iterating through the methods of a class using reflection, the methods that show up are only the ones within the portion of the partial class that is being examined. For example: When a user changes a method within a class, the IDE tells us that the type has changed and passes us a reference to that type. When we iterate through that class, we only get the methods that appear in the definition of that type within the file that was changed. This means that if you had a class definition like the following:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ClassLibrary1

{

public class Class1

{

public void Method1()

{

}

public void Method2()

{

}

public void Method3()

{

}

With minimal effort, you could separate it into two different files and reduce the effect of the background discovery of test methods by moving some of the methods into a separate file. The following class definitions show how the above referenced class was split into two partial class definitions.

Class1.cs

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ClassLibrary1

{

public partial class Class1

{

public void Method1()

{

}

public void Method3()

{

}

Class1a.cs

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ClassLibrary1

{

public partial class Class1

{

public void Method2()

{

}

Reduce the number of unit tests in one assembly:

This method reduces the effect of the background discovery of test methods by refactoring an assembly and separating test classes into smaller assemblies. Depending upon the complexity of the assembly, this can take some work and may be more than you wish to undertake.

Disable background discovery of test methods: (Available in VS2008 SP1 only)

1. Select the Tools | Options | Test Tools | Test Project menu option.

2. On the Test Project dialog (figure 1.1), select the “Disable background discover of test methods” checkbox option and press ok.

This will disable the background discovery of test methods and will dramatically speed up the IDE. However, since this is global option, this will disable the test method discovery on every project. If you wish to re-enable the discovery, simply load the Test Project dialog again and deselect the “Disable background discover of test methods” checkbox option.

Figure 1 - Test Project Dialog

Machine OS	Host Process Platform Flag	Test Host Process	CLR
32 Bit	x86	QTAgent32.exe	32 Bit
32 Bit	MSIL	QTAgent32.exe	32 Bit
64 Bit	x86	QTAgent32.exe	32 Bit
64 Bit	MSIL	QTAgent.exe	64 Bit if available, otherwise 32 Bit.

Lonny Kruger's Blog on Visual Studio Team System Load Testing

Switching Themes. Welcome to “It’s a Build Thing … “

Working around the AnyCPU Limitation with Visual Studio Unit Tests

TFS Build Service Host and the Local System Account

OK … A Little More Information

Unlocking multiple CPUs for load tests run from Visual Studio

Running Load Tests in Visual Studio and Virtual User Licenses

Stand-Alone Network Emulation and CodePlex

Creating a Stand-Alone Network Emulator using VS2010 – Beta 1 Release

Create a stub application

Create a class to hold the Simplified Network Emulation API

Copy and Modify the Predefined Network Profiles

Complete the Main Form in the application

Add member variables to the form:

Add a Load event handler to the form and add the following code:

Add a FormClosed event handler to the form and add the following code:

Add a Click event handler to the start button and add the following code:

Add a Click event handler to the stop button and add the following code:

Conclusion

Try it out and let me know

Introducing True Network Emulation in Visual Studio 2010

What is True Network emulation?

Why do I care about network emulation?

How do I use the Network Emulation that is provided in Visual Studio 2010?

Can I get a more detailed explanation?

Try it out and let me know

Visual Studio Team Test Load Agent Goes 64 Bit!

Great, so how can I get my tests to execute on the 64 bit CLR?

What do I need to know before I try out 64 bit test execution?

Machine OS Host Process Platform Flag Test Host Process CLR 32 Bit x86 QTAgent32.exe 32 Bit MSIL 64 Bit x86 QTAgent32.exe 32 Bit MSIL QTAgent.exe 64 Bit if available, otherwise 32 Bit.

There is a Gotcha!

Try it out and let me know

Where is the 64 bit Load Agent Installer?

Did you know?

Slow Performance in Visual Studio 2008 IDE after opening Test Project

Solutions

Reduce the number of unit tests in one single class by using Partial Classes:

Reduce the number of unit tests in one assembly:

Disable background discovery of test methods: (Available in VS2008 SP1 only)

Machine OS

Host Process Platform Flag

Test Host Process

CLR

32 Bit

x86

QTAgent32.exe

32 Bit

MSIL

64 Bit

x86

QTAgent32.exe

32 Bit

MSIL

QTAgent.exe

64 Bit if available, otherwise 32 Bit.