Ed Glas's blog on VSTS load testing : Visual Studio Load Controller

Getting Started with Visual Studio 2010 Beta 2

edglas — Tue, 20 Oct 2009 18:59:35 GMT

As you are probably already aware, Visual Studio Beta 2 is now available for download, see the landing page here.

For load testing, you will need to download and install:

Visual Studio Ultimate Beta 2 (x86)

Visual Studio 2010 Team Agents Beta 2 (x86)

Ultimate is the new equivalent of Team Suite, and contains Web performance tests and load tests.

Team Agents contains the controller and agents. Note that the x86 agent contains both 32 and 64 bit assemblies, so if you are looking for the 64 bit version of the load agent, this is the download you need.

As I posted here, we have moved to a virtual user license model. Ultimate is limited to 250 virtual users and one core of the CPU. To generate more than 250 users and use all cores on your machine, you will need to install a virtual user pack.

The most straight forward way to do this is to install a controller and agent (or multiple agents) on the same or separate machines.

We have re-architected the controller and agent setups such that setup does little more than lay down the assemblies, while a post-setup configuration tool does the heavy lifting to get the machine configured. The configuration steps are the ones that typically fail, so rather than having to reinstall the entire product and go back through all the configuration steps, if one part fails you only need to reconfigure that part. We have also enabled command line versions of both setup and the configuration tools.

Here is the controller config tool with options called out:

So you will need to fill in the controller service user using a user with counter collection access to your systems under test (same as in VS 2008), and then configure the load test database in the “Configure for load testing” section.

To add virtual users, start a Visual Studio Command prompt as Administrator (on Vista or Win7, just type Visual Studio Command Prompt in the search window, on XP you’ll find this under the Visual Studio group in the start menu). Then run this command line:

TestControllerConfig licenses /addkey:V10000

This will license the controller for 10,000 virtual users. Note that this license is temporary, and will only work in beta2. For RTM, we will only offer the 1,000 user license pack. If you have a Volume License, you will be able to get your license key from the volume licensing site, and then enter a count of licenses you want to install.

Once you have a virtual user license installed, that license can also be used for local runs (that is, the runs do not have to go through the controller).

If you just want to do local runs, and not use the controller and agent, you can run this from the command line to install the virtual user license:

VSTestConfig licenses /addkey:V10000

This enables you to exceed 250 users for local runs, and unlocks all the cores on your CPU.

Ed.

Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010

edglas — Tue, 20 Oct 2009 14:52:00 GMT

In Visual Studio 2010, we are changing our load test licensing model. In VS 2005 and VS 2008, we shipped the load agent, which was licensed by CPU on the machine generating load. In Visual Studio 2010 we are changing our license model to a virtual user based model. This new license model offers a number of advantages for you, which is ultimately why we chose to make the change:

Still way less expensive than our competition.
Enables better purchasing planning and allows an apples to apples price comparison with competitors.
Enables more flexible machine allocation for load generation.
Enables you to use agents for more than just load testing.

Still Way Less Expensive than our Competitors

Your initial reaction may be “Groan – that means that Microsoft is going to be just as expensive as those other tools!” But that is not the case at all, we were careful to maintain the per-virtual-user price we set out initially with VS 2005. With VS 2005 and VS 2008 load agents, the price was around $5,000 per load agent CPU, and our general guidance was that for a typical load test expect around 1,000 virtual users per CPU. We are not yet disclosing our pricing yet, but I can tell you our goal is to stay in this same ball park with VS 2010.

Better Planning and Comparison

With the VS 2008 load agent, we often get questions from customers doing purchasing planning asking “How many load agents do we need to purchase? We are planning to test up to 5,000 virtual users.” To which our answer with VS 2008 was “we do not know, it depends on your application and test scripts.” Of course this it difficult to know how many agents to purchase until actually doing testing, which was often too late. With virtual user licenses, purchasers will know exactly how many licenses to buy.

The old CPU-based license model also makes it very difficult to compare prices with competitive solutions. Most competitive products are licensed by virtual user and data collectors. If you know how many virtual users you want to test to, but do not know how many agents you will need to generate that number of users, it is impossible to make a comparison. With the Visual Studio Load Test Virtual User Packs 2010, if you know the virtual user count you want to test to, you will know exactly how many licenses to purchase, and be able to compare directly with competitive solutions you may consider.

Enables More Flexibility with Load Generators

The virtual user license offers flexibility with the machines you use as load generators.

With VS 2008, to get the full value of the per-CPU license, you had to run on fast CPUs, often requiring new hardware purchases. Also the license was locked to a particular machine, restricting which machines you could use as load generators. With Visual Studio 2010, you can now run virtual users on older hardware with no “penalty”, as you can line up as many older machines as you wish to generate the target number of virtual users. Also, the virtual user license is now tied to the controller, not the agent machine, so you are allowed to move agent machines in and out of the agent pool.

For some tests, you may run out of some other resources before running out of CPU. The most common case for this in VS 2008 was load agents running out of memory before CPU (especially for web tests with very large post bodies or very large responses), which prevented customers from realizing the full value of their license. With the virtual user license, if this happens you can simply add more machines to the agent pool to achieve the load you are targeting.

Visual Studio 2010 Agents: More than just Load Generators

With the high cost of the agent license, we did not have a viable solution for distributed functional testing. In Visual Studio 2010, agents are now included at no extra cost with VS Premium, Ultimate, and Test Elements. We’ve added Data Collection facility to agents, and added new UI automation test type, which opens up a bunch of new scenarios for using agents.

What if I already have VS 2008 Load Agents?

If you have purchased VS 2008 load agents with software assurance, you will be able to migrate your existing licenses to a commiserate set of Virtual User License Packs. We will iron out the details and mechanics of this once we get closer to RTM.

Conclusion

You can see there are many advantages to you with the new licensing model.

Ed.

VS 2008 SP1 Beta is available

edglas — Wed, 28 May 2008 00:39:16 GMT

Incase you haven't seen this, the beta for Visual Studio 2008 SP1 is now available.

We've put a lot of work into this SP1 release, mostly around bug fixes for bugs that you reported on the forums and through tech support (mostly forums), as well as bugs our test and dev teams reported. We've also managed to slip a few extra features into the release.

The download is here.

I highly recommend you install the beta and report back any issues ASAP so we can be sure to get them rolled into the final release.

One issue to be aware of is if you are running using test agents, the agent, controller, and all clients must match the SP1 version. Unfortunately the controller and agent uses .NET remoting object serialization, which makes it very sensitive to any changes in the objects that are being passed back and forth.

New features (we'll be blogging about these in separate posts):

Priority	Team Level 1	Title
1	Load Test Team	Support random extraction of matches using extract text rule
2	Load Test Team	DCR: Add Web test and load test extensibility points to support partners
1	Test Infrastructure Team	Enable Dual NIC controller scenarios

Bugs fixed (I’ve highlighted ones you may find interesting):

Priority	Team Level 1	Title
1	Load Test Team	VS crashes due to InvalidOperationException in load test monitor when running load test that hits many unique URLs
1	Load Test Team	Extra threads created when running load test containing unit tests on multi-agent rig causing out of memory exception
1	Load Test Team	Loadtest DB: Installing the controller should not drop an existing database.
2	Load Test Team	Goal-based profile - user load adjustment is based on 'ideal' user load instead of actual user load
2	Load Test Team	Cookies: We're not handling cookie expirations properly.
2	Load Test Team	LoadTest: When OutOfMemory exception stops load test on one agent in multi-agent test some load test results data is lost.
2	Load Test Team	Incorrect requests being recorded for Ajax site
2	Load Test Team	Form fields can be recorded twice
2	Load Test Team	Loadtest: Test times are inaccurate at less than 15 milliseconds.
2	Load Test Team	DTS: Web Recorder does not record request when Content type is missing
2	Load Test Team	DTS: Webtest Recorded in 2005 will not record in 2008
2	Load Test Team	WebTest.WebProxy property does not work
2	Load Test Team	Hidden Field Name appears as GUID when uploading file to sharepoint
2	Load Test Team	Custom Load Profiles: Having a MinUserCount > MaxUserCount throws the wrong exception.
2	Load Test Team	Performance overhead of new PostPage event is too high
2	Load Test Team	Webtest: Think times set on the first request are ignored.
2	Load Test Team	Orcas Web Test: View response in html editor does not work
2	Load Test Team	LoadTestPlugin can't set the user load higher than the max user load for the scenario
2	Load Test Team	Test View has huge delays
2	Load Test Team	Visual Studio crashes when calling AddCommentToResult in a WebTestRequestPlugin
2	Load Test Team	Web Browser tab in Web Test result shows broken page on JPN OS.
2	Load Test Team	Can't set Outcome on web test or request
2	Load Test Team	Copy/Paste from the webtest playback window on virtual PC crashes orcas.
2	Load Test Team	The default for the load test run setting "Run unit tests in application domain" should be true
2	Load Test Team	File upload parameters with no actual file causes the web test engine to throw an exception.
2	Load Test Team	Wininet recorder: ID field not correlated properly when uploading files to sharepoint.
2	Load Test Team	Copying a Load profile from the UI yields a cast error.
2	Load Test Team	Load test user load ramps up to quickly during warmup period
2	Load Test Team	Load Test should honor the ApartmentState setting in the Test Run Config file
2	Load Test Team	Thinktime that's set via plugin code is ignored
2	Load Test Team	LoadTest aborts when test throws an exception with a quote in the stack trace
2	Load Test Team	Loadtest Analyzer: Error when trying to bring up the loadtest analyzer after a test that stops very quickly.
2	Load Test Team	LoadTest: Goal profile performance counter should be sampled at least every 5 seconds
2	Load Test Team	WebTest: User code can't set values in the WebTestContext if the property has been set in an env variable on in the LoadTestContext
2	Load Test Team	Network emulation doesn't work correctly with compression
1	Test Infrastructure Team	Loc: String "Whidbey" appears in Microsoft.VisualStudio.QualityTools.Resource.dll
1	Test Infrastructure Team	VSPro 2008 won't show test lists created on VSTS2008
1	Test Infrastructure Team	Test runs with load test and any other test type get stuck
1	Test Infrastructure Team	CodeGen for IIS website code failed in generating the private accessor
1	Test Infrastructure Team	Upgrade a Whidbey test project in VSTD2008 will fail
1	Test Infrastructure Team	SP1 patch setup fails on VSTS with controller/agent installed
1	Test Infrastructure Team	Fix perf tests that are failing from time to time in QFE (TechEase ticket 143089)
1	Test Infrastructure Team	Publishing load tests fails in TFS SP1
2	Test Infrastructure Team	Multi-agent: Could not start two load tests in paralell on different agents
2	Test Infrastructure Team	Watson - unhandled TeamFoundationServerUnauthorizedException crashed devenv.exe
2	Test Infrastructure Team	Out of memory error when publishing test cases using MSTest
2	Test Infrastructure Team	MSTest can't run codedwebtests when alone errors with assembly is not trusted
1	Unit Test Team	Invalid AppDomain.CurrentDomain.BaseDirectory in VS 2008 Unit Test Runner
1	Unit Test Team	Unit Testing: App Domain naming creates issues with performance counters
1	Unit Test Team	Unit Test Generation fails for default WCF service
2	Unit Test Team	Ordered tests created in VS2005 are not automatically upgraded in VSTD 2008
2	Unit Test Team	DTS: IDE in 2008 responds slow compared to IDE in 2005 in a test Project with over 9000 unit test
2	Unit Test Team	VSTestHost specificies LoaderOptimization.MultiDomain as the default, causes various problems
2	Unit Test Team	Unit testing fails in asp.net hosted in iis

I hope you find this useful, and by all means please download and use SP1 beta!

Ed.

84,248 Ads per second!

edglas — Mon, 17 Mar 2008 23:59:13 GMT

The adCenter Display Ads Platform team recently ran a large test using VSTS. The goal of this testing was to validate the configuration and stability of 100+ servers under load.

The test used one Controller and 42 agents to produce the necessary load. The controller was also hosting a full install of SQL Server which acted as the result store.

Controller and Agent hardware information

Processor: 4 x AMD Opteron 280 (2405 MHz)
Memory: 4 GB RAM
Network: GigE NIC

High level test summary

Test Duration                               12 hours
Total requests generated             3,639,518,000 (3.63 Billion requests)
Average requests/sec                   84,248
Average Requests/Test                1
Failed Tests                                  0
Sampling rate                               30 secs
Average agent Proc utilization    45%

To minimize the risk they started off with one controller and one agent, then increasing to 8 agents, then going to 42.

They estimate they could have produced at least 50% more requests with 42 agents but restricted the load to avoid any agent overloading.

A note from the customer: Overall it was a happy experience running a test of this large scale. Thanks to VSTS for making load testing a pleasure!

Ed.

Updated Content for web tests and load tests

edglas — Mon, 17 Dec 2007 20:36:00 GMT

I've updated my collection of links for info on web tests and load tests.

I had been publishing this to a sticky post on the forum, but ran into the size limit so I'm moving it to my blog.

The page is here. Do let me know if you hit any links that are broken, the content gets moved around from time to time.

Ed.