Ed Glas's blog on VSTS load testing : Load Agent

Getting Started with Visual Studio 2010 Beta 2

edglas — Tue, 20 Oct 2009 18:59:35 GMT

As you are probably already aware, Visual Studio Beta 2 is now available for download, see the landing page here.

For load testing, you will need to download and install:

Visual Studio Ultimate Beta 2 (x86)

Visual Studio 2010 Team Agents Beta 2 (x86)

Ultimate is the new equivalent of Team Suite, and contains Web performance tests and load tests.

Team Agents contains the controller and agents. Note that the x86 agent contains both 32 and 64 bit assemblies, so if you are looking for the 64 bit version of the load agent, this is the download you need.

As I posted here, we have moved to a virtual user license model. Ultimate is limited to 250 virtual users and one core of the CPU. To generate more than 250 users and use all cores on your machine, you will need to install a virtual user pack.

The most straight forward way to do this is to install a controller and agent (or multiple agents) on the same or separate machines.

We have re-architected the controller and agent setups such that setup does little more than lay down the assemblies, while a post-setup configuration tool does the heavy lifting to get the machine configured. The configuration steps are the ones that typically fail, so rather than having to reinstall the entire product and go back through all the configuration steps, if one part fails you only need to reconfigure that part. We have also enabled command line versions of both setup and the configuration tools.

Here is the controller config tool with options called out:

So you will need to fill in the controller service user using a user with counter collection access to your systems under test (same as in VS 2008), and then configure the load test database in the “Configure for load testing” section.

To add virtual users, start a Visual Studio Command prompt as Administrator (on Vista or Win7, just type Visual Studio Command Prompt in the search window, on XP you’ll find this under the Visual Studio group in the start menu). Then run this command line:

TestControllerConfig licenses /addkey:V10000

This will license the controller for 10,000 virtual users. Note that this license is temporary, and will only work in beta2. For RTM, we will only offer the 1,000 user license pack. If you have a Volume License, you will be able to get your license key from the volume licensing site, and then enter a count of licenses you want to install.

Once you have a virtual user license installed, that license can also be used for local runs (that is, the runs do not have to go through the controller).

If you just want to do local runs, and not use the controller and agent, you can run this from the command line to install the virtual user license:

VSTestConfig licenses /addkey:V10000

This enables you to exceed 250 users for local runs, and unlocks all the cores on your CPU.

Ed.

Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010

edglas — Tue, 20 Oct 2009 14:52:00 GMT

In Visual Studio 2010, we are changing our load test licensing model. In VS 2005 and VS 2008, we shipped the load agent, which was licensed by CPU on the machine generating load. In Visual Studio 2010 we are changing our license model to a virtual user based model. This new license model offers a number of advantages for you, which is ultimately why we chose to make the change:

Still way less expensive than our competition.
Enables better purchasing planning and allows an apples to apples price comparison with competitors.
Enables more flexible machine allocation for load generation.
Enables you to use agents for more than just load testing.

Still Way Less Expensive than our Competitors

Your initial reaction may be “Groan – that means that Microsoft is going to be just as expensive as those other tools!” But that is not the case at all, we were careful to maintain the per-virtual-user price we set out initially with VS 2005. With VS 2005 and VS 2008 load agents, the price was around $5,000 per load agent CPU, and our general guidance was that for a typical load test expect around 1,000 virtual users per CPU. We are not yet disclosing our pricing yet, but I can tell you our goal is to stay in this same ball park with VS 2010.

Better Planning and Comparison

With the VS 2008 load agent, we often get questions from customers doing purchasing planning asking “How many load agents do we need to purchase? We are planning to test up to 5,000 virtual users.” To which our answer with VS 2008 was “we do not know, it depends on your application and test scripts.” Of course this it difficult to know how many agents to purchase until actually doing testing, which was often too late. With virtual user licenses, purchasers will know exactly how many licenses to buy.

The old CPU-based license model also makes it very difficult to compare prices with competitive solutions. Most competitive products are licensed by virtual user and data collectors. If you know how many virtual users you want to test to, but do not know how many agents you will need to generate that number of users, it is impossible to make a comparison. With the Visual Studio Load Test Virtual User Packs 2010, if you know the virtual user count you want to test to, you will know exactly how many licenses to purchase, and be able to compare directly with competitive solutions you may consider.

Enables More Flexibility with Load Generators

The virtual user license offers flexibility with the machines you use as load generators.

With VS 2008, to get the full value of the per-CPU license, you had to run on fast CPUs, often requiring new hardware purchases. Also the license was locked to a particular machine, restricting which machines you could use as load generators. With Visual Studio 2010, you can now run virtual users on older hardware with no “penalty”, as you can line up as many older machines as you wish to generate the target number of virtual users. Also, the virtual user license is now tied to the controller, not the agent machine, so you are allowed to move agent machines in and out of the agent pool.

For some tests, you may run out of some other resources before running out of CPU. The most common case for this in VS 2008 was load agents running out of memory before CPU (especially for web tests with very large post bodies or very large responses), which prevented customers from realizing the full value of their license. With the virtual user license, if this happens you can simply add more machines to the agent pool to achieve the load you are targeting.

Visual Studio 2010 Agents: More than just Load Generators

With the high cost of the agent license, we did not have a viable solution for distributed functional testing. In Visual Studio 2010, agents are now included at no extra cost with VS Premium, Ultimate, and Test Elements. We’ve added Data Collection facility to agents, and added new UI automation test type, which opens up a bunch of new scenarios for using agents.

What if I already have VS 2008 Load Agents?

If you have purchased VS 2008 load agents with software assurance, you will be able to migrate your existing licenses to a commiserate set of Virtual User License Packs. We will iron out the details and mechanics of this once we get closer to RTM.

Conclusion

You can see there are many advantages to you with the new licensing model.

Ed.

New, comprehensive technical documentation available for Web, Load, and Unit Tests

edglas — Tue, 31 Mar 2009 14:50:00 GMT

The VSTS Ranger team, led by Geoff Gray, just published the VSTT Quick Reference Guide 1.0.

This is a comprehensive collection of technical information on VSTT –- 83 pages of information!!! Many questions you may have on how stuff works in web, load, and unit test are answered in this doc. This doc is a must have for anyone working in VSTT.

Geoff and his team, the Microsoft Services Test Labs, deliver professional services for doing performance testing. They are experts at the tools, and work closely with my team (the product team) both to troubleshoot issues as they come up and to help steer the product in the right direction. Whenever Geoff saw an answer in email, figured it out himself, or saw it on a blog post, he put it in the document. He has been building it up over the past few years. I highly recommend you refer to it any time you are looking for more information or are stuck trying to get something to work in the product.

Here’s an extract from the document:

Congrats Geoff on getting this published! And thanks to others who helped along the way, and to Bijan Javidi for your support in sponsoring this project to get it published.

Ed.

More on 2008 SP1

edglas — Wed, 25 Jun 2008 04:34:34 GMT

Bill Barnett has posted a couple of great articles detailing the new APIs we've added in SP1. Check these out to see the powerful extensibility points for controlling load patterns in a load test, and more/better hooks in web tests that enable some key scenarios such as easily filtering some types of dependents and easily capturing page load times via the API.

Since the beta, we added support for IE8. The interesting thing about IE8 is that by default it opens 6 connections to the web server, rather than 2 which is used by previous browsers. FireFox 3 also uses 6 concurrent connections. While this makes playback of a single test faster, it doesn't actually increase throughput in a load test.

As I said before, and Bill details on his blog, the biggest thing to be aware of with SP1 if you are running load tests on an agent is that agent, controller, and VS clients must all be upgraded at the same time. So you can't run SP1 VS against an RTM controller and agents (and visa-versa).

We've also fixed a few additional bugs since the beta (all of these were in the RTM version of 2008, none were regressions in the beta):

Priority	Team Level 1	Title
1	Test Infrastructure Team	VSTS failed to install on top of VS Pro + SP1
1	Load Test Team	Web tests do not accurately emulate and measure the page response time using IE8
2	Load Test Team	Multi-agent -total user load and TestRunning are not calculated correctly
2	Test Infrastructure Team	Dialog clipping - the "Disable background discovery of test methods" string clips on localized versions.
2	Test Infrastructure Team	When a test file (test storage) has dependent files, we do not enumerate tests from it
2	Load Test Team	WebTest: No way for Web test code to get the ContentLength of a request
1	Load Test Team	Recorder: Low level recorder isn't working for multiline textboxes
2	Load Test Team	Web test hidden field correlation feature doesn't work for Ajax update panels
1	Load Test Team	Web test execution: file uploads are posted twice when uploading to sites that require NTLM authentication
1	Unit Test Team	Unit Test inheritance: user experience running unit tests with inheritance from command line is broken (mstest reports an error)

Short of finding a catastrophic error and clearing up one last problem we are seeing with applying the patch, we are now done with SP1 and making great headway on our next release.

Ed.

VS 2008 SP1 Beta is available

edglas — Wed, 28 May 2008 00:39:16 GMT

Incase you haven't seen this, the beta for Visual Studio 2008 SP1 is now available.

We've put a lot of work into this SP1 release, mostly around bug fixes for bugs that you reported on the forums and through tech support (mostly forums), as well as bugs our test and dev teams reported. We've also managed to slip a few extra features into the release.

The download is here.

I highly recommend you install the beta and report back any issues ASAP so we can be sure to get them rolled into the final release.

One issue to be aware of is if you are running using test agents, the agent, controller, and all clients must match the SP1 version. Unfortunately the controller and agent uses .NET remoting object serialization, which makes it very sensitive to any changes in the objects that are being passed back and forth.

New features (we'll be blogging about these in separate posts):

Priority	Team Level 1	Title
1	Load Test Team	Support random extraction of matches using extract text rule
2	Load Test Team	DCR: Add Web test and load test extensibility points to support partners
1	Test Infrastructure Team	Enable Dual NIC controller scenarios

Bugs fixed (I’ve highlighted ones you may find interesting):

Priority	Team Level 1	Title
1	Load Test Team	VS crashes due to InvalidOperationException in load test monitor when running load test that hits many unique URLs
1	Load Test Team	Extra threads created when running load test containing unit tests on multi-agent rig causing out of memory exception
1	Load Test Team	Loadtest DB: Installing the controller should not drop an existing database.
2	Load Test Team	Goal-based profile - user load adjustment is based on 'ideal' user load instead of actual user load
2	Load Test Team	Cookies: We're not handling cookie expirations properly.
2	Load Test Team	LoadTest: When OutOfMemory exception stops load test on one agent in multi-agent test some load test results data is lost.
2	Load Test Team	Incorrect requests being recorded for Ajax site
2	Load Test Team	Form fields can be recorded twice
2	Load Test Team	Loadtest: Test times are inaccurate at less than 15 milliseconds.
2	Load Test Team	DTS: Web Recorder does not record request when Content type is missing
2	Load Test Team	DTS: Webtest Recorded in 2005 will not record in 2008
2	Load Test Team	WebTest.WebProxy property does not work
2	Load Test Team	Hidden Field Name appears as GUID when uploading file to sharepoint
2	Load Test Team	Custom Load Profiles: Having a MinUserCount > MaxUserCount throws the wrong exception.
2	Load Test Team	Performance overhead of new PostPage event is too high
2	Load Test Team	Webtest: Think times set on the first request are ignored.
2	Load Test Team	Orcas Web Test: View response in html editor does not work
2	Load Test Team	LoadTestPlugin can't set the user load higher than the max user load for the scenario
2	Load Test Team	Test View has huge delays
2	Load Test Team	Visual Studio crashes when calling AddCommentToResult in a WebTestRequestPlugin
2	Load Test Team	Web Browser tab in Web Test result shows broken page on JPN OS.
2	Load Test Team	Can't set Outcome on web test or request
2	Load Test Team	Copy/Paste from the webtest playback window on virtual PC crashes orcas.
2	Load Test Team	The default for the load test run setting "Run unit tests in application domain" should be true
2	Load Test Team	File upload parameters with no actual file causes the web test engine to throw an exception.
2	Load Test Team	Wininet recorder: ID field not correlated properly when uploading files to sharepoint.
2	Load Test Team	Copying a Load profile from the UI yields a cast error.
2	Load Test Team	Load test user load ramps up to quickly during warmup period
2	Load Test Team	Load Test should honor the ApartmentState setting in the Test Run Config file
2	Load Test Team	Thinktime that's set via plugin code is ignored
2	Load Test Team	LoadTest aborts when test throws an exception with a quote in the stack trace
2	Load Test Team	Loadtest Analyzer: Error when trying to bring up the loadtest analyzer after a test that stops very quickly.
2	Load Test Team	LoadTest: Goal profile performance counter should be sampled at least every 5 seconds
2	Load Test Team	WebTest: User code can't set values in the WebTestContext if the property has been set in an env variable on in the LoadTestContext
2	Load Test Team	Network emulation doesn't work correctly with compression
1	Test Infrastructure Team	Loc: String "Whidbey" appears in Microsoft.VisualStudio.QualityTools.Resource.dll
1	Test Infrastructure Team	VSPro 2008 won't show test lists created on VSTS2008
1	Test Infrastructure Team	Test runs with load test and any other test type get stuck
1	Test Infrastructure Team	CodeGen for IIS website code failed in generating the private accessor
1	Test Infrastructure Team	Upgrade a Whidbey test project in VSTD2008 will fail
1	Test Infrastructure Team	SP1 patch setup fails on VSTS with controller/agent installed
1	Test Infrastructure Team	Fix perf tests that are failing from time to time in QFE (TechEase ticket 143089)
1	Test Infrastructure Team	Publishing load tests fails in TFS SP1
2	Test Infrastructure Team	Multi-agent: Could not start two load tests in paralell on different agents
2	Test Infrastructure Team	Watson - unhandled TeamFoundationServerUnauthorizedException crashed devenv.exe
2	Test Infrastructure Team	Out of memory error when publishing test cases using MSTest
2	Test Infrastructure Team	MSTest can't run codedwebtests when alone errors with assembly is not trusted
1	Unit Test Team	Invalid AppDomain.CurrentDomain.BaseDirectory in VS 2008 Unit Test Runner
1	Unit Test Team	Unit Testing: App Domain naming creates issues with performance counters
1	Unit Test Team	Unit Test Generation fails for default WCF service
2	Unit Test Team	Ordered tests created in VS2005 are not automatically upgraded in VSTD 2008
2	Unit Test Team	DTS: IDE in 2008 responds slow compared to IDE in 2005 in a test Project with over 9000 unit test
2	Unit Test Team	VSTestHost specificies LoaderOptimization.MultiDomain as the default, causes various problems
2	Unit Test Team	Unit testing fails in asp.net hosted in iis

I hope you find this useful, and by all means please download and use SP1 beta!

Ed.

84,248 Ads per second!

edglas — Mon, 17 Mar 2008 23:59:13 GMT

The adCenter Display Ads Platform team recently ran a large test using VSTS. The goal of this testing was to validate the configuration and stability of 100+ servers under load.

The test used one Controller and 42 agents to produce the necessary load. The controller was also hosting a full install of SQL Server which acted as the result store.

Controller and Agent hardware information

Processor: 4 x AMD Opteron 280 (2405 MHz)
Memory: 4 GB RAM
Network: GigE NIC

High level test summary

Test Duration                               12 hours
Total requests generated             3,639,518,000 (3.63 Billion requests)
Average requests/sec                   84,248
Average Requests/Test                1
Failed Tests                                  0
Sampling rate                               30 secs
Average agent Proc utilization    45%

To minimize the risk they started off with one controller and one agent, then increasing to 8 agents, then going to 42.

They estimate they could have produced at least 50% more requests with 42 agents but restricted the load to avoid any agent overloading.

A note from the customer: Overall it was a happy experience running a test of this large scale. Thanks to VSTS for making load testing a pleasure!

Ed.

Updated Content for web tests and load tests

edglas — Mon, 17 Dec 2007 20:36:00 GMT

I've updated my collection of links for info on web tests and load tests.

I had been publishing this to a sticky post on the forum, but ran into the size limit so I'm moving it to my blog.

The page is here. Do let me know if you hit any links that are broken, the content gets moved around from time to time.

Ed.

Agent Setup and Agent/Controller network connections

edglas — Fri, 05 Oct 2007 20:14:00 GMT

Check out Durga's blog post on setting up an agent in workgroup mode.

One thing I noticed is that the install guide is incorrect, as it says that the "Simple File Sharing" is turned off when installing on XP in workgroup mode. You have to clear that manually. I have corrected that in my post.

I also put together a visio diagram that shows which ports are used during setup and when the agent and controller run tests.

And here are the connections used during agent setup:

Ed.

What's new in Orcas in Web, Load, and Unit Tests?

edglas — Wed, 28 Mar 2007 05:19:00 GMT

Lot's of new stuff for web, load, and unit tests in Orcas. Follow the links for more info.

Web test

· Recorder now records AJAX and popups

· Test-level validation rules

· Auto-validate response URL

· Composition/decomposition, also here and here.

· Easily bind to csv or XML

· UI improvements in playback

· Support for multiple web test plugins and parameterized plugins

· API improvements

· Expected HTTP code

· Request-level plugins

· Correlation helper (in beta 2)

Load Test

· Load test result manager

· Summary report

· Support for multiple graphs, new performance graph, vertical zoom, synch zoom

· New load model for user pacing, users running

· User init/terminate functions, also here

· Support for iteration count, cool down

· Unit tests under load performance improvements

· Profiler Integration

Unit Test

See Naysawn's posts here and here for more info on these.

· Perf improvements

· Better code generation

· Better scheme for accessing privates

· Auto-cleanup of results (beta 2)

· Option to goto error (beta 2)

· New command key shortcuts (beta 2)

· Zero-deployment option (beta 2)

· Data binding improvements

o Easily bind to csv, XML

o Auto-deploy local files and fixup connect strings

o Iterate correctly in a load test

Ed.

Load Agent Install Guide

edglas — Mon, 06 Mar 2006 21:19:00 GMT

Here is the install guide for the agent and controller. It contains a bunch of useful sizing and configuration information, describing for example the different kinds of machine configs we support with VS, Controller, and Agent. This will be available on MSDN the next time we do a doc refresh.

Visual Studio Team Test Load Agent and Load Controller Installation and Configuration Guide

Ed.

---------------------------------------------