Ed Glas's blog on VS load testing

A New Role for Me

Ed Glas — Mon, 14 Feb 2011 21:52:00 GMT

I have been quiet on my blog for a while, mostly because I have switched jobs. I had been running dev, test, and PM for the Team Test team here in Raleigh NC, reporting to the PUM of Team Test, Amit Chatterjee in our India Development Center. My team built the load test tools and TCM server product in VS 2010. At the end of VS 2010 we decided to consolidate the test tools work to the IDC facility. Neelesh Komkolkar is now PM for the load testing tools.

I took on the role as QA manager for the Team Foundation Server (TFS) product team. I now run the TFS QA team, with half my team in being in NC and half in Redmond. While I was sad to let the load testing and TCM products go, I am excited by the new challenges and opportunity afforded in my new role.

Ed.

Interviewing for an SDET job at Microsoft

Ed Glas — Mon, 14 Feb 2011 21:44:32 GMT

I have been doing a lot of interviewing lately. First of all, if you are interested in coming to sunny Raleigh, NC to work as an SDET on my team, let me know! We are hiring.

Second, I want to give people a flavor of what to be prepared for in an interview.

In a typical interview here we ask you to code up some function on the whiteboard and then discuss test cases for the function.

String functions in particular are popular. So something like code a function to reverse the words in a string: “The cat jumped over the fence” to “ehT tac depmuj revo eht ecnef”.

The best approach for coding is to first think about the algorithm. For example, write down on the white board the most common simple example, then think through the algorithm. Once you have that, code it. Typically the coding should be done in about 20-30 minutes, so if it seems your algorithm is not simple you are probably doing something wrong.

Then come up with interesting test cases for it:

A common case: “I went to the park”

Edge cases:

really short: “A”, “”

really long (say a book)

Extra spaces:

“ I went to the park ”

“I went to the park”

“ I went to the park”

“I went to the park ”

“ I went to the park ”

Tabs (is this whitespace?)

Test cases with non-alpha characters would also be interesting.

You get the idea.

Another one might be insert an element into a sorted linked list. You can look on the internet for other types of coding questions.

It is also good to think about:

- running on different language OS, or different language VS (say a far east language, or bi-directional language)

- performance (how fast is it) and stress (e.g. memory leak detection)

- usability

and other types of non-functional testing.

You might also be asked to test some piece of software, like the file open dialog in excel, or some object, like a tire, coffee cup, or battery.

I hope that helps, and happy interviewing!

Ed.

Plugins for Handling JSON and msbin1 SilverLight Payloads on codeplex

Ed Glas — Sat, 07 Aug 2010 19:19:35 GMT

We enabled two new key extensibility points in VS 2010 to better handle rich post bodies that we often see with “web 2.0” style apps and with SilverLight. The first is you can create custom editors for the request body: Request Body Plugins (also here: How to: Create a Custom HTTP Body Editor for the Web Performance Test Editor). The second is you can add new tabs to the Web Test result viewer: How to: Create a Visual Studio Add-In for the Web Performance Test Results Viewer.

Mahipal Kante, the dev lead for load test, leveraged these extensibility points to build a great set of plugins for handling the WCF msbin1 format and JSON format. It enables both editing the parameter values in the editor as well as viewing them in the results viewer.

The beta is available here: http://teamtestplugins.codeplex.com/releases/view/43166.

Ed.

MSDN Docs Updated for July

Ed Glas — Sat, 07 Aug 2010 18:18:00 GMT

Of all the great things we did with the product in 2010, one area I was surprised and pleased by is how far our documentation came thanks to the heroics of Howie Hilliker. Howie has been busy since we shipped, continuing to beef up the MSDN documentation. We get regular updates to the docs through “continuous publication”, or CPub, and we recently dropped the July updates on the site.

Here is a list of the new and updated topics:

New topics:

How to: Specify the Maximum Size for the Log File

Troubleshooting Network Emulation in Load Tests

Walkthrough: Adding a Loop to a Web Performance Test

How to: Run a Load Test Containing Web Performance Tests that Collects ASP.NET Profiler Data

How to: Manually Create a Load Test Performance Report Using Microsoft Word

Updated topics (Customer feedback bugs):

Added information on ClassInitialize and TestInitialize attributes:

Anatomy of a Unit Test

Added C# and Visual Basic code snips displaying accessors

Unit Tests for Private, Internal, and Friend Methods

Added screenshots and information for using SQLExpress

How to: Select a Load Test Results Repository

Added codes snip for using "Tag Inner Text" validation rule and added links to validation and extraction walkthrough

Using Validation and Extraction Rules in Web Performance Tests

Added note regarding IIS application pool credentials (closed PU bug)

How to: Collect IntelliTrace Data to Help Debug Difficult Issues

Added Tip note suggesting the use of the ColorWebApp ASP.NET sample created in the walkthrough to experiment with

How to: Create a Coded Web Performance Test

Cleaned up art - numbering sequence

Running Unit Tests on Multiple Machines Using a Test Controller and Test Agents

Walkthrough: Using a Test Controller and Test Agents in a Load Test

Using Test Controllers and Test Agents with Load Tests

Configuring Test Controllers and Test Agents for Load Testing

Reported bug - waiting on resolution

How to: Specify Test Agents to Use in Load Test Scenarios

Fixed confusion over "Performance session" vs. "ASP.NET Profiler adapter" & added links

How to: Start a Performance Session on a Web Application from the Web Performance Test Editor

Load Test Analyzer Overview

Added Links to the following topics

Added link to new related walkthrough topic

Adding Loop and Branch Logic to Web Requests in Web Performance Tests

Added links to the various walkthrough topics on binding to db, csv and XML data sources.

Binding a Data Source to a Web Performance Test

Added links to "Binding a Data Source to a Web Performance Test"

Walkthrough: Adding Data Binding to a Web Performance Test

Walkthrough: Adding Data Binding to a Web Performance Test (CSV File)

Walkthrough: Adding Data Binding to a Web Performance Test (XML File)

Added link to new walkthrough on using a loop condition in a Web performance test

Web Performance Test Walkthroughs

What's New for Web Performance and Load Testing

How to: Add Loops to Web Performance Tests Requests

Added information on DLL version error work-around

How to: Create a Load Test Plug-In

How to: Create a Recorder Plug-In

How to: Create a Request-Level Plug-In

How to: Create a Web Performance Test Plug-In

Added link to new topic "How to: Run a Load Test Containing Web Performance Tests that Collects ASP.NET Profiler Data"

How to: Configure ASP.NET Profiler for Load Tests Using Test Settings

How to: Create a Test Setting for a Distributed Load Test

Added link to new topic "Troubleshooting Network Emulation in Load Tests"

How to: Add Networks Using the Load Test Editor

Specifying Virtual Network Types in a Load Test Scenario

Troubleshooting Load and Web Performance Tests

Troubleshooting Load Tests

Running Load Tests

Added link to new topic "How to: Specify the Maximum Size for the Log File"

Modifying Load Test Logging Settings

Added link to new topic "How to: Manually Create a Load Test Performance Report Using Microsoft Word"

Reporting Load Tests Results for Test Comparisons or Trend Analysis

How to: Create Load Test Performance Reports Using Microsoft Excel

Ed.

Version 3.5 of the Performance Testing Quick Reference Guide Available

Ed Glas — Sat, 07 Aug 2010 18:06:55 GMT

Geoff Gray has done it again, adding a bunch of new content to the Quick Reference Guide at

http://vsptqrg.codeplex.com/.

I strongly recommend you download this documentation and keep it in “your back pocket”. One thing to note is that in its current form it does not get indexed by search engines, so you need to actually download the pdf.

Here is a list of the new topics:

--NEW-- How cache settings can affect your testing and app performance 19

--NEW-- How a Load Test Plugin is executed on a test rig 29

--NEW-- Sharing State across agents in a load test rig is not supported out of the box 30

--NEW-- Customizing and extending the available network emulation settings 66

--NEW-- New Performance Counters in .NET 4.0

--NEW-- Content-Length=0 Header not sent resulting in HTTP 411 Length Required Error 99

--NEW-- Web and load tests stay in “Pending” state for a long time before starting 99

--NEW-- Error that test could not run because the network emulation is required 100

--NEW-- Error/Crash in “Open and Manage Load Test Results” dialog 100

--NEW-- Calls to CaptchaGenerator.aspx fail during playback 101

--NEW-- Request failure with improperly encoded query strings calling SharePoint 2010 101

--NEW-- Network Emulation does not work in any mode other than LAN 101

--NEW-- Error: Request failed: No connection could be made because the target machine actively refused it 103

--NEW-- MaxConnection value in App.Config is not honored when running a load test 103

--NEW-- Web Test Authoring and Debugging Techniques 104

--NEW-- Troubleshooting Network Emulation 123

--NEW-- Best Practice: Understanding the ROI for test automation 140

--NEW-- How to create guaranteed unique user IDs for UNIT tests 171

--NEW-- How to create a sync point for starting load tests 173

--NEW-- How to set default extensions that the WebTest recorder will ignore 173

--NEW-- Gotcha: VS 2008 and out of memory 174

--NEW-- Gotcha: Timeout attribute in coded web test does not work during a load test 174

--NEW-- How to extend recorder functionality with plugins 185

--NEW-- Stand-Alone Network Emulation and CodePlex 193

Thanks for putting this new content together Geoff!

Ed.

New SharePoint 2010 Load Test Toolkit Released

Ed Glas — Sat, 07 Aug 2010 17:39:59 GMT

The SharePoint team released the Administration Toolkit for SharePoint 2010 (SPAT2010).

One part of this toolkit is a great new load testing kit for 2010.

The toolkit was initially developed by the SharePoint product team’s perf testing team, and then made available publicly.

SharePoint is a real challenge to do performance testing against (or any test automation for that matter), since both field names and values contain GUIDs. Large SharePoint deployments can contain 1000s of SharePoint sites, so how can you effectively test this? If you only hit the same site over and over it could have a very different performance characteristic than if you spread load across all the sites. And typically some sites are much busier and more popular than other sites.

The load test toolkit handles both of these problems really well. It is particularly targeted initially at migrating a site from 2007 to 2010, and does a great job by reading the IIS log to get site distribution, and by then dynamically determining during a load test which sites to hit. For example, if the load test is testing actions against a document library, a web test plugin will go to the SharePoint site to dynamically select an appropriate document library to target, and then cache the url to the library on the load client. This is a brilliant way to solve both the distribution and parameterization problems. Hats off to Demetrious and Kfir on the SharePoint team for developing the solution, and James and Doron for making it available to customers.

Ed.

Visual Studio Performance Testing Quick Reference Guide (Version 2.0) Published

Ed Glas — Tue, 13 Apr 2010 02:09:26 GMT

Version 2 of the Quick Reference Guide has now been published. This guide is chock-full of information gathered through real experience with the product. This new version contains 97 new or updated articles.

Special thanks to Geoff Gray for once again spearheading this effort. Geoff is one of a handful of elite VS load test practitioners. As he encounter unexpected behavior or bugs in the product, or gets undocumented information on the internals or edge cases, he captures the info for the quick reference guide.

Here is a screen shot of some of the really nice artwork in the guide:

Here is a list of topics in the guide (the numbers after each article is the page number):

NOTE FROM THE AUTHOR
HOW IT WORKS
How Web Tests Handle HTTP Headers 9
General Info (including order of execution) of load and web test plugins and rules 9
Client Code does not execute because Web Tests Work at the HTTP Layer 12
File Downloads, Download Size and Storage of files during Web Tests 12
When is the “Run unit tests in application domain” needed? 12
How the “Test Iterations” Setting impacts the total number of tests executed 12
Test timeout setting for load test configuration does not affect web tests 13
How user pacing and “Think Time Between Test Iterations” work 13
Load test warmup and cool down behaviors 13
What is the difference between Unique, Sequential and Random Data Sources 14
Comparing new users to return users 14
Goal based user behavior after the test finishes the warmup period 17
Threading models in Unit tests under load 18
Simulation of Browser Caching during load tests 19
The difference between Load Test Errors and Error Details 20
How parameterization of HIDDEN Fields works in a webtest 21
Testing execution order in Unit Tests 23
How machines in the test rig communicate 25
Changing the Default Port for Agent-Controller Communication 26
How to Add Agents To A Test Rig 26
ITEMS NEW TO VS 2010
“Find” feature now available in Webtest playback UI 27
“Go To Web Test” feature now available in Webtest playback UI 28
Recorder Log Available 29
Add extraction rule directly from the playback UI 30
New “Reporting Name” property for web requests 31
LoadTestResultsTables now differentiate between GET and POST requests 32
Virtual user visualization now available 33
New Excel reporting features built into load test results 39
New Load Test and Load Test Rig Licensing and configurations 40
New test mix: “Sequential Test Mix” 44
Query String and FORM POST URLs get parameterized 46
New options on Load Test Scenarios 47
Loops and Conditionals 48
CONFIGURATIONS AND SETTINGS
How to Change the Location Where Agents Store Run Files 50
How to set a proxy server for web tests 50
How to configure Web Tests so Fiddler can capture playback info 50
Controlling the amount of memory that the SQL Server Results machine consumes 51
How to configure the timeouts for deployment of load tests to agents 51
How to set the number of Load Test Errors and Error Details saved 52
Multi-proc boxes used as agents should have .NET garbage collection set to server mode 53
Location of list of all agents available to a controller 53
NETWORKS, IP SWITCHING, TEST STARTUPS
IP Address Switching anatomy (how it works) 54
Gotcha: IP Address Switching is ONLY for WEB TESTS 54
Gotcha: IP Addresses used for switching are not permanent 54
How to Setup IP Switching 55
Troubleshooting invalid view state and failed event validation 58
Startup: Slowness Restarting a Test Rig with Agents Marked as “Offline” 58
Startup: Multiple Network Cards can cause tests in a rig to not start 59
Startup: Slow startup can be caused by _NT_SYMBOL_PATH environment variable 59
Startup: tests on a Rig with Agents on a Slow Link 60
“Not Bound” Exception when using IP Switching is not really an error 60
How to configure the timeout for deployment of load tests to agents 61
PERFORMANCE COUNTERS AND DATA
Customizing the Available Microsoft System Monitor counter sets 62
Performance Counter Considerations on Rigs with slow links 64
Increase the performance counter sampling interval for longer tests 65
Changing the default counters shown in the graphs during testing 65
Possible method for fixing “missing perfmon counters” issues 65
How and where Performance data gets collected 66
DATA AND RESULTS
Custom Data Binding in UNIT Tests 67
Verifying saved results when a test hangs in the “In Progress” state after the test has finished 67
The metrics during and after a test differ from the results seen. 68
How new users and return users affect caching numbers 69
data sources for data driven tests get read only once 70
Consider including Timing Details to collect percentile data 71
Consider enabling SQL Tracing through the Load Test instead of separately 72
How to collect SQL counters from a non-default SQL instance 72
How 90% and 95% response times are calculated 72
Transaction Avg. Response Time vs. Request Avg. Response Time 73
Considerations for the location of the Load Test Results Store 73
Set the recovery model for the database to simple 73
How to clean up results data from runs that did not complete 74
InstanceName field in results database are appended with (002), (003), etc. 74
Layout for VSTS Load Test Results Store 74
How to view Test Results from the GUI 75
SQL Server Reporting Services Reports available for download 75
How to move results data to another system 75
Load Test Results without SQL NOT stored 76
Web Test TRX file and the NAN (Not a Number) Page Time entry 77
Proper understanding of TRX files and Test Results directory 78
Understanding the Response Size reported in web test runs 79
ERRORS AND KNOWN ISSUES 80
CSV files created in VSTS or saved as Unicode will not work as data sources 80
Incorrect SQL field type can cause errors in web tests 80
Leading zeroes dropped from datasource values bound to a CSV file 80
Recorded Think Times and paused web test recordings 80
After opening a webtest with the VS XML Editor, it will not open in declarative mode. 81
Calls to HTTPS://Urs.Microsoft.Com show up in your script 81
Possible DESKTOP HEAP errors when driving command line unit tests 81
Goal based load tests in VSTS 2008 do not work after applying SP1 82
Using Named Transactions in a Goal-Based Load Profile can cause errors 82
Debugging Errors in Load Tests 83
Debugging OutOfMemory Exceptions in Load Tests 83
Memory leak on load test when using HTTPS 83
“Not Trusted” error when starting a load test 84
Detail Logging may cause “Out of disk space” error 85
Error details and stack traces no longer available in VSTS 2010 85
VSTS does not appear to be using more than one processor 85
Changes made to Web Test Plugins may not show up properly 85
Socket errors or “Service Unavailable” errors when running a load test 86
Error “Failed to load results from the load test results store” 87
Hidden Field extraction rules do not handle some fields 87
Test results iteration count may be higher than the max test iterations set 87
In flight test iterations may not get reported 88
Completion of Unit Test causes spawned CMD processes to terminate 88
Bug with LoadProfile.Copy() method when used in custom goal based load tests 89
Errors in dependent requests in a Load Test do not show up in the details test log 90
WCF service load test gets time-outs after 10 requests 92
Loadtestitemresults.dat size runs into GBs 92
TROUBLESHOOTING
How to enable logging for test recording 93
Diagnosing and fixing Web Test recorder bar issues 93
User Account requirements and how to troubleshoot authentication 94
How to enable Verbose Logging on an agent for troubleshooting 95
Error that Browser Extensions are disabled when recording a web test 95
Troubleshooting invalid view state and failed event validation 96
Troubleshooting the VSTS Load Testing IP Switching Feature 97
Troubleshooting Guide for Visual Studio Test Controller and Agent 99
HOW TO, GOTCHAS AND BEST PRACTICES
How to call one coded web test from another 111
How to use methods other than GET and POST in a web test 111
How to filter out certain dependent requests 111
How to handle ASP.NET Cookie-less Sessions 112
How to use Client-side certificates in web tests 112
How to remove the “If-Modified-Since” header from dependent requests 113
How to handle custom data binding in web tests 113
How to add a datasource value to a context parameter 113
How to test Web Services with Unit Tests 114
How to add random users to web tests 114
How to add think time to a Unit Test 114
How to add details of a validation rule to your web test 115
How to mask a 404 error on a dependent request 116
How to parameterize Web Service calls within Web Tests 117
How to pass Load Test Context Parameters to Unit Tests 117
How to create Global Variables in a Unit Test 117
How to use Unit Tests to Drive Load with Command Line Apps 118
How to add Console Output to the results store when running Unit tests under load 118
How to add parameters to Load Tests 119
How to Change the Standard Deviation for a NormalDistribution ThinkTime 119
How to programmatically access the number of users in Load Tests 120
How to create a webtest plugin that will only execute on a predefined interval 120
How to support Context Parameters in a plug-in property 121
How to stop a web test in the middle of execution 122
How To: Modify the ServicePointManager to force SSLv3 instead of TLS (Default) 122
How To: Stop a Test in the PreRequest event 123
How to make a validation rule force a redirection to a new page 123
How to add a Web Service reference in a test project 127
How to remotely count connections to a process 129
How to hook into LoadTest database upon completion of a load test 129
How to deploy DLLs with MSTEST.EXE 130
How to authenticate with proxy before the test iteration begins 131
How to enumerate WebTextContext and Unit TestContext objects 132
How to manually move the data cursor 132
How to programmatically create a declarative web test 133
How to modify the string body programmatically in a declarative web test 134
Gotcha: Check Your Validation Level in the Load Test Run Settings 134
Gotcha: Do not adjust goals too quickly in your code 134
Gotcha: Response body capture limit is set to 1.5 MB by default 134
Gotcha: Caching of dependent requests is disabled when playing back Web Tests 135
Best Practice: Blog on various considerations for web tests running under load 135
Best Practice: Coded web tests and web test plug-ins should not block threads 135
Best Practice: considerations when creating a dynamic goal based load test plugin: 136
Best Practice: Add an Analysis Comment 136
Best Practice – Using comments in declarative webtests 136
EXTENSIBILITY
New Inner-text and Select-tag rules published on Codeplex 138
ITEMS NOT SPECIFIC TO THE VSTS TESTING PLATFORM
Using the VSTS Application Profiler 146
VSTS 2008 Application Profiler New Features 146
Using System.NET Tracing to debug Network issues 146
Logparser tips and tricks 147
Logparser WEB Queries 147
LogParser Non-Web Queries 148
OLDER ARTICLES
Content-Length header not available in Web Request Object 149
SharePoint file upload test may post the file twice 149
Some Hidden Fields are not parameterized within AJAX calls 149
(FIX) Unit Test threading models and changing them 149
Bug in VSTS 2008 SP1 causes think time for redirected requests to be ignored in a load test 150
New Load Test Plugin Enhancements in VSTS 2008 SP1 150
Four New Methods added to the WebTestPlugin Class for 2008 SP1 150
INDEX

Ed.

Updated Load Test Links for April 2010

Ed Glas — Tue, 13 Apr 2010 01:15:50 GMT

I’ve updated my Content Index for Web Tests and Load Tests, which is a list of links to load test content. I added a bunch of links on what’s new in VS 2010, as well as a few additional links:

Dynamics AX Performance and Load Testing: Application Benchmark Toolkit

Parameterizing Tests to Run in Different Environments

Load Test Plug-in – Simulate Repeated Peak Load Modeling

Visual Studio Performance Testing Quick Reference Guide (Version 2.0)

I also moved some posts that are specific to VS 2008 into their own section.

Ed.

Great New Documentation for Performance Testing Now Online

Ed Glas — Mon, 12 Apr 2010 19:59:04 GMT

I am very excited about the new level of documentation we have in this release.

Howie Hilliker has done a fantastic job enhancing our documentation. We actually have screen shots and artwork in the docs now! Also thanks to the load test product team for reviewing the content.

The documentation has been completely reorganized, with performance testing topics under the Testing Application Performance and Stress topic.

For example, check out these topics:

Walkthrough: Correlating Dynamic Parameters

Load Test Analyzer Overview

Walkthrough: Using a Test Controller and Test Agents in a Load Test

We also have walkthroughs on creating plugins, which is very helpful.

Creating and Using Custom Plug-ins for Load and Web Performance Tests

Here’s a blowout of the table of contents. It is definitely worthwhile to scan through this list to familiarize yourself with what is there.

Testing Application Performance and Stress

Web Performance Test Walkthroughs

Load Test Walkthroughs

Creating and Editing Load and Web Performance Tests

Creating and Editing Load Tests

Creating Load Tests Using the New Load Test Wizard

Editing Load Test Using the Load Test Editor

Editing Load Test Scenarios Using the Load Test Editor

How to: Insert Additional Scenarios to an Existing Load Test

Editing Think Times to Simulate Web Site Human Interaction Delays in Load Tests Scenarios

How to: Change the Think Profile

Editing Load Patterns to Model Virtual User Activities

Editing Text Mix Models to Specify the Probability of a Virtual User Running a Test

Editing the Test Mix to Specify Which Web Performance, Unit and Coded UI Tests to Include in a Load Test Scenario

Specifying Virtual Network Types in a Load Test Scenario

Specifying Web Browsers Types in a Load Test Scenario

Configuring Test Iterations in a Load Test Scenario

Configuring Scenario Start Delays

Specifying the Counter Sets for Computers in a Load Test

Configuring Load Test Run Settings

Collecting SQL Trace Data to Monitor and Improve Performance in Load Tests

How to: Integrate SQL Trace Data Using the Load Test Editor

Using Coded UI Tests in Load Tests

Creating and Editing Web Performance Tests

Customizing Web Performance Test Recordings Using Web Performance Test Editor

Using Validation and Extraction Rules in Web Performance Tests

Adding Loop and Branch Logic to Web Requests in Web Performance Tests

How to: Configure Visual Studio Ultimate Options for Web Performance Tests

Running Web Performance Tests

Analyzing Load Tests Results Using the Load Test Analyzer

Analyzing Load Test Results in the Tables View of the Load Test Analyzer

Using the Counters Panel in Graphs View and Tables View

Analyzing Load Test Virtual User Activity in the Details View of the Load Test Analyzer

Examining Results Using the Web Performance Test Results Viewer

Managing Load Test Results in the Load Test Results Repository

Reporting Load Tests Results for Test Comparisons or Trend Analysis

How to: Create Load Test Performance Reports Using Microsoft Excel

Creating and Using Custom Plug-ins for Load and Web Performance Tests

Troubleshooting Load and Web Performance Tests

About JavaScript and ActiveX Controls in Web Performance Tests

Ed.

VS 2010 Ships!

Ed Glas — Mon, 12 Apr 2010 17:14:18 GMT

I am happy to say that VS 2010 is generally available starting today! Go to the Visual Studio home page for information.

As you are probably aware, we put a huge investment in our testing products into this release. I outlined the new features with my post Elevating the Role of the Tester with Visual Studio 2010. The Visual Studio Team Test blog is a great place to keep up with these features, as is our fantastic online documentation rooted here: Testing the Application.

For those of you doing performance testing, my team has been working very hard these past two years to deliver a boat load of new features to you in this release. Early customer feedback shows that these features will dramatically improve your productivity. Here is a list of links to blog posts and documentation on what’s new in VS 2010 for performance testers.

What’s new in VS 2010	What's New for Web Performance and Load Testing
	Product Transitions from VS 2008 to VS 2010 for Load Testing
	Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010
	Elevating the Role of the Tester with Visual Studio 2010
Web Test Debugging	Web Test Authoring and Debugging Techniques for Visual Studio 2010
	Debugging a Web Test with VS 2010
	Walkthrough: Correlating Dynamic Parameters
Web Test Recorder	How to: Create a Recorder Plug-In Adding Your Own Web Test Recorder Plugin to Make Record/Playback “Just Work” Web Test Recorder Plug-in Convert Comments entered in Web Test Recorder into Transactions Extensible recorder plugins for modifying recorded web tests
	How to: View Recording Log in Web Performance Test Playback VSTS 2010 Feature: View Recording Log in Web Test Playback Window
	New Tools Options for the Recorder
	More Recorder Enhancements in VS 2010
	VSTS 2010 Feature: File Upload record/playback just works
Web Test Editor	How to: Set Request Details in a Web Performance Test Request Details Editor
	Find/Replace in Web Test Editor
	Adding Loop and Branch Logic to Web Requests in Web Performance Tests Branching and Looping in Declarative Web Tests Looping and Branching in Web Tests
	Request Body Plugins How to: Create a Custom HTTP Body Editor for the Web Performance Test Editor
	How to: Use Reporting Names to Easily Identify Requests in a Web Performance Test
Web Test Result Viewer	How to: Create a Visual Studio Add-In for the Web Performance Test Results Viewer Extensible Web test result viewer plugins Tracking viewstate using a webtest playback add-in Result Viewer Plugins
	How to: Resolve Web Performance Test Playback Issues Caused by Dynamic Parameters
	How to: Jump to Web Request from the Web Performance Test Playback
	How to: Add an Extraction Rule from the Web Performance Test Playback
	API for Processing Web Test Results
	Enhancements for Web Test Playback UI
Load Test Modeling	VSTS 2010 Feature: Sequential Test Mix
	VSTS 2010 Feature: More flexible load modeling via new options on Load Test Scenarios Configuring Test Iterations in a Load Test Scenario How to: Delay the Start Time of a Scenario Using the Load Test Editor
	Multiple Loadtest Plug-ins Creating and Using Custom Plug-ins for Load and Web Performance Tests
Load Test Analysis	Using The Virtual User Activity Chart to Understand the VS Load Engine Load test virtual user activity visualization
	Modifying Load Test Logging Settings Saving Test Logs during a load test How to: Specify if Test Failures are Saved to Test Logs Using the Load Test Editor
	Deep integration with the ASP.NET profiler
	Deep integration with dynaTrace
	How to: Configure Event Log Collection Using Test Settings
	Create a Simple Data Collector to Integrate Data into Load Test Analyzer (Tables View) Integrate Data Collector Data into Load Test Analyzer (Tables View)
Performance Reporting	Performance Reporting in Excel Reporting Load Tests Results for Comparison or Trend Analysis
Agent/Controller	VS 2010 Test Agent and Test Controller Troubleshooting Guide
	VSTS 2010 Feature: Silent Installation of Controller and Agent
	Process Model Changes in VS 2010
	How to: Specify 64-Bit Process Using Test Settings
Network Emulation	How to: Configure Network Emulation Using Test Settings
	Introducing True Network Emulation in Visual Studio 2010
	Stand-Alone Network Emulation and CodePlex
	Specifying Virtual Network Types in a Load Test Scenario

Ed.

Simulating Peak Load with VS Load Tests

Ed Glas — Fri, 02 Apr 2010 12:58:46 GMT

In VS load tests come with a few load patterns out of the box:

Constant load: holds count of users steady
Step load: steps the count of users up to some level then holds them steady
Goal-based load: adjusts the user load up and down to reach some goal

But what if you want to simulate several peak loads, such as users coming to a site at 8:00 a.m. and again at noon?

In VS 2008 SP1, we enabled extensibility in the load test engine to allow you to define your own load patterns.

Yun Tong on the VS load test team developed a load test plugin that allows you to define multiple, repeatable load patterns, which enables peak load modeling.

Check out the cool plugin she developed at:

Load Test Plug-in – Simulate Peak Load Modeling

Ed.

Process Model Changes in VS 2010

Ed Glas — Mon, 29 Mar 2010 13:46:30 GMT

In VS 2010, we made several changes to the process models for local and remote execution.

These changes impact you when you:

Look in task manager or at performance counters to see how the test processes are running,
Turn on logging to debug a problem. You turn on logging in the .config file for the process you want to log from, so obviously the process name changes affect this.
Use other settings in the config files to control test execution or data collection.

Everyone working with load testing will probably do all three of these activities at some point or another.

A major new piece of infrastructure in VS 2010 is test data collectors, which give you the ability to collect data from the system under test. Some data collectors work against the test host process, and have to hook into the process before it starts (such as Intellitrace and Test Impact). To make this happen, we start the data collection process first, and give data collectors an opportunity to hook the test host process. Since the controller is orchestrating the test, for local execution we were required to move the controller out of the test host process and into devenv (Visual Studio).

Here are the changes in local execution:

The key changes here are:

The host process was renamed.
The local controller is now hosted in devenv rather than the agent process.
We have added a QTDCAgent.exe data collector process.
There is now an option to run tests in a 64 bit process on a 64 bit machine, or force tests to always run in a 32 bit process. The default is to force tests to run 32 bit, since Coded UI tests will only run in a 32 bit process. If you are running load tests you should under most circumstances change this to Run tests in a 64 bit process on a 64 bit machine.

Here is the place in Test Settings you have to change for #4.

For remote execution, you can see things remain mostly the same as in VS 2008, except for the addition of the data collector agent and 32 bit flavors of the processes.

Ed.

VS 2010 Test Agent and Test Controller Troubleshooting Guide

Ed Glas — Mon, 29 Mar 2010 00:49:46 GMT

Michael Koltachev, Geoff Staneff, and the rest of the TE team put together a valuable troubleshooting guide for troubleshooting a VS 2010 test rig.

Check it out at

http://social.msdn.microsoft.com/Forums/en-US/vststest/thread/df043823-ffcf-46a4-9e47-1c4b8854ca13.

In VS 2005 and 2008 agent and controller setup, all the configuration was done in setup. If any of the config steps failed, setup rolled back.

We made a huge improvement in VS 2010 by making setup super-simple, mostly just lay down the files. Then we have a config tool that launches after setup to do the configuration (opening ports, setting up groups, etc.). Now if a particular config step fails you can correct the issue and then re-run the config, rather than having to reinstall everything.

One of the great things about the troubleshooting guide is for many issues it points you back to running the config tool, which automatically checks all the required settings, showing output and any error information for each step.

VS 2010 agent setup now also supports command line install and command line config, a much requested feature for those of you who have automation to provision and configure agent machines (we see a lot of that in Microsoft product teams).

Ed.

Web Test Authoring and Debugging Techniques for Visual Studio 2010

Ed Glas — Wed, 24 Mar 2010 02:20:00 GMT

Overview
How Web Performance Tests Work
Overview of New Features in VS 2010
Debugging a Web Test to Find and Fix Dynamic Parameters
Adding Your Own Web Test Recorder Plugin to Make Record/Playback “Just Work”
More Recorder Enhancements in VS 2010
New Tools Options for the Recorder
Looping and Branching in Web Tests
Request Details Editor
Request Body Plugins
Result Viewer Plugins

Overview

This post follows similar papers done after VS 2005:

Web Test Authoring and Debugging Techniques, and again after VS 2008: Web Test Authoring and Debugging Techniques for VS 2008.

With each release of VS we have made major strides in Web Test Authoring and Debugging. With VS 2008, we added a number of features to address the most common challenges with Web test authoring, the most important being a low-level http recorder and a automatic correlation tool. This covered the most prevalent challenges outlined in Web Test Authoring and Debugging Techniques. Again with VS 2010 we have made major strides in Web test authoring and debugging:

More http recordings “just work”
New tools to help you debug and fix the ones that don’t, and
New extensibility points for the recorder, editor, and results viewer enable you, us and our community to release rich functionality “out of band” to handle custom applications and rich data types.

A New Name, But Under the Covers Still the Same

In this release we renamed "Web Test” to “Web Performance Test” to highlight the primary scenario for Web tests, which is using them as scripts in a load test to model user actions. Load tests are used to drive load against a server, and then measure server response times and server response errors. Because we want to generate high loads with a relatively low amount of hardware, we chose to drive Web performance tests at the protocol layer rather than instantiating a browser. While Web performance tests can be used as functional tests, this is not their primary focus (see my post Are Web Tests Functional Tests?). You will see that I still refer to “Web Performance Tests” as “Web Tests” for short.

If you really want to test the user experience from the browser, use a Coded UI test to drive the browser.

In order to be successful working with Web Performance Tests, it is important you understand the fundamentals about how they work.

Web Performance Tests Work at the HTTP Layer

The most common source of confusion is that users do not realize Web Performance Tests work at the HTTP layer. The tool adds to that misconception. After all, you record in IE, and when running a Web test you can select which browser to use, and then the result viewer shows the results in a browser window. So that means the tests run through the browser, right? NO! The Web test engine works at the HTTP layer, and does not instantiate a browser. What does that mean? In the diagram below, you can see there are no browsers running when the engine is sending and receiving requests:

What Does This Mean for You?

This design has fundamental and far-reaching impact if you are working with Web tests. It is critical that you understand this if you are going to be successful authoring and debugging Web tests. This escapes even customers who have worked extensively with Web tests, and is a major source of confusion. The Web test engine:

Sends and receives data at the HTTP layer.
Does NOT run a browser.
Does NOT run java script.
Does NOT host ActiveX controls or plugins.

Ok, so Web tests work at the HTTP layer. What about requests sent and received by javascript and/or browser plugins? The best example for java script generating HTTP traffic is AJAX calls. The most common example of browser plugins are SilverLight or Flash. The Web test recorder will record HTTP traffic from AJAX calls and from most (but not all) browser plugins.

Web Tests Can Succeed Even Though It Appears They Failed

A common source of confusion comes from the browser preview in the Web test result viewer. This browser control does not run java script nor host plugins, which is by design since the engine does not do this either, and for security reasons. A common technique in pages requiring java script is to sense this, and put up an alternate page when the browser is not running java script, such as “java script required on this page”:

This page looks like it failed, when if fact it succeeded! Looking closely at the response, and subsequent requests, it is clear the operation succeeded. As stated above, the reason why the browser control is pasting this message is because java script has been disabled in this control.

Another variant of this is plugins such as this page that is using SilverLight:

Again, it looks like the page failed, when in fact at the HTTP layer it succeeded.

A Common Challenge: Dynamic Parameters

One of the major challenges with working at the HTTP layer are “dynamic parameters”. A dynamic parameter is a parameter whose value changes every each time it is run. The most common case is a session parameter, such as a login session ID. Each time a user logs in to a site, he is given a new login session ID. In order to simulate this user’s actions, the test cannot simply replay the recorded session ID, it must replay the new session ID. Web tests handled most patterns of dynamic parameters automatically, but there are still some patterns it does does not automatically handle.

Huge Strides Forward with VS 2010

With ever more complicated applications being built on HTTP, it is getting harder and harder to develop scripts at the HTTP layer.

With VS 2010, we again have made tremendous strides across the tool, in recording, editing, and debugging, so help you be successful doing this. Some of the high-level features are:

Finding and fixing dynamic parameters
Enabling an extensibility point in the recorder such that record/playback “just works” for any app (effectively enabling you to automate #1).
Enabling extensibility points for editing and viewing results of rich data types

We have made a number of other improvements as well, most notably:

Editor Improvements

Support for looping and branching in Web tests
Request details editor
Create meaningful reports using the reporting name on Page
Goal validation rule

Recorder Improvements

Record/playback of file upload “just works”
Record all dependents by default
New recorder options in Tools Options
Improvements in hidden field and dynamic parameter correlation

Debugging a Web Test to Find and Fix Dynamic Parameters

The VS recorder automatically handles most classes of dynamic parameters: cookies, query string and form post parameter values, and hidden fields. In VS 2010 we have made incremental improvements on each of these. Yet there are still a few dynamic parameter patterns that cannot be deterministically detected and fixed up.

Our goal with this release was to build tooling around the flow for debugging a web test, mostly to help find and fix dynamic parameters. This flow is described in Sean’s seminal post, How to Debug a Web Test. The flow is this:

Record a Web test and play it back. Playback fails.
Look at the form post and query string parameters in a failed request and determine if any look to be dynamic
Go to the web test to determine if they are bound to an extracted value
If not, search through the log to find where the parameter is set. Or better yet, search through the recording log to find the unique value in order to find where it is getting set.
Add an extraction rule and bind the parameter value to the extracted value.

In VS 2010, you’ll find commands in Web test playback and the editor that seamlessly support this flow:

A new recorder log that enables you to see the http traffic that was generated from IE. This is a huge new feature critical for debugging tests. You can jump from a request, request parameter, or response in playback to the same context in the recording log to compare them.
Search in playback and search and replace in the Web test editor. These features are super-important for quickly finding and fixing dynamic parameters.
Jump from a request in playback to that same request in the editor. This greatly increases the efficiency of the workflow.
Create an extraction rule directly from playback, automatically setting the correct parameters on the extraction rule. Again, this increases efficiency.

Together, these features really grease the debugging and fix up workflow, and will make you much more efficient when working with web tests.

A quick overview of the flow:

From the Web test results viewer, select a from a failed request that looks like a dynamic parameter. Right-click from the parameter on the Request tab to jump to the editor to confirm it was not bound.

In the editor, you can see this value is not bound to a context parameter:

Now go back to the results viewer. At this point, you want to find the dynamic values in the response of one of the previous requests, as the dynamic parameter value had to have come from a response body (since that’s how http and browsers work). To do this, you want to go to the recorder log. The reason you want to do this from the recorder log is that the recording will have the original recorded value in it. To do this, click on the recorder log icon (we really should have put this on the context menu too!).

This will take you to the same request with the same parameter selected. Now right-click on the parameter and do a quick find to find the parameter value in a previous response. Again, you want to do this from the recording log, since the parameter is dynamic the value will be in the recording log but not the playback log.

Search up in response bodies to find the value. Note that if the dynamic string was constructed in java script, you may need to only search of the dynamic part of the value:

Once you find it, right click to add an extraction rule:

Once the extraction rule is added, you also need to bind the parameter values. Choose yes to the message box to launch search and replace from the Web test editor.

You can see that we have added tooling to make finding and fixing dynamic parameters much easier in VS 2010!!!

Engineering the Solution

To engineer this solution, we made several important design changes to Web tests and Web test results.

First, we changed the persistence mechanisms for Web test results to store results to a separate log file rather than the in the trx.
We created a full public API for the Web test result.
We stamp request ids in each http request (enables jumping between playback and the editor).
The recorder generates a Web test result file and saves it as part of the recording.

About the Web Performance Test Recorder Log

The Recorder Log is a file stored in the same directory as the web test is recorded into. You can get to the recorder log from the Web test results viewer as shown above. Or you can open it from VS, browse to the Web test folder and look for *.webtestresult to find recorder log files in your project folder. The name of the recorded result file is stored in the RecordedResultFile attribute in the web test xml file. This file is not added to the project by default, if you wish to share it with team members consider adding it to the solution so it will get checked in to source control.

The recorder log is persisted in the same file format as a Web test result. There is a full API over this data (see the WebTestResult and WebTestResultDetails classes).

Adding Your Own Recorder Plugins to Make Record/Playback “Just Work”

Once you have found and fixed up the dynamic parameters in a test, consider writing a recorder plugin to do this for you automatically each time you record a new test for this web site.

Recorder plugins are a new, super-powerful capability to the VS 2010 recorder. Recorder plugins are an extensibility hook that gives you full access to the recorded result and the recorded Web test, and move seamlessly from a recorded request to that corresponding request in the web test. This enables you to make any modifications you see fit to the generated Web test. This is in effect a “catch-all”, the ultimate power and productivity tool in your hands to save time fixing up Web tests.

I really can’t emphasize enough what a powerful solution this is. If you will be scripting a web site for any given period of time, and it requires you fix up the recordings, it will be worthwhile for you to invest in building a recorder plugin for it.

Recorder plugins can be used for any number of reasons: fixing up dynamic parameters (adding extraction rules and bindings), automatically adding validation rules, automatically adding data sources and doing data bindings, filtering out recorded dependents, etc.

Recorder plugins are pretty straightforward to code up and install. Recorder Plugins derive from the WebTestRecorderPlugin class. Once you have implemented a plugin, just drop the assembly into either of these directories, and then restart VS:

%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\WebTestPlugins
%USERPROFILE%\My Documents\Visual Studio 10\WebTestPlugins

Here’s sample recorder plugin code that adds an extraction rule and binds query string parameters to the extracted value.

using System;
using System.Collections.Generic;
using System.Text;
using System.ComponentModel;

using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;
using System.Diagnostics;

namespace RecorderPlugins
{
    [DisplayName("Correlate ReportSession")]
    [Description("Adds extraction rule for Report Session and binds this to querystring parameters that use ReportSession")]
    public class CorrelateSessionId : WebTestRecorderPlugin
    {
        public override void PostWebTestRecording(object sender, PostWebTestRecordingEventArgs e)
        {
            // Loop through the responses in the recording, looking for the session Id.
            bool foundId = false;
            foreach (WebTestResultUnit unit in e.RecordedWebTestResult.Children)
            {
                WebTestResultPage recordedWebTestResultPage = unit as WebTestResultPage;
                if (recordedWebTestResultPage == null)
                {
                    continue;
                }
                // If we haven't found the session Id yet, look for it in this response.
                if (!foundId)
                {
                    // Look for the "ReportSession" string in the response body of a recorded request
                    int indexOfReportSession = recordedWebTestResultPage.RequestResult.Response.BodyString.IndexOf("ReportSession");
                    if (indexOfReportSession > -1)
                    {
                        // Find the corresponding page in the test, this is the page we want to add an extraction rule to
                        WebTestRequest requestInWebTest = e.RecordedWebTest.GetItem(recordedWebTestResultPage.DeclarativeWebTestItemId) as WebTestRequest;
                        Debug.Assert(requestInWebTest != null);

                        if (requestInWebTest != null)
                        {
                            foundId = true;
                            string startsWith = "?ReportSession=";
                            string endsWith = "&";
                            string contextParamName = "ReportSession";
                            AddExtractTextRule(requestInWebTest, startsWith, endsWith, contextParamName);
                            e.RecordedWebTestModified = true;
                        }
                    }
                }
                else
                {
                    // Once we have extracted the session id, bind any session id parameters to the context parameter
                    // This call gets the corresponding request in the web test.
                    WebTestRequest requestInWebTest = e.RecordedWebTest.GetItem(recordedWebTestResultPage.DeclarativeWebTestItemId) as WebTestRequest;
                    Debug.Assert(requestInWebTest != null);

                    if (requestInWebTest != null)
                    {
                        BindQueryStringParameter(requestInWebTest, "SessionId", "SessionId");
                    }
                }
            }
        }

        /// 
        /// Code to add an ExtractText rule to the request.
        /// 
        /// 
        /// 
        /// 
        /// 
        private static void AddExtractTextRule(WebTestRequest request, string startsWith, string endsWith, string contextParameterName)
        {
            // add an extraction rule to this request
            // Get the corresponding request in the Declarative Web test
            ExtractionRuleReference ruleReference = new ExtractionRuleReference();

            ruleReference.Type = typeof(ExtractText);
            ruleReference.ContextParameterName = contextParameterName;
            ruleReference.Properties.Add(new PluginOrRuleProperty("EndsWith", endsWith));
            ruleReference.Properties.Add(new PluginOrRuleProperty("StartsWith", startsWith));
            ruleReference.Properties.Add(new PluginOrRuleProperty("HtmlDecode", "True"));
            ruleReference.Properties.Add(new PluginOrRuleProperty("IgnoreCase", "True"));
            ruleReference.Properties.Add(new PluginOrRuleProperty("Index", "0"));
            ruleReference.Properties.Add(new PluginOrRuleProperty("Required", "True"));
            ruleReference.Properties.Add(new PluginOrRuleProperty("UseRegularExpression", "False"));

            request.ExtractionRuleReferences.Add(ruleReference);
        }

        public static void BindQueryStringParameter(WebTestRequest requestInWebTest, string queryStringParameterName, string contextParameterName)
        {
            // This code adds data binds the SessionId parameter to the context parameter
            foreach (QueryStringParameter param in requestInWebTest.QueryStringParameters)
            {
                if (param.Name.Equals(queryStringParameterName))
                {
                    param.Value = "{{" + contextParameterName + "}}";
                }
            }
        } 
    }
}

More Recorder Enhancements in VS 2010

In addition to lighting up these powerful new scenarios, the VS 2010 does what it did in VS 2008 only better.

No More Empty Recorder Pane

With VS 2008, there were several cases for which the recorder would not record requests. Most of these involved the IE 7 and IE 8 process model, where these browsers start new processes when crossing security contexts (thus the need to run VS as Admin). These problems have been fixed in VS 2010, as the recorder now records across IE process boundaries.

More Apps “Just Work”

There were a few cases for which hidden field correlation and dynamic parameter detection did not work with VS 2008. You let us know about those cases, and we have improved the hidden field correlation and dynamic parameter detection tools to handle them in VS 2010. These were mostly around dynamic parameters in AJAX requests.

And binary post bodies are now handled correctly, which were not always handled correctly with VS 2008.

The recorder now also automatically handles File Uploads so they will “just work”. Files that are uploaded are automatically added to the project, and the file upload file name will be dynamically generated to enable you to upload the same file to different names automatically.

New Tools Options for the Recorder

You asked for more control over the recorder, you got it with new recorder options in Tools Options:

Web Test Editor Enhancements in VS 2010

One of our goals with VS 2010 was to enable you to stay in “declarative” Web tests for more use cases without having to move to a coded Web test. One reason you had to go to code with VS 2005 or VS 2008 was to do looping or conditional execution in your Web test.

Looping and Branching

The VS 2010 declarative editor now supports powerful new looping and branching constructs. Looping and branching are based on conditional rules, which follow the exact same extensibility model as validation rules and extraction rules. There are many rules “in the box”:

You can see above, there are a lot of flexible rules already built in.

A few scenarios this enables:

Conditional logins. In a load test, if you want to simulate a user logging in once and then doing many operations in the test, this can be accomplished easily in a conditional rule. Session IDs are typically handled by cookies, and you can easily set up a rule to only go to the login pages if the login has not happened yet.
Variability in your scripts. If you want users to occasionally skip steps in a script, or randomly repeat some steps, this is easily achieved with the probability rule which will only execute some requests based on the probability you specify.
Loop until some operation succeeds. If an operation is expected to fail for some users, but will succeed on retry, and you need to model the retry, you can do this by looping while the operation is not successful. To do this, use an extraction rule to indicate whether or not the action was successful, then use the Context Parameter Exists to loop until it is successful.

You can debug your loops and conditions using the results viewer, which shows the results of conditional evaluations.

A word of caution: do not use loops to do many, many loops within a given test. This will “gum up” the load test engine, since it function is to control virtual user execution. Also, an entire web test result is stored in memory, including all the loops. So running a web test with many loops will run your machine out of memory. You can still run these in a load test to avoid this, but for the reason stated above we recommend against this.

More Editor Features

I already talked about search and replace in the editor above. There is also a super-handy new Request Details editor that enables you to quickly see and edit the think times, reporting name, and goal for each page. You should use this view each time you record a new test.

Use the Reporting Name and Response Time Goals to really light up your excel load test reports, as both are propagated to the reports.

Setting the response time goal will also help you to find slow requests in a load test, as by default there is a new Response Time Goal validation rule added to the test. This rule will fail pages that exceed the goal by the threshold you specify (by default the tolerance is 0). This rule will cause slow requests to fail, and enable you to collect logs on the failures, which may help you determine why the page is slow.

New Extensibility Points to Handle Rich Data

One area we did not address in VS 2010 is better editor and result viewer handling of rich data types. If you have AJAX code sending and receiving Web services, REST, or JSON requests, you know how difficult these are to work with. Like other releases, our mantra was if we couldn’t get it in the box, we wanted to expose extensibility points to enable us and the community to add tooling to help in this area.

To this end, we have enabled two extensibility points that will enable us to address this out of band:

Web test editor request body editor plugins.
New tabs and menu items in Web test result viewer.

We plan to release new editor and playback plugins around the time we RTM, so keep an eye on codeplex.com\teamtestsplugins for new releases.

Web Test Editor Request Body Plugins

Web Test request body plugins provide a way to plug a custom editor into VS for editing form post bodies. These plugins implement either IStringHttpBodyEditorPlugin or IBinaryHttpBodyEditorPlugin, and enable you to customize the edit pane for different post body content.

The IStringHttpBodyEditorPlugin interface is super-simple:

public interface IStringHttpBodyEditorPlugin
{
    object CreateEditor(string contentType, string initialValue);
    string GetNewValue();
    bool SupportsContentType(string contentType);
}

Basically, SupportsContentType allows you to specify which content types your editor supports. When the editor encounters a particular content type, it will scan the list of editor plugins for the first one it finds to support that type, then host the editor control. The CreateEditor call is used by the plugin to instantiate an instance of the control and provides the initial value to be edited, and the GetNewValue is the way the plugin returns the result of the editing session. The IBinaryHttpBodyEditorPlugin is the same, except that it gets and puts byte arrays.

public interface IBinaryHttpBodyEditorPlugin
{
    object CreateEditor(string contentType, byte[] initialValue);
    byte[] GetNewValue();
    bool SupportsContentType(string contentType);
}

We are working on creating new editors for the most common formats now, and will ship “out of band” to codeplex.com\teamtestplugins around RTM. Here’s a screen shot of the editor handling msbin1 data in a rich way (I rubbed out some URLs of this public facing site):

Web test editor plugins must be deployed to %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\WebTestPlugins.

Web Test Result Viewer Plugins

The Web Test Result Viewer also supports design time plugins. There are many scenarios for these plugins, here are some example scenarios:

The coolest comes from dynaTrace.
Tools that automatically analyze the result to point out potential performance problems (see blogs.msdn.com\mtaute for an example).
Custom viewers for rich request and response data.

This third scenario is the one I want to delve into more in this section. Just as you want a rich editing experience for working with Web services, REST, or JSON requests, you want a rich way to view this data in the result viewer as well. The Web test result viewer plugins provide the perfect extensibility point for this.

Result viewer plugins are a bit more involved to code up and install the editor plugins.

Like the response body editor, we are working on out of band plugins for the Web test result viewer. Here is a screen shot of the result view plugin for binary data:

Notice the tree views in the bottom panes, showing binary data as a tree.

Conclusion

Your takeaway after reading this blog post should be - “Wow, VS 2010 is fantastic and will save me tons of time creating and maintaining Web tests, I have to have it!”

By working with you directly on the forums and through our blogs, we saw the types of problems you are hitting developing scripts. We also listened to your feedback and folded it back into the tool. In places we didn’t have time to address, we’ve added extensibility points to enable us to deliver features to you out of band, and for you to create your own solutions.

Now you can say: “I’m a performance tester, and Visual Studio 2010 was my idea!”

Ed.

dynaTrace + VS 2010 Load Tests = 1 Powerful Solution for Performance Analysis of Java Applications

Ed Glas — Wed, 10 Mar 2010 20:27:00 GMT

dynaTrace offers a terrific integration with VS 2008 to enable performance analysis of java and mixed .NET + java applications.

The integration is now even deeper with VS 2010. With VS 2010, you can drill straight from a specific slow Web page in a load test to the dynaTrace truePath for that page. The truePath shows the performance breakdown for the page, detailing which tiers of the application were slow for that page. This enables you to jump immediately to the performance problem.

This is similar to the functionality offered in the integration between VS load tests and the VS profiler, but goes even further as the dynaTrace solution works for both .NET and java applications, and can even trace performance across .NET and java tiers, for example if your .NET web application makes a call to a Web service running in a java application server.

Check out Andreas’s blog post to see a walkthrough of the integration.

The combination of these two tools makes a super-powerful load testing solution for shops that run both java and .NET applications.

Ed.

Application Benchmark Toolkit for Microsoft Dynamics AX

Ed Glas — Fri, 05 Mar 2010 15:23:30 GMT

The Dynamics AX product team is a long-time internal customer of VS Load Tests, and use VS to do all their product load testing.

They released the Benchmark Toolkit to customers, which enables Dynamics load testing. This is a huge win for Dynamics AX customers, as it provides a way for them to find performance problems before moving their Dynamics solutions to production. Before the combination of the toolkit and VS, performance testing was difficult and expensive, and therefore often did not happen. It is great to see Raymond and his team release this to their customers.

Ed.

Product Transitions from VS 2008 to VS 2010 for Load Testing

Ed Glas — Wed, 03 Mar 2010 14:05:00 GMT

If you are currently using VS 2008, here is how I compare the products and components you use for load testing.

VS 2008

VS 2010

Description

Key differences

VS Team System Test Edition

VS Team System Suite

VS Ultimate 2010

Client for developing load tests and web tests, running tests, and analyzing results. Enables limited “local” load generation from the same machine VS is on.

Test Edition is gone, load testing is only available in Ultimate.

Both VS 2008 and VS 2010 limit local load generation to one core. However, VS 2010 has the addition restriction of only allowing 250 virtual users.

Visual Studio Team System 2008 Test Load Agent

Visual Studio Load Test Virtual User Pack 2010

Product required for generating load

VU Packs are installed on the controller, and can be used on any number of agents.

VU Packs are just “paper licenses” that enable load generation.

VU Packs are tied to a controller, much like in VS 2008 Load Agent licenses were tied to the agent machine.

Test Controller

Visual Studio Test Controller

Distributes tests to agents, manages automated test runs, centralizes data collection from agents.

In VS 2008, the test controller is free when you purchase a load agent.

In VS 2010, the test controller is free when you purchase VS Ultimate, Premium, or Test Pro.

The 2010 test controller is used in a number of additional scenarios:

Lab management and provisioning test environment

Manages data collection agents for manual testing

Manages automated runs on Test Agents

Visual Studio Test Agent

The test agent allows distributed, parallel functional and unit test execution, as well as distributed data collection. The agent is also used as a load generator in conjuction with the Virtual User Pack on the controller.

VUser licenses are required on the controller to generate load.

See Test Controller for additional scenarios that are enabled.

If you licensed VS Team System Test Edition 2008 or Suite 2008 under SA, you can transition your license to VS Ultimate 2010 under the SA agreement. Similarly, if you purchased load agent licenses under SA we will also provide a transition from Visual Studio Team System 2008 Test Load Agent to Visual Studio Load Test Virtual User Pack 2010. Please refer to Licensing Whitepaper, EULA and/or Product Use rights for licensing requirements.

Ed.

Parameterizing Tests to Run in Different Environments

Ed Glas — Tue, 16 Feb 2010 14:24:53 GMT

Sean has a great post on running Web tests and load tests from the command line.

A hidden feature is that you can set parameter values from environment variables, which enables you to target different environments with the same test without editing the test.

You may develop your web tests and load tests in one test environment, and then need to re-run it against another environment. A common practice is to have test, preproduction, and production environments.

In Web tests, you can use the Parameterize Web Server command to put the server part of the URL in a context parameter.

Now my target server has been parameterized out to the WebServer1 context parameter. Depending on your test, you may also need to add additional parameters. For example, some Web applications send the URL in a query string. In VS 2010, use the search command to find all instances of the target server and replace it with a context parameter.

That’s great, but does that mean you need to edit the Web test to target a different environment? No!

There are two ways to override Web test context parameters:

1) override the value from an environment variable. An environment variable with the name Test.WebServer1 will override the value of the WebServer1 context parameter. In a similar way you can override variable values in a unit test.

2) create a context parameter with the same name in the load test. The load test parameter value will override the Web test value. Here I’ve created a run settings that targets the test environment. Notice that the run setting is configured to collect counters from the target web server, and it overrides the context parameter value to set the same web server.

Now I can create a second Run Settings in my load test to target the pre-production environment:

Now I have two run settings in my load test, each targeting a different environment.

Great, does that mean I have to edit my load test to target a different environment? No!

Here is the real hidden gem: you can override the default run settings in your load test by setting an environment variable from the command line:

Set Test.UseRunSetting=PreProdEnvironment
mstest /testcontainer:loadtest1.loadtest

These features have been in the product since VS 2005.

Ed.

Install Visual Studio 2010 RC Today

Ed Glas — Tue, 09 Feb 2010 02:29:18 GMT

Visual Studio 2010 RC is available on the Web today for MSDN subscribers. Go to the RC landing page for details.

We have a very short window, just a few weeks, for incorporating feedback from you back into the product for RTM. Please install the RC today, so we can fix any critical issues you find in the product before RTM. We’ve already fixed a couple of bugs since the RC in the load testing product, generated from internal adopters of the RC. Overall feedback so far has been very positive – people love this release!

Between Beta 2 and RC we’ve made substantial improvements across VS, especially focusing on performance and reliability of VS.

We have been very busy since releasing beta 2. Of course we’ve fixed bugs across the feature set and also rounded out some rough edges, including:

Web Test Recorder

With the RC release, we now have addressed the most common problems as described in Diagnosing and fixing Web Test recorder bar issues. We have also fixed some patterns around correlation and hidden field binding in AJAX requests that we were not handling properly before.

Handle security context switches in IE7 and IE8 in the recorder
Handle process boundaries in IE8 in the recorder
Handle Window.Open going across process boundaries
Two pass hidden field matching (better matching for ajax requests)
Better dynamic field detection (handles cases where referrer not set)

Web Test

Support for binary post bodies, which includes a new option in tools-options-web test, and an extensibility point to override the form post body editor control. Customers testing SilverLight clients calling via WCF will find this pattern now “just works” in the RC.
Parameterize server command now handles urls in query string parameters. This is a pretty common pattern, as seen in SharePoint.

Analyzer

We made really nice changes in the analyzer to address feedback you’ve given us.

Details view: filter by error type. This gives a super-powerful way to visualize certain errors.
Details view: Paint overview graph on details zoom bar. This enables you to see what was going on in the test at a particular point in time.
Maximize use of space in graph view for graphs: make zoombars and fonts smaller, move vertical zoom
Counter Range Groups sets like ranges for like counters (e.g. two instances of the same counter always use the same range, or related counters such as Bytes Sent, Bytes Received, and Bytes Total)
Use transaction request time, not elapsed time, in stats
Show transaction stats in tables view
Differentiate GET and POST for the same urls in reports

Reporting

Select default counters for excel reports. This makes it a lot easier to create your first report.
Excel reports now work on 64 bit Excel

Load Test API

Expose run id in load test object

Please install the RC and send us your feedback.

Thanks,

Ed.

Configuration Options for Load Testing with Visual Studio 2010

Ed Glas — Sun, 07 Feb 2010 22:14:54 GMT

In this post I outlined the products you need for load testing with Visual Studio 2010. I included a diagram with one configuration option for load tests. Here I will discuss other configurations available.

Configuration 1: “Local” Load Generation

This is what you get when you install Visual Studio Ultimate, which is the ability to generate load “locally” using the test host process on the same machine that VS is running on. In addition to limiting load to 250 users, it is also limited to one core on the client CPU.

Note that purchasing Ultimate also gives you the ability to collect ASP.NET profiler traces by using a Test Agent as a data collector on the Web server.

Configuration 2: Distributed Test Controller and Test Agents

This is a common configuration if you are scaling out your load agents. With this configuration, the Test Controller and each Test Agent is on a separate machine.

The advantage of this configuration is the controller is easily shared by team members, and overhead from the controller does not interfere with load generation nor operation of the client.

Note the Test Controller must have one or more Virtual User Packs installed to enable load testing. Load agents in this configuration always use all cores on the machine.

Configuration 3 A and B: Stacked Configuration

With configuration A, you install the Test Controller and Test Agent on the same machine as VS, then configure the Test Controller with Virtual User Packs. This enables you to generate >250 virtual users from the client machine, and unlocks all cores in the processor. Configuration B shows an alternative configuration, enabled if you configure the machine with Virtual User Packs using the VSTestConfig command line.

Note that a Virtual User Pack can only be used on one machine at a time, and configuring it on a machine ties it to that machine for 90 days. So you can’t have the same Virtual User Pack installed on both the VS client and a separate machine running the Test Controller. See the Virtual User Pack license for details.

Configuration 4: Stacked Controller, Distributed Agents

In this configuration, the controller is running on the same machine as the Test client, with distributed agents running as load generators. This configuration is recommended if you have a solo performance tester. If your test controller and test agents will be shared by a team, we recommend running the controller on a separate box. Note that test agents are tied to a single test controller. You can’t have two test controllers controlling the same agent.

Conclusion

If you are using Visual Studio 2008, these options should look familiar to you as the VS 2008 load agents and controller offered the same configuration options. The new twist with VS 2010 is the Virtual User Packs, which, as I explained in this post, offer you more flexibility in how you configure your load agents.

Ed.

VS 2010 Product SKU Lineup Finalized

Ed Glas — Sun, 07 Feb 2010 01:40:00 GMT

Here are the VS 2010 SKUs, and testing features available in each (see this link for more feature comparisons).

Client SKUs

Name

Test tools it contains

Visual Studio 2010 Ultimate

Test Tools in VS Premium

Load Testing

Network Emulation

IntelliTrace

Microsoft Test Manager (manual testing and test case management)

Test Agents, Test Controller

Visual Studio 2010 Premium

Test Tools in Pro

ASP.NET Profiler

Coded UI Tests (UI Automation)

Test Data Generator

Visual Studio 2010 Professional

Unit tests

Visual Studio Test Professional 2010

Microsoft Test Manager

Test Agents, Test Controller

Server SKUs

Name

Test tools it contains

TFS

Includes TCM Server, and some lab management components

Visual Studio Lab Management 2010

VMM Server, Visual Studio Agents

Visual Studio Agents 2010

Test Agent, Test Controller, Lab Agent

Visual Studio Load Test Virtual User Pack 2010

A license key (no extra software)

Note that the Lab Management product will be shipped sometime post VS 2010 RTM. An RC build will be available when VS 2010 ships. We are looking for more customer feedback before we RTM the product.

Now, looking at a typical load testing configuration:

Just using Visual Studio Ultimate enables you to generate 250 virtual users of load. To go higher than 250 users, you need to purchase a Virtual User Pack, which gives you 1000 users. You can use the 1000 users on any number of agents. Note that if you install the Virtual User Pack on the same machine as Visual Studio Ultimate, you do not get 1250 users on the controller. The 250 virtual users you get with Ultimate can only be used on “local” runs, not on a Test Controller. If you need to generate more 1000 users, you purchase additional Virtual User Packs, which aggregate or accumulate on the Test Controller. In other words, installing 2 Virtual User Packs on one controller gives you 2000 Virtual Users, which can be run on any number of agents.

The Test Controller and Test Agent are “free” when you purchase Ultimate.

Transitioning from Visual Studio 2008

If you are transitioning from Visual Studio 2008 Team System Edition for Testers, or Visual Studio 2008 Team Suite, and and have purchased Software Assurance, you will automatically be transitioned into Visual Studio Ultimate 2010.

If you have purchased Visual Studio Team Test Load Agent 2008 under Software Assurance, you will be granted 5 Virtual User Packs.

VS 2010 Compatibility with VS 2008

All of your test assets will transfer 100% from Visual Studio 2008: Load tests, web tests, unit tests, and test projects. On opening the test project, you will be prompted to convert the project to VS 2010. Clients and test controller/test agents are not compatible with each other. So 2008 clients can’t talk to a 2010 controller, and visa-versa. Also 2008 agents and controllers do not mix with 2010 agents or controller. This is the same constraints we had when moving from VS 2005 to VS 2008.

One thing that does not get upgraded is the load test results database. On installation of VS 2010, a new database is created. If you want to continue to work with previous load test results, you can use the load test results manager in VS 2008 to export them from the VS 2008 database and then use VS 2010 to import them into VS 2010.

Conclusion

I hope you find the move to VS 2010 a smooth one, I certainly expect you will. There are many, many new features in VS 2010 that you’ll want to take advantage of, overall I highly recommend you upgrade as soon as you can, it will be worthwhile for you to do so.

New Web and Load Test Features in Beta2

Ed Glas — Wed, 21 Oct 2009 20:42:00 GMT

In beta2 we added a number of new minor features, and a couple of major ones.

We also changed our licensing model, which you can read about here.

Here is a list of the minor features we added, along with a brief description of each. The last one in the list, compare two runs in Excel, is probably my most favorite feature in the entire release.

Area

Title

Description

Recorder

Recorder options in tools options

Starting URL supplies a default url to start recording from.

Two new options to control whether or not the recorder does hidden field matching, and whether or not it runs the dynamic parameter detection tool after recording completes.

Recorder

Recorder records all requests by default, fold dependents under page

In 2008, static dependents not found by the dependents parser were not recorded by default. Now they are.

Web test

search/replace

Search and replace in web test requests!

Playback

Add request bytes column

Request byte count is now shown in request list pane in playback (in addition to response byte count)

Playback

PreRequestDataBinding event

This new event fires before data binding occurs, allowing a plugin to dynamically change data bindings in a request.

Data binding

64 bit csv

This fixes a limitation we had in beta1, and now enables csv data sources on 64 bit machines.

Data binding

New APIs on WebTest to control data cursor position and reload data

Give full control over the cursor, enabling tight control of which user is using what data. Also reload enables refreshing a datasource during the test.

Load Test

Scrub counter sets

We refreshed the set of counters being collected, thresholds, and default counters.

Analyzer

Add standard deviation and 99^th percentile stats to load test analyzer

Analyzer

Graph instance 0 for CPU on local runs by default

Fixes the problem documented in this post.

Analyzer

Specify range for a counter (CPU range is 100)

Now you can specify a range for a given counter. Also percentage counters get a range of 100 by default.

Analyzer

Fixed range on perf graph

The performance graph now gets a default range set for it (range is rounded up response time of the slowest measurement)

Analyzer

Computer tags

Computer tags enable you to classify computers by function (e.g. web server, db server, etc.). Also enables run to run comparison even if the runs are on different physical machines.

Reporting

Compare two runs in Excel

This is a new, rich reporting option that enables you to compare performance and resource utilization data in two different runs.

Ed.

Getting Started with Visual Studio 2010 Beta 2

Ed Glas — Tue, 20 Oct 2009 18:59:35 GMT

As you are probably already aware, Visual Studio Beta 2 is now available for download, see the landing page here.

For load testing, you will need to download and install:

Visual Studio Ultimate Beta 2 (x86)

Visual Studio 2010 Team Agents Beta 2 (x86)

Ultimate is the new equivalent of Team Suite, and contains Web performance tests and load tests.

Team Agents contains the controller and agents. Note that the x86 agent contains both 32 and 64 bit assemblies, so if you are looking for the 64 bit version of the load agent, this is the download you need.

As I posted here, we have moved to a virtual user license model. Ultimate is limited to 250 virtual users and one core of the CPU. To generate more than 250 users and use all cores on your machine, you will need to install a virtual user pack.

The most straight forward way to do this is to install a controller and agent (or multiple agents) on the same or separate machines.

We have re-architected the controller and agent setups such that setup does little more than lay down the assemblies, while a post-setup configuration tool does the heavy lifting to get the machine configured. The configuration steps are the ones that typically fail, so rather than having to reinstall the entire product and go back through all the configuration steps, if one part fails you only need to reconfigure that part. We have also enabled command line versions of both setup and the configuration tools.

Here is the controller config tool with options called out:

So you will need to fill in the controller service user using a user with counter collection access to your systems under test (same as in VS 2008), and then configure the load test database in the “Configure for load testing” section.

To add virtual users, start a Visual Studio Command prompt as Administrator (on Vista or Win7, just type Visual Studio Command Prompt in the search window, on XP you’ll find this under the Visual Studio group in the start menu). Then run this command line:

TestControllerConfig licenses /addkey:V10000

This will license the controller for 10,000 virtual users. Note that this license is temporary, and will only work in beta2. For RTM, we will only offer the 1,000 user license pack. If you have a Volume License, you will be able to get your license key from the volume licensing site, and then enter a count of licenses you want to install.

Once you have a virtual user license installed, that license can also be used for local runs (that is, the runs do not have to go through the controller).

If you just want to do local runs, and not use the controller and agent, you can run this from the command line to install the virtual user license:

VSTestConfig licenses /addkey:V10000

This enables you to exceed 250 users for local runs, and unlocks all the cores on your CPU.

Ed.

Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010

Ed Glas — Tue, 20 Oct 2009 14:52:00 GMT

In Visual Studio 2010, we are changing our load test licensing model. In VS 2005 and VS 2008, we shipped the load agent, which was licensed by CPU on the machine generating load. In Visual Studio 2010 we are changing our license model to a virtual user based model. This new license model offers a number of advantages for you, which is ultimately why we chose to make the change:

Still way less expensive than our competition.
Enables better purchasing planning and allows an apples to apples price comparison with competitors.
Enables more flexible machine allocation for load generation.
Enables you to use agents for more than just load testing.

Still Way Less Expensive than our Competitors

Your initial reaction may be “Groan – that means that Microsoft is going to be just as expensive as those other tools!” But that is not the case at all, we were careful to maintain the per-virtual-user price we set out initially with VS 2005. With VS 2005 and VS 2008 load agents, the price was around $5,000 per load agent CPU, and our general guidance was that for a typical load test expect around 1,000 virtual users per CPU. We are not yet disclosing our pricing yet, but I can tell you our goal is to stay in this same ball park with VS 2010.

Better Planning and Comparison

With the VS 2008 load agent, we often get questions from customers doing purchasing planning asking “How many load agents do we need to purchase? We are planning to test up to 5,000 virtual users.” To which our answer with VS 2008 was “we do not know, it depends on your application and test scripts.” Of course this it difficult to know how many agents to purchase until actually doing testing, which was often too late. With virtual user licenses, purchasers will know exactly how many licenses to buy.

The old CPU-based license model also makes it very difficult to compare prices with competitive solutions. Most competitive products are licensed by virtual user and data collectors. If you know how many virtual users you want to test to, but do not know how many agents you will need to generate that number of users, it is impossible to make a comparison. With the Visual Studio Load Test Virtual User Packs 2010, if you know the virtual user count you want to test to, you will know exactly how many licenses to purchase, and be able to compare directly with competitive solutions you may consider.

Enables More Flexibility with Load Generators

The virtual user license offers flexibility with the machines you use as load generators.

With VS 2008, to get the full value of the per-CPU license, you had to run on fast CPUs, often requiring new hardware purchases. Also the license was locked to a particular machine, restricting which machines you could use as load generators. With Visual Studio 2010, you can now run virtual users on older hardware with no “penalty”, as you can line up as many older machines as you wish to generate the target number of virtual users. Also, the virtual user license is now tied to the controller, not the agent machine, so you are allowed to move agent machines in and out of the agent pool.

For some tests, you may run out of some other resources before running out of CPU. The most common case for this in VS 2008 was load agents running out of memory before CPU (especially for web tests with very large post bodies or very large responses), which prevented customers from realizing the full value of their license. With the virtual user license, if this happens you can simply add more machines to the agent pool to achieve the load you are targeting.

Visual Studio 2010 Agents: More than just Load Generators

With the high cost of the agent license, we did not have a viable solution for distributed functional testing. In Visual Studio 2010, agents are now included at no extra cost with VS Premium, Ultimate, and Test Elements. We’ve added Data Collection facility to agents, and added new UI automation test type, which opens up a bunch of new scenarios for using agents.

What if I already have VS 2008 Load Agents?

If you have purchased VS 2008 load agents with software assurance, you will be able to migrate your existing licenses to a commiserate set of Virtual User License Packs. We will iron out the details and mechanics of this once we get closer to RTM.

Conclusion

You can see there are many advantages to you with the new licensing model.

Ed.

Using The Virtual User Activity Chart to Understand the VS Load Engine

Ed Glas — Mon, 19 Oct 2009 03:07:14 GMT

In dev10 we’ve added a new Virtual User Activity Chart that shows virtual user activity. Each row on the chart is a virtual user.

Covering the Basics of the Chart

See Sean Lumley’s post VSTS 2010 Feature- Load test virtual user activity visualization to see an over view of the Virtual User Activity Chart.

Below is the test view of the chart, which shows which shows which test a user is running. This load test has 25 user constant load, with three unit tests in it: Fast (takes 1 second to execute), Medium (takes 5 seconds to execute), and Slow (takes 10 seconds to execute), using the Sequential Test Mix (which I will go into more detail on later in the post).

Zooming in and using Paint to group 5 users at a time for easy counting, the chart shows 25 rows, one for each user.

Zooming in using the zoom bar on the chart, you can see user 0 first ran the fast test, then the medium test, then the slow test:

Which is confirmed by hovering over the purple Medium test bar:

This view shows you exactly what each virtual user was executing during the test, so it will enable you to see patterns of user activity, load patterns, and correlate failed or slow tests and requests with other virtual user activity.

Another use of this view is it really helps you understand what the load engine does.

In this post, I’ll review various properties in the load engine and show the details view so you can see the effect these properties have.

Understanding Test Mixes

First, I’ll show different test mixes and their corresponding details view, using tests that will accentuate the differences in the mixes.

I’ll use three tests, long, medium and short. Long runs for 10 seconds, medium for 5 seconds, and short for 1 second.

Test Mix Based on Number of Tests Started

Here is the Activity Chart for this test mix, with 34% Fast, 33% Medium, 33% Slow. With this mix, as each user finishes a test, the user randomly selects the next test based on the test weightings.

One think you’ll notice is the predominance of Yellow, and lack of Blue. You wouldn’t think it to look at the user details view, but at the end of this test, the count for each test type is roughly the same. Since the Slow test runs 10x slower than the fast test, there is 10x more yellow than blue.

Name

Scenario

Total Tests

Avg. Test Time (sec)

Slow

Scenario1

912

10.0

Medium

Scenario1

934

5.00

Fast

Scenario1

1,012

1.00

34/33/33 Test Mix Based on the Number of Virtual Users Starting Tests

What this mix is doing is trying to keep, and any given time, the mix of virtual users running the tests to hit the percentage specified. Given the fast test starts and ends so quickly, you would expect a lot of more iterations of the fast test which is exactly what we see, a lot more blue:

Name

Scenario

Total Tests

Avg. Test Time (sec)

Slow

Scenario1

472

10.0

Medium

Scenario1

952

5.00

Fast

Scenario1

5,329

1.00

Test Mix Based on User Pace

In this test, I’ll run the same three tests, and specify they each run 120 times per hour. I chose 120 since it will take 16 seconds to run all three tests, so if I run each twice a minute, I should get a nice graph with about half the time spent running tests. Of course, this mix is not intended to be used this way, it should be used to model your user behavior (how many times an hour do you expect a user to do a particular action).

Now we see a very different graph, with a lot of pacing time between iterations:

Name	Scenario	Total Tests	Avg. Test Time (sec)
Slow	Scenario1	174	10.0
Medium	Scenario1	177	5.00
Fast	Scenario1	183	1.00

Test Mix Based on Sequential Test Order

With this mix, each user simply runs the tests in the order you specify in the mix, which is easy to conceptualize and super-easy to see:

The Effect of Think Time Between Iterations and Warmup Duration Properties

As you can see from this last detail view, this test has all users going lock step through the application, all doing exactly the same thing at the same time. While this may be useful for doing stress testing, it is not realistic user behavior, which is why we have the other test mixes which do a better job of varying what users are doing.

Another way to vary what users are doing and introduce pacing into a scenario is by using Warmup Duration and Think Time Between Iterations. Think Time Between Iterations is especially important if your test only has a single web test in it (thus no think time), or you are running a unit test with no pacing (for unit tests this is implemented via sleep calls, but never use sleep calls in a web test).

Here’s what happens when I put a 30 second warmup in the last test:

Now you can see the virtual users stagger their start over 30 seconds.

And here is the same test (no warmup) with a 5 second think time between iterations:

The variations are due to the Normal Distribution think time profile that is applied by default. This profile will vary the specified think time using a normal distribution (bell curve).

The Effect of Percentage of New Users

Now I am going to clear the delay between iterations, and use 100% new users. What I’d expect here is that each user will only run one test, leading to a lot of rows in the view, which is exactly what happens:

Scrolling up vertically one screen shows this:

So you can see the load engine repeatedly starting new users. As one group of 10 users completes, the next group of 10 starts. New users have a clean cache and cookie container, which matter for web tests.

Different mixes interact differently with the percentage of new users:

Sequential Mix as you can plainly see above, we made the design decision on sequential mix to have each new user run each test once.
With Test Mix Based on Sequential Test Order and Test Mix Based on Number of Tests Started, each user runs one test.
Test Mix Based on User Pace ignores the percentage of new users altogether, as the mix relies on a repeat user to establish the proper pacing.

The Effect of Test Mix Init and Terminate Tests

Init and terminate tests one once per each user. Switching my test back to 0 percent new users, and adding an init and terminate test shows the following:

Where the legend shows

This clearly shows the init test running once per virtual user, as the very first test executed, exactly as it should.

But wait! Where is the terminate test? It is not in the legend, and does not show up scrolling the view all the way to the end of the test. This shows the terminate test only gets called when a cooldown period is configured in the load test. Re-running with a 30 second cool down demonstrates this:

Understanding Load Patterns with the Virtual Users Activity Chart

Each of the above tests uses a constant user load of 25 users. Now I’ll use the chart to show how Step Load and Goal-based load based loads add users. For these tests, I’m going to use just one test in the mix, the 1 second fast test, to draw attention to the load pattern rather than the test mix.

Step Load

Here’s the Activity Chart for a step load pattern adding 10 users every 10 seconds until 100 users is reached. Notice the vertical scroll bar to scroll up to see the remaining users. Zooming in on the first 30 seconds shows the first 30 users created.

The step load Step Ramp Time property enables you to ramp the users added at each step. I highly recommend using this for every step load pattern, particularly if you are using large steps. Here I set a step ramp of 5 seconds, which shows users ramped up for each step over 5 seconds.

Goal Based Load

For a goal based load pattern, it is important that the counter you are seeking the goal on has a direct correlation with the load at each interval. So for this demonstration I put a tight loop in my test to provide a direct correlation between CPU on the load agent and the virtual user count. For this test, I also did a step load before hand to see what a good CPU range would be.

Here’s user load:

Let’s look at the Activity Chart for the areas where the dips are:

Here you can see virtual users stopping as they finish an iteration in order to achieve a lower load.

Scrolling up, you can see that new users are added to the test to move the load back up.

One thing that surprised me here is that users are not recycled. That looks like a bug to me!

Conclusion

Hopefully this post unravels some of the properties of the load engine, and it shows you just one of the ways you can leverage the new Virtual User Activity Chart in dev10 to understand exactly what your load test is doing!

Ed Glas's blog on VS load testing

A New Role for Me

Interviewing for an SDET job at Microsoft

Plugins for Handling JSON and msbin1 SilverLight Payloads on codeplex

MSDN Docs Updated for July

Version 3.5 of the Performance Testing Quick Reference Guide Available

New SharePoint 2010 Load Test Toolkit Released

Visual Studio Performance Testing Quick Reference Guide (Version 2.0) Published

Updated Load Test Links for April 2010

Great New Documentation for Performance Testing Now Online

VS 2010 Ships!

Simulating Peak Load with VS Load Tests

Process Model Changes in VS 2010

VS 2010 Test Agent and Test Controller Troubleshooting Guide

Web Test Authoring and Debugging Techniques for Visual Studio 2010

Contents

Overview

A New Name, But Under the Covers Still the Same

Web Performance Tests Work at the HTTP Layer

What Does This Mean for You?

Web Tests Can Succeed Even Though It Appears They Failed

A Common Challenge: Dynamic Parameters

Huge Strides Forward with VS 2010

Debugging a Web Test to Find and Fix Dynamic Parameters

Engineering the Solution

About the Web Performance Test Recorder Log

Adding Your Own Recorder Plugins to Make Record/Playback “Just Work”

More Recorder Enhancements in VS 2010

No More Empty Recorder Pane

More Apps “Just Work”

New Tools Options for the Recorder

Web Test Editor Enhancements in VS 2010

Looping and Branching

More Editor Features

New Extensibility Points to Handle Rich Data

Web Test Editor Request Body Plugins

Web Test Result Viewer Plugins

Conclusion

dynaTrace + VS 2010 Load Tests = 1 Powerful Solution for Performance Analysis of Java Applications

Application Benchmark Toolkit for Microsoft Dynamics AX

Product Transitions from VS 2008 to VS 2010 for Load Testing

Parameterizing Tests to Run in Different Environments

Install Visual Studio 2010 RC Today

Configuration Options for Load Testing with Visual Studio 2010

Configuration 1: “Local” Load Generation

Configuration 2: Distributed Test Controller and Test Agents

Configuration 3 A and B: Stacked Configuration

Configuration 4: Stacked Controller, Distributed Agents

Conclusion

VS 2010 Product SKU Lineup Finalized

Client SKUs

Server SKUs

Transitioning from Visual Studio 2008

VS 2010 Compatibility with VS 2008

Conclusion

New Web and Load Test Features in Beta2

Getting Started with Visual Studio 2010 Beta 2

Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010

Still Way Less Expensive than our Competitors

Better Planning and Comparison

Enables More Flexibility with Load Generators

Visual Studio 2010 Agents: More than just Load Generators

What if I already have VS 2008 Load Agents?

Conclusion

Using The Virtual User Activity Chart to Understand the VS Load Engine

Covering the Basics of the Chart

Understanding Test Mixes

Test Mix Based on Number of Tests Started

34/33/33 Test Mix Based on the Number of Virtual Users Starting Tests

Test Mix Based on User Pace

Test Mix Based on Sequential Test Order

The Effect of Think Time Between Iterations and Warmup Duration Properties

The Effect of Percentage of New Users

The Effect of Test Mix Init and Terminate Tests

Understanding Load Patterns with the Virtual Users Activity Chart

Step Load

Goal Based Load

Conclusion