Bill Barnett's blog

VSTS 2010 Feature: More flexible load modeling via new options on Load Test Scenarios

2009-06-11T20:56:00Z

In VSTS 2010 Beta 1 (available for download here), there are four new properties on each Scenario in a Load Test that give you more control of the time, location, and duration of the running of the Scenario during the load test. You can access the Scenario property sheet by right-clicking on the desired Scenario’s node in the Load Test Editor tree:

The new properties are:

“Agents to Use”
”Maximum Test Iterations”
”Delay Start Time”
”Disable During Warmup”

The screen shot below of the Scenario property sheet shows the new Scenario properties along with those that existed in VSTS 2008:

The above screen shot shows the initial default values for all four properties which in all four cases means that the new feature is not used and the load test behaves as it would with VSTS 2008 (all other things being the same).

The screen shots below show non-default values for each of these new properties. You can read the property descriptions in the screen shot, so no need to repeat those, but I’ve added some additional comments on the selected property below each screen shot:

The agent names that are entered should be the names of agents that are connected to the controller to which the load test will be submitted. They should be the simple computer names of the agents (as seen in the “Computer Name” field in the Control Panel). Unfortunately, at this time, if you switch to submitting the load test to a different controller, you will need to change the value for “Agents to Use” as there is no way to parameterize this list to vary depending on the controller used. This list of agents designates a subset of those the agents that are connected to the controller, and are in the Ready state when the load tests starts (they may be running a different load test or other test run when the load test is queued as long as they become Ready when the load test is taken out of the Pending state and starts running), and that meet any agent selection criteria to allow the test run to be run on the agent. The Scenario will run on all agents in the list that meet these criteria, and the user load for the Scenario will be distributed among these agents either evenly (by default) or according to any agent weightings specified in the Agent properties for the agents (from the “Administer Test Controllers” dialog in Visual Studio).

The screen shot below has a non-default value for “Maximum Test Iterations” which is selected to show the property description:

This property is a maximum on the total number of test iterations for tests in the selected Scenario for all users on all agents on which the Scenario is run, and does not mean “Test Iterations per User”. If the Load Test Run Settings dialog specifies a value for Test Iterations that is less than this value, or the timed duration of the load test expires before this maximum is reached the load test will stop. In other words, as the name implies this is a maximum, but not a minimum.

The screen shot below has non-default values for “Delay Start Time” and “Disable During Warmup”. The property description for “Delay Start Time” also describes how the “Disable During Warmup” setting affects when the Scenario actually starts:

The description implies but does not explicitly state this: if “Disable During Warmup” is false, the Delay Start Time is measured from the very beginning of the load test including any warm-up period.

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check out Ed Glas's list.

VSTS 2010 Feature: Sequential Test Mix

2009-06-11T00:20:00Z

VSTS 2010 Load Tests offer several new options that can be set on each Scenario in the load test that give you more control over the execution of the Scenarios within the load test and therefore allows you to more easily create a load test that accurately models the load that your are trying to simulate on your servers under test.

The first of these is a new “Text Mix” type based on sequential test order that simply allows you to set up a sequential ordering of the tests that each virtual user will go through. This option is available in the New Load Test and New Scenario Wizards:

The diagram shows an example with 4 virtual users and two tests in the Scenario, and the text below the diagram explains the behavior.

If you select “Based on sequential test order” and click “Next”, the next Wizard page allows to add to and order the tests:

Note that there is no need to enter percentages or pacing information as for the other test mix types.

Once you’ve created the load test Scenario through the Wizard with this test mix type, you can later use the Load Test editor’s Test Mix editor to change the Test Mix type, or add, remove, or change the order of the tests:

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check Ed Glas's list. You can download the beta here.

VSTS 2010 Load Test Feature: Saving Test Logs

2009-06-09T02:04:00Z

Now that VSTS 2010 beta 1 is available I and other Load Test development and test team members will be writing a series of blog posts highlighting the new Web and Load testing features. Ed Glas has a blog post that lists the new features here. You can download the beta here.

This post describes in detail the item in Ed’s list “Log entire test result on test failure …”:

With VSTS 2010, Load Test users now have the option of capturing and saving the entire result of individual tests run within the load test such as a failed Web test or a failed unit test. This feature makes it easier to debug problems that occur when running tests within a load test that do not occur when running the same tests outside the context of a load test.

The New Load Test Wizard exposes a new option “Save Log on Test Failure” which defaults to True:

Also, in the Load Test editor, the Load Test Run Setting’s property sheet includes this option as well as two other related properties in the new “Logging” category:

You can see the description of the new “Save Log Frequency for Completed Tests” property above. The “Maximum Test Logs” setting controls the maximum number of Test Logs that are saved in the load test results database for a single load test. You can increase this value from the default value of 200, but we aware that the test logs could potentially take up a considerable amount of space in the database especially for Web tests that contain many large requests and/or responses since they include the entire body of each Web test request and response (the exception to this is that details for dependent requests are not saved in order to save space).

For example, to see how this works with unit tests in a load test, suppose I have the two following unit test methods:

[TestMethod]
public void UnitTestThatLogsAndPasses()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(1000);
}

[TestMethod]
public void UnitTestThatLogsAndFails()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(500);
    Assert.Fail("This test fails for demo purposes");
}

Then I include these unit tests in a load test with the passing test running 90% of the time:

The Test Logs are not viewable while the load test is running, but can be viewed once the load test has completed and you go to the post-run Load Test Analyzer by responding Yes to this prompt at the end of the load test:

(You can also bring up the post-run Load Test Analyzer for a load test run from the “Open and Manage Load Test Results” dialog available on the toolbar in the Load Test Editor.)

This should bring you to the Summary page in the Load Test Analyzer; if you then click on the Errors link, the Errors table is displayed and includes links to the Test Logs in the “Details” column (the right-most column – you may need to scroll right to see it):

Clicking on one of the “Test log” links in this table opens a new Window in Visual Studio to view that unit test result which looks just like viewing the result of a unit test run outside of a load test:

In addition to links to the Test Logs in the Errors table, there is also a new “Test Details” table in the Load Test Analyzer that also may contains links to Test Logs:

In the load test result shown above, it was a Web test that failed, so clicking on the “Test Log” link opens a new Web Test result viewer window in Visual Studio; you can see the details of requests that failed, but you can also browse around the Web test result to see the details of previous requests that may have been considered successful but perhaps caused later requests to fail because they were missing expected content (and there was not an appropriate validation rule).

As with the unit test, the Web test result viewer is the same as when the Web test is run outside of a load test:

As a slight aside to this topic, the new Test Details table is available in the post-run load test analyzer as long as you have set the value for the Load Test Run Setting’s property “Timing Details Storage” to “All Individual Details”. This is true even if you disable the capturing of Test Logs by setting “Save Log on Test Failure” to False and leaving ““Save Log Frequency for Completed Tests” at the default value of 0 (which means never capture test logs for successful tests). In this case, the Test Details table would look like this:

When “Timing Details Storage” is set to “All Individual Details” (which is now the default value for this property for load tests created with VSTS 2010), the data shown in the table above is stored in the load test results database for all tests run within the load test (excluding the warm-up and cool-down periods). The Test Details table in the Load Test Analyzer will only display up to 1000 of these rows at a time. When the table is first loaded, it is sorted in chronological order (that is by ascending values of the Start Time column) as shown above. However, you can choose a different sort column and toggle the sort order by clicking on the column headers. This results in a new query to the load test database so that you see the top 1000 rows matching the sort column and order. For example, you can choose to sort by the slowest values in the Test Time column to see the 1000 slowest individual tests within the load test.

How is the user load adjusted for a load test with a goal based load pattern?

2008-09-12T16:11:00Z

This post describes how the user load is adjusted when a load test scenario uses a goal based load pattern.

First of all, if there is a warm-up period specified, the goal based pattern does not apply during the warmup period. Instead, the load test creates a temporary step load pattern that is used during the warm-up period to increase the user load until it reaches the Inital User Count setting in the goal based load pattern's properties. (In VSTS 2008 SP1 the warmup is done at a rate calculated to reach the Initial User Count just as the warm-up period ends; in previous versions, the warmup load pattern increased the user load at a rate of 10 users per second on each agent until the Initial User Count was reached, and this was too quick a warm-up for some tests.)

Here's what happens with a goal based load pattern after the warm-up period completes:

1. The user load starts at the value specified by the Initial User Count property of the Goal Based Load Pattern.

2. At each sampling interval (which defaults to 5 seconds, but can be modified by the “Sample Rate” property in the load test run settings), the performance counter defined in the goal based load pattern is sampled. (If it can’t be sampled for some reason, an error is logged and the user load remains the same.)

3. The value sampled is compared with the “Low End” and “High End” properties of the “Target Range for Performance Counter”.

4. If the value is within the boundaries of the “Low End” and “High End”, the user load remains the same.

5. If the value is not within the boundaries of the “Low End” and “High End”, the user load is adjusted as follows:

o The midpoint of the target range for the goal is divided by the sample valued for the goal performance counter to calculate an “adjustment factor”.

o For example, if the goal is defined as “% Processor Time” between 50 and 70, the midpoint is 60. If the sampled value for % Processor Time is 40, then AdjustmentFactor = 60/40 = 1.5, or if the sampled value is 80, the AdjustmentFactor = 60/80 = 0.75.

o The AdjustmentFactor is multiplied by the current user load to get the new user load.

o However, if the difference between the new user load and the current user load is greater than the “Maximum User Count Increase/Decrease” property (whichever applies), then the user load is only adjusted by as much as max increase/decrease property. My experience has been that keeping these values fairly small is a good idea; otherwise the algorithm tends to cause too much fluctuation (the perf counter keeps going above and below the target range).

o The new user load can also not be larger than the value specified by the goal based pattern’s MaximumUserCount property or less than the Minimum User Count property.

o Two more considerations based on special properties of the goal based load pattern:

§ If the property “Lower Values Imply Higher Resource Use” is True (which you might use for example for a performance count such as Memory\Available Mbytes), then the user load is adjusted in the opposite direction: the user load is decreased when the sampled counter value is less than the Low End of the target range and increased when the user load is greater than the High End of the target range.

§ If the property “Stop Adjusting User Count When Goal Achieved” is True, then once the sampled goal performance counter is within the target range for 3 consecutive sampling intervals, then the user load is no longer adjusted and remains constant for the remainder of the load test.

o Lastly, as is true for all of the user load patterns, in a test rig with multiple agents, the new user load is distributed among the agents equally by default, or according to the “Agent Weightings” if these are specified in the agent properties.

Hot fix available for think time bug introduced in VSTS 2008 SP1

2008-09-11T01:03:00Z

There is now a hot fix available at http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=15510 for the bug introduced in VSTS 2008 SP1 where think times were being ignored during a load test for Web test requests that resulted in redirect responses (described in my earlier post: http://blogs.msdn.com/billbar/archive/2008/08/04/bug-in-vsts-2008-sp1-causes-think-time-for-redirected-requests-to-be-ignored-in-a-load-test.aspx).

Collecting SQL counters from a non-default SQL instance in a VSTS Load Test

2008-08-06T17:24:00Z

If you want to collect performance counters from a SQL Server instance while running a load test, you can do this easily by selecting checking the SQL counter set in the "Manager Counter Sets" dialog in the VSTS load test editor. Doing this includes the default counter set for SQL Server in your load test. The performance counter category names that are specified in this counter set begin with "SQLServer:": for example "SQLServer:Locks". However, if you are trying to monitor a SQL Server instance that is not the default SQL server instance, the names of the performance counter categories for that instance will have different category names. For example, if your SQL server instance is named "INST_A", then this performance counter category will be named "MSSQL$INST_A:Locks". To change the load test to collect these performance counters, the easiest thing to do is open the .loadtest file with the XML editor or a text editor and replace all instances of "SQLServer:" by "MSSQL$INST_A:Locks" (correcting the replacement string for your instance name).

Article posted: Diagnosing OutOfMemoryExceptions that occur when running load tests

2008-08-05T21:14:00Z

I've posted a new article on diagnosing problems running out of memory while running a VSTS load test: http://blogs.msdn.com/billbar/pages/diagnosing-outofmemoryexceptions-that-occur-when-running-load-tests.aspx.

Bug in VSTS 2008 SP1 causes think time for redirected requests to be ignored in a load test

2008-08-04T22:08:00Z

Unfortunately, late last week we discovered a nasty bug with load testing in VSTS 2008 SP1 that is a regression from the original RTM version of VSTS 2008: for Web test requests that result in redirects, any think time that is specifed on the request is ignored when the Web test is run in a load test. Even more unfortunately, VSTS 2008 SP1 has been locked down for changes, so the fix for this problem will not make it in. We will need to create a QFE to be applied to SP1 to fix the problem. It takes some time to get through the QFE process. In the meantime, it is possible to workaround this bug by adding a Web test plugin to each Web test used in a load test that uses redirects and think times (probably most Web tests). The work around is to add the following WebTestPlugin to your test solution, and then add the Web test plug-in to each Web test (fortunately a Web test can have any number of plug-ins):

using Microsoft.VisualStudio.TestTools.WebTesting;

namespace TestProject1    // Change the name space as appropriate
{
    public class WebTestPlugin_FixThinkTime : WebTestPlugin
    {
        private WebTestRequest m_currentPageOriginalRequest;

        public override void PostRequest(object sender, PostRequestEventArgs e)
        {
            base.PostRequest(sender, e);

            if (!e.Request.IsRedirectFollow)
            {
                m_currentPageOriginalRequest = e.Request;
            }
            else
            {
                if (m_currentPageOriginalRequest.ThinkTime == 0 && e.Request.ThinkTime != 0)
                {
                    m_currentPageOriginalRequest.ThinkTime = e.Request.ThinkTime;
                }
            }
        }
    }
}

If you are already using VSTS 2008 SP1 Beta1, can you add this plug-in now to fix the existing bug, and the plug-in will still work when the final version of SP1 ships, and will not cause any harm once the QFE that fixes the bug is available (though it could be removed at that point).

Running Load Tests with VSTS 2008 SP1 requires that SP1 be installed on VS client, controller, and agents

2008-06-13T21:19:00Z

As mentioned in previous blog posts, the beta version of VSTS 2008 SP1 is now available. In order to implementation the Load Test API enhancements (described here: http://blogs.msdn.com/billbar/pages/load-test-api-enhancements-in-vsts-2008-sp1-beta.aspx) we moved some of the classes that are used to implement the different load test patterns into the LoadTestFramework.dll to make them public and also changed the namespace for these classes. One consequence of this change is that it introduced a dependency: if you install VSTS 2008 SP1 on the Visual Studio client, then you must also install it on the controller and all agents that are online and connected to that controller. If you install SP1 on the Visual Studio client and attempt to run a load test against a test rig without SP1 installed, the load test will fail immediately with the status Not Executed in the Test Results window, and if you click on the "Test run error" link in the Test Results window, the error shown will look something like:

Failed to queue test run 'username@MYCONTROLLER 2008-06-13 11:17:28': Object of type 'Microsoft.VisualStudio.TestTools.LoadTesting.LoadTestConstantLoadProfile' cannot be converted to type 'Microsoft.VisualStudio.TestTools.WebStress.WebTestLoadProfile'.

(The first type listed will vary depending on the type of load pattern used by your load test.)

This dependency exists in the beta version of SP1 and will still exist in the final version of VSTS 2008 SP1.

Disabling caching of all dependent requests

2008-06-06T22:56:00Z

In a previous blog post here: http://blogs.msdn.com/billbar/archive/2007/01/22/simulation-of-browser-caching-in-vsts-load-tests-and-web-tests.aspx, I described the behavior of caching with VSTS 2005.

With VSTS 2008, the default caching behavior is the same. However, with VSTS 2008, if you want completely disabling caching of all dependent requests and always fetch them, you can so by adding the following WebTestPlugin to your Web test:

public class WebTestPlugin_DisableDependentCaching : WebTestPlugin

{

public override void PostRequest(object sender, PostRequestEventArgs e)

{

foreach (WebTestRequest dependentRequest in e.Request.DependentRequests)

{

dependentRequest.Cache = false;

}

Engage!

2008-06-05T19:26:00Z

This is my first blog post of a personal nature. I heard on TV this morning that today marks the 40th anniversary of the assassination of Bobby Kennedy (http://www.robertfkennedy.net/).

Today though I'm remembering another great American who died a year ago today: my oldest son Thomas who was 22 years old. You can read his somewhat heartbreaking but also inspirational story here: http://www.curefa.org/families/thomas.asp. If Thomas were alive today, I'd guess he would have voted for Obama and might inwardly be excited about his nomination, but no doubt he'd have something cynical but funny to say about the campaign because he had a great cynical sense of humor about everything except Star Trek and baseball which were sacrosanct (as an avid Braves fan - he'd be much more excited that Chipper Jones is batting over .400 in June). The lesson I've tried to learn from my son is to live life to its fullest everyday, or in the word of his favorite character: "Engage!".

Bill

Articles Posted on Web and Load Test API enhancements available in VSTS 2008 SP1 beta

2008-06-04T23:56:00Z

The beta for Visual Studio 2008 SP1 is now available and includes enhancements to both the Web test and load test APIs that are described in these articles:

http://blogs.msdn.com/billbar/pages/web-test-api-enhancements-available-in-vsts-2008-sp1-beta.aspx

http://blogs.msdn.com/billbar/pages/load-test-api-enhancements-in-vsts-2008-sp1-beta.aspx

Article Posted: Moving VSTS 2005 Load Test results from one computer to another

2008-04-03T19:47:00Z

I've posted an article on "Moving VSTS 2005 Load Test results from one computer to another": http://blogs.msdn.com/billbar/pages/moving-vsts-2005-load-test-results-from-one-computer-to-another.aspx

Article posted: Adding Console Output to Load Tests running Unit Tests

2008-04-03T18:58:00Z

I've posted an article on "Adding Console Output to Load Tests running Unit Tests": http://blogs.msdn.com/billbar/pages/adding-console-output-to-load-tests-running-unit-tests.aspx.

Content Index for Web and Load testing

2007-12-17T22:44:00Z

Ed Glas has posted a "Content Index for Web and Load Tests" on his blog at: http://blogs.msdn.com/edglas/pages/content-index-for-web-tests-and-load-tests.aspx.

This provides links to a variety of online resources (forums, blogs, whitepapers) for VSTS web and load testing.