Bill Barnett's blog : Web test

VSTS 2010 Feature: More flexible load modeling via new options on Load Test Scenarios

billbar@microsoft.com — Thu, 11 Jun 2009 20:56:00 GMT

In VSTS 2010 Beta 1 (available for download here), there are four new properties on each Scenario in a Load Test that give you more control of the time, location, and duration of the running of the Scenario during the load test. You can access the Scenario property sheet by right-clicking on the desired Scenario’s node in the Load Test Editor tree:

The new properties are:

“Agents to Use”
”Maximum Test Iterations”
”Delay Start Time”
”Disable During Warmup”

The screen shot below of the Scenario property sheet shows the new Scenario properties along with those that existed in VSTS 2008:

The above screen shot shows the initial default values for all four properties which in all four cases means that the new feature is not used and the load test behaves as it would with VSTS 2008 (all other things being the same).

The screen shots below show non-default values for each of these new properties. You can read the property descriptions in the screen shot, so no need to repeat those, but I’ve added some additional comments on the selected property below each screen shot:

The agent names that are entered should be the names of agents that are connected to the controller to which the load test will be submitted. They should be the simple computer names of the agents (as seen in the “Computer Name” field in the Control Panel). Unfortunately, at this time, if you switch to submitting the load test to a different controller, you will need to change the value for “Agents to Use” as there is no way to parameterize this list to vary depending on the controller used. This list of agents designates a subset of those the agents that are connected to the controller, and are in the Ready state when the load tests starts (they may be running a different load test or other test run when the load test is queued as long as they become Ready when the load test is taken out of the Pending state and starts running), and that meet any agent selection criteria to allow the test run to be run on the agent. The Scenario will run on all agents in the list that meet these criteria, and the user load for the Scenario will be distributed among these agents either evenly (by default) or according to any agent weightings specified in the Agent properties for the agents (from the “Administer Test Controllers” dialog in Visual Studio).

The screen shot below has a non-default value for “Maximum Test Iterations” which is selected to show the property description:

This property is a maximum on the total number of test iterations for tests in the selected Scenario for all users on all agents on which the Scenario is run, and does not mean “Test Iterations per User”. If the Load Test Run Settings dialog specifies a value for Test Iterations that is less than this value, or the timed duration of the load test expires before this maximum is reached the load test will stop. In other words, as the name implies this is a maximum, but not a minimum.

The screen shot below has non-default values for “Delay Start Time” and “Disable During Warmup”. The property description for “Delay Start Time” also describes how the “Disable During Warmup” setting affects when the Scenario actually starts:

The description implies but does not explicitly state this: if “Disable During Warmup” is false, the Delay Start Time is measured from the very beginning of the load test including any warm-up period.

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check out Ed Glas's list.

VSTS 2010 Feature: Sequential Test Mix

billbar@microsoft.com — Thu, 11 Jun 2009 00:20:00 GMT

VSTS 2010 Load Tests offer several new options that can be set on each Scenario in the load test that give you more control over the execution of the Scenarios within the load test and therefore allows you to more easily create a load test that accurately models the load that your are trying to simulate on your servers under test.

The first of these is a new “Text Mix” type based on sequential test order that simply allows you to set up a sequential ordering of the tests that each virtual user will go through. This option is available in the New Load Test and New Scenario Wizards:

The diagram shows an example with 4 virtual users and two tests in the Scenario, and the text below the diagram explains the behavior.

If you select “Based on sequential test order” and click “Next”, the next Wizard page allows to add to and order the tests:

Note that there is no need to enter percentages or pacing information as for the other test mix types.

Once you’ve created the load test Scenario through the Wizard with this test mix type, you can later use the Load Test editor’s Test Mix editor to change the Test Mix type, or add, remove, or change the order of the tests:

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check Ed Glas's list. You can download the beta here.

VSTS 2010 Load Test Feature: Saving Test Logs

billbar@microsoft.com — Tue, 09 Jun 2009 02:04:00 GMT

Now that VSTS 2010 beta 1 is available I and other Load Test development and test team members will be writing a series of blog posts highlighting the new Web and Load testing features. Ed Glas has a blog post that lists the new features here. You can download the beta here.

This post describes in detail the item in Ed’s list “Log entire test result on test failure …”:

With VSTS 2010, Load Test users now have the option of capturing and saving the entire result of individual tests run within the load test such as a failed Web test or a failed unit test. This feature makes it easier to debug problems that occur when running tests within a load test that do not occur when running the same tests outside the context of a load test.

The New Load Test Wizard exposes a new option “Save Log on Test Failure” which defaults to True:

Also, in the Load Test editor, the Load Test Run Setting’s property sheet includes this option as well as two other related properties in the new “Logging” category:

You can see the description of the new “Save Log Frequency for Completed Tests” property above. The “Maximum Test Logs” setting controls the maximum number of Test Logs that are saved in the load test results database for a single load test. You can increase this value from the default value of 200, but we aware that the test logs could potentially take up a considerable amount of space in the database especially for Web tests that contain many large requests and/or responses since they include the entire body of each Web test request and response (the exception to this is that details for dependent requests are not saved in order to save space).

For example, to see how this works with unit tests in a load test, suppose I have the two following unit test methods:

[TestMethod]
public void UnitTestThatLogsAndPasses()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(1000);
}

[TestMethod]
public void UnitTestThatLogsAndFails()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(500);
    Assert.Fail("This test fails for demo purposes");
}

Then I include these unit tests in a load test with the passing test running 90% of the time:

The Test Logs are not viewable while the load test is running, but can be viewed once the load test has completed and you go to the post-run Load Test Analyzer by responding Yes to this prompt at the end of the load test:

(You can also bring up the post-run Load Test Analyzer for a load test run from the “Open and Manage Load Test Results” dialog available on the toolbar in the Load Test Editor.)

This should bring you to the Summary page in the Load Test Analyzer; if you then click on the Errors link, the Errors table is displayed and includes links to the Test Logs in the “Details” column (the right-most column – you may need to scroll right to see it):

Clicking on one of the “Test log” links in this table opens a new Window in Visual Studio to view that unit test result which looks just like viewing the result of a unit test run outside of a load test:

In addition to links to the Test Logs in the Errors table, there is also a new “Test Details” table in the Load Test Analyzer that also may contains links to Test Logs:

In the load test result shown above, it was a Web test that failed, so clicking on the “Test Log” link opens a new Web Test result viewer window in Visual Studio; you can see the details of requests that failed, but you can also browse around the Web test result to see the details of previous requests that may have been considered successful but perhaps caused later requests to fail because they were missing expected content (and there was not an appropriate validation rule).

As with the unit test, the Web test result viewer is the same as when the Web test is run outside of a load test:

As a slight aside to this topic, the new Test Details table is available in the post-run load test analyzer as long as you have set the value for the Load Test Run Setting’s property “Timing Details Storage” to “All Individual Details”. This is true even if you disable the capturing of Test Logs by setting “Save Log on Test Failure” to False and leaving ““Save Log Frequency for Completed Tests” at the default value of 0 (which means never capture test logs for successful tests). In this case, the Test Details table would look like this:

When “Timing Details Storage” is set to “All Individual Details” (which is now the default value for this property for load tests created with VSTS 2010), the data shown in the table above is stored in the load test results database for all tests run within the load test (excluding the warm-up and cool-down periods). The Test Details table in the Load Test Analyzer will only display up to 1000 of these rows at a time. When the table is first loaded, it is sorted in chronological order (that is by ascending values of the Start Time column) as shown above. However, you can choose a different sort column and toggle the sort order by clicking on the column headers. This results in a new query to the load test database so that you see the top 1000 rows matching the sort column and order. For example, you can choose to sort by the slowest values in the Test Time column to see the 1000 slowest individual tests within the load test.

Hot fix available for think time bug introduced in VSTS 2008 SP1

billbar@microsoft.com — Thu, 11 Sep 2008 01:03:00 GMT

There is now a hot fix available at http://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=15510 for the bug introduced in VSTS 2008 SP1 where think times were being ignored during a load test for Web test requests that resulted in redirect responses (described in my earlier post: http://blogs.msdn.com/billbar/archive/2008/08/04/bug-in-vsts-2008-sp1-causes-think-time-for-redirected-requests-to-be-ignored-in-a-load-test.aspx).

Bug in VSTS 2008 SP1 causes think time for redirected requests to be ignored in a load test

billbar@microsoft.com — Mon, 04 Aug 2008 22:08:00 GMT

Unfortunately, late last week we discovered a nasty bug with load testing in VSTS 2008 SP1 that is a regression from the original RTM version of VSTS 2008: for Web test requests that result in redirects, any think time that is specifed on the request is ignored when the Web test is run in a load test. Even more unfortunately, VSTS 2008 SP1 has been locked down for changes, so the fix for this problem will not make it in. We will need to create a QFE to be applied to SP1 to fix the problem. It takes some time to get through the QFE process. In the meantime, it is possible to workaround this bug by adding a Web test plugin to each Web test used in a load test that uses redirects and think times (probably most Web tests). The work around is to add the following WebTestPlugin to your test solution, and then add the Web test plug-in to each Web test (fortunately a Web test can have any number of plug-ins):

using Microsoft.VisualStudio.TestTools.WebTesting;

namespace TestProject1    // Change the name space as appropriate
{
    public class WebTestPlugin_FixThinkTime : WebTestPlugin
    {
        private WebTestRequest m_currentPageOriginalRequest;

        public override void PostRequest(object sender, PostRequestEventArgs e)
        {
            base.PostRequest(sender, e);

            if (!e.Request.IsRedirectFollow)
            {
                m_currentPageOriginalRequest = e.Request;
            }
            else
            {
                if (m_currentPageOriginalRequest.ThinkTime == 0 && e.Request.ThinkTime != 0)
                {
                    m_currentPageOriginalRequest.ThinkTime = e.Request.ThinkTime;
                }
            }
        }
    }
}

If you are already using VSTS 2008 SP1 Beta1, can you add this plug-in now to fix the existing bug, and the plug-in will still work when the final version of SP1 ships, and will not cause any harm once the QFE that fixes the bug is available (though it could be removed at that point).

Disabling caching of all dependent requests

billbar@microsoft.com — Fri, 06 Jun 2008 22:56:00 GMT

In a previous blog post here: http://blogs.msdn.com/billbar/archive/2007/01/22/simulation-of-browser-caching-in-vsts-load-tests-and-web-tests.aspx, I described the behavior of caching with VSTS 2005.

With VSTS 2008, the default caching behavior is the same. However, with VSTS 2008, if you want completely disabling caching of all dependent requests and always fetch them, you can so by adding the following WebTestPlugin to your Web test:

public class WebTestPlugin_DisableDependentCaching : WebTestPlugin

{

public override void PostRequest(object sender, PostRequestEventArgs e)

{

foreach (WebTestRequest dependentRequest in e.Request.DependentRequests)

{

dependentRequest.Cache = false;

}

Content Index for Web and Load testing

billbar@microsoft.com — Mon, 17 Dec 2007 22:44:00 GMT

Ed Glas has posted a "Content Index for Web and Load Tests" on his blog at: http://blogs.msdn.com/edglas/pages/content-index-for-web-tests-and-load-tests.aspx.

This provides links to a variety of online resources (forums, blogs, whitepapers) for VSTS web and load testing.

Mike Taute's blog post on Diagnosing and fixing Web Test recorder bar issues

billbar@microsoft.com — Mon, 12 Nov 2007 18:31:00 GMT

For anyone have trouble getting the VSTS Web test recorder to start recording properly, see Mike Taute's blog post on diagnosing and fixing these problems at: http://blogs.msdn.com/mtaute/archive/2007/11/09/diagnosing-and-fixing-web-test-recorder-bar-issues.aspx.

Simulation of browser caching in VSTS load tests and Web tests

billbar@microsoft.com — Mon, 22 Jan 2007 23:07:00 GMT

In a VSTS load test that contains Web tests, the load test attempts to simulate the caching behavior of the browser. Here are some notes on how that is done:

· There is a property named on each request in a Web test named “Cache Control” in the Web test editor (and named “Cache” on the WebTestRequest object in the API used by coded Web tests).

· When the Cache Control property on a request in the Web test is false, the request is always issued.

· When the Cache Control property is true, the VSTS load test runtime code attempts to emulate the Internet Explorer caching behavior (with the “Automatically” setting).

(This includes reading and following the HTTP cache control directives.)

· The Cache Control property is automatically set to true for all dependent requests (typically for images, style sheets, etc embedded on the page).

· In a load test, the browser caching behavior is simulated separately for each user running in the load test.

· When a virtual user in a load test completes a Web test and a new Web test session is started to keep the user load at the same level, sometimes the load test starts simulates a “new user” with a clean cache, and sometimes the load test simulates a return user that has items cached from a previous session. This is determined by the “Percentage of New Users” property on the Scenario in the load test. The default for “Percentage of New Users” is 100 in which case all user sessions are started with a clean cache. This is probably not correct for most applications where there are return users, so users should consider the most appropriate value to use for this setting depending on the actual usage of the application being load tested.

Important Note: When running a Web test by itself (outside of the load test), the Cache Control property is automatically set to false for all dependent requests so they are always fetched; this is so that they can be displayed in the browser pane of the Web test results viewer without broken images.

Update for VSTS 2008: The Web test API enhancement in VSTS 2008 now allow you to write a WebTestPlugin that disables caching of all dependent requests. See this blog post: http://blogs.msdn.com/billbar/archive/2008/06/06/disabling-caching-of-all-dependent-requests.aspx.