Bill Barnett's blog : VSTS2010

VSTS 2010 Feature: More flexible load modeling via new options on Load Test Scenarios

billbar@microsoft.com — Thu, 11 Jun 2009 20:56:00 GMT

In VSTS 2010 Beta 1 (available for download here), there are four new properties on each Scenario in a Load Test that give you more control of the time, location, and duration of the running of the Scenario during the load test. You can access the Scenario property sheet by right-clicking on the desired Scenario’s node in the Load Test Editor tree:

The new properties are:

“Agents to Use”
”Maximum Test Iterations”
”Delay Start Time”
”Disable During Warmup”

The screen shot below of the Scenario property sheet shows the new Scenario properties along with those that existed in VSTS 2008:

The above screen shot shows the initial default values for all four properties which in all four cases means that the new feature is not used and the load test behaves as it would with VSTS 2008 (all other things being the same).

The screen shots below show non-default values for each of these new properties. You can read the property descriptions in the screen shot, so no need to repeat those, but I’ve added some additional comments on the selected property below each screen shot:

The agent names that are entered should be the names of agents that are connected to the controller to which the load test will be submitted. They should be the simple computer names of the agents (as seen in the “Computer Name” field in the Control Panel). Unfortunately, at this time, if you switch to submitting the load test to a different controller, you will need to change the value for “Agents to Use” as there is no way to parameterize this list to vary depending on the controller used. This list of agents designates a subset of those the agents that are connected to the controller, and are in the Ready state when the load tests starts (they may be running a different load test or other test run when the load test is queued as long as they become Ready when the load test is taken out of the Pending state and starts running), and that meet any agent selection criteria to allow the test run to be run on the agent. The Scenario will run on all agents in the list that meet these criteria, and the user load for the Scenario will be distributed among these agents either evenly (by default) or according to any agent weightings specified in the Agent properties for the agents (from the “Administer Test Controllers” dialog in Visual Studio).

The screen shot below has a non-default value for “Maximum Test Iterations” which is selected to show the property description:

This property is a maximum on the total number of test iterations for tests in the selected Scenario for all users on all agents on which the Scenario is run, and does not mean “Test Iterations per User”. If the Load Test Run Settings dialog specifies a value for Test Iterations that is less than this value, or the timed duration of the load test expires before this maximum is reached the load test will stop. In other words, as the name implies this is a maximum, but not a minimum.

The screen shot below has non-default values for “Delay Start Time” and “Disable During Warmup”. The property description for “Delay Start Time” also describes how the “Disable During Warmup” setting affects when the Scenario actually starts:

The description implies but does not explicitly state this: if “Disable During Warmup” is false, the Delay Start Time is measured from the very beginning of the load test including any warm-up period.

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check out Ed Glas's list.

VSTS 2010 Feature: Sequential Test Mix

billbar@microsoft.com — Thu, 11 Jun 2009 00:20:00 GMT

VSTS 2010 Load Tests offer several new options that can be set on each Scenario in the load test that give you more control over the execution of the Scenarios within the load test and therefore allows you to more easily create a load test that accurately models the load that your are trying to simulate on your servers under test.

The first of these is a new “Text Mix” type based on sequential test order that simply allows you to set up a sequential ordering of the tests that each virtual user will go through. This option is available in the New Load Test and New Scenario Wizards:

The diagram shows an example with 4 virtual users and two tests in the Scenario, and the text below the diagram explains the behavior.

If you select “Based on sequential test order” and click “Next”, the next Wizard page allows to add to and order the tests:

Note that there is no need to enter percentages or pacing information as for the other test mix types.

Once you’ve created the load test Scenario through the Wizard with this test mix type, you can later use the Load Test editor’s Test Mix editor to change the Test Mix type, or add, remove, or change the order of the tests:

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check Ed Glas's list. You can download the beta here.

VSTS 2010 Load Test Feature: Saving Test Logs

billbar@microsoft.com — Tue, 09 Jun 2009 02:04:00 GMT

Now that VSTS 2010 beta 1 is available I and other Load Test development and test team members will be writing a series of blog posts highlighting the new Web and Load testing features. Ed Glas has a blog post that lists the new features here. You can download the beta here.

This post describes in detail the item in Ed’s list “Log entire test result on test failure …”:

With VSTS 2010, Load Test users now have the option of capturing and saving the entire result of individual tests run within the load test such as a failed Web test or a failed unit test. This feature makes it easier to debug problems that occur when running tests within a load test that do not occur when running the same tests outside the context of a load test.

The New Load Test Wizard exposes a new option “Save Log on Test Failure” which defaults to True:

Also, in the Load Test editor, the Load Test Run Setting’s property sheet includes this option as well as two other related properties in the new “Logging” category:

You can see the description of the new “Save Log Frequency for Completed Tests” property above. The “Maximum Test Logs” setting controls the maximum number of Test Logs that are saved in the load test results database for a single load test. You can increase this value from the default value of 200, but we aware that the test logs could potentially take up a considerable amount of space in the database especially for Web tests that contain many large requests and/or responses since they include the entire body of each Web test request and response (the exception to this is that details for dependent requests are not saved in order to save space).

For example, to see how this works with unit tests in a load test, suppose I have the two following unit test methods:

[TestMethod]
public void UnitTestThatLogsAndPasses()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(1000);
}

[TestMethod]
public void UnitTestThatLogsAndFails()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(500);
    Assert.Fail("This test fails for demo purposes");
}

Then I include these unit tests in a load test with the passing test running 90% of the time:

The Test Logs are not viewable while the load test is running, but can be viewed once the load test has completed and you go to the post-run Load Test Analyzer by responding Yes to this prompt at the end of the load test:

(You can also bring up the post-run Load Test Analyzer for a load test run from the “Open and Manage Load Test Results” dialog available on the toolbar in the Load Test Editor.)

This should bring you to the Summary page in the Load Test Analyzer; if you then click on the Errors link, the Errors table is displayed and includes links to the Test Logs in the “Details” column (the right-most column – you may need to scroll right to see it):

Clicking on one of the “Test log” links in this table opens a new Window in Visual Studio to view that unit test result which looks just like viewing the result of a unit test run outside of a load test:

In addition to links to the Test Logs in the Errors table, there is also a new “Test Details” table in the Load Test Analyzer that also may contains links to Test Logs:

In the load test result shown above, it was a Web test that failed, so clicking on the “Test Log” link opens a new Web Test result viewer window in Visual Studio; you can see the details of requests that failed, but you can also browse around the Web test result to see the details of previous requests that may have been considered successful but perhaps caused later requests to fail because they were missing expected content (and there was not an appropriate validation rule).

As with the unit test, the Web test result viewer is the same as when the Web test is run outside of a load test:

As a slight aside to this topic, the new Test Details table is available in the post-run load test analyzer as long as you have set the value for the Load Test Run Setting’s property “Timing Details Storage” to “All Individual Details”. This is true even if you disable the capturing of Test Logs by setting “Save Log on Test Failure” to False and leaving ““Save Log Frequency for Completed Tests” at the default value of 0 (which means never capture test logs for successful tests). In this case, the Test Details table would look like this:

When “Timing Details Storage” is set to “All Individual Details” (which is now the default value for this property for load tests created with VSTS 2010), the data shown in the table above is stored in the load test results database for all tests run within the load test (excluding the warm-up and cool-down periods). The Test Details table in the Load Test Analyzer will only display up to 1000 of these rows at a time. When the table is first loaded, it is sorted in chronological order (that is by ascending values of the Start Time column) as shown above. However, you can choose a different sort column and toggle the sort order by clicking on the column headers. This results in a new query to the load test database so that you see the top 1000 rows matching the sort column and order. For example, you can choose to sort by the slowest values in the Test Time column to see the 1000 slowest individual tests within the load test.