Search results matching tag 'tsbt-tst'

New Web and Load Test Features in Beta2

edglas — Wed, 21 Oct 2009 07:00:00 GMT

In beta2 we added a number of new minor features, and a couple of major ones.

We also changed our licensing model, which you can read about here.

Here is a list of the minor features we added, along with a brief description of each. The last one in the list, compare two runs in Excel, is probably my most favorite feature in the entire release.

Area

Title

Description

Recorder

Recorder options in tools options

Starting URL supplies a default url to start recording from.

Two new options to control whether or not the recorder does hidden field matching, and whether or not it runs the dynamic parameter detection tool after recording completes.

Recorder

Recorder records all requests by default, fold dependents under page

In 2008, static dependents not found by the dependents parser were not recorded by default. Now they are.

Web test

search/replace

Search and replace in web test requests!

Playback

Add request bytes column

Request byte count is now shown in request list pane in playback (in addition to response byte count)

Playback

PreRequestDataBinding event

This new event fires before data binding occurs, allowing a plugin to dynamically change data bindings in a request.

Data binding

64 bit csv

This fixes a limitation we had in beta1, and now enables csv data sources on 64 bit machines.

Data binding

New APIs on WebTest to control data cursor position and reload data

Give full control over the cursor, enabling tight control of which user is using what data. Also reload enables refreshing a datasource during the test.

Load Test

Scrub counter sets

We refreshed the set of counters being collected, thresholds, and default counters.

Analyzer

Add standard deviation and 99^th percentile stats to load test analyzer

Analyzer

Graph instance 0 for CPU on local runs by default

Fixes the problem documented in this post.

Analyzer

Specify range for a counter (CPU range is 100)

Now you can specify a range for a given counter. Also percentage counters get a range of 100 by default.

Analyzer

Fixed range on perf graph

The performance graph now gets a default range set for it (range is rounded up response time of the slowest measurement)

Analyzer

Computer tags

Computer tags enable you to classify computers by function (e.g. web server, db server, etc.). Also enables run to run comparison even if the runs are on different physical machines.

Reporting

Compare two runs in Excel

This is a new, rich reporting option that enables you to compare performance and resource utilization data in two different runs.

Ed.

Getting Started with Visual Studio 2010 Beta 2

edglas — Tue, 20 Oct 2009 07:00:00 GMT

As you are probably already aware, Visual Studio Beta 2 is now available for download, see the landing page here.

For load testing, you will need to download and install:

Visual Studio Ultimate Beta 2 (x86)

Visual Studio 2010 Team Agents Beta 2 (x86)

Ultimate is the new equivalent of Team Suite, and contains Web performance tests and load tests.

Team Agents contains the controller and agents. Note that the x86 agent contains both 32 and 64 bit assemblies, so if you are looking for the 64 bit version of the load agent, this is the download you need.

As I posted here, we have moved to a virtual user license model. Ultimate is limited to 250 virtual users and one core of the CPU. To generate more than 250 users and use all cores on your machine, you will need to install a virtual user pack.

The most straight forward way to do this is to install a controller and agent (or multiple agents) on the same or separate machines.

We have re-architected the controller and agent setups such that setup does little more than lay down the assemblies, while a post-setup configuration tool does the heavy lifting to get the machine configured. The configuration steps are the ones that typically fail, so rather than having to reinstall the entire product and go back through all the configuration steps, if one part fails you only need to reconfigure that part. We have also enabled command line versions of both setup and the configuration tools.

Here is the controller config tool with options called out:

So you will need to fill in the controller service user using a user with counter collection access to your systems under test (same as in VS 2008), and then configure the load test database in the “Configure for load testing” section.

To add virtual users, start a Visual Studio Command prompt as Administrator (on Vista or Win7, just type Visual Studio Command Prompt in the search window, on XP you’ll find this under the Visual Studio group in the start menu). Then run this command line:

TestControllerConfig licenses /addkey:V10000

This will license the controller for 10,000 virtual users. Note that this license is temporary, and will only work in beta2. For RTM, we will only offer the 1,000 user license pack. If you have a Volume License, you will be able to get your license key from the volume licensing site, and then enter a count of licenses you want to install.

Once you have a virtual user license installed, that license can also be used for local runs (that is, the runs do not have to go through the controller).

If you just want to do local runs, and not use the controller and agent, you can run this from the command line to install the virtual user license:

VSTestConfig licenses /addkey:V10000

This enables you to exceed 250 users for local runs, and unlocks all the cores on your CPU.

Ed.

Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010

edglas — Tue, 20 Oct 2009 07:00:00 GMT

In Visual Studio 2010, we are changing our load test licensing model. In VS 2005 and VS 2008, we shipped the load agent, which was licensed by CPU on the machine generating load. In Visual Studio 2010 we are changing our license model to a virtual user based model. This new license model offers a number of advantages for you, which is ultimately why we chose to make the change:

Still way less expensive than our competition.
Enables better purchasing planning and allows an apples to apples price comparison with competitors.
Enables more flexible machine allocation for load generation.
Enables you to use agents for more than just load testing.

Still Way Less Expensive than our Competitors

Your initial reaction may be “Groan – that means that Microsoft is going to be just as expensive as those other tools!” But that is not the case at all, we were careful to maintain the per-virtual-user price we set out initially with VS 2005. With VS 2005 and VS 2008 load agents, the price was around $5,000 per load agent CPU, and our general guidance was that for a typical load test expect around 1,000 virtual users per CPU. We are not yet disclosing our pricing yet, but I can tell you our goal is to stay in this same ball park with VS 2010.

Better Planning and Comparison

With the VS 2008 load agent, we often get questions from customers doing purchasing planning asking “How many load agents do we need to purchase? We are planning to test up to 5,000 virtual users.” To which our answer with VS 2008 was “we do not know, it depends on your application and test scripts.” Of course this it difficult to know how many agents to purchase until actually doing testing, which was often too late. With virtual user licenses, purchasers will know exactly how many licenses to buy.

The old CPU-based license model also makes it very difficult to compare prices with competitive solutions. Most competitive products are licensed by virtual user and data collectors. If you know how many virtual users you want to test to, but do not know how many agents you will need to generate that number of users, it is impossible to make a comparison. With the Visual Studio Load Test Virtual User Packs 2010, if you know the virtual user count you want to test to, you will know exactly how many licenses to purchase, and be able to compare directly with competitive solutions you may consider.

Enables More Flexibility with Load Generators

The virtual user license offers flexibility with the machines you use as load generators.

With VS 2008, to get the full value of the per-CPU license, you had to run on fast CPUs, often requiring new hardware purchases. Also the license was locked to a particular machine, restricting which machines you could use as load generators. With Visual Studio 2010, you can now run virtual users on older hardware with no “penalty”, as you can line up as many older machines as you wish to generate the target number of virtual users. Also, the virtual user license is now tied to the controller, not the agent machine, so you are allowed to move agent machines in and out of the agent pool.

For some tests, you may run out of some other resources before running out of CPU. The most common case for this in VS 2008 was load agents running out of memory before CPU (especially for web tests with very large post bodies or very large responses), which prevented customers from realizing the full value of their license. With the virtual user license, if this happens you can simply add more machines to the agent pool to achieve the load you are targeting.

Visual Studio 2010 Agents: More than just Load Generators

With the high cost of the agent license, we did not have a viable solution for distributed functional testing. In Visual Studio 2010, agents are now included at no extra cost with VS Premium, Ultimate, and Test Elements. We’ve added Data Collection facility to agents, and added new UI automation test type, which opens up a bunch of new scenarios for using agents.

What if I already have VS 2008 Load Agents?

If you have purchased VS 2008 load agents with software assurance, you will be able to migrate your existing licenses to a commiserate set of Virtual User License Packs. We will iron out the details and mechanics of this once we get closer to RTM.

Conclusion

You can see there are many advantages to you with the new licensing model.

Ed.

Using The Virtual User Activity Chart to Understand the VS Load Engine

edglas — Mon, 19 Oct 2009 07:00:00 GMT

In dev10 we’ve added a new Virtual User Activity Chart that shows virtual user activity. Each row on the chart is a virtual user.

Covering the Basics of the Chart

See Sean Lumley’s post VSTS 2010 Feature- Load test virtual user activity visualization to see an over view of the Virtual User Activity Chart.

Below is the test view of the chart, which shows which shows which test a user is running. This load test has 25 user constant load, with three unit tests in it: Fast (takes 1 second to execute), Medium (takes 5 seconds to execute), and Slow (takes 10 seconds to execute), using the Sequential Test Mix (which I will go into more detail on later in the post).

Zooming in and using Paint to group 5 users at a time for easy counting, the chart shows 25 rows, one for each user.

Zooming in using the zoom bar on the chart, you can see user 0 first ran the fast test, then the medium test, then the slow test:

Which is confirmed by hovering over the purple Medium test bar:

This view shows you exactly what each virtual user was executing during the test, so it will enable you to see patterns of user activity, load patterns, and correlate failed or slow tests and requests with other virtual user activity.

Another use of this view is it really helps you understand what the load engine does.

In this post, I’ll review various properties in the load engine and show the details view so you can see the effect these properties have.

Understanding Test Mixes

First, I’ll show different test mixes and their corresponding details view, using tests that will accentuate the differences in the mixes.

I’ll use three tests, long, medium and short. Long runs for 10 seconds, medium for 5 seconds, and short for 1 second.

Test Mix Based on Number of Tests Started

Here is the Activity Chart for this test mix, with 34% Fast, 33% Medium, 33% Slow. With this mix, as each user finishes a test, the user randomly selects the next test based on the test weightings.

One think you’ll notice is the predominance of Yellow, and lack of Blue. You wouldn’t think it to look at the user details view, but at the end of this test, the count for each test type is roughly the same. Since the Slow test runs 10x slower than the fast test, there is 10x more yellow than blue.

Name

Scenario

Total Tests

Avg. Test Time (sec)

Slow

Scenario1

912

10.0

Medium

Scenario1

934

5.00

Fast

Scenario1

1,012

1.00

34/33/33 Test Mix Based on the Number of Virtual Users Starting Tests

What this mix is doing is trying to keep, and any given time, the mix of virtual users running the tests to hit the percentage specified. Given the fast test starts and ends so quickly, you would expect a lot of more iterations of the fast test which is exactly what we see, a lot more blue:

Name

Scenario

Total Tests

Avg. Test Time (sec)

Slow

Scenario1

472

10.0

Medium

Scenario1

952

5.00

Fast

Scenario1

5,329

1.00

Test Mix Based on User Pace

In this test, I’ll run the same three tests, and specify they each run 120 times per hour. I chose 120 since it will take 16 seconds to run all three tests, so if I run each twice a minute, I should get a nice graph with about half the time spent running tests. Of course, this mix is not intended to be used this way, it should be used to model your user behavior (how many times an hour do you expect a user to do a particular action).

Now we see a very different graph, with a lot of pacing time between iterations:

Name	Scenario	Total Tests	Avg. Test Time (sec)
Slow	Scenario1	174	10.0
Medium	Scenario1	177	5.00
Fast	Scenario1	183	1.00

Test Mix Based on Sequential Test Order

With this mix, each user simply runs the tests in the order you specify in the mix, which is easy to conceptualize and super-easy to see:

The Effect of Think Time Between Iterations and Warmup Duration Properties

As you can see from this last detail view, this test has all users going lock step through the application, all doing exactly the same thing at the same time. While this may be useful for doing stress testing, it is not realistic user behavior, which is why we have the other test mixes which do a better job of varying what users are doing.

Another way to vary what users are doing and introduce pacing into a scenario is by using Warmup Duration and Think Time Between Iterations. Think Time Between Iterations is especially important if your test only has a single web test in it (thus no think time), or you are running a unit test with no pacing (for unit tests this is implemented via sleep calls, but never use sleep calls in a web test).

Here’s what happens when I put a 30 second warmup in the last test:

Now you can see the virtual users stagger their start over 30 seconds.

And here is the same test (no warmup) with a 5 second think time between iterations:

The variations are due to the Normal Distribution think time profile that is applied by default. This profile will vary the specified think time using a normal distribution (bell curve).

The Effect of Percentage of New Users

Now I am going to clear the delay between iterations, and use 100% new users. What I’d expect here is that each user will only run one test, leading to a lot of rows in the view, which is exactly what happens:

Scrolling up vertically one screen shows this:

So you can see the load engine repeatedly starting new users. As one group of 10 users completes, the next group of 10 starts. New users have a clean cache and cookie container, which matter for web tests.

Different mixes interact differently with the percentage of new users:

Sequential Mix as you can plainly see above, we made the design decision on sequential mix to have each new user run each test once.
With Test Mix Based on Sequential Test Order and Test Mix Based on Number of Tests Started, each user runs one test.
Test Mix Based on User Pace ignores the percentage of new users altogether, as the mix relies on a repeat user to establish the proper pacing.

The Effect of Test Mix Init and Terminate Tests

Init and terminate tests one once per each user. Switching my test back to 0 percent new users, and adding an init and terminate test shows the following:

Where the legend shows

This clearly shows the init test running once per virtual user, as the very first test executed, exactly as it should.

But wait! Where is the terminate test? It is not in the legend, and does not show up scrolling the view all the way to the end of the test. This shows the terminate test only gets called when a cooldown period is configured in the load test. Re-running with a 30 second cool down demonstrates this:

Understanding Load Patterns with the Virtual Users Activity Chart

Each of the above tests uses a constant user load of 25 users. Now I’ll use the chart to show how Step Load and Goal-based load based loads add users. For these tests, I’m going to use just one test in the mix, the 1 second fast test, to draw attention to the load pattern rather than the test mix.

Step Load

Here’s the Activity Chart for a step load pattern adding 10 users every 10 seconds until 100 users is reached. Notice the vertical scroll bar to scroll up to see the remaining users. Zooming in on the first 30 seconds shows the first 30 users created.

The step load Step Ramp Time property enables you to ramp the users added at each step. I highly recommend using this for every step load pattern, particularly if you are using large steps. Here I set a step ramp of 5 seconds, which shows users ramped up for each step over 5 seconds.

Goal Based Load

For a goal based load pattern, it is important that the counter you are seeking the goal on has a direct correlation with the load at each interval. So for this demonstration I put a tight loop in my test to provide a direct correlation between CPU on the load agent and the virtual user count. For this test, I also did a step load before hand to see what a good CPU range would be.

Here’s user load:

Let’s look at the Activity Chart for the areas where the dips are:

Here you can see virtual users stopping as they finish an iteration in order to achieve a lower load.

Scrolling up, you can see that new users are added to the test to move the load back up.

One thing that surprised me here is that users are not recycled. That looks like a bug to me!

Conclusion

Hopefully this post unravels some of the properties of the load engine, and it shows you just one of the ways you can leverage the new Virtual User Activity Chart in dev10 to understand exactly what your load test is doing!

Increasing the ROI of our Automation

edglas — Sat, 13 Jun 2009 07:00:00 GMT

A while back I posted this on automation. I want to resurface this topic now, as it has been on my mind a lot lately as we develop dev10. What is the “right” approach for automation? How do we make it pay off?

Automation is hugely important for us. It enables us to ensure our software works on a myriad of OS (XP, Vista, Win7, Server 2003, Server 2008), processor architecture (x86, ia64), and language (English, German, Japanese, Chinese, etc.) combinations.

At the end of the day, return on investment is what it is all about. Is it faster to develop and maintain an automated test case, or simply run the tests manually? If over the lifetime of a test case you spend more time fixing it than it would have taken to run the test manually for each time you ran the automated test, that test has not paid off. This is basic and obvious, but often times get overlooked when considering automation.

Going back a bit in history to the VS 2005 release when we initially introduced load testing, we made a few huge mistakes with our approach to automation, mistakes which I see others repeating over and over:

My test team wrote automation exclusively using UI automation.
My test team owned developing and maintaining the automation.
Dev “unit tests” were not considered “automation”.
Devs and testers used different test automation frameworks.

With dev10, we have reversed each of these, and are benefitting greatly from it.

One thing I really want to emphasize is the cost of maintaining our UI automation. I have not looked at the exact number, but in the course of developing dev10 nearly every single UI automation test case has broken. Contrast that with our tests that test at the API layer, I estimate that fewer than 10% have broken. To add insult to injury, the UI tests are hugely expensive to fix. API tests that break nearly always give compile errors, which are easy to fix. UI tests do not fail until you run them, and then you have to grovel through the test log to figure out what UI controls have changed to cause the failure, make a fix, rerun the test (perhaps to find another failure), all of which is extremely time consuming.

Increasing our ROI: A Cost-Based Approach to Automation

For dev10, we have developed four methodologies for automation, and consider each when considering how to automate testing of particular functionality. We give preference to the lowest cost approach. That is, if we can use a cheaper approach to automate, we do it that way and we don’t automate everything through the UI. In fact, the UI is the least favored approach, and is only done if none of the other approaches will give us the coverage we need. The approaches, in order of preference, are:

Unit tests that run in the test process. Really “unit test” is a misnomer for most of these tests, as they typically test a broader area than a pure unit test. I prefer to call these tests API tests. The defining characteristics of these tests is that they are hosted by the test process.

An example of this for our load test product is we have tests that run requests through our web test engine.

Another example is in our TCM server product. Nearly all of the automation for TCM server has been developed using this approach.
Command line tests. With this approach we have “unit tests” that spawn our command line. The test code then waits for the command to complete, and then uses APIs to validate the command did the right thing. We use this approach for both web and load test features and for our TCM server.

For example, for load tests we have tests that run a load test from the command line, then access the load test database to ensure the results were as expected. This gives us substantial coverage of our web and load test feature set.
API tests that run in the process under test. mstest has a feature that enables processes other than vstesthost.exe to host the test code. An example of this is ASP.NET unit tests, which run in the ASP.NET process. We take advantage of this extensibility point to add our client processes as host processes.

An example of this is we run our new TCM test client Camano (aka Microsoft Test and Lab Manager) as a test host. This is a bit of a mind bender, but we also actually run VS as a test host! So devenv starts vstesthost which starts a second copy of devenv. Once VS is loaded, our test code runs and has access to our architectural layer “just beneath the glass” and enables us to run web and load tests in the IDE (e.g. open a project, open and edit load tests in the project, run them, validate the UI through code).
UI Automation. With this approach, our test code instantiates the client under test, and drives it through the mouse and keyboard (we drive VS through MSAA, and we drive our TCM client using UIA).

The cost of UI automation is at least an order of magnitude greater than the other three approaches. Isn’t it ironic that we are introducing a new UI automation test type in dev10. If UI tests are so bad, why are we bringing them to market? While I maintain that they are very expensive, UI tests are also absolutely necessary. There are some things we simply cannot test without them (like dialogs, or some integrated, cross-process end-to-end scenarios). But if I go back and look at the UI tests we developed for web and load tests in VS 2005, the vast majority of them could have been implemented using a cheaper alternative.

Engineering for Automation: Developing Testable Software

Developing these other automation approaches did not come for free. We have deliberately architected our software to enable each of these approaches, and spent time developing test frameworks and reference patterns for each approach. It has really paid off in a big way. If you look at our TCM tests, we maintain an extremely high pass rate (98% plus). Our “legacy” UI tests from the 2005 release are in bad shape.

Whose job is it anyway?

At the end of the day, it is the test team’s job to help us understand whether or not we are ready to ship. Automation plays a huge role in that, since we are able to run our automation suites unattended and generate test results relatively quickly and concurrently on many different configurations. The test team is responsible for ensuring we have the right set of automation in place help us understand where we are at.

Getting Testers and Developers on the Same Page

However, it is not solely the test team’s job to develop and maintain the automation. The test team develops the test plan, which includes the scenarios and functionality we want to automate, but both the dev and test team develop and maintain the automation. If a developer is writing “unit tests” for a new API they are developing, they work off the test plan to develop one of the specified test cases. Tester do not duplicate this effort by developing “their own” automation to cover. A key tenet is that testers and developers use the same frameworks and methodologies for developing automation. Testers are able to run tests developed by devs, and visa-versa. Going back to the mistakes we made in VS 2005, this simple methodology, along with considering the most cost-effective way to automate a given test case, is the key to fixing each of these mistakes.

Similarly, it is the entire team’s job to maintain the automated tests. As I said earlier, maintenance is the hidden and huge cost of doing automation. Devs must fix any compile errors in tests introduced by their changes, and also must ensure that all P1 tests pass before checking in. Testers and devs share the load on ensuring all P2 and P3 tests maintain a high pass rate. We run these once a week and distribute the load on driving them to passing.

I hope you can also learn from our mistakes, and use our experience to improve your approach to automation.

Ed.

VSTS 2010 Feature: More flexible load modeling via new options on Load Test Scenarios

billbar@microsoft.com — Thu, 11 Jun 2009 07:00:00 GMT

In VSTS 2010 Beta 1 (available for download here), there are four new properties on each Scenario in a Load Test that give you more control of the time, location, and duration of the running of the Scenario during the load test. You can access the Scenario property sheet by right-clicking on the desired Scenario’s node in the Load Test Editor tree:

The new properties are:

“Agents to Use”
”Maximum Test Iterations”
”Delay Start Time”
”Disable During Warmup”

The screen shot below of the Scenario property sheet shows the new Scenario properties along with those that existed in VSTS 2008:

The above screen shot shows the initial default values for all four properties which in all four cases means that the new feature is not used and the load test behaves as it would with VSTS 2008 (all other things being the same).

The screen shots below show non-default values for each of these new properties. You can read the property descriptions in the screen shot, so no need to repeat those, but I’ve added some additional comments on the selected property below each screen shot:

The agent names that are entered should be the names of agents that are connected to the controller to which the load test will be submitted. They should be the simple computer names of the agents (as seen in the “Computer Name” field in the Control Panel). Unfortunately, at this time, if you switch to submitting the load test to a different controller, you will need to change the value for “Agents to Use” as there is no way to parameterize this list to vary depending on the controller used. This list of agents designates a subset of those the agents that are connected to the controller, and are in the Ready state when the load tests starts (they may be running a different load test or other test run when the load test is queued as long as they become Ready when the load test is taken out of the Pending state and starts running), and that meet any agent selection criteria to allow the test run to be run on the agent. The Scenario will run on all agents in the list that meet these criteria, and the user load for the Scenario will be distributed among these agents either evenly (by default) or according to any agent weightings specified in the Agent properties for the agents (from the “Administer Test Controllers” dialog in Visual Studio).

The screen shot below has a non-default value for “Maximum Test Iterations” which is selected to show the property description:

This property is a maximum on the total number of test iterations for tests in the selected Scenario for all users on all agents on which the Scenario is run, and does not mean “Test Iterations per User”. If the Load Test Run Settings dialog specifies a value for Test Iterations that is less than this value, or the timed duration of the load test expires before this maximum is reached the load test will stop. In other words, as the name implies this is a maximum, but not a minimum.

The screen shot below has non-default values for “Delay Start Time” and “Disable During Warmup”. The property description for “Delay Start Time” also describes how the “Disable During Warmup” setting affects when the Scenario actually starts:

The description implies but does not explicitly state this: if “Disable During Warmup” is false, the Delay Start Time is measured from the very beginning of the load test including any warm-up period.

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check out Ed Glas's list.

VSTS 2010 Feature: Sequential Test Mix

billbar@microsoft.com — Wed, 10 Jun 2009 07:00:00 GMT

VSTS 2010 Load Tests offer several new options that can be set on each Scenario in the load test that give you more control over the execution of the Scenarios within the load test and therefore allows you to more easily create a load test that accurately models the load that your are trying to simulate on your servers under test.

The first of these is a new “Text Mix” type based on sequential test order that simply allows you to set up a sequential ordering of the tests that each virtual user will go through. This option is available in the New Load Test and New Scenario Wizards:

The diagram shows an example with 4 virtual users and two tests in the Scenario, and the text below the diagram explains the behavior.

If you select “Based on sequential test order” and click “Next”, the next Wizard page allows to add to and order the tests:

Note that there is no need to enter percentages or pacing information as for the other test mix types.

Once you’ve created the load test Scenario through the Wizard with this test mix type, you can later use the Load Test editor’s Test Mix editor to change the Test Mix type, or add, remove, or change the order of the tests:

For a complete list of the new Web testing and Load Testing features in VSTS 2010 Beta 1 check Ed Glas's list. You can download the beta here.

VSTS 2010 Load Test Feature: Saving Test Logs

billbar@microsoft.com — Tue, 09 Jun 2009 07:00:00 GMT

Now that VSTS 2010 beta 1 is available I and other Load Test development and test team members will be writing a series of blog posts highlighting the new Web and Load testing features. Ed Glas has a blog post that lists the new features here. You can download the beta here.

This post describes in detail the item in Ed’s list “Log entire test result on test failure …”:

With VSTS 2010, Load Test users now have the option of capturing and saving the entire result of individual tests run within the load test such as a failed Web test or a failed unit test. This feature makes it easier to debug problems that occur when running tests within a load test that do not occur when running the same tests outside the context of a load test.

The New Load Test Wizard exposes a new option “Save Log on Test Failure” which defaults to True:

Also, in the Load Test editor, the Load Test Run Setting’s property sheet includes this option as well as two other related properties in the new “Logging” category:

You can see the description of the new “Save Log Frequency for Completed Tests” property above. The “Maximum Test Logs” setting controls the maximum number of Test Logs that are saved in the load test results database for a single load test. You can increase this value from the default value of 200, but we aware that the test logs could potentially take up a considerable amount of space in the database especially for Web tests that contain many large requests and/or responses since they include the entire body of each Web test request and response (the exception to this is that details for dependent requests are not saved in order to save space).

For example, to see how this works with unit tests in a load test, suppose I have the two following unit test methods:

[TestMethod]
public void UnitTestThatLogsAndPasses()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(1000);
}

[TestMethod]
public void UnitTestThatLogsAndFails()
{
    Debug.WriteLine("This line written using Debug.WriteLine()");
    Trace.WriteLine("This line written using Trace.WriteLine()");
    Console.WriteLine("This line written using Console.WriteLine()");
    System.Threading.Thread.Sleep(500);
    Assert.Fail("This test fails for demo purposes");
}

Then I include these unit tests in a load test with the passing test running 90% of the time:

The Test Logs are not viewable while the load test is running, but can be viewed once the load test has completed and you go to the post-run Load Test Analyzer by responding Yes to this prompt at the end of the load test:

(You can also bring up the post-run Load Test Analyzer for a load test run from the “Open and Manage Load Test Results” dialog available on the toolbar in the Load Test Editor.)

This should bring you to the Summary page in the Load Test Analyzer; if you then click on the Errors link, the Errors table is displayed and includes links to the Test Logs in the “Details” column (the right-most column – you may need to scroll right to see it):

Clicking on one of the “Test log” links in this table opens a new Window in Visual Studio to view that unit test result which looks just like viewing the result of a unit test run outside of a load test:

In addition to links to the Test Logs in the Errors table, there is also a new “Test Details” table in the Load Test Analyzer that also may contains links to Test Logs:

In the load test result shown above, it was a Web test that failed, so clicking on the “Test Log” link opens a new Web Test result viewer window in Visual Studio; you can see the details of requests that failed, but you can also browse around the Web test result to see the details of previous requests that may have been considered successful but perhaps caused later requests to fail because they were missing expected content (and there was not an appropriate validation rule).

As with the unit test, the Web test result viewer is the same as when the Web test is run outside of a load test:

As a slight aside to this topic, the new Test Details table is available in the post-run load test analyzer as long as you have set the value for the Load Test Run Setting’s property “Timing Details Storage” to “All Individual Details”. This is true even if you disable the capturing of Test Logs by setting “Save Log on Test Failure” to False and leaving ““Save Log Frequency for Completed Tests” at the default value of 0 (which means never capture test logs for successful tests). In this case, the Test Details table would look like this:

When “Timing Details Storage” is set to “All Individual Details” (which is now the default value for this property for load tests created with VSTS 2010), the data shown in the table above is stored in the load test results database for all tests run within the load test (excluding the warm-up and cool-down periods). The Test Details table in the Load Test Analyzer will only display up to 1000 of these rows at a time. When the table is first loaded, it is sorted in chronological order (that is by ascending values of the Start Time column) as shown above. However, you can choose a different sort column and toggle the sort order by clicking on the column headers. This results in a new query to the load test database so that you see the top 1000 rows matching the sort column and order. For example, you can choose to sort by the slowest values in the Test Time column to see the 1000 slowest individual tests within the load test.

dynaTrace Leverages new Extensibility Points to Provide Deep Integration with Dev10

edglas — Wed, 20 May 2009 07:00:00 GMT

A fantastic example of a partner taking advantage of the new extensibility points in Web test playback and load tests in dev10 is provided by dynaTrace. dynaTrace has developed an extension that provides enables you to drill from a particular request to the PurePath for that request. The PurePath provides a tiered performance breakdown for the request, detailing how much time the request spent at each tier of your application. Check out Andreas’s post on the integration (including some code samples!).

Ed.

Are You Reporting Inaccurate Performance Numbers with VS 2008?

edglas — Tue, 19 May 2009 07:00:00 GMT

There’s a hidden problem with VS 2008 load testing that you need be aware if you are running tests locally (as opposed to using a controller and agents).

If you purchase and run a load agent, load agents run on all cores and all CPUs on the machine.

However, when running locally VS limits load generation to one core on the machine. If this core is running at or close to 100% CPU utilization, it will throw off performance measurements and interfere with performance counter collection. In general it really invalidates your test results. This is why we graph agent CPU by default, and have performance counter thresholds that fire to warn you if the test is running too hot. However, by default the test is configured to look at the _Total instance, when it should be looking at the “0” instance.

This screen shot really tells the story:

CPU 0 is clearly pegged:

Yet the CPU counter in the load test is reporting ~50% utilization:

‘

What is going on here is that the counter reported on is the _Total counter.

To get an accurate measurement we should graph the instance 0 counter instead.

To change your tests to graph the instance 0 counter, change the counter instance in the load test editor:

Now if I re-run, alarm bells should go off as thresholds violations. But they don’t!

The reason is the analyzer is still displaying the _Total instance by default. However, you can see the threshold violation is now firing:

To graph instance 0, drill into the counter tree and drag it on the graph:

Now you can see the threshold violations displayed on the graph. To graph the "0" instance by default, edit the .loadtest file in the xml editor and change the instance:

You can fix this for all subsequent load tests you create by editing the <installdir>\Common7\IDE\Templates\LoadTest\CounterSets\AgentCounterSet.CounterSet file and adding a new instance tag for the “0” instance:

<CounterCategory Name="Processor">
<Counters>
    <Counter Name="% Processor Time">
      <ThresholdRules>
        <ThresholdRule Classname="Microsoft.VisualStudio.TestTools.WebStress.Rules.ThresholdRuleCompareConstant, Microsoft.VisualStudio.QualityTools.LoadTest">
          <RuleParameters>
            <RuleParameter Name="AlertIfOver" Value="True" />
            <RuleParameter Name="WarningThreshold" Value="90" />
            <RuleParameter Name="CriticalThreshold" Value="95" />
          </RuleParameters>
        </ThresholdRule>
      </ThresholdRules>
    </Counter>
    <Counter Name="% Privileged Time" />
    <Counter Name="% User Time" />
</Counters>
<Instances>
    <Instance Name="_Total" />
    <Instance Name="0" />
</Instances>
</CounterCategory>

Also change the settings for the default counters to show on the graph:

Hope that helps, and we certainly will get this fixed for dev10 (but the fix did not make it into beta1).

Ed.