Ed Glas's blog on VSTS load testing

Install Visual Studio 2010 RC Today

Visual Studio 2010 RC is available on the Web today for MSDN subscribers. Go to the RC landing page for details.

We have a very short window, just a few weeks, for incorporating feedback from you back into the product for RTM. Please install the RC today, so we can fix any critical issues you find in the product before RTM. We’ve already fixed a couple of bugs since the RC in the load testing product, generated from internal adopters of the RC. Overall feedback so far has been very positive – people love this release!

Between Beta 2 and RC we’ve made substantial improvements across VS, especially focusing on performance and reliability of VS.

We have been very busy since releasing beta 2. Of course we’ve fixed bugs across the feature set and also rounded out some rough edges, including:

Web Test Recorder

With the RC release, we now have addressed the most common problems as described in Diagnosing and fixing Web Test recorder bar issues. We have also fixed some patterns around correlation and hidden field binding in AJAX requests that we were not handling properly before.

• Handle security context switches in IE7 and IE8 in the recorder
• Handle process boundaries in IE8 in the recorder
• Handle Window.Open going across process boundaries
• Two pass hidden field matching (better matching for ajax requests)
• Better dynamic field detection (handles cases where referrer not set)

Web Test

• Support for binary post bodies, which includes a new option in tools-options-web test, and an extensibility point to override the form post body editor control. Customers testing SilverLight clients calling via WCF will find this pattern now “just works” in the RC.
• Parameterize server command now handles urls in query string parameters. This is a pretty common pattern, as seen in SharePoint.

Analyzer

We made really nice changes in the analyzer to address feedback you’ve given us.

• Details view: filter by error type. This gives a super-powerful way to visualize certain errors.
• Details view: Paint overview graph on details zoom bar. This enables you to see what was going on in the test at a particular point in time.
• Maximize use of space in graph view for graphs: make zoombars and fonts smaller, move vertical zoom
• Counter Range Groups sets like ranges for like counters (e.g. two instances of the same counter always use the same range, or related counters such as Bytes Sent, Bytes Received, and Bytes Total)
• Use transaction request time, not elapsed time, in stats
• Show transaction stats in tables view
• Differentiate GET and POST for the same urls in reports

Reporting

• Select default counters for excel reports. This makes it a lot easier to create your first report.
• Excel reports now work on 64 bit Excel

Load Test API

• Expose run id in load test object

Please install the RC and send us your feedback.

Thanks,

Ed.

Configuration Options for Load Testing with Visual Studio 2010

In this post I outlined the products you need for load testing with Visual Studio 2010. I included a diagram with one configuration option for load tests. Here I will discuss other configurations available.

Configuration 1: “Local” Load Generation

This is what you get when you install Visual Studio Ultimate, which is the ability to generate load “locally” using the test host process on the same machine that VS is running on. In addition to limiting load to 250 users, it is also limited to one core on the client CPU.

Note that purchasing Ultimate also gives you the ability to collect ASP.NET profiler traces by using a Test Agent as a data collector on the Web server.

 

 

Configuration 2: Distributed Test Controller and Test Agents

This is a common configuration if you are scaling out your load agents. With this configuration, the Test Controller and each Test Agent is on a separate machine.

The advantage of this configuration is the controller is easily shared by team members, and overhead from the controller does not interfere with load generation nor operation of the client.

Note the Test Controller must have one or more Virtual User Packs installed to enable load testing. Load agents in this configuration always use all cores on the machine.

 

Configuration 3 A and B: Stacked Configuration

With configuration A, you install the Test Controller and Test Agent on the same machine as VS, then configure the Test Controller with Virtual User Packs. This enables you to generate >250 virtual users from the client machine, and unlocks all cores in the processor. Configuration B shows an alternative configuration, enabled if you configure the machine with Virtual User Packs using the VSTestConfig command line.

Note that a Virtual User Pack can only be used on one machine at a time, and configuring it on a machine ties it to that machine for 90 days. So you can’t have the same Virtual User Pack installed on both the VS client and a separate machine running the Test Controller. See the Virtual User Pack license for details.

 

Configuration 4: Stacked Controller, Distributed Agents

In this configuration, the controller is running on the same machine as the Test client, with distributed agents running as load generators. This configuration is recommended if you have a solo performance tester. If your test controller and test agents will be shared by a team, we recommend running the controller on a separate box. Note that test agents are tied to a single test controller. You can’t have two test controllers controlling the same agent.

Conclusion

If you are using Visual Studio 2008, these options should look familiar to you as the VS 2008 load agents and controller offered the same configuration options. The new twist with VS 2010 is the Virtual User Packs, which, as I explained in this post, offer you more flexibility in how you configure your load agents.

Ed.

VS 2010 Product SKU Lineup Finalized

Here are the VS 2010 SKUs, and testing features available in each (see this link for more feature comparisons).

Client SKUs

Name	Test tools it contains
Visual Studio 2010 Ultimate	Test Tools in VS Premium Load Testing Network Emulation IntelliTrace Microsoft Test Manager (manual testing and test case management) Test Agents, Test Controller
Visual Studio 2010 Premium	Test Tools in Pro ASP.NET Profiler Coded UI Tests (UI Automation) Test Data Generator
Visual Studio 2010 Professional	Unit tests
Visual Studio Test Professional 2010	Microsoft Test Manager Test Agents, Test Controller

 

Server SKUs

Name	Test tools it contains
TFS	Includes TCM Server, and some lab management components
Visual Studio Lab Management 2010	VMM Server, Visual Studio Agents
Visual Studio Agents 2010	Test Agent, Test Controller, Lab Agent
Visual Studio Load Test Virtual User Pack 2010	A license key (no extra software)

Note that the Lab Management product will be shipped sometime post VS 2010 RTM. An RC build will be available when VS 2010 ships. We are looking for more customer feedback before we RTM the product.

Now, looking at a typical load testing configuration:

Just using Visual Studio Ultimate enables you to generate 250 virtual users of load. To go higher than 250 users, you need to purchase a Virtual User Pack, which gives you 1000 users. You can use the 1000 users on any number of agents. Note that if you install the Virtual User Pack on the same machine as Visual Studio Ultimate, you do not get 1250 users on the controller. The 250 virtual users you get with Ultimate can only be used on “local” runs, not on a Test Controller. If you need to generate more 1000 users, you purchase additional Virtual User Packs, which aggregate or accumulate on the Test Controller. In other words, installing 2 Virtual User Packs on one controller gives you 2000 Virtual Users, which can be run on any number of agents.

The Test Controller and Test Agent are “free” when you purchase Ultimate.

Transitioning from Visual Studio 2008

If you are transitioning from Visual Studio 2008 Team System Edition for Testers, or Visual Studio 2008 Team Suite, and and have purchased Software Assurance, you will automatically be transitioned into Visual Studio Ultimate 2010.

If you have purchased Visual Studio Team Test Load Agent 2008 under Software Assurance, you will be granted 5 Virtual User Packs.

VS 2010 Compatibility with VS 2008

All of your test assets will transfer 100% from Visual Studio 2008: Load tests, web tests, unit tests, and test projects. On opening the test project, you will be prompted to convert the project to VS 2010. Clients and test controller/test agents are not compatible with each other. So 2008 clients can’t talk to a 2010 controller, and visa-versa. Also 2008 agents and controllers do not mix with 2010 agents or controller. This is the same constraints we had when moving from VS 2005 to VS 2008.

One thing that does not get upgraded is the load test results database.  On installation of VS 2010, a new database is created. If you want to continue to work with previous load test results, you can use the load test results manager in VS 2008 to export them from the VS 2008 database and then use VS 2010 to import them into VS 2010.

Conclusion

I hope you find the move to VS 2010 a smooth one, I certainly expect you will. There are many, many new features in VS 2010 that you’ll want to take advantage of, overall I highly recommend you upgrade as soon as you can, it will be worthwhile for you to do so.

New Web and Load Test Features in Beta2

In beta2 we added a number of new minor features, and a couple of major ones.

We also changed our licensing model, which you can read about here.

Here is a list of the minor features we added, along with a brief description of each. The last one in the list, compare two runs in Excel, is probably my most favorite feature in the entire release.

Area	Title	Description
Recorder	Recorder options in tools options	Starting URL supplies a default url to start recording from. Two new options to control whether or not the recorder does hidden field matching, and whether or not it runs the dynamic parameter detection tool after recording completes.
Recorder	Recorder records all requests by default, fold dependents under page	In 2008, static dependents not found by the dependents parser were not recorded by default. Now they are.
Web test	search/replace	Search and replace in web test requests!
Playback	Add request bytes column	Request byte count is now shown in request list pane in playback (in addition to response byte count)
Playback	PreRequestDataBinding event	This new event fires before data binding occurs, allowing a plugin to dynamically change data bindings in a request.
Data binding	64 bit csv	This fixes a limitation we had in beta1, and now enables csv data sources on 64 bit machines.
Data binding	New APIs on WebTest to control data cursor position and reload data	Give full control over the cursor, enabling tight control of which user is using what data. Also reload enables refreshing a datasource during the test.
Load Test	Scrub counter sets	We refreshed the set of counters being collected, thresholds, and default counters.
Analyzer	Add standard deviation and 99th percentile stats to load test analyzer
Analyzer	Graph instance 0 for CPU on local runs by default	Fixes the problem documented in this post.
Analyzer	Specify range for a counter (CPU range is 100)	Now you can specify a range for a given counter. Also percentage counters get a range of 100 by default.
Analyzer	Fixed range on perf graph	The performance graph now gets a default range set for it (range is rounded up response time of the slowest measurement)
Analyzer	Computer tags	Computer tags enable you to classify computers by function (e.g. web server, db server, etc.). Also enables run to run comparison even if the runs are on different physical machines.
Reporting	Compare two runs in Excel	This is a new, rich reporting option that enables you to compare performance and resource utilization data in two different runs.

Ed.

Getting Started with Visual Studio 2010 Beta 2

As you are probably already aware, Visual Studio Beta 2 is now available for download, see the landing page here.

For load testing, you will need to download and install:

Visual Studio Ultimate Beta 2 (x86)

Visual Studio 2010 Team Agents Beta 2 (x86)

Ultimate is the new equivalent of Team Suite, and contains Web performance tests and load tests.

Team Agents contains the controller and agents. Note that the x86 agent contains both 32 and 64 bit assemblies, so if you are looking for the 64 bit version of the load agent, this is the download you need.

As I posted here, we have moved to a virtual user license model. Ultimate is limited to 250 virtual users and one core of the CPU. To generate more than 250 users and use all cores on your machine, you will need to install a virtual user pack.

The most straight forward way to do this is to install a controller and agent (or multiple agents) on the same or separate machines.

We have re-architected the controller and agent setups such that setup does little more than lay down the assemblies, while a post-setup configuration tool does the heavy lifting to get the machine configured. The configuration steps are the ones that typically fail, so rather than having to reinstall the entire product and go back through all the configuration steps, if one part fails you only need to reconfigure that part. We have also enabled command line versions of both setup and the configuration tools.

Here is the controller config tool with options called out:

So you will need to fill in the controller service user using a user with counter collection access to your systems under test (same as in VS 2008), and then configure the load test database in the “Configure for load testing” section.

To  add virtual users, start a Visual Studio Command prompt as Administrator (on Vista or Win7, just type Visual Studio Command Prompt in the search window, on XP you’ll find this under the Visual Studio group in the start menu). Then run this command line:

TestControllerConfig licenses /addkey:V10000

This will license the controller for 10,000 virtual users. Note that this license is temporary, and will only work in beta2. For RTM, we will only offer the 1,000 user license pack. If you have a Volume License, you will be able to get your license key from the volume licensing site, and then enter a count of licenses you want to install.

Once you have a virtual user license installed, that license can also be used for local runs (that is, the runs do not have to go through the controller).

If you just want to do local runs, and not use the controller and agent, you can run this from the command line to install the virtual user license:

VSTestConfig licenses /addkey:V10000

This enables you to exceed 250 users for local runs, and unlocks all the cores on your CPU.

Ed.

Introducing the Microsoft Visual Studio Load Test Virtual User Pack 2010

In Visual Studio 2010, we are changing our load test licensing model. In VS 2005 and VS 2008, we shipped the load agent, which was licensed by CPU on the machine generating load. In Visual Studio 2010 we are changing our license model to a virtual user based model. This new license model offers a number of advantages for you, which is ultimately why we chose to make the change:

1. Still way less expensive than our competition.
2. Enables better purchasing planning and allows an apples to apples price comparison with competitors.
3. Enables more flexible machine allocation for load generation.
4. Enables you to use agents for more than just load testing.

Still Way Less Expensive than our Competitors

Your initial reaction may be “Groan – that means that Microsoft is going to be just as expensive as those other tools!” But that is not the case at all, we were careful to maintain the per-virtual-user price we set out initially with VS 2005. With VS 2005 and VS 2008 load agents, the price was around $5,000 per load agent CPU, and our general guidance was that for a typical load test expect around 1,000 virtual users per CPU. We are not yet disclosing our pricing yet, but I can tell you our goal is to stay in this same ball park with VS 2010.

Better Planning and Comparison

With the VS 2008 load agent, we often get questions from customers doing purchasing planning asking “How many load agents do we need to purchase? We are planning to test up to 5,000 virtual users.” To which our answer with VS 2008 was “we do not know, it depends on your application and test scripts.” Of course this it difficult to know how many agents to purchase until actually doing testing, which was often too late. With virtual user licenses, purchasers will know exactly how many licenses to buy.

The old CPU-based license model also makes it very difficult to compare prices with competitive solutions. Most competitive products are licensed by virtual user and data collectors. If you know how many virtual users you want to test to, but do not know how many agents you will need to generate that number of users, it is impossible to make a comparison. With the Visual Studio Load Test Virtual User Packs 2010, if you know the virtual user count you want to test to, you will know exactly how many licenses to purchase, and be able to compare directly with competitive solutions you may consider.

Enables More Flexibility with Load Generators

The virtual user license offers flexibility with the machines you use as load generators.

With VS 2008, to get the full value of the per-CPU license, you had to run on fast CPUs, often requiring new hardware purchases. Also the license was locked to a particular machine, restricting which machines you could use as load generators. With Visual Studio 2010, you can now run virtual users on older hardware with no “penalty”, as you can line up as many older machines as you wish to generate the target number of virtual users. Also, the virtual user license is now tied to the controller, not the agent machine, so you are allowed to move agent machines in and out of the agent pool.

For some tests, you may run out of some other resources before running out of CPU. The most common case for this in VS 2008 was load agents running out of memory before CPU (especially for web tests with very large post bodies or very large responses), which prevented customers from realizing the full value of their license. With the virtual user license, if this happens you can simply add more machines to the agent pool to achieve the load you are targeting.

Visual Studio 2010 Agents: More than just Load Generators

With the high cost of the agent license, we did not have a viable solution for distributed functional testing. In Visual Studio 2010, agents are now included at no extra cost with VS Premium, Ultimate, and Test Elements. We’ve added Data Collection facility to agents, and added new UI automation test type, which opens up a bunch of new scenarios for using agents.

What if I already have VS 2008 Load Agents?

If you have purchased VS 2008 load agents with software assurance, you will be able to migrate your existing licenses to a commiserate set of Virtual User License Packs. We will iron out the details and mechanics of this once we get closer to RTM.

Conclusion

You can see there are many advantages to you with the new licensing model.

Ed.

Using The Virtual User Activity Chart to Understand the VS Load Engine

In dev10 we’ve added a new Virtual User Activity Chart that shows virtual user activity. Each row on the chart is a virtual user.

Covering the Basics of the Chart

See Sean Lumley’s post VSTS 2010 Feature- Load test virtual user activity visualization to see an over view of the Virtual User Activity Chart.

Below is the test view of the chart, which shows which shows which test a user is running. This load test has 25 user constant load, with three unit tests in it: Fast (takes 1 second to execute), Medium (takes 5 seconds to execute), and Slow (takes 10 seconds to execute), using the Sequential Test Mix (which I will go into more detail on later in the post).

Zooming in and using Paint to group 5 users at a time for easy counting, the chart shows 25 rows, one for each user.

Zooming in using the zoom bar on the chart, you can see user 0 first ran the fast test, then the medium test, then the slow test:

Which is confirmed by hovering over the purple Medium test bar:

This view shows you exactly what each virtual user was executing during the test, so it will enable you to see patterns of user activity, load patterns, and correlate failed or slow tests and requests with other virtual user activity.

Another use of this view is it really helps you understand what the load engine does.

In this post, I’ll review various properties in the load engine and show the details view so you can see the effect these properties have.

Understanding Test Mixes

First, I’ll show different test mixes and their corresponding details view, using tests that will accentuate the differences in the mixes.

I’ll use three tests, long, medium and short. Long runs for 10 seconds, medium for 5 seconds, and short for 1 second.

Test Mix Based on Number of Tests Started

Here is the Activity Chart for this test mix, with 34% Fast, 33% Medium, 33% Slow. With this mix, as each user finishes a test, the user randomly selects the next test based on the test weightings.

One think you’ll notice is the predominance of Yellow, and lack of Blue. You wouldn’t think it to look at the user details view, but at the end of this test, the count for each test type is roughly the same. Since the Slow test runs 10x slower than the fast test, there is 10x more yellow than blue.

Name	Scenario	Total Tests	Avg. Test Time (sec)
Slow	Scenario1	912	10.0
Medium	Scenario1	934	5.00
Fast	Scenario1	1,012	1.00

34/33/33 Test Mix Based on the Number of Virtual Users Starting Tests

What this mix is doing is trying to keep, and any given time, the mix of virtual users running the tests to hit the percentage specified. Given the fast test starts and ends so quickly, you would expect a lot of more iterations of the fast test which is exactly what we see, a lot more blue:

Name	Scenario	Total Tests	Avg. Test Time (sec)
Slow	Scenario1	472	10.0
Medium	Scenario1	952	5.00
Fast	Scenario1	5,329	1.00

Test Mix Based on User Pace

In this test, I’ll run the same three tests, and specify they each run 120 times per hour. I chose 120 since it will take 16 seconds to run all three tests, so if I run each twice a minute, I should get a nice graph with about half the time spent running tests. Of course, this mix is not intended to be used this way, it should be used to model your user behavior (how many times an hour do you expect a user to do a particular action).

Now we see a very different graph, with a lot of pacing time between iterations:

Name	Scenario	Total Tests	Avg. Test Time (sec)
Slow	Scenario1	174	10.0
Medium	Scenario1	177	5.00
Fast	Scenario1	183	1.00

Test Mix Based on Sequential Test Order

With this mix, each user simply runs the tests in the order you specify in the mix, which is easy to conceptualize and super-easy to see:

The Effect of Think Time Between Iterations and Warmup Duration Properties

As you can see from this last detail view, this test has all users going lock step through the application, all doing exactly the same thing at the same time. While this may be useful for doing stress testing, it is not realistic user behavior, which is why we have the other test mixes which do a better job of varying what users are doing.

Another way to vary what users are doing and introduce pacing into a scenario is by using Warmup Duration and Think Time Between Iterations. Think Time Between Iterations is especially important if your test only has a single web test in it (thus no think time), or you are running a unit test with no pacing (for unit tests this is implemented via sleep calls, but never use sleep calls in a web test).

Here’s what happens when I put a 30 second warmup in the last test:

Now you can see the virtual users stagger their start over 30 seconds.

And here is the same test (no warmup) with a 5 second think time between iterations:

The variations are due to the Normal Distribution think time profile that is applied by default. This profile will vary the specified think time using a normal distribution (bell curve).

The Effect of Percentage of New Users

Now I am going to clear the delay between iterations, and use 100% new users. What I’d expect here is that each user will only run one test, leading to a lot of rows in the view, which is exactly what happens:

Scrolling up vertically one screen shows this:

So you can see the load engine repeatedly starting new users. As one group of 10 users completes, the next group of 10 starts. New users have a clean cache and cookie container, which matter for web tests.

Different mixes interact differently with the percentage of new users:

• Sequential Mix as you can plainly see above, we made the design decision on sequential mix to have each new user run each test once.
• With Test Mix Based on Sequential Test Order and Test Mix Based on Number of Tests Started, each user runs one test.
• Test Mix Based on User Pace ignores the percentage of new users altogether, as the mix relies on a repeat user to establish the proper pacing.

The Effect of Test Mix Init and Terminate Tests

Init and terminate tests one once per each user. Switching my test back to 0 percent new users, and adding an init and terminate test shows the following:

Where the legend shows

This clearly shows the init test running once per virtual user, as the very first test executed, exactly as it should.

But wait! Where is the terminate test? It is not in the legend, and does not show up scrolling the view all the way to the end of the test. This shows the terminate test only gets called when a cooldown period is configured in the load test. Re-running with a 30 second cool down demonstrates this:

Understanding Load Patterns with the Virtual Users Activity Chart

Each of the above tests uses a constant user load of 25 users. Now I’ll use the chart to show how Step Load and Goal-based load based loads add users. For these tests, I’m going to use just one test in the mix, the 1 second fast test, to draw attention to the load pattern rather than the test mix.

Step Load

Here’s the Activity Chart for a step load pattern adding 10 users every 10 seconds until 100 users is reached. Notice the vertical scroll bar to scroll up to see the remaining users. Zooming in on the first 30 seconds shows the first 30 users created.

 

The step load Step Ramp Time property enables you to ramp the users added at each step. I highly recommend using this for every step load pattern, particularly if you are using large steps. Here I set a step ramp of 5 seconds, which shows users ramped up for each step over 5 seconds.

Goal Based Load

For a goal based load pattern, it is important that the counter you are seeking the goal on has a direct correlation with the load at each interval. So for this demonstration I put a tight loop in my test to provide a direct correlation between CPU on the load agent and the virtual user count. For this test, I also did a step load before hand to see what a good CPU range would be.

Here’s user load:

Let’s look at the Activity Chart for the areas where the dips are:

Here you can see virtual users stopping as they finish an iteration in order to achieve a lower load.

Scrolling up, you can see that new users are added to the test to move the load back up.

 

One thing that surprised me here is that users are not recycled. That looks like a bug to me!

Conclusion

Hopefully this post unravels some of the properties of the load engine, and it shows you just one of the ways you can leverage the new Virtual User Activity Chart in dev10 to understand exactly what your load test is doing!

Increasing the ROI of our Automation

A while back I posted this on automation. I want to resurface this topic now, as it has been on my mind a lot lately as we develop dev10. What is the “right” approach for automation? How do we make it pay off?

Automation is hugely important for us. It enables us to ensure our software works on a myriad of OS (XP, Vista, Win7, Server 2003, Server 2008), processor architecture (x86, ia64), and language (English, German, Japanese, Chinese, etc.) combinations.

At the end of the day, return on investment is what it is all about. Is it faster to develop and maintain an automated test case, or simply run the tests manually? If over the lifetime of a test case you spend more time fixing it than it would have taken to run the test manually for each time you ran the automated test, that test has not paid off. This is basic and obvious, but often times get overlooked when considering automation.

Going back a bit in history to the VS 2005 release when we initially introduced load testing, we made a few huge mistakes with our approach to automation, mistakes which I see others repeating over and over:

1. My test team wrote automation exclusively using UI automation.
2. My test team owned developing and maintaining the automation.
3. Dev “unit tests” were not considered “automation”.
4. Devs and testers used different test automation frameworks.

With dev10, we have reversed each of these, and are benefitting greatly from it.

One thing I really want to emphasize is the cost of maintaining our UI automation. I have not looked at the exact number, but in the course of developing dev10 nearly every single UI automation test case has broken. Contrast that with our tests that test at the API layer, I estimate that fewer than 10% have broken. To add insult to injury, the UI tests are hugely expensive to fix. API tests that break nearly always give compile errors, which are easy to fix. UI tests do not fail until you run them, and then you have to grovel through the test log to figure out what UI controls have changed to cause the failure, make a fix, rerun the test (perhaps to find another failure), all of which is extremely time consuming.

Increasing our ROI: A Cost-Based Approach to Automation

For dev10, we have developed four methodologies for automation, and consider each when considering how to automate testing of particular functionality. We give preference to the lowest cost approach. That is, if we can use a cheaper approach to automate, we do it that way and we don’t automate everything through the UI. In fact, the UI is the least favored approach, and is only done if none of the other approaches will give us the coverage we need. The approaches, in order of preference, are:

1. Unit tests that run in the test process. Really “unit test” is a misnomer for most of these tests, as they typically test a broader area than a pure unit test. I prefer to call these tests API tests. The defining characteristics of these tests is that they are hosted by the test process.
   
   An example of this for our load test product is we have tests that run requests through our web test engine.
   
   Another example is in our TCM server product. Nearly all of the automation for TCM server has been developed using this approach.
    
2. Command line tests. With this approach we have “unit tests” that spawn our command line. The test code then waits for the command to complete, and then uses APIs to validate the command did the right thing. We use this approach for both web and load test features and for our TCM server.
   
   For example, for load tests we have tests that run a load test from the command line, then access the load test database to ensure the results were as expected. This gives us substantial coverage of our web and load test feature set.
   
3. API tests that run in the process under test. mstest has a feature that enables processes other than vstesthost.exe to host the test code. An example of this is ASP.NET unit tests, which run in the ASP.NET process. We take advantage of this extensibility point to add our client processes as host processes.
   
   
   An example of this is we run our new TCM test client Camano (aka Microsoft Test and Lab Manager) as a test host. This is a bit of a mind bender, but we also actually run VS as a test host! So devenv starts vstesthost which starts a second copy of devenv. Once VS is loaded, our test code runs and has access to our architectural layer “just beneath the glass” and enables us to run web and load tests in the IDE (e.g. open a project, open and edit load tests in the project, run them, validate the UI through code).
   
4. UI Automation. With this approach, our test code instantiates the client under test, and drives it through the mouse and keyboard (we drive VS through MSAA, and we drive our TCM client using UIA).

The cost of UI automation is at least an order of magnitude greater than the other three approaches. Isn’t it ironic that we are introducing a new UI automation test type in dev10. If UI tests are so bad, why are we bringing them to market? While I maintain that they are very expensive, UI tests are also absolutely necessary. There are some things we simply cannot test without them (like dialogs, or some integrated, cross-process end-to-end scenarios). But if I go back and look at the UI tests we developed for web and load tests in VS 2005, the vast majority of them could have been implemented using a cheaper alternative.

Engineering for Automation: Developing Testable Software

Developing these other automation approaches did not come for free. We have deliberately architected our software to enable each of these approaches, and spent time developing test frameworks and reference patterns for each approach. It has really paid off in a big way. If you look at our TCM tests, we maintain an extremely high pass rate (98% plus). Our “legacy” UI tests from the 2005 release are in bad shape.

Whose job is it anyway?

At the end of the day, it is the test team’s job to help us understand whether or not we are ready to ship. Automation plays a huge role in that, since we are able to run our automation suites unattended and generate test results relatively quickly and concurrently on many different configurations. The test team is responsible for ensuring we have the right set of automation in place help us understand where we are at.

Getting Testers and Developers on the Same Page

However, it is not solely the test team’s job to develop and maintain the automation. The test team develops the test plan, which includes the scenarios and functionality we want to automate, but both the dev and test team develop and maintain the automation. If a developer is writing “unit tests” for a new API they are developing, they work off the test plan to develop one of the specified test cases. Tester do not duplicate this effort by developing “their own” automation to cover. A key tenet is that testers and developers use the same frameworks and methodologies for developing automation. Testers are able to run tests developed by devs, and visa-versa. Going back to the mistakes we made in VS 2005, this simple methodology, along with considering the most cost-effective way to automate a given test case, is the key to fixing each of these mistakes.

Similarly, it is the entire team’s job to maintain the automated tests. As I said earlier, maintenance is the hidden and huge cost of doing automation. Devs must fix any compile errors in tests introduced by their changes, and also must ensure that all P1 tests pass before checking in. Testers and devs share the load on ensuring all P2 and P3 tests maintain a high pass rate. We run these once a week and distribute the load on driving them to passing.

I hope you can also learn from our mistakes, and use our experience to improve your approach to automation.

Ed.

Read this Before Running a Load Test with Dev10 Beta 1

If you install the dev10 beta 1 on the same machine as VS 2008, and you are running load tests, read this.

When you run the first load test, the load test will check the schema of the load test database, and if it is the 2008 schema it will update the schema in place.

This will create problems for you in VS 2008: you will no longer be able to import and export results.

The workaround is to use two separate databases for your results, one for VS 2008, one for VS 2010. The new Excel reporting feature in dev10 looks for the load test database (we’re fixing that in beta2), so the best workaround is to rename the VS 2008 load test database.

Before running any 2010 load test on your machine, rename the VS 2008 load test database:

1. Close all Visual Studio instances.
2. Run a Visual Studio command prompt as Administrator (can be 2008 or 2010).
3. Run
   sqlcmd –S .\sqlexpress
   1> EXEC sp_renamedb 'LoadTest', 'LoadTest2008'
   2> GO
4. Start Visual Studio 2008
5. Run Test | Administer Test Controllers
6. Change the load test results store to the renamed database
   1. Click …
   2. Under “Select or Enter a Database Name” type “LoadTest2008”

If you run VS 2010 tests prior to renaming the database, you will need to remove the dev10 results from the database:

1. From Visual Studio 2010, open a load test.
2. Click Open and Manage Test Results icon
3. Select the 2010 test results and click export.
4. Delete the 2010 test results.
5. Go through the steps above to create two databases.
6. Open VS 2010 and use Open and Manage Test Results to import the 2010 results back into the database.
7. Delete the extra columns created in your database. We’ll blog a script for doing that soon.

Note the same thing will happen during setup if you install the Test Controller – controller setup will update the existing 2008 load test database. So be sure to rename the database before running controller setup.

Ed.

New Performance Testing Training Available from RTTS

RTTS, a premier testing consultancy and test tools training company, now has training available for performance and load testing with VSTS 2008. RTTS specializes in providing software quality services, and offers training on a variety of tools. It is great to see VSTS in the mix as they see demand for VSTS growing. RTTS offers VSTS performance testing training onsite, over the web, or at their site, as well as consulting services. More details available here.

Ed.

More posts on Testing SharePoint with VSTS

Shivaji Nagarajan has extensive experience testing SharePoint using VSTS, and has captured what he has learned in a series of blog posts. Be sure to check them out if you are testing SharePoint!

Ed.

dynaTrace Leverages new Extensibility Points to Provide Deep Integration with Dev10

A fantastic example of a partner taking advantage of the new extensibility points in Web test playback and load tests in dev10 is provided by dynaTrace. dynaTrace has developed an extension that provides enables you to drill from a particular request to the PurePath for that request. The PurePath provides a tiered performance breakdown for the request, detailing how much time the request spent at each tier of your application. Check out Andreas’s post on the integration (including some code samples!).

Ed.

Are You Reporting Inaccurate Performance Numbers with VS 2008?

There’s a hidden problem with VS 2008 load testing that you need be aware if you are running tests locally (as opposed to using a controller and agents).

If you purchase and run a load agent, load agents run on all cores and all CPUs on the machine.

However, when running locally VS limits load generation to one core on the machine. If this core is running at or close to 100% CPU utilization, it will throw off performance measurements and interfere with performance counter collection. In general it really invalidates your test results. This is why we graph agent CPU by default, and have performance counter thresholds that fire to warn you if the test is running too hot. However, by default the test is configured to look at the _Total instance, when it should be looking at the “0” instance.

This screen shot really tells the story:

CPU 0 is clearly pegged:

Yet the CPU counter in the load test is reporting ~50% utilization:

‘

 

 

 

 

 

What is going on here is that the counter reported on is the _Total counter.

To get an accurate measurement we should graph the instance 0 counter instead.

To change your tests to graph the instance 0 counter, change the counter instance in the load test editor:

Now if I re-run, alarm bells should go off as thresholds violations. But they don’t!

The reason is the analyzer is still displaying the _Total instance by default. However, you can see the threshold violation is now firing:

To graph instance 0, drill into the counter tree and drag it on the graph:

Now you can see the threshold violations displayed on the graph. To graph the "0" instance by default, edit the .loadtest file in the xml editor and change the instance:

      <DefaultCountersForAutomaticGraphs>
        <DefaultCounter CategoryName="Processor" CounterName="% Processor Time" InstanceName="0"/>
        <DefaultCounter CategoryName="Memory" CounterName="Available MBytes"/>
      </DefaultCountersForAutomaticGraphs>

You can fix this for all subsequent load tests you create by editing the <installdir>\Common7\IDE\Templates\LoadTest\CounterSets\AgentCounterSet.CounterSet file and adding a new instance tag for the “0” instance:

<CounterCategory Name="Processor">
  <Counters>
    <Counter Name="% Processor Time">
      <ThresholdRules>
        <ThresholdRule Classname="Microsoft.VisualStudio.TestTools.WebStress.Rules.ThresholdRuleCompareConstant, Microsoft.VisualStudio.QualityTools.LoadTest">
          <RuleParameters>
            <RuleParameter Name="AlertIfOver" Value="True" />
            <RuleParameter Name="WarningThreshold" Value="90" />
            <RuleParameter Name="CriticalThreshold" Value="95" />
          </RuleParameters>
        </ThresholdRule>
      </ThresholdRules>
    </Counter>
    <Counter Name="% Privileged Time" />
    <Counter Name="% User Time" />
  </Counters>
  <Instances>
    <Instance Name="_Total" />
    <Instance Name="0" />
  </Instances>
</CounterCategory>

Also change the settings for the default counters to show on the graph:

      <DefaultCountersForAutomaticGraphs>
        <DefaultCounter CategoryName="Processor" CounterName="% Processor Time" InstanceName="0"/>
        <DefaultCounter CategoryName="Memory" CounterName="Available MBytes"/>
      </DefaultCountersForAutomaticGraphs>

Hope that helps, and we certainly will get this fixed for dev10 (but the fix did not make it into beta1).

Ed.

Elevating the Role of the Tester with Visual Studio 2010

I am very excited and proud to introduce the new testing features you’ll find in Visual Studio 2010. In addition to continuing to work on the performance testing tools, I have also been focusing on delivering a new set of products for the entire test team. Since most testing is done manually, we have really focused on the manual testers with this release. If you do testing or test management you will love the new features. We also are introducing a new solution for UI automation as well as improvements in web, load, and unit testing, and our testing framework.

We have been working on this since before we shipped VS 2008, it is great to (finally) get the beta out. When we set out to define this release, we set forth a set of what we call value props that capture the value in the release. Using these as the frame, we built up the architecture and feature set to match.

In this post, I’ll go over the over-arching themes for the test tools in dev10. Our GM Amit Chatterjee has a terrific set of posts on each of these feature areas on his blog, be sure to check those out (I’ve linked to them throughout this post). Also be sure to check out the beta version of our documentation, which takes a refreshing approach to product documentation.

Introduction to Team Test for Dev10

Visual Studio Team Test (VSTT10) for Dev10 introduces key new functionality for test management, manual testing, test lab management, and automated testing. Specifically,

• VSTT10 introduces an entirely new tester-focused UI that will free Manual Testers from what they see as the developer-centric Visual Studio UI. This new UI (code named Camano) will be the console for test case authoring, management, execution and tracking.
• VSTT10 takes virtual test lab management to new heights by allowing virtual lab creation, configuration and deployment all integrated into the testing experience.
• VSTT10 significantly increases our investment in the Specialist Tester role by improving the capabilities and performance of load and web testing as well as introducing new UI automation capabilities through the Visual Studio IDE.

VSTT10 introduces substantial new value propositions for software testers:

1. Make manual testers more effective by integrating test activity into the ALM and providing tools for managing, developing, recommending and executing tests.
2. Enable manual testers to reliably and consistently create actionable bug reports via automatic data gathering, reducing debugging time and eliminating non-reproducible bugs.
3. Ensure that both requirements and source code are adequately and transparently tested, ultimately answering the question "can we release?"
4. Enable efficient collaboration between QA and the rest of the extended development team throughout all phases of the application lifecycle.
5. Foster a collaborative tester community with an extensible platform that makes it easy to create and share new features and components that further enhance a tester’s productivity and effectiveness.

The core VSTT10 areas of test management, manual testing, lab management, and automated testing cut across each of these value propositions and create lasting value for test and development teams.

Getting the Basics Right

VSTT10 lays the foundation to help you the tester be more effective at the essential aspects of your job. The idea is to “get the basics right” and create a toolset that appeals to all test teams as well as the extended development team. You can watch an overview video here. 

An Emphasis on Test Management and Manual Testing

In talking to you, our customers, we found that many of you track your testing effort using ad-hoc means and informal tools like Excel or Word. Most testing is performed manually by what we are calling generalist testers. To this end, the main focus for VSTT10 is to offer solid test case management tooling as well as a differentiating experience for manual testing. Because we are targeting generalist testers and test managers, we chose to develop a new client outside of the Visual Studio shell. We really wanted to provide a simple, focused experience for testers, and found we could not effectively do that with VS. We think you are going to really like the new client.

TFS-based Test Planning and Authoring

VSTT10 integrates deeply with TFS to offer rich functionality for test planning and authoring. Test planning features enable the test manager to define which test cases to run as part of a specific test plan or testing effort. VSTT10 also incorporates test case authoring, including the ability to define test steps, attachments and to factor and re-use parts of tests into shared steps. Test cases also allow the manual tester to associate data with the test case making it easy to step through different rows of data within the same test case.

A tester adds structure to their test plans by organizing test cases into hierarchical test suites. Query-based suites are used to automatically include any test case that meets specific criteria.

Test configurations allow the tester to define valid configurations that will be covered in a test plan, and then assign which tests will run on which configurations. Again the recurring theme is to better organize a tester to perform more rigorous testing, and track which configurations have or have not been tested.

Test cases are assigned to different testers on a project and tracked by a test manager to allow for better team organization and management of the test effort during the testing phase.

Linking Test Cases to Requirements and Bugs

Test cases are easily linked to requirements, which enables test planning, test execution, and traceability. Linking requirements to test cases also enables rich reporting, such as the count of test cases per requirement and pass/fail/not run testing status per requirement. A tester can easily add a requirements-based suite to a test plan, which automatically adds any test case linked to the requirement to the test plan.

When a new bug is filed while testing, the bug is automatically linked to the test case that generated it. This enables the developer to quickly open the test case that generated the bug, as well as interesting reports on which test cases generate the most bugs.

Documentation links: Managing New Testing Efforts with Team Test, Defining your testing effort using test plans

A Great Manual Testing Experience

The VSTT10 client offers a differentiating and highly productive manual testing experience. The capabilities around bug capture alone are not to be found in any competitors offering and will substantially increase productivity around bug finding, reporting and triaging. . In addition, the tester can easily capture screen shots, mark tests or steps as pass/fail and manage their bug finding techniques at a level not possible in other tools.

A key differentiator for manual testing is in fast forward manual testing. This innovative feature enables a tester to automatically record during testing, and then replay these actions the next time that same test is run. Automation can also be associated with shared steps, enabling testers to fast-forward through common steps that are repeated across test cases. This will allow testers to quickly fill in form fields and other types of repetitive data entry activity by replaying the form fill actions recorded in previous tests. The idea is to increase productivity by focusing automation on repetitive tasks and freeing a tester to concentrate on finding bugs and gaining better test coverage.

Documentation links:

Creating manual test cases, Running Manual Tests with Test Runner, Recording and Playing Back Manual Tests

Are We Ready to Ship?

A key goal for test managers is answering the question “are we ready to ship?” Arguably, answering this question is the primary mission of a test team, and doing so has many facets. The test team must shine a light in dark places of the application under test in order to find problems before customers do. This becomes as much an exercise in understanding what hasn’t been tested as what has. To that end, the VSTT10 will collect and report data to answer questions like:

• Do we have adequate test coverage of our requirements and features?
• Have we run all our tests?
• Have we tested all of our target configurations adequately?
• Have we tested the feature areas adequately?
• Have we tested the changes?
• What are we missing in our regression suite?
• Are we on track to complete our testing?

To answer these questions, VSTT10 makes it easy to:

• Define and organize test cases.
• Define and manage which test cases are to be executed in a test effort.
• Link test cases to requirements and features, and associate them with areas of the application.
• Manage which configurations test cases are run on.
• Seamlessly track which tests have been authored and run, and for those that have been run track pass/fail.
• View which tests have been impacted by code changes.
• View reports which surface this data to enable the test manager to answer the questions above.

VSTT10 achieves this by allowing the tester to define relationships of test cases to requirements and bugs in TFS, and to define the set of test cases and configs to run as part of a test plan. VSTT10 then seamlessly captures test results, ultimately storing this data in the TFS warehouse.

A basic set of reports are available through the process template and hosted on the SharePoint portal.Data is surfaced through reports on top of TFS. Since the data is in the TFS warehouse, reporting services or Excel reporting can be used to flexibly mine the data. A very rich set of data is available and can be related to metrics and measures currently captured by a product team. The warehouse stores test results over the life of a project, and those results can be sliced by build, area path, suite, and configuration.

This allows a test manager and stakeholders to generate reports such as the set of tests run for a given plan, the set of tests not run a given plan, and for those that were run what the results were. Historical results are available in the warehouse, providing stake holders rich visualizations of testing activity and quality trends.

Because test data can be related to requirements, a test manager can generate reports that show pass/fail per requirement, the number of bugs per requirement, test results per area, per suite, per build, and so forth. In all a very rich set of information is available in VSTT10. Check out Amit’s blog post on reporting.

Collaboration with Dev

VSTT10 provides tight integration between developers and testers through the free flow of information between them. Not only can developers and testers share the information stored in TFS, but information required by developers from ongoing testing is gathered automatically as described below. Also for the first time developers can get visibility into which areas of the product have and haven’t been tested, and better understand where their features are at.

Actionable Bug Capture

The basic flow of information from testers to developers is primarily through bugs. The effort wasted chasing down bugs and reproducing errant behavior is a severe pain point for developers and testers alike. “It works on my box” is the norm. VSTT10 takes this on head-on through information rich bug capture.

In order to enable this rich flow of information, VSTT10 delivers an innovative technology called data collectors. Data collectors enable testers to collect information from client and server machines in the test lab while they are testing. Data collectors enable a wide range of scenarios, including rich bug capture.

As test environments can be very complicated, and not all testers are experts at configuring the server system, an important aspect of data collectors is that they “just work” out of the box and are easy to configure and use.

A case in point is an essential new technology for bug capture known as Historical Debugging, which enables developers to debug exactly what was going on on a machine at the time a bug occurred.

As useful as historical debugging is to developers, the Action Recorder is a technology useful for developers and testers alike. It is based on VSTT10’s UI automation recorder, and captures user actions the tester takes and records that information as part of the bug report. The Screen Capture tool enables the tester to easily take screen captures, and the Video Recorder records the testers desktop and links the video to the bug.

Documentation links: Setting Up Machines and Diagnostic Information to be Collected Using Test Settings, Creating A Diagnostic Data Adapter to Collect Custom Data or Impact a Test System

The Essential Role of Builds

As bugs represent the primary flow of information form test to dev, the flow from developers to testers is through the build. Testers pick up new builds to get new features and requirements to test and bugs to regress. The developers’ efforts are realized in the build. As such the build becomes fulcrum of the dev/test relationship and takes a first class role in the VSTT10 product. When selecting a new build, the test manager sees which work items were completed as part of the build and act on those. She sees which tests were impacted by changes in the build and assign them to be run. The ability to see impacted tests is especially important in the end-game just before releasing to production, when the test manager and stakeholders need to make sure they have adequately tested the bug fixes that were made in the final builds.

Documentation links: Recommending Tests to Run That are Affected by Code Changes, Determining Which Builds Have Bug Fixes, New Features or Requirements

Preparing the Test Environment

The build is also central to ensuring high quality builds are passed to testers and assists in preparing the test environment for new builds. VSTT10 enables integration with VMM and Windows Workflow for automated provisioning and setting up the test environment as part of the build process. With VSTT10 build and deployment engineers can define a workflow script to provision machines from VMM, install the latest build, and then run functional BVT tests against the installed build. These can then be leverage by both the development and test teams to quickly set up complicated test environments. VS 2008 limited the level of testing to unit testing of assemblies whereas VSTT10 enables full functional build verification testing.

Testers leverage test environments from within the test client to quickly re-use environments. Network fencing enables testers to have more than one instance of the same VM running at the same time. This enables one person on the test or development team to get a test environment set up, and others on the team to quickly clone the environment for testing.

Test Automation in VSTT10

Test automation is a key and strategic investment area for VSTT10. Every test team wants to be more efficient, and automation is one way to increase efficiency. A goal for VSTT10 is to enable teams to define and run a automated regression suites to validate their primary scenarios are still intact. These automated regression suites are then used both as build verification tests and as a final set of regression tests to run before shipping.

The Record and Playback Engine

A primary investment in VSTT10 is the record and playback engine, which is leveraged in both automated tests and the fast forward for manual testing feature. A great deal of engineering has gone into ensuring record and playback “just works”. Record and Playback supports IE, Winforms, and WPF front ends. Silverlight support and playback on Firefox are planned for an out of band release (between dev10 and dev11). The RnP technology also supports a rich extensibility mechanism for extending to custom controls within these UI technologies as well as new browsers and future UI frameworks. This technology provides the underpinnings for our future UI Automation solutions.

UI Automation

VSTT10 includes a new “Coded UI” UI automation test type which is available in Visual Studio. Coded UI tests are squarely targeted at the more sophisticated end of the specialist tester segment, as it requires development skills. The “Coded UI” test type provides an action recorder, validation tool, and comprehensive API.

Coded UI Tests are built on the mstest unit testing framework, providing a familiar and common paradigm for testers and developers. Test authors leverage the full power of VS, the .NET framework, and managed languages to develop their automation.

Managing Automated Tests

Test management capabilities enable testers to track which test cases have or have not been automated, and to track automated test results over time. As part of the ability to scale-up test execution, VSTT10 enables you to define and run automated suites that span across VS solutions.

Documentation links: How to: Associate an Automated Test With a Manual Test Case and Run It, How to: Create Test Cases from an Assembly of Automated Tests

Performance Testing in VSTT10

VSTT10 builds on the success of VSTT 2008 performance testing tools, with innovative new tools that help testers find performance problems and enable development to fix them.

VSTT10 introduces WAN emulation for both manual and automated performance testing. Network emulation enables you to see how your application will perform over a slow link.

New advances in Web Performance Tests (formerly known as Web tests) make it far easier to debug them, and powerful new extensibility hooks in the web test recorder enable custom rules that can be targeted at specific types of sites so that record/playback will “just work” against those sites.

Load tests leverage data collector collectors to collect information from the system under test during a load test. A significant advancement in this area is the ability to remotely collect a profiler trace from the system under test, enabling testers to collect code-level performance data from the server during the test, so developers will have enough information to fix performance bugs.

The timing details view enables the performance tester to visualize virtual user activity, and quickly drill into a complete virtual user session log from a failed iteration. And new advances in reporting will enable you to easily share performance test results and trends with your team.

Advances in MSTest

Dev10 introduces significant advances in mstest extensibility, with new APIs for executing tests and getting results. This enables many new scenarios, including introducing new test clients outside of visual studio.

Dev10 also introduces test categories, which enable you to categorize tests (think tagging them) and dynamically select a set of tests that are in a particular category. A test can belong to multiple categories at a time. This enables you to move away from static vsmdi test lists, providing a richer way of selecting tests for BVTs.

Mapping Value Props to Feature Areas

1. Make manual testers more effective by integrating test activity into the ALM and providing tools for managing, developing, recommending and executing tests.
2. Enable manual testers to reliably and consistently create actionable bug reports via automatic data gathering, reducing debugging time and eliminating non-reproducible bugs.
3. Ensure that both requirements and source code are adequately and transparently tested, ultimately answering the question "can we release?"
4. Enable efficient collaboration between QA and the rest of the extended development team throughout all phases of the application lifecycle.
5. Foster a collaborative tester community with an extensible platform that makes it easy to create and share new features and components that further enhance a tester’s productivity and effectiveness.

 

	Effective Generalist Testers	Actionable Bugs	Can we Release?	Collab with Dev	Partners / Extensibility
Test Planning	Yes		Yes
Test Authoring	Yes	Yes		Yes
Test Case Linking	Yes		Yes	Yes	Yes
Test Runner	Yes	Yes	Yes
Test Reporting	Yes		Yes	Yes	Yes
Data Collectors	Yes	Yes		Yes	Yes
RnP	Yes	Yes		Yes
Test Automation	Yes		Yes	Yes	Yes
Test Impact	Yes		Yes	Yes
Build Integration	Yes			Yes
Lab Management	Yes	Yes		Yes
Public APIs	Yes	Yes	Yes	Yes	Yes

 

Conclusion

As you can see from this paper, this is a compelling release! But don’t take my word for it, try it out! Download dev10 beta1 today. You’ll need to download and install TFS server. The new test client is installed with VSTS in beta 1 (we are working on a stand-alone installer for beta 2). Once you have the product installed, here is a Quick Start Guide to get you up and running.

Dev10 Beta 1 Available!

Beta 1 was released to the web today for MSDN subscribers. If you are a MSDN subscriber, you can download the Beta today from here.

We have been very busy on this release of late (one reason why my blog posts have slowed down lately :))

My team and I will be blogging about all the great new features in dev10 beta 1 over the coming weeks.

As in the past, I will compose a list of links to any posts on the new features and update this blog post with the links as we get them.

New Web Test Features in Dev10:

• These features are about making it easier to get web test playback working:
• Extensible recorder plugins for modifying recorded web tests (including adding custom dynamic parameter correlation)
• Find in playback
• View recording log in playback
• Jump to request from playback
• Add extraction rule from playback
• WebTestResult in a separate file for better scalability
• New API for web test result
• New extensibility API in playback enables UI extensions to playback
• File Upload record/playback just works
• Uploaded file added to project and deployed
• Option for generating a unique file name on each iteration
• Looping and branching in web test
• New dialog for editing think times, goals, reporting string across all requests
• New validation and extraction rules for inner text and select tags
• Test level validation rule for pages meeting response time goal

New features in Load Test in Dev10:

• Logging and data collection improvements
• Log entire test result on test failure or log entire result every n requests
• Remote profiler integration
• Event Log Data Collection
• Extensible data collectors to support more platforms
• Analysis Improvements
• Load test virtual user activity visualization, also here.
• Reporting
• Run to run perf reports in Microsoft Excel
• Performance testing
• Run automated UI tests as performance tests
• True network emulation
• More flexible load modeling
• New Sequential Test Mix
• Delay start on scenario
• Iteration count on scenario
• Select which agent(s) to run the scenario on
• Disable during warmup
• Multiple load test plugins
• 64 bit test execution

In addition to continuing our testing and bug fixing efforts, right now we’re working on an additional set of features for beta 2. I’ll have to sit on those until beta2. :)

Ed.