Ed Glas's blog on VSTS load testing

Event ordering in Web Tests

Web tests have a number of extensibility points before and after a request. What order do they run in?

The hooks are:

WebTestPlugin.PreRequest

WebTestRequestPlugin.PreRequest

ExtractionRule.Extract

ValidationRule.Validate

Here is a sequence diagram that shows what order things fire:

There is no guaranteed order of firing for WebTestPlugin.PreRequest and RequestPlugin.PreRequest. Also if you have >1 Extraction or Validation rule on a request there is no guaranteed order they are run in.

Also note that Pre and Post request events do not fire by default for dependents. You can hook the PostRequest call on a top level request and iterate through dependents to wire up event handlers if you need that functionality.

Ed.

New Fiddler Timeline view is neat!

Eric Lawrence has added a cool new feature to fiddler to visualize what is happening in your http traffic. Here's a snippet from the online fiddler help:

Timeline

The Timeline Tab shows the transfer timeline of selected HTTP Sessions.  NOTE: You should enable Streaming Mode for the highest accuracy in Timeline rendering.

The vertical line indicates the time to first byte of the server's response.
The green circle before the bar shows that a connection was reused; a red circle means that the connection was newly created.
The green circle after the bar shows that the server didn't send a Connection: close header; a red circle means that the server sent Connection: close.
The gray arrow icon indicates that the server's response was a redirect (302).  The red ! icon indicates that the server returned an error code (404).

How cool is that?!

Ed.

JW post on MS Testing - my observations

Check out James's recent post with the eye-catching title “if Microsoft is so good at testing, why does your software suck?”

I have another angle on this, which I saw first hand with our Whidbey release. It goes to some of the themes I refer to in my So you want to automate your test cases? post. We hire developers as testers. Our tester role is "Software Development Engineer in Test", or SDET. Our SDETs are awesome, our testers here at Microsoft are way better than testers I have worked with previously. But one problem I've seen is that because of our huge emphasis on test automation at Microsoft, sometimes ad-hoc or exploratory testing suffers because of it. That is, sometimes we're so busy writing automation, automation harnesses, and automation tools, we don't spend enough time in our customers shoes banging on our features and trying to break them. Almost all of the bugs we find are found through manual testing, not through automated testing. On my team we brainstorm on tests to run in order to break new features we're developing, schedule specific tasks in TFS for testers to beat on features, and also have dedicated bug bashes for our features.

Of course we can always do better, the recent load test regression that got through in SP1 is evidence of that. I wish we had had an automated test for that one, we do now. :) The timing of finding this bug was unfortunate, and we had another like it in RTM, was that one of our internal customers from the Services Test Labs found this bug in SP1 beta just as the door closed on SP1. We need to drive more adoption of our betas so we can catch these problems before RTM. The other thing is we were pretty aggressive in SP1 about getting fixes and features in that we heard about from customers. This regression was caused by one of those customer-reported problems ("Thinktime that's set via plugin code is ignored"). I don't see that changing, SPs are a chance to get value back to you, we need to take advantage of that. Of course we want to be smart about that, and make incremental changes that pack a lot of power but do not introduce a lot of risk.

Ed.

JW on testing

James Whittaker, a testing luminary, joined Team Test a while back and has started blogging. You'll find his blogs very interesting!

Ed.

Updated load test links

I've updated my links to info on web and load test post with the following links:

New Item Templates for custom rule and plugin development

Data Driven Web Testing With Visual Studio 2008 Team System

InfoPath Forms Services 2007 Web Testing Toolkit

SharePoint Server 2007 Scalability and Performance Whitepaper

Ed.

New templates for plugins and rules

Inspired by templates developed by Dennis Bass of the services test labs, I added new item templates to the codeplex.com\TeamTestPlugins project.

If you install these on your machine it makes creating new plugins much easier:

If you work with web and load tests you'll find these very handy, and they are xcopy-installed on your machine (you'll have them working in less than 5 minutes).

Now to get these in our next release. :)

Ed.

So you want to automate your test cases?

Many CIOs and Test Managers are interested in increasing the level of automation in testing efforts. What is automation good for? What is the best way to go about this? How much automation is enough? And who should do it, and when?

I've heard on more than one occasion test managers say that 100% of all test cases should be automated. Really??? The first thing to assess is what are your goals for automation. Automation is about running the same tests over and over again in an efficient way, which can achieve two purposes:

1. Regression testing. This about finding bugs in functionality that at one time was working.
2. Matrix or configuration testing. This is about running the same test against the same build, but in different configurations.

Regression Testing

Regression testing comes in two flavors, depending on the phase of the development cycle you are in. The first is BVT or smoke tests, which determine whether or not the product is "off the floor" or has basic functionality working. For example, if you are doing regular builds as hand offs from dev to test, you want to make sure the product is "self-test", or basically working so the testers can use it. If, for example, login fails with no workaround, the test will be blocked, so there's no point promoting the build to test. BVTs are a small number of tests that validate the basic functionality of the software.

Another flavor of regression testing is at project shutdown time. Here you are stabilizing new features preparing to deploy the software to production or ship it to customers. In the process of building new features, did the dev team break any existing features? Automation can help find these types of failures.

A third important scenario is software maintenance. Say you have a hot bug fix you want to roll into production. How can you lower the risk of the bug fix breaking other features? Or if you are taking a security patch for the OS, how can you be sure your critical applications will continue to work? Just like the project shutdown scenario, automation can help here too.

So if you are doing automation for regression testing, a key question is how much testing needs to be done before you are ready to ship? Look at automating those parts. I always recommend crawl, walk, run. Before investing a ton of effort into a given strategy, be sure you understand the downstream costs. I recommend starting with a few key scenarios and running them through a lifecycle to get a feel for costs.

Config Testing

Another use of automation is if you have to run the same set of tests against many different configurations. This is most often the case for ISVs shipping software, where their software may be run in a variety of environments. For example, web and load tests are available in both VSTT and VSTS, both of which are available in 8 languages and on XP, Vista, Windows 2003, Windows 2008. And of course the OS comes in a variety of languages too, then there are other configs like running as Admin vs. normal user, SQL 2005 and 2008, etc. You can see the combinatorics are crazy here, each dimension is a multiplier. So we are smart about making sure we cover each in at least one test run, but you can see we have to run the same set of tests many, many times on different configs.

If you have a browser app, another example may be testing it with different browsers (although here you're likely interested in page layout type bugs, which are difficult to detect in automation).

Automation is no replacement for manual testing

Fundamentally automation is about running tests that used to work, or worked on one config, and then no longer work when re-run (except in the case of TDD of course, that's another topic :)).

You will definitely get the highest bug yields from manual testing, also manual testing allows testers to be creative in finding bugs. Automation is not creative at all, it typically just does the same thing over and over.

A typical cycle for testing is to make sure all features are tested manually first, then select which test cases to automate.

The Hidden Cost of Automation: Maintenance

In order to reliably determine regressions, your tests must be reliable and working. As the application changes, you will need to fix your tests to match the application. If you don't do this your automation will turn into a smolder pile that is really good for nothing. Take this example, let's say you've automated 100 tests for a given release. Then you do a second release, and as you go into feature complete and the stabilization phase you go to re-run your tests and find that 75 of them are now failing.  As you investigate, you find that the UI changed, which caused your scripts to break. So now you are left with fixing your automation up, which for UI tests is really no easy task and often it takes longer to fix an automated test case than it did to develop it in the first place.

If have seen on more than one occasion where the effort to maintain a set of tests outweighed the effort to run the test cases manually. I've also seen it where the test team is so wrapped up in building and maintaining automation that actual manual testing falls by the wayside, and as a result bugs make it out to customers or are found late in the cycle.

I can't say this enough, the downstream cost of automating a test is huge, particularly if automating at the UI layer. UI tests have the advantage that they are easy to create and get started on, and easy to understand, but the disadvantage that they are significantly less reliable than unit (or I prefer the term API) tests, and more prone to break due to changes in the application.  API tests have the huge advantage of failing with compile-time errors rather than run-time errors, and are easier to maintain.

Also where as unit tests are written by developers as they complete new features, you are likely best off holding off on automating at the UI layer until later in the cycle once the changes in the UI settle out.

I love James Bach's classic paper Test Automation Snake Oil. It's almost 10 years old, but still applicable today.

Best Practices for Automation

1. Keep the UI layer as thin as possible, and do as little automation at the UI layer as possible. Architect your application to allow testing at different layers via APIs, rather than driving all automation through the UI. This is referred to as proper architectural layering, where you have defined clear architectural layers in the software. Tests below the UI layer are developed as unit tests, so they have the advantages of lower cost of maintenance and less fragility.
   Here's the architecture of the test case management product we are developing, which enables this. We're currently focusing our test case development at the client API layer and action API layer, with a few UI tests for end-to-end testing.
   
   
   
2. Ensure developers and testers work together.
1. Developers should develop unit tests for the layers they are developing as they go. Testers should leverage developer unit tests for config testing. For example, if your application supports different server configurations, and you have dev unit tests that test the server, testers should run these tests against the different supported configs.
2. Avoid duplication of tests. If developers have a rich set of business object unit tests to test the business logic, all of these tests likely does not need to be repeated at the UI layer. For the products we are developing, our test team develops the test plan and test cases, we tag which ones we want to automate, and both developers and testers work off this list.

Getting started with VS 2008

You can start now with automation using VS 2008 unit tests to target the layers in your software, and web tests to test your server at the http layer. Web tests largely fall into the same boat as UI automation tests for maintenance and development cost. They were primarily developed for load testing, but can also be used for functional testing of the server. Since they do not drive the application at the UI layer, but rather the http layer, they are harder to work with than UI tests.

You can also automate your builds using TeamBuild, and configure BVT unit tests to run as part of the build to ensure the build quality. Test results are published to TFS, so the entire team can see the results of the tests and quality of the build.

There are several different test partners offering UI automation solutions for VS, including Automated QA and ArtOfTest.

We are also working on UI automation technology for our next release, as well as TFS server-based test case management and a manual test runner.

Ed.

SharePoint Server 2007 Scalability and Performance whitepaper available

The SharePoint team has published this awesome SharePoint Server 2007 Scalability and Performance whitepaper where they tested loading up SharePoint with 50 million documents. It outlines the architecture and testing methods they used to conduct the study.

Note that the team used VSTS web and load tests to load test the site.

Ed.

VS 2008 SP1 Released

SP1 has shipped. As I blogged about before here and here, there is a lot of good stuff in SP1 for TeamTest. The English downloads are available here. Note, if you are using a different language VS, the patch is available in all languages too.

I have to note, we found a serious regression in load tests last week that was introduced in SP1, but it was too late to fix it in SP1. The regression is that think times are not applied to requests that redirect.

If you are doing web load testing, this problem will almost certainly impact the pace at which your vUsers run. Bill has documented a web test plugin workaround for this problem, and we will be issuing a QFE for the fix.

Ed.

dynaTrace is cool!

dynaTrace just completed a nice integration with VSTS web, unit and load tests that enables them to do end-to-end tracing of performance. In addition to getting overall performance numbers, they have a feature called PurePath that let's you trace, down to a given web test request, what calls that request results in on the mid-tier and data-tier. Being the performance geek that I am, I love this stuff!

And the setup and deployment are a snap. This week Andreas Grabner, a perf architect for dynaTrace, came and helped us get it running to diagnose some TFS server performance problems we are working on. We also brainstormed on some more integration points in web and load test that will enable an even tighter integration in our next release.

Check out the two minute video when you have two minutes: http://www.dynatrace.com/twominutedemo/default.aspx

Ed.

Masking a 404 error in a dependent request

There are times when your web test may fail with a 404 error in a dependent request (like a css file or gif), but you may not actually care about this error. For example, it may be something that doesn't really impact the layout or content of the page. Or you might file a bug and temporarily want to suppress the error until it is fixed so they can find other errors.

If you get a 404 error in a dependent resource, here's how you can mask it.

Here's a failing web test, it is very simple. If I expand the top request I can see it is failing due to a failed dependent request.

To mask the request, right-click on the dependent and hit Copy.

Go to the test, right click on the corresponding request in the test and select "Add Dependent Request". Paste the url on the clipboard into the url property for the dependent request:

Now set the expected http status code on the dependent to 404.

Rerun the test to see it pass.

Hope that helps,

Ed.

How to upload a file in a Web test

From this post, I see many users are having trouble with file uploads.

Really this is not as easy as it should be.

To upload a file, first record the file upload. I have a simple site that I did that on to generate this web test.

The key parameter here is obviously the file upload parameter, which recorded the file as "MyImage.gif".

At runtime, the web test engine will look for a file with the name MyImage.gif in the test deployment directory.

If I just go ahead and run this the test fails with:

Request failed: Could not find file 'C:\Users\edglas.000\Documents\Visual Studio 2008\Projects\TestProject1\TestResults\edglas_EDGLAS-LT2 2008-08-05 08_57_58\Out\MyImage.gif'.

Of course the first thing you'll notice is the file path was not recorded in the web test. So how will this file be found? and why is it looking in that directory?

When you run tests in VS, the files required to run the tests are "deployed" to a directory for the test run (see my post on deployment), in this case it is edglas_EDGLAS-LT2 2008-08-05 08_57_58. You'll get a different directory every time you run your tests. Notice this directory is available in the web test context:

The best way to handle this in your test is to actually add the file to be uploaded to your test project, then add it as a deployment item on your test. That way it will be copied to the out directory prior to running the test, and will be available during execution. This also has the advantage that if you are working with others, the test will also run on their machine (hard-coded full paths in a test are bad, as these tests will fail on another person's machine if their drive isn't set up the same way).

Once you've added the file to your project, add it as a deployment item. There's two ways to do this, on the run config or on the test. Since this file is really associated with the test, I recommend putting it on the test. This is not discoverable at all. First, open test view (Test menu | Windows | Test View) and select the web test. Then set the Deployment Items property for the web test's test element.

Now add MyImage.gif as a deployment item by clicking ... next to the deployment items property. Since this is a property on the web test, the path is relative to the web test:

Now my test runs successfully:

Another approach is to create a folder in your project where you put all your file upload files.

Then specify the relative path in the deployment item properties (relative to the web test in the project). So if my file to upload is in the FileUpload folder

Now in deployment items specify the relative path, which again is relative to the web test. Note the path in the file upload web test parameter is not relative since it will be published "flat" with no subdirectories (no changes required):

Now the test runs successfully again.

Another option is to deploy the files or entire directory using the run config settings.

For the run config, go to the Deployment tab and use the Add Directory to add your folder with files to upload. Note that this path is solution relative, since the run config is in the solution directory. The <Solution Directory> macro is automatically inserted after you select the file or directory.

Now if I go to the deployment dir using Explorer (easiest way to find it is from the web test context parameter), I see that both of my images were deployed, and my test still runs successfully. Also any new files I want to upload I can just drop them in this folder in my solution and don't have to add it as a deployment item to my test. Note that all files are published "flat", which means you can't have two different deployment files in different folders with the same name.

Hope this helps,

Ed.

How to set a friendly name on an Extraction Rule

In VS2008, we changed the way custom Extraction rules, Validation rules, and web and load test Plugins return a friendly name and description.

In 2005, these were implemented as override on the abstract base class. The problem with this was that it required us to instantiate the class at design time, which we really didn't want to do for a number of reasons.

So we moved to a pretty standard (in .NET) way of doing this via attributes on the class. But the attribute names are not in the web test or load test namespace, so they are not discoverable, and our samples online are wrong (I added some wiki comments). Even the warning we give at compile time if you compile the old sample or an old rule isn't enough to figure out what attribute to use:

Warning: Member 'MyRules.MyValidationRule.RuleName' overrides obsolete member 'Microsoft.VisualStudio.TestTools.WebTesting.ValidationRule.RuleName'. Add the Obsolete attribute to 'WebAndLoadTestPluginsAndRules.CompareContextParams.RuleName'.

To set the name and description, simply add these to your class:

[System.ComponentModel.DisplayName("My Friendly Rule Name")]
[System.ComponentModel.Description("My rule description")]
public class MyValidationRule : ValidationRule

Ed.

WCF Load Test Tool is hot on codeplex

Check out the stats for the WCF load test tool on codeplex. The tool is approaching 2000 downloads! Kudos again to Rob Jarratt, who wrote this tool. It's great that we have a platform with the right extensibility hooks that enables us to produce out band solutions to key problems like this one.

I had the chance to actually meet Rob face to face and go to dinner with him last night. Rob is from the UK, so we'd never actually met! What a connected world we live in. We were both spending the day at TechReady, our Microsoft-internal technical field sales conference in Seattle (support, consultants, and technical sales come from around the world to exchange info and learn more about our products).

Ed.

Updated Links to Info

I updated my Links to Info on Web and Load Tests post with new links. The last update was in March, so it was due. I also moved some links into a separate table that are specific to 2005. Some of the 2005 context still applies to 2008, so I left that in the main table.

Most of the links are to new content my, Bill's, and Sean's blogs. Here's the new stuff:

New Web Test APIs for page boundaries and error handling

New Load Test APIs for load modeling

Support for IE8 and FireFox3 playback emulation

Support for random extraction of text

Description of web test execution

New rules for Inner Text parsing, list, and combo box

Disabling caching of all dependent requests

Collecting Code Coverage when Running Web and Load Tests

Sharing Test Results

How to create a custom counter set

SQL Record/Playback for Visual Studio Load Tests

Ed.