Ed Glas's blog on VSTS load testing : Guidance

Increasing the ROI of our Automation

edglas — Sat, 13 Jun 2009 02:55:00 GMT

A while back I posted this on automation. I want to resurface this topic now, as it has been on my mind a lot lately as we develop dev10. What is the “right” approach for automation? How do we make it pay off?

Automation is hugely important for us. It enables us to ensure our software works on a myriad of OS (XP, Vista, Win7, Server 2003, Server 2008), processor architecture (x86, ia64), and language (English, German, Japanese, Chinese, etc.) combinations.

At the end of the day, return on investment is what it is all about. Is it faster to develop and maintain an automated test case, or simply run the tests manually? If over the lifetime of a test case you spend more time fixing it than it would have taken to run the test manually for each time you ran the automated test, that test has not paid off. This is basic and obvious, but often times get overlooked when considering automation.

Going back a bit in history to the VS 2005 release when we initially introduced load testing, we made a few huge mistakes with our approach to automation, mistakes which I see others repeating over and over:

My test team wrote automation exclusively using UI automation.
My test team owned developing and maintaining the automation.
Dev “unit tests” were not considered “automation”.
Devs and testers used different test automation frameworks.

With dev10, we have reversed each of these, and are benefitting greatly from it.

One thing I really want to emphasize is the cost of maintaining our UI automation. I have not looked at the exact number, but in the course of developing dev10 nearly every single UI automation test case has broken. Contrast that with our tests that test at the API layer, I estimate that fewer than 10% have broken. To add insult to injury, the UI tests are hugely expensive to fix. API tests that break nearly always give compile errors, which are easy to fix. UI tests do not fail until you run them, and then you have to grovel through the test log to figure out what UI controls have changed to cause the failure, make a fix, rerun the test (perhaps to find another failure), all of which is extremely time consuming.

Increasing our ROI: A Cost-Based Approach to Automation

For dev10, we have developed four methodologies for automation, and consider each when considering how to automate testing of particular functionality. We give preference to the lowest cost approach. That is, if we can use a cheaper approach to automate, we do it that way and we don’t automate everything through the UI. In fact, the UI is the least favored approach, and is only done if none of the other approaches will give us the coverage we need. The approaches, in order of preference, are:

Unit tests that run in the test process. Really “unit test” is a misnomer for most of these tests, as they typically test a broader area than a pure unit test. I prefer to call these tests API tests. The defining characteristics of these tests is that they are hosted by the test process.

An example of this for our load test product is we have tests that run requests through our web test engine.

Another example is in our TCM server product. Nearly all of the automation for TCM server has been developed using this approach.
Command line tests. With this approach we have “unit tests” that spawn our command line. The test code then waits for the command to complete, and then uses APIs to validate the command did the right thing. We use this approach for both web and load test features and for our TCM server.

For example, for load tests we have tests that run a load test from the command line, then access the load test database to ensure the results were as expected. This gives us substantial coverage of our web and load test feature set.
API tests that run in the process under test. mstest has a feature that enables processes other than vstesthost.exe to host the test code. An example of this is ASP.NET unit tests, which run in the ASP.NET process. We take advantage of this extensibility point to add our client processes as host processes.

An example of this is we run our new TCM test client Camano (aka Microsoft Test and Lab Manager) as a test host. This is a bit of a mind bender, but we also actually run VS as a test host! So devenv starts vstesthost which starts a second copy of devenv. Once VS is loaded, our test code runs and has access to our architectural layer “just beneath the glass” and enables us to run web and load tests in the IDE (e.g. open a project, open and edit load tests in the project, run them, validate the UI through code).
UI Automation. With this approach, our test code instantiates the client under test, and drives it through the mouse and keyboard (we drive VS through MSAA, and we drive our TCM client using UIA).

The cost of UI automation is at least an order of magnitude greater than the other three approaches. Isn’t it ironic that we are introducing a new UI automation test type in dev10. If UI tests are so bad, why are we bringing them to market? While I maintain that they are very expensive, UI tests are also absolutely necessary. There are some things we simply cannot test without them (like dialogs, or some integrated, cross-process end-to-end scenarios). But if I go back and look at the UI tests we developed for web and load tests in VS 2005, the vast majority of them could have been implemented using a cheaper alternative.

Engineering for Automation: Developing Testable Software

Developing these other automation approaches did not come for free. We have deliberately architected our software to enable each of these approaches, and spent time developing test frameworks and reference patterns for each approach. It has really paid off in a big way. If you look at our TCM tests, we maintain an extremely high pass rate (98% plus). Our “legacy” UI tests from the 2005 release are in bad shape.

Whose job is it anyway?

At the end of the day, it is the test team’s job to help us understand whether or not we are ready to ship. Automation plays a huge role in that, since we are able to run our automation suites unattended and generate test results relatively quickly and concurrently on many different configurations. The test team is responsible for ensuring we have the right set of automation in place help us understand where we are at.

Getting Testers and Developers on the Same Page

However, it is not solely the test team’s job to develop and maintain the automation. The test team develops the test plan, which includes the scenarios and functionality we want to automate, but both the dev and test team develop and maintain the automation. If a developer is writing “unit tests” for a new API they are developing, they work off the test plan to develop one of the specified test cases. Tester do not duplicate this effort by developing “their own” automation to cover. A key tenet is that testers and developers use the same frameworks and methodologies for developing automation. Testers are able to run tests developed by devs, and visa-versa. Going back to the mistakes we made in VS 2005, this simple methodology, along with considering the most cost-effective way to automate a given test case, is the key to fixing each of these mistakes.

Similarly, it is the entire team’s job to maintain the automated tests. As I said earlier, maintenance is the hidden and huge cost of doing automation. Devs must fix any compile errors in tests introduced by their changes, and also must ensure that all P1 tests pass before checking in. Testers and devs share the load on ensuring all P2 and P3 tests maintain a high pass rate. We run these once a week and distribute the load on driving them to passing.

I hope you can also learn from our mistakes, and use our experience to improve your approach to automation.

Ed.

So you want to automate your test cases?

edglas — Fri, 15 Aug 2008 15:35:00 GMT

Many CIOs and Test Managers are interested in increasing the level of automation in testing efforts. What is automation good for? What is the best way to go about this? How much automation is enough? And who should do it, and when?

I've heard on more than one occasion test managers say that 100% of all test cases should be automated. Really??? The first thing to assess is what are your goals for automation. Automation is about running the same tests over and over again in an efficient way, which can achieve two purposes:

Regression testing. This about finding bugs in functionality that at one time was working.
Matrix or configuration testing. This is about running the same test against the same build, but in different configurations.

Regression Testing

Regression testing comes in two flavors, depending on the phase of the development cycle you are in. The first is BVT or smoke tests, which determine whether or not the product is "off the floor" or has basic functionality working. For example, if you are doing regular builds as hand offs from dev to test, you want to make sure the product is "self-test", or basically working so the testers can use it. If, for example, login fails with no workaround, the test will be blocked, so there's no point promoting the build to test. BVTs are a small number of tests that validate the basic functionality of the software.

Another flavor of regression testing is at project shutdown time. Here you are stabilizing new features preparing to deploy the software to production or ship it to customers. In the process of building new features, did the dev team break any existing features? Automation can help find these types of failures.

A third important scenario is software maintenance. Say you have a hot bug fix you want to roll into production. How can you lower the risk of the bug fix breaking other features? Or if you are taking a security patch for the OS, how can you be sure your critical applications will continue to work? Just like the project shutdown scenario, automation can help here too.

So if you are doing automation for regression testing, a key question is how much testing needs to be done before you are ready to ship? Look at automating those parts. I always recommend crawl, walk, run. Before investing a ton of effort into a given strategy, be sure you understand the downstream costs. I recommend starting with a few key scenarios and running them through a lifecycle to get a feel for costs.

Config Testing

Another use of automation is if you have to run the same set of tests against many different configurations. This is most often the case for ISVs shipping software, where their software may be run in a variety of environments. For example, web and load tests are available in both VSTT and VSTS, both of which are available in 8 languages and on XP, Vista, Windows 2003, Windows 2008. And of course the OS comes in a variety of languages too, then there are other configs like running as Admin vs. normal user, SQL 2005 and 2008, etc. You can see the combinatorics are crazy here, each dimension is a multiplier. So we are smart about making sure we cover each in at least one test run, but you can see we have to run the same set of tests many, many times on different configs.

If you have a browser app, another example may be testing it with different browsers (although here you're likely interested in page layout type bugs, which are difficult to detect in automation).

Automation is no replacement for manual testing

Fundamentally automation is about running tests that used to work, or worked on one config, and then no longer work when re-run (except in the case of TDD of course, that's another topic :)).

You will definitely get the highest bug yields from manual testing, also manual testing allows testers to be creative in finding bugs. Automation is not creative at all, it typically just does the same thing over and over.

A typical cycle for testing is to make sure all features are tested manually first, then select which test cases to automate.

The Hidden Cost of Automation: Maintenance

In order to reliably determine regressions, your tests must be reliable and working. As the application changes, you will need to fix your tests to match the application. If you don't do this your automation will turn into a smolder pile that is really good for nothing. Take this example, let's say you've automated 100 tests for a given release. Then you do a second release, and as you go into feature complete and the stabilization phase you go to re-run your tests and find that 75 of them are now failing. As you investigate, you find that the UI changed, which caused your scripts to break. So now you are left with fixing your automation up, which for UI tests is really no easy task and often it takes longer to fix an automated test case than it did to develop it in the first place.

If have seen on more than one occasion where the effort to maintain a set of tests outweighed the effort to run the test cases manually. I've also seen it where the test team is so wrapped up in building and maintaining automation that actual manual testing falls by the wayside, and as a result bugs make it out to customers or are found late in the cycle.

I can't say this enough, the downstream cost of automating a test is huge, particularly if automating at the UI layer. UI tests have the advantage that they are easy to create and get started on, and easy to understand, but the disadvantage that they are significantly less reliable than unit (or I prefer the term API) tests, and more prone to break due to changes in the application. API tests have the huge advantage of failing with compile-time errors rather than run-time errors, and are easier to maintain.

Also where as unit tests are written by developers as they complete new features, you are likely best off holding off on automating at the UI layer until later in the cycle once the changes in the UI settle out.

I love James Bach's classic paper Test Automation Snake Oil. It's almost 10 years old, but still applicable today.

Best Practices for Automation

Keep the UI layer as thin as possible, and do as little automation at the UI layer as possible. Architect your application to allow testing at different layers via APIs, rather than driving all automation through the UI. This is referred to as proper architectural layering, where you have defined clear architectural layers in the software. Tests below the UI layer are developed as unit tests, so they have the advantages of lower cost of maintenance and less fragility.
Here's the architecture of the test case management product we are developing, which enables this. We're currently focusing our test case development at the client API layer and action API layer, with a few UI tests for end-to-end testing.
Ensure developers and testers work together.

Developers should develop unit tests for the layers they are developing as they go. Testers should leverage developer unit tests for config testing. For example, if your application supports different server configurations, and you have dev unit tests that test the server, testers should run these tests against the different supported configs.
Avoid duplication of tests. If developers have a rich set of business object unit tests to test the business logic, all of these tests likely does not need to be repeated at the UI layer. For the products we are developing, our test team develops the test plan and test cases, we tag which ones we want to automate, and both developers and testers work off this list.

Getting started with VS 2008

You can start now with automation using VS 2008 unit tests to target the layers in your software, and web tests to test your server at the http layer. Web tests largely fall into the same boat as UI automation tests for maintenance and development cost. They were primarily developed for load testing, but can also be used for functional testing of the server. Since they do not drive the application at the UI layer, but rather the http layer, they are harder to work with than UI tests.

You can also automate your builds using TeamBuild, and configure BVT unit tests to run as part of the build to ensure the build quality. Test results are published to TFS, so the entire team can see the results of the tests and quality of the build.

There are several different test partners offering UI automation solutions for VS, including Automated QA and ArtOfTest.

We are also working on UI automation technology for our next release, as well as TFS server-based test case management and a manual test runner.

Ed.

Updated Content for web tests and load tests

edglas — Mon, 17 Dec 2007 20:36:00 GMT

I've updated my collection of links for info on web tests and load tests.

I had been publishing this to a sticky post on the forum, but ran into the size limit so I'm moving it to my blog.

The page is here. Do let me know if you hit any links that are broken, the content gets moved around from time to time.

Ed.

Performance testing guidance

edglas — Sun, 18 Feb 2007 00:38:00 GMT

The patterns and practices team, led by J.D. Meier and including Larry Brader and Carlos Farre, has teamed with a bunch of expert load testers as well as industry expert Scott Barber, to create best practices guidance for load testing. The guidance is hosted on codeplex at http://www.codeplex.com/PerfTesting.

I had a great meeting with JD last week when I was in Redmond last week. I love the way he thinks, looking for underlying patterns in things that are timeless and independent of other factors (like type of target app and tooling to drive the tests). One of the goals of the performance testing guidance is to distill out this timeless information into nuggets that will stand the test of time. Then tie that together with additional guidance on things like tooling and target apps (these things that are more temporal), and to give guidance on how to put the perf testing into practice with a given toolset against a given target app.

Ed.