Welcome to MSDN Blogs Sign in | Join | Help

For those of you dreaming the 100% automation dream...please wake up!

My respected colleague Keith Stobie recently posted to his blog my response an internal query regarding “unrealistically high” goals for the percentage of tests that are automated. (Since, I have been remiss in my public rants I will reuse Keith’s post. Thanks Keith.) Of course, the unrealistically high percentage is probably near the 100% mark. There has been a lot of talk recently about 100% automation goals especially in the Agile world. So, for all those out there that think 100% test automation is a “reasonable” or even achievable goal I have a suggestion. Learn to just say NO to drugs!

There are some areas of testing where I do see the need and value of approximately 100% of the tests being automated. BVT/BAT is a prime example. When I was the Setup Test Lead on Windows 95 I was responsible for developing the BVT/BAT suite. The suite contained approximately 1000 unique functional tests and ran simultaneously on 4 different language versions within the span of 30 minutes once a week for about 1½  years upon delivery of each new build. But it was only maintenance free for the last 6 months of the product cycle. Still, that’s a pretty good return on investment considering Ed Kit, and James Hancock have estimated that an automated test must run 15-17 times to break even on the cost of developing that test. Other areas which lend themselves to high percentages of automated tests include performance testing, stress testing, regression testing, etc. Areas of testing in which I need to establish baselines and run consistent tests in order to compare repeatedly predictable results against those baselines towards some ultimate predefined goal are prime candidates for high levels of automation. Scripted test automation does a very poor job of exposing "new" defects post test design, and GUI automation requires enormous maintenance especially if it is created too early in the product cycle when the U/I is still in transition. The bottom line is we need to be smart about what and when to automate, because good reusable test automation is damn expensive.

For my unabridged response to managers or others who throw around the 100% automation mantra see my comments  in Keith’s blog. For more information about making smart decisions of what and when to automate I highly recommend the book by Daniel J.Mosley and Bruce A. Posey entitled Just Enough Software Test Automation.

 

Published Wednesday, August 02, 2006 3:35 AM by I.M.Testy
Filed under:

Comments

# re: For those of you dreaming the 100% automation dream...please wake up!

"Ed Kit and James Hancock have estimated that an automated test must run 15-17 times to break even on the cost of developing that test."

This is a fascinating example of testing mythodology and bizarre accounting.  I particularly love the bogus precision of "15-17".

Kit, Hancock, and many others seem to assume that the value of automating a test should be calculated as a function of the number of times that it is run.  Wouldn't it be the case that an automated test that finds an important bug on the first run has a very high value?  

A good automated test extends human capabilities beyond their usual reach.  Does an automated test that stresses the system by simulating 10,000 users need to run 15 (or 17) times before it provides value?

A good automated (unit) test detects unwelcome changes, and thus returns some amount of value every time it is run.  Does it cross some magic threshold at 15 (or 17) times?

There are at least five value factors and five cost factors for every test, whether manual or automated.  Most of these involve some degree of uncertainty, so any attempt to calculate a precise value will return some number times uncertainty to some exponent.

In terms of your argument about 100% automation, I agree.  My proposal is that, if we're going to automate 100% of testing, why not go the whole hog and automate 100% of the programming?  It's a much better return on investment, because programmers outnumber testers, and the programmers typically get paid more.  So what if it would be hard to do--we should strive for it, shouldn't we?  <-satire

---Michael B.
Thursday, August 31, 2006 5:00 AM by Michael Bolton

# re: For those of you dreaming the 100% automation dream...please wake up!

Michael, you are confusing value and cost.

Kit and Hancock are specifically talking about the overall cost of GUI automated testing. They do not broach the subject of test value or effectiveness.

I assumed whomever might retort the conclusions that Kit and Hancock derived from their studies would be familiar with their work. (I also assumed the reader wouldn't confuse the terms 'cost' and 'value.')

So, perhaps instead of wild speculation you can share with us your reference of a detailed study or other empirical evidence that offers a counter-argument to Kit and Hancock's findings.)

A load test simulating 10000 concurrent users should be automated because its overall value is very high and it would be virtually impossible to perform manually. But, the cost of developing such a test is also high.

But a functional test that I might run only once (or once per milestone) may not be a good candidate for automating because the cost of developing that test might exceed a reasonable return on investment. (Oh, by the way, the majority of bugs are found during the design/development phase of GUI test automation; not during the execution of GUI test automation. Some studies demonstrate GUI test automation exposes between 5 - 15% of all defects discovered during a development lifecycle.)

My point in using the information presented by Kit and Hancock is to get professional testers to think about automation and make smart business decsions about what GUI tests to automate in order to derive the greatest value for their effort.

- Bj -

Friday, September 01, 2006 5:35 PM by I.M.Testy
Anonymous comments are disabled
 
Page view tracker