For those of you dreaming the 100% automation dream...please wake up!

re: For those of you dreaming the 100% automation dream...please wake up!

Michael Bolton — Thu, 31 Aug 2006 12:00:13 GMT

"Ed Kit and James Hancock have estimated that an automated test must run 15-17 times to break even on the cost of developing that test."

This is a fascinating example of testing mythodology and bizarre accounting. I particularly love the bogus precision of "15-17".

Kit, Hancock, and many others seem to assume that the value of automating a test should be calculated as a function of the number of times that it is run. Wouldn't it be the case that an automated test that finds an important bug on the first run has a very high value?

A good automated test extends human capabilities beyond their usual reach. Does an automated test that stresses the system by simulating 10,000 users need to run 15 (or 17) times before it provides value?

A good automated (unit) test detects unwelcome changes, and thus returns some amount of value every time it is run. Does it cross some magic threshold at 15 (or 17) times?

There are at least five value factors and five cost factors for every test, whether manual or automated. Most of these involve some degree of uncertainty, so any attempt to calculate a precise value will return some number times uncertainty to some exponent.

In terms of your argument about 100% automation, I agree. My proposal is that, if we're going to automate 100% of testing, why not go the whole hog and automate 100% of the programming? It's a much better return on investment, because programmers outnumber testers, and the programmers typically get paid more. So what if it would be hard to do--we should strive for it, shouldn't we? <-satire

---Michael B.

re: For those of you dreaming the 100% automation dream...please wake up!

I.M.Testy — Sat, 02 Sep 2006 00:35:05 GMT

Michael, you are confusing value and cost.

Kit and Hancock are specifically talking about the overall cost of GUI automated testing. They do not broach the subject of test value or effectiveness.

I assumed whomever might retort the conclusions that Kit and Hancock derived from their studies would be familiar with their work. (I also assumed the reader wouldn't confuse the terms 'cost' and 'value.')

So, perhaps instead of wild speculation you can share with us your reference of a detailed study or other empirical evidence that offers a counter-argument to Kit and Hancock's findings.)

A load test simulating 10000 concurrent users should be automated because its overall value is very high and it would be virtually impossible to perform manually. But, the cost of developing such a test is also high.

But a functional test that I might run only once (or once per milestone) may not be a good candidate for automating because the cost of developing that test might exceed a reasonable return on investment. (Oh, by the way, the majority of bugs are found during the design/development phase of GUI test automation; not during the execution of GUI test automation. Some studies demonstrate GUI test automation exposes between 5 - 15% of all defects discovered during a development lifecycle.)

My point in using the information presented by Kit and Hancock is to get professional testers to think about automation and make smart business decsions about what GUI tests to automate in order to derive the greatest value for their effort.

- Bj -