Search Form
Syndication
Administrivia

The Software Tester's Axioms

Sorting it all Out
Michael Kaplan's random stuff of dubious value
Be sure to read the disclaimer here first!
MSDN Blogs > Sorting it all Out > The Software Tester's Axioms

The Software Tester's Axioms

12 Nov 2005 4:01 AM
  • Comments 26

Like any science (and please do not for a moment doubt that Test is an engineering discipline, and at heart a good engineer is an applied scientist!), software testing has some basic principles which cannot be proven but are simply universal truths upon which other theorems can be based.

(The original version of these was put up years ago by me in a newsgroup, but after many reformulations, the current ones have stayed stable for some time. There have been other corollaries and theorems proposed but I have not captured all of them here)

Here are the Tester's axioms:

Axiom I: If you have not tested it, assume it is broken.

Axiom II: If someone changes it, they probably broke it.

Axiom III: Sometimes when it is not changed, it breaks too.

Axiom IV: What is broken on the test machine often works fine on the dev machine.

Axiom V:There is always another bug that a user can find.

And there is of course the first theorem of test, which is largely based on applying on the developer's doubt of Axioms 2, 3, and 5 created by the circumstances of Axiom 4:

THEOREM I: Developers will often not believe there is a bug until it is proven to them, sometimes with extreme prejudice.

And of course the corollary to Axiom V:

There are at least as many bugs left to be found as there are users who haven't yet run your code.

 

(Special thanks to David Fenton for pointing out the corollary to Axiom V, years ago!)

Comment on the blather
Leave a Comment
  • Please add 1 and 7 and type the answer here:
  • Post
Blog - Comment List
Comments
  • 12 Nov 2005 5:50 AM
    Theorem II: Developers believe (and sometimes even assume) that bugs "fix themselves" between builds
  • Jonathan Wilson
    12 Nov 2005 6:36 AM
    Theorem III: No developer can ever fix all the bugs in their own code on their own. Having one other experienced person go over your code once is significantly better than you reading your code many times over looking for bugs.
  • 12 Nov 2005 9:30 AM
    Hi Maurits --

    Ok, I agree with the principle of Theorem II -- though it may be a corollary of Axioms II, III, and IV? :-)

    But then it does also depend on Theorem I, so perhaps it is a theorem.....
  • 12 Nov 2005 9:32 AM
    Hi Jonathan --

    I agree with the statement, but I am not sure we can call it a theorem of Test without working on the specific language a bit (as currently worded it seems more like a dev-focused item).
  • Serge Wautier
    12 Nov 2005 9:52 AM
    The more a bug is difficult to identify, the more valuable it is. Because the more other unknown bugs it will help uncover during its debugging.

    [Speaking of bug, can somebody pleeease help the 'Remember me' checkbox actually remember me...]
  • 12 Nov 2005 10:52 AM
    I don't like the third axiom at all.

    If something breaks and nothing changed, you just don't KNOW what changed.
  • Marvin
    12 Nov 2005 1:14 PM
    Theorem VI: The more bugs you have found in a module, the more bugs there are left undetected.

    Corrolary: If you have two modules: one that had a few bugs in the past and another that had a lot, spend more time testing the second one.
  • 12 Nov 2005 1:29 PM
    Hi Nick -- the implication of the third axiom (which perhaps should be recast to include include this info) is that sometimes external changes cause a break -- so even if the code is not changed, something can break it.
  • 12 Nov 2005 1:30 PM
    Marvin, I love it! And I get your Theorem IV (we aee only up theorem IV so far!) pointed out by example all the time. :-)
  • 12 Nov 2005 2:21 PM
    Michael- Yeah, that's totally right. I'm just being pedagogical. A lot of software bugs get papered over instead of fixed because programmers don't think about what's going on outside their code.
  • 12 Nov 2005 4:39 PM
    I think your point is a good one though. How about changing from this:

    "Sometimes when it is not changed, it breaks too."

    to this:

    "Sometimes when it is not changed, external factors can break it, anyway."
  • Marvin
    12 Nov 2005 9:30 PM
    It was supposed to be Axiom VI. No idea why I typed theorem ;-)

    Seriously this fact is supported by statistics and is often mentioned in books about software testing (I once read a few for general education). I also had many chances to see how true it is.
  • 12 Nov 2005 9:40 PM
    Ah, that may take more thought (it is not that I disagree, it is that we have to be cautious about adding new axioms to a system!).
  • Jerry Pisk
    13 Nov 2005 3:27 AM
    I don't think the third axiom talks about external factors. I think it simply points out the fact that even code that did not change has bugs that you didn't find earlier. An external factor might have exposed the bug instead of causing it.
  • Sorting It All Out
    23 Feb 2006 3:54 AM
    Well, at the very least keep MSLU apps out of the VMWare shared folders? :-)
    The other day, Brian asked...
Page 1 of 2 (26 items) 12
MSDN Blogs > Sorting it all Out > The Software Tester's Axioms