Musings on Software Testing

re: Musings on Software Testing

kevinpilch-bisson — Sat, 08 Dec 2007 00:31:14 GMT

Hey Wes,

Great post! I'm always interested in learning better ways to test and write software. I think that you've misunderstood the purpose of TDD though. In TDD, the tests you write are _definitely not_ supposed to be the exhaustive set of tests, and in fact, their primary purpose is not to "test" the software. The primary purpose is to drive the _design_ or architecture of the software, based on having real consumers for the APIs that you are writing.

For this reason, many people have stopped using the term TDD, and have started using "Emergent Design Development (EDD)" instead.

I'd be interested to hear your opinions on the efficacy of unit tests as a methodology for driving design though...

I've begun thinking of TDD as hill-climbing

Peter Thatcher — Sat, 08 Dec 2007 00:56:40 GMT

Lately, I've been begun viewing TDD as a hill-climbing search, or perhaps some other search that hits a local maximum. It seems that if your development is entirely driven by those tests, they can only take you so far, and you'll hit the maximum.

What takes talent is to look beyond the test and write something that not only fulfills it but does so in an elegant way. The elegant solution, once found, can guide us in new directions which we hadn't thought of before. This may lead us to add to or alter our tests. In this way, the search can break past the local maximum.

re: Musings on Software Testing

Nathan Stott — Sat, 08 Dec 2007 02:31:27 GMT

I agree with you 100%.

I believe that unit testing is essential, but that functional testing often devolves into an exercise in unbiased learning.

There is certain core functionality to any framework or application that takes a finite set of inputs. An example of that is an implementation of ActiveRecord. The models will have methods that take a quite finite set of inputs. Perfect.

However, functional testing involves testing that which is interacted with. Interaction is hard to predict. To properly test a complicated windows or web form would take hundreds of functional tests, and use cases for forms evolve rapidly and often.

I know you're a bit of a Haskell buff. To put it in Haskell terms, I don't think that testing monads is nearly as useful as testing functions, and too much effort put into the testing of monads is a waste of time.

No one will ever write a bug-free application and putting too much effort into that costs more money than it saves.

Great blog, Wes. I'm glad you're finally blogging again. Three in 2 days!

re: Musings on Software Testing

wesdyer — Sat, 08 Dec 2007 18:55:40 GMT

Kevin:

Good to hear from you. I am being pedantic on purpose. I am just taking what TDD is defined to be and following it to the letter, so I am following the letter but not the spirit of what it is saying. Btw, I have seen it practiced this way. Testing is extremely important to me and any of my team mates will vouch for me that I write tests as early and often as possible. I used a brain-dead extreme form of TDD to show how process can degrade into suboptimal forms of machine learning. Tests should inform the design but not determine it.

re: Musings on Software Testing

EGESA RONALD LEONARD — Thu, 20 Dec 2007 03:15:45 GMT

This article is nice as it brings forward the concept of TDD. Infact one other thing to stress is this idea that a bug that customers cannot find is not worth finding by test. True, i have held the same idea, but then remeber that a problem postponed is not a problem solved.

The bug may never be found by customers, but may catch up with the programmer at a later stage when program upgrades are necessary.

re: Musings on Software Testing

Evan — Fri, 21 Dec 2007 20:21:26 GMT

"While tests are invaluable to programmers for debugging purposes, it would be better if programmers returned to the specification in order to understand what they are supposed to develop instead of returning to the tests."

Unless you make that step past TDD into BDD where the tests are specifications..

re: Musings on Software Testing

wesdyer — Sat, 22 Dec 2007 00:58:46 GMT

Evan:

I still stand by my conclusion if the input space is large enough (infinite definitely qualifies). It is impossible to test all of the values and so the problem may always lurk just out of reach of the tests. The specification doesn't suffer the same problem since it can talk in abstract generalizations that can cover infinite spaces. Tests that are generated either automatically or by humans without knowledge of the internals make this less probable but it still remains possible.

Don't get me wrong. Programmer written tests help a lot. They are absolutely necessary. But they do not solve all our problems.

re: Musings on Software Testing

wesdyer — Sat, 22 Dec 2007 01:03:39 GMT

EGESA RONALD LEONARD:

You are right that a bug that is not found in this version may adversely affect a future version. But it still is true that a bug that a user *never* encounters is not important to fix to please the customers.

That said, I still want to fix all of the bugs because I take personal pride in my work. Even if no one else in the whole world knows about the bug, I do. I care.

In fact, if developers don't take personal pride in their work then I think morale will slip over the long run which can be catastrophic to a project. So this business about users not finding bugs is more about prioritization and making hard decisions than anything else.

re: Musings on Software Testing

Bill Campbell — Thu, 27 Dec 2007 00:56:11 GMT

Hey - Interesting! Thanks for your thoughts. I find it interesting that folks talk about following TDD to the letter - if that were the case, you'd never have any production code that didn't have a test written for it 'first'. If there wasn't a test, there would be no need for the code, right?

Regards and Happy New Year!!

re: Musings on Software Testing

Keith Burgoyne — Sat, 29 Dec 2007 19:04:50 GMT

Good musing, Wes. I always enjoy reading a person's line of reasoning when it is well thought-out. I'll contribute that any "absolute" (theory in this case) always immediately raises a red flag for me. The selection of one theory absolutely to the exclusion of others is invariably wrong. My experience is that many of the design and testing theories have components that are useful at various times, and other components that are non-useful, a waste of time, or possibly even detrimental to a project. The elements of these sets can even shuffle from project to project or even on the same project over time. The true goal, as you state, is to reduce the total impact resulting from bugs in the code. If the focus is primarily on using theory-A or theory-B, then it is not the goal that is really in focus. The various components of theory-A or theory-B should be selected based on their contribution to meeting the true goal.

re: Musings on Software Testing

Frank Hileman — Sat, 29 Dec 2007 19:46:37 GMT

With design-by-contract, "tests" (precondition, postcondition, invariant assertions) are based on declarative specifications, and the tests are constantly running, at least for debug builds, so they have a better chance of encountering unforseen situations. You demonstrated clearly that TDD is only an adjunct to specification based design. Considering the cost of maintaining intrusive, fine grained tests created using TDD, design-by-contract has a better return on investment.

re: Musings on Software Testing

ZachZ — Tue, 08 Jan 2008 18:50:52 GMT

To paraphrase Godel, any sufficiently complex system contains inconsistency.

re: Musings on Software Testing

TJ — Mon, 14 Jan 2008 21:06:47 GMT

If the input space is infinite and EVERY input value generates more than one output value, the E (experiences) would theorethically outgrow the unknown (U) which is imposible because the U is unlimited (infinite).

Simply said: Program would know more than what's unknown (based on the fact that E is growing while U is shrinking, as well that there are more E records per U record).

re: Musings on Software Testing

wesdyer — Tue, 15 Jan 2008 10:04:05 GMT

TJ:

The input space is not filled with input values but input sequences. The output space is not filled with output values but output sequences. The size of each space is the same. There is no such contradiction.

re: Musings on Software Testing

idiot programmers — Wed, 21 May 2008 17:03:58 GMT

what a bunch of CRAP

what are you smoking dude