Most Of Each Test Case Exercises A Small Fraction Of The Code

From Accountant To Scientist

The Braidy Tester — Wed, 03 Aug 2005 20:45:43 GMT

In many of my posts I have alluded to the automation stack my team is building, but I have not provided...

From Accountant To Scientist

The Braidy Tester — Wed, 20 Jul 2005 20:32:09 GMT

In many of my posts I have alluded to the automation stack my team is building, but I have not provided...

re: Most Of Each Test Case Exercises A Small Fraction Of The Code

humbugreality — Wed, 11 May 2005 19:33:30 GMT

You're both correct - "all" is a (rather big? <g/>) bit of an overstatement, especially when it comes to data values. The great phrases Michael suggests are a much better description of what I was trying to say.

Pete: "...let's say that you can define a finite number of execution paths and data values and execute them without writing additional test cases -- that would be cool!" I'm saying you can do exactly that; more details coming soon to a blog near you! This and your other questions should be answered by the end of this series. If they aren't, though, let me know and I'll expand further.

re: Most Of Each Test Case Exercises A Small Fraction Of The Code

PeteVanC — Wed, 11 May 2005 00:31:57 GMT

I don't believe you :) In general using all possible execution paths and data values is not possible (in a finite amount of time).

For example you describe the following test case in a previous blog entry:

Typing Alt to invoke the main menu, repeatedly pressing the left arrow key until the File menu is selected, repeatedly pressing the down arrow key until the New submenu item is selected, pressing the left arrow key a single time to expand the New submenu, repeatedly pressing the down arrow key until the New Document menu item is selected, then pressing Enter to invoke the New Document menu item.

What if I press the left arrow key followed by the right arrow key, followed by the left arrow key, nearly forever before finishing the setup?

In that same blog entry you describe 8 different ways to perform the test's setup. What about mixing and matching snippets from the various different ways and undoing and redoing those snippets in different combinations nearly forever before finishing the setup?

How do you ensure that hardware failures or other non-software failures are properly executed?

That said, let's say that you can define a finite number of execution paths and data values and execute them without writing additional test cases -- that would be cool!

I'm assuming here that someone needs to at least define the set of execution paths and data values -- is that correct?

And then someone also needs to verify that the execution paths and data values correctly perform the setup, teardown, or other part of the test case -- but I guess these might not be defined as additional test cases.

When the test is run, are all the possible exection paths and data values excersized at once, or only across a set of similar test cases?

In any case, how do you ensure that they all get performed within a reasonable amount of time? I imagine that any sufficiently robust combination would quickly become quite large.

Anyway, it was cool to see your picture in the profiles -- although I had imagined you differently.

Lastly, a caveat -- I haven't read all of your previous posts on this subject so it's possible that I'm not fully grasping the context of your statements.

re: Most Of Each Test Case Exercises A Small Fraction Of The Code

Michael Bolton — Tue, 10 May 2005 10:28:01 GMT

Pardon my skepticism, but I'd be surprised if you have a method that uses /all/ possible execution paths and data values--it ain't possible.

In a previous posting, you did (IMO) a very good job of identifying something that most people ignore, which is that there are plenty of ways to open a file--via the File / Open menu with keyboard, File / Open with mouse, clicking on the file in Explorer, using the "start foo.doc" syntax from the command line, and so on and so forth. Far more than most testers will identify unless they're pushed, and unless they brainstorm for a while.

Your claim would be entirely credible if you said "a good variety" or "a large number" or "reasonably good coverage", but "all" isn't possible. For example, using Word:

Type Alt-F. Click on Open. Tab into the file list. Select the filename such that it appears in the filename combo box. Backspace over the extension. Change your mind, and type the extension again.

Click on File. Click on Open. Click on the filename. Backspace over the extension. Change your mind, and type the extension again.

Click on File. Type O for open. Tab down to the filename combo box. Hit the down-arrow key. Select a file that looks interesting. Hit the right-arrow key. Hit backspace. Hit the down arrow key.

Take all those cases, and vary ad infinitum. Instead of tabbing to the filename combo box, tab past it and keep tabbing until you get there again. Now shift-tab until you're back to square one. Now click. Et cetera.

Now: it may seem unreasonable that I would quibble over the use of the word "all". However, I contend that bugs are unreasonable, and will appear in lots of places that reasonable approaches won't find. Naturally, risk should drive the decision as to how recherche we want to get, but there will sometimes be bugs beyond that frontier.

The funny thing is that, in writing this up and reproducing the steps above, I've managed to find a bug in Word. It may be related to the fact that earlier, when I opened Word, I double-clicked on a filename twice (that's four clicks), really fast, such that I opened the file for a second time before the first instance was open. This led to complaints about Normal.dot being corrupted or some such. In any case, I've just now managed to get Word (2000) into a state where it's open, sort-of, but flashing on the task bar and won't stop until (I presume) I kill the process. That's a hang--a high-severity, pain in the ass bug.

My point is that a claim of "lots" is defensible; that "what we consider to be excellent coverage" is defensible; that "we've addressed what we consider to be the most important risks" is highly defensible; and even "in our test suite, we cover a lot of varying ways in a user could reasonably accomplish the same task, unlike just about every other test automation implementation on the planet." Those are great things, but they're not "all".

---Michael "So you think YOU'RE a pedant?!" B.
DevelopSense: Software Testing in Plain English
http://www.developsense.com