cashto's blog : So you think you write unit tests? Think again.

Other blogs

Archives

So you think you write unit tests? Think again.

Here's another thing that gets under my skin: the term "unit test". Not that I have anything against them, they can be a fine tool (and at some future date I'll flesh out my developer test philosophy a bit more). No, what bothers me is that I see and hear the term misused all the time, and it causes a lot of confusion.

Where I work, developers frequently claim to have written "unit tests" for their code, when really they haven't. More often than not, what they've really written is an integration test. Don't get me wrong -- both kinds of tests are very useful, but they are not synonyms for each other. It's important to distinguish between them, because unit tests have an entirely separate set of goals, techniques, and best practices which do not apply to integration tests or other sorts of tests.

Here's a quick list of some of the different kinds of tests. It's not a complete list, but it should cover a lot of common terms you'll often see tossed around at work or in the literature. As you can see, there can be some overlap between the terms: for example, you may have a developer test which is also a unit test which may also be a checkin test as well. But keep in mind they are all orthogonal concepts, and can vary independently.

Unit test: tests a function or class in complete isolation from other code in the system. If the item under test has dependencies, those dependencies are replaced by fake ("mock") objects. If it touches the filesystem, hits a database, access a network, or displays UI, it's not a unit test. Unit tests have practically no setup and run extremely fast.
Integration test: tests the interaction of two or more classes and functions working together. Multiple layers are usually involved. Whereas unit tests ensure that the "bricks" in your system are square, solid, and uniform, integration tests ensure that they are stacked up correctly and glued together the right way. A test that verifies data entering one end of a pipeline will come out the other end is an integration test. So is a test that verifies that data is unchanged when round-tripped through a reversible process.
Characterization test: tests unknown or "legacy" code for response to stimuli. Often used during refactoring to ensure that the behavior of the code is unchanged. Characterization tests are often written in total ignorance of the inner workings of the code, and the expected results are often completely unknown as well.
Developer test: any sort of test (unit test, integration test, acceptance test, stress test) written by a developer as a design and development aid, as opposed to a QA engineer ensuring the correct operation of the code after the it has been "thrown over the wall" to Test.
Automated UI test: a test that simulates a fixed series of user interactions, such as mouse clicks and key presses, and the resulting UI is then compared to an expected result.
Regression test: a test which ensures that bugs that get fixed stay fixed.
Checkin test: a series of tests used to verify that the product is not accidentally broken by new features or bugfixes, prior to the change being submitted. Checkin tests do not exhaustively test everything (they still need to run in a reasonable amount of time), but they should give good coverage over certain areas. Checkin test suites are usually mostly unit tests, because unit tests execute so quickly. But integration tests and regression tests can be good candidates for inclusion as well.

Posted: Wednesday, September 27, 2006 11:26 PM by cashto

Comments

Dan's Archive said:

# November 1, 2006 1:23 PM

Javid Jamae said:

I agree that you should use test doubles to remove dependencies on external systems, but I don't agree that it is not a unit test if you have two or more classes that you're testing together. The "unit under test" can be several classes.

You should probably read Fowler's article "Mocks aren't stubs". There are really two main styles of unit testing: interaction-based and state-based. With state-based unit testing, you don't necessarily have to mock out every other object that you are touching.

# May 31, 2007 2:42 PM

Tim Stewart said:

If you are designing some file formats, how would you write unit tests for code that needs to read and write these formats?

I suppose you could mock the file system and do everything in memory but if the mock does not behave exactly like the file system, you would have a false sense of security (and buggy code).

It sounds to me that this would be one exception to the "if you write to the filesystem, it's not a unit test" but maybe I'm just rationalizing.

Thanks.

Tim

# August 2, 2007 12:32 PM

Robert Ames said:

... you break things into components. You have a WriteFile(...) function and a ReadFile(...) function. Test the junk out of those functions (maybe those are integration tests?)

Then you have a Load( ReadFile() )

...and a WriteFile( Save() );

...which then you factor out the actual loading and saving:

$orig = $obj->Save();

$loaded = new Obj( $orig );

assertEquals( $orig, $loaded );

(for some definition of equals)

...basically: building on solid components increases your confidence.

00Robert

# August 13, 2007 3:43 PM

Observer said:

"building on solid components increases your confidence"

Yeah, the file system is known to be unreliable. Better write your own simulator, so you know that it works.

# August 13, 2007 3:47 PM

Bill Smith said:

It also helps if the test returns a "yes" or "no" result. I've seen plenty of tests that just print out a bunch of goo. The output is relevant to the person who wrote the test on the day the test was written. A few months later, no one remembers what the output means and, in particular, whether the output indicates success or failure.

# August 13, 2007 4:31 PM

Tony Morris said:

Here's another thing that gets under my skin: the term "unit test". It is a euphemism for "poor proof checking". While many languages make it difficult to write proofs; particularly with inferior type systems (ala Java, C++, C# et. al.), one should at least specify a proof statement and an automated system that attempts to falsify (though not disprove or prove) the statement ala QuickCheck, ScalaCheck, etc.

# August 13, 2007 6:21 PM

JPD said:

Indeed, Tony! What better time than now to bring the light down from the ivory tower to illuminate the dark and primitive cubicles of corporate America?

I'm going lead the charge to purge "inferior type systems" from our production environment, and get all new development done in Haskell. Easy case to make since Haskell is at the heart of so much successful and mission-critical software, like... uh... erm...

# August 14, 2007 9:20 AM

cashto's blog

Recent Posts

Tags