Motley says: "100% code coverage tells me there is nothing left to test"
Summary
Motley: If I hit 100% code coverage for my tests, then there is nothing left to test.
Maven: 100% line coverage does not necessarily mean you have full test coverage. You have to consider condition coverage as well.
______________________________
[Context: Motley is all excited as he has started to measure code coverage on his tests]
Motley: Hey Mave, check this out. I'm one up on you this time. I've been doing Test-Driven Development for a little while now and discovered that Visual Studio 2005 has code coverage built in! I can see exactly what lines of code my tests are executing and see an overall percentage of lines covered. Pretty cool, eh?
Maven: Very cool! Code coverage can be a very useful tool to help measure quality.
Motley: Of course you knew about it already Mr. Smartypants. I should have known.
Maven: Ah, don't feel bad. Visual Studio and I are best buddies and know each other intimately. What kind of overall code coverage numbers are you getting?
Motley: I'm doing pretty well, if I do say so myself. Check this out: 80% coverage on this module.
Maven: Pretty good, but why aren't you at 100%?
Motley: 100% code coverage?!?! Surely you can't be serious!
Maven: I am. And stop calling me "Shirley".
Motley: Ever been in a Turkish prison? Hehehe. I couldn't resist. "Airplane" is one of my favorite movies - EVER! And these quotes are classic. But seriously, I can't possibly hit 100%.
Maven: Sure you can! Doing TDD you should never be writing any code without a test. As a result, you should be very close to 100% if not at 100%. You can use code coverage to help gauge how thorough your tests are.
Motley: So I guess if I hit 100% on a module there is nothing left to test. Excellent!
Maven: Not so fast. You know there's always a catch. Are you measuring line coverage or condition coverage?
Motley: What's the difference? I think VS said line coverage but I'm not sure.
Maven: Line coverage measures your overall coverage in hitting all lines of code. Condition coverage validates that all possible blocks (accounting for conditions) have been hit.
Motley: Don't these basically boil down to the same thing? If I hit 100% either way, I'm pretty happy.
Maven: Check this example:
If (a == 1)
{
// do some stuff
}
else
{
// other stuff
}
If (b == 1)
{
// yet more stuff
}
else
{
// keep on going with more stuff
}
How many test cases do I need to get 100% line coverage?
Motley: Two. I can test the if/if and then the else/else.
Maven: Correct! 100% line coverage. Do I have enough tests to fully cover all my conditions?
Motley: Fine. No you don't. You need 4: if/if, if/else, else/if, and else/else. I see your point. Even though I have 100% line coverage I may be missing some tests. And by the way, I get diminishing returns by trying to hit 100% - some code constructs compile down to a lower level that make it difficult to get 100%. I'm not going to bother covering those.
Maven: As I have said before, you are a very smart man, Motley.
Motley: Tell me something I don't know. Oh wait, you try and do that every day! Hang on a second. I have to rush off to the kitchen.
Maven: What is it?
Motley: It's the room in the back with a fridge and counter, but that's not important right now.
______________________________
Maven's Pointer: Turning on code coverage in Visual Studio is not super intuitive. First set up your main application or library project. Then add a Test project to your VS solution. Once you have a test project in place, there is a file called localtestrun.testrunconfig that VS adds to your solution. Double-click that file. In the dialog that appears, click "Code Coverage" on the left side and check the box next to your non-test projects.
Maven's Resources: Don't be fooled by the coverage report, How to: Obtain Code Coverage Data (with Visual Studio 2005)
