How to make sense of Code Coverage metrics
(co-authored with Brad Wilson)
Even since the release of Visual Studio Team System 2005 with easy to use code coverage tools, people inside (and outside) of Microsoft have been talking about what it means. The most common approach we've seen is for a team leader to mandate a minimum code coverage number (like "all code must have at least 80% code coverage before being checked in"). As agile advocates, you'd expect that our team's TDD'd code would have 100% code coverage. And you'd be wrong.
We could all agree that test-covered code is better than not, so we probably also agree that a high coverage number is better than a low one. Why can't we come up with a hard and fast number that means "good"? The answer is that it's different for every project, and even for a single project, that number may change over time. It is inevitable that code is written which isn't covered by a unit test. A few examples of acceptable code without tests might include: web service wrappers generated by Visual Studio, views in a Model-View-Presenter system, and code for which failure is only possible because the underlying platform fails (like helper methods that pass default values into more complex .NET CLR methods).
So when you run that code coverage system for the first time and it pops up "73%", what next? Well, if it's the first time you've run the test, you're probably going to check out that other 27% and see how you categorize it. Does that code need coverage or not? If so, use the lack of code coverage to educate yourself on taking smaller steps in TDD. But if not, what then? Is 73% a magical number?
The answer lies in a metric that can be used to predict the weather, barometric pressure. Today, weather.com says the barometric pressure in Seattle is 29.66”Hg and falling. More important than the absolute measurement is the trend: it tells you that the overcast and drizzling rain outside is going to get worse, not better. The two pieces of information - value and trend - are used to predict what will happen next. The same can be said for code coverage. You can determine the relative health of your tests by using the value and the trend to determine the appropriate action. That’s the important bit: code coverage gives you relative measurements against itself, not an absolute measurement against a target value. If your coverage fell from 73% to 72%, would you be worried? What if it fell from 73% to 67%?
Rather than holding your team to an absolute goal, use the trend to determine when you should invest time to figure out where you're covered and not, and to determine what to do about it.
