mfp's two cents : Late night discussions

Late night discussion on software development #3: Unit testing

mfp — Fri, 07 Mar 2008 15:58:00 GMT

“It is a true blessing to have lots of automated unit tests”, I say aloud trawling for a response.

“It certainly is.” is the response I get back.

I throw out another statement: ”It just feels so good to have over 80% code coverage. It makes you almost invulnerable. You can change just about anything, and if the tests pass, you know you did a good job.”

“It makes you blind, is what it does.” is the unexpected answer I get back.

I look at Søren, who is sitting quietly and comfortably in his armchair. I lean forward. “What do you mean? I feel the exact opposite of being blind – I have full transparency into my tests cases, I know exactly what parts of the code they exercise, and I know if any of them fail.”

“Tell me, why do you write unit tests?”

“To have high coverage numbers.”

“And why would you like to have high coverage numbers?”

“So I know my code is fully exercised by the unit tests.”

“And why would you like the code to be fully exercised by unit tests?”, Søren keeps pushing.

I feel like this is a cat-after-the-mouse-chase, but I’m up to it. “So I know my code works.”

“Assuming your unit tests passed once, why do you keep them?”

“Because the code may stop working due to a change.”

“Ok, so let me ask you again: Why do you write unit tests?“

“So I know my code works at all times”, I answer.

“Or as I prefer to phrase it: ‘to prevent regressions.’ So let me ask you: How good is your protection against regressions if your code coverage is 80%?”

The answer is almost out of my mouth before I stop myself. Søren has a point. 80% code coverage doesn’t mean 80% protection. But what does it mean? Just because a unit test exercises the code, it doesn’t mean it will detect all possible failures in the code. I’m silent; I don’t know what to reply.

After a while Søren says: “Zero code coverage means you haven’t done enough, 100% code coverage means you don’t know what you got.”

“So why do we measure code coverage at all?”

“Because it’s easy to measure, easy to understand, and easy to fool yourself. It is like measuring the efficiency of a developer by the number of code lines he can produce. A mediocre developer writes 100 lines per hour, a good developer writes 200 lines; but a great developer only has to write 50 lines.”

I can see his point. I would never measure a developer on how many lines of code he writes.

“Tell me, how often do your unit tests fail?” Søren asks.

“Occasionally; but really not that frequently.”

“What is the value of a unit test that never fails?”

“It ensures the code still works.”, I say feeling I'm stating the obvious.

“Are you sure? If it never fails, how will it detect the code is buggy?”

“So you are saying a unit test that never fails doesn’t add protection?”

“Yes. If it never fails, it is just hot air, blinding you from real problems, and wasting your developers’ time”.

“So we should design our unit tests to fail once in a while?”

“Yes, a unit test that truly never fails is worthless! Actually it is even worse, it is counterproductive. It took time to write, it takes time to run, and it adds no value. “

“Well, it does add the value that the code has been exercised.”

“I have seen unit tests without a single assert statement. Yes, it certainly exercises some code, but unless the code crashes it offers no protection.”

“Why would anyone write a unit test without assert statements?”

“To get the high code coverage numbers their managers demand.”

I pour us another glass of red wine, while I’m mentally trying to grasp the consequences of what I just learned. By measuring code coverage we are actually fostering an environment where worthless unit tests are rewarded just as much as priceless unit tests.

“As I see it we need to distinguish the good from the bad unit tests. Any ideas?” I ask.

“The bad unit tests come in three flavors: Those that don’t assert anything, but are solely written to achieve a code coverage bar, those that test trivial code, such as getters and setters, and those that prevent good fixes.”

“I can understand the first two, but please elaborate on the last one.”

“Suppose you write a unit test that asserts an icon in the toolbar has a certain resource ID. When the icon eventually is updated this unit test will fail. It adds no value, as the icon was supposed to be update. This means the developer has two hard coded lists to maintain when changing icons: one in the product code and one in the unit tests.”

“Got it, how would you propose a better unit test for this case?”

“Well, a unit test verifying that all icons have a valid resource ID and that no two icons share the same resource would be a good start.”

I can certainly see the difference, the latter unit test wouldn’t need updating every time an icon was changed, but it would detect problems our customers would notice. I wonder how many of our unit tests fall into these three categories. I need to find a way to reward my developers for writing good unit tests.

Søren interrupts my chain of thought, “What do you do when a unit test fails?”

“We investigate. Sometimes it is the unit test that is broken; sometimes it is the product code that is broken.”

“Who makes that investigation?”

“The developer who has written the product code that makes the unit test fail.”

“So basically you are letting the wolf guard the sheep!”

“What do you mean?”

“If developer A writes a unit test to protect his code, and developer B comes along and breaks the code, you let developer B judge whether it is his own code or developer A’s code that is to blame. Developer A would grow quite sour if he detected that developer B was lowering the defense in the unit test to get his changes through.”

“Yes, that certainly wouldn’t be rewarding. I guess it would be much more rewarding for developer A to receive an alarm when developer B broke his test. He would then know that his test worked and he could work with developer B to resolve the issue.”

Søren nods.

It certainly dawns on me. Writing a bad unit test wouldn’t be rewarding, as it would never fail; whereas a good unit test would eventually catch a fellow developer’s mistake. I know the developers in my team well enough to understand the level of sound competition such an alert system would cause. I set my glass on the table. I have some infrastructure changes to make on Monday.

Late night discussion on software development #2: Estimation

mfp — Fri, 18 Jan 2008 13:05:00 GMT

“How can it be”, I wonder out loud, “that our projects always finish on time with a little less functionality than we anticipated?”

“As far as I understand, it is no wonder that your projects finish on time, as you have a fixed delivery date. At that date you basically ship what you’ve got.” Søren answers and sips his wine.

“Well, shipping is a big word. After the development phase we go into a stabilizing phase which can be long and demanding. We really need these fixed dates, as our customers expect us to deliver on the date we announce. ” I feel myself already in defense, and I try to steer the discussion in the direction I want it to go, “What I struggle to understand is, that even when we start out with the best intentions, we always end up cutting functionality away.”

“Ok,” Søren replies, folds his hands, and looks at me, “this is as good a topic as any. Let me hear how you figure out how much you can do within the development phase.”

“First of all we have a planning phase with a long list of features and functionality that we would like to implement. This list is based on customer feedback and market demands. As you might expect the list is so long, that even with all developers in the world and an undisturbed decade, we wouldn’t be able to get to the bottom of it.“ I exaggerate to drive home my point, ”The list is then prioritized, and the developers are asked to give estimates for the top items on the list.”

“So at this point you have already trimmed the list of features?”

“Yes, at this point we have a net list of features. The developers go away with this list, and do their magic; you know, prototyping, designs, specifications, and so on. Then they come back with estimates. With the estimates in hand we can derive a plan. Typically this includes trimming the list of features even further. We end up with what appears to be a realistic plan. We often leave a few more features in the plan than the estimates allow for, just in case a feature is finished early.”

“Do you have reason to believe your developers over-estimate?”

“No, on the contrary. If they were giving too long estimates the features would have been cut early, and we wouldn’t have this problem; where we cut features as we run out of time."

“So do you believe your developers under-estimate?”

“I have no reason to believe so. They have been doing this for so long, and are really a great team, and they have a remarkable tendency to hit the right dates. Having said this, they do miss the dates occasionally. “

“And what happens when they slip?”

“Well, since they set the initial date. It is their responsibility to catch up. Typically they do overtime to get back on track.”

“And when they don’t succeed in catching up?”

“Then we have a problem and have to cut features from the product. Which is my headache – why can’t we implement according to our plan?”

Søren is quiet for a moment. I use the pause in the discussion to refill our glasses.

“Let me ask you this,” Søren says, “Suppose I asked you to paint my house. You can take as long as you like, but you need to tell me up front how long it will take. If you are not finished by the time you promised me, you’ll have to pay me a compensation for each minute you are late. How would you estimate this painting project?”

“When you put it like that, I would naturally build a lot of safety into the estimate, so I’m certain I can finish well before time. I just wish that my developers could finish on time!”

“Do the rules your developers play by resemble this scenario?” Søren asks.

“Thinking about it, their behavior does, but the outcome certainly doesn’t. They are not finished well before time.”

“That is because building software is different from painting a house. You know you are done when the last board has been painted. That is not true for software. There is always something more you can do. Adding more comments, writing more tests, refactoring, just to name a few. This is the developer’s baby; they want it to be as perfect as possible before passing it on. When they finish before time, they will spend whatever time is left polishing it off.”

“I know, I was a developer once”, I smile, “and I’m fine with that. Delivering polished features sounds good to me. The problem is we end up cutting some.”

“Let us look at the painting project again. Regardless of how conservative your estimate is, you may get surprised: the foundation might not be in the condition you expected, you may run out of paint, the ladder might break, and the brush might be too small, and so on. Regardless of the precautions you take real life can and will play tricks on you.”

“Yeah, we have all been visited by Murphy.”, I smile.

“The only sane way to deal with it is to include variation in the planning of the project.”

“Hang on a second. “, I have to digest what I just heard, “Are you saying that whenever we finish a feature early it doesn’t mean the next feature will start early – and whenever a feature is late it delays the rest of the plan?”

Søren is silent. I shuffle my feet, feeling awkward. After what seems like a too long pause, I say, “I think you are right; even when we build in a safety buffer into all our estimates we run out of time. What else is there to do?”

“How about delivering even more features on time?” Søren takes a pen from his inner-pocket, leans forward and grabs a napkin. This is what he draws:

“Basically this is what you currently do. Each feature is protected from variation by the buffer your developers build in to avoid working overtime. Suppose you remove the buffer from each feature, and add a buffer in the end instead. It probably has to be somewhat larger than the buffers you use today. Something like this”, Søren continues to draw.

“I see the point. This way each feature will commence as soon as the previous is done, and any variation might be caught up during the project, and if not we have the buffer in the end.” This feels good, perhaps too good to be true; napkin-drawings and red wine have a tendency to simplify things. After a short while I’m able to put words to what disturbs me, “this would mean our developers have to give shorter estimates. I don’t want to go onto their turf, questioning their work.”

“Well, you don’t have to”, Søren answers, “you have to ask for estimates differently. Instead of asking ‘When can you be finished?’, ask ‘When can you be finished with a 50 percent probability?’ The word estimation intrinsically holds a probability factor; you have to get that information for the estimate to be of true value.”

Being a tech-guy I cannot help myself questioning something as scientific as a number, “Why 50 percent?”

“First of all it tells the developers that it is perfectly alright to be late, actually you anticipate half the features will be late. This way you reassure the developers that it doesn’t mean they have to work extra hours if they are late.”, Søren sips his wine, ” Secondly, it also means half the features are early. This way there is a good chance of the late features being balanced out by the early features.”

Søren draws again.

“If you try this out, you’ll be surprised at how short the new estimates will be. In theory a 90 percent probability estimate is about twice as long as a 50 percent estimate. But to get such short estimates your team needs insight into this way of planning. And the plan has to be transparent. Your developers will need this insight to understand the importance of letting go of a feature once it is completed; instead of keep polishing it and delaying the delivery.“

The bottle of red wine is almost empty. I feel sober.

Late night discussion on software development #1: Continuous publishing

mfp — Fri, 16 Nov 2007 17:09:00 GMT

“So tell me”, I start, “when was the last time you used a help system?”

Søren pours himself another glass of red wine and leans back in his armchair. He is silent for a while. “What do you mean when you say help system?”

I open my mouth, and close it again. It always amazes me, when I realize the depth of such a seemingly naïve question. My immediate response seems too narrow. I have to break out of my standard way of thinking. A help system is not just a part of a larger system that helps you when you are stuck. It is even more than this. What comes out of my mouth is: “A help system is a system that helps you accomplish your tasks.”

“In that case I use help systems daily”, Søren proclaims.

“Really? I thought you already knew all the answers.”, I cannot resist the temptation.

Søren smiles, “You are also a developer, how do you manage to get anything done without a help system?”

Apparently Søren is into one of his moods where he answers all my questions with a new question. I sip my wine, take a deep breath and start to elaborate. “As a developer my work is quite dependent of the phase my project is in. Let's look at my favorite, the implementation phase. While writing code I never press the F1 button. I have worked with my current code editor for so long, that I’m fully aware of what it can and cannot do for me.”

Søren says: “So you have turned off IntelliSense?”

“No way – I don’t even know any developer crazy enough to do that. Actually I don’t even know an code editor that allows you to turn off IntelliSense.”

“Please go back to your definition of a help system.”

It dawns on me. Help system are more than just F1 help. It is anything that enables me to do my job. “Ok”, I say, “So what other help systems do you use?” I ask. I’m starting to feel confident I will earn more than a hangover from tonight.

“Well,” Søren says, “as a developer I often need to lookup API reference information. To do that I use the web constantly.”

“So you actually find any useful information out there? I’ve tried a few times but given up. Instead I browse objects locally searching for something that will satisfy my need.”

“That is one of the industry’s dilemmas. Users who have been disappointed by the documentation available in the help system, usually never come back. So why bother writing documentation, if it is not going to be read?“

“Exactly, if the documentation isn’t there, then there is no point in looking.”

Søren continues, “The users also suffers here, instead of looking for information where it is obvious they eagerly jump into the haystack looking for the needle. How can this destructive habit be reversed?”

A simple question, I think to myself. I answer: “Well, if the software providers just wrote the documentation we need, we wouldn’t have the problem.”

Søren promptly replies: “The question to ask is: How do you know it is not already there?” I feel like I’m being maneuvered into a corner. “When I installed my environment about 2 years ago, I browsed through the reference documentation, and it was primarily empty placeholders.”, I pause to think about what I just said. “Naturally these placeholders might have been populated in the meantime.“ I make a mental note to reserve some time on Monday morning to give the online documentation a second chance.

Søren finishes off his glass of red wine. He looks content.