Progressive Development : checklists

Motley says: “Slow down to go fast? Do I look like a tortoise?”

James Waletzky — Wed, 22 Apr 2009 08:23:00 GMT

Summary

	Motley: There is no harm shortcutting a few steps of development to meet a deadline. Some up-front check-in steps just take too much time.
	Maven: Slow down to go fast. Obey check-in checklists or you will pay for it later in the development cycle. Pre-check-in tasks such as code reviews, unit testing, static analysis, and buddy builds of various flavors must not be avoided.

______________________________

[Context: The code check-in deadline is approaching. Motley has code that is in progress and he is hurrying to get his change in prior to the build snap]

Maven: Hey, Mot. How's it-

Motley: Quiet! I have to get this code checked-in by the end of the day to meet the deadline. I don't have time to chat and need to rush to get this code in.

Maven: Whoa - maybe we should chat. Please just give me a few minutes here - it won't affect anything in the bigger scheme of things. What do you mean you have to "rush"?

Motley: The daily build starts at 6pm and it's 5:35pm. I only have 25 minutes to get the code checked-in so I have to take a few shortcuts.

Maven: I cannot stand the word "shortcut". Care to clarify?

Motley: It doesn't really matter what you can and cannot stand. But since you asked, I am going to skip writing unit tests for this last class, I am going to forgo code review because my code is always perfect anyway, and I am going to skip running FxCop just this once.

Maven: Have you ever heard the phrase, "Pay me now, or pay me more later"? What you are doing is akin to racking up credit card debt. On the outside it looks like a $5000 purchase, but in the long run it is going to cost you a lot more in interest if you do not pay off the debt as soon as the bill comes due.

Motley: Oh, no. More Maven analogies. I can do without those. And by the way, Boyz II Men called - they want their shirt back.

Maven: Always insulting my clothes! This blue and pink sweater combo isn't that bad, is it? Anyway, taking shortcuts now when the code is cheap to fix pays off in the long run and leads to quicker ship cycles and shorter bug-fix cycles later in the development cycle.

Motley: How so? Bugs are bugs, I need to fix them anyway.

Maven: It's much cheaper to fix bugs now than later. Fixing them later usually involves someone else creating and maintaining bugs in a bug database, fixing them when the code is no longer in context and fresh in your memory, and the tendency to bolt-on fixes and shortcut fixes. Additionally, if the test team does not find an issue, the cost of fixing it post-release is large due to the overhead in working with the customer, the bug getting back to the product group, deploying the fix, the negative press, and other factors. If you find an issue that modifies the design later, you may have to change a lot of code possibly destabilizing the product.

Motley: Yeah, yeah - the cost of change curve. I know it.

Maven: Then you know that you shouldn't rush your development. Would you rather fix bugs later when they are more expensive or now when they are cheap without the overhead of a post-check-in bug? Leaving bugs until later forces long "stabilization" periods on the team, which, in my opinion, are evil and can be avoided by a diligent team.

It is well worth pushing back on the deadline to ensure that you can adhere to the following code check-in checklist:

Unit tests. We have talked about unit testing at length in other discussions. This is the number one quality-driven developer technique. You must have a set of unit tests in place that exercise all of your new code and you can check-in when all the tests pass. This items should be bolded on a checklist.
Code coverage. Hand-in-hand with unit testing is code coverage. Aim for 70%+ code coverage for all of your code. More important than the coverage number is the feedback you receive from the coverage tool regarding source lines that were missed in testing so that you can add new tests.
Debug and retail builds. A debug build is generally different than a retail build, particularly when you make use of debug-time primitives such as assertions. You need to build both flavors to ensure the build succeeds either way. Broken builds are not tolerable as they block the entire team. In fact, do a clean build (vs. incremental build) to be safe.
Static analysis. A static analysis tool like FxCop finds bugs for you. Per the cost of change discussion, why not fix them now instead of later when it is more expensive? Why risk security defects escaping your defenses and being released into the wild where hackers and customers will find them?
Reliability tools. Running a tool like AppVerifier is more relevant to C++ applications. Running this tool is a must prior to check-in as it will help identify heap corruptions, improper use of handles, and misuse of critical sections. Additionally, for native code it will point out memory leaks.
Warning level. Crank up the compiler to generate warnings at level 4 (or whatever is highest for your compiler) to ensure that you have addressed all possible compiler-identified issues. Get into the habit of perpetually building at the highest warning level. Get clean and stay clean.
Other tools. Run any other organization-wide tools such as a copyright header checker, XML documentation checker, or coding style checker (e.g. Source Analysis) prior to check-in. You don't want to be the grunt fixing a ton of these issues later. Trust me on this one.
Code reviews. All code should be at least peer-reviewed, with larger changes (e.g. new features) being put through a code inspection. Having at least one other pair of eyes look at your code is likely to find many issues before even the test team gets their hands on the code.
Change list description. Include complete descriptions for check-in change lists, including how you tested the code, why the change was made, how you built, possible regressions, and other risks.
Buddy build. To ensure you did not forget to add a file to your change list and to protect against other bad assumptions, package up your change and pass it to a buddy for her to build. Most source control systems allow you to package up a change list for distribution. If you have a continuous integration system that builds changes before check-in, this is less necessary.

Motley: You cannot possibly be serious! If I do all that now there is no way I would ever make the deadline, let alone make a check-in.

Maven: All of the above is necessary, particularly in a large, quality-focused organization. We want to exterminate bugs before they are even checked-in. The checklist above helps with that.

Motley: Can you say "Sloooooooooooooowwwwww"? I knew you could.

Maven: Get in touch with your inner tortoise. I heard that in a TED talk on Slowing Down in a World Built for Speed by Carl Honore. Going a little slower up front will save time in the overall development cycle, and I would claim that you ship sooner as a result. Of course, that was not the specific theme of the talk, but the lesson can be applied here. Also, be pragmatic. Perhaps you avoid a step or two for small bug fixes.

Motley: One thing I do not quite buy is the lack of long stabilization periods at the end of the development cycle. We still need to put all the pieces together!

Maven: Ultimately we want to avoid long stabilization cycles and instead be stable when you check-in. A tail of the development cycle needs to be dedicated to integration and overall acceptance test validation, but is not the same thing as "stabilization". We talked about this before in our discussion on developer testing.

Motley: So you are telling me that I need to pay my boss a visit as soon as possible to tell him that I am going to miss the deadline? He is going to kill me.

Maven: Well, seeing as you only have 5 minutes to the build snap-

Motley: <POW>. You deserved that shot in the nose for wasting all my time and missing the build snap!

Maven: Ouch! Okay, I deserved that. But seriously, your boss is a reasonable person and will understand. Firstly, you are communicating bad news as soon as possible (well, not really, but let's pretend), which all managers appreciate. Secondly, present it such that you are focused on quality and want to avoid taking shortcuts, and that you feel this course of action is best for the product. I will bet you lunch that he understands.

Motley: There are some days where I wish I stayed in bed...

______________________________

James' Pointer: The checklist items I mention above are from a real team in at Microsoft in Windows Mobile. Quality is our #1 value as a team. We want solid check-ins that adhere to the checklist to ensure we don't have a long tail of bugs later. Yes, it is a lot of work but ultimately saves time in the long run. Slow down. Don't rush. Nail the checklist on every check-in. You will thank yourself later. Okay, maybe not, but you should.

James' Double Pointer Indirection: A good developer knows how much work goes into each and every check-in and factors these things into their estimates. These things take time.

James' Triple Pointer Indirection: "Slow down to go fast" is real advice in car racing circles. Intuitively it does not make a lot of sense. However, if I go into the corner too fast I have to brake to avoid the wall and then take a lot of time and gas to come back up to speed exiting the corner. If I slow down on entry, roll through the corner and smoothly apply the accelerator on exit, I actually go faster even though I slowed down entering the corner.

Maven's Resources:

TED (http://www.ted.com): A great web site of some of the world's best speakers. There are talks on a multitude of topics. You are guaranteed to find something of interest.
AppVerifier (http://msdn.microsoft.com/en-us/library/ms220948.aspx): A mandatory tool for C++ developers on the Microsoft Windows platform.
Source Analysis (http://code.msdn.microsoft.com/sourceanalysis): A customizable tool that validates coding standards and is a good complement to FxCop.

Motley says: "Affinity exercises? How do I work that sideways "8" into brainstorming?"

James Waletzky — Tue, 11 Mar 2008 18:30:00 GMT

Summary

Motley: Affinity exercises? How do I work that sideways "8" into design brainstorming?

Maven: Use affinity exercises for generating lots of ideas and prioritizing them. Define the topic, generate ideas, categorize, discuss, prioritize using dot voting, add up the votes, and create the summary.

______________________________

[Context: After the talk on team checklists, Motley was motivated to create a team code review checklist, but had some difficulty]

Motley: After our recent discussion on checklists, I was inspired to create a team checklist for use in code reviews. A few common problems appear fairly regularly, and I want to ensure everyone looks for them during a code review.

Maven: Sweet! Glad to see you are implementing some quality practices. Checklists provide a great reminder of frequent errors, as we discussed.

Motley: I thought I would get everyone together to determine what issues to look for in code as a team. We ended up focusing on one or two things and had in-depth discussions. By the end of the hour-long meeting, we didn't have much that was useful. I then had a chat with Mort and we came up with too many ideas - the opposite problem. We didn't know how to narrow them down. In the end we had nothing. Pretty much a waste of my time and the team's time.

Maven: Ah, yes - the unstructured brainstorm session. Brainstorming with a team is actually a very difficult exercise if you don't set some ground rules and put some structure in place. Have you ever tried an affinity exercise?

Motley: Affinity exercises? How do I work that sideways "8" into brainstorming?

Maven: Affinity. Not "Infinity".

Motley: Oh.

Maven: An affinity exercise is a brainstorming technique for many types of problems. By following a very simple structure, you can generate a lot of ideas in a short amount of time and come out of the meeting with a list of the most important ideas to the team.

Motley: As long as I keep all my limbs and I don't end up taking anyone else's, I'm game to hear more. Doesn't "affinity" have something to do with attraction? Is this exercise appropriate for a work environment???

Maven: Sometime I'd like to get a look into the depths of your conscious as I am sure there are some scary things in there.

Motley: Get on with it.

Maven: Ok. The affinity comes from the fact that you are going to generate a lot of ideas and examine the relationship between those ideas. First you need to stock up on Post-It notes, find a room for the team, invite a few people, and schedule an hour for the exercise.

Motley: What are the Post-It notes for?

Maven: You will soon see.

Motley: Hmmm… I think I'll have to bring back a few pads that I, um, borrowed for home use because our supply cabinet is running low. How does the exercise work?

Maven: The first part of the exercise is to define the topic to be brainstormed. Provide an overview to the team on the problem for which ideas will be generated.

Motley: Ok - we are generating a checklist for reviewing production-quality code. Next?

Maven: Great. Individuals generate as many ideas as possible. An easy way to do this is by presenting each person in the room with a stack of Post-It notes to be used for writing down one idea per note. Give the team about 15 minutes of silent time to write down as many ideas as possible. Write down an idea, rip off the Post-It note and put it beside you, then repeat. There are no wrong answers, we don't worry about duplicates between individuals, and there is no discussion at this point. Anything goes.

Motley: One idea per Post-It note. Everyone is quiet, heads-down and writing for about 15 minutes, or presumably until they run out of ideas.

Maven: Exactly. Once all of the ideas have been exhausted or the time limit exceeded, everyone posts their ideas on a flat surface (wall, whiteboard, paper, etc.). If two people come up with the same idea, they overlap the Post-It notes. The overlap provides a real-time filtering of duplicates. I recommend using a Whiteboard for this if possible so that you can draw on the surface.

Motley: What if the ideas are similar but not identical?

Maven: The next step after everyone has their Post-It notes on the wall is to find the relationships between ideas and categorize the ideas together. Create logical groups based on the types of ideas that were generated. For example, if you are creating a code review checklist, there may be several items related to security like checking for correct buffer usage. Group those items together, draw a box around them, and label the category/theme. The category helps provide context around the ideas, and may even be a checklist item in itself if the ideas within the category are too specific or really represent one central idea.

Next, you want to discuss the ideas to ensure everyone understands the complete set of notes. Because the Post-It notes typically do not contain much text, some explanation is required from the authors. The explanation may also lead to a few more ideas, so as people in the group have them, they can post a few more on the board.

Motley: What about the dumb ideas - presumably we don't want to waste time discussing those?

Maven: Encourage people to be as creative as possible and think outside the proverbial box. There are no "dumb" ideas. If, in your head, you feel an idea is not feasible, keep it to yourself. It will come out later in the exercise.

Motley: We have the ideas generated and displayed, and we have an understanding of what they are. This still doesn't solve one of our big problems - too many ideas posted and either no idea how to prioritize them or some disagreements on how to do so. This technique looks great for generating the ideas, but prioritization is still an issue.

Maven: I'm not done yet! Of course, the members of the team may have differing views on which are the highest priority ideas. We discussed previously that checklists should be kept to around a page in length to be useful. Chances are good that after an exercise like this you have multiple pages worth of ideas. The group must narrow them down.

Motley: I'm waiting…

Maven: Here is a good technique: make sure everyone has a pen or marker. Each person gets seven votes to use how he or she pleases. The votes can be distributed across the ideas in any way. For example, perhaps Marcy really wants to see two of the items on the team checklist - she can put 4 votes on one idea and 3 votes on another. If Mark sees 7 ideas that he feels should be on the checklist, he can put 1 vote on each of 7 ideas. Voting is typically done by placing one dot per vote directly on the Post-It notes. Not surprisingly, we call this "dot voting."

Motley: Simple but effective. After everyone has had all their votes, we tally up the total number of votes for each of the items and take, say, the top 10 for the checklist.

Maven: You got it. Those top 10 will be the ideas the team is most concerned about for code reviews. Notice, too, that the ideas that are not feasible get left out, but no one feels bad because someone insulted their idea. The exercise allows us to wade through a large number of ideas, prioritize them appropriately, and produce a result that the team is happy with.

Motley: Awesome - we're done!

Maven: Not quite. Someone then needs to take action and write down the checklist in whatever form your team uses (e.g. wiki, InfoPath form, etc.), ensuring it is made available to everyone in a place they have access to. The exercise was useless unless we capture the results and the checklists is used by the team.

Motley: That goes without saying. I could see us using this technique for lots of different kinds of brainstorming activities, from where to go for dinner to what weekly practical joke to play on you.

Maven: Please, no more practical jokes. I'm still trying to get that tree sap off my jeans.

______________________________

Maven's Pointer: There is no magic to using 7 votes for dot voting. Using an odd number helps people prioritize more than using an even number that they can split evenly. For larger groups of people, you may want fewer dots, like 5 or 3. For smaller groups of people, stick with 7 or 9 or larger as there will be fewer dots to go around due to less people in the exercise. The risk with fewer people, however, is that one person can manipulate the results a little more by putting the majority of their dots on one item.

Maven's Double Pointer Indirection: When the exercise is complete, take a digital photo of the whiteboard containing the Post-It notes in case you want to review the session later or see what other ideas were generated should some items start disappearing off the checklist as they are no longer frequent errors.

Maven's Triple Pointer Indirection: Remember the discussions on Human Performance Technology. You can use the Six Boxes to guide people's thinking a little bit more in various areas that may be relevant to your problem, particularly if the number of ideas being generated by the team is low.

Maven's Resources:

In-depth description of affinity diagrams: http://www.mindtools.com/pages/article/newTMC_86.htm

Motley says: "Checklist? We don't need no stinking checklist!"

James Waletzky — Tue, 04 Mar 2008 19:29:00 GMT

Summary

Motley: Checklist? We don't need no stinking checklist! Checklists are for anal people with short memories.

Maven: Checklists are very useful for activities like code reviews, where it is difficult to remember everything we need to look for. Good checklists are short, consistent, easy to understand, accessible and provide support information. Personal and team checklists are recommended.

______________________________

[Context: Maven spent the morning looking over the code reviews in his Inbox and noticed a few odd things about one of Motley's code reviews from earlier in the day]

Maven: I was just looking over the code you sent out for review earlier in the day. Nice job on finding those errors you found. There are a few more things I might point out, however.

Motley: Did I miss a few? Hard to believe as I am the king of code reviews. I must have been having a bad morning. What did I miss?

Maven: Let me ask you this first: when you look for issues in a code review, what kinds of things do you look for?

Motley: The usual stuff - memory leaks, invalid usage of pointers, failure to check input parameters - that kind of thing. That covers the major ones.

Maven: What about issues like commenting, proper use of exceptions, freeing up resources, ensuring strings are localizable, assertions are used appropriately, unit tests are created, and checking for buffer overruns? I could go on and on.

Motley: I usually remember to check for those kinds of problems. Depends on my mood.

Maven: Do you ever use a checklist?

Motley: Checklist? We don't need no stinking checklist! Checklists are for anal people with short memories. Checklists will just slow us down. Plus, they get out of date quickly in my experience. Better to just get into the habit of knowing what to look for and then just do it!

Maven: Let me ask you this: do you remember everything you need to buy at the grocery store?

Motley: The common stuff I buy from week to week? Yes.

Maven: What about the one-off groceries? What if you want to bake a cake and you don't have any flour, which is not something you buy from week to week?

Motley: First of all, think about what you just said. Do you think I would ever be caught baking a cake? Wise up, Mave.

Maven: Good point. Sorry. The point is that for things you are in the habit of, your brain just automatically remembers them. I always know I need fruit when I go to the grocery store. However, for things that I don't buy regularly, like chocolate chips, I need a list.

Motley: Do what you need to do. I still try and get into the habit of looking for certain items in a code review, though, so I can avoid a checklist.

Maven: Getting into the habit is great, but with all the things a developer has to keep in his or her head, it's hard to remember everything when coding.

If we sat down and brainstormed every characteristic of production-quality code, we would probably fill a couple of whiteboards. Unfortunately, the human brain only retains about 7 +/-2 concepts in short term memory at a time. That's why phone numbers are seven digits (forget the area code, which is pretty static). For concepts outside of that core 9 - to be optimistic - we really need some tools that help us with reminders.

Motley: There are certain things that I "just remember", like checking for null pointers. I don't need a checklist to help me with that.

Maven: I agree.

Motley: You agree?!? Then what's the point?

Maven: For concepts that really are habit and don't require any extra effort to think about, consider not putting those on a checklist. The checklist is a helpful tool to ensure you remember those concepts that have not become habit. Additionally, if the team, for example, always remembers to pull strings out of resource files to help make the code localizable, then get that off the checklist and replace it with a reminder to check for a mistake that the team often makes. The highest priority items for the business or the release are also good candidates.

Motley: So now you are talking about maintaining checklists, which I indicated is cumbersome.

Maven: Keep checklists in a simple format that everyone can easily edit, like a wiki. Sure, you have to spend time maintaining them, but if it saves even one bug from shipping, it has just paid for itself 50 times over. Even better, you might want to have your personal checklist in a tool that you use every day and all day. For example, I noticed you check your e-mail a lot - you could have your checklist inside Microsoft Outlook as a task list. There it is easy to maintain and always available. The wiki works just as well though, particularly for team checklists.

Motley: Now you are talking about more than one checklist? It's getting worse.

Maven: A good practice is to have a team checklist and a personal checklist. The team checklist contains items that you want everyone to look for in a code review. It typically contains the most frequent errors made by the team, or major areas to focus on. Your personal checklist contains items specific to you, such as things you often forget to do or look for. For me, I often forget to make liberal use of assertions, so my checklist helps remind me to code defensively. A best practice is to use your person checklist to review your code prior to sending it out for review.

Motley: So I guess I start with the stuff I missed from this mornings code review. I am just going to keep it in a simple text file on my desktop where it is easy to find and edit.

Maven: Format is not important - whatever works for you. Here are a few more characteristics of good checklists:

Keep checklists short. By "short", I mean one page or less. Big long multi-page checklists do not do much good as they become unwieldy to use. Only have items that are the most important to remember.
Use consistent phrasing. Checklists with inconsistent phrasing are hard to use. Settle on one grammatical style. Some examples:

Affirmative statements with the "do" action verb: "Do free up any allocated handles."
Ask questions: "Are all pointers freed?"
Make direct statements: "Function header comments are accurate."

Mixing styles makes the checklist difficult to read.

Make checklists easily understandable. Avoid too much jargon unless the terminology is known by everyone on the team. Keep checklists simple.
Ensure checklists are accessible. For team-based checklists, in particular, make sure they are easily accessible and updatable. Put them on a team web site in an easy-to-find location. If they are hard to find, no one will use them. If they are hard to edit, they won't be updated.
Have multiple views, with supporting information. A good checklist actually supports multiple views. You may have a quick and dirty short form to act as your primary list of reminders, and you may have a more complete and verbose view with explanations for each checklist item. That way a new member of the team can easily get familiar with each item in detail. Links to resources for more information are great additions.

Motley: As always, you have lots of rules for everything. Don't you get tired of being so thorough?

Maven: Well, no. Ha ha ha. These "rules" are basically some intuitive guidelines (softer than a "rule") to help create a good checklist. For your personal checklist, you can do what you want as long as they help with the main goal of reminding you of important items.

Motley: Fine, fine, fine. Based on my poor code review performance this morning, I'll give it a shot. I'll create a quick checklist for code reviews.

Maven: Great! Checklists are also great for other activities, too, like design reviews, unit tests, code check-ins, and many other development activities.

Motley: Let's not get carried away. I said I'll create one for code reviews.

Maven: Ideally, you also think about building tools that automatically validate your checklist items without human interaction. That's what a tool like FxCop does - it takes a big checklist like the .NET Design Guidelines and automatically ensures your code satisfies those guidelines. Automated checklists are the best kind!

Motley: I'm always up for writing code to remove more manual labor. Let me give that some thought.

______________________________

Maven's Pointer: One interesting format for checklists is in XML. With XML you can apply any type of XSLT transform to the data to generate different views. You can have one view that is a summary of the checklist item, and another view that contains all of the details. Each view is based on the same central XML file. Developing checklists using a tool like Microsoft InfoPath abstracts away the XML data format and still allows multiple views. Store the XML on a Microsoft SharePoint server to make them centrally accessible, and you have a great checklist solution.

Maven's Resources:

Code Complete, 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.

There is a wealth of checklists for software developers in Code Complete. Check the end of each chapter for useful checklists on coding, commenting, assertions, code reviews, and much more.

Checklists.com, http://www.checklists.com/

A nice set of general checklists

How to Create a Better Checklist, http://hwebbjr.typepad.com/openloops/2005/09/how_to_create_a.html

A more detailed summary of characteristics of good checklists