Progressive Development : code reviews

Motley says: "Affinity exercises? How do I work that sideways "8" into brainstorming?"

James Waletzky — Tue, 11 Mar 2008 18:30:00 GMT

Summary

Motley: Affinity exercises? How do I work that sideways "8" into design brainstorming?

Maven: Use affinity exercises for generating lots of ideas and prioritizing them. Define the topic, generate ideas, categorize, discuss, prioritize using dot voting, add up the votes, and create the summary.

______________________________

[Context: After the talk on team checklists, Motley was motivated to create a team code review checklist, but had some difficulty]

Motley: After our recent discussion on checklists, I was inspired to create a team checklist for use in code reviews. A few common problems appear fairly regularly, and I want to ensure everyone looks for them during a code review.

Maven: Sweet! Glad to see you are implementing some quality practices. Checklists provide a great reminder of frequent errors, as we discussed.

Motley: I thought I would get everyone together to determine what issues to look for in code as a team. We ended up focusing on one or two things and had in-depth discussions. By the end of the hour-long meeting, we didn't have much that was useful. I then had a chat with Mort and we came up with too many ideas - the opposite problem. We didn't know how to narrow them down. In the end we had nothing. Pretty much a waste of my time and the team's time.

Maven: Ah, yes - the unstructured brainstorm session. Brainstorming with a team is actually a very difficult exercise if you don't set some ground rules and put some structure in place. Have you ever tried an affinity exercise?

Motley: Affinity exercises? How do I work that sideways "8" into brainstorming?

Maven: Affinity. Not "Infinity".

Motley: Oh.

Maven: An affinity exercise is a brainstorming technique for many types of problems. By following a very simple structure, you can generate a lot of ideas in a short amount of time and come out of the meeting with a list of the most important ideas to the team.

Motley: As long as I keep all my limbs and I don't end up taking anyone else's, I'm game to hear more. Doesn't "affinity" have something to do with attraction? Is this exercise appropriate for a work environment???

Maven: Sometime I'd like to get a look into the depths of your conscious as I am sure there are some scary things in there.

Motley: Get on with it.

Maven: Ok. The affinity comes from the fact that you are going to generate a lot of ideas and examine the relationship between those ideas. First you need to stock up on Post-It notes, find a room for the team, invite a few people, and schedule an hour for the exercise.

Motley: What are the Post-It notes for?

Maven: You will soon see.

Motley: Hmmm… I think I'll have to bring back a few pads that I, um, borrowed for home use because our supply cabinet is running low. How does the exercise work?

Maven: The first part of the exercise is to define the topic to be brainstormed. Provide an overview to the team on the problem for which ideas will be generated.

Motley: Ok - we are generating a checklist for reviewing production-quality code. Next?

Maven: Great. Individuals generate as many ideas as possible. An easy way to do this is by presenting each person in the room with a stack of Post-It notes to be used for writing down one idea per note. Give the team about 15 minutes of silent time to write down as many ideas as possible. Write down an idea, rip off the Post-It note and put it beside you, then repeat. There are no wrong answers, we don't worry about duplicates between individuals, and there is no discussion at this point. Anything goes.

Motley: One idea per Post-It note. Everyone is quiet, heads-down and writing for about 15 minutes, or presumably until they run out of ideas.

Maven: Exactly. Once all of the ideas have been exhausted or the time limit exceeded, everyone posts their ideas on a flat surface (wall, whiteboard, paper, etc.). If two people come up with the same idea, they overlap the Post-It notes. The overlap provides a real-time filtering of duplicates. I recommend using a Whiteboard for this if possible so that you can draw on the surface.

Motley: What if the ideas are similar but not identical?

Maven: The next step after everyone has their Post-It notes on the wall is to find the relationships between ideas and categorize the ideas together. Create logical groups based on the types of ideas that were generated. For example, if you are creating a code review checklist, there may be several items related to security like checking for correct buffer usage. Group those items together, draw a box around them, and label the category/theme. The category helps provide context around the ideas, and may even be a checklist item in itself if the ideas within the category are too specific or really represent one central idea.

Next, you want to discuss the ideas to ensure everyone understands the complete set of notes. Because the Post-It notes typically do not contain much text, some explanation is required from the authors. The explanation may also lead to a few more ideas, so as people in the group have them, they can post a few more on the board.

Motley: What about the dumb ideas - presumably we don't want to waste time discussing those?

Maven: Encourage people to be as creative as possible and think outside the proverbial box. There are no "dumb" ideas. If, in your head, you feel an idea is not feasible, keep it to yourself. It will come out later in the exercise.

Motley: We have the ideas generated and displayed, and we have an understanding of what they are. This still doesn't solve one of our big problems - too many ideas posted and either no idea how to prioritize them or some disagreements on how to do so. This technique looks great for generating the ideas, but prioritization is still an issue.

Maven: I'm not done yet! Of course, the members of the team may have differing views on which are the highest priority ideas. We discussed previously that checklists should be kept to around a page in length to be useful. Chances are good that after an exercise like this you have multiple pages worth of ideas. The group must narrow them down.

Motley: I'm waiting…

Maven: Here is a good technique: make sure everyone has a pen or marker. Each person gets seven votes to use how he or she pleases. The votes can be distributed across the ideas in any way. For example, perhaps Marcy really wants to see two of the items on the team checklist - she can put 4 votes on one idea and 3 votes on another. If Mark sees 7 ideas that he feels should be on the checklist, he can put 1 vote on each of 7 ideas. Voting is typically done by placing one dot per vote directly on the Post-It notes. Not surprisingly, we call this "dot voting."

Motley: Simple but effective. After everyone has had all their votes, we tally up the total number of votes for each of the items and take, say, the top 10 for the checklist.

Maven: You got it. Those top 10 will be the ideas the team is most concerned about for code reviews. Notice, too, that the ideas that are not feasible get left out, but no one feels bad because someone insulted their idea. The exercise allows us to wade through a large number of ideas, prioritize them appropriately, and produce a result that the team is happy with.

Motley: Awesome - we're done!

Maven: Not quite. Someone then needs to take action and write down the checklist in whatever form your team uses (e.g. wiki, InfoPath form, etc.), ensuring it is made available to everyone in a place they have access to. The exercise was useless unless we capture the results and the checklists is used by the team.

Motley: That goes without saying. I could see us using this technique for lots of different kinds of brainstorming activities, from where to go for dinner to what weekly practical joke to play on you.

Maven: Please, no more practical jokes. I'm still trying to get that tree sap off my jeans.

______________________________

Maven's Pointer: There is no magic to using 7 votes for dot voting. Using an odd number helps people prioritize more than using an even number that they can split evenly. For larger groups of people, you may want fewer dots, like 5 or 3. For smaller groups of people, stick with 7 or 9 or larger as there will be fewer dots to go around due to less people in the exercise. The risk with fewer people, however, is that one person can manipulate the results a little more by putting the majority of their dots on one item.

Maven's Double Pointer Indirection: When the exercise is complete, take a digital photo of the whiteboard containing the Post-It notes in case you want to review the session later or see what other ideas were generated should some items start disappearing off the checklist as they are no longer frequent errors.

Maven's Triple Pointer Indirection: Remember the discussions on Human Performance Technology. You can use the Six Boxes to guide people's thinking a little bit more in various areas that may be relevant to your problem, particularly if the number of ideas being generated by the team is low.

Maven's Resources:

In-depth description of affinity diagrams: http://www.mindtools.com/pages/article/newTMC_86.htm

Motley says: "Checklist? We don't need no stinking checklist!"

James Waletzky — Tue, 04 Mar 2008 19:29:00 GMT

Summary

Motley: Checklist? We don't need no stinking checklist! Checklists are for anal people with short memories.

Maven: Checklists are very useful for activities like code reviews, where it is difficult to remember everything we need to look for. Good checklists are short, consistent, easy to understand, accessible and provide support information. Personal and team checklists are recommended.

______________________________

[Context: Maven spent the morning looking over the code reviews in his Inbox and noticed a few odd things about one of Motley's code reviews from earlier in the day]

Maven: I was just looking over the code you sent out for review earlier in the day. Nice job on finding those errors you found. There are a few more things I might point out, however.

Motley: Did I miss a few? Hard to believe as I am the king of code reviews. I must have been having a bad morning. What did I miss?

Maven: Let me ask you this first: when you look for issues in a code review, what kinds of things do you look for?

Motley: The usual stuff - memory leaks, invalid usage of pointers, failure to check input parameters - that kind of thing. That covers the major ones.

Maven: What about issues like commenting, proper use of exceptions, freeing up resources, ensuring strings are localizable, assertions are used appropriately, unit tests are created, and checking for buffer overruns? I could go on and on.

Motley: I usually remember to check for those kinds of problems. Depends on my mood.

Maven: Do you ever use a checklist?

Motley: Checklist? We don't need no stinking checklist! Checklists are for anal people with short memories. Checklists will just slow us down. Plus, they get out of date quickly in my experience. Better to just get into the habit of knowing what to look for and then just do it!

Maven: Let me ask you this: do you remember everything you need to buy at the grocery store?

Motley: The common stuff I buy from week to week? Yes.

Maven: What about the one-off groceries? What if you want to bake a cake and you don't have any flour, which is not something you buy from week to week?

Motley: First of all, think about what you just said. Do you think I would ever be caught baking a cake? Wise up, Mave.

Maven: Good point. Sorry. The point is that for things you are in the habit of, your brain just automatically remembers them. I always know I need fruit when I go to the grocery store. However, for things that I don't buy regularly, like chocolate chips, I need a list.

Motley: Do what you need to do. I still try and get into the habit of looking for certain items in a code review, though, so I can avoid a checklist.

Maven: Getting into the habit is great, but with all the things a developer has to keep in his or her head, it's hard to remember everything when coding.

If we sat down and brainstormed every characteristic of production-quality code, we would probably fill a couple of whiteboards. Unfortunately, the human brain only retains about 7 +/-2 concepts in short term memory at a time. That's why phone numbers are seven digits (forget the area code, which is pretty static). For concepts outside of that core 9 - to be optimistic - we really need some tools that help us with reminders.

Motley: There are certain things that I "just remember", like checking for null pointers. I don't need a checklist to help me with that.

Maven: I agree.

Motley: You agree?!? Then what's the point?

Maven: For concepts that really are habit and don't require any extra effort to think about, consider not putting those on a checklist. The checklist is a helpful tool to ensure you remember those concepts that have not become habit. Additionally, if the team, for example, always remembers to pull strings out of resource files to help make the code localizable, then get that off the checklist and replace it with a reminder to check for a mistake that the team often makes. The highest priority items for the business or the release are also good candidates.

Motley: So now you are talking about maintaining checklists, which I indicated is cumbersome.

Maven: Keep checklists in a simple format that everyone can easily edit, like a wiki. Sure, you have to spend time maintaining them, but if it saves even one bug from shipping, it has just paid for itself 50 times over. Even better, you might want to have your personal checklist in a tool that you use every day and all day. For example, I noticed you check your e-mail a lot - you could have your checklist inside Microsoft Outlook as a task list. There it is easy to maintain and always available. The wiki works just as well though, particularly for team checklists.

Motley: Now you are talking about more than one checklist? It's getting worse.

Maven: A good practice is to have a team checklist and a personal checklist. The team checklist contains items that you want everyone to look for in a code review. It typically contains the most frequent errors made by the team, or major areas to focus on. Your personal checklist contains items specific to you, such as things you often forget to do or look for. For me, I often forget to make liberal use of assertions, so my checklist helps remind me to code defensively. A best practice is to use your person checklist to review your code prior to sending it out for review.

Motley: So I guess I start with the stuff I missed from this mornings code review. I am just going to keep it in a simple text file on my desktop where it is easy to find and edit.

Maven: Format is not important - whatever works for you. Here are a few more characteristics of good checklists:

Keep checklists short. By "short", I mean one page or less. Big long multi-page checklists do not do much good as they become unwieldy to use. Only have items that are the most important to remember.
Use consistent phrasing. Checklists with inconsistent phrasing are hard to use. Settle on one grammatical style. Some examples:

Affirmative statements with the "do" action verb: "Do free up any allocated handles."
Ask questions: "Are all pointers freed?"
Make direct statements: "Function header comments are accurate."

Mixing styles makes the checklist difficult to read.

Make checklists easily understandable. Avoid too much jargon unless the terminology is known by everyone on the team. Keep checklists simple.
Ensure checklists are accessible. For team-based checklists, in particular, make sure they are easily accessible and updatable. Put them on a team web site in an easy-to-find location. If they are hard to find, no one will use them. If they are hard to edit, they won't be updated.
Have multiple views, with supporting information. A good checklist actually supports multiple views. You may have a quick and dirty short form to act as your primary list of reminders, and you may have a more complete and verbose view with explanations for each checklist item. That way a new member of the team can easily get familiar with each item in detail. Links to resources for more information are great additions.

Motley: As always, you have lots of rules for everything. Don't you get tired of being so thorough?

Maven: Well, no. Ha ha ha. These "rules" are basically some intuitive guidelines (softer than a "rule") to help create a good checklist. For your personal checklist, you can do what you want as long as they help with the main goal of reminding you of important items.

Motley: Fine, fine, fine. Based on my poor code review performance this morning, I'll give it a shot. I'll create a quick checklist for code reviews.

Maven: Great! Checklists are also great for other activities, too, like design reviews, unit tests, code check-ins, and many other development activities.

Motley: Let's not get carried away. I said I'll create one for code reviews.

Maven: Ideally, you also think about building tools that automatically validate your checklist items without human interaction. That's what a tool like FxCop does - it takes a big checklist like the .NET Design Guidelines and automatically ensures your code satisfies those guidelines. Automated checklists are the best kind!

Motley: I'm always up for writing code to remove more manual labor. Let me give that some thought.

______________________________

Maven's Pointer: One interesting format for checklists is in XML. With XML you can apply any type of XSLT transform to the data to generate different views. You can have one view that is a summary of the checklist item, and another view that contains all of the details. Each view is based on the same central XML file. Developing checklists using a tool like Microsoft InfoPath abstracts away the XML data format and still allows multiple views. Store the XML on a Microsoft SharePoint server to make them centrally accessible, and you have a great checklist solution.

Maven's Resources:

Code Complete, 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.

There is a wealth of checklists for software developers in Code Complete. Check the end of each chapter for useful checklists on coding, commenting, assertions, code reviews, and much more.

Checklists.com, http://www.checklists.com/

A nice set of general checklists

How to Create a Better Checklist, http://hwebbjr.typepad.com/openloops/2005/09/how_to_create_a.html

A more detailed summary of characteristics of good checklists

Motley says: "Ugh, not again..."

James Waletzky — Tue, 13 Nov 2007 18:47:00 GMT

Summary

Motley: Developers keep making the same mistakes over and over. Can't they learn?!? Some kind of punishment is in order, like no more free soda!

Maven: Don't just address the symptom - get to the root cause of a problem. Leverage the "Five Why's" to find root cause and come up with solutions that are realistic, cost-effective, and within your control.

______________________________

[Context: After taking part in a few structured code reviews, Motley is getting frustrated by the patterns of errors he is seeing]

Motley: You know, I'm almost sorry we started this code inspection thing. As the senior developer, I have been participating in quite a few of them. It has really come to light that (a) human beings are creatures of habit, and (b) we don't learn from our mistakes! Every review that we have finds the same kinds of errors. Can't people learn?!?!? Perhaps we should revoke the free soda as punishment.

Maven: I feel for you. The real question is, what can you do about it? Taking away soda is not the answer.

Motley: I can think of lots of other things too - beating on people, making them wear dunce caps, cutting their pay, -

Maven: Whoa! Let's try to be more constructive. How can we proactively cut down on the problems that occur? If you keep getting flat tires on your car, you could first consider fixing the symptom. Patch the tire. Buy run-flat tires. Both of these solutions won't solve the real problem, and you'll still keep getting flat tires.

Motley: Well, duh! You want to figure out why you are getting the flat tires. This is just common sense. Is it a fender rub? Do I have a cheap brand of tires? Is there something protruding from the wheel? Am I driving in an area that is prone to junk all over the road causing flats? Could be a multitude of reasons.

Maven: Exactly! Why not apply this same principle to software bugs? If we can figure out what the root cause for some of the bugs we find, perhaps we can prevent the problem from occurring at the source and not make the same mistakes again.

Motley: Well, um, of course. There's nothing to say, however, that the developers will actually address the root cause and not make the same mistakes. And with software, it is usually quite difficult to actually determine the real reason for a bug.

Maven: Well, once you find the root cause, there are some creative solutions you can usually apply to help developers do the right thing. Before we address that, let's talk about how to do root-cause analysis. One simple way is to leverage "The Five Why's". The idea is that if you ask "Why?" five times you'll ultimately get to the root cause of an issue.

Motley: Why five? Oh, wait. I know. If someone keeps asking me "Why?" after about 5 times I'll get frustrated and provide a knuckle sandwich for their effort. Five is about all I can take.

Maven: Actually, I don't know why it's five. I treat it as a guideline anyway, and not a hard and fast rule. Want to try an example?

Motley: I'll humor you, just please don't annoy me.

Maven: The foreman of an assembly line shuts down the line because Mark got injured. The manager could just start up the line again or he could pursue the cause of the shut-down. You play the manager.

Motley: Ugh. Fine...

Maven: Well, ask the question!

Motley: Ok, why did the line shut down?

Maven: Very good - you are a good actor! The line shut down because Mark slipped in a puddle of water and hurt himself.

Motley: Why is there a puddle of water on the floor?

Maven: Very good! There is a leak in the roof and it rained last night.

Motley: Jeez. Why is there a leak in the roof?

Maven: The rain storm last night also exhibited high winds and some tree branches were blown into the roof.

Motley: Why was it windy?

Maven: Ha. Ha. Nice. The key is to keep the "why" questions realistic. If you were to ask me "Why did the tree branches poke a hole in the roof?" I might answer because we have some trees that are too close to the building. At this point, we're probably done. Our root cause for the line being shut-down and Mark getting hurt was because trees are too close to the building. These two seem unrelated at first glance, but once you delve into the details, it becomes clear how to fix the root issue.

Motley: That's a pretty ridiculous example!

Maven: Well, it's fictitious, but the same technique can be used with bugs. Keep asking "Why?" until you get to the root issue for the bugs.

Motley: Okay, back to my previous question. Once we find the root cause we can tell developers how to fix it, but there is no guarantee that they will.

Maven: Yes, sometimes that is true. However, there are certain types of solutions we can put into place to minimize the odds of the bugs creeping in again. Here are some examples:

Run tools proactively: Before we check-in code, we can run static analysis tools like FxCop to proactively find bugs on our behalf. FxCop is great at finding security and other quality issues. Even before check-in we should run this kind of tool prior to code review to nail the obvious ones before we use up the time of the team.
Update tools to find our common bugs: Most tools like FxCop have an extensibility mechanism. If we consistently make the same error and the tool currently does not look for it, consider creating an add-in that finds the issue in code.
Update coding standards: The developers should all be very familiar with our team coding standards. Make it mandatory reading. Add patterns of errors to the standards document to ensure new developers in particular do not make the same mistakes.
Update team checklists: Checklists are important for reviews. If a mistake keeps happening, add it to the checklist that developers use during code reviews.
Get training: If mistakes are happening in an area that team isn't very familiar with, consider books or training to fill the gap. If this is our first foray into multithreading, for example, take a course on how to develop concurrent applications.
Change development processes: Perhaps the steps that are required before code check-in need updating. For example, consider updating the process to require a mandatory personal checklist-based review before a team review before using the time of others.

Commonalities between all of these solutions are that:

They are realistic. Doing a rain dance most likely is not going to solve the assembly line issue.
They are cost-effective. Building a dome around the factory is not feasible.
They are within your control. Controlling the weather is not within your control.

Motley: We obviously cannot afford the time to do root cause analysis on every bug!

Maven: No one says you need to do this for every bug. Concentrate on using this technique for patterns of bugs. One place to make use of bug root-cause analysis is for a few minutes at the end of an inspection. Have a quick chat that covers the following:

What went well with the inspection?
What could we improve the next time we do an inspection?
What is the root cause of some of the major issues we found?
What can we do to prevent those issues from re-occurring in the future?

Motley: I am always up for preventing boneheaded mistakes from appearing over and over again. We'll try it at the end of the next inspection to start, and then perhaps think about some of the other patterns of bugs we've seen over the past couple of weeks and address them.

Maven: Awesome! I think the team will be better for it moving forward...

______________________________

Maven's Pointer: Root cause analysis is used in many different industries from accident investigation to automotive assembly lines to healthcare. Addressing the symptoms of a problem are usually temporary fixes. Individuals and businesses that believe in continuous improvement must dig deeper beneath the surface to ensure the real problem is being addressed. Fishbone diagrams (a.k.a cause-effect diagrams or Ishikawa diagrams) are a tool to help you visualize and delve deeper into problems to discover root causes. I prefer a classic mind map, but the principles are the same.

Maven's Resources:

FxCop: http://msdn2.microsoft.com/en-us/library/bb429476(vs.80).aspx
The Five Why's: http://en.wikipedia.org/wiki/5_Whys, http://software.isixsigma.com/library/content/c020610a.asp
Fishbone diagrams: http://en.wikipedia.org/wiki/Ishikawa_diagram, http://www.isixsigma.com/library/content/t000827.asp (with Word and Excel templates)

Motley says: "I just get picked on when my code is reviewed"

James Waletzky — Tue, 06 Nov 2007 18:57:00 GMT

Summary

Motley: I'm not sold on this inspection thing. There were too many problems during our session. Plus, everyone just picks on the author and they hate having their code reviewed. Back to informal peer reviews for me.

Maven: Follow inspection best practices: inspect limited amounts, don't inspect everything, use a facilitator, be hardcore, use a checklist, focus on the work not the people.

______________________________

[Context: Motley has just finished his first code inspection with the team and is explaining to Maven how it went]

Motley: Well, we did a code inspection to try out the process we talked about. I gave the team an overview of the process using the diagram you gave me. We picked a bunch of code to look at from multiple owners, everyone prepared and we did the meeting. As I suspected, it really wasn't much better than an informal peer review. I'm not sold on this inspection thing, and I'm not sure I would do it again.

Maven: Hold on! Before you make that conclusion, let's get to the root cause as to why the inspection was not as effective as it could have been. First of all, how much code did you pick and how long did people spend preparing?

Motley: We inspected about 1500 lines of C# code. I would say the average preparation time was about 45 minutes, and the meeting took about 45 minutes as well.

Maven: There's problem number one. Every study on inspections indicates that less code in one session is better. In terms of absolute numbers, the guideline for preparation is somewhere between 200 and 500 lines of code (LOC) per hour . Generally, developers cannot maintain the "zone" of code reading and issue finding for more than 1.5 to 2 hours, so you should inspect approximately 400 to 1000 lines in one two hour session.

Motley: Are you serious?!? How would a company like Microsoft use this technique on all 50 million LOC in Windows Vista?!?

Maven: They wouldn't. You don't want to inspect every LOC that the team writes. Pull out inspections in the following situations:

The first few LOC for each developer in a development cycle. Having the team review the first few LOC that each person produces helps nail some problems right at the start. If a junior developer does not understand how to properly throw and catch exceptions, for example, they are going to make the same mistakes throughout their development. Having a team-based inspection at the start gets them on the right path from the beginning.
Critical code. Code that sits on a network or other trust boundary (e.g. an ISAPI filter running in the IIS web server) is a great candidate for inspections. The reason is that the consequences of someone hacking that code can be severe, particularly if the code is running as SYSTEM.
Complex code. Complex algorithms and interactions (e.g. concurrent sections of a program) should be looked at closely by the team. In many cases, tools do not do a great job of finding issues automatically, and inspection is the best way to flesh them out.

The key message is that you should wisely choose code to inspect, and not count on inspecting all of it. You said average preparation time was 45 minutes, but was everyone prepared for the meeting?

Motley: Well, a couple people who shall remain nameless didn't spend much time on it at all. Coincidentally, they didn't report many bugs either.

Maven: If some people come unprepared, you have a few options:

Ask them to simply be an observer in the meeting. Since they will not have much to contribute anyway, you want to minimize their distractions.
If it happens multiple times, kick them out of the meeting with a clear message that they are not helping improve product quality, and that practice is not acceptable.
If people say they prepared, but you are skeptical, play a little game. In the code that is handed out for the inspection, inject a couple of bugs. Don't make them too difficult, but they should not be too obvious either. Good candidates are off-by-one errors and reversed Boolean values. Tell people how many bugs you put in, and challenge them to find those bugs.

What about the meeting itself? How did that go?

Motley: Not so good. We were all over the place. We had many tangential conversations where we tried to solve the problem we discovered. We also had trouble referring back to the lines of code that were troublesome and everyone had different copies of the code.

Maven: There's a few things we can do to help that situation.

First, all good meetings - and all good inspections - have a facilitator. With inspections this role is often called the "moderator." This person guides the meeting, collects the data and issues that are found, and keeps the meeting running efficiently. They have free reign to cut someone off that goes down a tangent.
Second, avoid discussing solutions. The inspection is all about finding problems. If you want to talk about solutions to specific bugs, put it on a list on the whiteboard (often called a "parking lot") and anyone that is interested can stay after the main collection meeting is over.
Third, it is important that everyone inspect the same copy of code. One way to do this is have the moderator print out the code for everyone. Inspecting code on paper removes you from the distractions of the computer. In addition, you ensure everyone has an identical copy, and by including line numbers, you have referral points for individual LOC.

If you follow those tips, you should have a better inspection. What kinds of problems did you find this time around?

Motley: Not much. We found a few cursory bugs, but nothing major. My gut tells me that there are more bugs lurking, but I cannot say for sure. The problem was that no one really knew exactly what to look for.

Maven: When preparing, you really need to focus. Be as thorough as possible when finding issues. Everyone should be familiar with the problem, team coding standards, and team development processes. You want to point out every little thing you find - be hard core. Don't just focus on pure logic errors. Anything that, as a quality-oriented developer, you would like to see fixed, point it out. By following this practice, we lead ourselves towards better quality products.

In addition, do an analysis of where your team makes frequent mistakes. Those items should be put on a checklist, and everyone should leverage that checklist while preparing for the inspection.

Motley: Everyone is familiar with the .NET Design Guidelines, but we need to be a little more hard core. It would also help if we ensure tools like FxCop are run prior to the inspection too. I assume that's why you have been using this word "issue" instead of "bug" over the past few minutes? You want devs to think more widely about the mistakes they look for? Comments that do not match the code count?

Maven: You are a very bright guy, Mot. That's exactly why I use the word "issue". One last thing: you mentioned you used code from several authors. Try and restrict the code you look at to one person to prevent context switching. Was that person happy to have their code inspected?

Motley: Quite the contrary! He felt he was picked on the whole time and even told me afterwards he wants to avoid this whole process in the future. I told him to quit being a baby and that it's his job.

Maven: Whoa. Probably not the best response. Anyway, authors should want to have their code inspected. It's a great learning experience. Inspectors can encourage this by avoiding a threatening environment. Direct comments at the work item and not personally at the author. Also, instead of making statements, ask questions. For example, "I'm not sure, but is there a memory leak on line 34?" is better than "There's a leak on line 34. What were you thinking? Did you not run the tools? Wise up dude!" Asking questions is less confrontational and who knows, you may be incorrect about your assertion. Let the author come to the conclusion herself. It's a better learning experience that way anyway.

Motley: That's a lot of stuff to remember.

Maven: It's not that bad. Just remember a few key points:

Inspecting less is better
Be wise about what you inspect
Use a facilitator in the meeting
Be hardcore about the issues you identify, and use a checklist
Remove the people from the problem (i.e. avoid threats, ask questions, focus on the work)

Motley: Ugh. Back to the drawing board. I can think of some nasty multi-threaded code that perhaps we should inspect next. I'll try this stuff out again if my team doesn't hate me and get back to you.

______________________________

Maven's Pointer: According to Steve McConnell in Code Complete 2nd Edition, inspections are one of the most effective practices for finding defects early in the software lifecycle. In fact, studies have shown that inspections find 45% to 75% of all defects, with other practices like unit testing and system testing falling below these values. Your experience may vary, but selectively using inspections on requirements specifications, designs, code, test plans and other artifacts will save you time in the long run.

Maven's Resources:

Design Guidelines for Developing Class Libraries (.NET): http://msdn2.microsoft.com/en-us/library/ms229042.aspx
Code Complete 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.

Motley says: "Inspections suck... time that is"

James Waletzky — Tue, 30 Oct 2007 17:47:00 GMT

Summary

Motley: Inspections seem a bit heavy. They suck up too much time for little gain.

Maven: Inspections are actually a fairly light-weight structured process that find more issues compared to ad-hoc peer reviews and team reviews.

______________________________

[Context: Maven and Motley are continuing a conversation about code reviews, now with a focus on inspections]

Maven: Nice restaurant choice, Mave. Although I am a little worried about what this Mexican food will do to me this afternoon.

Motley: Let's get this conversation over with quick so I don't have to be around you this afternoon! You were saying something about "injections" or something? Hopefully they aren't lethal!

Maven: Inspections. Inspections are a structured team-based review process for code, design, requirements, and any other artifact. Inspections are much more effective than ad-hoc peer or team reviews. The process is very simple - six steps, of which 2 are optional. Here is the overview:

Motley: That still looks pretty formal and heavy to me! Just the fact that you have a diagram describing all the steps is a little scary. I'm worried this practice is going to suck up a lot of time.

Maven: Ah, don't be afraid. The process is actually quite simple and not unlike other code reviews. It's just a little more structured with a couple of key practices to ensure that reviewers are highly effective. First we start with planning.

Motley: Yeah, the process is so complicated I bet you need a plan just to pull it off!

Maven: Please! Put that chimichanga back in your mouth. Planning is really just common sense. You figure out what you are going to review, who should be involved, and plan the meeting(s). Not much to it. Next comes the overview.

Motley: You mean the 2 hour overview meeting where you are going to describe this process to everyone?

Maven: Always a wise guy. The overview is actually optional. You want to ensure that everyone involved in the inspection is sufficiently familiar with the work item to be inspected to do a good job. An overview is an ad-hoc meeting that usually doesn't last more than half an hour. If you are doing a code inspection, the author of the code gives an overview of what the code is doing (i.e. the requirements) and how the code was designed. There may also be a discussion of areas to watch out for, assumptions that were made, and it's a chance for inspectors to ask questions. Next, we prepare for the inspection.

Motley: What's the point? Shouldn't we just get together in a meeting and review the work?

Maven: Preparation is actually the most important part of the process. Here, each individual inspector takes some personal alone time to scour the material for issues. With code, an inspector looks for bugs, maintainability issues, and anything else a quality-oriented developer would commit to fix. Just getting together in a meeting to review the work on-the-fly is not effective. The real issues are found during individual preparation. After everyone prepares, THEN you have the meeting and collect the issues from everyone.

Motley: Hmmm… hopefully that meeting is extremely quick! If everyone does most of the work on their own, then we won't need much time for the meeting. In fact, why do the meeting at all?

Maven: The meeting is quick, but it is important to discuss some of the issues with the author. Treat it as a big learning experience for everyone. Plus, some of the things that people find may need clarifying questions and may not actually be issues at all. The meeting is a chance to validate that everyone has prepared, collect some metadata about the meeting (such as meeting time, number of issues found), and learn from one another. After the meeting is over, we take a few minutes to talk about the root cause of some of the issues that were found, and try to address them such that we don't make the same mistakes in the future. After the meeting, some rework is usually necessary.

Motley: Well, duh! If you find bugs you need to fix them.

Maven: Exactly, provided we are not at the point in the development cycle where only the really major stuff gets fixed. In any event, after most reviews, the author goes away and follows the processes of the team to fix the identified issues. After that happens, there may be some follow-up.

Motley: Because we don't trust the author to make the correct fixes? That's babysitting!

Maven: It depends on the developer. This step of the process is another optional one. We may want to have someone follow up if the developer responsible for the code is junior or new to the company. Another reason for follow-up, and even re-inspection, is if many major issues were found.

Motley: And what's this process improvement thing down the side of the diagram?

Maven: Let's save that discussion for another time.

Motley: So I bet your next request is that I just go off and try this. Fine. I'll humor you and give it a shot on some code. I'm not convinced it will help us, but I'll give it a shot.

Maven: Great! We can pick some code to inspect this afternoon and get it going.

Motley: Hold on, Burrito Boy. I don't want you anywhere near me this afternoon. I'll try it out with the team and let you know how it went.

______________________________

Maven's Pointer: The lean way of thinking is that inspections should not be necessary if we build quality into early parts of our development. Unfortunately this is just not feasible with software and other domains due to the human element. Think of inspections of code as similar to an inspection of your house under construction. Someone does the inspection that knows the domain, knows what to look for, and they make it a mandatory part of the house construction process. It is worth the small time hit to ensure a quality deliverable.

Maven's Resources:

Introduction to the Team Software Process, by Watts Humphrey, Addison-Wesley Professional, ISBN: 020147719X, August 1999.
Software Inspection, by Tom Gilb and D. Graham, Addison-Wesley Professional, ISBN: 0201631814, December 1993.

Motley says: "Quick peer reviews are sufficient. Team reviews? Waste of time."

James Waletzky — Tue, 23 Oct 2007 18:21:00 GMT

Summary

Motley: Quick ad-hoc peer reviews are sufficient to nail bugs early in the development cycle.

Maven: A more structured team-based code review process can lead to more issues found.

______________________________

[Context: Maven overhears a code review that is happening in Motley's office]

Maven: Hey Mot. Were you and Marvin just having a code review? I overheard a little bit of it while you were at your desk.

Motley: Yeah, we just kicked butt - found a few nasty little problems, although nothing too significant due to my superior coding abilities.

Maven: I expect nothing less! I was just wondering what your process is for code reviews. Can you enlighten me?

Motley: Not really a process per se - we want to keep things nice and light. Code reviews are known to be an effective defect prevention technique so of course we leverage them prior to checking in code. When I am ready to check-in, I call over a fellow developer, I pull up a quick WinDiff display of the code that has changed, we do a quick over-the-shoulder check at my desk. I then fix whatever bugs we identify, and then I am good to check-in. We usually catch a few things on a code review, particularly when I review someone else's code. I am not a senior dev for nothing!

Maven: I applaud your use of code reviews to keep quality high! But-

Motley: I knew it! There's always a "but" with you. For your information, I've been doing some reading on code reviews and I think that given the amount of effort we are spending on the review relative to the number of bugs that we find, we are being effective. We focus on the diff, fix all the bugs on-the-fly, and move on.

Maven: Don't get me wrong - I think what you are doing is great. For many check-ins involving trivial changes that process can work. What do you do for more complicated check-ins like a few files of new code?

Motley: Same thing. It just takes a little longer.

Maven: Do you typically find more bugs than for other reviews?

Motley: I would say we find about the same number per thousand lines of code.

Maven: And do you still find problems later on in the testing cycle?

Motley: Of course! You can't expect us to find everything in a brief little code review. I still fee like we're effective though.

Maven: Informal, less structured peer code reviews as you describe are great for the following situations:

Small bug fixes
Minor code changes where the author is already very familiar with the code
Changes of very low complexity
Changes with few possible side-effects

More formal, structured, team-based code reviews are more suitable in these situations:

Large bug fixes
Scenarios where a lot of new code is being added to the system (e.g. a feature check-in)
The first bit of code a developer on the team writes
Changes on a trust boundary - i.e. security-related changes
Complicated code changes, such as those involving concurrency

Motley: Whoa! You used a really bad word. The word "formal" really scares the crap out of me. Whenever some one uses "formal" to describe a process, look out. Those processes typically have high overhead and occupy a lot of time for - in my experience - little gain.

Maven: I hear you. In fact, I often agree. By "formal" here, I mean slightly more structured than just calling someone over to your office to quickly look at a bit of code. There is a process, but it's very agile and lightweight.

Motley: You also said "team-based". Why would I want to involve more than one person in a code review? That seems like a waste of time!

Maven: Studies show that team-based reviews that follow some kind of process - even lightweight - are the most effective defect finding techniques in the development cycle. You want to add people to your review to the point of diminishing returns. The ideal number is somewhere in the area of 3-4 reviewers. A more structured review tracks metrics over time that can help you determine how many people to include. It's all about being as effective as possible while not expending too much effort to get the job done.

Motley: More eyes, more bugs. I'm not sure I totally buy it. Why did you say previously that one of the scenarios where we should practice a more structured review is on the first bit of code a developer writes?

Maven: There is great value in that. The idea is that you get a few developers on a new project to review the code in detail for someone contributing their first few lines of code. A team-oriented review is a fantastic learning experience that gets the developer going in the right direction from the start. You see, humans are creatures of habit - if you make a mistake in your first few lines of code you are likely to repeat it in all your code. Fixing all of those problems expends wasted effort later.

Motley: Seems reasonable, but I could do that with just that developer and me.

Maven: You won't find near the amount of issues as a team of people. Use the technique sparingly, however. Microsoft, for example, could never team-review all 50 million lines of Windows Vista. You want to do team review in intelligent situations.

Motley: What else makes a team review different than an ad-hoc peer review? Right now I can't believe it would be that much more effective. And does this different kind of review process have a name?

Maven: A more structured review is typically called an inspection. Let's talk about it over lunch. To whet your appetite, here are some of the key principles behind an effective team-based code review:

A structured, repeatable, but lightweight process
Diligent individual preparation at a reasonable preparation rate
Non-confrontational meetings to ensure "threats" to the author are removed
A checklist reminder of the most important issues to look for
Metrics that help gauge the effectiveness of your meeting
A focus on process improvement to help prevent the same bugs from occurring again

Motley: My interest is slightly peeked. Where do you want to go for lunch?

Maven: How about-

Motley: Never mind - I'll pick. You drive.

______________________________

Maven's Pointer: The term "inspection" has been used throughout software engineering history to denote a more formalized code review. Invented by Michael Fagan at IBM in the late 1970's and used by many companies, inspections typically yield 45-75% (Code Complete 2nd Edition) of the defects injected into software throughout the development cycle. There are several different inspection processes, but the one we will focus on here has roots in the Team Software Process (TSP) with some modifications for efficiency.

Maven's Resources:

Peer Reviews in Software, A Practical Guide, by Karl Weigers, Addison-Wesley, ISBN: 0201734850, December 2001.
Code Complete 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.