Progressive Development : quality

Motley says: “Slow down to go fast? Do I look like a tortoise?”

James Waletzky — Wed, 22 Apr 2009 08:23:00 GMT

Summary

	Motley: There is no harm shortcutting a few steps of development to meet a deadline. Some up-front check-in steps just take too much time.
	Maven: Slow down to go fast. Obey check-in checklists or you will pay for it later in the development cycle. Pre-check-in tasks such as code reviews, unit testing, static analysis, and buddy builds of various flavors must not be avoided.

______________________________

[Context: The code check-in deadline is approaching. Motley has code that is in progress and he is hurrying to get his change in prior to the build snap]

Maven: Hey, Mot. How's it-

Motley: Quiet! I have to get this code checked-in by the end of the day to meet the deadline. I don't have time to chat and need to rush to get this code in.

Maven: Whoa - maybe we should chat. Please just give me a few minutes here - it won't affect anything in the bigger scheme of things. What do you mean you have to "rush"?

Motley: The daily build starts at 6pm and it's 5:35pm. I only have 25 minutes to get the code checked-in so I have to take a few shortcuts.

Maven: I cannot stand the word "shortcut". Care to clarify?

Motley: It doesn't really matter what you can and cannot stand. But since you asked, I am going to skip writing unit tests for this last class, I am going to forgo code review because my code is always perfect anyway, and I am going to skip running FxCop just this once.

Maven: Have you ever heard the phrase, "Pay me now, or pay me more later"? What you are doing is akin to racking up credit card debt. On the outside it looks like a $5000 purchase, but in the long run it is going to cost you a lot more in interest if you do not pay off the debt as soon as the bill comes due.

Motley: Oh, no. More Maven analogies. I can do without those. And by the way, Boyz II Men called - they want their shirt back.

Maven: Always insulting my clothes! This blue and pink sweater combo isn't that bad, is it? Anyway, taking shortcuts now when the code is cheap to fix pays off in the long run and leads to quicker ship cycles and shorter bug-fix cycles later in the development cycle.

Motley: How so? Bugs are bugs, I need to fix them anyway.

Maven: It's much cheaper to fix bugs now than later. Fixing them later usually involves someone else creating and maintaining bugs in a bug database, fixing them when the code is no longer in context and fresh in your memory, and the tendency to bolt-on fixes and shortcut fixes. Additionally, if the test team does not find an issue, the cost of fixing it post-release is large due to the overhead in working with the customer, the bug getting back to the product group, deploying the fix, the negative press, and other factors. If you find an issue that modifies the design later, you may have to change a lot of code possibly destabilizing the product.

Motley: Yeah, yeah - the cost of change curve. I know it.

Maven: Then you know that you shouldn't rush your development. Would you rather fix bugs later when they are more expensive or now when they are cheap without the overhead of a post-check-in bug? Leaving bugs until later forces long "stabilization" periods on the team, which, in my opinion, are evil and can be avoided by a diligent team.

It is well worth pushing back on the deadline to ensure that you can adhere to the following code check-in checklist:

Unit tests. We have talked about unit testing at length in other discussions. This is the number one quality-driven developer technique. You must have a set of unit tests in place that exercise all of your new code and you can check-in when all the tests pass. This items should be bolded on a checklist.
Code coverage. Hand-in-hand with unit testing is code coverage. Aim for 70%+ code coverage for all of your code. More important than the coverage number is the feedback you receive from the coverage tool regarding source lines that were missed in testing so that you can add new tests.
Debug and retail builds. A debug build is generally different than a retail build, particularly when you make use of debug-time primitives such as assertions. You need to build both flavors to ensure the build succeeds either way. Broken builds are not tolerable as they block the entire team. In fact, do a clean build (vs. incremental build) to be safe.
Static analysis. A static analysis tool like FxCop finds bugs for you. Per the cost of change discussion, why not fix them now instead of later when it is more expensive? Why risk security defects escaping your defenses and being released into the wild where hackers and customers will find them?
Reliability tools. Running a tool like AppVerifier is more relevant to C++ applications. Running this tool is a must prior to check-in as it will help identify heap corruptions, improper use of handles, and misuse of critical sections. Additionally, for native code it will point out memory leaks.
Warning level. Crank up the compiler to generate warnings at level 4 (or whatever is highest for your compiler) to ensure that you have addressed all possible compiler-identified issues. Get into the habit of perpetually building at the highest warning level. Get clean and stay clean.
Other tools. Run any other organization-wide tools such as a copyright header checker, XML documentation checker, or coding style checker (e.g. Source Analysis) prior to check-in. You don't want to be the grunt fixing a ton of these issues later. Trust me on this one.
Code reviews. All code should be at least peer-reviewed, with larger changes (e.g. new features) being put through a code inspection. Having at least one other pair of eyes look at your code is likely to find many issues before even the test team gets their hands on the code.
Change list description. Include complete descriptions for check-in change lists, including how you tested the code, why the change was made, how you built, possible regressions, and other risks.
Buddy build. To ensure you did not forget to add a file to your change list and to protect against other bad assumptions, package up your change and pass it to a buddy for her to build. Most source control systems allow you to package up a change list for distribution. If you have a continuous integration system that builds changes before check-in, this is less necessary.

Motley: You cannot possibly be serious! If I do all that now there is no way I would ever make the deadline, let alone make a check-in.

Maven: All of the above is necessary, particularly in a large, quality-focused organization. We want to exterminate bugs before they are even checked-in. The checklist above helps with that.

Motley: Can you say "Sloooooooooooooowwwwww"? I knew you could.

Maven: Get in touch with your inner tortoise. I heard that in a TED talk on Slowing Down in a World Built for Speed by Carl Honore. Going a little slower up front will save time in the overall development cycle, and I would claim that you ship sooner as a result. Of course, that was not the specific theme of the talk, but the lesson can be applied here. Also, be pragmatic. Perhaps you avoid a step or two for small bug fixes.

Motley: One thing I do not quite buy is the lack of long stabilization periods at the end of the development cycle. We still need to put all the pieces together!

Maven: Ultimately we want to avoid long stabilization cycles and instead be stable when you check-in. A tail of the development cycle needs to be dedicated to integration and overall acceptance test validation, but is not the same thing as "stabilization". We talked about this before in our discussion on developer testing.

Motley: So you are telling me that I need to pay my boss a visit as soon as possible to tell him that I am going to miss the deadline? He is going to kill me.

Maven: Well, seeing as you only have 5 minutes to the build snap-

Motley: <POW>. You deserved that shot in the nose for wasting all my time and missing the build snap!

Maven: Ouch! Okay, I deserved that. But seriously, your boss is a reasonable person and will understand. Firstly, you are communicating bad news as soon as possible (well, not really, but let's pretend), which all managers appreciate. Secondly, present it such that you are focused on quality and want to avoid taking shortcuts, and that you feel this course of action is best for the product. I will bet you lunch that he understands.

Motley: There are some days where I wish I stayed in bed...

______________________________

James' Pointer: The checklist items I mention above are from a real team in at Microsoft in Windows Mobile. Quality is our #1 value as a team. We want solid check-ins that adhere to the checklist to ensure we don't have a long tail of bugs later. Yes, it is a lot of work but ultimately saves time in the long run. Slow down. Don't rush. Nail the checklist on every check-in. You will thank yourself later. Okay, maybe not, but you should.

James' Double Pointer Indirection: A good developer knows how much work goes into each and every check-in and factors these things into their estimates. These things take time.

James' Triple Pointer Indirection: "Slow down to go fast" is real advice in car racing circles. Intuitively it does not make a lot of sense. However, if I go into the corner too fast I have to brake to avoid the wall and then take a lot of time and gas to come back up to speed exiting the corner. If I slow down on entry, roll through the corner and smoothly apply the accelerator on exit, I actually go faster even though I slowed down entering the corner.

Maven's Resources:

TED (http://www.ted.com): A great web site of some of the world's best speakers. There are talks on a multitude of topics. You are guaranteed to find something of interest.
AppVerifier (http://msdn.microsoft.com/en-us/library/ms220948.aspx): A mandatory tool for C++ developers on the Microsoft Windows platform.
Source Analysis (http://code.msdn.microsoft.com/sourceanalysis): A customizable tool that validates coding standards and is a good complement to FxCop.

Motley says: “Branches are for trees, not source code”

James Waletzky — Tue, 03 Mar 2009 19:08:00 GMT

Summary

	Motley: Branches are too complicated. The last thing we need is a copy of the code that has to be maintained in two or more places!
	Maven: Branches are source code copies with a tie back to a mainline set of code that allow for easy integration between copy and original set (and vice versa). Branches bring a level of isolation from mainline as well as keep the quality of mainline (and the daily builds) high.

______________________________

[Context: The company has been having lots of trouble with build breaks in mainline of late. General quality of the product mid-way through the development cycle is lower than expected]

Motley: Somebody is going to get crazy glue stuck to their cup! The quality of our daily builds has been crap lately. Too many people checking in lousy code. Too may conflicts. Not enough time to get our mainline build stable and of high quality before the next check-in goes in. Something has to be done.

Maven: Crazy glue? Ouch. If some moron did that to me I would pay them back big time - when they least expect it. Anyway, I agree with you - having a daily build of poor quality slows down everyone. The key is to isolate all the feature teams from one another giving them freedom to check-in yet keep the main build high quality.

Motley: Thanks for stating the obvious, Mave. Your haircut sucks. Now I stated the obvious too.

Maven: I thought I fixed that crooked sideburn! Ugh. There is a good way to accomplish this isolation. If you have a good source control system (like we do) such as Perforce or CVS, you can leverage a branch.

Motley: Branches are for trees. What do they have to do with source code? Should I start working outside in the shade?

Maven: Think of a branch as a more-involved copy of a collection of source code.

Motley: Why would I want to make a copy? Copies are fraught with problems, such as keeping multiple copies synchronized. And to think, I was expecting a good idea from you. Although pigs don't fly quite yet.

Maven: The digs keep coming! A branch is more than a copy. When you branch a source tree, you still maintain the tie to the original source code. Think of it like the trunk that a tree branch is connected to. You can bring forward changes in mainline to your branch, and bring back changes in your branch to mainline relatively easily. You can even work on the same source files in both places and merge the changes together in either direction.

Motley: And how would I do that? Keep them tied together, I mean.

Maven: You bring code forward to your branch from mainline and back to mainline from your branch through an integration, or more specifically, a forward integration (FI) and reverse integration (RI) respectively. Here is an example of a tree structure I used in a previous role.

Mainline: The main source code depot where daily builds take place from

Feature Branches: As many of these as necessary to support isolated feature teams

Feature Sub-branches: Use only as necessary (e.g. a developer making major changes as part of a feature team)

Motley: That looks more complicated your morning routine! Do the benefits of a structure like that really outweigh the costs of doing all those integrations?

Maven: As the consultant always says, "it depends". For a small overall team where not too many people are checking in to mainline, perhaps the structure is too much overhead. However, for larger teams, branches bring a level of isolation allowing features teams the freedom to do what they need to get the job done. In addition, a central build team could maintain mainline and have a very high quality bar for check-ins to mainline. Teams are only allowed to reverse integrate into mainline when they meet that bar.

Motley: And what does that bar typically involve?

Maven: It depends on the team. For us, we may have a quality gate in place that mandates the following when an RI is done:

All changes must have been code reviewed
All changes must have been gone through a test pass
Test automation is in place for new features (if applicable)
New features have been stress and performance tested
No memory leaks have been detected by our tools
FxCop has been run and is violation-free on all changed code
Unit tests have been executed and pass at 100%

You can basically go as thorough as you like. As you move up the branch hierarchy, the criteria to check-in gets lighter and lighter so as not to slow down the team. RI operations get stricter and stricter as you get closer to mainline.

Motley: Should every team have its own branch?

Maven: Teams making changes to an isolated part of the code base with relatively few check-ins may not need a branch. Teams working on radical and risky changes or in common parts of the code base would likely benefit from a branch. Keep in mind, though, that the more levels of branching you have, the more overhead there is for integration operations. I recommend trying to stick to 1-2 levels of branching at most.

Motley: What about builds? We lose the advantage of having a central build team generate a daily build?

Maven: Yes, that can be true. Hopefully you can take their infrastructure on a couple of machines that your feature team owns and duplicate the build there so that you have your own daily build and verification going for your feature. For small features, this may not be necessary. For large features, it is a huge benefit as you nail down problems quicker. If a build break happens due to a bad check-in, you catch it long before you hit mainline. As a result, you block fewer people. Everyone wins!

Motley: Ok, Einstein. I have a you now. What if feature team A is working on some changes that feature team B needs to make further progress? You cannot go across branches. Hah!

Maven: You have several options:

Feature team A can pack up the changes and pass them to Feature team B. Most source control systems support packing up a change list for distribution.
You can do what's called a baseless merge. I don't recommend doing this as it loses your source history. Essentially you do an integration across branches but lose the tie to mainline. Yuck.
You can RI the changes of note to mainline with a test pass and then FI them up into Feature team B's branch. This doesn't work as well if the changes are in progress and not fully baked.

Motley: I can think of another problem - my work in my branch can get out-of-date with what is going on with other teams. That may cause problems later.

Maven: True. The onus is on the feature team to ensure it FIs on a regular basis to keep up. Additionally, teams should not "go dark" from mainline for too long. Regular RIs to mainline should be happening, say, at the end of each sprint. For feature teams practicing Scrum, the code at the end of each sprint should be of shippable quality and can be RId back to mainline.

Motley: Also, merging in conflicting changes could be a real pain!

Maven: Yes, but not if you have the right tools. A good merge tool automatically merges isolated changes in the same file (the majority case), and allows you to easily resolve conflicting changes.

Motley: You seem to have an answer for everything. Let me see if I understand this branching thing:

Branching helps keep mainline quality high
A feature teams considers a branch to isolate its changes
Branching provides more freedom for check-ins
Quality gates are more strict the closer your branch is to mainline
Branching ensures you don't lose ties to the original source code via integrations
A good tool makes integrations and merges relatively painless

Maven: Nice summary. Why don't you go approach our beloved development manager and propose that our team undertake a branching model?

Motley: I think I'll do just that.

______________________________

James' Pointer: I am always amazed at how little discussion there is around source code branching. Branches are an incredibly useful software tool employed all over Microsoft for the reasons discussed above. Branches are particularly useful when you are an agile team in a large waterfall-based organization. They give you that level of isolation needed to follow your own effective processes yet still merge your changes in with mainline on a regular basis. Our team would be much less efficient without branches.

Maven's Resources:

Wikipedia definition of branching: http://en.wikipedia.org/wiki/Branching_(software)
(Tool) Beyond Compare: http://www.scootersoftware.com/moreinfo.php
(Tool) Araxis Merge: http://www.araxis.com/merge/
(Paper) Streamed Lines: Branching Patterns for Parallel Software Development, by Appleton et al, 1998, http://www.hillside.net/plop/plop98/final_submissions/P37.pdf

This paper has many good branching tips embedded within it.

Motley says: "Features sell a product. When in doubt, add more features!"

James Waletzky — Tue, 08 Jul 2008 16:55:00 GMT

Summary

Motley: Features sell a product. When in doubt, add more features!

Maven: These days, software is less about features and more about reliability, fit 'n finish, performance, and usability. Use the Kano model to help you focus on the right scenarios for the user.

______________________________

[Context: Motley is struggling with a common project management problem]

Motley: I am wondering how we are ever going to make any money on this product. We had to make so many cuts to make it to market in a reasonable time that the product is not all that useful, in my opinion. We need more features, and we need them quickly.

Maven: Do you think it's just features that make a product successful and sell lots of copies?

Motley: Well, pretty much. If the software doesn't do what you need it to do, why would you bother buying it? Features trump everything else. For a v1, we should even consider shipping sooner with a few more bugs to get a few more features in.

Maven: Hmmm… you were telling me the other day that you have an iPod Touch, right?

Motley: Why are you wasting my time here? Yes, I have an iPod Touch. Get to the point.

Maven: One more question: why do you have an iPod Touch vs. a Creative Labs Player vs. a Zune? The iPod costs significantly more doesn't it? And don't the other players typically have more memory for the same money, and perhaps include FM tuners?

Motley: That was three questions, wise guy. I chose the iPod Touch after playing with it for a while. It does what I need it to do, it's reliable, and it's extremely well refined and polished. I could flick the list views and pan around for hours. I love watching the acceleration of the list view as you start to flick and slow down as "friction" takes over. They really did a nice job with the interface. I had to have it. It's a really fun device to use.

Maven: So it's more than just features that attracted you to that particular model?

Motley: Well, in this case, yes. I looked at the other MP3 players out there, but I wasn't impressed. They had the same features, but the iPod Touch was just, well, fun. Additionally, one of the other players that shall go unnamed crashed within just a few minutes of use. I was not impressed.

Maven: Does the iPod Touch do everything you want it to do?

Motley: Not necessarily. It would be great if it had a built in GPS, for example, so I could hop on a wi-fi network and get directions from where I am. An FM tuner would be nice for those nights I go to the races (they broadcast the commentary over FM). But seriously, I have to get some work done. Why are you bothering me about all this?

Maven: Let's go back to the software you are working on - you were saying we need more features and should even ship with slightly lower quality to get features in. Does that equate back to your iPod Touch experience?

Motley: Well, errrrr, ummmm…

Maven: That's what I thought. A decade ago, when there was little competition for many software products on the market, features won the war. End users actually put up with an operating system like Windows 95 that blue screened every other day when it was released. It was the best thing out there and the reliability issues were tolerated. Times have changed. Users expect software to "just work". They want it to solve their problems their way with a minimum of fuss. If it meets the majority of their scenarios and does it well, users are generally happy. These days there are always other options if a product does not meet their needs.

Motley: You did mention "their way" above, though. In order to do things their way, you need to make it as configurable as possible and including knobs and buttons allowing them to customize their experience. That takes time to build.

Maven: I would actually argue that too many knobs to twist, turn and pull just ends up confusing the user creating a frustrating experience. Look at the iPod Touch - two buttons. Compare the browser on the iPod Touch to something like Internet Explorer on the desktop - very few configurability options. But, users are happy. In fact, they are ecstatic about the device. It satisfies 80% of the core user scenarios. It may not keep the power users happy, but there are other options out there, including third party software updates that could potentially add some of the knobs and buttons they need.

Motley: There is still a minimum feature set we would need to be successful.

Maven: No argument there. However, that feature set is usually not as large you think it is. Nail the top priority user scenarios extremely well and you will likely be successful.

Motley: Easier said than done.

Maven: Sometimes. You need to really understand the user in order to make good decisions. Here are a few other keys to success:

Reliability: The user's first experience with the product should be positive. It should "just work". It should be working out-of-the-box within a few minutes. Spend more time during development ensuring it installs easily and doesn't crash vs. adding features. What features are there must be solid.
Fit 'n Finish: Were the gravity and friction capabilities on the iPod Touch really necessary for a decent user experience? No. Do they set the device apart from competitors? Yes. Do they make the device fun to use? Absolutely. Do they set the bar for interfaces on mobile devices? You bet. Nice icons, nice graphics, and little effects add up to a really positive experience with the device when combined with the other criteria listed here.
Performance: Waiting sucks. Apple put the right hardware in the iPod Touch to ensure performance would not suffer. I am sure they spent a lot of time tuning the experience to make it snappy. Users will no longer accept sluggish software. Performance actually is a feature.
Usability: The user interface must be incredibly intuitive such that you do not even need a user's manual. The user should be able to pick up the device and be productive within minutes. An understanding of human psychology is a requirement when developing a good interface.

As another example, how many features of Microsoft Word do you use?

Motley: Out of the 10000? Probably 20.

Maven: Exactly. A few extremely polished features will likely satisfy 80% of your users (see the Pareto Principle, or 80-20 rule). Spend time making an incredible experience for those core features, and add features once the polish is in place. Aim to delight the user.

Motley: Delight the user? That sounds a little "foo-foo" to me. What is "delight the user" supposed to mean?

Maven: Think about the Kano model, which comes from Six Sigma. A diagram illustrating the concept is shown below:

Pasted from <http://www.isixsigma.com/library/content/c030630a.asp>

The lower curve represents the basic needs of the user that must be present in the product for it to stand a chance to be successful. With the iPod Touch example, it better play MP3 files. That is a basic need that the device must fulfill, and if it didn't, it wouldn't sell, at least not as an MP3 player. These features likely won't set you apart from the competition either.

The middle curve represents those features or criteria that you can never have enough of. With the iPod Touch example, more memory, faster performance and better sound quality are always welcomed and can set the product apart from the competition.

The upper curve provides the greatest opportunity to beat competitors, and are those things that truly "delight" the user. With the iPod Touch example, the polished user interface, the flick and pan operations, the automatic playlists, etc. help set it apart from other players and make customers say: "Wow, this is a cool device!" Often these types of "features" are unexpected, but very welcome.

Motley: Interesting model. You are saying that we must met the basic needs, consider performance characteristics, but to really set ourselves apart, we need to consider those things that delight the user. This typically revolves around ideas that really catch the user's attention, and are often the result of polishing existing user experience paradigms and features.

Maven: Exactly. Although polish does not always equate to delighters, it does contribute to a solid user experience - moreso than the addition of several more half-baked features.

______________________________

Maven's Pointer: Although I (James) work for Microsoft, I will gladly give another company kudos for a job well done if it is warranted. I do own an iPod Touch and think it's a fabulous device that other MP3 players cannot keep up with. I have been extremely happy with my purchase, even though I could have gotten a significant discount (relative to other players) on one unnamed player. "It just works" really matters to me - the device and/or software needs to be polished, usable, reliable, performant (is that really a word?), and just do what I need it to do. That's not to say Microsoft doesn't make some fabulous products as well, such as OneNote, Money and Vista (yes, I am happy with Vista). Apple does a set a great example, however, when it comes to building software and devices that consumers love.

Maven's Double Pointer Indirection: There is more to Kano analysis and mentioned here. There is guidance available in some of the resources below on how to ask user positive and negative questions and plotting them on a 2-D table, to determine whether ideas are delighters, performers, or basics.

Maven's Resources:

http://www.isixsigma.com/library/content/c030630a.asp
http://learnsigma.com/kano-model/
George et. al, The Lean Six Sigma Pocket Toolbook: A Quick Reference Guide to 100 Tools for Improving Quality and Speed, McGraw-Hill, ISBN: 0071441190, August 2004.

Motley says: "Bug fix sprints are a Scrum anti-pattern"

James Waletzky — Tue, 27 May 2008 17:57:00 GMT

Summary

Motley: Bug fix sprints are a Scrum anti-pattern. Quality should be kept high so as not to have to focus on bug fixing.

Maven: A clear meaning of done for sprint tasks is important, but even if you follow this best practice, there will always be integration issues, bugs, and changes required late in a cycle. A sprint focused on these tasks is reasonable.

______________________________

[Context: Motley and his team of merry developers have been fixing bugs for the past week, and Maven has noticed something odd]

Maven: Morning, Mot. How are things going?

Motley: Not too bad, although I'm a bit late getting in this morning. It's tough to get out of bed when all you have to look forward to is fixing bugs for the next couple of weeks. The bug database is the team's to-do list for the next little while.

Maven: Yeah, this is a part of the development cycle that is rarely a developer's favorite. Is morale still high in the daily Scrum meeting?

Motley: Daily Scrum? Why bother? We are done with Scrum now that we are integrating and bug fixing. Besides, bug fix sprints are a Scrum anti-pattern. Didn't you tell me the other day that Scrum only works for new feature development?

Maven: Um, you must have heard that statement from someone else, or misunderstood something I said. I would partially agree with your statement that bug fix sprints are a Scrum anti-pattern. In fact, most of the Scrum/agile purists would likely say that you are absolutely right. However, past experience and the reality of software development have made me realize that bug fix sprints can be a necessity in large organizations practicing agile development.

Motley: Why?

Maven: One of the most important concepts of Scrum, and where many teams fail, is having an explicit meaning of done for each and every task on the sprint backlog. For example, a typical coding task may involve the following in its "done" definition:

Design document reviewed (if necessary)
Feature fully implemented (including error handling, performance, reliability, security, maintainability and other quality concerns)
Buddy build complete
Buddy test complete
Unit tests written
Code coverage from unit tests > 80%
Static analysis run
Code reviewed by at least one peer
Code checked-in to the source tree

Motley: That's getting pretty detailed isn't it? What's the point?

Maven: The point is that we all have a common understanding for what it means to complete a specific coding task, and understand what "shippable quality" means. If you do not explicitly define the task, some tasks may get missed costing you valuable time and effort later in the cycle when change is more difficult. With agile, we want to produce high quality deliverables very early and not rely on a long "stabilization" phase at the end. If we do everything we can to prevent bugs up front, we'll actually save time in the long run and theoretically ship sooner. Having defined meanings of "done" for all our tasks is a big key.

Motley: Which reinforces my point that a bug fix sprint is an anti-pattern. Jeez, Mave - did you not have your morning jolt of caffeine?

Maven: If everything goes according to plan and you integrate components as you go, then yes, longer bug fix periods are somewhat of an anti-pattern. But let's be realistic. There is some measure of integration and late discovery of unknowns in any large software project. For a small project of a few thousand lines of code, you may be able to avoid a bug fix stage like this. However, take James' experience in working for a previous company. One thing his team was responsible for was porting a large code base from one platform to another. The team added unit tests where possible but much of the code was inherited legacy code that did not come with tests. The test team worked closely with the development team to find as many problems as possible up front, but when you have tens of thousands of lines of code running on a platform that is changing underneath you, you must expect to find bugs later.

Motley: Not to mention when people in the organization start hammering on your code as real users would. I think James called that "dogfooding" when I talked to him, as in eating your own dogfood. That does not sound very appetizing.

Maven: Correct. You cannot possibly predict all the issues that will arise and fix them proactively. As a result, his team undertook what they called a "quality sprint", which focused on fixing bugs. They took an agile approach and refused to develop new features until the product was at shippable quality, and they hit ZBB.

Motley: What the heck is a "ZBB"?

Maven: "Zero Bug Bounce". It's a moment in time where you have no bugs that are older than, say, three days and there are no ship stoppers. At this point, the team can make a call that quality is high and new features can be developed.

Motley: You called it a "quality sprint". You could also call it an "integration sprint" depending on what your goals are. In fact, that might be a better name for what we are doing now.

Maven: Agreed. The term I don't like ,however, is "stabilization sprint". To me, this implies that the code you wrote was crap to begin with. In reality, there may be things beyond your control that crop up upon integration even though you focused on quality, so I prefer to give it a more positive spin. "Stabilization" also typically implies a huge focus on testing. Testing is part of what happens here, but testing is also happens in every sprint.

Motley: Game it if you want. If it looks like a duck, and quacks like a duck, well, you know. You still have not answered the question as to why I would want to practice Scrum when I am fixing bugs!

Maven: You don't necessarily have to practice Scrum while bug fixing, but you may find it valuable. When your team needs to fit within a larger ship cycle on a big team, you may not have full control over how to structure your bug database. In addition, your team may live within the confines of a waterfall-oriented organization, but you can be agile as a feature team by practicing Scrum and transferring prioritization of fixes to the sprint backlog. The team does the usual sprint planning figuring out which bugs are most important to fix. The sprint backlog is then a prioritized queue of bugs, and devs grab bugs from the queue as soon as they are done fixing their current bug and validating with test.

Motley: Is it really worth the overhead though? I already have my list of things to do in the bug database. Why duplicate it?

Maven: I generally don't consider Scrum as "overhead" - it's a very lightweight process, although I would agree that with any process you arguably get some level of overhead. You could fairly easily write a tool that automatically populated the sprint backlog with the top bugs from the bug database. Note, however, that you also gain the following benefits by practicing Scrum while fixing bugs:

The daily meetings ensure improved communication and collaboration
Data gathered is useful for future planning sessions
Frequent retrospectives allow the team to constantly improve
The team can adjust priorities during the sprint
The team need not rely on a rigid bug database schema, and can use the sprint backlog to customize information about the bugs
The team remains focused, and has a Scrum Master to help

Motley: Fine, fine, fine. We'll try it your way for a sprint and see if it pays off. I did kind of lose track of what Marvin was doing with his bugs and the daily meeting should help.

Note: I'd love to hear some better suggestions on how to deal with the problems discussed in this story. Arguments based on reality (vs. theory) are much appreciated in comments!

______________________________

Maven's Pointer: Here's another argument for tracking time spent on tasks in a sprint: over several short quality-oriented sprints you can gather data on the amount of time it takes to fix each bug. By mining that data, you can compute an average length of time per bug, which helps you with effort estimates for capacity planning on future bug fixes. Our team, for example, knows that it takes just under 5.4 hours of effort to fix an average bug. We can then estimate how many we can fix during any given sprint, and set expectations accordingly.

Maven's Resources:

Scaling Testing in Scrum, by Bob Galen, http://www.agile2007.org/downloads/handouts/Galen_532.pdf

Motley says: "Checklist? We don't need no stinking checklist!"

James Waletzky — Tue, 04 Mar 2008 19:29:00 GMT

Summary

Motley: Checklist? We don't need no stinking checklist! Checklists are for anal people with short memories.

Maven: Checklists are very useful for activities like code reviews, where it is difficult to remember everything we need to look for. Good checklists are short, consistent, easy to understand, accessible and provide support information. Personal and team checklists are recommended.

______________________________

[Context: Maven spent the morning looking over the code reviews in his Inbox and noticed a few odd things about one of Motley's code reviews from earlier in the day]

Maven: I was just looking over the code you sent out for review earlier in the day. Nice job on finding those errors you found. There are a few more things I might point out, however.

Motley: Did I miss a few? Hard to believe as I am the king of code reviews. I must have been having a bad morning. What did I miss?

Maven: Let me ask you this first: when you look for issues in a code review, what kinds of things do you look for?

Motley: The usual stuff - memory leaks, invalid usage of pointers, failure to check input parameters - that kind of thing. That covers the major ones.

Maven: What about issues like commenting, proper use of exceptions, freeing up resources, ensuring strings are localizable, assertions are used appropriately, unit tests are created, and checking for buffer overruns? I could go on and on.

Motley: I usually remember to check for those kinds of problems. Depends on my mood.

Maven: Do you ever use a checklist?

Motley: Checklist? We don't need no stinking checklist! Checklists are for anal people with short memories. Checklists will just slow us down. Plus, they get out of date quickly in my experience. Better to just get into the habit of knowing what to look for and then just do it!

Maven: Let me ask you this: do you remember everything you need to buy at the grocery store?

Motley: The common stuff I buy from week to week? Yes.

Maven: What about the one-off groceries? What if you want to bake a cake and you don't have any flour, which is not something you buy from week to week?

Motley: First of all, think about what you just said. Do you think I would ever be caught baking a cake? Wise up, Mave.

Maven: Good point. Sorry. The point is that for things you are in the habit of, your brain just automatically remembers them. I always know I need fruit when I go to the grocery store. However, for things that I don't buy regularly, like chocolate chips, I need a list.

Motley: Do what you need to do. I still try and get into the habit of looking for certain items in a code review, though, so I can avoid a checklist.

Maven: Getting into the habit is great, but with all the things a developer has to keep in his or her head, it's hard to remember everything when coding.

If we sat down and brainstormed every characteristic of production-quality code, we would probably fill a couple of whiteboards. Unfortunately, the human brain only retains about 7 +/-2 concepts in short term memory at a time. That's why phone numbers are seven digits (forget the area code, which is pretty static). For concepts outside of that core 9 - to be optimistic - we really need some tools that help us with reminders.

Motley: There are certain things that I "just remember", like checking for null pointers. I don't need a checklist to help me with that.

Maven: I agree.

Motley: You agree?!? Then what's the point?

Maven: For concepts that really are habit and don't require any extra effort to think about, consider not putting those on a checklist. The checklist is a helpful tool to ensure you remember those concepts that have not become habit. Additionally, if the team, for example, always remembers to pull strings out of resource files to help make the code localizable, then get that off the checklist and replace it with a reminder to check for a mistake that the team often makes. The highest priority items for the business or the release are also good candidates.

Motley: So now you are talking about maintaining checklists, which I indicated is cumbersome.

Maven: Keep checklists in a simple format that everyone can easily edit, like a wiki. Sure, you have to spend time maintaining them, but if it saves even one bug from shipping, it has just paid for itself 50 times over. Even better, you might want to have your personal checklist in a tool that you use every day and all day. For example, I noticed you check your e-mail a lot - you could have your checklist inside Microsoft Outlook as a task list. There it is easy to maintain and always available. The wiki works just as well though, particularly for team checklists.

Motley: Now you are talking about more than one checklist? It's getting worse.

Maven: A good practice is to have a team checklist and a personal checklist. The team checklist contains items that you want everyone to look for in a code review. It typically contains the most frequent errors made by the team, or major areas to focus on. Your personal checklist contains items specific to you, such as things you often forget to do or look for. For me, I often forget to make liberal use of assertions, so my checklist helps remind me to code defensively. A best practice is to use your person checklist to review your code prior to sending it out for review.

Motley: So I guess I start with the stuff I missed from this mornings code review. I am just going to keep it in a simple text file on my desktop where it is easy to find and edit.

Maven: Format is not important - whatever works for you. Here are a few more characteristics of good checklists:

Keep checklists short. By "short", I mean one page or less. Big long multi-page checklists do not do much good as they become unwieldy to use. Only have items that are the most important to remember.
Use consistent phrasing. Checklists with inconsistent phrasing are hard to use. Settle on one grammatical style. Some examples:

Affirmative statements with the "do" action verb: "Do free up any allocated handles."
Ask questions: "Are all pointers freed?"
Make direct statements: "Function header comments are accurate."

Mixing styles makes the checklist difficult to read.

Make checklists easily understandable. Avoid too much jargon unless the terminology is known by everyone on the team. Keep checklists simple.
Ensure checklists are accessible. For team-based checklists, in particular, make sure they are easily accessible and updatable. Put them on a team web site in an easy-to-find location. If they are hard to find, no one will use them. If they are hard to edit, they won't be updated.
Have multiple views, with supporting information. A good checklist actually supports multiple views. You may have a quick and dirty short form to act as your primary list of reminders, and you may have a more complete and verbose view with explanations for each checklist item. That way a new member of the team can easily get familiar with each item in detail. Links to resources for more information are great additions.

Motley: As always, you have lots of rules for everything. Don't you get tired of being so thorough?

Maven: Well, no. Ha ha ha. These "rules" are basically some intuitive guidelines (softer than a "rule") to help create a good checklist. For your personal checklist, you can do what you want as long as they help with the main goal of reminding you of important items.

Motley: Fine, fine, fine. Based on my poor code review performance this morning, I'll give it a shot. I'll create a quick checklist for code reviews.

Maven: Great! Checklists are also great for other activities, too, like design reviews, unit tests, code check-ins, and many other development activities.

Motley: Let's not get carried away. I said I'll create one for code reviews.

Maven: Ideally, you also think about building tools that automatically validate your checklist items without human interaction. That's what a tool like FxCop does - it takes a big checklist like the .NET Design Guidelines and automatically ensures your code satisfies those guidelines. Automated checklists are the best kind!

Motley: I'm always up for writing code to remove more manual labor. Let me give that some thought.

______________________________

Maven's Pointer: One interesting format for checklists is in XML. With XML you can apply any type of XSLT transform to the data to generate different views. You can have one view that is a summary of the checklist item, and another view that contains all of the details. Each view is based on the same central XML file. Developing checklists using a tool like Microsoft InfoPath abstracts away the XML data format and still allows multiple views. Store the XML on a Microsoft SharePoint server to make them centrally accessible, and you have a great checklist solution.

Maven's Resources:

Code Complete, 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.

There is a wealth of checklists for software developers in Code Complete. Check the end of each chapter for useful checklists on coding, commenting, assertions, code reviews, and much more.

Checklists.com, http://www.checklists.com/

A nice set of general checklists

How to Create a Better Checklist, http://hwebbjr.typepad.com/openloops/2005/09/how_to_create_a.html

A more detailed summary of characteristics of good checklists

Motley says: "Performance stinks, but I'll deal with it later"

James Waletzky — Tue, 20 Nov 2007 17:46:00 GMT

Summary

Motley: I'll deal with performance issues when I have something to test. Then I'll run the profiler and just optimize the slow methods.

Maven: Think about performance early in the development cycle. Set goals. Design for performance. Measure, measure, measure. Optimize the scenarios that matter.

______________________________

[Context: Motley is doing some performance testing and optimization on the feature he is building]

Motley: Hey, Mave. Check this out. I've been doing a bunch of performance tuning on this component. I've made these three methods twice as fast as they were before!

Maven: Cool! What tool are you using to measure how fast they are?

Motley: The built-in profiler in Visual Studio rocks. You can run a set of unit tests against your production code and VS will tell you what methods are your bottleneck. Based on that I go in and fix them. For ultimate flexibility, you can choose to either instrument your build with profiling prolog and epilog code in each method, or you can have the VS profiler take frequent samples to see how your application is behaving. I prefer instrumentation - it changes the binary but you get more accurate results.

Maven: Sweet. Why did you just jump to the methods that VS tells you were the slowest?

Motley: Because they're slowest! Duh. I'm not going to go spend valuable time optimizing the fast methods! Have you had your morning coffee today Mave?

Maven: The key is that just because a particular method is slower on a relative basis than other methods for a particular test scenario doesn't mean it's worthy of spending time optimizing. What if the scenario you are optimizing for is rarely used? Is it worth making the 5% of the code that is rarely executed twice as fast? What if the user already perceives that feature to be fast enough given today's hardware?

Motley: Reasonable questions, I guess. But faster is faster. It's never a waste of time to optimize a component.

Maven: With all due respect, Mot, I beg to differ. The time you spend optimizing some component that doesn't really need it from the user perspective is time you could have spent improving code quality, optimizing something that needed it, or building a new piece of functionality to delight the user.

Motley: Grrrr… you mean I am "generating waste", if we talk about being agile. I hate it when you're right.

Maven: Generating waste? Yeah, I didn't think of it that way, but exactly! Nice observation Mot!

Motley: Okay, I'll find out which components need optimizing first and then work on those.

Maven: Great. How are you going to determine which ones need optimization?

Motley: I'll… Hmmm… your telling me I can't just look at absolute numbers. Well… enlighten me Einstein. How am I going to know what to optimize???

Maven: There's nothing wrong with using the profiler for some scenarios as you mentioned previously. Profiling works well when you are executing end-to-end scenario tests that exhibit real-world behavior of the software. For example, use the software in a scenario the way the user would. The bottlenecks that appear in a real-world use case are usually worth fixing. However, if the user already thinks that the software performs well enough for their needs - on standard hardware of course - then focus your efforts elsewhere. You need the voice of the customer (e.g. a user experience person or program manager) to give you that information, however.

There are other types of performance testing you want to do as well that are above the method level. For example, if you are building a web application, you may need to test throughput, response time, and latency. These factors may have more to do with the way you use bandwidth and how machines are configured vs. how individual methods perform.

Motley: I'll have to agree with you there, as scary as that sounds. You have to admit that I was doing the right thing by measuring, however.

Maven: Absolutely! The golden rule of performance tuning is Measure! Measure! Measure! We usually say it three times to ensure the emphasis is made. Don't try to guess at what to optimize as much of the time you'll guess wrong. Leverage a profiler and other testing tools to provide some solid data. Another question: have you been thinking about performance all along?

Motley: What do you mean? I can't fix any performance problems until I know they exist! I deal with performance stuff late in the cycle like any other developer.

Maven: What if you make a bad design decision? I know one team that was trying to determine whether they should redesign their remoting infrastructure from using DCOM to Web Services. Early on they decided to take the plunge and made a heavy investment in web services. Know what happened? Although web services brought benefits in terms of standardization and security, the performance hit they took was huge. Web services infrastructure was tied to the entire application and they couldn't go back.

Motley: Sucks to be them! They should have prototyped or developed using an agile model in small pieces with performance testing throughout.

Maven: Exactly! So why not heed your own advice?

Motley: <silence>

Maven: You need to consider performance from day 1 of your development. Set performance goals to hit. The previous version, if it exists, can be your baseline for performance numbers. Aim to be within at least 10% of those numbers. Think about the scenarios where performance is critical. Plan and budget the resources such as memory and CPU you use accordingly. Prototype new technologies before making a heavy investment in them. If you discover a design flaw related to performance too late, you may not be able to fix it.

Motley: So your best practices for developers when it comes to performance are to:

Measure, measure, measure - don’t guess at bottlenecks
Don't leave performance work to too late in the cycle
Think about performance goals early on, and consider performance when doing design
Test and tune real customer scenarios
Prototype technology decisions that may impact performance

What about Test-Driven Development, though? Where does performance work fit in with that methodology?

Maven: The story doesn't change a whole lot. You still think about performance goals and design impacts. As my buddy Matt says, make sure you don't do TDD with the blinders on as a change in one unit or component can have a significant impact on performance in a seemingly unrelated feature of the product. With TDD you are testing units in isolation, and mock objects help with that. You should think about developing performant code according to the contract in your unit. However, you may not be able to predict the impact on your dependencies. The advice there is to integrate early and often, and start performance testing early and often. Don't leave it until the end.

______________________________

Maven's Pointer: At Microsoft, software performance is treated very seriously, and is basically a feature in a project from day 1. You need to make some intelligent design decisions around performance early in a development cycle, but you want to avoid painting yourself into a corner if you make the wrong decision. That's why designing to interfaces is such an important design concept. You want your design to accommodate change if you make the wrong decision or you need to plug in a different algorithm depending on some heuristic. Your sorting algorithm may work fine on small datasets, but what if in the field you run across a huge dataset for which there is a better algorithm?

Maven's Resources:

No specific resource around designing for performance and tuning performance due to the tight tie to technology. It is recommended that you search your favorite bookstore for books on performance specific to your technology (e.g. SQL, ASP.NET, C#, etc.) or activity (development, testing).

Motley says: "I just get picked on when my code is reviewed"

James Waletzky — Tue, 06 Nov 2007 18:57:00 GMT

Summary

Motley: I'm not sold on this inspection thing. There were too many problems during our session. Plus, everyone just picks on the author and they hate having their code reviewed. Back to informal peer reviews for me.

Maven: Follow inspection best practices: inspect limited amounts, don't inspect everything, use a facilitator, be hardcore, use a checklist, focus on the work not the people.

______________________________

[Context: Motley has just finished his first code inspection with the team and is explaining to Maven how it went]

Motley: Well, we did a code inspection to try out the process we talked about. I gave the team an overview of the process using the diagram you gave me. We picked a bunch of code to look at from multiple owners, everyone prepared and we did the meeting. As I suspected, it really wasn't much better than an informal peer review. I'm not sold on this inspection thing, and I'm not sure I would do it again.

Maven: Hold on! Before you make that conclusion, let's get to the root cause as to why the inspection was not as effective as it could have been. First of all, how much code did you pick and how long did people spend preparing?

Motley: We inspected about 1500 lines of C# code. I would say the average preparation time was about 45 minutes, and the meeting took about 45 minutes as well.

Maven: There's problem number one. Every study on inspections indicates that less code in one session is better. In terms of absolute numbers, the guideline for preparation is somewhere between 200 and 500 lines of code (LOC) per hour . Generally, developers cannot maintain the "zone" of code reading and issue finding for more than 1.5 to 2 hours, so you should inspect approximately 400 to 1000 lines in one two hour session.

Motley: Are you serious?!? How would a company like Microsoft use this technique on all 50 million LOC in Windows Vista?!?

Maven: They wouldn't. You don't want to inspect every LOC that the team writes. Pull out inspections in the following situations:

The first few LOC for each developer in a development cycle. Having the team review the first few LOC that each person produces helps nail some problems right at the start. If a junior developer does not understand how to properly throw and catch exceptions, for example, they are going to make the same mistakes throughout their development. Having a team-based inspection at the start gets them on the right path from the beginning.
Critical code. Code that sits on a network or other trust boundary (e.g. an ISAPI filter running in the IIS web server) is a great candidate for inspections. The reason is that the consequences of someone hacking that code can be severe, particularly if the code is running as SYSTEM.
Complex code. Complex algorithms and interactions (e.g. concurrent sections of a program) should be looked at closely by the team. In many cases, tools do not do a great job of finding issues automatically, and inspection is the best way to flesh them out.

The key message is that you should wisely choose code to inspect, and not count on inspecting all of it. You said average preparation time was 45 minutes, but was everyone prepared for the meeting?

Motley: Well, a couple people who shall remain nameless didn't spend much time on it at all. Coincidentally, they didn't report many bugs either.

Maven: If some people come unprepared, you have a few options:

Ask them to simply be an observer in the meeting. Since they will not have much to contribute anyway, you want to minimize their distractions.
If it happens multiple times, kick them out of the meeting with a clear message that they are not helping improve product quality, and that practice is not acceptable.
If people say they prepared, but you are skeptical, play a little game. In the code that is handed out for the inspection, inject a couple of bugs. Don't make them too difficult, but they should not be too obvious either. Good candidates are off-by-one errors and reversed Boolean values. Tell people how many bugs you put in, and challenge them to find those bugs.

What about the meeting itself? How did that go?

Motley: Not so good. We were all over the place. We had many tangential conversations where we tried to solve the problem we discovered. We also had trouble referring back to the lines of code that were troublesome and everyone had different copies of the code.

Maven: There's a few things we can do to help that situation.

First, all good meetings - and all good inspections - have a facilitator. With inspections this role is often called the "moderator." This person guides the meeting, collects the data and issues that are found, and keeps the meeting running efficiently. They have free reign to cut someone off that goes down a tangent.
Second, avoid discussing solutions. The inspection is all about finding problems. If you want to talk about solutions to specific bugs, put it on a list on the whiteboard (often called a "parking lot") and anyone that is interested can stay after the main collection meeting is over.
Third, it is important that everyone inspect the same copy of code. One way to do this is have the moderator print out the code for everyone. Inspecting code on paper removes you from the distractions of the computer. In addition, you ensure everyone has an identical copy, and by including line numbers, you have referral points for individual LOC.

If you follow those tips, you should have a better inspection. What kinds of problems did you find this time around?

Motley: Not much. We found a few cursory bugs, but nothing major. My gut tells me that there are more bugs lurking, but I cannot say for sure. The problem was that no one really knew exactly what to look for.

Maven: When preparing, you really need to focus. Be as thorough as possible when finding issues. Everyone should be familiar with the problem, team coding standards, and team development processes. You want to point out every little thing you find - be hard core. Don't just focus on pure logic errors. Anything that, as a quality-oriented developer, you would like to see fixed, point it out. By following this practice, we lead ourselves towards better quality products.

In addition, do an analysis of where your team makes frequent mistakes. Those items should be put on a checklist, and everyone should leverage that checklist while preparing for the inspection.

Motley: Everyone is familiar with the .NET Design Guidelines, but we need to be a little more hard core. It would also help if we ensure tools like FxCop are run prior to the inspection too. I assume that's why you have been using this word "issue" instead of "bug" over the past few minutes? You want devs to think more widely about the mistakes they look for? Comments that do not match the code count?

Maven: You are a very bright guy, Mot. That's exactly why I use the word "issue". One last thing: you mentioned you used code from several authors. Try and restrict the code you look at to one person to prevent context switching. Was that person happy to have their code inspected?

Motley: Quite the contrary! He felt he was picked on the whole time and even told me afterwards he wants to avoid this whole process in the future. I told him to quit being a baby and that it's his job.

Maven: Whoa. Probably not the best response. Anyway, authors should want to have their code inspected. It's a great learning experience. Inspectors can encourage this by avoiding a threatening environment. Direct comments at the work item and not personally at the author. Also, instead of making statements, ask questions. For example, "I'm not sure, but is there a memory leak on line 34?" is better than "There's a leak on line 34. What were you thinking? Did you not run the tools? Wise up dude!" Asking questions is less confrontational and who knows, you may be incorrect about your assertion. Let the author come to the conclusion herself. It's a better learning experience that way anyway.

Motley: That's a lot of stuff to remember.

Maven: It's not that bad. Just remember a few key points:

Inspecting less is better
Be wise about what you inspect
Use a facilitator in the meeting
Be hardcore about the issues you identify, and use a checklist
Remove the people from the problem (i.e. avoid threats, ask questions, focus on the work)

Motley: Ugh. Back to the drawing board. I can think of some nasty multi-threaded code that perhaps we should inspect next. I'll try this stuff out again if my team doesn't hate me and get back to you.

______________________________

Maven's Pointer: According to Steve McConnell in Code Complete 2nd Edition, inspections are one of the most effective practices for finding defects early in the software lifecycle. In fact, studies have shown that inspections find 45% to 75% of all defects, with other practices like unit testing and system testing falling below these values. Your experience may vary, but selectively using inspections on requirements specifications, designs, code, test plans and other artifacts will save you time in the long run.

Maven's Resources:

Design Guidelines for Developing Class Libraries (.NET): http://msdn2.microsoft.com/en-us/library/ms229042.aspx
Code Complete 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.

Motley says: "Inspections suck... time that is"

James Waletzky — Tue, 30 Oct 2007 17:47:00 GMT

Summary

Motley: Inspections seem a bit heavy. They suck up too much time for little gain.

Maven: Inspections are actually a fairly light-weight structured process that find more issues compared to ad-hoc peer reviews and team reviews.

______________________________

[Context: Maven and Motley are continuing a conversation about code reviews, now with a focus on inspections]

Maven: Nice restaurant choice, Mave. Although I am a little worried about what this Mexican food will do to me this afternoon.

Motley: Let's get this conversation over with quick so I don't have to be around you this afternoon! You were saying something about "injections" or something? Hopefully they aren't lethal!

Maven: Inspections. Inspections are a structured team-based review process for code, design, requirements, and any other artifact. Inspections are much more effective than ad-hoc peer or team reviews. The process is very simple - six steps, of which 2 are optional. Here is the overview:

Motley: That still looks pretty formal and heavy to me! Just the fact that you have a diagram describing all the steps is a little scary. I'm worried this practice is going to suck up a lot of time.

Maven: Ah, don't be afraid. The process is actually quite simple and not unlike other code reviews. It's just a little more structured with a couple of key practices to ensure that reviewers are highly effective. First we start with planning.

Motley: Yeah, the process is so complicated I bet you need a plan just to pull it off!

Maven: Please! Put that chimichanga back in your mouth. Planning is really just common sense. You figure out what you are going to review, who should be involved, and plan the meeting(s). Not much to it. Next comes the overview.

Motley: You mean the 2 hour overview meeting where you are going to describe this process to everyone?

Maven: Always a wise guy. The overview is actually optional. You want to ensure that everyone involved in the inspection is sufficiently familiar with the work item to be inspected to do a good job. An overview is an ad-hoc meeting that usually doesn't last more than half an hour. If you are doing a code inspection, the author of the code gives an overview of what the code is doing (i.e. the requirements) and how the code was designed. There may also be a discussion of areas to watch out for, assumptions that were made, and it's a chance for inspectors to ask questions. Next, we prepare for the inspection.

Motley: What's the point? Shouldn't we just get together in a meeting and review the work?

Maven: Preparation is actually the most important part of the process. Here, each individual inspector takes some personal alone time to scour the material for issues. With code, an inspector looks for bugs, maintainability issues, and anything else a quality-oriented developer would commit to fix. Just getting together in a meeting to review the work on-the-fly is not effective. The real issues are found during individual preparation. After everyone prepares, THEN you have the meeting and collect the issues from everyone.

Motley: Hmmm… hopefully that meeting is extremely quick! If everyone does most of the work on their own, then we won't need much time for the meeting. In fact, why do the meeting at all?

Maven: The meeting is quick, but it is important to discuss some of the issues with the author. Treat it as a big learning experience for everyone. Plus, some of the things that people find may need clarifying questions and may not actually be issues at all. The meeting is a chance to validate that everyone has prepared, collect some metadata about the meeting (such as meeting time, number of issues found), and learn from one another. After the meeting is over, we take a few minutes to talk about the root cause of some of the issues that were found, and try to address them such that we don't make the same mistakes in the future. After the meeting, some rework is usually necessary.

Motley: Well, duh! If you find bugs you need to fix them.

Maven: Exactly, provided we are not at the point in the development cycle where only the really major stuff gets fixed. In any event, after most reviews, the author goes away and follows the processes of the team to fix the identified issues. After that happens, there may be some follow-up.

Motley: Because we don't trust the author to make the correct fixes? That's babysitting!

Maven: It depends on the developer. This step of the process is another optional one. We may want to have someone follow up if the developer responsible for the code is junior or new to the company. Another reason for follow-up, and even re-inspection, is if many major issues were found.

Motley: And what's this process improvement thing down the side of the diagram?

Maven: Let's save that discussion for another time.

Motley: So I bet your next request is that I just go off and try this. Fine. I'll humor you and give it a shot on some code. I'm not convinced it will help us, but I'll give it a shot.

Maven: Great! We can pick some code to inspect this afternoon and get it going.

Motley: Hold on, Burrito Boy. I don't want you anywhere near me this afternoon. I'll try it out with the team and let you know how it went.

______________________________

Maven's Pointer: The lean way of thinking is that inspections should not be necessary if we build quality into early parts of our development. Unfortunately this is just not feasible with software and other domains due to the human element. Think of inspections of code as similar to an inspection of your house under construction. Someone does the inspection that knows the domain, knows what to look for, and they make it a mandatory part of the house construction process. It is worth the small time hit to ensure a quality deliverable.

Maven's Resources:

Introduction to the Team Software Process, by Watts Humphrey, Addison-Wesley Professional, ISBN: 020147719X, August 1999.
Software Inspection, by Tom Gilb and D. Graham, Addison-Wesley Professional, ISBN: 0201631814, December 1993.

Motley says: "Quick peer reviews are sufficient. Team reviews? Waste of time."

James Waletzky — Tue, 23 Oct 2007 18:21:00 GMT

Summary

Motley: Quick ad-hoc peer reviews are sufficient to nail bugs early in the development cycle.

Maven: A more structured team-based code review process can lead to more issues found.

______________________________

[Context: Maven overhears a code review that is happening in Motley's office]

Maven: Hey Mot. Were you and Marvin just having a code review? I overheard a little bit of it while you were at your desk.

Motley: Yeah, we just kicked butt - found a few nasty little problems, although nothing too significant due to my superior coding abilities.

Maven: I expect nothing less! I was just wondering what your process is for code reviews. Can you enlighten me?

Motley: Not really a process per se - we want to keep things nice and light. Code reviews are known to be an effective defect prevention technique so of course we leverage them prior to checking in code. When I am ready to check-in, I call over a fellow developer, I pull up a quick WinDiff display of the code that has changed, we do a quick over-the-shoulder check at my desk. I then fix whatever bugs we identify, and then I am good to check-in. We usually catch a few things on a code review, particularly when I review someone else's code. I am not a senior dev for nothing!

Maven: I applaud your use of code reviews to keep quality high! But-

Motley: I knew it! There's always a "but" with you. For your information, I've been doing some reading on code reviews and I think that given the amount of effort we are spending on the review relative to the number of bugs that we find, we are being effective. We focus on the diff, fix all the bugs on-the-fly, and move on.

Maven: Don't get me wrong - I think what you are doing is great. For many check-ins involving trivial changes that process can work. What do you do for more complicated check-ins like a few files of new code?

Motley: Same thing. It just takes a little longer.

Maven: Do you typically find more bugs than for other reviews?

Motley: I would say we find about the same number per thousand lines of code.

Maven: And do you still find problems later on in the testing cycle?

Motley: Of course! You can't expect us to find everything in a brief little code review. I still fee like we're effective though.

Maven: Informal, less structured peer code reviews as you describe are great for the following situations:

Small bug fixes
Minor code changes where the author is already very familiar with the code
Changes of very low complexity
Changes with few possible side-effects

More formal, structured, team-based code reviews are more suitable in these situations:

Large bug fixes
Scenarios where a lot of new code is being added to the system (e.g. a feature check-in)
The first bit of code a developer on the team writes
Changes on a trust boundary - i.e. security-related changes
Complicated code changes, such as those involving concurrency

Motley: Whoa! You used a really bad word. The word "formal" really scares the crap out of me. Whenever some one uses "formal" to describe a process, look out. Those processes typically have high overhead and occupy a lot of time for - in my experience - little gain.

Maven: I hear you. In fact, I often agree. By "formal" here, I mean slightly more structured than just calling someone over to your office to quickly look at a bit of code. There is a process, but it's very agile and lightweight.

Motley: You also said "team-based". Why would I want to involve more than one person in a code review? That seems like a waste of time!

Maven: Studies show that team-based reviews that follow some kind of process - even lightweight - are the most effective defect finding techniques in the development cycle. You want to add people to your review to the point of diminishing returns. The ideal number is somewhere in the area of 3-4 reviewers. A more structured review tracks metrics over time that can help you determine how many people to include. It's all about being as effective as possible while not expending too much effort to get the job done.

Motley: More eyes, more bugs. I'm not sure I totally buy it. Why did you say previously that one of the scenarios where we should practice a more structured review is on the first bit of code a developer writes?

Maven: There is great value in that. The idea is that you get a few developers on a new project to review the code in detail for someone contributing their first few lines of code. A team-oriented review is a fantastic learning experience that gets the developer going in the right direction from the start. You see, humans are creatures of habit - if you make a mistake in your first few lines of code you are likely to repeat it in all your code. Fixing all of those problems expends wasted effort later.

Motley: Seems reasonable, but I could do that with just that developer and me.

Maven: You won't find near the amount of issues as a team of people. Use the technique sparingly, however. Microsoft, for example, could never team-review all 50 million lines of Windows Vista. You want to do team review in intelligent situations.

Motley: What else makes a team review different than an ad-hoc peer review? Right now I can't believe it would be that much more effective. And does this different kind of review process have a name?

Maven: A more structured review is typically called an inspection. Let's talk about it over lunch. To whet your appetite, here are some of the key principles behind an effective team-based code review:

A structured, repeatable, but lightweight process
Diligent individual preparation at a reasonable preparation rate
Non-confrontational meetings to ensure "threats" to the author are removed
A checklist reminder of the most important issues to look for
Metrics that help gauge the effectiveness of your meeting
A focus on process improvement to help prevent the same bugs from occurring again

Motley: My interest is slightly peeked. Where do you want to go for lunch?

Maven: How about-

Motley: Never mind - I'll pick. You drive.

______________________________

Maven's Pointer: The term "inspection" has been used throughout software engineering history to denote a more formalized code review. Invented by Michael Fagan at IBM in the late 1970's and used by many companies, inspections typically yield 45-75% (Code Complete 2nd Edition) of the defects injected into software throughout the development cycle. There are several different inspection processes, but the one we will focus on here has roots in the Team Software Process (TSP) with some modifications for efficiency.

Maven's Resources:

Peer Reviews in Software, A Practical Guide, by Karl Weigers, Addison-Wesley, ISBN: 0201734850, December 2001.
Code Complete 2nd Edition, by Steve McConnell, Microsoft Press, ISBN: 0735619670, June 2004.

Motley says: "I am a developer - I don't test. Testing is for the test team."

James Waletzky — Thu, 29 Mar 2007 18:56:00 GMT

Summary

Motley: Unit testing is the test team's job! I am not paid to write tests.

Maven: White box (unit) tests help with code quality and give you confidence to change your code.

______________________________

[Context: During a previous conversation about debugging, Motley stated that he doesn't do unit testing]

Motley: Yeah, that's what I said: unit testing is the test team's job! I am not paid to write tests.

Maven: Did you know that unit testing is a developer technique with one of the biggest returns on investment?

Motley: Great. But it's still not my job. We hire a test team. They test. That's what they do.

Maven: But does the test team know the code the way you do? Unit tests need to be white box tests - they exercise as many code paths as possible - ideally, all - and require knowledge of the code.

Motley: Testers are technical people - they can figure it out.

Maven: But, they have a lot of other stuff to do. The test team has to do performance testing, stress testing, integration testing, platform testing, and of course, the all-important testing of real customer scenarios. Do you think they have time to find all those trivial little bugs that you should have caught before check-in? The answer is NO! The have expectations that developers write high quality code. They then do the validation of the code with test scenarios that would be difficult for developers without a full infrastructure to test.

Motley: They're also the reason why our ship dates are often missed!

Maven: Only because the developers are not doing their job with unit testing. You need to do a root-cause analysis on why we have these long "stabilization" cycles. We should not be stabilizing anything! As of our code complete milestone the code should be stable. The Test team validates on the black box level - without having knowledge of the code - to ensure integrated scenarios work properly.

Motley: So you're saying that developers need to write test code??? Bah!

Maven: YES! Try it on the unit level. In the object-oriented world, a unit is often a class. Write a set of tests using a unit test framework that exercises the contracts of your methods. Run them as often as possible - ideally, after every code change. By having tests in place, you have the confidence to change your code without breaking anything. Plus, you can generate code coverage numbers to help validate the thoroughness of your tests. The test team will thank you because now they can focus on their real jobs and not cleaning up after the developers. The test team also gets some nice sample code that helps them learn more about your APIs.

Motley: Sounds like lots of benefits to have unit tests, but I am still not totally convinced. If I take care in writing my code and ensure it's bug-free, test can still concentrate on doing their real jobs.

Maven: But you can't be sure your code has minimal bugs until you test it! Honestly, if I could get devs to add one technique to their quality arsenal, it would be unit testing.

Motley: Tell you what: I'll give this thing a try. However, if I don't get positive feedback from the test team that it is helping, I'm going to return to writing real code and let them figure out if my stuff works.

Maven: I'm good with that. I think you'll come out a winner! In the future we'll talk about this idea of contracts and code coverage, but start with some basic tests.

Motley: I really look forward to that conversation - kind of like a cat looks forward to being neutered.

______________________________

Maven's Pointer: These days, Microsoft hires testers that are just as technical as developers - they just have a different role. However, since developers know the code in detail, they need to be the ones to test the details. The test team can validate the code on a higher-level to ensure integrated scenarios function correctly. By combining the test responsibilities, we ship sooner and with higher quality.

Maven's Resources: Pragmatic Unit Testing in C# with NUnit, by Andy Hunt and Dave Thomas, Pragmatic Bookshelf, ISBN: 0977616673, 2007.