Progressive Development : root cause analysis

Motley says: "Reflection is for mirrors" (Scrum Part IV)

James Waletzky — Tue, 12 Feb 2008 19:50:00 GMT

Summary

Motley: Reflection is for mirrors. We just finished a sprint. Why open up old wounds?

Maven: At the end of a sprint, do a demo for stakeholders, customers, and the team. Do a retrospective to reflect on what went well and what requires improvement. Continually improve the team from one sprint to the next.

______________________________

[Context: In a continuing conversation about Scrum, Maven is about to enlighten Motley on the most effective ways to wrap up a sprint, or short iteration]

Motley: At the end of the sprint, we followed the Scrum process. Everyone did a quick one minute summary of the work they accomplished during the sprint. We then chatted for 5 minutes on how things went. That's it. Don't tell me there's more. This is a lightweight process, right?

Maven: I guess "lightweight" means different things to different people. There are some best practices for the demo and retrospective. It is a valuable exercise to take time to reflect on the past sprint.

Motley: Reflection is for mirrors. What's the point? We just finished the sprint - why open old wounds?

Maven: Hopefully ripping apart flesh is not a good analogy describing your last sprint. It is extremely important to reflect on your last iteration. Let's take the demo. You may wonder why we bother demoing the work we just did. We met on a day-to-day basis and everyone was aware of what each other was doing on a high level. Well, the demo serves a few purposes:

The demo allows stakeholders that may not be directly involved in the sprint to get a first-hand look at the progress the team made. Feel free to invite people not directly involved in sprint work.
The demo allows your customer(s) to provide feedback on the work you did, allowing the team to fold in changes in priority or requirements into the next sprint. Remember, customers love to see working software. Solicit them for feedback and changing priorities and update the product backlog. Incorporate that feedback into your next sprint planning session.
The demo is a chance for the team to celebrate their accomplishments over the past sprint. Seeing working software can really be a morale boost for the team.

Motley: One problem I have with all this is that I don't want the team spending a bunch of time creating demos for the end of sprint meeting when they could be doing real work. Seems like a waste of time.

Maven: Actually, that is a common misunderstanding. You are not out to create a really flashy demo to show off the product. You are simply doing a demonstration of the functionality you just built. There should be no extra gold plating just to look more impressive at the demo meeting. It's not about creating a glitzy demo.

Motley: But what if some members of the team did not work on software directly? They are going to feel left out at demo time.

Maven: Again, don't take the word "demo" too literally. A demo could be a review of a wiki page that was created, a tool somebody wrote, a document that is important to the product, a description of a consulting engagement, or anything else concrete that the team can see some created value from. The demo is typically a 1h meeting that occurs during the slack time between sprints. Make it fun. Bring donuts or burn CDs with interim builds for your stakeholders. Think of the demo as a team building exercise with a celebratory feel.

Motley: Ok, we obviously have room for improvement there. What about the retrospective? What's the point? We just finished the sprint, so we know what just happened.

Maven: The retrospective is a chance to do two primary activities:

Examine what went well in the sprint. It's important to analyze those processes and team procedures that lead to success and to continue to do them. Having a clear idea of what is working ensures everyone on the team knows the best practices and allows you to encourage them to continue doing them. It is also a bit of a morale boost knowing that what you are doing is working. You also want to avoid turning the discussion into a negative whine/gripe session. Remember the retrospective prime directive:

"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."

From <http://www.retrospectives.com/pages/retroPrimeDirective.html>

Examine what requires improvement. Perhaps the biggest reason for a retrospective is to focus on continuous improvement, which is one of the principles of agile development. In Japanese, this is called Kaizen ("Kai" = change, "Zen" = good), where you look a system and consider the results and process to get to the results with a non-judgmental focus and goal of improving how you generate results. We want to look at how we can be more effective in the next sprint. Think about what you would do differently next time. Use any metrics you gather for quantitative analysis and use those measures to drive improvements (e.g. Were the estimates off? Why? How can we improve?)

Motley: So we have a quick chat and then improve for next time. Easy.

Maven: Well, there is a bit more to it:

Clarify success. As a team, determine what success would have looked like for this sprint, and will look like for upcoming sprints. "Begin with the end in mind", as Dr. Stephen Covey says. If we apply Human Performance Technology (HPT) as we discussed previously, we can measure the gap between our current state and desired state, determine root causes, and implement interventions to help the team succeed.
Ask everyone to prepare. A bit of up-front reflection prior to the retrospective meeting helps ensure the conversation is valuable.
Give everyone a chance to speak. Chances are good that most people on the team are critical thinkers with a desire to improve. Ensure everyone gets a chance to contribute at least one thing positive about the sprint and one thing negative. Draw out the introverts - they have opinions too. Treat the exercise as a brainstorm - there are no wrong answers. As a last resort, make the comments anonymous.
Create tasks for improvement areas. Write down everything that is said in the meeting, particularly areas for improvement. Create concrete action items out of the improvement areas with assignments and due dates. Put tasks on the next sprint if needed. Make sure the discussion leads to positive change.

Motley: Jeez, Mave. You always put so much structure on everything. Can't we simplify?

Maven: Actually, this is quite simple. Most of this is just small tips to make you more effective. Just remember to focus on what went well, and what requires improvement.

Motley: One other thing that still isn't clear to me though. What the heck was the point of gathering all the data during the sprint? The data seems as useless as most other metrics. You didn't even mention it as part of the retrospective. We spend a lot of effort capturing them, and then no one ever looks at it again.

Maven: Actually, there is great reason to capture the data. And yes, we do use it in the retrospective and throughout the sprint. It's getting late though. Why don't you head home for the day and we can pick this up in the morning.

Motley: Ah, I get it. That's code for you having to head home and do whatever geeky thing you do in the evening. Have fun washing your pocket protector!

______________________________

Maven's Pointer: One of, if not the primary reason for short iterations in agile development is to ensure we get rapid feedback. We want to create a culture of continuous improvement of the product, the team, and ourselves. Although sometimes feedback may be difficult to accept, it is necessary for forward progress. Encourage feedback early and often on not just the product but all parts of your life. It is the best way to get better!

Maven's Resources:

http://www.restrospectives.com: A web site dedicated to the growing practice of looking back to move forward. The retrospective prime directive is found here.
Agile Retrospectives: Making Good Teams Great, by E. Derby, D. Larsen, K. Schwaeber, Pragmatic Bookshelf, ISBN: 0977616649, July 2006

Motley says: "Consulting? I guess your answer from here on out is 'It depends'" (HPT Part 1)

James Waletzky — Tue, 27 Nov 2007 19:08:00 GMT

Summary

Motley: How could a consulting methodology help me? There's no way - I work in a product company.

Maven: Human Performance Technology (HPT) is an effective consulting methodology that uses a simple process to identify the root cause of human performance issues and build creative solutions to those problems.

______________________________

[Context: Maven and Motley are having a discussion on what Maven used to do before joining the company]

Maven: Did I ever tell you that I was once in the consulting world?

Motley: It depends. Hahahaha. Isn't that the consultant's answer for everything?

Maven: Very funny, wise guy.

Motley: Yeah, I'm quite the comedian. I'm not surprised in your background though given that you are always trying to solve people's problems around here. I even heard you talking to Morton the other day about some problems his daughter is having. You get your hands into everything!

Maven: Sometimes I can't help myself. I love to see the world improve! Did you know there is a method to my madness? I use a technique called Human Performance Technology, or HPT. You don't have to be a consultant to use it - all you need is a systemic human performance problem to solve. It's more of a problem solving technique than a consulting technique.

Motley: "Human Performance Technology"??? You make it sound like we're all robots. So you use computers to pop out the answer when faced with a consulting problem?

Maven: Not exactly. "Technology" is used in a broader sense. Here is an overview of the technique:

Motley: Wow. That looks like a bunch of psychological gobbly-goop. I'm not sure I even want to hear all the background on that stuff. Looks complicated.

Maven: Actually, it's quite easy! Here's how-

Motley: Wait! I, errrr, have a plane to catch.

Maven: You have to work on your "stretching the truth" skills, bud. Anyway, this won't take long. What do you do when you have a problem to solve? Say, for example, someone comes to you indicating that builds keep breaking. What is your process for solving that problem?

Motley: Easy. Put continuous integration in place as we discussed.

Maven: You've been paying attention! Well, continuous integration may or may not be the right solution. The trick is not jumping to solutions, but examining the problem. The first step is to assess the business need.

Motley: Well of course the business needs no build breaks.

Maven: Yes, but it is important to understand why. What is the impact to the business' bottom line? Should we solve this problem over something else? What makes this problem more important than others? You want to ensure you are putting effort into the right problem.

Motley: Well, for build breaks, it's pretty obvious. The daily build is our lifeline. Without a build everyone on the product team is blocked and we potentially delay shipping the product, which affects the business. It's also a quality issue.

Maven: Cool. Next you want to think about what your ideal future state looks like. This is called your "to be" state, or where we want to be when we are executing near perfectly. What is that ideal future state?

Motley: Hmmm… for builds, we want a build without compiler and link errors every day. I would also think we want the Build Verification Tests (BVTs) to consistently pass. In addition, I want statistics gathered on every build around code churn, I want instantaneous communication in the rare event there is a problem with the build, I don't want a full-time build person looking after it, and most importantly, I want a latte automatically made for me every time the build passes.

Maven: Well, I'll try and forget the last one, but the rest are great! Before we can get there, however, we need to understand the current state. What is the state of the build today?

Motley: Periodic build breaks that still happen too frequently. Developers that are still too sloppy. We've been trying to address the issue with continuous integration (CI) but the CI server is being overloaded by the problems that developers are checking in. It's far less than ideal. Plus, I am not getting my lattes.

Maven: Now that we have an understanding of where we are and where we want to go, it's important to understand the gap that is getting in the way of us getting there. Not only do we want to understand the symptoms but the real problems that are causing the gap.

Motley: Right. Back to the root cause analysis discussion we had. The "Five Why's" for example.

Maven: You bet! In addition to asking "why", we can leverage the six dimensions shown in the diagram above to help us get to the root cause of our issues, and also help us come up with creative solutions to our problems.

Motley: I thought you said this won't take long!!!

Maven: Bear with me here. Doesn't this model sound great? Don't you want to know more?

Motley: I just felt another gray hair pop out, and it's your fault. Fine - finish off your model...

______________________________

Maven's Pointer: There is a central organization that focuses on HPT called the International Society for Performance Improvement. HPT analysis leverages various other disciplines including behavioral psychology, instructional systems design, organizational development, and human resources management. The central idea is to improve human performance by focusing on measurable results while looking at the big picture (a systems view). HPT is a systematic approach at identifying and solving performance problems in the workplace and beyond.

Maven's Resources:

International Society for Performance Improvement: http://www.ispi.org
What is HPT? http://www.ispi.org/hpt_institute/#What

Motley says: "Ugh, not again..."

James Waletzky — Tue, 13 Nov 2007 18:47:00 GMT

Summary

Motley: Developers keep making the same mistakes over and over. Can't they learn?!? Some kind of punishment is in order, like no more free soda!

Maven: Don't just address the symptom - get to the root cause of a problem. Leverage the "Five Why's" to find root cause and come up with solutions that are realistic, cost-effective, and within your control.

______________________________

[Context: After taking part in a few structured code reviews, Motley is getting frustrated by the patterns of errors he is seeing]

Motley: You know, I'm almost sorry we started this code inspection thing. As the senior developer, I have been participating in quite a few of them. It has really come to light that (a) human beings are creatures of habit, and (b) we don't learn from our mistakes! Every review that we have finds the same kinds of errors. Can't people learn?!?!? Perhaps we should revoke the free soda as punishment.

Maven: I feel for you. The real question is, what can you do about it? Taking away soda is not the answer.

Motley: I can think of lots of other things too - beating on people, making them wear dunce caps, cutting their pay, -

Maven: Whoa! Let's try to be more constructive. How can we proactively cut down on the problems that occur? If you keep getting flat tires on your car, you could first consider fixing the symptom. Patch the tire. Buy run-flat tires. Both of these solutions won't solve the real problem, and you'll still keep getting flat tires.

Motley: Well, duh! You want to figure out why you are getting the flat tires. This is just common sense. Is it a fender rub? Do I have a cheap brand of tires? Is there something protruding from the wheel? Am I driving in an area that is prone to junk all over the road causing flats? Could be a multitude of reasons.

Maven: Exactly! Why not apply this same principle to software bugs? If we can figure out what the root cause for some of the bugs we find, perhaps we can prevent the problem from occurring at the source and not make the same mistakes again.

Motley: Well, um, of course. There's nothing to say, however, that the developers will actually address the root cause and not make the same mistakes. And with software, it is usually quite difficult to actually determine the real reason for a bug.

Maven: Well, once you find the root cause, there are some creative solutions you can usually apply to help developers do the right thing. Before we address that, let's talk about how to do root-cause analysis. One simple way is to leverage "The Five Why's". The idea is that if you ask "Why?" five times you'll ultimately get to the root cause of an issue.

Motley: Why five? Oh, wait. I know. If someone keeps asking me "Why?" after about 5 times I'll get frustrated and provide a knuckle sandwich for their effort. Five is about all I can take.

Maven: Actually, I don't know why it's five. I treat it as a guideline anyway, and not a hard and fast rule. Want to try an example?

Motley: I'll humor you, just please don't annoy me.

Maven: The foreman of an assembly line shuts down the line because Mark got injured. The manager could just start up the line again or he could pursue the cause of the shut-down. You play the manager.

Motley: Ugh. Fine...

Maven: Well, ask the question!

Motley: Ok, why did the line shut down?

Maven: Very good - you are a good actor! The line shut down because Mark slipped in a puddle of water and hurt himself.

Motley: Why is there a puddle of water on the floor?

Maven: Very good! There is a leak in the roof and it rained last night.

Motley: Jeez. Why is there a leak in the roof?

Maven: The rain storm last night also exhibited high winds and some tree branches were blown into the roof.

Motley: Why was it windy?

Maven: Ha. Ha. Nice. The key is to keep the "why" questions realistic. If you were to ask me "Why did the tree branches poke a hole in the roof?" I might answer because we have some trees that are too close to the building. At this point, we're probably done. Our root cause for the line being shut-down and Mark getting hurt was because trees are too close to the building. These two seem unrelated at first glance, but once you delve into the details, it becomes clear how to fix the root issue.

Motley: That's a pretty ridiculous example!

Maven: Well, it's fictitious, but the same technique can be used with bugs. Keep asking "Why?" until you get to the root issue for the bugs.

Motley: Okay, back to my previous question. Once we find the root cause we can tell developers how to fix it, but there is no guarantee that they will.

Maven: Yes, sometimes that is true. However, there are certain types of solutions we can put into place to minimize the odds of the bugs creeping in again. Here are some examples:

Run tools proactively: Before we check-in code, we can run static analysis tools like FxCop to proactively find bugs on our behalf. FxCop is great at finding security and other quality issues. Even before check-in we should run this kind of tool prior to code review to nail the obvious ones before we use up the time of the team.
Update tools to find our common bugs: Most tools like FxCop have an extensibility mechanism. If we consistently make the same error and the tool currently does not look for it, consider creating an add-in that finds the issue in code.
Update coding standards: The developers should all be very familiar with our team coding standards. Make it mandatory reading. Add patterns of errors to the standards document to ensure new developers in particular do not make the same mistakes.
Update team checklists: Checklists are important for reviews. If a mistake keeps happening, add it to the checklist that developers use during code reviews.
Get training: If mistakes are happening in an area that team isn't very familiar with, consider books or training to fill the gap. If this is our first foray into multithreading, for example, take a course on how to develop concurrent applications.
Change development processes: Perhaps the steps that are required before code check-in need updating. For example, consider updating the process to require a mandatory personal checklist-based review before a team review before using the time of others.

Commonalities between all of these solutions are that:

They are realistic. Doing a rain dance most likely is not going to solve the assembly line issue.
They are cost-effective. Building a dome around the factory is not feasible.
They are within your control. Controlling the weather is not within your control.

Motley: We obviously cannot afford the time to do root cause analysis on every bug!

Maven: No one says you need to do this for every bug. Concentrate on using this technique for patterns of bugs. One place to make use of bug root-cause analysis is for a few minutes at the end of an inspection. Have a quick chat that covers the following:

What went well with the inspection?
What could we improve the next time we do an inspection?
What is the root cause of some of the major issues we found?
What can we do to prevent those issues from re-occurring in the future?

Motley: I am always up for preventing boneheaded mistakes from appearing over and over again. We'll try it at the end of the next inspection to start, and then perhaps think about some of the other patterns of bugs we've seen over the past couple of weeks and address them.

Maven: Awesome! I think the team will be better for it moving forward...

______________________________

Maven's Pointer: Root cause analysis is used in many different industries from accident investigation to automotive assembly lines to healthcare. Addressing the symptoms of a problem are usually temporary fixes. Individuals and businesses that believe in continuous improvement must dig deeper beneath the surface to ensure the real problem is being addressed. Fishbone diagrams (a.k.a cause-effect diagrams or Ishikawa diagrams) are a tool to help you visualize and delve deeper into problems to discover root causes. I prefer a classic mind map, but the principles are the same.

Maven's Resources:

FxCop: http://msdn2.microsoft.com/en-us/library/bb429476(vs.80).aspx
The Five Why's: http://en.wikipedia.org/wiki/5_Whys, http://software.isixsigma.com/library/content/c020610a.asp
Fishbone diagrams: http://en.wikipedia.org/wiki/Ishikawa_diagram, http://www.isixsigma.com/library/content/t000827.asp (with Word and Excel templates)