Progressive Development : estimation

Point for Maven - Wideband Delphi Estimation works!

James Waletzky — Tue, 06 Jan 2009 18:54:00 GMT

Summary

Motley: I admit it - Wideband Delphi works! It helped us generate fairly accurate estimates for longer-term planning and the documentation of our assumptions kept everyone on the same page and provided rationale for our numbers.

Maven: Um, what Motley said.

______________________________

[Context: For the past few months, Motley and his team have been using the Wideband Delphi process for their estimates. Over the winter holidays, Motley got curious about the effectiveness of the technique]

Maven: Hey, Mot. Happy New Year, and welcome back to the office! Did you have a good holiday season?

Motley: Yeah, it was a good break. Although most of the time was spent with family and friends, I did do a bit of an analysis of some previous estimation data.

Maven: You have me curious - what's up?

Motley: A few months ago you gave the team some pretty good advice, I must admit. We used Wideband Delphi Estimation as you talked about before to estimate some key features for the product. The team used Scrum and gathered a bunch of data on how long it actually took us to implement the features, and I thought I would show it to you.

Maven: Sweet! Glad you were able to put it to use and see some positive results. Don't keep me in suspense!

Motley: Hold your horses, Mr. Impatient. First, a bit of background. This feature team was responsible for developing a feature with a bunch of subcomponents. We used Wideband Delphi to estimate all of the subcomponents early on in the project when we really did not know a whole lot about what we were estimating. Wideband Delphi was perfect for this exercise due to the large amount of project variability at the beginning.

Maven: How many people were involved with the feature team?

Motley: There was one program manager (PM), two developers, and two testers for the majority of the project to this point. That's not relevant for this analysis, however, as all work was estimated for one developer and computed the hours across all developers.

Maven: Makes sense. Any more background?

Motley: Yeah. One thing I would recommend to everyone is to have whoever is responsible for the initial high-level requirements put together a short 2-3 page document that describes the feature at a basic level, including some mock screenshots if necessary. It doesn't have to be highly detailed but should provide a high-level idea of what we will be building. Think of user stories, but to a slightly lower level of detail (but not much). Then distribute the doc to everyone involved in the estimation session to prepare. Have the author of the short document give an overview just prior to following the Wideband Delphi process to set context.

Maven: Great suggestion. Who did you invite to the session?

Motley: We invited the PM, the two developers, one tester and one senior developer from another team. One thing that worked really well was having that expert external to the feature team to help bring another perspective.

Maven: Okay - let's see the data already!

Motley: Hmmm… maybe I should make you wait a little longer. I love seeing you squirm.

Maven: You really are a mean person.

Motley: Just the way I like it! Fine. Fine. Here are a couple of graphs of the data I gathered:

Maven: Not bad data, Mot! Looks like many of your estimates were fairly close to the actual work on inspection. How do you explain the estimate for feature 6, which was quite far off?

Motley: That's easy - what we thought we were going to build at the beginning of the iteration actually turned out to be quite different. We did the estimate fairly early in the project and made some key assumptions, but later we discovered a couple of those assumptions were not valid. As a result, when we planned the iteration, we revisited our assumptions and realized we had more work than we thought we did. Don't forget - I am comparing our original estimates at the start of the project to our actual numbers. Secondarily, a dependency that we relied upon for that feature was late on their deliverable, so we did a bit of extra work to mitigate that.

Maven: Sounds reasonable. What about intuition - do you feel that Wideband Delphi gave you increased accuracy in your estimates.

Motley: Overall, gut feel is that Wideband Delphi gave us greater accuracy than we would have gotten had we left the estimate up to one developer. Everyone was happy with the process and has committed to keep using it.

Maven: As we talked about in the past, Wideband Delphi helps the team get on the same page at the start of the project. Did you observe that?

Motley: Wideband Delphi really helped us get on the same page with assumptions and come up with an initial direction people were happy with. Even though some of our assumptions changed later in the project, for long-term planning purposes at the start of the project, the data was accurate enough for the release management team to make reasonable predictions.

Maven: So you would use Wideband Delphi again?

Motley: Absolutely. You hit a home run with that one, Mave.

______________________________

Maven's Pointer: Our team at Microsoft uses Wideband Delphi consistently for early-project planning. When the management team forces us to come up with estimates that help define the milestone plan for the next 3 months, we use Wideband Delphi to generate not only fairly accurate numbers but also document the rationale behind the numbers. Even though detailed planning and scheduling for the next 3-6 months is fraught with peril - a realization agile teams come to - to be an agile team in a waterfall organization you must periodically give in to the will of the release management team.

Maven's Resources:

Agile Estimating and Planning, by Mike Cohn, Prentice Hall PTR, ISBN: 0131479415, November 2005.

Motley says: "Individual gut-feel estimation is the only way to estimate"

James Waletzky — Tue, 25 Sep 2007 17:53:00 GMT

Summary

Motley: Estimation is always inaccurate. Individual gut-feel estimation is the only way to go.

Maven: Wideband Delphi is a valuable estimation technique that can immediately improve your accuracy.

______________________________

[Context: Motley is about to estimate the amount of effort to develop a new feature and is reflecting on the difficulty of estimation]

Motley: You know, sometimes I really wonder what the point is of estimation. Developers are never accurate on any project longer than 3 days! The only real way to estimate is to go off gut feel and leverage the PANOOMA technique.

Maven: You're right - estimation is hard. But, fortunately there are some techniques we can leverage to improve the accuracy of our estimates. And by the way, what is the PANOOMA technique of estimation? That's not one I have heard of.

Motley: Pull A Number Out Of My Ass! That's really what ends up happening anyway.

Maven: Ha! Let's just call that "gut feel" estimation from now on, shall we? Anyway, have you tried basing your estimates on past projects?

Motley: Yeah, I try to compare what I am working on now to what I have worked on in the past, but I must admit that I usually miss something and end up blowing the estimate. It's frustrating. My boss doesn't trust me to delivery anything on time these days. I've heard of some more scientific techniques of estimating and some software packages that facilitate a more accurate estimation, but I'm not sold. Many of them also require lots of historical data, which I don't have.

Maven: Have you ever heard of Wideband Delphi estimation? That technique could help you out.

Motley: No, never heard of it. Not sure I want to. It sounds like it's extremely mathematical suited for pocket protector geeks and way overkill for our projects. And NO, I don't want to hear about it.

Maven: That's not the attitude I'm looking for!

Motley: Yeah, well, you're going to tell me anyway, so please just get it over with so I can get back to more meaningful activities, like lunch.

Maven: That's the spirit! I think. Wideband Delphi is a very easy estimation technique to learn. There's really not much to it, and it's not scientific or highly mathematical. Instead, it leverages the wisdom of crowds.

Motley: Hey, I've read that book! It surmises that an uninformed crowd can often make a better prediction than an informed individual. My buddies and I use it all the time when buying our weekly lottery tickets, but it never seems to help.

Maven: You're right on the synopsis, although lottery tickets is not the best application. Wideband Delphi leverages the "wisdom of crowds" concept using the following lightweight process:

Get a group of people together that can help with the estimation exercise. It is not important that everyone be an expert in the domain or the technology upon which the estimate is based.
Have the person that knows the most about the requirements for the work to be estimated give a quick overview of the work. "Quick" implies a minute or two - nothing too detailed.
Next, everyone takes a bit of time to think about the problem and write down their assumptions on a piece of paper or index card. These assumptions are just for you, as a reminder when the discussion portion of the process arrives in step 5.
Once you have noted all of your assumptions, you come up with one effort estimate in unloaded hours (no buffers, distractions, but instead pure effort in ideal developer days). Write down the number on a Post-It note, index card, or piece of paper and place it in the middle of the table upside down.
Once everyone has finished their thinking and written down a number, the estimates are revealed. A facilitator (anyone on the team can play the role) then notes all the estimates and a discussion ensues, focusing on the assumptions that everyone made. The facilitator then notes the team decisions that are made. Once the discussion has completed, you go back to step (3) and iterate until the numbers suitably converge. The team could agree on a final number or just use the mean.

Motley: Interesting. You are leveraging the experience of other people to help you with your estimate. I bet once you turn over the estimates in the first round that the numbers are all over the map!

Maven: Almost always! In the first round there are lots of unknowns and everyone has a slightly different take on what it takes to get the job done. When estimating a coding task, Mary may include a code review but Marvin may not. Marty may assume that he is doing the work and has expertise, but Marcus assumes that the new intern is doing the work. The discussion helps focus the team on what is required to get the job done. By learning about everyone's assumptions and making some initial decisions as a team, you inherently lead to more refined and accurate estimates. Plus, you document the team decisions and can back up your numbers to whoever asks, like management.

Motley: But why does everyone have to put their number upside down on a sheet of paper. Why not just tell everyone your estimate?

Maven: Good question! We do this for a couple of reasons. Firstly, you don't want to bias anyone with your number, so you keep it quiet and they are all revealed at once. Additionally, keeping estimates anonymous allows you to go with your gut without being pressured by others in the room. For example, if there is a high ranking architect with lots of experience in the estimation session and everyone hears her number first, they may be inclined to just go with it. However, everyone in the room adds value and should have their assumptions heard. Keeping it anonymous removes a bit of pressure.

Motley: This seems way overkill for a smaller project with fewer unknowns. I'm still back to gut feel estimation and this won't help me much. Unless, genius, you have a solution for that too?

Maven: You know me, Mot. I have a solution for everything! And for that, we need to talk about a related estimation technique called Planning Poker...

______________________________

Maven's Pointer: The most effective Wideband Delphi sessions involve estimators with different perspectives that also use different estimation techniques to derive their numbers. If one person uses Proxy-based estimation, another uses gut feel, and another uses Function Points, you can be pretty confident that the overall team number is accurate as it was arrived at using various techniques.

Maven's Resources:

From Bouncy Balls to Better Estimates, by James Waletzky, MSDN Magazine, January 2007.
Wideband Delphi Wikipedia entry
Software Estimation: Demystifying the Black Art, by Steve McConnell, Microsoft Press, ISBN: 0735605351, March 2006
The Wisdom of Crowds, by James Surowiecki, Anchor, ISBN: 0385721706, August 2005.