I will share a straightforward method to help development teams understand and communicate their ability to address the bugs in their areas. Not all product teams in Microsoft use a method such as the one I will describe below, but many do and there is much to be learned from this exercise.

The advantage benefits of this method:

  • A simple method for multiple feature teams to have a product-wide consistent conversation about their bugs
  • Enables one to see the impact of other activities on bug count
  • When historical data is available, allows one to estimate how future milestones will affect the bug count

[UPDATE 2010/12/18 – A draft version of the text was originally posted by mistake. The version below has been revised to bring it closer to the intended final version]

BASICS

First, we need to pick a unit of time for this method. The most convenient is weeks instead of individual days. There are two benefits. First, there are fewer values to plot. Second, we don’t have to account for handling weekends in the calculations.

The number of bugs that are to be fixed at the start of a given week (Monday) is that week’s BUG COUNT. To compute the bug count for week n is simple. Take the bug count for week n-1 and add the net effect of all activities that affected the bug count.

How we compute the “net effect” (we’ll get to this later) is the strength behind the analysis.

image

 

A STATIC SYSTEM

We start with the simplest case. We have start with a bug count of 87 on week 0. Everyone is on vacation. Nobody is entering new bugs. Nobody is fixing them.

The chart below is the current model. It is nothing more than the BUG COUNT for each week.

image

As a convention, cells with red text with a gray background indicate calculated values. Cells not marked in this way are considered input that we can modify,

The table – its form will change throughout this post – is the numeric model of the bug count.

To assist in visualizing the predicted bug count, I will use a simple column chart with the table. Below, you see it matches the table – the bug count is static and does not change week-to-week.

image

 

INCOMING

Every week people enter new bugs. This is the INCOMING RATE.

For now I will simply assert that 40 new bugs are filed every week. I include it the model by adding a row called incoming. As you can see below, the bug count is growing linearly – 40bugs/week.

image

image

 

BUT HOW MANY ARE KEPT?

When a feature team looks at the new bugs filed, not every bug is kept. Some of them are dismissed immediately.

The reasons can be any of the following:

  • The bug is a duplicate of an existing bug
  • The underlying issue was recently fixed in a build later than than the bug filer is using
  • The person who filed the bug has a misunderstanding about the issue or performed incorrect steps – the bug report itself is invalid
  • The bug has nothing to do with the code – it is not actionable by a developer. So, We do not count it a “code defect”. An example of bugs we might exclude are those relating to out-of-date documentation.
  • The bug describes a behavior the filer of the bug considers erroneous but that the feature team identifies as an intended behavior.

It’s not uncommon that 50% of incoming bugs are discarded for such reasons. This ratio of kept versus the incoming is the KEEP RATIO. For example, if we keep half of all bugs filed then then ratio is 0.5 and so we apply  it to the weekly incoming number.

image

Instead of growing 40bugs/week, the BUG COUNT grows by 20bugs/week. The situation is still bad – it continues to trend up - but at least the trend is not as steep.

image

 

WHAT IS THE CAPACITY TO FIX?

Now we account for the bugs fixing activity of the developers.

Although the number might seem low, we start by assuming that a developer can fix 1 bug each day. Thus the rate is 5 bugs/week for 1 developer. With a team of 6 developers, the capacity to fix bugs is 6 * 5 = 30 bugs/week. This is the FIX CAPACITY.

Now we  include the FIX CAPACITY to the model. I’ve added a net change row to make the change per week obvious.

image

The model reveals the trend is now shrinking - down by 10bugs/day. This is good, the trend is desirable. Still, the team will not hit zero bugs by week 6. 

image

 

NOW WE MODEL

To MODEL the bug count is to take into account the activities of the team and see how it affects the bug count. In practical terms by considering the behavior of the team – we will directly modify the weekly values for incoming, keep ratio, and fix capacity and then see how those modifications affect estimatin.

EXAMPLE: THE “TEST PASS” ACTIVITY

Some teams may schedule a “test pass”. In a test pass, the test team will formal run through every set of test cases and scenarios they can think of.

Suppose that the test pass occurs in week 4. Based on some historical data (which would be best) or some guesswork (which is acceptable to start) we can state that the incoming rate will be 2x the normal one and we will keep 75% of the bugs in that week. The reason for the higher KEEP RATIO is that we believe that the test team is catching more real issues with the product via their concerted effort during this week. 

 

image

Accounting for the test pass, the team is further from the goal because of the influx of bugs from week 4.

image

 

EXAMPLE: RAMPING DOWN DEV & TEST ACTIVITIES

The focus of dev and test in a phase shift. For example, after the test pass the test team may not be looking as actively for bugs – they may be moving on to planning activities for the next phase of the project. We adjust for that effect by reducing the incoming rate in week 5.

Likewise the dev team may be moving from fixing bugs to writing design documents for the next milestone so we need to account for that as well – in this case we’ll say that a single developer is moving from bug fixing to planning. Thus the FIX CAPACITY goes from 30 to 25.

image

We are still at 57 by week six!

image

EXAMPLE: ASKING FOR HELP

The model reveals that the team is not going to hit zero bugs by week 6.

At this point the team requests for developers to be temporarily reallocated from other parts of the product to this one. The general manger will rightly ask how much of a difference will this make. And because we have a flexible way of modeling the FIX CAPACITY – we can show the effect and clearly explain the computation.

Suppose the team request for:

two additional developers for w0 and w1

only 1 additional developer for w2, w3, and w4.

It would affect the numbers like this …

image

 

Look at week 6. Week 6 starts with 22 bugs, at the bottom the net change will be –20 bugs. This means week 7 begins with 2 bugs, So now we have a reasonable expectation that with additional help we will be done on early in week 7.

image

 

BUG MODEL EXCEL WORKBOOK

You can find the XLSX file used in the calculations above on my skydrive:

http://cid-1ff099edb1c7ebfa.office.live.com/self.aspx/Public/Blog%20Posts/MSDN%20Saveenr%20Blog/2010/2010-12-18-Bug-Model

 

PARTING THOUGHTS

  • Where do the numbers come from? You’ve got to look back to the history of the project – there will always some guesswork and estimation involved. As you collect data week by week you can refine your models as needed.
  • This a simple model. As presented I only wanted to give you a flavor of how to start your own analysis and modeling.
  • It isn’t always necessary. This as an analytic tool to help teams think about their project. It may be too much overhead for some teams. But having overseen multiple teams for many years at Microsoft, I think it can bring some useful rigor to project execution.