Does it meet the bar?

Does it meet the bar?

Rate This
  • Comments 6

When the team moved up to Microsoft from Connectix one thing that was quickly evident was that there were a number of 'cultural' differences between the two companies.  This was highlighted by the fact that when we started the entire program management and development teams were from Connectix while the majority of the test team was from Microsoft.  We have spent a lot of time trying to resolve the differences and want to end up with the 'best from both worlds' - however we still have a number of areas that need work :-)

One 'Microsoftism' that I have enjoyed having as a program manager is the concept of a 'bug bar'.  This is a list of requirements that a bug must meet in order to be fixed in a milestone or release.  Examples for a service pack would include things like 'Customer reported problems', 'Application crashes' or 'Data loss bugs'.  Then as bugs come in during the development cycle they are assessed against the 'bar' and a decision is made to fix the bug or to postpone it to a future release.

This is invaluable for a number of reasons.  Firstly, it helps to ensure a consistent quality approach for a release as it stops us from ignoring major bugs, and from wasting time on trivial issues.  Secondly, it helps to be able to give a precise answer as to why a certain bug did not get fixed in a given release.  Finally, it is great to be able to review the fixed and postponed bugs before shipping a product and be confident that the right changes have been made to the product.

However the thing that sticks in my mind about the 'bug bar' is this:

At Microsoft there are regular bug triage meetings where bugs are reviewed and assigned to an appropriate person.  From time to time a contentious bug will be reported - where people are divided as to whether this bug should be fixed, and if it should be fixed how it should be fixed, and what the potential impact of the fix is.  Invariably a minute or two into the discussion someone will ask 'Does it meet the bar?' Answering this question will resolve all debate.  If it meets the bar it should be fixed.  If it does not meet the bar we will get it next time.

Cheers,
Ben

Leave a Comment
  • Please add 6 and 5 and type the answer here:
  • Post
  • This was along the lines of a bug triage discussion I was having just yesterday, and I'm curious as to what your response would be.

    The bug: The column widths in part of our application don't maintain their size across runs of the application.

    The resolution: It didn't meet the bug bar we had set(this is a minor UI issue) so it was deferred to the next release.

    Being new to this company's product release cycle, I asked, "At what point will this bug ever clear the bar to the point it has to be fixed?" Unfortunately, no real answer was given.

    So do you(Microsoft) just have bugs that get deferred from release to release for eternity, never getting fixed? Or is there a bug bar level that states fix all these very minor ones to get them out of the system?
  • We have something similar; but the length of time an "issue" has been around contributes towards the bar as well. Customer complaints would tend to lower the bar (-:

  • Hi Casey,

    Yes - I have shipped my fair share of similarly minor UI bugs because they didn't meet the bar. There are a couple of ways to deal with this.

    The first way is to wait for a customer to report it - and then it will meet the bar (and if no one reports it - it is quite arguable that you should not spend time fixing such a minor issue).

    The second is to have a development process where the bar is 'low' during the early development stages - but gets 'higher' as you get closer to shipping. This is the approach that is usually taken at Microsoft. In fact - a number of teams at Microsoft have instituted a 'quality' milestone at the begining of a new release cycle - which is dedicated to going back and fixing the bugs that did not meet the bar in the last release.

    Cheers,
    Ben
  • Regarding the "quality milestone" -- remember that at the end of a product cycle the bar gets extremely high. I've seen the situation where a crash is deemed to not meet the bar because we think users won't ever hit it, and will be able to cope if they do. (It's rare, but it's real.)

    I've also seen "dogfooding-only" bugs - bugs that only affect people internal to Microsoft using interim builds. At the end of the cycle, we may punt the bug because we think fixing it won't help users. This is a hard choice, because we know that compromising the experience of dogfooders will in turn lessen the quality of feedback we get from dogfooders.

    As soon as the current release ships, it makes sense to fix these kinds of issues right away.
  • Regarding customer feedback & the bar - whenever we write down the current bug bar, we include "has this been reported by a user?"

    If a single user notices a bug during a Beta, that probably means that many will notice it in the final release.

    With a bug from testers, you always have to guess whether it will really be something users will notice. With a bug from a user, you already know!

    I spend a lot of my time triaging bugs; user feedback always gets extra consideration.
Page 1 of 1 (5 items)