In Rusty Miller's earlier post about Product Feedback Center, there was a comment about how bugs are black-and-white - it's broken, so we should fix it - and therefore voting doesn't make as much sense. I’d like to delve into this a little more.
At a broad level we agree – bugs are bugs and we’d like to ship the product with 0 bugs. However, there comes a time in every release cycle where we start weighing getting what already works well shipped vs. fixing the remaining bugs. We get feedback like “The product already works well enough for my needs – please ship it so I can meet my own business needs.” In addition, we’ve learned through hard experience that every time we fix a bug there’s a chance we’ll break something else – it’s a large, complicated product and it’s virtually impossible for a developer to test every aspect before checking their fix in. Between these two factors, as we get closer and closer to a milestone (beta1, RTM, etc.) we raise the bar on bugs. 4 months before a milestone there may be no restrictions on what can be fixed. 2 weeks before a milestone, when most of the product looks pretty good (where “good” varies depending up on the milestone), we only fix very major bugs – loss of data, crashes in mainline scenarios, etc.. For betas this is understandable – if need be we can fix it after the beta. But this process happens even for the RTM release, for all the same reasons. A common question that gets asked during the final push is “why are we seeing this now?” (as in, if this were really important, it would have come up earlier). Sometimes it’s a important issue and we just weren’t thorough enough, but a lot of times it’s because the bug doesn’t affect many people.
Throughout the process there are a ton of judgment calls based on our understanding of how people use the product, where we are in the cycle, how reliable/fragile a piece of code is, the impact of the bug on the user experience, whether there’s a simple workaround, and probably a dozen other considerations. It’s not a perfect process (as every bug you see demonstrates) and we always seek to do better. It does enable us to ship though which in turn enables a lot of customers to start deployment, release their Whidbey-based product, etc..
Our goal in enabling people to vote on bugs is to help us make even better judgment calls, and provide a check/balance in the cases where we make the wrong decision. If a team isn’t sure what to do, having customers say “this bug does/doesn’t matter a lot to me” is very helpful. Similarly, if we’ve decided a bug isn’t going to affect many people, and customers disagree, we’re going to make that visible within the division – in some cases I’d expect us to change our plan. It’s sort of of a manual version of Watson (that “we’re sorry, there was a serious error, would you like to submit data to help us?” dialog you’ve seen) for non-crash bugs. That Watson data is super-helpful in prioritizing our work – we start with the Watson bugs with the most hits (we’re able to clump reports we receive together based on the data that is returned) and work our way towards the ones with few hits. Regardless of the time/resources we have, we know that that approach maximizes the impact of our work. Voting is an attempt to get a similar dynamic going with non-crash bugs.
One question we’ve had internally is whether voting for bugs is exposed the right way. Today we have validation (“I’ve seen that too”) and voting (“I think this is/isn’t really important”). With Watson (which has been a huge help for us) we just get the “me too” – but then we know that all Watson bugs are bad because they are crashes. Should we keep the “voting for bugs” feature, or should we go to a very simple “me too!” mechanism where people can tell us they’ve seen the problem? We’d lose the qualitative opinions - which few people are giving us now anyway J - but end up with a Watson-like view onto which bugs are affecting the most people.
We'd love your thoughts on this. Is voting going to take off if we just give it enough time and explain why it matters, or do we need to reconsider how we've exposed voting and validation?
Mark Cliggett
PM Lead - DevDiv Community Team
blogs.msdn.com/markcli
