Several people on the Windows Internet Explorer team have written blog posts about our feedback mechanisms (our use of automated telemetry in Windows, how to write a great bug, how to submit bugs, etc) for IE9. After looking at the many similarities and obvious differences between manual feedback systems and projects, we decided to use Microsoft Connect for IE9 and eliminate the invitation requirement for filing bugs. In this post, I want to take a step back and talk about how we made that decision.
Every Internet Explorer user is a Windows customer. Listening to customer feedback is vital to the success of any business. As communication methods have evolved, how a business listens to its customers has changed as well. Our customers have always been at the center of how we envision and design IE but the way we listen has evolved both with technology improvements and with the way our customers communicate with each other.
Our Internet Explorer 8 beta customers asked us to look at different options for how we listen. We took their feedback to heart and looked deeply into all aspects of our feedback systems as well as those used by other software companies and in other industries. We also looked into research regarding the effectiveness and efficiency of various ways of listening and responding to customers. Most importantly, we listened to real customers, real enterprises, and real developers.
Each time we start a new project we begin by stepping back and looking across the feedback from the previous project and the engineering process by which we collected it. We challenge all assumptions, especially those based on speed, size, or connectivity. We also talk with people to see what they like and dislike about our competitors’ product and engineering process choices to see if Microsoft’s customers might benefit from something similar or more advanced.
One of the assumptions most software organizations appear to take for granted is the need for beta releases. In fact, some companies have even taken this assumption well beyond its commonly understood meaning. So we asked ourselves, “Should we do betas and, if so, how many? Should we do nightly builds? Should we continue to use Microsoft Connect or move to something else?”
A group of us sat down and thought through the customer benefits and drawbacks of various models. Ultimately, there are two main benefits to public releases:
There is a continuum of feedback models that ranges from completely closed to completely open. Examples of completely closed betas include new PC models, cars, toasters, etc. Examples of completely open models include some academic open source projects, school PTA projects, etc. We’ve chosen something near the middle so we can get broad feedback on quality events but not completely anonymous or unstructured either because we believe in responsible engineering.
From our point of view, the most important thing about working on a consumer technology like a browser is respecting people’s scarce time and energy. You can do this while still achieving your goals of getting everybody ready (from web developers to support professionals to corporations) for product launch and getting feedback on quality issues that only surface in unique configurations. I want to call out a few important aspects about feedback systems:
We also looked at various feedback tools when we started the project. These included things like Bugzilla, Mantis, Launchpad, etc. We compared them to Microsoft Connect, which is what nearly every Microsoft product uses to get beta feedback from customers. It turns out that these tools are all amazingly similar.
We looked more deeply at Bugzilla and projects using it because it’s a tool that at least two other browser vendors use today. We looked at the tool itself and how it handles bug workflow both from a beta site perspective and a product engineering perspective. We found some really interesting similarities and differences:
We made some engineering decisions on how to get the best possible feedback from our beta customers after doing all the research into different release cadences, feedback models, tools, and bug management processes. We want to respect our customers’ time and energy so we’re going to distribute more focused Platform Preview builds when there are new platform features ready for people to test drive.
Thanks for all the great feedback in IE8. We’re looking forward to building a great IE9 release!
Jason Upton Test Manager – Internet Explorer