About all those 'bugs', Robert...
What a great way to kick off this blog. Someone sent me a great link this week, Robert McLaws has an interesting analysis of Vista and Longhorn feedback on his blog.
It's pretty good based on the way he extracted the data. In fact there are a lot of reports that arrive marked ‘private’ and ‘organization’ that are missing from the data though. It’s important to note that the universe of ‘bugs’ we’re talking about includes everything from actual Bugs, to suggestions, Design Change Requests, requests for driver support, spelling and localization errors to name a few.
We have a lot of different programs, and the actual number of reports we’ve received differs based on whether you count TAP programs, non-english bugs, the CPP and such so it’s difficult to answer in raw numbers without a long discussion of which programs we’re counting, are we including DCR’s and suggestions or not. Not all reports are visible directly to you because of the ‘organization’ and ‘private’ flags on bugs among other things.
We have less than 10% of the total reports received that are still ‘active’. Some of these are actual bugs and will be fixed before we ship Vista. Some are actual bugs but are server specific and will get fixed before Longhorn server ships. Some are suggestions or design changes and will remain active and be tracked for inclusion in the next version of Windows. Still others that are currently active will wind up with some other resolution after we continue to review them.
Over 25% of all reports received to date are resolved and can be tied directly to a code change. Almost 40% cannot be linked to a direct change but are not reproducible for us at this point. Most of those are well known as fixed, but it’s not efficient to spend hours digging through the code to find the precise change. The remainder are reports that are resolved as External (may require follow up with 3rd parties), By Design (Working as intended), Won’t Fix (No change is going to be made right now, often this is really ‘Can’t Fix’) and a few Dead Code (a case where someone filed a bug that exists in code that will be replaced before we ship). It makes no sense to go do work in this area say, at Beta 1, when you know that by Beta 2 your replacing the whole module.
About the increasing quantity of bug reports – what is not factored in is the quantity of people we are adding to our programs. We are constantly adding new people to our programs to get more feedback. That coupled with the sheer interest generated as we approach the final stages tends to account for the uptick. Just the technical beta program alone has gone from 10,000 people to over 26,000 since Beta 1 and we’ve created entirely new programs as well. What is significant is that the severity reported bugs is dropping. We’ve moved from seeing the serious issues you see in a Beta 1 to a very usable system that generates more ‘this doesn’t work quite right’ issues at Beta 2. You will still find an occasional person with a persistent system problem and we continue to investigate those, but we now have those down to a very manageable quantity.
The daily incoming rate that is graphed can be an inaccurate number. We do take the system down from time to time for maintenance and then we queue the bugs up. A weekends worth of bugs may arrive on Monday. That is the case with that near 600 bug day in June.
He is correct that our automated systems that send back failures help us immensely. We can quantify various types of failures without the user having to file a bug report about it.
Thanks, Robert for recognizing the work people are doing here! Everyone here realizes that for Vista to be great we have to include the feedback from our customers in the product. We’ve been doing that all along, but in particular after Beta 2 we’ve made great progress. RC1 should reflect this when it’s available later this year.
