Almost 9 months after shipping Beta1, Beta2 is shipped and in your hands.  One big difference between VS2005 Beta1 and betas of our previous products has been the amount of community involvement.  Beta1 was the debut of the MSDN Product Feedback Center, which improved upon the previous bug reporting mechanism in a couple key ways:

  • It improved transparency through the way it synchronizes data back and forth from our internal bug database to the MSDN Product Feedback site. In particular it enabled MS employee’s to add comments to bug reports, which get pushed out to you.
  • It (finally) provided a more objective way to rank issues through voting and the importance rating.  This gave us a better picture of how many users were affected a particular issue.
  • Now that it was opened to the public (as opposed to you only being able to see your own bugs), it encouraged sharing workarounds, searching for duplicates before entering bugs and discussing/debating a particular issue through discussion threads.

This helped remove some of the barriers between the internal product team and customers.  The results so far have been great!  Here is a summary of the resolutions of C# and the Visual Studio Debugger feedback.  I’ve also included resolutions for bugs entered by MS employees for the same period for comparison.

Resolution of bugs reported since Beta1:

IssueType Source Total Fixed Postponed Won'tFix ByDesign NotRepro
CodeDefects MSDNFeedback 555 240 59 8 101 142
MSDNFeedback 43% 11% 1% 18% 26%
Internal 45% 11% 4% 10% 10%
Suggestions/DCRs MSDNFeedback  567 51 286 39 131 45
MSDNFeedback 9% 50% 7% 23% 8%
Internal 23% 45% 13% 8% 3%

For code defects, the fixed and postponed rates are almost the same between bugs reported through MSDN Feedback and those reported internally.   I think the higher “Not Repro” rate for MSDN Feedback code defects is due to having older builds compared with internal teams.   Getting your bug reports improves the product is two key ways.  First, obviously, it identifies specific issues that we missed through our normal testing, but need to fix.  Test teams employ a variety of test activities in the course of their testing, but even then it’s hard to find all the issues (or know which ones to fix) before it goes out in Beta.  The second way it improves the product is by identifying deficiencies in our test approach, which we can then address in our processes so that the next milestone doesn’t suffer the same class of problems.  We are currently in the process of doing a test hole analysis for these MSDN Feedback bugs to see if there are trends or patterns to the bugs we missed.  Getting beta and early adopter coverage is really important part of our strategy for testing the product and improving how we develop software.

Suggestions and Design Change Request (DCRs) are less of an apples-to-apples comparison with internally entered suggestions.  The first observation is that a lot more suggestions are entered through MSDN Feedback Center in percentage and total than MS employees enter internally.  Over half of MSDN Feedback reports are suggestions, but internal suggestions account for less than 5% of the total.  I think a lot of the “suggestions” internally are expressed through email and hallway conversations instead of bug reports.  I think the best forum for suggestions is really through the community anyway.  The MSDN Feedback site enables us to objectively rank what is important to you through voting and the importance rating.  This has been useful in weighing the different suggestions and as a result we’ve been able to act on your feedback – the two notable examples being C# EnC (399 votes) and Update Icon sets (813 votes), the #1 and #2 voted items.

Overally, I’ve been impressed with the quality of the bug reports and the volume of reports has been manageable.  In my next post I'll include the list of fixed C# and Debugger bugs.

- Rusty Miller
Visual C# Test Manager