It’s easy to get carried away with tracking bug counts, lines of code, person-hours per bug fixed, and on and on, but that’s not a big risk here at Microsoft. We tend to be so anti-bureaucracy, that we shun this kind of process analysis. That’s too bad, because some attention to metrics can help a great deal.
The team should pick some metrics to track and should widely publish them regularly, at least monthly, preferably much more frequently. This can be bug arrival rates, bug closure rates, whatever, but tracking the process is essential. If the team is “flying blind” they’ll never get the plane off the ground.
The key to metrics is to track something meaningful. To do this, you should:
n Decide the overall goal that you want to accomplish. This could be as straightforward as “cutting priority 1 bugs,” or as complex as “increasing customer satisfaction.”
n Determine a metric that will display whether you are meeting your objective. Define what it means if the number goes up, and if the number goes down, and what you might do about it.
n Track it, report it on a regular basis, and review your performance using it.
n Change how you are working if you aren’t meeting your objective. If you can’t envision this happening, you’re tracking the wrong metric.
The pitfall here is to track metrics for their own sake, or to track things that don’t optimize your team’s behavior or measure your progress towards shipping. Tracking the wrong thing can be worse than tracking nothing, because you may lure yourself into a false sense of accomplishment.
There is a “center of gravity” for projects that can be observed fairly easily, and the PM team should be following that center as it moves over time. This center begins in program management as the team designs the project, and specs it. It then moves to development as they code the project, and then to test as they help to polish it for shipment.
The PMs on the team should be making the transitions with the center. As the focus moves to the coding team, PM should be working to close down issues, triaging bugs, and responding to difficulties the development team has in implementing the spec. As the focus changes to test, PM should focus on triaging bugs and preparing the product for shipment.
If you find the PMs worrying about new features after code complete, or planning the next version instead of closing down bugs, they are not helping to ship the product. Either they need to be part of a separate team focused on the next release, or they should be more closely working on the current release.