it's been nearly eight weeks since my last post, but i promise i've been doing something for you! and i can easily account for one of the eight with an explanation of what my whole team did just last week.
recently a couple teammates were trying to think about what issues they were concerned about resolving before we could ship a great first version of MSBuild. they weren't talking about specific bugs or work items, but more generally about the mileage between that day and the day we ship, and how we can make sure we cover that distance. reducing our bug count came up for several reasons:
1. lately the count has been pretty flat. in other words, the number of bugs we resolve (fixed, doesn't repro, by design, ...) has been roughly the same as the number of bugs people are opening.
2. devs have lots of other work to do aside from fixing bugs.
3. some important but lower priority bugs were starting to look ripe for kicking down to version 2 (which makes no one happy).
4. fixing bugs seemed like something program managers and testers could help the developers with. and if we (the testers) were fixing bugs, presumably we wouldn't be finding them....more on that later.
a plan was set: for one week, the MSBuild team - developers, program managers, and testers - would all sit in the same conference room in an unusual building (read: far away from the people most likely to distract us) while fixing MSBuild bugs on our laptops. we wouldn't answer mail, wouldn't have phones, wouldn't go to meetings, and wouldn't work on anything other than fixing bugs. period. there were obviously some initial concerns about this: who would take care of our dogfooders? which bugs would we fix? could testers and program managers even fix bugs? what would make this a success? we came up with the answers we could but otherwise dove straight in hoping we'd accomplish something. at the very least, this experiment would be a week of doing something different and exciting, and there was potential for significant results.
let's start with what we were facing: at 12:01 AM on monday the 26th, there were 437 bugs without an available fix open against MSBuild. to put this number in perspective a bit: the bug count sat just above 300 from early february to late april, before steadily rising to just under 450 from late april to mid june. for the next six weeks, we hovered there at about the same spot. over the past six months, our bug count had increased by about 50%.
so there was a lot of work to do. the first obstacle: the conference room we'd reserved as our temporary home wasn't on any building maps and the receptionist had no idea where it was. we feared the mysterious room was a black hole, but a recent admission by steven hawking allayed those fears. once we'd all discovered the room, we fixed bugs. and read bugs we thought maybe we could fix. and fixed bugs. and read the same bugs we thought we could fix before but still didn't know how to fix. and fixed bugs. and talked about fixing bugs. and fixed bugs. we also ate a few dozen krispy kreme donuts, a lot of bagels and cream cheese, sumedh's banana, and even some pizza with indian food on top of it.
by the end of the week, we'd all learned a lot and several of us had new nicknames (not all of which were desirable). the developers were especially patient while helping the others out with everything from process details to code reviews to discussions of potential fixes. the testers actually did open nine new bugs during the week. eight of those were opened by our resident star tester dan, who finds bugs while putting his shoes on. i personally learned a ton about our codebase and debugged more managed->unmanaged transitions than the mariners after piniella's been tossed.
here's the details though:
Monday 12:01 AM: 437 unfixed bugs
Friday 11:59 AM: 264 unfixed bugs
frankly, we'd hoped to reduce the count by about 100 for the week, or about 2/person/day. in fact, we were able to reduce the count by about 3.5/person/day, even including the 16 new bugs opened over the week. these numbers deserve a few qualifications: our goal was simply to reduce the bug count by as much as possible, so many of us chose the easiest bugs to fix rather than the most important bugs; we don't yet know how many new bugs we introduced, or how the regression rate compares to the usual scenario when only developers are working on the codebase; some of us actually did read our mail during the week (and deserve slaps on the wrists!). but no matter the qualifications, it was a great success and discussions are already underway about when we should do it again. it'd be nice if our work will encourage other teams to try similar experiments.
hopefully MSBuild will be a better product when we ship it because of this experiment; i personally think it will.
jeff.
ps - i'll write sooner next time!
[Edit: removed link]