To the left, is the shirt I wore to work last Monday. My brother gave it to me for Christmas. Most of the words are cut off in that picture but it says, "I Roll Twenties". For those of you that have never been exposed to any role playing games (which I don't expect is many of you besides my fiance), I'll take a quick detour to explain why "I Roll Twenties" is amusing to those in the know.
In a role playing game, one person acts as the "game master" (or "dungeon master" if you from the D&D crowd) who is essentially the storyteller of the game. All of the other players in the game have individual characters (think dragons and elves, ninjas and super spies, Teenage Mutant Ninja Turtles, whatever) that act of their own free will. Of course, it wouldn't be a game if there wasn't some chance thrown in. That's where the 20-sided die comes in.
The game master lays out a plot, say "A dragon and all of its gold is before you... but the dragon is not sleeping." (Yes, yes, very cliche). The characters make a decision, say "We'll slay the dragon and take its gold." And the roll of a 20-sided die decides the outcome. Roll a high number on the die and the characters get what they want. Roll a low number and, well, let's just say some players may be creating some new characters soon.
That's the basics but if you've ever played a role playing game or watched people play the game, you know there is a lot of mystique around the "natural twenty". A natural twenty occurs when a player rolls a 20 on a 20-sided die. When that happens very good things happen for the character.
Continuing our story, the game master could say, "You swing your sword spectacularly and bonk the dragon on the head with the flat of your sword... knocking it, the dragon, out cold." Anyway, you get the idea. There is nothing better in a role playing game than to roll a 20.
Now, back to my shirt. If there is nothing better than rolling a 20 then rolling twenties all the time would fundamentally kick ass. Right? Right.
So, what does this have to do with anything? Well, let me go back two weeks to Monday, March 20th.
Two weeks ago, I was staring down two features that had to be in for Vista Beta 2. More importantly, branch's reverse integration (RI) was scheduled for that Thursday, March 23rd (read Larry Ostermans's description of reverse integration). Before we could reverse integrate, our test team has to sign off that all is well in our branch. We wouldn't want code in our branch to pollute the other branches in Windows. The test team needs about 48 hours to run their verification tests.
That meant I had to be done with both of my features by Tuesday night. Fortunately, I had all of the code on my side ready the week before and just needed Monday and Tuesday to integrate the changes from other teams and make it all work together. At this point, I expect you know what happens. I entered integration hell.
Looking back, if my life was a role playing game, believing that I could make our reverse integration date with all the integration challenges ahead of me was something like going into a dungeon where you know there are going to be tons of monsters that are going to wear you down until you hit the huge dragon at the end. Some would say, "Brave". Some would say, "Gutsy". Others would say, "Stupid".
Of course this wouldn't be a story if we had made our RI on Thursday. We didn't, and here are a couple more pieces of data why. First, I'm working on technology that operates during the installation of the operating system. That means when something goes wrong, it hoses everything. One failure in setup prevents you from verifying that anything else works. Second, in the worst case, debugging a setup failure takes ninety minutes to reproduce. Imagine how careful you would be in the debugger and in your fixes when success means you only have to wait an hour and a half to verify the fix but failure means you are guaranteed another three hours before possible success.
By noon on Friday, March 24th, I was certain that I was going to be working all weekend with my test team to make the new RI date they provided us, Sunday March 26th at 8 PM. I tried to prepare Jenny that I would be gone for many hours all weekend (I was already coming home late on Wednesday, Thursday, and Friday nights) and into next week until we made this RI. I also thought about what was going to have to happen for us to make that date.
I was going to have to roll a 20 for each bug fix I attempted. There was no room for failure. There was no more time. I had to roll twenties until I died.
The problem was that we had no idea how long the bug tail was. Each time we fixed one problem and retested, we found another problem behind it. You see I was making a structural change to a set of components that spanned several different teams and were tightly bound together. The tight coupling was the reason I was making the change (it was causing bugs, surprise, surprise) but no one knew just how tightly integrated the problem had become.
I won't give you a bug by bug report but suffice it to say that we also missed our Sunday RI date and things were looking very grim. At this point, I just had to find the bottom and figure out where we were at from there. It was Tuesday night that I rolled my first 19.
I thought I had the perfect fix for the last bug blocking all of scenarios. It was a registry key handle leak in someone else's code that I had never seen before but I was pretty confident that I could close the handle and fix the leak with a couple small changes. By 5 PM on Wednesday March 29th, I had gone home to spend time with Jenny who hadn't really seen me for 9 days. By 6 PM, the test team found that there was some strange regression, likely caused by my fix Tuesday night.
Fortunately, another developer on my team was able to verify the new bug was in fact a regression caused by my fix and my dev manager made the call to pull out my fix and an old (but small) bug fix we made to our code that exposed the problems we were seeing. They undid those fixes and kicked off the builds for the last time before our RI.
On Thursday, March 30th, I went home around 11:30 AM with the 24-hour flu, dead to the world. The rest of my team verified everything was working and around 6:00 PM our branch reverse integrated all of the fixes (and "unfixes") we had made over the last week and a half. With a little dispensation we made the RI for Vista Beta 2.
Somewhere along the way, I heard that Microsoft chose to slip out the ship date for Vista. My Mom actually called me Friday asking if the shake up in the Windows organization she was reading in the newspaper about had any affect on me. I told her it didn't right now because we were shipping Beta 2 based on the original dates.
There are plenty of things that didn't go quite right the weeks before I stepped into that dungeon of integration hell two weeks ago. We'll learn from those mistakes. However, what I found most heartening was that when I failed to roll that last 20 and essentially died the next day, the team carried on to reach the goal. I don't care what the media says; the team I'm on is focused on shipping Vista. We'll ship Vista on time (or early, if they keep moving out the dates! <wink/>).
In the meantime, I'm back for a while and will get some writing done here. Keep rolling those twenties... you know I am.