Custer's Last Project

Recently,  I finished A Terrible Glory. It's a pretty good book chronicling Custer's destruction at Little Bighorn at the hands of the Sioux Indians led by Sitting Bull. When I started the book, I didn't expect to get a post out of it. After all, the military analogies are typically a little worn. But I should have known better. Here is one of the best-known disasters (at least from the POV of Custer) in military history, and somehow I didn't think there was any relationship to the politics or best practices associated with building and shipping quality software. Of course there are lessons to apply.

Just like any history, military or otherwise, the real situation is far more complex than any of the simple stories told afterwards. I had always imagined Custer out on an expansive plain with about a thousand troops getting annihilated by a huge number of hostile Indians. In reality he only had about 200 on hand and was fighting in a complex network of valleys and woods. You can see the actual terrain here. Also, while the contingent that Custer personally led was wiped out to a man, there were other parts of his command that were separate from him at the time that were not destroyed. The political wrangling over whether the failure was all Custer's or whether the surviving captains were to blame continued in the form of military inquiries and dime novels for years to come.

Naturally affixing blame and granting credit is a time-honored tradition in software projects as well. To get down to the level that approaches truth, you have to be skeptical of stories. Of course, everyone creates stories to explain past occurrences. "Project FOO failed because management couldn't say no to customer DCRs. It turned into endless feature creep."  "Project BAR was a success because we listened to customers and took their feedback seriously." Note that the same behavior was viewed as the key to success for one project and the cause of failure for the other.

Stories are like that sometimes. There is nothing wrong with trying to glean as much as possible from the story that becomes the official history, but those lessons are the easiest to get and the least likely help you along the next time around. You are going to learn a lot more from steering away from conventional wisdom and trying to piece together what was good about FOO and why BAR could have been a disaster. Also look for alternative explanations for what actually did happen.  Only by fully deconstructing a story will you get to something that helps you in your next project. Of course you still have to create stories, especially if you are a manager.

 Just don't believe them.

Even more challenging than evaluating a failed project after the fact is to try and figure out what a failed project looks while you are still in the middle of it and before its failed. It was interesting to read the accounts of the prelude to Custer's battle. It's full of soldiers making ominous pronouncements, Indian guides forecasting doom, and Custer laughing it all off. But you've got to take these sort of statements with a rather large grain of salt. It's likely that every battle has its share of folks predicting calamity. And much of it is probably apocryphal.  Unless you can somehow survey both successful and failed battles before they occur and correlate statements of immanent doom with actual outcomes, these pronouncements don't mean much. They do make good stories, though.

While it's important not to fall prey to the delusions generated by official history, it's even better to take a hard, cold look at a project underway and look for objective measurements that might be predictive. This is admittedly, a bit of a challenge. You don't get that many significant projects in the course of your career to generate statistically relevant data. Also, unless you are a researcher or a consultant you don't have the opportunity to evaluate a broad cross-section of projects. But it's still worth the try. Sometimes all it takes is the correct perspective to eliminate historical bias and The Halo Effect, but there is always some data that can help you on your quest. Make sure your metrics are as objective as possible. No, I don't have the answer. We know that bug counts are problematic. Lately I've been intrigued by code churn analysis. The nice thing about code churn is that there is no way to game the system. The only way to change the numbers is to stop checking in code. But paradoxically, this is also its greatest weakness. There is no management level action you can take from code churn analysis, except change the ship date. You can exhort people to fix bugs, you can get them to implement or cut a feature, but you can't do both of those and still control code churn. When management sees a metric they can't control, they typically become less enthusiastic, even if the metric is a useful one.