For those of you who might be interested in knowing more about what I work on, here's a link to my manager talking about and demonstrating Essential Business Server.

http://edge.technet.com/Media/Essential-Business-Server-EBS-demo-with-Bjorn/ 

I'm not much of a UI guy. I've spent my entire time at Microsoft working on servers of one sort or another. I learned a ton about UI design back when I was on the Terminal Services team helping out the http://www.mesh.com guys on their live remote desktop feature, but I was definitely learning more than contributing. My current gig on the Essential Business Server team has also made me think more about UI and user experience in general than I have on other server teams. But despite this lack of background,  today I got bit by a UI problematic enough to gestate into a blog post.

Today the ants started to march. Normally that's not so bad, but for some reason they decided that our kitchen was a particularly good place to march. I didn't appreciate that so much. So off I went to Home Depot to get some ant poison. As long as I was going, I took an empty propane tank to exchange. That's where the fun begins.

Our local Home Depot uses an automated tank exchange. I like self-serve options. But this one was a bit problematic. Basically there is a couple of rows of cages each of which houses a full (or possibly empty) propane tank. You go to a console where you swipe your card and select whether you ware making an exchange or getting a new tank. I had an exchange.

Here's what is supposed to happen in the 'happy day' scenario. If you have an exchange, the system finds an empty cage for you and opens it. You put your empty tank into the cage and close the door. Then the system opens a second door that contains a full tank. You take the tank, close the door, and go about your business.

But that's only if it works.

In my case, after I swiped my card, it told me to 'put your tank in cage #36.' Unfortunately, cage #36 was securely locked. There was a failure somewhere in the system. But the real problem was that there was nothing else I could do. I couldn't tell the computer that it was locked or to try another cage. I just stood there. I looked around for an employee, but it was one of the first nice weekends of the year and the home and garden area was hopping. Besides, I was in the middle of transaction and had just swiped my card, so I didn't want to go too far away. Eventually it timed out and said "do you want to open the cage again?" Evidently it was concerned about how long I took and offered some corrective action. Good. The time-out was very long, about five minutes, but I was willing to give it another shot. Who knows? Maybe it hadn't really tried the last time.

I pushed the 'yes' button. Nothing happened. I waited another five minutes. During my wait I thought about where the problem might be. I noticed that the doors were spring loaded. Presumably when they were unlocked they would just pop open by themselves. Perhaps cage #36 wasn't loaded enough and if I just pulled a bit during the time it was supposed to be opening, it might just work. Besides I'm not sure what would happen if I said 'no.'  Would it void the transaction or keep my money because I refused to give it a tank for exchange? In any case, I tried again, but pulling didn't work. I waited another five minutes. By this point I had lost fifteen minutes and was thoroughly frustrated. I pressed the 'no' button. Please don't try to open the cage again. The system properly voided my transaction and bid me adieu.

But somehow I couldn't leave it alone.  I have a lot of trouble just walking away from a bug. I thought perhaps after the transaction failed it would realize that cage #36 was causing problems and try a different cage. Nope. Another five minutes burned.

I'll have to admit, this was the second time I used this system. The first time I was able to deposit the empty tank, and the door opened for the new tank. But my six-year-old daughter saw the door open and closed it. I wasn't watching but the screen asked if it should open the cage again. I said no, and it dutifully left the tank locked and charged me money. Since I was out an empty tank as well as the cost of a refill, I had to track down an employee. I got a propane tank, but only because I found a good employee.  The only 'proof' I had of my problem was a completed receipt that said I had successfully completed my purchase.

I think main lesson to take from this example is that true end to end scenario development is much more than the easy 'happy day' flow. It also needs to encompass failures, confused users, and meddling Kindergarteners. If dependent systems always worked and users always understood what to do, our jobs would be much easier. But the real world just doesn't work that way.

Anyway, I left with ant poison and my original empty tank.

One author I've been reading lately is Atul Gawande. I recently finished Complications and before too long I expect to start Better. Atul Gawande is a staff writer for the New Yorker. Just like other staff writers like Malcolm Gladwell you find that his best articles are insightful twists that you can apply generally. And also just like Gladwell, each idea is obviously just a small viewpoint into the whole truth. What makes it compelling is that it's not comprehensive, and doesn't give you the caveats that always apply. That’s how both Blink and The Wisdom of Crowds can both be right. Anyway, that's the way folks write nowadays. In that spirit, I'll jump into some of the ideas Gawande advocates.

Medicine is a particularly apt metaphor for software development. Both are practiced by highly paid professionals with years of training and experience. Both have a built in tension between heroic individualism and rigorous procedures. In software development we all know about the charismatic rock star coder who gets things done just by exerting the force of his or her will and intellect. Yet there is a side of development that wants to emphasize the true engineering aspects of the job. But we just don't have quite the same mindset as other engineering professions. When you are a civil engineer, it's great when you can make a beautiful bridge but it better not fall down when cars drive across it. No matter how brilliant the engineer is, we don't trust their judgment without significant testing and oversight. Despite the life and death consequences, doctors are subject to a different mythos. Doctors often have a hero complex. They feel that it's their judgment in a time of crisis that makes the difference between life and death. All the rules and procedures just get in the way. Certainly this is how the media typically portrays doctors. Sure when you push them, doctors will acknowledge that systematic processes are important. But that's not how they think of themselves and that's not how they invest their energies. The same hero complex pervades much of software development. We seem to acknowledge this similarity with the medical profession, at least at some level. You can see it in our language from how we 'triage' bugs to how after we ship we hold 'post-mortems' to identify ways to improve. 

It's with this model in mind that I read a recent article from Gawande about checklists. It turns out that I.C.U.s are dangerous places. But it's often not the primary trauma that kills the patient. Rather it's the infections and complications that can arise from the treatment. There are hundreds of little things that have to go right to prevent one of these bad things from happening. There are countless opportunities for even the most diligent to make mistakes. A particularly pernicious area of risk is infections from intravenous lines. One doctor, Peter Pronovost, decided to address this problem systematically. He made a checklist of things doctors were supposed to do when they put in a line. Here is the list:

1. Wash your hands with soap
2. Clean the patient’s skin with chlorhexidine antiseptic
3. Put sterile drapes over the entire patient
4. Wear a sterile mask, hat, gown, and gloves
5. Put a sterile dressing over the catheter site once the line is in.

These are all obvious steps that everybody already knew. I bet I could have come up with some of them myself. But what he did with the list was the real innovation. He gave the list to nurses and empowered them to stop the doctor if they missed any step in the process. Here's what happened:

• The ten-day line-infection rate went from eleven per cent to zero.
• Only two infections occurred over fifteen months.
• This process prevented an estimated forty-three infections, eight deaths, and saved two million dollars.

The results were stunning, but perhaps predictably, he has had little uptake in the medical community. Doctors have had too much training to be humiliated by a piece of paper. They said they needed to focus on the patient, not on some process. Things happen too fast to be hindered by extra paperwork. But yet the same hospitals are willing to invest tens of millions of dollars on new line kits with a special coating that improve the infection rate only slightly.

When I look at this anecdote, there are two things that really jump out at me regarding the world of software. One is the idea of the checklist itself. We have many of the same problems where we think that individual developers or leadership make the best decisions on whether something is ready for checkin or if this or that DCR should be accepted. The procedure for doing this right is so well known that nobody finds it interesting any more, but yet we rely on experience rather than procedures. We know about code reviews and the best way to handle them. We know about buddy builds, BVTs and due diligence but when push comes to shove we exercise our judgment and cut corners. There are lots and lots of best practices that are easy to forget or ignore. Despite how mundane these issues are, nobody wants to give up control to a checklist. But checklists and simple procedures can be incredibly liberating. Anything that requires the 'stupid' part of your brain can be written down and forgotten. Executing on the checklist becomes a matter of simple hygiene, like washing hands and brushing teeth. Following a check list keeps you diligent and frees you to think about higher level problems. Software also has the same love of technology to solve our problems. We invest millions in automated code analysis tools, but our code reviews turn into rubber stamps.

But perhaps the real power lies with empowering the nurses. They are responsible for the overall quality of care, but are still subservient to doctors in many ways. We have much the same relationship between test and development. As a company, Microsoft has put enormous effort into ensuring test has a good career path. But yet many of the best testers view test as a stepping stone to development. Very few developers have the same aspirations to move to test. If we could empower test to become the true gatekeepers of quality, we could solve a number of perennial problems.

• Test gets more oversight and a stronger sense of ownership of quality. This inevitably leads to greater career growth opportunity and better engagement from the test team.
• We get an objective assessment of our adherence to best practices.
• Test has no interest in getting the bug count down or getting a dev out of bug jail, so there is no conflict of interest that leads to cutting corners.
• Development is forced to ensure test is informed. Surprising test at checkin time will only get the checkin delayed.

Of course there will still be plenty of places to use judgment.  But taking the role of judgment away from places where it really doesn't belong frees you to exercise judgment where it's needed. Imagine how debilitated you would be if getting dressed, brushing your teeth, or eating breakfast were all judgment calls. Development will slow down, but that's a good thing. It will be hard to checkin if you need to find a tester to approve your checkin. This is especially true for distributed teams where test is on a different continent. But the reality is that those situations are enormously difficult already, this process simply highlights the underlying reality.

Back when I was in graduate school for physics, there was a professor who was very interested in physics education. He had a set of questions that he would give to Freshman physics classes at the beginning and at the end of the semester. These weren't typical analytical physics questions, but rather subtle variations on them. For example, rather than asking a person to calculate forces a question might ask if the force of a car pushing a truck up hill was greater than, less than, or equal to the force of the truck pushing back on the car. Even good physics students would get these questions wrong. His conclusion was that our physics education techniques weren't doing the students justice. They still didn't 'get' the underlying laws they could recite so well.

But there was something that always bothered me about this analysis. Perhaps it was just that I got one of these questions wrong myself, but it just didn't jibe for me. Then during a colloquium when he was presenting on this topic, it finally came to me. His questions always tested the students' intuition about physical phenomena. Hence his unstated goal for physics education was to improve the students sense of how the world worked. That may actually be a laudable goal, but what I realized was that it was much more valuable to teach people to analyze a problem and overcome their intuition.  In many ways you could argue that the triumph of the scientific theory was the application of systematic analysis to questions that were once relegated to experience or gut feeling.

Another example that comes to mind was an experiment run by a classmate of mine in college. She was a math major but was doing some cross disciplinary research with the psychology department regarding math anxiety. She was going to set up an environment designed to instill anxiety about math and then give the subjects a test. What was striking was the conclusions she expected to draw. She told me that if the students did well they didn't experience anxiety, and if they did poorly they were anxious.  Anyway, I asked the probably inappropriate question "So why run the experiment?" After all, she had her conclusions and the experiment was constructed so that they couldn't be disproven.

This ability to critically analyze your intuition and determine if it's right or wrong is key to a scientific mind. It's easy to forget about  the power of Karl Popper's philosophy of falsification. In order for a theory to be valid scientifically, there must be a test that can prove it wrong.  Further, experiments can never prove a theory fully correct, they can only disprove it. Needless to say, this rigor of thinking is not applied to the worlds of business and software. Instead people learn through a series of successes and failures and try to apply the lessons from the last project to the next. Sometimes these practices get codified and enforced, or at least documented.

I was reminded of these incidents from my college days recently as I was reading The Halo Effect: … and Eight Other Business Delusions that Deceive Managers. One of the themes the author repeatedly comes back to is that business research and most management books do a very poor job of analysis. They do things like ask pre-selected successful companies what makes them successful and then categorize the results and present them with the authority of science. The problem is there is no way for them to be wrong. If they can't be objectively wrong, then they haven't stated anything. In order to really know if 'customer orientation' is predictive of success you need to objectively find a measure of customer orientation, randomly select companies, measure them, and track performance over time. Even that may not be enough, because it could be that customer orientation is correlated with other practices that are the real cause of performance. Perhaps it's even worse than that and all the identified best practices correlate with one another.

This same scourge of non-critical thinking applies to the execution level of software development as well. All too often teams will come up with bug glide paths using estimates like "We'll have an incoming rate of X and we can fix bugs and a rate of Y per dev per day. We'll probably find Z bugs after the bug bash when we check in feature W. That all means we'll reach zero bug bounce by date P." You come up with a rather complicated bug glide path that bears no resemblance to what actually happens. What's stunning is the explanations for the deviations that float around. If you spike bugs early, you can claim "We are ahead of schedule and finding bugs early." If you are under the expected bug count, you aren't wrong. Instead you are in good shape because the bugs are under control. If you slip, something unexpected and unforeseeable happened. The next milestone you do the exactly the same thing and shrug that every project you've ever been on has had similar course corrections. Peoples' ability to deceive themselves makes it surprisingly easy to overlook the irony. The glide paths can help tell if you are in trouble or not, but the way they are normally constructed they cannot systematically improve your ability to execute the next time around.

This doesn't mean that you can't make decisions without double blind tests that track multiple software projects over multiple releases. But when you make a schedule, use a metric to track quality, or plan a project, try to figure out how you will know if you are wrong. Make the effort to dive into the data when you estimate incoming rate or fix rate. Your numbers might not be accepted by those higher up, but at least they will have to put more thought into why they think the numbers will be different. Look for measures of quality that aren't so self-referential and subject to social pressures like bugs are and see what where they take you. But most importantly follow up on your predictions. If you have an objective measure shows great quality for a product in development, figure out ahead of time how good quality looks in the market and see if you measure was useful.

And above all, always look for the alternative explanation.  If conventional wisdom says one thing, try to see how it could be wrong and examine alternatives. Identify future measurements or events that would validate one story versus and check up on them when the time comes. Of course there is still some gut feelings to apply, but if you can ferret out a little bit a truth for each release, you'll actually learn something.

Right now I'm reading The Black Swan. I'm almost finished.  Here's my book report .

The Black Swan is one of the rare books that seems to express and clarify my own thoughts. It gave me a description and a framework for an outlook that I like to think I already had.  The author, Nassim Taleb calls this outlook "skeptical empricism."  I heard about this book some time ago but at the time I was only mildly interested. The high level descriptions that tend to accompany it just don't give it justice. The blurb usually says something like "Teleb posits that large unlikely events dominate most phenomenon." Then they add in a couple of examples from stock market crashes and 9-11. While this is not an inaccurate description, the real point is more subtle. The key is how we assess the likelihood of unlikely events, both positive and negative.

Much of how we think about risk is predicated on the idea of 'normal' distributions. This is true both when we are trying to be rigorous and use a sophisticated mathematical tools, as well as when we casually make estimates, try to predict something about the future, or simply make a plan. The normal distribution is implied anytime you discuss such things as averages or standard deviation. Taleb claims, and makes a convincing argument, that normal distributions hardly ever exist in the real world.  Instead power laws dominate. One example he uses is wealth distribution. If you have any two people the sum of whose net worth is 1 Billion dollars, you are far more likely to have one person with a net worth of nearly one billion and another person of more modest means than you are of having two people with a net worth of 500 million. Taleb refers to the unlikely events that dominate the overall distribution 'black swans.'

Other manifestations of this behavior can be seen in large-scale projects, whether it is a construction project or a software project. Vast cost and schedule over-runs are commonplace. The reason is that the problems you face in a complex system don't follow normal distributions. Enormous schedule-slipping issues don't get exponentially less common the more severe they are. Issues that can delay a project by years may be uncommon, but they aren't unheard of. The difference may not seem significant, but the implications are enormous. For example, if your project faces a 20% risk of slipping ten days, with a normal distribution it's essentially impossible that you would slip a year. With a more realistic power-law distribution, it could be 10%.  That's unlikely, but you can be sure to run into a couple of problems of that order a couple of times in your career. Anyone who has shipped a reasonable amount of software can spin a couple of stories about these 'black bugs' that created these serious distortions.

This understanding of risk really helped me understand what bothers me about typical schedule creation and end game mechanics. There is often an element that just doesn't seem to appreciate the enormous difficulty in shipping software. You often hear people estimate the incoming bug rate and the average fix rate and then project these rates linearly until you reach some stabilization period. What's really dangerous is that after people make these estimates and realize they have some number of "dev days" and start putting in more features based on how many days they expect to have left. Thirty years after the Mythical Man Month and the calculation continues to persist. They make the classic mistake and assume that all unknowns are 'known unknowns' and that their difficulty is normally distributed and possibly even declining as you get closer to ship.  They may never say 'normally distributed' or 'Gaussian' but just by projecting a fix rate, they are implicitly making that claim whether they know it or not.

Just for fun, I did a query against a bug data-base and looked at the distribution in the time it took to resolve a bug. For those developers among you, I queried for code change bugs resolved as fixed and calculated the difference between open date and resolved date. While it's tough to prove that a distribution actually follows a power-law, the resulting plot sure looks like it. There are lots of bugs that get resolved in just a day or so, but the long tail just goes on and on.

The critique to all of this is obvious, and Taleb addresses it in his book. The problem is that power laws are not very good for prediction. Just deriving the exponent is enormously problematic because the extrapolation is very sensitive to the initial data. But even if you have the real distribution, what should you do with this information? You can't just add a year to a project because there is a ten percent chance that it might slip that long. You can't even multiply the probability by the length and slip by a month. That time is either too short or too long depending on whether the bug exists. But yet you have to plan.

The first step is to acknowledge that there is a problem. Black bugs do exist and you will get bit by them at some point. You've seen it happen to other teams, and it's folly to think you are just better than they are. This realization might be enough for some people, the fear of destabilizing changes might seep into your system deeply enough that you can make the right decisions. But Taleb has some more concrete strategies that he applies to the stock market. His strategy is that for areas that are subject to negative black swan events, be enormously conservative. Don't trust any risk assessments because the assessments are all derived through models that use the normal distribution. At the same time, try to take advantage of positive black swans with small investments you can afford to lose. In the stock market this means that the majority of your wealth (say 80%) should be invested in the most secure investment you can find, treasury bonds for example. The rest should be in invested in small amounts in highly risky, enormously leveraged instruments. It only takes one of these to pay off for you to be able to retire in comfort.

For software, this means that once the feature set is locked and you are marching towards ship, you should be as conservative and hard core as possible. You need to be even more conservative than you think you need to. Go ahead and make schedule estimates. Feel free to plot glide paths. Just don't believe them. And don't look for extra work to fill up the gaps. Any change, any code churn generates and obscures black bugs. It obscures them because, when you check in a new feature you'll fix a lot obvious issues for a while before the mist clears and you get to the deeper, harder-to-find problems.  It's like looking into a pool of water. If the water is still churning, you might not see the barracuda just below the surface. Don't assume that just because a bug is hard to find that it isn't serious.

Raise the bug bar slowly. If you raise the bug bar to exclude trivial issues early, then you've stilled the waters and increased the amount of time you have to look for the one bug that could cause you to slip. Conversely, if you keep putting in features and fixing unimportant bugs right until the final lockdown, you miss much of the focused test pass necessary to ferret out possible black bugs. Don't assume that if the incoming rate starts to decline that you are starting to scrape the bottom of the bucket. You may have gotten past the surface issues. No matter how long you test and how many customers run your product, you can never be sure that you won't still run into a black bug.

If there is some design change that you are just itching to make and you are convinced that you just can't ship without this feature, try to think of the worst thing that could happen. If you decide to fix this in code, you have traded a known bug for a unknown number bugs. Perhaps you have traded a bug you can alleviate with documentation for a black bug that tanks the product in the press. Since this came in with a DCR, it's late and it's much less likely that you'll find it in time.  Most importantly, don't schedule to the hilt. Even if you are sure you can squeeze more into the schedule, and some feature would really make customers happy, just say no. Unless you know you will lose sales it can wait until the next version. You may think the risk is low, but never forget you just can't calculate risk. All you know is that risk a lot higher than you think it is.

If you actually do find some slack in the schedule, don't add features, start experimenting with ideas for the next version. Look for opportunities for positive black swans. These are higher risk, speculative features that could have reap enormous benefits if you they pan out. These features are your chance for real innovation. While I am a also a big fan of careful value proposition work, the real game changing features always come from left field from some individual contributor with an idea. As you stop checking in changes and start looking for black swans, don't worry about your team not having enough to do. Setup a system to let the experiment and create something truly unique. You'll have time to plan a product that aligns to the markets needs and moves forward key strategic initiatives, but you'll also create something unexpected. Most of this experimental work probably won't make it into the product. But the level of commitment and passion going into it will far exceed anything a top down strategy can achieve.

This conservative approach to shipping may sound boring and like a  recipe for non-innovative products, but it's not. Remember that what you have is probably a whole lot greater than you think it is. You've probably lived with your product in one way or another for years now. When you start your application, all you can see is what it could have been. You see cut features and opportunities lost.  This is natural, but that's not what your customers will see. They will see the features that did get in. And if you executed well and didn't sully the project with last-minute changes that didn't have time to integrate cleanly, the product will be a joy to use and customers will love it. Just keep reminding yourself of that.

One book I've really gotten into recently is "The Nine."  It's a very readable history of the United State Supreme court over last thirty years or so. It purports to show the inner workings of the court and the political factors and maneuvering involved. It's very entertaining and gives you a sense of how the court operates and how it relates to public opinion and the other branches of government. Listening to descriptions of all of these court cases made me think about what 'rule of law' really means and, of course, how it applies to any decision making process including software development.

If your view of the court is based on cable news, it's easy to get a misguided view of its role in government. To a casual observer, it looks like they are making all the decisions, or at least all the exciting ones. There is a distinctly hierarchical feeling to the whole process. Intense court battles keep getting appealed to higher courts and if the case is important enough, the supreme court gets called in because they are the smartest ones around and they are the ones who know how to interpret the constitution.

But while there is an element of truth to the simplistic view, that's not where the power of the system lies. The key observation is that the supreme court does not make all of the decisions, nor does it necessarily make the tough decisions, or even the big ones. Instead they make highly theatrical sample decisions. A lot of people miss the point when they focus on the authority the supreme court has to overrule the lower courts. This line of thinking tends to overemphasize the supreme court justices need to be the ultimate experts in constitutional law. While it's certainly true that supreme court justices are well served by being smart and having a deep background in constitutional law, lacking these qualities does not preclude them from being highly effective.

What the court really does is enable distributed decision making. This perspective reaffirms one of my favorite truisms about management. Leaders aren't the ones who make all the right decisions, they are the ones who create systems in which the right decisions get made. An effective supreme court decision frees the lower courts to make decisions easily without fear of reversal. It provides guidance on how to interpret the law. This makes it possible to have many courts around the country operate freely and consistently.

But the real power isn't the ease at which cases can be solved. The real power is the cases that are never filed in the first place. Just imagine the chaos if property law was so murky that any mortgage could result in a court hearing to clarify the terms. If this were to happen, banks would never lend large sums of money with nothing but the house itself as collateral. Likewise, people wouldn't commit such a large fraction of their wealth and future income if they knew that the rules could change at any time. Risk of arbitrary intervention makes it impossible to enter into these kinds of deals. Incidentally, that's why it's so dangerous to meddle with foreclosures during the current housing crisis. If banks learn that the government can and will override legal contracts, they will need to raise their interest rates to compensate for this risk.

This is really what the 'rule of law' is all about. When you don't have this level of confidence in the law, then individual people in positions of power start making more executive decisions. Because people are fallible even in the best of cases, focusing the decision making on a single person or even a group of people is a sure recipe for tyranny. At best, the leader is forced to make orders of magnitude more decisions. People feel free to angle for a more favorable decision if the normal procedures rule against them. Or they don't feel that the decision will stick unless the leader personally makes a call.  At worst, management starts to make arbitrary or otherwise bad decisions and the cycle builds on itself. You can always spot a leader who hasn't put together an effective system. They are endlessly making decisions about the most arcane and minute issues.

In many ways shiprooms operate as courts. For the longest time I was amazed at how the central Windows shiproom worked. Here are a bunch of release managers making calls on bugs that they can't possibly all understand and yet somehow it all comes together (usually.) In reality, the central shiproom operates on reputation and trust. They only make decisions on the thinnest veneer of issues. But one way to understand how this works is that shiproom operates very much like a court system. There are even tiers as you go from feature team triage, to product group shiproom, to server shiproom, and finally to central shiproom. The bug bars as devised by the leadership team function as the constitution. Each time a bug gets raised at any level of shiproom, the attendees get an object lesson in how to apply the bug bar to individual cases. This application should reduce the need to make the same kind of call in the future. In order for this to work, the bug constitution needs to be clear and precise enough that most bugs can get be handled without resort to a appellate review. But of course, bug bars can never be inclusive enough to cover all possible bugs. But above all, shiproom has to be consistent in the application of the bug bars. That's where precedent comes in.

Coming back to the actual supreme court, this is the crux of the argument between originalism and precedent. It's obvious when you look at how the court system is intended to enable uniform, distributed decisions that it's critical that the guidance provided by the supreme court is consistent. All justices agree with this point. The only real difference is how to obtain that consistency. Some justices focus on precedent and give greater weight to recent decisions as opposed the actual text of the constitution. This approach is usually construed as liberal. The conservative originalists, on the other hand, try to divine the original intent of the framers. If you are able to ignore the implications in hot-button issues like Roe v Wade, then it’s pretty obvious that each approach has some advantages and disadvantages. The originalists run the risk of less stability in the short run because they have fewer qualms about overruling precedent, but they have a greater chance of keeping the direction of the court consistent over the long term. They always try to tack back to the original heading as defined by the framers. If you give greater weight to precedent, then you are unlikely to make sudden changes in course but over long periods of time the accumulated rulings could have little resemblance to the constitution. To the originalist, the only proper mechanism for changing course is through the amendment system.

Of course, this isn't quite how shiprooms work. Shiprooms do operate as an appeals court, but only for approved bugs. In a sense, once the bug bar raises high enough, all bugs approved in the lower shiprooms are automatically 'appealed' to the higher level for final approval. It doesn't operate the other way around. Bugs that are rejected by a lower triage or shiproom usually die then and there. But it might help overall product quality if someone who disagrees with rejecting a bug had the opportunity to appeal the decision to next level and argue their case. If the shiproom is very clear about how they apply the bug bar  and previous precedent to reach their decision, then everyone gets a better sense of how to apply the bars. Fewer issues go through the appeals process and the management gets a more precise, and more accurate measure of the quality of the product.  Ideally the shiproom would even write down the thinking and lay out the argument in terms both of the bug bars and previous decisions. This way shipping automatically builds institutional knowledge instead of just individual expertise.

Even for the limited way in which software development adheres to the rule of law, it's actually quite amazing what happens when an executive decision maker deviates from the process and pushes through a bug. No matter how right the executive is, bypassing shiproom and the court system weakens the team's belief in the system and the rule of law. It sends a signal that there are some people for whom the rules don't apply. People feel free to subvert it themselves and generally become cynical about their own contribution. This undermines confidence, encourages gaming the system, and does more damage to the culture and ultimately the product than could possibly be compensated for by the issue pushed through by the exec.

Regular visitors will get a pretty good sense of the books I am reading. I find that books on subjects that aren't directly related to software development or any other topic I'm 'really' writing about can offer deeper insights than a dry analysis of the actual topic. It forces you to look for patterns rather than just grabbing direct quotes. It also provides some distance. It's much easier to think discuss the leadership abilities of Lincoln than it is to analyze someone in your direct management chain. So please bear with me if my posts seem a bit like a book report. I promise there is a point in there somewhere.

Anyway, on the way back from a customer in Chicago, I was looking for a new book. The selection wasn't that great at the airport Hudson store. And of course I can't download an Audible title there, so I had to pick something that looked like I had a good chance of finishing it. After much browsing, I eventually settled on "The Smartest Guys in the Room: The Amazing Rise and Scandalous Fall of Enron." I'm quite enjoying it. Despite the rather arcane details of a company that no longer exists, It's a surprisingly readable book.

One thing I found very revealing is how human all of the main players are. Sure they stole millions and defrauded an industry, but for the most part they thought they were doing the right thing. They thought they were special and they were entranced with the idea of transforming the energy industry and becoming enormously wealthy in the process. The point of the Enron story is not that there were some evil corporate masters who stole money from good working people. The real case study is how smart and ambitious people can transition so easily from innovation to criminality without ever knowing exactly when they crossed the line, or indeed for some even knowing that they crossed the line at all.

The Enron story is about delusion. It wasn't just a single person or even a group of people maliciously fabricating information to defraud customers and investors. Instead it was entire system of delusion. Executives deluded themselves into believing they were part of a new economy where the normal rules no longer applied. Investors and analysts were full parties to this delusion and didn't look at the data with an appropriately skeptical eye. The consultants and accountants were all part of this vicious cycle. It's easy from a distance to talk about the rapacious and illegal practices of all involved, but who among us is strong enough to speak the truth when there are millions of dollars available for playing along? Even the lowly frontline worker who lost their retirement savings were party to the delusion by putting everything in their 401K plans in Enron stock.

How does this happen? With Enron it happened when the primary goals were shifted from the fundamental economics of value offered and revenues obtained to abstract secondary metrics like earning statements and the stock price. They moved from looking at data that can't be forced to lie and instead generated metrics that gave ample opportunity to make things up. Almost everything they did was tainted by this approach. For example, they bought companies and held them like a private equity firm. This in itself isn't wrong, but when you do this you have to report the value of the companies in your earnings statements. Privately held companies are notoriously hard to value and it really comes down to a judgment call when you put a final number on them. Opportunities to make judgment calls are opportunities to go astray.

Similarly, when deals go bad, you are supposed to track write off the loss in your earnings statement. But at Enron it was common to keep them off the balance sheet in the hope that the deals might come back in the future. Again it was a judgment call. It's not a hard line when a deal is no longer recoverable, perhaps the negotiations are just stalled. Another practice that helped enable Enron's downfall was the use of 'mark to market accounting.' This is an accounting practice where you count the eventual lifetime inflows from a deal at the time you make the deal before the cash has ever started to flow. In conventional accounting you only count the money as it actually comes in and out the door. Mark to market account also gives the opportunity to adjust the value of the deal over time to reflect changing economic circumstances. Again much of the valuations that show up on earnings reports came down to judgment calls.

These and other practices had two fundamental traits that eventually erupted into the biggest bankruptcy in history. First, there was enormous latitude and judgment required to give assets and earnings a proper valuation. In a highly money-charged environment like Enron the ability to use judgment essentially amounts to giving yourself a license to lie. This natural tendency was magnified enormously because Enron valued earnings reports and stock valuations, not economic fundamentals. In an environment like this, it's almost inevitable that things will go bad.

It's an insidious effect. Let's imagine that you are an Enron executive charged with making a certain target. One of your underlings reports the value of an asset that doesn't help you make your goal. You know full well that a different set of assumptions will enable you to hit your target. Not only that, but there is nothing wrong with your new assumptions. They just happen to be a shade more optimistic. Add to this to small fact that millions of dollars of your own bonus are on the line. What do you do? Let's just say that it takes a remarkable fortitude to go with the pessimistic estimate. The thing that's most insidious about this latitude in judgment is that the first person you lie to is yourself. You fully believe you are providing greater insight and greater knowledge to your underlings analysis. The fact that this makes you enormously wealthy in the promise can perversely give you 'proof' in the value you add in dispelling fear, uncertainty, and doubt. You go home that night reveling in your superior judgment and leadership abilities.

The other aspect that ties these practices together is the tendency to 'double down.' Every time you close on a mark to market deal you have to close a bigger deal in the next quarter to keep the profits up. Because you also have the freedom to use your judgment in the deal valuations, it doesn't always matter if these deals are good or not. You just have to keep making them at an ever increasing rate. The same thing happens when you defer writing off bad deals. Not only do you still have this deal hanging around next quarter but you probably have another one as well. The dirty laundry keeps piling up until economic reality imposes its will. The thing to remember in cases like this is that reality will impose its will eventually.

How do you avoid the perils of Enron? For one thing, no matter how nice it sounds judgment is a dangerous thing. Let's imagine that the leadership team says "Use your judgment to decide how many bugs to punt. And by the way we think that about 30% of them can go away because we really want to do this other feature and we have to ship by such and such a date." What are you going to do? It's quite likely that by playing along, punting bugs, and implementing a feature you can get a promotion. Even if the software is ultimately a failure, there is a good chance you will be on another team by the time that happens anyway. You can leverage that new promotion to get a lead gig somewhere else. Even if you stick around long enough for the project to truly tank, we all know of careers where failed project builds on failed project but the individual keeps climbing the ladder. Who among us would really choose the greater good? Sure, the use of good judgment is critical, but you should NEVER construct a situation where there are implicit pressures to make a particular call. Since there are always schedule pressures, restricting the use of judgment calls to truly critical and rare events is very important. If everything becomes a judgment call, then what kind of call are people going to make late in the cycle when they are over-worked and feeling beat? The best way to solve this problem is to have a culture that values the underlying reality rather than abstract metrics. This can be hard to implement, but an easy way to start is to have strict procedures and rules to guide judgment. Even in the most accomplished teams, the majority of decisions should be driven by objective, real criteria. Make the rules precise enough so that it is enormously obvious when management decides to exercise judgment and overrule the procedures.

The other key practice is to never defer pain. Don't use the next milestone as a dumping grounds for bugs you can't fix in this one. You can do this for a while without paying a cost, but eventually you will ship and real customers will use your products. Every time you use the schedule constraints to defer issues to the next milestone you are doubling down every bit as much as Enron did with its bad investments. Reality will come. Perhaps you as an individual can get a new job before things really hit the fan, but that certainly isn't the way I would want to run a project.

Most importantly you have to foster the right culture. All of the things that I discuss above predate Andrew Fastow's overt earnings manipulations. But because Enron fostered a culture of pushing the rules without reference to underlying reality, the transition to blatant fraud was subtle and unnoticed. Ultimately rules and procedures are just beacons to warn you of the real dangerous areas. They operate much like lighthouses. The real concern of any ship's captain is to avoid the rocks, not the lighthouse. Indeed the false security provided by a complex set of rules can lead people to think that as long as they can hew to the letter of the law, they are protected from danger. It's easy to think that breaking the rules are the actual danger. But if you start to feel that the rules are the reality, you are headed for the rocks.

I've been reading a bit about economics lately. Mostly I just want to have a coherent opinion on subjects like trade, budgets, and growth and other such stuff. The thing that has surprised me is how some economic principles have applications in the world of software development and bug management. Here are a couple of thoughts on how you can apply lessons from economics to shipping quality software.

People respond to incentives

This seems like such a simple statement of supply and demand, but yet the implications are quite profound. It's very powerful to predict the results of a policy change based on the aggregate self interest of the individuals affected. For example, if you institute price controls you will get shortages. That's simply because producers have no incentive to meet demand and consumers have no incentive to conserve. This effect was in fine display during last winter's windstorm. I waited in line for three hours to get a generator and spent half of my remaining time driving around looking for gas cans or a gas station that actually had gas. If prices are allowed to float then people who really need the gas and are willing to pay for it can still get it, while those who can do without cut back. Of course anyone who actually did this would get slammed for price gouging.

If you remove the mediation of prices, other factors jump in. These include black markets, the willingness to wait in line, and political factors. When I waited for the generator, a guy stopped by the line and said there was someone in the parking lot with a couple of generators on sale. You can see these factors in play with the release of new video game systems as well. The releases are always underpriced relative to demand so they are always accompanied by long lines, exorbitant 'black market' prices on E-bay, and back channel dealing. I've always wondered how many 'new friends' J Allard gets when a new version of Xbox is released. Even I got calls from naïve long lost relatives when the Xbox 360 came out. Say what you want about the inequality caused by pure market forces, at least prices provide transparency.

The key thing to realize when you try to apply this this idea is that people don't respond to the well-meaning intent of a policy. They may in fact respond in a way that is exactly opposite to the intent but in a way that is still quite rational. The economic effects come into play even during times of national emergency like WWII or local disasters when everyone is supposed to be banding together. We shouldn't expect something as prosaic as our jobs to be immune to this rule.

How does this work in the bug economy? What happens when you try to institute controls on bug counts? You get a black market in bugs. Issues start to drop out of product studio and get tracked informally, if at all. Some bugs get changed to tracking or are moved to other milestones, often without the guiding hand of a bug-bar. Bugs become hot-potatoes even when the product would be best served by deliberate consideration.  The lesson to learn is that you should never drive the indicator. If you institute gas price controls, that doesn't magically make any more gas appear. (Ask Jimmy Carter.) Similarly if you drive bug counts, that doesn't magically increase the quality of the product. What really happens is that you lose the value of your leading indicator, bug counts. This leads to my next observation.

Sound monetary policy is critical

Money has no intrinsic value. Money is only a mechanism for making it convenient to exchange the value you provide when you work for the goods someone else produces when they work. When you erode the value of money through inflation you introduce uncertainty. High inflation makes it impossible to make long term investments. People don't want to invest for future returns because that future cash will be greatly eroded.

One of the biggest problems in post-revolutionary America was the lack of hard currency. This impeded trade and made it difficult for the likes of Thomas Jefferson to get out of debt. That doesn't mean there isn't a 'real' economy. It just means that it's much harder to measure because the real economy retreats to black markets and barter.  The real economy is also much less efficient and creates less real wealth than one that is mediated through prices. This barter economy was the real reason Alexander Hamilton came up with the whiskey tax which in turn led to the Whiskey Rebellion. Whiskey was the only commodity and currency in wide circulation. Taxing whiskey was the only way to pull westerners into the larger federal system.

In the bug economy, bugs have no intrinsic value. Their presence does not mean that quality is low, and their absence does not mean quality is high. For example, when you reward finding a lot of bugs, you introduce bug inflation. The value of a bug degrades and the devs scoff at the bug count because there are so many trivial issues.  A single check-in can wipe out a score or more. This is just like the German buying a loaf of bread with a wheelbarrow of money in the 1920s. Again the real problem isn't that you have a lot of bugs, but you don't know what the bug count means anymore. By driving the meaningless indicator, rather than the underlying value you have introduced uncertainty and destroyed the value of the bug-count as a predictive tool. Deflation has similar problems. When the bug count gets driven down artificially you can have a hornets’ nest of problems hiding in an apparently small number of bugs.  Again, the 'real' quality economy retreats into the black market.

There is one big problem with instituting policies based on this fact. It's hard to tell the president that interest rates must go up to slow the economy and prevent inflation, especially in an election year. This leads us to my next observation:

The fed must be politically independent

No president would raise interest rates so that his successor can have a strong economy. I have some doubts that there has ever been a president who would sacrifice reelection to do the right thing if they had the power. The temptation is just too great. For example, Bob Woodward quoted Bill Clinton as saying "You mean to tell me that the success of the program and my re-election hinges on the Federal Reserve and a bunch of f****** bond traders?'" Only a politically independent fed can do the right thing according to its charter.

The closest equivalent we have in the software world is shiproom. Unfortunately, it's rare that it has the leverage and independence to make sure that bugs remain a meaningful indicator. Even if the person driving the meeting has some independence,  the people providing data to ship have their own conflicts of interest.  Sure, quality is never intentionally sacrificed, but remember the point about "people respond to incentives." The quality incentive doesn't hit until after the next review, if ever. The bug count and shipdate incentives hit much sooner and much more directly.

This group-think and conflict of interest can lead to overly optimistic schedules. All too often schedule estimates sound more like a refrain from "Bob the Builder." Bob says: "Can we build it?" His team responds "Yes we can!" Ironically, strong leadership can actually be a detriment. Inspiring a team to do the impossible can have long term negative effects. For myself, I've always had a soft spot for Lofty who always adds "er, well, I think so."

A closely related issue is that you can't delegate shipping. Yes, it's important to have long term vision about new features and come up with exciting new stuff, but at the very least executive management needs to be able to call BS. I'll never forget what my first GPM told me when I was getting beat up about putting together a schedule and driving a product. "Driving a project is a core part of what every PM does." At the time I was fresh off of a successful presentation I gave to BillG and every manager and VP in the chain of command (Well, SteveB wasn't there)  as well as a handful of other assorted VPs. I didn't understand how my ability to handle executive presentations didn't trump the nitty -gritty, boring parts of the job. Let’s just say that five years later, I am a convert.

Resources are scarce

Despite the fact that it can be very easy to fool yourself  about working harder to get more things done and how the magic of technology has destroyed the business cycle, productivity is not easily changed. Bugs may not indicate what you want them to, but the product still has a certain quality level, even if you can't measure it well. There is still work to fix issues, no matter how good the team is. Spending resources in one part of the economy means that another part will not have those resources available. If you over produce steel, then you'll have excess inventory and a recession until that inventory is worked off. Leadership cannot change this fact. Software has its own laws. Builds take a long time. Even the most trivial of bugs takes time to open, time to fix, time to check in, and time to regress. Code churn leads to regressions, glide paths can't be forced, and software that isn't tested is always broken.

The thing is that there is a real quality level lurking behind all of those bugs and a real economic output hiding behind the stats at http://www.bea.gov/. The best you can do is treat those stats with respect and do everything you can to make sure that those stats continue to reflect the bug (or non-bug) economy.