Kill Your Intuition

Back when I was in graduate school for physics, there was a professor who was very interested in physics education. He had a set of questions that he would give to Freshman physics classes at the beginning and at the end of the semester. These weren't typical analytical physics questions, but rather subtle variations on them. For example, rather than asking a person to calculate forces a question might ask if the force of a car pushing a truck up hill was greater than, less than, or equal to the force of the truck pushing back on the car. Even good physics students would get these questions wrong. His conclusion was that our physics education techniques weren't doing the students justice. They still didn't 'get' the underlying laws they could recite so well.

But there was something that always bothered me about this analysis. Perhaps it was just that I got one of these questions wrong myself, but it just didn't jibe for me. Then during a colloquium when he was presenting on this topic, it finally came to me. His questions always tested the students' intuition about physical phenomena. Hence his unstated goal for physics education was to improve the students sense of how the world worked. That may actually be a laudable goal, but what I realized was that it was much more valuable to teach people to analyze a problem and overcome their intuition.  In many ways you could argue that the triumph of the scientific theory was the application of systematic analysis to questions that were once relegated to experience or gut feeling.

Another example that comes to mind was an experiment run by a classmate of mine in college. She was a math major but was doing some cross disciplinary research with the psychology department regarding math anxiety. She was going to set up an environment designed to instill anxiety about math and then give the subjects a test. What was striking was the conclusions she expected to draw. She told me that if the students did well they didn't experience anxiety, and if they did poorly they were anxious.  Anyway, I asked the probably inappropriate question "So why run the experiment?" After all, she had her conclusions and the experiment was constructed so that they couldn't be disproven.

This ability to critically analyze your intuition and determine if it's right or wrong is key to a scientific mind. It's easy to forget about  the power of Karl Popper's philosophy of falsification. In order for a theory to be valid scientifically, there must be a test that can prove it wrong.  Further, experiments can never prove a theory fully correct, they can only disprove it. Needless to say, this rigor of thinking is not applied to the worlds of business and software. Instead people learn through a series of successes and failures and try to apply the lessons from the last project to the next. Sometimes these practices get codified and enforced, or at least documented.

I was reminded of these incidents from my college days recently as I was reading The Halo Effect: … and Eight Other Business Delusions that Deceive Managers. One of the themes the author repeatedly comes back to is that business research and most management books do a very poor job of analysis. They do things like ask pre-selected successful companies what makes them successful and then categorize the results and present them with the authority of science. The problem is there is no way for them to be wrong. If they can't be objectively wrong, then they haven't stated anything. In order to really know if 'customer orientation' is predictive of success you need to objectively find a measure of customer orientation, randomly select companies, measure them, and track performance over time. Even that may not be enough, because it could be that customer orientation is correlated with other practices that are the real cause of performance. Perhaps it's even worse than that and all the identified best practices correlate with one another.

This same scourge of non-critical thinking applies to the execution level of software development as well. All too often teams will come up with bug glide paths using estimates like "We'll have an incoming rate of X and we can fix bugs and a rate of Y per dev per day. We'll probably find Z bugs after the bug bash when we check in feature W. That all means we'll reach zero bug bounce by date P." You come up with a rather complicated bug glide path that bears no resemblance to what actually happens. What's stunning is the explanations for the deviations that float around. If you spike bugs early, you can claim "We are ahead of schedule and finding bugs early." If you are under the expected bug count, you aren't wrong. Instead you are in good shape because the bugs are under control. If you slip, something unexpected and unforeseeable happened. The next milestone you do the exactly the same thing and shrug that every project you've ever been on has had similar course corrections. Peoples' ability to deceive themselves makes it surprisingly easy to overlook the irony. The glide paths can help tell if you are in trouble or not, but the way they are normally constructed they cannot systematically improve your ability to execute the next time around.

This doesn't mean that you can't make decisions without double blind tests that track multiple software projects over multiple releases. But when you make a schedule, use a metric to track quality, or plan a project, try to figure out how you will know if you are wrong. Make the effort to dive into the data when you estimate incoming rate or fix rate. Your numbers might not be accepted by those higher up, but at least they will have to put more thought into why they think the numbers will be different. Look for measures of quality that aren't so self-referential and subject to social pressures like bugs are and see what where they take you. But most importantly follow up on your predictions. If you have an objective measure shows great quality for a product in development, figure out ahead of time how good quality looks in the market and see if you measure was useful.

And above all, always look for the alternative explanation.  If conventional wisdom says one thing, try to see how it could be wrong and examine alternatives. Identify future measurements or events that would validate one story versus and check up on them when the time comes. Of course there is still some gut feelings to apply, but if you can ferret out a little bit a truth for each release, you'll actually learn something.