Wash your hands

One author I've been reading lately is Atul Gawande. I recently finished Complications and before too long I expect to start Better. Atul Gawande is a staff writer for the New Yorker. Just like other staff writers like Malcolm Gladwell you find that his best articles are insightful twists that you can apply generally. And also just like Gladwell, each idea is obviously just a small viewpoint into the whole truth. What makes it compelling is that it's not comprehensive, and doesn't give you the caveats that always apply. That’s how both Blink and The Wisdom of Crowds can both be right. Anyway, that's the way folks write nowadays. In that spirit, I'll jump into some of the ideas Gawande advocates.

Medicine is a particularly apt metaphor for software development. Both are practiced by highly paid professionals with years of training and experience. Both have a built in tension between heroic individualism and rigorous procedures. In software development we all know about the charismatic rock star coder who gets things done just by exerting the force of his or her will and intellect. Yet there is a side of development that wants to emphasize the true engineering aspects of the job. But we just don't have quite the same mindset as other engineering professions. When you are a civil engineer, it's great when you can make a beautiful bridge but it better not fall down when cars drive across it. No matter how brilliant the engineer is, we don't trust their judgment without significant testing and oversight. Despite the life and death consequences, doctors are subject to a different mythos. Doctors often have a hero complex. They feel that it's their judgment in a time of crisis that makes the difference between life and death. All the rules and procedures just get in the way. Certainly this is how the media typically portrays doctors. Sure when you push them, doctors will acknowledge that systematic processes are important. But that's not how they think of themselves and that's not how they invest their energies. The same hero complex pervades much of software development. We seem to acknowledge this similarity with the medical profession, at least at some level. You can see it in our language from how we 'triage' bugs to how after we ship we hold 'post-mortems' to identify ways to improve. 

It's with this model in mind that I read a recent article from Gawande about checklists. It turns out that I.C.U.s are dangerous places. But it's often not the primary trauma that kills the patient. Rather it's the infections and complications that can arise from the treatment. There are hundreds of little things that have to go right to prevent one of these bad things from happening. There are countless opportunities for even the most diligent to make mistakes. A particularly pernicious area of risk is infections from intravenous lines. One doctor, Peter Pronovost, decided to address this problem systematically. He made a checklist of things doctors were supposed to do when they put in a line. Here is the list:

1. Wash your hands with soap
2. Clean the patient’s skin with chlorhexidine antiseptic
3. Put sterile drapes over the entire patient
4. Wear a sterile mask, hat, gown, and gloves
5. Put a sterile dressing over the catheter site once the line is in.

These are all obvious steps that everybody already knew. I bet I could have come up with some of them myself. But what he did with the list was the real innovation. He gave the list to nurses and empowered them to stop the doctor if they missed any step in the process. Here's what happened:

• The ten-day line-infection rate went from eleven per cent to zero.
• Only two infections occurred over fifteen months.
• This process prevented an estimated forty-three infections, eight deaths, and saved two million dollars.

The results were stunning, but perhaps predictably, he has had little uptake in the medical community. Doctors have had too much training to be humiliated by a piece of paper. They said they needed to focus on the patient, not on some process. Things happen too fast to be hindered by extra paperwork. But yet the same hospitals are willing to invest tens of millions of dollars on new line kits with a special coating that improve the infection rate only slightly.

When I look at this anecdote, there are two things that really jump out at me regarding the world of software. One is the idea of the checklist itself. We have many of the same problems where we think that individual developers or leadership make the best decisions on whether something is ready for checkin or if this or that DCR should be accepted. The procedure for doing this right is so well known that nobody finds it interesting any more, but yet we rely on experience rather than procedures. We know about code reviews and the best way to handle them. We know about buddy builds, BVTs and due diligence but when push comes to shove we exercise our judgment and cut corners. There are lots and lots of best practices that are easy to forget or ignore. Despite how mundane these issues are, nobody wants to give up control to a checklist. But checklists and simple procedures can be incredibly liberating. Anything that requires the 'stupid' part of your brain can be written down and forgotten. Executing on the checklist becomes a matter of simple hygiene, like washing hands and brushing teeth. Following a check list keeps you diligent and frees you to think about higher level problems. Software also has the same love of technology to solve our problems. We invest millions in automated code analysis tools, but our code reviews turn into rubber stamps.

But perhaps the real power lies with empowering the nurses. They are responsible for the overall quality of care, but are still subservient to doctors in many ways. We have much the same relationship between test and development. As a company, Microsoft has put enormous effort into ensuring test has a good career path. But yet many of the best testers view test as a stepping stone to development. Very few developers have the same aspirations to move to test. If we could empower test to become the true gatekeepers of quality, we could solve a number of perennial problems.

• Test gets more oversight and a stronger sense of ownership of quality. This inevitably leads to greater career growth opportunity and better engagement from the test team.
• We get an objective assessment of our adherence to best practices.
• Test has no interest in getting the bug count down or getting a dev out of bug jail, so there is no conflict of interest that leads to cutting corners.
• Development is forced to ensure test is informed. Surprising test at checkin time will only get the checkin delayed.

Of course there will still be plenty of places to use judgment.  But taking the role of judgment away from places where it really doesn't belong frees you to exercise judgment where it's needed. Imagine how debilitated you would be if getting dressed, brushing your teeth, or eating breakfast were all judgment calls. Development will slow down, but that's a good thing. It will be hard to checkin if you need to find a tester to approve your checkin. This is especially true for distributed teams where test is on a different continent. But the reality is that those situations are enormously difficult already, this process simply highlights the underlying reality.