I. M. Wright’s “Hard Code” : Software Quality--More Than a Dream

Nailing the nominals

ericbrec — Wed, 01 Oct 2008 09:00:00 GMT

People are always looking for that amazing breakthrough technology or process that solves all their problems—enhances their love life, trims their waist, and improves the productivity of their development team. That's why process manias like Agile and...(read more)

Crash dummies: Resilience

ericbrec — Thu, 01 May 2008 09:00:00 GMT

I heard a remark the other day that seemed stupid on the surface, but when I really thought about it I realized it was completely idiotic and irresponsible. The remark was that it's better to crash and let Watson report the error than it is to catch the...(read more)

Software performance: What are you waiting for?

ericbrec — Thu, 01 Nov 2007 10:25:00 GMT

You hurt your shoulder playing volleyball, so you make an appointment to see your doctor. You enter the office and wait in line for five minutes just to let the receptionist know you've arrived. He has you verify your contact and insurance information, which haven't changed in ages, and then tells you to sit in the waiting room.

You sit in the waiting room for ten minutes, inhaling all kinds of ailments from the crowd, seething about how you're going to leave sicker than you were when you came in, till a nurse shows you to an exam room.

After five minutes in the exam room, another nurse comes in, takes your vital signs, and has you repeat the reason you gave for the appointment when you originally made it. Ten minutes later, your doctor arrives and actually addresses your shoulder injury.

Welcome to a user's experience running our software. You wait forever just for it to launch. You provide your credentials again, even though you gave them when you logged in. You wait again for your personalized environment to load. You click a few menu items or buttons to launch the specific functionality you want. Finally, you wait again while the feature prepares to do what you actually launched the software to do in the first place. That's assuming there aren't network delays.

Just one moment, please

Waiting is dull. Waiting is frustrating. Waiting is agonizing. Waiting is just an unbelievably bad time all around, on every level. Nobody likes to wait. Nobody asks to wait. So, why the heck do we make people wait?

Actually, why do our customers even put up with it? Why do I put up with it at my doctor's office? I guess because all doctors' offices are slow. But, if one of my friends told me about a quick doctor's office that provided comparable care, I'd switch in a heartbeat.

This means a competitor's quicker program could flatline our business. How do we make our programs quick, before our competitors do? I'm glad you asked.

Eric Aside

If your doctor is good enough, you may put up with significant hardship instead of switching. However, giving people a reason to switch is inexcusable, especially when it's easy to do better.

You're faster than this

So, you want better performance from your software. Where do you start? "I know, I know!" says our resident performance pariah, Mr. Speedy. "Profile your code, find out where it's spending all its time, and then optimize, perhaps even parallelize, those inner loops."

Well, Mr. Speedy, aren't you clever. Let's profile our doctor's office, shall we? Whoa! It turns out the doctor is always busy, and that's the bottleneck. Who would have guessed? According to Mr. Speedy, all we need to do is speed up the doctor, find a faster doctor, or get two doctors to do the job of one. Right? Wrong!

There's nothing wrong with my algorithm—I mean, my doctor. If you made her faster, she wouldn't be any better. In fact, she'd probably be worse. Doing the job right takes time, and making all kinds of optimizations might improve things a little, but might also cause mistakes. I don't want a different doctor who happens to be faster, either. I like my doctor. I know her well, and she knows me.

I also don't want two doctors. Even if they are twins, I'll never know which thread—I mean, which doctor—I'll get. They might both try to treat me. They'll have to communicate with each other all the time to avoid mistakes. They might even get stuck waiting on each other. It's way more complicated, and it really doesn't solve the problem even if two doctors are twice as fast. I've still got to deal with the receptionist, waiting room, exam room, and nurses.

Eric Aside

"But what about multi-core processors?" you might ask. Look, there's using technology for the sake of technology, and there's using technology for a purpose. If the user experience demands threading across multiple processors, I'm all for it. If not, you're just giving yourself a cheap thrill at the expense of the customer.

Should I keep a copy?

"Hold on," says Mr. Speedy. "What you need is a cache–that always speeds things up." Hello! We've got cache fever in the doctor's office. That's part of what's slowing us down. We've got a reception line cache, a waiting room cache, and an exam room cache.

It seems like everyone at the doctor's office is concerned about speeding up their own work, so they all created their own caches. The receptionist created a cache, the nurses created a cache, and the doctor created a cache. The result is that patients spend all their time waiting and moving between caches instead of being processed by the doctor.

Think this doesn't happen in code? You've obviously never looked inside those database, shell, and system calls you use. All that data you're caching for "performance" is already being cached for the same reason by those functions. Sometimes there are as many caches as there are layers. Every cache has a fetch and memory cost to it. Well, I'm cached out.

You're not being the ball

Let's start over, shall we? Instead of speeding up the existing doctor's office, as marginally effective as that might be, let's think about things from the patient's perspective. What would you and I, as patients, like the experience to be?

Here's what I'd like. Check my contact info and insurance when I call in for an appointment. Write down my symptoms and include them in the appointment. Heck, let me do it all online (wait, that's crazy talk)!

When I show up at the doctor's office, I can walk right to my exam room (just one level of caching). It's the room with my name above it, just like at the good rental car agencies. A big sign in the room says, "Please take off your shirt and have positive identification ready for the nurse, then hit the big 'I'm ready' button."

The nurse, seeing that I've hit the "I'm ready" button, comes in, checks my ID, takes my vitals, and hits the 'Vitals taken' button, which adds me to my doctor's queue. As soon as my doctor's available, she comes in and addresses my needs. That would be great! Heck, there could even be a monitor in the exam room with queue stats and predicted wait times. The stats could be used to fine tune the number of appointments available per hour to minimize wait times, while still ensuring that each doctor is fully utilized.

Eric Aside

If you haven't read about the Theory of Constraints or its drum-buffer-rope approach to optimizing results, you are in for such a treat. They should be required reading for anyone trying to rethink and revolutionize performance of everything from software to cafeterias.

Have you ever been experienced?

There, that wasn't so hard. Setting up a doctor's office the way I described would be easy and not that expensive. It doesn't require more doctors or faster doctors, and it actually saves floor space. Sure, the online appointments and predicted wait time monitors would require special software, but those aren't essential to get a better and faster experience. What is essential is to think through the customer's experience with a view toward minimizing wait time.

Here are some questions for your consideration:

§ When was the last time your team thought through the end-to-end customer experience, including the wait time?

§ How would the customer want to deal with the inevitable constraints that every process has, besides giving them a CANCEL button? (Associating our software and services with "cancel" seems unwise.)

§ How could you minimize the impact of errors, network delays, and device I/O in a way that customers would find natural and unobtrusive?

§ What measures and statistics could you use to fine tune the experience, minimizing wait times while getting full utilization of key resources?

Right now we design experiences as if these performance constraints don't exist. Everything is modal and synchronous as if functions always return and people never select the wrong option. We design a feature at a time, instead of end-to-end; or if we do design end-to-end, we only think about the ideal scenarios, not the likely ones. We assume exceptions and delays are unusual, even exceptional. That's naïve, which is a kind way of saying "stupid."

Eric Aside

Of course, there are great examples in Microsoft software for thinking through the end-to-end experience quite nicely. I mention some in the next section.

You'll be ready

Performance tuning does have its place. There are functions and services that must scale up and out. There are issues with blocking and locking that require special care, which a real performance expert can help you resolve. It's just that those aren't the common case.

The common case is the ordinary case. A customer is trying to get something done. It involves network access and I/O. Those interactions could fail or cause delays. The customer generally has experienced these problems and knows they exist. The best way to handle them is to talk to customers and understand ideally what customers want to happen.

Perhaps the customer would be happy if the I/O completed asynchronously—the solution Outlook and OneNote use to vastly improve the customer experience. Perhaps the customer would be happy to work on a local copy and synchronize on demand—the solution ActiveSync and FrontPage use. Perhaps the customer is happy to queue their requests and get a status report later—the solution build systems and test harnesses use.

What about me?

The key is to look at the world from the customer's perspective and to design an experience that anticipates failure modes and minimizes their impact on users. Performance should be specified in the experience with specific measures and guidelines, not left to chance or hope. It typically doesn't require complex algorithms or fancy caching, both of which can be overdone. It requires being thoughtful and deliberate.

When performance is specified through the experience, it's built-in and tested from the start. No one gets surprised or has to scramble at the end of the project to suddenly become a performance expert. The only surprise is on the customer's face when what they thought would be an agonizing doctor visit turns out to be a delightful one.

October 1, 2006: “Bold predictions of quality”

ericbrec — Sun, 01 Oct 2006 02:48:00 GMT

I. M. Wright's "Hard Code" book excerpt

I’ve been busy dogfooding lately. It’s an ideal diversion for masochists. When it gets to be too much, I can always take respite in a nice horror film. Thank goodness what passes for dogfood now is a vast improvement over years past.

Eric Aside

Dogfooding is the practice of using prerelease builds of products for your day-to-day work. It encourages teams to make products right from the start, and it provides early feedback on the products’ value and usability.

Years ago, running a dogfood build and having your machine unplugged were almost indistinguishable in terms of productivity. These days, substantial parts of dogfood builds are fully functional, while others remain unusable, unreliable, or unconscionable. This begs the question, “Why?”

Why are some portions of new builds solid, thoughtful, and engaging, while others remain flaky, unfathomable, and exasperating? How can that be? Ask managers and they’ll say, “Well, it’s tough to tell ahead of time what’s going to be good or rancid.” Sounds like they’re washing down my dogfood with male bovine dung.

Enigma? I don’t think so

Software quality is unpredictable? Don’t make me gag. Poor-quality software has all the subtlety of a neighborhood ice cream truck. You know it’s bad for you; you know it’s coming a mile away; yet you can’t resist. Managers choose to ignore the signs and buy the ice cream (poor software) because they hate disappointing the children (upper management) and can’t resist the instant gratification (“progress”).

We’ve gotten so used to poor software that many people have forgotten the early signs. Let me summarize the rest of this column and make it simple for you. Good software is solid and originated out of complete and critical customer scenarios. Bad software is buggy and originated out of someone’s behind.

Twins of evil

How do you spot bad software before it’s integrated into the main branch? First, remember there are two aspects to quality—engineering and value. Most engineers get caught up in the engineering side of quality—the bugs. However, flawlessly engineered features can be glorified crud to customers if the ideas came from the wrong place. I talk about this more in “The other side of quality” in Chapter 6, “Software Design If We Have Time.”

We’re looking to predict both buggy code and code with questionable pedigree. Predicting buggy code is easier, so we’ll start there.

The usual suspects

In 2003, Pat Wickline studied the root causes of late cycle bugs in Windows Server 2003. The results were similar to his 2001 study of bugs in Windows 2000 Hotfixes. Simply put, more than 90% of bugs could have been found by design reviews, code reviews, code analysis (like PREfast), and planned testing. No one method would have found every bug, but the combination nearly finds them all.

In 2004, Nachiappan Nagappan studied measurable attributes in an engineering system that correlated well to bugs found later. Those attributes were code churn (the percentage of code lines added or changed per file) and code analysis results (the number of PREfast or PREfix defects found per line of code).

Eric Aside

He’s updated his thinking to focus on churn and complexity measures.

If you want to prevent poorly engineered code from getting into the main branch, have your build track code churn and code analysis results. If those measures go beyond the norms for quality code, then reject the check-in. If your developers don’t like it, tell them to do more design and code reviews and write more unit tests.

“What if there’s too much code churn, but the feature enables a complete and critical customer scenario?” you might ask. Allow me first to congratulate you on coming up with the only decent reason to not junk the code entirely. Then junk the code entirely. It’s time for a rewrite of that section. That’s the only way the feature will ever reach your engineering quality goals.

You’re gonna love it

Let’s move on to predicting questionable feature pedigree. Buggy code is easy to measure and control, though it does require management to set a bar and stick to it. The value of software is harder to measure, but in the end it requires the same thing—management must set a bar and stick to it.

How do you know if a feature or check-in will really be valued by customers? That’s easy. If it’s part of a complete and critical customer scenario, then users will love it. How do you know if a scenario is complete and critical? That’s the hard part.

Luckily, you don’t have to do that work. We pay marketing, product planning, and upper management to figure out the complete and critical customer scenarios for a release. No one feature team or product group can do it, because complete scenarios cut across product groups. Instead, engineering’s job is to tell the planners what’s possible, and then solidly implement the planned critical scenarios from end to end.

Quit fooling around

Of course, overzealous engineers of all kinds, not just PMs, will try to sneak in features that aren’t part of planned complete and critical scenarios. While doing so might relieve that engineer’s creative constipation, what comes out is predictably putrid for customers.

To trap poorly conceived features before scarce resources get wasted, you must take two steps:

1. Have a clearly documented vision or value proposition that lists the complete and critical scenarios for the release. Prototypes, personas, user experience designs, and high-level architectures also help clarify what’s needed immensely.

2. Convene a governing board who owns the vision or value proposition and have them review every feature. If the feature doesn’t fit a complete and critical scenario, it’s cut. Period. At the beginning of each major milestone, every GPM reviews their list of upcoming features with the governing board. While the board may not review every feature in great detail, they must still ruthlessly and relentlessly uphold the quality, value, and integrity of the release.

Eric Aside

The best groups at Microsoft have been following this process for years.

These two steps precisely correspond to setting the bar and sticking to it. While this bar is more subjective than the engineering quality bar, both require the same disciplined commitment by management to be successful.

Quality is no accident

It’s not difficult to predict quality. In fact, it is straightforward. Yet managers at all levels rarely apply the rigor necessary to assure quality.

Maybe managers are afraid assuring quality will add too much time to the schedule. As if doing it right the first time and sticking to only the critical needs takes longer. In fact, when quality is what customers expect, then focusing on quality is always the fastest way to ship.

Maybe managers are afraid engineers won’t like assuring quality. As if engineers take no pride in their work or enjoy ambiguity and wasting their time. In fact, engineers take great pride in the quality of their work, prefer to know what’s expected, and hate wasting effort.

The truth is that quality is expected, quality is fundamental, quality is central to our success. It is because our customers say it is.

Quality is the right thing to do and the right way to do it. It is the key to our future survival and prosperity. Quality is no accident. You can predict and control it. All you need is a brain and a backbone. Get yours today.

Eric Aside

While they both vigorously advocate quality, it’s worth noting the differences between “Where’s the beef” and “Bold predictions of quality.” The first discusses why quality is needed and the mechanics of getting it. The second describes how to measure and refine your work to push the quality bar higher. We’ve made significant progress, but quality is an ideal that demands eternal vigilance.