I. M. Wright’s “Hard Code” : Inefficiency Eradicated

Your World. Easier

ericbrec — Wed, 01 Apr 2009 09:00:00 GMT

During difficult economic times like these, people tend to whine less about common complaints that now seem trite. Mostly, I'm relieved not to hear how much e-mail is in Ingrid's Inbox, how Brian broke the build again, and how Suresh's service schedule slipped successive sprints.

However, it's during difficult days that we should patch plaguing problems. When are you going to be more motivated to mend malignant maladies? Surely, no additional alliteration is advisable.

A surprising number of common issues can be solved using two simple techniques—single-piece flow and checklists. There's a ton of behavioral and process theory behind why these simple methods are effective. The point is that they are effective and you and your team are less effective without them.

All too easy

Take Ingrid's Inbox. Like most Inboxes, Ingrid's is overflowing. She spends tons of time on it, yet it only gets bigger. She constantly loses track of mail, discovers mail, and revisits mail. It's hopeless.

Ingrid would have her mail under control if she followed single-piece flow. Single-piece flow tells her to handle one piece (one message) at a time till it's done. By "done" I mean she'll never look at that message again (except to answer a related message).

Here's how it works. Each time after Ingrid reads an e-mail message she does one of four actions:

1. She deletes the message (my favorite).

2. She files it away in a folder.

3. She forwards the message to someone else and then deletes or files it.

4. She answers the message and then deletes or files it.

That's it. She never opens a message and then leaves it in her Inbox. By the end of each day, every day, her Inbox is empty. She never misses a mail message, loses a message, or revisits a message.

Single-piece flow is efficient because it removes overhead and rework. There's overhead every time you context switch to look at a new message, and there's rework when you re-read a message. In single-piece flow, overhead is minimized and rework is eradicated.

Eric Aside
Sure, there are exceptions to the read-it-once rule, but they account for less than 5 percent of the mail I get in a day. Even those e-mail messages get filed in a special folder for more intensive attention. I also use Inbox rules to pre-filter mail from discussion groups.

If you're wondering how to get started and don't want to just delete all the mail in your Inbox, follow these steps:

1. Take mail you are actively working on right now and move it into a special folder.

2. Create folders for your remaining mail that correspond to your obligations and interests.

3. Search your Inbox for mail that fits each folder and move that mail into the folder.

4. Go through your special folder and clear out 90% of it using single-piece flow.

Now you are on your way.

Deja vu – all over again

Brian keeps breaking the build. You can punish Brian till he's afraid of checking in code regularly, but doing so only causes other problems.

A better solution is to give Brian a checklist. Checklists are wildly misunderstood, improperly developed, and underutilized. Regrettably, well-meaning compulsive people list everything possible Brian should check before he submits code to the build. That's not only a waste of time, it's also ineffective.

Brian's checklist should list only common causes of build breaks (less than one page’s worth). It should be in Brian's sight when he submits code. The goal is to be quick, easy, and effective.

Too long or complex and Brian won't follow it. Too short and it's not effective. Luckily, most mistakes are common mistakes. Thus, all the team needs to do is collect a list of the common or critical causes for build breaks and turn that into a checklist. The same is true for design review lists, code review lists, and all checklists.

Remember to update your checklists as your failure patterns change or they will become stale. Checklists prevent common errors and promote good habits. Any structured manual process you follow should have a simple checklist—unless you like being Brian the build breaker.

Eric Aside
You may have battles deciding what goes in a checklist. You shouldn't. Remember, you list what the data says are the common or critical failure points, not everything that ever happened.

For example, I added a checklist to my e-mail signature several months ago, which I check and then delete before I send every mail. It lists the two mistakes I've made for years, but haven't made since:

· Check the Cc and Bcc lines for undesired aliases

· Ensure the subject line is accurate

Slip sliding away

Suresh's software squad slipped their schedule on successive sprints. Bad news for a service—or any project. Is it time to work weekends? No. Is it time to slap the squad silly? No. Is it time for single-piece flow and checklists? Yes.

Suresh's squad is made up the usual way—a few PMs including a service engineer, a bunch of developers, and a similar-sized bunch of testers. The PMs write specs, the developers code them, and the testers test them. The squad is using Scrum-like Sprints with a nicely prioritized backlog of features.

Unfortunately, the PMs are creating specs faster than can be implemented. They waste time and effort on specs that change or are cut before they see daylight. The developers and testers jump from feature to feature as they get blocked. Nothing is in sync. Nothing gets finished. The schedule slips incessantly. In retrospectives, all Suresh's squad talks about is unblocking people.

Even though Suresh's squad is using Scrum-like Sprints, they aren't using single-piece flow. They aren't splitting into cross-discipline teams, with each team working on one feature at a time till it's done. They don't even have a checklist that defines done. It's doomed.

Once they create feature teams that spec, code, and test one feature at a time (sometimes called, feature crews) and a checklist that defines done, there's a chance for progress. The single-piece flow removes blocking issues because everyone is working on the same problem. Instead of jumping ahead, the checklist keeps the team honest, motivating them to work together, finish, and stay focused. Now the team doesn't waste time context switching and can tell how long it really takes to complete a feature, leading to confident scheduling and higher quality finished products and services.

Eric Aside
Many people wonder what to do with feature crew PMs once they finish specing, or developers once they hit code complete. For PMs there are two solutions—be part of multiple feature crews or be prepared to work with their dev and test peers on problem solving and development. For developers, the right solution is to work with the test team on tools, automation, unit tests, and component tests.

There's another clever solution written up by a former Microsoft employee, Corey Ladas. He calls it Scrumban and you can read about it on his and Bernie Thompson’s Lean Software Engineering site.

Our two weapons are

So there you have it—single-piece flow and checklists. Two enormously useful, remarkably simple, and yet woefully underutilized techniques for managing workload and building quality software.

Single-piece flow and checklists can be applied to individuals, small teams, and large divisions. They aren't controversial when used pragmatically, and have years of documented case history supporting their effectiveness.

Sure, you could create a whole grassroots movement around single-piece flow and checklists, but that seems a bit overblown for such simple ideas. Maybe you should just use them wisely. Enjoy the time you get back and the improvement in your results. The best things in life are often the most basic and simple.

Eric Aside

James Waletzky, has an amusing blog entry with a few checklist references entitled, Checklist? We don't need no stinking checklist!

Green fields are full of maggots

ericbrec — Sun, 01 Feb 2009 10:00:00 GMT

As I said in Nailing the nominals , the two keys to successful big projects (100K+ LOC) are thinking ahead and defining done. Thinking ahead is about design and planning. Defining done is about setting a quality bar and sticking to it. Yet many big projects...(read more)

De-optimization

ericbrec — Mon, 01 Dec 2008 10:00:00 GMT

Why? Why! Why do managers make stupid decisions that cause devastating churn and tawdry results? And it's not just managers, though they are particularly proficient at promoting poor performance—architects, leads, and individual contributors flood the...(read more)

So far away: Distributed development

ericbrec — Fri, 01 Feb 2008 10:00:00 GMT

If you are a software geek, like me, being the product support technician for your friends and family comes with the territory. While it's painful to watch your family struggle with software, particularly if you helped write it, at least you can tell them, "Back off, I'm a computer scientist," and repair whatever is wrong. Sure, you'll cringe as you undo their failed "fixes," but in time you'll set things straight. That is, if you live nearby.

If your mom lives a thousand miles away, the call might sound more like this:

Mom: Honey, they changed my password and now my e-mail doesn't work.

Me: Okay, log onto your computer and open Outlook.

Mom: Using what password?

Me: Use whatever password you normally use.

Mom: Not the new e-mail one, just my old one.

Me: Yes. Then open Outlook.

Mom: Where do I type in the password?

Me: On the main screen where you click on your name and then type a password.

Mom: What main screen?

Me: Wait, can you just open Outlook right now?

Mom: Yes, I've got it open, but you told me to log on.

[Another hour like this moving through menus, dialog boxes, and buttons…]

This experience is far better if you can get a tool like Remote Assistance to work, but getting that up and running behind firewalls and routers can be equally entertaining. What should be a five-minute fix becomes an hour of acute aggravation. This order of magnitude multiplier for time and trouble comes into play any time you work across time and distance, which brings us to the topic of distributed development.

Doesn't anybody stay in one place anymore?

It's becoming far more common to run development across global locations for the same project. While the added diversity and talent should be a huge advantage, the results are often frustration, delays, and disconnects in quality and functionality. Why? It's due to the big, bad Bs—bandwidth, boundaries, and being there.

There's insufficient bandwidth for clear communication and fast network access to central services. The boundaries between project work at the different locations are poorly defined causing additional communication, conflicts, calamities, and clean-up. And because the different teams are separated, the one-on-ones, drop-ins, hallway conversations, and other daily human interactions you get from being there don't occur.

With all this trouble you might wonder why we bother with distributed development. No, you fool, it's not the money. Engineering salaries and costs are converging for many roles in China and India, and distributed development adds overhead. The real reasons for distributed development stem from the talent and the markets. The computer science talent pool in Brazil, Russia, India, and China is growing at nearly 20% per year and will have 1.5 times the number of software engineers in all of the United States by 2010. They can't all move to Redmond, and we wouldn't want them to because their home markets are critical to our success.

So if you're running a team, you'd better learn how to deal with the big, bad Bs. Let's break them down.

I get so tired when I have to explain

The first big, bad B is bandwidth—bandwidth between people and between computers.

Clear communication between people and teams is critical to success, as I've said many times before. The amount of information that is communicated between two people in the same room far exceeds that between people over video teleconference (VTC), which in turn exceeds Live Meeting, which exceeds the telephone (as shown in the exchange with my mother), which exceeds IM, which exceeds e-mail. In other words, e-mail is incredibly lame as a communication vehicle, so naturally it's the most popular.

Actually, people use e-mail because it's convenient, asynchronous, and works across time zones. Unfortunately, because e-mail has such little bandwidth, the communication is poor and often several round trips are needed to gain comprehension and answers. Due to time zone differences, a round trip often takes a day; thus e-mail communication proceeds at a glacial pace. Bandwidth between computers can also be quite slow, which means quick source control, file, or database operations in Redmond may take hours overseas.

How do you beat the big, bad bandwidth issue? There are only two ways—increase the bandwidth or cut down on the necessary communication.

§ You can increase the bandwidth by using higher bandwidth communication tools, like VTC and Live Meeting. These work even over low network bandwidth lines. However, the tools are synchronous, so you must reserve overlapping work time for communication with distributed team members. That means if you're in Redmond and other team members are in China, you should reserve 4 – 5 pm Pacific Time everyday for scheduled and impromptu meetings with your peers in China.

§ You can cut down on necessary communication by crafting far more careful e-mails. Mitigate questions and confusion by providing added information and answering common questions in advance. It will take you a bit longer to write e-mail this way, but it won't take you days, which is how long the communication will take otherwise.

§ You can cut out the need for a whole class of e-mails and phone calls by keeping a SharePoint wiki with project information. Fill it with how-to topics, discussion archives, checklists, documents, and project mail, schedule, and status. Any time a new team member arrives at any location, assign them to update the site with improved instructions or missing documentation.

§ You can also cut down on necessary communication by creating clear project boundaries between the work that is happening at different locations. Isolate local communication, including caching of source control and databases. Then you need cross-group communication only when people or information cross boundaries. That brings me to the next big, bad B.

Doesn't help to know that you're just time away

The second big, bad B is boundaries—boundaries between work at different project locations.

If you've never worked on distributed projects or teams before, you might foolishly think that distributed groups are all just one happy team. It's this kind of naïve thinking that causes managers to allow individual team members to work alone from remote locations. Remember, being ignorant, naïve, and foolish is just one step from being stupid.

A project team residing in three separate sites is not one happy team; it's three potentially happy teams working on one potentially successful project. By team I mean a group of individuals with a common understanding working in unison toward a common goal.

Due to the bandwidth issues I described earlier, you cannot hold and maintain a common understanding and work in unison unless you are likely to pass by your teammates on the way to the restroom. That's why shared collaborative spaces are the best. That's also why teams split across floors experience many of the same problems as teams split across continents.

To run a successful distributed project, you must create clear boundaries between the distributed teams that provide enough isolation to allow them to work independently with only minimal coordination; the clearer the boundaries the better the result. I talk about this more in my column, "Blessed isolation."

Typically, boundaries are architectural that isolate components, but they could be boundaries between versions or responsibilities. You could also have distributed teams focus on local market scenarios. Treat each distributed team like a dependency or a vendor and you'll be on the right track. This doesn't add unnecessary overhead; that communication overhead is there regardless. Instead, it reduces unnecessary conflicts, catastrophes, and clean-up.

But if your teams are isolated, how do you keep that sense of unity and sharing as you drive toward common goals? That's where the last big, bad B comes into play.

It would be so fine to see your face at my door

The third big, bad B is being there—being there to create the human bonds necessary to achieve real cooperation and connection.

Making the best use of limited bandwidth and clear boundaries will mitigate much of the difficulties with distributed development. However, you are still one group of people working on the same project with the same goals. There are important relationships you must maintain between teams, peers, and reporting structures that require a human face. Being there is important.

VTC and other live presence tools can help. You should use them regularly for one-on-ones and meetings, perhaps not every time but at least once per month. Remember to reserve the overlap time between locations for these functions.

Of course, nothing can replace actual human contact, so plan to have everyone visit with each other at least twice a year. Plan one visit in October for the company meeting and performance reviews, and plan a second visit in February or March for budgeting and fiscal year planning. Don't visit for a few days; visit for a week or two. Have morale events, one-on-ones, and training, as well as planning and review meetings. Make the most of your time together.

A number of teams do rotations or exchanges. In these cases, team members from one location spend three months, or even six months, with team members in another location. It can be a swap or an assignment. These rotations provide tremendous knowledge transfer while creating understanding, empathy, and connection across the groups.

One last thought… Sometimes giving the illusion of being there can be very effective at maintaining relationships and improving communication. Some companies have tried continuous video feeds of common areas. A cheaper and fun solution that's been effective at Microsoft involves big pictures and even life-sized cutouts of team members placed in high traffic areas. As people leave meetings or walk the halls thinking about the project, they see one of these cutouts and are reminded to include that person or team in the conversation.

Where are you when the sun goes down?

With a little effort, you can make distributed development work for your projects and teams. Soon the sun may literally never set on your team. The advantages for you and Microsoft are enormous.

The diversity of experience, culture, and ideas might initially cause discomfort for you and your group, but over time it will make you better, your group better, and your products and services better. Your work will be more relevant in more markets for more people, and you will learn and grow in the process. Take the time to beat the Bs and you, your group, and your work will become worldly wonders.