I. M. Wright’s “Hard Code” : Process

Right on schedule

ericbrec — Tue, 01 Sep 2009 09:00:00 GMT

My older son can now drive. This adds two new worries to my life—how ancient I feel and thoughts of my son in a ditch somewhere. To mitigate the second worry, my wife and I enforce a curfew and insist my son call if he's running late. The other night, he arrived home 20 minutes late without notice. My wife was furious that he was late. I was furious that he didn't call.

Why didn't my son call to say he was running late? Because, like my wife, he was focused on the schedule. He avoided facing conflict until he got home. He said, "I got home as fast as I could"—presumably breaking numerous traffic regulations along the way. My son completely missed the point. The purpose of the rules was to mitigate risk, yet his response to them was to drive recklessly.

Software engineers do this all the time. They come up with a development schedule, unexpected issues come up, and they end up being late. Instead of informing their managers of the delay, they avoid facing conflict, rush the work, sacrifice quality, and slip the schedule, all with little control or visibility. It's the opposite of what managers should want, yet those same managers insist on following the schedule precisely. Why? Because most managers and engineers don't distinguish between the two types of scheduling—meeting a commitment and managing risk.

Those who understand binary and those who don't

Yes, that's right. There are two types of scheduling and project management.

§ Meet a commitment. You made a commitment to customers or partners and you must meet it at the quality and time period promised. Period.

§ Manage risk. There is a mix of critical and desirable work. People can make bad choices. Issues can arise. You must manage risk to ensure critical work gets done.

These two approaches to scheduling and project management often get confused. Why?

§ They typically appear together. The overall project has commitments, but it is made up of smaller tasks that require risk management.

§ They both use dates. The difference is that dates for commitments are untouchable and drive everything. Dates for risk management are simply checkpoints to make sure work stays on track.

§ They both are called scheduling. Most people simply don't know the difference.

§ Meeting commitments is the only type most people are taught. From the time kids enter school, they are exposed to inflexible due dates and commitments they must meet. When they later learn project management, it is all about Gantt charts and milestones, with some risk management thrown in as an aside.

§ Managing risk is usually self-taught and informal. A small number of people are formally taught risk management. Most of us learn it from peers in college as we juggle large workloads. Instead of completing everything on-time, we form workgroups, focus on the critical work, and minimize the damage to our grades.

This tragic lack of understanding leads to horrible decisions, poor engineering, and scheduling disasters. You need to know the difference and apply the right scheduling to the right problems. Let's start with meeting commitments.

That's the only thing you're committed to

Meeting commitments is essential to working with partners, which in turn is essential to running most businesses. You can't coordinate work across internal dependencies or external agreements without synchronizing on dates and deliverables. Because commitments cascade, you must meet them or face catastrophe.

Say your daughter's birthday is coming and you've promised her a new game she wants. How would you feel if it wasn't delivered to you when promised? There is a chain of handoffs between the developer, the manufacturer, the seller, and you that all must be met to ensure your daughter's happiness on her birthday.

Of course, this is much easier to do for Web-based products, but the same problems come up with handoffs between teams or departments. Delivering on commitments builds trust and lasting relationships between partners. Missing commitments does the opposite. While many software projects require little or no coordination across groups, running large and mature projects typically involves meeting commitments, and thus formal project management techniques.

Eric Aside

To help them make their commitments, companies typically use inventory, buffers, and other forms of risk management for individual steps in the process. That's the whole point. You use formal project management for meeting your high-level commitments and risk management for properly completing the individual steps in your process.

Don't you think it's a little risky?

Risk management is about making sure the critical work gets done properly, even in highly variable environments. Scrum and Feature Crews are excellent examples of software development practices that focus on risk management—in particular, the risk that you won't efficiently ship value and quality to the customer.

As I pointed out in my very first column, Dev schedules, flying pigs, and other fantasies, dev schedules and test schedules are in the risk management category. All the feature dates are checkpoints to mitigate risk. Only the small number of cross-group synchronization points (major milestones) are commitments.

What matter in risk management are focus, order, and status—not precision. Focus on what's important, do those items in priority order, and track the status as the situation changes. Notice that hitting precise dates for tasks isn't important, so long as you finish the critical tasks at the expected quality on time. Everything else can be cut.

That's why you must tell your engineers the same thing I told my son, "Coming home right on time isn't important. Telling us you aren't coming home on time is." You can only manage the risks if you know about them. The dates are there only to alert you when plans need to change.

Of course, you can't slip a critical task past a commitment date (a major milestone), and my son can't stay out past 1:00 A.M. or his license will be suspended. But curfew is well before then, as it should be, to avoid any chance of catastrophe.

Eric Aside

There are many popular risk management techniques. Here are a few handy ones for software development (many taken from prior columns):

· Hold daily stand-up meetings—15 minute meetings to talk about progress, future work, and blocking issues (called Scrums in Scrum).

· Assign a backup for all task assignments (partner less experienced folks with more experienced folks).

· Use buffers—set aside time for task fluctuations (personally, I never liked this practice; I'd rather have a well-prioritized list with no intention of completing everything).

· Under promise and over deliver—also known as setting appropriate expectations.

· Establish a fallback plan—have a plan in mind in case a risky task fails, like cutting back the feature or going back to the previous version.

· Balance risk—keep the overall risk of your project at a constant state of "scary but not terrifying" by adding and removing risk as things change. For example, if a team member has a parent fall ill, your risk has increased so you should cut a risky feature or reassign that tough task you gave a junior engineer.

You pick the one right tool

So before you start working on a schedule, stop and think about what's on it. Is it a set of dates and deliverables you've committed to a partner? Or is it a set of tasks with various priorities that you need to track and avoid messing up?

In the case of my son coming home by curfew, it was a task we didn't want him to wreck, literally. Thus, we were doing risk management and the focus needed to be on giving timely status as opposed to coming home at a precise time. Most software development tasks fit that mold. You just want your engineers to tell you promptly if they are running late so you can adjust. Precision is unnecessary and potentially counter-productive.

However, if you are running a large project with multiple teams and partnerships, then commitments and synchronization are critical at a high level. The high-level schedule is full of milestones and classic project management tools.

Just don't confuse the high-level schedule with the low-level tasks. If you treat the low-level tasks like your high-level commitments, your engineers will take shortcuts and drive too fast. Instead of managing risk, you might cause them to crash one of your critical tasks, which in turn breaks your high-level commitments. Use the right tool for the right level. You'll sleep better at night.

De-optimization

ericbrec — Mon, 01 Dec 2008 10:00:00 GMT

Why? Why! Why do managers make stupid decisions that cause devastating churn and tawdry results? And it's not just managers, though they are particularly proficient at promoting poor performance—architects, leads, and individual contributors flood the...(read more)

Nailing the nominals

ericbrec — Wed, 01 Oct 2008 09:00:00 GMT

People are always looking for that amazing breakthrough technology or process that solves all their problems—enhances their love life, trims their waist, and improves the productivity of their development team. That's why process manias like Agile and...(read more)

At your service

ericbrec — Tue, 01 Apr 2008 09:03:00 GMT

Remember this one, "The microprocessor changes everything!" No, it didn't. Yes, it had a big impact, but people still fretted about the same problems and tried to accomplish the same things. They just created problems and accomplished things more efficiently....(read more)

Things have got to change: Change management

ericbrec — Sat, 01 Mar 2008 13:13:00 GMT

It's the political season in the United States, making "change" a happy word around here. Politicians fight over who better represents change. They proclaim themselves to be agents of change. Hysterical admirers jump up and down waving "Change" signs....(read more)

So far away: Distributed development

ericbrec — Fri, 01 Feb 2008 10:00:00 GMT

If you are a software geek, like me, being the product support technician for your friends and family comes with the territory. While it's painful to watch your family struggle with software, particularly if you helped write it, at least you can tell them, "Back off, I'm a computer scientist," and repair whatever is wrong. Sure, you'll cringe as you undo their failed "fixes," but in time you'll set things straight. That is, if you live nearby.

If your mom lives a thousand miles away, the call might sound more like this:

Mom: Honey, they changed my password and now my e-mail doesn't work.

Me: Okay, log onto your computer and open Outlook.

Mom: Using what password?

Me: Use whatever password you normally use.

Mom: Not the new e-mail one, just my old one.

Me: Yes. Then open Outlook.

Mom: Where do I type in the password?

Me: On the main screen where you click on your name and then type a password.

Mom: What main screen?

Me: Wait, can you just open Outlook right now?

Mom: Yes, I've got it open, but you told me to log on.

[Another hour like this moving through menus, dialog boxes, and buttons…]

This experience is far better if you can get a tool like Remote Assistance to work, but getting that up and running behind firewalls and routers can be equally entertaining. What should be a five-minute fix becomes an hour of acute aggravation. This order of magnitude multiplier for time and trouble comes into play any time you work across time and distance, which brings us to the topic of distributed development.

Doesn't anybody stay in one place anymore?

It's becoming far more common to run development across global locations for the same project. While the added diversity and talent should be a huge advantage, the results are often frustration, delays, and disconnects in quality and functionality. Why? It's due to the big, bad Bs—bandwidth, boundaries, and being there.

There's insufficient bandwidth for clear communication and fast network access to central services. The boundaries between project work at the different locations are poorly defined causing additional communication, conflicts, calamities, and clean-up. And because the different teams are separated, the one-on-ones, drop-ins, hallway conversations, and other daily human interactions you get from being there don't occur.

With all this trouble you might wonder why we bother with distributed development. No, you fool, it's not the money. Engineering salaries and costs are converging for many roles in China and India, and distributed development adds overhead. The real reasons for distributed development stem from the talent and the markets. The computer science talent pool in Brazil, Russia, India, and China is growing at nearly 20% per year and will have 1.5 times the number of software engineers in all of the United States by 2010. They can't all move to Redmond, and we wouldn't want them to because their home markets are critical to our success.

So if you're running a team, you'd better learn how to deal with the big, bad Bs. Let's break them down.

I get so tired when I have to explain

The first big, bad B is bandwidth—bandwidth between people and between computers.

Clear communication between people and teams is critical to success, as I've said many times before. The amount of information that is communicated between two people in the same room far exceeds that between people over video teleconference (VTC), which in turn exceeds Live Meeting, which exceeds the telephone (as shown in the exchange with my mother), which exceeds IM, which exceeds e-mail. In other words, e-mail is incredibly lame as a communication vehicle, so naturally it's the most popular.

Actually, people use e-mail because it's convenient, asynchronous, and works across time zones. Unfortunately, because e-mail has such little bandwidth, the communication is poor and often several round trips are needed to gain comprehension and answers. Due to time zone differences, a round trip often takes a day; thus e-mail communication proceeds at a glacial pace. Bandwidth between computers can also be quite slow, which means quick source control, file, or database operations in Redmond may take hours overseas.

How do you beat the big, bad bandwidth issue? There are only two ways—increase the bandwidth or cut down on the necessary communication.

§ You can increase the bandwidth by using higher bandwidth communication tools, like VTC and Live Meeting. These work even over low network bandwidth lines. However, the tools are synchronous, so you must reserve overlapping work time for communication with distributed team members. That means if you're in Redmond and other team members are in China, you should reserve 4 – 5 pm Pacific Time everyday for scheduled and impromptu meetings with your peers in China.

§ You can cut down on necessary communication by crafting far more careful e-mails. Mitigate questions and confusion by providing added information and answering common questions in advance. It will take you a bit longer to write e-mail this way, but it won't take you days, which is how long the communication will take otherwise.

§ You can cut out the need for a whole class of e-mails and phone calls by keeping a SharePoint wiki with project information. Fill it with how-to topics, discussion archives, checklists, documents, and project mail, schedule, and status. Any time a new team member arrives at any location, assign them to update the site with improved instructions or missing documentation.

§ You can also cut down on necessary communication by creating clear project boundaries between the work that is happening at different locations. Isolate local communication, including caching of source control and databases. Then you need cross-group communication only when people or information cross boundaries. That brings me to the next big, bad B.

Doesn't help to know that you're just time away

The second big, bad B is boundaries—boundaries between work at different project locations.

If you've never worked on distributed projects or teams before, you might foolishly think that distributed groups are all just one happy team. It's this kind of naïve thinking that causes managers to allow individual team members to work alone from remote locations. Remember, being ignorant, naïve, and foolish is just one step from being stupid.

A project team residing in three separate sites is not one happy team; it's three potentially happy teams working on one potentially successful project. By team I mean a group of individuals with a common understanding working in unison toward a common goal.

Due to the bandwidth issues I described earlier, you cannot hold and maintain a common understanding and work in unison unless you are likely to pass by your teammates on the way to the restroom. That's why shared collaborative spaces are the best. That's also why teams split across floors experience many of the same problems as teams split across continents.

To run a successful distributed project, you must create clear boundaries between the distributed teams that provide enough isolation to allow them to work independently with only minimal coordination; the clearer the boundaries the better the result. I talk about this more in my column, "Blessed isolation."

Typically, boundaries are architectural that isolate components, but they could be boundaries between versions or responsibilities. You could also have distributed teams focus on local market scenarios. Treat each distributed team like a dependency or a vendor and you'll be on the right track. This doesn't add unnecessary overhead; that communication overhead is there regardless. Instead, it reduces unnecessary conflicts, catastrophes, and clean-up.

But if your teams are isolated, how do you keep that sense of unity and sharing as you drive toward common goals? That's where the last big, bad B comes into play.

It would be so fine to see your face at my door

The third big, bad B is being there—being there to create the human bonds necessary to achieve real cooperation and connection.

Making the best use of limited bandwidth and clear boundaries will mitigate much of the difficulties with distributed development. However, you are still one group of people working on the same project with the same goals. There are important relationships you must maintain between teams, peers, and reporting structures that require a human face. Being there is important.

VTC and other live presence tools can help. You should use them regularly for one-on-ones and meetings, perhaps not every time but at least once per month. Remember to reserve the overlap time between locations for these functions.

Of course, nothing can replace actual human contact, so plan to have everyone visit with each other at least twice a year. Plan one visit in October for the company meeting and performance reviews, and plan a second visit in February or March for budgeting and fiscal year planning. Don't visit for a few days; visit for a week or two. Have morale events, one-on-ones, and training, as well as planning and review meetings. Make the most of your time together.

A number of teams do rotations or exchanges. In these cases, team members from one location spend three months, or even six months, with team members in another location. It can be a swap or an assignment. These rotations provide tremendous knowledge transfer while creating understanding, empathy, and connection across the groups.

One last thought… Sometimes giving the illusion of being there can be very effective at maintaining relationships and improving communication. Some companies have tried continuous video feeds of common areas. A cheaper and fun solution that's been effective at Microsoft involves big pictures and even life-sized cutouts of team members placed in high traffic areas. As people leave meetings or walk the halls thinking about the project, they see one of these cutouts and are reminded to include that person or team in the conversation.

Where are you when the sun goes down?

With a little effort, you can make distributed development work for your projects and teams. Soon the sun may literally never set on your team. The advantages for you and Microsoft are enormous.

The diversity of experience, culture, and ideas might initially cause discomfort for you and your group, but over time it will make you better, your group better, and your products and services better. Your work will be more relevant in more markets for more people, and you will learn and grow in the process. Take the time to beat the Bs and you, your group, and your work will become worldly wonders.

Software performance: What are you waiting for?

ericbrec — Thu, 01 Nov 2007 10:25:00 GMT

You hurt your shoulder playing volleyball, so you make an appointment to see your doctor. You enter the office and wait in line for five minutes just to let the receptionist know you've arrived. He has you verify your contact and insurance information, which haven't changed in ages, and then tells you to sit in the waiting room.

You sit in the waiting room for ten minutes, inhaling all kinds of ailments from the crowd, seething about how you're going to leave sicker than you were when you came in, till a nurse shows you to an exam room.

After five minutes in the exam room, another nurse comes in, takes your vital signs, and has you repeat the reason you gave for the appointment when you originally made it. Ten minutes later, your doctor arrives and actually addresses your shoulder injury.

Welcome to a user's experience running our software. You wait forever just for it to launch. You provide your credentials again, even though you gave them when you logged in. You wait again for your personalized environment to load. You click a few menu items or buttons to launch the specific functionality you want. Finally, you wait again while the feature prepares to do what you actually launched the software to do in the first place. That's assuming there aren't network delays.

Just one moment, please

Waiting is dull. Waiting is frustrating. Waiting is agonizing. Waiting is just an unbelievably bad time all around, on every level. Nobody likes to wait. Nobody asks to wait. So, why the heck do we make people wait?

Actually, why do our customers even put up with it? Why do I put up with it at my doctor's office? I guess because all doctors' offices are slow. But, if one of my friends told me about a quick doctor's office that provided comparable care, I'd switch in a heartbeat.

This means a competitor's quicker program could flatline our business. How do we make our programs quick, before our competitors do? I'm glad you asked.

Eric Aside

If your doctor is good enough, you may put up with significant hardship instead of switching. However, giving people a reason to switch is inexcusable, especially when it's easy to do better.

You're faster than this

So, you want better performance from your software. Where do you start? "I know, I know!" says our resident performance pariah, Mr. Speedy. "Profile your code, find out where it's spending all its time, and then optimize, perhaps even parallelize, those inner loops."

Well, Mr. Speedy, aren't you clever. Let's profile our doctor's office, shall we? Whoa! It turns out the doctor is always busy, and that's the bottleneck. Who would have guessed? According to Mr. Speedy, all we need to do is speed up the doctor, find a faster doctor, or get two doctors to do the job of one. Right? Wrong!

There's nothing wrong with my algorithm—I mean, my doctor. If you made her faster, she wouldn't be any better. In fact, she'd probably be worse. Doing the job right takes time, and making all kinds of optimizations might improve things a little, but might also cause mistakes. I don't want a different doctor who happens to be faster, either. I like my doctor. I know her well, and she knows me.

I also don't want two doctors. Even if they are twins, I'll never know which thread—I mean, which doctor—I'll get. They might both try to treat me. They'll have to communicate with each other all the time to avoid mistakes. They might even get stuck waiting on each other. It's way more complicated, and it really doesn't solve the problem even if two doctors are twice as fast. I've still got to deal with the receptionist, waiting room, exam room, and nurses.

Eric Aside

"But what about multi-core processors?" you might ask. Look, there's using technology for the sake of technology, and there's using technology for a purpose. If the user experience demands threading across multiple processors, I'm all for it. If not, you're just giving yourself a cheap thrill at the expense of the customer.

Should I keep a copy?

"Hold on," says Mr. Speedy. "What you need is a cache–that always speeds things up." Hello! We've got cache fever in the doctor's office. That's part of what's slowing us down. We've got a reception line cache, a waiting room cache, and an exam room cache.

It seems like everyone at the doctor's office is concerned about speeding up their own work, so they all created their own caches. The receptionist created a cache, the nurses created a cache, and the doctor created a cache. The result is that patients spend all their time waiting and moving between caches instead of being processed by the doctor.

Think this doesn't happen in code? You've obviously never looked inside those database, shell, and system calls you use. All that data you're caching for "performance" is already being cached for the same reason by those functions. Sometimes there are as many caches as there are layers. Every cache has a fetch and memory cost to it. Well, I'm cached out.

You're not being the ball

Let's start over, shall we? Instead of speeding up the existing doctor's office, as marginally effective as that might be, let's think about things from the patient's perspective. What would you and I, as patients, like the experience to be?

Here's what I'd like. Check my contact info and insurance when I call in for an appointment. Write down my symptoms and include them in the appointment. Heck, let me do it all online (wait, that's crazy talk)!

When I show up at the doctor's office, I can walk right to my exam room (just one level of caching). It's the room with my name above it, just like at the good rental car agencies. A big sign in the room says, "Please take off your shirt and have positive identification ready for the nurse, then hit the big 'I'm ready' button."

The nurse, seeing that I've hit the "I'm ready" button, comes in, checks my ID, takes my vitals, and hits the 'Vitals taken' button, which adds me to my doctor's queue. As soon as my doctor's available, she comes in and addresses my needs. That would be great! Heck, there could even be a monitor in the exam room with queue stats and predicted wait times. The stats could be used to fine tune the number of appointments available per hour to minimize wait times, while still ensuring that each doctor is fully utilized.

Eric Aside

If you haven't read about the Theory of Constraints or its drum-buffer-rope approach to optimizing results, you are in for such a treat. They should be required reading for anyone trying to rethink and revolutionize performance of everything from software to cafeterias.

Have you ever been experienced?

There, that wasn't so hard. Setting up a doctor's office the way I described would be easy and not that expensive. It doesn't require more doctors or faster doctors, and it actually saves floor space. Sure, the online appointments and predicted wait time monitors would require special software, but those aren't essential to get a better and faster experience. What is essential is to think through the customer's experience with a view toward minimizing wait time.

Here are some questions for your consideration:

§ When was the last time your team thought through the end-to-end customer experience, including the wait time?

§ How would the customer want to deal with the inevitable constraints that every process has, besides giving them a CANCEL button? (Associating our software and services with "cancel" seems unwise.)

§ How could you minimize the impact of errors, network delays, and device I/O in a way that customers would find natural and unobtrusive?

§ What measures and statistics could you use to fine tune the experience, minimizing wait times while getting full utilization of key resources?

Right now we design experiences as if these performance constraints don't exist. Everything is modal and synchronous as if functions always return and people never select the wrong option. We design a feature at a time, instead of end-to-end; or if we do design end-to-end, we only think about the ideal scenarios, not the likely ones. We assume exceptions and delays are unusual, even exceptional. That's naïve, which is a kind way of saying "stupid."

Eric Aside

Of course, there are great examples in Microsoft software for thinking through the end-to-end experience quite nicely. I mention some in the next section.

You'll be ready

Performance tuning does have its place. There are functions and services that must scale up and out. There are issues with blocking and locking that require special care, which a real performance expert can help you resolve. It's just that those aren't the common case.

The common case is the ordinary case. A customer is trying to get something done. It involves network access and I/O. Those interactions could fail or cause delays. The customer generally has experienced these problems and knows they exist. The best way to handle them is to talk to customers and understand ideally what customers want to happen.

Perhaps the customer would be happy if the I/O completed asynchronously—the solution Outlook and OneNote use to vastly improve the customer experience. Perhaps the customer would be happy to work on a local copy and synchronize on demand—the solution ActiveSync and FrontPage use. Perhaps the customer is happy to queue their requests and get a status report later—the solution build systems and test harnesses use.

What about me?

The key is to look at the world from the customer's perspective and to design an experience that anticipates failure modes and minimizes their impact on users. Performance should be specified in the experience with specific measures and guidelines, not left to chance or hope. It typically doesn't require complex algorithms or fancy caching, both of which can be overdone. It requires being thoughtful and deliberate.

When performance is specified through the experience, it's built-in and tested from the start. No one gets surprised or has to scramble at the end of the project to suddenly become a performance expert. The only surprise is on the customer's face when what they thought would be an agonizing doctor visit turns out to be a delightful one.