I. M. Wright’s “Hard Code” : Project Mismanagement

Right on schedule

ericbrec — Tue, 01 Sep 2009 09:00:00 GMT

My older son can now drive. This adds two new worries to my life—how ancient I feel and thoughts of my son in a ditch somewhere. To mitigate the second worry, my wife and I enforce a curfew and insist my son call if he's running late. The other night, he arrived home 20 minutes late without notice. My wife was furious that he was late. I was furious that he didn't call.

Why didn't my son call to say he was running late? Because, like my wife, he was focused on the schedule. He avoided facing conflict until he got home. He said, "I got home as fast as I could"—presumably breaking numerous traffic regulations along the way. My son completely missed the point. The purpose of the rules was to mitigate risk, yet his response to them was to drive recklessly.

Software engineers do this all the time. They come up with a development schedule, unexpected issues come up, and they end up being late. Instead of informing their managers of the delay, they avoid facing conflict, rush the work, sacrifice quality, and slip the schedule, all with little control or visibility. It's the opposite of what managers should want, yet those same managers insist on following the schedule precisely. Why? Because most managers and engineers don't distinguish between the two types of scheduling—meeting a commitment and managing risk.

Those who understand binary and those who don't

Yes, that's right. There are two types of scheduling and project management.

§ Meet a commitment. You made a commitment to customers or partners and you must meet it at the quality and time period promised. Period.

§ Manage risk. There is a mix of critical and desirable work. People can make bad choices. Issues can arise. You must manage risk to ensure critical work gets done.

These two approaches to scheduling and project management often get confused. Why?

§ They typically appear together. The overall project has commitments, but it is made up of smaller tasks that require risk management.

§ They both use dates. The difference is that dates for commitments are untouchable and drive everything. Dates for risk management are simply checkpoints to make sure work stays on track.

§ They both are called scheduling. Most people simply don't know the difference.

§ Meeting commitments is the only type most people are taught. From the time kids enter school, they are exposed to inflexible due dates and commitments they must meet. When they later learn project management, it is all about Gantt charts and milestones, with some risk management thrown in as an aside.

§ Managing risk is usually self-taught and informal. A small number of people are formally taught risk management. Most of us learn it from peers in college as we juggle large workloads. Instead of completing everything on-time, we form workgroups, focus on the critical work, and minimize the damage to our grades.

This tragic lack of understanding leads to horrible decisions, poor engineering, and scheduling disasters. You need to know the difference and apply the right scheduling to the right problems. Let's start with meeting commitments.

That's the only thing you're committed to

Meeting commitments is essential to working with partners, which in turn is essential to running most businesses. You can't coordinate work across internal dependencies or external agreements without synchronizing on dates and deliverables. Because commitments cascade, you must meet them or face catastrophe.

Say your daughter's birthday is coming and you've promised her a new game she wants. How would you feel if it wasn't delivered to you when promised? There is a chain of handoffs between the developer, the manufacturer, the seller, and you that all must be met to ensure your daughter's happiness on her birthday.

Of course, this is much easier to do for Web-based products, but the same problems come up with handoffs between teams or departments. Delivering on commitments builds trust and lasting relationships between partners. Missing commitments does the opposite. While many software projects require little or no coordination across groups, running large and mature projects typically involves meeting commitments, and thus formal project management techniques.

Eric Aside

To help them make their commitments, companies typically use inventory, buffers, and other forms of risk management for individual steps in the process. That's the whole point. You use formal project management for meeting your high-level commitments and risk management for properly completing the individual steps in your process.

Don't you think it's a little risky?

Risk management is about making sure the critical work gets done properly, even in highly variable environments. Scrum and Feature Crews are excellent examples of software development practices that focus on risk management—in particular, the risk that you won't efficiently ship value and quality to the customer.

As I pointed out in my very first column, Dev schedules, flying pigs, and other fantasies, dev schedules and test schedules are in the risk management category. All the feature dates are checkpoints to mitigate risk. Only the small number of cross-group synchronization points (major milestones) are commitments.

What matter in risk management are focus, order, and status—not precision. Focus on what's important, do those items in priority order, and track the status as the situation changes. Notice that hitting precise dates for tasks isn't important, so long as you finish the critical tasks at the expected quality on time. Everything else can be cut.

That's why you must tell your engineers the same thing I told my son, "Coming home right on time isn't important. Telling us you aren't coming home on time is." You can only manage the risks if you know about them. The dates are there only to alert you when plans need to change.

Of course, you can't slip a critical task past a commitment date (a major milestone), and my son can't stay out past 1:00 A.M. or his license will be suspended. But curfew is well before then, as it should be, to avoid any chance of catastrophe.

Eric Aside

There are many popular risk management techniques. Here are a few handy ones for software development (many taken from prior columns):

· Hold daily stand-up meetings—15 minute meetings to talk about progress, future work, and blocking issues (called Scrums in Scrum).

· Assign a backup for all task assignments (partner less experienced folks with more experienced folks).

· Use buffers—set aside time for task fluctuations (personally, I never liked this practice; I'd rather have a well-prioritized list with no intention of completing everything).

· Under promise and over deliver—also known as setting appropriate expectations.

· Establish a fallback plan—have a plan in mind in case a risky task fails, like cutting back the feature or going back to the previous version.

· Balance risk—keep the overall risk of your project at a constant state of "scary but not terrifying" by adding and removing risk as things change. For example, if a team member has a parent fall ill, your risk has increased so you should cut a risky feature or reassign that tough task you gave a junior engineer.

You pick the one right tool

So before you start working on a schedule, stop and think about what's on it. Is it a set of dates and deliverables you've committed to a partner? Or is it a set of tasks with various priorities that you need to track and avoid messing up?

In the case of my son coming home by curfew, it was a task we didn't want him to wreck, literally. Thus, we were doing risk management and the focus needed to be on giving timely status as opposed to coming home at a precise time. Most software development tasks fit that mold. You just want your engineers to tell you promptly if they are running late so you can adjust. Precision is unnecessary and potentially counter-productive.

However, if you are running a large project with multiple teams and partnerships, then commitments and synchronization are critical at a high level. The high-level schedule is full of milestones and classic project management tools.

Just don't confuse the high-level schedule with the low-level tasks. If you treat the low-level tasks like your high-level commitments, your engineers will take shortcuts and drive too fast. Instead of managing risk, you might cause them to crash one of your critical tasks, which in turn breaks your high-level commitments. Use the right tool for the right level. You'll sleep better at night.

It starts with shipping

ericbrec — Fri, 01 May 2009 09:00:00 GMT

Call me "old school" but I believe in shipping. Trying isn't enough. Getting close isn't enough. Good ideas aren't enough. You've got to ship.

It used to be that interviews started with, "What have you shipped?" If you hadn't shipped recently, "Why?" Why? Because you can't deliver customer value if you don't deliver. You can't iterate and improve without finishing an iteration. You can't get customer feedback without customers.

People used to complain that promotions and rewards were disproportionally distributed to those who shipped. I say, "Absolutely, that's how it should be." Does this hurt quality? No, you set a high minimum quality bar and ship. Does it hurt innovation? No, innovators have always risked an initial drop in pay to receive a big payoff should they deliver.

Eric Aside

Some people complain that the big payoff doesn't exist at Microsoft for innovative ideas. Those people haven't shipped. The people who successfully ship innovative ideas are the ones who become our organizational and technical leaders.

It all starts with shipping. This is particularly apt with services, where everything literally starts with shipping, and where I'm focusing the rest of this column. Our critics claim that in the new world of services Microsoft has forgotten how to ship. Perhaps, but Microsoft has forgotten more about shipping than most companies will ever know. We just need some reminders and reeducation, especially when it comes to services.

Eric Aside

Does a focus on shipping drive death marches? No, death marches delay shipping. As I wrote in “Marching to death” (chapter 1), death marches result from a lack of planning and courage. This is particularly important to understand in the services world where sustainable shipping is critical to long term success.

I offer you my service

How much about shipping services has Microsoft forgotten or doesn't get, according to critics? Not as much as they would have you believe, but enough to make you think. Let's go over the herrings and the heartaches, mixed with a little happiness.

The red herrings:

§ Services make you think about everything differently.

§ Services center on data while packaged products center on functionality.

§ Services have greater security concerns than packaged products.

§ Services have serious issues with dependencies.

§ Services demand higher quality and faster iterations than packaged products.

The heartaches (and happiness):

§ Services run across hundreds of machines, not on a single client.

§ Services must scale out automatically.

§ Services are easier to switch than packaged products.

§ Service upgrades hit everyone instantly.

§ Services are living, changing things.

Let's break these down, starting with the red herrings.

What is that smell?

The first services red herring is a big one, "Services change everything." As I addressed in At your service, this is total bovine fertilizer. Services start and end with helping customers achieve their goals, just like all products ever. You focus on the customer experience and what they hope to accomplish or you lose. End of story.

The next three red herrings—centering on data, security concerns, and dependency issues—all apply just as well to shipping packaged products, though it may have taken us longer to realize it. You can't expose data format changes to customers without chasing them away, on the client or the server. There isn't a computer product or service today that isn't vulnerable to attack—you must secure them all. Finally, if you think external dependencies aren't problematic on the client, you clearly don't use many drivers. I'm not saying these aren't real issues—I'm saying they aren't new or specific to services.

The last red herring is among the most common concerns raised about why shipping services differs from shipping packaged products—high availability and Internet time. Look, it's not okay for packaged products to never work or require a reboot every time you use them; at least it hasn't been for quite some time. The quality bar is no different for services, though there are plenty of services that fail constantly.

As for Internet time, that hit packaged products a decade ago with the introduction of Windows Update. And if you think that those patches are just security fixes, you haven't been paying attention. More and more we are fixing all kinds of experience issues shortly after customers report them, for services and packaged products. That's a great thing for customers.

However, gradually improving the customer experience every month or every day isn't enough. Both services and packaged products need to ship significant, orchestrated updates to deliver breakthrough customer value. Facebook wasn't going to gradually update itself into Twitter any more than Vista would gradually update itself into Windows 7. You must focus on what the customer is trying to accomplish, and sometimes that isn't a quick change.

Eric Aside

The best way to learn how to ship is to do it early and often. Make every build a shippable build. Build every day and rebuild the entire system at least every week. Deploy regular tech previews and betas. Deploy regular incremental updates and fixes into production. Ship early, ship often. Practice makes perfect.

There are too many of them

However, not everything about shipping packaged products applies to shipping services. There are mental, process, and team adjustments that you need to make.

First and foremost is that services run across hundreds or thousands of machines dispersed in multiple data centers worldwide. Sometimes functionality and data are replicated. Sometimes functionality and data are specialized. Usually, it's a combination of both for scale and reliability. Naturally, this presents design and synchronization problems but plenty of books have been written about that (read don't rediscover). The less obvious challenges are around debugging and deployment.

Why is debugging a service so tough? Timing issues are killer given multiple threads on multiple processors across multiple machines. Yikes! However, that's not even the toughest challenge.

What's the first thing you do when debugging an issue? Analyze the stack, right? With services the stack is split across servers and requests, making it nearly impossible to trace a specific user action. The good news is that there are new tools that help tie user actions together across machines. The bad news is that this isn't the toughest challenge either. The toughest challenge is that you're always debugging in the live environment. You don't get symbols, breakpoints, or the ability to step through code.

So let's recap. Debugging services means debugging nasty timing issues across multiple machines with no stack, symbols, or breakpoints on live code. There's only one solution—instrumentation—and lots of it, designed in from the beginning, knowing you'll soon be debugging across live machines with no stack, symbols, or breakpoints.

They're multiplying too rapidly!

Solving debugging brings us to the other huge challenge—deployment. Deployment needs to be completely automated and lightning fast. We're talking file copy installation, with fast file copy. No registry, no custom actions, and no manual anything.

Why does deployment need to be so fast and simple? Two reasons:

§ You're installing onto hundreds or thousands of machines worldwide while they are live. Installation must work and work fast with zero human intervention ever. The slightest bit of complexity will cause failures. Remember, five minutes times 1,000 machines equals three-and-a-half days. It had better just work.

§ The number of servers needs to grow and shrink dynamically based on load. Otherwise, you are wasting hardware, power, cooling, and bandwidth in order to meet the highest demand. Because your scale depends on load, it can change any time. When it changes, you need to build out more systems automatically and instantly.

The happiness around deployment is that Azure will do most of the heavy lifting for you (so let it, don't reinvent). However, you still need to design your services to support file copy installation.

Life is so uncertain

Enough of the challenges you can predict, how about the unpredictable ones? The services landscape is in constant change. While some services are sticky because they hold your data (like Facebook or eBay), many aren't sticky at all (like search or news). A few minutes of downtime can cost you thousands of customers. Data compromise or loss can cost you millions of customers. They'll just switch. Our competitors will be happy to accept them. It cuts both ways so you need work hard to both welcome and keep new users.

When you update a service everyone gets the new version instantly, not over years. If there's a bug that only one customer in a thousand experiences, then that bug will hit thousands of customers instantly (law of truly large numbers). That means you need to resolve the issue quickly or rollback. Either way, it's a bad idea to update a service on a Friday and a good idea to have an emergency rollback button always at the ready.

Finally, it's important to realize that services are living, changing things. You'd think that because the servers are all yours with your image and your configuration that it would be a controlled environment—and it is until you turn on the switch. Once the server goes live, it changes. The memory usage changes, the data and layout on the disks change, the network traffic changes, and the load on the system changes. Services are like rivers not rocks. You can't ship and forget services. They need constant attention. To make your life easier, bake resilience in by automating the five Rs—retry, restart, reboot, reimage, and replace (though replace may require human hands at some point).

The happiness that comes with these heartaches are customers willing to switch; an ideal idea testing platform because you can show customers different ideas and see which they prefer on a daily basis; and the ability to ship now and find the tricky intermittent Heisenbugs later (using your five Rs resilience to keep up availability).

Back to basics

There you have it. Some food for thought mixed in with the old basics of writing solid code that focuses on customers and their goals.

However, none of this is worth anything without shipping. Make shipping a priority and we all win. Sure, the quality bar has gone up, but we're not kids selling lemonade anymore. We need to ship quality experiences regularly, on both long and short time scales. We need to ship on the Internet, on the PC, and on the phone. We need to serve our customers well and delight them into sticking with us. It's a long journey, but it doesn't start until we ship.

I would estimate

ericbrec — Mon, 01 Sep 2008 09:00:00 GMT

When I'm discussing challenges with fellow engineers, the first topic that comes up isn't estimation—it's career and people challenges. That's why those issues are so rampant in these rants. However, "How do you generate task estimates?" is always among...(read more)