I. M. Wright’s “Hard Code”

Permanently high plateau

ericbrec — Fri, 01 Jun 2012 06:00:00 GMT

A friend asked me recently about one of his reports. He had a few concerns going into annual review calibration. His employee was a smart, strong, consistent contributor, well beyond entry level and independence (see Level up for reference), but he had plateaued. My friend was concerned that his employee didn’t recognize he had plateaued, and that he might calibrate poorly against his peers who still have growth potential.

Managers and employees frequently fret about plateaus and growth potential—especially around review time. At what level is it safe to plateau, if any? Does a lack of growth potential mean a 4 or 5 review? Are people who’ve plateaued delusional if they don’t see it? Allow me to clear this up—growth potential is a fairy tale and plateaus are nothing more than places to regroup. Think I’m delirious? I think you’re clueless.

Eric Aside

In Microsoft’s current rating system for reviews, a 1 means an employee rates in the top 20 percent relative to peers in the same discipline and level band, a 2 indicates the next 20 percent, a 3 is the middle 45 percent, a 4 is the lower 10 percent, and a 5 indicates the employee is in the bottom 5 percent. Rewards are directly determined by an employee’s rating (with the very top 5 percent receiving extra). Reviews happen once a year during the summer, though many divisions have a midyear calibration to help determine promotions and recognize performance issues.

What... what seems to be the problem?

There are two different issues revolving around my friend and his employee: the nature of plateaus and the value of growth potential. Let’s take them one at a time.

Are plateaus permanent? Nope. Sure, everyone has limitations, and a man’s got to know his (just ask Hal Holbrook). However, put the right people in the right situations and they can surprise you. When people plateau, that simply means they need to regroup, consider their situation, and either accept the plateau because they love their work or pursue a different situation that better suits their gifts—even if that situation isn’t at Microsoft.
Is growth potential valuable? Nope. Growth potential is an intoxicating fantasy born of those who remind you of people you admire from long ago who became great (like yourself, I’m guessing). You know what is valuable? Actual growth. Actual results. We should hire people who show potential, but only reward them for realizing it.

Eric Aside

Microsoft’s review system used to place value on growth potential. Thankfully, since 2011 that is no longer the case. The review system is now solely based on demonstrated results over time (both what and how).

I can't go any further

What should my friend do about his employee who doesn’t realize he’s plateaued? Treat him like he hasn’t. Give the employee assignments appropriate for his continued growth. Either the employee will do well, proving the plateau was temporary, or the employee will fail to excel, sending both of you a clear message that you can discuss.

Mercurial managers might mention, “But what if the employee fails miserably? We can’t risk our business on someone who’s clearly not capable of reaching the next level!” First of all, cut the assumptions—they’re bad for you and your employees. Second, any high risk assignment should have a mitigation plan. If you don’t have one, stop being a manager—you’re incompetent (worse than plateauing).

They expect results

Will my friend’s employee calibrate poorly against his peers who still have growth potential? That depends—what did his employee accomplish this fiscal year? How does that compare to his perky peers?

People with stalled career growth can still achieve more than their current peers—much more at times. Their perky peers may be less experienced and make more mistakes, technically and interpersonally. We shouldn’t reward perky people for their poor judgment, poor behavior, and poor results. Ignorance is a reason, but not an excuse. Our current rating system values proven results, not postulated potential.

“But if we don’t pay them extra, how do we retain people who have growth potential?” Help them grow! Give them stretch assignments to challenge their abilities and accelerate their learning. Give them mentors to teach and guide them through all the twists and turns of real experience. Give them thoughtful, constructive, and actionable feedback that hones their skills. Treat them with the respect and dignity that rewards people for exceptional work, not hotness.

Are we there yet?

What if you are the employee who has plateaued? What’s your next move? That depends. What do you want to accomplish…with your life? What role best puts you on the path to achieving your goals?

Knowing what you truly wish to do with your life is critical to happiness and true success. The trick is knowing which “you” to follow—ego, heart, or spirit.

Following your ego means chasing your perception of success through others’ eyes. This path may lead to fame and fortune, but people can end up feeling isolated, lonely, and hollow.
Following your heart means pursuing what makes you happy at your core. This may not lead to western culture’s vision of success or even a steady paycheck, but it does lead to happiness.
Following your spirit means seeking a higher calling for your life and serving a greater purpose beyond yourself. This path yields powerful feelings of connection and meaning for your life. It can also lead to cults and sappy memoirs.

As with all things, you want a balance: ego to help you earn a living, heart to make you happy, and spirit to provide meaning in your life. Too much of any one of these can lead to trouble.

One thing I’ve learned after a long career is that when you need to make a choice, and the options are balanced with no clear favorite, go with your heart. Your ego is insatiable and your spirit is with you wherever you go and whatever you choose. Your heart knows what truly makes you happy.

Eric Aside

I wrote about being fulfilled at a plateau in “When the journey is the destination” (chapter 7) and how to best use your heart (gut) to help you make decisions in You have to make a decision.

It takes all kinds

What advice did I give my friend with the plateaued employee? Don’t assume the employee has plateaued. Give him assignments appropriate for his continued growth, mitigating in case he falls short (true for any employee assignment). Discuss his strong results at calibration—ensure he’s alongside those with similar proven results (not imagined). If the employee falters, take time with him to discuss his true aspirations as a balanced, complete human being. Help the employee choose a career path in line with his unique skills and personal vision.

We don’t give up—we evolve. We don’t reward potential—we shape it. Great people come together at Microsoft to change the world. We need every talent and every insight. Instead of fretting about plateaus, we should be helping each other realize our dreams. When we do, we turn those dreams into a reality for the world.

Hired helpers

ericbrec — Tue, 01 May 2012 06:00:00 GMT

There are never enough resources to complete our ambitious plans, so Microsoft is constantly hiring help—vendors and contingent staff (CSG). Full-time employees (FTEs) are hired too, but the relationship is different—at least it’s supposed to be.

Ever since the $97 million December 2000 settlement of the “permatemps” case, Microsoft has been very careful to treat vendors and CSG differently than FTEs. These days, processes and policies prevent permatemps. However, that hasn’t wiped out stupidity on this front. FTEs are still asked to do work with little growth potential that is far better assigned to vendors or CSG. Vendors and CSG are still asked to cover work that, if restructured, could yield far better results at a lower cost when accomplished by FTEs.

Why are FTEs doing vendor and CSG work? Why is work that is better accomplished by FTEs left to vendors and CSG? Two reasons: people are lazy and people are ignorant. Let’s expose the lazy and cure the ignorant.

Eric Aside

FTE doesn’t actually stand for full time employee; it means full time equivalent. Go figure. CSG stands for contingent staff g-something (group?). CSG is used at Microsoft to refer to individual temporary employees (“give that to the CSG”) and all contingent staff, although officially vendors are a form of contingent staff.

What am I supposed to do?

First and foremost, the ignorant must learn the difference between FTEs, vendors, and CSG and understand which assignments they should be given.

FTEs are employees managed and paid directly by Microsoft.
Vendors are companies contracted by Microsoft to do a specific statement of work (SOW). The vendor company selects which vendor employees are involved, with input from Microsoft, and the work can be ongoing (janitorial service) or fixed length (a marketing campaign).
CSG are temporary workers employed by an outside agency. They are contracted by Microsoft to do a specific assignment over a limited timeframe (365 days or fewer). Microsoft selects the individuals involved from resumes provided by the agency, and Microsoft oversees the individuals, providing feedback to the agency.

Here’s a set of rules to help you assign work. Treat these rules as strong suggestions—every situation is unique.

Primary assignment is super-secret. Use FTEs. At the very least, your VP should be involved in setting the policy of who works on what.
Primary assignment is ambiguous or open-ended. Use FTEs. Vendors and CSG should only be used for well-defined work.
Primary assignment does not provide career growth at Microsoft. Use vendors or CSG. Don’t assign FTEs to dead-end roles.
Primary assignment is ongoing. Use FTEs or vendors. CSG folks, also known as temps, should be used for temporary assignments.
Primary assignment is short and must be filled within days. Use CSG. FTEs and vendors take a minimum of a few weeks to hire.

There are many other policies and guidelines for hiring and using vendors and CSG, and those policies vary by region. This list above helps you decide what sort of people should be assigned to what work, particularly within the United States. Let’s consider some examples.

Eric Aside

Within Microsoft, you can tell who are FTEs, vendors, or CSG by their email aliases—an alias starting with “a–” indicates a CSG, an alias starting with “v-” denotes a vendor, and an alias with no dashes signifies an FTE. There are other dash prefixes, but a’s and v’s are the most common.

In addition, the Microsoft badges people use to gain access to company buildings and resources are colored differently for different workers. FTE badges have a blue outline around the photo, while vendor and CSG badges have an orange outline. This has led to the FTE nickname “blue badge.”

No future for you

The first two rules are basic—highly confidential, ambiguous, or open-ended roles are for FTEs. What about the third rule—jobs with limited growth potential? Receptionists, gardeners, and the janitorial staff play important roles at Microsoft, but they aren’t growth roles at the company. They are done by vendors.

What about folks who send the daily build mail, localizers, or event planners? Those jobs could be designed to be entry level roles that lead to careers in engineering or marketing, but if that’s not the plan, then FTEs shouldn’t do them.

If you as a manager can’t describe how a role is leading an FTE toward his or her career aspirations, then that role needs to transition to a vendor or CSG. The fourth rule often determines the selection. If the role is ongoing, it should be filled by a vendor; otherwise, a CSG could probably handle it (the exception being highly specialized vendor work).

Eric Aside

A strong indicator of an ongoing dead end role that should be vended out is an uncommon, yet low-level role. An example is localization. Microsoft used to have localizers on staff. These were low-level jobs that served an ongoing need, yet didn’t have much of a growth path. Our localization leadership reexamined our business model and decided to use a combination of vendors and crowd sourcing for our localization needs, with FTEs continuing to guide the overall process.

This is just temporary

Looking at the list of rules, most seem to point to FTEs or vendors. When are CSG folks the right choice? When it’s a temporary assignment for a specific purpose that requires oversight or must be filled quickly.

For example, let’s say you need to hire three people immediately to code and/or test a straightforward new feature to meet one of those random requirements that crops up from time to time. You could pull FTEs off their current assignments to handle this fire drill, but they are working on other critical work items. Hiring three temps to code and/or test the feature makes sense.

Let’s say an engineer on your team suddenly needs to leave for a few months to care for a family member. You can’t afford to scope back the engineer’s work, but you can hire a temp for a few months until your engineer returns. The temp probably won’t replace your engineer’s skill set precisely, but between the temp and the rest of your team, you can handle all the loose ends.

There are many other examples, but they all have the same basic structure—you’ve got a temporary assignment for a specific purpose that requires oversight or quick action. Otherwise, you’d use FTEs or vendors.

What now?

Your temp does a great job on the new feature or covering for an employee on leave. Now the feature is done or the employee returns. What’s next for this great CSG? Surely you don’t let her go—there’s plenty of other work to do! Wrong. You let her go.

Temporary employees are, well, temporary. They have a specific purpose. When that purpose is served, the CSG contract is over and the person leaves. Perhaps at a later date you’d hope to hire the CSG into another temporary assignment or even into an open FTE position. However, for now, temporary means temporary, and a contract is a contract.

What are we doing?

What happens when you’ve got vendors or CSG doing work that does have Microsoft career growth? Work that is ongoing and requires frequent oversight? Perhaps the work started out specific and temporary, but grew in scope. This sounds like work better suited for FTEs, but you don’t have the headcount. What should you do? You should reexamine your business.

What are you trying to accomplish? How has it changed? How should you operate going forward? Business changes all the time, yet people don’t often reexamine the way they operate their business. Why not? Because they are lazy. Because they get caught up in the day-to-day work. Because the changes come slowly enough to go unnoticed.

This is a test

For example, say you used to hire CSG to test your service at release time. You didn’t need as many testers during the long product cycle, but hired a bunch for the crunch at the end. Over time, competition and enlightenment moved you to a monthly or even weekly release cycle for your service. Now you’ve got CSG testers on staff all the time, rotating through (no permatemps). It doesn’t make sense anymore. What do you do?

You’ve got a few options:

Continue to use a rotating crew of CSG testers because you are a lazy, backward fool who enjoys release cadence disruption caused by turnover.
Convert your CSG dollars to FTE dollars and hire full-time testers. Yes, this can be done and is often cheaper when designed well. You do need to think carefully about your hiring ramp so as not to be left with periods of reduced capacity.
Change how you test your service to use a combination of instrumentation, statistical methods, testing in production, and increased unit and component pair testing by developers. This allows you to reduce the size of your test staff over time, while increasing its statistical and strategic capability. However, this is a big technical and people change that takes time and patience to implement.

Business changes are business opportunities. You can ignore them, or use them as a catalyst to remake your team to be more competitive. The opportunities do require you to embrace change, break old habits, and rethink your business.

Doing it well

Do you have the right people doing the right work? Has your business environment changed, yet you and your team haven’t? Get off your butt, and take a close look at how your business is run.

Have you got some folks in dead-end jobs? Fill those roles with vendors instead. Got a quick job to do right away, but you need extra staff to do it? If it can be done offsite, hire a vendor; otherwise, hire some CSG. Are you using CSG for work that has now become commonplace? Convert them to FTEs or adapt to a more competitive approach.

Don’t be lazy—be proactive. Don’t be ignorant—be informed. Use the right folks for the right jobs. Keep your team growing, competitive, and attuned to your business and customers. Make your staff your means to win.

Don’t be a tool

ericbrec — Sun, 01 Apr 2012 06:00:00 GMT

A recent flood of build breaks triggered a wave of tool suggestions to plug the cracks in our code. Some argued for faster builds. Some argued for deeper branching. Some argued for a “gauntlet” service that simulates official builds and blocks problem code submissions. All of these suggestions are awash in the seeping sewage of the flood—none of them address the root cause, and many would only pressurize the leak until it truly exploded in a tsunami of stifled stench.

Build breaks and other quality issues aren’t created or resolved by tools any more than the code is. Problems are created by people, and they are resolved by people. Misguided, well-meaning nitwits protest, “Of course people create and resolve problems, but tools can have a huge impact.” Heck yeah! Tools can have an enormous impact—they can make checkins slower, costlier, and less frequent; they can frustrate engineers to the point of leaving the project and company; and they can remove all creativity, agility, and pride from development until our code is a mindless mush molded to match the meaningless mechanisms of our monochromatic, masochistic machine.

Tools serve us, not the other way around. Before you suggest a tool, before you jump to a solution, before you make mayhem with mechanism, start with the human problem. What are people trying to accomplish? What’s getting in the way of success? What alternatives are available for the range of situations? Once you understand the true goals, then you can ask how tools might help. Don’t start with a tool. Don’t be a tool.

What are you trying to do?

The problem in this case is a bunch of build breaks. Actually, the problem is that you need your product to build or you can’t deploy it or sell it—so you build it all the time (a best practice). When someone checks in code that breaks the build, then no one on the team can retrieve the current code and build it. That can slow down value being added to your product in the form of code enhancements.

What are you really trying to accomplish? You’re trying to maintain the pace of value being added to your built product. Bad code checkins slow that pace for the individual and the team.

He looks to me to make things right

What alternatives are available to avoid bad code checkins and maintain the pace of value being added to your built product?

Block checkins that don’t build. This is known at Microsoft as a “gauntlet” system. Every checkin is built and tested by the system. Checkins are only submitted if they pass.
Pros: Build never breaks when the system is working properly.
Cons: Can significantly slow the pace of any value being added to the product, and these systems often don’t work properly so build breaks still occur.

Eric Aside

Gauntlet systems only succeed when their results match official build and test results. Of course, the gauntlet system can’t be identical to the official build system (different queuing mechanism, different code signing, different build machines and environment, different publishing, and different performance optimizations). Maintaining identical results for separate systems isn’t feasible, thus gauntlet systems often don’t work properly—in addition to often adding hours to checkins to perform their validation.

Work in a private branch. Checkins are not submitted to the main branch. Instead, small groups of engineers work in private branches off the main branch. Sets of changes are integrated into the main branch only when they build properly on the private branch.
Pros: Bad checkins only slow down a small group of engineers.
Cons: Slows value being added because product changes must travel from the private branch to the main branch to reach the built product. The pain and disruption of build breaks becomes the pain and disruption of integrations. When many teams integrate their private branches during the same week (a common occurrence), conflicts and breaks are common.
Work against the last known good (LKG) build. When a build succeeds, it is labeled as an LKG along with its source code. Instead of using the latest checkins, engineers sync to the LKG, and thus their builds are never broken.
Pros: The team can continue adding value to the product even when someone submits a bad checkin.
Cons: The longer the time between LKG builds, the more the LKG gets out of sync with the current code. This leads to all the problems you have with private branches, without the convenience of isolating them to a private branch. You can alleviate the problem somewhat by continuously creating LKGs with a rolling build, but when that rolling build breaks, you’re almost back to where you started.
Validate work before checkin. Engineers perform their own validation, using build and test tools to check their work before checkin. Often this means having two enlistments—one for development and one for running clean builds. You can also build on a buddy’s machine or have some form of shared buddy build service.
Pros: Easy and catches issues at the source before they become team problems.
Cons: Trusts engineers to be diligent. Forces engineers to use their own machines for validation or a shared buddy build service, which can be slow and seriously deplete storage and CPU resources on work machines. Breaks inevitably get through.

Each alternative has its strengths and weaknesses. Which one is right? Ah, we haven’t considered the range of situations.

Eric Aside

The complexity of your build system itself could be causing breaks. Many of these systems evolved slowly over time and haven’t received the engineering rigor we take for granted with modern production software. Investing in build systems isn’t sexy work, but it has a huge force multiplier when every engineer gains an hour or more of productivity per day.

You got something for me?

There are three general categories of changes that can cause build breaks. Each has a different risk profile.

Trivial code or resource change. The checkin alters minor logic or resources, like a change to an existing string or icon. (Adding or deleting a resource is more significant.) While these changes can lead to build breaks, the occurrence is quite rare. Yet often these changes are valuable to the business or customer—you want them added quickly. A simple build should do. Waiting for a buddy build, let alone a gauntlet pass or an integration period, seems disproportionate to the risk. People hate disproportionate risk responses—engineers especially.
Isolated change confined to a known area. The checkin alters code within a relatively small and well-defined scope, like refactoring a single class while leaving its interface the same. These changes can break builds, but rarely beyond their defined scope. Local builds and testing should be sufficient to catch a break.
Interface or behavioral change to shared components. The checkin impacts written or unwritten contracts between components. These changes often lead to build breaks, impacting the entire project. Great care must be taken to perform clean builds on buddy machines for all build configurations followed by a broad spectrum of build verification tests.

A gauntlet system can protect you from all three categories, but it’s only worth it for the last category, and it’s prone to failure. A private branch helps the last two categories, but it’s overkill for the first category and just moves the pain around. LKG builds and trusting engineers to be diligent work for all three categories, but will let build breaks through.

Wow, what do you do? Oh wait, that’s right—it’s a people problem, not a tool problem.

Power to the people

Tools can help, but you start with people. Too much reliance on tools quickly makes them a crutch, causing people to shut off their brains and hit the button instead of applying discretion.

How do you best avoid bad code checkins and maintain the pace of value being added to your built product? Allow and expect people to use engineering discretion.

For a trivial change, let them check code in after a simple build check.
For an isolated change, give them solid local build and testing tools to run, and then let them checkin.
For changes to shared components, give them a buddy build service with full build verification checks to run before checkin.
For a nice combination of protection and speed, use an LKG system on a private branch with a fast rolling build that integrates every single good build into the main branch.

How do you keep people from breaking the LKG? You don’t. As long as breaks are rare, the pace of value being added to your built product will be high. So you trust engineers to be professionals and to apply the appropriate level of verification for their changes.

When you make the impact and choices clear to people, and you publicly expect and trust them to make the right decisions, they feel empowered and responsible for their work. The right example is set and folks fall in line—for each other as teammates. The risk is minimized, build breaks are rare, and value to customers is maximized. That’s what happens when people are the solution.

Eric Aside

Many teams highlight the trust placed with engineers by conferring a lighthearted, embarrassing token to folks who fall short of earning that trust. My old team used a big stuffed bear called “Buster, the Build Break Bear.” The last person to leave the build broken for an hour had to keep Buster on display in his office. As long as it’s not mean spirited, this kind of token is an effective reminder.

For enormous teams, the sheer number of engineers causes even rare events to become common. To protect the LKG, you’ll need separate private branches and LKGs for each large subgroup. You’ll want rolling build integrations in both directions to keep the private branches in sync.

How large should subgroups be? You want as few private branches as possible because each extra branch slows down code movement. So you want subgroups as large as you can make them and still have only a few LKG breaks a week (100 to 500 people each has been my experience).

Out of focus

ericbrec — Thu, 01 Mar 2012 07:00:00 GMT

Are you sensing a rush coming as we complete midyear career discussions at Microsoft and head into the stretch toward annual reviews? Worried about keeping up with your peers when you already have far too much to do and far too little time in which to do it? Feeling lost already? Please. Get a grip and get a map.

When you have to take an urgent trip somewhere, you don’t drive off without knowing where you’re going as if you are on a carefree vacation. So why the heck are you overwhelming yourself with tasks without having a personal plan?

“But my manager says I’ve got to do all this stuff!” Grow up! Think! What really matters? What will your manager talk about in calibration this summer? Focus on the few things that matter most, and manage everything else so it doesn’t get in the way. How? Read on.

Eric Aside

Focusing on the few things that matter most is one of those principles that apply to everyone and every project regardless of level or scope. It’s just as important to an individual engineer as it is to an engineering director or division president.

But that’s not important

Step one for knowing where to focus is to differentiate between what’s urgent and what’s important. Urgent tasks can fill up your day if you let them. A year will go by and you’ll have accomplished nothing important. That’s a problem.

Work with your manager to scope your responsibilities so that you have time for what’s important as well as what’s urgent. Naturally, your manager will want urgent work completed promptly, but it doesn’t have to all fall on you. Usually, urgent work can be shared across the team. Important work demands ownership and responsibility. That’s where you want your focus to be.

Not much to say

How do you determine what’s important? That’s easy. What business are you in? Who are your customers? What portion of the business does your team own? Tie those things to what your leadership team, your manager, and your skip-level manager care about, and you’ve got what’s important.

Let me put it another way. During review calibration, you are compared to 50 to 100 peers in similar roles across your organization. Even in a four- to eight-hour calibration meeting, managers only have about five minutes on average to talk about each employee—not a lot of time. Your manager can only discuss a few things you’ve focused on in your work.

Keep in mind, your manager will prepare and take your entire year into account. Out of that, she will pick out key work you’ve accomplished and how you accomplished it. What are the few items you want her to bring up that best represent your efforts—that will stand out as valuable among your peers? That’s what’s important. That’s what you focus upon.

Eric Aside

For more on calibration, please read Out of calibration.

Trust, but verify

“But what about all the other work that needs to get done? I can’t get away with doing just a few things all year.” Actually, you can if those few things are important, but I know what you mean. There’s a ton of work that comes across your desk and inbox each day. People count on you to be responsive and a good partner. You don’t want to let them down, and sometimes small items can turn out to be important.

First and foremost, know what few important things you are focused on currently—don’t lose sight of them. Never let noisy urgency distract you away from being laser-focused on what really matters.

As for the rest of what gets thrown your way, you’ve got a few options that I described in Don't panic:

You can ask for clarification—this delays the task and sometimes causes it to resolve itself.
You can do it or defer it based on priority—this keeps you focused on your important work.
You can entrust it to someone else—this gets the task off your desk and quite possibly to someone better positioned to handle it.

What do you do if an important request arrives, but you can’t prioritize it ahead of your other important work? Entrust it to someone else, but keep track—stay on the mail thread and verify the request is being resolved. (I keep a special email folder for work I’m tracking.) Trust, but verify. That way, you ensure the important work gets done and you are able to scale.

Two to tango

If you only need to have a few important accomplishments per year, plus be responsive to day-to-day requests, how many important things should you work on at once? Two.

As I mentioned in “Time enough” (chapter 8), people are most productive when they are working on precisely two projects: a primary project, where they focus most of their time, and a secondary project to fill-in when the primary project is stalled or they need a change of pace.

You should handle mail and ad hoc requests as you reach nice stopping points during the day. Don’t disrupt your flow.

Eyes on the prize

As you consider what you’ll be working on for the next six months, clarify which important projects you’re responsible for delivering—the ones you own. At any given time, know which project is primary and which is secondary.

It may sound strange to focus only on a few important projects, but that’s what makes the difference between busy work and achievement. Focus delivers experiences instead of disjointed sets of features. Focus keeps people aligned and in sync instead of falling into dysfunctional chaos. Focus achieves your goals instead of simply keeping you occupied.

It’s never too late to get focused. Understand your business (that’s the purpose of those all-hands meetings). Know your customer. Work with your manager to determine where you and your work fit. Then focus on a few important deliverables. Achieve that and you’re sure to give your managers something good to talk about at calibration—something that makes you proud.

Software engineering—what’s missing?

ericbrec — Wed, 01 Feb 2012 07:00:00 GMT

To start the new year, my boss gave an all-hands speech to a large group of developers about being an engineer. He equated being an engineer with taking responsibility for quality and using methods that ensure high quality at checkin (Nailing the nominals). Naturally, a developer in the crowd took issue with calling software developers “engineers.” He claimed, “Engineers assemble parts out of tables and charts of proven components. Developers create something new and unproven every time.” Some disputes never get old.

Is software development engineering or craft? I’ve already argued that it’s pointless to argue. Developers must act like engineers or we all lose our jobs. No one bets his company or health on well-crafted software—it had better be engineered. As I claimed in “A software odyssey” (chapter 5), developers need to know instead of guess. They need to use instrumentation data from their products and engineering systems to guide decision making and drive quality and improvement.

But what about all those handy tables and charts “real” engineers use to assemble parts? Does software have an equivalent? Why do new language paradigms claim to solve the reuse problem, only to later fail? Can we be engineers without repositories and reuse? Hey, control yourself! We do have code warehouses, language paradigms are misguided, and we are engineers. Quit waving your rationalization crutches around, hang up your cowboy hat, and be a professional.

Eric Aside

I had an email conversation with the guy who asked the question. He was interested in setting up a repository, which got me thinking.

If at first you don’t succeed

Since the 1960s, there have been numerous attempts to establish software repositories and reuse. Many of the advances in object-oriented languages were accompanied by sincere claims that they finally solved the reuse problem, and soon software development would take on more of the character of other engineering disciplines. Unfortunately, those claims haven’t been realized.

At Microsoft, there have been many failed attempts to establish code repositories. The most recent internal attempt is CodeBox. It enjoys great success as an internal “open-source-like” collaborative development environment for tools, but it’s not a reusable code repository beyond copy/paste (a separate subject I address later).

Why have these attempts failed? Is there no hope? Oh, please. Folks have been thinking about the problem the wrong way. Imagining parts as class instances failed as an analogy—engineering reuse is quite alive and well in software.

I got what you need

There are two proven instances of code repositories and reuse that work and work well—libraries and design patterns.

Software libraries, including DLLs, interfaces, platforms (like .NET), SDKs, web services (mashups), and even JavaScript libraries like jQuery, have been wildly successful. They have achieved broad adoption, driven tremendous savings in development effort, and resulted in significant quality and security improvements to software systems.

Design patterns have also been highly successful, though not quite as broadly adopted as libraries. They provide proven design approaches for common engineering problems that result in more robust and reliable code that is easier to explain and understand. Design patterns are an old idea used in many other engineering disciplines. They aren’t reusable parts—they are approaches to common engineering challenges.

Something borrowed, something blue

There is a third, broadly utilized example of code reuse—copy/paste. Code “borrowed” from another engineer is prevalent in every significant codebase ever studied. Borrowing code responsibly can save you time and effort by using the best accumulated knowledge available. Borrowing code irresponsibly can cost you your career and your company millions of dollars. The specifics of who owns the code and how it is licensed are critical, but that’s a discussion for a different time.

Microsoft is extremely restrictive about borrowed code. Even internal efforts like CodeSearch are secured so that only people working on the same product can search their product code. The areas with the most code “sharing” are engineering tools and appropriately licensed code samples from SDKs.

Look, I love “sharing” and “borrowing.” My kids were taught these concepts in toddler group. “Borrowing” responsibly is part of being a professional, but it shouldn’t be confused with reusing documented, measured, tested, and published libraries and patterns from an established repository. Those proven repositories provide predictability and repeatability to engineering—the attributes that guy was seeking when he questioned whether or not developers are engineers.

A tremendous sense of isolation

Why do libraries and design patterns succeed where other repositories and reuse fail? I think the key is isolation.

Libraries are isolated from the rest of the code. They form a separate logical layer that developers can program against almost as a black box. There’s no need for the vast majority of developers to change the library, and most never even see the library code. This makes for stable libraries and clear separation of responsibilities—quite sustainable.
Design patterns are isolated from code by their very nature—they aren’t code. They are abstractions of code structure that defy fixed implementation, which also makes them stable and sustainable in their own way.

Note that while both libraries and design patterns provide abstraction, abstraction isn’t enough—you need isolation. If the developer ever has to mess with the repository for a typical project, then the isolation fails, and the repository becomes unstable and unsustainable for reuse. It’s a bummer, but that’s how things have gone over the past 50 years.

Eric Aside

Note that copy/paste succeeds as a reuse approach for the same reason—isolation. Copying code from another source does not impact the source in any way. This is a blessing and a curse. Should the original source have a serious error, such as a security vulnerability, that error is propagated. However, unlike libraries, a fix to the original code does not propagate to where the code was copied.

What are we doing here?

If software libraries are the only form of proven code repositories (like charts and tables for other engineering disciplines), does software engineering fall short of engineering? No, software development is just like every other form of engineering practice.

If you are an electrical engineer designing circuit boards, each new circuit board is unique. You determine what the circuit board’s inputs and outputs are and what it needs to do. You look through your tables and charts of circuit board components to find the ones that meet your specs. You then use common electrical design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the components on your new board.

If you are a mechanical engineer designing transmissions, each new transmission is unique. You determine what the transmission’s inputs and outputs are and what it needs to do. You look through your tables and charts of gears, shafts, and clutches to find the ones that meet your specs. You then use common mechanical design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the components in your new transmission.

If you are a software engineer designing an application or interface, each new piece of software is unique. You determine what the software’s inputs and outputs are and what it needs to do. You look through your online documentation of software libraries to find the ones that meet your specs. You then use common software design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the software components in your new application or interface.

Living the dream

There are a few lessons to be learned about being a software engineer from analyzing software repositories and reuse.

We can encourage reuse by designing well-isolated architectural layers, creating robust libraries and interfaces that reflect how developers use them, and communicating effective design patterns that propagate engineering expertise.
Having great software repositories isn’t enough. We must follow other essential engineering practices, such as basing our decisions on data as well as experience, investigating applicable design patterns and libraries we can use, and taking responsibility for quality by having our designs and code peer reviewed, our code analyzed, and our units and components tested.
Put those pieces together and there’s nothing missing from being an engineer other than demanding that minimum quality bar from each other and exceeding it ourselves as professionals.

You can be an engineer and you should be an engineer. Stand proud. Be strong. Win the hearts, minds, and well-earned capital of our customers.

Eric Aside

This is my 100^th column. I love that it’s about quality and engineering—two things I really care about.

Many thanks to all the readers who’ve spent a little of their valuable time reading my cathartic rants each month. Ranting to you sure beats talking to myself.

Who’s in charge here?

ericbrec — Sun, 01 Jan 2012 07:00:00 GMT

I was talking with a friend from another Microsoft division. He complained about gridlock on his team because “no one can make a decision.” He lamented, “We discuss issues and come to some conclusions, but rarely get a resolution that sticks.” I bit my tongue. He is a friend.

Later in that same conversation, my friend asked if he should switch groups since he didn’t think there were opportunities to be promoted to the principal band. “There aren’t any broad challenges on my team,” he said. I shot back, “What, are you kidding? There’s nothing but opportunity on team mollusk. All you have to do is act like a vertebrate. Assume authority, you wuss!”

Okay, I didn’t call him a “wuss.” He is a friend. But he was also being a wuss. He said, “I can’t just assume authority. Why would they listen to me?” Let me let you in on a secret. People are sheep. Folks are afraid to risk taking responsibility and all that comes with it. If you step forward, people will follow.

I assume so

“It can’t be that easy.” Yup, it is. If you are willing to take responsibility, to stick your neck out, then people will gladly give you that responsibility. After all, if they were desperate to have it, they would have stepped forward.

How do you assume authority in a situation where others hesitate? Three steps:

Choose your battles. You must first decide if this issue is worth putting your reputation at stake. Wise people choose their battles carefully. You should assume authority over issues you own and issues that matter to you. Every other issue is a charity case that you should only champion out of the goodness of your heart.
Choose your direction. You must next decide for yourself in which direction to move forward. Your decision should be based on the best information at hand and be reasonable to those around you. For advice on making decisions with incomplete information, read You have to make a decision.
State the direction. You then take responsibility by laying out the decision as plainly as possible—no embellishment. It’s not, “I think we should go north,” or “Everyone okay with going north?” It’s, “Let’s go north.” Speak with confidence. Speak with authority. Off to the north you go.

Eric Aside

When other people are willing to take responsibility, you can’t just assume authority. In that situation, authority is shared, and you need to negotiate. Read my columns Controlling your boss for fun and profit and My way or the highway for more on persuasion and negotiation.

Follow you, follow me

“Will anyone follow me north?” Absolutely. Remember, we are talking about situations in which others aren’t willing to make the decision—after all, making a decision risks failure. By taking responsibility, you are relieving others of that burden. They are happy about that. They are also happy to have a direction. The only price you ask for all this happiness is to follow. Of course they’ll follow.

“But what if I’m wrong?” Then take responsibility for the mistake, move on, and try again. Iterate. Be willing to absorb a little failure for a lot of learning. The experience will help you long term, and no one is perfect. At least you made a decision, and the team made progress in some direction. As I discussed in You have to make a decision, that’s far better than deadlock.

Eric Aside

For more on what to do when you make a mistake, read I messed up.

Who do you think you are?

“But what if my boss or some high-level leader is in the meeting? I can’t assume authority then, right?” Oh yes, you can! Sometimes that’s the best time for you to act—assuming no one else seems willing to step forward.

If your boss or a high-level leader is present and he makes the decision, then you should respect that and follow his guidance. However, leaders often don’t have enough information to make a decision. If you own the area or simply feel you know the best direction and no one else is taking responsibility, then go ahead and state the direction.

“But I’m not the boss!” Look, there’s a big difference between deference and respect. You should respect your management, but you shouldn’t defer to it to the point of inaction. You have an opinion. You have knowledge and experience. Your management pays you to use your head. Respect that and use it.

We’ve gone too far

Once you build up your confidence and get accustomed to taking responsibility, you might get overzealous. Be careful. Not every battle is worth fighting. Stay focused on what you own, what role you play, and what’s important to you and your team.

If you take on too much responsibility, you’ll be taking it away from others. If they object, you’ll create unconstructive conflict. If they acquiesce, they will lose confidence and the capability to act with conviction. Know your boundaries. Exceeding them will eventually hurt yourself and your team.

With great power

Who’s in charge? You are. Remember when you were young and thought the president ran the government, CEOs ran companies, and the principal ran the school? You were misguided. It’s ordinary people who make the decisions and drive what happens.

There are no special qualifications to be a decision maker. All you need to do is decide to decide, and then be thoughtful using the best data available. Be willing to take responsibility, and people will gladly hand it over.

Assuming leadership is like assuming power—it can be intoxicating and hazardous. With great power comes great responsibility, so know your limits. Understand the situation, choose your battles, speak from confidence and conviction, state the direction, and people will follow.

You can do it. It helps your team and your career. Choose to lead. Our company depends upon you.

That's not funny

ericbrec — Thu, 01 Dec 2011 07:00:00 GMT

Tension fills the conference room a few weeks before the Client release. The Client team wasn’t told that the Database team had added a parameter to the AddClient API. The Client broke spectacularly—the latest in a series of miscommunication and miscues by both teams. Only now were the Client and Database teams meeting to discuss the breaking change. Contempt oozes over the conference room table.

The Database lead program manager (PM) opens the meeting with, “As you know, the Client team requested the new parameter for AddClient a couple of months ago. [Uproar.] We realize now that the change came in too late. We’ve rolled back the AddClient API and removed the new parameter. We’ve got a build running and should have the fix for the Client team later today.”

After a moment, the Database lead PM adds, “By the way, did we tell you about the other new API function?” The room gets quiet. Everyone stares at the Database PM. The PM looks around the table and then says, “It’s called ‘BlindsideClientTeam.’ We’re thinking of adding a parameter called ‘lastMinute.’”

Eric Aside

This opening doesn’t follow the usual I.M. Wright rant, because the column is about using humor to diffuse rants. It’s a holiday meta topic for “Hard Code.”

It’s a funny thing

How do you show a partner that you’re sorry for a mistake, understand the trouble you’ve caused, and rebuild trust between your teams?

You could apologize, though apologies alone sound hollow when people and schedules are compromised.
You could show your understanding by recounting the situation, though that often seems patronizing.
You should take steps to fix the problem and avoid it going forward, but those take time, and you need to repair relationships now.

Humor in these situations works wonders. You can’t lead with humor—it’s a serious situation. You lead with taking responsibility and outlining steps to fix the problem. But then, to repair hurt feelings and regain trust, nothing works quite like humor.

Eric Aside

For more on handling a bad mistake, read I messed up.

No one understands me

Building and maintaining trust between people is difficult. Mistakes happen. When trust breaks down, relationships become dysfunctional. Teams that don’t trust each other tend to duplicate work or break ties entirely to mitigate risk. Individuals that don’t trust each other can’t compromise and resolve issues.

To build trust, you must understand each other’s situation—what’s important and what’s scary—and you must empathize. Otherwise, people won’t believe you’ll take care of their interests and concerns.

However, showing empathy is tricky. You don’t want to sound like some new-age life coach: “So you’re saying that if I repeat how you feel one more time you’ll slit my throat? You must feel angry.” This is where humor can make a big difference.

That is illogical

When it comes to issues people care about, like their time, effort, and future earning potential, people aren’t rational. They assume others will ruin their plans and cause them to fail at every turn. That’s why people resort to hyperboles like, “They change direction every day.” They ask contrived questions like, “Would it be possible for you to share your schedule?” Of course it’s possible, especially for partner teams—are we children?

You might think that responding to irrationality with a calm and rational response would be enough to quell fears. It often solves the immediate problem, but the underlying distrust still lingers.

To change the dynamic of a relationship, you must expose the distrust and address it. However, doing so is like exposing an open wound—it’s ugly and sensitive. You need to ease the pain, release the tension, and allow everyone to relax. Humor, when used judiciously, provides just the right laughing gas to reduce the pain.

The fab five

There are five general categories for humor in the workplace. Each has its purpose (breaking tension) and pitfalls (not everyone or everything is funny). Let’s break them down.

Self-deprecation. Making fun of yourself is one of the easiest and least risky forms of humor to use at work to diffuse tension. Purpose: acknowledge bad behavior or weaknesses without being mean. The Database lead PM used self-deprecation to admit his team’s culpability to the Client team. Pitfalls: appearing insincere by not acknowledging true weaknesses. The classic example is when someone admits to “caring too much” (not funny).
Absurd contradiction. Denying a request or fact that is not remotely in dispute can make all disputes appear silly. Purpose: expose distrust or disagreement in a nonthreatening manner, and encourage your peers to feel at ease. A disarming response to “Would it be possible for you to share your schedule?” is “No.” Wait for your partner to be startled, and then follow with “I’d be happy to send you the URL.” Pitfalls: appearing passive aggressive or even sincerely aggressive. You can seem passive aggressive if your denial appears sarcastic instead of absurd (don’t embellish). You’ll seem sincerely aggressive if the issue is so emotionally charged (perhaps a bad history) that everyone thinks your denial is serious regardless of how crazy you make it.
Broad exaggeration. Making a significant overstatement or understatement can be funny while providing perspective. Purpose: draw attention to something important without sounding whiny. An overstatement, like “The build broke 57 times yesterday,” draws attention to build problems without being too particular or placing blame. An understatement, like listing “future earning potential” among lighter things people care about, actually adds more weight to the area than saying “it’s so important.” Pitfalls: appearing serious and thus whiny, uninformed, or uncaring. Don’t exaggerate a little—people might miss it. Don’t make a big show of your exaggeration—people might think you’re not joking. Large overstatements and understatements work best with a straight face and solemn delivery.
Outrageous ideation. Suggesting radical ideas during a brainstorming session to enhance creativity. Purpose: get people thinking “outside the box.” Say you are brainstorming ideas for driving resiliency of a service. One idea could be “Blow up the datacenter.” That’s a ridiculous idea, but could lead to people considering testing random service shutdowns and geo-distribution that they might not have otherwise considered. Pitfalls: appearing dumb, flippant, or unconstructive. Outrageous ideation can be overdone easily. Be sure to have a number of pragmatic ideas as well and start with some of those.
Deliberate dissonance. Intentionally breaking the flow of an interaction to shift direction (a generalization of absurd contradiction and outrageous ideation). Purpose: relieve tension and change the dynamic of the conversation. When people are introducing themselves and mentioning the important jobs they’ve had as if it were a contest, you can lower the pompous pretense with “I’m a former McDonald’s fry cook.” Pitfalls: appearing foolish, unprofessional, and even offensive. Dissonance is strongly based on cultural norms and personal background. Your witty comments can easily fail to amuse the wrong audience. I usually have serious things to add in case people miss the joke. I then treat my witticisms like a dog whistle that only a few may hear.

As you can see, humor has many pitfalls, and not everyone has the timing and delivery to make people laugh. However, most people can find ways to break tension and take themselves a little less seriously. Use your best judgment, practice among friends, and find what works best for you.

Tell the truth

The magic of humor is how it serves unadorned truth in palatable portions. It’s inappropriate to call someone crazy, especially in front of a group. It’s awkward to acknowledge someone’s fears in a professional setting. It’s incendiary to rehash failure when people are frustrated and fuming. However, a little respectful silliness can expose irrationality, break through barriers, and show some remorse without humiliation.

Of course, humor can be overdone or overused. We are professionals in a work environment—there are limits to what is appropriate. Knowing this, there are times when it’s important to tell the truth clearly and candidly. Humor can provide a bridge that enables us to speak openly and honestly about difficult issues when we need to most. Doing so resolves problems, builds trust, and permits us to move forward as one team to solve the real problems facing our customers.

Eric Aside

You can read more about humor in the workplace from the write-up of a little known Microsoft Engineering Competency: Humor that includes a recommended reading list.

Destabilization

ericbrec — Tue, 01 Nov 2011 07:00:00 GMT

It breaks my heart and sickens my stomach to witness the tremendous productivity and quality gains of Lean Software Development practices at Microsoft: feature crews in Office, scrum teams in Xbox, and improvement teams in SQLServer, to name a few. These Lean approaches yield less-incomplete work, higher-quality builds throughout the product cycle, earlier feedback on features, and timelier cross-discipline communication.

Eric Aside

Feature crews are small, cross-discipline teams each tasked with a single feature, or closely related small features, to design, spec, develop, and test together from start to finish. Improvement teams are quite similar to feature crews, but have a different name to emphasize their work on infrastructure improvements, as well as features.

Why is my heart broken by greater Lean productivity? Why is my stomach in knots over higher Lean quality? Because there’s an insidious, deceptive, soul-crushing beast lurking in many of Microsoft’s Lean Software Development efforts, slowing them down, driving poor engineering practices, lengthening our ship cycles, and reducing the feature set we deliver to customers. What is this horrifying, hellacious, heathen of a heartbreaker? It goes by the name “stabilization.”

Stabilization periods between development milestones don’t just add superfluous time to schedules. Stabilization periods also make products worse with fewer features and lower quality, while also dragging down morale and encouraging crappy code. Oh, my forlorn heart. Oh, my aching stomach.

We don't need you anymore

Legacy McLoser says, “You’ve lost it this time, Mr. Wright. Stabilization periods are essential for integration quality.” No, they’re not—at least not when using Lean methods, like feature crews, improvement teams, scrum, or kanban.

In traditional Waterfall methodology, the codebase isn’t functional or complete during development periods. Stabilization between development periods provides an opportunity to run integration and system testing, fix bugs, and get the codebase close to a shippable state. Without stabilization periods, integration and system issues build up to the point that past Microsoft projects have failed or slipped years. Thus, stabilization is required when running traditional Waterfall.

In Lean methods, a cross-discipline team works on a feature from start to finish. The feature enters the codebase integrated, complete, and fully tested (or at least it should—more on this later). The codebase is always kept close to a shippable state.

Integration issues that occasionally arise in Lean projects resemble sustained engineering issues far more than the sustained insanity of cleanup during Waterfall stabilization. Teams should deal with sustained engineering issues as they go. Therefore, stabilization for Lean methods is unnecessary and quite harmful, as I describe below, because it demoralizes your best teams and encourages your worst.

Eric Aside

For more on the basics of Lean Software Development, please read my 2004 column on the subject, “Lean: more that good pastrami” (chapter 2).

Shake and bake

Note that I’m only talking about stabilization periods between development periods. I’m not talking about the lengthy stabilization time at the end of a long product cycle. That “bake time” for release candidates is necessary for traditional, packaged products like Office and SQLServer to catch edge cases that take time to emerge, regardless of whether projects use a Waterfall or Lean approach.

Eric Aside

Properly designed services can be gradually rolled out, instantly rolled back, and iteratively improved after they are released. Thus, properly designed services don’t need “bake time” at the end of a project.

Packaged products have limited opportunity to roll out gradually, instantly roll back, or iteratively improve once they are released to a wide audience. Thus, packaged products (and critical trust services, like handling credit cards) need “bake time” to minimize risk.

I talk about properly designing services with gradual rollout and instant rollback in my column, There's no place like production.

Get a clue

Stabilization periods between development milestones aren’t just unnecessary, they are harmful. Why? Imagine a feature crew committed to delivering every feature completed and fully tested. We’ll call this feature crew “the clue crew.”

What does the clue crew do during stabilization? Sure, there might be a few integration issues, but certainly not enough to keep the clue crew busy the whole time. Mostly, the clue crew fixes other team’s bugs and works secretly on more features.

Meanwhile, there’s another feature crew, “the poo crew,” which is sloppy. This crew takes on too much work and doesn’t deliver completed and fully tested features. Instead, its features are full of bugs and lack fleshed-out tests. The poo crew uses the stabilization period to clean up its mess—with the help of the clue crew.

What happens next? Naturally, management notices how the poo crew ended up completing more officially booked features than the clue crew—even though the poo crew could only finish its features with the extra few weeks and the help of the clue crew. Management chastises the clue crew for its lack of ambition and encourages it to book more work, like the poo crew, the following development period.

What’s the matter here?

Maybe you think the poo crew is the smart one. After all, it appears to be more productive than the clue crew. If you are that superficial and stupid, please take your way-back machine to the 1990s, and get back to wasting valuable time and effort producing buggy products using traditional Waterfall.

Hopefully you recognize that the poo crew wouldn’t have completed as much over the entire development and stabilization period if it had been forced to fix its own bugs and produce tested, sustainable features. Likewise, the clue crew could have produced more tested, sustainable features had it not needed to spend time fixing poo crew bugs, and instead used the stabilization time to complete more official features.

However, as long as there is a stabilization period, poo crews will have permission to leave their features incomplete and clue crews will be punished for writing quality code by being slapped with poo crew bugs and chastised for lacking ambition. In other words, stabilization periods provide the means to demoralize your best teams and encourage your worst. Soon every team is writing poo, and stabilization periods become essential in a self-fulfilling, Waterfall way.

Eric Aside

Why is the Lean clue crew consistently more productive with higher quality than the poo crew?

Because they test their features early, before bad designs are baked into the code.
Because they fix bugs early, while the code is still fresh.
Because they work tightly across disciplines, while designs are still iterating, improving communication and causing less rework.

Are we done yet?

What’s the alternative to stabilization periods? Hard definitions of “done” for features, enforced by strong release management. I talked about “done” definitions for features in Cycle time—the soothsayer of productivity. Here they are again:

All updated designs and code are reviewed (includes security, etc.).
All automated tests are written and passed (includes security, etc.).
No ship-stopping bugs exist (includes unacceptable fit-and-finish).
All monitoring and feedback is in place.

Typically, feature crews, infrastructure teams, and scrum teams will develop in a branch off the release code branch and not integrate back into the release branch with the new feature until the done definitions are met. There are two rules for working in branches:

Never be more than one branch away from the release branch, unless you enjoy waiting weeks for fixes to integrate up and down the branches. There are several variations on being one branch away—a branch per feature, per crew, per team, or per group. Any can work if you forward integrate daily from the release branch, build your branch at least daily, and never create branches of branches.
Never stay on a branch for more than a month, unless you enjoy merge conflicts and integration hell. If your feature is larger than a month’s worth of effort, break down the feature into a series of smaller, testable pieces, satisfy the “done” definitions for each piece, and then reverse integrate back into the release branch.

Eric Aside

An easy way to avoid missing key concerns like security or accessibility is to use checklists. I discuss them in detail in Your World. Easier and “Bad specs: Who is to blame?” (both in chapter 3).

Being in a stable relationship

With proper enforcement of “done” and smart use of branches, the poo crew can’t integrate its unstable, unfinished, unsustainable ugliness into the release branch. It doesn’t get credit for features it didn’t really finish.

Meanwhile, the clue crew integrates its features regularly and spends its time adding value to the release branch instead of stabilizing some other team’s poo. There’s no need for stabilization in this model because the release branch is always stable with the latest completed features. If there is an integration issue, it’s handled like any other issue with released code: the crew responsible stops work as needed and fixes the issue (see Sustained engineering idiocy for details).

What happens to all those weeks of stabilization time between development milestones? You can either cut them from the schedule and ship earlier or use the time to create and deliver more customer value.

Hasta la vista, baby

Teams still need to plan regularly, coordinate, and synchronize. Packaged products still need significant “bake time” for release candidates at the end of a product cycle. However, stabilization’s days are numbered.

If you haven’t switched to a Lean approach, do so today. Lean is far more efficient and yields higher quality than Waterfall, it scales to huge organizations, and there are enough variations of Lean that one is sure to fit your team.

If you have switched to a Lean approach, do away with stabilization periods. By removing this ugly old vestige of inefficient, low-quality development, and by insisting on only completed features in the release branch, you make your product better, your team better, and your customers happier. May stabilization rest in peace.

Master of your domain

ericbrec — Sat, 01 Oct 2011 06:00:00 GMT

If you had to choose between hiring an outstanding candidate with only related domain knowledge and a solid candidate with specific domain knowledge, who would you select? At Microsoft, we generally select the outstanding candidate, figuring a talented employee can quickly learn a domain. That is, unless the employee already works here.

If the outstanding candidate is already a Microsoft employee, then specific domain knowledge generally trumps capability (aside from poor performers). Why are we this stupid? Because we know there are dysfunctional managers who don’t always use the review system properly, so we give candidates the benefit of the doubt, and because we are obsessed with domain expertise. It’s wrong.

A bunch of Microsoft readers are riled up now, denying Microsoft favors domain knowledge over capability for roles. Really? Think back to your last reorganization. Did the most capable or the incumbents get slotted into positions? That’s right—we’re idiots.

Eric Aside

There are many factors for slotting people into roles during reorganization. You must consider continuity, growth potential and opportunity, and a general fairness and morale. There are difficult trade-offs and mistakes are inevitable. Feel better? Good, because those are all noble reasons, but poor excuses.

My boy's wicked smart

For all the Microsoft readers who’d deny we favor domain expertise over capability, there are other Microsoft readers who support that bias. After all, we’re all smart, and deep domain knowledge isn’t easily transferred. An engineer with specific domain expertise for the job will be more productive. Right? Wrong.

Deep domain knowledge is frequently transferred. It takes a little while, but a team’s existing experts are usually pleased to instruct an outstanding new team member. If they aren’t or an area is far too complex for even sharp people to comprehend, then you’ve got other serious problems.

Yes, the company consistently hires smart people, but being smart isn’t enough to make someone outstanding. Outstanding engineers who make difficult tasks look easy can do more than write great specs, code, or tests. They know how to understand, influence, collaborate, and get things done. They multiply the performance of those around them. Yet in a reorg or job change, these people are eschewed in favor of those with more domain knowledge.

Eric Aside

How do outstanding employees multiply the performance of those around them?

They seek first to understand—the problem, the people, the system, past solution attempts, and proven solutions from inside and outside Microsoft.
They use their understanding of the entire situation to reassure their partners and construct an approach in which everyone is engaged and committed to the successful outcome.
They nurture that partner commitment with inclusive decision making, transparency, and trust-driven collaboration.
With their group of dedicated partners, aligned behind shared goals, they work together to make achieving the impossible a regular occurrence.

You can call it my ghoulish obsession

How does Microsoft culture come to perpetuate an obsession of being masters of our domains instead of overall capability? My educated guess is that it starts with how engineers initially differentiate themselves from their peers and influence their teams—they become “go-to” people for some area. (Individual leadership describes this in detail.) We are conditioned to value subject matter expertise from the beginning.

In addition, outstanding engineers differentiate themselves by how they go about their business. Yet Microsoft is primarily a results company—it’s about what, not how. Subject expertise is about what. Thus, expertise has become king.

Eric Aside

Domain knowledge is important—you can’t put just anybody in any job. In addition, specialization is necessary at the group level for complex software projects, as I argue in “Undisciplined” (Chapter 4). However, your overall team productivity and capability depends upon more than domain knowledge alone.

That is quite logical

Clearly, we don’t need outstanding engineers in every position—there’s plenty of everyday work that solid engineers can do. However, we do need outstanding engineers in key technical and organizational leadership roles. One division that has figured this out is Office.

For years, Office tried to find ways to challenge and reward its best performers by giving them what they wanted—pet projects, incubation labs, and the opportunity to introduce new applications. Then as Office 2007 was wrapping up, the executive leadership of Office decided to try a radical new approach—what if we put our best people on our most critical projects?

In hindsight, the decision to put your top people on your top projects seems absurdly obvious. Of course, it had been done in isolated situations many times before, but Office was breaking new organizational ground at Microsoft by deliberately applying this approach across the entire division. Windows and Windows Phone have done something similar to Office, but in other divisions the practice of actively managing your top talent is still working its way down from the executive ranks to group levels.

Eric Aside

Microsoft has long had a talent management process for top employees called “People Review.” Executives discuss how top performers are progressing, their current assignments, and their potential future assignments. However, actual slotting of key jobs is usually left to the divisions to occur within their release cadences, which typically don’t line up with People Review. This relegates People Review to a status report rather than a Microsoft-wide talent management exercise.

How bad is it?

How bad can it be to put perfectly solid domain experts, rather than outstanding employees, into technical and organizational leadership roles? Obviously, it’s not catastrophic. Microsoft has been operating this way for decades and has done quite well.

Many outstanding employees rise into key technical and organizational leadership roles without the intervention of a deliberate talent management strategy. We generally hire strong people, so solid folks do fine and ship successful products.

There are two problems: opportunity cost and abandonment.

How much better would our products and execution have been if our top people were actively selected for our critical projects? Of course, there’s no way to know. However, I do love my Windows Phone, and the newer versions of Office and Windows. I guess it’s a lost opportunity.
How do outstanding employees feel when they lose assignments to solid everyday employees due to a lack of specific domain knowledge? How do they feel when their talents are squandered, even though they’ve quickly picked up new domains in the past? I have many such friends—enough that this feeling is almost cliché. They feel abandoned and unappreciated—not the way you want your best employees treated.

Eric Aside

Strong employees tend to pick up expertise in multiple domains, but often at Microsoft only the primary domain is valued. Skills in collaboration, influence, vendor management, open source, distributed development, talent management, agile development, and other important but secondary domains tend to be marginalized—a tragic loss of opportunity, value, and appreciation.

Tell me what I must do

There are a few things Microsoft employees can do to ensure our best people are taking on our most critical roles:

If you’ve got an open position on your team, consider the leadership requirements. If leadership is critical, value how candidates influence, collaborate, and get things done over what technologies they last mastered.
If one of your current technological or organizational leaders is knowledgeable, but not particularly gifted at leadership, demand better from whoever will listen. If a position opens, reach out to great leaders you know—they may not feel appreciated where they are.
If you’re undergoing reorganization, write to your executives and demand that leadership trump domain knowledge. Don’t accept excuses for less than the best available.

Domain knowledge is important. Every employee within a division should be a master of his domain. However, when slotting people for key roles, domain knowledge should not be the primary deciding factor, as long as candidates have a track record of picking up new domains. Instead, leadership skills should take precedence and guide us to greater productivity, greater customer value, and greater leadership in the marketplace.

Eric Aside

Unfortunately, Microsoft does little to evaluate or track leadership competence, aside from a fairly superficial, midyear manager appraisal that isn’t formally part of our review or reward system. In addition, our engineering interview process has traditionally focused on domain knowledge and only recently started adding clear guidance on other areas of capability. Perhaps this will improve as our HR systems continue to be revamped.

Production is a mixed blessing

ericbrec — Thu, 01 Sep 2011 06:00:00 GMT

There is one service design flaw that engineers repeat day after day, month after month, year after year. Scalability? Nope, though it’s popular. Security? Happens, but not that frequently. Serviceability? Getting warmer. Give up? Don’t care because you don’t work on services? You will, and when you do, you’ll care.

The pervasive service design flaw is incompatibility. Service neophytes might wonder how this could be. After all, a wonderful thing about services is that you own the machines and versions of software running on them. If there was ever a situation where you could finally escape legacy software and control your environment, it would be services. Right? Wrong. Dead wrong.

Production services are a mixed blessing.

A blessing because you control much of the environment, including the versions of software and hardware running.
Mixed because those versions are always mixed, even if it’s just for a few minutes at a time. Thus, your services need to work in mixed environments—new versions need to be compatible with old versions. Not scared yet? Just wait.

The horror

If you’re unfamiliar with services, perhaps you’re familiar with Windows applications. In particular, you may remember what used to happen when you installed two applications, each with a different version of the same shared dynamic linked library (DLL). The applications would often get confused as to which version to load, causing one or both applications to fail.

This problem was compounded by poorly written installers (that is, nearly all installers), which would overwrite a new version of a DLL with an old version. Uninstalling applications was even worse since you needed to reference count shared DLLs to avoid removing them prematurely, which only the best installers did well (that is, almost none).

The net result was that installing a PC game or utility would break a completely unrelated productivity application, and vice-versa. Many DLLs were shared by multiple applications, so there might be only one install sequence that could make all the applications work, or perhaps no sequence would work if one of the applications used a new version of one DLL and an old version of another. We had a name for this situation: “DLL Hell.”

These days Windows prevents installers from overwriting shared DLLs and runs different versions of DLLs at the same time, giving each application the version it wants. Thank goodness we’ll never see DLL Hell again—right?

Eric Aside

I’m only giving a brief overview of DLL Hell. Follow the Wikipedia link if you’re interested or a masochist.

Synchronicity

Most services run on multiple machines—some run on thousands of machines. When you deploy a new version of a website or service to production, the update isn’t instantaneous. There will always be a time when some clients will be using the new version and some will be using the old version, unless you shut down your service during deployment (yuck!). What’s more, your website or service probably depends on other services that also independently update in a mixed way. Is this problem sounding familiar?

Unlike Windows, no one controls the World Wide Web operating system. There’s no one protecting old services you depend on from being overwritten or rolled back. There’s no one insisting that services run multiple versions side-by-side.

“Wait, what if you use a virtual IP address in front of a load balancer? You can take machines out of rotation, update them, and then put them back in.” Clever, but what about the first set of machines you put back? Unless you shut down your service and let all the connections drain, there will be several minutes when both the new and old versions of the service are in use. The same problem happens when using Azure’s DNS flip. So, what can you do?

Forward and back

Mixed environments mean you must design new versions of your service to work with old clients (backward compatibility) and old versions of your service to work with new clients (forward compatibility). Shared resources must be forward and backward compatible as well—images, media, xap files (Silverlight), JavaScript, and CSS—because they will all be shared between old and new versions. If that sounds a bit subtle or tricky, then you’re beginning to understand why incompatibility is such a pervasive service design flaw.

Luckily, most modern services use versioning in the service contract to ensure the right version of interfaces is invoked and/or run multiple versions of services side by side using exposure control. If you use exposure control as I described in There's no place like production, it’s relatively easy to direct traffic to the right version of your service. Then you can deploy new versions of your service without worrying about breaking old clients. That is, if you forget about the data. Oh yeah, that’s right—the data.

Commander Data

Many services manipulate data—client data, application state, and shared settings. Naturally, this data is shared across all roles within a web service or application—it provides the context and persistence to allow clients to do meaningful work with a service. Purists will tell you that services shouldn’t carry application state and shared settings should never change. That’s nice, but not always practical. Even if it were, there’d still be client data.

Remember that production is a mixed blessing. There’s always a time when a mix of old and new versions of a service is running. That means new versions of a service will run against old data, and old versions will run against new data. Don’t see the problem yet?

Let’s say you update your data schema. Your old code doesn’t understand the new schema, so first you deploy your new code, which you’ve written to understand both the old and new schemas. (You’re so clever!) Next you update the data schema and let your service run, until you realize there’s a horrible bug in the new code. So you must roll back to the old code. Uh oh. The old code doesn’t work against the new schema. You can’t roll back the database because you’d lose data that changed since the update. Updating your data schema has resulted in either having a horrible bug in production or losing client data.

Let’s say you have a setting that disables an incomplete or malfunctioning service feature—such settings are typically shared and dynamic in order to kill features that unexpectedly misbehave. Your new code has a fix for the feature, so you’ve changed the setting to re-enable it. Your team has also added other settings for other new features. Naturally, you deploy the new settings first—otherwise the new code wouldn’t have its needed settings. (You’re so clever!) The old code grabs the new settings, re-enables the broken feature, and yowsa!

Eric Aside

Many thanks to Alexey Kouzmitch, who broke down these scenarios for my team.

The double switch

The issue of a setting change impacting old code is easy to fix—either always use new settings (basically version your settings), always deploy settings with roles (don’t make them shared), or always enable features as a post-deployment step.

The issue of updating your service data schema is slightly more complex. Schema changes must be forward and backward compatible—they work with the new code and work with the old code should a rollback become necessary. How do you do this? Either don’t make breaking changes to schemas (don’t edit/delete columns or stored procedures, only add them) or spread the schema change over two deployments.

If you must make breaking schema changes, first deploy new code that can handle the old and new schemas. Let it run a while to ensure it’s stable and functional. Later, install the schema changes in the next deployment. If that new deployment is faulty, a rollback will take you to the previous deployment, which works with the new schema.

Eric Aside

You can handle service API changes in similar ways. Instead of versioning the service contract, you just add parameters to interfaces using backwardly compatible default values. If you must edit or delete parameters, you do so in a new interface and keep the old interface around. If you must make breaking changes to existing interfaces, you do so using two deployments, just like the case of the breaking schema change.

A plan comes together

Even with the best of plans, there are times when a new service version can’t be compatible with the old version. An example of a recent disaster was when a third party made a breaking schema change to its service that also contained a security fix. We had to take the change right away, so naturally the service was down as soon as the schema was updated.

In cases where downtime is unavoidable, let operations and all other stakeholders know at least a week beforehand. That way, you can brainstorm ways to minimize the impact and properly communicate that impact to clients in advance.

Get in the mix

Production is a mixed blessing. You can control nearly every aspect of production, but we live in a rapidly changing world, and those changes mean old and new software are frequently intermingled. Thus, your service changes need to be forward and backward compatible.

The good news is that service changes don’t need to be backward or forward compatible for more than two or three revisions, and there are straightforward mechanisms for ensuring your service continues running smoothly during new deployments. However, if your development and test teams only focus on the new version, your next deployment is likely to create widespread panic.

Don’t just be smart—be smarter. Think ahead and behind two deployments. Test strange configurations as necessary. Patiently deploy data changes. And always put the customer first. After all, when the customer isn’t first, neither are we.

A change would do you good

ericbrec — Mon, 01 Aug 2011 06:00:00 GMT

Few Microsoft engineers change positions between mid-May and mid-August—they don’t want a role change to adversely impact their annual performance ratings, which lock around mid-August. Of course, managers shouldn’t allow position changes to unduly impact ratings. Then again, eating a taco from a roadside stand should not result in dysentery, but sometimes it does.

Once ratings are locked in mid-August, there’s usually a spike in people transferring amongst teams. Should you transfer? Don’t be a fool. Determine if it’s worth it.

Transferring jobs costs you career momentum, especially cross-division transfers (details below). If you hate your boss and your team, a change may be due. If you’re stuck at the same level with no growth in sight, a transfer may be necessary. If you just want to try something new or play with some cool technology, then I hope you like your current level because you’re staying there a while.

Transferring jobs costs you career momentum—it had better be worth it. If you like your current team and manager, and you’re making good progress on your career goals, stay put! Enjoy! Don’t transfer arbitrarily—it will set you back. How much? When is it worth it? Glad you asked.

How long will it take?

Job searches take about a month, maybe two, for entry and intermediate engineers. Senior engineer searches can take a few months, and principal engineer searches can take four months or more. Likewise, it usually takes longer for principal engineers to acclimate and return to full productivity (six to nine months) than senior engineers (three to six months) or entry-level engineers (one to three months). That’s because principal engineers are expected to impact the entire group, and it takes longer to learn the entire group’s social and technological dynamics.

Eric Aside

You can read all about these Microsoft career stages in my Level up column. The times I give are rough based on ranges I’ve seen with friends, colleagues, and personally over the past 15 years.

Adding up these time spans, a job transfer can cost up to five months for an entry-level or intermediate engineer, nine months for a senior engineer, and a year or more for a principal engineer—and that doesn’t count the potential impact on annual review rating. Surprised? Wake up! Transfers aren’t free.

If you are a senior engineer and transfer in February, you may not be adjusted to your new team by June, when annual calibration is done. You may need an additional six months to regain your momentum after a poor rating. I talk about how to establish yourself quickly on a new team in The new guy, but timing of transfers is an important consideration.

Transfers within your division or, better yet, your current group cut search and acclimation times significantly. People are more likely to know you, and you’re more likely to be familiar with the people, their engineering methods, and their technology. All of this makes landing a role and attaining full productivity faster. The basic rule is, “The further away you travel, the longer it takes.”

Quit stalling

Hopefully, you are enjoying your current role and have a supportive manager who’s helping you meet your career aspirations. If not, you’ll want to compare the transfer cost I just outlined with the cost of staying put. If you are in hell, then you probably want to transfer at nearly any cost—but what if your career is stalled?

You’re stalled if you’re not achieving your career goals in terms of level, skills, or role—whatever is important to you. Will your stall last longer than the search and acclimation of a transfer? That depends on the kind of stall: a crowded elevator, out in the void, or a caged animal.

Crowded elevator

If you’re seeking a senior or principal promotion at the same time as several other members in your team, your elevator to the next level is pretty crowded. You may not be able to get in.

A team with 20 members will promote roughly six people a year. This rate is based on a bunch of variables, like promotion budget (typically around 2 percent) and the size of promotion raises (typically around 6 percent, but can vary widely based on current salary). Out of those six available promotions, you’d expect perhaps one or two senior promotions, zero or one principal promotions, and the rest entry-level or intermediate promotions. Thus, if you and three peers are due for a senior promotion at the same time, two of you are likely to be stalled for six months at least.

Even if only two people are due for promotion, they may not both be promoted—budget is a necessary but not sufficient condition. There also needs to be a business need and readiness of the people involved.

If you are stalled due to a crowded elevator, you can…

Work with your manager to ensure a spot for you on the elevator (fast, but risky).
Wait a year for the elevator to clear out (costs six to 12 months, and others could take your place).
Hope other people transfer, leaving space in the elevator (risky, and may cost a year).
Transfer to a group with a larger or less-crowded lift (risky, and costs search and acclimation time).

Your best choice is the option most likely to get your career moving in the least amount of time. Be honest with yourself about the risks, and choose wisely.

Eric Aside

Concentrate on one promotion stage at a time. If you’re looking for a senior promotion, don’t worry about the principal stage elevator being full. That’s far enough away for many factors to change by the time it’s your turn.

Also consider how important promotion velocity is for you. If you have unrealistic expectations, you may never be satisfied. If you enjoy your current role, you may not want to rush into a larger one. Either way, talk to your manager about your goals. At the very least, you’ll find out what your manager is thinking.

Out in the void

If your team is working on a fringe project—one that isn’t part of a major product line, doesn’t ship, seems like a pet project, or involves unusual work for Microsoft engineers—you’re out in the void, and career growth may be limited. Basically, there’s only so much investment Microsoft puts into fringe projects—you don’t want to be caught unaware.

Being out in the void isn’t necessarily bad. Often these projects provide unique growth experiences that can accelerate your career. However, if you stay too long, you might find yourself stalled or your project canceled. Usually, a two- to three-year stay is enough to gain experience without significant risk.

Caged animal

The worst situation is when your manager prevents your career growth—you become a caged animal. While this is relatively rare, it can happen. Sometimes it’s not intentional. Perhaps you’ve outgrown your current role, but you and your manager don’t realize it until it’s too late. Now you’re stuck.

While transferring out of your cage may seem like the obvious solution, consider your situation carefully first.

Is your manager about to leave? If he leaves in less time than a search and acclimation would take, you might be better off staying. Who knows? You might help fill the leadership void.
Was the cage unintended? If so, you and your manager can probably identify opportunities opening up in the near future on your current team.
Was the cage one of your own making? If so, transferring won’t help. You need to deal with your own demons directly before throwing them at someone else. Remember, focus on your strengths and mitigate your weaknesses in order to grow.
Do you work for a filthy barnyard animal that plays favorites, lacks integrity, and garners little trust or respect? If so, by all means, leave as soon as possible—involving HR as appropriate.

No one should remain a caged animal. Release yourself and get your career back on track.

Leaving on a jet plane

What about transferring outside of Microsoft? Putting aside my belief that, even with all its faults, Microsoft is an awesome place where you can truly change the world, transferring to another company is an option that will certainly impact your search and acclimation times. Your search takes longer because interviews are tougher to schedule, there’s often travel involved, and there’s more paperwork. Your acclimation takes longer because you may need to move to a new city, the corporate culture will be quite different, and the technology may be quite different. Be prepared for your transfer time to be as much as double what it would within Microsoft. I hope that new job is twice as good.

Eric Aside

Personally, I’ll never leave Microsoft for another software job. Allow me to quote from my introduction to the last chapter of my book, Hard Code:

I love Microsoft. Having worked for academia (RPI), the government (JPL), small business (GRAFTEK), mid-size business (SGI), and large business (Boeing), Microsoft is by far the best place I’ve been. For all its faults and gaffes, Microsoft has three amazing strengths:

The people in charge earnestly want to make the world a better place…through software. At every other company I’ve known, the approach is the opposite: be successful in a field and hope it improves the world.
The people in charge hire the best and then fundamentally trust them at the lowest levels to make decisions and run the business. Naturally, this has led to some problems over the years and forced checks to be put in place, but empowerment is more than a nice buzzword.
The people in charge embrace and encourage change and then have the patience and perseverance to see it through. So many companies fight to keep their comfortable status quo. Microsoft fights its size and legacy to constantly improve and adapt to the world it helped shape.

Keep moving

The bottom line is that you never want your career to stall for long. If you are stuck with little chance for growth due to a crowded elevator, being out in the void, or being caged, take action to open your opportunities.

Sometimes creating opportunities means transferring jobs. Sometimes it means altering the situation in your current role. You might also consider a discipline change or change in area ownership. Keep your options open and your career moving.

Since many divisions are now functional organizations where movement within discipline is easy, the perfect time to recharge your career may be at the end of a ship cycle. Often divisions encourage people movement at the end of a cycle, so take advantage of the short search and acclimation times available.

Take charge of your career. Avoid chasing the latest trends or mimicking your peers. Instead, be thoughtful about your career. Choose to be around people you respect, trust, and enjoy. Choose a path that leads toward the growth you seek. Prepare for success, and it will be yours.

Eric Aside

If you decide to transfer, be sure to check out Get a job for advice on finding the right role and The new guy for advice on establishing yourself in your new group.

Out of calibration

ericbrec — Sun, 12 Jun 2011 11:01:00 GMT

It’s calibration time at Microsoft. Time for managers to rank everyone in your peer group (same discipline, same career stage, same division) into five (and a half) ranges: the top 20 percent (and top 5 percent), the near top 20 percent, the middle 40 percent, the lower 13 percent, and the bottom 7 percent.

Calibration brings out the best in us—the best in our acidic, reproachful disdain. Engineers hate calibration because it’s not fair to great teams, in which everyone deserves high ratings, and because it discourages teamwork, since team members compete against each other for rewards. Managers hate calibration because it forces them to make hard choices, it punishes them for having a great team while rewarding their peers for having poor teams, and it creates uncomfortable conversations with their employees.

Well, I love calibration. That’s right, I love it! You weenies and whiners can go join some puritan, petite startup, while I count our billions and continue working with a top-notch staff. Hey, I’m huge on rewarding strong teams and teamwork. The fact that you think calibration discourages teamwork shows your ignorance. It’s time you got a clue.

Wisdom of crowds

There are teams and then there are divisions. They are not the same. A division has thousands of engineers. A team has between one and 12. You are calibrated against your peers in your division, not your team.

Yes, the few engineers on your team that are in your discipline career stage are among the hundreds in your calibration group. So what? That’s rounding error. You’re not competing against your teammates—you’re being compared across your division.

“Yeah, but my boss says every team has to fit the curve!” Typically, group managers like their entire teams to fit the percentages as a starting point. I used to misunderstand this, thinking it applied within career stage and would persist in calibration. Now that I’ve been through many calibrations, I realize it’s like the initial guess in a root-finding algorithm. You’ve got to start somewhere, but it’s rarely where you end up. By starting with teams roughly meeting the percentages, you at least cover the common cases quickly. Nonetheless, managers still talk about everyone in the calibration group.

Eric Aside

Employees often complain that the HR text descriptions of each rating range don’t match the actual definitions—the percentages. True, but those definitions are a handy guide for managers to determine a starting point for calibration.

What are you trying to say?

A common concern is, “Instead of rating teams against their results, we’re rating engineers against each other! Doesn’t that discourage teamwork?” No, it doesn’t. Remember, you’re compared against all the engineers in your discipline and career stage across your division—not just the few on your team. If you and your teammates perform better than others in your division because you collaborate well, then you and your teammates will rank higher.

Don’t get me wrong—managers can certainly use calibration to create a competitive environment within their teams, making them dysfunctional. But managers can create competitive, dysfunctional environments any number of ways. I discussed this at length in “Beyond comparison” (Chapter 9).

Calibration doesn’t discourage teamwork. However, calibration does have a message for employees—Microsoft pays for performance.

If you are not performing as well as other engineers in your discipline at your career stage, then you will not be paid as well as your peers. If you perform better than other engineers in your discipline at your career stage, then you will be paid extra—sometimes a great deal extra. True, “perform better” is subjective, and any subjective system can be abused, but managers are calibrated too. In the end, Microsoft seeks to reward sustained excellence.

We must prepare

How does the actual calibration meeting work? How do managers decide who is “performing better”? It starts with preparation. HR provides a standard spreadsheet that has tabs for every employee and a main table to capture the calibration results. Because it’s a spreadsheet, metrics are automatically calculated to help groups understand the distribution of performance.

The tab for each employee in the spreadsheet is filled out in advance of the calibration meeting. HR puts in the employee’s past review results and basic information (such as the employee’s name, level, discipline, and date of last promotion). The employee’s manager adds:

What the employee accomplished against and beyond his or her commitments.
How the employee accomplished those results. (Did the employee make friends or enemies along the way?)
Proven capability the employee has demonstrated in the past (for context).
Feedback the employee has received from peers.
A promotion indicator with explanation.
The recommended choice out of the five ranges.

Naturally, these tabs are easier to fill out after employees have submitted their own assessments and after the manager has received peer feedback. That’s why everyone is encouraged to fill out assessments and feedback at least a few days before calibration. Yes, I know that doesn’t always happen. (That’s another reason to talk to your manager regularly.)

Getting to know you

The actual calibration meeting is usually six to eight hours long (no joke). Typically, names of employees are put on 3x5 cards or Post-it notes. For each career stage, the cards of employees in that career stage are placed in one of the five ranges as a starting point.

One range at a time (typically highest first), managers talk about every employee in that range. In addition to describing what, how, proven capability, and feedback for each employee, the managers talk about why they feel those aspects align with the selected range. Other managers typically ask questions, particularly if the reasoning doesn’t align with their own. In cases where employees clearly don’t fit the initial choice of range, they are moved up or down accordingly, regardless of percentages.

Once everyone in a range is discussed, the number of employees in the range is compared to the target percentage. If there are too few in the range, managers discuss who might move up as they review the next range. If there are too many in the range, managers go back and further question the employees that seemed to fit least solidly in the range. This all continues until every range is discussed and the calibration model is complete.

That's a bit extreme

A naïve manager exclaims, “But I’ve worked hard to create a high-functioning team. They should all be in the top range!” Congratulations! Is there no room for improvement? I’ll bet there is.

Remember, you aren’t comparing teammates to each other—you are comparing employees to their peers across the division. Usually, you’ve got a mix of engineers that work well together. Reward each accordingly, and help every engineer become the best employee possible.

“But what about lame managers and lame teams? They all should suffer!” That’s harsh—calm down. But there are bad managers who hire bad people, yet present them as average or good people. These managers manipulate the system until they are caught.

Even the best of the worst managers get caught within a few years—usually faster. They are replaced or their teams are disbanded. No system is perfect, but ours does get things right, given time, and my experience is that the process has gotten better since Microsoft started focusing more on management excellence.

That's not fair at all

The naïve manager laments, “But what do I tell a solid employee who was in the bottom 7 percent? He completed all his commitments.” Perhaps the commitments were too easy for his level—but what’s done is done. The employee is still in the bottom 7 percent. He is not getting a bonus, a raise, stock, or a promotion. Instead, he is getting a tough message about moving up or moving out.

“But he’s a solid employee. How is that fair?” You’ve got a solid employee who’s not nearly as good as other engineers in the same division, in the same discipline, at the same career stage. That means you can replace that employee and likely achieve better long-term performance. So, either your employee can substantially improve, or he can find another place to be successful. I describe this in detail in “The toughest job” (Chapter 9).

Eric Aside

It’s important to have appropriate commitments for your career stage—achievable yet challenging enough to meet expectations for growth. Some divisions calibrate commitments within career stages by having managers review the commitments of their staff with their leads and with their peers. You can read more about writing great commitments in “I'm deeply committed.”

I’m still here

Yes, Microsoft compares employees against other employees in similar roles at the same career stage. Microsoft pays for individual performance. But why not pay for team performance instead, or at least in addition?

Personally, I’d like an element of my pay to be based on my team’s performance. Perhaps it will someday. However, I wouldn’t want all my pay to be team-based. I work for Microsoft, not my team. When I switch teams, I’m still working for Microsoft. My pay needs to be at least partly based on my own performance, not my associates’.

I also believe Microsoft could better recognize and reward the wide range of personality types and skill sets needed to create a high-functioning team. We need to find more ways to ensure that teamwork is recognized when it enables as well as when it produces.

Even with its imperfections, Microsoft’s system of pay for performance sustains the top-notch engineering workforce that I have the pleasure and privilege to collaborate with every day. That quality wouldn’t exist if calibration didn’t force us to have hard conversations and value the best among us. I love it.

Eric Aside

The percentages ranges I mention at the start of the column are part of the new review model introduced this year. In my 16 years with the company, I’ve seen three review models:

2.5 – 4.5/A – C: The model when I started in 1995. A 2.5 meant you were done—up or out within 6 months. A 3.0 was a warning—you’re off your game. A 3.5 was standard goodness. A 4.0 was great, and a 4.5 was wow! The stock ratings (A best, C worst) were hidden and unknown unless you were a group manager. Large divisions had a discipline calibration. All organizations had a cross-discipline calibration (in large divisions it followed the discipline calibration). The approach varied by vice president, and your rewards were determined separately from your ratings. Reviews happened twice a year.
Underperformed, Achieved, Exceeded/20-70-10: The model we’ve had since 2006. Theoretically, the commitment ratings were not subject to percentages, but in actual practice they did conform to rough percentages in order to differentiate bonus amounts within a fixed budget. The contribution rankings were transparent and directly corresponded to percentages—top 20 percent, middle 70 percent, and bottom 10 percent. All divisions had a calibration based on discipline and career stage. Some organizations followed with a second cross-discipline calibration by career stage. The approach varied by vice president, and your rewards were determined separately from your ratings. Reviews happened once a year.
1 – 5: The new system starting this summer. A 1 means the top 20 percent, 2 is the next 20 percent, 3 is the middle 40 percent, 4 is the lower 13 percent, and 5 is the bottom 7 percent. There is no separate stock/contribution ranking. All divisions have a calibration based on discipline and career stage. The approach is dictated by HR (though I imagine different vice presidents will introduce small variations), and your rewards are directly determined by your rating (with the very top 5 percent receiving extra). Reviews happen once a year.

Personally, I like the new system. We’ll have to see how it works out, but initially I like the simplicity of a single rating, that it’s on a curve that better matches historic percentages, that there’s no cross-discipline calibration (made no sense), and that there is a the standard approach across divisions. And I am overjoyed with the direct, fixed, and transparent tying of rewards to ratings.

I can think of three more improvements beyond the ones I mention in the column.

Have six ratings (a separate one for the top 5 percent). With everything else so transparent, why hide the top 5 percent?
Get credit toward your bottom 7 percent if you worked with HR to dismiss or implement a career change for struggling employees during the past review cycle. You can’t claim people who simply transferred or left—HR has to have been actively engaged regarding these folks who would have otherwise gotten a 5 rating. This would encourage managers to connect with HR and actively manage performance issues.
Do calibration twice a year, and share the midyear calibration rating (perhaps tied to a semi-annual bonus, as it used to be, or used just as a tracking number). As a manager, I hate that I can’t provide my employees an unambiguous message about their calibrated performance at midyear. I can tell people they are in jeopardy, but psychologically that’s not the same as giving them a number.

Quality is in the eye of the customer

ericbrec — Wed, 01 Jun 2011 06:00:00 GMT

Not every bug is the same. A bug that frequently freezes an app gets more attention than an extra line of green pixels in a border. An embarrassing typo in a prominent feature is more urgent to fix than an inappropriate exception thrown by a misused API. Customer data loss or compromise requires instant attention, while a rare app crash can be fixed later. We adjust our approach to bugs based on the bug.

Engineers get bug priorities, so why are they clueless about test priorities? We approach testing credit card transactions the same way we approach testing web page layout. Of course, we spend more time scrutinizing credit card flows—we’re not complete idiots—but we go through the same test planning, test case development, test automation, and test triage for both. That’s crazy.

Not all features are viewed by customers the same way. Some are critical to quality—like those protecting customers or ensuring a smooth customer experience. Some are what my dad calls “fruit salad”—nice to do well, but not as important. In these days of the Internet and cloud services, there’s no reason to continue testing everything the same way.

You've only got one shot

It used to be that we got one shot at releasing a quality product. Even then, our packaged product releases would have numerous issues that wouldn’t receive attention until the first service pack was released six to nine months later.

These days, our packaged products are much better. They are of higher quality from the start thanks to instrumentation, early bug fixes, and integration of only those features that are complete and tested using practices like Scrum and feature crews. Packaged products now go through multiple layers of internal and external previews and betas that resolve most issues before final release. These products also integrate with services that can be updated and improved after release.

Our websites and cloud services are even more flexible. Many of them update monthly or even daily. This rapid iteration and release cadence unveils new approaches to quality and testing that previously were not practical. But we have to break with the past before we can embrace our future.

I'm a beta unit

Customers aren’t shocked when a beta is less than perfect. Some imperfection here and there is fine, as long as those imperfections don’t compromise customer trust and don’t disrupt the overall customer experience.

Customers take the same “beta” attitude toward the Web in general. Connectivity is shoddy and browsers are finicky, so customers expect Web properties and applications to be a little flaky. That’s fine as long as websites add value, perform quickly, and protect customers from hackers or misuse.

Customers are particularly enamored with betas and Web properties that improve regularly in response to customer needs—spoken and unspoken. You see this all over forums and customer feedback.

There lies the opportunity to change our approach to quality and testing for betas, websites, and cloud services.

Focus on protecting trust and overall customer experience rather than specific features.
Focus on repeatable issues rather than flaky issues.
Focus on being responsive to customer needs rather than one-time perfection.

My experiment worked!

We can’t compromise on security, privacy, or reliability (trustworthy computing areas), so we must continue their thorough, in-depth, pre-release testing. The overall customer experience must be delightful and compelling, so end-to-end scenario testing is critical.

Therefore for rapidly updated products and services, the test team should focus its in-depth work on trustworthy computing and the overall customer experience—including penetration testing, injection testing, stress testing, and end-to-end scenario testing. All other areas can rely on design and code reviews, code analysis (like compilers and FxCop®), unit and component testing, and system sanity checks.

Yes, this puts more responsibility for initial quality on developers. That’s good. Testers should be focused on critical areas and overall quality assurance, and developers should take more direct responsibility for quality. This change is not a big risk for betas and Web properties because customers prefer that we be responsive to what they truly care about rather than perfect about everything. Now we can ship more frequently and actually experiment with different solutions for customers—just like Amazon has been doing for years.

Microsoft old-timers might ask, “But what about those nasty bugs that take so long to find?” That’s why we differentiate between flaky issues (Heisenbugs) and repeatable issues (Bohrbugs). Heisenbugs are the nasty ones because they are so unpredictable. However, they are also the issues that customers dismiss and that websites and cloud services can easily retry and resolve.

Eric Aside

The terms Bohrbug and Heisenbug date back to Jim Gray’s 1985 paper, "Why Do Computers Stop and What Can Be Done About It?"

So instead of dragging every feature through test planning, test case development, test automation, and test triage, we focus on being responsive to customer needs, protecting customer trust, and assuring an overall great experience. We ship frequently with constantly improving customer solutions that are well instrumented, so the product team can analyze usage patterns and continue the virtuous cycle.

There’s no place like production

It’s the old-timers again: “Wait a minute! You can’t be serious about releasing untested code!” First of all, it’s not untested. It’s design and code reviewed, code analyzed, unit and component tested, and sanity checked. The trustworthy computing areas are fully tested pre-release and the end-to-end scenarios are validated. That’s more testing than most of our competitors do.

However, the old-timers are right. There is still significant functionality being released without the traditional heavily planned, managed, and executed test pass. Shouldn’t we be worried? No, we should be embarrassed to think all that extra overhead was worth it. Our test labs aren’t the same as customer systems. As I wrote last year, There's no place like production.

We roll out changes to only part of the whole customer base at a time through betas or exposure control. We validate that features are working in real-world scenarios. We roll back if there’s a serious issue. And we focus on continuously improving, using the real customer feedback and usage data that would otherwise be guesswork.

Have you decided to join the 21st century?

We can’t change our testing approach for all products. The change works best for betas, websites, and cloud services. However, we owe it to our customers to utilize modern, feedback-based, rapid iteration methods; grow our testers into quality assurance, trustworthy computing, and experimentation experts; and put more responsibility on our developers for quality.

It’s time Microsoft joined the modern era of software development. With all our years of enterprise experience, we can lead the industry with quality, while delighting our customers with rapid updates that address their every spoken and unspoken need.

Test don’t get no respect

ericbrec — Sun, 01 May 2011 06:00:00 GMT

I love Microsoft®. We’ve been together happily for many years. If you’ve been in a healthy long-term relationship, then you know what this means—there are things about Microsoft that make me curse, stomp, and spit. I’ve learned to tolerate them, but they still make me cringe.

A prime example is our disrespect for critical disciplines like testing. The test discipline is one of the two largest engineering disciplines at Microsoft and one of three key engineering triad disciplines. How can we not grant testers the same respect and opportunities we give the other two engineering triad disciplines—program managers (PMs) and developers? Perhaps our history provides the answer.

Microsoft was started by developers and run by developers for years. We’re now run by a former program manager of sorts, so PMs receive begrudging respect. Developers can’t draw and have no sense of style, so designers are becoming appreciated. Developers hate writing, so content publishing is at least considered necessary. Localization and media are magical things that just appear in the build. But test? Developers believe testing is easy, if not remedial, so developers think testers are beneath them.

Eric Aside

Please note that I’m not giving any of these disciplines proper credit for all the work they do. I’m merely listing the superficial ways they are often viewed by developers.

It’s a different kind of flying altogether

Since developers think testing is easy compared to development, they think they can do a tester’s job. After all, isn’t that the cool, agile way? Aren’t we all just software engineers? Yeah, and everyone would get along if we just gave peace a chance. Don’t be naïve.

Developers can verify that their individual components work as specified in isolation (unit testing). They have much more trouble verifying that their components work as a system, outside of isolation and spec. Why? Tunnel vision. Developers design and write their code for a certain purpose. That’s the way developers think about it—as they should.

Real-world testing must verify that the code works appropriately when used in ways that defy all logic and purpose. To test code properly, you need to completely forget how and why it was written and instead think about how it might be used in arbitrary and even insidious ways. Developers don’t think that way—they can’t think that way and still develop with purpose. Testers do think that way. That’s why we need testers.

That's easy!

Even though the test discipline is essential for high quality software, some groups still consider converting all their testers to developers, expecting the combined development team to write all tests—it’s a great way to get more developer headcount! The teams that actually go through with this change experience the following problems:

They lose their test leads and leaders. The former development leads tend to lead the combined engineering teams. (I talk about why in the next section.) This relegates the former test leads to individual contributor roles (ICs), who usually don’t take this reduction in responsibility well and leave. The former top test ICs also tend to leave—after all, clearly their skill set is not appreciated, no matter what sparkly purple lipstick the management team might put on this pathetic pig of a plan.
They lose their testers. After the test leads and leaders go, the former test ICs gravitate toward a development mindset. They gravitate there because testers get no respect, they are calibrated against developers even if they continue to work differently as testers, and they can see there’s no career growth in test. After all, their role models left the team. Now the only path up is through development.
Team morale drops, especially among the testers. Losing team members and team leaders impacts morale. The most impacted are the testers still clinging to their discipline who lose their role models, their self-identity, and their solid reviews. (Even great apples get bad reviews in an orange grove.)
Their code quality is initially higher and then gradually drops. The quality improves initially because developers suddenly realize they’ve lost their safety net and actually start writing unit tests, doing design and code reviews, and paying attention to those build warnings. (Of course, developers should have been doing this all along.) However, after a while, the system and boundary errors start creeping in and building up. No one is looking for them, so they are discovered by the wrong kind of testers—customers.

Combining development and test makes sense at the unit level (such as Test-driven Development). This practice can also work at the component level for well-understood and well-factored components on teams that also have strong QA. However, combining development and test doesn’t make sense at the system level. I talked about this at length in “Undisciplined” (chapter 4).

I just can’t get enough

The lack of respect for testers is most apparent in leveling and career development. Depending on where they work in the company, testers are ranked one to three levels below developers and PMs. In other words, the test counterpart to a PM and developer who are all working on the same project with the same scope will tend to be a full career stage below his or her peers. Unbelievable!

“Yeah, but testing isn’t as difficult as development and program management,” says the dim developer. Really? Try it sometime. Try writing automation that works every time, even as developers alter configurations and data. Try performing penetration testing that closes gaps that foil sophisticated hacks. Try producing injection testing that discovers failure modes, system testing that finds sneak conditions, or end-to-end scenario testing that validates the billion-dollar bets we make. And I’m just scratching the surface.

“Yeah, but our test team isn’t that advanced—they don’t do all those things.” Exactly! We don’t value the test discipline enough to advance our testing capability to the same levels as PM and development.

Testing is frigging hard! We need great people to do it well, and then we need to pay, develop, and expect them to be world class. Our treatment of the test discipline is astonishing and pathetic. Believe or not, it used to be even worse until senior test leaders from across the company started tracking the numbers and driving promotions.

Yes, it’s true that testers do different jobs from PMs and developers. But we can’t design and construct complex systems and think that testing those systems will be any less complex. You get what you pay for, and by spending less on testing, we create imbalance. We sacrifice quality, productivity, and efficiency as a result. The sacrifice in quality is obvious. The sacrifice in productivity and efficiency comes from incomplete and fragile testing that results in higher error rates, more rework, and higher support and sustained engineering costs.

Eric Aside

The refrain I hear when I complain about our commitment to testing is, “Yeah, but Google and Amazon have far fewer testers.” Amazon carefully tests the systems that matter, like billing and account management. Google does endless analysis of its search results.

As for Microsoft, we are in a broader business than Amazon or Google. We are a platform company and an enterprise company. Our customers expect more from us. Quality is a differentiator and a necessity. While we don’t need perfection in all things, we do need the right quality in the right products.

The world just wants us to fit in

It’s bad enough to give up quality, productivity, and efficiency by expecting less from our testers than we do from their PM and developer counterparts. What’s worse is that lowering expectations sends a clear message that to get ahead, a tester needs to become something else. This philosophy is nothing short of irresponsible and tragic.

Lowering expectations for testers is irresponsible because some of our most challenging engineering problems involve testing—testing in production, testing many-core highly-parallel systems, testing 1000+ machine services, testing globally distributed cloud services, testing secure and private cloud systems, testing hybrid procedural and functional languages, testing natural user interfaces, and on and on.
Lowering expectations for testers is tragic because we send a message that testing is not a legitimate career path, when in fact it is a robust career path to the highest engineering stages for both test managers and test ICs. Instead of following this path, testers abandon their discipline to do something else, often with mixed results.

By expecting less from our testers, we are encouraging them to move away from a career they love—one that is essential to Microsoft’s success and offers tremendous opportunity—and toward a career not of their choice that may inhibit their growth. It is a travesty.

Eric Aside

Microsoft recently announced changes to its compensation plans that increase the base salary for most engineers, including testers. That’s wonderful and I’m grateful. Investing in our test discipline is a separate matter. It is choosing to be just as sophisticated in testing as we are in program management and development.

Executives might reasonably ask, “Why would we increase our spending in testing? What’s the return on investment?” While I believe a strong financial return exists, I’d turn that statement around. If we want to save money, why don’t we decrease how much we pay developers by a couple of levels? Because our quality and innovation would suffer. Why would they suffer? Our products are platforms with wide and varied usage that require sophisticated engineering to orchestrate and improve—and test.

Tell me what it means to me

At a time when we should be investing in test, we continue to demean the discipline. We have great leadership at the highest levels within the test discipline, but far too few testers join these ranks each year. Even though test is far behind PM and development, they receive a smaller promotion budget. (The promotion budget is equal in proportion to the other disciplines, but that proportion is a distortion due to the higher salaries in PM and development.) Only a handful of test ICs and test managers reach the principal and partner stages to serve as role models.

How can we allow a critical and central engineering discipline to be so disrespected and damaged? Are we that vain or foolish to think testers aren’t really needed or the problems aren’t really that difficult?

It’s time we put our money where our priorities are and push testing to the next stage. We hire the best—let’s challenge them accordingly. Let’s lay out the test career path all the way through to vice president and technical fellow. Let’s start aggressively recognizing the talent we have and developing the talent we need. Testing deserves our respect—our customers, our partners, and our business depend upon it.

Eric Aside

What can you do to help if you aren’t a test or multidisciplinary leader?

Accept and appreciate that the test mindset and skillset are different from development, yet their problems are equally complex and critical to our product quality.
Write high quality code from the beginning using design and code reviews, code analysis (like PREfast and FxCop), and thorough unit testing, all of which allow testers to focus on their unique value of providing quality assurance and exposing system issues developers wouldn’t normally detect.
Encourage your test team to hire great fulltime testers that focus on the truly challenging test problems we face—problems that when solved will improve the quality of our products, our testing, and our testers.

BTW, I could write a similar column on service engineers, who are even less understood than testers.

You have to make a decision

ericbrec — Fri, 01 Apr 2011 05:00:00 GMT

What’s worse—a flawed decision or no decision? That’s easy. Decisions keep a business moving. An imperfect decision might move your business slightly in the wrong direction, but at least it will be moving. Make a few adjustments, and you’re back on track.

Making no decision brings business to a halt. Even if you make a great decision during the delay, the time needed to regain momentum will leave you far behind. Like boulders and trains, groups of people have inertia. It takes great effort to get them going, so it’s better to keep them moving forward and adjust, rather than halt progress and restart.

Unfortunately, leaders often don’t have enough information to make a clear decision. What do you call leaders who are indecisive when dealing with ambiguity? Incompetent, ineffective imposters. They are losers, not leaders. If you want to lead, you better know how to make good decisions with partial data.

I’m the decider

Before discussing making decisions with incomplete data, I should point out that being a decisive dunce is no better than being an ineffective imposter. Decisive dunces make decisions quickly (good), but without clear thinking or cause (bad). This results in random or backward progress and an exasperated team. Having too little information is no excuse for being arbitrary or clueless.

Instead, you want to inform yourself as much as time will allow, and then make your decision based on context and experience. How much time do you have? Only until the point your team becomes blocked. How do you apply context and experience? Develop a consistent framework for making your decisions that helps you spot patterns, and then rely on your well-informed intuition to make the best choice.

This is easier to describe with examples. Let’s talk about hiring decisions, product tradeoffs, and triage decisions.

Eric Aside

If you want more time to make decisions, then plan ahead. Map your business rhythm to a calendar, and begin your decision making early in anticipation of events.

If you want better data to make decisions, then put measurements in place, utilize business intelligence, and be better informed.

Both strategies for better decisions require forethought. Don’t like a vacuum? Fill it.

To hire or not to hire

I’ve been an “as appropriate” for a decade, meaning that I am the last interviewer on an interview loop. My role is to make the final hiring decision about the candidate. Sometimes that decision is very easy because the prior interview comments are compelling in one direction or another and they fit my personal assessment of the candidate well.

However, the hiring decision is often difficult. The prior interview feedback is mixed and inconclusive. For an interview, I have at most one hour to gather information directly from the candidate. In addition, the recruiting team and hiring manager are blocked immediately after my interview, so I have only an hour or two to decide.

To make the best possible decision, I follow the same routine each interview. I prepare before the interview by reading the candidate’s resume, checking out any online content it references, and reading the prior interview feedback. I look for patterns in the comments that might indicate the root cause(s) of concerns. Then I interview the candidate with the goal of confirming the root cause(s) and whether or not the concerns can be overcome.

Now, I’ve collected all the facts I possibly can without significantly delaying the offer. I’ve done the analysis, and my intuition is informed. How do I put this all together? I step back and let my intuition guide me. After all the analysis, and all my years of experience, what decision seems right?

As Malcolm Gladwell points out in Blink, an informed intuition is an exceptional instrument. I’m not talking about judging a candidate from a quick glance. I’m talking about learning all I can and then, instead of focusing only on the individual details, I take a step back and looking at the candidate as a whole with everything I know about her and the other candidates I’ve met. That’s hard to do consciously—your informed intuition can be your guide.

Eric Aside

Even if I did take time to call every reference, bring the candidate back for more interviews, and do a background search online, my decision might not be much better. As Gladwell also points out, sometimes the extra analysis and second guessing is worse because you start to ignore your overall assessment and cumulative experience in favor of individual facts.

If I’m still undecided about a candidate, the decision is easy—don’t hire the candidate. You always want to feel confident about the candidate’s potential for success for the sake of the candidate, the team, and the company.

Now consider the alternative

Product tradeoffs can be very difficult to make. There are often many opposing viewpoints and rarely conclusive data. Great data from telemetry and web services can make a huge difference, but data informs—it doesn’t decide (necessary, but not sufficient). You must make the call quickly based on the data and your collective experience or risk losing your market position.

The individual decision may be about a user story, an architectural approach, or even a development tool. Regardless, there will be pros and cons for each choice. First, use your value proposition, architecture principles, or other relevant framework to clarify your options. If you still have a number of valid choices, you can quickly analyze your tradeoffs using one of my favorite tools, Pugh Concept Selection—a fancy name for a simple technique. (I described it briefly in My experiment worked!)

In Pugh Concept Selection, you make a table where the rows are choices and the columns are decision criteria. (Excel® is perfect for doing this.) The result is like a pros and cons table with a twist. Instead of indicating which choices meet the criteria best on some absolute scale, you pick one choice as your default and give it zeros for all criteria. Then you rate all the other choices relative to your default choice. If a choice is better than the default for a criterion, you put a +1 in that row and column. If a choice is worse, you fill in -1, and if it’s about the same, you fill in 0. Add up all the columns, and you have the rating for each row. Now you can see how all the choices compare.

Eric Aside

You can weight criteria (columns) differently to indicate what’s most important. Typically, people use power series for weights (1, 3, 9). To determine the sum a row, you multiply its ratings (-1, 0, +1) by their associated weights and add them together. The Excel SUMPRODUCT function does this nicely. Click here for a sample spreadsheet.

Filling out the table of choices as a group can be fun and insightful. You learn a great deal about what the tradeoffs between choices really are and what you care about most. However, Pugh Concept Selection won’t make your choice for you (though it could reject a bunch). While one option might get the best score, there are likely several top options worth considering.

Once the table of choices is complete and you’ve talked through the results and insights gained, it’s time to make a decision. That’s when you throw out the table and rely on your newly informed intuition. The analysis is necessary and instructive, but it’s your experience and overall context—your intuition—that can take that analysis and sense the right decision.

Tell me why

When you are managing product issues toward the end of a cycle, there are many issues to cover, so decisions must be made quickly. In addition, you are usually making those decisions in a group of at least three people representing different interests, so you must reach a consensus in which no one objects.

Eric Aside

At Microsoft®, this decision-making process is called triage. It can be one of the most divisive activities in the development cycle if done improperly.

How can you quickly convince three or more people to agree with confidence about important issues? You need three things:

§ The root of the issue—without knowing the root cause, you can’t apply your past experience to the problem with confidence.

§ The scope of the issue—without knowing how many customers and partners are impacted and how are they impacted, you can’t judge the benefit of acting.

§ The risk of the issue—without knowing the risks involved, you can’t balance the costs against the benefits.

Once you’ve understood these three things, which every late-stage issue report should contain, you’ve informed your collective intuition. Now, the group decision makers can depend upon their visceral reaction to a suggested resolution. Of course, that’s no guarantee you’ll agree. For more on my framework for these kinds of tense group decisions, read my column “Are we having fun yet?” (chapter 1).

Wait, there’s more

“But what if more information arrives later that significantly changes your thinking?” No problem. We want to make a decision that lets us move forward, not a perfect decision, necessarily. The way we gain insight and experience is to try, receive feedback, adjust, and try again.

That’s no reason to change decisions constantly, however. To maintain your forward momentum, you should iterate, making slight adjustments as you go. Yet, occasionally there are enough new internal insights or external expectations to suggest a substantial course correction. That’s okay—it’s what makes our lives interesting and our business engaging.

Each time you analyze new information, your intuition becomes smarter. You get more experience and a better perspective. You make better decisions. Keep an open mind, listen well to customers and contrarians, apply a consistent framework, and trust your well-informed instincts. Don’t wait to be certain. Decide, iterate, and constantly lead your team closer to its goals.

You're no bargain either

ericbrec — Tue, 01 Mar 2011 07:00:00 GMT

“Can I talk to you about Bozo? He gets on people’s nerves. His communication style causes trouble. He’s bringing the whole team down. He’s a freaking clown.” If you’re a manager, you’ve probably heard this before. Every team seems to have its share of Bozos. What do you do about Bozo? Discipline him? Move him to another team? Fire him? No, don’t be a fool.

Bozo needs to reconsider his behavior—no doubt. If you’re Bozo’s manager, you need to evaluate the situation and take appropriate action. But make no mistake, the problem isn’t just Bozo. The problem is with the whole team.

Let me make it perfectly clear. You are a Bozo. We are all Bozos. No one is perfect. Nobody always says the right thing and behaves the right way. Even if you did, someone would take offense.

Eric Aside

Bozo the Clown was the host of a famous children’s show when I was young and dinosaurs ruled the earth.

The good, the bad, and the ugly

“Sure, but not all Bozos are created equal. The Bozo on my team is completely useless—a disaster. He’s got to go.” Really? Why was he hired? What is his review history? Has he been a Bozo from the start?

Sometimes we do make bad hiring decisions and need to send our clowns to a new circus. I designate such truly troublesome teammates as “negative headcount.” If you were to remove Bozo from his team and not replace him, how would his team perform long-term with one less person? If you believe productivity would be higher and stay higher, because Bozo just adds work to everyone else, then Bozo is negative headcount—he subtracts instead of adds to his team.

Most of the time, however, Bozo is a good employee who’s made some mistakes. Unfortunately, his coworkers aren’t sufficiently patient, understanding, or accommodating when dealing with him.

“Dynamics of Software Development” is one of my favorite books on its subject. It’s written by Jim McCarthy, a former Microsoft employee. Jim’s “Don’t flip the Bozo bit” essay stresses the importance of giving people a chance. The key is understanding the root cause of the trouble and addressing it with Bozo and the team.

Take all of me

Remember, everyone is a Bozo. We are all package deals. You can’t take the benefits of people without also taking their faults. You can fight this notion at your peril, or you can embrace it and accept people as they are.

Why do people “flip the Bozo bit” on others, instead of accepting them as they are? Because it’s so easy and natural to project your opinion of people onto them, instead of seeing them as complete individuals. You believe a coworker, a lover, or a celebrity is wonderful, and you see that person as wonderful. That is, until she disappoints you. Then the bit is flipped and she is intolerable.

Stop projecting. Take a look in the mirror. You aren’t wonderful and neither is anyone else. You are a real, whole, human being—a marvelous complex package of positives and problems. So is everybody else. Accept it, embrace it, and move on.

I'll accommodate you

How do you deal with all of life’s imperfections? You make accommodations.

Certainly, you should talk candidly and courteously to your teammates one-on-one about their weaknesses. In response, they should try to improve or at least help you both adapt. Remember, you can’t fix everything about everybody as hard as you might try. Instead, know yourself and know your coworkers—adjust accordingly.

I’ve had inventive teammates who were flakey, brilliant teammates who were arrogant, and insanely productive teammates who were uptight. I could have admonished the flakiness, but then lost the creativity. I could have browbeaten the arrogance, but then lost the brilliance. I could have completely unwound the uptightness, but then lost the productivity. Instead, I provided structure for the inventive flakey folks, tolerance and coaching for the brilliant arrogant folks, and safe places to work and vent for the productive uptight folks.

For myself, I can’t keep any details in my head for long, so I write myself emails and tell others to write me emails. Everyone has a way of working and dealing with his weaknesses. Ask and learn what works well for your teammates and your management. Then get over yourself and accommodate.

Eric Aside

As a manager, it’s important to recognize when the best accommodation for an employee is to switch teams. Sometimes you’ve got a good person who fits poorly with the team. That’s not very common, but it happens. Instead of a diversity of ideas, you get a clash of styles. After understanding the root cause, encourage your employee to seek a new role. Help your employee find and attain a happier and more successful future.

We've gotta play to your strengths

Why should others compromise and make accommodations for your weaknesses? Let’s say you corrected all your problems, as if that were possible. What would the result be? You’d be ordinary in every area, with no improvement to the strengths that truly differentiate you. You’d certainly get along better with everyone, and that’s quite valuable. However, that’s not why you were chosen for your role. Accommodate and defuse your weaknesses? Absolutely! Focus on weaknesses while starving your strengths? Never!

Your value as an individual rests in your unique strengths. Those are the areas you want to develop and deploy the most. To the extent that your weaknesses inhibit exploiting your strengths, you must correct or accommodate those weaknesses. However, remember who you are and the core of your success. Adapt, and then accelerate your growth by doing what you do best.

Warts and all

Everyone can be a Bozo at times to people in his work life and home life. Give yourself a break. Accept your faults. Do what you can about them, and accommodate the rest. Help your coworkers, your friends, and your family to do the same.

When talking honestly and respectfully to teammates one-on-one about their shortcomings isn’t enough, make accommodations. Tolerate their weaknesses, utilize their strengths, and they’ll likely do the same for you. Yeah, you could argue and complain instead, and sometimes that’s cathartic, but it’s rarely beneficial to anyone.

Swallowing your pride and getting along is healthy for you, your team, and your loved ones. There are always areas where you can’t compromise, but typically they are few. Your life will be less stressful, your relationships will be stronger, and you will be happier and more successful if you embrace others as the whole human beings they are.

Cycle time—the soothsayer of productivity

ericbrec — Tue, 01 Feb 2011 07:00:00 GMT

Nothing infuriates me more than wasted time and wasted effort. I’m not talking about training, reorgs, moves, morale events, or vacations. Those at least have the potential to be valuable in your life. I’m talking about build time, integration time, unused specs, incomplete features, blocking issues, excessive and persistent bugs, and over-engineered code and processes. You know—hours and days you’ll never get back.

I broke down all this real waste years ago in my column “Lean: More than good pastrami” (chapter 2). While I provided examples and suggested solutions for individual areas, I didn’t really map the path to a better life for your particular team. Every situation is different. You need a path that discovers your most harmful waste, drives your team to resolve it, and rewards your team with a visceral sense of relief.

In manufacturing, the secret path to success is reducing inventory. Inventory hides your manufacturing issues. As you reduce inventory, problems appear. You fix the issues, and then reduce inventory further. Gradually, your waste is eradicated, and your efficiency soars. In software development, the secret path to success is reducing cycle time. The shorter you make the time between concept and completion, the more roadblocks you face that have little to do with actual engineering. Fixing those problems unleashes productivity. Let me show you the way to free your engineering soul.

What’s done is done

The first step to shortening your development cycle time is determining how long your cycle takes. For services, it’s the time between releases. However, for packaged products, shortening time between releases often can’t be supported by the market. Thus, a better definition of a cycle is the time between starting detailed specification of a feature and having that feature completed. What does it mean to complete a feature? There’s the rub.

You’ve got to define “done” for your features and for release of your services. Below are the definitions my team uses. We insist the first four are done for every feature and the second four are done for every release. We release the Xbox.com sites every four weeks—and we LOVE IT!

“Done” for every feature:

1. All updated designs and code are reviewed

2. All automated tests written and passed

3. No ship-stopping bugs

4. All monitoring and health checks in place (feedback tools for packaged products)

“Done for every release:

1. All localization and world readiness completed

2. Full test pass completed successfully

3. All quality areas signed off and partners signed off (including LCA)

4. All necessary release documentation completed

As you attempt to shorten the time between starting and finishing work, it’s these eight “done” criteria that expose issues. Let’s briefly discuss the common problems that arise and how to respond.

If you build it

This first requirement for “done,” reviewing updated designs and code, only saves time, so let’s talk about automated tests—unit tests, component tests, stress tests, acceptance tests, system tests, fault injection tests, and so on. Developers and testers should share in writing these tests. Who writes which tests varies by team. As they attempt to shorten cycle time, most teams struggle with their test harnesses and the time it takes to run the tests.

When it comes to reducing cycle time, you’ve got to distinguish between tests that run quickly and often and tests that run slowly and infrequently. Any tests that fall in the middle need to pick a side and be rewritten or refactored.

While it’s nice to have one test harness, you can get away with two—one for fast and reliable check-in tests and one for full test passes. If you’ve got such a big team that even quick check-in tests take more than 10 to 20 minutes, then you’ve probably got a large enough team to invest in test prioritization and parallelization technology.

Likewise, if you’ve got such a large codebase that it takes more than 10 to 20 minutes to rebuild, then you’ve probably got a large enough team to invest in a highly parallelized build lab and build dependency logic. Remember, build, test, and check-in form software development’s inner loop. Anything done to speed up your development inner loop creates a huge multiplier to overall productivity.

As for code branches, you never want to be more than one branch deep from the main branch. Integration is expensive, and each branch level adds another integration layer. Think about it. Say you were building personalized laptops. Having the distinctive components go through customs would crush your delivery schedule. Every branch level is like another customs station between your fixes and features and the main branch.

Roaches check in, but they don't check out!

Cleaning up a large bug backlog before release can really slow down cycle time. Bugs take progressively longer to fix the longer you wait. Design and code reviews plus automated testing will help (as will refactoring spaghetti code and switching to test-driven development). Regardless, what really matters is finding and fixing bugs early. Short cycle times demand immediate bug fixing.

No matter what you do, you’ll still have bugs—we are human. Some of those bugs will be very difficult to find and fix, which will slow you down. The good news is that an architecture that is resilient to failure can alleviate the need to fix the toughest bugs—the intermittent ones. Resilient architectures allow you to collect data on these stubborn, sporadic slipups and fix them once they are finally understood.

Eric Aside

I wrote more about resiliency in one of my more controversial columns, Crash dummies: Resilience.

How am I doing?

Monitoring and health checks are often treated as afterthoughts. This amateur-hour action increases the time needed to track down customer issues, which lengthens cycle time. This is just as true with designing and implementing customer feedback tools for packaged products.

Monitoring and health checks need to become forethoughts, designed in from the start. Consider why you are building your feature, and ask how you’ll know if it is performing as envisioned. That will tell you exactly how to monitor its use and inquire about its health.

All this data and your quick cycle times enable fast feedback loops and constant improvement. Be sure to spend time during every cycle reflecting on what you can do better in your product and your engineering team. My feature teams and leads do this twice every release (every two weeks).

Eric Aside

Making monitoring and health checks a forethought is my team’s most recent addition to the “done” list. Poor monitoring and insufficient health checks caused us to stumble in the fall, while we watched a partner team of ours shine in the same area.

Sign me up

We already talked about automation for full test passes, and localization processes are quite refined and fast at Microsoft, so the next area that typically causes trouble is sign-off. Quality areas (security, privacy, and so on) should be addressed and bugs fixed by all engineers as part of normal feature work. However, sign-off on these areas, as well as partner sign-off, can really slow down cycle time.

Even though quality is the responsibility of every engineer, sign-off works best if one engineer is assigned to shepherd each quality area through its process. Those engineers become the team specialists in their areas, a nice career opportunity for them that provides cross-group scope.

Since team specialists deal with their quality areas all the time, sign-off requirements and activities are far faster and easier for them than for other team members. In addition, team specialists develop relationships across the team and with corporate specialists in their area, which also speeds the process and provides growth for the entire team.

How about a few more details

Using many of the techniques I’ve described, my team has managed to reduce the cycle time for our production releases from a few times a year to every four weeks. It’s fantastic! Before I get into all the advantages we are seeing, let me cover two topics people often question.

How do you develop features or architecture changes that take longer than four weeks? There are two basic approaches: horizontal and vertical.

§ The horizontal approach is to work on the large change a layer of the stack at a time. For example, first make the schema change, then ship the new service, then write the new model, then the new controller, and finally the new view. Each layer can ship within a four-week cycle.

§ The vertical approach is to break the large change into smaller slices of functionality. You then complete each vertical slice end-to-end within a four-week cycle. If the slices lead to a disjointed user experience, you hide the slices from users until enough of them are complete.

§ People often use a combination of horizontal and vertical techniques. Unfortunately, the horizontal approach often leads to over-engineering of layers and hampers iterative feedback. I much prefer the vertical technique, using the horizontal approach only as a last resort.

How do you handle sustained engineering? Sustained engineering fixes usually take about a month from identification through testing and release. Since we ship every four weeks, sustained engineering is just part of our regular work. There are no sustained engineering releases except in the most unusual of cases, and more importantly, there is no special sustained engineering team. We are in it together—we all feel the pain of our mistakes and the joy of our advances.

Life is good

Now that we’ve been releasing every four weeks for the last six months, we’re really feeling the benefits.

§ Much of the overhead that engineers complain about is gone—we had to remove it to succeed.

§ Slipping is manageable—if a feature misses a release by a week or two, it still goes out within a month.

§ Releases aren’t scary or crazy anymore—we do them all the time, and you can cause only so much trouble in four weeks.

§ Our team gets more done in less time—we’re faster due to the streamlining that frequent releases demand.

§ We serve our customers well, and they notice—our dramatically improved response times to issues and feedback is greatly appreciated by our customers.

While there is pain involved in any change, shortening cycle time provides the immediate gratification of less overhead and quicker results. The team loves it, and I love it.

In the future, we want to be able to ship in one or two days, like some of our competitors (who are probably laughing at my team’s long, four-week cycles). We don’t plan to release that quickly all the time, but being able to do so will mean being even more streamlined. Once you start down this path, you get hooked on having so little between you and your customers, and that is a great place for everyone.

Eric Aside

Why did we move push the Xbox.com team to four-week cycles? Because several members of the leadership, including me, knew it was the best way to improve our team’s productivity and customer quality. Trimming cycle time and work-in-progress is an old technique from lean manufacturing, which dates back to the 1930s.

Individual leadership

ericbrec — Sat, 01 Jan 2011 07:00:00 GMT

Want to create a noxious gas? Combine ambitious yet clueless engineers, a flat functional organizational structure, and the upcoming midyear career discussions. Soon toxic fumes will emanate from individual contributors (ICs) in response to impotent explanations of upward mobility by overwhelmed managers.

Why the wanton whining from wishful workers? Because the technical leadership path is poorly understood, and the organizational leadership path is narrowing. The flat functional structure of most of Microsoft’s divisions today reduces the number of engineering managers and tends to up-level those positions. So now a developer can reach the level of vice-president within discipline, but it’s tougher for that developer to start out in a lead position.

Functional organizational structures tend to drive most engineers into the IC ranks. The good news is that Microsoft® has a robust growth path for IC engineers. At the time of this post, most senior and principal band developers are ICs. It’s not until you reach the partner band that most developers are managers. The trouble is most ICs and their managers have trouble describing technical leadership. How do you get started? How do you reach the senior band as an IC? Glad you asked.

There are many ways of going forward

I covered the Microsoft career stages and how to generally move up in my column Level up. I covered how to get from senior to principal as an IC in my columns Lead, follow, or get out of the way and “Get yourself connected” (chapter 7). The question remains, “How do you first establish your technical leadership and become a senior IC?”

For completeness and clarity, there are three ways to reach the senior band:

1. Become an organizational leader (boss), typically an engineering lead. As I mentioned earlier, these positions are dwindling, especially among the developer ranks.

2. Become a technical leader (guru), typically an expert in some area. This is the focus of the rest of this column.

3. Become a historical resource (sage), typically a longtime team member. This path takes the longest and has the most limited growth potential.

In each case, you are influencing the work of other engineers—as a manager, as a thought leader, or as a resource. Of course, many engineers are combinations of bosses, gurus, and sages. However, being assigned a position as a boss is getting tough, and becoming a sage is a very slow process, so let’s focus on becoming a guru.

Ask the expert

Being an effective technical leader means influencing the opinions and actions of other engineers. In science fiction, that influence is accomplished through mind control. In real life, that influence typically starts with expertise.

Basically, you want to become the “go-to” person for some area. The area could be

§ A technology, like LINQ®.

§ A performance area, like power consumption.

§ A functional area, like copy/paste.

§ A quality area, like accessibility.

The key is that when someone on your team has a problem in that area, he goes to you for a solution.

Eric Aside

Most senior and principal engineers have at least two or three areas of expertise, but are typically known for only one or two.

I know I’ve got a bad reputation

Once you’ve picked an area, you must convince your peers to see you as the area expert. You need to build your reputation around that area—also known as your personal brand. You want to be known as the “LINQ legend” or “power pro.”

Your first step is to become well-versed in your desired area of expertise. It helps to pick an area you are passionate about and already know fairly well. You can read books and attend talks on the subject, join related distribution lists, seek out other experts around the company, and volunteer to take on significant problems in that area for your group.

Support from your manager can make a big difference. Your manager can help you get the right assignments, approve the right training and conferences, and generally help you to establish yourself. Of course for any of this to happen, you must tell your manager about your interest in becoming an expert in your chosen area.

The next step is being opportunistic. When your area of expertise comes up in conversation (written or verbal), have something intelligent to say that is supported by evidence. “Yeah, Eric Meijer talked about that in his LINQ webcast. The right approach is to …” You can do this in a code review, on your blog, on a distribution list, in a wiki, on Stack Overflow®, MSDN®, or some other forum—wherever or whenever expertise is needed. The more you contribute in a helpful—not showy—manner, the more people your expertise will influence.

Quality is job one

“But what area should I choose?” Some people naturally migrate to particular areas of interest, but many engineers like technology in general. In addition, often all the “good” areas already have experts on your team. So what’s a fledgling expert to do?

Eric Aside

If you’d really like to be an expert in an area already taken by another engineer, you can always become the backup for that area. Ask the current expert to be your mentor. Participate in all the design and code reviews for that area. Fix bugs in that area. Cover that area when the current expert is away. Eventually, when the current expert moves on, you’ll become the new team expert.

Go for quality! There are plenty of “-ilities” that don’t get enough love—maintainability, manageability, accessibility, compatibility, sustainability, availability, interoperability, reliability, traceability, scalability, survivability, testability, and usability, not to mention the big three: world readiness, security, and privacy. Heck, I’ve probably missed some. Surely, there’s an -ility for you.

The great thing about being a guru in a quality area is that the knowledge is transferable to other teams. Many quality areas don’t get the attention they deserve, and quality is important to every product. You can continue developing your expertise your entire career, and your personal brand will stick with you wherever you go.

That’s a stretch

There are a few pitfalls on the path to expertise. You want your choice of expertise to be your own. Your lead or co-worker might suggest a work area to you that appears challenging, but really doesn’t hold your interest or seems like a death trap. Stay away from that area. Remember, if you are going to be the point person for an area, you’d better like it.

Eric Aside

There are some areas, like setup and compatibility, that aren’t sexy but are quite complex and particularly valuable. Before you dismiss an area, learn more about it. You may find it’s a place you like and can really shine.

Even if you like the area suggested, you need to watch out for stretch assignments. Stretch assignments are terrific—they can really accelerate your career. However, nothing comes for free. If you fail at a stretch assignment, then you fail. You don’t get an exceptional review for failing, no matter how much time you put into the assignment. Instead, your reward is extra experience.

Why do stretch assignments? Because the extra experience you receive is extremely valuable. Your career will move forward faster in the long term. In addition, if you are lucky enough to succeed (and there is always an element of luck), then you can get an exceptional review. Regardless, you want to accept a stretch assignment knowing the risks and as well as the potential rewards.

What?

Your success in being a respected expert depends greatly upon your ability to effectively communicate your expertise and influence the work of others. If you don’t listen well in order to understand people’s problems and communicate well in order to describe your solutions, then your vast knowledge is for naught. People might as well consult a brick.

For pointers on listening and providing feedback, try reading I’m listening. For pointers on general communication, try “You talking to me?” (chapter 8). The point is smarts will only take you so far. Think of all the successful experts you know—they are all good communicators. Think of all the expert wannabes you know—they all have something in common too.

You can do it

You’ve got to be a smart, capable engineer to reach the brink of the senior band at Microsoft. The next step as a technical leader is available to everyone. It requires hard work, but it’s not especially complicated.

Choose an area you are passionate about. Learn all you can about it. Tell your manager about your interest. Work in that area and on your communication skills. Advise others in that area. Socialize with experts in that area. Become the go-to person for your team.

Soon you’ll be spreading your influence across your team and turning heads in the process. It’s rewarding, it’s fun, and it provides real meaning and value to your work. It’s your road to individual leadership—time to buckle up and drive!

There's no place like production

ericbrec — Wed, 01 Dec 2010 07:00:00 GMT

As much as I love Microsoft®, and as many advantages as we have as a company in the intelligence of our people, the breadth of our products, and the boldness of our vision, there are times when people here are frigging clueless. It’s not everyone—Microsoft is a wildly diverse company. But there’s just enough ignorance to drive you insane.

A great example of nerve-racking naiveté surrounds our service environments. My current team has separate environments for development, check-in testing, scenario testing, stress testing, cross-division integration, partner integration, certification, and production. That’s eight different environments—and we’re planning to build out a preproduction environment next year. Here’s the punch line of this pathetic joke: even with all these environments, there are a ton of issues and scenarios that can only be exposed in production.

Why are we being so witlessly wasteful? Because we can afford it (good situation but bad reason), and because there are so many old-school enterprise engineers who don’t understand the most basic truth about services: there’s no place like production. These engineers conjure requirements for testing and integration environments based on hard-won lessons from business software, yet they fail to fathom their folly. Close your eyes, tap Ctrl-Alt-Delete together three times, and think to yourself, “There's no place like production. There's no place like production. There's no place like production.”

How did I get here?

What kind of fools build out and maintain useless environments? The kind who got burned building enterprise software.

Large businesses rely on enterprise software—it’s got to work or they won’t buy it. Once they buy it, they own it. You don’t get to fix enterprise software anytime you want. That’s right, not even with security patches.

Remember, enterprise paychecks depend on having the software run smoothly. Software changes represent risk to an enterprise business. If the software doesn’t work, work well, and continue working well, enterprises businesses aren’t buying it. And they’ll tell you when they are darn well ready to accept a patch.

An entire generation of Microsoft engineers learned the hard way that you can’t release software until the code is fully tested. There are no “retries” in enterprise software.

Enterprise engineers heave at the thought of releasing code that hasn’t been fully vetted into production environments. They’d burst into convulsions if they understood the real truth about services.

Surely, you can’t be serious?

What is the real truth about software services? There’s no place like production.

Let’s break down these myths about testing and integration environments one at a time.

§ If your check-in tests pass in one environment, they’ll pass in all environments. Okay, that one obviously is wrong, but here’s what’s worse. It’s not difficult to write critical check-in tests that pass in production, but fail everywhere else (like tests of broad fan-out or database mirroring). Instead of kidding yourself, write a small set of automated sanity checks that developers can run quickly in their development environment before they check in.

§ You need a separate environment to test scenarios before integrating code with partners. There are two reasons people believe this—they don’t want unstable code to break their partners, and they don’t want their partners’ unstable code to block testing. The first reason is perfectly rational—you need a test environment to do preliminary acceptance and stress testing, especially for critical components. The second reason is laughable—like your partners are actually going to maintain your test environment in some working state. They won’t. They can’t. (More below.)

§ You can’t use production for stress testing. Why not? Are you worried production will fall over? Wouldn’t you want to know? Isn’t that the whole point? Wouldn’t it be great to watch that happen in a controlled way and back off as needed? Hello?

§ You need integration environments to check cross-division scenarios prior to release and provide preproduction access to external partners. Assume cross-division scenarios worked perfectly prior to production. Assume external partners signed-off in a separate environment before release. Do you now have quality assurance? No. None. Scenarios don’t work the same in production, where there are more machines, different load conditions, different routing and load balancing, different configurations, different settings, different data and certificates, different OS setups and patches, different networking, and different hardware. You’ll catch some integration issues, but not enough to make this enormous expense worthwhile.

Eric Aside

Does a virtual cloud environment, like Azure®, take care of these issues? No, it only resolves the different OS setups and patches and different hardware. It helps a bit with the other issues, but only production is production.

§ You need a protected certification environment. Why do you certify products in advance? Because you want to ensure they’ll work in production. Oh wait.

Let’s recap. There’s no place like production. You need a development environment to run a small set of automated check-in tests, a test environment to run preliminary acceptance and stress testing to help avoid catastrophic failures, and production. Anything more is superfluous.

Eric Aside

It’s nice for your partners to provide “one-boxes” for you to use with your dev and test environments. One-boxes are preconfigured virtual machines that run the services you depend upon in a compressed image. Of course, one-boxes are nothing like production.

Then it’s hopeless

“Wait a minute! We can’t throw untested code at customers. They’ll plotz! And don’t get me started about exposing prerelease, uncertified, partner code. Have you lost your mind?!” Shut up and grow up. There’s no place like production. The problem becomes configuring production to permit the testing and certifying of prerelease code.

The solution is called “continuous deployment.” The concept is simple: deploy multiple builds to production, and use custom routing to direct traffic as desired. It’s like a source control system for regulating services instead of source files. That it isn’t built into Azure and other cloud systems is inconceivable.

There are a variety of different approaches to continuous deployment, which basically differ in regard to the sophistication of the deployment system and custom routing. However, continuous deployment can be quite simple to achieve.

The toughest part is dealing with data, which must function properly across multiple builds. However, if a service is designed to handle at least one rollback after a new build is deployed, even if that new build introduces new data, then that service will function well in a continuous deployment environment.

Eric Aside

You also need to worry about variations of settings across builds. This is a little tricky, but not too bad. Hopefully, your settings aren’t changing all the time.

If new builds depend on new versions of the .NET® framework or operating system, those have to be hosted on new machines—just as you’d have to do without continuous deployment.

How do I work this?

How can you use continuous deployment for integration testing, partner testing, stress testing, and certification? Let’s run through those.

§ Integration testing. You deploy your new build to production, but set the custom routing to direct traffic only from your engineering team to the build. (The default is no routing at all.) The rest of the world continues to see your last release. This technique is called “exposure control.” Now, your team can test against real production with real production data and real production load using a build not exposed to customers.

Eric Aside

You’ll need good diagnostics to analyze any failures you see in production. That’s true with or without continuous deployment.

§ Partner testing. Partners deploy their new builds to production, but set the custom routing to only direct traffic from their engineering teams to their builds. The rest of the world sees no change. Now, partners can test against your production services without anyone seeing their new work, including their competitors.

§ Stress testing. You deploy your new build to production and test it out. Once verified, you use exposure control to increase the live traffic to your new build by increments—first one percent, then three percent, then 10 percent, then 30 percent, then 100 percent. You monitor service health throughout the process. If your services ever show signs of trouble, you capture the data and route traffic back to your last release (instant rollback).

§ Certification. Partners deploy their new builds to production and test them. Once verified, they use exposure control to direct the certification team to their new builds. The certification team certifies their builds in production, before customers or competitors see their new work. Once certified, partners can choose when to direct live traffic to their new builds.

§ Beta bonus! You deploy a beta build. Once verified, you use exposure control to direct beta users to the beta build.

§ Experimentation bonus! You deploy a variation of your current build. Once verified, you use exposure control to direct half the live traffic to your current build and half to the new variation. You utilize the differences you see in usage patterns to improve your services.

§ Auto-rollback bonus! After you direct all live traffic to your new build, you leave the previous release in place. You connect your health checks to the exposure control. Now if your health checks ever indicate trouble, your exposure control automatically and near instantly redirects traffic back to your previous release—day or night.

We’re not in Kansas anymore

Microsoft engineers learned a great deal from our move into enterprise software a decade ago. Unfortunately, those lessons have misdirected our recent service efforts, driving us to build out extraneous environments in the name of service quality.

Maintaining extraneous environments drains our bandwidth, power, and hardware budgets, and dramatically burdens our engineers, without providing real quality assurance. This needs to stop, and thankfully it is stopping as teams adopt continuous deployment.

With continuous deployment, you get service quality without the added costs. You also bag a bunch of bonus benefits to help you improve your services and better serve your internal and external partners.

There was a time when software development was done without source control systems. Now such a notion is not only laughable, it’s unconscionable. Continuous deployment provides a similar capability for services. Hopefully, we’ll soon look back and wonder how anyone ever worked without it.

Eric Aside

Currently, Bing and the Ads Platform have the only production implementations of continuous deployment I’m aware of at Microsoft. Amazon has one of the best known systems in the industry.

My team is currently building a very simple form of continuous deployment. It uses an on-machine IIS proxy to provide exposure control to multiple versions of the same roles on the same machine.

From the perspective of the engineering team, we still deploy the same roles to the same machines as we always have. The difference is that those machines now host multiple versions of the roles, with exposure control directing the traffic we want to the version we want. Sweet!

Am I bugging you? Bug Reports

ericbrec — Mon, 01 Nov 2010 07:00:00 GMT

Some developers hate seeing bugs. They think bugs indicate a failure on their part—that their code seemed perfect until bugs were found. These developers are called “amateurs.” Real developers know the only reason you haven’t found bugs is that you haven’t looked.

I love seeing bugs. It’s better for me to see them than for my customers to see them. What I hate seeing are poorly written bug reports—misleading or generic titles, unclear or missing reproduction steps, exaggerated priority, overstated severity, and inappropriate, cowardly, and poorly documented resolutions.

Why can’t people write decent bug reports? It’s not like a decent report is longer or much harder to write than a lame report. It’s not like clear definitions for everything in a bug report don’t exist. Ah, but those definitions do vary and sometimes conflict from team to team. What are the right definitions to use for everything in a bug report? I’m glad you asked.

Eric Aside

Every piece of software ever written has hundreds or thousands of bugs, depending on its size and complexity. Some bugs are innocuous, like “I’d prefer the close button to be wider.” Some bugs are misunderstandings, like “It wouldn’t let me use an obscene name for my gamer tag.” And some bugs are nasty issues that must be fixed no matter what, like exposing personal information. Since bugs are often found by people outside of the development team, bug reports must be written and tracked to closure—typically using a work item tracking database, like Product Studio® (legacy MS tool) or Team Foundation Server®.

Bug dissection

All bug reports have the same basic set of information.

§ Title—a short description of the issue

§ Assignment—who’s taking care of the issue at the moment

§ Repro steps—the steps necessary to reproduce the problem

§ Priority—the urgency and importance of the issue

§ Severity—the fallout from the issue

§ Resolution—how the issue was resolved

There are a bunch of other fields that are helpful in reproducing the issue and understanding the root cause, but the basic set is short and simple. Let’s cut through the controversy and lay out the rules for each field.

Title and assignment

The title of a bug should be a terse one-line description of the issue that is specific enough to identify that issue uniquely, making bug report searches and identification easy. “The screen blanks when you hit the Cancel button” is a poor title. “Blank screen after canceling avatar editor” is a great title. The second version is shorter, yet provides more specific context around where and when the issue occurs.

When you create a new bug report, you must assign it to someone to resolve. However, unless you are part of the development team, you shouldn’t assign the bug to an individual, even if you know the whole team personally. Instead, assign the bug to the team. This is typically done by specifying the area or team in the bug report and accepting the default assignment. The default assignment is typically “Active” or “Triage.” You don’t know better. Trust the team to know who should work on the issue.

Eric Aside

There are some groups who want all bugs assigned to individuals. This ensures no bugs are ignored. However, even those teams must check for bugs assigned to Active or Triage to ensure they aren’t missed. After all, people outside the team don’t know what other values to use.

As a general rule, all bugs should be assigned to individuals or groups who will check them regularly. Since most triage teams meet daily, I’ve always liked the idea of assigning bugs to Active or Triage as a default.

Repro steps

There is nothing more frustrating than a bug report without a decent repro (reproduction steps). It’s like your significant other telling you, “You know what you did!” with no further explanation. Now I know I messed up and have no way to correct it. Terrific.

Repro steps should be short and sweet—the minimal set that triggers the issue. You should also include the build number (typically a separate field), the environment you used (the version of the operating system, browser, and any other relevant details), and any preparation necessary (like signing into Xbox.com with a gold account).

Sometimes you aren’t sure how you triggered the bug because it’s intermittent or associated with a weird state. In this case, provide the build number, environment, and set up, then describe the circumstances, acknowledging that precise repro steps aren’t clear.

After describing the minimal repro, you must indicate what you expected to happen (“Expected”), followed by what actually happened (“Actual”). All repro steps should have these three sections—the setup, the expected results, and the actual results. That way someone reading the bug report knows exactly what went wrong and how to reproduce it.

Often a picture or video tells a thousand words. There are many tools to create screen captures of both stills and compressed videos. Attaching these files to a bug report can be the difference between a properly fixed issue and an elusive one.

Eric Aside

It’s annoying to see a bug report with 15 repro steps when the issue can be reproduced in four steps. Not only are four steps shorter and easier to understand, but they also allow the developer and the tester to close the bug far faster. It takes less time to reproduce the bug, less time to determine the cause (fewer possibilities), and less time to verify the issue has been fixed.

Priority

There are endless arguments over the meaning of the priority field, a value that typically ranges from 0 to 3. Surely you have better ways to spend your time. Instead, let’s lay out a few simple rules then define priority based on those rules.

§ Priority should never have to be adjusted once properly set, unless the bug itself changes character. If priority 1 means “fix this sprint or milestone” and priority 2 means “fix next sprint or milestone,” then you’ve got to change the priority of bugs at the end of every sprint or milestone. Not only is that a waste of time, but it updates the “last changed date” on the bug, an act that causes the loss of important information.

§ Priority should be easy to assign and differentiate. You don’t want the team spending a bunch of time arguing over the priority of every bug. It should be obvious, both when writing the bug report and when reading it.

§ Priority should be memorable and actionable. No one should have to ask, “What was pri 2 again?” No one should have to question what needs to be done for each priority level.

Based on these three rules, here are priority definitions that serve well.

Priority	Description	Timeframe
Pri 0	A CRITICAL failure that requires URGENT attention that doesn’t have a known WORKAROUND. This is a blocking bug.	You can use the bathroom after you resolve the issue or find a workaround.
Pri 1	A CRITICAL failure that requires URGENT attention.	Must be resolved in the current sprint or milestone.
Pri 2	A CRITICAL failure.	Must be resolved before release.
Pri 3	A failure or suggestion.	Should be resolved before release.

Pri 0 issues typically block testing, deployment, or some other time-sensitive work. Given the seriousness of pri 0 bugs, you can’t submit them and expect something to happen. You must send mail to the individual or team, and then call or walk over and talk to them until someone is actively working on resolving the issue. If a viable workaround is found, pri 0 bugs should be changed to pri 1.

Eric Aside

Teams do vary their definitions of priority. Some start at pri 1 instead or pri 0. Some break the rules I listed at the start of this section or have a separate field to indicate a blocking bug.

If you open a bug in a different team’s work-item database, be sure to use their definitions. The definitions typically pop up in a tooltip or help screen.

Severity

Severity is even simpler than priority, yet it’s also often misused. Severity refers to the fallout of the issue, NOT how important it is. The definitions are:

§ Severity 1—the issue causes a CRASH or customer data LOSS

§ Severity 2—the issue causes a MALFUNCTION that inhibits action

§ Severity 3—the issue causes an INCONVENIENCE or unfinished LOOK

Please note that severity is independent of priority—in other words, severity has nothing to do with priority. A priority 1 bug is more important than a priority 2 bug, regardless of severity. Displaying offensive content is severity 3 but priority 1. Crashing when a user does a forced reboot is severity 1 but priority 3. Nothing makes you the subject of engineer ridicule like claiming a non-crashing bug is severity 1 just because it’s high priority. You sound like an idiot.

Resolution

One of the most important and most often misused fields in a bug report is “Resolution”—the indicator of what was done to resolve the issue. Resolving a bug means you are no longer concerned about the issue and you don’t plan any further work once the bug originator verifies that the resolution closes the bug.

If the issue requires more work before you release, even if it’s not the responsibility of your team, then the bug should remain active and assigned to one of your team members to track.

Here are the possible values for the resolution field in alphabetical order:

§ By Design—the bug report describes the intended behavior. It works as designed.

§ Duplicate—the bug has the same cause and nearly the same user experience as an earlier reported bug. Never resolve an older bug as a duplicate of a newer bug—regardless of how much nicer the newer bug report is—unless you like making enemies of the originator and losing the “first seen” date.

§ External—the bug is caused by something outside of your control AND you can release without the bug being fixed. If you can't release without having someone outside your group fix the issue, then the bug should remain active and assigned to someone in your group to track, linking to the issue on the other team.

§ Fixed—the bug is fixed. My favorite resolution.

§ Not Repro—you couldn’t get the bug to recur in the build and environment noted. Saying “It works on my machine” doesn’t cut it—check with the originator first whenever possible.

§ Postponed—you won’t fix this bug in this release. Postponed is for the same gutless slackers who say they’ll start writing unit tests tomorrow. Real engineers leave the bug active and use a “Fix By” field in the bug report to indicate a future release when they truly intend to fix the issue.

§ Won't Fix—you won’t fix the bug ever. My second favorite resolution—it shows you have enough experience to know when a bug simply isn’t worth fixing, usually because the fix causes more trouble than the bug itself.

When you resolve a bug, you must provide a description as well as fill in the resolution field. That description is important. You get fewer arguments about resolutions, understand the issue better upon recurrence, and protect yourself and the company if the issue later makes headlines. This happened to an old team of mine once—we saved the company millions in penalties when our resolution description for an offensive content bug proved our lack of malice.

When a bug is resolved, it is automatically assigned to the person who opened it. If that person isn’t on the team, the bug should be assigned to another team member who can verify the resolution with the originator of the bug. You can’t always count on people outside the team to validate a resolution in a thorough and timely manner. Of course, if the resolution isn’t satisfactory, the bug should be reactivated.

Eric Aside

I first defined resolutions for my development team ten years ago. Looking back at that mail, the definitions here still stand.

Keep it simple

There are many other fields in a bug report. I mentioned using the build and environment fields to capture reproduction information and the “Fix By” field to indicate when a bug will be addressed. There are also fields to track root cause, how the bug was discovered, area of the product or service where the bug occurred, potential security impact, and countless other variations of information.

When setting bug report requirements, demand no less than what you need and no more than what you’ll use. Requiring more than necessary will cause people to complain and stop submitting bug reports—neither of which serves you or your customers well.

By keeping bug reports easy to write and easy to read, you encourage people to submit clear bug reports for the issues they find. Using bug templates that prefill some fields also helps. There’s no better gift to our engineers and the customers we care about than a well-written bug report that averts an issue before it ever reaches our users.

You can depend on me

ericbrec — Fri, 01 Oct 2010 06:00:00 GMT

We’re getting into the end game before a big release, and I’m already tired of people whining about unstable and overdue dependencies. Of course they are unstable and overdue, what planet are you from?

Yeah, yeah, a package should only depend upon packages that are more stable than it is (the Stable Dependencies Principle). I’ve pushed this principle countless times. Yet when you work for a big ambitious technology company like Microsoft®, no one wants to wait for cool technology to stabilize before coding against it—at least no executive I’ve ever met.

That means your dependencies are unstable and likely running late. It’s not the fault of the teams you depend upon, and it’s not going to be much better next time. Tough luck—quit whining and deal with it. Don’t know how? I figured.

Amongst our weaponry are

There are five methods of dealing with unstable dependencies.

1. Convert them from hard dependencies to soft dependencies or knowledge dependencies.

2. Over-communicate and project manage the heck out of them.

3. Get as close as possible to them personally, physically, logistically, and logically.

4. Automate ingestion of their work for you and testing of their work for them.

5. Create multi-release plans, stable interfaces, realistic schedules, and a vision that leads instead of chases.

Hold on, that last method is a pipedream—there are four plausible methods of dealing with unstable dependencies. Let’s break them down.

Eric Aside

Unstable and overdue dependencies are avoidable by creating multi-release plans, stable interfaces, realistic schedules, and a vision that leads instead of chases. The teams at Microsoft (and elsewhere) that have figured this out live better lives and deliver great, dependable experiences on a predictable schedule.

These thoughtful teams sacrifice a little bit on timely innovation. However, keep in mind that Apple is widely considered a highly innovative company, yet Apple’s innovations don’t utilize bleeding-edge technology. Instead, they craft innovative new experiences from proven technology.

I think your brain is going soft

A hard dependency is one that you literally can’t ship without. If it fails, you fail. A soft dependency is a dependency with a fallback position. If it fails, you can still ship with reduced functionality.

Unstable hard dependencies are a recipe for panic followed by disaster. You want to convert them to soft dependencies by agreeing to a fallback plan. Typically, fallback plans involve shipping with a previous version of the dependency, reducing functionality, taking ownership of the dependency’s module, or some combination.

Fallback plans are wonderful psychologically. They remove the fear and uncertainty around failure. Everyone knows what will happen, and it doesn’t involve bloodshed—only lackluster reviews and a less compelling release. Everyone is still motivated to deliver something great. With raw fear off the table, people collaborate and problem solve far better.

Taking a snapshot of your partner’s code converts a hard or soft dependency to a knowledge dependency. You aren’t actually dependent on the other team for anything but its knowledge and past experience.

Knowledge dependencies are underutilized—they don’t get the respect they deserve. Just because your team may not want to take on any hard or soft dependencies doesn’t mean you can’t take advantage of the knowledge and experience of people who’ve done something similar before. I talked about this in NIHilism and other innovation poison.

Failure to communicate

When you are dealing with overlapping and overcommitted schedules, like when you’re working on almost any project ever, you need to over-communicate to your partners and project manage them. It doesn’t matter how reliable they are or how well-coordinated you appear to be. Assumptions will be made, and important details will get missed. You need to you say everything to everyone regularly and repeatedly, and track every deliverable.

You’d think all this extra communication would become noise, but it doesn’t when handled properly—with regular face-to-face meetings, item tracking (think Product Studio or TFS®), and formal email for plan changes.

§ Regular face-to-face meetings (once a week or so) are great for coordinating small changes, fixing issues that arise, and doing all-important sanity checks. A sanity check is five minutes spent validating high-level assumptions. (“We’re still getting these key deliverables in two weeks, right? You’re still gainfully employed, right?”)

§ Item tracking in a work-item database, like Product Studio, TFS, or any number of other commercial packages, is perfect for tracking resolution of bugs and work items across teams. Share the database queries you use with your partners so that everyone sees the same status.

§ When your or your partners’ plans change, everyone needs to know. Start with a formal email to everyone involved (scrum masters, leads, managers, and directly impacted team members). If any work items have been dropped, changed, or added, update the work item database accordingly. Follow up at the next face-to-face meeting with a full description of what changed and why it changed. This would appear to be obvious, but one person’s big change is another person’s minor detail. That’s why you also do sanity checks.

Eric Aside

Clearly this extra communication and project management is extra work. So is everything else in this column. The extra work typically hits program managers and testers the hardest, but developers are also impacted. The amount of extra work is proportional to the type of dependency (hard, soft, or knowledge) and the level of associated chaos. Plan accordingly.

We two are one

The easiest way to stay in close contact and resolve issues quickly is to practically join teams.

§ Personally—get to know your partners personally. Meet together, socialize together, and truly understand each other. A good working relationship helps in all sorts of ways. You become committed to each other’s success.

§ Physically—sit with your partners physically. The whole team probably won’t fit, but having one or two individuals spending significant time in your partners’ space will do wonders for catching issues early on both sides.

§ Logistically—tie yourself to your partners logistically. When they deploy, you deploy. When they beta, you beta. When they ship, you ship. Staying in sync will save you oodles of trouble—trust me on this.

Eric Aside

Being flexible and agile really pays off in this case. Using short iterations and always being ready to ship not only helps you minimize work in progress and reduces technical debt, it also helps you stay synchronized with your partners’ releases.

§ Logically—engage with your partners’ tools, work item databases, and source code. The deeper you know what’s really going on in their work, the better you’ll foresee, understand, and resolve issues.

Eric Aside

Even if your partners haven’t finalized their interfaces, a starter interface can often help you get an early jump on development and testing. You can write your own emulator, use an early drop from your partners, and otherwise code and test against the upcoming interface in advance of receiving the final version.

It’s totally automatic

One critical engagement with your partners’ tools is around handoffs. Before they deploy a new version, they should run a set of build verification tests YOU WROTE—only you know what you are expecting from your partners. After they deploy a new version, you should run a set of ingestion tools THEY WROTE—only they know all the moving parts, tricky ordering, and special steps necessary.

The build verification tests you wrote should quickly check that the new version works the way you intend to use it. Writing these tests can be a bit tricky, because you have to understand your usage patterns, and you have to author the tests in their test system. Of course, all that effort is well worth it when every handoff works as expected.

The ingestion tools they wrote should give you all you need to use the new version. This should include setup, libraries, content, configuration, and validation. Writing these ingestion tools shouldn’t be wasted effort since they help your partners as much as they help you. That said, all the effort is well worth it when they don’t spend two days after every handoff getting your systems functioning again.

No whining!

Even in the best of circumstances, there are always surprises in any development cycle. Being flexible, using short cycles to react faster, and communicating well with partners, customers, and within your own team does wonders for dealing with the unexpected.

What isn’t constructive is blaming your partners for costly mistakes, even if they were at fault. We’re all human and mess up occasionally. We win and lose together. If you can’t handle the problem or didn’t know about it in time, then you’re at fault too. You all can improve your communication and issue management next time—the issue will recur.

For all the headaches and heartaches that unstable dependencies can cause, they also can be exciting, build a larger sense of team, and bring faster, broader, and bolder innovation to customers.

Don’t play the victim. Create fallback plans, over-communicate, integrate your teams, and automate quality handoffs. You can be part of something big if you embrace the challenge.

Making the big time

ericbrec — Wed, 01 Sep 2010 06:00:00 GMT

Review time is almost over. Maybe you got promoted. Maybe your head is filled with thoughts of making it to the big time—calling the shots, getting paid, and having everyone hang on your every word. For entry and independent ICs, that means being a senior or principal engineer (manager or architect). For senior ICs and leads, that means being a principal or partner engineer. For principal and partner ICs and managers, it means being a vice president (VP) or technical fellow (TF).

From the time we are young, western society drowns us with images and icons of successful, powerful, and connected executives controlling their enterprises and living the good life. You grow up wanting that life for yourself—the life you see in movies and media. Your families groom you for that life—telling you to study, sending you to good schools, and encouraging you to be ambitious. Get a grip. Life isn’t a movie.

There are great things about being in charge and crappy things about being in charge. Before you go chasing some fantasy, you should know the pitfalls as well as the perks. You should be ready and willing to deal with the crud in order to taste the cream.

It’s a question of time

As you increase your scope of influence as an organizational or technical leader, there are many changes. However, there’s one dynamic that trumps all the others—the time it takes for your decisions to make an impact grows longer. This is a subtle, yet critical point.

§ When you are a college hire, your manager tells you to do something and you do it. The time between articulating what needs to happen and having that happen is usually measured in hours or days.

§ When you are a lead telling your reports to complete a project, the time between articulating what needs to happen and having that happen is measured in days or weeks.

§ When you are a discipline manager delegating work to leads, the time expected for results increases to weeks or months.

§ By the time you are a VP or TF, the time between making any substantial change in direction and seeing the result is six months to three years.

There is nothing you can do about this slowing of sway. Even flattening an organization doesn’t help because communication is so imperfect. The more people you influence, the more people who must understand your intent and the longer it takes for everyone to comprehend it and execute it.

The best you can do is state your direction, and then state it again and again—in new ways and through feedback on progress. Remember, most people hold tight to the status quo—you have to continuously nudge folks and sometimes shove and drag them.

The future’s so bright

Lengthening the time between decision and effect impacts almost every aspect of leadership. The broader your scope gets, the further out you need to look.

This means discipline managers and architects need to focus on issues coming next month, not next week. Then they need to direct their teams and remind them several times in advance of the change. Discipline managers who focus on controlling day-to-day issues are called “micromanagers,” also known as ineffective, limited, and despised future failures.

For a VP or TF, telling people about what’s coming next week is useless. Executives need to be focused on next year at a minimum, while providing feedback and small adjustments to this year.

Blame Canada!

By moving up, you trade fast results for broad impact. You can still get small things done quickly, but you aren’t paid to do small things—you are paid for big impact, and that takes time.

This brings us to accountability. If an engineering lead makes a bad decision, the fallout will typically hit within a few weeks. The feedback and correction are quick. If an executive makes a bad decision, it may not have repercussions for years. Those intervening years slow feedback and weaken accountability.

Personally, I like the sweet spot of a discipline manager or architect. Your decisions show results within a few months—fast enough to adjust and learn, yet you still can accomplish great things. In contrast, executives can get away with horrible mistakes for years before being held responsible, assuming they are still in the same division.

Of course, you need to find your own sweet spot. If you want instant gratification, you should probably not venture higher than the senior IC or lead role. If you want to help dictate corporate strategy and influence thousands—and you don’t mind playing politics and waiting years for results—being a VP or TF might suit you well.

Insects don't have politics

Why do VPs and TFs, as well as GMs and directors, get caught up in politics? Because when decisions are widely separated from results, the right decisions become a matter of opinion. When decisions are a matter of opinion, politics play a significant role. This dynamic is reduced at an engineering company, like Microsoft®, since many executives are former engineers and insist upon the support of fact, data, and logic. However, the further you get from actual engineering, the more the door opens to personalities, gamesmanship, and alliances.

“But what about history—you know, case studies and best practices?” The applicability of the past to the unique situation in the present is subjective. There are always exceptions and counterexamples. If you are going to be a successful executive, you need to learn to play politics. You must know who trusts whom, who influences whom, who has what agenda, who owes whom favors, who is likely to support or oppose you and why, and what are the hot button issues for the key stakeholders.

It’s politics, plain and simple. It appears in every company and government once you get to a certain level of abstraction, where nothing is definitive.

Eric Aside

Even worse than politics, delayed accountability can create opportunities for corporate psychopaths—pathological liars with little empathy or conscience. In chapter 8 of their book “Snakes In Suits,” Drs. Babiak and Hare claim in working with almost 200 high-potential executives they found about 3.5% of fit the profile of a psychopath (around 1 in 30). That’s significantly higher than the 1% incidence of psychopathy they found in the general public.

Make it so

Of course, making the big time has plenty of advantages. In addition to the pay, prestige, and perks, you gain enormous influence. Your vision becomes the entire organization’s vision—all the more reason to create, articulate, and drive a clear direction. However, even the powerful advantage of influence has its drawbacks.

As a leader, anything you say—no matter how slight—will be taken as direction. Discipline managers soon realize this when they step into their roles. The most minor opinion becomes gospel. It’s like a macro version of the Heisenberg Uncertainty Principle—you can’t observe a discussion without impacting its dynamic.

Unintended influence becomes a bigger problem as your influence grows. An executive’s casual comment often results in person-months of effort. You must be very careful about what you say, when you say it, and to whom you say it. Ideally, you should say the minimum needed in order to guide those around you in the right direction.

Prefacing your words with “This is just my offhand opinion” or “As an ordinary customer, I think …” doesn’t help. People still take everything you say seriously. There is no cure—people take things out of context and like to please leaders. All you can do is say what you mean and mean what you say. Otherwise, close your mouth.

Eric Aside

If they are supposed to focus on a year out and not talk about much else, how do executives spend their time?

· The bulk of time is spent providing feedback, resolving issues, and directing the work in progress associated with reaching their vision. They conceive a vision and then drive it, and drive it, and drive it. Remember, people hold tight to the past.

· The next large chunk of time is spent building and sustaining relationships with partners, peers, superiors, and the staff. Relationships make everything else possible.

· Of course, any plan a year out will need to adjust to changing conditions. Re-planning is a regular activity, as is preparing for the next big push.

· Finally, there’s the minutia associated with any large role—budgets, resources, busy work, random requests from superiors, events, and personal projects.

It’s alright for you

Now you should have a balanced view of making the big time. Everyone has his or her own limits and views of success. The key is to know yourself and understand what makes you feel happy and fulfilled.

As I talked about in “Level up,” being ambitious means doing whatever it takes to achieve your goals. The higher you want to go, the more of your personal life you will likely sacrifice to get there. It’s a choice you should make before you the choice is made for you.

I know many executives, and I’m quite thankful they’ve made the choice to devote themselves to the leadership of our company. I know many senior ICs and leads, who love the jobs they do so well and feel no need to advance further. Now you know both sides. Strive to make the right choice for you.

I messed up

ericbrec — Sun, 01 Aug 2010 06:00:00 GMT

Ever make a bad mistake? One that makes you feel like there’s a hollow in your chest—you know you’ve messed up badly. Maybe you were even trying to do the right thing, but it just ended up wrong unintentionally. This happens to me regularly. It recently happened to a friend of mine—I feel for him.

The worst part of making a bad mistake is the panic that ensues. Your stress level skyrockets. You desperately seek a way to make it better. You can’t sleep. You feel guilty and responsible. The panic can last for days. Or you could be indifferent and dismissive because you are pond scum.

Make it right

For those of us with a soul, who care about rectifying our mistakes, the thing we want to know most is, “How do I make this right?” I’m glad you asked. Here’s what you do, in order.

1. Take responsibility—you made the mistake, you need to admit it.

2. Deeply understand the fallout—don’t make it worse by fixing the wrong problem.

3. Invite help to repair the damage—acknowledge the problem isn’t yours alone.

4. Ensure it doesn’t happen again—talk about the way forward.

This can be a tough time to stay calm and in control of your emotions. However, staying calm is essential to rectify the situation. You made a big mess—there are no shortcuts to cleaning it up. Let’s talk through each step.

Eric Aside

Whatever you did or said wrong is done, so don’t bother trying to “take it back.” Email recall only attracts attention to your mail. Hiding mistakes only makes them far worse. Be an adult and a professional—own your mistakes.

Take responsibility

You’ve made a mistake, proving once again that you are human. Being human is a reason, but not an excuse. You really did mess up. The first thing to do is fess up.

Don’t blame others, even if you think they are more at fault. Have some integrity. You acted and others suffered. Admit it plainly. “I made a mistake,” is all you need to say.

For a mistake at work, “I’m so sorry” and “Please forgive me” can be unnecessary and even be legally damaging in particularly sensitive circumstances. Why is saying you’re sorry unnecessary in some work situations?

§ You aren’t sorry in a legal sense—claiming responsibility for everything that happened. You regret making the mistake, but it wasn’t malicious.

§ You want to inspire confidence in your ability to handle the problems you’ve caused—not sound weak and helpless.

§ You want to focus on the future, when the problems are corrected and don’t happen again—not dwell on the past.

Eric Aside

Of course, in personal situations it’s often important to say you’re sorry. However, this is not your personal life. This is business. The best forgiveness comes when you’ve competently and confidently acknowledged the issue and rectified it.

There’s no need to belabor the point. Simply say, “I made a mistake,” and move forward.

Deeply understand the fallout

The biggest blunder people make after a mistake is suggesting or implementing a quick solution. Take a breath and reconsider. You never would have made the mistake in the first place if you fully understood the problem. Don’t pretend that botching something makes you an expert.

You need to deeply understand the trouble you caused.

§ Who was impacted?

§ Did your mistake impact different people in different ways?

§ What was the nature of the trouble (personal, schedule, resource, or something else)?

§ What solution do people really desire?

§ How can you help, if at all?

§ How can this kind of mistake be avoided going forward?

Only after listening carefully to everyone impacted, asking questions, and truly understanding what happened, can you possibly know how to make it right.

Invite help to repair the damage

The second biggest blunder people make after a mistake is fixing it alone, out of guilt, hubris, or desire for pity. “I made the mistake—I’ll suffer the cost of fixing it” is the rationale. Get over yourself and quit wallowing. Sure, the mistake was yours, but the problem belongs to everyone involved.

Once you know what needs to happen to rectify the situation, ask for help. If you’ve acted with integrity and listened with care, people will be happy to assist you. Repairing the damage together will rebuild relationships and ensure the solution meets everyone’s needs.

Please note the word “relationships.” That’s what this is all about. When you make a mistake, the lasting harm is damaged relationships between you and your partners and customers. Trust is what you are working hardest to recover.

By asking for help, you aren’t trying to avoid responsibility or work. You still own the brunt of the effort and are accountable to see the problems resolved. However, to bring the event to a conclusion that satisfies your partners, they need to be engaged and will want to be included.

Of course, your partners may suggest solutions you don’t like—all the more reason to deeply understand the fallout. That analysis will either help you successfully convince your partners to apply an alternate solution or help you accept the solution they desire. After all, you are in no position to dictate to those you’ve harmed.

The stronger your relationships were before the mistake, the better you will work through the situation together, and the stronger your relationships will be when the trouble is long forgotten. Relationships are everything.

Eric Aside

If you run into trouble, contact Human Resources (HR) or Legal and Corporate Affairs (LCA) as needed. Remember, solving things yourself doesn’t make you a hero or martyr, it makes you an idiot.

If someone else runs into trouble, forgive mistakes and help your peers recover. The relationships you support may be the relationships you need in the future.

Ensure it doesn’t happen again

Once you’ve taken responsibility, understood the issues, and worked toward a broad solution, the only thing left is to avoid any chance of recurrence. People understand the occasional error. It’s when an error is repeated that people question your true intentions and the value you put on your relationships.

As I mentioned earlier, you must deeply understand how this kind of mistake can be avoided in the future. Then you must clearly articulate that intention. There are two magic words you should use, “going forward.”

“Going forward I will confer with our stakeholders before finalizing a decision.” “Going forward I will run the entire automated test suite before checking in a global change.” “Going forward I will ask others before taking the last donut.”

Going forward is far better than looking backward. Forget the blame. Stop the whining. Move toward the future. Everyone will appreciate it.

Eric Aside

As I mentioned in “Your World. Easier,” people tend to repeat certain mistakes, which is why checklists are so handy. Keep track of your personal patterns, and put procedures in place to prevent problems.

All better

Sometimes people simply need to hear you acknowledge a mistake. They’ll fix the problem or be happy to help. They only want to know that you realize your error, appreciate the problem you caused, and know how to avoid it going forward.

Sometimes there is no way to repair the tangible damage, or the damage lingers for years. However, you can still repair relationships by taking responsibility and ensuring the ruckus is not repeated.

Why are people so forgiving? Because everyone makes mistakes. We’ve all experienced the horror of realizing we’ve messed up. We are here for each other. Trust others to understand. Work hard to understand them. One day, this too shall pass.

Culture clash

ericbrec — Thu, 01 Jul 2010 06:00:00 GMT

Culture is management’s boogeyman—a monster that can’t be controlled, an immoveable object that can’t be overcome. If you ever want to see managers become whiny, petulant infants, ask them to challenge an issue ingrained in the corporate culture.

When managers say, “That’s a cultural issue,” they mean, “There’s nothing I can do about it.” When managers say, “You’d have to change the culture,” they mean, “I’m wimping out on this issue.” Culture is a manager’s favorite excuse to maintain the status quo and avoid confronting change. That is not only lame—it is naïve and indefensible.

Culture as concept seems interwoven into the very fabric of our lives. What hope would anyone have to change such a thing? Yet corporate culture doesn’t arise in a vacuum. It is instilled by the people who founded the company and changes gradually as the people change. Therefore, you can affect culture, and it can change. You simply need to know how to do it, and stop being a wimp.

You see the whole culture

Countless books, articles, blogs, and commentaries discuss what drives and sustains culture. It basically comes down to expectations, modeling, and rewards.

Your parents, peers, neighbors, and friends expect you to behave a certain way. They model it for you. They correct or shun you when you fail to comply. They applaud and encourage you when you set the “right” example. This is how culture perpetuates itself.

Can you change culture? Sure—simply expect, model, and reward different behavior. Really, it’s that easy.

Ever notice how the culture of a group or organization changes when the leader changes? That’s what’s happening—the new leader expects, models, and rewards different behavior.

Eric Aside

The behaviors a new leader expects, models, and rewards aren’t necessarily the ones the leader intends. The leader may say one thing, but actually expect, model, and reward another. The old saying, “Do as I say, not as I do” is pretty ineffective at driving cultural change.

Back off, man. I'm a scientist.

Is there a science to driving behavioral change in organizations? Of course there is—it’s called Human Performance Technology (HPT). You’d think every manager would know about it, but few do. Maybe they prefer whining and impotence.

The basis of HPT is B.F. Skinner's model of operant conditioning. Basically, this model says that voluntary behavior breaks down into stimulus, response, and reinforcement. You receive a stimulus, like your manager telling you to start coding. You respond by coding. You receive reinforcement, like pizza.

Thomas Gilbert analyzed the types of stimulus, response, and reinforcement in his book on Human Competence, differentiating between the environment and the person.

§ The environment refers to workplace influences, like expectations which stimulate a response (such as instructions and metrics), tools and processes that shape the response (such as build systems), and incentives that reinforce the response (such as bonuses and recognition).

§ The person refers to individual influences, like knowledge to interpret a stimulus, capacity to respond, and motivation to receive the incentives. (If you’re not hungry, pizza isn’t an incentive. Hunger is the motivation.)

To drive behavioral change, you need to consider and potentially shift the environmental and individual elements of stimulus, response, and reinforcement, known as the “Six Boxes.”

	Stimulus	Response	Reinforcement
Environment	Expectations	Tools & Process	Incentives
Individual	Knowledge	Capacity	Motivation

Eric Aside

The concept of putting Gilbert’s six areas into plain language as “Six Boxes” comes from Carl Binder.

Environment

Expectations: Suppose you want your software development team to focus more on unit testing, perhaps even adopt Test-driven development (TDD). From the environment side, you need to ensure that writing unit tests is expected by management and the team. It would be helpful to have metrics that indicate how much of the code has unit tests in place. However, that’s not enough.

Tools & Process: The team will need to have a good unit testing framework in place—one that is easy to run, easy for developers to add tests to, and easy to use for measuring results. Of course, that’s not enough.

Incentives: If group managers reward developers for checking in code quickly, with or without unit tests, developers will stop writing the unit tests. To drive unit testing, only developers who write thorough unit tests should be chosen for recognition and rewards. But that’s not enough.

Eric Aside

Which reward works best? Typically, it’s not salary. Salary can actually demotivate when people aren’t paid fairly. Also, recent studies show that salary doesn’t align well with what people truly desire when doing difficult work—autonomy, mastery, and purpose (check out Dan Pink's talk on the subject).

Instead of more money, offer more responsibility and opportunity to improve, and tie people’s work to outcomes that matter. These rewards better target engineers’ motivations and also naturally lead to raises and promotions.

Individual

Knowledge: Developers will need to know what the different kinds of unit tests are and how to write them well. They will need to know how to use the unit testing framework, how to get metrics on their progress, and what targets they are expected to reach. Now, we’re almost there.

Capacity: Naturally, the developers will need to be capable of writing unit tests. Hopefully, in this particular case that’s not a problem. However, you might also expect your test team to unit test its automation. If your test team is using an automation tool, the team members may not be capable of scripting it or writing external code to test it. You might need to hire testers who have that capacity.

Motivation: Finally, your developers need to care about meeting these new expectations and receiving the recognition and rewards you offer. If your developers are completely apathetic or entrenched, they won’t be motivated sufficiently to make the change.

Is it deliberate?

I’ve outlined a whole bunch of system thinking for a simple cultural change. You might say that’s overkill. However, look back. If any one of the six boxes isn’t properly addressed, the change won’t happen properly, if at all. “I know—we’ll just train everyone” isn’t enough.

The environment impacts everyone and thus is more potent than any individual. No amount of training will overcome the wrong expectations, tools, and incentives. However, clueless, incapable, and apathetic engineers are also quite debilitating.

Review all six boxes. In the unit testing example, you’ll see we need clear manager expectations, metrics, targets for those metrics, a unit testing framework, a change to rewards and recognition agreed upon by all the group managers, training for the team, and the right team members in place—both from a capacity and motivation perspective.

How do you think all that through in advance? Easy—know where you are today, know what you want to accomplish, and then go through the six boxes with that gap in mind.

You must avoid jumping to the first sexy solution (“I know—we’ll use that new tool!”). You must be deliberate about your goals. You must also be deliberate about measuring progress toward achieving your goals, not just making pretty charts.

As I covered in How do you measure yourself?, measuring lots of values and putting them on graphs achieves little. The numbers don’t have meaning and the metrics will be gamed. You must know what you want to accomplish and deliberately measure your desired end results.

I'm going to ask you a few questions

HPT also provides a clear framework for performance improvement in general. Before proposing a solution, HPT requires that you understand the problem, the desired result, and how to measure success. The next time someone runs into your office suggesting that the team adopt a new estimation method, patiently hear him out and then ask him a few questions.

§ What are you trying to accomplish?

§ How is it done today?

§ How will you know when you’ve accomplished your goal?

§ What should managers say and do to support this direction?

§ What processes and tools will the team need?

§ How will managers need to recognize effort and progress differently?

§ How will the team learn about the new direction, tools, metrics, and targets?

§ Are the right people in the right roles to accomplish your goal?

§ Does the team care at all about this?

All together now

Remember, if you are serious about doing things differently, then you are working on changing the culture. This applies to big changes, like moving to a services model, and little changes, like using code review checklists. Of course, big changes take longer than small ones, which is why goals, metrics, and targets are so important.

Changing the culture is possible and even algorithmic. When you ask the right questions, focus on the result, and consider all six elements that drive behavior, you can move molehills as well as mountains.

Level up

ericbrec — Tue, 01 Jun 2010 06:00:00 GMT

If you’re not a Microsoft® engineer and you’re not interested in finding a new reason to bash Microsoft, save yourself some time and skip this column. If you want to know how to build your skills and systematically grow your career as an engineer at Microsoft, read on.

I’ve been managing Microsoft engineers for nearly 15 years. I’ve worked for eight different organizations and more managers than I can recall. While every manager and organization has its own take on promotions, there are some basics everyone follows. Those basics are documented nowhere, until now—right here.

Why am I giving up these “secrets” to getting promoted? Because they aren’t secret. My peers all know them, and I’ve always told my employees this information. After hearing it, my employees all ask, “Why isn’t this written down?” I could speculate that HR is too scared to make it simple, that engineers are too precise to generalize, or that Microsoft organizations cling to the notion that they are each special and unique. Whatever, I don’t care. I’m just going to tell you.

The basics

Most Microsoft engineers know there are six career stages, each with their own Career Stage Profile (CSP), but don’t know what differentiates one stage from the next.

§ Entry level (e.g., SDE I)—you’re straight out of college. Your education hasn’t prepared you to work in a professional team environment with money on the line and customers that aren’t figments of your professor’s imagination. You’re clueless.

Eric Aside

SDE stands for Software Development Engineer. There are similar stages for the other engineering disciplines.

§ Independence (e.g., SDE II)—you can code up anything your lead asks of you. Period. If you don’t know how, you know who to ask, or who to ask who to ask. This is the last time you’ll be differentiated by your coding skills.

§ Team leadership (e.g., Senior SDE)—you influence the rest of your team of roughly 3 – 12 engineers. You influence them as their managing lead or as their technical lead. Either way, your impact extends beyond what you can do alone.

§ Group leadership (e.g., Principal SDE)—you influence the entire group of roughly 12 – 80 engineers. You influence them as their discipline manager, group manager, architect, or core technology guru. Regardless, you have become a key person in your organization.

§ Organization leadership (e.g., Partner SDE)—you influence the entire organization of 80 – 500 engineers. You are the discipline director, general manager, partner architect, or core technology worldwide expert. You have become a partner at the company.

§ Industry leadership (e.g., Technical Fellow)—you influence the entire division and with it the entire industry (from 500 to millions of engineers). You are the distinguished engineer, technical fellow, or vice president. Money is probably not a concern of yours anymore unless you have a serious gambling problem.

How do you get promoted from one stage to the next? Let’s take that a stage at a time.

Entry level (e.g., SDE I)

To get promoted from entry level to independence, you need to become independent. Duh. It will likely take you two to three years, depending on your past experience. Don’t try to save the world. Don’t try to impress your vice president. Do your job. Do it well. When something is broken, fix it. Show your manager and your peers that you can handle assignments and get past roadblocks on your own.

Handling assignments on your own doesn’t mean doing so in isolation. It means knowing what questions to ask, who can answer them, and how to get the answers. Even the Lone Ranger had Tonto. (I’m dating myself.)

Often career stages are divided into two levels (in Redmond, SDE I has levels 59 and 60). You’ll get promoted to the second level (level 60) after a year or two if you show progress toward independence. You’ll get promoted to the next stage (level 61) when you actually are independent, assuming your team hasn’t run out of promotion budget.

Eric Aside

A common failing at the entry level is complaining instead of solving, aka whining. The world is a messed up place. The workplace is no different. No one else is going to fix your problems. If you’ve got an issue, then propose a solution to your manager and peers and, ideally, implement it. Otherwise, live with the problem and quit your whining.

Independence (e.g., SDE II)

Now you are independent. You can handle any individual assignment with little direction. You still ask plenty of questions and keep your manager informed, but you are the initiator. No one has to tell you what to do.

Why must you be independent before you get promoted into the independence stage? Microsoft wants you to be successful. If you are promoted and fail at the next stage, Microsoft has lost a valuable employee at the previous stage. Demotion is not an option because it kills morale and motivation. So, you only get promoted when Microsoft knows you’ll be successful at the next stage, which is when you’re behaving as if you have already been promoted.

Microsoft expects every engineer to become independent and a team leader, either organizationally or technically. That means you are expected to make it to the team leadership stage, typically after three to five years of independence. If you don’t, you’ll likely be managed out of the company. You can’t plateau as an independent engineer at Microsoft—you need to influence your peers at least at the team level.

When you have established that you are fully independent, regardless of the assignment, you’ll get promoted to the second level within independence. When you are influencing your peers and demonstrating sustained leadership (for 6 – 12 months), you’ll likely get promoted to the team leadership stage.

Eric Aside

Switching managers and groups often impacts time between promotions because your new manager will not know how long you’ve been displaying leadership skills. There are a couple of actions you can take to avoid resetting your promotion timetable every time you switch groups or managers:

· Ask your old manager and new manager to discuss your current leadership skills and how long you’ve been demonstrating them. This is different than asking for a promotion—you are simply encouraging a smooth personal development transition.

· If you are going to switch groups and managers, do so within a year or two after being promoted, ideally with the blessing of your current manager. Switching then won’t likely impact your next promotion.

Team leadership (Senior)

Sweet! You’ve made it to team leadership. That doesn’t necessarily mean you’re a people manager. It means your impact extends beyond yourself and your own work. Your influence guides the work of your teammates (for example, mentoring, setting the example others follow, making broad design decisions that stick, and establishing quality practices that stick).

From here on out, your coding skills no longer differentiate you from your peers. Everyone has great coding skills when they reach the independence stage. Now, your encouragement and influence skills set you apart. Soft skills. There, I’ve said it. You’ve got to apply all that written and oral persuasion crap you reluctantly muddled through in high school and college.

Microsoft’s ranking of the future potential contribution of employees—the top 20% of potential, the middle 70%, and the bottom 10%—changes its character at this stage. That’s because team leadership is the first stage where you can plateau, though Microsoft would prefer that you make it to the group leadership stage. The following table shows the differences:

Potential	Entry Level & Independence Stages	Team Leadership Stages and Above
Top 20%	Likely to reach team leadership stage	Promotable depending on progress
Middle 70%	Promotable depending on progress	Promotable, but increasingly difficult to justify due to fewer high level positions
Bottom 10%	In jeopardy of dismissal unless there is substantial improvement	In jeopardy or reached a plateau—ask your manager to clarify

As for promotions, when you show occasional broad influence within the entire group, you’ll likely be ready for advancement to the second level within team leadership. When you regularly and sustainably (for 6 – 12 months) influence the entire group, you’ll be ready to reach the group leadership stage. Both promotions can take many years. There is no longer an expected timeline, and no guarantee of a promotion. It now depends on business need and ambition, as well as ability.

Eric Aside

What about managing cross-team and cross-division dependency relationships? Isn’t that group leadership? No, it’s not. Managing relationships across teams and divisions is part of your regular job. Even entry level engineers need to become good at it. However, when you start influencing strategy and systems of other teams and divisions, you have graduated to group leadership.

Group leadership (Principal)

Wow, you’ve made it to the group leadership stage as a discipline manager, group manager, architect, or core technology guru. Congratulations, that’s a big, difficult step. You got here by thinking and influencing outside the confines of your team.

Influencing a large group necessarily means influencing strategy and systems. I’ve written many columns on each, in particular Controlling your boss for fun and profit and Lead, follow, or get out of the way. It means thinking big and setting a vision. It means less following, less critiquing, and less complaining. Instead, you step out in front and lead toward your vision. It’s scary to stick your neck out, with no one else to cover you. That’s what makes you a leader.

You can be promoted to the second level within the group leadership stage simply by establishing yourself as a strong group leader. However, getting to the organization leadership stage is far more difficult. The company doesn’t need many organization leaders. Very few engineers make it past the group leadership stage, so ambition begins to affect your continued advancement. This is true of any sizable company.

There will be peers of yours who have ambition. They’ll be willing to live overseas, work on unattractive projects with difficult people, travel frequently and at a moment’s notice, be on-call at all hours, and make other sacrifices that you may not be willing to make. All other things being equal, an ambitious person will win because of this willingness to do whatever is asked to reach the next stage. If that doesn’t sound like you, be prepared to plateau at the group leadership stage. You might make it further, but ambition is a key element.

Organization and industry leadership (Partner or higher)

As an organization leader—a discipline director, general manager, partner architect, or core technology worldwide expert—you have become a partner at the company. You have special meetings with Steve Ballmer and the executive staff. You get special stock plans and perks because you are responsible for business results as well as work results.

You also are expected to give back to the company. This can mean being moved onto projects that need your expertise, even if you’d prefer to stay where you are physically and organizationally. It can mean being involved in leadership development programs for group leaders, other partners, and new senior hires. It can also mean bumping up against other leaders just as ambitious as you. At this point, you are a grown up. You should be able and willing to handle the politics.

Things only get more competitive to become an industry leader—a distinguished engineer, technical fellow, or vice president. You must be world-famous and the voice for a major business that you helped create. Clearly, ambition and business opportunity play key roles in reaching this stage. It’s not just about being smart. Everyone who reaches the team leadership stage is smart. You’ve really got to persist and persevere, doing whatever is necessary, if you plan to be an industry leader.

Set your course

If you haven’t realized it by now, career development is a series of choices—choosing to work independently, to influence your peers, to think strategically, and to be willing to make sacrifices. How far do you want to go and what are you willing to do in order to achieve your goals? Those are the key questions you need to answer.

Your career is yours. It belongs to you. Mentors can guide you. Loved ones can support you. But in the end, you own your career. Decide what you want to be. Decide what makes you happy and fulfilled as a whole human being. Accept those choices, set your course, and pursue your dreams. Make it so.