Statistics as reported in most newspapers never quite satisfy me. It's not that they are lying, but there is never quite enough there. The flu is a great example. I was just reading this great article in Slate about how they came up with the 33,000 deaths per year from the flu http://www.slate.com/id/2218367. Turns out they did regression analysis and looked at the correlation between the severity of the flu season and the number deaths on a week by week basis. From this they estimate that 34,000 people died that would not have died otherwise. This is all well and good, but it doesn't answer the real question. How should the government react? What will the economic impact be? What's the difference between pandemic flu and seasonal flu?

Because, let's face it. The flu doesn't kill anybody. After all, we are all going do die. All the flu does is speed death up for some folks. Some articles make some cursory acknowledgment of this fact by noting whether the flu is killing young people or not. But this there is never a good measure associated with it. Reminds me of a case I was reading at about the time of the peanut contamination stories. A man was talking about how his mother had fought through cancer and was still weak and on a respirator but seemed to be making a comeback when she got sick and died from contaminated peanuts. Sure this is a tragic story and there is a chance that his mother would have had many more years if it hadn't been for the peanuts. But really, cancer killed her. Let's take a look at making these numbers meaningful.

For example, you could do the same regression analysis you did above but then continue the analysis moving forward to see if there is a subsequent dip death rate in the coming months. If the flu was killing people who were already pretty close to death, you might expect to see a decrease in death rate over the next year as those deaths that were 'paid forward' aren't recorded. After some work, rather than recording a 'mortality' rate you could state a rate for 'having more than 6 months taken off your life ' or 'number of years taken from population.'

Another death related statistic I've never liked is life expectancy. I've go "The Economist Pocket World in Figures 2008 edition" on my desk (of course.) I open it up and see that the life expectancy for Japan is 82.5 years. For the United States it's 78.2, just below Cuba and Denmark. But then on another page I can see that the median age in Japan 42.9. The United States doesn't even show up on the list because it's too low. Well, that puts the life expectancy in perspective. It's a lot easier to have a longer life expectancy if you don't have young people around to die and mess up your stats. It also shows how life expectancy in isolation doesn't help much for policy because you don't know who is dying.

For example, I am personally more interested in the life expectancy for a 40 year old male than I am for the entire population. A better chart would be percent chance of dying in a given year as a function of age. There are lots of cool things we could learn about and apply to policy. I would expect a spike in the first few years because of birth defects and childhood illnesses. After that you are almost certainly going to get a spike around 16 when US kids get driver's licenses and some relatively high rates for the next few years due to suicide and teenagers being crazy. Perhaps there would be another spike at 21 for hitting the legal US drinking age. It goes down until thirty and stays low through the peak parenting and earning years. Your just too busy to die. Things pick up in your fifties as bad living habits start to take their toll. I would guess you would see another spike around retirement but I suspect things level off for quite a while until you get truly old. These are all guesses and it would take a lot more data to come up with this sort of chart, but it sure seems more useful and more actionable.

And don't even get me started on dramatic illustrations of the amount of money in the economy especially since the banking crisis. The lesson is that real numbers take work to figure out. Even stats presented in good faith don't tell the real story, so you can be sure that numbers presented to further an agenda they are useless.

According to my Kindle, I am about 80% through reading Leadership on the Line. That's close enough that I'll write a brief review. Don't get wrong. I'll finish the book. But business books rarely finish strong. If I wait until I've clicked the last "next page" button, I probably won't write a review at all.

But getting to the point, I highly recommend the book.

The authors' primary thesis is that there are two kinds of leadership challenges. The first is technical. Technical challenges are the sort that can be solved through traditional management techniques. With a technical challenge, you don't need your team to do anything differently. They just need to do things better. An adaptive challenge, on the other hand, requires you to truly lead and make fundamental changes in your organization. It requires you to change hearts and minds and even shift the culture in a new direction. The book's authors posit that one of the biggest mistakes in leadership is to mistake an adaptive challenge for a technical one.

One example that's been making the headlines lately is the car industry. GM has been having problems for quite a while now, even before the current financial crisis. A technical response to this problem would be to improve marketing and reduce costs for its highest margin vehicles. GM would still focus primarily on large vehicles, at least in the U.S. An adaptive response would have been to invest in smaller, more efficient vehicles even when Hummer sales were robust. Getting the suppliers, unions, and dealers in lined up to make this change would have been enormously difficult and dangerous to your career. But if successful, GM would be in a better position today.

Technical challenges respond well to straight-forward management in existing frameworks. Adaptive challenges require leadership. Adaptive challenge is also dangerous. Most of the book deals with the pitfalls and hazards of trying to make lead an adaptive change.

For example, when you deliver an uncomfortable message, people will complain about your style without addressing the underlying issue. Another pitfall is getting seduced by your most ardent supporters and moving to far from what people will accept. You may not understand the nature of your powerbase and start to issue edicts that aren't followed regardless of your role authority. Someone's values may be threatened by the change you represent. You have to tread carefully to avoid destruction.

One of the things I liked best about this book is that the examples are almost all failures. Too many business books focus only on the success stories and are easily skewed by survivor bias. It also brought to mind the techniques of some of the best books on leaders I've read including Abraham Lincoln, Lyndon Johnson, and Alexander Hamilton, both for there successes and their failures.

The one criticism I have is that there are few precise techniques. There are a lot of abstract warnings and things to keep in mind. But without actionable steps these theories can be debilitating to someone facing an adaptive challenge. However, over the past couple of weeks I've come to think of this book as a partner to other sources of leadership information including Influencer and Manager Tools. Manager tools does a great job on the nuts and bolts of technical management. I don't mean this in a pejorative sense. After all, the day to day business of getting things done and working within an organization is critical. "Influencer" gives some specific methods for creating adaptive change. "Leadership on the Line" puts these two in context by describing the different kinds of challenges leaders face and describing the pitfalls and general approaches to tackling adaptive challenges.

Having kids can provide great examples of pathological team behavior. It's not that your team behaves like children. On the contrary, the reasonable explanations and subtle arguments deployed by adults can serve to obfuscate fundamental issues. But having kids helps to emphasize that all human beings exhibit the same basic traits. Kids just tend to exhibit these behaviors in such an obvious way that they can shed light on team and work dynamics. In particular, one recurring scene in our household has made me think more critically about what it means to be "done."

I am usually the one to drive the kids to school. I have two kids that go to two different campuses of the same school. One is in first grade the other is in second. If I can get both kids to the North campus by 7:50 AM, the South campus kid can hop on a shuttle and make it the rest of the way herself. Then I can go straight to work. If I'm late, we miss the bus and I have to drive a few miles to south campus and drop off the South campus kid myself. Missing the bus gets me to work almost a half hour later than I otherwise would have. You might say I'm motivated to get to get the kids ready on time.

I've learned that it's not enough to ask the kids if they are ready or to list off the things they need to get done before they can get in the car. The only thing that really matters is if they are in the car or not. Often the biggest delays happen between the time they believe they are ready to go and when they actually buckle up their seat belts. They have to go to the bathroom, get a stuffed animal, brush their teeth, or any number of last minute items. What's worse is that because they think they are done they tend to play around, argue, and generally make noise as I'm trying to make sure I have everything I need. I can't concentrate and deep in my heart I know that the kids aren't really ready if they aren't in the car.

That means in the morning you'll often hear me tell the kids to get in the car with a fair degree of urgency. Getting them in the car focuses their minds on all the little things they need and also gives me thirty seconds of quiet to help me make sure I haven't forgotten anything.

You know where this is going. It's not enough to say you are done. You have to report that you are done, something must be delivered, and there has to be some objective measure or someone acting as a gatekeeper. The bar doesn't have to be high, but it has to be higher than the developer's say so. How often have you heard "It's done, there are just a couple of bugs." or "It's done, but I have to check-in and run some final tests."

That just isn't good enough. In order to run a project successfully you need to identify simple, objective criteria for when every task is done. It's also not good enough to have milestone criteria. If you don't know how much of your work is done on at least a week by week basis if not a day by day basis, human nature will kick in and things will pile up.

And I do mean simple. It doesn't have to be end to end scenario tests, although it is always nice if a task can light up actual functionality. For example, here is one possible approach:

• Developers must write accompanying unit test for every task.
• Tester code reviews unit test to verify that it does what it should do.
• Developer runs unit test in presence of tester and sees that it passes. If you are using Smiley Project Management, the tester can place a star sticker by the task.

I was thinking about government analogies for software development organizations while I was unclogging our bathroom sink. I originally made this analogy after in this post about the supreme court. But I didn’t make any analogies between the roles of software development and the roles of government.

The first couple were easy to figure out:

• Judicial—This is clearly test. They enforce the rules and decide when things are done.
  
• Legislative—This is the real work done legislation, so it must be development.
  
• Executive—This one is management. True, management has more direct power over what gets done than the President does. But they both set direction and have veto power.

Note that program management doesn’t show up here. But then it finally dawned on me. PMs don’t exist at all companies. They are sometimes despised, but at their best they represent customer voices.

PMs are lobbyists.

Over the last week I went from not knowing about a new technological movement to seeing it everywhere and becoming a believer. Here's the story.

I'm something of a data wonk. On, my desk you'll find a copy of the economist book of facts and figures. I read it for fun. I often download the latest spreadsheets from OMB on the latest economic figures. I'm also the guy that spends time digging through the Customer Experience Improvement Program data looking for gems on how our customers use our products. And yes, I mine Twitter too.

But what's always frustrated me about this data is that it's hard to analyze and link together. The OMB data comes in a pretty nice spreadsheet but I always have to tweak it to get what I want. I can't link it to other data sets without significant work. This seriously limits the insights I might be able to achieve. I have to be pretty confident about what I'm going to find before I look for anything beyond the obvious. I've got so many data streams available and I just want to have the freedom to dump into an OLAP cube and play. Dealing with the limitations is an eternal source of pain for me. I live this stuff so I was primed.

And then there was the past week.

First I hear about Wolfram Alpha. It looks like a search engine but it calculates answers from data gleaned from the web rather than matching strings. Cool, I thought. But nothing started to flash in my head. Then I heard about a project from Amazon where they are publishing government data in an accessible format based on AWS. Also cool. I made a mental note to check it out the next time I had a hankering to go to OMB. But it still didn't click.

But then I was whiling away the time with my Zune at the kids swimming lessons and I thought I'd listen to a TED talk from Tim Berners-Lee. It's an idea of his for something called linked data. He's advocating for everyone to provide raw data over http for tools and services to build on. This talk is when everything came together. When you see three news items in the same area in a short period of time, you know something is up.

In some ways these projects seem unrelated, but that makes the meme even more powerful. Different people are coming to the same problem from different angles. Wolfram Alpha needs to data to work on. Amazon may be just looking for something to show off their platform with, but then again that's all Microsoft had in mind when it initially built Terraserver. Now Google Maps and Virtual Earth are big deals. Let's just I'm stoked.

I recently bought my wife an Amazon Kindle for Christmas. Yes, I know it's March but they were backordered. But at least this way I was able to give the Kindle two. Anyway, she loves it.

The thing that's striking about it and the reason I am writing this post is that I'm impressed with how the Kindle completes the scenario. By this I mean that the designers were relentlessly focused on the book reading experience and didn't stop at the natural boundaries if it meant that this experience would be compromised. There are always compromises to be made, but whenever they were faced with a decision they compromised in areas outside of the core scenario and never strayed from the fundamental vision.

Take Whispernet for example. It allows the kindle user to buy and download books from anywhere. They easy solution would have been to require synching with a PC in V1, WiFi in V2, and building mobile apps later on. But they wanted to make sure there were nothing in the way of buying and reading books . I don't know how they pulled this off, but I'm sure it included some animated discussions both internally and with the various wireless providers. It's a big bet.

The next big call is the screen. It's a monochrome LCD screen. It's not well suited to rapidly changing graphics or other applications. The screen flashes a bit when you turn the page. But when you think about it, a book has a similar experience. The there are few really nice things about it from a perspective of completing the scenario. One obvious benefit is that's cheaper than the alternatives. But the two factors that I believe really drove the decision are battery life and readability. The screen doesn't draw any power just to display a page. It only draws a minimal amount to change the display. This lets you treat the Kindle much more like a book. You don't have to think about charging it the same way you need to with a cell phone or a notebook. The second benefit is the readability. I was a little surprised at this one, but reading something that's not backlit is might easier on the eyes and much more suited to a book-like experience.

But it's the compromises that really illuminate these decisions. It's easy enough to articulate how to complete a given scenario. It's difficult to make the tradeoffs necessary to make it happen. When you start out you have to know that you won't be able to complete every experience. Some of them will be decidedly sub-par. Some you'll cut altogether. For example, don't rely on your Kindle to be a web browser. But you can't nail the important scenarios unless you make these compromises.

This is where leadership comes in. Someone needs to provide the vision to keep the project on track. Even more importantly they need to provide the vision in a way that is easily digestible and enables everyone to easily make decisions based on the same clear criteria. Completing the scenario is just as much about saying 'no' to other great features as it is to spending the resources to get that last mile.

Joel Spolsky talks about this a bit too. http://www.joelonsoftware.com/items/2006/12/09.html. He focuses more on the benefits to simplicity to the design then on clarity of vision. He is certainly right, but I'm talking more about the less appealing underside to the same discussion. Sometimes you have to have to make decisions that cut or reduce really good ideas and perfectly good scenarios just to make the core scenario complete and not just 80%.

Anyway, back to my book.

One of my favorite theorems is Bayes' theorem. (There, I admit it. I'm a geek.) At its heart, Bayes' theorem is a way to update a probability estimate with new information. Say for instance you hear that there is a one in ten chance of rain today. Later on, you see some ominous clouds. It's reasonable to think that the probability is now a bit higher in light of this new observation. But yet the original ten percent forecast still has some value. Bayes' theorem calculates the updated probability based on the original ten percent forecast, the chance of ominous clouds showing up independent of any forecast of rain, and finally the probability of ominous clouds on days that it really does rain. Frankly, you don't have to have the precise formulation memorized to make some of it. The Wikipedia link above is good enough if you want to dive in a little bit deeper.

I first encountered Bayes' theorem in graduate school. One of the atomic physics grad students who had gotten his PhD a couple of years earlier came back to talk about his current work. It was some government job tracking radon in houses. I don't remember exactly how he used Bayesian analysis, but what struck me is that it seemed to be able to create data from nothingness. He had some very sparse data about Radon instances and went on to construct a pretty thorough model of radon concentrations across the country. It retrospect I understand that he took raw data and applied additional knowledge like Gaussian distributions and other assumptions that don't seem like knowledge in themselves but helped extend the base data.

But at the time, something just didn't seem right about it. So I looked into it more. Later on I used it my own research. There was a Bayesian classification algorithm that I found somewhere that would identify clusters of data from a raw, uncorrelated source. My research involved finding optimal structures for silicon clusters and I used this Bayesian classification algorithm to find trends among the vast library of locally stable clusters I found though simulated annealing, Monte Carlo, and genetic search algorithms.

But the title of this post is "The Bayesian approach to life" not "The Bayesian approach to classifying silicon clusters" (Although I think you can still find references to my abortive attempts out there somewhere.) The thing is that Bayes can shine light statistics and probabilities you hear every day. Let's say you hear some expert talking head say there is a 50% chance that Twinkies cure cancer. You might think of this as knowledgeable assessment of the situation. But on the other hand, you see that the expert knows there are two options (cures cancer or doesn't) and he offers no additional insight.

Mostly it's just a nice way to think about uncertainty. Probably in an everyday sense is really about updating estimates based on new evidence. You can't do that right without Bayes. It's also useful when you have to deal with an observation that can have more than one cause.

I've been posting some highly abstract theories for project management and effectiveness. Well, actually there are even a level removed from that. They are more like analogies for theories. I'll probably still have one of those kinds of posts every once and a while. But I'm also a huge fan of http://www.manager-tools.com and one of the things that makes them so unique and so valuable is that they always give very concrete actions to take and they have a strict focus on behaviors. Heck they even provide ten step instructions on handshakes. So here it goes.

I've always struggled with how to drive and determine schedules. So I thought for my latest project I would focus on them with much more energy than I have in the past. At first I was drawn to abstract tools and prediction techniques, but I knew that it would be a huge job to try and roll out a change in operations across the larger team. I also didn't want to get bogged down in tool complexities. Instead I focused on what I could do on a feature team level. Out of necessity I finally boiled it down to the basics. Also, having recently finished The Influencer I realized that making broader change requires that you change vital behaviors and provide a highly visible example. Rhetoric doesn't work.

My key frustration on every project I have worked on is that there is so many opportunities to game the system. People mean well, but ultimately the bug economy takes over. I think ultimately this is because project leaders focus on the macro approach and confuse the reports with the project itself. I wanted to strip that all away.

The core of a project is tasks. Project management is getting those tasks done on time. (Yes, more management-tools jumping in here. Who Does What by When. ) Once the project tasks were broken down I did the following:

• Printed them all out in a large font

• Taped them to a huge roll of paper

• Drew circles by the tasks where each circle represented a half day of the estimated time it would take to complete the task.
• Pinned it to the wall

• Put a large number of smiley stickers next to the chart. The total number of smiley stickers represents our capacity for the milestone.

Now every day we have a fifteen minute standup meeting. Each developer on the project has to put two stickers on the chart. If they were working on a task, they put the sticker in a circle next to that task. Declares when a task is done and has a special sticker to signify it.

The great thing about this model is that it strips away the fluff and brings you back to the world of actual work. Test is there to make sure things are done when the dev says they are done. Plus you get a highly visible representation of the work done, the work left, and the capacity available in the milestone. Even better, the devs become better estimators because they get a feedback cycle.

So far so good. For the one or two folks who read this, please feel free to jump in and offer opinions.

9:17 AM

One counter intuitive rule of economics is that a country shouldn't allocate its resources based on how productive it is relative to other countries but rather relative to other things that country could do with those resources. For example, imaging that The United States was the most product textile manufacturer in the world. I doubt this is the case, but it illustrates the point. IOW a dollar invested in creating textiles in the US provides more return than other countries.

That doesn't necessarily imply that all textiles should be manufactured in the US. It really doesn't matter if the US can produce textiles with a 5% profit while other countries can only get 3%. What matters is if there are things that could be done with that dollar that produce more than 5%. If the US can produce boats at 10% profit it makes more sense to allocate the limited resources to produce boats than it would be to produce textiles. This is true even if it results in higher textile prices. I've hear economics defined as the study of allocating scarce resources that have alternative uses. This example goes a long way towards show exactly what that means.

It's also one of the turning points for making your own work more effective and it's key to becoming a leader in an organization. I'm sure that there are many tasks done by your reports or other people in an organization that you could do better. That doesn't mean you should do them. Instead ask yourself a few questions. What would happen if I let the other person run with it? Maybe it wouldn't be as good. But how much worse, really? What else could I do with my time? Is there a an area where my time has more leverage? Is there some other task I am uniquely qualified for?

Of course, this rule is even more interesting in the breach. Classical economics doesn't care if you generate your GDP from housing, financial markets, corn production, or car manufacturing. But because you can't really sell a house located in the US to someone in China and because the underlying assumptions of classical economics on individual behavior just isn't right, it turns out that it really does matter what generates your wealth. Similarly while you may be able to maximize use of your time by focusing on your strengths, the greatest lever you have to improve your overall skill portfolio is to focus on your weaknesses. Besides, there is rarely an area you can ignore and expect to be successful. 

Yesterday I spent an hour teaching origami at “Japan Day” for my son’s first grade class. It was frenetic. I’ve done this sort of thing before and it’s always the same.

It’s very popular. Kids love origami.

Most aren’t very good at it naturally. Being able to visualize and understand most origami folds is a rare trait.

But yet, there are always a couple who do quite well.

The kids are enormously demanding. No matter how hard I try to keep them on the same step they all want individual help and they want it now.

Then finally there are a couple who decided to open one of the advanced books lying around and start folding an eagle. It’s hard to convince them that they need to master the basics first.

By the end I had given out almost all of my “complex” origami examples to assuage the kids I couldn’t spend enough time with. There were misshapen star boxes and deformed cranes everywhere.

Of course, as I am in the middle of writing specs right now I can’t help but be reminded of my day job. Everyone wants something different from the spec and every audience has different levels of expertise and different concerns to be addressed. And everyone wants to be heard.

The only way to get through this process is engage and give them what they want. In the past I would sometimes push back and try to dictate what would be a proper question for a spec and what should be expected knowledge about file systems, IPv6, Windows architecture, or whatever. But that was a mistake. You end up with more bugs and a distrustful team. It takes a lot less energy to just keep pounding through the questions and shovel information to each audience as fast as you can. Eventually they will be sated and the team will be better off for the work. Everyone feels like they’ve been heard and you have a spec that will save a lot of heart-ache in the long run.

I sure picked an interesting time to start learning more about economics. Everything I've read, and experienced since my first post a while ago has gone against my first tentative steps. The first book I read the contradicts classical economics was The Black Swan, which argues against the linear Gaussian assumptions behind  much classical economic theory. Next came Predictably Irrational which shows how people are systematically irrational. Most people understand that we aren't all rational in the economic sense of the word, but we expect it to average out in a 'wisdom of the crowds' sort of way. "Predictably irrational" shows that the way in which we are irrational breaks economic models.

Finally I plowed through The Origin of Wealth. This book tries to reconstruct economic theory on the basis of  evolutionary theory, network theory, and automata. And then finally, we had real life impose itself in the form of a rather severe economic crisis.

Seeing the direction economics is going reminds me of the history of physics. Starting in the 17th and 18th centuries we developed analytical tools that allowed us to solve exactly for many physical phenomena. They work well, but they require very artificial circumstances to be solvable. That's why physics students have to solve problems like describing the electrical field around a charged sphere with stripes or a semicircular waveguide. Of course these things don't really exist in real life but since they can be solved in terms of Bessel functions and other things they can be written in a homework assignment. It's easy enough to solve for the motions of a pendulum, but only if you assumed a small angle.

Then came chaos. It turns out that there are lots of things you can't solve for. Sure we knew this all along, but now we have computational tools to show the behaviors that emerge from this. And even many of the things you can solve for exactly are better off dealing with numerically. But we also had the broad realization that you can't know the initial conditions well enough to predict results with any specified accuracy.

Now computational and storage power are moving us away from all of these abstractions. You just simulate based on first principles. This is clearly the way that micro economics can be unified with macro economics and there are those that believe that the same applies to physics as well. I was talking to an astrophysicist friend of mine and he said that for some projects they no longer think of light in the classical sense at all but rather count and ray-trace each and every photon. The computational power is there, and it's actually easier and more accurate given how dim the objects are. The telescope becomes the business end of a high-energy accelerator at the edge of the universe. 

What does this mean for policy, both for economic policy and and for planning complex projects like software?

The first stage of classical economics and 18th century physics corresponds to "laissez faire" free markets. When you hear an Ayn Rand follower like Greenspan talk about "irrational" investors, it's used in a pejorative sense. You can hear the echos of "Atlas Shrugged" and get the sense that all that's needed is for the market apply it's corrective power.

Once you realize that markets can get into a serious, non-linear state it becomes obvious that laissez faire is not the best solution. You institute damping functions in the form of regulations and fed action to prevent the markets from getting into this state. In software development you slow down progress through the use of check-in criteria, static code analysis, stabilization periods, and other overhead. But these are crude tools. You slow down the overall economy to minimize your risk on the downside.  It also treats developers and individual actors as fungible, interchangeable actors.

Economics is using simulation to move to the next step, but clearly it's a long way away from policy. The closest thing we have in software development is agile methods. Ultimately I think what this all comes down to is better tools with extremely fine grained tracking and integration of everything you know about a project. Rather than trying to remember to integrate vacations and meeting over head, integrate the calendar system and account for actual working hours per developer. With this data you can estimate the code churn introduced by feature as well as the expected bug count per LOC churned. Given the back and forth between churn and bugs, you can even plot out a reasonable glide path.

The problem with most project planning (and economic models for that matter) is that there are too many externalities. You can be enormously precise in your estimate to develop a particular module, but you have only rough swags on the external factors that really make or break a project. The answer to those external factors is to make them internal, based on actual data, and predictive.

 The funny thing is that pretty much everything a software developer does is tracked in some manner or another in software. Some bit of software can track code churn, meeting schedule, bugs, and work items. All it takes is integration.

You'll often hear disgruntled participants in any endeavor complain that they aren't being listened to or that they can't advance because of 'politics.'  This has always seemed to me like a lazy excuse. 'Politics' is just a word for how organizations work together to make decisions. Politics is the blend of formal decision making procedures, organizational hierarchy, and personalities that occur in the day by day business of getting things done.

I thought about this a recently as I read The Years of Lyndon Johnson: Master of the Senate. It turns out that in order to really understand the political career of  Lyndon Johnson you need to know about his time in the senate. This was when he was at the peak of his powers. Johnson transformed the senate from an institution of long speeches, but zero action into an institution that could actually pass bills. In the process he changed the meaningless role of majority leader into one with immense power. And he did it with the most brilliant execution of pure, cynical politics. Lyndon Johnson came into the senate with an overriding ambition to have power, but no apparent goal in using it. His stance on any given issue is exactly what it needed to be to increase his power. There were no burning ideas for him, just opportunities.

Once LBJ was elected to the senate, he made the first step that any good book on transitions will tell you. He quickly identified the real power brokers in the senate and closely allied himself to them. The most powerful man in the Senate at the time was Richard Russell.  Russell was an ardent racist and segregationist. For normal day to day business at the senate, there was much to commend him. But when it came to civil rights, he and the southern caucus were rock solid in blocking every bill. This was the real era of the filibuster. Because of this, and because LBJ was a southerner, Johnson ardently fought AGAINST civil rights throughout his first years in the senate. Indeed his first speech was a strong defense of southern segregation.

But ultimately there was something real in his rise to power as well. The way he was able to cement his power was not through speeches or even back channel maneuvering, but rather by taking a job that nobody else really wanted and turning into a seat of power simply by being useful. That job was Senate majority leader.

The role of Senate majority leader has no intrinsic power. You can't punish people and you can't reward them. Even today you typically don't see the most powerful senators in the spot of majority leader. But Johnson used the role to became a project manager without peer and made the senate function again. He could count votes better than anyone, knew when any bill was coming up from a sub-committee and knew what every senator needed and what they had to offer. He became the best source of knowledge in the senate and could use that knowledge to schedule the floor efficiently. He could horse-trade votes and control the workflow better than anyone had even attempted to in the past.

The most interesting example of all of this is the 1957 civil rights act. After his 1956 humiliation at the Democratic convention, Johnson realized that he had to actually pass a civil rights bill if he had any chance of being president. The problem for LBJ was that his power base was still the southern caucus and they wouldn't pass anything meaningful. The northern liberals stood on principle and demanded that they not compromise on a substantive bill with far more far reaching implications. Johnson didn't care so much about the contents of the bill, but he needed something to pass.

He squeaked through a toothless bill by using some of the most stunning political maneuvering ever seen in the Senate. What I find most remarkable about this incident is how difficult the Northern liberals were. Here they were, with the opportunity to pass the first civil rights bill since reconstruction and they were fighting for stronger wording that surely would have doomed its passage. Perhaps they were the truly cynical politicians because they apparently cared more about the press back home than they did about getting something accomplished. In the end they may have preferred the role of saying the truth but not being listened to. There is much to commend this role. It gives you the luxury of saying "I told you so" without taking responsibility or assuming the burden of success or greater leadership.

This is the danger of the principled stand. You can publicly decry the state of affairs in your team, watch in glowering anger as nothing changes, and then complain about the lack of promotion or how you were shot down for political reasons. The hard truth is that it's easy to be right. It's vastly harder to actually produce change. The real kicker is that even if you recognize this basic fact and try a couple of techniques to create change, you still may fail. Rather than give up, this is the moment to embrace politics, identify the key opinion leaders and rather than fight them, engage them. Small concrete steps beat flowery speeches (or long blog posts) any day.

Right now some of you might be thinking of some people with more of a knack for politics than software who always seem to get ahead. They seem to be corporate ladder climbers without any particular vision to push or principled stands. But before you dismiss those kinds of people as exactly the kind of employee you don't want to be, remember that among presidents Lyndon Johnson is surpassed only by Abraham Lincoln in what he actually accomplished for civil rights.

Recently,  I finished A Terrible Glory. It's a pretty good book chronicling Custer's destruction at Little Bighorn at the hands of the Sioux Indians led by Sitting Bull. When I started the book, I didn't expect to get a post out of it. After all, the military analogies are typically a little worn. But I should have known better. Here is one of the best-known disasters (at least from the POV of Custer) in military history, and somehow I didn't think there was any relationship to the politics or best practices associated with building and shipping quality software. Of course there are lessons to apply.

Just like any history, military or otherwise, the real situation is far more complex than any of the simple stories told afterwards. I had always imagined Custer out on an expansive plain with about a thousand troops getting annihilated by a huge number of hostile Indians. In reality he only had about 200 on hand and was fighting in a complex network of valleys and woods. You can see the actual terrain here. Also, while the contingent that Custer personally led was wiped out to a man, there were other parts of his command that were separate from him at the time that were not destroyed. The political wrangling over whether the failure was all Custer's or whether the surviving captains were to blame continued in the form of military inquiries and dime novels for years to come.

Naturally affixing blame and granting credit is a time-honored tradition in software projects as well. To get down to the level that approaches truth, you have to be skeptical of stories. Of course, everyone creates stories to explain past occurrences. "Project FOO failed because management couldn't say no to customer DCRs. It turned into endless feature creep."  "Project BAR was a success because we listened to customers and took their feedback seriously." Note that the same behavior was viewed as the key to success for one project and the cause of failure for the other.

Stories are like that sometimes. There is nothing wrong with trying to glean as much as possible from the story that becomes the official history, but those lessons are the easiest to get and the least likely help you along the next time around. You are going to learn a lot more from steering away from conventional wisdom and trying to piece together what was good about FOO and why BAR could have been a disaster. Also look for alternative explanations for what actually did happen.  Only by fully deconstructing a story will you get to something that helps you in your next project. Of course you still have to create stories, especially if you are a manager.

 Just don't believe them.

Even more challenging than evaluating a failed project after the fact is to try and figure out what a failed project looks while you are still in the middle of it and before its failed. It was interesting to read the accounts of the prelude to Custer's battle. It's full of soldiers making ominous pronouncements, Indian guides forecasting doom, and Custer laughing it all off. But you've got to take these sort of statements with a rather large grain of salt. It's likely that every battle has its share of folks predicting calamity. And much of it is probably apocryphal.  Unless you can somehow survey both successful and failed battles before they occur and correlate statements of immanent doom with actual outcomes, these pronouncements don't mean much. They do make good stories, though.

While it's important not to fall prey to the delusions generated by official history, it's even better to take a hard, cold look at a project underway and look for objective measurements that might be predictive. This is admittedly, a bit of a challenge. You don't get that many significant projects in the course of your career to generate statistically relevant data. Also, unless you are a researcher or a consultant you don't have the opportunity to evaluate a broad cross-section of projects. But it's still worth the try. Sometimes all it takes is the correct perspective to eliminate historical bias and The Halo Effect, but there is always some data that can help you on your quest. Make sure your metrics are as objective as possible. No, I don't have the answer. We know that bug counts are problematic. Lately I've been intrigued by code churn analysis. The nice thing about code churn is that there is no way to game the system. The only way to change the numbers is to stop checking in code. But paradoxically, this is also its greatest weakness. There is no management level action you can take from code churn analysis, except change the ship date. You can exhort people to fix bugs, you can get them to implement or cut a feature, but you can't do both of those and still control code churn. When management sees a metric they can't control, they typically become less enthusiastic, even if the metric is a useful one.

I've often wondered what makes an effective leader. While I certainly haven't completely figured it out, I have noticed one trait that seems essential. Great leaders aren't necessarily the people who make the right decisions, rather great leaders are people who create systems in which the right decisions get made. This is why you can judge the quality of a leader by watching what happens when they leave. If it's barely noticed, that is not an indication of their dispensability rather it's shows the durability of what they've created. Some of the most obvious examples come from the lives of brutal dictators. Just looking at how countries have changed with the death of a charismatic but brutal leader, bears this maxim out. But rather than delve on an obvious example, I want to jump back to the birth of America to talk about the creation of one of the most enduring systems in the world, the US economy.

The key figure in this drama is Alexander Hamilton. He was born in dubious circumstances in the east indies and found his way to New York as a young man. With the exception of Benjamin Franklin, he was undoubtedly the smartest of our founding fathers. He was also arrogant and brash with dreams of military glory. During the revolution he served as Washington's Aid de Camp and handled much of his correspondence, becoming a trusted advisor in the process. Oddly enough, he didn't play a large role at the constitutional convention, but was very instrumental in advocating its passage through writing a large portion of the Federalist Papers. He became the first expert in constitutional law and helped establish much of the early constitutional interpretation. His writings have been cited in supreme court cases up into the modern times. The Federalist papers were Hamilton's first great contribution to the American system.

Washington named Hamilton the Secretary of the Treasury and Thomas Jefferson the Secretary of State. And this is where Hamilton had his most lasting influence. He argued for and succeeded in getting the federal government to assume the state's debt in from the Revolution. This was controversial partially because the states had differing amounts of debt but mostly because of the centralizing influence a federal debt would incur. But Hamilton knew the importance of a strong financial system in enabling the growth of an economy. This first financial report also insisted that the bonds paid to Revolutionary veterans be honored. This sounds straight-forward enough, but it was controversial because many of the bonds had been bought by speculators for 10 cents on the dollar. Jefferson and others thought this was unfair and that the bonds should be invalidated and the veterans paid directly.

The distinction is important. No matter how on the side of right Jefferson may have been in opposing the assumption of debt or the honoring of government bonds,  the failure of either one of these measures would have been disastrous for the economy. More than anything, the government needed a good credit rating to continue operations, raise armies, and develop into a strong nation. If the government had shown a predilection to void government bonds or became almost impossible to work with because of the thirteen potential debtors, America would have had no ability to raise cash or act as a single entity. Besides, the bond purchases were legitimate transactions. The veterans got hard cash early and the speculators took on a lot of risk.

But what's interesting about this issue is that negating the bonds would require an explicit a decision, either by a ruler or a ruling body. Once it's expected that these kinds of issues are the fair and proper place for political decisions, the country is suddenly much more dependent on the ability of its leaders to make the right decision each and every time. Hamilton's solution didn't require ongoing government intervention to decide which bonds to honor. There will always be bad leaders, a robust system can survive them and wait them out. A system where leaders are required to make good decisions will crumble when a bad leader comes around.

The last component of Hamilton's financial plans that I'll discuss here was the imposition of an excise tax on whiskey. To set the background on this tax, it's important to understand how the economy worked in those early times.  In early America, the south and the west was always short of cash. Even the large plantations were largely cashless and self sufficient. They grew their own food, they built their own buildings, and used unpaid slaves for labor. When they did trade, it was often on the barter system. This was the world of the gentleman farmer. He had slaves and land, but couldn't interact with the larger economy. Even immediately after the revolution, England was the primary source of both finance and manufactured goods. This led to one of the bigger points of conflict between north and south, Federalists and Republicans. The Federalists were generally northern and were trying to foster a manufacturing society, hence they favored high prices for manufactured goods and supported a high import tariff. The Republicans were southern and provided raw materials but had to purchase manufactured goods. This meant the that tax burden fell heavier on them then it did on the Northerners.

The southern farmers despised banks because they owed money to them and had no steady flow of hard currency. They desperately wanted a greater money supply and the accompanying inflation because then it would be easier to pay back loans. They weren't too fond of England, because those were the folks to whom they owed money. But above all, they felt they had a right to live their chosen lifestyle and not be burdened by the leverage the banks had over them.  This freedom certainly sounds nice, but it rested not on inherent rights but on a untenable economic system.

Hard currency was not a fundamental part of the economy for much of the country. Only manufacturing states had a functioning monetary system. That means it was very hard to tax the western states. The one thing that came close to the pre-requisites of a currency, was whiskey. It acted both as a currency itself and a source of hard cash. It was the one product that farmers could easily manufacture and transport to cities. It was largely fungible and easily measured. It was drunk by city folk and country folk, young and old. And just like today, as a 'sin' tax, it was easier to justify politically. When you really got down to the details of the tax, it wasn't a very good deal for western producers. Small distillers paid more than large, commercial distillers. Also, the tax was difficult for westerners to pay because it was a production tax. Of course, the westerners didn't have money to pay at production time. This ultimately lead to the whiskey rebellion where a small insurrection over the tax was put down and Hamilton got his wish to be named a Major General.

But no matter how imperfectly the system was conceived and implemented, the system of broad tax burden and centralized monetary policy created American capitalism. It created a system that sustained itself and fueled the economic growth. Other countries and leaders have to decide in more detail how money is collected and from whom. This leads to corruption and lower confidence in the transparency of the government. The freedom for leaders to make low level decisions can be disastrous.

This pattern plays out in the corporate world all too often. Good executives understand how this works. Usually the misunderstanding comes from the junior participant, but not always. Inexperienced team members will come out of an executive review saying inane things like "We have to do this feature or implement things this way because so-and-so said so." What the person doesn't understand is that the exec was asking detailed questions and drilling down to make sure that the team is capable and can be trusted to make the right calls. They weren't trying to act as Super Program Manager. Even if they blow up over some specific detail, that doesn’t mean you have to go and update the spec right after the meeting (although you might.) It just means that the exec is nervous because he thought you got something wrong in an area he knows something about. As long as you can show you have thought through your decision and it wasn't being made by default, you are fine.

This kind of misunderstanding is common, but not dangerous. At worst the junior team member leaves the room confused about why none of the executive guidance was being followed slavishly. The truly dangerous cases come when an executive actually thinks their job is to make these decisions and act as Super PM.  This just doesn't work in our business. The problem is that once you are in a room with two other people, no matter how junior they may be, those other two people know more than you do. Even if there is just one other person, that person knows something that you don't. Once you get into a leadership role, that disparity accelerates geometrically. That trick you learned back when you hacked FORTRAN 66 may not mean much to today's C# developer. The problem is that we are all competitive, and sometimes we are most competitive with our underlings. Leaders feel justified in their position when they win an argument. But often, those underlings are right, they just aren't as good at arguing or they don't have the review structure to back up their claims. Since brains are our key asset in this business the worst thing you can do with such a person is win an argument. By 'winning' the argument you have just ensured that whatever unique knowledge that person has will remain locked up and unusable by you or your organization.  Instead you have to create a system that doesn't require you to make these decisions, or even better sees it as an organizational failure when you do. This creates an economy of ideas which can be much more powerful and unifying than even the whiskey economy.

For those of you who might be interested in knowing more about what I work on, here's a link to my manager talking about and demonstrating Essential Business Server.

http://edge.technet.com/Media/Essential-Business-Server-EBS-demo-with-Bjorn/