I'm re-reading "The Brothers Karamazov." I read it for the first time and college and it affected me deeply. It's not quite the same to read it again almost twenty years later. I've changed and I think I'm past my days of existential anguish. But I'm finding new revelations that I'm sure I missed the first time around.

For example, you wouldn't think that Dostoevsky would have anything to say about software development. But any project is really about human relationships and nobody wrote about that better than 19th century Russians.

One of the characters is the elder Zosima. He is a saintly monk who took on of the brothers (Alexei) under his wing. Zosima dies fairly early on in the book, but on his death bed he gives a very long speech where he tells his life story and outlines his overall philosophy. One of the themes that recurs in the book that Zosima expounds upon greatly is the subject of personal responsibility. At one point he quotes his dying brother as saying:

Know that this is the truth and that every one of us is answerable for everyone else and for everything.

And later on

If the evil deeds of men sadden you too greatly and arouse in you an anger you cannot overcome and fill you with the desire to wreak vengeance on the evil-doers--fear this feeling most of all, and at once go and seek suffering for yourself, because you too are responsible for the evil deeds of all men. Bear that ordeal and your desire for revenge will be quenched when you understand that you were guilty yourself for having failed to show the light to the wicked as a man without sin could.

Just substitute the word "bug" for "sin" and you've got one of the clearest explanations of teamwork ever written. The real power of this formulation is that there is no limit to its inclusiveness. The responsibility that Zosima talks about extends beyond your formal responsibility, beyond your team, beyond your product, and even beyond your company.

In some ways this is absurd, you can't expect me to be responsible for a bug in a proprietary application developed for another platform in another country. But that's just the point. People are loathe to take responsibility for things over which they have no power. But the fact of the matter is that nobody has power.

I was talking with a tester the other day who was bemoaning the fact that all he could do was file bugs and argue that some behavior should be changed. He didn't have the power to make the decision like I did. It was difficult to explain that I had no power at all and that any influence I had came from that realization. All I could do was persuade and learn. Mostly it's learn. If I had real power, I wouldn't have to explain my reasoning. As it is, I've found the most effect way to get people behind what I advocate is to tell them what they already think. Usually it's right, or right enough. The times I truly try to change opinion about something it's a long laborious process to move the needle. When I do change opinion, the change happens long after my argument and can't be attributed to anything I said or did. More often it's my opinion that's changed. Some people may call that losing an argument. I call it learning.

This all means it's wrong to complain about getting overruled by management or by a teammate.  If you are overruled it's your fault for not presenting your case in a way that could be consumed. It may even be your fault because you are simply wrong. Some people seem to enjoy getting overruled because it helps them avoid responsibility. If they get their way in one thing, they keep coming back with more extreme arguments until someone just says no. At that point they gripe and complain, but secretly they rejoice because they have reserved the right to say "I told you so."

I once asked a manager about five levels above me what the most frustrating part of his job was. He said that he wasn't free to run his business the way he saw fit. He wasn't given complete discretionary control of his budget. I don't think there is any level at which this frustration disappears. The Elder Zosima teaches us that once you are able to accept responsibility for things that are manifestly outside of your influence, it becomes much easier to accept responsibility for things over which you have limited influence. Once you are responsible for everything you are free to focus on the actual problem rather than making sure blame is placed where it belongs.

There is an interesting corollary to this maxim. Once you accept responsibility for things outside of your nominal authority you can find yourself greatly expanding your actual influence. Contrary to popular opinion, influence at the highest levels is about working with the limited authority to mandate and force. The bottom line is that if you think the only thing that's holding you back is lack of power, then you aren't ready to lead.

People tend to argue from three perspectives, or ways of thinking. Some of the greatest disconnects occur when people are argue the same point from different perspectives.

• Principles--The principled thinker holds on to basic truths and measures other ideas against these truths. "We hold these truths to be self-evident…"

• Systems--Systems thinkers are less concerned with right, wrong, or absolute truth. Instead they focus on the outcome and putting together the appropriate system to produce the desired result.

• Details--Bottoms up work. Detailed thinkers figure if each step is handled correctly the journey will take care of itself.

Some of the great political debates have at their core a conflict between these different modes of thought. The systems thinker comes along with studies showing how they can reduce crime or some other undesirable behavior through different programs or regulations. The mistake the system thinker makes is that they assume that the principled thinker has a shared goal with them. That isn't always the case. Sure the principled thinker has no problem reducing crime, for example. But perhaps they value individual freedom greater than a world with less crime. It's more important that the individual  has the freedom to make the choice and then bears the consequence of that choice. More crime does not disprove their perspective. All the while the detailed thinker looks at individual cases and thinks that both principled and systems thinkers don't know the reality on the ground.

If you haven't noticed from reading some of my other posts like The Bug Constitution, I tend to be a systems thinker. I figure as long as you get the right rules together, and get the economy structured correctly, the right outcome will happen almost automatically. Principled thinkers tend to confuse this approach with detailed thinking, but it's not quite the same thing.

The trick is that all three modes of thought are critical, none of them are inherently superior to the others. One of the great pitfalls is that people can become very successful with one mode of thought and stick to it even when other approaches would be more effective.

Of course, you can see all three modes in full display during a project shiproom. Everyone is trying gauge the quality of the product and our ability to meet schedule. The systems thinker says "We just need the right triage rules and bug bars in place and we can track quality and schedule analytically." The detail thinker claims "Bug counts are meaningless. You have to look at all the individual bugs and run the product daily to understand what the quality is." The principled thinker says "You are both wrong. Product quality is nothing but features that add value. It's more important that we find time to had high value features than every pixel gets properly aligned. Also, we are experienced developers we don't need a bunch of rules to help us determine which bugs to fix."

As you can imagine, nobody changes anybody's mind. The real problem is that they are all correct. Sometimes a value and a vision for a product can be so compelling, that users will forgive a lot of niggling bugs and design problems. (Spolsky has a great post on this.) Other times a relentless focus on every last bit of fit and finish can transform a product from a relentless pile of annoyances into a pure joy to use. Other times only broad customer analytics and trends get help you get your heads around what's really going on. What's tough about this is that all perspectives are right. It takes leadership to balance them and know which one is more important for a given product, product version, and stage of release. The answer is not always the same and people who get it right once, often get it wrong the next time because they don't adapt to a different world.

I've got another anecdote from Atul Gawande. This one is about hernias.  Repairing hernias is one of the simplest procedures in the surgical field.  When you are learning to be a surgeon, you can start doing this during your first year. It takes about an hour and a half to repair a hernia and the recurrence rate is somewhere around ten or fifteen percent, depending on the hospital. It doesn't seem like an area that requires greater efficiencies. It's not an expensive procedure and it's generally quite successful. But there is a hospital near Toronto called the Shouldice Hospital that challenges this conventional viewpoint.

At Shouldice, repairs take somewhere between thirty and forty five minutes and the recurrence rate is approximately one percent. There is no magic knowledge or other breakthrough. They just have a single minded focus on hernia repairs. Each surgeon at Shouldice performs between six and eight hundred repairs a year. This is more than a typical surgeon performs in a lifetime.

This story resonated with me during my visits to various customers over the past couple of years. I've notice that no matter how knowledgeable the customer was, there was always something that they used a consultant for. They especially used consultants for tasks that they didn't perform regularly, like deploying new services or upgrading Exchange. At first this was a little surprising. After all, we put a lot of work into making our products simple to install and it was a little unnerving that no matter how simple it was, there was always a specialist doing it. This was especially surprising given the depth of knowledge of the customers we visited.

The hernia story helps explain it. Most businesses don't install Exchange often enough that they have acquired any particular expertise. It's far more efficient to have a specialist that services a number of customers and installs Exchange on a very regular basis. It doesn't speak to any deficiency on the part of the in-house IT staff, or even to the intelligence of the consultant. It's all about practice. Outsourcing can greatly minimize risk. A ten percent recurrence rate on a hernia repair may not sound all that bad. But surgery is unpleasant enough that I know if I needed a hernia repaired, I would much prefer the one percent rate demonstrated by Shouldice.

Statistics as reported in most newspapers never quite satisfy me. It's not that they are lying, but there is never quite enough there. The flu is a great example. I was just reading this great article in Slate about how they came up with the 33,000 deaths per year from the flu http://www.slate.com/id/2218367. Turns out they did regression analysis and looked at the correlation between the severity of the flu season and the number deaths on a week by week basis. From this they estimate that 34,000 people died that would not have died otherwise. This is all well and good, but it doesn't answer the real question. How should the government react? What will the economic impact be? What's the difference between pandemic flu and seasonal flu?

Because, let's face it. The flu doesn't kill anybody. After all, we are all going do die. All the flu does is speed death up for some folks. Some articles make some cursory acknowledgment of this fact by noting whether the flu is killing young people or not. But this there is never a good measure associated with it. Reminds me of a case I was reading at about the time of the peanut contamination stories. A man was talking about how his mother had fought through cancer and was still weak and on a respirator but seemed to be making a comeback when she got sick and died from contaminated peanuts. Sure this is a tragic story and there is a chance that his mother would have had many more years if it hadn't been for the peanuts. But really, cancer killed her. Let's take a look at making these numbers meaningful.

For example, you could do the same regression analysis you did above but then continue the analysis moving forward to see if there is a subsequent dip death rate in the coming months. If the flu was killing people who were already pretty close to death, you might expect to see a decrease in death rate over the next year as those deaths that were 'paid forward' aren't recorded. After some work, rather than recording a 'mortality' rate you could state a rate for 'having more than 6 months taken off your life ' or 'number of years taken from population.'

Another death related statistic I've never liked is life expectancy. I've go "The Economist Pocket World in Figures 2008 edition" on my desk (of course.) I open it up and see that the life expectancy for Japan is 82.5 years. For the United States it's 78.2, just below Cuba and Denmark. But then on another page I can see that the median age in Japan 42.9. The United States doesn't even show up on the list because it's too low. Well, that puts the life expectancy in perspective. It's a lot easier to have a longer life expectancy if you don't have young people around to die and mess up your stats. It also shows how life expectancy in isolation doesn't help much for policy because you don't know who is dying.

For example, I am personally more interested in the life expectancy for a 40 year old male than I am for the entire population. A better chart would be percent chance of dying in a given year as a function of age. There are lots of cool things we could learn about and apply to policy. I would expect a spike in the first few years because of birth defects and childhood illnesses. After that you are almost certainly going to get a spike around 16 when US kids get driver's licenses and some relatively high rates for the next few years due to suicide and teenagers being crazy. Perhaps there would be another spike at 21 for hitting the legal US drinking age. It goes down until thirty and stays low through the peak parenting and earning years. Your just too busy to die. Things pick up in your fifties as bad living habits start to take their toll. I would guess you would see another spike around retirement but I suspect things level off for quite a while until you get truly old. These are all guesses and it would take a lot more data to come up with this sort of chart, but it sure seems more useful and more actionable.

And don't even get me started on dramatic illustrations of the amount of money in the economy especially since the banking crisis. The lesson is that real numbers take work to figure out. Even stats presented in good faith don't tell the real story, so you can be sure that numbers presented to further an agenda they are useless.

According to my Kindle, I am about 80% through reading Leadership on the Line. That's close enough that I'll write a brief review. Don't get wrong. I'll finish the book. But business books rarely finish strong. If I wait until I've clicked the last "next page" button, I probably won't write a review at all.

But getting to the point, I highly recommend the book.

The authors' primary thesis is that there are two kinds of leadership challenges. The first is technical. Technical challenges are the sort that can be solved through traditional management techniques. With a technical challenge, you don't need your team to do anything differently. They just need to do things better. An adaptive challenge, on the other hand, requires you to truly lead and make fundamental changes in your organization. It requires you to change hearts and minds and even shift the culture in a new direction. The book's authors posit that one of the biggest mistakes in leadership is to mistake an adaptive challenge for a technical one.

One example that's been making the headlines lately is the car industry. GM has been having problems for quite a while now, even before the current financial crisis. A technical response to this problem would be to improve marketing and reduce costs for its highest margin vehicles. GM would still focus primarily on large vehicles, at least in the U.S. An adaptive response would have been to invest in smaller, more efficient vehicles even when Hummer sales were robust. Getting the suppliers, unions, and dealers in lined up to make this change would have been enormously difficult and dangerous to your career. But if successful, GM would be in a better position today.

Technical challenges respond well to straight-forward management in existing frameworks. Adaptive challenges require leadership. Adaptive challenge is also dangerous. Most of the book deals with the pitfalls and hazards of trying to make lead an adaptive change.

For example, when you deliver an uncomfortable message, people will complain about your style without addressing the underlying issue. Another pitfall is getting seduced by your most ardent supporters and moving to far from what people will accept. You may not understand the nature of your powerbase and start to issue edicts that aren't followed regardless of your role authority. Someone's values may be threatened by the change you represent. You have to tread carefully to avoid destruction.

One of the things I liked best about this book is that the examples are almost all failures. Too many business books focus only on the success stories and are easily skewed by survivor bias. It also brought to mind the techniques of some of the best books on leaders I've read including Abraham Lincoln, Lyndon Johnson, and Alexander Hamilton, both for there successes and their failures.

The one criticism I have is that there are few precise techniques. There are a lot of abstract warnings and things to keep in mind. But without actionable steps these theories can be debilitating to someone facing an adaptive challenge. However, over the past couple of weeks I've come to think of this book as a partner to other sources of leadership information including Influencer and Manager Tools. Manager tools does a great job on the nuts and bolts of technical management. I don't mean this in a pejorative sense. After all, the day to day business of getting things done and working within an organization is critical. "Influencer" gives some specific methods for creating adaptive change. "Leadership on the Line" puts these two in context by describing the different kinds of challenges leaders face and describing the pitfalls and general approaches to tackling adaptive challenges.

Having kids can provide great examples of pathological team behavior. It's not that your team behaves like children. On the contrary, the reasonable explanations and subtle arguments deployed by adults can serve to obfuscate fundamental issues. But having kids helps to emphasize that all human beings exhibit the same basic traits. Kids just tend to exhibit these behaviors in such an obvious way that they can shed light on team and work dynamics. In particular, one recurring scene in our household has made me think more critically about what it means to be "done."

I am usually the one to drive the kids to school. I have two kids that go to two different campuses of the same school. One is in first grade the other is in second. If I can get both kids to the North campus by 7:50 AM, the South campus kid can hop on a shuttle and make it the rest of the way herself. Then I can go straight to work. If I'm late, we miss the bus and I have to drive a few miles to south campus and drop off the South campus kid myself. Missing the bus gets me to work almost a half hour later than I otherwise would have. You might say I'm motivated to get to get the kids ready on time.

I've learned that it's not enough to ask the kids if they are ready or to list off the things they need to get done before they can get in the car. The only thing that really matters is if they are in the car or not. Often the biggest delays happen between the time they believe they are ready to go and when they actually buckle up their seat belts. They have to go to the bathroom, get a stuffed animal, brush their teeth, or any number of last minute items. What's worse is that because they think they are done they tend to play around, argue, and generally make noise as I'm trying to make sure I have everything I need. I can't concentrate and deep in my heart I know that the kids aren't really ready if they aren't in the car.

That means in the morning you'll often hear me tell the kids to get in the car with a fair degree of urgency. Getting them in the car focuses their minds on all the little things they need and also gives me thirty seconds of quiet to help me make sure I haven't forgotten anything.

You know where this is going. It's not enough to say you are done. You have to report that you are done, something must be delivered, and there has to be some objective measure or someone acting as a gatekeeper. The bar doesn't have to be high, but it has to be higher than the developer's say so. How often have you heard "It's done, there are just a couple of bugs." or "It's done, but I have to check-in and run some final tests."

That just isn't good enough. In order to run a project successfully you need to identify simple, objective criteria for when every task is done. It's also not good enough to have milestone criteria. If you don't know how much of your work is done on at least a week by week basis if not a day by day basis, human nature will kick in and things will pile up.

And I do mean simple. It doesn't have to be end to end scenario tests, although it is always nice if a task can light up actual functionality. For example, here is one possible approach:

• Developers must write accompanying unit test for every task.
• Tester code reviews unit test to verify that it does what it should do.
• Developer runs unit test in presence of tester and sees that it passes. If you are using Smiley Project Management, the tester can place a star sticker by the task.

I was thinking about government analogies for software development organizations while I was unclogging our bathroom sink. I originally made this analogy after in this post about the supreme court. But I didn’t make any analogies between the roles of software development and the roles of government.

The first couple were easy to figure out:

• Judicial—This is clearly test. They enforce the rules and decide when things are done.
  
• Legislative—This is the real work done legislation, so it must be development.
  
• Executive—This one is management. True, management has more direct power over what gets done than the President does. But they both set direction and have veto power.

Note that program management doesn’t show up here. But then it finally dawned on me. PMs don’t exist at all companies. They are sometimes despised, but at their best they represent customer voices.

PMs are lobbyists.

Over the last week I went from not knowing about a new technological movement to seeing it everywhere and becoming a believer. Here's the story.

I'm something of a data wonk. On, my desk you'll find a copy of the economist book of facts and figures. I read it for fun. I often download the latest spreadsheets from OMB on the latest economic figures. I'm also the guy that spends time digging through the Customer Experience Improvement Program data looking for gems on how our customers use our products. And yes, I mine Twitter too.

But what's always frustrated me about this data is that it's hard to analyze and link together. The OMB data comes in a pretty nice spreadsheet but I always have to tweak it to get what I want. I can't link it to other data sets without significant work. This seriously limits the insights I might be able to achieve. I have to be pretty confident about what I'm going to find before I look for anything beyond the obvious. I've got so many data streams available and I just want to have the freedom to dump into an OLAP cube and play. Dealing with the limitations is an eternal source of pain for me. I live this stuff so I was primed.

And then there was the past week.

First I hear about Wolfram Alpha. It looks like a search engine but it calculates answers from data gleaned from the web rather than matching strings. Cool, I thought. But nothing started to flash in my head. Then I heard about a project from Amazon where they are publishing government data in an accessible format based on AWS. Also cool. I made a mental note to check it out the next time I had a hankering to go to OMB. But it still didn't click.

But then I was whiling away the time with my Zune at the kids swimming lessons and I thought I'd listen to a TED talk from Tim Berners-Lee. It's an idea of his for something called linked data. He's advocating for everyone to provide raw data over http for tools and services to build on. This talk is when everything came together. When you see three news items in the same area in a short period of time, you know something is up.

In some ways these projects seem unrelated, but that makes the meme even more powerful. Different people are coming to the same problem from different angles. Wolfram Alpha needs to data to work on. Amazon may be just looking for something to show off their platform with, but then again that's all Microsoft had in mind when it initially built Terraserver. Now Google Maps and Virtual Earth are big deals. Let's just I'm stoked.

I recently bought my wife an Amazon Kindle for Christmas. Yes, I know it's March but they were backordered. But at least this way I was able to give the Kindle two. Anyway, she loves it.

The thing that's striking about it and the reason I am writing this post is that I'm impressed with how the Kindle completes the scenario. By this I mean that the designers were relentlessly focused on the book reading experience and didn't stop at the natural boundaries if it meant that this experience would be compromised. There are always compromises to be made, but whenever they were faced with a decision they compromised in areas outside of the core scenario and never strayed from the fundamental vision.

Take Whispernet for example. It allows the kindle user to buy and download books from anywhere. They easy solution would have been to require synching with a PC in V1, WiFi in V2, and building mobile apps later on. But they wanted to make sure there were nothing in the way of buying and reading books . I don't know how they pulled this off, but I'm sure it included some animated discussions both internally and with the various wireless providers. It's a big bet.

The next big call is the screen. It's a monochrome LCD screen. It's not well suited to rapidly changing graphics or other applications. The screen flashes a bit when you turn the page. But when you think about it, a book has a similar experience. The there are few really nice things about it from a perspective of completing the scenario. One obvious benefit is that's cheaper than the alternatives. But the two factors that I believe really drove the decision are battery life and readability. The screen doesn't draw any power just to display a page. It only draws a minimal amount to change the display. This lets you treat the Kindle much more like a book. You don't have to think about charging it the same way you need to with a cell phone or a notebook. The second benefit is the readability. I was a little surprised at this one, but reading something that's not backlit is might easier on the eyes and much more suited to a book-like experience.

But it's the compromises that really illuminate these decisions. It's easy enough to articulate how to complete a given scenario. It's difficult to make the tradeoffs necessary to make it happen. When you start out you have to know that you won't be able to complete every experience. Some of them will be decidedly sub-par. Some you'll cut altogether. For example, don't rely on your Kindle to be a web browser. But you can't nail the important scenarios unless you make these compromises.

This is where leadership comes in. Someone needs to provide the vision to keep the project on track. Even more importantly they need to provide the vision in a way that is easily digestible and enables everyone to easily make decisions based on the same clear criteria. Completing the scenario is just as much about saying 'no' to other great features as it is to spending the resources to get that last mile.

Joel Spolsky talks about this a bit too. http://www.joelonsoftware.com/items/2006/12/09.html. He focuses more on the benefits to simplicity to the design then on clarity of vision. He is certainly right, but I'm talking more about the less appealing underside to the same discussion. Sometimes you have to have to make decisions that cut or reduce really good ideas and perfectly good scenarios just to make the core scenario complete and not just 80%.

Anyway, back to my book.

One of my favorite theorems is Bayes' theorem. (There, I admit it. I'm a geek.) At its heart, Bayes' theorem is a way to update a probability estimate with new information. Say for instance you hear that there is a one in ten chance of rain today. Later on, you see some ominous clouds. It's reasonable to think that the probability is now a bit higher in light of this new observation. But yet the original ten percent forecast still has some value. Bayes' theorem calculates the updated probability based on the original ten percent forecast, the chance of ominous clouds showing up independent of any forecast of rain, and finally the probability of ominous clouds on days that it really does rain. Frankly, you don't have to have the precise formulation memorized to make some of it. The Wikipedia link above is good enough if you want to dive in a little bit deeper.

I first encountered Bayes' theorem in graduate school. One of the atomic physics grad students who had gotten his PhD a couple of years earlier came back to talk about his current work. It was some government job tracking radon in houses. I don't remember exactly how he used Bayesian analysis, but what struck me is that it seemed to be able to create data from nothingness. He had some very sparse data about Radon instances and went on to construct a pretty thorough model of radon concentrations across the country. It retrospect I understand that he took raw data and applied additional knowledge like Gaussian distributions and other assumptions that don't seem like knowledge in themselves but helped extend the base data.

But at the time, something just didn't seem right about it. So I looked into it more. Later on I used it my own research. There was a Bayesian classification algorithm that I found somewhere that would identify clusters of data from a raw, uncorrelated source. My research involved finding optimal structures for silicon clusters and I used this Bayesian classification algorithm to find trends among the vast library of locally stable clusters I found though simulated annealing, Monte Carlo, and genetic search algorithms.

But the title of this post is "The Bayesian approach to life" not "The Bayesian approach to classifying silicon clusters" (Although I think you can still find references to my abortive attempts out there somewhere.) The thing is that Bayes can shine light statistics and probabilities you hear every day. Let's say you hear some expert talking head say there is a 50% chance that Twinkies cure cancer. You might think of this as knowledgeable assessment of the situation. But on the other hand, you see that the expert knows there are two options (cures cancer or doesn't) and he offers no additional insight.

Mostly it's just a nice way to think about uncertainty. Probably in an everyday sense is really about updating estimates based on new evidence. You can't do that right without Bayes. It's also useful when you have to deal with an observation that can have more than one cause.

I've been posting some highly abstract theories for project management and effectiveness. Well, actually there are even a level removed from that. They are more like analogies for theories. I'll probably still have one of those kinds of posts every once and a while. But I'm also a huge fan of http://www.manager-tools.com and one of the things that makes them so unique and so valuable is that they always give very concrete actions to take and they have a strict focus on behaviors. Heck they even provide ten step instructions on handshakes. So here it goes.

I've always struggled with how to drive and determine schedules. So I thought for my latest project I would focus on them with much more energy than I have in the past. At first I was drawn to abstract tools and prediction techniques, but I knew that it would be a huge job to try and roll out a change in operations across the larger team. I also didn't want to get bogged down in tool complexities. Instead I focused on what I could do on a feature team level. Out of necessity I finally boiled it down to the basics. Also, having recently finished The Influencer I realized that making broader change requires that you change vital behaviors and provide a highly visible example. Rhetoric doesn't work.

My key frustration on every project I have worked on is that there is so many opportunities to game the system. People mean well, but ultimately the bug economy takes over. I think ultimately this is because project leaders focus on the macro approach and confuse the reports with the project itself. I wanted to strip that all away.

The core of a project is tasks. Project management is getting those tasks done on time. (Yes, more management-tools jumping in here. Who Does What by When. ) Once the project tasks were broken down I did the following:

• Printed them all out in a large font

• Taped them to a huge roll of paper

• Drew circles by the tasks where each circle represented a half day of the estimated time it would take to complete the task.
• Pinned it to the wall

• Put a large number of smiley stickers next to the chart. The total number of smiley stickers represents our capacity for the milestone.

Now every day we have a fifteen minute standup meeting. Each developer on the project has to put two stickers on the chart. If they were working on a task, they put the sticker in a circle next to that task. Declares when a task is done and has a special sticker to signify it.

The great thing about this model is that it strips away the fluff and brings you back to the world of actual work. Test is there to make sure things are done when the dev says they are done. Plus you get a highly visible representation of the work done, the work left, and the capacity available in the milestone. Even better, the devs become better estimators because they get a feedback cycle.

So far so good. For the one or two folks who read this, please feel free to jump in and offer opinions.

9:17 AM

One counter intuitive rule of economics is that a country shouldn't allocate its resources based on how productive it is relative to other countries but rather relative to other things that country could do with those resources. For example, imaging that The United States was the most product textile manufacturer in the world. I doubt this is the case, but it illustrates the point. IOW a dollar invested in creating textiles in the US provides more return than other countries.

That doesn't necessarily imply that all textiles should be manufactured in the US. It really doesn't matter if the US can produce textiles with a 5% profit while other countries can only get 3%. What matters is if there are things that could be done with that dollar that produce more than 5%. If the US can produce boats at 10% profit it makes more sense to allocate the limited resources to produce boats than it would be to produce textiles. This is true even if it results in higher textile prices. I've hear economics defined as the study of allocating scarce resources that have alternative uses. This example goes a long way towards show exactly what that means.

It's also one of the turning points for making your own work more effective and it's key to becoming a leader in an organization. I'm sure that there are many tasks done by your reports or other people in an organization that you could do better. That doesn't mean you should do them. Instead ask yourself a few questions. What would happen if I let the other person run with it? Maybe it wouldn't be as good. But how much worse, really? What else could I do with my time? Is there a an area where my time has more leverage? Is there some other task I am uniquely qualified for?

Of course, this rule is even more interesting in the breach. Classical economics doesn't care if you generate your GDP from housing, financial markets, corn production, or car manufacturing. But because you can't really sell a house located in the US to someone in China and because the underlying assumptions of classical economics on individual behavior just isn't right, it turns out that it really does matter what generates your wealth. Similarly while you may be able to maximize use of your time by focusing on your strengths, the greatest lever you have to improve your overall skill portfolio is to focus on your weaknesses. Besides, there is rarely an area you can ignore and expect to be successful. 

Yesterday I spent an hour teaching origami at “Japan Day” for my son’s first grade class. It was frenetic. I’ve done this sort of thing before and it’s always the same.

It’s very popular. Kids love origami.

Most aren’t very good at it naturally. Being able to visualize and understand most origami folds is a rare trait.

But yet, there are always a couple who do quite well.

The kids are enormously demanding. No matter how hard I try to keep them on the same step they all want individual help and they want it now.

Then finally there are a couple who decided to open one of the advanced books lying around and start folding an eagle. It’s hard to convince them that they need to master the basics first.

By the end I had given out almost all of my “complex” origami examples to assuage the kids I couldn’t spend enough time with. There were misshapen star boxes and deformed cranes everywhere.

Of course, as I am in the middle of writing specs right now I can’t help but be reminded of my day job. Everyone wants something different from the spec and every audience has different levels of expertise and different concerns to be addressed. And everyone wants to be heard.

The only way to get through this process is engage and give them what they want. In the past I would sometimes push back and try to dictate what would be a proper question for a spec and what should be expected knowledge about file systems, IPv6, Windows architecture, or whatever. But that was a mistake. You end up with more bugs and a distrustful team. It takes a lot less energy to just keep pounding through the questions and shovel information to each audience as fast as you can. Eventually they will be sated and the team will be better off for the work. Everyone feels like they’ve been heard and you have a spec that will save a lot of heart-ache in the long run.

I sure picked an interesting time to start learning more about economics. Everything I've read, and experienced since my first post a while ago has gone against my first tentative steps. The first book I read the contradicts classical economics was The Black Swan, which argues against the linear Gaussian assumptions behind  much classical economic theory. Next came Predictably Irrational which shows how people are systematically irrational. Most people understand that we aren't all rational in the economic sense of the word, but we expect it to average out in a 'wisdom of the crowds' sort of way. "Predictably irrational" shows that the way in which we are irrational breaks economic models.

Finally I plowed through The Origin of Wealth. This book tries to reconstruct economic theory on the basis of  evolutionary theory, network theory, and automata. And then finally, we had real life impose itself in the form of a rather severe economic crisis.

Seeing the direction economics is going reminds me of the history of physics. Starting in the 17th and 18th centuries we developed analytical tools that allowed us to solve exactly for many physical phenomena. They work well, but they require very artificial circumstances to be solvable. That's why physics students have to solve problems like describing the electrical field around a charged sphere with stripes or a semicircular waveguide. Of course these things don't really exist in real life but since they can be solved in terms of Bessel functions and other things they can be written in a homework assignment. It's easy enough to solve for the motions of a pendulum, but only if you assumed a small angle.

Then came chaos. It turns out that there are lots of things you can't solve for. Sure we knew this all along, but now we have computational tools to show the behaviors that emerge from this. And even many of the things you can solve for exactly are better off dealing with numerically. But we also had the broad realization that you can't know the initial conditions well enough to predict results with any specified accuracy.

Now computational and storage power are moving us away from all of these abstractions. You just simulate based on first principles. This is clearly the way that micro economics can be unified with macro economics and there are those that believe that the same applies to physics as well. I was talking to an astrophysicist friend of mine and he said that for some projects they no longer think of light in the classical sense at all but rather count and ray-trace each and every photon. The computational power is there, and it's actually easier and more accurate given how dim the objects are. The telescope becomes the business end of a high-energy accelerator at the edge of the universe. 

What does this mean for policy, both for economic policy and and for planning complex projects like software?

The first stage of classical economics and 18th century physics corresponds to "laissez faire" free markets. When you hear an Ayn Rand follower like Greenspan talk about "irrational" investors, it's used in a pejorative sense. You can hear the echos of "Atlas Shrugged" and get the sense that all that's needed is for the market apply it's corrective power.

Once you realize that markets can get into a serious, non-linear state it becomes obvious that laissez faire is not the best solution. You institute damping functions in the form of regulations and fed action to prevent the markets from getting into this state. In software development you slow down progress through the use of check-in criteria, static code analysis, stabilization periods, and other overhead. But these are crude tools. You slow down the overall economy to minimize your risk on the downside.  It also treats developers and individual actors as fungible, interchangeable actors.

Economics is using simulation to move to the next step, but clearly it's a long way away from policy. The closest thing we have in software development is agile methods. Ultimately I think what this all comes down to is better tools with extremely fine grained tracking and integration of everything you know about a project. Rather than trying to remember to integrate vacations and meeting over head, integrate the calendar system and account for actual working hours per developer. With this data you can estimate the code churn introduced by feature as well as the expected bug count per LOC churned. Given the back and forth between churn and bugs, you can even plot out a reasonable glide path.

The problem with most project planning (and economic models for that matter) is that there are too many externalities. You can be enormously precise in your estimate to develop a particular module, but you have only rough swags on the external factors that really make or break a project. The answer to those external factors is to make them internal, based on actual data, and predictive.

 The funny thing is that pretty much everything a software developer does is tracked in some manner or another in software. Some bit of software can track code churn, meeting schedule, bugs, and work items. All it takes is integration.

You'll often hear disgruntled participants in any endeavor complain that they aren't being listened to or that they can't advance because of 'politics.'  This has always seemed to me like a lazy excuse. 'Politics' is just a word for how organizations work together to make decisions. Politics is the blend of formal decision making procedures, organizational hierarchy, and personalities that occur in the day by day business of getting things done.

I thought about this a recently as I read The Years of Lyndon Johnson: Master of the Senate. It turns out that in order to really understand the political career of  Lyndon Johnson you need to know about his time in the senate. This was when he was at the peak of his powers. Johnson transformed the senate from an institution of long speeches, but zero action into an institution that could actually pass bills. In the process he changed the meaningless role of majority leader into one with immense power. And he did it with the most brilliant execution of pure, cynical politics. Lyndon Johnson came into the senate with an overriding ambition to have power, but no apparent goal in using it. His stance on any given issue is exactly what it needed to be to increase his power. There were no burning ideas for him, just opportunities.

Once LBJ was elected to the senate, he made the first step that any good book on transitions will tell you. He quickly identified the real power brokers in the senate and closely allied himself to them. The most powerful man in the Senate at the time was Richard Russell.  Russell was an ardent racist and segregationist. For normal day to day business at the senate, there was much to commend him. But when it came to civil rights, he and the southern caucus were rock solid in blocking every bill. This was the real era of the filibuster. Because of this, and because LBJ was a southerner, Johnson ardently fought AGAINST civil rights throughout his first years in the senate. Indeed his first speech was a strong defense of southern segregation.

But ultimately there was something real in his rise to power as well. The way he was able to cement his power was not through speeches or even back channel maneuvering, but rather by taking a job that nobody else really wanted and turning into a seat of power simply by being useful. That job was Senate majority leader.

The role of Senate majority leader has no intrinsic power. You can't punish people and you can't reward them. Even today you typically don't see the most powerful senators in the spot of majority leader. But Johnson used the role to became a project manager without peer and made the senate function again. He could count votes better than anyone, knew when any bill was coming up from a sub-committee and knew what every senator needed and what they had to offer. He became the best source of knowledge in the senate and could use that knowledge to schedule the floor efficiently. He could horse-trade votes and control the workflow better than anyone had even attempted to in the past.

The most interesting example of all of this is the 1957 civil rights act. After his 1956 humiliation at the Democratic convention, Johnson realized that he had to actually pass a civil rights bill if he had any chance of being president. The problem for LBJ was that his power base was still the southern caucus and they wouldn't pass anything meaningful. The northern liberals stood on principle and demanded that they not compromise on a substantive bill with far more far reaching implications. Johnson didn't care so much about the contents of the bill, but he needed something to pass.

He squeaked through a toothless bill by using some of the most stunning political maneuvering ever seen in the Senate. What I find most remarkable about this incident is how difficult the Northern liberals were. Here they were, with the opportunity to pass the first civil rights bill since reconstruction and they were fighting for stronger wording that surely would have doomed its passage. Perhaps they were the truly cynical politicians because they apparently cared more about the press back home than they did about getting something accomplished. In the end they may have preferred the role of saying the truth but not being listened to. There is much to commend this role. It gives you the luxury of saying "I told you so" without taking responsibility or assuming the burden of success or greater leadership.

This is the danger of the principled stand. You can publicly decry the state of affairs in your team, watch in glowering anger as nothing changes, and then complain about the lack of promotion or how you were shot down for political reasons. The hard truth is that it's easy to be right. It's vastly harder to actually produce change. The real kicker is that even if you recognize this basic fact and try a couple of techniques to create change, you still may fail. Rather than give up, this is the moment to embrace politics, identify the key opinion leaders and rather than fight them, engage them. Small concrete steps beat flowery speeches (or long blog posts) any day.

Right now some of you might be thinking of some people with more of a knack for politics than software who always seem to get ahead. They seem to be corporate ladder climbers without any particular vision to push or principled stands. But before you dismiss those kinds of people as exactly the kind of employee you don't want to be, remember that among presidents Lyndon Johnson is surpassed only by Abraham Lincoln in what he actually accomplished for civil rights.