Designing software is fun fun fun. During the OneNote project we had countless hours of brainstorming design sessions full of laughs and great ideas. There's really nothing quite as intellectually stimulating as a tough intellectual problem, 3-5 smart, articulate people, and a whiteboard. Ideas come and go. Analogies drawn. Metaphors extended. One of my goals in such meetings to is to inject hilarity where possible, and some of the deepest laughs I have ever had have come from those sessions. Surfing the "froth" at the tip of the idea wavefront in sessions like these is incomparable.

Then, the test comes. The idea we have laughed and cried over and finally reached an agreement on is prototyped, and we put it in front of "normal people". These heathens then have the gall to not understand how to use the feature we have designed, or not discover it among all the other stuff, or otherwise demonstrate through their confounded normality that our ivory-tower design sessions have produced a masterpiece of hokum. The program manager "owner" of the design drops his or her tail, and sulks through the remaining sessions (usually 6-8 in total) glumly verifying that yes, seven normal people out of eight could not make heads or tails of their masterwork.

By the next morning, the feedback has been internalized, the design ideas flow again, and we try once more to come up with the design which somehow meets all the criteria, fits within all the constraints, actually solves the problem at hand, and this time will in fact be usable by the dreaded "user".

Iterating in this way, we can work out most of the fundamental errors in a product's design. But usability tests are not a panacea. In fact they can be quite dangerous. Although they ostensibly test normal people, these kind souls are people who come to the hallowed halls of Microsoft in Redmond. They are very aware that they are going to be using something new, and they should be on the lookout for changes - plus they really want to be helpful, so they take extra special care to examine what’s in front of them. This is all quite unlike true normal people, who don’t have time to notice software and just need to get their work done.

Another bias that usability tests introduce is the "optimization for first use" phenomenon. You've probably seen this. There's a feature you use which seems innocuous the first time you use it, but as you use it more and more, it bugs you that it takes so many steps to get it done. Or that it seems so verbose, with a lot of user interface for something which is really quite simple. What has happened is that the usability test showed that people who have never seen a feature before have trouble with it in the first hour of using it. So the designer makes the feature hold your hand through the process. That improves the results in the test, but ruins the feature for people who know what they are doing.

Remember Microsoft Bob? That was an extreme. People who didn't know anything about computers could use Bob to write a letter. But Bob was so optimized for the first time user that even those utter novices would quickly tire of its helpfulness and want to move on to something that got out of their way and just let them do their thing.