We have a number of ongoing, long-term projects designed to help us test the overall usability and learning curve of the new Office 12 user interface.

One of the most important tasks has been a full deployment of Office 12 Beta 1 within a local company.  We chose a dozen people with all different job types (but none of them computer or software related) and replaced their Office 2003 with Office 12 Beta 1.

On the surface, this doesn't sound that novel.  Obviously, if you want to find out how well a product works, you need to get it in front of people.  What is unusual is how early we started this study.

If you've been in an Office beta program, you know that Beta 1 is typically extremely rough.  It's designed to give people a glimpse of where the technology is headed but not really suitable for daily work unless you're willing to put up with a lot of bugs.  We typically do not get the quality of Beta 1 far enough along so that we would feel comfortable with "normal" people using it.  It's designed for our hard-core MVPs, partners, and technology enthusiasts.

That said, we knew that if a long-term, longitudinal study with real people was going to impact the product, it had to happen early enough for the feedback to be acted in the product.  By the time Beta 2 is out, we're mostly focusing on quality, performance, and fit-and-finish issues and we don't typically revisit the design of a major component of the product.

Thus, in early November, an unsuspecting group of volunteers at a local company got their rock stable, menus-based Office 2003 replaced with Office 12 Beta 1.

We've learned at enormous amount from this study, and I'll be sharing many of the results with you both anecdotally and statistically in the months ahead.  The overall feedback has been very positive and, in the places there are have been problems, the feedback has been totally actionable.  We have made many improvements to the product to address feedback garnered from all sources about Beta 1.

The single most interesting takeaway for me, personally, has been seeing the incredibly strong link between the quality and the usability of the product.  Normal people simply can't tell the difference between something that's broken and something that was designed poorly.  Most of the "usability" feedback we get is actually bug reporting.

When the quality sucks, the usability sucks.  And it's not just the obvious things like crashes and data loss; the fit-and-finish ultimately makes or breaks the usability.  When a menu disappears too quickly, or the selection handles aren't intuitive, or when a feature doesn't work in a certain scenario--all of these may be accounted for as bugs which will be fixed in a later version, but these things will add up and taint your usability numbers.

One of the hardest parts of synthesizing all of the usability feedback we get is knowing when to write off things as "probably fixed in a later build" without neglecting real issues.