We built OneNote in relative secrecy during 2001-2002. Well, it wasn’t cloak and dagger stuff - we just didn't make any effort to talk about it outside of Office - although rumours that Office was building some "TabletPC app" did spread to other parts of the company, a misconception which until recently was probably worse inside the company than outside, since we were able to communicate pretty clearly to the public the message we wanted to get across after we announced. The result of this lack of exposure was that we had become a little insular and no longer had a good perspective on the product as a whole - was it useful? Did it suck?
Although we kept the project secret, there was a select group of total outsiders whom we enlisted to burst our bubbles and give us raw real world feedback. Now, we had done usability tests on individual features, but we had not yet had the whole app up and functioning well enough to do real "dogfooding" - that is, living with the product and trying to do real work with it, rather than just test it or play with it. For that we wanted people in a variety of life situations - students, professionals - anyone who used a computer and tried to take notes or do research.
We recruited around 20 people in the Seattle area to become part of our "Field trial". We narrowed down a pool of about a hundred people to a set of around 30 whom we thought would make good candidates based on their answers to our screening questions. We then approached these people personally to verify if they would be a good fit, and if they were, we invited them to join. Since we had 6 program managers on the team and I had decided about 3 participants per PM was a good ratio given the time investment we were going to make, we wanted 18 participants, with a few as backup in case some had to drop out. We got 20, and prepared the NDAs (non-disclosure agreements, whereby the participants agreed not to blab to others or the press about what they were doing for cheap glory). We had 6 students at the University of Washington, and the rest were spread across quite a spectrum, from a dean at the university, to a health care professional, to a writer, to a belly dancing instructor.
Although OneNote could barely stay up for a few minutes at a time, we already had very good file robustness, so people were not likely to lose any data even if they crashed. Always saving helps in that regard, as does a new file structure designed explicitly to prevent corruption and allow super-fast incremental saves. Although I am a huge XML fan, sometimes binary file formats just ARE better at some things. (note: that doesn’t mean we won't do XML in the future - you just gotta admit that XML isn’t always the best at everything)
We collected a bunch of tablets (mostly beta hardware and prototypes, since there were so few to go around in the fall of 2002 before they had launched) and gave out the machines with an "alpha" build of OneNote on them. We agreed with our participants that we would meet them for 2 hrs a week - one visit was to watch them in action doing whatever they do with the tablet and OneNote, and the other hour was an interview where we asked them to describe their experiences, what they liked and didn’t like, what they had discovered that week, etc. We also committed to be on call to save them if a disaster struck with their notes. We kept a tally of what features were discovered when, and if people deviated too much off the norm, we stepped in and told them about some things so that they wouldn't miss out on the later parts of the trial. The trial ;lasted 8 weeks, getting progressively more advanced. We lost about 5 of the participants over that time – some just didn’t use the product, and a couple found they couldn’t handle the time commitment.
The trial went pretty well, although the instability of the alpha meant people were taking notes in class and had to switch to paper when the app came tumbling down. We also found unsurprisingly that the willingness to adopt new technology more or less correlated with age (can’t teach an old dog new tricks!). And on top of that the students were just animals in terms of adopting OneNote – it was clearly a good fit for them.
We learned a lot from the trial - it helped answer critical questions about our user model, our assumptions about how people would use the product and to what end, what features just didn’t work, what features would get used a lot, and importantly, what expectations people would have before they used the product. This latter was important because as we crafted our message to analysts, the press and the public, we had to anticipate what conclusions they would jump to when we announced, and tune our statements to make sure they did not come away with the wrong impression (like, it is a Tablet-only app!). One thing that the field trial helped us with was to make abundantly clear that the ink model we had was weighted too heavily on structure, as I discuss below. That prompted the ink-o-rama sessions to get ink to a shippable state.
BTW if any of our field trial participants read this and want to drop me a line, that would be great. We’ve lost track of several of you and we wanted to give you a free copy...