I’ve hinted around at what we called the Field Trials in Tablet before and I know that Chris Pratley talked about Field Trials in his work with One Note, but unless you happen to read the HCI (Human Computer Interaction) literature, you might not know what it was that the tablet team did and why it was so important and novel inside of Microsoft.
So what is a Field Trial?Basically it’s an opportunity to get real world feedback on your product by getting it to a state that a small group of users can use it for a prolonged period of time. In the case of the tablet (when I was still working on it), we ran a 2 week trial, a 5 week trial, a 8 week trial and a 9 month trial.
Before I go too much further… How is a Field Trial different than a Beta?Well at Microsoft a “Beta” is more about finding bugs in the process as well as getting customer feedback, it’s also usually at a point just before making a release candidate which means that if there are major changes needed in the overall user model, they generally don’t get made or they wait for the next version. Also betas are generally huge with hundreds or hundreds of thousands of people participating in the process; whereas a field trial is relatively small since you’re going to spend a lot of time with each of the individuals participating.
Okay tell me more… Some backgroundWhen I first joined the Tablet team, I pitched to the team (all 5 of us) that in order for us to really know if we’re building the right product that we have to get it out into the users hands early enough in the process for us to understand how they want to use it (the plan for the 5 week trial). The way I proposed to do this was to give some users prototype tablets and follow them around so that we can understand how it’s being used, what things are still needed and where usability issues exist. At this same time, Chuck Thacker was busy sketching the schematics for prototypes and I was beginning to make plans for how to take advantage of the fact that we would be building full functional PCs with integrated digitizers without a keyboard and the potential for several buttons. Surprisingly Chuck pushed through the engineering relatively quickly including utilizing a super-secret start-ups new processor (Transmeta) who hadn’t officially announced anything at the time. However actually having these built and debugged took a lot longer than expected but that was a good thing since the logistics of doing a field trial are immense.
TabletsChuck had a run of 35 initial tablets built for the team. At that time there were probably around 50 people who were involved with the tablet, but of those 35 units, 30 of them were dedicated for the field trial which meant that the developers who were busy writing prototype code and putting together features and other influentials around the company had to fight over the 5 other tablets that were in circulation. In retrospect it was funny since many of the team members who eventually came out to the field trials had not had any direct experience with one of the working tablets. Of course these devices were extremely fragile as well as the software that we developed and as such we wanted to have spares on hand in case anything failed.
Given delays in getting the hardware we performed an initial trial about 5 months before the first main one with some very limited hardware from a company that we acquired. Wish I could talk more about the company and the hardware but suffice it to say it wasn’t a full-pc but rather a CE style tablet. The main purpose at this time was to understand if users really needed Windows and also for us to get some experience at piloting the methodology and refining it so that when we were ready to roll out the more extensive trial we’d be ready. This one lasted 2 weeks with mixed reactions, some users really like the functionality but wanted XYZ (name your favorite Windows application) to be included while some of the other users had a really hard time just even getting started with the device. We learned quite a bit about the methodology, infrastructure needs, support and how to get buy-in from participants and the corporations they work for.
CorporationsWe ran the field trials in a large mid-western city during the winter months of 2001. We wanted to have at least 5 participants from 4 different large companies or corporations involved with the study. The main reason was to have a wide variety of “horizontal” knowledge workers who performed various different tasks in different ways at these different companies. This way we could have a small sample of users who fit this idea of a “meeting go-er” who was central to the Tablet software. So for many months I fly out to this city and had meetings with the IT staff at these companies since we knew that to get into the site we were going to need cooperation from the IT department as well as senior management. Once we got the IT folks interested, then it was on to selling it up the chain and getting them all interested. Unfortunately we had lots of false starts as we’d show the concept to the IT folks using one of the very early prototypes (there was only 2 or 3 of them available at that time) and they’d be really interested and the CEO or CIO thought it would be a distraction. Some times we had the CEO wanting to participate and be one of the people we’d follow around, but the IT folks would simply just dig in their heals and not let us put the equipment on their network. So we went around and around and finally ended up at 3 different companies from which we recruited 21 people.
UsersOnce we had the companies, then we went on to start screening users. As you might imagine the IT staffs volunteered themselves as the first candidates and we had to work pretty hard to either get to the fringe of their organizations or get to some of the users that they supported whom they felt would be amenable. This was a pretty elaborate process to as we wanted to interview the people and “shadow” them (follow them around for part of their day) to make sure that these people were relatively “normal” and fit our profile. Almost all of the people we choose were not early adopters of technology nor were these people gadget freaks – we tried to screen them out so as to get those people whom in 2 or 3 years their company would deploy tablets to them as standard gear since it was a tool that would make them more productive, etc. We ended up with a great selection of people ranging from a low level analyst to a couple of corporate executives.
Over the course of the next 5 weeks we interviewed the users, followed them around for parts of the day, send them questionnaires to fill out and had them perform some exercises to assess how much they’ve learned or used of the device. This was a lot of fun as we had multiple teams out in the field and every day we would spend between 8 and 12 hours following around 2-3 different users. At night the teams would get together and we’d all discuss what we had seen, go out for a late dinner and then crash to start it again the next day. Some of the users called us their “entourage” or “posse” as we often had 2 or 3 people following them around as they went about their activity with video and digital cameras. It was quite an experience as almost every meeting was interrupted by at least 10 minutes to either explain who we (the observers) were and/or what the cool device they had was and what it could do. Even when were not there, we often had the user relate about how many demos they had to give during the day and how it was interfering with the “work” that they had to get done. To us they were the “celebs” and we were the paparazzi taking notes and photos of absolutely everything they were doing.
This process went on for 5 weeks which I spent living in a hotel and flying in different team members to observer Mon-Wed and Wed-Fri. I’m still upset about the Business Week article that got it wrong… it said I missed my daughter’s 7th birthday when in fact I missed her 2nd birthday, but we did have a party on the weekend. I flew back to Seattle on the weekends or down to San Antonio where my wife and daughter were spending the time with her parents while I was clearly occupied. This was to me one of the most exciting projects I’ve ever done.
InfrastructurePerforming a study like this is no simple matter, we had to insure the IT department that the devices would be compatible on their network, not cause any issues, that their main applications would work, that we’d provide unparalleled support for the users if anything went wrong, etc. In doing this we ended up hiring our own tech support guy for the area who had the job of making sure that the user’s machines were up and running at all times. He was on call 24/7 and we set the goal of less than 2 hours for the user to up and running. Thus having extra units helped since he was able to remove the hard disk and place in the other units; as well as having extra hard drives with the company’s image, providing regular backup service to all the users so that the data could be restored quickly on a backup etc. And his job was pretty rough since I’d be out in the field every day and I think I ended up running into him almost every day while he was fixing someone’s machine. Like I said these were prototype units with prototype software.
And then there was the product team with everyone triaging bugs and making sure that we had the right functionality for the users actually in the build and working on the prototypes. The entire tablet team deserves a lot of credit for pulling this all together, but the testers on the team a great deal of gratitude as they were the ones that were constantly banging on the product and then working with the developers late into the night to get fixes not only before we started but all throughout the trials when we’d find things breaking left and right.
In addition we hired an outside firm to help manage some of the logistics and do some of the reporting of the user data. This way in addition to my skills and those of some of my team, we were able to have multiple sets of eyes out there that are trained in psychology, user research, usability, etc. Thus we could really follow around multiple participants and have a large sample set than if it was just me.
Stay tuned for part 2….