I haven't talked about methodology in a long time (at least not on here). I was recently in a conversation with someone about doing some research using a remote screen capture tool on a PC. The person (who shall remain nameless -- but you know who you are!) basically recruited participants to try out a build of a new concept for a particular software program and was using a tool that allows all of the interaction on the computer to be encoded to a video file that can be sent back over the internet. This way the researcher could watch a video of how the user interacted with the software while sitting back at her office.
I happened upon her in her office when she was watching the video. During the video I noticed that there were gaps in the video, in that there was a pause in what the user was doing while in the midst of the task trying to be accomplished. We talked a little about what we had seen in the video, and I brought up this observation to her. She didn't think that it was all that relevant -- and I disagreed and suggested that unless she knew why the user wasn't finishing the activity in a continuous fashion, then she really didn’t really understand how the user was doing the task.
- Was the user trying to get the task done and the kids were fighting in the background?
- Was the user puzzled by what he were trying to do and just sat there trying to figure out what to click next?
- Was the user asking someone for help?
- Did the user forget what they were doing when he was returning to the task and thus sitting motionless for a while before starting up again?
- Did the user really make errors as a result of not remembering where they were at in the task or were the errors the result of something else?
At this point she wasn’t very happy. A lot of planning had gone into setting this study up and getting the software properly instrumented and set up for the participants. At the time she was planning the study, it hadn’t dawned on her that the context in which the user is performing the tasks is as important as the task at hand. (Design solutions to address where to click next are much different than a solution focused on helping the user remember what they just did if they’re being distracted while performing the task.)
Technology is great, it can really help to better understand a situation, but if you’re not getting the full context then you’re missing a part of the bigger picture. A lot of products are built in a vacuum, not because there’s missing data about the user, but rather they don’t take into account the larger ecosystem. If you can capture the user in that context, when they’re doing a task you’re interested in, then you’ll have a much richer set of data about how to design a product.