On Friday I was taking part of a day long series of talks on engineering excellence.  While listening to a talk on getting user feedback in the engineering process, I had some thought about the work that I’ve done in the past and something that most product teams don’t really know.  A lot of time the help that is created is a result of what the team thinks that the user will need to know or what they believe should be the common task flow across the application/product.  But if you really know how a product is being used by your end-users then you can better target the help messaging to the areas where there are problems and build help topics that support the tasks that they are doing.  (Not to mention build a product that really addresses the user's needs.)

How do you figure out how your product is really being used?

To me this is one of the really fun questions to get an answer to, but also one of the more difficult ones.  There are techniques out there like instrumentation where you can add logs to the product so that at some later point in time you can track how the user used the product.  What did they click on, when did they click on it, what they did next etc.  Doing analysis of instrumentation data can be extremely difficult and if you haven’t specified clear goals of what you want to understand up front, you’ll soon be overwhelmed with the amount of data that comes back.  But another downside to instrumentation is that you don’t know what else is going on?  For example when we were studying how people managed their finances for MS Money, we knew we could instrument Money as much as we wanted, so that every mouse click or keystroke and place would be recorded.  But would users accept that?  Now I would know ever intimate detail about that person’s finances that are being entered, but I wouldn’t necessarily understand the environment.  For example unless I thought about recording the balance of each account at the time the data is being entered, I might not know that the user is trying to deal with the fact that they just bounced a check.  Or if they are using a website to look up some information I wouldn’t know that either unless I could instrument that as well.  Or since there is a competitor, I might want to see how their program compares to ours – and surely I can’t instrument that either!

So there are limitations to instrumentation in this particular usage.  So instead, what if I stood behind the person as they did their finances?  This is a pretty typical way that many people use to understand how people use a product – site/field visits or lab studies or focus groups.  Here I could write a thesis just on this topic alone, but instead, let’s just simplify this to say that the act of observing something can have profound effects on the outcome of the event (Schroedingers’ cat paradox).  I’ll illustrate this with our personal finance example.

In a lab test (e.g. a usability lab),  a participant might come in and you might ask her to show us how they manage their finances by having her bring in her computer (or file) plus any relevant information that they might use in the process.  At least two major issues here – the context in which she’s doing her finances doesn’t match where it’s normally done; and secondly she may have saved up more things to do in the given session than what would normally happen.  For example the participant might have saved all the bills for the last week to bring in, but normally she would pay the bills as they came in.

In a focus group or interview, a participant is only going to describe the process and that is more of an abstracted description of what really happens at any given session when using the software or how any particular task is really done.

In the field it gets to be a little bit richer, but also can have some problems.  For example the user might save up a bunch of stuff for you to view since they know they have an audience which again wouldn’t be what they normally do.  Alternatively you might drive a bunch of people a good distance to view a user who has literally only 3 minutes worth of tasks to do.  Plus the user might not be performing the task in the same way or at the same time that it normally occurs.  Maybe they do the task usually at 3 in the morning in their bathrobe, surely you’re not going to be going out then to do an observation.  Or it maybe that they normally get interrupted by phone calls since they do it right before dinner, but since you’re there they have someone else screening the calls.  For the most part you’re getting richer data in this situation, but it really requires thinking beyond the easy staple of techniques.

So thinking more out of the box on this, what you really want to do is to have a video camera pointed in the direction of the workspace where finances are done and the camera rolling all the time while the participant is doing tasks related to her finances.  I’m not suggesting that this is actually what you do, but it’s in the ball park of the kind of information that you really want to collect.

Of course how much you actually observe or are able to eavesdrop on the user is going to depend on the complexity of the product and the level of user interaction required at any individual occasion or on an on-going basis.  Studying something as rich in interaction as the Tablet PC or even a regular notebook takes a lot of work…  Eventually I’ll get up to writing my thought on that topic.

But for now I’ll leave you with the question of do you really know how users use your product or do you just think you they use it like you designed it?  Sad reality is that most products never get the level of attention that I describe above, but I’d love to hear examples of products that do!