stevencl's WebLog : Usability

HOWTO: Run an API usability study

stevencl — Thu, 05 May 2005 20:31:00 GMT

With the task list in place and participants recruited, it's time to run the study. My experience has been that running an API usability study is really no different from running any other type of study. Here's a description of what I do.

The day before the study I make sure that the machines I'll be using in the usability lab are set up. I install the necessary builds of Visual Studio and associated SDKs etc to be able to run the study. I always do this first thing in the morning just in case anything goes wrong as setting the machines up properly can be quite time consuming. Once that is done, I then go about setting up the screen capture software so that I have video recordings of each participant's session in the lab.

While our usability labs are well equipped with audio and video recording equipment, I prefer to use HyperCam for screen recording since the quality is decent and the format (AVI) makes it easy for me to create video clips to show highlights from each session. Running HyperCam and VisualStudio on the same machine can sometimes cause the machine to slow down pretty significantly though, so I run HyperCam on one machine and then use remote desktop from that machine to connect to the machine running Visual Studio. Setting up HyperCam, including setting up the PC microphone so that the recording level is just right usually takes no more than 10 or 15 minutes.

Our usability labs are split between the participant side and the observer side. After setting up the participant side of the room, I set the machine up on the observer side. We have in-house tools that we use to record our observations during each session. The tools allow us to create a simple tagging scheme that we use to tag our observations as we take them so that afterwards we can easily refer to all events that involved use of the ServiceContract attribute for example, or events that were particular highlights for whatever reason. I take some time to think about how I might want to analyze the observations after all the sessions and set the tools up to use the particular tagging 'schema' that I come up with. Sometimes the schema is as simple as three tags labeled 'Highlight', 'Success' and 'Failure' but other times it will be more complex. It all depends on the nature of the study.

Lastly, I print out the task list and a copy of the tutorial materials that were sent to participants, just in case participants forget to bring their own copy in. I always leave printing the task list to last as invariably I'll make small changes to the task list right up the last minute.

After all this setup, all that is left is to run each participant's session.

The first time each participant comes into the lab I spend some time getting them familiar with the lab. It can be quite disconcerting to be sitting in a lab knowing that people are watching you behind a one way mirror so I reassure participants that the purpose of the study is to learn how usable the API is, not to test each participant that comes into the lab. I also give some participants time to practice talking out loud while they work. We always ask participants to provide us with a running commentary of what they are doing as they are working as that is the best way to get at the intent behind the actions that they take. However it can feel a little odd to talk so much while working since it's not something that most people are used to doing. So for participants who have not been in the lab before, I will sometimes ask them to do some task (such as using Paint to draw a picture of a mouse) and ask them to talk out loud while they are working. Many participants find that after just a few minutes of practice that it's actually not that difficult to keep talking while working.

Then participants start working on the tasks. While they work on each task, I try to take as many notes about what they are doing as I can. I typically try to record a transcript of everything they say as they are working. When something interesting happens, I'll make sure to annotate my notes accordingly so that I can easily refer back to that event (our note taking software timestamps each note that I take so that I can associate a note with a point in time on the video recording of the session). Paying attention to what is going on and taking good notes can be very tiring but it's critical, especially if you don't want to be spending hours reviewing video tapes and other logs afterwards. Time and effort spent here pays off in the long run.

Probably the most important thing to be aware of when running a usability session is the impact that anything you say to the participant has on their behavior. The most obvious way in which you influence the behavior of the participant is in how you describe the task you ask them to perform. If you mention the specific classes, namespaces, methods etc that you want them to use, then you've led them down a path that they might not necessarily have taken had you not mentioned it. But during the study there are other less obvious ways that you can inadvertently influence participants. Very often, participants will get stuck with a task and might get a little frustrated. They'll ask for your help and the natural inclination is to offer some help, particularly if they are very frustrated. You can simply tell them how to get unstuck, but that doesn't really provide you with useful data about why they got stuck and how they expected to be able to fix the problem themselves. All you know is that when told to do the right thing, they were able to do it. Instead, you should see this as an opportunity to understand what it is about the design of the API that leads participants to get stuck.

These sort of situations are where the benefits of running a usability study really become apparent. As soon as a participant gets stuck on some task, they are in the best position to describe how and why they are stuck because they are literally going through the experience right there and then in front of you. They don't have to rely on their memory to tell you what happened, as they would if you simply interviewed somebody about their use of an API after they were done using it. They are motivated to explain things to you because that will help them recover the situation, unlike if you were asking them to describe the situation after the event.

Whenever participants ask me questions when they are stuck, I'll very often not tell them anything and will instead ask them to describe what they think is happening and how they expect the API to work. With that information I'm able to start forming hypotheses about the conceptual model that participants are forming while working with the API and how that leads to breakdowns in using the API. I can then start testing these hypotheses by asking participants to provide feedback about the way that these conceptual models are formed. I don't ask them to comment explicitly on their conceptual model but instead I'll ask them about salient aspects of the API that I think might lead to a particular model being formed. These are the points in a usability study during which I learn the most from participants.

Outside of these situations I try to say very little during the session. For example, to begin with, lots of participants ask me when they should move on to the next task. I try never to answer these questions, telling participants that they should move on to the next task when they think they are done with a task. I don't want participants to get used to my confirmation that they have done the right thing. If I keep telling participants that they are doing the right thing, participants tend to do something then wait to hear my confirmation before moving on. Instead of using their own intuition to figure out if they have done the right thing, they simply wait for an acknowledgement from me. At that point the data I'm getting is questionable.

The key criteria for success in running a usability study are: good observation and note taking skills, and the ability to understand observed behavior from the participant's perspective, instead of your own. You'll notice that running a successful study does not depend on having expensive usability labs. They're definitely nice to have, but not necessary.

Ideas for talks at upcoming conferences?

stevencl — Fri, 21 Jan 2005 00:13:00 GMT

I was just in a meeting with the Visual Studio UX (user experience) team. Someone brought up the topic of upcoming developer conferences and whether or not folks from the UX team would be attending. Then someone suggested that we could even consider doing a talk (or talks), as well as attend the conferences.

Nobody from the UX team has presented at any of the recent (last five years) developer conferences so we have no idea whether or not this would be worthwhile and of interest to other folks. We'd love to know what you think - are there topics that you'd like the UX team to present on at any developer conferences in the near future? If so, what are they? For example, would a talk on API usability be of interest?

Whyline

stevencl — Wed, 04 Aug 2004 22:21:00 GMT

Brad Myers from CMU was here today giving a talk which included some details of the Whyline system, a debugger that allows programmers to ask questions about the behavior of their application in order to be able to figure out what the problem is (there's some discussion of the system on Slashdot).

There were many interesting points made during the talk. In the lab and field studies that Brad Meyer's group had run on how programmers debug, they found that when starting out debugging a particular issue, developers start by asking questions about the application behavior they are observing, as opposed to asking questions about the code that is being executed (or not as the case may be). So, they'll ask questions such as “Why isn't the error message being appended to the application log?” instead of “Why doesn't applicationLog.Write(someError) write the error message to the log?”

The reasons for this were that developers aren't completely confident about the particular line or lines of code that are responsible for the issue when they start investigating the issue. The implications of this in terms of debugger functions and features are pretty interesting because it indicates that one piece of the puzzle that developers currently have very little tool assistance for is in correlating program behavior to specific lines of code. Breakpoints, data inspectors etc, are all useful tools, but in order to be able to use these effectively, the developer needs to know what lines of code to place a breakpoint on, or what data to watch.

One other interesting point made during the talk was that “print statements are still the most popular style of debugging”. Perhaps this is because they are one of the only available features that help developers trace execution of their application so that they know what lines of code or what data they should focus their attention on.

VS automation samples

stevencl — Wed, 14 Jul 2004 18:08:00 GMT

I've been asked for suggestions for a list of VS automation samples that would help solve tricky tasks that I've observed users working on during usability studies. Just wanted to check if anybody has such a list of samples that they would like us to provide.

One example from a recent study that I can think of would be a sample to show how to automate migrating or copying a bunch of project settings from one project to a collection of other projects.

Any other suggestions?

This scares me!

stevencl — Tue, 06 Jul 2004 17:00:00 GMT

Aleksei Guzev was scared by my last post on readability vs writability.

I think what might have scared Aleksei (feel free to correct me if I am wrong Aleksei) is the thought that the results of this study would be used to make a case for adding a Count property to the DataReader class and removing the ability to iterate through the contents of the class with IEnumerator. Far from it - no such case was made as a result of this study and none will be. What the study shows though is that for some developers, using IEnumerator will not be as intuitive as it will be for others. Therefore, it is worth considering how to make this more obvious to those developers. For example, Intellisense in Visual Studio could highlight particular programming patterns or tasks that a class supports (it might show a list of tasks in addition to a list of methods such as 'Iterate through the contents of myDataReader') such that when the user selects one of these tasks, the code to accomplish this is generated and inserted into the editor. In this case, Intellisense might generate code that creates an instance of an enumerator, sets it to the first element, creates the loop to MoveNext through the enumerator and calls Current to get each element. The user would then just have to add code to do whatever they wanted with each element.

You can think of other examples - for instance, classes that implement IDisposable might have a task called 'Use this resource' (or some better wording...) that would automatically insert a using statement if the language supported it, or would just insert the appropriate Dispose call.

Other alternatives to addressing the usability feedback would be to make supporting these tasks much clearer in the documentation. The key point that I am trying to make though, is just because participants find some tasks in an API usability study difficult, doesn't always mean that the only solution available to you is to change the API. In many cases it is, but there are times when changing the API to address usability feedback won't always be the best thing to do.

Psychology of Programming Workshop

stevencl — Tue, 27 Apr 2004 21:24:00 GMT

Another paper that caught my attention at the recent Psychology of Programming workshop was presented by Matt Jadud from the University of Kent. Matt's research focuses on understanding how novices learn to program and to develop teaching methodologies that take this understanding into account. In his paper, Matt described how he had analysed data collected from an instrumented version of BlueJ, a popular IDE used to teach Java to Computing Science students.

Matt presented an excellent analysis of the behavior of students when the IDE reports a compiler error and when students have a syntactically correct program. Matt’s analysis indicated that students prefer to write large amounts of code, and then repeatedly compile their code to find all of the syntax errors in the code. The data indicated that 51% of compilation events occurred less than 30 seconds after the previous event and that the amount of code changed between these events was minimal (along the lines of adding or removing a semi-colon).

There was an interesting discussion at the presentation about whether or not the students coding strategy was a good strategy and what things the tool could do to shape students behavior to improve their strategies. The discussion centred around the extent to which guidance in a teaching tool simply conditions students to behave in one way or another. In Visual Studio .Net we do a lot of things to aid developers such as highlighting syntax errors, indicating matching parentheses etc etc. While Visual Studio .Net is not primarily designed to be a teaching tool, are there still some disadvantages to these features even for developers who are not using the tool to learn how to program but to get a job done? Are we at risk of conditioning developers not to pay attention to these details?

Usability and prototypes

stevencl — Fri, 27 Feb 2004 23:10:00 GMT

It's been a while since I last posted and the main reason for my silence has been that I've been busy preparing and running a usability study this week which involved the creation of an HTML prototype to mock up the UI of the product we were studying.

The study is investigating the user experience for a portion of the Whidbey product that has yet to be fully implemented. In order to get good feedback about the design from participants we needed to create a prototype of the product. I suggested that we create HTML prototypes with separate web pages for each step that participants needed to complete in the usability study. Each page would contain a screenshot of VS in a particular state (e.g., the solution explorer showing the current project selected) and would have a set of hotspots within which the user could click to move on to the next page in the sequence. With correctly defined pages and hotspots we could simulate different actions that the user could take with VS. For example, to simulate a right click on a project in the solution explorer, we would have one page containing an image of the Visual Studio IDE with the solution explorer open and a hotspot over the appropriate project in the solution explorer. When the user clicks in the hotspot, we show the next page containing an image of the VS IDE with the solution explorer open as well as the context menu for the project that the user clicked on.

It seemed like a good idea but I underestimated how much work it would involve. For example, to accomodate the different ways that a developer could create a new project in Visual Studio required that we create almost 50 separate web pages with links and hotspots relating each of the pages together as appropriate. To create prototypes for each of the four tasks in the study eventually required two of us to spend three full days creating images etc.

The effort was worthwhile though. We're finishing up the study today and we've been able to use the prototypes to get some pretty useful feedback. It's reasonably early on in the lifecycle of this feature that we should be in a good position to respond to most of the feedback we've collected. You can tell the prototype has worked when you have to keep reminding participants that they are working with a prototype and that what they are looking at is just a bitmap and so they won't be able to scroll through the code they are examining, let alone change the code.

An interesting side effect that I think arises from the fact that we were using a prototype is that participants were more likely to talk about the interaction flow as opposed to the lower level details of the UI such as button labels, colors, positions etc. I think the nature of the prototype made it more obvious to participants that we were more interested in how the different parts of the product flowed together in the different tasks. So the effort involved in creating the prototype, although large, was definitely worthwhile. Something to keep in mind if you're thinking about using prototypes in one of your own usability studies.

OK/Cancel

stevencl — Fri, 30 Jan 2004 17:20:00 GMT

I just got forwarded this link to a pretty amusing and interesting usability site: http://www.ok-cancel.com/index.html

Sign up for a usability study

stevencl — Fri, 14 Nov 2003 19:49:00 GMT

If you'd like to get the chance to participate in an API usability study and give us your feedback about Avalon, Indigo, WinFS or other APIs, sign up as a usability participant at http://www.microsoft.com/usability/.

We're running studies fairly regularly (for example, next week we are running studies on the Avalon Sidebar API and on the new C++ on .Net syntax). For each study, participants normally come into the labs on the Microsoft campus for anywhere between 2 and 4 hours and work on a number of tasks using a specific API. We get great feedback from people while watching them work. As a token of our appreciation, participants get to choose one or two pieces of software (depending on how long their session is) from a list of software that includes Visual Studio .Net if you come in for a developer study.

We tend to do most of our studies here on the main campus in Redmond, WA, but I'd encourage people to sign up even if you don't live anywhere near here. We always try to set up usability studies wherever and whenever a major conference takes place. For example, there were a number of usability studies that took place in LA during the PDC. So, if you're interested, please sign up and help us improve the quality of our products.

Experts and non experts

stevencl — Mon, 03 Nov 2003 23:42:00 GMT

I had an interesting conversation with a colleague of mine last week. We were discussing the extent to which full details of a technology or feature should be exposed to end-users. We were discussing this issue in the context of the IDE specifically, and how details should be exposed in the UI, but the issue is also relevant when considering APIs.

For example, consider the System.Xml namespace. This namespace exposes a set of types that provide standards-based support for processing XML. For example, the XmlDocument class implements the W3C Document Object Model (DOM) Level 1 Core and the Core DOM Level 2.

XmlDocument exposes members such as CreateDocumentFragment, CreateElement and CreateNode. If you're familiar with the details of the W3C DOM you'll know which method to choose when you want to add a new data record to an Xml document that you're using as a database. But if you're not as familiar with the W3C DOM, you may not know which method to use to accomplish your task. You'd need to become familiar with the DOM before fully understanding what the different methods do and when you should use which one. Likewise, if you wanted to run a query over your data, you'd need to know about XPath and that you create an XPath expression to select the nodes in your document that meet the search criteria.

This is all fine if you are comfortable and familiar with all things XML. But if you're not, it may be somewhat daunting - there is a lot to learn. The rewards are potentially pretty large, but the tax you have to pay to reap those rewards may also be pretty large.

The question is, does the developer always have to be forced to pay this much tax or is there a way that the API can be designed such that the developer who does not have as much familiarity with all things Xml can still reap the benefits?

It's basically a trade-off between making experts comfortable and providing them with an API that maps very well on to their expert view of the domain at the expense of requiring non domain experts to become domain experts or to forever remain uncomfortable using the API. Or, making non-domain experts comfortable with an API by abstracting over some of the domain specific aspects of the API but by so doing making experts less comfortable since everything that they know about the domain will not be represented in the API as they expect it to be.

My colleague and I concluded that it really comes down to the percentage of domain experts and non-domain experts in the user population. We figured that if 10% of users were experts and 90% were non experts, it would be unreasonable to force the non-experts to become experts before they could use the API effectively. But we did not come up with a point at which we felt it would be reasonable to make users learn a bit more before they could use an API. Is it 50/50, or higher or lower? One argument suggests that over time people will become experts in the domain the more that they use the API. But another argument suggests that if the cost of being able to use an API is too high, people won't even bother with it, or may use the API incorrectly, leading to inefficient code etc.

Given these arguments, at what point do you think it is reasonable to have to invest significant time to learn about the domain that an API abstracts over? We'd love to hear your opinion on this.

Usability at the PDC

stevencl — Sat, 25 Oct 2003 00:32:00 GMT

Graeme Mott, a colleague of mine in the Visual Studio Usability group will be at the PDC next week. He's planning a few interesting activities:

He'll be walking around the conference with a video camera giving attendees thirty seconds of 'air-time' to provide their feedback to the Visual Studio product team.
He'll also be running usability studies on various features of the product, including some of the new features around data access etc.

Graeme will be hanging around the Microsoft booth wearing a 'Usability' t-shirt and he'd love for you to introduce yourself to him and let him know what you think.

I wanted to make sure that you knew about this ahead of time so that you can take some time to think about the feedback you'd like to give Graeme and the rest of the Visual Studio team.

Also, feel free to make fun of his t-shirt...

Aesthetics, language design and usability

stevencl — Wed, 15 Oct 2003 20:09:00 GMT

Here's an interesting interview with Anders Hejlsberg in which he makes some points about programming language usability.

Anders says that "good language design boils down to assembling a team of people who have good taste." I couldn't agree more. It's absolutely vital to understand your target customers so that you know what they will like and what they won't like. Developing that sense of good taste takes time and experience. You can build that experience up by designing languages or APIs that you yourself would like to use, if you consider yourself a good representative for your target users. But what do you do if you are designing a language or API for a group of users who you don't represent, such as children? This is where usability data is useful.

One of the biggest benefits of doing any kind of usability study is not the list of usability issues or bugs that you collect. Instead, it's the increase in customer empathy and understanding that comes of watching someone work and observing their work style and behavior. It's illuminating to see how differently someone thinks or works to the way that you would in that same situation. Gaining such an insight is incredibly useful when you are designing any kind of product, be it a programming language, an API or a GUI, especially when you are not representative of your users. Instead of basing design decisions on what you would do, you can now base design decisions on what your users would do.

Anders suggests that you don't get as high a yield from usability studies for language features as you do for IDE features. I think this is a reasonable statement to make when you compare the results obtained from a 2 hour usability study on a programming language or API with the results obtained from a 2 hour usability study on a GUI (with some exceptions - see below). Most APIs and programming languages take more than 2 hours to really get to grips with. Instead of just relying on one single two hour session per participant, API and programming language studies tend to demand more longitudinal studies where you can observe developers working with the API or programming language over a longer period of time, measured at least in days or weeks instead of hours (the exception is an API or language that supports tasks that you expect users to be able to complete in much shorter periods of time).

I'd argue that APIs and programming languages are interactive products, just as GUIs are. They expose affordances which indicate to the user the actions that the artifact makes possible. The big difference between GUIs and APIs or programming languages is how those affordances are discovered and learned, not whether or not they exist. That interaction between the user and the perceived affordances of the product is an important aspect that needs to be understood and designed. Usability studies are a very useful tool to help build up an understanding of how that two way interaction between a programmer and the API or programming language develops over time. With a programming language or API the interaction typically takes place over a longer period of time than with a GUI, hence the need for studies that take place over a longer period of time than GUI based studies.

As always, data obtained from usability studies should not be the only source of data that you collect about your design. It's always important to get a broad spectrum of data. But usability data should be part of that spectrum.

API usability and the cognitive dimensions framework

stevencl — Thu, 09 Oct 2003 03:35:00 GMT

Brad Abrams posted a slide deck that I use to talk about the cognitive dimensions framework which we use at Microsoft to evaluate and describe API usability. I thought I would take this space to provide a little more explanation about how we use the framework.

The cognitive dimensions framework presents a set of dimensions that describe aspects of an API that impact its usability. For example, one of the dimensions is 'Abstraction Level' which describes the types of abstractions that the API exposes. Another dimension is 'Working Framework' which describes the mental load that the API places on its users. An API may be found to be unusable because the abstractions it exposes are too low level. Or it might place too much of a mental load on its users, requiring them to keep track of multiple objects in their head as they write code to perform a task. Or it could suffer from a combination of both problems, exposing multiple primitives that the developer has to keep track of in their head as they write code to accomplish a given task.

By evaluating an API with respect to each of these dimensions, you can form a picture of what the API looks like overall with respect to the framework. We like to create pretty graphs to do this. For example, here is a graph of an evaluation of some fictional API:

The black dotted line in the graph represents the API being evaluated. The spokes in the 'wheel' represent each of the different dimensions. End points (the middle and outer edges) on each spoke represent the end points on the scale for each dimension. Thus the point on each spoke where the black dotted line crosses it corresponds to the point on the scale for that dimension that the API has been valued at.

The most common way that we use to perform such an evaluation is to perform a usability study in our usability labs here at Microsoft. We invite developers from outside of Microsoft to come in to the labs and use an API on a given set of tasks, over a number of hours. We observe each developer as they work from behind a one way mirror and make notes about what parts of the API work well and what parts don't work well. We usually bring at least five developers in for at least two hours each for every study we do. So after ten hours worth of watching developers write code against the API, we spend time reviewing and analysing the data we collected and producing our evaluation of the API with respect to the cognitive dimensions framework.

However, there really isn't much point in going to all this bother if all we end up with is a graph like that shown above. It's not very helpful to provide the team developing the API with a graph showing how their API fits on the cognitive dimensions - there is nothing on that graph to tell them whether or not the API was judged usable or not. There is no baseline with which to make a comparison. That is where the individual developer personas come in to play.

We have defined what the three developer personas (Opportunistic, Pragmatic and Systematic) would judge to be the ideal API with respect to the cognitive dimensions framework. In other words, we have created profiles for each of the personas that describe for each dimension, the ideal point on the scale for that persona. The personas define quite different working styles so the profiles have ended up being quite unique. Thus we now have a way to judge whether or not an API is usable by comparing its evaluation with the profile we produced for the developer persona that best matches the target audience for that API. We can do this comparison in the same graph:

The blue line on the graph represents the profile of a particular developer persona. For example, it shows that this persona prefers APIs that expose aggregates (the outer edge on the abstraction level scale) and APIs that have small working frameworks (the inner edge, or middle, of the working framework scale). If you compare the points where the blue lines cross each spoke with the point where the black lime crosses each spoke, you get an idea for how well the API matches the persona's ideal API. Hopefully you can see that this particular API doesn't really hit the mark for this persona (there are big differences between the points where both lines cross on most, if not all, of the spokes in the graph).

Fundamentally the cognitive dimensions framework provides developers with a common vocabulary with which to talk about and discuss API usability. The benefits of such a common vocabulary are twofold. First, the language shapes the distinctions that are considered important to attend to. By explicitly describing each of the dimensions, it is more likely that they will be attended to. Secondly, the language allows developers to discuss issues using terminology that they can assume will be understood by others.

We've found that the cognitive dimensions have really helped us get to the root cause of issues that we have observed in the usability labs. For example, in one study, we observed lots of developers spending a large amount of time in the help docs looking for code samples that would show them how to accomplish a given task. The first interpretation of this data was simply 'Fix the help docs!'. However, when we used the cognitive dimensions framework to describe the issues, it became clear that the reason the developers weren't successful when they were searching through the help was because what they were looking for in the help simply didn't exist. The API they were working with exposed a set of abstractions that were at the wrong level for these particular developers. They expected a particular type of abstraction to be exposed by the API but since it wasn't, they couldn't find anything about it in the help docs. As a result of this, the API team redesigned this API to expose abstractions more in line with what developers were expecting. When we retested the API, it worked much better.

The dimensions in the framework have been adapted from decades worth of work done at the University of Cambridge on programming language usability. So far we've used the framework successfully around the company to evaluate APIs and to help to design usable APIs. It's still evolving though, so it would be great to hear your comments and feedback on this. I'd love to know about other frameworks or methods for evaluating and describing API usability also.