Brad Abrams posted a slide deck that I use to talk about the cognitive dimensions framework which we use at Microsoft to evaluate and describe API usability. I thought I would take this space to provide a little more explanation about how we use the framework.

The cognitive dimensions framework presents a set of dimensions that describe aspects of an API that impact its usability. For example, one of the dimensions is 'Abstraction Level' which describes the types of abstractions that the API exposes. Another dimension is 'Working Framework' which describes the mental load that the API places on its users. An API may be found to be unusable because the abstractions it exposes are too low level. Or it might place too much of a mental load on its users, requiring them to keep track of multiple objects in their head as they write code to perform a task. Or it could suffer from a combination of both problems, exposing multiple primitives that the developer has to keep track of in their head as they write code to accomplish a given task.

By evaluating an API with respect to each of these dimensions, you can form a picture of what the API looks like overall with respect to the framework. We like to create pretty graphs to do this. For example, here is a graph of an evaluation of some fictional API:


The black dotted line in the graph represents the API being evaluated. The spokes in the 'wheel' represent each of the different dimensions. End points (the middle and outer edges) on each spoke represent the end points on the scale for each dimension. Thus the point on each spoke where the black dotted line crosses it corresponds to the point on the scale for that dimension that the API has been valued at.

The most common way that we use to perform such an evaluation is to perform a usability study in our usability labs here at Microsoft. We invite developers from outside of Microsoft to come in to the labs and use an API on a given set of tasks, over a number of hours. We observe each developer as they work from behind a one way mirror and make notes about what parts of the API work well and what parts don't work well. We usually bring at least five developers in for at least two hours each for every study we do. So after ten hours worth of watching developers write code against the API, we spend time reviewing and analysing the data we collected and producing our evaluation of the API with respect to the cognitive dimensions framework.

However, there really isn't much point in going to all this bother if all we end up with is a graph like that shown above. It's not very helpful to provide the team developing the API with a graph showing how their API fits on the cognitive dimensions - there is nothing on that graph to tell them whether or not the API was judged usable or not. There is no baseline with which to make a comparison. That is where the individual developer personas come in to play.

We have defined what the three developer personas (Opportunistic, Pragmatic and Systematic) would judge to be the ideal API with respect to the cognitive dimensions framework. In other words, we have created profiles for each of the personas that describe for each dimension, the ideal point on the scale for that persona. The personas define quite different working styles so the profiles have ended up being quite unique. Thus we now have a way to judge whether or not an API is usable by comparing its evaluation with the profile we produced for the developer persona that best matches the target audience for that API. We can do this comparison in the same graph:


The blue line on the graph represents the profile of a particular developer persona. For example, it shows that this persona prefers APIs that expose aggregates (the outer edge on the abstraction level scale) and APIs that have small working frameworks (the inner edge, or middle, of the working framework scale). If you compare the points where the blue lines cross each spoke with the point where the black lime crosses each spoke, you get an idea for how well the API matches the persona's ideal API. Hopefully you can see that this particular API doesn't really hit the mark for this persona (there are big differences between the points where both lines cross on most, if not all, of the spokes in the graph).

Fundamentally the cognitive dimensions framework provides developers with a common vocabulary with which to talk about and discuss API usability. The benefits of such a common vocabulary are twofold. First, the language shapes the distinctions that are considered important to attend to. By explicitly describing each of the dimensions, it is more likely that they will be attended to. Secondly, the language allows developers to discuss issues using terminology that they can assume will be understood by others.

We've found that the cognitive dimensions have really helped us get to the root cause of issues that we have observed in the usability labs. For example, in one study, we observed lots of developers spending a large amount of time in the help docs looking for code samples that would show them how to accomplish a given task. The first interpretation of this data was simply 'Fix the help docs!'. However, when we used the cognitive dimensions framework to describe the issues, it became clear that the reason the developers weren't successful when they were searching through the help was because what they were looking for in the help simply didn't exist. The API they were working with exposed a set of abstractions that were at the wrong level for these particular developers. They expected a particular type of abstraction to be exposed by the API but since it wasn't, they couldn't find anything about it in the help docs. As a result of this, the API team redesigned this API to expose abstractions more in line with what developers were expecting. When we retested the API, it worked much better.

The dimensions in the framework have been adapted from decades worth of work done at the University of Cambridge on programming language usability. So far we've used the framework successfully around the company to evaluate APIs and to help to design usable APIs. It's still evolving though, so it would be great to hear your comments and feedback on this. I'd love to know about other frameworks or methods for evaluating and describing API usability also.