stevencl's WebLog

Help us learn about how you work

2006-01-30T23:24:00Z

We want to make better products.

One way of doing that would be to design a product that we would like to use and then hope that others outside of Microsoft would also like to use it. Sometimes that approach works, other times it doesn't. Not everybody writes the same kind of software that we do and not everybody likes to work the same way that we do. So it's likely that not everybody will like tools that we design for ourselves.

But the only way to learn if we need to design products differently is to understand how people work and how they use the tools that they use. One of the best ways to gain that understanding is to visit people at their work and observe what they do. Only then can we really understand what they do, how they do it and why they do it that way. Other approaches such as focus groups and lab studies are useful, but one thing that we've learned is that it's easier for somebody to really explain what they do by doing it rather than just talk about it.

So, we're trying to find people who are willing to have us come and visit them for a few hours to simply watch them work, and ask a few questions along the way. We're not asking people to take time out of their work. Instead, we'd like them to do whatever they would normally be doing, just with us watching them. If you'd be willing to participate, please let me know.

Obviously security concerns are an issue - we will not share any proprietary information that we learn about while visiting you.

What would you get out of it you ask? Well, in addition to having the opportunity to directly impact and influence the future of Visual Studio you'll also get your pick of free software from our gratuity list (there is no handy link to the list of software on the gratuity list but in the past this has included Visual Studio Professional).

Waterfall conference

2006-01-30T23:13:00Z

This conference looks interesting... :-)

In particular, this workshop looks like it might have some good content.

(Thanks to Frank Wales for posting this to the PPIG discussion list)

Using CDs to design a new programming language

2005-12-02T18:05:00Z

I came across an interesting Master's thesis the other day. Chad Austin developed a functional shading language and used the cognitive dimensions framework to guide and evaluate the design of the language.

I'd be really interested in hearing more about this type of work. I'm keen to learn from others experiences using the cognitive dimensions framework.

Studying design patterns

2005-12-01T01:11:00Z

The Channel 9 video prompted some interesting questions. One question asked about common patterns and design guidelines. Seemed like a good opportunity to mention that we're currently lucky enough to have Jeff Stylos from Carnegie Mellon University working with us as an intern until the end of December. While he's here he's studying different object creation patterns, particularly the use of required constructors versus default constructors. This is one of the first studies we've done that doesn't focus on a specific API but looks at design guidelines and patterns instead.

Check out one of Jeff's recent projects: http://gem.pebbles.cs.cmu.edu:8080/mica/.

Usability study video

2005-12-01T00:46:00Z

In October, James Conard made a video of one of the usability sessions we ran on the Windows Workflow API. The video is now up on Channel 9.

Somebody asked if the results of studies like these are generalised beyond the specific APIs being tested. The answer is yes, they are. There are a couple of ways we do this.

I work with Krzysztof, Brad, Joe, Mitch and others on the WinFX review team. Whenever we're reviewing an API, I'll look for API designs that I think will exhibit the same issues that have been observed in the labs. We'll discuss during the review the extent to which the same issue may or may not manifest itself in the API we're reviewing and the API team will decide whether or not they will make changes. An example of such an issue is the use of attributes.
When we observe patterns of behavior across multiple studies we roll these up and produce a design guideline that aims to inform API designers how to avoid the same issue. An example of a pattern we've seen that turned into a design guideline is the recommendation to favor using enums instead of bools to improve readability of code.

Maintaining state in an Indigo service

2005-08-26T16:39:00Z

I've been working on usability studies for Indigo for the last 12 months now and have been learning a lot. I just learned another very valuable lesson this week, thanks to Steve Swartz.

During a study I was running last week, participants worked with a service that maintained state between calls for each session but that did not share the same state across multiple sessions. However, participants expected that the state would be persisted across sessions such that one client proxy could call an operation that would change the state and another proxy would see that state change in a different operation. Participants were writing code to bind the results of the operation to Avalon UI and the fact that state was not persisted across sessions caused them to have to rewrite a lot of their data binding code.

The contract for the service was as follows:

[ServiceContract(Namespace = "http://Usability.Task")]
interface IAlbumService
{
[OperationContract]
Album[] GetAlbumList();

[OperationContract]
void AddAlbum(string title);

[OperationContract]
int GetNumberOfAlbums();

[OperationContract]
void SellAlbum(string title);

}

You'll notice that the contract says nothing about whether or not state is persisted. I had been setting the InstanceMode property on the service implementation to maintain state but of course, this is part of the implementation, not the contract, and so consumers of the service have no visibility into the setting of this property. I had thought that it might be useful to communicate to users the setting of the InstanceMode property but Steve made it clear that this was not appropriate, since this really has nothing to do with the statefulness of the service. This property is provided as a way to allow service developers to deal with perf and scalability issues in their service implementation, not necessarily to provide service developers with a mechanism by which to preserve state between calls or between sessions.

So I asked how best to communicate the stateful nature of a contract? Steve offered two suggestions (hopefully Steve doesn't mind me quoting him below). The first involved adding Open and Close methods to the contract:

"I would add an Open() and Close() method to the contract. Open() would be the only method that could initiate a connection; Close would be the method that would always close a connection and spill the instance on the service. Open() would take a key that would identify the app session being dealt with. The service would be storing data in a database (or in a file – whatever works for the particular app)."

The second involves passing an id in each of the operations to identify the 'session' that the client is participating in:

"Many people would argue that a stateless “per-call” service is more scalable and robust. The way to build that into the contract is to put an album-list identifier into each of these methods. If any particular client was only going to have one album list, then you might use cookies so that all calls from machine X would go back to the same album list. These strategies scale better because they don’t rely on connections. You might use these strategies even with per-session instances, because they would let the client app use a session for three or four calls in a particular part of the UI (caching security information in the channel), and then let them get at the same state from some other session."

In both suggestions, the fact that state is persisted in some way is made explicit by the design of the contract. In terms of usability, being explicit is almost always a good thing. It means that there is one less assumption that users have to make about how an API works and in the case of the usability study last week, would likely have prevented participants from rewriting all their data binding code after realizing that the initial assumptions they made about how the service operated were incorrect.

I think that one of the reasons that I hadn't designed the contract this way originally was because I was still thinking in an object oriented fashion instead of a service oriented fashion. I was still thinking in terms of objects that have state instead of services that are stateless. Looking back now it seems like such an obvious mistake to make but I expect that it's one that many people might make when they transition from building objects to building services. It strikes me that being explicit about maintaining state in the design of the service contract could be a best practice that might be codified in tools such as FxCop and others. Having a rule that fires whenever it appears that a service implementation is storing state and does not expose Open or Close operations or does not appear to take some id or token in each operation might be a useful way to guide developers towards best practices and help them think in terms of services. Or even better might be some tool that helps generate a contract for you based on certain requirements that the user feeds into the tool and thus would generate the appropriate operations for you.

I'm not sure how feasible it would be to write such an FxCop rule or build such a tool but the idea of packaging up best practice guidance such as this in tools is appealing. I'd be interested in hearing about other best practices that people follow when building services and to learn about ways in which people ensure that those best practices are followed.

HOWTO: Analyze the data collected in a usability study

2005-07-11T17:39:00Z

I just realised that I never got around to finishing off a series of posts on how to design and run an API usability study.

After designing and running the study, the fun part begins - making sense of all the data that you have collected.

The first thing to do is to collect together all the patterns of behaviour that you observed. For example, any problems that were experienced by two or more participants, or similar expectations that two or more participants had. Typically I list out all of the problems that people had in terms of the tasks that they were working on.

The next question to ask is why did the participants experience problems? To understand the reason for the problems, I use the cognitive dimensions framework. For each problem, I try to relate it to one or more of the cognitive dimensions. I look carefully at the notes I took, the comments made by participants while working on the task and the code that they wrote while working on the task, carefully considering the changes that they make to their code during the session. The cognitive dimensions framework provides multiple perspectives with which to view a problem. Consider the first dimension, abstraction level. While watching a series of video clips of participants struggling to find a code snippet to show them how to write a line of text to a file, I found myself asking "Is the problem due to the abstraction level of the API?". I then started looking more closely for behaviours and comments that would help me answer the question. Further probing into the data found that many participants made comments about the types they were browsing in the documentation being too "low level". Looking at the query strings created by participants when searching the documentation provided me with a hint about the type of abstraction they were looking for. Queries performed by many participants used the words "File" quite often. Thus it started looking as if the fundamental problem was due to the level of abstraction offered by the API not matching users expectations.

After identifying one dimension which helps explain the issue it's important to continue to look for other dimensions which might offer another perspective. Each dimension in the framework is non-orthogonal and changes in one dimension can affect other dimensions. I continued to look for other perspectives and it became clear that the API also suffered from a larger work step unit than expected. Participants often needed to work with more than one object to accomplish the task, often using a factory object to create an instance of some other object. It was clear from watching the sequence of code written by users that this was completely unexpected.

After explaining all of the probems that I listed at the start with the use of the cognitive dimensions, I then start going through the dimensions looking for other issues. Sometimes there are issues that aren't immediately obvious during the actual participant sessions or that don't cause significant enough problems for participants to stop them from being successful at the different tasks. But these might still be significant enough to make the experience of using the API an unpleasant one. API usability isn't just about ensuring that users will be successful. Users also need to enjoy working with an API, otherwise they might not choose to use your API. I call these type of issues "paper cuts". You want to avoid the "death by a thousand paper cuts" phenomenon. I use the cognitive dimensions to look for these paper cut issues.

For each dimension I go through the data looking for instances where the API might not meet users expectations with respect to that particular dimension. For these types of issues I don't limit myself to instances experienced by two or more participants. Particular paper cuts might not be experienced by everybody, but if everybody experiences a subset of the set of paper cuts, even if everybody experiences a different subset, that could be enough to tip them over the edge from wanting to use the API to not wanting to use it.

The last thing I do is collect video clips that demonstrate each issue collected and present my findings to the team. You'll notice that I do not spend time designing solutions to the problems. My job as a usability engineer is not to design an API. It's to provide the people designing the API with the information and knowledge they need to be able to design as usable an API as they can. I do this through the use of the cognitive dimensions framework and have found that this provides the necessary insight and detail to understand why problems were experienced with an API such that a good solution to those problems can be designed.

PPIG 2005 and blogging

2005-07-11T17:24:00Z

I attended the Psychology of Programming Interest Group workshop at the end of June in Brighton. It was a really interesting workshop, finished off nicely by a keynote from Marc Eisenstadt. He encouraged all attendees at the workshop to consider starting up a blog and posting on various programming usability issues. I'm looking forward to seeing lots of diverse and interesting posts on API and programming usability in the future. Marc has a blog and posted about PPIG here.

Factory design pattern and usability

2005-06-21T20:29:00Z

Luke Church watched the API usability presentation and contacted me with a question about a point I made regarding the factory design pattern and usability.

One thing that I probably didn’t make clear in the presentation is that the difficulties with the factory design pattern are greatest for opportunistic programmers (or programmers working in an opportunistic style). There are three reasons why these programmers have difficulties with the factory design pattern:

These programmers like to learn by exploration and make heavy use of design time tools to tell them what they can do. A common work style is to create an instance of an object and then use intellisense on that instance to see what methods the type exposes, thus learning what that type is capable of.
In many cases these programmers have a narrow code focus but a broad task focus. What I mean by that is that when they are writing code they tend to concentrate on the particular line of code that they are writing. They tend not to focus on higher levels of code abstraction such as the method that the line of code is contained within, nor the class exposing the method or the application that defines the class. Thus solutions that require them to consider their code at higher levels of code abstraction (e.g., configuring some object in a constructor and using it in another method, using another type to create an instance of a desired type) will be more difficult to find. In terms of their broad task focus, this means that they tend not to break a task down into many multiple, fine grained steps. Instead they consider the steps required to accomplish the task at a fairly high level. If a task has a high work-step unit it’s unlikely that these developers will be able to discover all the steps required.
These programmers have a very domain centered perspective as opposed to an implementation centered perspective. Sometimes the factory pattern is used for implementation details. Perhaps using the factory will lead to better performance if the factory can control instance creation. However, these details are often not considered by these programmers since they focus on the domain, not the implementation.

For these same reasons, other types of programmers have fewer difficulties with the factory design pattern. The pragmatic programmers are similar to opportunistic programmers (certainly in terms of point 1 above) but the difference is that when they discover that the type they want to create does not expose a constructor they expand their code focus and narrow their task focus as well as start to consider reasons why the type does not expose a constructor. Thus they are likely to end up finding the factory that lets them create the instance of that type.

Systematic programmers approach the task differently and, having a much wider code focus to begin with will be more likely to assume or to learn that type instances are created by a factory before they get started writing code.

We're hiring

2005-06-09T00:13:00Z

We have several open positions in the Visual Studio user experience team. One of those positions is for somebody to work with me on the API usability efforts. We've got lots of exciting ideas about research projects that we'd like to work on to provide API teams at Microsoft with data about their customers that helps them create truly usable APIs. If you're passionate about understanding how developers work and if you love programming, learning about new APIs and playing with the latest technologies then this is the job for you.

Please apply or contact me if you're interested.

API usability evaluation

2005-05-05T21:53:00Z

Christopher Oezbek is keeping notes about an API usability evaluation he is doing using the cognitive dimensions.

HOWTO: Run an API usability study

2005-05-05T17:31:00Z

With the task list in place and participants recruited, it's time to run the study. My experience has been that running an API usability study is really no different from running any other type of study. Here's a description of what I do.

The day before the study I make sure that the machines I'll be using in the usability lab are set up. I install the necessary builds of Visual Studio and associated SDKs etc to be able to run the study. I always do this first thing in the morning just in case anything goes wrong as setting the machines up properly can be quite time consuming. Once that is done, I then go about setting up the screen capture software so that I have video recordings of each participant's session in the lab.

While our usability labs are well equipped with audio and video recording equipment, I prefer to use HyperCam for screen recording since the quality is decent and the format (AVI) makes it easy for me to create video clips to show highlights from each session. Running HyperCam and VisualStudio on the same machine can sometimes cause the machine to slow down pretty significantly though, so I run HyperCam on one machine and then use remote desktop from that machine to connect to the machine running Visual Studio. Setting up HyperCam, including setting up the PC microphone so that the recording level is just right usually takes no more than 10 or 15 minutes.

Our usability labs are split between the participant side and the observer side. After setting up the participant side of the room, I set the machine up on the observer side. We have in-house tools that we use to record our observations during each session. The tools allow us to create a simple tagging scheme that we use to tag our observations as we take them so that afterwards we can easily refer to all events that involved use of the ServiceContract attribute for example, or events that were particular highlights for whatever reason. I take some time to think about how I might want to analyze the observations after all the sessions and set the tools up to use the particular tagging 'schema' that I come up with. Sometimes the schema is as simple as three tags labeled 'Highlight', 'Success' and 'Failure' but other times it will be more complex. It all depends on the nature of the study.

Lastly, I print out the task list and a copy of the tutorial materials that were sent to participants, just in case participants forget to bring their own copy in. I always leave printing the task list to last as invariably I'll make small changes to the task list right up the last minute.

After all this setup, all that is left is to run each participant's session.

The first time each participant comes into the lab I spend some time getting them familiar with the lab. It can be quite disconcerting to be sitting in a lab knowing that people are watching you behind a one way mirror so I reassure participants that the purpose of the study is to learn how usable the API is, not to test each participant that comes into the lab. I also give some participants time to practice talking out loud while they work. We always ask participants to provide us with a running commentary of what they are doing as they are working as that is the best way to get at the intent behind the actions that they take. However it can feel a little odd to talk so much while working since it's not something that most people are used to doing. So for participants who have not been in the lab before, I will sometimes ask them to do some task (such as using Paint to draw a picture of a mouse) and ask them to talk out loud while they are working. Many participants find that after just a few minutes of practice that it's actually not that difficult to keep talking while working.

Then participants start working on the tasks. While they work on each task, I try to take as many notes about what they are doing as I can. I typically try to record a transcript of everything they say as they are working. When something interesting happens, I'll make sure to annotate my notes accordingly so that I can easily refer back to that event (our note taking software timestamps each note that I take so that I can associate a note with a point in time on the video recording of the session). Paying attention to what is going on and taking good notes can be very tiring but it's critical, especially if you don't want to be spending hours reviewing video tapes and other logs afterwards. Time and effort spent here pays off in the long run.

Probably the most important thing to be aware of when running a usability session is the impact that anything you say to the participant has on their behavior. The most obvious way in which you influence the behavior of the participant is in how you describe the task you ask them to perform. If you mention the specific classes, namespaces, methods etc that you want them to use, then you've led them down a path that they might not necessarily have taken had you not mentioned it. But during the study there are other less obvious ways that you can inadvertently influence participants. Very often, participants will get stuck with a task and might get a little frustrated. They'll ask for your help and the natural inclination is to offer some help, particularly if they are very frustrated. You can simply tell them how to get unstuck, but that doesn't really provide you with useful data about why they got stuck and how they expected to be able to fix the problem themselves. All you know is that when told to do the right thing, they were able to do it. Instead, you should see this as an opportunity to understand what it is about the design of the API that leads participants to get stuck.

These sort of situations are where the benefits of running a usability study really become apparent. As soon as a participant gets stuck on some task, they are in the best position to describe how and why they are stuck because they are literally going through the experience right there and then in front of you. They don't have to rely on their memory to tell you what happened, as they would if you simply interviewed somebody about their use of an API after they were done using it. They are motivated to explain things to you because that will help them recover the situation, unlike if you were asking them to describe the situation after the event.

Whenever participants ask me questions when they are stuck, I'll very often not tell them anything and will instead ask them to describe what they think is happening and how they expect the API to work. With that information I'm able to start forming hypotheses about the conceptual model that participants are forming while working with the API and how that leads to breakdowns in using the API. I can then start testing these hypotheses by asking participants to provide feedback about the way that these conceptual models are formed. I don't ask them to comment explicitly on their conceptual model but instead I'll ask them about salient aspects of the API that I think might lead to a particular model being formed. These are the points in a usability study during which I learn the most from participants.

Outside of these situations I try to say very little during the session. For example, to begin with, lots of participants ask me when they should move on to the next task. I try never to answer these questions, telling participants that they should move on to the next task when they think they are done with a task. I don't want participants to get used to my confirmation that they have done the right thing. If I keep telling participants that they are doing the right thing, participants tend to do something then wait to hear my confirmation before moving on. Instead of using their own intuition to figure out if they have done the right thing, they simply wait for an acknowledgement from me. At that point the data I'm getting is questionable.

The key criteria for success in running a usability study are: good observation and note taking skills, and the ability to understand observed behavior from the participant's perspective, instead of your own. You'll notice that running a successful study does not depend on having expensive usability labs. They're definitely nice to have, but not necessary.

HOWTO: Design a task list for an API usability study

2005-04-19T22:33:00Z

In my previous post I talked about setting up an API usability study. In this post, I'll talk about how I design the tasks that participants work on in a study.

After getting some level of familiarity with the API, I tend to start designing a task list by asking the API team what they want to learn. Sometimes teams will have specific research questions they would like answers to, such as 'Will users be more successful with the FileObject class for doing file I/O than they would be with the StreamReader and StreamWriter classes?'. Very often though, teams tell me that they want to know whether or not the API is usable and what the usability problems are.

In that case, I ask the team what scenarios the API is designed to help the user accomplish and which of the three personas the API is designed for. I then start working on the scenarios, trying my best to think and act like that persona (some personas are easier to act like than others :-) ). As I work on the scenarios, I take notes about how I expected the scenario to work, what I actually had to do to make it work, naming issues that tripped me up, conceptual issues that caused me some difficulties etc etc. Each of these notes turns into research questions that I might consider trying to answer during the study. For example, if I thought that a particular class was named poorly, one research question would be to determine whether or not that class name made sense for participants. I'd design a task that required participants to use that class and then measure the effect that the name of the class had on participants abilities to complete the task.

I also look for assumptions that the team has made about how the API will be used so that I can design tasks that verify those assumptions. For example, in the Indigo study I completed recently, one assumption that the team had made was that separating out the configuration of a service from it's implementation would make users more successful. This separation was achieved by using configuration files to set things like the address of a service and using the implementation of a service to set things like the instance mode of that service. So I wanted to make sure that I designed tasks that determined whether or not participants knew when to work with the configuration files and when to work with the implementation of the service. I needed tasks that required the user to modify the configuration of the service and tasks that required the user to modify the implementation of the service.

After coming up with a list of tasks that participants will work on, you then have to think about the order in which you present these tasks. This is a pretty critical aspect of designing a good usability study and if done wrong can really ruin the results that you get. When thinking about the order of the tasks, you need to be aware of the side effects of each task, (the additional things that the user will learn or do while working on a task that isn't critical to that task), the dependencies between tasks and the degree of difficulty of each task. You don't want participants to learn something critical about task B while working on task A since that will compromise the data you collect about task B. It effectively means that the behavior you observed while participants were working on task B will only generalise to all users who work on task B, after working on task A. It tells you nothing about how users will work when they work on task B alone. It's also a good idea to start participants off with easier tasks since being successful in the usability lab early on makes participants a lot more comfortable.

The last thing I work on is the wording of the tasks. There are two things that you need to make sure of when wording the tasks:

The description of the task will not lead participants to attempt a particular solution unless that is what you want them to do.
The description of the task is clear enough that participants know what you would like them to do, but do not know why you are asking them to do the task.

Obviously you don't want to bias the way that participants work while in the lab. The best study is one that mimics how real users work in the real world as closely as possible. You also don't want participants to know why they are doing what you are asking of them, nor do you want them to know about the measures that you are taking, otherwise participants (whether consciously or sub-consciously) will start behaving in a way that they think will give you the measures or data that you need, instead of behaving in a way that will help them accomplish the task that you have asked them to work on.

You might also want to consider repeat tasks so that you can compare the observations you made the first time the participant worked on a task to the next time they work on a similar task. I never repeat a task word for word, but I do ask participants to work on something that is very similar to something they have done earlier. Most often I'll make sure that there is a reasonable space of time in between the repeat tasks, at least an hour, but sometimes a couple of days or more.

When I complete the task list, I review it by asking myself whether or not the data that I think I will collect from each task will help me find answers to the research questions that I collected at the beginning. I also try to picture myself presenting the results from the study to the team and defending the way that I designed the study - sometimes people can be a bit defensive when hearing the results of a usability study on their API and they will try to explain away the results by attacking the study design rather than looking at the API. If I can't explain why I designed a task the way that I did, I'll take another look at it to see how I might do a better job.

HOWTO: Design and run an API usability study

2005-03-29T16:26:00Z

A few people have asked me about how I design and run API usability studies. I'm running an Indigo study this week so I thought I would describe the steps that went into setting this study up.

The study is a follow up to the Indigo study we ran in October and which identified a number of usability issues, the biggest of which was the extensive use of attributes. Since then, the team redesigned parts of the API and were keen to get fresh usability data on the new design to see if the problems were resolved. To get the new study kicked off, I met with Steve Swartz (one of the Indigo architects) and David Palmann (a program manager on the Indigo team responsible for usability) in December 2004 so that they could describe the design changes to me.

We discussed the earliest date that we could run the follow up study. The implementation of the new API design wouldn't be complete until some time in February and then with the necessary testing that needs to take place, it looked likely that the earliest a stable build would be available wouldn't be until some time in March. We tentatively scheduled early March to run the study.

One of the first trade-offs to make in setting up a study is deciding when to do it. You need to make sure that you run the study with a version of the API that looks pretty much the way that the team have designed it but you don't want to wait until the API is completely implemented and tested since there probably won't be much time to respond to any changes suggested by the usability study. We thought that we would be able to secure a reasonable build of the new Indigo API by early March which would still give the team enough time to respond to any usability issues.

With a rough timeframe in mind, I drafted out a plan for the work that I would need to do to prepare and run the study. Typically the very first thing that I like to do is to get hold of a build of the API such that I can spend some time coding against the API to get a feel for the API and to identify any potential usability problems in the API. This build needn't be the exact same build that I will use in the study itself, just something that I can get up and running and playing with. At this point in time (late December), a build wasn't yet available so I had to put off being able to play with the API until some time later in January.

This would leave me with at most four weeks to play with the API and prepare the study if we were to run the study in early March. Normally this would concern me since one risk is that once I start playing with the API, I discover that it is pretty complex and that preparing the study materials is likely to take a significant amount of time. Since I am typically running/designing more than one study at a time it can be quite difficult to find the time to prepare the materials for a complex API when there are other studies going on.

In this case though I wasn't worried since I already had materials ready from the October study. I just needed to update those materials based on the new changes. The study materials that need to be prepared are basically an API introduction document that each participant gets before they come into the lab and the task list that participants work on when they are in the lab. The API introduction is a very important document - it gives participants enough of an intro to the API so that when they come in to the lab they have some idea of what to expect and what they will be working on. However, the intro needs to be tuned to the needs of the study. It's important that the intro gives participants enough information that they feel comfortable working on a task in the usability lab, but not so much that we just tell them what to do. Very often I end up writing this document myself from scratch based on my experiences using the API for a few weeks. I try to look back on my experiences using the API and try to document answers to questions that I had while getting to grips with the API. Quite often these will be things that I needed help from the API team to figure out.

We try to send the intro document to participants at least a week before they come into the lab so if I wanted to run the study in early March., I really needed to have a draft of the document complete by mid February so that it could be reviewed before being delivered to participants. I was able to reuse the document that was sent to participants for the October study and edit this to reflect the new API. In late January I spent an hour with Steve Swartz making changes to the original document.

At this point I still hadn't used the new API myself. I found that when I was working on the changes to the tutorial document that Steve had suggested there were still some niggling questions that remained which would have been easily answered had I been able to use the API and explore for myself. So I waited until a reasonably stable build was available for me to play with. In mid-February David was able to provide me with a Virtual PC image containing a build of the API that I could use. This helped me complete the edits to the intro doc so that it reflected the new API.

Around the same time, other projects I was working on required some attention. There were UI reviews for Visual Studio Team System and Joey Lawrance had just started his internship with us. He and I were working on a pretty big study that was to take place throughout February and this took higher priority. This meant that our original plan of running the study in early March wasn't feasible so we had to postpone the study for a couple of weeks. But I had a build of the API now so I could get at least think about getting started on the task list.

In my next posting I'll talk about how I go about designing tasks for an API usability study.

API usability talk available on MSDN from tomorrow

2005-02-17T21:30:00Z

The API usability portion of the Designing .NET Class Libraries class will be available from tomorrow. Apologies in advance if my Scottish accent makes it difficult to understand me! Let me know if there is anything that I need to translate...