Hi. Thanks for all the feedback on my last Blog entry. The C# team did a lot of work trying to understand our users, and it’s great to see where you all do and don’t agree with our conclusions. I thought it might be interesting for you all to get a peek into a different part of our product development process, when we usability test new features. In this entry I’ll briefly cover the C# specific usability study that we did for Generics, including a little on how we tested it, and what we found. It was interesting for me to study a language feature as apposed to a set of UI, and I think we learned a lot.
We're going way back here, to October 2002. I was working with Anson Horton, who was at the time the PM for the C# compiler. We wanted to get some understanding of how users would react to generics and our documentation for them. We wanted to know if users would be able to master the syntax with a minimum of frustration, and would they get how to use them to write clean code. We recruited 5 Visual C# developers to come to the Microsoft Usability labs, and asked them to try some programming with generics.
The Set Up
Anson wrote a simple application that used a regular old hashtable of objects from the Visual C# 2003 (Everett). We asked the documentation writer to get us the best version of the Generics documentation that they could before the study. We put a copy of the docs in the waiting room so the participants could read as much or as little of the documentation as they wanted before they started coding. We also taped a copy of the documents down to the table next to their computer where we could keep a remote camera on the docs and get an idea about what the users were reading. There were 56 pages of documentation altogether that covered the concepts of Generics, but mostly focused on the syntax of all the different parts of consuming and creating Generic classes.
We wrote out some things that we wanted the users to do with the Anson’s program. Here are two of the tasks we gave them:
(Note: This task was used to get a sense of how well users understood the Generics concept. The code used a hashtable, and we wanted to see if they could recognize that the hashtable was a good candidate for Generics. Then we wanted to see their first experience using Generics was good and productive.)
(Note: This task was used to get a sense for how easy or hard it would be for users to create a class that was generic. We started them with a non-generic version so they could focus on just the generic code in the study.)
We also created an interview, and Anson interviewed each participant about their impressions of Generics after they tried it.
Here are some of the key results from the study.
In general, users will be able to consume generic collection classes. However, they will find the syntax of specifying the type parameters in the instantiation clause to be unintuitive.
Basically, what we saw was that when the developers should have written something like this:
List<string> myListOfStrings = new List<string>();
They would look at the docs and write:
But they struggled to get: myListOfStrings = new List<string>();
There were a few of factors that made this hard it hard to write the full line of code (note that we were using very early bits for this, so all the namespaces and class names have since changed):
Included type parameters with declaration clause, but not with instantiation clause: “error CS0691: ‘ArrayList’ type not found. ‘ArrayList’ has the wrong number of type parameters”
Included type parameters with instantiation clause, but not with declaration clause: “error CS0029: Cannot implicitly convert type ‘Experimental.Collections.Generic.ArrayList<int>’ to ‘System.Collections.ArrayList’”
I think you can see why the error messages didn’t help much for developers trying to learn the syntax. I think the current error messages are slightly better.
Users want to use code snippets to learn the syntax
Not much to say here. Basically, all the users except 1 skipped everything in the docs, and just skimmed for code until they saw a code fragment that did what they wanted. We see this effect all over the place, not just learning syntax, but it was especially noticeable here. I guess it figures that people would want to learn syntax by reading examples, but one of the developers did use the “grammar” section that describes the usage of the syntax in a general way, not with examples.
Users want specific code snippets for the class they are trying to consume
This observation followed the last one, but we saw that the developers skimmed the documentation for examples using the particular class the wanted to use, not just any example. For instance, if they were trying to instantiate a HashTable<K,V>, but the first example was for a List<T> they kept skimming. After they found out that the only example was for a List<T> they went back and used that. During the interview they said they would have been happier with examples for every generic class they could use.
After consuming a Generic class, users will be able to create their own Generic classes
We saw that by following the examples, the developers were able to convert Anson’s LinkedList to be strongly typed. However, a few of the developers felt that they wouldn’t really need to do so, because they don’t need much more then the .NET Framework was providing. Good news, it sounds like we have the right set of classes in the Framework.
C# coding will be better with Generics
After trying them, all 5 developers said that Generics was definitely a great addition to the language, and would make their code cleaner to read and run better because they would never get runtime exceptions related to casting. This was congruous from the feedback that we got from our other customer channels as well. The difference was that this feedback was based on actual experience of Generics in code, so we felt that our predictions about the customer reaction to Generics were probably true.
Templates don’t kill developers, developers kill developers
This harkens back a little to the “Battle Scars from C++” comment in my last blog entry. During the interview, the developers recounted stories of working on C++ projects where templates were hideously misused, and caused them endless pain trying to use other developers template code. They thought that this was a risk with Generics too. I think we need to provide strong guidelines on when to use, and especially when not to use, generic classes.