Here's an interesting interview with Anders Hejlsberg in which he makes some points about programming language usability.

Anders says that "good language design boils down to assembling a team of people who have good taste." I couldn't agree more. It's absolutely vital to understand your target customers so that you know what they will like and what they won't like. Developing that sense of good taste takes time and experience. You can build that experience up by designing languages or APIs that you yourself would like to use, if you consider yourself a good representative for your target users. But what do you do if you are designing a language or API for a group of users who you don't represent, such as children? This is where usability data is useful.

One of the biggest benefits of doing any kind of usability study is not the list of usability issues or bugs that you collect. Instead, it's the increase in customer empathy and understanding that comes of watching someone work and observing their work style and behavior. It's illuminating to see how differently someone thinks or works to the way that you would in that same situation. Gaining such an insight is incredibly useful when you are designing any kind of product, be it a programming language, an API or a GUI, especially when you are not representative of your users. Instead of basing design decisions on what you would do, you can now base design decisions on what your users would do.

Anders suggests that you don't get as high a yield from usability studies for language features as you do for IDE features. I think this is a reasonable statement to make when you compare the results obtained from a 2 hour usability study on a programming language or API with the results obtained from a 2 hour usability study on a GUI (with some exceptions - see below). Most APIs and programming languages take more than 2 hours to really get to grips with. Instead of just relying on one single two hour session per participant, API and programming language studies tend to demand more longitudinal studies where you can observe developers working with the API or programming language over a longer period of time, measured at least in days or weeks instead of hours (the exception is an API or language that supports tasks that you expect users to be able to complete in much shorter periods of time).

I'd argue that APIs and programming languages are interactive products, just as GUIs are. They expose affordances which indicate to the user the actions that the artifact makes possible. The big difference between GUIs and APIs or programming languages is how those affordances are discovered and learned, not whether or not they exist. That interaction between the user and the perceived affordances of the product is an important aspect that needs to be understood and designed. Usability studies are a very useful tool to help build up an understanding of how that two way interaction between a programmer and the API or programming language develops over time. With a programming language or API the interaction typically takes place over a longer period of time than with a GUI, hence the need for studies that take place over a longer period of time than GUI based studies.

As always, data obtained from usability studies should not be the only source of data that you collect about your design. It's always important to get a broad spectrum of data. But usability data should be part of that spectrum.