Error messages: diagnostic is preferable to prescriptive

Error messages: diagnostic is preferable to prescriptive

Rate This
  • Comments 35

The new LINQ features are going to create new failure modes for the compiler, so we're going to need to create some new error messages. The compiler development team got together the other day to discuss what makes an error message good or bad. I thought I'd share with you guys what we came up with. We believe that good error messages are:

  • Polite: making the user feel like an idiot is very, very bad.
  • Readable: poor grammar and tortured sentence structure is bad.
  • Accurate: error messages must accurately describe the problem.
  • Precise: "Something is wrong" is an accurate error message but not a very precise one!
  • Diagnostic but not prescriptive: describe the problem, not the solution.

The first four are obviously goodness. That last one is a little more controversial. Surely a good error message not only tells you what is wrong but helps you fix it, no?

The issue is that deducing what is wrong with bad code is hard enough. Trying to read the user's mind and figure out what they were thinking when they wrote the bad code, and then telling them how to correctly implement that thought is not something that we feel we can do with sufficiently high accuracy in most situations.

Look at it this way: suppose we pull off a miracle and manage to produce error messages which 90% of the time tell the user the correct way to fix their code so that it does what they want it to do. That means that 10% of the time we are telling people how to write a syntactically correct program that does something different than they intended! Pushing people towards writing buggy programs that still compile is very bad, and we do not want to go there.

It's instructive to look at a few places where we violated these guidelines in earlier versions of the compiler:

Static member 'Baz' cannot be marked as override, virtual or abstract

If the user wrote static virtual, then we don't know what the heck they meant to do. Assuming that they meant to say static and that the virtual is wrong is a little presumptuous. Maybe the static is the wrong part! Also, if the user said static virtual, then why is the error message mentioning override and abstract? That's accurate but not precise. A better error message in this case would be something like

Member 'Baz' cannot be both static and virtual

Here's another place where we get it wrong, but this one is more subtle:

A params parameter must be the last parameter in a formal parameter list

This is an example of an English sentence that can be interpreted different ways depending on the context. If I said "a punctuation mark must be the last symbol of a sentence" then I mean that every sentence must end in a punctuation mark, but I do not mean that punctuation marks are only legal at the end of a sentence. If I said "a period must be the last symbol of a statement" then I mean that every statement must end in a period, and furthermore that periods are forbidden anywhere else in the statement.

You and I know that what the error message is trying to say is that if there is a params then it must go at the end. But based solely on this error message, a user would be entirely logically justified in thinking that (int i) is an illegal parameter list because it doesn't end with a params parameter. Or, under another interpretation, they'd also be logically justified in concluding that (params int[] foo, params int[] bar) is legal, because it does end with a params parameter.

The portion of the specification which the error message is attempting to draw attention to is of course "If a formal parameter list includes a parameter array then it must be the last parameter in the list. There can only be one parameter array for a given method" which is nicely unambiguous. Why not simply use this quote from the specification for the error message? That's a reasonable idea, but it sounds a little stiff and doesn't call out where the problem is. I'd prefer:

Method 'Foo' has a parameter array parameter which is not the last parameter in the formal parameter list.

This tells you what is wrong without telling you how to fix it. Since we don't know how to fix it – whether the user should be removing the params modifier, or moving it to the end, or rewriting their method from scratch – we should just report the spec violation and let them sort it out.

There are times when we do want to tell the user what to do, but only when we are highly likely to be correct. For example:

User-defined operator 'Blah' must be declared static and public.

Here we are both diagnosing the problem and prescribing a solution. If they're trying to make a user-defined operator, this is what they absolutely must do to be successful. It is very unlikely that they wanted to make a private instance function and made a private instance operator by mistake!

This illustrates another principle of good error messages that I didn't call out before: good error messages use precise terminology from the standard rather than making up new jargon. Yes "formal parameter list" and "user-defined operator" are a little bit stiff, but they are also clearly defined in the standard.

Sometimes we get the error right but the wording could be improved:

Foo: static classes cannot be used as constraints

Why are they trying to use a static class as a constraint? Who knows? How should they fix it? Beats me! The best we can do is to tell them that it hurts when they try to do that. But the wording! Good heavens! Would you ever say "Pizza: delicious foods should be eaten while they're fresh!" ??? Clearly

Static class 'Foo' cannot be used as a constraint

is much better.

Anyone have additional suggestions for what makes a good error message? Or other examples of places where we got it wrong?

  • Re: localizing error messages: I'm strongly against localized development tools. Most developers have to know English anyway, because all the programming languages are based on English, all library functions names are in English, and most documentation is in English anyway. In addition to this, the localization team usually has to invent a lot of terms on their own when there's no common translation of it in the foreign language at the moment of localization. These newborn terms are much less informative to a programmer, compared to the International English terms.
  • PingBack from http://hasalotofnothingtosay.wordpress.com/2006/10/10/what-makes-a-good-error-message/

  • If people aren't searching by error codes, then perhaps the error message needs a more specific coding.  If the code is unique, then perhaps the people in question need to be re-educated about error messages.

  • oh men! error messages should be as unclear and messy as possible; only then someone who is facing problems will look into books and understand why is something an error and how to do it correctly instead of just playing with letters in source code, trying to satisfy short error report;

  • Emitting the compiler errors in the correct sequence and recognizing when compiler errors are the result of previous errors would be helpful as well. In one case, I was attempting to compile a solution and the compiler emitted a large list of errors. This led me on a wild goose chase until I finally uncovered a specific compiler error that led me to the root of the problem, but this error was buried toward the end of the list. I'm not certain why because it was the obvious cause of all other errors. I fixed one line of code and the solution compiled fine.

    Other types of errors that lead to a WTF? moment are those that provide no error source (e.g., a compiler error that provides no code file, no line number). Today I attempted to compile a solution and the compiler emitted something like "Culture ID blah (0xblah) is not a valid culture. Parameter: culture" without giving any information on what caused the problem or how to diagnose the problem. I suspect the Framework or Visual Studio is corrupt, but the error isn't documented and is extremely confusing. One of the first questions I asked myself: "What method takes culture as a parameter and why is it invalid?" I even searched my code just to be certain it wasn't something I had caused. There are other instances in which something in a project (other than code) causes a compiler error, but the compiler emits the error without a reference to the cause. This sometimes leads me to chase down project options, resources, settings, dependencies, etc. to try to read the compiler's mind.

  • Localization

    "The common case is non-english-speaking programmer gets an error, if the error is in their language then they stay productive, and if it isn't, then they're stuck.  Googling it in English is hardly going to help them!"

    English language being not my mother tongue, I think I may say something here. Please localize KEYWORDS or forget the localization of programming languages altogether. No programmer (or at least a bit interested in programming) in Lithuania doesn't know English. I discourage my colleagues coming up with non-English variable names, because the code just gets more messy with that. You can't write "do{}while(teisybė)" what's "while" what's "do" in that? And if not, then what's "teisybė"?

    For professional programmers non-localized error messages is not a problem and it even is better (for Googling and stuff). And for non-professional ones... I don't really think error message localization is a solution here, but if it is and if variable names with international characters make sense, then keywords should follow that route too (although I would rather not use that kind of compiler:).

  • Hello. I'am french man and my english sucks. Nevertheless I completly agree with Marius. I work with C# for 3 years and i had never used the exception ID. I ever don't know if it exists. I use the google technique because it is the fastest one. I am used to english error message as I am used to english keyword, but when a tester with french configuration tells me: "I have found a 'La reference de l'objet n'existe pas'", I do not understand quicly taht he speaks of a NullReferenceException.

    Every programmers can understand basic english!

    (Ok, for an any user world application error message must be localised. But a compilator is not a any-user application.)

  • Count another vote against localizing error messages. I am too coming from a Russian-speaking country, and agree that pretty much everybody knows enough English to understand error messages. Also, if you translate the error message, and that message refers to the C# standard (which is only available in English), it's not going to be helpful. You are wrong on the "I would say that if the forum is international then one should expect discussions in any language" point as well. Trying to discuss C# topic in an international forum using Russian is simply counterproductive. Only Russian folks will be able to help you out. But if you use English (however bad) instead -- the pool of people who will understand you widens by couple orders of magnitude. For better or worse, English is the de-facto standard in the software development community.

  • I came here to see the terrible purple color of your blog as reported by Coding Horror <http://www.codinghorror.com>.

    However, I actually like the color because it is very subtle and helps separates the blog entry from the rest of the page. Maybe Jeff needs to switch to decaf.

    Now that I'm here, all of your suggestions are good, but I'd like to add one more: GIVE ME AN ERROR CODE!

    Once you have an error number, you can look up a detailed explanation of what went wrong, all the possible scenarios of what might have happened,  and why. Maybe even give a suggestion or two of what I might need to do to fix the problem. An error message can't be longer than a line, but a reference book (or web page) can prattle on for pages.

    This is good: <i>Method 'Foo' has a parameter array parameter which is not the last parameter in the formal parameter list</i>

    But, this would be nicer: <i>Error: AB324: Method 'Foo' has a parameter array parameter which is not the last parameter in the formal parameter list</i>

    As for humor in error messages, it's good the first few times around, but it gets a little irritating the 21st time you see it.

  • "Method 'Foo' has a parameter array parameter which is not the last parameter in the formal parameter list."

    Really? You really think that's an improvement? Because I parse that as "Method 'Foo' has a blah array blah which is not the blah in the blah list." Way too many blahs. Prefer:

    "In method 'Foo', the keyword 'params' appears somewhere other than at the end of the parameter list."

    or "MyClass.Foo, line %d: 'params' must appear only at the end of a parameter list" (which is darn close to what Microsoft started out with, although I agree with your criticism of Microsoft's original wording).

  • For the same reason that

    "Static member 'Baz' cannot be marked as override, virtual or abstract."

    is a bad error message,

    "Static class 'Foo' cannot be used as a constraint."

    is also bad because we do not know if the user wants to declare a static class, or use it as a constraint.

    "Class 'Foo' cannot be both marked as static and used as a constraint."

    would be much better.

  • I agree in not localizing error messages. One example is in NullReferenceException. As a programmer you should/must learn in terms like Null, reference, object, exception. See is as names of something, not as explanations. What to do when you see a NullReferenceException while debugging, and not knowing this name/term? I think all terms should always be in english everywhere. It is part of the programming language/framework.

  • A little bit late, but I would cast another vote for not localizing error messages, or at least giving us the choice to select English messages if we want to.  I think people does not understand how many trouble they can cause us by trying to help us!

    Besides what has been said here (impossibility to Google, and errors or different understandings on the translation) there is other place where it is annoying. I support a third-party library with customers all over the world, and very ofter I get some screenshot or a description of an error message on other language, that is not English and it is not mine either. It is very hard to help on those cases, when you can not even read what it is on the screen. On a world more and more globalizated, we need some "lingua franca", like it or not.

    A side note, but I remember old times, trying to program in Excel 6 VBA. Someone had had the idea to translate not only the messages but the language itself.  I think they actually though they were helping us, but can you imagine having to code things like:

    si x>1 entonces z=1 siotro z=2  

    (instead of if x>1 then z= 1 else z=2)

    Those things are hard-coded in your head. Even when English is not my native language (as you could guess by my writing) I find it very hard to switch to a Spanish programming language, and I found myself all the time "translating" the commands from English to Spanish. I want to make a while... so it should be "mientras"... ok. And not always it was even a direct translation. I would translate "else" to "sino", but the translator decided it was "siotro", and so on.

    And note that on the example above I only used "x" and "z", not "y".  This is because in Spanish, "y" means AND, and so it is a reserved word. Of course, due to an error message that was not "Accurate", I lost one complete day trying to find out the error on a line like this:

    y = h * sinAlpha

    I don't remember the error message right now, but it did not even remotely hint at "y" being a reserved word.

    Other pet peeve with localization are keyboard shortcuts. I know "Open" in Spanish is "Abrir", but did we really need to change Ctrl-O to Ctrl-A?  (And of course, since Ctrl-A is Select all in English, move Ctrl-A to Ctr-E and so on) Even when on a perfect world we would have all our apps translated, in real life we have some translated and some not. So, in a common Spanish machine, sometimes you have to press ctrl-a and sometimes ctrl-o, depending on the application.

    Sorry if this went a little off-topic, but I just wanted to give a feeling of why we might actually not want this  kind of "help" to keep us "productive" :)

  • What would be very nice if someone rewrote the System Error Messages. Most are incomprehesible and for some the .h files are the only public documents containing the terminology.

  • And the oscar goes to... "Catastrophic Failure"

Page 2 of 3 (35 items) 123