Fabulous Adventures In Coding : Rants

Queueing Theory In Action, plus, frogs

Eric Lippert — Thu, 20 Aug 2009 16:32:00 GMT

Well that was a lovely vacation. It got off to a poor start but then it improved dramatically.

Suppose you've got an "entrance" that is producing some largish number of "customers" on some schedule. You've got a bunch of "servers" who are handling the customer requests. Once a customer request is satisfied, the customer leaves through an "exit". What happens when there are more customers arriving in quick succession than there are available servers to serve them? This is the domain of analytic queueing theory; this theory is germane to a great many human and technical problems, from obtaining a double cheeseburger, onion rings and a large... orange... drink... to routing telephone calls over satellite networks.

(Some of you might be wondering what on earth the first two paragraphs have to do with each other. It'll come together eventually, I promise.)

An interesting example of two different queue servicing algorithms is exemplified by two popular fast food restaurant chains. At restaurant "M", if there are, say, four cashiers then there are four queues. Customers arrive, choose a queue and wait. At restaurant "W", there is one long serpentine queue; when a cashier becomes available, the person at the front of the line goes to that cashier.

The principle downside of the W system is that the single queue looks like it will take much longer than four short queues in the M system, which can be daunting. But by almost every relevant objective metric, by almost every relevant social factor, and in almost every common real-world business scenario the W system is preferable:

The W system requires no customer to make a choice based on incomplete information; the M system basically presents the customer with a roll of the dice. Which queue is fastest? It depends not just on the competence of the cashier, but on whether the transactions pending in a given queue are simple or complex.
Suppose you are in the M system and two queues -- yours, and the one beside you -- are being serviced at approximately the same rate on average. It is entirely possible and indeed, perhaps common, that even though both queues are in the long run moving at the same rate, that due to sudden starts and stops in both queues you will perceive that you are spending more time "being passed" by the people next to you than "passing" them. It is quite possible for people in both queues to have this perception simultaneously! Everyone feels like they've chosen the wrong queue, even if there is no "wrong queue" on average. In the W system, there is only one queue, so it is automatically the right one.
The W system is fair; the customer who has waited the longest is always served next. The M system is not fair; customers who happen to be in a "fast" queue can get served before customers in a "slow" queue where they are waiting behind a complex ongoing transaction (or worse, behind a customer who has reached the front of the queue before deciding what to order.)
The W system has in theory higher possible throughput; the only time that a customer with a fast transaction pending at the front of the queue has to wait a long time is in the unlikely situation that every cashier happens to be busy with a complex transaction. If any cashier is at present serving fast transactions, then they're clearing the front of the queue quickly. In the M system, many fast transactions can get delayed by a single slow one. This drives average throughput way down.
The W system is far more flexible. New cashiers can be added dynamically when the queue gets too long and removed to perform less pressing tasks when it shortens. In the M system, when a new cashier comes online there can be a disorganized rush to form a new queue; customers are again asked to make a decision about whether to try the new queue or stay in the old one, and this produces new opportunities for perceived unfairness. But worse, when a cashier is removed in the M system, what happens to their queue?

It is the importance of this last question that was driven home to me on day one of my vacation. Let's call the airline "D". The D Air Lines baggage check in Seattle-Tacoma International Airport operates on the "M" queueing model. You check yourself in at one of the kiosks, print out your boarding passes, pay your fifteen dollars to check a bag, and then choose one of ten or so queues.

Now, D has at least in theory eliminated one problem with the M model; the last thing the checkin system tells you is which queue to stand in. I do not know how that system decides which queue is best, or whether any customer actually reads the message. It's a subtle thing; I personally was not expecting the system to give me this information, so I could see how someone could miss it entirely and just pick any old queue.

But anyways, we're standing in our assigned queue and its moving along. We don't particularly care how long its taking because our flight has been delayed, allegedly by one hour, due to an "unexpected maintenance issue". So we have plenty of time. (As it turns out, we in reality have over three hours of extra time. Which is fine. Airline mechanics reading this: please take all the time you need to ensure that the wings stay on.) And as we're waiting, I point out something to my wife: the conveyor belt that the luggage is supposed to be going on is not moving. Or rather, it's moving by about one bag width per minute -- jerkily starting up and then halting a second later. Luggage is of course arriving at the front of the queues far, far faster than that, so quite an accumulation of luggage is forming. Most of the "servers" aren't having to walk around, but a few people are walking behind the counter, and it's quite comical to see them trying to manoevre around the increasingly tall stacks of luggage piled beside the now-saturated conveyor.

Leah informs me that her friend C used to be a baggage handler for D Air Lines, but was recently laid off. "Oh, is the recession and slowdown in air travel making for fewer working hours available?" I ask. No, apparently C claims that there was plenty of work for them to do, but the precarious financial state of the airlines has led them to lay off staff and overwork the remaining handlers.

So anyway, we get to the front of the queue and I look up, trying to make eye contact with the D Air Lines representative directly in front of me. She is fixedly looking at the floor and says loudly in the vicinity of the representative serving the queue beside her "I have to leave now." Which she does, continuing to look fixedly at the floor as she walks away. The other representatives, all busy serving other customers, do not react to this news. They may not have heard it. Or, they might have interpreted it as mere information -- as it was stated -- rather than as a request for someone to deal with the now-abandoned queue.

I have plenty of time to wait. So I do. I continue to try to make eye contact with the representatives serving the queues on either side of me, but they are either looking at the person they're serving, or looking with dismay at the growing towers of baggage now thoroughly engulfing the unmoving conveyor belt. I wonder whether it is difficult to miss someone less than two metres away staring at you with an expectant smile for minutes on end. Perhaps D Air Lines trains its staff to be good at that, I muse.

The minutes continue to pass. The queue behind me continues to grow. I reflect upon how this is a failure not just of customer service, but of application of basic results in queueing theory that you'd think an airline would be good at. I realize that I have a blog article in here somewhere, which makes me happy. I realize that I am now thinking about work while I'm on vacation, which makes me vexed. So on balance, it's a wash.

About five minutes into this, the traveller behind me politely taps me on the shoulder and asks "You're going to Michigan too, right? Am I in the right line?"

I turn to half face her, and half face the D Air Lines employee who has been so successfully ignoring me and the couple dozen people behind me. "Ma'am," I say, "look at the larger picture. You are standing in a queue that is being served by no one to put your bags on a conveyor belt that is not moving. Most of the baggage handlers who would be taking your bags off the belt have been fired, and even if there were baggage handlers, the aircraft cannot fly, and probably is not even at this airport. Clearly you and I have not chosen the wrong queue; we have chosen the wrong airline."

Amazingly enough, that aforementioned representative immediately starts servicing our queue.

Though no indication whatsoever that there had been any sort of problem in the first place was forthcoming, to his credit he was polite, seemed reasonably competent at taking my bag, and added my bag to the tower. As we walked away I looked back and saw the fellow now servicing both queues; those queues were now each running at half speed, which I suppose is better than nothing, though I imagine that the people who had chosen either queue were less than thrilled. The airplane did eventually arrive and we did eventually retrieve the bag, so it all worked out in the end.

Now you know why most airlines use the "serpentine" W model rather than the M model. It prevents some of these sorts of problems from happening in the first place.

Things improved considerably after that. A sampling of stuff I saw on my vacation:

Jupiter in opposition
Ganymede
the Perseid meteor shower
trout
frogs
toads
tadpoles
turtles
turkey vultures
swallows
hummingbirds
mergansers
loons
great blue herons
some kind of predatory bird that was against the sun so I couldn't see it clearly but I suspect it was an osprey
chipmunks
bunny rabbits
silver birch
snapdragons
bright green dragonflies
sheep
roosters
fossilized clams
suspiciously damaged wooden kayak paddles: that is, evidence of beaver-shark activity. But how did they get into my kayak storage over the winter? Are beaver-sharks amphibious?
my family
old friends

The flight home -- where the baggage check was based on the W model -- was uneventful.

And so, back to more fabulous adventures in coding. I hope you enjoyed the pre-canned posted I prepared before my vacation.

Coming up next: one more addendum about iterator blocks.

Making it easier

Eric Lippert — Mon, 15 Jun 2009 19:56:00 GMT

I read an article in a technology column on MSNBC a while back, the upshot of which was “I have umpteen-dozen passwords I’ve got to have memorized these days; I thought technology was supposed to make my life easier!”

Really?

First of all, let’s leave aside the obvious fact that our column writer has a technology-driven standard of living which includes affordable access to varieties of food, drink, shelter, clothing, medicine, travel, and entertainment all of which were beyond the wildest dreams of the crowned heads of Europe less than two centuries ago, much less the peasants. Technology does seem to have made lives a lot easier all around. But I rather want to address the actual claim: is the purpose of technology to make your life easier?

I don’t think it is. I think the purpose of technology is to make specific tasks easier.

Let’s take a task like “communicate with a person in Australia”. Suppose you live in, say, London. In the early 1800’s, the best way to get a message to a colleague in Australia was to write the message out with your goose quill pen, making multiple copies. Wrap each up in oilcloth, the most waterproof substance of the time. Find several convenient Royal Navy ships or merchant ships going to Australia by different routes and at different times; Napoleon might sink several of them but one of them will probably get through. Entrust your waterproofed messages to the captains of those ships, and then wait the several months it takes to get the message around either the Cape or the Horn and the reply back to you.

I take this opportunity to point out that this already requires an extremely high level of technology. Safely navigating a vessel from England to Australia is not an easy task under the best of circumstances. But obviously we can do better now; the 1800’s solution strikes me as somewhat more arduous and time-consuming as a whole than having to remember your email password.

The task of communicating with Australia is now much easier, thanks to improved technology. And because we have increased the scope of what is feasible, we can take advantage of new capabilities; in the 1800’s, real-time arbitraging of price differences between Australian and European commodity market derivatives was not anyone’s job description, but I’m sure that there are people pursuing this highly complex task today. The technology enables us to find new ways to complicate our lives, ways that were never possible before because we were too busy spending effort on solving other problems.

I think about this kind of thing a lot in the context of programming language design.

When we add a new feature to the language, we almost always make a specific task easier. But in doing so, we also almost always make the language as a whole more complex and therefore harder to learn. We make it more likely that there will be a communications divide between those who know how the new feature works and those who do not, making it harder for everyone to read and understand each other’s code.

This is one of the major reasons why new features start with “-100 points”, as I’ve often said. Coming up with features that make specific tasks easier is, well, easy. But that’s not enough; the feature has to make things so much better that it justifies the additional complexity added to the language.

******

Wikimedia Commons photo by Ted “Rufus” Ross.

Bug Psychology

Eric Lippert — Mon, 01 Jun 2009 16:48:00 GMT

Fixing bugs is hard.

For the purposes of this posting, I’m talking about those really “crisp” bugs -- those flaws which are entirely due to a failure on the developer’s part to correctly implement some mechanistic calculation or ensure some postcondition is met. I’m not talking about oops, we just found out that the product name sounds like a rude word in Urdu, or the specification wasn’t quite right so we changed it or the code wasn’t adequately robust in the face of a buggy caller. I mean those bugs where you were asked to compute some value and you just plain get the result wrong for some valid inputs.

Let me give you an example.

The first bug I ever fixed at Microsoft as a full-time employee was one of those. To understand the context of the bug, start by reading this post from the early days of FAIC, and then come back.

Welcome back, I hope you enjoyed that little trip down memory lane as much as I did.

Now that you understand how a VT_DATE is stored, that explains this bizarre behaviour in VBScript:

print DateDiff("h", #12/31/1899 18:00#, #12/30/1899 6:00#) / 24
print DateDiff("h", #12/31/1899 18:00#, #12/29/1899 6:00#) / 24

This prints –1.5 and –2.5, as you’d expect. There’s a day and a half between 6 AM December 30th and 6 PM December 31st, and two and a half days between the other two dates. This is perfectly understandable. But if you just subtract the dates:

print #12/31/1899 18:00# - #12/30/1899 6:00#
print #12/31/1899 18:00# - #12/29/1899 6:00#

You get 1.5 and 3, not 1.5 and 2.5. Because of the bizarre date format that VT_DATE chooses, when you convert dates to numbers, you cannot safely subtract them if they straddle the magic zero date. That’s why you need the helpful “DateDiff”, “DateAdd” and so on, methods.

The bug I was assigned was that testing had discovered a particular pair of dates which DateDiff was not subtracting correctly. I took a look at the source code for one of the helper methods that DateDiff used to do one of the computations it needed along the way. To my fresh-out-of-college eyes it looked something like this:

if (frob(x) > 0 && blarg(y)) return x – y;
else if (frob(x) < blarg(y) && blah_blah(x) > 0 || blah_de_blah_blah_blah(x,y)) return frob(x) – x + y + 1;
else if…

There were seven such cases.

My urge was to dive right in and add an eighth special case that fixed the bug. But my ability to get it right in the face of all this complexity concerned me. It seemed like this was an awfully complicated function already for what it was trying to do.

I researched the history of the code a bit and discovered that in fact variations on this bug had been entered… seven times. Each special case in the code corresponded to a particular bug that had been “fixed”, a term I use guardedly in this case. A great many of those “fixes” had actually introduced new bugs, regressing existing correct behaviour, which then in turn were “fixed” by adding special cases on top of the broken special cases that had been added to “fix” previous bugs.

I decided that this coding horror would end here. I deleted all the code (all seven lines of it! I was bold!) and started over.

Deep breath.

Spec the code requirements first. Then design the code to meet the spec. Then write the code to the design.

Spec:

* Input: two doubles representing dates in VT_DATE format.
* VT_DATE format: signed integer portion of double is number of days since 12/30/1899, unsigned fractional part is portion of day gone by.
* For example: –1.75 = 12/29/1899, 6 PM.
* Output: double containing number of days, possibly fractional, between two dates. Differences due to daylight savings time, and so on, to be ignored.

Design strategy:

* Problem: Some doubles cannot simply be subtracted because negative dates are not absolute offsets from epoch time
* Therefore, convert all dates to a more sensible date format which can be simply subtracted.

Code:

double DateDiffHelper(double vtdate1, double vtdate2)
{
return SensibleDate(vtdate2) – SensibleDate(vtdate1);
}
double SensibleDate(double vtdate)
{
// negative dates like –2.75 mean “go back two days, then forward .75 days”:
// Transform that into –1.25, meaning “go back 1.25 days”.
return DatePart(vtdate) + TimePart(vtdate);
}

I already had helper methods DatePart and TimePart, so I was done. The new code was shorter, far more readable, generated smaller, faster machine code and most important, was clearly correct. No special cases; no bugs.

It’s not that my coworkers were dummies. Far from it. These were smart people. But computer geek psychology is such that it is very easy to narrow-focus on the immediately wrong thing, and try to tweak it until it does the right thing.

When faced with these sorts of “crisp” bugs, I try to restrain myself from diving right in. Rather, I try to psychoanalyze the person – who is, of course, usually my past self – who caused the bug. I ask myself “how was the person who wrote the buggy code fooled into thinking it was correct?” Did they not have a clear specification of what the method was supposed to do? Was it misleading? Did they have a clear plan for how to proceed? If so, where did it go wrong?

If there never was either a spec or a plan, then for all you know the whole thing might only be working by sheer accident. There could be any number of design flaws in the thing that just haven’t come to light yet. Editing such a beast means adding unknown to unknown. which seldom leads to good results. Sometimes coming up with a new spec, a new plan and scrapping an existing bug farm is the best way to proceed.

For many years after that, I would ask how to implement DateDiffHelper as my technical question for fresh-out-of-college candidates that I was interviewing for the scripting dev team. I reasoned that if that was the sort of problem I was given on my first day in the office, then maybe that would be a reasonable question to ask a candidate.

When you ask the same question over and over again, you really get to see the massive difference in aptitude between candidates. I had some candidates who just picked up a marker, wrote a solution straight out on the board, wrote down the test cases they’d use to verify it, mentally ran a few of the tests in their head, and then we’d have another half hour to chat about the weather. And I had some candidates who tried earnestly to write the version using special cases, despite my specifically telling them “you might consider transforming this bad format into something more pleasant to work with”, after they got stuck on the third special case. I’d point out a bug and immediately they’d write down code for another special case, rather than stopping to think about the fact that they’d just written buggy code three times already and told me it was correct three times.

Arrays considered somewhat harmful

Eric Lippert — Mon, 22 Sep 2008 21:04:00 GMT

I got a moral question from an author of programming language textbooks the other day requesting my opinions on whether or not beginner programmers should be taught how to use arrays.

Rather than actually answer that question, I gave him a long list of my opinions about arrays, how I use arrays, how we expect arrays to be used in the future, and so on. This gets a bit long, but like Pascal, I didn't have time to make it shorter.

Let me start by saying when you definitely should not use arrays, and then wax more philosophical about the future of modern programming and the role of the array in the coming world.

You probably should not return an array as the value of a public method or property, particularly when the information content of the array is logically immutable. Let me give you an example of where we got that horridly wrong in a very visible way in the framework. If you take a look at the documentation for System.Type, you'll find that just looking at the method descriptions gives one a sense of existential dread. One sees a whole lot of sentences like "Returns an array of Type objects that represent the constraints on the current generic type parameter." Almost every method on System.Type returns an array it seems.

Now think about how that must be implemented. When you call, say, GetConstructors() on typeof(string), the implementation cannot possibly do this, as sensible as it seems.

public class Type {
   private ConstructorInfo[] ctorInfos;
   public ConstructorInfo[] GetConstructors()
   {
     if (ctorInfos == null) ctorInfos = GoGetConstructorInfosFromMetadata();
     return ctorInfos;
   }

Why? Because now the caller can take that array and replace the contents of it with whatever they please. Returning an array means that you have to make a fresh copy of the array every time you return it. You get called a hundred times, you’d better make a hundred array instances, no matter how large they are. It’s a performance nightmare – particularly if, like me, you are considering using reflection to build a compiler. Do you have any idea how many times a second I try to get type information out of reflection? Not nearly as many times as I could; every time I do it’s another freakin’ array allocation!

The frameworks designers were not foolish people; unfortunately, we did not have generic types in .NET 1.0. clearly the sensible thing now for GetConstructors() to return is IList<ConstructorInfo>. You can build yourself a nice read-only collection object once, and then just pass out references to it as much as you want.

What is the root cause of this malaise? It is simple to state: The caller is requesting values. The callee fulfills the request by handing back variables.

An array is a collection of variables. The caller doesn’t want variables, but it’ll take them if that’s the only way to get the values. But in this case, as in most cases, neither the callee nor the caller wants those variables to ever vary. Why on earth is the callee passing back variables then? Variables vary. Therefore, a fresh, different variable must be passed back every time, so that if it does vary, nothing bad happens to anyone else who has requested the same values.

If you are writing such an API, wrap the array in a ReadOnlyCollection<T> and return an IEnumerable<T> or an IList<T> or something, but not an array. (And of course, do not simply cast the array to IEnumerable<T> and think you’re done! That is still passing out variables; the caller can simply cast back to array! Only pass out an array if it is wrapped up by a read-only object.)

That’s the situation at present. What are the implications of array characteristics for the future of programming and programming languages?

Parallelism Problems

The physics aspects of Moore’s so-called “Law” are failing, as they eventually must. Clock speeds have stopped increasing, transistor density has stopped increasing. The laws of thermodynamics and the Uncertainty Principle are seeing to that. But manufacturing costs per chip are still falling, which means that our only hope of Moore’s "Law" continuing to hold over the coming decades is to cram more and more processors into each box.

We’re going to need programming languages that allow mere mortals to write code that is parallelizable to multiple cores.

Side-effecting change is the enemy of parallelization. Parallelizing in a world with observable side effects means locks, and locks means choosing between implementing lock ordering and dealing with random crashes or deadlocks. Lock ordering requires global knowledge of the program. Programs are becoming increasingly complex, to the point where one person cannot reasonably and confidently have global knowledge. Indeed, we prefer programming languages to have the property that programs in them can be understood by understanding one part at a time, not having to swallow the whole thing in one gulp.

Therefore we tools providers need to create ways for people to program effectively without causing observable side effects.

Of all the sort of “basic” types, arrays most strongly work against this goal. An array’s whole purpose is to be a mass of mutable state. Mutable state is hard for both humans and compilers to reason about. It will be hard for us to write compilers in the future that generate performant multi-core programs if developers use a lot of arrays.

Now, one might reasonably point out that List<T> is a mass of mutable state too. But at least one could create a threadsafe list class, or an immutable list class, or a list class that has transactional integrity, or uses some form of isolation or whatever. We have an extensibility model for lists because lists are classes. We have no ability to make an “immutable array”. Arrays are what they are and they’re never going to change.

Conceptual Problems

We want C# to be a language in which one can draw a line between code that implements a mechanism and code that implements a policy.

The “C” programming language is all about mechanisms. It lays bare almost exactly what the processor is actually doing, providing only the thinnest abstraction over the memory model. And though we want you to be able to write programs like that in C#, most of the time people should be writing code in the “policy” realm. That is, code that emphasizes what the code is supposed to do, not how it does it.

Coding which is more declarative than imperative, coding which avoids side effects, coding which emphasizes algorithms and purposes over mechanisms, that kind of coding is the future in a world of parallelism. (And you’ll note that LINQ is designed to be declarative, strongly abstract away from mechanisms, and be free of side effects.)

Arrays work against all of these factors. Arrays demand imperative code, arrays are all about side effects, arrays make you write code which emphasizes how the code works, not what the code is doing or why it is doing it. Arrays make optimizing for things like “swapping two values” easy, but destroy the larger ability to optimize for parallelism.

Practical Problems

And finally, given that arrays are mutable by design, the way an array restricts that mutability is deeply weird. All the contents of the collection are mutable, but the size is fixed. What is up with that? Does that solve a problem anyone actually has?

For this reason alone I do almost no programming with arrays anymore. Arrays simply do not model any problem that I have at all well – I rarely need a collection which has the rather contradictory properties of being completely mutable, and at the same time, fixed in size. If I want to mutate a collection it is almost always to add something to it or remove something from it, not to change what value an index maps to.

We have a class or interface for everything I need. If I need a sequence I’ll use IEnumerable<T>, if I need a mapping from contiguous numbers to data I’ll use a List<T>, if I need a mapping across arbitrary data I’ll use a Dictionary<K,V>, if I need a set I’ll use a HashSet<T>. I simply don’t need arrays for anything, so I almost never use them. They don’t solve a problem I have better than the other tools at my disposal.

Pedagogic Problems

It is important that beginning programmers understand arrays; it is an important and widely used concept. But it is also important to me that they understand the weaknesses and shortcomings of arrays. In almost every case, there is a better tool to use than an array.

The difficulty is, pedagogically, that it is hard to discuss the merits of those tools without already having down concepts like classes, interfaces, generics, asymptotic performance, query expressions, and so on. It’s a hard problem for the writer and for the teacher. Fortunately, for me, it's not a problem that I personally have to solve.

High maintenance

Eric Lippert — Mon, 08 Sep 2008 19:36:00 GMT

The other day I went to buy some snack from the snack machine in the kitchen. The snack I wanted was in slot B-10, so I put in my coins, press B - one - zero, hey wait a minute there's no zero button! And why is it serving me up the snack on the left end of the machine instead of the right? Aha, there is a button marked "10", which is the one I was supposed to press. Instead I got snack B1. How irksome!

And then I laughed at my plight, because of course Steve Maguire told the same story about Microsoft vending machines in Writing Solid Code fifteen years ago. Maguire went on to make an analogy between bad candy machine interfaces and bad software interfaces; a "candy machine interface" is one which leads the caller to make a plausible but wrong choice.

Coincidentally, I was asked to review a fragment of code recently that got me thinking again about candy machine interfaces. This isn't exactly the kind of bad interface that Maguire was talking about -- but I'm getting ahead of myself. Let's take a look at the code:

public static class StreamReaderExtensions
{
    public static IEnumerable<string> Lines(this StreamReader reader)
    {
        if (reader== null)
            throw new ArgumentNullException("reader");
        reader.BaseStream.Seek(0, SeekOrigin.Begin);
        string line;
        while ((line = reader.ReadLine()) != null)
            yield return line;
    }
}

The idea of this code is awesome. I use this technique all the time in my own programs when I need to manipulate a file. Being able to treat a text file as a sequence of lines is very handy. However, the execution of it has some problems.

The first flaw is the bug I discussed in my earlier post on psychic debugging. Namely, the null check is not performed when the method is called, but rather, when the returned enumerator is moved for the first time. That means that the exception isn't thrown until possibly far, far away from the actual site of the error, which is potentially confusing.

The second flaw is that the "while" loop condition is a bit hard to read because it tries to do so much in one line. Shorter code is not magically faster than longer code that does the same thing; write the code so that it is maximally readable.

But there's a deeper flaw here. To get at it, first let me state a crucial fact about the relationship between a stream reader and a stream:

A StreamReader “owns” its underlying stream. That is, once a stream has been handed to a stream reader by a caller, the caller should not muck with the stream ever again. The stream reader will be seeking around in the stream; mucking around with the stream could interfere with the stream reader, and the stream reader will interfere with anyone trying to use the stream. The stream reader emphasizes its ownership by taking responsibility for disposing the stream when the stream reader is itself disposed.

Now, “Lines” defines an object, and what does that object do right off the bat? It mucks around with the underlying stream. This is big red flag #1. This smells terrible. No one should be messing with that stream but the reader.

Furthermore, think about it from the caller’s point of view. Maybe the caller knows that there are a bunch of bytes that it wants to skip, so it deliberately hands a StreamReader to Lines() which has been positioned somewhere past the beginning of the file. But Lines() thinks that it knows best and ignores the information that the caller has given it. This is big red flag #2.

(The reason why the original code was seeking back was because the same reader was being "recycled" many times to read the same file over and over again, which is yet another red flag. Readers are cheap; you can have multiple readers on one file, there's no need to hoard them and reuse them.)

The third big red flag for me here is the ownership issue. When you hand a stream to a reader, the reader takes care of everything for you from then on – that’s the “contract” between the reader and the caller. “Lines()” does not have that contract. Lines()’s contract says “attention caller: I am taking ownership of this reader. I am going to muck with its underlying stream. You must never use this reader for anything ever again – except that I’m not going to dispose it for you. If you want the reader and its stream closed then you are going to have to keep a reference to it around until you can prove that I am done with it. And if you get it wrong, either I crash or you get a resource leak. So get it right.”

This is a terrible contract to impose upon a caller, and of course, the imposition here is in no way represented by the signature of the method. You just have to know that “Lines()” owns the reader for the purposes of all its functionality, but not for its cleanup, which is deeply weird.

In short, the caller needs to know all the details of what is happening in the method in order to use it correctly; this is a violation of the whole purpose of creating methods. Methods are supposed to abstract away an operation so that you do not have to know what they are doing.

A fourth red flag is that the task performed by Lines() does not have a clear meaning in terms of the logic of the program. In looking at every caller of this method it became clear that the desired semantics were “give me the lines of this text file one at a time”. But we haven’t written a method that does that. In order to get the lines of a text file from Lines() the caller is required to do all kinds of work to make Lines() happy – open the stream, create a reader, call Lines() but keep the reader around, close the reader after we know that the iterator is done.

This code makes the caller do a whole bunch of things -- things that have to be done in exactly the right order but potentially distributed out over long stretches of time and disparate code locations. This method is very "high-maintenance"! Everything has to be just right all the time in order for it to be happy and get along with others; anything out of place and things will start going wrong.

Finally, we have an example of premature generality here -- if the intention is to read a text file, then write a method that reads a text file. If you don't need the power of reading lines from an arbitrary stream, maybe don't implement that. Cut down on your testing burden.

Some relatively simple changes fix it up:

public static class FileUtilities
{
    public static IEnumerable<string> Lines(string filename)
    {
        if (filename == null)
            throw new ArgumentNullException("filename");
        return LinesCore(filename);
    }
    private static IEnumerable<string> LinesCore(string filename)
    {
        Debug.Assert(filename != null);
        using(var reader = new StreamReader(filename))
        {
            while (true)
           {
                string line = reader.ReadLine();
              if (line == null)
                  yield break;
               yield return line;
          }
        }
    }
}

And now everything is crystal clear. The purpose of this thing is to read the lines out of a text file. The caller gives it the name of a file, it gives you the lines, and we’re done. The caller does not have to worry about anything else – the iterator takes care of opening the file, cleaning up when its done, and so on. There’s no messing around with streams at all; we have now provided an abstraction over the file system to the caller.

Whenever you write a method think about the contract of that method. What burdens are you imposing upon the caller? Are they reasonable burdens? The purpose of a method should be to make the caller’s life easier; the original version of Lines() makes life harder on the caller. The new version makes life easier. Don't write high-maintenance methods.

Tasty Beverages

Eric Lippert — Wed, 20 Aug 2008 00:09:00 GMT

“Diet Dr. Pepper tastes more like regular Dr. Pepper.”

That was a previous advertising slogan for Diet Dr. Pepper, my personal favourite source of both caffeine and phenylalanine; I’m drinking it right now as I write this.

The present slogan is the brain-achingly oxymoronic “Diet Dr. Pepper: There’s Nothing Diet About It” – really? Seems like one ought to change the name then, if the name of one’s product is so misleading as to require its complete and utter disavowal in the slogan.

But that’s not what I want to talk about today. I actually want to talk about predicates.

The word “predicate” is one of those slippery words that has multiple technical meanings depending on the domain, all related but subtly different enough that one really ought to carefully call out how one is using the term.

In C# programming a predicate is a function which takes a bunch of arguments and returns a Boolean. customer=>customer.Age < 21, for example.
In mathematical logic, a predicate is a function (or relation) which indicates the membership of a set. For example, “all integers x such that x is both even and prime” is a predicate for the set containing only the number 2.
In English grammar, a predicate is that part of a statement which claims to represent a fact about the subject. For example, in “Thucydides fought in the Peloponnesian War”, “Thucydides” is the subject and the rest is the predicate. This predicate happens to be true for the subject “Thucydides”, and false for a different subject, say, “Julius Caesar”.

What all of these things have in common is that a predicate is something into which you can "substitute" a value to obtain a result of either truth or falsity.

What on earth does this have to do with Diet Dr. Pepper tasting more like regular Dr. Pepper?

Is “tastes more like regular Dr. Pepper” actually a predicate? If it is then when it is applied to a subject it must produce a statement which can be classified as true or false. Let’s leave the subjective nature of taste aside for a moment; that’s not the fundamental logical problem here.

Rather, consider this utterance: “Diet Dr. Pepper tastes more like” Is “tastes more like” a predicate? Of course not. That utterance doesn’t make any sense. Tastes more like… what? This utterance cannot be classified as true or false for any subject, so the latter part of it cannot actually be a predicate.

But the same thing goes for “tastes more like regular Dr. Pepper”! Tastes more like regular Dr. Pepper… than what?

In order to actually be a predicate it needs more objects. For example, “Diet Dr. Pepper tastes more like regular Dr. Pepper than a pint of Guinness tastes like a mango lassi” is a statement which actually has a truth value. Perhaps a subjective and arguable truth value, but at least this sentence has the form of a statement with a subject and a real predicate now. The original slogan’s “predicate” isn’t really a predicate at all; one might think of it as a pseudopredicate.

Advertisers love pseudopredicates. Once you realize that they exist you see them all the time. Advertisers love them because they make no testable claim which could be shown to be false in a court of law. Rather, they rely on either the irrational belief that “more” anything means “better”, or upon your brain’s ability to fill in the rest of the objects which they intend you to infer.

In this particular case, I imagine that the crafters of this slogan intended your brain to fill in “Diet Dr. Pepper tastes more like regular Dr. Pepper than our previous formulation of Diet Dr. Pepper tasted like regular Dr. Pepper” – that is, they want to make the claim that the product has improved without making the admission that the previous formulation was less than delicious.

Or, perhaps they want you to fill in “Diet Dr. Pepper tastes more like regular Dr. Pepper than Diet Coke tastes like regular Coke” – that is, they want to assert that their product is superior to a competing product. This assertion is in my personal opinion true, but the Coca Cola company could potentially take issue with if stated baldly as an objective claim. By relying upon a pseudopredicate to make a slogan which actually is so malformed as to have no truth value at all, the copywriters duck these thorny issues. There are lots of ways that clever advertisers leverage our tendancies to "fill in the blanks" in order to sell products.

That’s not actually what I want to talk about today either. I actually want to talk about writing secure code.

What on earth does Diet Dr. Pepper tasting more like regular Dr. Pepper have to do with writing secure code?

The other day I got a question about the characteristics of a particular bit of source code obfuscation technology, which we shall call X; what the technology actually consists of and what the precise question was are irrelevant to this discussion.

I answered the question with a question; I asked why it was that the questioner wanted to use technology X. The answer was “To protect the source code”. Leave aside for the moment the fact that I could probably have deduced from the original query that the questioner was interested in protecting some resource. There’s a deeper problem here. In the utterance “technology X protects the source code”, is “protects the source code” a predicate, or a pseudopredicate?

It’s a pseudopredicate. There is an object missing. To make this a predicate, it needs to be something like “protects the source code from casual inspection and editing by snoopy people” – as it happens, this predicate was true for technology X. What I was rather worried about was that the questioner actually had in his head the predicate “protects the database administrator password that I’ve stuck into my source code from discovery and misuse by a determined and intelligent attacker”. That predicate happens to be utterly false for technology X. Because he was not actually stating a predicate that could be true or false of X I was unable to answer the guy's question about X.

I never, ever lock my car doors anymore. Why? I drive a soft-top convertible. One day I woke up to discover that someone had sliced open the top and unlocked the car. The two bucks in quarters I keep in the car for parking meters was a trivial loss compared to the hundreds my insurance company paid to get the top replaced and the hours of my time wasted in dealing with the situation. The locks are not a mitigation to that vulnerability at all! Locking my car doors makes it more prone to be damaged, not less.

I do, however, lock my house, to protect it against random people wandering in. However, the locks are hardly any mitigation to the vulnerability of the house to determined attack from a wily, hostile burglar. It would be foolish of me to say that “the locks protect my house” without mentioning the threat.

What I’m rambling on about here is this: the fitness of a particular security technology to mitigate a vulnerability can only be evaluated in the context of a stated threat against a stated resource. That’s because every security technology is designed to mitigate specific vulnerabilities to particular threats. When you’re evaluating the benefit of a particular security system, make sure that the predicates you are using to talk about the system are actually predicates, not pseudopredicates; state the threats.

Customer Service Is Not Rocket Science, Part Two

Eric Lippert — Mon, 23 Jun 2008 18:20:00 GMT

I find it irritating, but not surprising, when I get absurdly bad customer service from a business whose business model is based on volume and high margins. But I find it quite surprising, and indeed, greatly amusing, to get absurdly bad customer service from a business whose business model is entirely based on quality of service.

This was so amusing to me that I thought I would share it with you all. A little fun for the first work day of the summer.

I have difficulty keeping up with my lawn and gardening. I decided to go to an internet based company which recommends and rates home service contractors. We'll call them "Referral Service". I used "Referral Service" to find a lawn guy in my area, who we will refer to as "Lawn Guy".

After arranging an appointment time and a reasonable price, I got a phone call from "Referral Service". The guy on the phone was polite, enthusiastic, and asked me all kinds of questions about what other projects I might have going on around my house that they could help me with. I told the guy that yes, I have lots of projects -- I have a fence that needs rebuilding, I have some rooms that need painting, I have an unfinished renovation project in my basement. I told the guy that I would likely use their web site in the fall to arrange contractors for these various other projects.

Fine. Seems like everyone is pretty competent so far. But as it happened, Lawn Guy didn't work out, for reasons which will rapidly become apparent.

A few days later I got an email -- clearly a form letter -- from Referral Service, asking me to fill out a form to say how well Lawn Guy did. I thought about it for a minute and realized that my concern was sufficiently outside of the normal experience that I was not comfortable just filling out a number on a Likert Scale. I wanted to clearly express to Referral Service exactly what my problem was so that they could deal with it appropriately. Here are the emails, slightly reformatted to make them easier to read in this medium, and with names changed to protect the guilty.

*****

Good afternoon [Referral Service]

[Lawn Guy] made rude and deprecating comments about Poles to my housemate on his first day on the job. That he did not know that she was of Polish descent is hardly an excuse. I've fired him.

Eric Lippert

*****

Dear Eric,

Thank you for your email. Ratings & Reviews are perhaps the most important service we offer to our members. This area of our web site includes valuable word-of-mouth feedback from [Referral Service] customers. For all requests you place with [Referral Service], you'll be asked to submit Ratings & Reviews for the service professionals you are presented. When you do, you not only help other people make their choice, but you contribute to the overall quality of service we offer.

To submit a Rating and Review please visit our site [there then follows a list of the nine different things that you have to click on to submit a review]

Please let us know if there is anything else we can do, and we hope you tell others about our service. We look forward to helping you with all of your future home improvement needs.
Best Regards,

Andrew Throatwobbler
[Referral Service]

*****

Mr. Throatwobbler,

You asked that I let you know if there is anything else we could do. What you could have done is respond to my concern with something other than a canned form letter -- a letter, which I note, asks me to do work for your benefit. My job is not to provide content for your website; I don't care a bit about your website. I care about my lawn. If you want my business you really ought to be concentrating on that.

You said that you hoped I would tell others about your service. No, no, you don't. You ought to hope very much that I do not tell others. Responding to a report of a gross, offensive, personal insult with a form letter is the very depth of poor service.

Eric

*****

Dear Eric,

Thank you for your email and reply; we appreciate your taking the time to provide your feedback. Please know that we take these matters seriously and this is not the level of professionalism we have come to expect of our professionals. By providing your rating it additionally notates their account and should we ever notice a negative trend we do reserve the right to remove them from the service.

Thank you for your time and for using [Referral Service].

Regards,
Nicole Otterbach

*****

Ms. Otterbach,

I care not a bit about your policies for deciding who gets to be part of your service or not. I already told you that I don't care about that, and yet you go on about it.

You seem to have completely forgotten that you are in the business of recommending someone to mow my lawn! A lawn which is now going unmowed because I had to fire the racist you sent me after he insulted my housemate to her face.

The smart thing to do would have been to concentrate on the fact that you have failed utterly in what ought to be your core competency -- supplying me with a competent lawn care person. Instead, you've concentrated on sending me emails about how my doing work for free on your rating system benefits your business.

Eric Lippert

*****

Eric,

Thank you for your response. Both Ms. Otterbach and myself have expressed to you the improtance [sic] of submitting a rating on this service provider. If you submit a negative review about this service provider, then it will be followed up by our Ratings department and the necessary steps will take place. At this time, I can not locate your account in our network and am unable to see if you have this submitted. If not, please do so. Thank you.

Andrew Throatwobbler

*****

Mr. Throatwobbler,

Yes, you certainly have expressed that this is important to you, just as I have expressed that it is not important to me. Yet you continue to concentrate on it! An important principle of customer service, or, for that matter, life in general, is "when you find yourself in a hole, stop digging", yet you continue to dig. I find this interesting from a psychological point of view. (However, though interesting, psychological musings are certainly not getting my lawn mowed.)

As for the "necessary steps" -- my filling out a form is not a precondition of the necessary steps; I think you and I have a different view on what is "necessary" in such a situation. When I was in the service industry I was taught that the necessary steps for dealing with an upset, angry or disappointed customer were these three:

1) First, and most important, express regret that the problem occured, particularly with regard to the actions you took that precipitated the situation. As yet, no one in your organization has said that you in any way regret referring a boorish racist who insulted my housemate, a racist whom I had the unpleasant and upsetting task of firing. I would have thought that you would sincerely regret that your recommendation caused several people pain and distress, but apparently not.

2) Second, take this as an opportunity put a good light on the organization by distancing yourself from the situation. Ms. Otterbach did that, by saying that this is not the level of professionalism you would expect.

3) Third, take steps to make the customer's problem better. You've certainly not done that. You've concentrated solely on your issues -- your web site content, your rating system, your policies, your strange inability to find a customer in your own system. I really don't care whether you continue to refer Lawn Guy to others; that is not my problem. That's your problem. I've told you repeatedly that I don't care about that; I care about getting my lawn mowed. And yet you keep on telling me about your rating system.

One out of three is not good, particularly since none of these steps are difficult.

Studies have shown that customers who have a negative experience that is dealt with rapidly and respectfully have a higher opinion of an organization than customers who have never had any problems! Customers who are angry, upset or disappointed are actually a gold mine for you, because when you go the extra mile to solve their problems, they turn around and advertise your business to others for free.

Of course, the opposite is also true. Customers who report a problem and get back form letters, no apology, and repeated requests to do work that is to your benefit, not theirs, also tell others.

However, one good thing that has come out of this is that I now have an article for my blog.

Cheers,
Eric Lippert

*****

I haven't heard back from Mr. Throatwobbler yet. I'm in so much suspense, as I am sure are you all! Will he and Ms. Otterbach continue to attempt to tag team a former customer into submitting to their bureaucratic policy machinery? If they ever get back to me I'll post an update.

How to not get a question answered

Eric Lippert — Wed, 20 Feb 2008 18:28:00 GMT

Raymond has had lots of great posts over the years on how to not get a question answered. Some of the ways he points out that help ensure that a question goes unanswered are:

Use a difficult or meaningless subject line.
Ask a grammatically unclear question.
Forget to actually ask a question (this is a personal favourite of mine; back when I was working on scripting I would practically daily get an email consisting of a stack trace and a crash report, with no indication whatsoever of what the sender wanted me to do with it. Investigate it? Commiserate? Give career advice?)
Ask an unanswerable question over and over again.

I would add that phrasing the question in terms of the attempted solution rather than the underlying problem is a great way to have a question remain unanswered. But I already did that. Today's rant, rather, is:

A great way to not get your question answered is to treat the person that you're asking to share their time and knowledge like they're not a human being.

I understand that this is easy to do.

I understand that when someone is asking a technical question, it is almost always because they're deeply frustrated by their inability to make some important specific complicated thing work the way they'd expect.

I understand that software makers are a big source of frustration in a more general sense. We all use imperfect software every day; little frustrations mentally accumulate in a way that little victories do not.

I understand that programming languages and software in general is often counterintuitive, and that understanding the complex reasoning behind a counterintuitive result is often seemingly of little practical import compared to just getting the problem at hand solved.

I understand that it is very easy to use an email client as an impersonal and automatic information resource something like a search engine, where you just type in your query, you get your result back, and you move on with your day.

I understand that there is an inherent and pervasive bias in pure-text communication which makes statements intended to be good-humoured sound sophomoric, makes statements which were intended to be friendly sound smarmy, makes statements which were intended to be enthusiastic sound brash, makes statements intended to be helpful sound condescending, makes statements which were intended to be precise and accurate sound brusque and pedantic, makes statements which were intended to be positive sound neutral, and makes statements which were intended to be neutral seem downright hostile.

(Boy, do I ever understand that. For over four years I have deliberately tried to pitch the tone of this blog as good-humoured, friendly, enthusiastic, helpful, precise, accurate and positive; I suspect that most of the time I fail badly at most or all of those. Writing is hard.)

I understand all that, from both sides; I've certainly been on both the sending and receiving ends of all of the above, many times.

Which is why I try very hard to be helpful to the complete strangers who send me overtly rude and frequently profane emails capped off with a request for me to to take time out of my day to explain something to them.

I try to see an upset or confused user as an opportunity to make a good impression; sometimes people do a complete 180 when you help them out and are appreciative and grateful. Sometimes they even send an unsolicited note to your manager, which is always pleasant.

But more often, I never hear from them again after their question is answered.

None of this is in their own best interests. It makes the human information sources they rely upon less likely to help them now and in the future.

So, just a friendly reminder. (A friendly, good-humoured, helpful, enthusiastic, positive reminder!) The people you are sending email to about your technical problems are people. It would be smart to briefly introduce yourself, describe your problem without insulting the people who created it, indicate what you've done to attempt to solve it yourself, actually ask for help, and later say thank-you for their time whether they manage to actually help you or not.

It's smart because doing so is in your own best interests.

It would also be nice, but I actually am not particularly concerned about "nice" today. Nice is nice to have I suppose. But not being smart is just dumb.

Packet Gnomes

Eric Lippert — Thu, 11 Oct 2007 17:04:00 GMT

The other morning I got the following question from a reader:

I have created a client-server application in C# using asynchronous socket methods [... blah blah blah, a long description of a scenario in which the socket misses packets under certain conditions]

My reply was that I believe networks are run by tiny "packet gnomes" who move the packets around on your behalf. Wireless networks are run by "packet fairies", who can fly. When I have to debug a network problem usually I either leave cookies out for the gnomes, or I call in a friend who has less magical/more scientific beliefs about networks.

Readers: if you have a question about something I actually know about -- design and implementation of programming language tools, the life and works of J.R.R. Tolkien, sailing small craft, piano repair, and so on -- I'm happy to consider it. But there is really very little point asking me about anything else, since odds are good that I am far less clueful on the subject than you already are.

Writing Code Isn't Rocket Science (It's Worse Than That)

Eric Lippert — Mon, 10 Apr 2006 17:30:00 GMT

Today, an old joke:

Q: What do rocket scientists say when they want to describe a portion of their work as easy?
A: "This bit isn't exactly brain surgery."

I think that pretty much everyone would agree that rocket science and brain surgery are both intellectually demanding pursuits. But it seems to me that there's a fundamental qualitative difference between them.

Rockets are devices constructed by humans for specific purposes. Though there may be considerable systemic interactions between all the parts of a rocket which must work harmoniously together, those interactions are the result of careful design. The rocket as a whole can be decomposed into its original parts, and those parts can be independently tested to see if they meet their design criteria.

Furthermore, the aim of rocket science is usually to deliver a specific payload to a specific place at a specific time. Though it is no mean feat to get to the moon, we can least for all practical purposes calculate the future position of the moon at any time you care to name to any degree of precision.

And yes, it's hard, and yes, mistakes are made, sometimes with tragic consequences. But fundamentally, rocket science is about shaping raw matter into precise forms to achieve precise tasks.

Brain surgery isn't nearly so much like that. We're presented with a lump of thinking, dreaming meat that we barely understand how it works and is presently being used to solve all manner of problems that evolution did not design it for. Our understanding of brains is strongest at the very low level – the neurochemical level – and at the very high level – the gross divisions of the brain into the centers that control particular muscles, responses, etc. But we have practically no understanding whatsoever of any level between those – we are no where close to understanding what algorithms the brain uses to recognize faces or write music. And a working, living brain has trillions of interacting parts which were not designed with orthogonality or functional decomposition in mind.

I've been thinking about the difference between brain surgery and rocket science lately in the context of my coworker Peter Hallam's essay on the difference between writing new code and modifying old code.

Naively, one would think that adding new features to code would be like rocket science. You understand all the parts, you figure out how to redesign the parts to admit the new desired behaviour, you implement it, test it, and you're done. It's an intellectual challenge, but it's fundamentally amenable to analysis because every part has a clear, well-designed function that can be tested independently.

But as Peter points out, anyone who has actually tried to add major new functionality to existing code knows that most of the time its more like brain surgery. We know what the code does on a gross level. (Say, tokenize source, parse, create symbol tables, bind type annotations, check reachability, generate code.) We know what any part of it does on the microscopic level. (Say, move the contents of this register to that memory location.) The hard part is understanding what the algorithms are that make up the large-scale behaviour, and understanding how to tweak them to admit the new desired behaviour without breaking anything. (How does tweaking early nullable realization during binding affect rewriting lambdas into expression trees? Who knows? Not me, that's for sure!) Understanding all that stuff is the incredibly hard part; actually putting the code under the knife is comparatively trivial.

A big part of good implementation of software-in-the-large is making sure that the program is more like a rocket than a brain. Coming up with tools to enable that admirable goal is the hard part.

Democracy in Action

Eric Lippert — Wed, 03 Nov 2004 16:48:00 GMT

I saw Tom Stoppard's play "Jumpers" recently. The line that got the biggest audience response:

"It's not the voting that's democracy; it's the counting."

True, that.

I've written a bunch more code for my "Google On The Cheap" series but I've been incredibly heads-down between work, planning a wedding, and my latest book project. Expect it "real soon now".

The National Coin Flipping League Championship Series

Eric Lippert — Thu, 21 Oct 2004 19:09:00 GMT

No tech today, but a little basic math.

In baseball, a sport I know little about, apparently the Boston Red Sox have recently come back from a three game deficit to win a best-of-seven series against their traditional rival team, the New York Yankees.

Baseball is a game which attracts statisticians, and many have noted that this is the first time in major league baseball history that a team has won a best of seven series after being down three games to none.

However, it has happened twice in hockey.

I have a modest proposal. Suppose once a year, the National Hockey League and Major League Baseball decide all their various championships without going to all the trouble and expense of playing the game. Rather, they could simply hold a best-of-seven coin-flipping championship. (Call it the Numismatic Hockey League if you'd like.)

Suppose Boston calls heads. The odds of Boston flipping T T T and then coming back to win with H H H H are one in 128.

Therefore, there should be one such occurrence on average every 128 series. There are four such series a year: the American and National League finals, one "world" series (for which only North American teams are eligible, strangely enough), and one Stanley Cup. You'd expect to wait 128 / 4 = 32 years on average between occurrences.

We've been playing pro baseball and hockey, what, about a hundred years in North America?

Three such series, in about a hundred years -- or, roughly one every 32 years. It seems like the math works out rather nicely. Maybe they have been deciding the games via coin flipping and just not telling anyone. Hmm...

Is Boston's victory really that impressive? I mean, the last time I played Risk I rolled three sixes on three dice and England crushed Iceland -- odds of that are 1/216, almost twice as long as Boston coming back from a three tail deficit in the National League Coin Flipping Championship. That's because my blue plastic army guys really worked together as a team and gave 110%!

And yet it didn't make headlines in even the local paper.

In related news, if Houston wins their championship, and it ends up being Texas vs. Massachussets in both baseball AND the presidential election, that's going to be freaky weird. What are the odds of that?

Anthimeria weirds languages

Eric Lippert — Fri, 01 Oct 2004 21:44:00 GMT

A little non-technical rant for a Friday.

Professor Thingo, in a recent blog entry, decries the use of "Gestalt" as a verb and asks "Does the English language now allow parts of speech to be used entirely interchangeably? Did I miss a memo?"

Though I also would personally balk at verbing "Gestalt" and "architecture" (but not "architect", a perfectly good verb!) I feel compelled to answer the rhetorical question -- yes, by and large English does allow parts of speech to be used interchangeably. In fact, the very notion of "part of speech" arose from the study of Latin grammar, a language with precious little in common with English. The whole notion of "parts of speech" maps poorly to English, a language which cheerfully uses "green" as an adjective, verb, adverb, noun and interjection.

Latin, unlike English, is a highly inflected language. In inflected languages there are roots. You then do things to them that make them into nouns, verbs, plurals, diminutives, whatever you want. The part of speech can usually be determined by the inflection. Some Latin verbs have over a hundred inflections.

English, by way of contrast, never has more than five verb inflections for a given verb (except the perennial exception, "to be"). Drive-drives-driving-driven-drove, throw a couple nouns for good measure (driver-drivers) and we’re done. Every other form of "drive" is formed by adding more words into the mix. You can figure out whether a noun or a verb is meant from cues such as phrasal verb particles ("back" is ambiguous, "back away" is probably a verb), auxiliary verbs ("turn" is ambiguous, "will turn" is not) and other contextual cues.

A huge number of English words are nouns that became verbs without benefit of any kind of inflection or derivation. That's just what English does, and what it's done for centuries, and yet prescriptivists continue to decry it. (Of course, they’ve also done so for decades, so it’s a bit silly for me to decry prescriptivism!)

I found this page of bad advice to be particularly hilarious. This line in particular:

If you look at a dictionary entry carefully, you'll often see that the word you're looking at was used exclusively as a noun up until 1983 or something like that.

Unlike those guys, I actually did look at a dictionary carefully and discovered that in fact many of the verbings they were decrying had been used as verbs in English for centuries. Two in particular stood out. "Impact", which is actually a verb that became a noun in the late 18^th century. (Though, to be fair, the 17^th-century meaning of "impact" as a verb was more along the sense of "impacted molar" than the physics sense of things colliding – that usage didn’t arise until the 20^th century.) More ridiculous though is "parent", which they decry as "idiotically new-age" but has been used as a verb in English since at least the mid 1600’s. (Again, to be fair, the intransitive usage is modern, but the transitive verb sense is very, very old.)

The earliest known recorded usage of each as a noun and verb is telling.

contact: v: 1834 n: 1626
impact: v: 1601 n: 1781
focus: v: 1875 n: 1656
parent: v: 1663 n: 1450
medal: v: 1822 n: 1578

"Parent" has been a verb almost as long as "impact" has even been a recorded English word! And anyone who tells you that you shouldn’t use "medal" as a verb because it’s only been used in that sense since 1822 should also be decrying the use of "mail" as a verb (in the postal sense). "Mail" as a verb dates from as recently as 1827. And not to mention "access", which only dates from 1962 as a verb!

Furthermore, when English introduces new words it frequently takes on both noun and verb forms. Is "Spackle" (a trademark, incidentally) a noun or verb? What about "blog"?

Look at the over two dozen words I’ve used just in this short essay that are clearly both verbs and nouns – part, miss, map, green, root, contrast, throw, figure, answer, cue, back, down, turn, will, line, record, sense, mention, date, take, look, tell, form, use, essay … Using nouns as verbs is just what English does, and I think it’s great. Go verb!

I Have A Mysterious Fifth Sense

Eric Lippert — Sat, 21 Aug 2004 02:35:00 GMT

A little fun for a Friday afternoon.

The economy must be picking up -- I'm getting cold calls from recruiters again for the first time in about four years. Today was the second (and third!) this month.

However, apparently some of them are just a wee bit disorganized. I just had the following conversations:

[Ring ring]

Me: Hi, this is Eric.

Her: Hi, this is Barbara at XYZ Recruiters. How are you today?

Me: I am extremely well! How are you, Barbara?

[This seemed to completely flummox Barbara. Perhaps people who are interrupted at work by cold callers do not usually inquire as to her health?]

Her: Uh. Um. Me? Uh, I'm fine I guess! Thanks for asking!

[Brief pause -- OK, I guess keeping this conversation going is up to me.]

Me: So what's up, Barbara?

Her, back to the script: There's a small company in downtown Seattle that is 60% ex-Microsoft people and they're looking to hire C++ devs. They've just landed a big contract with Foo corp.

Me: Well thanks for thinking of me Barbara. I'm happy to speak with you but to be fair I first must warn you that I am intensely loyal.

Her: Oh. Well, thanks for your time. Bye.

Me: Bye!

Wow, she didn't put up a fight at all. Maybe the economy isn't picking up so much. Four years ago recruiters -- who were for some reason invariably female -- would flirt with me and then try to get me interested in crappy database admin jobs in the Cayman Islands, of all places.

Ah well, back to work. I resolve an old bug that got fixed a while back but never removed from the bug database. I start looking at another bug and researching the history of a particular code change. We've made a minor change to the formatting of an XML file and Mario wants to know whether that was by design or an accident, when...

[Ring Ring -- hey, the caller id looks familiar...]

Me: Hi, this is Eric.

Her: Hi this is...

Me: Barbara at XYZ recruiters?

Her: Uh, yes. How...

Me: What's new?

[Like Hobbes, I love the moment of dawning comprehension. Barbara hits it.]

Her: Wait... did I call you already?

Me: Yes, about ten minutes ago.

Her, trying to place me: Uh… are you a C++ developer?

Me: Why yes I am as a matter of fact!

Her, paging in at last: You're the one who's "intensely loyal", right?

Me: Indeed. And I still am. Cold-calling recruiting really is kind of about looking for disloyal people, isn't it? People who will just pick up when something better comes along, right?

Her: Hey, some people are looking for a change! Wanting change in your life doesn't make you disloyal, does it?

Me: Well, you're the expert. I'll take your word for it.

This brought back fond memories of my teenage days. Unlike my crazy friends, I never prank called people, but I prank answered them all the time. You phone me, you take that risk. The moral: send email.

Having two voice lines in the house (one for my 300 baud modem on the Commodore 64, of course) led to ample opportunities for consecutive calls from clueless telemarketers. On the second call I'd just answer with "Don't say anything! I have a mysterious fifth sense! My psychic powers tell me that your name is Helen, and you want to sell me... magazine subscriptions! Yes?"

Freaked them out every time.

The Attribute Of Manliness

Eric Lippert — Wed, 18 Aug 2004 21:38:00 GMT

This is a technical, not a political, current-events, linguistic or academic blog. (You know of course that as soon as I say that, it's because I'm about to post something that is political, timely, linguistic and academic. Foreshadowing: your sign of a quality blog!) Despite all that, I was so struck by this passage I read last night that I felt I had to share it. We'll get back to error handling in VBScript or some such topic later this week.

The writer is discussing semantics, specifically how word meanings and popular opinions change in political debates during wartime. The writer is... well, I'll just let him say it, and talk about the writer afterwards.

Words had to change their ordinary meaning:

reckless audacity came to be considered the courage of a loyal ally ; prudent hesitation, specious cowardice
moderation was held to be a cloak for unmanliness, frantic violence became the attribute of manliness
ability to see all sides of a question, inaptness to act on any
cautious plotting, a justifiable means of self-defense
the advocate of extreme measures was always trustworthy; his opponent was a man to be suspected
the fair proposals of an adversary were met with jealous precautions by the stronger of the two, and not with a generous confidence
revenge also was held of more account than self-preservation

The cause of all these evils was the lust for power arising from greed and ambition; and from these passions proceeded violence.

Thus Thucydides of Athens, 2435 years ago. (Translation by Richard Crawley. I've changed the formatting and trimmed it a bit -- Crawley gets a little wordy, but I love the balanced sentences.)

The first reaction I had upon reading this was "isn't it astonishing how modern Thucydides sounds across the ages? If he'd only thought to coin the snappy term 'doublespeak', he'd have scooped Orwell by a couple millenia!"

And then I gave my head a shake, because of course I was reasoning backwards. This shouldn't be astonishing in the least; I live in a culture where general opinions on government, politics, warfare, sports and art are more or less just as they were in Classical Greece. It would be more astonishing if Thucydides insights into human nature were not applicable today.