Fabulous Adventures In Coding : Cargo Cult Programming

Cargo Cultism, Part Four: Other Disciplines

Eric Lippert — Thu, 04 Mar 2004 21:01:00 GMT

Just a couple quick follow-ups.

Mike Spille has some good comments tying the relationship between cargo cult programmers and abstraction layers together.

Mike Gunderloy wonders if I've ever read Steve “Code Complete“ McConnell's Cargo Cult Software Engineering essay. I have now -- thanks! However, this essay is about cargo cultism amongst managers of software teams, not the developers themselves.

What this really shows is that imitating form without understanding content is a methodology in many disciplines. It is certainly not limited to the engineering professions! In fact, this problem in a completely unrelated discipline has been on my mind lately.

As Professor Thingo mentions in his recent blog entries, in addition to knowing a thing or two about scripting languages, I'm also an expert on the life and work of JRR Tolkien. I have the dubious distinction of being the first person to put a Tolkien fan page up on the Internet. For many years that page was the first result returned by Google searches for "Tolkien", which got me a lot of email from kids doing book reports and a number of unintentionally hilarious interviews with newspapers when the first Peter Jackson movie was released. The admins of the university server upon which the page resided finally noticed that I'd graduated eight years earlier and removed it some time last year; it had not been updated since 1997 and was a mass of dead links.

But I digress. I think that it is safe to say that the vast majority of fantasy genre fiction produced in the last fifty years apes the form of The Lord of the Rings without coming close to the quality of the content. Want to sell books? Put a map in the back, have a few plucky short guys with swords go up against evil incarnate to save the world, and wait for the royalties to roll in. (See Tom Shippey“s excellent book Author of the Century for an in-depth analysis of this fact.)

Peter Jackson, in one of his Oscar acceptance speeches thanked the Academy for looking beyond the form -- the elves and wizards and hobbits -- to the underlying moral thematic core of the work. And you have to look beyond it; most genre fantasy is derivative dreck, not to put too fine a point on it, and that prejudices people against it. (A notable exception would be the early work of Guy Kay, who was Christopher Tolkien's assistant when he was editing The Silmarillion. I haven't read any of Kay's recent work, so I don't know if it is still of such high quality.)

Cargo cultism is everywhere.

Cargo Cultists, Part Three: Is Mort A Cargo Cultist?

Eric Lippert — Tue, 02 Mar 2004 20:14:00 GMT

My colleague Mike, in a comment in yesterday's entry, mentions "Mort". Who is this Mort guy?

At Microsoft, we do a lot of market research, and frequently discover that large segments of our customers have characteristics in common. It's a lot easier to talk about these customer segments if we take one prototypical example of that segment and give them a name. It's a lot easier to say "Mort likes Intellisense" than "the professional line-of-business programmer who lacks a degree in computer science but has a great deal of familiarity with Office and VBA, and who typically writes productivity applications shared amongst his coworkers, likes Intellisense."

There are many of these prototypical customers but the three we talk about most often on the Visual Studio team are Mort (the line-of-business developer), Elvis (the professional application developer) and Einstein (the expert on both low level bit-twiddling and high-level object oriented architectures.) Elvis and Einstein got their jobs by studying computer science and going into development as a career. Mort comes to a development position via his line of business -- he's an expert on frobnicating widgets, and one day realizes that his widget tracking spreadsheets could benefit from a little VBA magic, so he picks up enough VBA to get by.

Clearly Elvis and Einstein are not Cargo Cult Programmers -- Elvis wouldn't last a month and Einstein's entire reason for continued breathing is directly opposed to Cargo Cult Programming. Is Mort a Cargo Cult Programmer?

Though Mort is more likely to be a Cargo Cult Programmer, I don't think that he necessarily has to be, and I certainly think that he does his job better if he is not! Yes, Mort doesn't understand OOP, but for the kinds of problems Mort solves, he doesn't need to know what inheritance is or how polymorphism works. Mort's programs tend to be straightforward procedural manipulation of object models and strings, with a few loops and subroutines thrown in here and there. Mort does understand that stuff, and often has a quite sophisticated understanding of the practical uses of the object model.

What makes Mort cargo-cultish is that he tends to proceed from a not-quite-working solution (often obtained off the web, or by macro recording) to a working solution by piecemeal local changes. Mort is a very local programmer -- he wants to make a few changes to one subroutine and be done. Mort does not want to understand how an entire system works in order to tinker with it. And my goodness, Mort hates reading documentation -- more on that tomorrow. Intellisense is Mort's favourite feature in the whole world because it lets him look stuff up by hitting the period key. Mort's primary job is to frobnicate widgets -- code is just a means to that end -- so every second spent making the code more elegant takes him away from his primary job.

But what makes Mort not cargo-cultish is that Mort actually does enjoy writing code, Mort is interested in writing code, Mort wants to do the right thing to write more effective code. But Mort, like all of us, sometimes fails to put in the necessary investment in time now that will pay off later. That's not because Mort is a lazy idiot; Mort's a very smart, dedicated guy, but you know that he's got this huge pile of widgets to frobnicate by Wednesday…

In a sense, my advice for novice developers doesn't apply so much to Mort, because that advice was directed towards people who want to be Elvis. Mort does his job better if he avoids cargo cultism. Elvis can't do his job at all if he falls into cargo cultism. In the future I'll try to be more clear about what developer segment I'm blogging about.

Creating developer tools that work for Elvis and Einstein is relatively easy. Creating dev tools that make Mort productive and effective is hard. Creating tools that work effectively given the widely varying goals, learning styles and knowledge levels of those three is the hardest and most interesting thing about this job. I'll certainly talk a lot more about Mort, Elvis and Einstein (as well as Matt, Thomas and Erin) as I start blogging more and more about VSTO.

Cargo Cultists, Part Two

Eric Lippert — Tue, 02 Mar 2004 18:20:00 GMT

Thanks all for your kind words and cogent thoughts on yesterday's post. I'd like to sum up some of the wisdom in the comments:

My advice for novice programmers was

Don't be a cargo cultist -- understand the meaning and purpose of every line of code before you try to change it.
Understand abstraction, and use it appropriately.

To that, you guys add:

learn to use a debugger to step through the code and find problems
learn to NOT use a debugger -- learn to build programs that are debuggable by reading and understanding them
reading code is harder than writing it -- read lots of code
sound methodology is important -- you need to know that you got it right
object models carry semantics too -- learn their semantics
realize that mastery of any skill doesn't take ten days, it takes ten years

Good advice!

I'd also like to respond to some of the points raised in the comments. I'll do that over the next couple of days.

Syntax, Semantics, Micronesian cults and Novice Programmers

Eric Lippert — Mon, 01 Mar 2004 19:18:00 GMT

I've had this idea in me for a long time now that I've been struggling with getting out into the blog space. It has to do with the future of programming, declarative languages, Microsoft's language and tools strategy, pedagogic factors for novice and experienced programmers, and a bunch of other stuff. All these things are interrelated in some fairly complex ways. I've come to the realization that I simply do not have time to organize these thoughts into one enormous essay that all hangs together and makes sense. I'm going to do what blogs do best -- write a bunch of (comparatively!) short articles each exploring one aspect of this idea. If I'm redundant and prolix, so be it.

Today I want to blog a bit about novice programmers. In future essays, I'll try to tie that into some ideas about the future of pedagogic languages and languages in general.

Novice programmers reading this: I'd appreciate your feedback on whether this makes sense or it's a bunch of useless theoretical posturing.

Experienced programmers reading this: I'd appreciate your feedback on what you think are the vital concepts that you had to grasp when you were learning to program, and what you stress when you mentor new programmers.

An intern at another company wrote me recently to say "I am working on a project for an internship that has lead me to some scripting in vbscript. Basically I don't know what I am doing and I was hoping you could help." The writer then included a chunk of script and a feature request. I've gotten requests like this many times over the years; there are a lot of novice programmers who use script, for the obvious reason that we designed it to be appealing to novices.

Well, as I wrote last Thursday, there are times when you want to teach an intern to fish, and times when you want to give them a fish. I could give you the line of code that implements the feature you want. And then I could become the feature request server for every intern who doesn't know what they're doing… nope. Not going to happen. Sorry. Down that road lies cargo cult programming, and believe me, you want to avoid that road.

What's cargo cult programming? Let me digress for a moment. The idea comes from a true story, which I will briefly summarize:

During the Second World War, the Americans set up airstrips on various tiny islands in the Pacific. After the war was over and the Americans went home, the natives did a perfectly sensible thing -- they dressed themselves up as ground traffic controllers and waved those sticks around. They mistook cause and effect -- they assumed that the guys waving the sticks were the ones making the planes full of supplies appear, and that if only they could get it right, they could pull the same trick. From our perspective, we know that it's the other way around -- the guys with the sticks are there because the planes need them to land. No planes, no guys.

The cargo cultists had the unimportant surface elements right, but did not see enough of the whole picture to succeed. They understood the form but not the content. There are lots of cargo cult programmers -- programmers who understand what the code does, but not how it does it. Therefore, they cannot make meaningful changes to the program. They tend to proceed by making random changes, testing, and changing again until they manage to come up with something that works.

(Incidentally, Richard Feynman wrote a great essay on cargo cult science. Do a web search, you'll find it.)

Beginner programmers: do not go there! Programming courses for beginners often concentrate heavily on getting the syntax right. By "syntax" I mean the actual letters and numbers that make up the program, as opposed to "semantics", which is the meaning of the program. As an analogy, "syntax" is the set of grammar and spelling rules of English, "semantics" is what the sentences mean. Now, obviously, you have to learn the syntax of the language -- unsyntactic programs simply do not run. But what they don't stress in these courses is that the syntax is the easy part. The cargo cultists had the syntax -- the formal outward appearance -- of an airstrip down cold, but they sure got the semantics wrong.

To make some more analogies, it's like playing chess. Anyone can learn how the pieces legally move. Playing a game where the strategy makes sense is the hard (and interesting) part. You need to have a very clear idea of the semantics of the problem you're trying to solve, then carefully implement those semantics.

Every VBScript statement has a meaning. Understand what the meaning is. Passing the right arguments in the right order will come with practice, but getting the meaning right requires thought. You will eventually find that some programming languages have nice syntax and some have irritating syntax, but that it is largely irrelevant. It doesn't matter whether I'm writing a program in VBScript, C, Modula3 or Algol68 -- all these languages have different syntaxes, but very similar semantics. The semantics are the program.

You also need to understand and use abstraction. High-level languages like VBScript already give you a huge amount of abstraction away from the underlying hardware and make it easy to do even more abstract things.

Beginner programmers often do not understand what abstraction is. Here's a silly example. Suppose you needed for some reason to compute 1 + 2 + 3 + .. + n for some integer n. You could write a program like this:

n = InputBox("Enter an integer")

Sum = 0
For i = 1 To n
Sum = Sum + i
Next

MsgBox Sum

Now suppose you wanted to do this calculation many times. You could replicate the middle four lines over and over again in your program, or you could abstract the lines into a named routine:

Function Sum(n)
      Sum = 0
      For i = 1 To n
            Sum = Sum + i
      Next
End Function

n = InputBox("Enter an integer")
MsgBox Sum(n)

That is convenient -- you can write up routines that make your code look cleaner because you have less duplication. But convenience is not the real power of abstraction. The power of abstraction is that the implementation is now irrelevant to the caller. One day you realize that your sum function is inefficient, and you can use Gauss's formula instead. You throw away your old implementation and replace it with the much faster:

Function Sum(n)
Sum = n * (n + 1) / 2
End Function

The code which calls the function doesn't need to be changed. If you had not abstracted this operation away, you'd have to change all the places in your code that used the old algorithm.

A study of the history of programming languages reveals that we've been moving steadily towards languages which support more and more powerful abstractions. Machine language abstracts the electrical signals in the machine, allowing you to program with numbers. Assembly language abstracts the numbers into instructions. C abstracts the instructions into higher concepts like variables, functions and loops. C++ abstracts even farther by allowing variables to refer to classes which contain both data and functions that act on the data. XAML abstracts away the notion of a class by providing a declarative syntax for object relationships.

To sum up, Eric's advice for novice programmers is:

Don't be a cargo cultist -- understand the meaning and purpose of every line of code before you try to change it.
Understand abstraction, and use it appropriately.

The rest is just practice.