Buggin' My Life Away : Programming

Craig's Bug

Rick Schaut — Fri, 01 Feb 2008 05:01:00 GMT

One of the things I love about this job is tracking downinteresting bugs. It’s like readinga good who-done-it, complete with plot twists, multiple potential suspects and a fascinating array of characters. Late last September, Craig Eisler submitted a bug that had all of these elements.

Now, when your general manager submits a bug, you don’t necessarily have to get on it right away. But, it’s not a good idea to let it linger for too long. So, after this bug had been on the list for about a week or so, I decided it was time to figure out what was up.

The bug involved a document Craig had been editing with revision marking turned on. He’d made some changes to the document, saved it, quit Word and, when he opened the document back up, some of the spaces between words were missing.

With a bug like this, the first thing I do is open it up in Win Word. If the problem appears there, then there’s a strong likelihood that the bug is in the code that writes the file. If the bug doesn’t appear in Win Word, then I know to start looking at the code that reads files. In this case, the problem appeared in Win Word, so my first suspect is going to be somewhere in the save code.

Also, the missing spaces seemed to be in or around areas of the document where revisions had been made. So, the first suspect was somewhere in the code that writes revisions. I created a document, added some text, turned on revisions made some changes and saved the file stepping through the code that handles revision marking.

Nothing. Everystep along the way, everything was happy. The code was writing all the appropriate XML tags, and none of the text was missing. Of course those two sentences summarize about two-three hours of sleuthing, after which I was able to exclude the revision marking save code as a suspect.

Well, maybe it’s something that only appears to be related to revision marking. Let’s take two files, one that exhibits the problem and one that doesn’t, and compare the contents (much easier to do now that the document content is XML).

Again, nothing. Other than a few gratuitous differences (revision save ID’s, for example, which are uniquely generated every time new text is saved in a document), everything in and around the missing spaces was identical. Another couple of hours spent cracking open various documents and comparing the XML, and I still haven’t a clue whothe real culprit is.

Well, but, if the documents are identical, maybe the problem happens when we read the file. So, I set a breakpoint on the code that handles runs of text from the XML parser, and, sure enough, our “characters” callback wasn’t getting called for just the missing spaces. Hm… Progress.

About another hour or so, I had tracked it down to some general XML parsing code we’d ported from Win Office. One of the problems with writing filing code that’s intended to handle future situations is that, when you write the code, the future hasn’t arrived yet. Well, for Win Office, Mac Office represented the future, and some of the stuff we’d added exposed a subtle bug in the code we’d ported (note: the problem has been fixed in WinOffice before Mac Office shipped, so people shouldn’t run into this at all).

By the time I had finished all of this, implemented a fix and verified the fix, it was about 1:30 in the morning. When I get engrossed in a good mystery, I sometimes find it difficult to put the book down. Same thing goes with good mystery bugs.

And, the bug being a good mystery bug that geeks like Craig and I appreciate, I sent off an e-mail to Craig outlining the basic nature of the bug and the fix (which turned out to be just a couple lines of code—this is almost always the case with a good mystery bug). This being in the middle of the FIFA Women’s World Cup, I ended the mail by saying that I was going to go home, drink some coffee, and stay up to catch the USA match (I forget who the opponent was).

The next day, I got a response from Craig with some questions about the bug. I replied to Craig’s questions and, at the end of the e-mail, lamented over the fact that, after all that effort and all that coffee to stay up to watch a soccer match, our local cable company saw fit to pre-empt the last 30 minutes of the match with a taped question-and-answer program starring Mike Holmgren, the Seahawks coach.

For a born-in-Green-Bay, prick-me-and-I-bleed-green-and-gold, Packers fan like me, having a soccer match for which I’d stayed up all night to be able to watch be preempted by Mike Holmgren is seriously rubbing salt into the wound, and I conveyed my displeasure at this to Craig by saying that the only reason we still have a working television in the house is the fact that there were people asleep in the house at that time.

What ensued was a series of e-mails between Craig and I with a dual thread about the bug he’d reported and my television set. His response to my previous e-mail was to suggest that I should have taken the TV outside and smashed it there. Wake up the neighbors instead of the kids, and I get to vent my hostilities.

In my reply, I explained that this was not an option giventhe size (36”) and weight (over 260 lbs) of my TV set, to which Craig replied, “A CRT!? How quaint! Do you get slivers when you go to the bathroom after watching that TV?”

Having an uber-geek general manager is cool. You get great bug reports, and he likes to hear about the nature of the bug and the fix. Just be a little careful when you talk about the technology around your house.

Rick

Currently playing in iTunes: New World Blues by Gov't Mule

There Comes a Time

Rick Schaut — Thu, 05 Apr 2007 04:48:00 GMT

In every project, there's a point where you have to stop tweaking this and touching up that little bit of behavior and focus on just fixing bugs. If you don't, then you never ship. Nadyne gave some of the details over on mac mojo.

Well, we're getting close to that time with Mac Office 12. In honor of nearing such a milestone, I prepared something for our program management friends. These are the folks who are responsible for designing how the various features are supposed to behave from the user's point of view. It's a suggestion box, and it looks like this:

Mind you, this suggestion box is for PGM use only. Think of it as my way of saying that now is the time for us to shut up and listen to the real users who are trying out the product during the private beta.

Enough fun. I need to get back to fixing bugs.

Rick

Currently playing in iTunes: Beautifully Broken by Gov't Mule (w/ George Porter Jr.)

BBEdit 8.5

Rick Schaut — Fri, 15 Dec 2006 01:14:00 GMT

MacWorld has a decent review of BBEdit 8.5, but it fails to mention a new feature that is of particular use to programmers.

One of the features of CodeWarrior that I'd missed was the syntax highlighting that the IDE applied to symbols that were defined within the project. In fact, I hadn't realized just how much I'd come to rely on that feature when writing new code until I started working with BBEdit and XCode. I had a sudden, new-found paranoia about whether or not I'd typed in a symbol name correctly.

Well, if you use ctags in association with BBEdit, BBEdit will apply syntax highlighting to any symbol that's in the tags file that it would use for the "Find Definition" command on the Search menu.

Rick

Currently playing in iTunes: The Rainy Season by Marc Cohn

What's Wrong with this Code: The Answer

Rick Schaut — Wed, 14 Jun 2006 18:21:00 GMT

Bjarne Stroustrup is a brilliant man, which means he's very good at defining programming languages. It also means he's not very good at writing books about the programming languages he's defined. His writing goal is concise precision. Unfortunately, for those of us who have to read what he writes, that quest for concise precision often translates into some of the driest reading in existence.

Yesterday, I asked What's Wrong with this Code? Unfortunately, the answer to that question involves one of the driest sentences in Stroustrup's book on C++. In fact, it's the very first sentence of section 6.2.2 of The C++ Programming Language:

The order of evaluation of subexpressions within an expression is undefined.

Even as I write this, I can hear the collective, "Huh?" So, let's take at look at the key line of code in yesterday's example. It's in the function, Foo():

_val += Bar(val);

This is an expression with subexpressions. It's a contractional form of the expression:

_val = _val + Bar(val);

It's rather like the contraction "it's"--a way of saying "it is" using fewer letters and syllables. The subexpressions are the bits on either side of the + sign and the = sign.

Now, I'd like to translate that expression back into English for you, but I can't. At least I can't give you a definitive translation. The expression is ambiguous. We can read it as:

Evaluate the function, Bar(val), add the result to the value in _val, and place the sum back into _val.

Or, we can read it as:

Take the value that's in _val, add it to the result of the function, Bar(val), and place the sum back into _val.

Thing is, the actual result of the code example I gave will be different depending upon which interpretation we use. That's because Bar() changes _val (programmers call this a subexpression side-effect). If we choose the first interpretation, then the result of yesterday's program will leave _val with a value of 2. If we take the second interpretation, then the result of that code will leave _val with the value of 43.

Well, that's fine, Rick, but what does this have to do with what Stroustrup said? Stroustrup is saying that compiler writers are free to choose either of these interpretations that I've given above. In fact, compiler writers are free to choose one interpretation at one line of code, and another interpretation in a different instance of the same line of code. While this sometimes happens, compiler writers tend to choose one interpretation and stick to it.

And that's where the real problem lies. Remember, this is a contrived example. Given this kind of contrived coding example, most programmers worth half their paychecks can spot the problem a mile away. But, real-world software models real-world problems. And, real-world problems are complex. So, in the real world, someone writes Bar() without the line of code that modifies _val. Some time later, someone else adds Foo() which uses the computation done in Bar(). A little time after that, someone changes Bar() so that it now modifies _val.

All of these changes get checked into the code base, built into the product, and tested. They might even get tested over and over again. We might even ship several versions of the product, without ever running into any problem that any user will see. And, so long as the compiler we're using takes the "correct" interpretation, there won't be any problem in the code that we actually ship.

Then someone comes along and forces us to start using a new compiler, like when Apple announced the move to Intel processors. Should the new compiler interpret these expressions differently than the way the old compiler interprets these expressions, then we'll have a few subtle bugs in our code.

Now, I've read articles where people describe this programming style as "Microsoft's patchwork programming style," or some such similar--as if the word "patchwork" accurately describes incremental development with modular programming and as if Microsoft is the only entity in existence that uses these programming techniques. Actually, everyone does it this way, and any group engaged in trying to write programs to solve complex problems will end up with some code like this in their products. It's inevitable. I'm quite certain that Adobe has encountered a handful of these kinds of issues in their code, and I'll bet that Apple had to deal with a few of them as well.

Worse yet, in all of the compiler errors and warnings that Erik Schwiebert mentioned in the post that prompted me to offer this little programming example, not one of them involved problems of this type. Those of you who have XCode installed on your machines, copy and paste the code from my example into a test file, and compile it using GCC with every warning you can think of turned on. You'll get nary a peep out of GCC.

We know that many people are anxiously awaiting universal binaries for Office and Adobe's products. Please, be patient. You really do want us to find all of these subtle issues before we ship these products to you, and that takes time--mostly testing time even when you have a lab like this one where you can run those tests.

Kudos to Doug for being the first to say where the problem lies, and kudos to Joshua Ochs for being the first to quote the correct section number from Stroustrup. And, Joshua's guess was almost correct. CodeWarrior uses the first interpretation above. GCC uses the second.

Update: Upon reflection, I thought it worth adding, for some of the more experienced programmers out there, some further details that indicate just how insidious this can be. Imagine some code that began its life as Java. Later, it got ported to C++. The key part of the body of Foo() is a tight loop that terminates when _val reaches 0, and Bar() is really invoked through a pointer to member function in an array of pointers to member functions indexed by _val. This is the kind of stuff that keeps me awake at night.

Rick

Currently playing in iTunes: Everyday (I Have the Blues) by The Marshall Tucker Band

What's Wrong with this Code?

Rick Schaut — Tue, 13 Jun 2006 22:44:00 GMT

A while back, Erik Schwiebert wrote about some of the travails of moving our build system from one using CodeWarrior to one using XCode and GCC. Erik mentioned the huge number of errors and warnings we've had to resolve, but, in reading some of the reactions around the web, I'm afraid that some people, particularly people who know nothing about compilers and developing software, seem to have completely mistaken Erik's remarks.

Someday, I'll understand why people are motivated to render opinions on matters of which they are entirely ignorant, but, for now, I'll have to satisfy myself by offering some remedial education. I'll start by asking, what's wrong with the following code:


class FooBar
{
public:
    FooBar(int val): _val(val) {};
    ~FooBar() {};

    void Foo(int val)
        {
        _val += Bar(val);
        };

    int Val() { return _val; };

private:
    int Bar(int val)
        {
        _val = val;
        return _val;
        };

    int _val;
};

int main()
{
    FooBar fb(42);
    fb.Foo(1);

    int val = fb.Val();

    return 0;
}

Some of my readers will spot the problem immediately. To those who do, I'll remind you that this is a contrived example. A real-world example will have a class that has 10 or more methods, and none of those methods will be as trivial as Foo or Bar.

Update: Fixed code problems due to .Text's handling of HTML entities.

Rick

Currently playing in iTunes: One Way Out by The Allman Brothers

Compound Statement Macros

Rick Schaut — Thu, 01 Sep 2005 03:57:00 GMT

I had thought this was a rather well-known trick, but, after reacquanting myself with Alexandrescu's Modern C++ Design , I've come to believe that it's not at as well-known as I thought. So, I thought I'd share it here for posterity. It's common to write...(read more)

Got Frontier and MacBird Experience?

Rick Schaut — Wed, 15 Jun 2005 17:52:00 GMT

Dave Winer is looking for a Mac developer with knowledge of Frontier and MacBird to work on his latest project: an OPML editor. According to Dave, the Mac version is stalled.

If I didn't just get a load of work heaped on me, I'd be sorely tempted to toss my hat in. It'd be a great, and really fun, chance to work with Dave Luebbert again.

At one point in my life, I aspired to be a guitar player. I still play guitar, but my day gig is just too good to give up. Back in those early days, however, I absorbed as much as I could. I still have a dog-eared copy of the late Tommy Tedesco's For Guitar Players Only.

Tommy gave a lot of great advice in that book, and one of the best is that there are really only three reasons to ever take a gig: you'll make money from it, you'll learn from it or you'll just plain have fun doing it. Winer's gig looks like it has two out of those three.

Rick

Currently playing in iTunes: The Life by Alicia Keys

Flipping Out

Rick Schaut — Wed, 08 Jun 2005 01:12:00 GMT

I've seen quite a few people comment in blogs that the endian issue I mentioned yesterday won't turn out to be much of an issue. Well, you can't say I didn't warn you about drinking the Kool-Aid, but it you still have some lingering doubts that there isn't a significant number of developers out there who are going to have issues, go wade your way through this stuff.

For a more concrete example, Apple have produced documentation for developers who use PowerPlant. PowerPlant uses various resources to describe the UI objects that are created--windows, menus, dialog boxes, etc. Reading these resources correctly in the context of a universal binary will be crucial. They've included some sample code to flip bytes in PowerPlant resources. It's more than 780 lines of code--780 lines of tedious, byte flipping code.

Now, there are quite a few people who are likely to say that this is really just an extreme example of where byte-swapping is problematic; that, if developers followed all the "rules" they will have no problem. The fact is that these "rules" are nothing more than rules-of-thumb--general guidelines that many developers will disregard for very legitimate reasons.

Consider a drawing program like Adobe Illustrator (I almost said FreeHand--does that date me?). Undo is an important feature in a drawing program. There are two ways to do Undo: save off an entire copy of an object for each user action, or encode user actions as operation codes that can be transacted. In the latter case, you save the object in its original state plus all of the changes the user made to the object as opcodes. Chances are that the opcode solution will result in much smaller files. It does, however, break the "rules" that unknowing people tend to tout when these kinds of discussions arise.

No. The PowerPlant resource example isn't extreme. It's just a common, specific example for which Apple have been kind enough to provide a solution. Anyone shipping serious applications that have not already had to solve this problem for cross-platform scenarios has quite a bit of tedious work to do. While I've welcomed this change for my own circumstances, I don't envy the position in which a number of my compatriots down in San Jose find themselves.

Rick

Currently playing in iTunes: If 6 Was 9 by Jimi Hendrix

The Beginning of the Endian

Rick Schaut — Tue, 07 Jun 2005 11:04:00 GMT

So the rumors have been true. Apple is moving to Intel processors, which, in some ways, makes sense. At least it does from Apple's point of view. The Kool-Aid has never really affected their ability to figure out what's right for Apple regardless of the direction they've taken.

For users, the transition won't be all that painful either. Indeed, the prospect of being able to run VirtualPC on a native Intel processor has me, as a user, licking my chops. Buy a Mac and you'll get the best of all three worlds: Mac, Unix and Windows.

When you're a developer, though, the Kool-Aid can be rather heady. For those of us who've been around for a while, developing software for Apple's computers has often been rather like being a farm animal with a ring through its nose. In fact, the primary difference between Apple and Microsoft as platform vendors is that Apple generates excitement in the user base while Microsoft tends toward generating excitement in the independent software developer base.

So, when Steve Jobs starts talking about certain software developers having little trouble making the transition from the PowerPC to x86 processors, you can pretty well bet that the Kool-Aid is laced with all sorts of compounds that are not there for your benefit.

No doubt, some, perhaps even many, applications will have little difficulty making the transition. There is, however, a class of applications that will require a significant amount of work. The feature they all have in common: they produce documents--files with data in binary form. To understand why, we need a little lesson in the history of microprocessor design.

Microprocessors have data registers. These are analogous to little scratch-pads on your desktop. When the computer needs to do some manipulation of data, it pulls that data out of memory and puts it into these little scratch pads. When the computer is done with that data, yet wants to save it for later use, it writes the data on the scratch pad out to memory.

Early microprocessors had 8-bit, or byte-sized, data registers. This was generally convenient, because memory is considered to be just a very huge array of bytes. However, when the first 16-bit microprocessors were designed, the size of the scratch pads, or data registers, was twice the size of a single, addressable piece of memory.

This introduced a design problem. When you read a 16-bit value from a byte-addressable data store, like memory, which byte do you load first? You have two choices. You can load the most-significant byte first (known as big-endian), or you can load the least-significant byte first (known as little-endian). The difference between big-endian and little-endian systems is often referred to as "byte-sex". It's a term I've used before, and, for those who've wondered what it means, now you know.

As luck would have it, the two most active designers of microprocessors at the time, Intel and Motorola, answered that question in opposite ways. I'm not well-versed in VLSI design, but I have little doubt that each group of chip designers had legitimate reasons for the choices they made. None of that is important here. What is important is that, way back in those days, Apple chose Motorola processors while IBM chose Intel. Thus began the great endian divide.

Independent of each other, this isn't really a significant issue. Big-endian is slightly more difficult to work with than little-endian, but the parameter type provided by modern compilers renders the problem negligible. The real problem starts when binary data gets saved to disk and shared.

To see how this works, imagine a program that keeps track of people's salaries. Your boss decides to give you a raise, so this program reads your salary from memory into one of these data registers, adds your raise to it, and then writes it back out to memory. At the end of the day, the program stores all employee records to disk as just an array of bytes as they reside in memory. Let's also pretend that the accounts use computers with Intel processors, so the data got written in little-endian format.

Now suppose you want to look to make sure you got the correct raise. You fire up a program on your Mac that reads your employee record, but when it reads your salary, it's going to read the bytes of that number in reverse order from the way it was written on an Intel processor. If your new salary is supposed to be $128 a week, unless the data was byte-swapped when it was read from the disk, your Mac will lead you to believe that your salary is really $2,147,483,648 per week. Now that's a raise!

If you think this is messy, then know that I've oversimplified the problem significantly. Rarely do programs maintain various numbers in isolation from each other. Employee records often have other numbers associated with them, like hours, hire dates, number of children, exemption status, etc. These are kept in compound data structures generally known as records. Many of them are quite huge.

When transferring data between x86 and PowerPC based computers, each numeric value in each of these records needs to be byte-swapped. Code that doesn't properly byte-swap data written on the other processor type has a bug. I call these "unsafe byte-sex" bugs. If you're going to move documents with binary data between PowerPC and Intel processors, you have to write code that practices safe byte-sex.

For some developers, this problem will be huge. Word, for example, has a total of 176 distinct data structures in its binary file format, each of which needs to be properly byte-swapped whenever one of them is read from the disk. Word's example is probably a bit extreme, but even 20 or 30 data structures requiring proper byte-swapping represents a good chunk of work for both developers and testers.

That said, life for Word will actually get quite a bit easier. Why? Because Word's files are always written in Intel byte-sex. Given the number of unsafe byte-sex bugs I've had to fix over the course of my career working on Word, I'm actually looking forward to the day when I can forget about them almost entirely.

Rick

Currently playing in iTunes: Saint Augustine In Hell by Sting

C++ Function Objects

Rick Schaut — Sun, 15 May 2005 20:01:00 GMT

C++ provides us with a number of ways we can shoot ourselves in the foot. If you work as a developer on any sizeable software project, there is quite likely a set of rules about limitations as to the C++ language features you’re allowed to use: no memory allocation in constructors, no operator overloading, exceptions used only for error handling, etc. Repentance for violating these rules often resembles that for breaking the master build—i.e. some obligation to perform some menial yet necessary task for the group and/or the possession of some object of indistinction (“The Hat” or “The Hose”).

There are, however, reasons for flexibility where at least some of these rules are concerned, and I’ll offer as an example some practical considerations in favor of allowing the use of operator overloading for operator() (that’s read as “function-call operator”). If you happen to be one of the lucky ones who work on projects with only one or two other programmers, or even just by yourself, stick around. ‘Cause function objects are cool, and I’m about to tell you why.

Those of us who write software for the Macintosh know that the world is really divided into three: Windows, Macintosh and Unix. How do we know this? Because we’re constantly having to manipulate the line endings of various text files. Curiously enough, the standard Mac OS X install doesn’t have a neat little command-line tool for converting line endings (or at least “apropos” with various forms of the phrase “convert line endings” yields “nothing appropriate”). There isn’t even one among the little more than two dozen command line tools in Apple’s Developer Tools.

So, if you often find yourself having to flip the line endings of various text files, you’ll either open them all up in, say, XCode or BBEdit, and manipulate the line endings by hand, or you’ll write a little command line tool that will do what you need. The benefit of the latter is that it can handle large numbers of files all at once.

Should you write one such tool, you’re quite likely to have three different functions in your code that look something like this:

/*---------------------------------------------------------------------------- %%Function: ClnConvertToWin %%Contact: richs ----------------------------------------------------------------------------*/ int ClnConvertToWin(InputFile &in, OutputFile &out) { char chPrev = chNul; int cln = 0; for (;;) { int ch = in.ChGet() if (in.FEof()) break; switch(ch) { default: out.PutChar(ch); break; case chCR: out.PutChar(ch); if (chLF != in.ChPeek()) { out.PutChar(chLF); cln++; } break; case chLF: if (chPrev != chCR) { cln++; out.PutChar(chCR); } out.PutChar(ch); break; } chPrev = ch; } return cln; }

This code, which converts an arbitrary input file to Windows’ line endings, looks simple enough. It reads a character from the input file one character at a time, and performs some specific action based on which character is just read. It keeps track of the number of lines that it’s converted, and returns that count when it’s all done.

The two other versions of this function likely have the exact same loop control and differ only by the structure of the switch statement that does the actual conversion of line endings.

This is bad, because you now have three separate loops in your code that are almost identical. Suppose, for example, that you move this code to a system where the “read a character, then test for end-of-file” construct isn’t the most efficient or robust way to read characters from a file. You now have three separate loops of code to change, and three separate opportunities to create bugs in the code.

In the old days, we might have resolved this problem by using function pointers, but they’re clumsy. Also, function pointers provide no opportunity for the compiler to optimize out the function-call semantics. You’re going to be stuck with full procedure prologue and epilogue with every iteration through that loop. For performance reasons, as well as maintenance reasons, we don’t want to use function pointers in this particular application.

With C++, however, we can encapsulate the switch statement into a function object, and put the control loop in a template function that takes as a parameter a reference to an object that overloads operator(). The template that encapsulates the loop might look like:

/*---------------------------------------------------------------------------- %%Function: ClnConvertLines %%Contact: richs ----------------------------------------------------------------------------*/ template <class CharConverter> int ClnConvertLines(InputFile &in, CharConverter &cnv) { int cln = 0; for (;;) { int ch = in.ChGet(); if (in.FEof()) break; cnv(ch, cln); } return cln; }

And the function object that converts arbitrary line endings to Windows might look like:

/*---------------------------------------------------------------------------- %%Class: ToWin %%Contact: richs ----------------------------------------------------------------------------*/ class ToWin { public: ToWin(InputFile &anIn, OutputFile &anOut) : in(anIn), out(anOut), chPrev(chNul) {}; ~ToWin() {}; void operator()(int ch, int &cln) { switch(ch) { default: out.PutChar(ch); break; case chCR: out.PutChar(ch); if (chLF != in.ChPeek()) { out.PutChar(chLF); cln++; } break; case chLF: if (chPrev != chCR) { cln++; out.PutChar(chCR); } out.PutChar(ch); break; } chPrev = ch; }; private: int chPrev; OutputFile &out; InputFile ∈ };

With that, our original conversion function becomes:

Inline int ClnConvertToWin(InputFile &in, OutputFile &out) { ToWin cnv(in, out); return ClnConvertLines(in, cnv); }

I should point out that there is no a priori reason for ClnConvertLines to be a template. We could have defined a base class, CharConverter, that virtualized operator(), and made ToWin a subclass of CharConverter. In this particular case, however, the virtualized base class approach isn’t any better than the old-style, function pointer approach. In fact, on some systems, it’s worse, because you have the double-dereference through an object’s v-table instead of the single dereference of a function pointer.

The template-based solution, while it yields more object code in that ClnConvertLines will get instantiated for every different flavor of cnv object we give it, is much faster for our application. Because the template-based solution gets expanded in line, there is an opportunity for the compiler to optimize out the function-call semantics where the overloaded operator() is invoked—one of those rare instances where we get to have our cake and eat it too.

Now, if that weren’t cool enough, the fact that we’ve abstracted out the actual conversion of line endings into a separate piece of source code leads to a flexibility one wouldn’t want to entertain in the purely functional approach. For example, suppose we know that a particular input file has Macintosh line endings. Scanning the beginning of an input file to figure out the existing line endings isn’t all that hard, and is well worth the time if it greatly simplifies our inner loop. The implementation of the line conversion from Macintosh to Windows line endings is almost trivial:

/*---------------------------------------------------------------------------- %%Class: MacToWin %%Contact: richs ----------------------------------------------------------------------------*/ class MacToWin { public: MacToWin(OutputFile &anOut) : out(anOut) {}; ~MacToWin() {}; void operator()(int ch, int &cln) { out.PutChar(ch); if (ch == chCR) { out.PutChar(chLF); cln++; } }; private: OutputFile &out; };

You wouldn’t entertain something like this in the purely functional approach, because the proliferation of code with the same loop semantics is something you want to avoid. If having just three duplicates of that outer loop is bad, having one for every possible known combination of input and output line endings is that much more of a maintenance headache. With function objects, we can proliferate to our heart’s content without increasing the level of maintenance required should we decide to change the semantics of the loop control.

By now, there’s at least one astute reader who’s thinking, “Gosh, Schaut, flipping line endings isn’t all that different from iterating through one of the Standard Template Library’s collection classes. Using function objects should be obvious. What’s all the fuss about?”

Such an astute reader would be absolutely correct: they way I’ve used function objects here is almost exactly the way function objects are used in the STL. In fact, we can take that line of thought and extend it to the concept of an input iterator.

Think about how one might use a command-line tool to convert line endings. Some times, you’ll want to just invoke the tool on a single file. Other times, you’ll want to invoke the tool on a whole bunch of files in a single directory. On still other occasions, you’ll want to use some complex find command to generate a list of files in an entire directory tree, and pipe the output of that command through the line converter’s standard input file.

So, you’ll have two distinct ways of getting a list of files to convert: as an array of C-style strings provided on the command line or as a list of file names coming in via your standard input file. The structure of the loop to convert files and report the progress of that conversion to the user ought not change simply because we’re getting a list of files in two distinctly separate ways. This problem screams for a solution where input iterators are implemented as function objects.

I’ll leave the actual implementation of this as an exercise for the reader, but there is one thought to consider. The input iterator is in an outer loop, not an inner loop, and the function that figures out which particular conversion loop to invoke is likely to be complex enough that we wouldn’t want multiple copies of it in our object code. In this case, I would avoid a template-based approach in favor of defining a base class for our input iterators where the operator() is virtualized.

Hopefully, this will lead some of you to think more about using function objects in your daily work—in particular, I’d want you to think that function objects are useful outside something as complex as the Standard Template Library. If function objects can improve our implementation of something as mundanely simple as flipping line endings in text files, they just have to be cool enough to use in a wide variety of contexts.

Rick Schaut

Currently playing in iTunes: Sierra Leone by Derek Trucks Band

Update: Fixed the template definition for ClnConvertLines (convert to HTML entities).

Subtle Bugs

Rick Schaut — Mon, 11 Apr 2005 05:53:00 GMT

It wouldn't surprise me for people to wonder how it’s possible for reasonably competent programmers to introduce bugs by making seemingly innocuous changes to some program’s code. The short answer is that we’re often too clever for our own good. We’ll employ little tricks that subtly change the semantics of the data that we’re manipulating. These small changes in semantics can result in weird bugs.

There’s a good example of this in the sample code that Apple distributes with the developer tools for Mac OS X. One bit of sample code implements a rudimentary framework for Carbon Human Interface objects. Among this code is the implementation of a wrapper C++ class for Carbon Events. The following is a method implemented in this class. It extracts a Boolean parameter from a Carbon Event, and returns it in a pointer to a bool.

//-----------------------------------------------------------------------------------
//     GetParameter
//-----------------------------------------------------------------------------------
//
OSStatus
TCarbonEvent::GetParameter(
    EventParamName  inName,
    bool*           outValue )
{
    return GetParameter<Boolean>( inName, sizeof( Boolean ), (Boolean*) outValue );
}

For my non-programming readers, both a Boolean and a bool represent values in Boolean algebra. Boolean algebra is the algebra of logic, and Boolean variables take on only one of two values: true or false. In computing practice, and in the C++ Language, false is equivalent to 0, and true is any non-zero value.

The problem, however, is that Boolean and bool are different language types. Boolean represents the traditional Macintosh implementation of the concept, while bool is the concept as specified in the C++ Language standard. Boolean is type-equivalent to an unsigned char—a single-byte quantity, while the underlying implementation of bool can change from one compiler to the next. In Metrowerks, for example, bool is usually type equivalent to an unsigned int—a four-byte quantity.

And therein lies the problem. The code above blandly takes a pointer to a four-byte quantity and passes it to a piece of code that expects a pointer to a single-byte quantity. This would not be a problem on Intel processors, but the PowerPC is big endian. On a big endian system, when you take the address of a four-byte quantity the resulting pointer actually points to the highest-order byte (i.e. the “big” end of the four-byte quantity) rather than the lowest-order byte (i.e. the “little” end of the four-byte quantity).

Suppose the called code uses the number 1 as a constant value representing true. If the requested parameter of the wrapped Carbon event is true, then this implementation of GetParameter will put the number 1 in the high-order byte of outValue. Instead of getting the number 1, then, outValue actually gets the number 16777216 (or 0x1000000 hexadecimal).

Now, in terms of language semantics, there’s nothing specifically wrong with this notion. Both numbers, 1 and 16777216, being non-zero values are valid representations for true. However, most compilers favor using the number 1 to represent the actual value for the constant true, and “clever” programmers often take advantage of this fact, and will use the result of a Boolean expression in arithmetic expecting a true result to be the same as the number 1.

If some other programmer has written a piece of code that takes advantage of this tendency to use 1 to represent true, the implementation above breaks that code. This is how subtle bugs enter programs: two programmers being “clever” in mutually exclusive ways.

The immediate problem with the above code is that it doesn’t make its cleverness explicit. One way to do this is to add a CompileAssert:

#define CompileAssert(_expr) extern int _compile_assert_failed[!!(_expr)]
//-----------------------------------------------------------------------------------
//     GetParameter
//-----------------------------------------------------------------------------------
//
OSStatus
TCarbonEvent::GetParameter(
    EventParamName  inName,
    bool*           outValue )
{
    CompileAssert( sizeof(bool) >= sizeof(Boolean) );
    return GetParameter<Boolean>( inName, sizeof( Boolean ), (Boolean*) outValue );
}

Adding the CompileAssert not only makes the assumption explicit, it will generate a compiler error should we ever move this code to a system where the assumption is no longer valid. But that doesn’t really resolve the underlying problem.

Another attempt to resolve the problem might to replace the C-style cast with a C++ explicit static_cast:

This actually works provided that our particular compiler defines bool in a manner that’s type equivalent to Boolean. It has the added benefit that it will generate a compiler error should the underlying type-equivalence ever change. Unfortunately, if we’re already working with a compiler where Boolean and bool are not type equivalent, then our code no longer compiles. Metrowerks, for example, generates a compiler error with the message, illegal explicit type cast from bool* to Boolean*.

For my money, the best solution is the pedantically correct solution:

//-----------------------------------------------------------------------------------
//     GetParameter
//-----------------------------------------------------------------------------------
//
OSStatus
TCarbonEvent::GetParameter(
    EventParamName  inName,
    bool*           outValue )
{
    OSStatus result;
    Boolean value;

    result = GetParameter<Boolean>( inName, sizeof( Boolean ), &value );
    if (noErr == result)
       *outValue = value != 0;

    return result;
}

Note that this solution doesn’t simply assign value to the location in memory pointed to by outValue. Rather it assigns the result of the Boolean expression value != 0 to value.

When you work on a large software project with a number of other programmers, being so pedantically correct as to use real Boolean expressions for Boolean results will often mean the difference between code that works and plays well with others and code that introduces subtle bugs that can take you hours to track down, or worse, months before users actually discover that the bugs exist.

Rick

Currently playing in iTunes: Rhyme & Reason by Dave Matthews Band

MPW C Error Messages

Rick Schaut — Fri, 01 Apr 2005 13:54:00 GMT

Those of us who've been programming the Mac for a few years have some fond memories of the old Macintosh Programmer's Workbench (MPW). It was very much like the Terminal Window on OS X, but had a few quirks of its own.

Among those quirks were the error messages that the C compiler would give you for certain errors. Messages like, "a typedef name was a complete surprise to me at this point in your program," and "String literal too long (I let you have 512 characters, that's 3 more than ANSI said I should)" almost made you want to write broken code just to see what the compiler had to say.

The most complete list of the error messages I could find is here.

Well, rumor has it that there's a skunk works at Apple who are busy porting some of these error messages into XCode 2.0. And they say that Apple have lost their sense of humor...

Rick

Currently playing in iTunes: Won't Get Fooled Again by The Who

Hungarian Notation (Again)

Rick Schaut — Fri, 11 Mar 2005 08:21:00 GMT

Cyrus is having difficulty parsing out the example given in Simonyi's original article on the subject. I've blogged about this before, but it's worth reiterating the main point of my earlier post.

It's somewhat unfortunate, because the example that Simonyi chose happens to be a rather bad example of Hungarian notation, though, arguably, it serves well to illustrate Simonyi's point about "dimensional anaylsis."

There's a really simple rule we can use to figure out whether or not we're using Hungarian notation correctly. If the name of the variable denotes nothing more than the language type of the variable, then we're using it incorrectly. If the name of the variable reflects how it's being used in the program, then we're using Hungarian notation correctly. In this sense, we can think of Hungarian notation as acronyms applied to variable names.

As an exercise, you might want to take Simonyi's example, and correct the usage. In particular, see if you can't think of a better prefix for the computed hash value. perhaps "hsz", or maybe simply "sh"?

By the way, there's a potential bug in Somonyi's example that's implied by the use of Hungarian notation. There is an extern integer declared as "bsyMac" in the code. This is a current maximum value within a collection. PsySz computes a new bsy, but doesn't update bsyMac. It should, either directly or indirectly.

Rick

Currently playing in iTunes: Book Of Liars (Live) by Steely Dan

AppleScript and VBA: the Story of Sid and Nancy

Rick Schaut — Thu, 05 Aug 2004 14:30:00 GMT

In the few spare moments I’ve had over the past several weeks, I’ve followed some discussions of Word and AppleScript both in the comments to my earlier post and in the Microsoft newsgroups. A particularly good discussion, with input from both Matt Neuburg and Paul Berkowitz, can be found here.

Both Matt and Paul have pointed out that there is a very strong correspondence between AppleScript and Word’s VBA (Visual Basic for Applications) implementation. The same is true for both PowerPoint and Excel, though not for Entourage. There is no VBA implementation in Entourage.

I want to talk about why AppleScript so closely resembles VBA in Word, Excel and PowerPoint, but I first want to take a step back and talk about objects and object models. These are terms you’ll find quite often in any discussion of scripting regardless of the application or the scripting environment. It has a lot to do with how programmers think about the data they manipulate and how they manipulate it.

An “object” is any entity that can respond to some form of input or command, and often has some form of data associated with it. Word, as an application, is an object. The documents that you have open in Word are objects. The windows that display those documents are objects. The sections within those documents are objects. The paragraphs within the sections are objects. In formal terms, an object is both the data it contains and the operations that can be performed on the data it contains. We use the word “properties” to talk about the data, and the word “methods” (or “commands” in AppleScript parlance) to talk about the operations that can be performed on that data.

An “object model” is a conceptual definition, or description, of how the objects in an application are related to each other. Quite often this takes shape in the form of a “containment hierarchy”. The “containment hierarchy” is really just a fancy way of saying that some objects are properties of other objects. For example, in Word, a paragraph is a property of the document object. This relationship can include the notion of collections (or “plural forms” in AppleScript parlance). Really, a collection is nothing more than a group of similar objects within the same container.

Any given object model is very application centric. Word’s object model, for example, has some significant differences from, say, BBEdit’s object model, and that’s not just because Word does some things that BBEdit doesn’t. The way one describes the “selection” object is highly dependant upon how the concept of a “selection” is implemented in that application. The “selection” object for a drawing program, for example, is going to be very different from the “selection” object in a text editor.

Commands and how they relate to objects are also extremely application centric. For example, when you tell an application to “select foo of bar,” what does that really mean? Are you telling “foo of bar” to select itself, are you telling the selection object to incorporate “foo of bar,” or are you telling the application to change its selection to foo of bar? The answer to that question will depend largely on the application you’re working with (though, in Word, it generally means that you’re asking “foo of bar” to become the selection).

The important point in all of this is that, at the conceptual level, the object model one defines for any given application depends on the application, and not the scripting environment or scripting language you use to expose that object model. Had Word never had a VBA implementation of its object model, Word’s AppleScript implementation of that object model would not have differed substantially from its current implementation. Any differences would be more attributable to the way particular individuals approached the design of the object model than they would be attributable to differences between VBA and AppleScript.

This is the primary reason why there is such a strong correspondence between the VBA implementation and the AppleScript implementation of the object models in Word, Excel and PowerPoint. One could, if one wanted to, implement VBA as a pass-through to an AppleScript implementation of these object models, though there is a fundamental difference between VBA and AppleScript that would tend to favor doing things the other way around.

To understand this fundamental difference, we have to understand something known as “application intercommunication,” or “interprocess communication” (the word “process” is really just a synonym for “application program”). Within modern operating systems, applications have fences built around them. The idea is to keep bugs in one application from affecting the behavior of other applications running on the system. Within this design, if applications are going to communicate with each other, we need to give them a way to pass information back and forth over the fences that have been built around them.

With AppleScript, interprocess communication is done via AppleEvents. Every command that you send to Word via AppleScript is packaged up into an AppleEvent. When Word’s done processing that AppleEvent, it packages up a reply, puts the reply in the AppleEvent, and sends it back “over the fence” as it were. With VBA, interprocess communication is done using the Component Object Model (or COM). COM provides programming interfaces that are amenable to being called directly within the programming languages we use to write the applications themselves. COM is often referred to as a “binary interface,” because it lends itself to direct programmatic manipulation in this way.

In essence, COM was designed so that objects in the system would look an awful lot like they way objects are implemented in the C++ programming language. In fact, you can take a file written in the COM Object Description Language (ODL) and, using a command-line tool, generate header files that can be included directly into C++ code.

For interprocess communication, COM has a technical problem that I won’t describe in detail, but involves the fact that applications have fences built around them. Basically, one program cannot directly call a binary interface that’s provided by another program. That would be like reaching through the fence instead of passing something over the fence. To solve this, COM uses something called “marshalling”. None of this has to be done for AppleScript, because AppleScript doesn’t provide the same kind of binary interface.

So, COM is messy for interprocess communication, but it’s really nice for intraprocess communication. AppleScript isn’t quite so messy when it comes to interprocess communication, but it’s a royal PITA when it comes to intraprocess communication. What do I mean by intraprocess communication? Well intraprocess communication is how various software components in an application, and in any shared libraries used by that application, communicate with each other. In this case, the kind of binary interface that COM provides is perfect. To the programmer, the COM objects look like any other component in the program (which is where the name “COM” comes from).

This is, in fact, how all of the external wizards that we ship are implemented. They’re written in C++ and they talk to Word, or Excel or PowerPoint, via the binary interfaces each of the applications expose through the VBA implantations of their respective object models.

For those of you who are familiar with COM and the standard programming interfaces that COM defines, I should point out that all of these interfaces are “dual dispatch” interfaces. While every object does implement the standard IDispatch interface, every method and every property of every object is also available through direct programmatic manipulation through individual interfaces that descend from IDispatch. The point is that neither the Wizards we ship nor the AppleScript implementation goes through IDispatch::Invoke. Rather, both call the appropriate method or property implementation directly on the appropriate object itself.

If you don’t understand anything in the above paragraph, don’t worry. The important point to understand is that, in light of the previous point about the design of an object model for any given application, it makes perfect sense to use an existing VBA implementation of an object model to expose that same object model via AppleScript. In fact, not only does it make perfect sense to do this, it’s the most efficient way you can possibly do the job. The VBA implementation already packages up the object model in a manner that lends itself to direct programmatic manipulation.

It makes so much sense to do things this way that you can even get lazy about it. You can design a language that describes your object model both in VBA and AppleScript terms, and then you can write a tool that parses a description file and generates the code for you. In only a “few” (quoted because “few” is relative—there are 215 distinct objects in Word’s object model, and I won’t even begin to count all the properties and methods on those objects) instances do you have to get your hands dirty and write some code yourself.

As an insight into Jim Murphy’s sense of humor, the object description language is called “Sid,” and the tool that compiles files written in that language is called “Nancy.” One of these days, I’m going to have to go rent that movie.

Rick

Anatomy of a Software Bug

Rick Schaut — Wed, 19 May 2004 20:50:00 GMT

Anatomy of a Software Bug

Chris Mason is the person who hired me to work at Microsoft. By the time he hired me, he’d already spent a great deal of time looking into the issue of general software quality, and had written a memo (known as the “Zero Defects” memo) that underlies much of our software practices today. The ideas have been refined since then, but they haven’t changed much in terms of the basic concepts.

One of my favorite Chris Mason quotes comes from that memo, “Since human beings themselves are not fully debugged yet, there will be bugs in your code no matter what you do.” We work to minimize the bugs in the software we ship, but they’ll always be there.

The problem stems from the overall complexity of the software. In this context, “complexity” doesn’t refer to the code itself. Rather, we’re talking about the shear volume of things the user can do. In Word, for example, we have:

More than 850 command functions (e.g. Bold and Italic are the same command function)
More than 1600 distinct commands (e.g. Bold and Italic are distinct commands)
At any given time roughly 50% of these commands are enabled (conservative estimate)
With just 3 steps, the possible combinations of code execution paths exceeds 500 million

Now, there’s a philosophical issue about the desirability of increasingly complex software, but I’m not going to discuss it here. For all practical purposes, I don’t think there’s much benefit to getting into a discussion about it. It may be an interesting question on some level, but it’s one we’ll never fully resolve. And I’m just not all that interested in getting bogged down in an endless debate without the possibility of resolution.

I mention the issue of complexity because it leads to subtle interactions that can be difficult to track down. To illustrate the point even further, I thought I’d discuss the anatomy of one of the more famous bugs we’ve had in Word: the “Disk is full” on save error. Before I do, however, I should point out that Pierre Igot, after some prodding on my part, did provide us with a sample document that helped us to track down one of the more subtle interactions involved in this particular problem. For that, Pierre, I thank you and so do Word users everywhere.

The story of this problem begins with a basic design decision made when Richard Brodie was Word’s primary software architect. Brodie came to Microsoft along with Charles Simonyi after working at the Xerox PARC where he’d worked on Bravo—their version of the GUI word processor. A number of the ideas used in Word came from that early effort. Brodie joined Microsoft in 1981, began work on Word in the summer of 1982, and finished version 1.0 in October of 1983. You can read about much of the story in Microsoft First Generation by Cheryl Tsang.

Brodie figured out that a document is really just a collection of pieces of text, and that it didn’t really matter where each piece of text is physically located within the document’s file. For that matter, you could have one piece of text that came from one file and another piece of text that came from another file. We refer to this collection of pieces of text as the “piece table.” This design has a number of benefits. For example, if you copy text from one document to another, you don’t have to actually copy the text from one file to another—at least not right away. All you really need to do is copy the appropriate entries in the piece table in the source document to the piece table in the destination document. Of course, you do need to copy the physical text and formatting from one file to the other when you save the destination document, but delaying that physical copy until save time meant that the actual copy/paste could be done very quickly.

This design also made implementing undo rather simple. In fact, according to Brodie, implementing undo was the primary reason to use this design. With this design, all you have to do is create an internal undo document. When the user deletes some text from the current document, for example, you copy the deleted entries from the piece table to the undo document and save some information about where those piece-table entries had been located in the original document. To undo the delete, you just copy the piece-table entries from the undo document back to the original document.

This design does have one problem: where do you put the text that the user types into the document if it doesn’t go into the file that’s behind the document? To solve that, Brodie added something called a “scratch” file, and the scratch file remains a core part of Word’s design to this day. On the Mac, Word creates this file in your TemporaryItems folder. On Mac OS X, this folder is located at /private/tmp/<UID>/TemporaryItems, where “UID” is your user ID number (for most people, that’s 501, but it can be a different number altogether depending on how your user account was created). If start up Word, open the terminal window and get a listing of your TemporaryItems folder, you’ll see a file named something like “Word Work File S_.” There may be a number after the “_” character. That’s Word’s scratch file (the “S” standing for scratch). You might also see one or more files named “Word Work File D_” with some number after the “_” character. This is a back-up copy of a document file (the “D” standing for “Document”).

At this point, we need to fast-forward the story by a decade to the next major feature that brought this problem to the fore: multiple undo. For ten years, Word had undo, but it was just a single-level undo. For Word 6, we added the ability to go back and undo every change you’d made to the document since you first started editing it. And, with Word’s document/file architecture, this wasn’t all that difficult to do: just make the undo document contain multiple records with one record for each change to the main document. It’s a very cool feature, and most of us couldn’t think of how we’d survive without it. But it leads to a problem.

It’s not uncommon for users to make a few edits to a document, save the document, make a few more edits, save the document again, make a few more changes, and continue this process of edit/save for hours on end. Each time you delete text, however, the actual text itself exists in the last-saved file for the document you’re editing, and, with multiple levels of undo, the undo records for text deletions still point back to the last-saved version of the document’s file before you deleted the text. The next time you save, Word can’t close the last-saved version of the file, because the undo document still contains a reference to it. So, if you keep editing and saving, you’ll eventually hit an open file limit. At least this was true of Word 6. It’s changed quite a bit since then.

Arguably, this is something we should have figured out before we shipped Word 6, but, as Chris Mason pointed out, we humans haven’t been fully debugged yet. Moreover, it’s easy to say that one should have thought of a particular interaction in a complex piece of software, but that’s way easier said than done. When you’re implementing any given feature, you’re totally focused on the basic problems involved in the feature itself. To put this into perspective, the person who implemented multiple undo in Word is one of the best developers who has ever worked on Word, and has, since, been recognized as a Microsoft Distinguished Engineer.

The reality is, that we hadn’t realized we’d created this situation when we added multiple levels of undo. Moreover, this problem has several different variations on the basic theme. At this point, the story involves our efforts to understand the nature and scope of the problem, and to come up with the “best” way to fix it. Because of the variations, however, the problem has been like an onion. We’d peel away one layer of the onion, only to find some other variation that we hadn’t, for various reasons, figured out before.

As we weave our way through the rest of the story, there are some important points to keep in mind. The first is that I can’t fix what I can’t see, and, where software bugs are concerned, “seeing” means being able to watch the program execute, via some debugging tool, at the key point in the execution of the code where the problem occurs. In order to do this, I have to have a precise set of steps that consistently reproduces the problem. This not all that different from the problem a mechanic faces when trying to figure out the cause of that mysterious engine noise that only occurs after you’ve been driving the car around town for a few hours.

The second is that this particular problem is a developer’s worst nightmare. The fundamental cause is a basic design decision that you made more than a decade ago, and the only way to really fix it for certain is to rewrite the entire application from the ground up. Since that’s simply not an option for a product that you’ve shipped several times, you’re left with trying to make the problem difficult for most users to run into while trying to also minimize the negative effects if the user should ever run into the problem. This approach can, unfortunately, lead you to believe that you’ve come up with an “optimal” fix only to discover later that there’s another facet you haven’t taken into consideration (because you didn’t even know it existed until you peeled away the previous layer of the onion).

The third point to keep in mind is that we in Mac BU have relatively limited resources. When there’s a problem that’s fundamental to Word itself, we tend to let our Win Word siblings focus on that problem. Our efforts tend to have little chance of adding to their efforts, and this frees us up to focus on problems specific to Mac users. In general, this is the most efficient way to handle problems that our users are having, but there can be instances where there’s a Mac-specific dimension to a problem. As we’ll see soon enough, this particular problem had a Mac-specific dimension that complicated our efforts to fix it, and it took us a while to find that Mac-specific dimension.

Lastly, the fact that Mac Word’s code base has been forked from Win Word’s means that the Win Word people can make a change in the code for one reason, and that change can have other side-effects that we won’t see in the Mac version until we run into some very specific circumstances that show us the different behaviors caused by this change. In this particular case, Win Word added two lines of code in a routine that would seemingly be completely unrelated to this problem, but also made this problem much more difficult for users to run into in Win Word than it was in Mac Word. This one is the last piece (maybe I should say latest piece) in the puzzle that we discovered only a few months ago.

Whew! That’s a lot to keep in the back of our heads, but, nonetheless, let’s rewind back about ten years. Word 6.0 has just shipped on Windows, and we’re pretty happy with people’s reactions to the product. It doesn’t take long, though, for us to figure out that there’s a fly in the ointment. Reports start trickling in about people editing their documents “for a while” at which point they try to save their document and they get a “Disk is full” error. We’d ask people what they were doing, and the response was always some form of vague notion that they’d just been editing their document “for a while.” The precise measurement of “for a while” varied from user to user. For some folks, it was a little over an hour. For others, it was several hours. Reproducing the problem appeared to be highly dependant upon the user’s work habits.

After several months of trying to figure out the problem, someone in testing wrote a macro that inserted a large amount of text into the document and then, in a loop, replace successive words within the document saving it after each replace. Run this macro for a while, and you get a “Disk is full” error on one of the saves, at which point you can no longer save your document. Cool! We now have steps that reproduce the problem.

So, this document got handed off to a developer, who then fired up Word under the debugger, opened the document and ran the macro. The problem “reproduced,” but, for reasons that weren’t apparent at the time, the error that the developer ran into was subtly different from the error that the tester ran into. The developer thought about the problem he was seeing, and came up with one of those “optimal” fixes I mentioned above. It was the “right” fix in terms of the problem the developer saw, but it wasn’t the “right” fix for the problem that the tester saw.

What was this subtle difference between what the developer saw and what the tester saw? As I mentioned above, the basic theme of the problem is to hit an open file limit. In this case, there are two limits: Word’s internal open file limit and the OS’ open file limit. It turns out that the debugger bumps the OS’ open file limit from what it would normally be when you run Word outside the debugger. When the tester ran the macro, Word hit the OS’ open file limit. When the developer ran the macro, with Word running under the debugger, Word ran into its own, internal, file limit.

After a few iterations of the tester saying, “Sorry, but the bug’s not fixed yet,” and the developer saying, “What are you talking about? I don’t see the problem!” they both figured out that they were seeing different errors. Crap! The problem only reproduces when you’re not running under the debugger, which removes the one case where the developer can actually see what’s going on. At this point, we have yet to figure out that the problem involves hitting the OS’ open file limit. At this point, though, the developer isn’t completely in the dark, and comes up with a fix for the tester’s problem.

As I pointed out, the problem involves the undo document having a reference to the previously saved-version of the document’s file. The developer’s original fix was to add some code, in the case where Word hit its internal open file limit, that would basically remove everything from the undo document (what we refer to as “nuking the undo stack”). Nuking the undo stack allows the save to proceed, because Word can now close the open files that were referenced by the undo document. However, since the tester was seeing a different error, the developer’s fix didn’t handle that case.

Nonetheless, the developer took a different approach. Knowing that the undo document was very likely to be involved, one could walk through the undo document, and copy the text for any pieces that pointed to the previously-saved version of the document’s file to the scratch file. He coded up the solution, and handed a buddy-build off to the tester. The tester ran the macro, and the problem was fixed. The first layer of the onion had been peeled away, but the fix still wasn’t an “optimal” fix. As it stood, the chances that a user would run into the problem had been greatly reduced, but we still hadn’t dealt with the “minimize the damage if they do hit it” side of the issue. That’s because, at this point, we had yet to understand that the problem outside the debugger had to do with the OS’ open file limit. Because this problem wouldn’t reproduce under the debugger, the developer had no way of knowing exactly where the failure was occurring. Without knowing that, the developer didn’t know where to add the code that would “nuke the undo stack.”

To give you a sense of the time frame, this fix was ported from Win Word to Mac Word during the Office 2001 development cycle, and was back-ported into Word 98 for a service release that was done not too long after that. It’s also at this point where the Win Word and Mac Word stories diverge. There are two reasons for this. The first is that this was the point in time where Win Word got that two-line code change that I mentioned above. The second is that the open file limit under Mac OS is different than it is under Windows. I might be mistaken on this point, but I think the open file limit under Mac OS X is different from the limit under Mac OS 9 as well.

At this point, we still didn’t know that the basic problem involved hitting the OS’ open file limit. After a while, though, we did know that Mac Word users were seeing this problem way more often than Win Word users. In fact, the difference was enough for Mac Word testers to start investigating the problem directly. One of the things we did know is that the problem involved file references in the undo document. So, we came up with a variation of the original fix.

In order to understand this, we have to understand a basic principle of fixes. You make the simplest code change required to fix the problem. This reduces the chances that the fix will cause some other problem that is, potentially, worse than the one you’re trying to fix. When you’re mucking about with the locations where data is stored in files, the potential for catastrophic problems resulting from your fix is high. In that sense, the original fix for this problem was limited to copying what might be known as “simple” pieces. A “simple” piece has only text. A “complex” piece might have a graphic, or it might involve a field in the document, both of which are likely to have data in the file in addition to the text itself.

With this in mind, for Mac Word X, we modified the notion of what would be a “simple” piece of text for the sake of deciding whether or not to copy a piece from the previously-saved document’s file over to the scratch file. To view this in a slightly different way, we made the code that copies undo document referents more aggressive. This resolved another test case that the Mac Word testers had developed, again using a slightly different macro that would eventually cause the “Disk is full” error to occur. This fix didn’t actually make it into the shipping release of Mac Office X, but it was included in a subsequent SR (I don’t recall specifically which one).

At this point, we still don’t know that the problem involves the OS’ open file limit. That discovery didn’t happen until this past summer when, through the very persistent efforts of Mac Word’s current lead tester, we were able to use some tools on Mac OS X to figure out exactly what was happening. While we were able to verify this, we still didn’t know the exact location where Word was failing to open a file due to having hit the OS’ open file limit. Again, we still can’t get this to reproduce under the debugger, and there are a couple of places in the save process where it can fail because the OS won’t let Word open the file. So, rather than scatter fixes all over the place, we went with the sure fix: lower Word’s internal open file limit so we hit it before we hit the OS’ open file limit. This allows the code that nukes the undo stack to kick in, and then save the succeeds.

This brings us to late February/early March of this year, and the discussion I’d had with Pierre. While we still can’t reproduce the actual file open failure under the debugger, we now have enough information about what causes the problem to be able to predict when the failure will eventually occur. From that, we knew enough about the bug for me to believe that Pierre shouldn’t still be hitting that “Disk is full” save error in the version of Word he was using. Yet, he was still running into the problem.

That was the bad news: there was something about this problem that we still didn’t fully understand. However, armed with a sample document and the ability to predict when the error will occur, I could do something we’d never been able to do before: set up Word under the debugger, perform some steps in Word to see if those steps caused the predictive condition to occur, and set breakpoints that would tell me exactly why we weren’t able to copy pieces from the undo document over to the scratch file.

This is where I discovered those two lines of code that had been added to Win Word so long ago, yet hadn’t been added to Mac Word. When Word lays out a page in page layout view and a header or footer is visible, it updates any fields in the header or footer. If you have, say, a page field in the visible header/footer, Word will update that field. This is particularly necessary when you have the footer of one page and the header of the following page both visible in the document window. Word has to layout two pages in the same update, so it updates fields for the first page footer, lays out that page, then updates the fields for the next page header and lays out that page.

Now, why would this result in a field being copied over to the undo document? Well, Word has something called “auto undo tracking.” Basically, when you’re typing, Word automatically tracks the changes you’re making until you do something that causes Word to close out the “typing” undo record. You can see this when you click on the “Undo” dropdown on the standard toolbar. You’ll see “Typing <text you typed>” at various locations in the dropdown interspersed with other actions you’ve taken.

The two lines of code that were added to Win Word paused automatic undo tracking while updating these fields in the header or footer during page layout, then un-paused automatic undo tracking once the field update was finished. Ugh! How, on earth, were we to ever figure out that these two lines of code were the primary reason Win Word users weren’t seeing this problem nearly as often as Mac Word users? In any event, if you’ve stayed with me long enough, here’s a tip you can use until we release an SR of Word X (or earlier) with this fix. If you have a document that has headers and footers with page fields in them, do your editing in Normal view, and you’ll likely never hit the “Disk is full” save error.

Right about now, you’re probably asking, “Why did it take so long to figure out what was up with this?” Well, you might as well ask why police departments continue to have a large number of unsolved crimes on the books. The issue is the same: the investigation stalls for the lack of any further leads to follow. For the same reason that the police can’t just go out and start arresting anyone who might be a suspect, we can’t go scattering potential fixes throughout the code. Until we figure out what the precise nature of the problem is, we need leads that we can follow. The mere fact that you’re running into a particular problem isn’t a lead that I can follow. Specific details about potential suspects, however, are leads I can follow. When it comes to software problems, leads I can follow consist of information that helps me to reproduce the problems consistently.

And, always remember that I can’t fix what I can’t see. I have to be able to reproduce the problem while being able to run some kind of diagnostic tool. The key to fixing a bug is predictability. Without predictability, I can’t fix it, because without predictability I have no way to understand how the complex interactions in modern software cause the specific problem to occur.

Rick