February, 2009

  • The Old New Thing

    Why do my file properties sometimes show an Archive check box and sometimes an Advanced button?


    When you view the properties of a file and go to the General page, there are some check boxes at the bottom for file attributes. There's one for Read-only and one for Hidden, and then it gets weird. Sometimes you get Archive and sometimes you get an Advanced button. What controls which one you get?

    It depends on whether there is anything interesting in the Advanced dialog.

    If the volume supports either compression or encryption (or both), then you will get an Advanced dialog with check boxes for Archive, Compress and Encrypt. On the other hand, if the volume supports neither compression nor encryption, then you will just get an Archive check box, since it looks kind of silly having an Advanced button that shows you a dialog box with just one check box on it. (Note that these features can also be disabled by group policy, so it's not purely a file system decision.)

    In Windows, the most commonly encountered file system that does not support compression or encryption is probably FAT, and the most commonly encountered one that does is almost certainly NTFS, so in a rough sense, you can say that FAT gives you an Archive check box and NTFS gives you an Advanced button.

  • The Old New Thing

    Guest TCP psychic debugging: Why the remote server keeps RSTing the connection


    My colleague Keith Moore (who occasionally comments on this site) shared with me one of his recent triumphs of psychic debugging. First the question:

    The customer is getting an RST response from IIS and they would like to know why. Here is a fragment from a network capture that illustrates the problem. (Fragment deleted.) The full capture is available on ...

    Keith didn't look at the full capture; he barely even glanced at the fragment. Because his psychic powers told him the answer:

    The amount of data they are sending does not match the Content-Length header.

    He goes on to explain:

    If there's unread data pending when a connection is closed, it is automatically reset.

    Sure enough, the customer was uploading data to the IIS server, specifying a Content-Length of 1289 but actually sending 1291 bytes. The server stopped reading after 1289 bytes (respecting the Content-Length), and the client was upset because Hey, you forgot two bytes!

    As is occasionally the case with these types of problems, the misunderstanding goes deeper than the question itself. The customer replied, "I don't understand why the server sees 1291 bytes. If you look at the network capture, the client machine sends two frames, one of size 1289 and one of size 2. Both frames have the correct size specified in the frame header. The size of the frames is correct; I don't see what the problem is. I mean, sure, if the first frame had a header specifying 1289 bytes and the payload contained 1291 bytes, then that would be a problem, but that is not the case here."

    The problem is not with the frames; it's at a higher level. The client machine promised via the HTTP protocol to send 1289 bytes, and it sent 1289 bytes, and then sent two more bytes. The reset occurs because the client machine lied about how many bytes it was going to send. The frames themselves are fine; the problem is that they are the wrong frames.

  • The Old New Thing

    Another Seattle bus tool: One Bus Away


    I was recently tipped off to yet another Seattle bus tool: One Bus Away, which not only provides real-time bus arrival information for every stop in the Metro system, but does so in a variety of formats. You can use the Web-based interface (which is close to what you can already get from the Tracker Location View), but the real magic is that the information is also available via SMS, Web-enabled handheld devices, and—most important to me—any mobile phone.

    I have One Bus Away on my speed dial, and I've set bookmarks on the stops I use the most, so I'm never more than a few taps of the phone away from knowing exactly when my bus will arrive. This is a fantastic anxiety-reliever when I arrive at my stop very close to the posted time and am left wondering, "Did I just miss the bus? Or is it running late?"

    Bonus trivia: I had assumed that the Metro buses transmitted their locations to the central computer with the assistance of GPS technology, but in fact, it's much lower-tech: They just transmit the current odometer reading. Since buses travel along a fixed route, knowing the odometer is enough to give you the bus location. (Assuming the bus has not diverged from the official route...)

    Bonus update: I used One Bus Away just this morning. I was running a bit too tight on my morning schedule, and as I hopped on my bicycle, I called One Bus Away to find out how much time I had to get to my normal bus stop. It was a bit too close for comfort, so I took a detour to a different bus stop (a little further down the route to buy me some more time) and caught the bus easily.

  • The Old New Thing

    Why doesn't the file system have a function that tells you the number of files in a directory?


    There are any number of bits of information you might want to query from the file system, such as the number of files in a directory or the total size of the files in a directory. Why doesn't the file system keep track of these things?

    Well, of course, one answer is that it certainly couldn't keep track of every possible fragment of information anybody could possibly want, because that would be an infinite amount of information. But another reason is simply a restatement of the principle we learned last time: Because the file system doesn't keep track of information it doesn't need.

    The file system doesn't care how many files there are in the directory. It also doesn't care how many bytes of disk space are consumed by the files in the directory (and its subdirectories). Since it doesn't care, it doesn't bother maintaining that information, and consequently it avoids all the annoying problems that come with attempting to maintain the information.

    For example, one thing I noticed about many of the proposals for maintaining the size of a directory in the file system is that very few of them addressed the issue of hard links. Suppose a directory contains two hard links to the same underlying file. Should that file be double-counted? If a file has 200 hard links, then a change to the size of the file would require updating the size field in 200 directories, not just one as one commenter postulated. (Besides, the file size isn't kept in the directory entry anyway.)

    Another issue most people ignored was security. If you're going to keep track of the recursive directory size, you have to make sure to return values consistent with each user's permissions. If a user does not have permission to see the files in a particular directory, you'd better not include the sizes of those files in the "recursive directory size" value when that user goes asking for it. That would be an information disclosure security vulnerability. Now all of a sudden that single 64-bit value is now a complicated set of values, each with a different ACL that controls which users each value applies to. And if you change the ACL on a file, the file system would have to update the file sizes for each of the directories that contains the file, because the change in ACL may result in a file becoming visible to one user and invisible to another.

    Yet another cost many people failed to take into account is just the amount of disk I/O, particular writes, that would be required. Generating additional write I/O is a bad idea in general, particularly on media with a limited number of write cycles like USB thumb drives. One commenter did note that this metadata could not be lazy-written because a poorly-timed power outage would result in the cached value being out of sync with the actual value.

    Indeed the added cost of all the metadata writes is one of the reasons why Windows Vista no longer updates the Last Access time by default.

    Bonus chatter: My colleague Aaron Margosis points out a related topic over on the ntdebugging blog: NTFS Misreports Free Space? on the difficulties of accurate accounting, especially in the face of permissions which don't grant you total access to the drive.

  • The Old New Thing

    Don't keep track of information you don't need


    This is sort of an extreme corollary to Don't save anything you can recalculate. Sure, it sounds like such an obvious principle, but many people fail to understand its consequences.

    Let's look at the principle again. Don't keep track of information you don't need. I remember being asked to look at a customer's program, and one thing that struck me was that the program had a bazillion different flag variables that it spent a lot of time setting and clearing. Here's an oversimplified example:

    void CConfiguration::ShowDialog(HWND hwnd)
      m_fShowingDialog = true;
      m_fShowingDialog = false;
      m_fDialogHasBeenShown = true;

    A naïve reading of the code would lead one to believe that the main purpose of the program was to set and clear flags! Upon closer inspection, I found that most of the flag variables were write-only. Although the code went to great pains to update the flags as it ran, there was no code that actually cared whether the flag was set or not.

    I asked the customer liaison about this, thinking that maybe I missed something. Maybe the variables were being used in a subtle way that I failed to appreciate. Maybe they were used during debugging.

    The customer liaison got back to me. "When I asked them about it, they just said, 'Ah, yes.' Apparently that code was written by one of their guys named Bob. I guess he likes to set and clear flags a lot."

    But is there any code that actually checks the values of those flags?

    "Probably not. He just set and cleared those flags because he figured maybe someday he might need them."

    Indeed, once in a while, he actually tested one of the dozens of flags he spent most of his time setting and resetting. But the vast majority of the flags were write-only. (And since most of the flags went unused, I suspect that many of them weren't even accurate.)

    Obviously, this is an extreme case of write-only variables, but it illustrates the point. Next time, we'll look at one of the consequences of the principle of not keeping track of information you don't need.

  • The Old New Thing

    The love bus, also known as Metro Bus Route number 308


    Four years ago, Troy Kleweno spotted Christine Hsieh on the 308 bus. He saw her twice a day, once in the morning on the way to work, and again in the evening on the way home. His co-workers teased him about the "bus girl" he pined after for six months before he finally said hello. That led to a date, and soon they were a couple.

    It's only fitting then that when Troy decided that it was time to propose marriage, that he do it on the 308 bus. Watch it here.

    Happy Valentine's Day, everybody.

  • The Old New Thing

    The checkbox: The mating call of the loser


    (Cultural note: The phrase the mating call of the loser is a term of derision. I used it here to create a more provocative headline even though it's stronger than I really intended, but good writing is bold.)

    When given a choice between two architectures, some people say that you should give users a checkbox to select which one should be used. That is the ultimate cowardly answer. You can't decide between two fundamentally different approaches, and instead of picking one, you say "Let's do both!", thereby creating triple, perhaps quadruple the work compared to just choosing one or the other.

    It's like you're remodeling a book library and somebody asks you, "Should we use Dewey Decimal or Library of Congress?" Your answer, "Let's do both and let the user choose!"

    Imagine if there were a checkbox somewhere in the Control Panel that let you specify how Windows XP-styled controls were implemented. Your choices are either to require applications to link to a new UxCtrl.DLL (let's call this Method A) or to link to COMCTL32.DLL with a custom manifest (let's call this Method B). Well, it means that every component that wanted styled common controls would have to come in two versions, one that linked to UxCtrl and used the new class names in its dialog boxes and calls to CreateWindow, and one that used a manifest and continued to use the class names under their old names.

    #ifdef USE_METHODA
     hwnd = CreateWindow(TEXT("UxButton"), ...);
     hwnd = CreateWindow(TEXT("Button"), ...);
    DLG_WHATEVER DIALOG 36, 44, 230, 94
    CAPTION "Whatever"
    #ifdef USE_METHODA
        CONTROL         "Whatever",IDC_WHATEVER,"UxButton",
                        BS_AUTOCHECKBOX | WS_TABSTOP, 14,101,108,9
        CONTROL         "Whatever",IDC_WHATEVER,"Button",
                        BS_AUTOCHECKBOX | WS_TABSTOP, 14,101,108,9

    At run time, every program would have to check this global setting and spawn off either the "Method A" binary or the "Method B" binary.

    Now you might try to pack this all into a single binary with something like this:

     hwnd = CreateWindow(TEXT("Button"), ...);
    } else {
     hwnd = CreateWindow(TEXT("UxButton"), ...);
    } else {

    But it's not actually that simple because a lot of decisions take place even before your program starts running. For example, if your program specifies a load-time link to UXCTRL.DLL, then that DLL will get loaded before your program even runs, even if the system switch is set to use Method B. A single-binary program that tries to choose between the two methods at runtime will have to do some activation context juggling and delay-loading. Hardly a slam dunk.

    Okay, so now you have two versions of every program. And you also have to decide what should happen if somebody writes and ships a program that uses Method A exclusively, even when the system switch is set to Method B. Does everything still work within that program as if Method A were the system setting, while the rest of the system uses Method B? (If you go this route, then you've completely undermined the point of Method B. The whole point of Method B is to allow programs that rely on specific class names to continue working, but this rogue Method A program is running around using the wrong class names!)

    Now the entire operating system and application compatibility work needs to be done with the checkbox set both to Method A and to Method B, because the compatibility impact of each of the methods is quite different. Okay, that's double the work. Where is triple and quadruple?

    Well, the two different versions of the program need to be kept in sync, since you want them to behave identically. This can of course be managed with judicious use of #ifdefs or runtime branches. But you have to remember both ways of doing things and be mindful of the two method each time you modify the program. Somebody else might come in and "fix a little bug" or "add a little feature" to your program, unaware of how your program manages the shuffle of Method A versus Method B. The mental effort necessary to remember two different ways of doing the same thing plus having to expend that effort to correct mistakes in the code, that's the triple.

    The quadruple? I'm not sure, maybe the ongoing fragility of such a scheme, especially one that, at the end of the day, is a choice between two things that have no real impact on the typical end user.

    Engineering is about making tradeoffs. If you refuse to make the choice, then you're taking the cowardly route and ultimately are creating more work for your team. Instead of solving problems, you're creating them.

    All because you're too chicken to make a hard decision.

  • The Old New Thing

    Superthunderstingcar is go!


    If you are an old geezer, you'll remember supermarionated shows, the most famous of which is Thunderbirds. And if you're not an old geezer, then this amazing spoof by Peter Cook and Dudley Moore will mean nothing to you. (You will also have no idea who Peter Cook and Dudley Moore are.)

  • The Old New Thing

    What does the COM Surrogate do and why does it always stop working?


    The dllhost.exe process goes by the name COM Surrogate and the only time you're likely even to notice its existence is when it crashes and you get the message COM Surrogate has stopped working. What is this COM Surrogate and why does it keep crashing?

    The COM Surrogate is a fancy name for Sacrificial process for a COM object that is run outside of the process that requested it. Explorer uses the COM Surrogate when extracting thumbnails, for example. If you go to a folder with thumbnails enabled, Explorer will fire off a COM Surrogate and use it to compute the thumbnails for the documents in the folder. It does this because Explorer has learned not to trust thumbnail extractors; they have a poor track record for stability. Explorer has decided to absorb the performance penalty in exchange for the improved reliability resulting in moving these dodgy bits of code out of the main Explorer process. When the thumbnail extractor crashes, the crash destroys the COM Surrogate process instead of Explorer.

    In other words, the COM Surrogate is the I don't feel good about this code, so I'm going to ask COM to host it in another process. That way, if it crashes, it's the COM Surrogate sacrificial process that crashes instead of me process. And when it crashes, it just means that Explorer's worst fears were realized.

    In practice, if you get these types of crashes when browsing folders containing video or media files, the problem is most likely a flaky codec.

    Now that you know what the COM Surrogate does, you can answer this question from a customer:

    I'm trying to delete a file, but I'm told that "The action can't be completed because the file is open in COM Surrogate." What is going on?
  • The Old New Thing

    In order to serve you bett... wait, it really is better: Fuel surcharges


    Normally, the phrase In order to serve you better means that you're about to get screwed.

    Imagine my surprise to discover that United has stopped imposing a fuel surcharge for flights between Canada and the United States due to the decrease in fuel prices. But wait, that's only for flights between Canada and the United States. Flights within the United States appear to have the surcharge in place as usual.

    Hey, I've got an idea. How about getting rid of this surcharge nonsense and just raise the price of the ticket? Naw, that's just crazy talk.

    I can see the Calvin & Hobbes cartoon already. "Lemonade, 5¢ a glass (plus 25¢ water surcharge, 50¢ sugar surcharge, 75¢ rent reclamation fee, $1.50 Moe appeasement fee, 25¢ capital depreciation fee...)"

Page 2 of 4 (32 items) 1234