September, 2010

  • The Old New Thing

    Flushing your performance down the drain, that is


    Some time ago, Larry Osterman discussed the severe performance consequences of flushing the registry, which is a specific case of the more general performance catch: Flushing anything will cost you dearly.

    A while back, I discussed the high cost of the "commit" function, and all the flush-type operations turn into a commit at the end of the day. FlushViewOfFile, [see correction below] FlushFileBuffers, RegFlushKey, they all wait until the data has been confirmed written to the disk. If you perform one of these explicit flush operations, you aren't letting the disk cache do its job. These types of operations are necessary only if you're trying to maintain transactional integrity. If you're just flushing the data because "Well, I'm finished so I want to make sure it gets written out," then you're just wasting your (and the user's) time. The data will get written out, don't worry. Only if there is a power failure in the next two seconds will the data fail to get written out, but that's hardly a new problem for your program. If the power went out in the middle of the call to FlushFileBuffers (say, after it wrote out the data containing the new index but before it wrote out the data the index points to), you would've gotten partially-written data anyway. If you're not doing transactional work, then your call to FlushFileBuffers didn't actually fix anything. You still have a window during which inconsistency exists on the disk.

    Conclusion: View any call to FlushViewOfFile, [see correction below] FlushFileBuffers, and RegFlushKey with great suspicion. They will kill your program's performance, and even in the cases in which you actually would want to call it, there are better ways of doing it nowadays.

    More remarks on that old TechNet article: The text for the Enable advanced performance check box has been changed in Windows 7 to something that more accurately describes what it does: Turn off Windows write-cache buffer flushing on the device. There's even explanatory text that explains the conditions under which it would be appropriate to enable that setting:

    To prevent data loss, do not select this check box unless the device has a separate power supply that allows the device to flush its buffer in case of power failure.

    Hard drives nowadays are more than just platters of magnetic media. There's also RAM on the hard drive circuit board, and this RAM is used by the hard drive firmware as yet another buffer. If the drive is told, "Write this data to the hard drive at this location," the drive copies the data into its private RAM buffer and immediately returns a successful completion code to the operating system. The drive then goes about seeking the head, looking for the sector, and physically writing out the data.

    When your program issues a write command to the file system (assuming that file system buffering is enabled), the write goes into the operating system disk cache, and periodically, the data from the operating system disk cache is flushed to the hard drive. As we saw above, the hard drive lies to the operating system and says "Yeah, I wrote it," even though it hasn't really done it yet. The data the operating system requested to be written is just sitting in a RAM buffer on the hard drive, that in turn gets flushed out to the physical medium by the hard drive firmware.

    If you call one of the FlushBlahBlah functions, Windows flushes out its disk cache buffers to the hard drive, as you would expect. But as we saw above, this only pushes the data into the RAM buffer on the hard drive. Windows understands this and follows up with another command to the hard drive, "Hey, I know you're one of those sneaky hard drives with an internal RAM buffer. Yes, I'm talking to you; don't act all innocent like. So do me a favor, and flush out your internal RAM buffers too, and let me know when that's done." This extra "I know what you did last summer" step ensures that the data really is on physical storage, and the FlushBlahBlah call waits until the "Okay, I finished flushing my internal RAM buffer" signal from the hard drive before returning control to your program.

    This extra "flush out your internal RAM buffer too" command is the right thing to do, but it can safely be skipped under very special circumstances: Consider a hard drive with a power supply separate from the computer which can keep the drive running long enough to flush out its internal RAM, even in the event of a sudden total loss of external power. For example, it might be an external drive with a separate power supply that is hooked up to a UPS. If you have this very special type of set-up, then Windows doesn't need to issue the "please flush out your internal RAM buffers too" command, because you have a guarantee that the data will make it to the disk no matter what happens in the future. Even if a transformer box explodes, cutting off all power to your building, that hard drive has enough residual power to get the data from the internal RAM buffer onto the physical medium. Only if your hard drive has that type of set-up is it safe to turn on the Turn off Windows write-cache buffer flushing on the device check box.

    (Note that a laptop computer battery does not count as a guarantee that the hard drive will have enough residual power to flush its RAM buffer to physical media. You might accidentally eject the battery out of your laptop, or you might let your battery run down completely. In these cases, the hard drive will not have a chance to finish flushing its internal RAM buffer.)

    Of course, if the integrity of your disks is not important then go ahead and turn the setting on even though you don't have a battery backup. One case where this may be applicable is if you have a dedicated hard drive you don't care about losing if the power goes out. Many developers on the Windows team devote an entire hard drive to holding the files generated by a build of the operating system. Before starting a build, they reformat the drive. If the power goes out during a build, they'll just reformat the drive and kick off another build. In this case, go ahead and check the box that says Enable advanced performance. But if you care about the files on the drive, you shouldn't check the box unless you have that backup power supply.

  • The Old New Thing

    It rather involved being on the other side of this airtight hatchway: If you grant users full control over critical files, then it's not the fault of the system for letting users modify them


    Today's dubious security vulnerability is another example of If you reconfigure your computer to be insecure, don't be surprised that there's a security vulnerability.

    This example comes from by an actual security vulnerability report submitted to Microsoft:

    I have found a critical security vulnerability that allows arbitrary elevation to administrator from unprivileged accounts.

    1. Grant Full Control of the Windows directory (and all its contents and subdirectories) to Everyone.
    2. Log on as an unprivileged user and perform these actions...

    I can just stop there because your brain has already stopped processing input because of all the alarm bells ringing after you read that first step. That first step gives away the farm. If you grant control to the entire contents of the Windows directory to non-administrators, then don't be surprised that they can run around and do bad things!

    "If I remove all the locks from my doors, then bad guys can steal my stuff."

    Yeah, so don't do that. This is not a security vulnerability in the door.

    Bonus chatter: There are many variations on this dubious security vulnerability. Actual vulnerability reports submitted to Microsoft include
    • "First, grant world-write permission to this registry key..."
    • "First, reconfigure Internet Explorer to allow scripting of ActiveX controls not marked safe for scripting..."
    • "On a compromised machine, you can..."

    That last one is impressive for its directness. "Starting on the other side of this airtight hatchway..."

  • The Old New Thing

    What's up with the strange treatment of quotation marks and backslashes by CommandLineToArgvW


    The way the CommandLineToArgvW function treats quotation marks and backslashes has raised eyebrows at times. Let's look at the problem space, and then see what algorithm would work.

    Here are some sample command lines and what you presumably want them to be parsed as:

    Command line Result
    program.exe "hello there.txt" program.exe
    hello there.txt
    program.exe "C:\Hello there.txt" program.exe
    C:\Hello there.txt

    In the first example, we want quotation marks to protect spaces.

    In the second example, we want to be able to enclose a path in quotation marks to protect the spaces. Backslashes inside the path have no special meaning; they are copied as any other normal character.

    So far, the rule is simple: Inside quotation marks, just copy until you see the matching quotation marks. Now here's another wrinkle:

    Command line Result
    program.exe "hello\"there" program.exe

    In the third example, we want to embed a quotation mark inside a quotated string by protecting it with a backslash.

    Okay, to handle this case, we say that a backslash which precedes a quotation mark protects the quotation mark. The backslash itself should disappear; its job is to protect the quotation mark and not to be part of the string itself. (If we kept the backslash, then it would not be possible to put a quotation mark into the command line parameter without a preceding backslash.)

    But what if you wanted a backslash at the end of the string? Then you protect the backslash with a backslash, leaving the quotation mark unprotected.

    Command line Result
    program.exe "hello\\" program.exe

    Okay, so what did we come up with?

    We want a backslash before a quotation mark to protect the quotation mark, and we want a backslash before a backslash to protect the backslash (so you can end a string with a backslash). Otherwise, we want the backslash to be given no special treatment.

    The CommandLineToArgvW function therefore works like this:

    • A string of backslashes not followed by a quotation mark has no special meaning.
    • An even number of backslashes followed by a quotation mark is treated as pairs of protected backslashes, followed by a word terminator.
    • An odd number of backslashes followed by a quotation mark is treated as pairs of protected backslashes, followed by a protected quotation mark.

    The backslash rule is confusing, but it's necessary to permit the very important second example, where you can just put quotation marks around a path without having to go in and double all the internal path separators.

    Personally, I would have chosen a different backslash rule:

    Warning - these are not the actual backslash rules. These are Raymond's hypothetical "If I ran the world" backslash rules.

    • A backslash followed by another backslash produces a backslash.
    • A backslash followed by a quotation mark produces a quotation mark.
    • A backslash followed by anything else is just a backslash followed by that other character.

    I prefer these rules because they can be implemented by a state machine. On the other hand, it makes quoting regular expressions a total nightmare. It also breaks "\\server\share\path with spaces", which is pretty much a deal-breaker. Hm, perhaps a better set of rules would be

    Warning - these are not the actual backslash rules. These are Raymond's second attempt at hypothetical "If I ran the world" backslash rules.

    • Backslashes have no special meaning at all.
    • If you are outside quotation marks, then a " takes you inside quotation marks but generates no output.
    • If you are inside quotation marks, then a sequence of 2N quotation marks represents N quotation marks in the output.
    • If you are inside quotation marks, then a sequence of 2N+1 quotation marks represents N quotation marks in the output, and then you exit quotation marks.

    This can also be implemented by a state machine, and quoting an existing string is very simple: Stick a quotation mark in front, a quotation mark at the end, and double all the internal quotation marks.

    But what's done is done, and the first set of backslash rules is what CommandLineToArgvW implements. And since the behavior has been shipped and documented, it can't change.

    If you don't like these parsing rules, then feel free to write your own parser that follows whatever rules you like.

    Bonus chatter: Quotation marks are even more screwed up.

  • The Old New Thing

    Yes, the Windows 7 beta wallpaper was a picture of a betta fish


    It wasn't long before people realized that the fish in the default wallpaper for the Windows 7 beta was a betta fish. This was intended to be a cute little pun, though some people took the semiotics to an extreme,

    Dude, this is Windows, not The Da Vinci Code. It's not like the people who chose the wallpaper are using a backchannel to pass secret messages to you like "I know I'm not supposed to tell you, but here's the Windows product schedule" or "Help, I'm trapped in a wallpaper factory!" They're just having a bit of fun.

    I have yet to see anybody point out that the fish was blowing seven bubbles. And no, I don't know what it "signifies".

    (I always wonder about the people who claim that the government embeds subtle messages about their complex conspiracies in plain sight. If you want to keep a secret, you don't run around talking about it!)

  • The Old New Thing

    How do I create a UNC to an IPv6 address?


    Windows UNC notation permits you to use a raw IPv4 address in dotted notation as a server name: For example, net view \\ will show you the shared resources on the computer whose IP address is But what about IPv6 addresses? IPv6 notation contains colons, which tend to mess up file name parsing since a colon is not a valid character in a path component.

    Enter the domain.

    Take your IPv6 address, replace the colons with dashes, replace percent signs with the letter "s", and append This magic host resolves back to the original IPv6 address, but it avoids characters which give parsers the heebie-jeebies.

    Note that this magic host is resolved internally by Windows and never hits the network. It's sort of a magic escape sequence.

  • The Old New Thing

    What is the effect of the /LARGEADDRESSAWARE switch on a DLL?


    Nothing. Large-address-awareness is a process property, which comes from the EXE.

  • The Old New Thing

    Was there really an Opera billboard outside Microsoft main campus?


    In an interview with the Seattle Times, Rod Hamlin of Opera Software claimed,

    We put a big red billboard out by Microsoft last year that said, "Want to be a real Internet explorer?" We got some interesting feedback on that. All of the AT&T executives could see it and all the Microsoft guys driving back home past Marymoor Park.

    Okay, so where was this billboard? He says it was near Marymoor Park, and that it could be seen from AT&T executive offices, which makes sense so far because AT&T Wireless has offices in the Redmond Town Center business and shopping center, which lies right across the highway from the park.

    But then things fall apart. First of all, there is no billboard stand anywhere along the stretch of highway that goes between Marymoor Park and Redmond Town Center.

    Second, if you go to the regulations governing highway advertising in the State of Washington [easier-to-read PDF version], section 47.42.040 describes the types of signs allowed, and the alleged Opera billboard does not appear to be any of permissible types. (The closest match would be 47.42.040(4), if Opera had offices within twelve miles of Redmond Town Center.)

    Third, you'd think there'd be plenty of pictures of an advertising campaign this cheeky. But I haven't been able to find any online. POIDH.

    Here's an actual cheeky prank (and the response). The Internet Explorer team since learned their lesson, and now they send congratulatory cake.

    Stop the presses: A colleague of mine says that he saw the sign. But it wasn't a billboard. Actually, it wasn't even a sign. It was a sponsorship banner hung on the fence of one of the sports fields at Marymoor Park, the sort of sign that more traditionally might read Bob's Auto Repair proudly supports youth sports. Go Mustangs!) I asked him why he didn't take a picture. "I guess we've all become pretty jaded. Either that or everybody figured somebody else would take a picture (so then nobody did)."

    After our conversation, he went and took a picture.

  • The Old New Thing

    Does anybody actually like Brazil nuts?


    Brazil nuts are perhaps best known for floating to the top of a jar of mixed nuts. According to Wikipedia,¹ the reason for the phenomenon is not well understood.

    At least in my house, the reason for the phenomenon is quite clear: Brazil nuts float to the top because nobody in my house likes Brazil nuts. When you reach in and grab a handful of nuts, you toss the Brazil nuts back into the jar, which is why they end up on top.

    A few months ago, I asked "Does anybody actually like Brazil nuts?" A lot of people agreed with my opinion of them, but there was a notable dissenter: Larry Osterman. There is now a symbiotic relationship between my family and Larry Osterman. Every so often, I will skim off the Brazil nuts and bring them to Larry. Both sides win: We get rid of the annoying Brazil nuts, and Larry gets a small bag of Brazil nuts.

    ¹ Translation: I could just as well be making this up.

  • The Old New Thing

    Speculation around Microsoft Company Meeting 2010


    Today is Microsoft's annual Company Meeting.

    Back in August, the Real Estate and Facilities department sent a message to our group of buildings to inform us that the locker rooms would be closed "due to the filming of an upcoming corporate initiative."

    Speculation swirled as to what sort of "upcoming corporate initiative" would require filming in a locker room. The Company Meeting was only a month away, and I suggested that Steve Ballmer might be filming a (shudder) parody of the Old Spice Guy commercial.

    Picking up the ball, one of my colleagues wrote the proposed script for this parody:

    Hello developers. How are you? Fantastic.

    Does your CEO look like me? Yes.

    Can he code like me? Yes.

    Should you be using Windows 7? I don't know.

    Do you like the feel of productivity? Do you want an operating system that can stream your favorite movies while you tweet about the interesting article you're currently reading on Wikipedia? Of course you do.

    SWANDIVE into the best user experience of your life!

    So should you be using Windows 7? You tell me.

    Here's the commercial being parodied, and the first commercial in the series.

  • The Old New Thing

    Where did my mail control panel icon go?


    A customer ran into the following problem:

    I was trying to add another email account to Outlook, and the instructions say that I should go to the mail icon in the Control Panel, which to my surprise is nowhere to be found! How can I figure out what went wrong?

    A little bit of psychic debugging will solve this.

    The customer was running Windows Vista, 64-bit edition. On 64-bit versions of Windows XP and Windows Vista, the Control Panel shows only 64-bit control panels. The 32-bit control panels are off in a separate 32-bit control panel, which you can find by clicking the View 32-bit Control Panel Items icon.

    The separation of 32-bit and 64-bit control panels is an unfortunate consequence of the rule that 64-bit processes cannot load 32-bit DLLs and vice versa. On 64-bit Windows, Explorer is a 64-bit process, which means that it can't load traditional 32-bit control panels. (Recall that control panel applications run as DLLs inside the host process.) Therefore, Explorer has to pass off the work of working with 32-bit control panel applications to a 32-bit alter ego process.

    Fortunately, Windows 7 no longer segregates control panel applications by bitness: They all appear in the main Control Panel. This was done by running a puppet 32-bit copy of the Control Panel behind the scenes and making the puppet do the main Control panel's bidding whenever it needed to access information about 32-bit control panel applications.

    "Hey, go enumerate the 32-bit control panel applications for me."

    "Hey, go get the icon for this 32-bit control panel application."

    "Hey, go launch this 32-bit control panel application."

    "Hey, go make me a sandwich."

    "Hey, give me a backrub."

Page 1 of 4 (31 items) 1234