October, 2010

  • The Old New Thing

    Why does each drive have its own current directory?


    Commenter Dean Earley asks, "Why is there a 'current directory' AND an current drive? Why not merge them?"

    Pithy answer: Originally, each drive had its own current directory, but now they don't, but it looks like they do.

    Okay, let's unwrap that sentence. You actually know enough to answer the question yourself; you just have to put the pieces together.

    Set the wayback machine to DOS 1.0. Each volume was represented by a drive letter. There were no subdirectories. This behavior was carried forward from CP/M.

    Programs from the DOS 1.0 era didn't understand subdirectories; they referred to files by just drive letter and file name, for example, B:PROGRAM.LST. Let's fire up the assembler (compilers were for rich people) and assemble a program whose source code is on the A drive, but sending the output to the B drive.

    A>asm foo       the ".asm" extension on "foo" is implied
    Assembler version blah blah blah
    Source File: FOO.ASM
    Listing file [FOO.LST]: NUL throw away the listing file
    Object file [FOO.OBJ]: B: send the object file to drive B

    Since we gave only a drive letter in response to the Object file prompt, the assembler defaults to a file name of FOO.OBJ, resulting in the object file being generated as B:FOO.OBJ.

    Okay, now let's introduce subdirectories into DOS 2.0. Suppose you have want to assemble A:\SRC\FOO.ASM and put the result into B:\OBJ\FOO.OBJ. Here's how you do it:

    A> B:
    B> CD \OBJ
    B> A:
    A> CD \SRC
    A> asm foo
    Assembler version blah blah blah
    Source File: FOO.ASM
    Listing file [FOO.LST]: NUL
    Object file [FOO.OBJ]: B:

    The assembler reads from A:FOO.ASM and writes to B:FOO.OBJ, but since the current directory is tracked on a per-drive basis, the results are A:\SRC\FOO.ASM and B:\OBJ\FOO.OBJ as desired. If the current directory were not tracked on a per-drive basis, then there would be no way to tell the assembler to put its output into a subdirectory. As a result, DOS 1.0 programs were effectively limited to operating on files in the root directory, which means that nobody would put files in subdirectories (because their programs couldn't access them).

    From a DOS 1.0 standpoint, changing the current directory on a drive performs the logical equivalent of changing media. "Oh look, a completely different set of files!"

    Short attention span.

    Remembering the current directory for each drive has been preserved ever since, at least for batch files, although there isn't actually such a concept as a per-drive current directory in Win32. In Win32, all you have is a current directory. The appearance that each drive has its own current directory is a fake-out by cmd.exe, which uses strange environment variables to create the illusion to batch files that each drive has its own current directory.

    Dean continues, "Why not merge them? I have to set both the dir and drive if i want a specific working dir."

    The answer to the second question is, "They already are merged. It's cmd.exe that tries to pretend that they aren't." And if you want to set the directory and the drive from the command prompt or a batch file, just use the /D option to the CHDIR command:

    D:\> CD /D C:\Program Files\Windows NT
    C:\Program Files\Windows NT> _

    (Notice that the CHDIR command lets you omit quotation marks around paths which contain spaces: Since the command takes only one path argument, the lack of quotation marks does not introduce ambiguity.

  • The Old New Thing

    Wildly popular computer game? The Windows product team has you covered


    In Windows 95, the most heavily-tested computer game was DOOM. Not because the testers spent a lot of time developing test plans and test harnesses and automated run-throughs. Nope, it was because it was by far the most popular game the Windows 95 team members played to unwind.

    It was a huge breakthrough when DOOM finally ran inside a MS-DOS box and didn't require booting into MS-DOS mode any more. Now you could fire up DOOM without having to exit all your other programs first.

    I've learned that in Windows Vista, the most heavily tested game was World of Warcraft. Most members of the DirectX application compatibility team are WoW players, in addition to a large number in the Windows division overall.

    So if you have a wildly popular computer game for the PC, you can be pretty sure that the Windows team will be all over it. "For quality control purposes, I assure you."

    Related story: How to make sure your network card works with Windows.

  • The Old New Thing

    The evolution of the ICO file format, part 4: PNG images


    We finish our tour of the evolution of the ICO file format with the introduction of PNG-compressed images in Windows Vista.

    The natural way of introducing PNG support for icon images would be to allow the biCompression field of the BITMAP­INFO­HEADER to take the value BI_PNG, in which case the image would be represented not by a DIB but by a PNG. After all, that's why we have a biCompression field: For forward compatibility with future encoding systems. Wipe the dust off your hands and declare victory.

    Unfortunately, it wasn't that simple. If you actually try using ICO files in this format, you'll find that a number of popular icon-authoring tools crash when asked to load a PNG-compressed icon file for editing.

    The problem appeared to be that the new BI_PNG compression type appeared at a point in the parsing code where it was not prepared to handle such a failure (or the failure was never detected). The solution was to change the file format so that PNG-compressed images fail these programs' parsers at an earlier, safer step. (This is sort of the opposite of penetration testing, which keeps tweaking data to make the failure occur at a deeper, more dangerous step.)

    Paradoxically, the way to be more compatible is to be less compatible.

    The format of a PNG-compressed image consists simply of a PNG image, starting with the PNG file signature. The image must be in 32bpp ARGB format (known to GDI+ as Pixel­Format­32bpp­ARGB). There is no BITMAP­INFO­HEADER prefix, and no monochrome mask is present.

    Since we had to break compatibility with the traditional format for ICO images, we may as well solve the problem we saw last time of people who specify an incorrect mask. With PNG-compressed images, you do not provide the mask at all; the mask is derived from the alpha channel on the fly. One fewer thing for people to get wrong.

  • The Old New Thing

    Why is the origin at the upper left corner?


    Via the Suggestion Box, Dirk Declercq asks why the default client-area coordinate origin is at the upper left corner instead of the lower left corner. (I assume he also intends for the proposed client-area coordinate system to have the y-coordinate increasing as you move towards the top of the screen.)

    Well, putting the client area origin at the lower left would have resulted in the client coordinate space not being a simple translation of the screen coordinate space. After all, the screen origin is at the upper left, too. Windows was originally developed on left-to-right systems, where the relationship between client coordinates and screen coordinates was a simple translation. Having the y-coordinate increase as you move down the screen but move up the client would have just been one of those things you did to be annoying.

    Okay, so why not place the screen origin at the lower left, too?

    Actually, OS/2 does this, and DIBs do it as well. And then everybody wonders why their images are upside-down.

    Turns out that the people who designed early personal computers didn't care much for mathematical theory. The raster gun of a television set starts at the upper left corner, continues to the right, and when it reaches the right-hand edge of the screen, it jumps back to the left edge of the screen to render the second scan line. Why did television sets scan from the top down instead of from the bottom up? Beats me. You'll have to ask the person who invented the television (who, depending on whom you ask, is Russian or American or German or Scottish or some other nationality entirely), or more specifically, whoever invented the scanning model of image rendering, why they started from the top rather than from the bottom.

    Anyway, given that the video hardware worked from top to bottom, it was only natural that the memory for the video hardware work the same way. (The Apple II famously uses a peculiar memory layout in order to save a chip.)

    Who knows, maybe if the design of early computers had been Chinese, we would be wondering why the origin was in the upper right corner with the pixels in column-major order.

    Bonus chatter: Even mathematicians can't get their story straight. Matrices are typically written with the origin element at the upper left. Which reminds me of a story from the old Windows 95 days. The GDI folks received a defect report from the user interface team, who backed up their report with a complicated mathematical explanation. The GDI team accepted the change request with the remark, "We ain't much fer book lernin."

  • The Old New Thing

    The evolution of the ICO file format, part 1: Monochrome beginnings


    This week is devoted to the evolution of the ICO file format. Note that the icon resource format is different from the ICO file format; I'll save that topic for another day.

    The ICO file begins with a fixed header:

    typedef struct ICONDIR {
        WORD          idReserved;
        WORD          idType;
        WORD          idCount;
        ICONDIRENTRY  idEntries[];

    idReserved must be zero, and idType must be 1. The idCount describes how many images are included in this ICO file. An ICO file is really a collection of images; the theory is that each image is an alternate representation of the same underlying concept, but at different sizes and color depths. There is nothing to prevent you, in principle, from creating an ICO file where the 16×16 image looks nothing like the 32×32 image, but your users will probably be confused.

    After the idCount is an array of ICONDIRECTORY entries whose length is given by idCount.

    struct IconDirectoryEntry {
        BYTE  bWidth;
        BYTE  bHeight;
        BYTE  bColorCount;
        BYTE  bReserved;
        WORD  wPlanes;
        WORD  wBitCount;
        DWORD dwBytesInRes;
        DWORD dwImageOffset;

    The bWidth and bHeight are the dimensions of the image. Originally, the supported range was 1 through 255, but starting in Windows 95 (and Windows NT 4), the value 0 is accepted as representing a width or height of 256.

    The wBitCount and wPlanes describe the color depth of the image; for monochrome icons, these value are both 1. The bReserved must be zero. The dwImageOffset and dwBytesInRes describe the location (relative to the start of the ICO file) and size in bytes of the actual image data.

    And then there's bColorCount. Poor bColorCount. It's supposed to be equal to the number of colors in the image; in other words,

    bColorCount = 1 << (wBitCount * wPlanes)

    If wBitCount * wPlanes is greater than or equal to 8, then bColorCount is zero.

    In practice, a lot of people get lazy about filling in the bColorCount and set it to zero, even for 4-color or 16-color icons. Starting in Windows XP, Windows autodetects this common error, but its autocorrection is slightly buggy in the case of planar bitmaps. Fortunately, almost nobody uses planar bitmaps any more, but still, it would be in your best interest not to rely on the autocorrection performed by Windows and just set your bColorCount correctly in the first place. An incorrect bColorCount means that when Windows tries to find the best image for your icon, it may choose a suboptimal one because it based its decision on incorrect color depth information.

    Although it probably isn't true, I will pretend that monochrome icons existed before color icons, because it makes the storytelling easier.

    A monochome icon is described by two bitmaps, called AND (or mask) and XOR (or image, or when we get to color icons, color). Drawing an icon takes place in two steps: First, the mask is ANDed with the screen, then the image is XORed. In other words,

    pixel = (screen AND mask) XOR image

    By choosing appropriate values for mask and image, you can cover all the possible monochrome BLT operations.

    mask image result operation
    0 0 (screen AND 0) XOR 0 = 0 blackness
    0 1 (screen AND 0) XOR 1 = 1 whiteness
    1 0 (screen AND 1) XOR 0 = screen nop
    1 1 (screen AND 1) XOR 1 = NOT screen invert

    Conceptually, the mask specifies which pixels from the image should be copied to the destination: A black pixel in the mask means that the corresponding pixel in the image is copied.

    The mask and image bitmaps are physically stored as one single double-height DIB. The image bitmap comes first, followed by the mask. (But since DIBs are stored bottom-up, if you actually look at the bitmap, the mask is in the top half of the bitmap and the image is in the bottom half).

    In terms of file format, each icon image is stored in the form of a BITMAPINFO (which itself takes the form of a BITMAPINFOHEADER followed by a color table), followed by the image pixels, followed by the mask pixels. The biCompression must be BI_RGB. Since this is a double-height bitmap, the biWidth is the width of the image, but the biHeight is double the image height. For example, a 16×16 icon would specify a width of 16 but a height of 16 × 2 = 32.

    That's pretty much it for classic monochrome icons. Next time we'll look at color icons.

    Still, given what you know now, the following story will make sense.

    A customer contacted the shell team to report that despite all their best efforts, they could not get Windows to use the image they wanted from their .ICO file. Windows for some reason always chose a low-color icon instead of using the high-color icon. For example, even though the .ICO file had a 32bpp image available, Windows always chose to use the 16-color (4bpp) image, even when running on a 32bpp display.

    A closer inspection of the offending .ICO file revealed that the bColorCount in the IconDirectoryEntry for all the images was set to 1, regardless of the actual color depth of the image. The table of contents for the .ICO file said "Yeah, all I've got are monochrome images. I've got three 48×48 monochrome images, three 32×32 monochrome images, and three 16×16 monochrome images." Given this information, Windows figured, "Well, given those choices, I guess that means I'll use the monochrome one." It chose one of images (at pseudo-random), and went to the bitmap data and found, "Oh, hey, how about that, it's actually a 16-color image. Okay, well, I guess I can load that."

    In summary, the .ICO file was improperly authored. Patching each IconDirectoryEntry in a hex editor made the icon work as intended. The customer thanked us for our investigation and said that they would take the issue up with their graphic design team.

  • The Old New Thing

    Why are the keyboard scan codes for digits off by one?


    In Off by one what, exactly?, my colleague Michael Kaplan wrote

    And this decision long ago that caused the scan codes to not line up for these digits when they could have...

    The word that struck me there was "decision".

    Because it wasn't a "decision" to make the scan codes almost-but-not-quite line up with digits. It was just a coincidence.

    If you look at the scan code table from Michael's article

    you can see stretches of consecutive scan codes, broken up by weird places where the consecutive pattern is violated. The weirdness makes more sense when you look at the original IBM PC XT keyboard:


    With this presentation, it becomes clearer how scan codes were assigned: They simply started at 01 and continued through the keyboard in English reading order. (Scan code 00 is an error code indicating keyboard buffer overflow.) The reason for the keyboard scan code being off-by-one from the digits is merely due to the fact that there was one key to the left of the digits. If there were two keys to the left of the digits, they would have been off by two.

    Of course, if the original keyboard designers had started counting from the lower left corner, like all right-thinking mathematically-inclined people, then this sort-of-coincidence would never have happened. The scan codes for the digits would have been 2E through 37, and nobody would have thought anything of it.

    It's a testament to the human brain's desire to find patterns and determine a reason for them that what is really just a coincidence gets interpreted as some sort of conspiracy.

  • The Old New Thing

    Secret passages on Microsoft main campus


    They aren't really "secret passages" but they are definitely underutilized, and sometimes they provide a useful shortcut.

    At the northwest corner of Building 50, there are two doors. One leads to a stairwell that takes you to the second floor. That's the one everybody uses. The other door is a service entrance that takes you to the cafeteria. If your office is on the second or third floor in the northwest corner, it's faster to use the service hallway to get to the cafeteria than it is to walk to the core of the building and take the main stairs.

    There is a service tunnel that runs from the first floor of Building 86 (entrance next to the first floor cafeteria elevator) through the loading dock to the Central Garage, where you can continue to Building 85 or 84. This is not really any faster than the regular route, but it does have the advantage of being underground and mostly indoors, which is a major benefit when it is cold or raining.

    What is your favorite secret passage at your workplace?

  • The Old New Thing

    The evolution of the ICO file format, part 2: Now in color!


    Last time, we looked at the format of classic monochrome icons. But if you want to include color images, too? (Note that it is legal—and for a time it was common—for a single ICO file to offer both monochrome and color icons. After all, a single ICO file can offer both 16-color and high-color images; why not also 2-color images?)

    The representation of color images in an ICO file is almost the same as the representation of monochrome images: All that changes is that the image bitmap is now in color. (The mask remains monochrome.)

    In other words, the image format consists of a BITMAPINFOHEADER where the bmWidth is the width of the image and bmHeight is double the height of the image, followed by the bitmap color table, followed by the image pixels, followed by the mask pixels.

    Note that the result of this is a bizarre non-standard bitmap. The height is doubled because we have both an image and a mask, but the color format changes halfway through!

    Other restrictions: Supported color formats are 4bpp, 8bpp, 16bpp, and 0RGB 32bpp. Note that 24bpp is not supported; you'll have to convert it to a 0RGB 32bpp bitmap. Supported values for biCompression for color images are BI_RGB and (if your bitmap is 16bpp or 32bpp) BI_BITFIELDS.

    The mechanics of drawing the icon are the same as for a monochrome image: First, the mask is ANDed with the screen, then the image is XORed. In other words,

    pixel = (screen AND mask) XOR image

    On the other hand, XORing color pixels is not really a meaningful operation. It's not like people say "Naturally, fuchsia XOR aqua equals yellow. Any idiot knows that." Or "Naturally, blue XOR eggshell equals apricot on 8bpp displays (because eggshell is palette index 56, blue is palette index 1, and palette index 57 is apricot) but is equal to #F0EA29 on 32bpp displays." The only meaningful color to XOR against is black, in which case you have "black XOR Q = Q for all colors Q".

    mask image result operation
    0 Q (screen AND 0) XOR Q = Q copy from icon
    1 0 (screen AND 1) XOR 0 = screen nop
    1 Q (screen AND 1) XOR Q = screen XOR Q dubious

    For pixels you want to be transparent, set your mask to white and your image to black. For pixels you want to come from your icon, set your mask to black and your image to the desired color.

    We now have enough information to answer a common question people have about icons. After that break, we'll return to the evolution of the ICO file format.

    For further reading: Icons in Win32.

  • The Old New Thing

    The evolution of the ICO file format, part 3: Alpha-blended images


    Windows XP introduced the ability to provide icon images which contain an 8-bit alpha channel. Up until this point, you had only a 1-bit alpha channel, represented by a mask.

    The representation of an alpha-blended image in your ICO file is pretty straightforward. Recall that the old ICO format supports 0RGB 32bpp bitmaps. To use an alpha-blended image, just drop in a ARGB 32bpp bitmap instead. When the window manager sees a 32bpp bitmap, it looks at the alpha channel. If it's all zeroes, then it assumes that the image is in 0RGB format; otherwise it assumes it is in ARGB format. Everything else remains the same as for the non-alpha version.

    Note carefully that everything else remains the same. In particular, you are still required to provide a mask. I've seen some people be a bit lazy about providing a meaningful mask and just pass in all-zeroes. And everything seems to work just fine, until you hit a case where it doesn't work. (Read on.)

    There are basically three ways of drawing an alpha-blended icon image.

    1. Draw­Icon(DI_NORMAL): This is by far the most common way icons are drawn. In the alpha-blended case, this is done by blending the image with the destination according to the alpha channel.
    2. Draw­Icon(DI_IMAGE): This draws the image portion of the icon image, completely overwriting the destination.
    3. Draw­Icon(DI_MASK): This draws only the mask portion of the icon image, completely overwriting the destination.

    The DI_IMAGE and DI_MASK flags let an application draw just one of the two images contained in an icon image. Applications do this if they want finer control over the icon-drawing process. For example, they might ask for the mask so they can build a shadow effect under the icon. The mask tells them which parts of the icon are opaque and therefore should cast a shadow.

    If you understand this, then you can see how people who set their mask image to all-zeroes managed to get away with it most of the time. Since most programs just use DI_NORMAL to draw icons, the incorrect mask is never used, so the error never shows up. It's only when the icon is used by a program that wants to do fancy icon effects and asks for DI_MASK (or calls Get­Icon­Info and looks at the hbmMask) that the incorrect mask results in an ugly icon.

    The ironic thing is that the people who incorrectly set the mask to all-zeroes are probably the same people who will then turn around and say, "When I try to use alpha-blended icons, the result is hideously ugly under conditions X and Y. Those Microsoft programmers are such idiots. More proof that Windows is a buggy pile of manure." What they don't realize is that the hideous ugliness was caused by their own error.

  • The Old New Thing

    Hacking Barney the dinosaur for fun (no profit)


    Many years ago, Microsoft produced a collection of interactive toys called ActiMates, and one of the features was that television programs could broadcast an encoded signal which would enable the toy to interact with the program. The idea would be that the Barney doll would do something that was coordinated with what was happening on Barney & Friends.

    When this came out, a bunch of us wondered what it would take to hack into the device and get Barney to say and do, um, very un-Barneyish things. One of us managed to get a schematic for the device, but since none of us was an electrical engineer, that pretty much dead-ended the project.

    Over ten years later, I learned that we weren't the only people to get that idea. I met someone who told me that he managed to get his hands on the internal devkit for the ActiMates series and control a Barney doll from his PC. Not satisfied with being limited to the built-in Barney phrases, he was able to "take additional creative steps with the devkit" to stream his own replacement audio to the device (although he was never able to get the sound quality of his streamed audio to sound as good as the built-in phrases). As a result, he could make Barney say whatever he wanted, and if he really felt like it, he could wake up all the Barney toys in his apartment complex at midnight and give orders to his robot army of purple dinosaurs.

    The catch was that his robot army most likely would have consisted of just one robot.

    Bonus reading: SWEETPEA: Software Tools for Programmable Embodied Agents [pdf], Michael Kaminsky, Paul Dourish, W. Keith Edwards, Anthony LaMarca, Michael Salisbury and Ian Smith, CHI'99.

Page 1 of 3 (27 items) 123