December, 2007

  • The Old New Thing

    Why aren't console windows themed on Windows XP?

    • 124 Comments

    Commenter Andrej Budja asks why cmd.exe is not themed in Windows XP. (This question was repeated by Serge Wautier, proving that nobody checks whether their suggestion has already been submitted before adding their own. It was also asked by a commenter who goes by the name "S", and then repeated again just a few hours later, which proves again that nobody reads the comments either.) Knowledge Base article 306509 explains that this behavior exists because the command prompt window (like all console windows) is run under the ClientServer Runtime System (CSRSS), and CSRSS cannot be themed.

    But why can't CSRSS be themed?

    CSRSS runs as a system service, so any code that runs as part of CSRSS creates potential for mass havoc. The slightest mis-step could crash CSRSS, and with it the entire system. The CSRSS team decided that they didn't want to take the risk of allowing the theme code to run in their process, so they disabled theming for console windows. (There's also an architectural reason why CSRSS cannot use the theming services: CSRSS runs as a subsystem, and the user interface theming services assume that they're running as part of a Win32 program.)

    In Windows Vista, the window frame is drawn by the desktop window manager, which means that your console windows on Vista get the glassy frame just like other windows. But if you take a closer look, you will see that CSRSS itself doesn't use themed windows: Notice that the scroll bars retain the classic look.

    The window manager giveth and the window manager taketh away, for at the same time console windows gained the glassy frame, they also lost drag and drop. You used to be able to drag a file out of Explorer and drop it onto a command prompt, but if you try that in Windows Vista, nothing happens. This is a consequence of tighter security around the delivery of messages from a process running at lower integrity to one running at a higher integrity level (see UIPI). Since CSRSS is a system process, it runs at very high security level and won't let any random program (like Explorer) send it messages, such as the ones used to mediate OLE drag and drop. You'll see the same thing if you log on as a restricted administrator and then kick off an elevated copy of Notepad. You won't be able to drag a file out of Explorer and drop it onto Notepad, for the same reason.

  • The Old New Thing

    What was the role of MS-DOS in Windows 95?

    • 134 Comments

    Welcome, Slashdot readers. Remember, this Web site is for entertainment purposes only.

    Sean wants to know what the role of MS-DOS was in Windows 95. I may regret answering this question since it's clear Slashdot bait. (Even if Sean didn't intend it that way, that's what it's going to turn into.)

    Here goes. Remember, what I write here may not be 100% true, but it is "true enough." (In other words, it gets the point across without getting bogged down in nitpicky details.)

    MS-DOS served two purposes in Windows 95.

    • It served as the boot loader.
    • It acted as the 16-bit legacy device driver layer.

    When Windows 95 started up, a customized version of MS-DOS was loaded, and it's that customized version that processed your CONFIG.SYS file, launched COMMAND.COM, which ran your AUTOEXEC.BAT and which eventually ran WIN.COM, which began the process of booting up the VMM, or the 32-bit virtual machine manager.

    The customized version of MS-DOS was fully functional as far as the phrase "fully functional" can be applied to MS-DOS in the first place. It had to be, since it was all that was running when you ran Windows 95 in "single MS-DOS application mode."

    The WIN.COM program started booting what most people think of as "Windows" proper. It used the copy of MS-DOS to load the virtual machine manager, read the SYSTEM.INI file, load the virtual device drivers, and then it turned off any running copy of EMM386 and switched into protected mode. It's protected mode that is what most people think of as "the real Windows."

    Once in protected mode, the virtual device drivers did their magic. Among other things those drivers did was "suck the brains out of MS-DOS," transfer all that state to the 32-bit file system manager, and then shut off MS-DOS. All future file system operations would get routed to the 32-bit file system manager. If a program issued an int 21h, the 32-bit file system manager would be responsible for handling it.

    And that's where the second role of MS-DOS comes into play. For you see, MS-DOS programs and device drivers loved to mess with the operating system itself. They would replace the int 21h service vector, they would patch the operating system, they would patch the low-level disk I/O services int 25h and int 26h. They would also do crazy things to the BIOS interrupts such as int 13h, the low-level disk I/O interrupt.

    When a program issued an int 21h call to access MS-DOS, the call would go first to the 32-bit file system manager, who would do some preliminary munging and then, if it detected that somebody had hooked the int 21h vector, it would jump back into the 16-bit code to let the hook run. Replacing the int 21h service vector is logically analogous to subclassing a window. You get the old vector and set your new vector. When your replacement handler is called, you do some stuff, and then call the original vector to do "whatever would normally happen." After the original vector returned, you might do some more work before returning to the original caller.

    One of the 16-bit drivers loaded by CONFIG.SYS was called IFSMGR.SYS. The job of this 16-bit driver was to hook MS-DOS first before the other drivers and programs got a chance! This driver was in cahoots with the 32-bit file system manager, for its job was to jump from 16-bit code back into 32-bit code to let the 32-bit file system manager continue its work.

    In other words, MS-DOS was just an extremely elaborate decoy. Any 16-bit drivers and programs would patch or hook what they thought was the real MS-DOS, but which was in reality just a decoy. If the 32-bit file system manager detected that somebody bought the decoy, it told the decoy to quack.

    Let's start with a system that didn't contain any "evil" drivers or programs that patched or hooked MS-DOS.

    Program calls int 21h

    32-bit file system manager
    checks that nobody has patched or hooked MS-DOS
    performs the requested operation
    updates the state variables inside MS-DOS
    returns to caller
    Program gets result

    This was paradise. The 32-bit file system manager was able to do all the work without having to deal with pesky drivers that did bizarro things. Note the extra step of updating the state variables inside MS-DOS. Even though we extracted the state variables from MS-DOS during the boot process, we keep those state variables in sync because drivers and programs frequently "knew" how those state variables worked and bypassed the operating system and accessed them directly. Therefore, the file system manager had to maintain the charade that MS-DOS was running the show (even though it wasn't) so that those drivers and programs saw what they wanted.

    Note also that those state variables were per-VM. (I.e., each MS-DOS "box" you opened got its own copy of those state variables.) After all, each MS-DOS box had its idea of what the current directory was, what was in the file tables, that sort of thing. This was all an act, however, because the real list of open files was kept in by the 32-bit file system manager. It had to be, because disk caches had to be kept coherent, and file sharing need to be enforced globally. If one MS-DOS box opened a file for exclusive access, then an attempt by a program running in another MS-DOS box to open the file should fail with a sharing violation.

    Okay, that was the easy case. The hard case is if you had a driver that hooked int 21h. I don't know what the driver does, let's say that it's a network driver that intercepts I/O to network drives and handles them in some special way. Let's suppose also that there's some TSR running in the MS-DOS box which has hooked int 21h so it can print a 1 to the screen when the int 21h is active and a 2 when the int 21h completes. Let's follow a call to a local device (not a network device, so the network driver doesn't do anything):

    Program calls int 21h

    32-bit file system manager
    notices that somebody has patched or hooked MS-DOS
    jumps to the hook (which is the 16-bit TSR)

    16-bit TSR (front end)
    prints a 1 to the screen
    calls previous handler (which is the 16-bit network driver)

    16-bit network driver (front end)
    decides that this isn't a network I/O request calls previous handler (which is the 16-bit IFSMGR hook)

    16-bit IFSMGR hook
    tells 32-bit file system manager
      that it's time to make the donuts

    32-bit file system manager
    regains control
    performs the requested operation
    updates the state variables inside MS-DOS
    returns to caller

    16-bit network driver (back end)
    returns to caller

    16-bit TSR (back end)
    prints a 2 to the screen
    returns to caller
    Program gets result

    Notice that all the work is still being done by the 32-bit file system manager. It's just that the call gets routed through all the 16-bit stuff to maintain the charade that 16-bit MS-DOS is still running the show. The only 16-bit code that actually ran (in red) is the stuff that the TSR and network driver installed, plus a tiny bit of glue in the 16-bit IFSMGR hook. Notice that no 16-bit MS-DOS code ran. The 32-bit file manager took over for MS-DOS.

    A similar sort of "take over but let the crazy stuff happen if somebody is doing crazy stuff" dance took place when the I/O subsystem took over control of your hard drive from 16-bit device drivers. If it recognized the drivers, it would "suck their brains out" and take over all the operations, in the same way that the 32-bit file system manager took over operations from 16-bit MS-DOS. On the other hand, if the driver wasn't one that the I/O subsystem recognized, it let the driver be the one in charge of the drive. If this happened, it was said that you were going through the "real-mode mapper" since "real mode" was name for the CPU mode when protected mode was not running; in other words, the mapper was letting the 16-bit drivers do the work.

    Now, if you were unlucky enough to be using the real-mode mapper, you probably noticed that system performance to that drive was pretty awful. That's because you were using the old clunky single-threaded 16-bit drivers instead of the faster, multithread-enabled 32-bit drivers. (When a 16-bit driver was running, no other I/O could happen because 16-bit drivers were not designed for multi-threading.)

    This awfulness of the real-mode mapper actually came in handy in a backwards way, because it was an early indication that your computer got infected with an MS-DOS virus. After all, MS-DOS viruses did what TSRs and drivers did: They hooked interrupt vectors and took over control of your hard drive. From the I/O subsystem's point of view, they looked just like a 16-bit hard disk device driver! When people complained, "Windows suddenly started running really slow," we asked them to look at the system performance page in the control panel and see if it says that "Some drives are using MS-DOS compatiblity." If so, then it meant that the real-mode mapper was in charge, and if you didn't change hardware, it probably means a virus.

    Now, there are parts of MS-DOS that are unrelated to file I/O. For example, there are functions for allocating memory, parsing a string containing potential wildcards into FCB format, that sort of thing. Those functions were still handled by MS-DOS since they were just "helper library" type functions and there was no benefit to reimplementing them in 32-bit code aside from just being able to say that you did it. The old 16-bit code worked just fine, and if you let it do the work, you preserved compatibility with programs that patched MS-DOS in order to alter the behavior of those functions.

  • The Old New Thing

    Psychic debugging: The first step in diagnosing a deadlock is a simple matter of following the money

    • 26 Comments

    Somebody asked our team for help because they believed they hit a deadlock in their program's UI. (It's unclear why they asked our team, but I guess since our team uses the window manager, and their program uses the window manager, we're all in the same boat. You'd think they'd ask the window manager team for help.)

    But it turns out that solving the problem required no special expertise. In fact, you probably know enough to solve it, too.

    Here are the interesting threads:

      0  Id: 980.d30 Suspend: 1 Teb: 7ffdf000 Unfrozen
    ChildEBP RetAddr  
    0023dc90 7745dd8c ntdll!KiFastSystemCallRet 
    0023dc94 774619e0 ntdll!ZwWaitForSingleObject+0xc 
    0023dcf8 774618fb ntdll!RtlpWaitOnCriticalSection+0x154 
    0023dd20 00cd03f2 ntdll!RtlEnterCriticalSection+0x152 
    0023dd38 00cd0635 myapp!LogMsg+0x15 
    0023dd58 00cd0c6a myapp!LogRawIndirect+0x27 
    0023fcb8 00cb64a7 myapp!Log+0x62 
    0023fce8 00cd7598 myapp!SimpleClientConfiguration::Cleanup+0x17 
    0023fcf8 00cd8ffe myapp!MsgProc+0x1a9 
    0023fd10 00cda1a9 myapp!Close+0x43 
    0023fd24 761636d2 myapp!WndProc+0x62 
    0023fd50 7616330c USER32!InternalCallWinProc+0x23 
    0023fdc8 76164030 USER32!UserCallWinProcCheckWow+0x14b 
    0023fe2c 76164088 USER32!DispatchMessageWorker+0x322 
    0023fe3c 00cda3ba USER32!DispatchMessageW+0xf 
    0023fe9c 00cd0273 myapp!GuiMain+0xe8 
    0023feb4 00ccdeca myapp!wWinMain+0x87 
    0023ff48 7735c6fc myapp!__wmainCRTStartup+0x150 
    0023ff54 7742e33f kernel32!BaseThreadInitThunk+0xe 
    0023ff94 00000000 ntdll!_RtlUserThreadStart+0x23 
     
       1  Id: 980.ce8 Suspend: 1 Teb: 7ffdd000 Unfrozen
    ChildEBP RetAddr  
    00f8d550 76162f81 ntdll!KiFastSystemCallRet 
    00f8d554 76162fc4 USER32!NtUserSetWindowLong+0xc 
    00f8d578 76162fe5 USER32!_SetWindowLong+0x131 
    00f8d590 74aa5c2b USER32!SetWindowLongW+0x15 
    00f8d5a4 74aa5b65 comctl32_74a70000!ClearWindowStyle+0x23 
    00f8d5cc 74ca568f comctl32_74a70000!CCSetScrollInfo+0x103 
    00f8d618 76164ea2 uxtheme!ThemeSetScrollInfoProc+0x10e 
    00f8d660 00cdd913 USER32!SetScrollInfo+0x57 
    00f8d694 00cdf0a4 myapp!SetScrollRange+0x3b 
    00f8d6d4 00cdd777 myapp!TextOutputStringColor+0x134 
    00f8d93c 00cd04c4 myapp!TextLogMsgProc+0x3db 
    00f8d960 00cd0635 myapp!LogMsg+0xe7 
    00f8d980 00cd0c6a myapp!LogRawIndirect+0x27 
    00f8f8e0 00cd6367 myapp!Log+0x62 
    00f8faf0 7735c6fc myapp!remote_ext::ServerListenerThread+0x45c 
    00f8fafc 7742e33f kernel32!BaseThreadInitThunk+0xe 
    00f8fb3c 00000000 ntdll!_RtlUserThreadStart+0x23 
    

    The thing about debugging deadlocks is that you usually don't need to understand what's going on. The diagnosis is largely mechanical once you get your foot in the door. (Though sometimes it's hard to get your initial footing.)

    Let's look at thread 0. It is waiting for a critical section. The owner of that critical section is thread 1. How do I know that? Well, I could've debugged it, or I could've used my psychic powers to say, "Gosh, that function is called LogMsg, and look there's another thread that is inside the function LogMsg. I bet that function is using a critical section to ensure that only one thread uses it at a time."

    Okay, so thread 0 is waiting for thread 1. What is thread 1 doing? Well, it entered the critical section back in the LogMsg function, and then it did some text processing and, oh look, it's doing a SetScrollInfo. The SetScrollInfo went into comctl32 and ultimately resulted in a SetWindowLong. The window that the application passed to SetScrollInfo is owned by thread 0. How do I know that? Well, I could've debugged it, or I could've used my psychic powers to say, "Gosh, the change in the scroll info has led to a change in window styles, and the thread is trying to notify the window of the change in style. The window clearly belongs to another thread; otherwise we wouldn't be stuck in the first place, and given that we see only two threads, there isn't much choice as to what other thread it could be!"

    At this point, I think you see the deadlock. Thread 0 is waiting for thread 1 to exit the critical section, but thread 1 is waiting for thread 0 to process the style change message.

    What happened here is that the program sent a message while holding a critical section. Since message handling can trigger hooks and cross-thread activity, you cannot hold any resources when you send a message because the hook or the message recipient might want to acquire that resource that you own, resulting in a deadlock.

  • The Old New Thing

    How do I mark a shortcut file as requiring elevation?

    • 25 Comments

    Specifying whether elevation is required is typically something that is the responsibility of the program. This is done by adding a requestedExecutionLevel element to your manifest. (Bart De Smet shows you how. Calvin Hsia does the same for your Visual FoxPro programs.) But if the program you're running doesn't have such a manifest—maybe it's an old program that you don't have any control over—you can create a shortcut to the program and mark the shortcut as requiring elevation.

    To do this, you set the SLDF_RUNAS_USER flag in the shortcut attributes. Here's a skeleton program that sets the flag on the shortcut whose path is passed on the command line. For expository purposes, I've skimped on the error reporting, and just to shake things up, I've used ATL smart pointers.

    #include <windows.h>
    #include <shlobj.h>
    #include <atlbase.h>
    
    void MarkShortcutRunAs(LPCWSTR pszShortcut)
    {
     CComPtr<IPersistFile> sppf;
     if (FAILED(sppf.CoCreateInstance(CLSID_ShellLink))) return;
     if (FAILED(sppf->Load(pszShortcut, STGM_READWRITE))) return;
     CComQIPtr<IShellLinkDataList> spdl(sppf);
     if (!spdl) return;
     DWORD dwFlags;
     if (FAILED(spdl->GetFlags(&dwFlags))) return;
     dwFlags |= SLDF_RUNAS_USER;
     if (FAILED(spdl->SetFlags(dwFlags))) return;
     if (FAILED(sppf->Save(NULL, TRUE))) return;
     wprintf(L"Succeeded\n");
    }
    
    int __cdecl wmain(int argc, wchar_t *argv[])
    {
     if (argc == 2 && SUCCEEDED(CoInitialize(NULL))) {
      MarkShortcutRunAs(argv[1]);
      CoUninitialize();
     }
     return 0;
    }
    

    There's not really much to this program. It creates a shell link object (CLSID_ShellLink) and asks it to load from the file whose path is given on the command line. It then uses IShellLinkDataList::GetFlags and IShellLinkDataList::SetFlags to fetch the old flags and set new flags that include SLDF_RUNAS_USER. Once that's done, it saves the result back out.

    The hard part was knowing that the SLDF_RUNAS_USER flag existed in the first place.

    (I fear that most people will read this article and say, "Awesome! My program requires elevation, and this is how I can mark my Start menu shortcut to prompt for elevation. Thanks, Raymond!" These people will have completely ignored the opening paragraph, which explains that that is the wrong thing to do.)

  • The Old New Thing

    Book review: Advanced Windows Debugging (Mario Hewardt and Daniel Pravat)

    • 19 Comments

    Ever so often, somebody sends me a book, and most of the time I glance through it and say, "Eh."

    But not this time.

    Advanced Windows Debugging will make you the envy of your friends (if your friends are computer nerds). Even the section with the "Oh come on every moron knows this already" title Basic Debugger Tasks has stuff that I didn't know. Fortunately, you don't have to slog through the stuff you already do know in order to find it, because the nifty new debugger commands are set off in the snippets of debugger conversation. (And by debugger conversation, I mean output of a debugger based on the Windows debug engine, debuggers like ntsd, kd and windbg.)

    Once you get past the "basics", you still have loads more ahead of you. The book covers debugging scenarios like a corrupted heap, a deadlock, or 100% CPU usage, as well as debugging tasks, like following the trail of an LPC request from the client to the server, peeking at the token count of a semaphore, and reconstructing a partially-corrupted stack—and illustrates each investigation with both discussion and annotated debugger output. All the things that seasoned developers take for granted (because they have become instinctual after years of experience) are spelled out for you. Learn more from the book's web site, not unsurprisingly named advancedwindowsdebugging.com.

    I'm keeping this book on my shelf. You can borrow it, but I'm going to insist that you return it when you're done.

  • The Old New Thing

    If you need anything other than natural alignment, you have to ask for it

    • 27 Comments

    If you need variables to be aligned a particular way, you need to ask for it.

    Let's say I have the following code:

    void fn() 
    { 
     int a; 
     char b; 
     long c; 
     char d[10];
    } 
    

    What would the alignment of the starting adresses of a,b,c and d be?

    What would the alignment be if the memory were allocated on heap?

    If this alignment varies for different data types within the same translation unit, is there a way to force uniform alignment for all types?

    If you need a particular alignment, you have to ask for it. By default, all you can count on is that variables are aligned according to their natural requirements.

    First, of course, there is no guarantee that local variables even reside on the stack. The optimizer may very well decide that particular local variables can reside in registers, in which case it has no alignment at all!

    There are a few ways to force a particular alignment. The one that fits the C language standard is to use a union:

    union char_with_int_alignment {
     char ch;
     int Alignment;
    } u;
    

    Given this union, you can say u.ch to obtain a character whose alignment is suitable for an integer.

    The Visual C++ compiler supports a declaration specifier to override the default alignment of a variable.

    typedef struct __declspec(align(16)) _M128 {
        unsigned __int64 Low;
        __int64 High;
    } M128, *PM128;
    

    This structure consists of two eight-byte members. Without the __declspec(align(#)) directive, the alignment of this structure would be 8-byte, since that is the alignment of the members with the most restrictive alignment. (Both unsigned __int64 and __int64 are naturally 8-byte-aligned.) But with the directive, the aligment is expanded to 16 bytes, which is more restrictive than what the structure normally would be. This particular structure is declared with more restrictive alignment because it is intended to be use to hold 128-bit values that will be used by the 128-bit XMM registers.

    A third way to force alignment with the Visual C++ compiler is to use the #pragma pack(#) directive. (There is also a "push" variation of this pragma which remembers the previous ambient alignment, which can be restored by a "pop" directive. And the /Zp# directive allows you to specify this pragma from the compiler command line.) This directive specifies that members can be placed at alignments suitable for #-byte objects rather than their natural alignment requirements, if the natural alignment is more restrictive. For example, if you set the pack alignment to 2, then all objects that are bigger than two bytes will be aligned as if they were two-byte objects. This can cause 32-bit values and 64-bit values to become mis-aligned; it is assumed that you know what you're doing any can compensate accordingly.

    For example, consider this structure whose natural alignment has been altered:

    #pragma pack(1)
    struct misaligned_members {
     WORD w;
     DWORD dw;
     BYTE b;
    };
    

    Given this structure, you cannot pass the address of the dw member to a function that expects a pointer to a DWORD, since the ground rules for programming specify that all pointers must be aligned unless unaligned pointers are explicitly permitted.

    void ExpectsAlignedPointer(DWORD *pdw);
    void UnalignedPointerOkay(UNALIGNED DWORD *pdw);
    
    misaligned_members s;
    ExpectsAlignedPointer(&s.dw); // wrong
    UnalignedPointerOkay(&s.dw);  // okay
    

    What about the member w? Is it aligned or not? Well, it depends.

    If you allocate a single structure on the heap, then the w member is aligned, since heap allocations are always aligned in a manner suitable for any fundamental data type. (I vaguely recall some possible weirdness with 10-byte floating point values, but that's not relevant to the topic at hand.)

    misaligned_members *p = (misaligned_members)
        HeapAllocate(hheap, 0, sizeof(misaligned_members));
    

    Given this code fragment, the member p->w is aligned since the entire structure is suitably aligned, and therefore so too is w. If you allocated an array, however, things are different.

    misaligned_members *p = (misaligned_members)
        HeapAllocate(hheap, 0, 2*sizeof(misaligned_members));
    

    In this code fragment, p[1].w is not aligned because the entire misaligned_members structure is 2+4+1=7 bytes in size since the packing is set to 1. Therefore, the second structure begins at an unaligned offset relative to the start of the array.

    One final issue is the expectations for alignment when using header files provided by an outside component. If you are writing a header file that will be consumed by others, and you require special alignment, you need to say so explicitly in your header file, because you don't control the code that will be including your header file. Furthermore, if your header file changes any compiler settings, you need to restore them before your header file is complete. If you don't follow this rule, then you create the situation where a program stops working if a program changes the order in which it includes seemingly-unrelated header files.

    // this code works
    #include <foo.h>
    #include <bar.h>
    
    // this code doesn't
    #include <bar.h>
    #include <foo.h>
    

    The problem was that bar.h changed the default structure alignment and failed to return it to the original value before it was over. As a result, in the second case, the structure alignment for the foo.h header file got "infected" and no longer matched the structure alignment used by the foo library.

    You can imagine an analogous scenario where deleting a header file can cause a program to stop working.

    Therefore, if you're writing a header file that will be used by others, and you require nonstandard alignment for your structures, you should use this pattern to change the default alignment:

    #include <pshpack1.h> // change alignment to 1
    ... stuff that assumes byte packing ...
    #include <poppack.h>  // return to original alignment
    

    In this way, you "leave things the way you found them" and avoid the mysterious infection scenarios described above.

  • The Old New Thing

    The magical healing properties of safe mode - bonus content

    • 35 Comments

    Okay, so you already read The healing properties of safe mode in TechNet Magazine. Here's the bonus content that was cut for space.

    First, the original title was "The Magical Healing Powers of Safe Mode," but it got trimmed for space reasons. (Ich bin mit der deutschen Übersetzung des ersten Satzes ein bisschen enttäuscht. Die eingeklammerte Phrase bittet um einen von den berühmten nur auf Deutsch gesehenen unverständlich langen adjektivischen Ausdrücken. Anstatt dessen hat der Übersetzer aufgegeben und die Phrase einfach weggelassen. Anderseits benutzt die deutsche Version den ursprünglichen Titel, so vielleicht ist es ja nicht so schlecht.)

    Useless Windows history: The feature now known as safe mode went through many other names before the final name was settled upon.

    • Fail-safe boot
    • FailSafe boot
    • Fail-safe mode
    • Safe mode
  • The Old New Thing

    2007 year-end link clearance

    • 5 Comments

    A few random links that I've collected over the last six months.

    And then the obligatory plug for my column in TechNet Magazine, which, despite the fact that Microsoft's name is on the magazine cover, does not establish the official Microsoft position on anything.

  • The Old New Thing

    How did wildcards work in MS-DOS?

    • 48 Comments

    The rules were simple but led to complicated results.

    MS-DOS files were eleven characters long with an implicit dot between characters eight and nine. Theoretically, spaces were permitted anywhere, but in practice they could appear only at the end of the file name or immediately before the implicit dot.

    Wildcard matching was actually very simple. The program passed an eleven-character pattern; each position in the pattern consisted either of a file name character (which had to match exactly) or consisted of a question mark (which matched anything). Consider the file "ABCD····TXT", where I've used · to represent a space. This file name would more traditionally be written as ABCD.TXT, but I've written it out in its raw 11-character format to make the matching more obvious. Let's look at some patterns and whether they would match.

    PatternResultExplanation
    ABCD····TXT Match exact
    ??????????? Match all positions are a wildcard so of course they match
    ABCD????··· No match space (position 9) does not match T
    A?CD····??? match perfect match at A, C, D, and the spaces; wildcard match at the question marks

    The tricky part is converting the traditional notation with dots and asterisks into the eleven-character pattern. The algorithm used by MS-DOS was the same one used by CP/M, since MS-DOS worked hard at being backwards compatible with CP/M. (You may find some people who call this the FCB matching algorithm, because file names were passed to and from the operating system in a structure called a File Control Block.)

    1. Start with eleven spaces and the cursor at position 1.
    2. Read a character from the input. If the end of the input is reached, then stop.
    3. If the next character in the input is a dot, then set positions 9, 10, and 11 to spaces, move the cursor to position 9, and go back to step 2.
    4. If the next character in the input is an asterisk, then fill the rest of the pattern with question marks, move the cursor to position 12, and go back to step 2. (Yes, this is past the end of the pattern.)
    5. If the cursor is not at position 12, copy the input character to the cursor position and advance the cursor.
    6. Go to step 2.

    Let's parse a few patterns using this algorithm, since the results can be surprising. In the diagrams, I'll underline the cursor position.

    First, let's look at the traditional "ABCD.TXT".

    InputPatternDescription
    ··········· Initial conditions
    A A·········· Copy to cursor and advance the cursor
    B AB········· Copy to cursor and advance the cursor
    C ABC········ Copy to cursor and advance the cursor
    D ABCD······· Copy to cursor and advance the cursor
    . ABCD······· Blank out positions 9, 10, and 11 and move cursor to position 9
    T ABCD····T·· Copy to cursor and advance the cursor
    X ABCD····TX· Copy to cursor and advance the cursor
    T ABCD····TXT  Copy to cursor and advance the cursor

    The final result is what we expected: ABCD····TXT.

    Let's look at a weird case: the pattern is ABCDEFGHIJKL.

    InputPatternDescription
    ··········· Initial conditions
    A A·········· Copy to cursor and advance the cursor
    B AB········· Copy to cursor and advance the cursor
    C ABC········ Copy to cursor and advance the cursor
    D ABCD······· Copy to cursor and advance the cursor
    E ABCDE······ Copy to cursor and advance the cursor
    F ABCDEF····· Copy to cursor and advance the cursor
    G ABCDEFG···· Copy to cursor and advance the cursor
    H ABCDEFGH··· Copy to cursor and advance the cursor
    I ABCDEFGHI·· Copy to cursor and advance the cursor
    J ABCDEFGHIJ· Copy to cursor and advance the cursor
    K ABCDEFGHIJK  Copy to cursor and advance the cursor

    Sure, this was extremely boring to watch, but look at the result: What you got was equivalent to ABCDEFGH.IJK. The dot is optional if it comes after exactly eight characters!

    Next, let's look at the troublesome A*B.TXT.

    InputPatternDescription
    ··········· Initial conditions
    A A·········· Copy to cursor and advance the cursor
    * A??????????  Fill rest of pattern with question marks and move to position 12
    B A??????????  Do nothing since cursor is at position 12
    . A???????··· Blank out positions 9, 10, and 11 and move cursor to position 9
    T A???????T·· Copy to cursor and advance the cursor
    X A???????TX· Copy to cursor and advance the cursor
    T A???????TXT  Copy to cursor and advance the cursor

    Notice that the result is the same as you would have gotten from the pattern A*.TXT. Any characters other than a dot that come after an asterisk have no effect, since the asterisk moves the cursor to position 12, at which point nothing changes the parse state except for a dot, which clears the last three positions and moves the cursor.

    I won't work it out here, but if you stare at it for a while, you'll also discover that *.* is the same as * by itself.

    In addition to the rules above, the MS-DOS command prompt had some quirks in its parsing. If you typed DIR .TXT, the command prompt acted as if you had typed DIR *.TXT; it silently inserted an asterisk if the first character of the pattern was a dot. This behavior was probably by accident, not intentional, but it was an accident that some people came to rely upon. When we fixed the bug in Windows 95, more than one person complained that their DIR .TXT command wasn't working.

    The FCB matching algorithm was abandoned during the transition to Win32 since it didn't work with long file names. Long file names can contain multiple dots, and of course files can be longer than eleven characters, and there can be more than eight characters before the dot. But some quirks of the FCB matching algorithm persist into Win32 because they have become idiom.

    For example, if your pattern ends in .*, the .* is ignored. Without this rule, the pattern *.* would match only files that contained a dot, which would break probably 90% of all the batch files on the planet, as well as everybody's muscle memory, since everybody running Windows NT 3.1 grew up in a world where *.* meant all files.

    As another example, a pattern that ends in a dot doesn't actually match files which end in a dot; it matches files with no extension. And a question mark can match zero characters if it comes immediately before a dot.

    There may be other weird Win32 pattern matching quirks, but those are the two that come to mind right away, and they both exist to maintain batch file compatibility with the old 8.3 file pattern matching algorithm.

  • The Old New Thing

    AppInit_DLLs should be renamed Deadlock_Or_Crash_Randomly_DLLs

    • 26 Comments

    I have no idea why the window manager team added this feature to Windows NT. It basically says, "Hi, use this key to violate all the rules known to mankind about what can legitimately be done in a DllMain function. Oh, and be an attractive malware attack vector, too."

    I've debugged a few crashes that were traced back to the AppInit_DLLs key. What makes them particularly fun is that the offending DLL is usually not on the stack. Rather, the fact that a foreign DLL is being loaded inside USER32's initialization code means that you're violating the rule against calling LoadLibrary inside a DllMain function. The result of this madness is that DLLs get initialized out of order, and typically manifests itself in some DLL crashing trying to use an object (often a critical section) that it is supposed to have initialized in its DLL_PROCESS_ATTACH handler. It crashed because the loader got tricked into initializing DLLs out of order. The dependent DLL received its DLL_PROCESS_ATTACH before the prerequisite DLL.

    I end up looking at these failures because the victim DLL is often a DLL that my group is responsible for.

    The window manager folks came to the same conclusion about AppInit_DLLs, and it doesn't work any more in Windows Vista by default. (Nick Kramer describes how to re-enable it.)

Page 1 of 4 (37 items) 1234