February, 2005

  • The Old New Thing

    You cannot globally reserve user-mode address space

    • 41 Comments

    Occasionally, somebody asks for a way to reserve user-mode address space globally. In other words, they want to allocate address space in all processes in the system (current and future). Typically this is because they want to map some memory into each process and don't want to go through the trouble of designing the shared memory blocks so that they can be relocated.

    This is obviously not possible.

    Why obviously? Well, imagine if this were possible.

    "Imagine if this were possible" is one of the logic tests you can apply to a theory to see if it can possibly be true. Here, we're using it to determine whether a proposed behavior is possible. [Typo fixed 10am.] (There is a corresponding thought experiment, "Imagine if things actually worked that way.")

    What are the consequences of global address space allocation?

    Well, first of all, there's no guarantee that by the time you request your global address space, there will be any available addresses at all. Consider a program that uses every last scrape of user-mode address space. (It can do this by just calling VirtualAlloc(MEM_RESERVE) in a loop; since no memory is being committed, this doesn't actually require 2GB of RAM.) Run such a program and no global address space allocations are possible until that program exits.

    So even if it were possible, it wouldn't be reliable. Your program would have to be prepared for the situation where no global address space was available. Since you're going to have to write fallback code anyway, you didn't save yourself any work.

    Next, suppose it were possible, that there were some imaginary GlobalVirtualAlloc function. Well, then I can write a program that calls this imaginary function in a loop, sucking up all available global virtual address space, and run it as a non-administrator.

    I just violated security. (The general principle here is that a non-administrator should not be allowed to affect other users. We've already seen one scenario where a non-administrator can crash a program running as administrator due to insecure use of shared memory.)

    My imaginary program sucked up all global virtual address space, reducing the address space available to programs running as administrator. If there aren't many programs running on the system, my imaginary program will probably be able to suck up quite a lot of address space this way, which in turn will cause a corresponding reduction in address space to those administrative programs. I can therefore cause those programs to run out of address space sooner, resulting in premature failure (denial of service).

    Yes, you could decide that "global" address space reservation is available only to administrators, but that wouldn't help a lazy programmer, since the program would not work when run as a non-administrator - you have to write the fallback code anyway. Or you could decide that "global" address space reservation applies only to users within the same session with the same security token, but again, you have to write the fallback code anyway if the global reservation fails; you didn't save yourself any work. When the time comes, you can use VirtualAlloc and pass a preferred address to try to get the memory at that address; if it fails, then use the fallback code that you already wrote.

    The moral of the story is that each process gets its own address space and each process manages its address space independently of other processes. (Of course, a process can grant a user PROCESS_VM_OPERATION permission, which gives that user permission to mess with the address space of that process. But changes to that process's address space does not affect the address space of other processes.)

  • The Old New Thing

    How to act like you know Chinese even though you don't

    • 34 Comments

    I was riding in a car with a friend when a Chinese pop album spun up on the CD. I made a remark about the song, and my friend, knowing that I was studying Mandarin, asked me, "Do you know what she's singing about?"

    I immediately replied, "She's singing about love."

    "Wow," my friend responded. "Your Chinese is quite good."

    "Nope," I said. "But all pop songs are about love."

    If somebody asks you what a pop song is about, you can answer "love", and you've got a very high chance of being correct.

  • The Old New Thing

    LoadLibraryEx(DONT_RESOLVE_DLL_REFERENCES) is fundamentally flawed

    • 25 Comments

    There is a flag to the LoadLibraryEx function called DONT_RESOLVE_DLL_REFERENCES. The documentation says,

    If this value is used, and the executable module is a DLL, the system does not call DllMain for process and thread initialization and termination. Also, the system does not load additional executable modules that are referenced by the specified module.

    If you are planning only to access data or resources in the DLL, it is better to use LOAD_LIBRARY_AS_DATAFILE.

    In my opinion, the above text that "suggests" the LOAD_LIBRARY_AS_DATAFILE flag is not strong enough.

    DONT_RESOLVE_DLL_REFERENCES is a time bomb.

    Look carefully at what the flag does and doesn't do. The module is loaded into memory, but its initialization function is not called and no dependent DLLs are loaded. [Typo fixed, 10am.] As a result, you cannot run code from this DLL. (More accurately, if you try, it will crash because the DLL hasn't initialized itself and none of its imports to DLLs have been resolved.) However, unlike the LOAD_LIBRARY_AS_DATAFILE flag, the loaded DLL can be found by GetModuleHandle and can be used by GetProcAddress.

    Clearly, GetProcAddress is a bad idea for something loaded by DONT_RESOLVE_DLL_REFERENCES, because as we already noted, you can't run any code from the DLL. What's the point of getting a procedure address from a DLL if you can't call it, after all?

    The GetModuleHandle part triggers the time bomb.

    It is common for somebody to call GetModuleHandle to see if a DLL is loaded, and if so, use GetProcAddress to get a procedure address and call it. If the DLL had been loaded with DONT_RESOLVE_DLL_REFERENCES, both the GetModuleHandle will succeed, but the resulting function will crash when called. The code doing this has no idea that the DLL was loaded with DONT_RESOLVE_DLL_REFERENCES; it has no way of protecting itself.

    (Note that code that does this is unsafe anyway, because the code that originally loaded the DLL might decide to do a FreeLibrary on another thread, causing the code to be ripped out from underneath the first thread. This second problem can be "fixed" by using GetModuleHandleEx, which can be instructed to increment the DLL reference count, but that doesn't fix the first problem.)

    Even if you used LoadLibrary to load the DLL and passed that handle to GetProcAddress, you still crash, because the LoadLibrary notices that the DLL is already loaded and merely increments the reference count.

    #include <windows.h>
    
    typedef HINSTANCE (WINAPI *SXA)(HWND, LPCSTR, LPCSTR,
                                    LPCSTR, LPCSTR, int);
    
    int __cdecl main(int argc, char* argv[])
    {
     if (argc > 1) // set the time bomb
      LoadLibraryEx("shell32.dll", NULL, DONT_RESOLVE_DLL_REFERENCES);
    
     // victim code runs here
     HINSTANCE h = LoadLibrary("shell32.dll");
     if (h) {
      SXA f = (SXA)GetProcAddress(h, "ShellExecuteA");
      if (f) {
       f(NULL, NULL, "notepad.exe", NULL, NULL, SW_SHOWNORMAL);
      }
      FreeLibrary(h);
     }
    }
    

    If you run this program with no command line arguments, then everything works just fine: Notepad is launched without incident. However, if you pass a command line argument, this sets the time bomb, and the call to ShellExecuteA crashes in flames because shell32.dll was loaded without having its DLL references resolved.

    In other words, DONT_RESOLVE_DLL_REFERENCES is fundamentally flawed and should be avoided. It continues to exist solely for backwards compatibility.

  • The Old New Thing

    Windowless controls are not magic

    • 61 Comments

    It seems that when people notice that the Internet Explorer rendering engine doesn't use HWNDs for screen elements, they think that Internet Explorer is somehow "cheating" and doing something "undocumented" and has an "unfair advantage".

    Nevermind that windowless controls have been around since 1996. They aren't magic. Mind you, they're a lot of work, but they aren't magic.

    Clearly Internet Explorer cannot create a real HWND for every element in an HTML page. There is a limit of 10,000 USER handles per process, and you are likely to run out of desktop heap long before then.

    The Internet Explorer team went and reimplemented all of the controls that a web page would need. They have their own windowless checkbox control, a windowless listbox control, a windowless edit box, and so on. In addition to reproducing all the functionality of the windowed controls, the Internet Explorer folks also had to reproduce the "look" of the windowed controls, down to the last pixel. (Functions like DrawThemeBackground and DrawFrameControl prove extremely helpful here.)

    If I recall correctly, the only element that is still windowed is the <SELECT> element.

    If you squint, you can see some places where they didn't quite nail it. For example, if you right-click in a text box, options like "Right to left reading order" and "Insert Unicode control character" are missing. As another example, notice that IE's scroll bars do not light up when you hover over them.

    Remember, I never worked on Internet Explorer; all I know is what I learn from people from that team. (jeffdav for example, joins the shell team for lunch nearly every day.) If you have questions about Internet Explorer, you would likely have much better success asking the Internet Explorer team yourself via their team blog.

  • The Old New Thing

    Why do minimized windows have an apparent size of 160x31?

    • 31 Comments

    We discussed a few months ago the issue of where windows minimized to before the taskbar was invented. In the modern taskbar world, why do minimized windows have an apparent size of 160x31?

    The size isn't just apparent. That's really their size. You can see them, for example, if you fire up a program that uses the Multiple Document Interface.

    Observe the appearance of the window "Book1". This is a minimized window (though minimized to its MDI host rather than to the desktop). With the introduction of Windows Explorer, which put files on the desktop in the form of icons, it became necessary to change the appearance of minimized windows in order to avoid confusing a minimized program icon from a desktop icon. A minimized program, therefore, took the form of a miniature title bar.

    The programming interface to minimized windows remained the same, for compatibility reasons. (And please let's just agree to disagree on whether backwards compatibility is a good thing or not.) That's why the function to tell whether a window is minimized continues to be called IsIconic, the message you receive when someone tries to restore a minimized program is still called WM_QUERYOPEN, and the OpenIcon function can still be used to "open" a minimized "icon". All even though minimized windows haven't looked like icons for nearly ten years.

    The OpenIcon function is just an old-fashioned way of saying ShowWindow(hwnd, SW_NORMAL), in the same way that the CloseWindow function (dating back to Windows 1.0) is an extremely old-fashioned way of saying ShowWindow(hwnd, SW_MINIMIZE).

  • The Old New Thing

    The dangers of filtering window messages

    • 21 Comments

    The GetMessage and PeekMessage functions allow you to pass a filter, restricting the window handle or range of messages that the function will retrieve from the message queue. While it's okay to use these filters, make sure you eventually get around to making an unfiltered call so that any straggling messages can come through.

    A common mistake is to use a window-filtered GetMessage in your message loop. In our scratch program, a window-filtered GetMessage would look like this:

            while (GetMessage(&msg, hwnd, 0, 0)) { // Wrong!
                TranslateMessage(&msg);
                DispatchMessage(&msg);
            }
    

    Even though the program creates but one window, this program is nevertheless incorrect.

    "How can this be?" you ask. "My program has only one window. Why would there possibly be any messages for any other windows? The filter, while redundant, isn't harmful, is it?"

    Many system services create windows on your behalf. For example, if input method editing is enabled, the method editor may create helper windows to assist in character input. If you initialize COM, then COM may decide to create a helper window to assist in inter-thread marshalling. If you use only a filtered GetMessage, then messages destined for these helper windows will never be retrieved, and you will be left scratching your head wondering why your program occasionally hangs when it tries to perform a drag/drop operation, for example.

    Moral of the story: Make sure your message loop eventually performs an unfiltered message retrieval so that these services can operate properly.

  • The Old New Thing

    What other effects does DS_SHELLFONT have on property sheet pages?

    • 22 Comments

    Once you invent a new flag you can start using it to fix errors of the past without breaking backwards compatibility.

    One of the errors of the past was that property sheet page dimensions were taken relative to the "MS Sans Serif" font, even if the page used some other font.

    DLG_SAMPLE DIALOGEX 32, 32, 212, 188
    CAPTION "Caption"
    FONT "Lucida Sans Unicode"
    ...
    

    This sample dialog template says that it is 212dlu wide and 188dlu tall. If the dialog template were used for a standalone dialog, those DLU values would be calculated relative to the font on the dialog, namely Lucida Sans Unicode.

    However, if the dialog template were used for a property sheet page, earlier versions of Windows would interpret the values 212 and 188 relative to the font of the property sheet frame (usually MS Sans Serif), not relative to the font associated with the page itself. Many people worked around this problem by giving their pages pre-adjusted sizes, so that when Windows measured the dialog against MS Sans Serif, the adjustments cancelled out.

    In other words, suppose that Lucida Sans Unicode is 25% wider than MS Sans Serif. (I'm just making up numbers.) Then to get a 212dlu-wide dialog relative to Lucida Sans Unicode, the dialog template would specify a width of 212dlu + 25% = 265dlu.

    Since people were now relying on this behavior, it couldn't be changed. If you went in and "fixed" it, all those pre-adjusted dialogs would come out at the wrong size.

    Ah, but now there is a new flag, DS_SHELLFONT. Starting in Windows 2000, if you specify the DS_SHELLFONT dialog style for your DIALOGEX dialog template, then the dialog dimensions are taken relative to the font you specified in your template (which is probably what you wanted) rather than relative to the property sheet frame font. If you leave off the flag (as older programs will), then the property sheet measurement code remains bug-for-bug compatible with previous versions.

  • The Old New Thing

    Why does DS_SHELLFONT = DS_FIXEDSYS | DS_SETFONT?

    • 23 Comments

    You may have noticed that the numerical value of the DS_SHELLFONT flag is equal to DS_FIXEDSYS | DS_SETFONT.

    #define DS_SETFONT          0x40L   /* User specified font for Dlg controls */
    #define DS_FIXEDSYS         0x0008L
    #define DS_SHELLFONT        (DS_SETFONT | DS_FIXEDSYS)
    

    Surely that isn't a coincidence.

    The value of the DS_SHELLFONT flag was chosen so that older operating systems (Windows 95, 98, NT 4) would accept the flag while nevertheless ignoring it. This allowed people to write a single program that got the "Windows 2000" look when running on Windows 2000 and got the "classic" look when running on older systems. (If you make people have to write two versions of their program, one that runs on all systems and one that runs only on the newer system and looks slightly cooler, they will usually not bother writing the second one.)

    The DS_FIXEDSYS flag met these conditions. Older systems accepted the flag since it was indeed a valid flag, but they also ignored it because the DS_SETFONT flag takes precedence.

    This is one of those backwards-compatibility exercises: How do you design something so that it is possible to write one program that gets the new features on new systems while at the same time degrading gracefully on old systems?

  • The Old New Thing

    What's the deal with the DS_SHELLFONT flag?

    • 34 Comments

    It indicates that you want the Windows 2000 default shell font. But that doesn't mean that you're going to get it.

    In order to indicate that you would like the "Windows 2000" look for your dialog, you have to do three things and hope for a fourth:

    1. Use a DIALOGEX template instead of a DIALOG template.
    2. Set the DS_SHELLFONT flag in your dialog styles.
    3. Set the dialog font to "MS Shell Dlg".
    4. Hope that you're running on Windows 2000 or later on a system that has the new Windows 2000 dialog font enabled.

    If all four conditions are satisfied, then your dialog gets the "Windows 2000" look.

    If any condition fails, then you get the "classic" dialog font. Note that the fourth condition is not within your program's control. Consequently, you have to make sure your dialog looks good in both the "classic" dialog font and the new one.

    For property sheet pages, things are more complicated.

    It would be visually jarring for there to be a mix of fonts on a property sheet. You wouldn't want the "Advanced" button to be in MS Sans Serif but the "Apply" button in Tahoma. To avoid this problem, the property sheet manager looks at all the pages in the property sheet. If they are all using the "Windows 2000" look, then the property sheet uses the "Windows 2000" look also. But if there is even a single page that does not use the "Windows 2000" look, then the property sheet reverts to the "classic" look and also converts all the "Windows 2000"-look pages to "classic" look.

    This way, all the pages in the property sheet have the "classic" look instead of having a mishmash of some pages with the classic look and some with the Windows 2000 look.

    That's why you will occasionally find that a shell property sheet has reverted to the "classic" look. Some shell extension infected the property sheet with a page that does not have the "Windows 2000" look, and for the sake of visual consistency, the property sheet manager set all the pages on the property sheet to "classic" look.

    This is another reason it is important that you test your property sheet pages both with the "Windows 2000" look and the "classic" look. If your property sheet page ends up sharing a property sheet with a non-"Windows 2000"-look page, your page is going to be reverted to "classic" look.

  • The Old New Thing

    Why is breadth-first searching better for file system tree walking?

    • 17 Comments

    Earlier, Eric Lippert discussed one scenario where breadth-first searching is better than depth-first searching. Today, I'll tell you about another.

    If you go back to the old MS-DOS file enumeration functions, you'll find that there is a "Find first file" function and a "Find next file" function, but no "Find close" function. That's because the MS-DOS file enumeration functions maintained all their state in the find structure. The FAT file system was simple enough that the necessary search state fit in the reserved bytes and no external locking was necessary to accomplish the enumeration. (If you did something strange like delete a directory while there was an active enumeration on it, then the enumeration would start returning garbage. It was considered the program's responsibility not to do that. Life is a lot easier when you are a single-tasking system.)

    However, moving these functions to a network protocol or a non-FAT file system created problems. When enumerating files over a network, the server needs to allocate additional memory to track the enumeration, memory which it needs to free when the enumeration is complete. Similarly, non-FAT file systems may have additional state to track that (1) doesn't fit in the reserved space in the find structure, (2) shouldn't be stored in the reserved space (because it would compromise security), or (3) can't be stored in the reserved space (because the kernel needs to update it asynchronously when the state of the directory changes).

    Since MS-DOS has no "Find close" function, how do these alternate file systems know when it is safe to free the memory associated with the file search? [Context clarified - 10am] Clairevoyance is not yet a viable option, so the file system is forced to guess.

    Typically, a file enumeration is considered "abandoned" if it is not used for "a long time" or if too many pending file enumerations are in progress.

    Let's see how depth-first search compares to breadth-first search on these points.

    In a typical depth-first search, you recurse into a subdirectory as soon as you see it. Thus, if you are, say, six levels deep in recursion, you will have six active enumerations, one for each level. The most recent one is most active, and the one at the root is least active. This means that the enumeration at the root is likely to be considered "abandoned" by the server because it hasn't been used for the longest time, and because there is a lot of other enumeration activity going on in the meantime.

    On the other hand, with a breadth-first search, when a directory is found, it is not processed immediately but is rather put on a list for later consideration. Consequently, there is only one active enumeration, and it runs to completion before the next enumeration begins. This means that the enumeration is unlikely to be considered "abandoned" by the server.

    This problem is not purely theoretical. If you performed a depth-first search on a large directory tree on a network server from a MS-DOS program, you often found that the enumeration ended prematurely because the older enumerations became abandoned by the server. You also saw this from a Win32 program if you were enumerating against an older network server that was designed for MS-DOS clients (and which therefore doesn't implement FindClose).

    You can still use depth-first enumeration while avoiding the abandonment problem. Instead of recursing into a directory immediately after enumerating it, save it on a list. When you are finished enumerating the files in a directory, go back and process the list. You're still enumerating depth-first, but you are delaying the recursive call until after the directory enumeration has completed.

    Explorer uses this "delayed recursion" trick to avoid the problem of prematurely-terminated enumerations when walking through directories on these old servers.

Page 2 of 3 (22 items) 123