December, 2003

  • The Old New Thing

    What is the command line length limit?

    • 16 Comments

    It depends on whom you ask.

    The maximum command line length for the CreateProcess function is 32767 characters. This limitation comes from the UNICODE_STRING structure.

    CreateProcess is the core function for creating processes, so if you are talking directly to Win32, then that's the only limit you have to worry about. But if you are reaching CreateProcess by some other means, then the path you travel through may have other limits.

    If you are using the CMD.EXE command processor, then you are also subject to the 8192 character command line length limit imposed by CMD.EXE.

    If you are using the ShellExecute/Ex function, then you become subject to the INTERNET_MAX_URL_LENGTH (around 2048) command line length limit imposed by the ShellExecute/Ex functions. (If you are running on Windows 95, then the limit is only MAX_PATH.)

    While I'm here, I may as well mention another limit: The maximum size of your environment is 32767 characters. The size of the environment includes the all the variable names plus all the values.

    Okay, but what if you want to pass more than 32767 characters of information to a process? You'll have to find something other than the command line. We'll discuss some options tomorrow.

  • The Old New Thing

    What's the difference between CreateMenu and CreatePopupMenu?

    • 16 Comments
    CreateMenu creates a horizontal menu bar, suitable for attaching to a top-level window. This is the sort of menu that says "File, Edit", and so on.

    CreatePopupMenu creates a vertical popup menu, suitable for use as a submenu of another menu (either a horizontal menu bar or another popup menu) or as the root of a context menu.

    If you get the two confused, you can get strange menu behavior. Windows on rare occasions detects that you confused the two and converts as appropriate, but I wouldn't count on Windows successfully reading your mind.

    There is no way to take a menu and ask it whether it is horizontal or vertical. You just have to know.

    Answers to other questions about menus:

    When a window is destroyed, its menu is also destroyed. When a menu is destroyed, the entire menu tree is destroyed. (All its submenus are destroyed, all the submenu's submenus, etc.) And when you destroy a menu, it had better not be the submenu of some other menu. That other menu would have an invalid menu as a submenu!

    If you remove a submenu from its parent, then you become responsible for destroying it, since it no longer gets destroyed automatically when the parent is destroyed.

    It is legal for a menu to be a submenu of multiple parent menus. Be extra careful when you do this, however, because if one of the parents is destroyed, it will destroy the submenu with it, leaving the other parent with an invalid submenu.

    And finally: The menu nesting limit is currently 25 on Windows XP. That may change in the future, of course. (As with window nesting, Windows 95 let you go ahead and nest menus all you wanted. In fact, you could go really evil and create an infinite loop of menus. You crashed pretty quickly thereafter, of course...)

  • The Old New Thing

    What is the window nesting limit?

    • 10 Comments

    In the old days, Windows didn't bother enforcing a nesting limit because, well, if you want to nest windows 200 deep, that's your decision. Many window operations are recursive, but since everything happened on the application's stack, it was your own responsibility to make your stack big enough so it didn't overflow.

    But Windows NT moved the window manager off the application stack (first into a separate process, then into kernel mode). So now the OS needs to watch out for stack overflow attacks from people creating too many nested windows.

    The window nesting limit was set to 100 for the early days of Windows NT. For Windows XP, it dropped to 50 because increased stack usage in some internal functions caused us to overflow at around 75. Dropping to 50 created some breathing room.

    Disclaimer: I was not personally involved in this issue. I'm just reporting what I was able to figure out from reading checkin logs.

  • The Old New Thing

    When programs grovel into undocumented structures...

    • 58 Comments

    Three examples off the top of my head of the consequences of grovelling into and relying on undocumented structures.

    Defragmenting things that can't be defragmented
    In Windows 2000, there are several categories of things that cannot be defragmented. Directories, exclusively-opened files, the MFT, the pagefile... That didn't stop a certain software company from doing it anyway in their defragmenting software. They went into kernel mode, reverse-engineered NTFS's data structures, and modified them on the fly. Yee-haw cowboy! And then when the NTFS folks added support for defragmenting the MFT to Windows XP, these programs went in, modified NTFS's data structures (which changed in the meanwhile), and corrupted your disk.

    Of course there was no mention of this illicit behavior in the documentation. So when the background defragmenter corrupted their disks, Microsoft got the blame.

    Parsing the Explorer view data structures
    A certain software company decided that they wanted to alter the behavior of the Explorer window from a shell extension. Since there is no way to do this (a shell extension is not supposed to mess with the view; the view belongs to the user), they decided to do it themselves anyway.

    From the shell extension, they used an undocumented window message to get a pointer to one of the internal Explorer structures. Then they walked the structure until they found something they recognized. Then they knew, "The thing immediately after the thing that I recognize is the thing that I want."

    Well, the thing that they recognize and the thing that they want happened to be base classes of a multiply-derived class. If you have a class with multiple base classes, there is no guarantee from the compiler which order the base classes will be ordered. It so happened that they appeared in the order X,Y,Z in all the versions of Windows this software company tested against.

    Except Windows 2000.

    In Windows 2000, the compiler decided that the order should be X,Z,Y. So now they grovelled in, saw the "X" and said "Aha, the next thing must be a Y" but instead they got a Z. And then they crashed your system some time later.

    So I had to create a "fake X,Y" so when the program went looking for X (so it could grab Y), it found the fake one first.

    This took the good part of a week to figure out.

    Reaching up the stack
    A certain software company decided that it was too hard to take the coordinates of the NM_DBLCLK notification and hit-test it against the treeview to see what was double-clicked. So instead, they take the address of the NMHDR structure passed to the notification, add 60 to it, and dereference a DWORD at that address. If it's zero, they do one thing, and if it's nonzero they do some other thing.

    It so happens that the NMHDR is allocated on the stack, so this program is reaching up into the stack and grabbing the value of some local variable (which happens to be two frames up the stack!) and using it to control their logic.

    For Windows 2000, we upgraded the compiler to a version which did a better job of reordering and re-using local variables, and now the program couldn't find the local variable it wanted and stopped working.

    I got tagged to investigate and fix this. I had to create a special NMHDR structure that "looked like" the stack the program wanted to see and pass that special "fake stack".

    I think this one took me two days to figure out.

    I hope you understand why I tend to go ballistic when people recommend relying on undocumented behavior. These weren't hobbyists in their garage seeing what they could do. These were major companies writing commercial software.

    When you upgrade to the next version of Windows and you experience (a) disk corruption, (b) sporadic Explore crashes, or (c) sporadic loss of functionality in your favorite program, do you blame the program or do you blame Windows?

    If you say, "I blame the program," the first problem is of course figuring out which program. In cases (a) and (b), the offending program isn't obvious.

  • The Old New Thing

    Why not just block the apps that rely on undocumented behavior?

    • 47 Comments
    Because every app that gets blocked is another reason for people not to upgrade to the next version of Windows. Look at all these programs that would have stopped working when you upgraded from Windows 3.0 to Windows 3.1.
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Compatibility
    

    Actually, this list is only partial. Many times, the compatibility fix is made inside the core component for all programs rather than targetting a specific program, as this list does.

    (The Windows 2000-to-Windows XP list is stored in your C:\WINDOWS\AppPatch directory, in a binary format to permit rapid scanning. Sorry, you won't be able to browse it easily. I think the Application Compatibility Toolkit includes a viewer, but I may be mistaken.)

    Would you have bought Windows XP if you knew that all these programs were incompatible?

    It takes only one incompatible program to sour an upgrade.

    Suppose you're the IT manager of some company. Your company uses Program X for its word processor and you find that Program X is incompatible with Windows XP for whatever reason. Would you upgrade?

    Of course not! Your business would grind to a halt.

    "Why not call Company X and ask them for an upgrade?"

    Sure, you could do that, and the answer might be, "Oh, you're using Version 1.0 of Program X. You need to upgrade to Version 2.0 for $150 per copy." Congratulations, the cost of upgrading to Windows XP just tripled.

    And that's if you're lucky and Company X is still in business.

    I recall a survey taken a few years ago by our Setup/Upgrade team of corporations using Windows. Pretty much every single one has at least one "deal-breaker" program, a program which Windows absolutely must support or they won't upgrade. In a high percentage of the cases, the program in question was developed by their in-house programming staff, and it's written in Visual Basic (sometimes even 16-bit Visual Basic), and the person who wrote it doesn't work there any more. In some cases, they don't even have the source code any more.

    And it's not just corporate customers. This affects consumers too.

    For Windows 95, my application compatibility work focused on games. Games are the most important factor behind consumer technology. The video card that comes with a typical computer has gotten better over time because games demand it. (Outlook certainly doesn't care that your card can do 20 bajillion triangles a second.) And if your game doesn't run on the newest version of Windows, you aren't going to upgrade.

    Anyway, game vendors are very much like those major corporations. I made phone call after phone call to the game vendors trying to help them get their game to run under Windows 95. To a one, they didn't care. A game has a shelf life of a few months, and then it's gone. Why would they bother to issue a patch for their program to run under Windows 95? They already got their money. They're not going to make any more off that game; its three months are over. The vendors would slipstream patches and lose track of how many versions of their program were out there and how many of them had a particular problem. Sometimes they wouldn't even have the source code any more.

    They simply didn't care that their program didn't run on Windows 95. (My favorite was the one that tried to walk me through creating a DOS boot disk.)

    Oh, and that Application Compatibility Toolkit I mentioned above. It's a great tool for developers, too. One of the components is the Verifier: If you run your program under the verifier, it will monitor hundreds of API calls and break into the debugger when you do something wrong. (Like close a handle twice or allocate memory with GlobalAlloc but free it with LocalAlloc.)

    The new application compatibility architecture in Windows XP carries with it one major benefit (from an OS development perspective): See all those DLLs in your C:\WINDOWS\AppPatch directory? That's where many of the the compatibility changes live now. The compatibility workarounds no longer sully the core OS files. (Not all classes of compatibility workarounds can be offloaded to a compatibility DLL, but it's a big help.)
  • The Old New Thing

    What order do programs in the startup group execute?

    • 22 Comments

    The programs in the startup group run in an unspecified order. Some people think they execute in the order they were created. Then you upgraded from Windows 98 to Windows 2000 and found that didn't work any more. Other people think they execute in alphabetical order. Then you installed a Windows XP multilingual user interface language pack and found that didn't work any more either.

    If you want to control the order that programs in the startup group are run, write a batch file that runs them in the order you want and put a shortcut to the batch file in your startup group.
  • The Old New Thing

    What's with those blank taskbar buttons that go away when I click on them?

    • 41 Comments

    Sometimes you'll find a blank taskbar button that goes away when you click on it. What's the deal with that?

    There are some basic rules on which windows go into the taskbar. In short:

    • If the WS_EX_APPWINDOW extended style is set, then it will show (when visible).
    • If the window is a top-level unowned window, then it will show (when visible).
    • Otherwise it doesn't show.

    (Though the ITaskbarList interface muddies this up a bit.)

    When a taskbar-eligible window becomes visible, the taskbar creates a button for it. When a taskbar-eligible window becomes hidden, the taskbar removes the button.

    The blank buttons appear when a window changes between taskbar-eligible and taskbar-ineligible while it is visible. Follow:

    • Window is taskbar-eligible.
    • Window becomes visible ? taskbar button created.
    • Window goes taskbar-ineligible.
    • Window becomes hidden ? since the window is not taskbar-eligible at this point, the taskbar ignores it.

    Result: A taskbar button that hangs around with no window attached to it.

    This is why the documentation also advises, "If you want to dynamically change a window's style to one that doesn't support visible taskbar buttons, you must hide the window first (by calling ShowWindow with SW_HIDE), change the window style, and then show the window."

    Bonus question: Why doesn't the taskbar pay attention to all windows as they come and go?

    Answer: Because that would be expensive. The filtering out of windows that aren't taskbar-eligible happens inside USER32 and it then notifies the taskbar (or anybody else who has installed a WH_SHELL hook) via one of the HSHELL_* notifications only if a taskbar-eligibie window has changed state. That way, the taskbar code doesn't get paged in when there's nothing for it to to.
  • The Old New Thing

    How do I pass a lot of data to a process when it starts up?

    • 18 Comments

    As we discussed yesterday, if you need to pass more than 32767 characters of information to a child process, you'll have to use something other than the command line.

    One method is to wait for the child process to go input idle, then FindWindow for some agreed-upon window and send it a WM_COPYDATA message. This method has a few problems:

    • You have to come up with some way of knowing that the child process has created its window so you can start looking for it. (WaitForInputIdle is helpful here.)
    • You have to make sure the window you found belongs to the child process and isn't just some other window which happens to have the same name by coincidence. Or, perhaps, not by coincidence: If there is more than once instance of the child process running, you will need to make sure you're talking to the right one. (GetWindowThreadProcessId is helpful here.)
    • You have to hope that nobody else manages to find the window and send it the WM_COPYDATA before you do. (If they do, then they have effectively taken over your child process.)
    • The child process needs to be alert for the possibility of a rogue process sending bogus WM_COPYDATA messages in an attempt to confuse it.

    The method I prefer is to use anonymous shared memory. The idea is to create a shared memory block and fill it with goodies. Mark the handle as inheritable, then spawn the child process, passing the numeric value of the handle on the command line. The child process parses the handle out of its command line and maps the shared memory block to see what's in it.

    Remarks about this method:

    • You need to be careful to validate the handle, in case somebody tries to be sneaky and pass you something bogus on your command line.
    • In order to mess with your command line, a rogue process needs PROCESS_VM_WRITE permission, and in order to mess with your handle table, it needs PROCESS_DUP_HANDLE permission. These are securable access masks, so proper choice of ACLs will protect you. (And the default ACLs are usually what you want anyway.)
    • There are no names that can be squatted or values that can be spoofed (assuming you've protected the process against PROCESS_VM_WRITE and PROCESS_DUP_HANDLE).
    • Since you're using a shared memory block, nothing actually is copied between the two processes; it is just remapped. This is more efficient for large blocks of data.

    Here's a sample program to illustrate the shared memory technique.

    #include <windows.h>
    #include <shlwapi.h>
    #include <strsafe.h>
    
    struct STARTUPPARAMS {
        int iMagic;     // just one thing
    };
    

    In principle, the STARTUPPARAMS can be arbitrarily complicated, but for illustrative purposes, I'm just going to pass a single integer.

    STARTUPPARAMS *CreateStartupParams(HANDLE *phMapping)
    {
        STARTUPPARAMS *psp = NULL;
        SECURITY_ATTRIBUTES sa;
        sa.nLength = sizeof(sa);
        sa.lpSecurityDescriptor = NULL;
        sa.bInheritHandle = TRUE;
        HANDLE hMapping = CreateFileMapping(INVALID_HANDLE_VALUE,
                    &sa, PAGE_READWRITE, 0,
                    sizeof(STARTUPPARAMS), NULL);
        if (hMapping) {
            psp = (STARTUPPARAMS *)
                        MapViewOfFile(hMapping, FILE_MAP_WRITE,
                            0, 0, 0);
            if (!psp) {
                CloseHandle(hMapping);
                hMapping = NULL;
            }
        }
    
        *phMapping = hMapping;
        return psp;
    }
    

    The CreateStartupParams function creates a STARTUPPARAMS structure in an inheritable shared memory block. First, we fill out a SECURITY_ATTRIBUTES structure so we can mark the handle as inheritable by child processes. Setting the lpSecurityDescriptor to NULL means that we will use the default security descriptor, which is fine for us. We then create a shared memory object of the appropriate size, map it into memory, and return both the handle and the mapped address.

    STARTUPPARAMS *GetStartupParams(LPSTR pszCmdLine, HANDLE *phMapping)
    {
        STARTUPPARAMS *psp = NULL;
        LONGLONG llHandle;
        if (StrToInt64ExA(pszCmdLine, STIF_DEFAULT, &llHandle)) {
            *phMapping = (HANDLE)(INT_PTR)llHandle;
            psp = (STARTUPPARAMS *)
                    MapViewOfFile(*phMapping, FILE_MAP_READ, 0, 0, 0);
            if (psp) {
                //  Now that we've mapped it, do some validation
                MEMORY_BASIC_INFORMATION mbi;
                if (VirtualQuery(psp, &mbi, sizeof(mbi)) >= sizeof(mbi) &&
                    mbi.State == MEM_COMMIT &&
                    mbi.BaseAddress == psp &&
                    mbi.RegionSize >= sizeof(STARTUPPARAMS)) {
                    // Success!
                } else {
                    // Memory block was invalid - toss it
                    UnmapViewOfFile(psp);
                    psp = NULL;
                }
            }
        }
        return psp;
    }
    

    The GetStartupParams function is the counterpart to CreateStartupParams. It parses a handle from the command line and attempts to map a view. If the handle isn't a file mapping handle, the call to MapViewOfFile will fail, so we get that part of the parameter validation for free. We use VirtualQuery to validate the size of the memory block. (We can't use a strict equality test since the value we get back will be rounded up to the nearest page multiple.)

    void FreeStartupParams(STARTUPPARAMS *psp, HANDLE hMapping)
    {
        UnmapViewOfFile(psp);
        CloseHandle(hMapping);
    }
    

    After we're done with the startup parameters (either on the creation side or the consumption side), we need to free them to avoid a memory leak. That's what FreeStartupParams is for.

    void PassNumberViaSharedMemory(HANDLE hMapping)
    {
        TCHAR szModule[MAX_PATH];
        TCHAR szCommand[MAX_PATH * 2];
        DWORD cch = GetModuleFileName(NULL, szModule, MAX_PATH);
        if (cch > 0 && cch < MAX_PATH &&
            SUCCEEDED(StringCchPrintf(szCommand, MAX_PATH * 2,
                      TEXT("\"%s\" %I64d"), szModule,
                      (INT64)(INT_PTR)hMapping))) {
            STARTUPINFO si = { sizeof(si) };
            PROCESS_INFORMATION pi;
            if (CreateProcess(szModule, szCommand, NULL, NULL,
                              TRUE,
                              0, NULL, NULL, &si, &pi)) {
                CloseHandle(pi.hProcess);
                CloseHandle(pi.hThread);
            }
        }
    }
    

    Most of the work here is just building the command line. We run ourselves (using the GetModuleFileName(NULL) trick), passing the numerical value of the handle on the command line, and passing TRUE to CreateProcess to indicate that we want inheritable handles to be inherited. Note the extra quotation marks in case our program's name contains a space.

    int CALLBACK
    WinMain(HINSTANCE hinst, HINSTANCE hinstPrev,
            LPSTR pszCmdLine, int nShowCmd)
    {
        HANDLE hMapping;
        STARTUPPARAMS *psp;
        if (pszCmdLine[0]) {
            psp = GetStartupParams(pszCmdLine, &hMapping);
            if (psp) {
                TCHAR sz[64];
                StringCchPrintf(sz, 64, TEXT("%d"), psp->iMagic);
                MessageBox(NULL, sz, TEXT("The Value"), MB_OK);
                FreeStartupParams(psp, hMapping);
            }
        } else {
            psp = CreateStartupParams(&hMapping);
            if (psp) {
                psp->iMagic = 42;
                PassNumberViaSharedMemory(hMapping);
                FreeStartupParams(psp, hMapping);
            }
        }
        return 0;
    }
    

    At last we put it all together. If we have a command line parameter, then this means that we are the child process, so we convert it into a STARTUPPARAMS and display the number that was passed. If we don't have a command line parameter, then this means that we are the parent process, so we create a STARTUPPARAMS, stuff the magic number into it (42, of course), and pass it to the child process.

    So there you have it: Passing a "large" (well, okay, small in this example, but it could have been megabytes if you wanted) amount of data securely to a child process.

  • The Old New Thing

    The unsafe device removal dialog

    • 27 Comments

    In a comment, somebody asked what the deal was with the unsafe device removal dialog in Windows 2000 and why it's gone in Windows XP.

    I wasn't involved with that dialog, but here's what I remember: The device was indeed removed unsafely. If it was a USB storage device, for example, there may have been unflushed I/O buffers. If it were a printer, there may have been an active print job. The USB stack doesn't know for sure (those are concepts at a higher layer that the stack doesn't know about) - all it knows is that it had an active channel with the device and now the device is gone, so it gets upset and yells at you.

    In Windows XP, it still gets upset but it now keeps its mouth shut. You're now on your honor not to rip out your USB drive before waiting two seconds for all I/O to flush, not to unplug your printer while a job is printing, etc. If you do, then your drive gets corrupted / print job is lost / etc. and you're on your own.
  • The Old New Thing

    How to stop delivery of telephone books

    • 29 Comments

    Like many of you (I suspect), I don't use the paper telephone book. If I want to look something up, I go online. Yet every year I get a dozen different telephone books. I don't like them because a telephone book sitting on my front porch screams, "Rob this house! Nobody's home!" Besides, it's a waste of paper.

    So for the past few years I've been trying to stop delivery of all the telephone books. It's harder than you think.

    The first hurdle is figuring out what the "please take me off your mailing list" number is. Because they sure don't advertise it.

    I've discovered that the best way to get through to somebody who can take you off the list is to call the "How to order more copies of this wonderful telephone directory."

    Note: WorldPages added another wrinkle to the procedure. You see, they misprinted their own telephone number. Why anybody would voluntarily pay money to be listed by a telephone directory company that can't even get their own phone number right is beyond me.

    You have to be polite but firm when dealing with these people. Qwest is particularly tricky. I had called last year to stop delivery of all three of their phone books (they have three!), but in June another one showed up. I called them and they confirmed, "Yes, I see that we have a zero on your account, I don't know what happened. Let me try again."

    Aside: Why does Qwest want my telephone number to stop delivery of my telephone book? They deliver the book to a house, not to a telephone.

    Anyway, so that seems to keep the telephone book delivery gnomes at bay, until December, when yet another Qwest telephone book arrives at my doorstep. So I call again.

    "Yes, we have you marked as 'do not deliver'."

    "So why did I get one?"

    "This wasn't one of our standard phone books. This was a promotional phone book."

    Aha, so when you say "Do not deliver" it doesn't actually mean, "Do not deliver." It just means "Don't deliver the one that I am specifically complaining about." But there's this double-secret phone book delivery list that you have to specifically ask to be removed from, and we're not going to tell you about it until you ask.

    "Please remove me from the promotional phone book delivery list as well."

    "I'm sorry, I can't do that. There is no way for us to enter that in our computers." Always blame the computers. One response to "Our computer can't do that" I haven't yet resorted to is "Well, then I guess you'll have to do it by hand.")

    "Who delivers the promotional phone books?"

    "We contract that out to a local delivery company."

    "Can I talk to them?"

    "Hang on a second."

    <wait>>

    "Okay, I can fill out a form so you don't receive promotional phone books either." (Aha, so she is going to do it by hand.)

    "Thank you. Good-bye."

    We'll see how long this lasts. I predict May.
Page 1 of 5 (45 items) 12345