• The Old New Thing

    Why does a corrupted binary sometimes result in "Program too big to fit in memory"?


    If you take a program and corrupt the header, or just take a large-ish file that isn't a program at all and give it a ".exe" extension, then try to run it (Warning: Save your work first!), you will typically get the error "Program too big to fit in memory". Why such a confusing error message? Why doesn't it say "Corrupted program"?

    Because the program isn't actually corrupted. Sort of.

    A Win32 executable file begins with a so-called "MZ" header, followed by a so-called "PE" header. If the "PE" header cannot be found, then the loader attempts to load the program as a Win16 executable file, which consists of an "MZ" header followed by an "NE" header.

    If neither a "PE" nor an "NE" header can be found after the "MZ" header, then the loader attempts to load the program as an MS-DOS relocatable executable. If not even an "MZ" header can be found, then the loader attempt to load the program as an MS-DOS non-relocatable executable (aka "COM format" since this is the format of CP/M .COM files).

    In pictures:

    elseMS-DOS relocatable
    else MS-DOS non-relocatable

    Observe that no matter what path you take through the chart, you will always end up at something. There is no exit path that says "Corrupted program".

    But where does "Program too big to fit in memory" come from?

    If the program header is corrupted, then various fields in the header such as those which specify the amount of memory required by the program will typically be nonsensical values. The loader sees an MS-DOS relocatable program that requires 800KB of conventional memory, and that's where "Out of memory" comes from.

    An MS-DOS non-relocatable program contains no such information about memory requirements. The rule for loading non-relocatable programs is simply to load the program into a single 64KB chunk of memory and set it on its way. Therefore, a program with no "MZ" header but which is larger than 64KB in size won't fit in the single 64KB chunk and consequently results in an "Out of memory" error.

    And since people are certain to ask:

    • "MZ" = the legendary Mark Zbikowski.
    • "NE" = "New Executable", back when Windows was "new".
    • "PE" = "Portable Executable", because one of Windows NT's claims to fame was its portability to architectures other than the x86.
    • "LE" = "Linear Executable", used by OS/2 and by Windows 95 device drivers.
  • The Old New Thing

    Liquefied NBA points apparently cannot be sold on eBay


    Of course we don't find this out until somebody tries. (Google cache for a limited time only.)

  • The Old New Thing

    Waiting for all handles with MsgWaitForMultipleObjects is a bug waiting to happen


    The MsgWaitForMultipleObjects and MsgWaitForMultipleObjectsEx functions allow you to specify whether you want to want for any or all of the handles (either by passing bWaitAll = TRUE or by passing dwFlags = MWMO_WAITALL, accordingly). But you never want to wait for all handles.

    Waiting for all handles means that the call does not return unless all the handles are signalled and a window message meeting your wake criteria has arrived. Since you are typically waiting for messages out of a sense of obligation (keeping the UI thread responsive) rather than one of expectation, even if all the handles you pass become signalled, your program will still stall until a window message arrives. (And if you thought you were being a good UI citizen by using MsgWaitForMultipleObjectsEx, you aren't actually helping any because waiting for all objects means that the call will not return even if a message is ready, since it's also waiting for those handles you passed.) Functions which are built on top of the MsgWaitForMultipleObjectsEx function such as MsgWaitForMultipleObjects and CoWaitForMultipleHandles suffer from the same problem.

    The reason for this can be gleaned from the MsgWaitForMultipleObjectsEx documentation; you just have to put on your thinking cap. Notice that if a message arrives when you are waiting for any handle, the return value is WAIT_OBJECT_0 + cHandles. Notice also that the maximum number of objects you can wait on is MAXIMUM_WAIT_OBJECTS - 1. Obviously, what's happening under the covers is that the MsgWaitForMultipleObjectsEx function creates a handle that will be signalled when the message queue reaches one of the states you requested in the wake mask, adds that handle to the end of the array you passed in, and then passes the whole thing to the WaitForMultipleObjectsEx function. (Note that the getting access to that internal handle won't be of any use to you, the application, since you don't know how to tell the window manager what wait states should result in the event being set.)

    (Larry Osterman reminded me that he covered the same topic a while back. So now you get to see it twice.)

  • The Old New Thing

    Stephen Tolouse's reminiscences of Windows 95 RTM day


    Stephen Tolouse (known around Microsoft as "stepto", pronounced "step-toe") from the Microsoft Security Response Center reminisces about Windows 95 RTM.

    Stephen mentions that "the build numbers were artificially inflated to reach 950". There's actually a technical reason for this inflation, which I intend to write about when I have the time to give the topic the treatment it deserves.

  • The Old New Thing

    Pumping messages while waiting for a period of time


    We can use the MsgWaitForMultipleObjects function (or its superset MsgWaitForMultipleObjectsEx) to carry out a non-polling "sleep while processing messages".

    #define MSGF_SLEEPMSG 0x5300
    BOOL SleepMsg(DWORD dwTimeout)
     DWORD dwStart = GetTickCount();
     DWORD dwElapsed;
     while ((dwElapsed = GetTickCount() - dwStart) < dwTimeout) {
      DWORD dwStatus = MsgWaitForMultipleObjectsEx(0, NULL,
                        dwTimeout - dwElapsed, QS_ALLINPUT,
                        MWFMO_WAITANY | MWMO_INPUTAVAILABLE);
      if (dwStatus == WAIT_OBJECT_0) {
       MSG msg;
       while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)) {
        if (msg.message == WM_QUIT) {
         return FALSE; // abandoned due to WM_QUIT
        if (!CallMsgFilter(&msg, MSGF_SLEEPMSG)) {
     return TRUE; // timed out

    This function pumps messages for up to dwTimeout milliseconds. The kernel of the idea is merely to use the MsgWaitForMultipleObjects/Ex function as a surrogate for WaitMessageTimeout, pumping messages until the cumulative timeout has been reached. There are a lot of small details to pay heed to, however. I've linked them to earlier postings that discuss the specific issues, if you need a refresher. The CallMsgFilter you might find gratuitous, but you'll change your mind when you realize that users might press a keyboard accelerator while you're sleeping, and you presumably want it to go through somebody's TranslateAccelerator. The message filter lets you hook into the modal loop and do your accelerator translation.

    Extending this function to "wait on a set of handles up to a specified amount of time, while pumping messages" is left as an exercise. (You can do it without changing very many lines of code.)

    [Call the right function. -2pm]

  • The Old New Thing

    Welcome to the United States, unless you're a Canadian technologist who is an invited guest at a Microsoft conference, in which case, keep out


    Vancouver technologist Darren Barefoot was invited to Redmond by the MSN Search team but was stopped by Immigration and denied entry.

    Ultimately, the customs agents concluded that because Microsoft was covering my flight and accommodation, I was being compensated for consulting activities. In order to enter the country, I'd need a work permit.

    He says Customs, but I'm pretty sure it was really Immigration that stopped him.

    Sorry, Darren. We miss you already. We even asked the rain to take a few days off and invited the sun out to play just for you.

  • The Old New Thing

    You can call MsgWaitForMultipleObjects with zero handles


    There is no WaitMessageTimeout function, but you can create your own with the assistance of the MsgWaitForMultipleObjects function.

    BOOL WaitMessageTimeout(DWORD dwTimeout)
     return MsgWaitForMultipleObjects(
        0, NULL, FALSE, dwTimeout, QS_ALLINPUT)
          == WAIT_TIMEOUT;

    To wait for a message with timeout, we use the MsgWaitForMultipleObjects in a vacuous sense: You pass it a list of objects you want to wait for, as well as a timeout and a set of queue states, asking that the function return when any of the objects is signalled or when a message is ready. By passing no objects, the only thing left to wait for is an incoming message.

    Next time, we'll see how this basic idea can be used to build a slightly more complex function.

    [1/26: Fix call to MsgWaitForMultipleObjects; had it confused with MsgWaitForMultipleObjectsEx. That's what happens when you write entries on an airplane.]

  • The Old New Thing

    Creepy messages in your baggage


    Heather Leigh had a creepy experience with her baggage last week. The story continued yesterday. I'm watching to see how things turn out.

  • The Old New Thing

    Performance consequences of polling


    Polling kills.

    A program should not poll as a matter of course. Doing so can have serious consequences on system performance. It's like checking your watch every minute to see if it's 3 o'clock yet instead of just setting an alarm.

    First of all, polling means that a small amount of CPU time gets eaten up at each poll even though there is nothing to do. Even if you tune your polling loop so its CPU usage is only, say, a measly one tenth of one percent, once this program is placed on a Terminal Server with 800 simultaneous connections, your 0.1% CPU has magnified into 80% CPU.

    Next, the fact that a small snippet of code runs at regular intervals means that it (and all the code that leads up to it) cannot be pruned from the system's working set. They remain present just to say "Nope, nothing to do." If your polling code touches any instance data (and it almost certainly will), that's a minimum of one page's worth of memory per instance. On an x86-class machine, that 4K times the number of copies of the program running. On that 800-user Terminal Server machine, you've just chewed up 3MB of memory, all of which is being kept hot just in case some rare event occurs.

    Finally, polling has deleterious effects even for people who aren't running humongous Terminal Server machines with hundreds of users. A single laptop will suffer from polling, because it prevents the CPU from going to more power-efficient sleep states, resulting in a hotter laptop and shorter battery life.

    Of course, Windows itself is hardly blame-free in this respect, but the performance team remains on the lookout for rogue polling in Windows and "politely reminds" teams they find engaging in polling that they should "strongly consider" other means of accomplishing what they're after.

  • The Old New Thing

    There are two types of rebates, and you need to be on the alert


    Many commenters to my earlier entry on sales in France had questions about rebates. Slate explained the whole rebate thing back in 2003. The short version: There are two types of rebates, manufacturer rebates and retailer rebates. Manufacturer rebates exist because they want the retail price to go down, but they are afraid that if they just lowered the wholesale price, retailers would not pass the savings on to the consumer. A manufacturer's rebate ensures that all the benefit of the price drop goes to the consumer and not to any middle-men. Retailer rebates, on the other hand, are carefully crafted schemes designed to trick the consumer into buying the product and then failing to meet all the requirements for redeeming the rebate coupon. Read the Slate article for details.

Page 355 of 460 (4,596 items) «353354355356357»