• The Old New Thing

    News flash: Companies change their product to appeal to their customers

    • 24 Comments

    There was some apparent uproar because there was an industry which "changed the flavoring of their product depending on which market segment they were trying to appeal to."

    Well duh, don't all industries do this?

    The reason why this even remotely qualified as news didn't appear until the last five words of the article!

  • The Old New Thing

    The window manager needs a message pump in order to call you back unexpectedly

    • 8 Comments

    There are a bunch of different ways of asking the window manager to call you when something interesting happens. Some of them are are in response to things that you explicitly asked for right now. The enumeration functions are classic examples of this. If you call EnumWindows and pass a callback function, then that callback is called directly from the enumerator.

    On the other hand, there is a much larger class of things that are in response either to things that happen on another thread, or in response to things that happen on your thread, but not as a direct result of an immediate request. For example, if you use the SendMessageCallback function, and the window manager needs to trigger your callback, the window manager needs a foot in the door of your thread in order to get control. It can't just interrupt code arbitrarily; that way lies madness. So we're looking for some way the window manager can regain control of the CPU at a time when the program is in a stable, re-entrant state.

    That foot in the door for the window manager is the message pump. That's the one component that the window manager can have some confidence that the program is going to call into periodically. This solves the first problem: How do I get control of the CPU.

    Furthermore, it's a known quantity for programs that when you call GetMessage or PeekMessage, incoming sent messages are dispatched, so your program had better be in a stable, re-entrant state when you call those functions. That solves the second problem: How do I get control of the CPU when the program is in a stable state?

    Take-away: When you register a callback with the window manager, you need to pump messages. Otherwise, the window manager has no way of calling you back.

    Related: The alertable wait is the non-GUI analog to pumping messages.

  • The Old New Thing

    If you're waiting for I/O to complete, it helps if you actually have an I/O to begin with

    • 12 Comments

    We saw earlier the importance of waiting for I/O to complete before freeing the data structures associated with that I/O. On the other hand, before you start waiting, you have to make sure that you have something to wait for.

    A customer reported a hang in Get­Overlapped­Result waiting for an I/O to cancel, and the I/O team was brought in to investigate. They looked at the I/O stack and found that the I/O the customer was waiting for was no longer active. The I/O people considered a few possibilities.

    • The I/O was active at one point, but when it completed, a driver bug prevented the completion event from being signaled.
    • The I/O was active at one point, and the I/O completed, but the program inadvertently called Reset­Event on the handle, negating the Set­Event performed by the I/O subsystem.
    • The I/O was never active in the first place.

    These possibilities are in increasing order of likelihood (and, perhaps not coincidentally, decreasing order of relevance to the I/O team).

    A closer investigation of the customer's code showed a code path in which the Read­File call was bypassed. When the bypass code path rejoined the mainline code path, the code continued its work for a while, and then if it decided that it was tired of waiting for the read to complete, it performed a Cancel­Io followed by a Get­Overlapped­Result to wait for the cancellation to complete.

    If you never issue the I/O, then a wait for the I/O to complete will wait forever, since you're waiting for something that will never happen.

    Okay, so maybe this was a dope-slap type of bug. But here's something perhaps a little less self-evident:

    // there is a flaw in this code - see discussion
    // assume operating on a FILE_FLAG_OVERLAPPED file
    if (ReadFile(h, ..., &overlapped)) {
     // I/O completed synchronously, as we learned earlier
    } else {
     // I/O under way
     ... do stuff ...
     // okay, let's wait for that I/O
     GetOverlappedResult(h, &overlapped, &dwRead, TRUE);
     ...
    }
    

    The Get­Overlapped­Result call can hang here because the comment "I/O is under way" is overly optimistic: The I/O may never even have gotten started. If it never started, then it will never complete either. You cannot assume that a FALSE return from Read­File implies that the I/O is under way. You also have to check that Get­Last­Error() returns ERROR_IO_PENDING. Otherwise, the I/O failed to start, and you shouldn't wait for it.

    // assume operating on a FILE_FLAG_OVERLAPPED file
    if (ReadFile(h, ..., &overlapped)) {
     // I/O completed synchronously, as we learned earlier
    } else if (GetLastError() == ERROR_IO_PENDING) {
     // I/O under way
     ... do stuff ...
     // okay, let's wait for that I/O
     GetOverlappedResult(h, &overlapped, &dwRead, TRUE);
     ...
    } else {
     // I/O failed - don't wait because there's nothing to wait for!
    }
    
  • The Old New Thing

    Charlie Sheen v Muammar Gaddafi: Whose line is it anyway?

    • 23 Comments

    I got seven out of ten right.

  • The Old New Thing

    Although the x64 calling convention reserves spill space for parameters, you don't have to use them as such

    • 18 Comments

    Although the x64 calling convention reserves space on the stack as spill locations for the first four parameters (passed in registers), there is no requirement that the spill locations actually be used for spilling. They're just 32 bytes of memory available for scratch use by the function being called.

    We have a test program that works okay when optimizations are disabled, but when compiled with full optimizations, everything appears to be wrong right off the bat. It doesn't get the correct values for argc and argv:

    int __cdecl
    wmain( int argc, WCHAR** argv ) { ... }
    

    With optimizations disabled, the code is generated correctly:

            mov         [rsp+10h],rdx  // argv
            mov         [rsp+8],ecx    // argc
            sub         rsp,158h       // local variables
            mov         [rsp+130h],0FFFFFFFFFFFFFFFEh
            ...
    

    But when we compile with optimizations, everything is completely messed up:

            mov         rax,rsp 
            push        rsi  
            push        rdi  
            push        r13  
            sub         rsp,0E0h 
            mov         qword ptr [rsp+78h],0FFFFFFFFFFFFFFFEh 
            mov         [rax+8],rbx    // ??? should be ecx (argc)
            mov         [rax+10h],rbp  // ??? should be edx (argv)
    

    When compiler optimizations are disabled, the Visual C++ x64 compiler will spill all register parameters into their corresponding slots. This has as a nice side effect that debugging is a little easier, but really it's just because you disabled optimizations, so the compiler generates simple, straightforward code, making no attempts to be clever.

    When optimizations are enabled, then the compiler becomes more aggressive about removing redundant operations and using memory for multiple purposes when variable lifetimes don't overlap. If it finds that it doesn't need to save argc into memory (maybe it puts it into a register), then the spill slot for argc can be used for something else; in this case, it's being used to preserve the value of rbx.

    You see the same thing even in x86 code, where the memory used to pass parameters can be re-used for other purposes once the value of the parameter is no longer needed in memory. (The compiler might load the value into a register and use the value from the register for the remainder of the function, at which point the memory used to hold the parameter becomes unused and can be redeployed for some other purpose.)

    Whatever problem you're having with your test program, there is nothing obviously wrong with the code generation provided in the purported defect report. The problem lies elsewhere. (And it's probably somewhere in your program. Don't immediately assume that the reason for your problem is a compiler bug.)

    Bonus chatter: In a (sadly rare) follow-up, the customer confessed that the problem was indeed in their program. They put a function call inside an assert, and in the nondebug build, they disabled assertions (by passing /DNDEBUG to the compiler), which means that in the nondebug build, the function was never called.

    Extra reading: Challenges of debugging optimized x64 code. That .frame /r command is real time-saver.

  • The Old New Thing

    No, not that M, the other M, the one called Max

    • 29 Comments

    Code names are rampant at Microsoft. One of the purposes of a code name is to impress upon the people who work with the project that the name is only temporary, and that the final name will come from the marketing folks (who sometimes pull through with a catchy name like Zune, and who sometimes drop the ball with a dud like Bob and who sometimes cough up monstrosities like Microsoft WinFX Software Development Kit for Microsoft® Pre-Release Windows Operating System Code-Named "Longhorn", Beta 1 Web Setup).

    What I find amusing are the project which change their code names. I mean, the code name is already a placeholder; why replace a placeholder with another placeholder?

    One such example is the experimental project released under the code name Max. The project founders originally named it M. Just the letter M. Not to be confused with this thing code named M or this other thing code named M.

    In response to a complaint from upper management about single-letter code names, the name was changed to Milkshake, and the team members even made a cute little mascot figure, with a straw coming out the top of his head like a milkshake.

    I'm not sure why the name changed a second time. Perhaps those upper level managers didn't think Milkshake was a dignified-enough name. For whatever reason, the name changed yet again, this time to Max. (Wikipedia claims that the project was named after the pet dog of one of the team members; I have been unable to confirm this. Because I haven't bothered trying.)

    There's no real punch line here, sorry. Just one example of the naming history of a project that went by many names.

    Bonus chatter: Apparently the upper management folks who complained about the single-letter code name M were asleep when another product was code-named Q (now known as Windows Home Server).

  • The Old New Thing

    On the linguistic productivity of the word spam

    • 13 Comments

    The word spam has spawned off its own corner of the English language.

    spamdexing inflating one's rank in a Web index
    sping Trackback spam
    splog A fake blog
    spim Spam instant messages
    spasms Spam SMS messages
    spit VoIP spam
    sporgery Spamming with a forged From line
    spambot A computer which sends spam
    blam spam comments on a blog

    I'm impressed by how productive the root word spam has become.

    Gratuitous cross-promotion: My colleague Terry Zink occasionally writes about spam on his Cyber Security Blog.

  • The Old New Thing

    If you want to use GUIDs to identify your files, then nobody's stopping you

    • 37 Comments

    Igor Levicki proposes solving the problem of file extensions by using a GUID instead of a file name to identify a file.

    You can do this already. Every file on an NTFS volume has an object identifier which is formally 16-byte buffer, but let's just call it a GUID. By default a file doesn't have an object identifier, but you can ask for one to be created with FSCTL_CREATE_OR_GET_OBJECT_ID, which will retrieve the existing object identifier associated with a file, or create one if there isn't one already. If you are a control freak, you can use FSCTL_SET_OBJECT_ID to specify the GUID you want to use as the object identifier. (The call fails if the file already has an object identifier.) And of course there is FSCTL_GET_OBJECT_ID to retrieve the object identifier, if any.

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <stdio.h>
    #include <tchar.h>
    #include <ole2.h>
    #include <winioctl.h>
    
    int __cdecl _tmain(int argc, PTSTR *argv)
    {
     HANDLE h = CreateFile(argv[1], 0,
                     FILE_SHARE_READ | FILE_SHARE_WRITE |
                     FILE_SHARE_DELETE, NULL,
                     OPEN_EXISTING, 0, NULL);
     if (h != INVALID_HANDLE_VALUE) {
      FILE_OBJECTID_BUFFER buf;
      DWORD cbOut;
      if (DeviceIoControl(h, FSCTL_CREATE_OR_GET_OBJECT_ID,
                     NULL, 0, &buf, sizeof(buf),
                     &cbOut, NULL)) {
        GUID guid;
        CopyMemory(&guid, &buf.ObjectId, sizeof(GUID));
        WCHAR szGuid[39];
        StringFromGUID2(guid, szGuid, 39);
        _tprintf(_T("GUID is %ws\n"), szGuid);
      }
      CloseHandle(h);
     }
     return 0;
    }
    

    This program takes a file or directory name as its sole parameter and prints the associated object identifier.

    Big deal, now we have a GUID associated with each file.

    The other half is, of course, using this GUID to open the file:

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <stdio.h>
    #include <tchar.h>
    #include <ole2.h>
    
    int __cdecl _tmain(int argc, PTSTR *argv)
    {
     HANDLE hRoot = CreateFile(_T("C:\\"), 0,
                     FILE_SHARE_READ | FILE_SHARE_WRITE |
                     FILE_SHARE_DELETE, NULL,
                     OPEN_EXISTING,
                     FILE_FLAG_BACKUP_SEMANTICS, NULL);
     if (hRoot != INVALID_HANDLE_VALUE) {
      FILE_ID_DESCRIPTOR desc;
      desc.dwSize = sizeof(desc);
      desc.Type = ObjectIdType;
      if (SUCCEEDED(CLSIDFromString(argv[1], &desc.ObjectId))) {
       HANDLE h = OpenFileById(hRoot, &desc, GENERIC_READ,
                     FILE_SHARE_READ | FILE_SHARE_WRITE |
                     FILE_SHARE_DELETE, NULL, 0);
       if (h != INVALID_HANDLE_VALUE) {
        BYTE b;
        DWORD cb;
        if (ReadFile(h, &b, 1, &cb, NULL)) {
         _tprintf(_T("First byte of file is 0x%02x\n"), b);
        }
        CloseHandle(h);
       }
      }
      CloseHandle(hRoot);
     }
     return 0;
    }
    

    To open a file by its GUID, you first need to open something—anything—on the volume the file resides on. Doesn't matter what you open; the only reason for having this handle is so that OpenFileById knows which volume you're talking about. In our little test program, we use the C: drive, which means that the file search will take place on the C: drive.

    Next, you fill in the FILE_ID_DESCRIPTOR, saying that you want to open the file by its object identifier, and then it's off to the races with OpenFileById. Just as a proof of concept, we read and print the first byte of the file that was opened as a result.

    Notice that the file you open by its object identifier does not have to be in the current directory. It can be anywhere on the C: drive. As long as you have the GUID for a file, you can open it no matter where it is on the drive.

    You can run these two programs just to enjoy the thrill of opening a file by its GUID. Notice that once you get the GUID for a file, you can move it anywhere on the drive, and OpenFileById will still open it.

    (And if you want to get rid of those pesky drive letters, you can use the volume GUID instead. Now every file is identified by a pair of GUIDs: the volume GUID and the object identifier.)

    So Igor's dream world where all files are referenced by GUID already exists. Why isn't everybody switching over to this utopia of GUID-based file identification?

    You probably know the answer already: Because people prefer to name things with something mnemonic rather than a GUID. Imagine a file open dialog in this dream world. "Enter the GUID of the file you wish to open, or click Browse to see the GUIDs of all the files on this volume so you can pick from a list." How long would this dialog survive?

    For today, you don't have to call me Raymond. You can call me {7ecf65a0-4b78-5f9b-e77c-8770091c0100}, or "91c" for short.

    (And I've totally ignored the fact that using GUIDs to identify files does nothing to solve the problem of trying to figure out what program should be used to open a particular file.)

    Bonus chatter: You can also open files by their file identifer, which is a volume-specific 64-bit value. But I chose to use the GUID both for the extra challenge, and just to show that Igor's dream world already exists.

  • The Old New Thing

    DRUNK HULK's insightful commentary in all-caps (and faux-rudimentary English)

    • 7 Comments

    DRUNK HULK. Just read it.

    • 11:07 AM Feb 5th: DRUNK HULK SAW SPEED 3: LAKE HOUSE! WHAT BORE WAY TO END TRILOGY!
    • 6:58 AM Feb 8th: DRUNK HULK HOPE SMALLVILLE FINALE ANSWER ALL BIG QUESTION! LIKE HOW STUPID SHOW STAY ON AIR FOR 10 YEAR!
    • 7:33 AM Jan 19th: DRUNK HULK INTEREST IN BUY KINDLE 3! BUT WILL DRUNK HULK UNDERSTAND WHAT GO ON IF NO READ PREVIOUS ONE!
  • The Old New Thing

    Why does WaitForMultipleObjects return ERROR_INVALID_PARAMETER when all the parameters look valid to me?

    • 14 Comments

    A customer asked for assistance with the WaitForMultipleObjects function:

    I am getting ERROR_INVALID_PARAMETER when calling Wait­For­Multiple­Objects even though all the parameters are valid as far as I can tell. I've narrowed it down to this simple program.

    int main()
    {
     int i;
     HANDLE Handles[4];
    
     // Create the events
     for (i = 0; i < 4; i++) {
      Handles[i] = CreateEvent(NULL, FALSE, FALSE, TEXT("Test"));
      if (Handles[i] == NULL) {
       printf("Failed to create event - test failed\n"); return 0;
      }
     }
    
     // Set them all to signaled
     for (i = 0; i < 4; i++) SetEvent(Handles[i]);
    
     // Wait for all of them - we expect this to return WAIT_OBJECT_0
     printf("WaitForMultipleObjects returned %d\n",
            WaitForMultipleObjects(4, Handles, TRUE, INFINITE));
    
     return 0;
    }
    

    First of all, thank you for narrowing the issue down to a minimal program that illustrates the problem. You'd be surprised how often a customer says, "I'm having problem with function X. Here's a program that illustrates the problem." And then attaches a huge project that doesn't compile because it is written in some development environment different from the one you have on your machine.

    The problem here is that you are passing four handles to the same event to Wait­For­Multiple­Objects with the bWait­All parameter set to TRUE. The Wait­For­Multiple­Objects function rejects duplicates if you ask it to wait for all of the objects. Why is that?

    Well, consider this program: It creates a named auto-reset event (as is "obvious" from the FALSE second parameter passed to Create­Event) and stores a handle to it in Handles[0]. The second through fourth calls to Create­Event merely create new handles to the same auto-reset event because the name matches an existing event. The second loop sets that same event four times. And then the Wait­For­Multiple­Objects asks to wait for all of the handles to be signaled. But since all four handles refer to the same object, it's being asked to wait until the event has reached the state where the wait can complete four times simultaneously. (Huh?)

    Recall that Wait­For­Multiple­Objects does not alter the state of any of the waited objects until the wait completes. If you ask it to wait for both an event and a semaphore, and the event is signaled but the semaphore is not, then the function will leave the event signaled while it waits for the semaphore. Only when all the items being waited for are signaled will the Wait­For­Multiple­Objects function perform whatever actions are appropriate for acquiring a signaled object and return.

    Okay, so we asked it to wait on the same auto-reset event four times. But that's nonsense: An auto-reset event is just a stupid semaphore which can have at most one token. But in order for the wait to succeed, it needs four tokens. That's never going to happen, so the wait is nonsensical.

    More generally speaking, Wait­For­Multiple­Objects returns ERROR_INVALID_PARAMETER if you pass bWaitAll = TRUE and there are any duplicates in the handle array (either identical handles, or different handles to the same underlying object). It doesn't try to puzzle out the objects and say, "Well, let me see if this is a reasonable combination of objects to wait on more than once"; it just sees the duplicate and says "Forget this!"

    Going back to the customer's original problem: We asked why they were creating four handles to the same object, and what they expected when waiting for an auto-reset event to have four available tokens (which it never will), and the customer admitted that it was just an error in their code. The original version of the code used a named event and waited on it with Wait­For­Single­Object, and when they modified the code to make it support multiple events, they forgot to give each event a different name.

Page 124 of 441 (4,404 items) «122123124125126»