December, 2012

  • The Old New Thing

    2012 year-end link clearance


    Another round of the semi-annual link clearance.



    Culture (or what passes for it)

    Other Stuff

    And, as always, the obligatory plug for my column in TechNet Magazine:

  • The Old New Thing

    Why can't you rename deleted items in the Recycle Bin?


    I misread a question from commenter Comedy Gaz, so let's try it again. (Good thing I held one last Suggestion Box Monday of the year in reserve.)

    Why can't you rename deleted items in the Recycle Bin?

    Okay, first of all, "Why would you want to do this?"

    I see no explanation for how this could possibly escape the 100-point hole every feature starts out in. I mean, these are items you deleted. Why do you care what their names are? Are you renaming it so you can find it again later? Why would you go to the effort of locating an item in the Recycle Bin, and then not bother recovering it? It's like calling the Lost and Found at Grand Central Terminal, and saying, "Hi, I left my umbrella on the Danbury train last Tuesday. It's blue with white snowflakes. Yes, that's the one. No, I don't want to come in and get it. Could you just dye it green, and paint a yellow smiley face on it? Thanks."

    The purpose of the Recycle Bin is not to provide another place where you can organize your data. The purpose of the Recycle Bin is to give you one last chance to recover the data you deleted by mistake!

    What would be the point of writing the code to allow the name to be edited, update the name in the Recycle Bin databases (watch out for the cross-process race conditions!), then locate all the other open Recycle Bin windows and tell them, "Hey, if you were showing the name of deleted item number 51462, please go refresh it, because it has a new name"? That's a lot of code to be written and tested (and re-tested every build) for a pretty dubious scenario in the first place. (And why stop at just the file name? Why not let people edit the Original Location and Date Deleted too?)

    From an information-theoretical standpoint, renaming an item in the Recycle Bin would be a falsification of the historical record. The information about the items in the Recycle Bin describe the item at the time it was deleted. Its name, the folder it was deleted from, the date it was deleted. If you could change the name of an item in the Recycle Bin, then that record would be incorrect. "This icon represents the file that you deleted from folder Q, except that the name I'm showing you isn't actually the name. It's some bogus name that somebody edited."

    It'd be like asking the church to go update its registry to change your birth name. "Yes, I know that I was born baptized with the name Amélie Bernadette, but please change your files so it says that I was baptized with the name Chloë Dominique. Thanks."

    The church isn't going to do that because that would now be lying. You were baptized with the name Amélie Bernadette. You are welcome to change your name to Chloë Dominique, but that doesn't change the fact that you were baptized with the name Amélie Bernadette.

  • The Old New Thing

    What do HeapLock and HeapUnlock do, and when do I need to call them?


    You never need to call the Heap­Lock and Heap­Unlock functions under normal operation. Assuming the heap is serialized (which is the default), all the standard heap functions like Heap­Allocate and Heap­Free will automatically serialize.

    In fact, the way they serialize is by calling the¹ Heap­Lock and Heap­Unlock functions!

    Nearly all heap operations complete in a single call. If your heap is serialized, this means that the heap operation takes the heap lock, does its work, and then releases the heap lock and returns. If all heap operations were like this, then there would be no need for Heap­Lock or Heap­Unlock.

    Unfortunately, there is also the Heap­Walk function, which does a little bit of work, and then returns with a partial result. The design for Heap­Walk is that the application calls the function repeatedly until it either gets all the results it wants, or gets bored and gives up. But wait, what if the heap changes while the application is trying to walk through it? To prevent that from happening, the program can call Heap­Lock before starting the enumeration, and Heap­Unlock when it is done. During the time the heap is locked, other threads which attempt to call a Heap­Xxx function with that same heap will block until the heap is unlocked.

    The ability to lock the heap creates a lot of potential for craziness, because the heap is a high-traffic area. As a result, it is very important that any code which calls Heap­Lock do very little while the lock is held. Take the lock, do your thing, and get out quickly.

    But wait, there's more. Holding the heap lock blocks all other threads from allocating or freeing memory. This puts the heap lock very low in your lock hierarchy. Therefore, while you hold the heap lock, you cannot block on synchronization objects whose owners might try to access the heap you just locked. Consider the following:

    // Code in italics is wrong.
    void BadIdea()

    Sending a message is a big deal. The thread that is the target of the message may be waiting for the heap lock, and now you've created a deadlock. You won't proceed until that thread processes the message, but that thread can't process the message until you unlock the heap.

    You might accidentally do something wrong while hold the heap lock if you happen to trigger a delay-loaded DLL, in which case your call into that other DLL turns into a call to Load­Library, and now you've lost control. In practice, the only thing you should be doing while holding the heap lock is calling Heap­Walk and saving the results locally, and in a way that doesn't allocate or free memory on the heap you are walking! Wait until after you unlock the heap to start studying the results you collected or transfer the raw data into a more suitable data structure.

    Bonus chatter

    Note that if you call Heap­Lock or Heap­Unlock on a heap that was created without serialization (HEAP_NO_SERIALIZATION), then the results are undefined. That's because passing the HEAP_NO_SERIALIZATION flag means "Hey, Heap Manager, don't bother locking this heap. I will take responsibility for ensuring that only one thread operates on this heap at a time." If you later call Heap­Lock on a no-serialization heap, the heap manager will say, "Wha? You said that you would take care of serialization, not me!"

    It's like ordering a car and saying, "Don't bother installing door locks. I will take responsibility for ensuring the safety of the car. (Say, by never letting the car leave a secured facility.)" And then a month later, calling OnStar and saying, "Hi, can you remotely lock my car for me? Thanks." Dude, you explicitly opted out of door locks.

    (Amazingly, I encountered one developer who thought that calling Heap­Lock on a no-serialization heap would cause other heap operations on the heap to be blocked, even if they passed the HEAP_NO_SERIALIZATION flag to those operations. Um, no, the Heap­Lock function cannot lock a no-serialization heap because a no-serialization heap doesn't have lock in the first place, at your request.)

    Nitpicker's corner

    ¹ s/the/the functional equivalents of/

  • The Old New Thing

    What is so special about the instance handle 0x10000000?

    A customer wanted to know what it means when the Load­Library function returns the special value 0x10000000.

    Um, it means that the library was loaded at 0x10000000?

    Okay, here's some more information: "We're trying to debug an application which loads DLLs and attempts to hook their registry accesses when they call Dll­Register­Server. It looks like when the special handle is returned from Load­Library, the registry writes go through and bypass the hook. On the other hand, when a normal value is returned by Load­Library, the hook works."

    There is nothing special about the value 0x10000000. It's an address like any other address.

    At this point, your psychic powers might start tingling. Everybody who does Win32 programming should recognize that 0x10000000 is the default DLL base address assigned by the linker. If you don't specify a custom base address, the linker will base you at 0x10000000.

    Now things are starting to make sense. The DLL being monitored was probably built with the default base address. The value 0x10000000 is special not because of its numeric value, but because it matches the DLL's preferred address, which means that no rebasing has occurred. And this in turn suggests that there's a bug in the registry hooks if the DLL is loaded at its preferred address.

    The code in question was copied from a book, so now they get to debug code copied from a book.

    Wait, we're not finished yet.

    You may have answered the customer's question, but you haven't solved their problem.

    Hooking and patching DLLs like this is not supported. But what is supported is the Reg­Override­Predef­Key function. In fact, the Reg­Override­Predef­Key was designed specifically to solve this very problem:

    The Reg­Override­Predef­Key function is intended for software installation programs. It allows them to remap a predefined key, load a DLL component that will be installed on the system, call an entry point in the DLL, and examine the changes to the registry that the component attempted to make.

    The documentation continues, explaining how such an installation program might use the Reg­Override­Predef­Key function to accomplish the desired task.

  • The Old New Thing

    What celebration of Kwanzaa would be complete without Sandra Lee's infamous Kwanzaa cake?


    For those unfamiliar with Sandra Lee, her gimmick is that the ingredients in her recipes are primarily prepared foods, a food-preparation technique (I'm reluctant to call it cooking) which she calls "semi-homemade."

    First, the creation one critic called "an edible hate crime."

    Next, the apology from the creator.

    I guess I imagined something more refined. And I know the Corn Nuts were disgusting, but she didn't. As a matter of fact, the more tasteless the recipes got the more she liked them, the faster she approved them.
  • The Old New Thing

    A question about endian-ness turns out to be the wrong question

    Via a customer liaison, we received what seemed like a simple question: "How can I detect whether a Windows machine is big-endian or little-endian?"

    You could actually answer this question (say by coughing up a code fragment that stores a 16-bit value to memory and then takes it apart into bytes to see how it got stored, or by simply hard-coding it based on the target architecture you are compiling for), but you'd be making the mistake of answering the question instead of solving the problem.

    The customer liaison explained, "My customer is having a problem that is caused by a bug in the SAP BI connector. According to the Knowledge Base article, the problem occurs when the SAP BI server is installed on a big-endian system."

    Okay, with that background, we immediately recognize that the question is wrong. The problem occurs when the SAP BI server is running on a big-endian system. It doesn't matter what the endian-ness of the Windows machine is, so any mechanism for detecting whether the Windows machine is big-endian or little-endian is barking up the wrong tree.

    But it turns out that the customer never even had to do this detection at all. If you read the Knowledge Base article, it says that the problem is already fixed.

    The fix for this issue was first released in Cumulative Update 4 for SQL Server 2008 Service Pack 1.

    So just make sure you're running Cumulative Update 4 for SQL Server 2008 Service Pack 1 or higher (which, if you've been making any attempt at keeping your server up to date, you've been doing for three years), and the problem will go away.

    The customer liaison thanked us for our assistance, but nevertheless asked for the code that would detect the endian-ness of the Windows system. I asked, "How will that help you solve your problem?" but before the customer liaison answered, some other people just gave the customer code that detects the machine endian-ness.

    Even though that will do absolutely nothing to solve the customer's problem.

    That was the last we heard from the customer liaison. I'm hoping that they actually installed the service pack and solved their problem. And I'm afraid what they're going to do with that code fragment.

  • The Old New Thing

    You too can use your psychic powers: Spaces in paths


    I'm going to be lazy today and make you solve the problem.

    Did a recent security hotfix change change the way we handle UNC paths with spaces? Normally, if we open the Run dialog and type \\server\share\Support Library\, the Support Library folder opens. But recently it stopped working. Instead of opening the Support Library folder, we get the Open With dialog:

    Open with

    Choose the program you want to use to open this file:

    File: Support

      Contoso Chess Challenger 5.0
      Contoso Music Studio
      Fabrikam Chart 2.0
      Litware 2010
    $   Woodgrove Bank Online
      Happy Holidays!
      Always use the selected program to open this kind of file


    Can you figure out what happened recently that introduced this problem? You have all the information you need.

  • The Old New Thing

    Miss Denmark, she is dressed as a mermaid because we couldn't secure the rights from LEGO


    Once again, the commentary is NSFW, but once again, they so deserve it.

    Miss Universe 2012 National Costumes, Part 1 my favorite is the commentary for the Dominican Republic)
    Miss Universe 2012 National Costumes, Part 2
    Miss Universe 2012 National Costumes, Part 3 (but Venezuela wins the things on top her head category)

    By the way, here's your dress made of LEGO.

    And for completeness, because I forgot to link to it last year:

    Miss Universe 2011 National Costumes, Part 1
    Miss Universe 2011 National Costumes, Part 2

  • The Old New Thing

    What is the proper handling of WM_RENDERFORMAT and WM_RENDERALLFORMATS?


    Jeremy points out that the documentation for Set­Clipboard­Data says that the clipboard owner must not call Open­Clipboard when responding to the WM_RENDER­FORMAT and WM_RENDER­ALL­FORMATS messages. On the other hand, the documentation for WM_RENDER­ALL­FORMATS says that the owner must call Open­Clipboard and Empty­Clipboard. Which is it?

    It's none of them!

    Let's start with WM_RENDER­FORMAT. The reference implementation for a WM_RENDER­FORMAT handler goes like this, with all error handling deleted for expository purposes:

     hData = GenerateFormat(cf);
     SetClipboardData(cf, hData);
     return 0;

    In response to WM_RENDER­FORMAT, you simply place the format on the clipboard. No opening is required. In fact, attempting to open will fail because the clipboard is already open: It has been opened by the application whose call to Get­Clipboard­Data triggered the delay-render!

    Next comes WM_RENDER­ALL­FORMATS. The original reference implementation goes like this, again with error checking deleted:

    // code in italics is wrong -- see discussion below
     SendMessage(hwnd, WM_RENDERFORMAT, CF_FORMAT1, 0);
     SendMessage(hwnd, WM_RENDERFORMAT, CF_FORMAT2, 0);
     return 0;

    In response to WM_RENDER­ALL­FORMATS, you open the clipboard, then render all your formats into it, and then close the clipboard. And one to render your formats is simply to send yourself a fake WM_RENDER­FORMAT message, which gets the code in the earlier code block to generate the format and place it on the clipboard.

    So you see that everybody is wrong!

    The WM_RENDER­ALL­FORMATS handler does call Open­Clipboard—if you tried it without the Open­Clipboard call, you'd notice that the data never made it to the clipboard—and it doesn't call Empty­Clipboard. (If you did, you'd notice that the Empty­Clipboard would have wiped out your non-delay-rendered data!)

    Where did I get these reference implementations from? I got them from the Windows 3.1 SDK. (And that explains the bug; read on.)

    In real life, you probably would also listen for the WM_DESTROY­CLIPBOARD message so you would know that you are no longer the clipboard owner, in which case you wouldn't bother rendering anything.

    I haven't written code in a while, so let's write some code. Start with our scratch program and make these changes. We'll start by writing it incorrectly:

    OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)
     if (OpenClipboard(hwnd)) {
      SetClipboardData(CF_UNICODETEXT, NULL);
     return TRUE;
    const WCHAR c_szText[] = L"hello";
    OnRenderFormat(HWND hwnd, UINT fmt)
     if (fmt == CF_UNICODETEXT)
      HGLOBAL hglob;
      if (SUCCEEDED(CreateHGlobalFromBlob(
                                c_szText, sizeof(c_szText),
                                GMEM_MOVEABLE, &hglob))) {
       if (!SetClipboardData(fmt, hglob)) GlobalFree(hglob);
     return 0;
    OnRenderAllFormats(HWND hwnd)
     if (OpenClipboard(hwnd)) {
      OnRenderFormat(hwnd, CF_UNICODETEXT);
        HANDLE_MSG(hwnd, WM_RENDERFORMAT, OnRenderFormat);
        HANDLE_MSG(hwnd, WM_RENDERALLFORMATS, OnRenderAllFormats);

    This program puts delay-rendered text on the clipboard when it starts up, When the request for text arrives, we just return the word hello. If we are asked to render all our formats, we render all our formats by calling our internal function once for each format we support. (All one of them.)

    There's a tiny race condition in that implementation above, though. What if somebody takes ownership of the clipboard while you're trying to render all your formats? Let's force the race condition. Set a breakpoint on the On­Render­All­Formats function, run the program, and close the window. The breakpoint will hit.

    Switch away from the debugger and open Notepad. Type 123 into Notepad, then select it and type Ctrl+C to copy it to the clipboard.

    Notepad will hang for a while, since the window manager is trying to send a WM_DESTROY­CLIPBOARD message to tell the previous clipboard owner that it is no longer responsible for the data on the clipboard. Let the call time out, at which point Notepad will wake back up and put 123 text on the clipboard. Now resume execution of the scratch program, so that it puts the Unicode word hello onto the clipboard.

    Okay, go back to Notepad and hit Ctrl+V. Look, it pasted hello instead of 123. Oops, our delay-rendering program destroyed the clipboard as it exited. If the application had put something more complicated on the clipboard, then our scratch program would have created a mishmash of old and new data.

    To protect against this race condition, make the following small change:

    OnRenderAllFormats(HWND hwnd)
     if (OpenClipboard(hwnd)) {
      if (GetClipboardOwner() == hwnd) {
        OnRenderFormat(hwnd, CF_UNICODETEXT);

    After opening the clipboard, we check if we are still the window responsible for the clipboard contents. Only if so do we render our delay-rendered formats.

    Exercise: Why is the Get­Clipboard­Owner test done after the Open­Clipboard? Wouldn't it be better to bail out quickly if we are not the clipboard owner and avoid opening the clipboard in the first place?

  • The Old New Thing

    Why do BackupRead and BackupWrite require synchronous file handles?


    The Backup­Read and Backup­Write functions require that the handle you provide by synchronous. (In other words, that they not be opened with FILE_FLAG_OVERLAPPED.)

    A customer submitted the following question:

    We have been using asynchronous file handles with the Backup­Read. Every so often, the call to Backup­Read will fail, but we discovered that as a workaround, we can just retry the operation, and it will succeed the second time. This solution has been working for years.

    Lately, we've been seeing crash when trying to back up files, and the stack traces in the crash dumps appear to be corrupted. The issue appears to happen only on certain networks, and the problem goes away if we switch to a synchronous handle.

    Do you have any insight into this issue? Why were the Backup­Read and Backup­Write functions designed to require synchronous handles?

    The Backup­Read and Backup­Write functions have historically issued I/O against the handles provided on the assumption that they are synchronous. As we saw a while ago, doing so against an asynchronous handle means that you're playing a risky game: If the I/O completes synchronously, then nobody gets hurt, but if the I/O goes asynchronous, then the temporary OVERLAPPED structure on the stack will be updated by the kernel when the I/O completes, which could very well be after the function that created it has already returned. The result: A stack smash. (Related: Looking at the world through kernel-colored glasses.)

    This oversight in the code (blindly assuming that the handle is a synchronous handle) was not detected until 10 years after the API was originally designed and implemented. During that time, backup applications managed to develop very tight dependencies on the undocumented behavior of the backup functions. The backup folks tried fixing the bug but found that it ended up introducing massive compatibility issues. On top of that, there was no real business case for extending the Backup­Read and Backup­Write functions to accept asynchronous handles.

    As a result, there was no practical reason for changing the function's behavior. Instead, the requirement that the handle be synchronous was added to the documentation, along with additional text explaining that if you pass an asynchronous handle, you will get "subtle errors that are very difficult to debug."

    In other words, the requirement that the handles be synchronous exists for backward compatibility.

Page 1 of 3 (26 items) 123