August, 2012

  • The Old New Thing

    Adventures in undefined behavior: The premature downcast


    A customer encountered the following problem:

    class Shape
        virtual bool Is2D() { return false; }
    class Shape2D : public Shape
        virtual bool Is2D() { return true; }
    Shape *FindShape(Cookie cookie);
    void BuyPaint(Cookie cookie)
        Shape2D *shape = static_cast<Shape2D *>(FindShape(cookie));
        if (shape->Is2D()) {
           .. do all sorts of stuff ...

    The Buy­Paint function converts the cookie back to a Shape object, and then checks if the object is a Shape2D object by calling Is­2D. If so, then it does some more stuff to figure out what type of paint to buy.

    (Note to nitpickers: The actual scenario was not like this, but I presented it this way to illustrate the point. If you say "You should've used RTTI" or "You should've had a BuyPaint method on the Shape class", then you're missing the point.)

    The programmers figured they'd save some typing by casting the result of Find­Shape to a Shape2D right away, because after all, since Is­2D is a virtual method, it will call the right version of the function, either Shape::Is­2D or Shape2D::Is­2D, depending on the actual type of the underlying object.

    But when compiler optimizations were turned on, they discovered that the call to Is­2D was optimized away, and the Buy­Paint function merely assumed that it was always operating on a Shape2D object. It then ended up trying to buy paint even for one-dimensional objects like points and lines.

    Compiler optimization bug? Nope. Code bug due to reliance on undefined behavior.

    The C++ language says (9.3.1) "If a nonstatic member function of a class X is called for an object that is not of type X, or of a type derived from X, the behavior is undefined." In other words, if you are invoking a method on an object of type X, then you are promising that it really is of type X, or a class derived from it.

    The code above violates this rule, because it is invoking the Is­2D method on a Shape2D*, which therefore comes with the promise "This really is a Shape2D object (or something derived from it)." But this is a promise the code cannot deliver on, because the object returned by Find­Shape might be a simple Shape.

    The compiler ran with the (false) promise and said, "Well, since you are guaranteeing that the object is at least a Shape2D, and since I have studied your code and determined that no classes which further derive from Shape2D override the Is­2D method, I have therefore proved that the final overrider is Shape2D::Is­2D and can therefore inline that method."

    Result: The compiler inlines Shape2D::Is­2D, which returns true, so the "if" test can be optimized out. The compiler can assume that Buy­Paint is always called with cookies that represent two-dimensional objects.

    The fix is to do the annoying typing that the original authors were trying to avoid:

    void BuyPaint(Cookie cookie)
        Shape *shape = FindShape(cookie);
        if (shape->Is2D()) {
          Shape2D *shape2d = static_cast<Shape2D *>(shape);
           .. do all sorts of stuff (with shape2d) ...
  • The Old New Thing

    I'm not calling a virtual function from my constructor or destructor, but I'm still getting a __purecall error


    Some time ago, I described what __purecall is for: It's to detect the cases where you call a virtual method with no implementation (a so-called pure virtual method). This can happen during object constructor or destruction, since those are times when you can validly have a partially-implemented object.

    Well, there's another case this can happen: If the object has already been destructed.

    If you call a method on an object that has already been destructed, the behavior is undefined. In practice, what happens is that the method runs on whatever leftover values are in memory where the object used to be. Depending on how lucky or unlucky you are, this may resemble the actual object closely enough that the method doesn't notice, or at least doesn't notice for a while, or maybe it can't even get off the ground.

    The __purecall error is a case where it can't even get off the ground. The leftover object still has a vtable, namely the vtable of the base class, the one that has __purecall entries for all the pure virtual functions. (It still has that vtable because that's the object's last identity before finally going invalid.) And the method you tried to call is a virtual method that is pure in the base class. Not only are you calling a pure virtual function after destruction has begun, you're calling it after destruction is complete.

  • The Old New Thing

    Psychic debugging: Why Chinese characters don't work


    Here's a walkthrough of an actual case of psychic debugging.

    I'm using the internal XYZ table-generating tool, but it appears to be useless in my case. Does this tool work? Right now, I'm just resorting to the manual method of generating my data table (Chinese and English).

    Hooray for vague questions! "It appears to be useless."

    But there were enough tiny clues in the question that made it possible to make a psychic guess.

    First of all, the parenthetical "Chinese and English" suggests that there are Chinese characters in the data being passed to the XYZ table-generating tool.

    Second of all, one might make a guess that the file encoding is wrong. Maybe the XYZ program expects the file to be in UTF-8 format, but you're saving it in UTF-16 format. Maybe the XYZ program supports only ASCII data.

    I never heard back, which could have meant that my psychic powers were right, or it could have meant that my psychic powers were wrong, but the person figured, "No point asking this guy, all he does is make random guesses based on incomplete information."

  • The Old New Thing

    Command line tool to manage Windows 7 Libraries, with source code


    A customer asked if there was a command-line tool for managing Windows 7 Libraries so that they could create and deploy libraries across their organization.

    Not only is there such a tool, it even comes with source code.

    The Shell Library Command Line Sample shows how to manage Windows 7 libraries programmatically via the IShell­Library interface. And it's actually a useful program on its own.

    Usage: shlib.exe SUBCOMMAND
    Displays and modifies the attributes of Shell Libraries.
    Supported commands:
      create      Creates a library at the specified path.
      info        Prints info about the given library.
      enum        Enumerates the folders in the library.
      setattrib   Modifies the attributes of the library.
      add         Adds the specified folder to the specified library.
      remove      Removes the specified folder from the library.
      setsaveloc  Sets the default save location of the library.
      resolve     Resolves the specified folder in the library.
      resolveall  Resolves all locations in the library in bulk.
      manage      Displays the Manage Library Dialog for the library.

    For example, to see all the folders that are part of your Documents library, use the command

    shlib enum FOLDERID_DocumentsLibrary

    Each of the commands has further sub-options.

    Usage: create shlib.exe [OPTIONS]  [...]
    Creates a library at the specified path.
     -create[:ARG]       Specifies that a new library should be created.
       <none>              Fail if the library already exists.
       overwrite           Overwrite any existing library.
       uniquename          Generate a unique name in case of conflict.
    The library may be specified by a file system path,
    or by a KNOWNFOLDERID (e.g. "FOLDERID_DocumentsLibrary").

    And the source code shows how to perform each of these actions programmatically.

    The IShellLibrary interface is also supported by the Windows API Code Pack for .NET, so you C# folks can use it too.

    (One thing I noticed that is missing from the sample is a seticon command. Shouldn't be hard to implement. You just call IShell­Library::Set­Icon.)

  • The Old New Thing

    Well at least nobody's parking there any more


    There is a field next to the Microsoft building I used to work in, and for a time, people parked their cars out on the field, presumably because finding a proper parking space in the garage became difficult due to overcrowding. To prevent people from parking in the field, Security placed a large log across the access to the field. The technique worked: Nobody parked in the field any more.

    Some months later, our building had a fire drill, and everybody dutifully filed out of the building and waited until the all-clear signal was given to return. Normally, people would wait in the field, because that is the designated assembly area for the building, but oh wait, Security blocked off access to the field. Instead, people waited in the fire lane.

    Good thing this was just a drill, because they would have gotten run over by a fire truck.

    I pointed out to the Life Safety Security representative who was running the fire drill that Parking Security had created a life safety hazard. My observations were duly noted, and it looks like somebody actually paid attention, because a few weeks later, the log was removed. Now if there's a real fire, we can actually reach our designated assembly area.

    I just found it ironic that the Security department created a safety hazard.

  • The Old New Thing

    How does the taskbar decide whether to show Recent or Frequent documents on the jump list?


    Some jump lists show Recent documents, and others show Frequent documents.

    (Images stolen from Sasha Goldshtein.)

    Our friend xpclient wanted to know how the taskbar decides which one to use, because it seemed random. (Ironically, xpclient is asking a question about the Window 7 client.)

    The default is to show Recent documents. But an application can customize its jump list, and in that case, the application can select which category to show based on the guidance in the documentation. (Or the application might choose to ignore the guidance in the documentation and show both.)

    Okay, but what about Explorer itself? You can find that Explorer will sometimes show Frequent locations (e.g., on My Computer) and sometimes Recent locations (e.g., on Control Panel). How is this possible if the decision is made on per-application basis?

    Because the taskbar uses the shell application user model to associate windows with applications. We've taken advantage of this model a number of times before, like when we prevented a program from being pinned, or made one application look like two or many. Explorer sets its jump lists to show Frequent, but if an Explorer window sets a custom application ID, then that window is treated by the taskbar as a separate program and therefore the Explorer default does not apply (and it gets Recent instead).

    To make things even more complicated, xpclient was running a third-party taskbar hacking application which manipulated application ID of windows which belong to other applications. And just like in real life, if you mess with stuff that isn't yours, you can cause trouble for the real owner. In this case, manipulating the application IDs broke Explorer's jump list customizations.

    This is another case of somebody doing some weird customization and then complaining that their system acts all strange (which often leads to the accusation that Microsoft developers are a bunch of idiots) when the strange behavior was caused by their customization. I mean, if you're running a taskbar hacking application, then you really shouldn't be surprised if the taskbar occasionally behaves erratically.

    Bonus chatter: The Recent and Frequent categories are variable-length, which you can customize by setting the Number of recent items to display in Jump Lists.

  • The Old New Thing

    Dumping a hash table with external chaining from the debugger


    I was doing some crash dump debugging, as I am often called upon to do, and one of the data structure I had to grovel through was something that operated basically like an atom table, so that's what I'll call it. Like an atom table, it manages a collection of strings. You can add a string to the table (getting a unique value back, which we will call an atom), and later you can hand it the atom and it will give you the string back. It looked something like this:

    struct ENTRY
      ENTRY *next;
      UINT   atom;
      PCWSTR value;
    #define ATOMTABLESIZE 19
    struct ATOMTABLE
      ENTRY *buckets[ATOMTABLESIZE];

    (It didn't actually look like this; I've reduced it to the smallest example that still illustrates my point.)

    As part of my debugging, I had an atom and needed to look it up in the table. A program would do this by simply calling the "here is an atom, please give me the string" function, but since this was a crash dump, there's nothing around to execute anything. (Not that having a live machine would've made things much easier, because this was a kernel-mode crash, so you don't get any of this edit-and-continue froofy stuff. This is real debugging™.)

    But even though the crashed system can't execute anything, the debugger can execute stuff, and the debugger engine used by kd comes with its own mini-programming language. Here's how I dumped the atom table:

    // note: this was entered all on one line
    // broken into two lines for readability
    0: kd> .for (r $t0=0; @$t0 < 0n19; r $t0=@$t0+1)
             { dl poi (fffff8a0`06b69930+@$t0*8) 99 2 }
    fffff8a0`06ad3b90  fffff8a0`037a3fc0 fffff8a0`0000000c \
    fffff8a0`037a3fc0  fffff8a0`037a4ae0 00000000`00000025 | $t0=0
    fffff8a0`037a4ae0  fffff8a0`02257580 00000000`00000036 |
    fffff8a0`02257580  00000000`00000000 00000000`00000056 /
    fffff8a0`06ad3b30  fffff8a0`06ad3ad0 a9e8a9d8`0000000d \
    fffff8a0`06ad3ad0  fffff8a0`037a4700 000007fc`0000000e |
    fffff8a0`037a4700  fffff8a0`01f96fb0 00000000`0000003f | $t0=1
    fffff8a0`01f96fb0  fffff8a0`06cfa5d0 fffff8a0`00000044 |
    fffff8a0`06cfa5d0  00000000`00000000 00181000`00000060 /
    fffff8a0`033e7a70  fffff8a0`037a4770 00000020`00000023 \
    fffff8a0`037a4770  fffff8a0`023b52f0 00000000`0000003e | $t0=2
    fffff8a0`023b52f0  fffff8a0`03b6e020 006f0063`00000059 |
    fffff8a0`03b6e020  00000000`00000000 00000000`00000075 /
    fffff8a0`037a0670  fffff8a0`02b08870 fffff8a0`00000026 \ $t0=3
    fffff8a0`03b9e390  00000000`00000000 00240000`00000071 /

    Let's take that weirdo command apart one piece at a time.

    The overall command is

    .for (a; b; c) { d }

    This operates the same as the C programming language. (Sorry, Delphi programmers, for yet another C-centric example.) In our case, we use the $t0 pseudo-register as our loop control.

    • r $t0=0 — this sets $t0 to zero
    • @$t0 < 0n19 — this stops once $t0 reaches 19.
    • r $t0=@$t0+1 — this increments $t0.

    Note that here as well as in the loop body, I prefix the register name with the @ character when I want to obtain its value, in order to force it to be interpreted as a register. (Otherwise, the debugger will look for a symbol called $t0.)

    The command being executed at each iteration is { dl poi (fffff8a0`06b69930+@$t0*8) 99 2 }. Let's break this down, too:

    • dl — this command dumps a singly-linked list.
    • poi (fffff8a0`06b69930+@$t0*8) — this is the head of the linked list. In this example, 0xfffff8a0`06b69930 is the address of the buckets array, so we add the loop counter times the size of a pointer (8, in this case) to get the address of the $t0'th entry, and then dereference it (poi) to get the address of the head of the linked list.
    • 99 — This is the maximum length of the linked list. I picked an arbitrary large-enough number. I like using 9's because it carries the most value per keypress. Other people like to use nice round numbers like 1000, but 999 saves you a whole keypress and is just one less. (On the other hand, I don't use fff because that runs the risk of being misinterpreted as a symbol.)
    • 2 — This is the number of pointer-sized objects to dump at the start of each node. For 32-bit code, I use 4, whereas for 64-bit code, I use 2. Why those values? Because those are the maximum number of pointer-sized objects that the debugger will print on one line.

    Okay, so now I have that linked list dump. The value I'm looking for is the atom whose value is 0x3F, so I skim down the last column looking for 0000003f, and hey there it is.

    fffff8a0`037a4700  fffff8a0`01f96fb0 00000000`0000003f

    Now I have my ENTRY, and I can dump it to see what the corresponding value is.

    0: kd> dt contoso!ENTRY fffff8a0`037a4700
        +0x000 next: 0xfffff8a0`01f96fb0
        +0x008 atom: 0x0000003f
        +0x010 value: 0xffff8a0`01f97e20 -> "foo"
  • The Old New Thing

    Wait, you never said that I had to initialize the object before I used it!


    A customer reported that they were having trouble creating slim reader/writer locks at runtime. They simplified the issue to a short program:

    #include <windows.h>
    #include <iostream>
    using namespace std; // this is just a quick test app
    int a = 10;
    // This version works
    int working_version()
     SRWLOCK lock;
     cout<<"Acquired exclusively"<<endl;
    // This one doesn't
    int broken_version_1()
     SRWLOCK *lock = new SRWLOCK;
     cout<<"Acquired exclusively"<<endl;
     // ignore the memory leak - this is just a quick test app
    // This one doesn't either
    int broken_version_2()
     SRWLOCK *lock = new SRWLOCK[2];
     cout<<"Acquired exclusively"<<endl;
     // ignore the memory leak - this is just a quick test app
    int main(int argc, char **argv)
     switch (argv[1][0]) {
     case '0': working_version(); break;
     case '1': broken_version_1(); break;
     case '2': broken_version_2(); break;
     return 0;

    "What is the correct way of creating an SRWLOCK via the new operator?"

    It wasn't long before somebody noted that nowhere in the code is the function Initialize­SRW­Lock called.

    "Oh, yeah, thanks for catching that. It looks like one needs to initialize SRW locks which are created via the new operator. Otherwise it's not required."

    No, the function is always required. It's just that you got lucky in the local variable case and the initial stack garbage looks enough like an initialized SRW lock that you don't notice the problem.

    MSDN doesn't say "You must initialize an SRW lock before using it" because the statement was believed to be so obvious that it never occurred to anybody that somebody would think the opposite was true. I mean, what's the point of having an Initialize­SRW­Lock function if initialization is not required? Think of it as one of the ground rules for programming: If an object has an initialization method, you must initialize the object before using it.

    But just to be sure, I've submitted a documentation change request to add the requirement.

    Bonus chatter: A common coding pattern is to wrap the low-level C-style object inside a C++style RAII-style object.

    Bonus chatter 2: If you're creating a highly-concurrent system, then you should probably put each lock on its own cache line.

  • The Old New Thing

    The shifting sands of "Run as different user"


    A customer liaison asked the following question on behalf of his customer.

    When I do a shift-right-click on a shortcut to a program, I see the following:

    • On Windows Server 2008, shift-right-click does not offer the option Run as different user.
    • On Windows 7, shift-right-click does offer the option Run as different user.

      On Windows Server 2008 R2 (the server counterpart to Windows 7), shift-right-click does offer the option Run as different user.

    The option to run a program as another user (other than Administrator) was present in Windows XP, but it was lost in Windows Vista. It appears that we responded to those complaints by restoring the functionality in Windows 7.

    Is that right?

    The odd thing is that my customer has the Run as different user option available on their Windows 7 machines, but not on their Windows Server 2008 R2 machines. Does whether you have access to Run as different user depend on how you installed Windows Server 2008 R2? (If it matters, my customer installed it via the Microsoft Deployment Toolkit.)

    I found this question interesting for a variety of reasons.

    First of all, it comes dangerously close to being one of those Please tell me I'm not hallucinating type of requests. These support requests take the following peculiar form:

    We did X, then Y, then Z, and then Q happened. Is that right?

    "Um, if you say so. I wasn't there."

    But in this case, it started out looking like it was going to turn into a Please tell me I'm not hallucinating request, then veered into "Is X the reason the feature was added back?"

    This is a rather peculiar question, because knowing the answer one way or the other doesn't actually take you any closer to a solution to the problem. (And I don't know the answer, but fortunately, it wasn't relevant to solving the customer's problem.)

    Another interesting thing about the customer's question is that he never actually comes out and asks the question. He sort of says a few related things, and asks a tangential question, but never comes right out and asks, "How do I get the Run as different user option to work on my Windows Server 2008 R2 machine?"

    It's like a kid who pointedly hangs around a candy bowl, hoping that an adult will say, "Here, have a piece of candy."

    You're a grown-up now. You don't have to linger around the candy bowl hoping somebody will figure out that you want some candy. You should just ask, "May I have a piece of candy?"

    My psychic powers tell me that they have set the Require trusted path for credential entry policy. The Run as different user feature is disabled if you set this policy.

    The customer liaison replied, "It appears that the option for Require trusted path for credential entry is not enabled. The customer is going to do a clean install and test on a non-domain-joined machine to avoid any GPOs."

    Some time passed, and the customer liaison reported back with a resolution.

    The culprit was indeed the Require trusted path for credential entry policy. It didn't show up in a GPO search because they were setting the policy via a script rather than deploying a group policy object.

    It was very nice of the customer liaison to reply with confirmation that the problem was solved. This is, unfortunately, a notable event. Most of the time, people never report back if your suggestion solved their program; they only come back if your suggestion didn't help. Which means you're not sure if your suggestion solved the problem, or if it didn't solve the problem but they decided to continue the investigation somewhere else.

    Bonus chatter: This shows yet another reason why you should use Group Policy Objects to manage group policies rather than custom scripts that whack registry keys. In addition to the fact that registry key whacking may not work, there are tools for processing Group Policy Objects that make this sort of investigation much easier.

  • The Old New Thing

    Sorry we got cut off, my phone just auto-rebooted


    Some time ago, I was on the phone when the connection suddenly cut out. I looked at the my phone display and it said, Auto-restart in 7 seconds. The seconds counted down, and then the phone rebooted, and after about ten more seconds, it was back in business, acting like nothing was wrong. (Except that it just terminated a phone call without warning, but hey, who's keeping track?)

    Sometimes I think phones are getting too smart for their own good.

    Historical note: This entry was written over four years ago, and the model of IP phone in question has long since been replaced.

Page 1 of 3 (29 items) 123