February, 2004

  • The Old New Thing

    Raymond's comment policy


    Okay, I was hoping it wasn't going to be needed but it takes only one bad apple...

    Here are the ground rules.

    • I reserve the right to edit, delete, or ignore any comment.
    • If I edit your comment in any significant way, I promise to make that fact clear in the edit. (Exception: Broken links and typos will be repaired without comment.)

    Things that increase the likelihood that your comment will be edited or deleted:

    • Offensive or abusive language or disrespectful behavior. (Insults count as disrespectful behavior. Examples of insulting words/phrases: "moron", "designed without adult supervision".)
    • Misrepresentation. (I.e., claiming to be somebody you're not.) If you don't want to use your real name, that's fine, as long as your "handle" isn't offensive, abusive, or misrepresentative.
    • Comment spamming. This includes posting multiple comments in rapid succession, even if they are different. If you are prone to spurts of creativity, collect them into batches and post them as a single comment.
    • Comments that are off-topic, particular when the discussion turns into a shouting match.
    • Comments that attempt to "out" a company/program/person. E.g., trying to guess the identity of a program whose name I did not reveal.
    • More generally, comments that identify a specific company, program, or person. You can talk about a program, but you have to call it "Program X" or something like that. The purpose is to discuss problems and solutions, not to assign blame and ridicule.
    • Comments that expose me to potential legal action. Examples: Disclosing any company's trade secrets, posting copyrighted source code, violating Microsoft's company guidelines.
    • Giving yourself a star. (This is another case of mispresentation.)

    If a wave of comment spam is under way, I may choose to moderate or even close comments until the problem subsides.

    More rules may be added later, but I hope it won't be necessary.

    Disclaimer: All postings are provided "AS IS" with no warranties and confer no rights. Opinions expressed are those of the respective authors. More legal stuff here.

    [31 May 2004: Add exceptions for broken link repair.

    2 Dec 2004: Add disclaimer and exception for fixing typos.

    13 Dec 2004: Add remark on temporary closure of comments during spam attacks.

    15 Mar 2005: Add remark for off-topic comments.

    20 Mar 2007: No "outing".

    25 Dec 2008: No disrespectful behavior.

    12 March 2009: Examples of insults.

    8 July 2010: Self-starring.]

  • The Old New Thing

    TEXT vs. _TEXT vs. _T, and UNICODE vs. _UNICODE

    So what's with all these different ways of saying the same thing?

    There's actually a method behind the madness.

    The plain versions without the underscore affect the character set the Windows header files treat as default. So if you define UNICODE, then GetWindowText will map to GetWindowTextW instead of GetWindowTextA, for example. Similarly, the TEXT macro will map to L"..." instead of "...".

    The versions with the underscore affect the character set the C runtime header files treat as default. So if you define _UNICODE, then _tcslen will map to wcslen instead of strlen, for example. Similarly, the _TEXT macro will map to L"..." instead of "...".

    What about _T? Okay, I don't know about that one. Maybe it was just to save somebody some typing.

  • The Old New Thing

    Sure, we do that

    The DirectX video driver interface for Windows 95 had a method that each driver exposed called something like "DoesDriverSupport(REFGUID guidCapability)" where we handed it a capability GUID and it said whether or not that feature was supported.

    There were various capability GUIDs defined, things like GUID_CanStretchAlpha to ask the driver whether it was capable of stretching a bitmap with an alpha channel.

    There was one driver that returned TRUE when you called DoesDriverSupport(GUID_XYZ), but when DirectDraw tried to use that capability, it failed, and in a pretty spectacular manner.

    So one of the DirectDraw developers called the vendor and asked them, "So does your card do XYZ?"

    Their response: "What's XYZ?"

    Turns out that their driver's implementation of DoesDriverSupport was something like this:

    BOOL DoesDriverSupport(REFGUID guidCapability)
      return TRUE;

    In other words, whenever DirectX asked, "Can you do this?" they answered, "Sure, we do that," without even checking what the question was.

    (The driver must have been written by the sales department.)

    So the DirectDraw folks changed the way they queried for driver capabilities. One of the developers went into his boss's office, took a network card, extracted the MAC address, and then smashed the card with a hammer.

    You see, this last step was important: The GUID generation algorithm is based on a combination of time and space. When you ask CoCreateGuid to create a new GUID, it encodes the time of your request in the first part of the GUID and information that uniquely identifies your machine (the network card's MAC address, which is required to be unique by the standards that apply to network card).

    By smashing the network card with a hammer, he prevented that network card from ever being used to generate a GUID.

    Next, he added code to DirectDraw so that when it starts up, it manufactures a random GUID based on that network card (which - by its having been destroyed - can never be validly created) and passes it to DoesDriverSupport. If the driver says, "Sure, we do that", DirectDraw says, "Aha! Caught you! I will not believe anything you say from now on."
  • The Old New Thing

    Pointers to member functions are very strange animals


    Pointers to member functions are very strange animals.

    Warning: The discussion that follows is specific to the way pointers to member functions are implemented by the Microsoft Visual C++ compiler. Other compilers may do things differently.

    Well, okay, if you only use single inheritance, then pointers to member functions are just a pointer to the start of the function, since all the base classes share the same "this" pointer:

    class Simple { int s; void SimpleMethod(); };
    class Simple2 : public Simple
      { int s2; void Simple2Method(); };
    class Simple3 : public Simple2
      { int s3; Simple3Method(); };

    Since they all use the same "this" pointer (p), a pointer to a member function of Base can be used as if it were a pointer to a member function of Derived2 without any adjustment necessary.

    The size of a pointer-to-member-function of a class that uses only single inheritance is just the size of a pointer.

    But if you have multiple base classes, then things get interesting.

    class Base1 { int b1; void Base1Method(); };
    class Base2 { int b2; void Base2Method(); };
    class Derived : public Base1, Base2
      { int d; void DerivedMethod(); };

    There are now two possible "this" pointers. The first (p) is used by both Derived and Base1, but the second (q) is used by Base2.

    A pointer to a member function of Base1 can be used as a pointer to a member function of Derived, since they both use the same "this" pointer. But a pointer to a member function of Base2 cannot be used as-is as a pointer to a member function of Derived, since the "this" pointer needs to be adjusted.

    There are many ways of solving this. Here's how the Visual Studio compiler decides to handle it:

    A pointer to a member function of a multiply-inherited class is really a structure.

    Address of function
    The size of a pointer-to-member-function of a class that uses multiple inheritance is the size of a pointer plus the size of a size_t.

    Compare this to the case of a class that uses only single inheritance.

    The size of a pointer-to-member-function can change depending on the class!

    Aside: Sadly, this means that Rich Hickey's wonderful technique of Callbacks in C++ Using Template Functors cannot be used as-is. You have to fix the place where he writes the comment

    // Note: this code depends on all ptr-to-mem-funcs being same size

    Okay, back to our story.

    To call through a pointer to a member function, the "this" pointer is adjusted by the Adjustor, and then the function provided is called. A call through a function pointer might be compiled like this:

    void (Derived::*pfn)();
    Derived d;
      lea  ecx, d       ; ecx = "this"
      add  ecx, pfn[4]  ; add adjustor
      call pfn[0]       ; call

    When would an adjustor be nonzero? Consider the case above. The function Derived::Base2Method() is really Base2::Base2Method() and therefore expects to receive "q" as its "this" pointer. In order to convert a "p" to a "q", the adjustor must have the value sizeof(Base1), so that when the first line of Base2::Base2Method() executes, it receives the expected "q" as its "this" pointer.

    "But why not just use a thunk instead of manually adding the adjustor?" In other words, why not just use a simple pointer to a thunk that goes like this:

    Derived::Base2Method thunk:
        add ecx, sizeof(Base1)  ; convert "p" to "q"
        jmp Base2::Base2Method  ; continue

    and use that thunk as the function pointer?

    The reason: Function pointer casts.

    Consider the following code:

    void (Base2::*pfnBase2)();
    void (Derived::*pfnDerived)();
    pfnDerived = pfnBase2;
      mov  ecx, pfnBase2            ; ecx = address
      mov  pfnDerived[0], ecx
      mov  pfnDerived[4], sizeof(Base1) ; adjustor!

    We start with a pointer to a member function of Base2, which is a class that uses only single inheritance, so it consists of just a pointer to the code. To assign it to a pointer to a member function of Derived, which uses multiple inheritance, we can re-use the function address, but we now need an adjustor so that the pointer "p" can properly be converted to a "q".

    Notice that the code doesn't know what function pfnBase2 points to, so it can't just replace it with the matching thunk. It would have to generate a thunk at runtime and somehow use its psychic powers to decide when the memory can safely be freed. (This is C++. No garbage collector here.)

    Notice also that when pfnBase2 got cast to a pointer to member function of Derived, its size changed, since it went from a pointer to a function in a class that uses only single inheritance to a pointer to a function in a class that uses multiple inheritance.

    Casting a function pointer can change its size!

    I bet that you didn't know that before reading this entry.

    There's still an awful lot more to this topic, but I'm going to stop here before everybody's head explodes.

    Exercise: Consider the class

    class Base3 { int b3; void Base3Method(); };
    class Derived2 : public Base3, public Derived { };
    How would the following code be compiled?
    void (Derived::*pfnDerived)();
    void (Derived2::*pfnDerived2();
    pfnDerived2 = pfnDerived;

    Answer to appear tomorrow.

  • The Old New Thing

    The arms race between programs and users


    There is a constant struggle between people who write programs and the people who actually use them. For example, you often see questions like, "How do I make my program so the user can't kill it?"

    Now, imagine if there were a way to do this. Ask yourself, "What would the world be like if this were possible?"

    Well, then there would be some program, say, xyz.exe, that is unkillable. Now suppose you're the user. There's this program xyz.exe that has gone haywire, so you want to exit it. But it won't let you exit. So you try to kill it, but you can't kill it either.

    This is just one of several arms races that you can imagine.

    • "I don't want anybody to kill my process." vs. "How do I kill this runaway process?"
    • "I want to shove this critical dialog in the user's face." vs. "How do I stop programs from stealing focus?"
    • "I don't want anybody to delete this file." vs. "How do I delete this file that refuses to be deleted?"
    • "How do I prevent this program from showing up in Task Manager?" vs. "How can I see all the programs that are running on my computer?"

    Eventually you have to decide which side wins, and Windows has decided to keep users in control of their own programs and data, and keep administrators in control of their own computer. So users can kill any process they want (given sufficient privileges), they can stop any program from stealing focus, and they can delete any file they want (again, given sufficient privileges).

    Programs can try to make themselves more difficult to kill (deny PROCESS_TERMINATE access, deny PROCESS_CREATE_THREAD access so people can't CreateRemoteThread(EndProcess), deny PROCESS_VM_WRITE so people can't scribble into your stack and make you doublefault, deny PROCESS_SUSPEND_RESUME so they can't suspend you), but eventually you just can't stop them from, say, elevating to Debug privilege, debugging your process, and moving EIP to "ExitProcess".

    Notice that you can kill CSRSS.EXE and WINLOGON.EXE if you like. Your computer will get very angry at you, but you can do it. (Save you work first!)

    Another useful question to ask yourself: "What's to prevent a virus from doing the same thing?" If there were a way to do these things, then a virus could take advantage of them and make itself invisible to Task Manager, undeletable, and unkillable. Clearly you don't want that, do you?

  • The Old New Thing

    What's so special about the desktop window?

    The window returned by GetDesktopWindow() is very special, and I see people abusing it all over the place.

    For example, many functions in the shell accept a window handle parameter to be used in case UI is needed. IShellFolder::EnumObjects, for example.

    What happens if you pass GetDesktopWindow()?

    If UI does indeed need to be displayed, you hang the system.


    • A modal dialog disables its owner.
    • Every window is a descendant of the desktop.
    • When a window is disabled, all its descendants are also disabled.

    Put this together: If the owner of a modal dialog is the desktop, then the desktop becomes disabled, which disables all of its descendants. In other words, it disables every window in the system. Even the one you're trying to display!

    You also don't want to pass GetDesktopWindow() as your hwndParent. If you create a child window whose parent is GetDesktopWindow(), your window is now glued to the desktop window. If your window then calls something like MessageBox(), well that's a modal dialog, and then the rules above kick in and the desktop gets disabled and the machine is toast.

    So what window do you pass if you don't have a window?

    Pass NULL. To the window manager, a parent of NULL means "Create this window without an owner." To the shell, a UI window of NULL typically means "Do not display UI," which is likely what you wanted anyway.

    Be careful, though: If your thread does have a top-level unowned window, then creating a second such window modally will create much havoc if the user switches to and interacts with the first window. If you have a window, then use it.

  • The Old New Thing

    The layout of a COM object


    The Win32 COM calling convention specifies the layout of the virtual method table (vtable) of an object. If a language/compiler wants to support COM, it must lay out its object in the specified manner so other components can use it.

    It is no coincidence that the Win32 COM object layout matches closely the C++ object layout. Even though COM was originally developed when C was the predominant programming language, the designers saw fit to "play friendly" with the up-and-coming new language C++.

    The layout of a COM object is made explicit in the header files for the various interfaces. For example, here's IPersist from objidl.h, after cleaning up some macros.

    typedef struct IPersistVtbl
        HRESULT ( STDMETHODCALLTYPE *QueryInterface )(
            IPersist * This,
            /* [in] */ REFIID riid,
            /* [iid_is][out] */ void **ppvObject);
            IPersist * This);
            IPersist * This);
            IPersist * This,
            /* [out] */ CLSID *pClassID);
    } IPersistVtbl;
    struct IPersist
        const struct IPersistVtbl *lpVtbl;

    This corresponds to the following memory layout:

    p    lpVtbl    QueryInterface

    What does this mean?

    A COM interface pointer is a pointer to a structure that consists of just a vtable. The vtable is a structure that contains a bunch of function pointers. Each function in the list takes that interface pointer (p) as its first parameter ("this").

    The magic to all this is that since your function gets p as its first parameter, you can "hang" additional stuff onto that vtable:

    p    lpVtbl    QueryInterface
    other stuff

    The functions in the vtable can use offsets relative to the interface pointer to access its other stuff.

    If an object implements multiple interfaces but they are all descendants of each other, then a single vtable can be used for all of them. For example, the object above is already set to be used either as an IUnknown or as an IPersist, since IUnknown is a subset of IPersist.

    On the other hand, if an object implements multiple interfaces that are not descendants of each other, then you get multiple inheritance, in which case the object is typically laid out in memory like this:

    p    lpVtbl    QueryInterface (1)
    q    lpVtbl    QueryInterface (2) AddRef (1)
    other stuff
    AddRef (2) Release (1)
    Release (2) ...

    If you are using an interface that comes from the first vtable, then the interface pointer is p. But if you're using an interface that comes from the second vtable, then the interface pointer is q.

    Hang onto that diagram, because tomorrow we will learn about those mysterious "adjustor thunks".

  • The Old New Thing

    Dunkin Donuts vs. Krispy Kreme

    Having grown up on the east coast, I imprinted on Dunkin Donuts. Once a month we would stop at DD on the way home and buy a shoebox of doughnuts. Toasted coconut and butternut, those were my favorites.

    Ironically, Dunkin Donuts is really a coffee shop disguised as a doughnut shop. (Doughnuts account for only 20% of their sales; coffee 50%.)

    So during my travels through Manhattan, I walked past one of the twenty-five zillion Dunkin Donuts stores there and popped in for a toasted coconut doughnut. One bite and I was a little kid again.

    Some people say that DD's doughnuts are awful, but that's pretty much irrelevant to me by now. It's all about the memories that are invoked.

    And besides, those people are wrong. I don't understand the appeal of KK donuts. They have no flavor; it's just sugar.

  • The Old New Thing

    Mismatching scalar and vector new and delete

    In a previous entry I alluded to the problems that can occur if you mismatch scalar "new" with vector "delete[]" or vice versa.

    There is a nice description of C++ memory management in C++ Gotchas: Avoiding Common Problems in Coding and Design on www.informit.com, and I encourage you to read at least the section titled Failure to Distinguish Scalar and Array Allocation before continuing with this entry, because I'm going to use that information as a starting point.

    Here's how the Microsoft C++ compiler manages vector allocation. Note that this is internal implementation detail, so it's subject to change at any time, but knowing this may give a better insight into why mixing scalar and vector new/delete is a bad thing.

    The important thing to note is that when you do a scalar "delete p", you are telling the compiler, "p points to a single object." The compiler will call the destructor once, namely for the object you are destructing.

    When you do "delete[] p", you are saying, "p points to a bunch of objects, but I'm not telling you how many." In this case, the compiler needs to generate extra code to keep track of how many it needs to destruct. This extra information is kept in a "secret place" when the vector is allocated with "new[]".

    Let's watch this in action:

    class MyClass {
      MyClass(); // constructor
      ~MyClass(); // destructor
      int i;
    MyClass *allocate_stuff(int howmany)
        return new MyClass[howmany];

    The generated code for the "allocate_stuff" function looks like this:

        push    esi
        mov     esi, [esp+8] ; howmany
       ;  eax = howmany * sizeof(MyClass) + sizeof(size_t)
        lea     eax, [esi*4+4]
        push    eax
        call    operator new
        test    eax, eax
        pop     ecx
        je      fail
        push    edi
        push    OFFSET MyClass::MyClass
        push    esi
        lea     edi, [eax+4] ; edi = eax + sizeof(size_t)
        push    4 ; sizeof(MyClass)
        push    edi
        mov     [eax], esi ; howmany
        call    `vector constructor iterator'
        mov     eax, edi
        pop     edi
        jmp     done
        xor     eax, eax
        pop     esi
        retd    4

    Translated back into pseudo-C++, the code looks like this:

    MyClass* allocate_stuff(int howmany)
      void *p = operator new(
         howmany * sizeof(MyClass) + sizeof(size_t));
      if (p) {
        size_t* a = reinterpret_cast<size_t*>(p);
        *a++ = howmany;
        vector constructor iterator(a,
          sizeof(MyClass), &MyClass::MyClass);
        return reinterpret_cast<MyClass*>(a);
      return NULL;

    In other words, the memory layout of the vector of MyClass objects looks like this:


    The pointer returned by the new[] operator is not the start of the allocated memory but rather points to MyClass[0]. The count of elements is hidden in front of the vector.

    The deletion of a vector performs this operation in reverse:

    void free_stuff(MyClass* p)
        delete[] p;
        mov     ecx, [esp+4] ; p
        test    ecx, ecx
        je      skip
        push    3
        call    MyClass::`vector deleting destructor`
        ret     4
    Translated back into pseudo-C++, the code looks like this:
    void free_stuff(MyClass* p)
      if (p) p->vector deleting destructor(3);
    The vector deleting destructor goes like this (pseudo-code):
    void MyClass::vector deleting destructor(int flags)
      if (flags & 2) { // if vector destruct
        size_t* a = reinterpret_cast<size_t*>(this) - 1;
        size_t howmany = *a;
        vector destructor iterator(p, sizeof(MyClass),
          howmany, MyClass::~MyClass);
        if (flags & 1) { // if delete too
          operator delete(a);
      } else { // else scalar destruct
        this->~MyClass(); // destruct one
        if (flags & 1) { // if delete too
          operator delete(this);

    The vector deleting destructor takes some flags. If 2 is set, then a vector is being destructed; otherwise a single object is being destructed. If 1 is set, then the memory is also freed.

    In our case, the flags parameter is 3, so we will perform a vector destruct followed by a delete. Observe that this code sucks the original "howmany" value out of its secret hiding place and asks the vector destructor iterator to run the destructor that many times before freeing the memory.

    So now, armed with this information, you should be able to describe what happens if you allocate memory with scalar "new" and free it with vector "delete[]" or vice versa.

    Bonus exercise: What optimizations can be performed if the destructor MyClass::~MyClass() is removed from the class definition?

    Answers to come tomorrow.

  • The Old New Thing

    The social skills of a thermonuclear device


    Somebody@somewhere.else described me as having the social skills of a thermonuclear device. I don't remember the incident in question, but I'll have to accept that it happened.

    I have a very low tolerance for laziness. If you come to me for help, I expect you to have done your homework. (Though I try to scale my expectations to your position. If you're in a coding position then you had better know how to use a debugger. If you're in a testing position, then you had better know at least how to install the debugger. This is considered part of the standard skill set for a tester.)

    But the absolute worst thing you can do is start a conversation by saying, "Hey, Raymond, since you know everything, can you do this for me...?"

Page 1 of 4 (32 items) 1234