The Old New Thing

The difference between assignment and attachment with ATL smart pointers

oldnewthing — Fri, 20 Nov 2009 15:00:00 GMT

Last time, I presented a puzzle regarding a memory leak. Here's the relevant code fragment:

CComPtr<IStream> pMemoryStream;
CComPtr<IXmlReader> pReader;
UINT nDepth = 0;

//Open read-only input stream
pMemoryStream = ::SHCreateMemStream(utf8Xml, cbUtf8Xml);

The problem here is assigning the return value of SHCreateMemStream to a smart pointer instead of attaching it.

The SHCreateMemStream function creates a memory stream and returns a pointer to it. That pointer has a reference count of one, in accordance with COM rules that a function which produces a reference calls AddRef, and the responsibility is placed upon the recipient to call Release. The assignment operator for CComPtr<T> is a copy operation: It AddRefs the pointer and saves it. You're still on the hook for the reference count of the original pointer.

ATLINLINE ATLAPI_(IUnknown*) AtlComPtrAssign(IUnknown** pp, IUnknown* lp)
{
        if (lp != NULL)
                lp->AddRef();
        if (*pp)
                (*pp)->Release();
        *pp = lp;
        return lp;
}

template <class T>
class CComPtr
{
public:
        ...

        T* operator=(T* lp)
        {
                return (T*)AtlComPtrAssign((IUnknown**)&p, lp);
        }

Observe that assigning a T* to a CComPtr<T> AddRefs the incoming pointer and Releases the old pointer (if any). When the CComPtr<T> is destructed, it will release the pointer, undoing the AddRef that was performed by the assignment operator. In other words, assignment followed by destruction has a net effect of zero on the pointer you assigned. The operation behaves like a copy.

Another way of putting a pointer into a CComPtr<T> is with the Attach operator. This is a transfer operation:

        void Attach(T* p2)
        {
                if (p)
                        p->Release();
                p = p2;
        }

Observe that there is no AddRef here. When the CComPtr<T> is destructed, it will perform the Release, which doesn't undo any operation performed by the Attach. Instead, it releases the reference count held by the original pointer you attached.

Let's put this in a table, since people seem to like tables:

Operation	Behavior	Semantics
Attach()	Takes ownership	Transfer semantics
operator=()	Creates a new reference	Copy semantics

You use the Attach method when you want to assume responsibility for releasing the pointer (ownership transfer). You use the assignment operator when you want the original pointer to continue to be responsible for its own release (no ownership transfer).

There is also a Detach method which is the opposite of Attach: Detaching a pointer from the CComPtr<T> means "I am taking over responsibility for releasing this pointer." The CComPtr<T> gives you its pointer and then forgets about it; you're now on your own.

The memory leak in the code fragment above occurs because the assignment operator has copy semantics, but we wanted transfer semantics, since we want the smart pointer to take the responsibility for releasing the pointer when it is destructed.

pMemoryStream.Attach(::SHCreateMemStream(utf8Xml, cbUtf8Xml));

The CComPtr<T>::operator=(T*) method is definitely one of the more dangerous methods in the CComPtr<T> repertoire, because it's so easy to assign a pointer to a smart pointer without giving it a moment's thought. (Another dangerous method is the T** CComPtr<T>::operator&(), but at least that has an assertion to try to catch the bad usages. Even nastier is the secret QI'ing assignment operator.) I have to say that there is merit to Ben Hutchings' recommendation simply not to allow a simple pointer to be assigned to a smart pointer, precisely because the semantics are easily misunderstood. (The boost library, for example, follows Ben's recommendation.)

Here's another exercise based on what you've learned:

Application Verifier told us that we have a memory leak, and we traced it back to the function GetTextAsInteger.
BSTR GetInnerText(IXMLDOMNode *node)
{
    BSTR bstrText = NULL;
    node->get_text(&bstrText);
    return bstrText;
}

DWORD GetTextAsInteger(IXMLDOMNode *node)
{
    DWORD value = 0;

    CComVariant innerText = GetInnerText(node);
    hr = VariantChangeType(&innerText, &innerText, 0, VT_UI4);
    if (SUCCEEDED(hr))
    {
        value = V_UI4(&innerText);
    }

    return value;
}
Obviously, the problem is that we passed the same input and output pointers to VariantChangeType, causing the output integer to overwrite the input BSTR, resulting in the leak of the BSTR. But when we fixed the function, we still got the leak:
DWORD GetTextAsInteger(IXMLDOMNode *node)
{
    DWORD value = 0;

    CComVariant innerText = GetInnerText(node);
    CComVariant textAsValue;
    hr = VariantChangeType(&innerText, &textAsValue, 0, VT_UI4);
    if (SUCCEEDED(hr))
    {
        value = V_UI4(&textAsValue);
    }

    return value;
}
Is there a leak in the VariantChangeType function itself?

Hint: It is in fact explicitly documented that the output parameter to VariantChangeType can be equal to the input parameter, which results in an in-place conversion. There was nothing wrong with the original call to VariantChangeType.

We're using a smart pointer, so we can't possibly be the source of the leak

oldnewthing — Thu, 19 Nov 2009 15:00:00 GMT

A customer reported that there was a leak in the shell, and they included the output from Application Verifier as proof. And yup, the memory that was leaked was in fact allocated by the shell:

VERIFIER STOP 00000900 : pid 0x3A4: A heap allocation was leaked.

        497D0FC0 : Address of the leaked allocation.
        002DB580 : Adress to the allocation stack trace.
        0D65CFE8 : Address of the owner dll name.
        6F560000 : Base of the owner dll.

1: kd> du 0D65CFE8
0d65cfe8  "SHLWAPI.dll"

1: kd> !heap -p -a 497D0FC0
...
    ntdll!RtlpAllocateHeap+0x0003f236
    ntdll!RtlAllocateHeap+0x0000014f
    Kernel32!LocalAlloc+0x0000007c
    shlwapi!CreateMemStreamEx+0x00000043
    shlwapi!CreateMemStream+0x00000012
    <Unloaded_xyz.dll>+0x000642de
    <Unloaded_xyz.dll>+0x0005e2af
    <Unloaded_xyz.dll>+0x0002d49a
    <Unloaded_xyz.dll>+0x0002a0fd
    <Unloaded_xyz.dll>+0x000289cb
    <Unloaded_xyz.dll>+0x0002a25c
    <Unloaded_xyz.dll>+0x00027225
    <Unloaded_xyz.dll>+0x0002252b
    <Unloaded_xyz.dll>+0x00025394
    <Unloaded_xyz.dll>+0x0004d70f
    Kernel32!BaseThreadInitThunk+0x0000000d
    ntdll!RtlUserThreadStart+0x0000001d

1: kd> dps 002DB580
shlwapi!CreateMemStreamEx+0x43
shlwapi!CreateMemStream+0x12
<Unloaded_xyz.dll>+0x642de
<Unloaded_xyz.dll>+0x5e2af
<Unloaded_xyz.dll>+0x2d49a
<Unloaded_xyz.dll>+0x2a0fd
<Unloaded_xyz.dll>+0x289cb
<Unloaded_xyz.dll>+0x2a25c
<Unloaded_xyz.dll>+0x27225
<Unloaded_xyz.dll>+0x2252b
<Unloaded_xyz.dll>+0x25394
<Unloaded_xyz.dll>+0x4d70f
Kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d

On the other hand, SHCreateMemStream is an object creation function, so it's natural that the function allocate some memory. The responsibility for freeing the memory belongs to the caller.

We suggested that the customer appears to have leaked the interface pointer. Perhaps there's a hole where they called AddRef and managed to avoid the matching Release.

"Oh no," the customer replied, "that's not possible. We call this function in only one place, and we use a smart pointer, so a leak is impossible." The customer was kind enough to include a code snippet and even highlighted the lines that proved they weren't leaking.

CComPtr<IStream> pMemoryStream;
CComPtr<IXmlReader> pReader;
UINT nDepth = 0;

//Open read-only input stream
pMemoryStream = ::SHCreateMemStream(utf8Xml, cbUtf8Xml);

The exercise for today is to identify the irony in the highlighted lines.

Hint. Answers (and more discussion) tomorrow.

News flash: Healthy people live longer

oldnewthing — Wed, 18 Nov 2009 15:00:01 GMT

Researchers have determined that people in good physical condition live longer. Who'd'a thunk it?

How do I move the Windows.edb and other search index files?

oldnewthing — Wed, 18 Nov 2009 15:00:00 GMT

Nothing profound today, just a little tip.

My customer is looking out for a way to change the location of the windows.edb file to another (larger) drive.

From the Indexing Options Control Panel, click Advanced, and then under Index location, click Select new.

We found the author of Notepad, sorry you didn't go to the award ceremony

oldnewthing — Tue, 17 Nov 2009 15:00:01 GMT

I've received independent confirmations as to the authorship of Notepad, so I'm inclined to believe it. Sorry you didn't get to go to the award ceremony.

The original author of Notepad also served as the development manager for Windows 95. His job was to herd the cats that made up the programmers who worked on Windows 95, a job which you can imagine falls into the "not easy" category.

After Windows 95, he retired from the software industry and became a high school science teacher. At a social event some years later, I met him again and asked about the transition from software development manager to high school science teacher.

His response: "You'd be surprised how many of the skills transfer."

How to tell when your patent has been approved

oldnewthing — Tue, 17 Nov 2009 15:00:00 GMT

There are a variety of ways of finding out when your patent is granted, but the quickest mechanism is to check your mailbox. But the thing to look for is not what you might think.

Even before you receive word from your company's patent department, you will start receiving junk mail delivered to your home address from companies that sell patent-related novelties, pointless trinkets like pencils and mugs with your patent number on it. Nevermind that you can probably order personalized pencils for much less than what the patent-chasers were offering.

How to pretend that you attended my talk at UIUC Reflections|Projections 2009

oldnewthing — Mon, 16 Nov 2009 15:00:01 GMT

Step 1: Buy a 1.55-ounce Hershey's Milk Chocolate Bar from a convenience store, supermarket, or (if truly desperate) online.

Step 2: Print out this candy bar wrapper.

Step 3: Trim wrapper on registration marks and wrap around candy bar.

Step 4: Stay up late the night before you plan on watching the video by partying with Ryan North and teaching him how to play beer pong.

Step 5: Force yourself to wake up the next morning and watch the recorded video of my talk while trying desperately to stay awake. The candy bar might help.

Note: Although most steps are optional, they are essential if you want an accurate simulation.

Why does shlwapi import a nonexistent function?

oldnewthing — Mon, 16 Nov 2009 15:00:00 GMT

Commenter charless asks why shlwapi.dll imports a nonexistent function from mpr.dll, which shows up in dependency tools as a broken import.

Because that function did exist at one point, although it doesn't exist any more.

The function in question was available only on Windows 95-series versions of Windows. It never existed on Windows NT or any of its successors. But remember that shlwapi.dll was originally developed for Internet Explorer, which ran on Windows 95 as well as Windows NT. Internet Explorer checked the operating system and called the Windows 95-only function only after verifying that it was running on Windows 95. If it was running on Windows NT, then it never called the function and therefore never stepped on the land mine known as ERROR_PROC_NOT_FOUND.

Okay, so why does shlwapi still link to the function long after the Windows 95 series of operating systems have become obsolete?

Removing a function, even a function that doesn't do anything, even an undocumented function that doesn't do anything, is a dangerous endeavor. Suppose you have a program that links to the function, but just like Internet Explorer, it is clever and checks whether it is running on Windows NT before calling it. If you remove the useless function from shlwapi, then that program will fail to load, even though it never calls the offending function, and now you have an application compatibility problem on your hands.

Since it's a small function that doesn't do anything, it's a lot less risky simply to leave the function in.

Even though it doesn't do anything except fail.

What a drag: You can be a drag in managed code, too

oldnewthing — Fri, 13 Nov 2009 15:00:01 GMT

David Anson digests my earlier series on virtual drag/drop and translates it into managed code. His example of dragging his entire RSS feed is an excellent illustration of dragging dynamically-generated virtual content. (I didn't use an example like that because the purpose of the What a drag series was to get something done in the least amount of code, and generating a stream from a URL takes an awful lot of code when doing it from the unmanaged side, which would ultimately detract from the point of the example.)

Bonus: He takes the example further by adding asynchronous support.

You thought reasoning about signals was bad, reasoning about a total breakdown of normal functioning is even worse

oldnewthing — Fri, 13 Nov 2009 15:00:00 GMT

A customer came to the Windows team with a question, the sort of question which on its face seems somewhat strange, which is itself a sign that the question is merely the tip of a much more dangerous iceberg.

Under what circumstances will the GetEnvironmentVariable function hang?

This is kind of an open-ended question. I mean, for example, somebody might sneak in and call SuspendThread on your thread while GetEnvironmentVariable is running, which will look like a hang because the call never completes because the thread is frozen.

But the real question for the customer is, "What sort of problem are you seeing that is manifesting itself in an apparent hang in the GetEnvironmentVariable function?"

The customer was kind enough to elaborate.

We have a global unhandled exception filter in our application so we can log all failures. After we finish logging, we call ExitProcess, but we find that the application never actually exits. If we connect a debugger to the stuck application, we see it hung in GetEnvironmentVariable.

Your gut response should be, "Holy cow, I'm surprised you even got that far!"

This isn't one of those global unhandled exception filters that got installed because your program plays some really clever game with exceptions, No, this is an "Oh no, my program just crashed and I want to log it" exception handler. In other words, when this exception handler "handles" an exception, it's because your program has encountered some sort of serious internal programming error for which the program did not know how to recover. We saw earlier that you can't do much in a signal handler because you might have interrupted a block of code which was in the middle of updating some data structures, leaving them momentarily inconsistent. But this exception filter is in an even worse state: Not only is there a good chance that the program is in the middle of updating something and left it in an inconsistent state, you are in fact guaranteed that the system is in a corrupted state.

Why is this a guarantee? Because if the system were in a consistent state, you wouldn't have crashed!

Programming is about establishing invariants, perturbing them, and then re-establishing them. It is a game of stepping-stone from one island of consistency to another. But the code that does the perturbing and the re-establishing assumes that it's starting from a consistent state to begin with. For example, a function that removes a node from a doubly-linked list manipulates some backward and forward link pointers (temporarily violating the linked list invariant), and then when it's finished, the linked list is back to a consistent state. But this code assumes that the linked list is not corrupted to begin with!

Let's look again at that call to ExitProcess. That's going to detach all the DLLs, calling each DLL's DllMain with the DLL_PROCESS_DETACH notification. But of course, those DllMain are going to assume that the data structures are intact and nothing is corrupted. On the other hand, you know for a fact that these prerequisites are not met—the program crashed precisely because something is corrupted. One DLL might walk a linked list—but you might have crashed because that linked list is corrupted. Another DLL might try to delete a critical section—but you might have crashed because the data structure containing the critical section is corrupted.

Heck, the crash might have been inside somebody's DLL_PROCESS_DETACH handler to begin with, for all you know.

"Yeah, but the documentation for TerminateProcess says that it does not clean up shared memory."

Well, it depends on what you mean by clean up. The reference count on the shared memory is properly decremented when the handle is automatically closed as part of process cleanup, and the shared memory will be properly freed once there are no more references to it. It is not cleaned up in the sense of "corruption is repaired"—but of course the operating system can't do that because it doesn't know what the semantics of your shared memory block are.

But this is hardly anything to get concerned about because your program doesn't know how to un-corrupt the data either.

"It also says that DLLs don't receive their DLL_PROCESS_DETACH notification."

As we saw before, this is a good thing in the case of a corrupted process, because the code that runs in DLL_PROCESS_DETACH assumes that your process has not been corrupted in the first place. There's no point running it when you know the process is corrupted. You're just making a bad situation worse.

"It also says that I/O will be in an indeterminate state."

Well yeah, but that's no worse than what you have now, which is that your I/O is in an indeterminate state. You don't know what buffers your process hasn't flushed, but since your process is corrupted, you have no way of finding out anyway.

"Are you seriously recommending that I use TerminateProcess to exit the last chance exception handler?!?"

Your process is unrecoverably corrupted. (This is a fact, because if there were a way to recover from it, you would have done it instead of crashing.) What other options are there?

Quit while you're behind.

Why can you create a PIF file that points to something that isn't an MS-DOS program?

oldnewthing — Thu, 12 Nov 2009 15:00:00 GMT

James MAstros asked why it's possible to create a PIF file that refers to a program that isn't an MS-DOS program. (That's only part of the question; I addressed other parts last year.)

Well, for one thing, there was indeed code to prevent you from setting PIF properties for something that isn't an MS-DOS program, so the precaution was already there. But it didn't stop anybody who was really determined to try. All you had to do was create an MS-DOS program, then create a PIF file for it, and then overwrite the MS-DOS program file with something else. Since time travel has not been invented, the PIF creator code can't retroactively go back in and say, "Well, if I knew you were going to pull this trick, I wouldn't have let you create it in the first place!" You can't even enforce it at the time the PIF file is launched, because somebody could replace the file during the split second between checking the file type and actually using it.

Of course, the real question is "Why, if you create a PIF file that describes something that isn't an MS-DOS program, does it still work?" It still works because the PIF file did exactly what it was supposed to do. It created an MS-DOS virtual machine with the specified parameters and then ran the program in it.

Now, back in the days when PIF files were invented, if you tried to run something that wasn't an MS-DOS program inside an MS-DOS virtual machine, all that happened was that the MS-DOS stub ran, the same thing that happened if you tried to run the program from MS-DOS. The program was treated not as a Windows program but as an MS-DOS program.

Windows 95 changed that. If you tried to run a Windows application from the MS-DOS command prompt, it would run the Windows application instead of telling you, "This program cannot be run in DOS mode." This change was made for two reasons.

First, the existing behavior seemed pretty stupid. You're running Windows, and if you open a command prompt and try to run a Windows program, you're told, "You need Windows to run this program." It's like one of those bizarro-land government red tape nightmares, where you go to the courthouse to file some papers in person, and the clerk at the desk says, "I'm sorry, you have to file this document in person at the courthouse."

Second, it was necessary to allow 32-bit console programs to run when launched from a MS-DOS command window. Since Windows 95 used the MS-DOS prompt as its command line interface (as opposed to Windows NT which used a 32-bit command prompt), it was kind of important that you be able to run 32-bit console programs from a virtual machine. Without it, the whole idea of a console program became kind of weak. "Yeah, we have console programs, but you can't launch them from a console."

What happens, then, if you create PIF file that points to a 32-bit program? Well, the operating system goes to all the effort to create a virtual machine to the specifications you indicated in the PIF file. You want a particular amount of extended memory? Okay, we'll set that up. You want a custom icon? Sure, no problem. You want it to disable DPMI memory? You got it. Once that's all set up, the virtual MS-DOS driver says, "Okay, and set the initial CS:IP for the virtual machine to the MS-DOS EXEC call to run the program the PIF file specified." The EXEC call executes, and the interop code kicks in and launches the 32-bit Windows program.

From the virtual machine's point of view, nothing is actually wrong; you merely did something in a really roundabout way. It's like booking a meeting room, specifying that you would like a slide projector, that the chairs and tables be set in a particular arrangement, that everybody be provided with water and a notebook, and then putting a sign on the door that says, "This meeting has moved to Location Z." You go to all this effort to get the conference room to be set up exactly the way you want it, and then you end up not using it. The conference center doesn't care (as long as it still gets paid).

PIF files are like shortcuts, but with the added effort of creating an MS-DOS virtual machine. And just like with shortcuts, bad guys can choose a dangerous target and make your day miserable.

Leave it to the Taiwanese to think of wrapping a donut inside another donut

oldnewthing — Wed, 11 Nov 2009 15:00:01 GMT

The food known in Mandarin Chinese as 油條 (yóutiáo), but which in Taiwanese goes by the name 油炸粿, is basically a fried stick of dough, similar to a cruller, but puffier rather than cakey. The traditional way of eating it is to wrap it inside a 燒餅 (a sesame-coated flatbread), and dip the entire combination into a bowl of hot soy milk. I prefer salty soy milk, but some people prefer sweet. (Those people who prefer the sweet version are clearly wrong.)

Obviously, the donut sandwich was invented before the low-carb diet craze.

Sidebar: Salty soy milk (鹹豆漿) is one of those nostalgia breakfasts for me, or more accurately, one of those manufactured nostalgia breakfasts, because I didn't actually eat it that much as a child.

Sidebar 2: For authentic Chinese food in Seattle, my choice is Chiang's Gourmet. They have an extensive menu of standard Chinese breakfast foods. The service is surly, but that somehow just adds to the experience.

Trying to avoid double-destruction and inadvertently triggering it

oldnewthing — Wed, 11 Nov 2009 15:00:00 GMT

We saw some time ago the importance of artificially bumping an object's reference count during destruction to avoid double-destruction. However, one person's attempt to avoid this problem ended up triggering it.

ULONG MyObject::Release()
{
 LONG cRef = InterlockedDecrement(&m_cRef);
 if (cRef > 0) return cRef;
 m_cRef = MAXLONG; // avoid double-destruction
 delete this;
 return 0;
}

The explanation for the line m_cRef = MAXLONG was that it was done to avoid the double-destruction problem if the object receives a temporary AddRef/Release during destruction.

While it's true that you should set the reference count to an artificial non-zero value, choosing MAXLONG has its own problem: integer overflow.

Suppose that during the object's destruction, the reference count is temporarily incremented twice and decremented twice.

Action	m_cRef
Just before call to Release()	1
InterlockedDecrement	0
m_cRef = MAXLONG	2147483647
destructor does temporary `AddRef()`	−2147483648 (integer overflow)
destructor does temporary `AddRef()`	−2147483647
destructor does temporary `Release()`	−2147483648
since `m_cRef` < 0, we re-destruct

Sure, choosing a huge DESTRUCTOR_REFCOUNT means that you have absolutely no chance of decrementing the reference count back to zero prematurely. However, if you choose a value too high, you introduce the risk of incrementing the reference count so high that it overflows.

That's why the most typical values for DESTRUCTOR_REFCOUNT are 1, 42, and 1000. The value 1 is really all you need to avoid double-destruction. Some people choose 42 because it's cute, and other people choose 1000 because it's higher than any "normal" refcount, so it makes it easier to spot during debugging. But even then, the "high" value of 1000 still leaves room for over two billion AddRef()s before overflowing the reference count.

On the other hand, if you choose a value like MAXLONG or MAXDWORD, then you're taking something that previously never happened (reference count integer overflow) and turning it into an almost certainty.

I reorganized your kitchen for you, sweetie

oldnewthing — Tue, 10 Nov 2009 15:00:01 GMT

I suspect most people are familiar with the It may be a mess, but it's my mess and I know where everything is phenomenon. That doesn't necessarily mean that items are in the best location, but at least you know which suboptimal location you chose.

:: Wendy :: told me a story some time ago about something that happened while her parents were visiting. When she returned from work, her mother said, "Oh, Wendy, darling, I reorganized your kitchen for you. You had everything in the wrong place."

Wendy's mother was trying to be helpful, but of course it was a net loss for poor Wendy, who couldn't find anything in her kitchen for weeks. Yes, there was the whole Oh great where did my mother put my food processor? problem, but even after she found it, the "improved" location was far worse than its original location. In fact, in many cases, it was in the exact opposite location from where it should be.

You see, Wendy is left-handed, and her mother is right-handed.

Little-known command line utility: clip

oldnewthing — Tue, 10 Nov 2009 15:00:00 GMT

Windows Vista includes a tiny command line utility called clip. All it does is paste its stdin onto the clipboard.

dir | clip
echo hey | clip

For the opposite direction, I use a little perl script:

use Win32::Clipboard;
print Win32::Clipboard::GetText();