• The Old New Thing

    The unattend file lets you configure Windows while it's installing, and in some cases it's your only chance


    Some Windows settings can only be established as part of the installation process. This is done with a so-called unattend file. (Remember, no matter where you put an advanced setting, somebody will tell you that you are an idiot.) In earlier versions of Windows, the unattend file took the form of an INI file, but Windows Vista hopped aboard the XML bandwagon, and the unattend file format changed to XML. The nice thing about using XML is that you can publish a schema so people can validate their unattend file without having to perform a test install (only to discover twenty minutes later that a typo resulted in an entire section of the unattend file being ignored, say).

    If you spend a lot of time setting up computers, you can use an unattend file to answer all the Setup questions (like "enter your product key") so all you have to do is type "setup /unattend:myconfiguration.xml" and go out to lunch. When you come back, your machine will be installed and ready.

    Here are two of the most popular unattend settings which must be set during installation. (There are a bunch of popular unattend settings for things that can also be changed post-install; for those other settings, the unattend file is not your only chance.)

    Wait, the C:\Program Files directory isn't on the list of directories that can be relocated. There's a reason for that, which we'll look at next time.

  • The Old New Thing

    Beyoncé, the giant metal chicken has a Facebook page


    In my 2011 mid-year link clearance, I linked to the story And that's why you should learn to pick your battles. I thought that was the end of the story, but no, it's the gift that keeps on giving. Beyoncé, the giant metal chicken has a Facebook page, populated with all sorts of crazy things like pictures of Beyoncé's relatives spotted in the wild (some of them knocking on doors or peeking in windows), a No Soliciting sign just for giant metal chickens, and an updated version of the chart of anniversary gifts which lists BIG METAL CHICKEN as the modern 15th anniversary present.

  • The Old New Thing

    Adjusting your commute to avoid being at work quite so much


    Commenter Bernard remarked that he moved from one side of the company campus to the other, and his commute distance (and time) was cut in half.

    That reminds me of a short story from a now-retired colleague of mine. He bicycled to work, and over the years, the gradual expansion of the Microsoft main corporate campus resulted in nearly three quarters of his commute to work taking place on Microsoft property. This bothered him to the point where he changed his route just so he wouldn't be "at work" practically the moment he left his house.

    Bonus chatter: Looks like Michael Kaplan also has one of these unbalanced commutes.

  • The Old New Thing

    Do not access the disk in your IContextMenu handler, no really, don't do it


    We saw some time ago that the number one cause of crashes in Explorer is malware.

    It so happens that the number one cause of hangs in Explorer is disk access from context menu handlers (a special case of the more general principle, you can't open the file until the user tells you to open it).

    That's why I was amused by Memet's claim that "would hit the disk" is not acceptable for me. The feedback I see from customers, either directly from large multinational corporations with 500ms ping times or indirectly from individual users who collectively click Send Report millions of times a day, is that "would hit the disk" ruins a lot of people's days. It may not be acceptable to you, but millions of other people would beg to disagree.

    The Windows team tries very hard to identify unwanted disk accesses in Explorer and get rid of them. We don't get them all, but at least we try. But if the unwanted disk access is coming from a third-party add-on, there isn't much that can be done aside from saying, "Don't do that" and hoping the vendor listens.

    Every so often, a vendor will come back and ask for advice on avoiding disk access in their context menu handler. There's a lot of information packed into that data object that contains information gathered from when the disk was accessed originally. You can just retrieve that cached data instead of going off and hitting the disk again to recalculate it.

    I'm going to use a boring console application and the clipboard rather than building a full IContext­Menu, since the purpose here is to show how to get data from a data object without hitting the disk and not to delve into the details of IContext­Menu implementation.

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <ole2.h>
    #include <shlobj.h>
    #include <propkey.h>
    #include <tchar.h>
    void ProcessDataObject(IDataObject *pdto)
     ... to be written ...
    int __cdecl _tmain(int argc, PTSTR *argv)
     if (SUCCEEDED(OleInitialize(NULL))) {
      IDataObject *pdto;
      if (SUCCEEDED(OleGetClipboard(&pdto))) {

    Okay, let's say that we want to check that all the items on the clipboard are files and not directories. The HDROP way of doing this would be to get the path to each of the items in the data object, then call Get­File­Attributes on each one to see if any of them has the FILE_ATTRIBUTE_DIRECTORY flag set. But this hits the disk, which makes baby context menu host sad. Fortunately, the IShell­Item­Array interface provides an easy way to check whether any or all the items in a data object have a particular attribute.

    void ProcessDataObject(IDataObject *pdto)
     IShellItemArray *psia;
     HRESULT hr;
     hr = SHCreateShellItemArrayFromDataObject(pdto,
     if (SUCCEEDED(hr)) {
      SFGAOF sfgaoResult;
      hr = psia->GetAttributes(SIATTRIBFLAGS_OR, SFGAO_FOLDER,
      if (hr == S_OK) {
       _tprintf(TEXT("Contains a folder\n"));
      } else if (hr == S_FALSE) {
       _tprintf(TEXT("Contains no folders\n"));

    In this case, we want to see if any item (SI­ATTRIB­FLAGS_OR) in the data object has the SFGAO_FOLDER attribute. The IShell­Item­Array::Get­Attributes method returns S_OK if all of the attributes you requested are present in the result. Since we asked for only one attribute, and since we asked for the result to be the logical or of the individual attributes, this means that it returns S_OK if any item is a folder.

    Okay, fine, but what if the thing you want to know is not expressible as a SFGAO flag? Well, you can dig into each of the individual items. For example, suppose we want to see the size of each item.

    #include <strsafe.h>
    void ProcessDroppedObject(IDataObject *pdto)
     IShellItemArray *psia;
     HRESULT hr;
     hr = SHCreateShellItemArrayFromDataObject(pdto,
     if (SUCCEEDED(hr)) {
      IEnumShellItems *pesi;
      hr = psia->EnumItems(&pesi);
      if (SUCCEEDED(hr)) {
       IShellItem *psi;
       while (pesi->Next(1, &psi, NULL) == S_OK) {
        IShellItem2 *psi2;
        hr = psi->QueryInterface(IID_PPV_ARGS(&psi2));
        if (SUCCEEDED(hr)) {
         ULONGLONG ullSize;
         hr = psi2->GetUInt64(PKEY_Size, &ullSize);
         if (SUCCEEDED(hr)) {
          _tprintf(TEXT("Item size is %I64u\n"), ullSize);

    I went for IEnum­Shell­Items here, even though a for loop with IShell­Item­Array::Get­Count and IShell­Item­Array::Get­Item­At would have worked, too.

    File system items in data objects cache a bunch of useful pieces of information, such as the last-modified time, file creation time, last-access time, the file size, the file attributes, and the file name (both long and short). Of course, all of these properties are subject to file system support. the shell just takes what's in the WIN32_FIND_DATA; if the values are incorrect (for example, if last-access time tracking is disabled), then the shell is going to cache the incorrect value. But don't say, "Well, if the cache is no good, then I won't use it; I'll just go hit the disk", because if you hit the disk, the file system is going to give you the same incorrect value anyway!

    If you just want to order the combo platter, you can ask for PKEY_Find­Data, and out will come a WIN32_FIND_DATA. This might be the easiest way to convert your old-style context menu that hits the disk into a new-style context menu that doesn't hit the disk: Take your calls to Get­File­Attributes and Find­First­File and convert them into calls into the property system, asking for PKEY_File­Attributes or PKEY_Find­Data.

    Okay, that's the convenient modern way to get information that has been cached in the data object provided by the shell. What if you're an old-school programmer? Then you get to roll up your sleeves and get your hands dirty with the CFSTR_SHELL­ID­LIST clipboard format. (And if your target is Windows XP or earlier, you have to do it this way since the IShell­Item­Array interface was not introduced until Windows Vista.) In fact, the CFSTR_SHELL­ID­LIST clipboard format will get your hands so dirty, I'm writing a helper class to manage it.

    First, go back and familiarize yourself with the CIDA structure.

    // these should look familiar
    #define HIDA_GetPIDLFolder(pida) (LPCITEMIDLIST)(((LPBYTE)pida)+(pida)->aoffset[0])
    #define HIDA_GetPIDLItem(pida, i) (LPCITEMIDLIST)(((LPBYTE)pida)+(pida)->aoffset[i+1])
    void ProcessDataObject(IDataObject *pdto)
     FORMATETC fmte = {
     STGMEDIUM stm = { 0 }; // defend against buggy data object
     HRESULT hr = pdto->GetData(&fmte, &stm);
     if (SUCCEEDED(hr) && stm.hGlobal != NULL) {
      LPIDA pida = (LPIDA)GlobalLock(stm.hGlobal);
      if (pida != NULL) { // defend against buggy data object
       IShellFolder *psfRoot;
       hr = SHBindToObject(NULL, HIDA_GetPIDLFolder(pida), NULL,
       if (SUCCEEDED(hr)) {
        for (UINT i = 0; i < pida->cidl; i++) {
         IShellFolder2 *psf2;
         PCUITEMID_CHILD pidl;
         hr = SHBindToFolderIDListParent(psfRoot,
                    HIDA_GetPIDLItem(pida, i),
                    IID_PPV_ARGS(&psf2), &pidl);
         if (SUCCEEDED(hr)) {
          VARIANT vt;
          if (SUCCEEDED(psf2->GetDetailsEx(pidl, &PKEY_Size, &vt))) {
           if (SUCCEEDED(VariantChangeType(&vt, &vt, 0, VT_UI8))) {
             _tprintf(TEXT("Item size is %I64u\n"), vt.ullVal);

    I warned you it was going to be ugly.

    First, we retrieve the CFSTR_SHELL­ID­LIST clipboard format from the data object. This format takes the form of an HGLOBAL, which needs to be Global­Lock'd like all HGLOBALs returned by IData­Object::Get­Data. You may notice two defensive measures here. First, there is a defense against data objects which return success when they actually failed. To detect this case, we zero out the STG­MEDIUM and make sure they returned something non-NULL in it. The second defensive measure is against data objects which put an invalid HGLOBAL in the STG­MEDIUM. One of the nice things about doing things the IShell­Item­Array way is that the shell default implementation of IShell­Item­Array has all these defensive measures built-in so you don't have to write them yourself.

    Anyway, once we get the CIDA, we bind to the folder portion, walk through the items, and get the size of each item in order to print it. Same story, different words.

    Exercise: Why did we need a separate defensive measure for data objects which returned success but left garbage in the STG­MEDIUM? Why doesn't the Global­Lock test cover that case, too?

  • The Old New Thing

    2011 Q3 link clearance: Microsoft blogger edition


    It's that time again: Linking to other Microsoft bloggers.

  • The Old New Thing

    There's also a large object heap for unmanaged code, but it's inside the regular heap


    Occasionally, a customer will ask for assistance explaining some strange heap behavior, or at least heap behavior that appears to be strange if you assume that the heap behaves purely classically.

    I need to understand the heap behavior we're seeing. I have a sample program which allocates five blocks of memory from the process heap, each of size 100 bytes. When we dump the heap blocks with the !heap -x command, we find that all of them belong to the same heap segment, and when we do a !vadump -v, we find that they all live on the same page. On the other hand, if we allocate five blocks of size 512KB, then we find that each one gets placed in its own virtually-allocated range, and the range is not size 512KB but rather 516KB.


    1. Why does the system combine the 100-byte allocations? Why is there no extra overhead for the small allocations, but the large allocations have a 4KB overhead?
    2. Why does the system allocate a separate virtual address range for the 512KB block? Which component makes this decision and what are the factors involved?
    3. Why does allocating a 512KB block require 4KB of overhead?
    4. Is there any documentation for this behavior?

    Let's look at the last question first. There is no documentation for this behavior because it is all implementation detail. All the heap is required to do is return you a block of memory of the size you requested. How it goes about getting this memory block is at the discretion of the heap manager, and as we've seen in the past, the precise algorithm has varied over time.

    The claim that there is no extra overhead for the small allocations is incorrect. There is still a small amount of overhead, but you can't see it because the !heap -x command doesn't report it. There is still overhead in the heap to keep track of the memory block's actual size, the fact that the memory block is allocated (and not free), and other overhead-type things. Indeed, even for large blocks, you didn't see the overhead reported by the !heap -x command; you had to drop below the heap and use the !vadump command to see it. (And how do you know that the other 3.9KB are being lost to overhead? Maybe they are being held for a future 3.9KB allocation?)

    Okay, so fine, small blocks have overhead, but why do larger blocks have significantly higher overhead? The answer to this is in the second question: The system allocates large blocks in a separate virtual address range because the default process heap treats large memory blocks differently from small memory blocks, in the same way that the CLR treats large and small objects differently.

    You might say that the unmanaged heap also has a large-object sub-heap.

    When objects get large, the heap switches to VirtualAlloc. You can think of it as creating a custom segment just for that object. And since VirtualAlloc allocates memory in integer multiples of the page size, any nonzero overhead will result in a full extra page being allocated since the requested allocation size was itself already an integral multiple of the page size. Mathematically:

    roundup(n × PageSize + v, PageSize) = n × PageSize + roundup(v, PageSize).

    Therefore, even if a heap allocation has only one byte of overhead, you will have to pay a full page for it due to rounding.

    The precise point at which the heap will switch to VirtualAlloc has changed over time, so don't rely on it. In Windows 95, the switchover point was around 4MB, but Windows NT set the cutover to something close to 512KB. If you're interested in details of the internal heap bookkeeping, you can check out Advanced Windows Debugging or Windows Internals. Note that information about the heap implementation is to be used for diagnostic and educational purposes only. Do not write code that takes advantage of the internals of the heap architecture, because those details change all the time.

    The purpose of a heap is to reduce memory and address space overhead compared to direct allocation via methods like VirtualAlloc by combining multiple small allocations into a single 64KB block, at a cost of some overhead per item. Although there is a small amount of overhead per item, the overall savings makes it a win. If you need eight hundred 100-byte chunks of memory, and you didn't have a heap manager, you would allocate eight hundred 64KB blocks (since VirtualAlloc allocates address space in 64KB chunks) and commit the first page in each, for a total memory usage of around 3MB and address space usage of around 50MB. Even if the heap overhead were a ridiculous 100%, the one thousand allocations would fit into forty 4KB pages, reducing the memory usage to 160KB and the address space usage to 192KB, a massive savings. (Looking at things in terms of relative overhead, 4KB out of a 512KB allocation is still less than one percent. You wish your index fund had such a low overhead!)

    The advantage of chunking allocations together diminishes as the size of the allocations increase. By the time you reach 512KB, the heap is not really buying you much savings, since you're already buying in bulk. In fact, the switchover to direct VirtualAlloc is an admission by the heap manager that the allocation size has become so large that it is starting to make the overall heap degrade. If the 512KB allocation were made in the classic heap, it would probably not start on an exact page boundary; when you went to free the memory, there would be little fragments left over at the start and end which could not be decommitted and which would end up as little committed pages scattered about fragmenting your address space. (Besides, I don't think the heap decommits partial segments anyway.)

    If you know ahead of time that your allocation is large, and you control both the code which allocates the memory and the code which frees the memory, you can switch to VirtualAlloc directly and avoid burdening the heap with very large allocations. (On the other hand, the heap does this cutover automatically, so perhaps you're better off just letting the heap designers decide how big is "too big.")

    Bonus chatter: And no, the heap manager will never ask VirtualAlloc for large pages.

  • The Old New Thing

    Appearing to succeed is a valid form of undefined behavior, but it's still undefined


    A customer requested a clarification on the MSDN documentation for the HeapFree function.

    The MSDN documentation says that if the lpMem parameter is NULL, then the behavior is undefined. Is this true?

    As explicitly stated in MSDN, the behavior is undefined. Observe that the annotation on the lpMem parameter is __in, which means that the parameter must be a non-NULL value provided by the caller. (If NULL were permitted, the annotation would have been __in_opt.)

    Undefined behavior means that anything can happen. The program might crash immediately. It might crash five minutes later. It might send email to your boss saying that you screwed up and then read you Vogon poetry. Or maybe not.

    MSDN says don't do it, so don't do it.

    The customer explained why they were interested in knowing more information about undefined behavior:

    We were interested because there is a mismatch between the semantics of a function we are implementing (where NULL is valid and ignored) and the function HeapFree we are using as the implementation. It looks like Windows Vista returns TRUE if you pass NULL.

    If there is a mismatch in semantics between the function you are implementing and the function you are calling, it is your responsibility as the programmer to bridge the gap. The customer didn't say what function they were implementing, but I'm guessing it was something like operator delete. Since your function accepts NULL but HeapFree doesn't, it is your responsibility to filter out NULL parameters.

    void operator delete(void* ptr) throw ()
     if (ptr != NULL)
      HeapFree(CustomHeap, 0, ptr);

    This concept goes by the fancy name of the Adapter Pattern. The less fancy name is wrapper function.

    And the value returned by HeapFree on Windows Vista is irrelevant. Pretending to succeed is a valid form of undefined behavior, because anything qualifies as undefined behavior.

    (Of course, you can't assume that returning TRUE will always be the result of triggering undefined behavior. After all, if you could rely on it, then it wouldn't be undefined any more!)

  • The Old New Thing

    Does this operation work when impersonating? The default answer is NO


    I'll often see a customer ask for assistance with a scenario like this: "We're having trouble doing X. We're doing X1, X2, and X3, but it looks like we're getting the wrong answer back."

    The next step in the conversation goes something like "There must be something else going on, because X1, X2 and X3 is the correct way of doing X. To demonstrate, I've written the following sample program that illustrates doing X by the X1, X2, X3 technique. When I run it, I get the correct answer. What do you get when you run it?"

    "When we run your program we get the correct answer, but it doesn't work when we do it from our program." And then, as if by afterthought, "Could the problem be that we're impersonating?"

    Ohhhhhh, you're impersonating. Thanks for not mentioning that.

    By default, nothing works when impersonating. Impersonation requires end-to-end awareness. A function might create a worker thread—the worker thread runs with the identity of the process. A function might use a function like Queue­Usere­Work­Item—by default, the work item runs with the identity of the process. (You have to pass WT_TRANSFER_IMPERSONATION if you want the work item to respect impersonation.) A function might send a message to another window—that window will do its work under its own security token, not the token of the sender. A function might invoke a method on a remote COM object—that object will run under its own security token, not the token of the invoker. (COM requires you to call Co­Set­Proxy­Blanket to enable impersonation transfer during marshaling, and the server needs to call CoImpersonateClient. For some reason, this is called cloaking.) The registry keys HKEY_CURRENT_USER and HKEY_CLASSES_ROOT don't work when you're impersonating. (You have to use RegOpenCurrentUser or RegOpenUserClassesRoot.) Functions like SHGetKnownFolderPath have a token parameter which is used when impersonating; if you pass NULL, then it assumes you aren't impersonating.

    The requirements go beyond just code that runs during the execution of the function in question. If you have a function which caches information across calls, the cache needs to be made impersonation-aware so that a value calculated when called while impersonating user X isn't mistakenly used while impersonating user Y.

    In order for impersonation to work, every function all the way down the chain needs to be impersonation-safe. Sure, you might be careful to call QueueUserWorkItem with the WT_TRANSFER_IMPERSONATION flag, and your work item is careful to call SetProxyBlanket on its COM objects, and your COM server is careful to call CoImpersonateClient when servicing the call, but if your COM server then calls a helper object which calls SHGetKnownFolderPath and passes NULL for the impersonation token, then all your careful work has been for naught.

    This is another special case of When you create an object with constraints, you have to make sure everybody who uses the object understands those constraints.

    The Programming Golden Rule can be applied here as well: When you write your own code, do you do this? Since most people who write code do not think about impersonation (indeed, the operating system even encourages not-impersonation-safe coding when it provides conveniences like HKEY_CURRENT_USER) the default answer to "Does this work when I'm impersonating" is "No."

  • The Old New Thing

    Ah, the exciting world of cross-forest dogfood


    The Windows group has its own domain (known as NTDEV for historical reasons) which operates separately from the domain forest operated by the Microsoft IT department. Various trust relationships need to be set up between them so that people on the Windows team can connect to resources managed by the Microsoft IT department and vice versa, but it generally works out okay.

    There are some glitches, but that's the price of dogfood. What better way to make sure that Windows works well in a cross-forest environment than by putting the entire Windows division in its own domain separate from the rest of the company?

    Subject: Newbie domain join question

    My user account is in the REDMOND domain. Should I join my machine to the REDMOND domain or the NTDEV domain?

    I was much amused by this cynical recommendation:

    Subject: RE: Newbie domain join question

    It's pretty simple, really.

    If you want to have trouble accessing NTDEV resources, then join the REDMOND domain.

    If you want to have trouble accessing REDMOND resources, then join the NTDEV domain.

  • The Old New Thing

    Sending a window a WM_DESTROY message is like prank calling somebody pretending to be the police


    A customer was trying to track down a memory leak in their program. Their leak tracking tool produced the stacks which allocated memory that was never freed, and they all seemed to come from uxtheme.dll, which is a DLL that comes with Windows. The customer naturally contacted Microsoft to report what appeared to be a memory leak in Windows.

    I was one of the people who investigated this case, and the customer was able to narrow down the scenario which was triggering the leak. Eventually, I tracked it down. First, here's the thread that caused the leak:

    DWORD CALLBACK ThreadProc(void *lpParameter)
     // This CreateWindow caused uxtheme to allocate some memory
     HWND hwnd = CreateWindow(...);
     MSG msg;
     while (GetMessage(&msg, NULL, 0, 0)) {
     return 0;

    This thread creates an invisible window whose job is to do something until it is destroyed, at which point the thread is no longer needed. The window procedure for the window looks like this:

                             WPARAM wParam, LPARAM lParam)
     switch (uMsg) {
     ... business logic deleted ...
     case WM_DESTROY:
     return DefWindowProc(hwnd, uMsg, wParam, lParam);

    Sinec this is the main window on the thread, its destruction posts a quit message to signal the message loop to exit.

    There's nothing obviously wrong here that would cause uxtheme to leak memory. And yet it does. The memory is allocated when the window is created, and it's supposed to be freed when the window is destroyed. And the only time we exit the message loop is when the window is destroyed. So how is it that this thread manages to exit without destroying the window?

    The key is how the program signals this window that it should go away.

    void MakeWorkerGoAway()
     // Find the worker window if it is registered
     HWND hwnd = GetWorkerWindow();
     // If we have one, destroy it
     if (hwnd) {
      // DestroyWindow doesn't work for windows that belong
      // to other threads.
      // DestroyWindow(hwnd);
      SendMessage(hwnd, WM_DESTROY, 0, 0);

    The authors of this code first tried destroying the window with DestroyWindow but ran into the problem that you cannot destroy a window that belongs to a different thread. "But aha, since the DestroyWindow function sends the WM_DESTROY message, we can just cut out the middle man and send the message directly."

    Well, yeah, you can do that, but that doesn't actually destroy the window. It just pretends to destroy the window by prank-calling the window procedure and saying "Ahem, um, yeah, this is the, um, window manager? (stifled laughter) And, um, like, we're just calling you to tell you, um, you're being destroyed. (giggle) So, um, you should like pack up your bags and (snort) sell all your furniture! (raucous laughter)"

    The window manager sends the WM_DESTROY message to a window as part of the window destruction process. If you send the message yourself, then you're making the window think that it's being destroyed, even though it isn't. (Because it's DestroyWindow that destroys windows.)

    The victim window procedure goes through its "Oh dear, I'm being destroyed, I guess I'd better clean up my stuff" logic, and in this case, it unregisters the worker window and posts a quit message to the message loop. The message loop picks up the WM_QUIT and exits the thread.

    And that's the memory leak: The thread exited before all its windows were destroyed. That worker window is still there, because it never got DestroyWindow'd. Since the window wasn't actually destroyed, the internal memory used to keep track of the window didn't get freed, and there you have your leak.

    "You just got punk'd!"

    The correct solution is for the MakeWorkerGoAway function to send a message to the worker window to tell it, "Hey, I'd like you to go away. Please call DestroyWindow on yourself." You can invent a private message for this, or you can take advantage of the fact that the default behavior of the WM_CLOSE message is to destroy the window. Since our window procedure doesn't override WM_CLOSE, the message will fall through to DefWindowProc which will convert the WM_CLOSE into a DestroyWindow.

    Now that you understand the difference between destroying a window and prank-calling a window telling it is being destroyed, you might be able to help Arno with his problem.

Page 120 of 455 (4,547 items) «118119120121122»