• The Old New Thing

    Got errands? Now is the time

    • 8 Comments

    This upcoming Sunday is the Super Bowl, the championship game for a sport played only in the United States.¹

    The entire country stops doing anything when the game is on. This makes it a perfect time to get out and run your errands, because the streets will be completely empty.

    Check out this traffic map at the kickoff of the 2014 Super Bowl. For fun, you can go backward in time in 10-minute increments and watch the traffic slowly die out as the start of the game approaches.

    If you're a photographer, it's a good time to go take pictures of public places, because they will all be deserted.

    It's also a good time to go to Costco, though you should wait until the game has started. It takes time for all the people getting last-minute party supplies to drain out.

    ¹ Well, Canada has their own variant. They're so cute, those Canadians.

    Bonus chatter: The American Football League of China is a real thing.

  • The Old New Thing

    Creating a shared memory block that can grow in size

    • 7 Comments

    A little-known feature of shared memory blocks in Win32 is that it is possible to resize them, sort of.

    When you create a shared memory block, you can pass the SEC_RESERVE flag to Create­File­Mapping, then the size you pass to the function is treated as a maximum rather than an exact size. (Don't forget that Create­File­Mapping is used for creating both memory-mapped files and for creating plain old shared memory. The name of the function is misleading unless you're wearing kernel-colored glasses.)

    When you map this shared memory block, you are reserving address space, but no memory is committed yet. You call Virtual­Alloc to commit memory into the shared memory block.

    This means that you can create a growable shared memory block by creating an initially empty block, and then committing a small amount of memory into it. When you want to grow the block, you commit more. However, you cannot shrink the shared memory block. Once the memory is committed, it cannot be decommitted.

    Here's a demonstration. Note that most error checking has been elided for expository purposes. Note also that since the memory isn't actually being shared with anybody, this program working too hard; it could have just used plain old Virtual­Alloc. So pretend that the memory is being shared with somebody else.

    #include <windows.h>
    #include <stdio.h>
    
    #define ONE_GIGABYTE (1024 * 1024 * 1024)
    #define VIEW_SIZE (ONE_GIGABYTE / 2) // We will map half of it
    
    void ReportMemoryPresence(void *p)
    {
     MEMORY_BASIC_INFORMATION mbi;
     VirtualQuery(p, &mbi, sizeof(mbi));
     printf("Memory at %p is %s\n", p,
            (mbi.State & MEM_COMMIT) ? "committed" : "not committed");
    }
    
    void WaitForEnter()
    {
     char dummy[64];
     fgets(dummy, 64, stdin);
    }
    
    int __cdecl wmain(int, wchar_t **)
    {
     BYTE *pView;
     HANDLE h = CreateFileMapping(INVALID_HANDLE_VALUE, NULL,
                                  PAGE_READWRITE,
                                  0, VIEW_SIZE,
                                  NULL);
     printf("Created the file mapping\n");
     WaitForEnter();
    
     pView = (BYTE*)MapViewOfFile(h, FILE_MAP_WRITE, 0, 0, VIEW_SIZE);
     printf("Mapped half of it at %p\n", pView);
    
     ReportMemoryPresence(pView);
     ReportMemoryPresence(pView + VIEW_SIZE - 1);
     WaitForEnter();
    
     return 0;
    }
    

    In this version, we create a one-gigabyte shared memory block with no special flags, which means that all the memory gets committed up front. When you run this program, it reports that the memory at the start and end of the mapping is present. That's because the normal mode for shared memory is to commit it all at creation.

    You can watch the effect of commit by running Task Manager, going to the Performance tab, and looking at the value under Committed. It should jump by a gigabyte when "Created the file mapping" is printed. (For some reason, the Commit size in the Details pane counts the view as commitment, even though the view consists almost entirely of reserved rather than committed pages.)

    Now let's add the SEC_RESERVE flag:

     HANDLE h = CreateFileMapping(INVALID_HANDLE_VALUE, NULL,
                                  PAGE_READWRITE | SEC_RESERVE,
                                  0, VIEW_SIZE,
                                  NULL);
    

    Now when you run the program, Task Manager's Committed memory does not increase. That's because we created an empty shared memory block with the potential to grow up to one gigabyte, but right now it is size zero. This is confirmed by the memory presence check, which reports that the memory at the start and end of the mapped view is not committed.

    Okay, well, a zero-length shared memory block isn't very useful, so let's make it, say, 100 megabytes in size.

    #define BLOCK_SIZE (100 * 1024 * 1024)
    
    int __cdecl wmain(int, wchar_t **)
    {
     BYTE *pView;
     HANDLE h = CreateFileMapping(INVALID_HANDLE_VALUE, NULL,
                                  PAGE_READWRITE | SEC_RESERVE,
                                  0, VIEW_SIZE,
                                  NULL);
     printf("Created the file mapping\n");
     WaitForEnter();
    
     pView = (BYTE*)MapViewOfFile(h, FILE_MAP_WRITE, 0, 0, VIEW_SIZE);
     printf("Mapped half of it at %p\n", pView);
    
     ReportMemoryPresence(pView);
     ReportMemoryPresence(pView + VIEW_SIZE - 1);
     WaitForEnter();
    
     VirtualAlloc(pView, BLOCK_SIZE, MEM_COMMIT, PAGE_READWRITE);
     printf("Committed some of it at %p\n", pView);
    
     ReportMemoryPresence(pView);
     ReportMemoryPresence(pView + BLOCK_SIZE - 1);
     ReportMemoryPresence(pView + BLOCK_SIZE);
     ReportMemoryPresence(pView + VIEW_SIZE - 1);
     WaitForEnter();
    
     return 0;
    }
    

    Watch the Committed memory in Task Manager go up by 0.1 gigabytes when we commit some of it. Also observe that the memory presence checks show that we have exactly 100 megabytes of memory available; the byte at 100 megabytes + 1 is not present.

    Okay, so we were able to grow the shared memory block from zero to 100 megabytes. Let's grow it again up to 200 megabytes.

    int __cdecl wmain(int, wchar_t **)
    {
     ...
    
     VirtualAlloc(pView + BLOCK_SIZE, BLOCK_SIZE, MEM_COMMIT, PAGE_READWRITE);
     printf("Committed some of it at %p\n", pView + BLOCK_SIZE);
    
     ReportMemoryPresence(pView);
     ReportMemoryPresence(pView + 2 * BLOCK_SIZE - 1);
     ReportMemoryPresence(pView + 2 * BLOCK_SIZE);
     ReportMemoryPresence(pView + VIEW_SIZE - 1);
     WaitForEnter();
    
     return 0;
    }
    

    Okay, well there you go, a growable shared memory block. If you wanted to conserve address space, you could use Map­View­Of­File to map only the number of bytes you intend to commit, and each time you want to grow the memory block, you create a new larger view. I didn't bother with that because I'm lazy.

    Bonus chatter: Another way to get the effect of growable and shrinkable shared memory blocks is to cheat and create multiple shared memory blocks, but map them right next to each other.

    Bonus chatter 2: You can get sort of the effect of decommitting memory from the block by resetting it (MEM_RESET). The memory is still committed, but you told the memory manager that if the memory needs to be paged out, just discard it rather than writing it to disk.

    Bonus chatter 3: Be aware that creating very large SEC_RESERVE sections can incur high commit charges for the page tables themselves. This is significantly improved in Windows 8.1, which defers committing the page tables until you actually use them.

  • The Old New Thing

    Color-aware ClearType requires access to fixed background pixels, which is a problem if you don't know what the background pixels are, or if they aren't fixed

    • 37 Comments

    ClearType is a technology that blends text pixels with background pixels with the goal of reducing visual artifacts during text rendering.¹ If you go for the full-meal version of ClearType, it uses knowledge about the physical properties of LCD screens in order to carry out its work. Some time ago, I noted one catch with this model, which is that overdraw gives the ClearType algorithm the wrong background pixels.

    Another gotcha is that if you don't know what the background pixels are at all, then you can't use ClearType. For example, you might be rendering a semi-transparent bitmap that will be drawn on top of other content. If you try to use ClearType to render text onto this bitmap, the ClearType engine will see transparent pixels as the background, and blend accordingly. (My guess is that it will treat them as black pixels.) But when you actually draw this bitmap to the screen, those transparent pixels will allow the pixels below the bitmap to shine through, and those underlying pixels are not transparent.

    The result is color fringes because the ClearType engine was given incorrect background pixels.

    Another assumption that the ClearType engine makes is that the bitmap will be drawn on exact pixel boundaries without any stretching or shrinking or rotation. Since ClearType is doing math based on the physical LCD, you break the ClearType model if you scale the bitmap, rotate it, or render it on a fractional-pixel boundary because each carefully-crafted ClearType pixel does not end up as a single pixel on the LCD panel. (The mapping of original pixels to transformed pixels is controlled by something called an interpolation mode. For example, Direct2D offers a variety of interpolation modes.)

    If you break this rule and use ClearType anyway, you will once again get the dreaded color fringes.

    Nearly all of these factors come into play on the Windows 8 Start screen.

    The word Start on that page is rendered onto a transparent bitmap because it needs to overlay on top of your Start background, and most of the Start backgrounds are not fixed; they scroll slowly as you pan through your tiles.

    The tiles themselves translate when you pan the Start screen, and they scale when you go into Customize mode or when you press them, and they rotate when you tap on them. This means that any text drawn onto a tile cannot use the color-aware version of ClearType.

    These concerns apply more generally to any bitmap that will be scrolled. A bitmap that scrolls with subpixel positioning cannot use the color-aware version of ClearType because ClearType assumes integer-pixel positioning. (This is why Internet Explorer doesn't use color-aware ClearType. Scrolling is performed with Direct Manipulation, and Direct Manipulation does subpixel scrolling.)

    Now, of course, you could work around this problem. You could design your interface so that it doesn't require transparent bitmaps, say by using fixed backgrounds. And you could design your interface so it doesn't use subpixel positioning, scaling, or rotation. Or you could simply stop pre-rendering text and instead rerender them on-the-fly each time something changes. The first two workarounds impair your design. The second impairs your performance, since moving a bitmap is no longer a simple update of a sprite's coordinates and transformation matrix; you have to go through a full text rendering pass, including a pixel read-back in order to figure out what the current background pixels are. (And pixel read-back from video memory is not cheap since it forces the composition tree to flatten.)

    Both trade-offs are pretty expensive, and the cheaper alternative is usually simply to stop using color-aware ClearType.

    ¹ Yes, there are people who don't like ClearType. The point of today's article isn't about whether ClearType is good or bad; it's about ClearType's limitations. I tried to remain neutral on the subject by saying that improved text rendering is the goal of ClearType, making no statement about whether it achieves that goal.

  • The Old New Thing

    The compiler can make up its own calling conventions, within limits

    • 19 Comments

    A customer was confused by what they were seeing when debugging.

    It is our understanding that the Windows x86-64 calling convention passes the first four parameters in registers rcx, rdx, r8, and r9. But we're seeing the parameters being passed some other way. Given the function prototype

    int LogFile::Open(wchar_t *path, LogFileInfo *info, bool verbose);
    

    we would expect to see the parameters passed as

    • rcx = this
    • rdx = path
    • r8 = info
    • r9 = verbose

    but instead we're seeing this:

    rax=0000000001399020 rbx=0000000003baf238 rcx=00000000013c3260
    rdx=0000000003baf158 rsi=000000000139abf0 rdi=00000000013c3260
    rip=00007ffd69b71724 rsp=0000000003baf038 rbp=0000000003baf0d1
    r8=0000000001377870  r9=0000000000000000 r10=000000007fffffb9
    r11=00007ffd69af08e8 r12=00000000013a3b80 r13=0000000000000000
    r14=0000000001399010 r15=00000000013a3b90
    contoso!LogFile::Open:
    00007ffd`69b71724 fff3        push rbx
    0:001> du @rdx          // path should be in rdx
    00000000`03baf158  "`"
    0:001> du @r8           // but instead it's in r8
    00000000`01377870  "C:\Logs\Contoso.txt"
    

    Is our understanding of the calling convention incomplete?

    There are three parties to a calling convention.

    1. The function doing the calling.
    2. The function being called.
    3. The operating system.

    The operating system needs to get involved if something unusual occurs, like an exception, and it needs to go walking up the stack looking for a handler.

    The catch is that if a compiler knows that it controls all the callers of a function, then it can modify the calling convention as long as the modified convention still observes the operating system rules. After all, the operating system doesn't see your source code. As long as the object code satisfies the calling convention rules, everything is fine. (This typically means that the modification needs to respect unwind codes and stack usage.)

    For example, suppose you had code like this:

    extern void bar(int b, int a);
    
    static void foo(int a, int b)
    {
      return bar(b + 1, a);
    }
    
    int __cdecl main(int argc, char **argv)
    {
     foo(10, 20);
     foo(30, 40);
     return 0;
    }
    

    A clever compiler could make the following analysis: Since foo is a static function, it can be called only from this file. And in this file, the address of the function is never taken, so the compiler knows that it controls all the callers. Therefore, it optimizes the function foo by rewriting it as

    static void foo(int b, int a)
    {
      return bar(b + 1, a);
    }
    

    It makes corresponding changes to main:

    int __cdecl main(int argc, char *argv)
    {
     foo(20, 10); // flip the parameters
     foo(40, 30); // flip the parameters
     return 0;
    }
    

    By doing this, the compiler can generate the code for foo like this:

    foo:
            inc     ecx
            jmp     bar
    

    rather than the more conventional

    foo:
            mov     eax, edx
            inc     eax
            mov     ecx, edx
            mov     edx, eax
            jmp     bar
    

    You can look at this transformation in one of two ways. You can say, "The compiler rewrote my function prototype to be more efficient." Or you can say, "The compiler is using a custom calling convention for foo which passes the parameters in reverse order."

    Both interpretations are just two ways of viewing the same thing.

  • The Old New Thing

    Tip for trying to boost morale: Don't brag about your overseas trips

    • 25 Comments

    Once upon a time, a senior manager held a team meeting to address low team morale. Attempting to highlight how important the project was, he opened by saying, "I just got back from ⟨faraway country⟩, meeting with some of our important clients, and..."

    This remark had exactly the opposite effect from what the manager intended. Instead of revitalizing the team, the team became even more despondent. "Here we are, working late and coming in on weekends, and this senior manager is telling us about his recent overseas junket."

    (After he left, I heard that he still pulled this stunt with his new team. "Over the past two weeks, I've been all over ⟨continent⟩...")

  • The Old New Thing

    Customizing item enumeration with IShellItem

    • 5 Comments

    If you are using the original IShell­Folder interface, then you can use SHCONTF values to customize how child items are enumerated. But what if you're using IShell­Item?

    Let's take it one step at a time. First, the basic program. (Remember, Little Programs do little to no error checking.)

    #define UNICODE
    #define _UNICODE
    #define STRICT
    #define STRICT_TYPED_ITEMIDS
    #include <windows.h>
    #include <shlobj.h>
    #include <atlbase.h>
    #include <atlalloc.h>
    #include <propsys.h>
    #include <stdio.h>
    
    int __cdecl wmain(int argc, wchar_t **argv)
    {
     CCoInitialize init;
    
     if (argc < 2) return 0;
     CComPtr<IShellItem> spsiFolder;
     SHCreateItemFromParsingName(argv[1], nullptr,
                                 IID_PPV_ARGS(&spsiFolder));
    
     CComPtr<IEnumShellItems> spesi;
     spsiFolder->BindToHandler(nullptr, BHID_EnumItems,
                                  IID_PPV_ARGS(&spesi));
    
     for (CComPtr<IShellItem> spsi;
          spesi->Next(1, &spsi, nullptr) == S_OK;
          spsi.Release()) {
      PrintDisplayName(spsi, SIGDN_NORMALDISPLAY, L"Display Name");
      PrintDisplayName(spsi, SIGDN_FILESYSPATH, L"File system path");
      wprintf(L"\n");
     }
     return 0;
    }
    

    Run this program with the fully-qualified path to a directory as the command line argument, and it enumerates all the items in the folder. This uses the default enumeration, which is "include both folders and non-folders, and include hidden items, but not super-hidden items."

    But what if we want to customize the enumeration?

    We saw that the IBindCtx parameter acts as a catch-all options parameter. In this case, we need to look at the options available for BHID_Enum­Items and see if any of them help us.

    Fortunately, we have STR_ENUM_ITEMS_FLAGS which lets us override the default enumeration mode. Let's say that we want only folders, and we want to respect the user's preferences for hidden items (which means that we omit SHCONTF_HIDDEN).

    I'm goint to do this two ways. First the flat version:

     ...
     CComPtr<IBindCtx> spbcEnum;
     CreateDwordBindCtx(STR_ENUM_ITEMS_FLAGS, SHCONTF_FOLDERS,
                        &spbcEnum);
    
     CComPtr<IEnumShellItems> spesi;
     spsiFolder->BindToHandler(spbcEnum, BHID_EnumItems,
                                  IID_PPV_ARGS(&spesi));
    

    Now the fluent version:

     ...
     CBindCtxBuilder builder;
     builder.SetVariantDword(STR_ENUM_ITEMS_FLAGS, SHCONTF_FOLDERS);
    
     CComPtr<IEnumShellItems> spesi;
     spsiFolder->BindToHandler(builder.GetBindCtx(), BHID_EnumItems,
                                  IID_PPV_ARGS(&spesi));
    

    (Don't forget that error checking has been elided for expository purposes.)

    The STR_ENUM_ITEMS_FLAGS bind context string was added in Windows 8, so it has no effect on older versions of Windows. We'll address this next week.

    Note that the IEnum­Shell­Items interface is incorrectly-named. The convention for enumeration interfaces is to name them IEnum­XXX, where XXX is singular.

  • The Old New Thing

    Helper functions to make shell bind contexts slightly more manageable

    • 11 Comments

    Last time, we learned about the wonderful world of shell bind context strings, and I promised some helper functions to make this slightly more manageable.

    Here are some helper functions which supplement the Create­Bind­Ctx­With­Opts function we created some time ago.

    #include <propsys.h>
    
    HRESULT EnsureBindCtxPropertyBag(
        IBindCtx *pbc, REFIID riid, void **ppv)
    {
     *ppv = nullptr;
     CComPtr<IUnknown> spunk;
     HRESULT hr = pbc->GetObjectParam(STR_PROPERTYBAG_PARAM, &spunk);
     if (FAILED(hr)) {
      hr = PSCreateMemoryPropertyStore(IID_PPV_ARGS(&spunk));
      if (SUCCEEDED(hr)) {
       hr = pbc->RegisterObjectParam(STR_PROPERTYBAG_PARAM, spunk);
      }
     }
     if (SUCCEEDED(hr)) {
      hr = spunk->QueryInterface(riid, ppv);
     }
      return hr;
    }
    
    HRESULT AddBindCtxDWORD(
        IBindCtx *pbc, LPCWSTR pszName, DWORD dwValue)
    {
     CComPtr<IPropertyBag> sppb;
     HRESULT hr = EnsureBindCtxPropertyBag(pbc, IID_PPV_ARGS(&sppb));
     if (SUCCEEDED(hr)) {
      hr = PSPropertyBag_WriteDWORD(sppb, pszName, dwValue);
     }
     return hr;
    }
    
    HRESULT AddBindCtxString(
        IBindCtx *pbc, LPCWSTR pszName, LPCWSTR pszValue)
    {
     CComPtr<IPropertyBag> sppb;
     HRESULT hr = EnsureBindCtxPropertyBag(pbc, IID_PPV_ARGS(&sppb));
     if (SUCCEEDED(hr)) {
      hr = PSPropertyBag_WriteStr(sppb, pszName, pszValue);
     }
     return hr;
    }
    
    HRESULT CreateDwordBindCtx(
        LPCWSTR pszName, DWORD dwValue, IBindCtx **ppbc)
    {
     CComPtr<IBindCtx> spbc;
     HRESULT hr = CreateBindCtx(0, &spbc);
     if (SUCCEEDED(hr)) {
      hr = AddBindCtxDWORD(spbc, pszName, dwValue);
     }
     *ppbc = SUCCEEDED(hr) ? spbc.Detach() : nullptr;
     return hr;
    }
    
    HRESULT CreateStringBindCtx(
        LPCWSTR pszName, LPCWSTR pszValue, IBindCtx **ppbc)
    {
     CComPtr<IBindCtx> spbc;
     HRESULT hr = CreateBindCtx(0, &spbc);
     if (SUCCEEDED(hr)) {
      hr = AddBindCtxString(spbc, pszName, pszValue);
     }
     *ppbc = SUCCEEDED(hr) ? spbc.Detach() : nullptr;
     return hr;
    }
    

    The Ensure­Bind­Ctx­Property­Bag function puts a property bag in the bind context if there isn't one already.

    The Add­Bind­Ctx­DWORD function adds a DWORD to that associated property bag. If you wanted to add multiple DWORDs to a bind context, you would call this function multiple times. You can also use the Add­Bind­Ctx­String if the thing you want to add is a string.

    The Create­Dword­Bind­Ctx function handles the simple case where you want to create a bind context that contains a single DWORD. Similarly, Create­String­Bind­Ctx.

    But now things are starting to get kind of unwieldy. What if you want a bind context with a string and a DWORD? Let's go for something a bit more fluent.

    But first, some scaffolding.

    class CStaticUnknown : public IUnknown
    {
    public:
     // *** IUnknown ***
     IFACEMETHODIMP QueryInterface(
      _In_ REFIID riid, _Outptr_ void **ppv)
     {
      *ppv = nullptr;
      HRESULT hr = E_NOINTERFACE;
      if (riid == IID_IUnknown) {
       *ppv = static_cast<IUnknown *>(this);
       AddRef();
       hr = S_OK;
      }
      return hr;
     }
    
     IFACEMETHODIMP_(ULONG) AddRef()
     {
      return 2;
     }
    
     IFACEMETHODIMP_(ULONG) Release()
     {
      return 1;
     }
    
    };
    
    CStaticUnknown s_unkStatic;
    

    This static implementation of IUnknown is one we'll use for the bind context strings whose mere presence indicates that a flag is set.

    class CBindCtxBuilder
    {
    public:
     CBindCtxBuilder()
     {
      m_hrCumulative = CreateBindCtx(0, &m_spbc);
     }
    
     CBindCtxBuilder& SetMode(DWORD grfMode);
     CBindCtxBuilder& SetFindData(const WIN32_FIND_DATA *pfd);
     CBindCtxBuilder& SetFlag(PCWSTR pszName);
     CBindCtxBuilder& SetVariantDword(PCWSTR pszName, DWORD dwValue);
     CBindCtxBuilder& SetVariantString(PCWSTR pszName, PCWSTR pszValue);
    
     HRESULT Result() const { return m_hrCumulative; }
    
     IBindCtx *GetBindCtx() const
     { return SUCCEEDED(m_hrCumulative) ? m_spbc : nullptr; }
    private:
     HRESULT EnsurePropertyBag();
    
    private:
     CComPtr<IBindCtx> m_spbc;
     CComPtr<IPropertyBag> m_sppb;
     HRESULT m_hrCumulative;
    };
    

    The bind context builder class is a helper class that creates a bind context, and then fills it with stuff. For now, we let you set the following:

    After you build up the bind context, you can check the Result() to see if it was built successfully, and use Get­Bind­Ctx to extract the result.

    Here's the implementation. It's really not that exciting. We accumulate any error in m_hrCumulative, and once an error occurs, all future methods do nothing aside from preserving the error. To make the object fluent, the methods return a reference to themselves.

    There is a special bind context method for setting the mode:

    CBindCtxBuilder&
    CBindCtxBuilder::SetMode(DWORD grfMode)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      BIND_OPTS bo = { sizeof(bo), 0, grfMode, 0 };
      m_hrCumulative = m_spbc->SetBindOptions(&bo);
     }
     return *this;
    }
    

    Find data is set as a direct object on the bind context, as we saw some time ago:

    CBindCtxBuilder&
    CBindCtxBuilder::SetFindData(const WIN32_FIND_DATA *pfd)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = AddFileSysBindCtx(m_spbc, pfd);
     }
     return *this;
    }
    

    Flags are set by there mere presence, so we associate them with a dummy IUnknown that does nothing:

    CBindCtxBuilder&
    CBindCtxBuilder::SetFlag(PCWSTR pszName)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = m_spbc->RegisterObjectParam(
        const_cast<PWSTR>(pszName), &s_unkStatic);
     }
     return *this;
    }
    

    If a property is set in the property bag, we need to proceed in two steps. First, we create the property bag if we don't have one already. Second, we put the value into the property bag:

    CBindCtxBuilder&
    CBindCtxBuilder::SetVariantDword(
        PCWSTR pszName, DWORD dwValue)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = EnsurePropertyBag();
     }
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative =  PSPropertyBag_WriteDWORD(
        m_sppb, pszName, dwValue);
     }
     return *this;
    }
    
    CBindCtxBuilder&
    CBindCtxBuilder::SetVariantString(
        PCWSTR pszName, PCWSTR pszValue)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = EnsurePropertyBag();
     }
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative =  PSPropertyBag_WriteStr(
        m_sppb, pszName, pszValue);
     }
     return *this;
    }
    

    And finally, the helper function that creates a property bag if we don't have one already.

    HRESULT CBindCtxBuilder::EnsurePropertyBag()
    {
     HRESULT hr = S_OK;
     if (!m_sppb) {
      hr = PSCreateMemoryPropertyStore(
        IID_PPV_ARGS(&m_sppb));
      if (SUCCEEDED(hr)) {
       hr = m_spbc->RegisterObjectParam(
        STR_PROPERTYBAG_PARAM, m_sppb);
      }
     }
     return hr;
    }
    

    The idea here is that the class is used like this:

    CBindCtxBuilder builder;
    builder.SetMode(STGM_CREATE)
           .SetFindData(&wfd)
           .SetFlag(STR_FILE_SYS_BIND_DATA_WIN7_FORMAT)
           .SetFlag(STR_BIND_FOLDERS_READ_ONLY);
    hr = builder.Result();
    if (SUCCEEDED(hr)) {
     hr = psf->ParseDisplayName(hwnd, builder.GetBindCtx(),
       pszName, &cchEaten, &pidl, &dwAttributes);
    }
    

    You create the bind context builder, then use the various Set­Xxx methods to fill the bind context with goodies, and then you check if it all worked okay. If so, then you use Get­Bind­Ctx to get the resulting bind context and proceed on your way.

  • The Old New Thing

    The wonderful world of shell bind context strings

    • 6 Comments

    Some time ago, we saw how the IBindCtx parameter to IShell­Folder::Parse­Display­Name can be used to modify how the parse takes place. More generally, the IBindCtx parameter to a function is a catch-all miscellaneous options parameter.

    The interesting part of the bind context is all the stuff that has been added to it via the IBindCtx::Register­Object­Param method. You can attach arbitrary objects to the bind context, using a string to identify each one.

    Some bind context parameters are like Boolean parameters that simply change some default behavior of the operation. For these operations, the object that is associated with the bind context string is not important; the important thing is that there is something associated with it. Traditionally, you just connect a dummy object that implements just IUnknown.

    In the most general case, the object associated with a bind context string implements some operation-specific interface. For example, the STR_BIND_DELEGATE_CREATE_OBJECT bind context string expects you to associate an object that implements the ICreate­Object interface, because the whole point of the STR_BIND_DELEGATE_CREATE_OBJECT bind context string is to say, "Hey, I want to create objects in a nonstandard way," so you need to tell it what that nonstandard way is.

    At the other extreme, you may have a chunk of data that you want to associate with the bind context string. Since bind contexts want to associate objects, you need to wrap the data inside a COM object. We saw this earlier when we had to create an object that implemented the IFile­System­Bind­Data interface in order to babysit a WIN32_FIND_DATA structure.

    Rather than having to create a separate interface for each data type (hello, IObject­With­Folder­Enum­Mode), and rather than going to the opposite extreme and just using IStream to pass arbitrary unstructured data, the shell folks decided to take a middle-ground approach: Use a common interface that still has a modicum of type safety, namely, IProperty­Bag. Other nice things about this approach is that there are a lot of pre-existing helper functions for property bags and property variants. Also, you need to attach only one object instead of a whole bunch of tiny little ones.

    Under this new regime (which took hold in Windows 8), the bind context has an associated property bag, and you put your data in that property bag.

    In pictures:

      IBindCtx::Register­Object­Param IProperty­Bag::Write
    IBindCtx Boolean parameter IUnknown
    Interface parameter object with custom interface
    STR_PROPERTY­BAG_PARAM IPropertyBag Property bag DWORD parameter VT_UI4
    Property bag string parameter VT_BSTR
    Property bag Boolean parameter VT_BOOL

    If you want a Boolean-style parameter to be true, then set it in the bind context with a dummy object that implements IUnknown. If you want a Boolean-style parameter to false, then omit it from the bind context entirely.

    To set an interface-style parameter, set it in the bind context with an object that implements the desired interface.

    To set a property bag-based parameter, set it in the property bag with the appropriate variant type.

    Here are the bind context strings defined up through Windows 8.1 and the way you set them into the bind context.

    Bind context string Model Operation
    STR_AVOID_DRIVE_RESTRICTION_POLICY Boolean Binding
    STR_BIND_DELEGATE_CREATE_OBJECT Interface ICreateObject Binding
    STR_BIND_FOLDER_ENUM_MODE Interface IObjectWith­FolderEnumMode Parsing
    STR_BIND_FOLDERS_READ_ONLY Boolean Parsing
    STR_BIND_FORCE_FOLDER_SHORTCUT_RESOLVE Boolean Binding
    STR_DONT_PARSE_RELATIVE Boolean Parsing
    STR_DONT_RESOLVE_LINK Boolean Binding
    STR_ENUM_ITEMS_FLAGS Property bag: VT_UI4 Binding for enumeration
    STR_FILE_SYS_FIND_DATA Interface IFileSys­BindData or IFileSys­BindData2 Parsing
    STR_FILE_SYS_BIND_DATA_WIN7_FORMAT Boolean Parsing
    STR_GET_ASYNC_HANDLER Boolean GetUIObjectOf
    STR_GPS_BEST­EFFORT Boolean Binding for IProperty­Store
    STR_GPS_DELAY­CREATION Boolean Binding for IProperty­Store
    STR_GPS_FAST­PROPERTIES­ONLY Boolean Binding for IProperty­Store
    STR_GPS_HANDLER­PROPERTIES­ONLY Boolean Binding for IProperty­Store
    STR_GPS_NO_OPLOCK Boolean Binding for IProperty­Store
    STR_GPS_OPEN­SLOW­ITEM Boolean Binding for IProperty­Store
    STR_IFILTER_FORCE_TEXT_FILTER_FALLBACK Boolean Binding for IFilter
    STR_IFILTER_LOAD_DEFINED_FILTER Boolean Binding for IFilter
    STR_INTERNAL_NAVIGATE Boolean Loading history
    STR_INTERNET­FOLDER_PARSE_ONLY_URLMON_BINDABLE Boolean Parsing
    STR_ITEM_CACHE_CONTEXT Interface IBindCtx Parsing and initiailzing
    STR_NO_VALIDATE_FILE­NAME_CHARS Boolean Parsing
    STR_PARSE_ALLOW_INTERNET_SHELL_FOLDERS Boolean Parsing
    STR_PARSE_AND_CREATE_ITEM Interface IParse­And­Create­Item Parsing
    STR_PARSE_DONT_REQUIRE_VALIDATED_URLS Boolean Parsing
    STR_PARSE_EXPLICIT_ASSOCIATION_SUCCESSFUL Property bag: VT_BOOL Parsing
    STR_PARSE_PARTIAL_IDLIST Interface IShell­Item Parsing
    STR_PARSE_PREFER_FOLDER_BROWSING Boolean Parsing
    STR_PARSE_PREFER_WEB_BROWSING Boolean Parsing
    STR_PARSE_PROPERTY­STORE Interface IProperty­Bag Parsing
    STR_PARSE_SHELL_PROTOCOL_TO_FILE_OBJECTS Boolean Parsing
    STR_PARSE_SHOW_NET_DIAGNOSTICS_UI Boolean Parsing
    STR_PARSE_SKIP_NET_CACHE Boolean Parsing
    STR_PARSE_TRANSLATE_ALIASES Boolean Parsing
    STR_PARSE_WITH_EXPLICIT_ASSOCAPP Property bag: VT_BSTR Parsing
    STR_PARSE_WITH_EXPLICIT_PROGID Property bag: VT_BSTR Parsing
    STR_PARSE_WITH_PROPERTIES Interface IProperty­Store Parsing
    STR_PROPERTYBAG_PARAM Interface IProperty­Bag holds property bag parameters
    STR_SKIP_BINDING_CLSID Interface IPersist Parsing and binding

    There are some oddities in the above table.

    • All of the STR_GPS_* values would be more conveniently expressed as a single VT_UI4 property bag-based value. (Exercise: Why isn't it?)
    • The STR_ITEM_CACHE_CONTEXT bind context parameter is itself another bind context! The idea here is that you, the caller, are enabling caching during the parse, and the inner bind context acts as the cache.
    • The STR_PARSE_EXPLICIT_ASSOCIATION_SUCCESSFUL value is unusual in that it is something set by the parser and passed back to the caller.
    • As we have been discussing, STR_PROPERTY­BAG_PARAM is a bind context string that doesn't mean anything on its own. Rather, it provides a property bag into which more parameters can be stored.

    Next time, I'll write some helper functions to make all this slightly more manageable.

  • The Old New Thing

    Why does my synchronous overlapped ReadFile return FALSE when the end of the file is reached?

    • 16 Comments

    A customer reported that the behavior of Read­File was not what they were expecting.

    We have a synchronous file handle (not created with FILE_FLAG_OVERLAPPED), but we issue reads against it with an OVERLAPPED structure. We find that when we read past the end of the file, the Read­File returns FALSE even though the documentation says it should return TRUE.

    They were kind enough to include a simple program that demonstrates the problem.

    #include <windows.h>
    
    int __cdecl wmain(int, wchar_t **)
    {
     // Create a zero-length file. This succeeds.
     HANDLE h = CreateFileW(L"test", GENERIC_READ | GENERIC_WRITE,
                   0, nullptr, CREATE_ALWAYS,
                   FILE_ATTRIBUTE_NORMAL, nullptr);
    
     // Read past EOF.
     char buffer[10];
     DWORD cb;
     OVERLAPPED o = { 0 };
     ReadFile(h, buffer, 10, &cb, &o); // returns FALSE
     GetLastError(); // returns ERROR_HANDLE_EOF
    
     return 0;
    }
    

    The customer quoted this section from The documentation for Read­File:

    Considerations for working with synchronous file handles:

    • If lpOverlapped is NULL, the read operation starts at the current file position and Read­File does not return until the oepration is complete, and the system updates the file pointer before Read­File returns.
    • If lpOverlapped is not NULL, the read operation starts at the offset that is specified in the OVERLAPPED structure and Read­File does not return until the read operation is complete. The system updates the OVERLAPPED offset before Read­File returns.
    • When a synchronous read operation reads the end of a file, Read­File returns TRUE and sets *lpNumberOfBytesRead to zero.

    and then added

    According to the third bullet point, the Read­File should return TRUE, but in practice it returns FALSE and the error code is ERROR_HANDLE_EOF.

    The problem here is that there are two concepts here, and they confusingly both use the word synchronous.

    • A synchronous file handle is a handle opened without FILE_FLAG_OVERLAPPED. All I/O to a synchronous file handle is serialized and synchronous.
    • A synchronous I/O operation is an I/O issued with lpOverlapped == NULL.

    The sample program issues an asynchronous read against a synchronous handle. The third bullet point applies only to synchronous reads.

    To reduce confusion, the documentation would have been clearer if it hadn't switched terminology midstream.

    • If lpOverlapped is NULL, the read operation starts at the current file position and Read­File does not return until the oepration is complete, and the system updates the file pointer before Read­File returns.
    • If lpOverlapped is not NULL, the read operation starts at the offset that is specified in the OVERLAPPED structure and Read­File does not return until the read operation is complete. The system updates the OVERLAPPED offset before Read­File returns.
    • If lpOverlapped is NULL and the read operation reads the end of a file, Read­File returns TRUE and sets *lpNumberOfBytesRead to zero.

    We asked what the customer was doing that caused them to trip over this confusion in the documentation.

    The customer's original code opened a file (synchronously) and read from it (synchronously). The customer is parallelizing the computation in a way that will read that single file from multiple threads. A single file pointer is therefore not suitable, because different threads will want to read from different positions.

    One idea would be to have each thread call Create­File so that each handle has its own file position. Unfortunately, this won't work for the customer because the sharing mode on the file handle denies read sharing.

    The solution they came up with was to open the file synchronously (without FILE_FLAG_OVERLAPPED) but to read asynchronously (by using an OVERLAPPED structure). The OVERLAPPED structure lets you specify where you want to read from, so multiple threads can issue reads against the file position they want.

    This solution works, but the customer is concerned because this hybrid model is not well-documented in MSDN. They found a blog entry that discusses it, but even that blog entry does not discuss what happens in the multithreaded case.) In particular, they are seeing that the end-of-file behavior acts according to asynchronous rather than synchronous rules.

    Any advice you have on how we can pursue this model would be appreciated. Another concern is that since we do not set the hEvent in the OVERLAPPED structure, the file handle itself is used as the signal that I/O has completed, and this will cause problems if multiple I/O's are active simultaneously.

    The problem is that the customer confused the two senses of synchronous, one when applied to files and one when applied to I/O operations. Since they opened a synchronous file handle, all I/O operations are serialized and execute synchronously. Passing an OVERLAPPED structure issues an asynchronous I/O, but since the underlying handle is synchronous, the I/O is serialized and synchronous. The customer's code therefore is not actually performing I/O asynchronously; its requests for asynchronous I/O is overridden by the fact that the underlying handle is synchronous.

    The hybrid model doesn't actually realize any gains of asynchronous I/O. The use of the OVERLAPPED structure merely provides the convenience of combining the seek and read operations into a single call. Since the benefit is rather meager, the hybrid model is not commonly used, and consequently it is not covered in depth in the documentation. (The facts are still there, but there is relatively little discussion and elaboration.)

    Based on this feedback, the customer considered switching to using an asynchronous file handle and setting the hEvent in the OVERLAPPED structure so that each thread can wait for its specific I/O to complete. In the end, however, they decided to stick with the hybrid model because switching to an asynchronous handle was too disruptive to their code base. They are satisfied with the OVERLAPPED technique that lets them perform the equivalent of an atomic Set­File­Pointer + Read­File (even if the I/O is synchronous and serialized).

  • The Old New Thing

    Why does the copy dialog give me the incorrect total size of the files being copied?

    • 31 Comments

    If you try to copy a bunch of files to a drive that doesn't have enough available space, you get an error message like this:

    1 Interrupted Action

    There is not enough space on Removable Disk (D:). You need an additional 1.50 GB to copy these files.

    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB
    Try again Cancel

    "But wait," you say. "I'm only copying 5GB of data. Why does it say Total size: 14.9 GB?"

    This is a case of information being presented out of context and resulting in mass confusion.

    Suppose you saw the information like this:

    Computer
    ◢ Hard Disk Drives (1)   
     
    ▭  Windows (C:)
    Space free: 31.5 GB
    Total size: 118 GB
    ◢ Drives with Removable Storage (1)   
     
    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB

    In this presentation, it is clear that Total size refers to the total size of the drive itself.

    So the original dialog is not saying that the total size of data being copied is 14.49 GB. It's trying to say that the total size of the removable disk is 14.9 GB.

    Mind you, the presentation is very confusing since the information about the removable disk is presented without any introductory text. It's just plopped there on the dialog without so much as a hello.

    I'm not sure how I would fix this. Maybe reordering the text elements would help.

    1 Interrupted Action

    There is not enough space on Removable Disk (D:).

    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB

    You need an additional 1.50 GB to copy these files.

    Try again Cancel

    However, the design of the dialog may not allow the information tile to be inserted into the middle of the paragraph. It might be restricted to a layout where you can have text, followed by an information tile, followed by buttons. In that case, maybe it could go

    1 Interrupted Action

    You need an additional 1.50 GB to copy these files. There is not enough space on Removable Disk (D:).

    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB
    Try again Cancel

    But like I said, I'm not sure about this.

Page 1 of 441 (4,410 items) 12345»