• The Old New Thing

    Tip for trying to boost morale: Don't brag about your overseas trips

    • 13 Comments

    Once upon a time, a senior manager held a team meeting to address low team morale. Attempting to highlight how important the project was, he opened by saying, "I just got back from ⟨faraway country⟩, meeting with some of our important clients, and..."

    This remark had exactly the opposite effect from what the manager intended. Instead of revitalizing the team, the team became even more despondent. "Here we are, working late and coming in on weekends, and this senior manager is telling us about his recent overseas junket."

    (After he left, I heard that he still pulled this stunt with his new team. "Over the past two weeks, I've been all over ⟨continent⟩...")

  • The Old New Thing

    Customizing item enumeration with IShellItem

    • 4 Comments

    If you are using the original IShell­Folder interface, then you can use SHCONTF values to customize how child items are enumerated. But what if you're using IShell­Item?

    Let's take it one step at a time. First, the basic program. (Remember, Little Programs do little to no error checking.)

    #define UNICODE
    #define _UNICODE
    #define STRICT
    #define STRICT_TYPED_ITEMIDS
    #include <windows.h>
    #include <shlobj.h>
    #include <atlbase.h>
    #include <atlalloc.h>
    #include <propsys.h>
    #include <stdio.h>
    
    int __cdecl wmain(int argc, wchar_t **argv)
    {
     CCoInitialize init;
    
     if (argc < 2) return 0;
     CComPtr<IShellItem> spsiFolder;
     SHCreateItemFromParsingName(argv[1], nullptr,
                                 IID_PPV_ARGS(&spsiFolder));
    
     CComPtr<IEnumShellItems> spesi;
     spsiFolder->BindToHandler(nullptr, BHID_EnumItems,
                                  IID_PPV_ARGS(&spesi));
    
     for (CComPtr<IShellItem> spsi;
          spesi->Next(1, &spsi, nullptr) == S_OK;
          spsi.Release()) {
      PrintDisplayName(spsi, SIGDN_NORMALDISPLAY, L"Display Name");
      PrintDisplayName(spsi, SIGDN_FILESYSPATH, L"File system path");
      wprintf(L"\n");
     }
     return 0;
    }
    

    Run this program with the fully-qualified path to a directory as the command line argument, and it enumerates all the items in the folder. This uses the default enumeration, which is "include both folders and non-folders, and include hidden items, but not super-hidden items."

    But what if we want to customize the enumeration?

    We saw that the IBindCtx parameter acts as a catch-all options parameter. In this case, we need to look at the options available for BHID_Enum­Items and see if any of them help us.

    Fortunately, we have STR_ENUM_ITEMS_FLAGS which lets us override the default enumeration mode. Let's say that we want only folders, and we want to respect the user's preferences for hidden items (which means that we omit SHCONTF_HIDDEN).

    I'm goint to do this two ways. First the flat version:

     ...
     CComPtr<IBindCtx> spbcEnum;
     CreateDwordBindCtx(STR_ENUM_ITEMS_FLAGS, SHCONTF_FOLDERS,
                        &spbcEnum);
    
     CComPtr<IEnumShellItems> spesi;
     spsiFolder->BindToHandler(spbcEnum, BHID_EnumItems,
                                  IID_PPV_ARGS(&spesi));
    

    Now the fluent version:

     ...
     CBindCtxBuilder builder;
     builder.SetVariantDword(STR_ENUM_ITEMS_FLAGS, SHCONTF_FOLDERS);
    
     CComPtr<IEnumShellItems> spesi;
     spsiFolder->BindToHandler(builder.GetBindCtx(), BHID_EnumItems,
                                  IID_PPV_ARGS(&spesi));
    

    (Don't forget that error checking has been elided for expository purposes.)

    The STR_ENUM_ITEMS_FLAGS bind context string was added in Windows 8, so it has no effect on older versions of Windows. We'll address this next week.

    Note that the IEnum­Shell­Items interface is incorrectly-named. The convention for enumeration interfaces is to name them IEnum­XXX, where XXX is singular.

  • The Old New Thing

    Helper functions to make shell bind contexts slightly more manageable

    • 7 Comments

    Last time, we learned about the wonderful world of shell bind context strings, and I promised some helper functions to make this slightly more manageable.

    Here are some helper functions which supplement the Create­Bind­Ctx­With­Opts function we created some time ago.

    #include <propsys.h>
    
    HRESULT EnsureBindCtxPropertyBag(
        IBindCtx *pbc, REFIID riid, void **ppv)
    {
     *ppv = nullptr;
     CComPtr<IUnknown> spunk;
     HRESULT hr = pbc->GetObjectParam(STR_PROPERTYBAG_PARAM, &spunk);
     if (FAILED(hr)) {
      hr = PSCreateMemoryPropertyStore(IID_PPV_ARGS(&spunk));
      if (SUCCEEDED(hr)) {
       hr = pbc->RegisterObjectParam(STR_PROPERTYBAG_PARAM, spunk);
      }
     }
     if (SUCCEEDED(hr)) {
      hr = spunk->QueryInterface(riid, ppv);
     }
      return hr;
    }
    
    HRESULT AddBindCtxDWORD(
        IBindCtx *pbc, LPCWSTR pszName, DWORD dwValue)
    {
     CComPtr<IPropertyBag> sppb;
     HRESULT hr = EnsureBindCtxPropertyBag(pbc, IID_PPV_ARGS(&sppb));
     if (SUCCEEDED(hr)) {
      hr = PSPropertyBag_WriteDWORD(sppb, pszName, dwValue);
     }
     return hr;
    }
    
    HRESULT AddBindCtxString(
        IBindCtx *pbc, LPCWSTR pszName, LPCWSTR pszValue)
    {
     CComPtr<IPropertyBag> sppb;
     HRESULT hr = EnsureBindCtxPropertyBag(pbc, IID_PPV_ARGS(&sppb));
     if (SUCCEEDED(hr)) {
      hr = PSPropertyBag_WriteStr(sppb, pszName, pszValue);
     }
     return hr;
    }
    
    HRESULT CreateDwordBindCtx(
        LPCWSTR pszName, DWORD dwValue, IBindCtx **ppbc)
    {
     CComPtr<IBindCtx> spbc;
     HRESULT hr = CreateBindCtx(0, &spbc);
     if (SUCCEEDED(hr)) {
      hr = AddBindCtxDWORD(spbc, pszName, dwValue);
     }
     *ppbc = SUCCEEDED(hr) ? spbc.Detach() : nullptr;
     return hr;
    }
    
    HRESULT CreateStringBindCtx(
        LPCWSTR pszName, LPCWSTR pszValue, IBindCtx **ppbc)
    {
     CComPtr<IBindCtx> spbc;
     HRESULT hr = CreateBindCtx(0, &spbc);
     if (SUCCEEDED(hr)) {
      hr = AddBindCtxString(spbc, pszName, pszValue);
     }
     *ppbc = SUCCEEDED(hr) ? spbc.Detach() : nullptr;
     return hr;
    }
    

    The Ensure­Bind­Ctx­Property­Bag function puts a property bag in the bind context if there isn't one already.

    The Add­Bind­Ctx­DWORD function adds a DWORD to that associated property bag. If you wanted to add multiple DWORDs to a bind context, you would call this function multiple times. You can also use the Add­Bind­Ctx­String if the thing you want to add is a string.

    The Create­Dword­Bind­Ctx function handles the simple case where you want to create a bind context that contains a single DWORD. Similarly, Create­String­Bind­Ctx.

    But now things are starting to get kind of unwieldy. What if you want a bind context with a string and a DWORD? Let's go for something a bit more fluent.

    But first, some scaffolding.

    class CStaticUnknown : public IUnknown
    {
    public:
     // *** IUnknown ***
     IFACEMETHODIMP QueryInterface(
      _In_ REFIID riid, _Outptr_ void **ppv)
     {
      *ppv = nullptr;
      HRESULT hr = E_NOINTERFACE;
      if (riid == IID_IUnknown) {
       *ppv = static_cast<IUnknown *>(this);
       AddRef();
       hr = S_OK;
      }
      return hr;
     }
    
     IFACEMETHODIMP_(ULONG) AddRef()
     {
      return 2;
     }
    
     IFACEMETHODIMP_(ULONG) Release()
     {
      return 1;
     }
    
    };
    
    CStaticUnknown s_unkStatic;
    

    This static implementation of IUnknown is one we'll use for the bind context strings whose mere presence indicates that a flag is set.

    class CBindCtxBuilder
    {
    public:
     CBindCtxBuilder()
     {
      m_hrCumulative = CreateBindCtx(0, &m_spbc);
     }
    
     CBindCtxBuilder& SetMode(DWORD grfMode);
     CBindCtxBuilder& SetFindData(const WIN32_FIND_DATA *pfd);
     CBindCtxBuilder& SetFlag(PCWSTR pszName);
     CBindCtxBuilder& SetVariantDword(PCWSTR pszName, DWORD dwValue);
     CBindCtxBuilder& SetVariantString(PCWSTR pszName, PCWSTR pszValue);
    
     HRESULT Result() const { return m_hrCumulative; }
    
     IBindCtx *GetBindCtx() const
     { return SUCCEEDED(m_hrCumulative) ? m_spbc : nullptr; }
    private:
     HRESULT EnsurePropertyBag();
    
    private:
     CComPtr<IBindCtx> m_spbc;
     CComPtr<IPropertyBag> m_sppb;
     HRESULT m_hrCumulative;
    };
    

    The bind context builder class is a helper class that creates a bind context, and then fills it with stuff. For now, we let you set the following:

    After you build up the bind context, you can check the Result() to see if it was built successfully, and use Get­Bind­Ctx to extract the result.

    Here's the implementation. It's really not that exciting. We accumulate any error in m_hrCumulative, and once an error occurs, all future methods do nothing aside from preserving the error. To make the object fluent, the methods return a reference to themselves.

    There is a special bind context method for setting the mode:

    CBindCtxBuilder&
    CBindCtxBuilder::SetMode(DWORD grfMode)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      BIND_OPTS bo = { sizeof(bo), 0, grfMode, 0 };
      m_hrCumulative = m_spbc->SetBindOptions(&bo);
     }
     return *this;
    }
    

    Find data is set as a direct object on the bind context, as we saw some time ago:

    CBindCtxBuilder&
    CBindCtxBuilder::SetFindData(const WIN32_FIND_DATA *pfd)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = AddFileSysBindCtx(m_spbc, pfd);
     }
     return *this;
    }
    

    Flags are set by there mere presence, so we associate them with a dummy IUnknown that does nothing:

    CBindCtxBuilder&
    CBindCtxBuilder::SetFlag(PCWSTR pszName)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = m_spbc->RegisterObjectParam(
        const_cast<PWSTR>(pszName), &s_unkStatic);
     }
     return *this;
    }
    

    If a property is set in the property bag, we need to proceed in two steps. First, we create the property bag if we don't have one already. Second, we put the value into the property bag:

    CBindCtxBuilder&
    CBindCtxBuilder::SetVariantDword(
        PCWSTR pszName, DWORD dwValue)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = EnsurePropertyBag();
     }
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative =  PSPropertyBag_WriteDWORD(
        m_sppb, pszName, dwValue);
     }
     return *this;
    }
    
    CBindCtxBuilder&
    CBindCtxBuilder::SetVariantString(
        PCWSTR pszName, PCWSTR pszValue)
    {
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative = EnsurePropertyBag();
     }
     if (SUCCEEDED(m_hrCumulative)) {
      m_hrCumulative =  PSPropertyBag_WriteStr(
        m_sppb, pszName, pszValue);
     }
     return *this;
    }
    

    And finally, the helper function that creates a property bag if we don't have one already.

    HRESULT CBindCtxBuilder::EnsurePropertyBag()
    {
     HRESULT hr = S_OK;
     if (!m_sppb) {
      hr = PSCreateMemoryPropertyStore(
        IID_PPV_ARGS(&m_sppb));
      if (SUCCEEDED(hr)) {
       hr = m_spbc->RegisterObjectParam(
        STR_PROPERTYBAG_PARAM, m_sppb);
      }
     }
     return hr;
    }
    

    The idea here is that the class is used like this:

    CBindCtxBuilder builder;
    builder.SetMode(STGM_CREATE)
           .SetFindData(&wfd)
           .SetFlag(STR_FILE_SYS_BIND_DATA_WIN7_FORMAT)
           .SetFlag(STR_BIND_FOLDERS_READ_ONLY);
    hr = builder.Result();
    if (SUCCEEDED(hr)) {
     hr = psf->ParseDisplayName(hwnd, builder.GetBindCtx(),
       pszName, &cchEaten, &pidl, &dwAttributes);
    }
    

    You create the bind context builder, then use the various Set­Xxx methods to fill the bind context with goodies, and then you check if it all worked okay. If so, then you use Get­Bind­Ctx to get the resulting bind context and proceed on your way.

  • The Old New Thing

    The wonderful world of shell bind context strings

    • 6 Comments

    Some time ago, we saw how the IBindCtx parameter to IShell­Folder::Parse­Display­Name can be used to modify how the parse takes place. More generally, the IBindCtx parameter to a function is a catch-all miscellaneous options parameter.

    The interesting part of the bind context is all the stuff that has been added to it via the IBindCtx::Register­Object­Param method. You can attach arbitrary objects to the bind context, using a string to identify each one.

    Some bind context parameters are like Boolean parameters that simply change some default behavior of the operation. For these operations, the object that is associated with the bind context string is not important; the important thing is that there is something associated with it. Traditionally, you just connect a dummy object that implements just IUnknown.

    In the most general case, the object associated with a bind context string implements some operation-specific interface. For example, the STR_BIND_DELEGATE_CREATE_OBJECT bind context string expects you to associate an object that implements the ICreate­Object interface, because the whole point of the STR_BIND_DELEGATE_CREATE_OBJECT bind context string is to say, "Hey, I want to create objects in a nonstandard way," so you need to tell it what that nonstandard way is.

    At the other extreme, you may have a chunk of data that you want to associate with the bind context string. Since bind contexts want to associate objects, you need to wrap the data inside a COM object. We saw this earlier when we had to create an object that implemented the IFile­System­Bind­Data interface in order to babysit a WIN32_FIND_DATA structure.

    Rather than having to create a separate interface for each data type (hello, IObject­With­Folder­Enum­Mode), and rather than going to the opposite extreme and just using IStream to pass arbitrary unstructured data, the shell folks decided to take a middle-ground approach: Use a common interface that still has a modicum of type safety, namely, IProperty­Bag. Other nice things about this approach is that there are a lot of pre-existing helper functions for property bags and property variants. Also, you need to attach only one object instead of a whole bunch of tiny little ones.

    Under this new regime (which took hold in Windows 8), the bind context has an associated property bag, and you put your data in that property bag.

    In pictures:

      IBindCtx::Register­Object­Param IProperty­Bag::Write
    IBindCtx Boolean parameter IUnknown
    Interface parameter object with custom interface
    STR_PROPERTY­BAG_PARAM IPropertyBag Property bag DWORD parameter VT_UI4
    Property bag string parameter VT_BSTR
    Property bag Boolean parameter VT_BOOL

    If you want a Boolean-style parameter to be true, then set it in the bind context with a dummy object that implements IUnknown. If you want a Boolean-style parameter to false, then omit it from the bind context entirely.

    To set an interface-style parameter, set it in the bind context with an object that implements the desired interface.

    To set a property bag-based parameter, set it in the property bag with the appropriate variant type.

    Here are the bind context strings defined up through Windows 8.1 and the way you set them into the bind context.

    Bind context string Model Operation
    STR_AVOID_DRIVE_RESTRICTION_POLICY Boolean Binding
    STR_BIND_DELEGATE_CREATE_OBJECT Interface ICreateObject Binding
    STR_BIND_FOLDER_ENUM_MODE Interface IObjectWith­FolderEnumMode Parsing
    STR_BIND_FOLDERS_READ_ONLY Boolean Parsing
    STR_BIND_FORCE_FOLDER_SHORTCUT_RESOLVE Boolean Binding
    STR_DONT_PARSE_RELATIVE Boolean Parsing
    STR_DONT_RESOLVE_LINK Boolean Binding
    STR_ENUM_ITEMS_FLAGS Property bag: VT_UI4 Binding for enumeration
    STR_FILE_SYS_FIND_DATA Interface IFileSys­BindData or IFileSys­BindData2 Parsing
    STR_FILE_SYS_BIND_DATA_WIN7_FORMAT Boolean Parsing
    STR_GET_ASYNC_HANDLER Boolean GetUIObjectOf
    STR_GPS_BEST­EFFORT Boolean Binding for IProperty­Store
    STR_GPS_DELAY­CREATION Boolean Binding for IProperty­Store
    STR_GPS_FAST­PROPERTIES­ONLY Boolean Binding for IProperty­Store
    STR_GPS_HANDLER­PROPERTIES­ONLY Boolean Binding for IProperty­Store
    STR_GPS_NO_OPLOCK Boolean Binding for IProperty­Store
    STR_GPS_OPEN­SLOW­ITEM Boolean Binding for IProperty­Store
    STR_IFILTER_FORCE_TEXT_FILTER_FALLBACK Boolean Binding for IFilter
    STR_IFILTER_LOAD_DEFINED_FILTER Boolean Binding for IFilter
    STR_INTERNAL_NAVIGATE Boolean Loading history
    STR_INTERNET­FOLDER_PARSE_ONLY_URLMON_BINDABLE Boolean Parsing
    STR_ITEM_CACHE_CONTEXT Interface IBindCtx Parsing and initiailzing
    STR_NO_VALIDATE_FILE­NAME_CHARS Boolean Parsing
    STR_PARSE_ALLOW_INTERNET_SHELL_FOLDERS Boolean Parsing
    STR_PARSE_AND_CREATE_ITEM Interface IParse­And­Create­Item Parsing
    STR_PARSE_DONT_REQUIRE_VALIDATED_URLS Boolean Parsing
    STR_PARSE_EXPLICIT_ASSOCIATION_SUCCESSFUL Property bag: VT_BOOL Parsing
    STR_PARSE_PARTIAL_IDLIST Interface IShell­Item Parsing
    STR_PARSE_PREFER_FOLDER_BROWSING Boolean Parsing
    STR_PARSE_PREFER_WEB_BROWSING Boolean Parsing
    STR_PARSE_PROPERTY­STORE Interface IProperty­Bag Parsing
    STR_PARSE_SHELL_PROTOCOL_TO_FILE_OBJECTS Boolean Parsing
    STR_PARSE_SHOW_NET_DIAGNOSTICS_UI Boolean Parsing
    STR_PARSE_SKIP_NET_CACHE Boolean Parsing
    STR_PARSE_TRANSLATE_ALIASES Boolean Parsing
    STR_PARSE_WITH_EXPLICIT_ASSOCAPP Property bag: VT_BSTR Parsing
    STR_PARSE_WITH_EXPLICIT_PROGID Property bag: VT_BSTR Parsing
    STR_PARSE_WITH_PROPERTIES Interface IProperty­Store Parsing
    STR_PROPERTYBAG_PARAM Interface IProperty­Bag holds property bag parameters
    STR_SKIP_BINDING_CLSID Interface IPersist Parsing and binding

    There are some oddities in the above table.

    • All of the STR_GPS_* values would be more conveniently expressed as a single VT_UI4 property bag-based value. (Exercise: Why isn't it?)
    • The STR_ITEM_CACHE_CONTEXT bind context parameter is itself another bind context! The idea here is that you, the caller, are enabling caching during the parse, and the inner bind context acts as the cache.
    • The STR_PARSE_EXPLICIT_ASSOCIATION_SUCCESSFUL value is unusual in that it is something set by the parser and passed back to the caller.
    • As we have been discussing, STR_PROPERTY­BAG_PARAM is a bind context string that doesn't mean anything on its own. Rather, it provides a property bag into which more parameters can be stored.

    Next time, I'll write some helper functions to make all this slightly more manageable.

  • The Old New Thing

    Why does my synchronous overlapped ReadFile return FALSE when the end of the file is reached?

    • 16 Comments

    A customer reported that the behavior of Read­File was not what they were expecting.

    We have a synchronous file handle (not created with FILE_FLAG_OVERLAPPED), but we issue reads against it with an OVERLAPPED structure. We find that when we read past the end of the file, the Read­File returns FALSE even though the documentation says it should return TRUE.

    They were kind enough to include a simple program that demonstrates the problem.

    #include <windows.h>
    
    int __cdecl wmain(int, wchar_t **)
    {
     // Create a zero-length file. This succeeds.
     HANDLE h = CreateFileW(L"test", GENERIC_READ | GENERIC_WRITE,
                   0, nullptr, CREATE_ALWAYS,
                   FILE_ATTRIBUTE_NORMAL, nullptr);
    
     // Read past EOF.
     char buffer[10];
     DWORD cb;
     OVERLAPPED o = { 0 };
     ReadFile(h, buffer, 10, &cb, &o); // returns FALSE
     GetLastError(); // returns ERROR_HANDLE_EOF
    
     return 0;
    }
    

    The customer quoted this section from The documentation for Read­File:

    Considerations for working with synchronous file handles:

    • If lpOverlapped is NULL, the read operation starts at the current file position and Read­File does not return until the oepration is complete, and the system updates the file pointer before Read­File returns.
    • If lpOverlapped is not NULL, the read operation starts at the offset that is specified in the OVERLAPPED structure and Read­File does not return until the read operation is complete. The system updates the OVERLAPPED offset before Read­File returns.
    • When a synchronous read operation reads the end of a file, Read­File returns TRUE and sets *lpNumberOfBytesRead to zero.

    and then added

    According to the third bullet point, the Read­File should return TRUE, but in practice it returns FALSE and the error code is ERROR_HANDLE_EOF.

    The problem here is that there are two concepts here, and they confusingly both use the word synchronous.

    • A synchronous file handle is a handle opened without FILE_FLAG_OVERLAPPED. All I/O to a synchronous file handle is serialized and synchronous.
    • A synchronous I/O operation is an I/O issued with lpOverlapped == NULL.

    The sample program issues an asynchronous read against a synchronous handle. The third bullet point applies only to synchronous reads.

    To reduce confusion, the documentation would have been clearer if it hadn't switched terminology midstream.

    • If lpOverlapped is NULL, the read operation starts at the current file position and Read­File does not return until the oepration is complete, and the system updates the file pointer before Read­File returns.
    • If lpOverlapped is not NULL, the read operation starts at the offset that is specified in the OVERLAPPED structure and Read­File does not return until the read operation is complete. The system updates the OVERLAPPED offset before Read­File returns.
    • If lpOverlapped is NULL and the read operation reads the end of a file, Read­File returns TRUE and sets *lpNumberOfBytesRead to zero.

    We asked what the customer was doing that caused them to trip over this confusion in the documentation.

    The customer's original code opened a file (synchronously) and read from it (synchronously). The customer is parallelizing the computation in a way that will read that single file from multiple threads. A single file pointer is therefore not suitable, because different threads will want to read from different positions.

    One idea would be to have each thread call Create­File so that each handle has its own file position. Unfortunately, this won't work for the customer because the sharing mode on the file handle denies read sharing.

    The solution they came up with was to open the file synchronously (without FILE_FLAG_OVERLAPPED) but to read asynchronously (by using an OVERLAPPED structure). The OVERLAPPED structure lets you specify where you want to read from, so multiple threads can issue reads against the file position they want.

    This solution works, but the customer is concerned because this hybrid model is not well-documented in MSDN. They found a blog entry that discusses it, but even that blog entry does not discuss what happens in the multithreaded case.) In particular, they are seeing that the end-of-file behavior acts according to asynchronous rather than synchronous rules.

    Any advice you have on how we can pursue this model would be appreciated. Another concern is that since we do not set the hEvent in the OVERLAPPED structure, the file handle itself is used as the signal that I/O has completed, and this will cause problems if multiple I/O's are active simultaneously.

    The problem is that the customer confused the two senses of synchronous, one when applied to files and one when applied to I/O operations. Since they opened a synchronous file handle, all I/O operations are serialized and execute synchronously. Passing an OVERLAPPED structure issues an asynchronous I/O, but since the underlying handle is synchronous, the I/O is serialized and synchronous. The customer's code therefore is not actually performing I/O asynchronously; its requests for asynchronous I/O is overridden by the fact that the underlying handle is synchronous.

    The hybrid model doesn't actually realize any gains of asynchronous I/O. The use of the OVERLAPPED structure merely provides the convenience of combining the seek and read operations into a single call. Since the benefit is rather meager, the hybrid model is not commonly used, and consequently it is not covered in depth in the documentation. (The facts are still there, but there is relatively little discussion and elaboration.)

    Based on this feedback, the customer considered switching to using an asynchronous file handle and setting the hEvent in the OVERLAPPED structure so that each thread can wait for its specific I/O to complete. In the end, however, they decided to stick with the hybrid model because switching to an asynchronous handle was too disruptive to their code base. They are satisfied with the OVERLAPPED technique that lets them perform the equivalent of an atomic Set­File­Pointer + Read­File (even if the I/O is synchronous and serialized).

  • The Old New Thing

    Why does the copy dialog give me the incorrect total size of the files being copied?

    • 31 Comments

    If you try to copy a bunch of files to a drive that doesn't have enough available space, you get an error message like this:

    1 Interrupted Action

    There is not enough space on Removable Disk (D:). You need an additional 1.50 GB to copy these files.

    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB
    Try again Cancel

    "But wait," you say. "I'm only copying 5GB of data. Why does it say Total size: 14.9 GB?"

    This is a case of information being presented out of context and resulting in mass confusion.

    Suppose you saw the information like this:

    Computer
    ◢ Hard Disk Drives (1)   
     
    ▭  Windows (C:)
    Space free: 31.5 GB
    Total size: 118 GB
    ◢ Drives with Removable Storage (1)   
     
    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB

    In this presentation, it is clear that Total size refers to the total size of the drive itself.

    So the original dialog is not saying that the total size of data being copied is 14.49 GB. It's trying to say that the total size of the removable disk is 14.9 GB.

    Mind you, the presentation is very confusing since the information about the removable disk is presented without any introductory text. It's just plopped there on the dialog without so much as a hello.

    I'm not sure how I would fix this. Maybe reordering the text elements would help.

    1 Interrupted Action

    There is not enough space on Removable Disk (D:).

    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB

    You need an additional 1.50 GB to copy these files.

    Try again Cancel

    However, the design of the dialog may not allow the information tile to be inserted into the middle of the paragraph. It might be restricted to a layout where you can have text, followed by an information tile, followed by buttons. In that case, maybe it could go

    1 Interrupted Action

    You need an additional 1.50 GB to copy these files. There is not enough space on Removable Disk (D:).

    ▭  Removable Disk (D:)
    Space free: 2.50 GB
    Total size: 14.9 GB
    Try again Cancel

    But like I said, I'm not sure about this.

  • The Old New Thing

    Why does Outlook use a semicolon to separate multiple recipients by default?

    • 35 Comments

    Microsoft Outlook by default uses a semicolon to separate multiple recipients. You can change this to a comma, but why is the semicolon the default?

    Microsoft Outlook was originally positioned as a business product, and many businesses complained that the use of a comma as a separator created havoc because they have a policy of setting names in the address book as "Last, First".

    In 2000, the Outlook folks tried to change the default, but the outcry from corporations made them go back to having the semicolon be the default separator.

    Besides, there are a lot of people who have commas in their names, such as Martin Luther King, Jr.

  • The Old New Thing

    Why does a single integer assignment statement consume all of my CPU?

    • 21 Comments

    Here's a C++ class inspired by actual events. (Yes, the certificate on that Web site is broken.) It is somebody's attempt to create a generic value type, similar to VARIANT.

    class Value
    {
    public:
     Value(Type type) : m_type(V_UNDEFINED) { }
    
     Type GetType() const { return m_type; }
     void SetType(Type type) { m_type = type; }
    
     int32_t GetInt32() const
     {
      assert(GetType() == V_INT32);
      return *reinterpret_cast<const int32_t *>(m_data);
     }
    
     void SetInt32(int32_t value)
     {
      assert(GetType() == V_INT32);
      *reinterpret_cast<int32_t *>(m_data) = value;
     }
    
     // GetChar, SetChar, GetInt64, SetInt64, etc.
    
    private:
     char m_data[sizeof(int64_t)];
     char m_type;
    };
    
    ...
    
    Value CalculateTheValue()
    {
     int32_t total;
     // ... a bunch of computation ...
    
     Value result;
     result.SetType(V_INT32);
     result.SetInt32(total);
     return result;
    }
    

    Profiling showed that over 80% of the time spent by Calculate­The­Value was inside the Set­Int32 method call, in particular on the line

      *reinterpret_cast<int32_t *>(m_data) = value;
    

    Why does it take so much time to store an integer to memory, dwarfing the actual computation to calculate that integer?

    Alignment.

    Observe that the underlying data for the Value class is declared as a bunch of chars. Since a char is just a byte, it has no alignment restrictions. On the other hand, data types like int32_t typically do have alignment restrictions. For example, accessing a 32-bit value is usually more efficient if the value is stored in memory starting at a multiple of 4.

    How much more efficient depends on the processor and the data type.

    Of the processors that allow unaligned memory access, the penalty can be zero, or only 10% or maybe 100%.

    Many processor architectures are less forgiving of misaligned data access and raise an alignment exception if you break the rules. When such an exception occurs, the operating system might choose to terminate the application. Or the operating system may choose to emulate the instruction and fix up the misaligned access. The program runs much slower, but at least it still runs. (In Windows, the decision how to respond to the alignment exception depends on whether the process asked for alignment faults to be forgiven. See SEM_NO­ALIGNMENT­FAULT­EXCEPT.)

    It appears that the original program is in the last case: An alignment exception occurred, and the operating system handled it by manually reading the four bytes from m_data[0] through m_data[4] and assembling them into a 32-bit value, then resuming execution of the original program.

    Dispatching the exception, parsing out the faulting instruction, emulating it, then resuming execution. That is all very slow. Probably several thousand instruction cycles. This can easily dwarf the actual computation performed by Calculate­The­Value.

    Okay, but why is the result variable unaligned?

    Since, as we noted a while back, the way the Value class is defined requires only byte alignment, the compiler is not constrained to align it in any particular way. If there were a int16_t local variable in the Calculate­The­Value function, the compiler might choose to arrange its stack frame like this:

    • Start at an aligned address X.
    • Put int32_t total at X+0 through X+3.
    • Put int16_t whatever at X+4 through X+5.
    • Put Value result at X+6 through X+22.

    Since X is a multiple of 4, X+6 is not a multiple of 4, so the m_data member is misaligned and incurs an alignment fault at every access.

    What's more, since the Value class has an odd number of total bytes, if you create an array of Values, you are guaranteed that three quarters of the elements will be misaligned.

    The solution is to fix the declaration of the Value class so that the alignment requirements are made visible to the compiler. Instead of jamming all the data into a byte blob, use a discriminated union. That is, after all, what you are trying to emulate in the first place.

    class Value
    {
    public:
     Value(Type type) : m_type(V_UNDEFINED) { }
    
     Type GetType() const { return m_type; }
     void SetType(Type type) { m_type = type; }
    
     int32_t GetInt32() const
     {
      assert(GetType() == V_INT32);
      return m_data.m_int32;
     }
    
     void SetInt32(int32_t value)
     {
      assert(GetType() == V_INT32);
      m_data.m_int32 = value;
     }
    
     // GetChar, SetChar, GetInt64, SetInt64, etc.
    
    private:
     union
     {
      char    m_char;
      int32_t m_int32;
      int64_t m_int64;
      // etc.
     } m_data;
     char m_type;
    };
    

    Exercise: One guess as to the cause of the problem is that the assignment statement is incurring paging. Explain why this is almost certainly not the reason.

    Bonus chatter: I'm ignoring RVO here. If you are smart enough to understand RVO, you should also be smart enough to see that RVO does not affect the underlying analysis. It just shifts the address calculation to the caller.

  • The Old New Thing

    Why does CreateFile take a long time on a volume handle?

    • 32 Comments

    A customer reported that on Windows XP, a call to Create­File was taking a really, really long time if it was performed immediately after a large file copy. They were kind enough to include a demonstration program:

    #include <windows.h>
    
    int main(int argc, char **argv)
    {
     HANDLE h = CreateFile("\\\\.\\D:",
                           GENERIC_READ | GENERIC_WRITE,
                           FILE_SHARE_WRITE | FILE_SHARE_READ,
                           NULL,
                           OPEN_EXISTING,
                           FILE_ATTRIBUTE_NORMAL,
                           NULL);
     Sleep(5000);
     return 0;
    }
    

    If this program is run on its own, the Create­File completes quickly. But if you copy 1.7GB of data immediately before running the program, then Create­File takes longer. The customer would like to know the reason for this issue and whether there is a way to avoid it.

    The reason is that you just copied a lot of data, so there is a lot of dirty data in the disk cache that is waiting to get flushed out. And when you create the volume handle, Windows needs to flush out all that data so that the volume handle sees a consistent view of the volume. Flushing out 1.7GB of data can take a while.

    There is no way to avoid this problem because the speed of data transfer to the drive is limited by the drive hardware. It will take N seconds to transfer 1.7GB of data, so the time between the start of the file copy operation and the successful opening of the volume handle will be N seconds. If you want the Create­File to go faster, you could do a Flush­File­Buffers on the file being copied so that the cost of writing the data gets charged to the copy operation rather than the Create­File, but that's just creative accounting. You didn't actually make any money; you just moved it around.

    Now, a lot of programs open a volume handle but don't actually read from it or write to it, such as the sample program above. Therefore, newer versions of Windows (I don't know exactly whether it was Windows Vista or Windows 7) defer the flush until somebody actually tries to use the handle for reading or writing. So at least for the sample program above, the Create­File will complete quickly. However, the first read or write operation will be slow.

    Again, the total time doesn't change. All that changes is where the cost of the flush is incurred.

  • The Old New Thing

    Where can I find the standard asynchronous stream?

    • 8 Comments

    In the documentation for XmlLite, one of the features called out is that XmlLite is a non-blocking parser. If the input stream returns E_PENDING, then XmlLite propagates that status to its caller, and a subsequent request to XmlLite to parse will resume where it left off.

    That documentation calls out two scenarios in which this can happen, the second of which is

    2. The input Stream is a standard asynchronous stream. The E_PENDING HRESULT may be raised when the data is temporarily unavailable on the network. In this case, you need to try again later in a callback or after some interval of time.

    A customer was kind of confused by this explanation. "Where do I get a standard asynchronous stream so I can use it in scenario 2?"

    The documentation here is trying to be helpful by expanding on the original statement that XmlLite is a non-blocking parser and providing examples of how you can take advantage of this non-blocking behavior. The normative statement is the one that says, "XmlLite propagates the E_PENDING from the input stream to its caller, and a subsequent request to read data from the XmlLite parser will resume where it left off." The rest is informational, but it seems that the informational text was more confusing than helpful.

    The informational text is trying to say, "Here are some examples where you can take advantage of this behavior." The first scenario is an example where you provided an IStream that returns E_PENDING when it wants to force the XmlLite parser to stop parsing. You might do this, for example, if you have out-of-band data in your XML stream. The stream would return E_PENDING when it encounters the out-of-band data, and this causes the XmlLite parser to stop parsing and return E_PENDING. You can then process the out-of-band data, and then when you are ready to resume parsing, you reissue the call that returned E_PENDING so the parser can resume where it left off.

    The second scenario is an example where you provided an IStream that returns E_PENDING to indicate that there is more data in the stream, but it is not available right now. For example, the stream may be the result of a streaming download, and the next chunk of the download hasn't arrived yet. Instead of blocking the read, the stream returns E_PENDING to say, "There is more data, but I can't provide it right now. Go do something else for a while." The download stream presumably has some way of notifying when the next download chunk is ready. Your program can subscribe to that notification, and when it is received, you can resume parsing with XmlLite.

    The adjective "standard" here in the phrase "a standard asynchronous stream" does not refer to a specific reference implementation. It's using the word "standard" in the sense of "regularly and widely used, seen, or accepted; not unusual or special." (This was subtly implied by the use of the indefinite rather than the definite article, but that use of the indefinite could be interpreted to mean "an instance of the standard asynchronous stream".) In other words, the opening sentence is saying, "The input Stream is any asynchronous stream that behaves in the usual manner."

    By analogy, consider the sentence "This service is available from a standard touch-tone phone." This doesn't mean "There is a specific model of touch-tone phone that is the standard touch-tone phone, and you must use that one." It's just saying "Any touch-tone phone (that conforms to the standard) will work."

Page 1 of 441 (4,406 items) 12345»