• The Old New Thing

    Further adventures in trying to guess what encoding a file is in

    • 8 Comments

    The Is­Text­Unicode function tries to guess the encoding of a block of memory purporting to contain text, but it can only say "Looks like Unicode" or "Doesn't look like Unicode", and there some notorious examples of where it guesses wrong.

    A more flexible alternative is IMulti­Language2::Detect­Code­page­In­IStream and its buffer-based equivalent IMulti­Language2::Detect­Input­Code­page. Not only can these methods detect a much larger range of code pages, they also can report multiple code pages, each with a corresponding confidence level.

    Here's a Little Program that takes the function out for a spin. (Remember, Little Programs do little to no error checking.)

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <shlwapi.h>
    #include <ole2.h>
    #include <mlang.h>
    #include <shlwapi.h>
    #include <atlbase.h>
    #include <stdio.h>
    
    bool IsHtmlFile(PCWSTR pszFile)
    {
     PCWSTR pszExtension = PathFindExtensionW(pszFile);
     return
      CompareStringOrdinal(pszExtension, -1,
                           L".htm", -1, TRUE) == CSTR_EQUAL ||
      CompareStringOrdinal(pszExtension, -1,
                            L".html", -1, TRUE) == CSTR_EQUAL;
    }
    
    int __cdecl wmain(int argc, wchar_t **argv)
    {
     if (argc < 2) return 0;
     CCoInitialize init;
     CComPtr<IStream> spstm;
     SHCreateStreamOnFileEx(argv[1], STGM_READ, 0, FALSE, nullptr, &spstm);
    
     CComPtr<IMultiLanguage2> spml;
     CoCreateInstance(CLSID_CMultiLanguage, NULL,
         CLSCTX_ALL, IID_PPV_ARGS(&spml));
    
     DetectEncodingInfo info[10];
     INT cInfo = ARRAYSIZE(info);
    
     DWORD dwFlag = IsHtmlFile(argv[1]) ? MLDETECTCP_HTML
                                        : MLDETECTCP_NONE;
     HRESULT hr = spml->DetectCodepageInIStream(
         dwFlag, 0, spstm, info, &cInfo);
     if (hr == S_OK) {
      for (int i = 0; i < cInfo; i++) {
       wprintf(L"info[%d].nLangID = %d\n", i, info[i].nLangID);
       wprintf(L"info[%d].nCodePage = %d\n", i, info[i].nCodePage);
       wprintf(L"info[%d].nDocPercent = %d\n", i, info[i].nDocPercent);
       wprintf(L"info[%d].nConfidence = %d\n", i, info[i].nConfidence);
      }
     } else {
      wprintf(L"Cannot determine the encoding (error: 0x%08x)\n", hr);
     }
     return 0;
    }
    

    Run the program with a file name as the command line argument, and the program will report all the detected code pages.

    One thing that may not be obvious is that the program passes the MLDETECTCP_HTML flag if the file extension is .htm or .html. That is a hint to the detector that it shouldn't get faked out by text like <body> and think it found an English word.

    Here's the output of the program when run on its own source code:

    info[0].nLangID = 9
    info[0].nCodePage = 20127
    info[0].nDocPercent = 100
    info[0].nConfidence = 83
    info[1].nLangID = -1
    info[1].nCodePage = 65001
    info[1].nDocPercent = -1
    info[1].nConfidence = -1
    

    This says that its first guess is that the text is in language 9, which is LANG_ENGLISH, code page 20127, which is US-ASCII, That text occupies 100% of the file, and the confidence level is 83.

    The second guess is that the text is in code page 65001, which is UTF-8, but the confidence level for that is low.

    The language-guessing part of the function is not very sophisticated. For a higher-quality algorithm for guessing what language some text is in, use Extended Linguistic Services. I won't bother writing a sample application because MSDN already contains one.

  • The Old New Thing

    Non-capturing C++ lambdas can be converted to a pointer to function, but what about the calling convention?

    • 24 Comments

    First, let's look at how lambdas are implemented in C++.

    It is similar in flavor to the way lambdas are implemented in C#, but the details are all different.

    When the C++ compiler encounters a lambda expression, it generates a new anonymous class. Each captured variable becomes a member of that anonymous class, and the member is initialized from the variable in the outer scope. Finally, the anonymous class is given an operator() implementation whose parameter list is the parameter list of the lambda, whose body is the lambda body, and whose return value is the lambda return value.

    I am simplifying here. You can read the C++ language specification for gory details. The purpose of this discussion is just to give a conceptual model for how lambdas work so we can get to answering the question. The language also provides for syntactic sugar to infer the lambda return type and capture variables implicitly. Let's assume all the sugar has been applied so that everything is explicit.

    Here's a basic example:

    void ContainingClass::SomeMethod()
    {
     int i = 0, j = 1;
     auto f = [this, i, &j](int k) -> int
        { return this->calc(i + j + k); };
     ...
    }
    

    The compiler internally converts this to something like this:

    void ContainingClass::SomeMethod()
    {
     int i = 0, j = 1;
    
     // Autogenerated by the compiler
     class AnonymousClass$0
     {
     public:
      AnonymousClass$0(ContainingClass* this$, int i$, int& j$) :
       this$0(this$), i$0(i$), j$0(j$) { }
      int operator(int k) const
         { return this$0->calc(i$0 + j$0 + k); }
     private:
      ContainingClass* this$0; // this captured by value
      int i$0;                 // i captured by value
      int& j$0;                // j captured by reference
     };
    
     auto f = AnonymousClass$0(this, i, j);
     ...
    }
    

    We are closer to answering the original question. but we're not there yet.

    As a special bonus: If there are no captured variables, then there is an additional conversion operator that can convert the lambda to a pointer to a nonmember function. This is possible only in the case of no captured variables because captured variables would require an AnonymousClass$0 instance parameter, but there is nowhere to pass it.

    Here's a lambda with no captured variables.

    void ContainingClass::SomeMethod()
    {
     auto f = [](int k) -> int { return calc(k + 42); };
     ...
    }
    

    The above code gets transformed to

    void ContainingClass::SomeMethod()
    {
     class AnonymousClass$0
     {
     public:
      AnonymousClass$0()  { }
      operator int (*)(int k) { return static_function; }
      int operator(int k) const { return calc(k + 42); }
     private:
      static int static_function(int k) { return calc(k + 42); }
     };
    
     auto f = AnonymousClass$0();
     ...
    }
    

    Okay, now we can get to the actual question: How can I specify the calling convention for this implicit conversion to a pointer to nonmember function?

    (Note that calling conventions are not part of the C++ standard, so this question is necessarily a platform-specific question.)

    The Visual C++ compiler automatically provides conversions for every calling convention. So with Visual C++, the transformed code actually looks like this:

    void ContainingClass::SomeMethod()
    {
     class AnonymousClass$0
     {
     public:
      AnonymousClass$0()  { }
      operator int (__cdecl *)(int k) { return cdecl_static_function; }
      operator int (__stdcall *)(int k) { return stdcall_static_function; }
      operator int (__fastcall *)(int k) { return fastcall_static_function; }
      int operator(int k) { return cdecl_static_function(k); }
     private:
      static int __cdecl cdecl_static_function(int k) { return calc(k + 42); }
      static int __stdcall stdcall_static_function(int k) { return calc(k + 42); }
      static int __fastcall fastcall_static_function(int k) { return calc(k + 42); }
     };
    
     auto f = AnonymousClass$0();
     ...
    }
    

    In other words, the compiler creates all the conversions, just in case. (The versions you don't use will be removed by the linker.)

    But only for noncapturing lambdas.

  • The Old New Thing

    What's the point of using a custom timer queue if there is a default one already?

    • 4 Comments

    A customer observed that when you create a timer via Create­Timer­Queue­Timer, you have the option of passing either NULL to get the default timer queue or a handle to a custom timer queue created by the Create­Timer­Queue function. Why would you want to create a custom timer queue?

    If you create a custom timer queue, then you know that all the timers in the queue are yours. For example, there might be another component in the process that creates a lot of timers in the default timer queue. If you had put your timers in the default timer queue, then your timers will risk starvation.

    It's the same sort of logic that might lead you to create a custom heap instead of using the default heap: You create a custom heap if you want to isolate your allocations from other components. Another component that creates a lot of allocations in the default heap may create fragmentation pressure in the default heap, but your custom heap is unaffected.

    Note, however, that in both cases (timer starvation and heap fragmentation), this is not something you typically worry about because the cost of the insurance often doesn't outweigh the benefit. For example, if you create your own timer queue, then the threads associated with your timer queue cannot be used to service timers from the default timer queue and vice versa, resulting in more active threads in the process than necessary, which increases scheduler overhead and memory usage. Since the threads from the two timer queues are not coordinating their efforts (since that's the point of creating a private timer queue), you can end up with CPU thrashing if both timer queues have collectively created more threads than there are CPU cores, and they are all running.

    After all, the whole point of a timer queue in the first place is to consolidate a lot of little pieces of work into a small number of threads. Creating a private timer queue is moving in the opposite direction.

    But if you want to do it, then the facility is there.

    For example, if you create a private heap, then you can free all the allocations in that heap by destroying the heap. Similarly, having a private timer queue means that you can cancel and delete all the timers by deleting the timer queue. Though at least in the timer case, you can get the best of both worlds (shared timer queue plus bulk cancellation) by associating all the timers with a cleanup group.

  • The Old New Thing

    When asking about the capacity of a program, you also need to consider what happens when you decide to stop using the program

    • 27 Comments

    An internal customer had a question about a tool, let's call it Program Q.

    As part of our gated checkin system, the system creates a new record in a Program Q table to record details of the checkin. (What tests were run, who did the code review, that sort of thing.) We are considering incorporating the Program Q record number as part of our version number: major.minor.service­pack.hot­fix.record­number. What is the maximum number of records per table supported by Program Q?

    Now, the easy way out is to just answer the question: The theoretical maximum number of records per table is 2³²−1. Even if your gated checkin system's throughput is one checkin per second, that gives you over a century of record numbers.

    But answering the question misses the big picture: The limiting factor is not the capacity of Program Q. The limiting factor is how long you plan to keep using Program Q! Before the century is up, you probably won't be using Program Q. What is your transition plan?

    In this case, it's probably not that complicated. Suppose that at the time Program Q is retired and replaced with Program R, the highest record number is 314159. You could just say that the version number in the binary is the Program R record number plus 400000.

    If you're clever, you can time the switch from Program Q to Program R to coincide with a change to the major.minor.service­pack.hot­fix, at which point you can reset the record­number to 1.

  • The Old New Thing

    Microspeak: Light up

    • 16 Comments

    In Microspeak, a feature lights up if it becomes available in an application when run on an operating system that supports it.

    The idea is that you write your application to run on, say, Windows versions N and N + 1. There is a new feature in Windows version N + 1, and new functionality in the application when the code detects that the underlying Windows feature is availble.

    Here are a few citations:

    I have had some requests lately for details on the path to Windows 7 compatibility and how to light up key features.
    Top 7 Ways to light Up Your Apps on Windows Server 2008.

    The idea is that the program takes advantage of new Windows features when running on new versions of Windows. if run on older versions of Windows without those features, nothing happens.

    Inside the product group, discussion about "lighting up" often takes the form of deciding how much new hotness will be automatically applied to old applications and how much needs to be explicitly opted in. Applying new features to old applications makes the old applications and the new feature more valuable because the user sees the feature everywhere, as opposed to working only in new applications designed to take advantage of it. On the other hand, applying them automatically to old applications creates a compatibility risk, because the application may observe a behavior that hadn't occurred before, and it may begin behaving erratically as a result.

  • The Old New Thing

    The 2015/2016 Seattle Symphony subscription season at a glance

    • 6 Comments

    For many years, I've put together a little pocket guide to the Seattle Symphony subscription season for my symphony friends to help them decide which ticket package they want. For the past several years now, we haven't ordered any tickets at all because we all have young children, but I still make this guide out of some sense of obligation.

    So here's the at-a-glance season guide for the 2015/2016 season anyway, again with no comments from me because nobody I know is going to use them to decide which tickets to order. Besides, you can probably preview nearly all of the pieces nowadays (minus the premieres) by searching on YouTube.

    Here is the official brochure for those who want to read the details, and you can see what The Seattle Times thinks of it.

    Week Program 21 13 7A
    7B
    7C
    7D
    7E
    7F
    7G 4A SU
    09/24
    2015
    Mendelssohn: String Quartet #6
    Beethoven: Symphony #4
    Mahler: Symphony #1

     
     

     
     
       
     
     
       
     
     

    10/01
    2015
    R. Strauss: Don Quixote
    Brahms: Symphony #3
                   
    10/08
    2015
    Dvořák: A Hero's Song
    Britten: Violin Concerto
    R. Strauss: Also Sprach Zarathustra
                   
    10/22
    2015
    Stravinsky: Symphony in C
    Beethoven: Piano Concerto #1
    Mozart: Symphony #41, "Jupiter"
                   
    11/05
    2015
    Giya Kancheli: World Premiere
    Brahms: Violin Concerto
    Martinů: Symphony #4
                   
    11/12
    2015
    R. Strauss: Till Eulenspiegel's Merry Pranks
    Bruch: Violin Concerto #1
    Nielsen: Symphony #4, "The Inextinguishable"
                 
     
     
    11/19 Mahler: Symphony #10 (Cooke)                
    12/03
    2015
    Debussy: Danses sacrée et profane
    Messiaen: Poèmes pour Mi
    Fauré: Requiem
                   
    01/07
    2016
    Rimsky-Korsakov: Overture to The Tsar's Bride
    Rachmaninov: Piano Concerto #2
    Tchaikovsky: Suite #3 in G
                   
    01/21
    2016
    Mozart: Selections from Idomeneo Ballet Music
    Mozart: Violin Concerto #3
    Haydn: Symphony #104, "London"
                   
    02/04
    2016
    R. Strauss: Don Juan
    Beethoven: Piano Concerto #3
    Berio: Sinfonia (for 8 voices and orchestra)
                   
    02/11
    2016
    Ives: Three Places in New England
    Bartók: Piano Concerto #3
    Beethoven: Symphony #3, "Eroica"
                   
    03/10
    2016
    Haydn: Symphony #88
    Mozart: Piano Concerto #23
    Schoenberg: Transfigured Night
                   
    03/17
    2016
    John Adams: Scheherezade.2, Violin Concerto
    Elgar: Pomp and Circumstance March #3
    Respighi: Pines of Rome
                 
     
     

    03/24
    2016
    Glinka: Summer Night in Madrid
    Glazunov: Violin Concerto
    Rimsky-Korsakov: Scheherezade
                   
    04/07
    2016
    Mussorgsky: Introduction to Khovanshchina
    Prokofiev: Violin Concerto #2
    Brahms: Symphony #4
                   
    04/14
    2016
    Glinka: Overture to Ruslan and Ludmilla
    Dvořák: Cello Concerto
    Silvestrov: Symphony #5
                 
     
     

    04/21
    2016
    Mendelssohn: Overture to A Midsummer Night's Dream
    Britten: Nocturne
    Szymanowski: Symphony #3
    Tchaikovsky: Romeo & Juliet Fantasy Overture
                   
    04/28
    2016
    Dutilleux: Timbres, espace, mouvement
    Beethoven: Piano Concerto #4
    Ewald: Symphony for Brass Quintet #3
    Prokofiev: Symphony #7
     
     

     
     
     

     
     
     
     

     
         

     
     
    06/02
    2016
    Stravinsky: Symphony of Psalms
    Shostakovich: Symphony #4
                   
    06/05
    2016
    Fauré: Masques et bergamasques
    Ravel: Piano Concerto in G
    Dvořák: Symphone #9, "New World"
                   
    06/09
    2016
    Anna Clyne: U.S. Premiere
    Gershwin: Concerto in F
    Beethoven: Symphony #7
                   
    Week Program 21 13 7A
    7B
    7C
    7D
    7E
    7F
    7G 4A SU

    Insider tip: Click a column header to focus on a specific series. (This feature has been around for several years, actually.)

    Legend:

    21Masterworks 21-concert series (Choice of Thursdays or Saturdays)
    13Masterworks 13-concert series (Choice of Thursdays or Saturdays)
    7AMasterworks 7-concert series A (Thursdays)
    7BMasterworks 7-concert series B (Saturdays)
    7CMasterworks 7-concert series C (Thursdays)
    7DMasterworks 7-concert series D (Saturdays)
    7EMasterworks 7-concert series E (Thursdays)
    7FMasterworks 7-concert series F (Saturdays)
    7GMasterworks 7-concert series G (Sunday afternoons)
    4AMasterworks 4-concert series A (Friday afternoons)
    SUSymphony Untuxed (Fridays, reduced program)

    For those not familiar with the Seattle Symphony ticket package line-ups: Most of the ticket packages are named Masterworks nX where n is the number is the number of concerts in the package, and the letter indicates the variation. Ticket packages have been combined if they are identical save for the day of the week. For example, 7C and 7D are the same concerts; the only difference is that 7C is for Thursday nights, while 7D is for Saturday nights.

    This chart doesn't include concert series such as the Distinguished Artists series which share no concerts with any of the Masterworks concerts.

    Notes and changes:

    • The 7[AB], 7[CD], and 7[EF] concert series do not overlap, so you can create your own 14-concert series by taking any two of them, or recreate the 21-concert series by taking all three.
    • The 13-concert series is the same as the 7[CD] and 7[EF] series combined, minus the June 9 concert.
    • The non-Masterworks series line-up continues to be tweaked: Gone are the Mozart series and the The Sunday Untuxed short concerts for families. Two children's series were renamed but otherwise unchanged: Discover Music became Family Concerts, and Soundbridge Presents became Symphony Kids.
    • The long-time Wolfgang club, which targeted adults under age 40, and its accompanying series appear to be gone.
    • A Shakespeare-themed concert in April commemorates the 400th anniversary of his death.

    This is the first season of a two-season cycle of Beethoven symphonies and piano concerti. Although there are no ticket packages specifically for the Beethoven concerts, tickets are available individually so you can make your own festival.

  • The Old New Thing

    Reading the word under the cursor via UI Automation

    • 12 Comments

    A while back, I showed how to use UI Automation to extract the text under the mouse cursor. Let's refine the program so it can extract the word under the cursor.

    The ability to extract subunits of text is provided by the Text­Pattern class. You can ask for the text around a specific point by asking Range­From­Point to create a zero-length range directly under the cursor, and then adjust the range to grab the text you want.

    Start with the first Little Program and make these changes:

    using System;
    using System.Windows;
    using System.Windows.Forms;
    using System.Windows.Automation;
    using System.Windows.Automation.Text;
    
    class Program
    {
     static Point MousePos {
      get { var pos = Control.MousePosition;
            return new Point(pos.X, pos.Y); }
     }
    
     public static void Main()
     {
      for (;;) {
       AutomationElement e = AutomationElement.FromPoint(MousePos);
       if (e != null) {
        object o;
        if (e.TryGetCurrentPattern(TextPattern.Pattern, out o)) {
         var textPattern = (TextPattern)o;
         var range = textPattern.RangeFromPoint(MousePos);
         range.ExpandToEnclosingUnit(TextUnit.Word);
         Console.WriteLine(range.GetText(-1));
        }
       }
       System.Threading.Thread.Sleep(1000);
      }
     }
    }
    

    We grab the element under the mouse and see if it supports Text­Pattern. If so, we ask the text pattern for a Text­Pattern­Range that corresponds to the position of the cursor. It's where the insertion point would go if you clicked there. Since the insertion point always goes between characters, the initial range has zero length. We expand it to the word enclosing the imaginary insertion point, then extract the text, then print it.

    Note that the Text pattern is typically supported only by editable text controls, so you will need to practice with Notepad or some other text editor.

    Just to show how annoying you can push things, let's secretly select the text under the cursor.

    using System;
    using System.Windows;
    using System.Windows.Forms;
    using System.Windows.Automation;
    using System.Windows.Automation.Text;
    
    class Program
    {
     static Point MousePos {
      get { var pos = Control.MousePosition;
            return new Point(pos.X, pos.Y); }
     }
    
     public static void Main()
     {
      for (;;) {
       AutomationElement e = AutomationElement.FromPoint(MousePos);
       if (e != null) {
        object o;
        if (e.TryGetCurrentPattern(TextPattern.Pattern, out o)) {
         var textPattern = (TextPattern)o;
         var range = textPattern.RangeFromPoint(MousePos);
         range.ExpandToEnclosingUnit(TextUnit.Word);
         // Console.WriteLine(range.GetText(-1));
         range.Select();
        }
       }
       System.Threading.Thread.Sleep(1000);
      }
     }
    }
    

    Once a second, the program autoselects the word under the cursor. This gets really annoying fast, but it's just a demonstration.

  • The Old New Thing

    What is this inconsistent heap state that MSDN warns me about during DLL_PROCESS_DETACH?

    • 23 Comments

    In the MSDN documentation for the Dll­Main entry point, MSDN notes:

    When handling DLL_PROCESS_DETACH, a DLL should free resources such as heap memory only if the DLL is being unloaded dynamically (the lpReserved¹ parameter is NULL). If the process is terminating (the lpvReserved parameter is non-NULL), all threads in the process except the current thread either have exited already or have been explicitly terminated by a call to the Exit­Process function, which might leave some process resources such as heaps in an inconsistent state. In this case, it is not safe for the DLL to clean up the resources. Instead, the DLL should allow the operating system to reclaim the memory.

    A customer wanted to know, "What is this inconsistent heap state that MSDN is talking about here?"

    The information is actually right there in the sentence. "... explicitly terminated by a call to the Exit­Process function, which might leave some process resources such as heaps in an inconsistent state."

    When you see text that says "X might lead to Y," then when you ask "What could lead to Y?" you might consider that it is X.

    Background reading: Quick overview of how processes exit on Windows XP, plus bonus electrification of critical sections and slim reader/writer locks.

    Okay, I'll give the customer the benefit of the doubt and assume that the question was more along the lines of "Why would termination by a call to the Exit­Process function lead to an inconsistent state?"

    Remember why Terminate­Thread is a horrible idea: It terminates the thread in the middle of whatever it is doing, not giving it a chance to restore consistency or otherwise provide for an orderly end to operations. The thread may have been in the middle of updating a data structure, which usually involves perturbing an invariant, then re-establishing it. If it was terminated before it could re-establish the invariant, you now have an inconsistent data structure.

    That's the sort of inconsistency the paragraph is talking about. If one of the threads terminated by Exit­Process was executing heap code at the moment it was terminated, then the heap may be inconsistent because the thread never got a chance to re-establish consistency. (The heap tries to detect that this has happened but all that does is transform one failure mode into another. The failure is still there.)

    Of course, the heap isn't the only resource that suffers from this problem. Any resource that is available to more than one thread is susceptible to this. It's just that the heap is a very popular shared resource, so it gets an extra mention in the documentation.

    ¹ Typo. Should be lpvReserved.

  • The Old New Thing

    Who is this rogue operative that filled my object with 0xDD, then sent me a message?

    • 21 Comments

    A failure occurred during stress testing, and the component team came to the conclusion that their component was actually the victim of memory corruption and they asked for help trying to see if there was anything still in memory that would give a clue who did the corrupting.

    /* static */ LRESULT CALLBACK CContoso::WndProc(
        HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
    {
        CContoso *pThis = reinterpret_cast<CContoso *>
                               (GetWindowLongPtr(hwnd, GWLP_USERDATA));
        ...
        pThis->... // crash on first dereference of pThis
    

    According to the debugger, the value of pThis is a valid pointer to memory that is complete nonsense.

    0: kd> dv
               hwnd = 0xf0040162
               uMsg = 0x219
               ...
              pThis = 0x10938bf0
               ...
    0: kd> dt pThis
    Type CContoso*
       +0x000 __VFN_table : 0xdddddddd 
       +0x004 m_cRef           : 0n-572662307
       +0x008 m_hwnd           : 0xdddddddd HWND__
       ...
    

    The CContoso object was filled with the byte 0xDD. Who would do such a thing?

    There are a few clues so far, and if you're psychic, you may have picked up on their aura.

    But I had a suspicion what happened, so I dug straight into the code to check my theory.

    BOOL CContoso::StartStuffInBackground()
    {
     AddRef(); // DoBackgroundWork will release the reference
     BOOL fSuccess = QueueUserWorkItem(&CContoso::DoBackgroundWork, this, 0);
     if (!fSuccess) Release();
     return fSuccess;
    }
    
    /* static */ DWORD CALLBACK CContoso::DoBackgroundWork(void *lpParameter)
    {
     CContoso *pThis = static_cast<CContoso *>(lpParameter);
     pThis->DoThis();
     pThis->DoThat();
     pThis->Release();
     return 0;
    }
    

    So far, we have a standard pattern. An extra reference to the object is kept alive as long as the background thread is still running. This prevents the object from being destroyed prematurely.

    (Note that this object is not necessarily a COM object. It could be a plain object that happens to have chosen the names Add­Ref and Release for the methods that manipulate the reference count.)

    What people often forget to consider is that this means that the final release of the CContoso object can occur on the background thread. I mean, this is obvious in one sense, because they are adding the extra reference specifically to handle the case where we want to delay object destruction until the background thread completes. But what happens if that scenario actually comes to pass?

    CContoso::~CContoso()
    {
     if (m_hwnd != nullptr) DestroyWindow(m_hwnd);
     ...
    }
    

    As part of the destruction of the CContoso object, it destroys its window. But Destroy­Window must be called on the same thread which created the window: "A thread cannot use Destroy­Window to destroy a window created by a different thread."

    This means that if the final release of the CContoso object comes from the background thread, the destructor will run on the background thread, and the destructor will try to destroy the window, but the call will fail because it is on the wrong thread.

    The result is that the object is destroyed, but the window still hangs around, and the window has a (now dangling) pointer to the object that no longer exists.

    Since the window in question was a hidden helper window, the program managed to survive like this for quite some time: Since the program thought the window was destroyed, there was no code that tried to send it a message, and the normal system-generated messages were not anything the object cared about, so they all fell through to Def­Window­Proc and nobody got hurt. But eventually, some other stress test running on the machine happened coincidentally to broadcast the WM_SETTING­CHANGE message 0x0219, and when the object tried to check what settings changed, that's when it crashed. (That was one of the clues I hinted at above: The message that triggered the crash is 0x0219. This is a good number to memorize if you spend time studying stress failures because it is often the trigger for crashes like this where a window has been orphaned by its underlying object.)

    The root cause is that the object was treated as a free-threaded object even though it actually had thread affinity.

    One way to fix this is to isolate the parts with thread affinity so that they are used only on the UI thread. The one we identified is the destructor due to its use of Destroy­Window. So at a minimum, we could marshal destruction to the UI thread.

    LONG CContoso::Release()
    {
     LONG cRef = InterlockedDecrement(&this->m_cRef);
     if (cRef == 0)
     {
      if (m_hwnd == nullptr) {
        delete this;
      } else {
        PostMessage(m_hwnd, CWM_DESTROYTHIS, 0, 0);
      }
     }
     return cRef;
    }
    
    /* static */ LRESULT CALLBACK CContoso::WndProc(
        HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
    {
        CContoso *pThis = reinterpret_cast<CContoso *>
                               (GetWindowLongPtr(hwnd, GWLP_USERDATA));
        ...
    
        case CWM_DESTROYTHIS:
         delete pThis;
         return 0;
        ...
    

    (The original code had better have been using an interlocked operation on Release because it was releasing from a background thread already.)

    If the final Release happens before we have a window, then we just destruct in-place, on the theory that if no window is created, then we are being destroyed due to failed initialization and are still on the original thread. Otherwise, we post a message to the window to ask it to destroy the object.

    Note that this design does have its own caveats:

    • Even if the final Release happens on the UI thread, we still post a message, even though we could have destroyed it inline.
    • Posting a message assumes that the message pump will continue to run after the object is released. If somebody releases the object and then immediately exits the thread, the posted message will never arrive and the object will be leaked.
    • Posting a message makes destruction asynchronous. There may be some assumptions that destruction is synchronous with the final release.

    As for the first problem, we could do a thread check and destruct in-place if we are on the UI thread. This would most likely solve the second problem because the exiting thread is not the one that will process the message. It will still be a problem if the background thread does something like

      Release();
      DoSomethingThatCausesTheUIThreadToExitImmediately();
    

    For the second problem, we could change the Post­Message to a Send­Message, but this creates its own problems because of the risk of deadlock. If the UI thread is blocked waiting for the background thread, and the background thread tries to send the UI thread a message, the two threads end up waiting for each other and nobody makes any progress. On the other hand, making the destruction synchronous would fix the third problem.

    Another approach is to push the affinity out one more step:

    /* static */ DWORD CALLBACK CContoso::DoBackgroundWork(void *lpParameter)
    {
     CContoso *pThis = static_cast<CContoso *>(lpParameter);
     pThis->DoThis();
     pThis->DoThat();
     pThis->AsyncRelease();
     return 0;
    }
    
    void CContoso::AsyncRelease()
    {
     PostMessage(m_hwnd, CWM_RELEASE, 0, 0);
    }
    
    /* static */ LRESULT CALLBACK CContoso::WndProc(
        HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
    {
        CContoso *pThis = reinterpret_cast<CContoso *>
                               (GetWindowLongPtr(hwnd, GWLP_USERDATA));
        ...
        case CWM_RELEASE:
         pThis->Release();
         return 0;
    
        ...
    

    In this design, we make the asynchronicity explicit in the name of the function, and we require all background threads to use the asynchronous version. This design again assumes that the only reason the window wouldn't exist is that something went wrong during initialization before any background tasks were created.

    Unfortunately, this design also retains the original constraint that Release can be called only from the UI thread. That makes the object rather fragile, because it is not obvious that Release has such constraints. If you go this route, you probably should rename Release to Release­From­UI­Thread.

    If this object is a COM object, then another option is to use COM marshaling to marshal the IUnknown to the background thread and use IUnknown::Release to release the object. Since you used COM to marshal the object, it knows that Co­Uninitialize should wait for all outstanding references marshaled to other threads, thereby avoiding the "lost message" problem.

    Anyway, those are a few ideas for addressing this problem. None of them are particularly beautiful, though. Maybe you can come up with something better.

    (The component team fixed this problem by taking advantage of a detail in the usage pattern of the CContoso object: The client of the CContoso object is expected to call CContoso::Stop before destroying the object, and after calling CContoso::Stop, the only valid thing you can do with the object is destroy it. Furthermore, that call to CContoso::Stop must occur on the UI thread. Therefore, they moved the part of the cleanup code that must run on the UI thread into the Stop method. The object's background tasks already knew to abandon work once they detected that the object had been stopped.)

  • The Old New Thing

    Why was the replacement installer for recognized 16-bit installers itself a 32-bit program instead of a 64-bit program?

    • 34 Comments

    Even though 64-bit Windows does not support 16-bit applications, there is a special case for 16-bit installers for 32-bit applications. Windows detects this scenario and substitutes a 32-bit replacement installer which replicates the actions of the 16-bit installer. Commenter Karellen is horrified at the fact that the replacement installer is a 32-bit program. "You're writing a program that will run exclusively on 64-bit systems. Why not built it to run natively on the OS it's designed for? Why is this apparently not the "obvious" Right Thing(tm) to do? What am I missing?"

    Recall that a science project is a programming project that is technically impressive but ultimately impractical. For example it might be a project that nobody would actually use, or it attempts to add a Gee-Whiz feature that nobody is really clamoring for.

    But at least a science project is trying to solve a problem. This proposal doesn't even solve any problems! Indeed, this proposal creates problems. One argument in favor of doing it this way is that it is satisfies some obsessive-compulsive requirement that a 64-bit operating system have no 32-bit components beyond the 32-bit emulation environment itself.

    Because! Because you're running a 64-bit system, and running apps native to that system is just more elegant.

    Okay, it's not obsessive-compulsive behavior. It's some sort of aesthetic ideal, postulated for its own sake, devoid of practical considerations.

    Remember the problem space. We have a bunch of 32-bit applications that use a 16-bit installer. Our goal is to get those applications installed on 64-bit Windows. By making the replacement installer a 32-bit program, you get the emulator to do all the dirty work for you. Things like registry redirection, file system redirection, and 32-bit application compatibility.

    Suppose the original installer database says

    • Copy X.DLL file into the %Program­Files%\AppName directory.
    • Copy Y.DLL into the %windir%\System32 directory.
    • If the current version of C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\AcroPDF.dll is 7.5 or higher, then set this registry key.

    If you write the replacement installer as a 32-bit program, then other parts of the 32-bit emulation engine do the work for you.

    • The environment manager knows that 64-bit processes get the environment variable Program­Files pointing to C:\Program Files, whereas 32-bit processes get Program­Files pointing to C:\Program Files (x86).
    • The file system redirector knows that if a 32-bit process asks for %windir%\System32, it should really get %windir%\SysWOW64.
    • The registry redirector knows that if a 32-bit process tries to access certain parts of the registry, they should be sent to the Wow­64­32­Node instead.

    If you had written the replacement installer as a 64-bit program, you would have to replicate all of these rules and make sure your copy of the rules exactly matched the rules used by the real environment manager, file system redirector, and registry redirector.

    Now you have to keep two engines in sync: the 32-bit emulation engine and the 64-bit replacement installer for 32-bit applications. This introduces fragility, because any behavior change in the 32-bit emulation engine must be accompanied by a corresponding change in the 64-bit replacement installer.

    Suppose the application compatibility folks add a rule that says, "If a 32-bit installer tries to read the version string from C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\AcroPDF.dll, return the version string from C:\Program Files (x86)\Common Files\Adobe\Acrobat\ActiveX\AcroPDF.dll instead." And suppose that rule is not copied to the 64-bit replacement installer. Congratulations, your 64-bit replacement installer will incorrectly install any program that changes behavior based on the currently-installed version of AcroPDF.

    I don't know for sure, but I wouldn't be surprised if some of these installers support plug-ins, so that the application developer can run custom code during installation. It is possible for 16-bit applications to load 32-bit DLLs via a technique known as generic thunking, and the 16-bit stub installer would use a generic thunk to call into the 32-bit DLL to do whatever custom action was required. On the other hand, 64-bit applications cannot load 32-bit DLLs, so if the 64-bit replacement installer encountered a 32-bit DLL plug-in, it would have to run a 32-bit helper application to load the plug-in and call into it. So you didn't escape having a 32-bit component after all.

    And the original obsessive-compulsive reason for requiring the replacement installer to be 64-bit was flawed anyway. This is a replacement installer for a 32-bit application. Therefore, the replacement installer is part of the 32-bit emulation environment, so it is allowed to be written as a 32-bit component.

    Let's look at the other arguments given for why the replacement installer for a 32-bit application should be written as a 64-bit application.

    Because complexity is what will be our undoing in the end, and reducing it wherever we can is always a win.

    As we saw above, writing the replacement installer as a 64-bit application introduces complexity. Writing it as a 32-bit application reduces complexity. So this statement itself argues for writing the replacement installer as a 32-bit application.

    Because we can't rewrite everything from scratch at once, but we can create clean new code one small piece at a time, preventing an increase to our technical debt where we have the opportunity to do so at negligible incremental cost to just piling on more cruft.

    As noted above, the incremental cost is hardly negligible. Indeed, writing the replacement installer as a 64-bit application is not merely more complex, it creates an ongoing support obligation, because any time there is a change to the 32-bit emulation environment, that change needs to be replicated in the 64-bit replacement installer. This is a huge source of technical debt: Fragile coupling between two seemingly-unrelated components.

    And writing the replacement installer as a 32-bit application does not create a future obligation to port it to 64 bits when support for 32-bit applications is dropped in some future version of Windows. Because when support for 32-bit applications disappears (as it already has on Server Core), there will be no need to port the replacement installer to 64-bit because there's no point writing an installer for a program that cannot run!

    Writing the replacement installer as a 32-bit program was the right call.

Page 5 of 447 (4,467 items) «34567»