August, 2006

  • The Old New Thing

    Accessibility is not just for people with disabilities

    • 21 Comments

    When the topic of accessibility comes up, most people think it's all about people with disabilities. While it's true that people with disabilities were the original target audience for accessibility, more recent developments have highlighted the value of accessible technologies to everyone.

    For one thing, making your program accessible means that test automation can manipulate your program in a uniform manner. They can enumerate the buttons, get text for them, press a button programmatically, and so on. Even if you don't use a standard button control but instead opt to go windowless, as long as you expose your controls via accessibility, test automation can find them.

    Also, we saw how programs can use accessibility to retrieve text from the screen. I use this in my English/Chinese dictionary program to "pluck" text off the screen and paste it into the "translate this" box, where I can arrow through the sentence and have the program translate each word or phrase on the fly. (Rats, now you know where I'm going with the program.) And once I started using this feature, I discovered to my dismay how many of the programs that I use on a regular or semi-regular basis simply fail to support this simple task. Microsoft products have a much higher success rate, but even there the support is not 100%.

    Finally, check out this screencast showing off Vista's speech recognition system. The "Say what you see" feature which Chris discusses at time code 8:52 needs to get the name of every element on the screen so it can take what you say and look for it on the screen. If your program doesn't expose these names, the "Say what you see" feature won't know what the user needs to say to click on your button, and users will say, "Harumph, why doesn't this program work with voice recognition? All my other programs do."

    My secret hope is that "Say what you see" will finally be enough to prod people into taking accessibility seriously. Because it's not just for people with disabilities.

  • The Old New Thing

    Cool web site: Bill Monk

    • 9 Comments

    One of my friends introduced me to Bill Monk, which is like PayPal but without the money. You and your friends sign up, and then you can let the site keep track of who owe whom how much.

    Of course, whether this is useful to you depends on how you and your friends actually deal with money and small debts. With some of my friends, we use the "It'll all even out in the end" strategy. We'll take turns paying for dinner and events based on the "I think it's my turn" honor system.

    With another group of friends, the bills are neatly tabulated and reckonings sent out by email for settlement. They use this system since they've known each other since college (I joined later), back when money was in short supply, and in order to accommodate group members of varying financial means. For this group, Bill Monk works great (and replaces a similar system some of us had set up to do exactly this).

  • The Old New Thing

    Do not change program semantics in the debug build

    • 36 Comments

    What you don't want is a problem that goes away when you debug it.

    It is expected that a program have additional debugging code inside #ifdef DEBUG blocks. After all, that's why it's a debug build. But what you definitely don't want to do is have that debugging to fundamentally change the program's behavior. You can perform additional validation. You can raise assertion failures. You can track resources. It can be slower and consume additional resources, but you had better not alter code flow.

    // This is wrong
    BOOL DoSomething(Thing *p)
    {
    #ifdef DEBUG
     // Do some extra parameter checking
     if (!p) {
      Log("Error! p parameter must not be NULL.");
      return FALSE; // WRONG!
     }
    #endif
      ... remainder of function ...
    }
    

    This code is wrong: The debug version behaves fundamentally differently from the retail version. If somebody calls this function with NULL for the p parameter, the retail version of the program will crash but the debug build will trap the error and fail the call.

    Do not change the function's semantics in the debug build. If the retail build crashes, then the debug build must also crash in the same way. Sure, you can log the failure before you crash, but you still need to crash.

    An analogous mistake in the C# world might go like this:

    // This is wrong
    void DoSomething()
    {
    #if DEBUG
      try {
    #endif
       ... guts of function ...
    #if DEBUG
      } catch (Exception ex) {
         LogException(ex);
      }
    #endif
    }
    

    In this C# example, the debug build logs and swallows exceptions, while the retail version allows them to escape.

    If you mess up and write code like this, where the retail and debug versions behave in some fundamentally different way, you will eventually get yourself into this situation: The retail version has some problem, but the debug version works okay. Your customer can't figure out what the difference is, so they switched to the debug version on their production servers. It runs twice as slow and consumes three times as much memory, requiring significant capital expenses to scale up to their previous level of service. But it's the best they can do because the problem doesn't occur on the debug version (and therefore cannot be debugged there).

    I have seen reports of software getting into this predicament, and it reflects very poorly on the developers.

  • The Old New Thing

    The cultural anthropology of getting on a bicycle

    • 53 Comments

    I can tell where you grew up by watching you get on a bicycle.

    Well, sort of.

    In my limited experience, I've observed two distinct ways of getting on a bicycle. The first is what I'll call the Chinese method, since it's the dominant technique in China, Taiwan, and Japan, as far as I can tell. To get on a bicycle using the Chinese method, stand to the side of the bicycle (let's say, the left side) with both hands on the handlebars in the normal riding position. Put the left pedal at the bottom of its stroke and place your left foot on it. Now ride the bicycle like a scooter, pushing off with your right foot, until you pick up some speed. While coasting forward, swing your right leg over the back of the bicycle and place your right foot upon the right pedal (which is at the top of its stroke). Sit down and begin pedaling.

    The second method I will call the U.S. method since it is the dominant technique in the States. To get on a bicycle using this method, straddle the bicycle with both hands on the handlebars in the normal riding position, placing one foot on the ground and the other foot on its corresponding pedal at the top of its stroke. Simultaneously push forward with the ground foot and put all your weight on the pedal foot (driving it forward). This will get you coasting very slowly. Use this time to take the foot that was on the ground and place it on its corresponding pedal (which by now has reached the top of its stroke). Sit down and begin pedaling.

    Each of these methods has its own drawbacks. The U.S. method requires you to be able to maintain your balance on a bicycle at very low speeds and then get your ground foot into position quickly before you run out of momentum. (This is particularly tricky when you have to strap or clip your foot into place.) On the other hand, the Chinese method requires you to shift your weight while balancing on a single foot. That bit is my downfall. When all my weight gets on that one foot, I start wobbling and can't quite finish the move.

    Now, using this method to determine where someone learned to ride a bicycle is not foolproof, of course. Some of my childhood friends who grew up in the United States to Taiwanese parents use the Chinese method.

    During my brief travels in Europe, I neglected to take note of how the locals got on their bicycles. Maybe there's a European way of getting on a bicycle?

  • The Old New Thing

    Why can't I see all of the 4GB of RAM in my machine?

    • 34 Comments

    Here's a question that came from a customer:

    I have set my boot options to /3GB /NOPAE. Even though the server has 4GB of physical memory, we see only 3.5GB of physical memory on the server after it boots up. However, when we change the boot options to /PAE, we can observe 4GB of memory. Furthermore, when I enable only /PAE, even though the physical memory is 4GB, I can see 4GB. Without /PAE, even though the physical memory is 4GB, I can see 3.5GB. What is the reason for this?

    Remember that in the absence of the /PAE switch, the Windows memory manager is limited to a 4GB physical address space. Most of that address space is filled with RAM, but not all of it. Memory-mapped devices (such as your video card) will use some of that physical address space, as will the BIOS ROMs. After all the non-memory devices have had their say, there will be less than 4GB of address space available for RAM below the 4GB physical address boundary. The rest of it went above the boundary. Here's a picture:

    4.5GB
    4GB Potential RAM (0.5GB)   Requires PAE to see
    3.5GB Hardware/ROM (0.5GB)   Visible without PAE
    0GB Potential RAM (3.5GB)

    On this imaginary system, the motherboard assigned the ROMs and the hardware devices to the physical address space between 3.5GB and 4GB (occupying 0.5GB of address space). When you start plugging in your memory chips, then, they are assigned physical addresses starting at the bottom, and then skip over the address space that has already been assigned to the hardware and ROM, then resume.

    On this imaginary system, then, the 0.5GB of address space used for hardware and ROMs causes that much memory to get shoved upwards, and it ends up above the 4GB boundary. Without PAE, the processor is capable only of addressing memory below the 4GB boundary, which means that the memory above that boundary is inaccessible. It's consuming electricity but isn't doing anything.

    The solution is to enable /PAE so that the processor can access the physical address space above the 4GB boundary. Because that's what PAE stands for: Physical Address Extension.

  • The Old New Thing

    Exploiting the inattentive, episode 2: The unlabeled line

    • 42 Comments

    In Episode 1 we learned that "Snide" brand powdered laundry detergent comes with a scoop that holds 5/8 cup detergent, even though the directions call for only 3/8 cup. The intent here is to get people to use more detergent than necessary, thereby increasing sales. A box of detergent which should be sufficient for 30 loads ends up running out after only 18. Consider it "technically legal" mislabeling.

    I returned to Snide brand powdered laundry detergent this week and discovered that they have taken this little trick to a new level. The markings on the side of the scoop read like this:

      2  
      1  
         
         

    The accompanying instructions give these instructions for how much detergent to use:

    Fill to bottom line  (3/8 cup)  for normal loads.
    Fill to line 1  (1/2 cup)  for large loads.
    Fill to line 2  (3/4 cup)  for extra-soiled loads.

    Notice that the amount of detergent you're supposed to use on a regular basis is indicated by the bottom unmarked line, the one you will barely notice. Meanwhile, if you're inattentive and use the entire scoop, you've used one cup of detergent, over twice as much as you were intending.

    My prediction: The next time I buy Snide brand powdered laundry detergent, the scoop will be 1½ cups, with an unlabeled line at 1/2 cup, another line at 3/4 cup labeled "1" and a third line at 1 cup labeled "2". The instructions will go something like this:

    Fill halfway to line 1  (3/8 cup)  for normal loads.
    Fill to bottom line  (1/2 cup)  for large loads.
    Fill to line 1  (3/4 cup)  for extra-soiled loads.
  • The Old New Thing

    Why does SetWindowsHookEx take an HINSTANCE parameter?

    • 10 Comments

    An anonymous commenter asked why the SetWindowsHookEx function takes an HINSTANCE parameter if the first thing it's going to do is convert the instance handle into a file name.

    Because that's not how it worked in 16-bit Windows.

    In 16-bit Windows there was no such thing as hook injection. All 16-bit Windows applications ran in the same address space, so there was no need to inject anything anywhere. Consequently, there was no need to convert the instance handle into a file name in order to inject it.

    Instead, the instance handle was used to increment the reference count on the module so that the hook procedure wouldn't get unloaded while the hook was active. When the hook was uninstalled, the module reference count was decremented.

    Even in 32-bit Windows, the window manager needs the instance handle in order to convert the function pointer back to an RVA so the function can be located when the module is loaded into another process. If you passed a LPCWSTR with a path to the module, the window manager would have to do a GetModuleHandle anyway to recover the instance handle. Since most programs have their instance handle readily available, that was the more natural choice. (Not to mention that it would preserve source compatibility with 16-bit Windows, which was an important criterion when trying to get people interested in porting their code to Win32.)

  • The Old New Thing

    Space scientist James Van Allen (1914–2006)

    • 16 Comments

    Discoverer of the Van Allen radiation belts and one of the TIME magazine men of the year for 1960 James Van Allen died yesterday. (What a coincidence. What are the odds that the person to discover the Van Allen radiation belts would also be named Van Allen?)

    I never met the man, but I met his granddaughter. She has an amazing facility with languages and became fluent in Japanese after spending a year as an exchange student. Her Japanese was so good that at one point, a girl came up to her and asked, "How do you get your skin so pale?", mistaking her for a native who had undergone some sort of bleaching process to get milky white skin.

    That's when you know you've made it. When native speakers mistake you for one of their own who merely has had a lot of cosmetic surgery.

    (Whenever we tell this story, she tries to downplay the experience. "Yeah, well, the girl didn't have her glasses on." Like that changes the story in any significant way. Now, if the girl were deaf...)

  • The Old New Thing

    One way people abused hooks in 16-bit Windows

    • 11 Comments

    We saw last time how windows hooks were implemented in 16-bit Windows. Even though the HHOOK was an opaque data type that should have been treated like a handle, many programs "knew enough to be dangerous" and took advantage of the fact that the HHOOK was just a pointer to the previous hook procedure.

    The most common way of abusing this knowledge was by unhooking from the windows hook chain the wrong way. Instead of calling the UnhookWindowsHook function to unhook a windows hook, they called SetWindowsHook again! Specifically, they removed their hook by simply reinstalling the previous hook at the head of the chain:

    HHOOK g_hhkPrev;
    
    // install the hook
    g_hhkPrev = SetWindowsHook(WH_KEYBOARD, MyHookProc);
    ...
    
    // crazy! uninstall the hook by setting the previous hook "back"
    SetWindowsHook(WH_KEYBOARD, g_hhkPrev);
    

    This code worked in spite of itself; it's as if two wrongs made a "sort of right". If nobody else messed with the hook chain in between the time the hook was installed and it was subsequently "uninstalled", then reinstalling the hook at the head of the chain did restore the chain variables in the same way they would have been restored if they had uninstalled the hook correctly.

    But if somebody else installed their own WH_KEYBOARD hook in the meantime, then setting the previous hook "back" would have the effect of not only "uninstalling" the MyHookProc but also all other hooks that were installed in the meantime. (This is exactly the same problem you have if you aren't careful in how you remove subclassed window procedures.)

    I still have no idea why they used this strange technique instead of doing the right thing, which is just swapping out one line of code for another:

    UnhookWindowsHook(WH_KEYBOARD, MyHookProc);
    

    Windows 3.1 introduced the SetWindowsHookEx/CallNextHookEx model, which doesn't use the external linked list technique but rather manages the hook chain internally. This protected the hook chain from programs that corrupted it by mismanaging the external hook chain, but it meant that when these crazy programs tried to unhook by hooking, they ended up corrupting the internal hook chain. Special code had to be written to detect these crazy people and turn their bad call into the correct one so that the hook chain wouldn't get corrupted.

  • The Old New Thing

    How were window hooks implemented in 16-bit Windows?

    • 9 Comments

    The mechanism for keeping track of window hooks was very different in 16-bit Windows. The functions involved were SetWindowsHook, UnhookWindowsHook and DefHookProc. The first two functions still exist today, but the third one has been replaced with a macro:

    // 16-bit prototype
    DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                             LPARAM lParam, HHOOK FAR *phk);
    
    // 32-bit macro
    #define DefHookProc(nCode, wParam, lParam, phhk)\
            CallNextHookEx(*phhk, nCode, wParam, lParam)
    

    Disclaimer: All code below is "reconstructed from memory". The spirit of the code is intact, but the precise details may be off.

    To install a windows hook in 16-bit Windows, you started by calling SetWindowsHook:

    HHOOK g_hhkPrev;
    
    g_hhkPrev = SetWindowsHook(WH_WHATEVER, MyHookProc);
    

    The return value from SetWindowsHook must be saved in a global variable, which we gave the somewhat provocative name g_hhkPrev. The hook procedure itself went something like this:

    // In Win16, hook procedures returned a DWORD, not an LRESULT.
    DWORD CALLBACK MyHookProc(int nCode, WPARAM wParam, LPARAM lParam)
    {
      if (nCode >= 0) { ... }
      return DefHookProc(nCode, wParam, lParam, &g_hhkPrev);
    }
    

    And then when you were finished, you removed the hook by calling UnhookWindowsHook:

    UnhookWindowsHook(WH_WHATEVER, MyhookProc);
    g_hhkPrev = NULL;
    

    Internally, the chain of hook functions was managed as a linked list, but instead of using some internal data structure to keep track of the hooks, the linked list was managed inside the HHOOK variables themselves.

    The internal implementation of SetWindowsHook was simply this:

    // This array is initialized with a bunch
    // of "do nothing" hook procedures.
    HOOKPROC g_rgHook[NUMHOOKS];
    
    HHOOK WINAPI SetWindowsHook(int nType, HOOKPROC pfnHookProc)
    {
     HHOOK hhkPrev = (HHOOK)g_rgHook[nType];
     g_rgHook[nType] = pfnHookProc;
     return hhkPrev;
    }
    

    Installing a hook merely set your hook procedure as the head of the hook chain, and it returned the previous head. Invoking a hook was a simple matter of calling the hook at the head of the chain:

    DWORD CallHook(int nType, int nCode, WPARAM wParam, LPARAM lParam)
    {
     return g_rgHook[nType](nCode, wParam, lParam);
    }
    

    Each hook procedure did its work and then sent the call down the hook chain by calling DefHookProc, passing the HHOOK by address.

    DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                             LPARAM lParam, HHOOK FAR *phk)
    {
     HOOKPROC pfnNext = (HOOKPROC)*phk;
     if (nCode >=0) {
      return pfnNext(nCode, wParam, lParam);
     }
     ... more to come ...
    }
    

    As you can see, it's all blindingly simple: Invoking a hook calls the first hook procedure, which then calls DefHookProc, which knows that a HHOOK is just a HOOKPROC, and it forwards the call down the chain by merely calling the next hook procedure directly.

    The real magic happens when somebody wants to unhook. Recall that the rule for hook procedures is that a negative hook code should be passed straight to DefHookProc (or in modern times, CallNextHookEx). This convention allows the hook system to use negative codes to manage its own internal bookkeeping. In this case, we're using -1 as the "unhook this hook procedure" code.

    BOOL WINAPI UnhookWindowsHook(int nType, HOOKPROC pfnHookProc)
    {
     return DefHookProc(-1, 0, (LPARAM)pfnHookProc,
                        (HHOOK FAR*)&g_rgHook[nType]);
    }
    
    And then the real magic begins:

    DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                             LPARAM lParam, HHOOK FAR *phk)
    {
     HOOKPROC pfnNext = (HOOKPROC)*phk;
     if (nCode >=0) {
      return pfnNext(nCode, wParam, lParam);
     }
     switch (nCode) {
     case -1: // trying to unhook a node
      if (pfnNext == (HOOKPROC)lParam) { // found it
       *phk = (HHOOK)pfnNext(-2, 0, 0);
       return TRUE;
      }
      // else keep looking
      return pfnNext(nCode, wParam, lParam);
    
     case -2: // report the next hook procedure
       return (DWORD)*phk;
     }
     return 0;
    }
    

    And there you have it, the entire window hook system in two dozen lines of code. You have to give 16-bit Windows credit for being small.

    Let's walk through hook installation, dispatch, and removal to see how this all works. Suppose there is one WH_KEYBOARD hook in the system. Our variables are therefore set up like this:

    // In USER
    g_rgHook[WH_KEYBOARD] = Hook1;
    
    // In HOOK1.DLL
    HHOOK g_hhkPrev1 = DoNothingHookProc;
    
    DWORD CALLBACK Hook1(int nCode, WPARAM wParam, LPARAM lParam)
    {
     if (nCode >= 0) { ... work ... }
     return DefHookProc(nCode, wParam, lParam, &g_hhkPrev1);
    }
    

    Now suppose you want to install a new hook, Hook2.

    // In HOOK2.DLL
    HHOOK g_hhkPrev2;
    
    g_hhkPrev = SetWindowsHook(WH_KEYBOARD, Hook2);
    

    The SetWindowsHook function just puts your function in as the new "head" hook function and returns the old one.

    // In USER
    g_rgHook[WH_KEYBOARD] = Hook2;
    
    // In HOOK2.DLL
    HHOOK g_hhkPrev2 = Hook1;
    
    DWORD CALLBACK Hook2(int nCode, WPARAM wParam, LPARAM lParam)
    {
     if (nCode >= 0) { ... work ... }
     return DefHookProc(nCode, wParam, lParam, &g_hhkPrev2);
    }
    
    // In HOOK1.DLL
    HHOOK g_hhkPrev1 = DoNothingHookProc;
    
    DWORD CALLBACK Hook1(int nCode, WPARAM wParam, LPARAM lParam)
    {
     if (nCode >= 0) { ... work ... }
     return DefHookProc(nCode, wParam, lParam, &g_hhkPrev1);
    }
    

    Now suppose the window manager decides it's time to fire the WH_KEYBOARD hook. It starts with CallHook which calls g_rgHook[WH_KEYBOARD] that takes us to Hook2. That hook function does its work, then calls DefHookProc(..., &g_hhkPrev2), which dispatches the hook to g_hhkPrev2 == Hook1. Similarly, the hook travels through Hook1, then DefHookProc(..., &g_hhkPrev1), where it finally reaches the DoNothingHookProc which does nothing and ends the hook chain.

    Now suppose that HOOK1.DLL decides to uninstall its hook. It therefore calls UnhookWindowsHook(WH_KEYBOARD, Hook1). This starts off the hook chain with the internal hook code -1 and &g_rgHook[WH_KEYBOARD] as the first hook pointer. This activates the case -1 in DefHookProc code path, which dereferences its phk parameter and obtains g_rgHook[WH_KEYBOARD] == Hook2. Since this is not equal to Hook1, the call forwards down the chain to Hook2.

    Like a good hook function, Hook2 reacts to the negative hook code by handing the call directly to DefHookProc(-1, ..., &g_hhkPrev2). This time, *phk == g_hhkPrev2 == Hook1, so the test succeeds and we dispatch the hook down the chain with a new internal code of -2, which means, "Tell me what the next hook procedure is".

    This dispatch calls Hook1 which (since the notification code is negative) immediately passes the call to DefHookProc(-2, ..., &g_hhkPrev1). This now triggers the case -2 code path, which just returns *phk == g_hhkPrev1 == DoNothingHookProc. This value is returned to the DefHookProc(-1, ...) which stores the result into *phk == g_hhkPrev2; the result is that you have g_hhkPrev2 = DoNothingHookProc. Finally, DefHookProc returns TRUE to indicate that the hook was successfully uninstalled. This value is then returned out from all the nested function calls to the original caller of UnhookWindowsHook.

    Observe that at the end of this unhook exercise, we get the desired result:

    // In USER
    g_rgHook[WH_KEYBOARD] = Hook2; // unchanged
    
    // In HOOK2.DLL
    g_hhkPrev2 = DoNothingHookProc; // updated!
    
    DWORD CALLBACK Hook2(int nCode, WPARAM wParam, LPARAM lParam)
    {
     if (nCode >= 0) { ... work ... }
     return DefHookProc(nCode, wParam, lParam, &g_hhkPrev2);
    }
    

    And Hook1 is out of the hook chain, as we desired.

    This really isn't all that complicated. All we did was delete a node from a linked list. It's just that this particular linked list cannot be traversed by just dereferencing pointers. Instead, we have to issue a function call and ask the recursive function to perform the work on the "next" node for us. That's what the negative nCode values are for.

    Every time I work through this exercise, I am impressed by how compactly 16-bit Windows was written. In just two dozen lines of code, we managed a linked list of function calls, including a dispatching system as well as arbitrary deletion from the middle of the linked list, and all without any memory allocation.

    (And because I know people are going to try to change the topic: Remember, I'm talking about 16-bit Windows, not 32-bit window hooks.)

    Next time, we'll look at one way people abused this simple system.

Page 3 of 4 (39 items) 1234