December, 2010

  • The Old New Thing

    How do I simulate input without SendInput?

    • 15 Comments

    Michal Zygmunt wants to create a system where multiple applications can have focus, with different users generating input and directing them at their target applications. Attempting to simulate this by posting input messages didn't work. "Can you tell us maybe how SendInput is internally implemented so that we can use it to simulate only part of the actions (like without acquiring focus)?"

    SendInput operates at the bottom level of the input stack. It is just a backdoor into the same input mechanism that the keyboard and mouse drivers use to tell the window manager that the user has generated input. The SendInput function doesn't know what will happen to the input. That is handled by much higher levels of the window manager, like the components which hit-test mouse input to see which window the message should initially be delivered to. So if your goal is to change the way you call SendInput so it changes the focus management rules, you're barking up the wrong tree. It's like asking, "Please tell me how RAM chips work so I can use it to change the way Lotus 1-2-3 resolves circular references."

  • The Old New Thing

    Developing the method for taking advantage of the fact that the OVERLAPPED associated with asynchronous I/O is passed by address

    • 26 Comments

    You can take advantage of the fact that the OVERLAPPED associated with asynchronous I/O is passed by address, but there was some confusion about how this technique could "work" when kernel mode has no idea that you are playing this trick.

    Whether kernel mode is in on the trick is immaterial since it is not part of the trick.

    Let's start with a version of the code which does not take advantage of the OVERLAPPED structure address in the way described in the article. This is a technique I found in a book on advanced Windows programming:

    #define MAX_OVERLAPPED 10 // let's do 10 I/O's at a time
    
    // data to associate with each OVERLAPPED
    struct OTHERDATA { ... };
    
    OVERLAPPED MasterOverlapped[MAX_OVERLAPPED];
    OTHERDATA OtherData[MAX_OVERLAPPED];
    
    OTHERDATA* FindOtherDataFromOverlapped(OVERLAPPED *lpOverlapped)
    {
     ptrdiff_t index = lpOverlapped - MasterOverlapped;
     return &OtherData[index];
    }
    
    // I/O is issued via
    // ReadFileEx(hFile, lpBuffer, nNumberOfBytesToRead,
    //            &MasterOverlapped[i], CompletionRoutine);
    
    void CALLBACK CompletionRoutine(
        DWORD dwErrorCode,
        DWORD dwNumberOfBytesTransferred,
        LPOVERLAPPED lpOverlapped)
    {
     OTHERDATA *lpOtherData =
                           FindOtherDataFromOverlapped(lpOverlapped);
     ... do stuff with lpOverlapped and lpOtherData ...
    }
    

    This version of the code uses the address of the OVERLAPPED structure to determine the location in the MasterOverlapped table and uses the corresponding entry in the parallel array at OtherData to hold the other data.

    Let's make this code worse before we make it better:

    OTHERDATA* FindOtherDataFromOverlapped(OVERLAPPED *lpOverlapped)
    {
     for (int index = 0; index < MAX_OVERLAPPED; index++) {
      if (&MasterOverlapped[index] == lpOverlapped) {
       return &OtherData[index];
      }
     }
     FatalError(); // should never be reached
    }
    

    Instead of doing simple pointer arithmetic to recover the index, we walk the array testing the pointers. This is naturally worse than doing pointer arithmetic, but watch what this step allows us to do: First, we reorganize the data so that instead of two parallel arrays, we have a single array of a compound structure.

    struct OVERLAPPEDEX
    {
     OVERLAPPED Overlapped;
     OTHERDATA OtherData;
    };
    
    OVERLAPPEDEX Master[MAX_OVERLAPPED];
    
    OTHERDATA* FindOtherDataFromOverlapped(OVERLAPPED *lpOverlapped)
    {
     for (int index = 0; index < MAX_OVERLAPPED; index++) {
      if (&Master[index].Overlapped == lpOverlapped) {
       return &Master[index].OtherData;
      }
     }
     FatalError(); // should never be reached
    }
    
    // I/O is issued via
    // ReadFileEx(hFile, lpBuffer, nNumberOfBytesToRead,
    //            &Master[i].Overlapped, CompletionRoutine);
    

    All we did was consolidate the parallel arrays into a single array.

    Now that it's an array of compound structures, we don't need to carry two pointers around (one to the OVERLAPPED and one to the OTHERDATA). We can just use a single OVERLAPPEDEX pointer and dereference either the Overlapped or the OtherData part.

    OVERLAPPEDEX* FindOverlappedExFromOverlapped(
        OVERLAPPED *lpOverlapped)
    {
     for (int index = 0; index < MAX_OVERLAPPED; index++) {
      if (&Master[index].Overlapped == lpOverlapped) {
       return &Master[index];
      }
     }
     FatalError(); // should never be reached
    }
    
    void CALLBACK CompletionRoutine(
        DWORD dwErrorCode,
        DWORD dwNumberOfBytesTransferred,
        LPOVERLAPPED lpOverlapped)
    {
        OVELRAPPEDEX *lpOverlappedEx =
                        FindOverlappedExFromOverlapped(lpOverlapped);
        ... do stuff with lpOverlappedEx ...
    }
    

    Finally, we can optimize the FindOverlappedExFromOverlapped function that we de-optimized earlier. Observe that the de-optimized loop is an example of the "for/if" anti-pattern.

    The "for/if" anti-pattern goes like this:

    for (int i = 0; i < 100; i++) {
     if (i == 42) do_something(i);
    }
    

    This can naturally be simplified to

    do_something(42);
    

    Our FindOverlappedExFromOverlapped function is a special case of this anti-pattern. It becomes more evident if we do some rewriting. Start with

    &Master[index].Overlapped == lpOverlapped
    

    Apply CONTAINING_RECORD to both sides.

    CONTAINING_RECORD(&Master[index].Overlapped, OVERLAPPEDEX, Overlapped) ==
        CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped)
    

    The left-hand side of the comparison simplifies to

        &Master[index]
    

    resulting in

    &Master[index] ==
       CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped)
    

    Recall that a[b] is equivalent to *(a+b), and therefore &a[b] is equivalent to a+b.

    Master + index ==
       CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped)
    
    Now subtract Master from both sides:
    index == CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped) - Master
    

    We have transformed the test into a clear case of the for/if anti-pattern, and the function can be simplified to

    OVERLAPPEDEX* FindOverlappedExFromOverlapped(
        OVERLAPPED *lpOverlapped)
    {
     ptrdiff_t index =
       CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped) - Master;
     return &Master[index];
    }
    

    Again, rewrite &a[b] as a+b:

     return Master + index;
    

    Substitute the value of index computed on the previous line:

     return Master + 
       CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped) - Master;
    

    The two occurrences of Master cancel out, leaving

    OVERLAPPEDEX* FindOverlappedExFromOverlapped(
        OVERLAPPED *lpOverlapped)
    {
     return CONTAINING_RECORD(lpOverlapped, OVERLAPPEDEX, Overlapped);
    }
    

    And there you have it. By a series of purely mechanical transformations, we have rediscovered the technique of extending the OVERLAPPED structure.

  • The Old New Thing

    What happened to the return code from WinMain in 16-bit Windows?

    • 7 Comments

    Commenter S asks, "What happened to the return code from WinMain in a Windows 3.1 app?" After all, there was no GetExitCodeProcess function in 16-bit Windows.

    Basically, the exit code vanished into the ether.

    Unless you captured it.

    The Toolhelp library provided a low-level hook into various parts of the kernel, allowing you to monitor, among other things, the creation and destruction of tasks. That was how you captured the return code of a Windows program in 16-bit Windows.

    But if you didn't catch it as it happened, it was gone forever, lost in the ether.

  • The Old New Thing

    The OVERLAPPED associated with asynchronous I/O is passed by address, and you can take advantage of that

    • 26 Comments

    When you issue asynchronous I/O, the completion function or the I/O completion port receives, among other things, a pointer to the OVERLAPPED structure that the I/O was originally issued against. And that is your key to golden riches.

    If you need to associate information with the I/O operation, there's no obvious place to put it, so some people end up doing things like maintaining a master table which records all outstanding overlapped I/O as well as the additional information associated with that I/O. When each I/O completes, they look up the I/O in the master table to locate that additional information.

    But it's easier than that.

    Since the OVERLAPPED structure is passed by address, you can store your additional information alongside the OVERLAPPED structure:

    // in C
    struct OVERLAPPEDEX {
     OVERLAPPED o;
     CClient *AssociatedClient;
     CLIENTSTATE ClientState;
    };
    
    // or in C++
    struct OVERLAPPEDEX : OVERLAPPED {
     CClient *AssociatedClient;
     CLIENTSTATE ClientState;
    };
    

    When the I/O completes, you can use the CONTAINING_RECORD macro or just static_cast the LPOVERLAPPED to OVERLAPPEDEX* and bingo, there's your extra information right there. Of course, you have to know that the I/O that completed is one that was issued against an OVERLAPPEDEX structure instead of a plain OVERLAPPED structure, but there are ways of keeping track of that. If you're using a completion function, then only use an OVERLAPPEDEX-aware completion function when the OVERLAPPED structure is part of an OVERLAPPEDEX structure. If you're using an I/O completion port, then you can use the completion key or the OVERLAPPED.hEvent to distinguish OVERLAPPEDEX asynchronous I/O from boring OVERLAPPED I/O.

  • The Old New Thing

    Why does SHCOLUMNINFO have unusually tight packing?

    • 22 Comments

    Alternate title: News flash: Sometimes things happen by mistake

    rbirkby asks why the SHCOLUMNINFO structure has 1-byte packing. "Was the expectation that there would be so many columns in a details view that the saving would be worthwhile?"

    Hardly anything that clever or ingenious. It's just the consequence of a mistake.

    When the SHCOLUMNINFO structure was added to the header file in the Windows 2000 timeframe, it was added with no specific packing directive. But it turns out that there was a specific packing directive; it just wasn't obvious. Near the top of the shlobj.h header file was the following:

    #include <pshpack1.h>   /* Assume byte packing throughout */
    

    (There was of course a matching #include <poppack.h> at the bottom.) This set the default packing for the entire header file to byte packing instead of natural alignment.

    By the time this mistake was identified, it was too late. Windows 2000 had already shipped, byte packing and all. And once the code ships, it's done. You're stuck with it.

    Sorry.

  • The Old New Thing

    There is no interface for preventing your notification icon from being hidden

    • 38 Comments

    Yes, it's another installment of I bet somebody got a really nice bonus for that feature. A customer had this question for the Windows 7 team:

    Our program creates a notification icon, and we found that on Windows 7 it is hidden. It appears properly on all previous versions of Windows. What is the API to make our icon visible?

    First of all, I'd like to congratulate you on writing the most awesome program in the history of the universe.

    Unfortunately, Windows 7 was not prepared for your awesomeness, because there is no way to prevent your notification icon from being hidden. That's because if there were, then every other program (the ones that aren't as awesome as you) would use it, thereby causing your awesome icon to be lost among all the non-awesome ones.

    I'm sorry. You will just have to find some other medium for self-actualization.

  • The Old New Thing

    The subtleties of a Will Ferrell movie, and other observations from the in-flight entertainment on a Chinese airline

    • 20 Comments

    My flights to and from Beijing were on Hainan Airlines, a Chinese airline. One consequence of this is that Mandarin Chinese is the primary language of communication; English is a distant second. It also means that the in-flight movies are subtitled in Chinese, so if you can't read Chinese, you are restricted to movies in languages you understand.

    I wasn't interested in the English-language movies, although I did watch a little bit of "The Other Guys", a Will Ferrell vehicle. In one scene, Ferrell's character and his friend have dinner in a Chinese restaurant. Ferrell's character says to the waiter, "謝謝" which means "Thank you" in Mandarin. The waiter responds, "唔該" which means (in this context) "You're welcome" in Cantonese.¹

    Part of my brain wondered if this language mismatch was some sort of subtle commentary about the nature of Will Ferrell's character, that he's perhaps a bit of a poseur, or that he's out of place and doesn't realize it?

    And part of my brain couldn't believe that the other part of my brain used "subtle" and "Will Ferrell" in the same sentence.

    Anyway, the only other language I knew that was offered by the in-flight entertainment system was German. So I watched Willi und die Wunder dieser Welt, a movie-length version of the German children's television show Willi wills wissen. And watching the movie reminded me that Germans are obsessed with poop. During the course of the movie, you see a flying fox pooping, you see a polar bear pooping, and you investigate a Japanese toilet. I didn't stick around for the whole movie, but I wouldn't be surprised if you also saw a scorpion pooping in the final segment.

    (In the Canadian segment, somebody talks with Willi in heavily Canadian-accented German which was apparently learned phonetically. I could barely understand him. It reminded me of my high school German class and the students who couldn't shake their thick American accents.)

    Footnote

    ¹There are several phrases that roughly mean "Thank you" in Cantonese. The two primary ones are the aforementioned "唔該" and "多謝", and the rules governing proper use of each one are complicated.

  • The Old New Thing

    Microspeak: Informing a product

    • 11 Comments

    Microspeak is not always about changing a word from a verb to a noun or changing a noun to a verb or even changing a noun into a verb and then back into a noun. Sometimes it's about data compression.

    This testing won't inform RC, but we'll need it to inform an RTM release.

    First, you need to familiarize yourself with a less-used sense of the verb to inform, namely to guide or direct. A typical use would be something like "This data will inform the decision whether to continue with the original plan in Product Q." In other words, this data will be used to help decide whether to continue with the original plan.

    But of course, at Microsoft, it's all rush rush hurry hurry no time for slow sentences just get to the point. So we drop out a few words from the sentence and instead of informing a decision about something, we just inform the thing itself: "This data will inform Product Q."

    Therefore, the sentence at the start of this article is shorthand for "This testing won't inform [some sort of decision regarding] RC, but we'll need it to inform [that same decision in] an RTM release." In other words, the result of this testing won't have any effect on the release candidate, but it will be used to help make some sort of decision regarding the RTM release.

    Another citation, taken from an executive slide deck:

    • Other take-aways
      • What learnings can we get today to inform the release?

    I like how that brief snippet combines three pieces of Microspeak: take-away, learnings, and inform.

  • The Old New Thing

    Why are the Compression and Encryption options check boxes instead of radio buttons?

    • 55 Comments

    Tanveer Badar asks why the file properties Advanced dialog shows two checkboxes (compression and encryption) even though NTFS supports only one or the other at a time. "Why not have two radio buttons instead of these silly check boxes?"

    Actually, if you want radio buttons, you'd need three, one to cover the "neither" case. Let's look at what those radio buttons would look like:

    Compress contents to save disk space
    Encrypt contents to secure data
    Neither compress nor encrypt

    This takes an implementation detail (that NTFS currently does not support simultaneous compression and encryption) and elevates it to the user interface, so that it can provide maximum confusion to the user. "What a strange way of exposing compression and encryption. If I want to turn on encryption, I should just check a box that says 'Encryption'."

    The current implementation (two check boxes) matches user expectations, but then says "Sorry, we can't do that" if the user picks a combination that is not currently supported. But who knows, maybe someday, NTFS will support simultaneously encryption and compression, and all that will happen is that the error message goes away. You don't have to redesign the property sheet (and invalidate all the training materials that had been produced in the meantime).

    Either way, it's a leaky abstraction that sucks, but at least the suckiness isn't shoved in the user's face.

  • The Old New Thing

    Some notes on my trip to Beijing disguised as travel tips

    • 34 Comments

    Single-use tickets purchased from subway vending machines are valid only on the day of purchase for use in that station. Do not buy your return ticket at the same time as your outbound ticket because it will not work. This detail is clearly explained on the ticket that you receive after you have paid for it. (Also, the vending machine will ask you how many "sheets" you want. It's asking how many tickets you want.)

    Subway station names are printed in both Chinese and pinyin, but the pinyin omits the tone markers, which means that you will have no idea how you're supposed to pronounce the station name, should anybody ask you to say it. (See below.)

    Unlike in some cities, where the subway logo is bright and distinctive (and therefore easy to spot from a long way away), the logo for the Beijing subway is not consistent in color, which makes it hard to pick out from a busy streetscape. (One sign I saw used the high-visibility color scheme of beige-on-brown.) The logo is a monogram of the Latin letters "D" and "G", because the Mandarin word for "subway" is pronounced "dì-tiě". You might think they would use "D" and "T", but you're being too literal.

    Even worse, the Olympic Green station entrance nearest the convention center is so unobtrusively marked that if you aren't standing on the correct side of the building looking directly at it, you won't see the sign at all and you will end up spending an hour walking around Beijing looking for it. (For reference, the station entrance is opposite convention center entrance C-2.)

    When you fail to find the Olympic Green station entrance, you might consider going to a security guard booth (they are all over Beijing, the city being somewhat security-obsessed in a mostly-theater sort of way) holding a subway map with the Olympic Green station circled and asking, "火車?" because you don't know the Mandarin word for "subway" and have to make do by asking for the "train". Do not expect the security guard to have any clue what you're asking for. (Okay, I sort of undermined myself by pronouncing the first word in Mandarin but the second in Cantonese, because the two languages occupy similar portions of my brain and I often get them mixed up. But still, the first two cues...)

    In general, if you ask for directions but don't know more than a few dozen words of Mandarin, you're going to be in a world of hurt. My plan was to hold a map, point at it, and ask, "我在哪裡?" ("Where am I?"), and then calculate what direction to head based on the answer. Do not expect people to answer the question you ask. They will instead ask you other questions like "你去哪兒?" ("Where are you going?") but since answering that question is beyond your vocabulary, all you can do is repeat your original question, and they will give up, frustrated. Telling them that you're from the United States doesn't help, because they don't speak English.

    (The "Where am I?" technique worked great in Germany. The person I asked would locate me on the map, and even orient the map, then follow up with additional questions that I struggled to understand and answer.)

    It has been suggested that my profound difficulty in getting directions was exacerbated by the fact that I look like somebody who should know Mandarin. If I were some European-looking person, I wouldn't have had as much of a problem because, I'm told, Chinese people naturally assume that if they see a Chinese person on the street, that person speaks Mandarin. (The person may speak a regional language as well, but you can count on them speaking at least Mandarin.) If you look Chinese but don't speak Mandarin, then they will just get frustrated at this Chinese person who refuses to speak Chinese.

    This roughly matched my experience. Pretty much everybody assumed that I spoke Mandarin. The exception? The street hustlers and scam artists. They had me pegged for a foreigner.

    By the way, I eventually solved my problem by looking for a bank. The manager on duty spoke some English, and combined with my rudimentary Mandarin, Cantonese, and Hokkien (thank heaven for cognates), I was able to get the information I needed. Of course, by that time, I had wandered so far astray that the nearest subway station was nowhere near the one I was looking for originally!

    Oh, and do not expect the hotel concierge to give you an up-to-date map. The information on the map had not been updated to take into account recent subway expansion, which means that its directions on how to get to points of interest were unnecessarily cumbersome. (What's more, the hotel itself did not appear on the map, because it was covered by an inset of the Olympic stadiums. This makes it hard to orient yourself once you step outside.) In fact, most of the time the street I was standing on didn't appear anywhere on the map (or at least I couldn't find it), so I had no clue where I was.

    The air pollution in Beijing is legendary. The week before I arrived, the United States Embassy declared that the Air Pollution Index in Beijing topped 500, earning the rating "Hazardous for all people." On the other hand, the official Chinese government pollution index was "only" 341. (Mind you, 341 is still off the chart. For grins, compare the scales used by mainland China, Hong Kong, and Malaysia.) You don't really notice the effect of the air pollution until you return to your hotel at the end of a day outdoors and wonder why your throat is sore and you feel like you spent the day in a smoke-filled bar.

    Walking through the Forbidden City makes you feel like Frodo in The Lord of the Rings. You fight your way across a courtyard to reach the building at the other end. Upon reaching the building, you cross the gate and before you lies... another seemingly-identical courtyard. This repeats about twenty-five bazillion times. They should really call it the Forbidden County.

    If you're trying to get to the Summer Palace, do not accidentally leave your map in your hotel room thinking that "This is such a prominent tourist location it must certainly have adequate signage, or at least be present on the 'things nearby' map at the station." The only nearby attraction on the map at the station is the Old Summer Palace. The only directions to the Summer Palace is a single arrow on the plaza level of the station. The arrow tells you to Frogger across a busy four-lane street (and over the fence). If you go to the crosswalk some distance away, you end up wandering in the wrong direction for a while and turning around when you figure "This can't be right." On the way back, you try a slight variation on the path out of the station and notice that there's a directional sign for the Summer Palace facing away from the station. (I.e., only people returning to the station can see it.) You follow that arrow and wonder if you're on the right street since it's pretty much an empty street as far as the eye can see, but you gradually find tour buses so you figure you're getting closer. You then find the Summer Palace parking lot and say, "Cool, there must certainly be a sign to the Summer Palace from the parking lot" but you'd be wrong. You then see a tour group in the distance and take your chances that they are going into the Summer Palace (rather than returning) and follow them down an unmarked side street, then another unmarked side street, before spotting the entrance to the Summer Palace.

    China clearly has yet to figure out this "foreign tourists not visiting as part of a guided tour" thing. My guess is that since it was a closed country for so long, there was no need for directional signage because all foreign tourists were necessarily accompanied by a government-approved tour guide, and the government-approved tour guide knows how to get there.

    The Summer Palace is very scenic. I bet it's even prettier in the summer.

    Orthographic note: Out of habit, I use traditional characters even though China uses simplified characters. With one exception, I'm reasonably comfortable with reading both sets of characters, though when writing I prefer traditional. Traditional characters feel more formal and "standard" to me, whereas simplified characters feel too casual for normal use. (Like writing "u" and "b4" instead of "you" and "before".)

    The one exception? The character for "car": 車. The simplified version is 车 which to me is unrecognizable because it destroys the ideographic representation of the top view of a car. (The central box is the body of the car, and the horizontal bars at the top and bottom are the axles. The simplified version is just a number "4" with some extra bars.)

    Mostly-theater: X-ray machines are omnipresent, but they are largely ignored. People just walk right on past them. The TechEd conference I attended had three security checkpoints: One with an X-ray machine and a metal detector, and two additional checkpoints where security personnel checked that you had a valid badge before letting you pass. What nobody appeared to notice is that if you took the publically-accessible skybridge from the Intercontinental Beijing Beichen hotel next door, you could enter the inner sanctum of the conference without ever passing through a security checkpoint.

Page 2 of 4 (31 items) 1234