July, 2012

  • The Old New Thing

    Reading the output of a command into a batch file variable

    • 28 Comments

    It's Day Two of Batch File Week. Don't worry, it'll be over in a few days.

    There is no obvious way to read the output of a command into a batch file variable. In unix-style shells, this is done via backquoting.

    x=`somecommand`
    

    The Windows command processor does not have direct backquoting, but you can fake it by abusing the FOR command. Here's the evolution:

    The /F flag to the FOR command says that it should open the file you pass in parentheses and set the loop variable to the contents of each line.

    for /f %%i in (words.txt) do echo [%%i]
    

    The loop variable in the FOR command takes one percent sign if you are executing it directly from the command prompt, but two percent signs if you are executing it from a batch file. I'm going to assume you're writing a batch file, so if you want to practice from the command line, remember to collapse the double percent signs to singles.

    I'm cheating here because I know that words.txt contains one word per line. By default, the FOR command sets the loop variable to the first word of each line. If you want to capture the entire line, you need to change the delimiter.

    for /f "delims=" %%i in (names.txt) do echo [%%i]
    

    There are other options for capturing say the first and third word or whatever. See the FOR /? online help for details.

    Now, parsing files is not what we want, but it's closer. You can put the file name in single quotes to say "Instead of opening this file and reading the contents, I want you to run this command and read the contents." For example, suppose you have a program called printappdir which outputs a directory, and you want a batch file that changes to that directory.

    for /f "delims=" %%i in ('printappdir') do cd "%%i"
    

    We ask the FOR command to run the printappdir program and execute the command cd "%%i" for each line of output. Since the program has only one line of output, the loop executes only once, and the result is that the directory is changed to the path that the printappdir program prints.

    If you want to capture the output into a variable, just update the action:

    for /f %%i in ('printappdir') do set RESULT=%%i
    echo The directory is %RESULT%
    

    If the command has multiple lines of output, then this will end up saving only the last line, since previous lines get overwritten by subsequent iterations.

    But what if the line you want to save isn't the last line? Or what if you don't want the entire line?

    If the command has multiple lines of output and you're interested only in a particular one, you can filter it in the FOR command itself...

    for /f "tokens=1-2,14" %%i in ('ipconfig') do ^
        if "%%i %%j"=="IPv4 Address." set IPADDR=%%k
    

    The above command asked to execute the ipconfig command and extract the first, second, and fourteenth words into loop variable starting with %i. In other words, %i gets the first word, %j gets the second word, and %k gets the fourteenth word. (Exercise: What if you want to extract more than 26 words?)

    The loop then checks each line to see if it begins with "IPv4 Address.", and if so, it saves the fourteenth word (the IP address itself) into the IPADDR variable.

    How did I know that the IP address was the fourteenth word? I counted!

       IPv4 Address. . . . . . . . . . . : 192.168.1.1
       ---- -------- - - - - - - - - - - - -----------
         1      2    3 4 5 6 7 8 9  11  13      14
                                  10  12
    

    That's also why my test includes the period after Address: The first dot comes right after the word Address without an intervening space, so it's considered part of the second "word".

    Somebody thought having the eye-catching dots would look pretty, but didn't think about how it makes parsing a real pain in the butt. (Note also that the above script works only for US-English systems, since the phrase IPv4 Address will change based on your current language.)

    Instead of doing the searching yourself, you can have another program do the filtering, which is important if the parsing you want is beyond the command prompt's abilities.

    for /f "tokens=14" %%i in ('ipconfig ^| findstr /C:"IPv4 Address"') do ^
      set IPADDR=%%i
    

    This alternate version makes the findstr program do the heavy lifting, and then saves the fourteenth word. (But this version will get fooled by the line Autoconfiguration IPv4 Address.)

    Yes I know that you can do this in PowerShell

    foreach ($i in Get-WmiObject Win32_NetworkAdapterConfiguration) {
      if ($i.IPaddress) { $i.IPaddress[0] }
    }
    

    You're kind of missing the point of Batch File Week.

  • The Old New Thing

    Why don't any commands work after I run my batch file? I'm told that they are not recognized as an internal or external command, operable program, or batch file.

    • 30 Comments

    I sort of forgot to celebrate CLR Week last year, so let's say that CLR week is "on hiatus" until next year. To fill the summertime time slot, I'm going to burn off a busted pilot: This week is Batch File Week 2012. Remember, nobody actually enjoys batch programming. It's just something you have to put up with in order to get something done. Batch programming is the COBOL of Windows. (Who knows, if people actually like Batch File Week [fat chance], maybe it'll come back as a regular series.)

    We'll open Batch File Week with a simple puzzle.

    A customer reported that after running their batch file, almost no commands worked any more!

    C:\> awesomebatchfile.bat
    ... awesome batch file does its work ...
    
    C:\> reg query "HKLM\Software\Clients\Mail" /ve
    'reg' is not recognized as an internal or external command,
    operable program or batch file.
    

    Wha? Maybe I can run regedit.

    C:\> regedit
    'regedit' is not recognized as an internal or external command,
    operable program or batch file.
    

    OMG OMG OMG OMG.

    C:\> notepad
    'notepad' is not recognized as an internal or external command,
    operable program or batch file.
    

    Okay, first, sit down and take a deep breath. Maybe take a Chill Pill.

    My first question was "Does awesomebatchfile.bat modify the PATH variable?" (This was, strictly speaking, a psychic debugging question, but a rather obvious one.)

    The customer replied, "Nope. Here, I'll send you the whole thing."

    And there it was, right there at the top of awesomebatchfile.bat:

    set path=C:\awesomedir
    if NOT "%1"=="" set path=%1
    cd /d %path%
    echo Awesomeness commencing in the %path% directory!
    ...
    

    The customer figured it would be convenient to have a variable called path, unaware that this variable has special meaning to the command interpreter. The customer didn't make the connection that their seemingly private variable called path was connected to the system variable of the same name (but by convention capitalized as PATH).

  • The Old New Thing

    A brief and also incomplete history of Windows localization

    • 32 Comments
    The process by which Windows has been localized has changed over the years.

    Back in the days of 16-bit Windows, Windows was developed with a single target language: English.

    Just English.

    After Windows was complete and masters were sent off to the factory for duplication, the development team handed the source code over to the localization teams. "Hey, by the way, we shipped a new version of Windows. Go localize it, will ya?"

    While the code that was written for the English version was careful to put localizable content in resources, there were often English-specific assumptions hard-coded into the source code. For example, it may have assumed that the text reading direction was left-to-right or assumed that a single character fit in a single byte. (Unicode hadn't been invented yet.)

    The first assumption is not true for languages such as Hebrew and Arabic (which read right-to-left), and to a lesser degree Chinese and Japanese (which read top-to-bottom in certain contexts). The second assumption is not true for languages like Chinese, Japanese, and Korean, which use DBCS (double-byte character sets).

    The localization teams made the necessary code changes to make Windows work in these other locales and merged them back into the master code base. The result was that there were three different versions of the code for Windows, commonly known as Western, Middle-East, and Far-East. If you wanted Windows to support Chinese, you had to buy the Far-East version of Windows. And since the code was different for the three versions, they had different sets of bugs, and workarounds for one version didn't always work on the others. (Patches didn't exist back then, there being no mechanism for distributing them.)

    If you ran into a problem with a Western language, like say, German, then you were out of luck, since there was no German Windows code base; it used the same Western code base. Windows 95 tried out a crazy idea: Translate Windows into German during the development cycle, to help catch these only-on-German problems while there was still time to do something about it. This, of course, created significant additional expense, since you had to have translators available throughout the product cycle instead of hiring them just once at the end. I remember catching a few translation errors during Windows 95: A menu item Sort was translated as Art (as in "What sort of person would do this?") rather than Sortieren ("put in a prearranged order"). And a command line tool asked the user a yes/no question, promting "J/N" (Ja/Nein), but if you wanted to answer in the affirmative, you had to type "Y".

    The short version of the answer to the question "Why can't the localizers change the code if they have to?" is "Because the code already shipped. What are you going to do, recall every copy of Windows?"

    At least in Windows 95, the prohibition on changing code was violated if circumstances truly demanded them, but doing so was very expensive. The only one I can think of is the change to remove the time zone highlighting from the world map. And the change was done in the least intrusive possible way: Patching four bytes in the binary to make the highlight and not-highlight colors the same. You dare not do something like introduce a new variable; who knows what kinds of havoc could result!

    Having all these different versions of Windows made servicing very difficult, because you had to develop and test a different patch for each code base. Over the years, the Windows team has developed techniques for identifying these potential localization problems earlier in the development cycle. For a time, Windows was "early-localized" into German and Japanese, so as to cover the Western and Far-East scenarios. Arabic was added later, expanding coverage to the Mid-East cases, and Hindi was added in Windows 7 to cover languages which are Unicode-only.

    Translating each internal build of Windows has its pros and cons: The advantage is that it can find issues when there is still time to make code changes to address them. The disadvantage is that code can change while you are localizing, and those code changes can invalidate the work you've done so far, or render it pointless. For example, somebody might edit a dialog you already spent time translating, forcing you to go back and re-translate it, or at least verify that the old translation still works. Somebody might take a string that you translated and start using it in a new way. Unless they let you know about the new purpose, you won't know that the translation needs to be re-evaluated and possibly revised.

    The localization folks came up with a clever solution which gets most of the benefits while avoiding most of the drawbacks: They invented pseudo-localization, which simulates what Michael Kaplan calls "an eager and hardworking yet naïve intern localizer who is going to translate every single string." This was so successful that they hired a few more naïve intern localizers, one which performed "Mirrored pseudo-localization" (covering languages which read right-to-left) and "East Asian pseudo-localization" (covering Chinese, Japanese, and Korean).

    But the rule prohibiting code changes remains in effect. Changing any code resets escrow, which means that the ship countdown clock gets reset back to its original value and all the testing performed up until that point needs to be redone in order to verify that the change did not affect them.

  • The Old New Thing

    Why do some font names begin with an at-sign?

    • 21 Comments

    It was a simple question.

    For some reason, my font selection dialog (CFont­Dialog) shows a bunch of font names beginning with the at-sign (@). These fonts don't work correctly if I use them. Any idea what they are? (I tried searching the Internet, but search engines don't seem to let you search for @ so it's hard to make much headway.)

    (And that's why I wrote "at-sign" in the subject instead of using the @ character.)

    Fonts which begin with an @-sign are vertically-oriented fonts. They are used in languages like Chinese, Japanese, and (less often) Korean. The idea is that if you want to generate vertical text, you start with the horizontal version of the font and compose your document, then switch to the vertical version for printing.

     x x x 

    I wasn't able to detect that your browser supports the @SimSun font, so I'll give an example with fake Chinese characters. Pretend that the shapes and Latin letters are actually Chinese characters. First, you compose your document with the horizontal font:

    ▴❤❦Quo123▴‌̥ 

    When it's time to print, switch to the vertical version of the font.

    ◀❥❧℺ᴝᴑ123◀°

    Hm, it looks like the Chinese characters got rotated 90° to the left, so they're all lying on their side. The result is not really all that readable, but wait, here's the trick: After the paper comes out of the printer, rotate the paper right 90°:

    ◀❥❧℺ᴝᴑ123◀°

    Notice that the vertical version of a font does not simply rotate every character 90°. Non-CJK characters typically remain in their original orientation (which means that when you turn the paper, they will come out rotated). And some CJK characters change form between horizontal and vertical variants, like the period in the example above, so it's not a simple rule like "rotate all CJK characters and leave non-CJK characters alone."

    This is basically a hack to get rudimentary vertical font support in software that doesn't support vertical text natively. (Web browsers support vertical text natively with the proposed writing-mode property.)

    If you don't want vertical fonts to show up in your font dialog, pass the CF_NO­VERT­FONTS flag. Of course, if you pass that flag, then your users won't be able to use the vertical-font trick any more.

    Supplemental reading which served as the source material for this article:

    Bonus head-to-head competition: You can read how Michael Kaplan blogged this exact same subject in his own Kaplanesque way.

  • The Old New Thing

    Why doesn't the Low Disk Space warning balloon show up as soon as I run low on disk space

    • 66 Comments

    A customer reported an issue with the title "The notification balloon for Low Disk Space does not appear even if the free disk is very low." They provided the following steps:

    • Install Windows 7 64-bit on a SATA drive.
    • Copy files to the system drive until disk space becomes low.
    • Observe that the notification balloon for Low Disk Space does not immediately appear.
    • The balloon appears approximately ten minutes later.

    You read through the steps nodding, "uh huh, uh huh", and then you get to the last step and you say, "Wait a second, the subject of your report was that the balloon doesn't appear at all, and now you're saying that it appears after ten minutes. So it does appear after all. What is the problem?"

    The customer explained that on earlier versions of Windows, the Low Disk Space warning balloon appeared within one minute, whereas in Windows 7 it can take up to ten minutes for the balloon to appear.

    Yup, that's right.

    In previous versions of Windows, Explorer checked for low disk space once a minute. The Windows performance folks requested that the shell reduce the frequency of checks to improve overall system performance, and the shell team agreed to reduce the frequency to once every ten minutes. (The performance team made other suggestions to reduce the impact of that code that runs every ten minutes.)

    So yes, in Windows 7, it may take up to ten minutes for Explorer to report that you are low on disk space. But Explorer never promised that those reports would be timely. Or that they would even appear in the first place. The behavior is not contractual; it's just a courtesy notification.

    Related: How full does a hard drive have to get before Explorer will start getting concerned? and How do I disable the low disk space notifications?

  • The Old New Thing

    Why hasn't the API hook mechanism for x64 been standardized like it was for x86?

    • 28 Comments
    Joshua posted to the Suggestion Box, "Around the time of WinXP SP2 x86, the API hook mechanism was standardized. Why wasn't the same thing done for x64?"

    Who said it was standardized for x86?

    Hooking APIs is not supported by Windows. There may be specific interfaces that expose hooks (like Co­Register­Initialize­Spy to let you monitor calls to CoInitialize and CoUninitialize, and Set­Windows­Hook­Ex to let you hook various window manager operations) but there is no supported general API hooking mechanism provided by the operating system.

    So I don't know where you got that idea from.

  • The Old New Thing

    How your taskbar auto-hide settings can keep getting overwritten

    • 23 Comments
    A customer reported that they were observing that some users were finding their taskbar set to auto-hide even though the standard configuration in the company is for the auto-hide feature to be disabled. Going into Taskbar Properties shows Auto-hide the taskbar checked. None of the users had changed their setting to auto-hide manually, so the question was raised to the Windows team, "Are there any cases where Explorer will set the auto-hide setting on its own?"

    Explorer does not set the auto-hide checkbox on its own. Now, the taskbar does auto-hide even when the setting is unchecked if it detects that the application is trying to go full-screen, say, in order to show a slide show or play World of Warcraft. But that doesn't check the check-box.

    Further investigation revealed that the check-box was being checked programmatically by one of the programs that the company used. And it wasn't custom software but a commercial product which targets the corporate market.

    The customer reported back that the problem was sporadic. They could not reproduce it consistently.

    My guess is that the application in question was trying to enable auto-hide temporarily for whatever reason. At program startup, it checks the current auto-hide setting, and if it's off, it programmatically turns auto-hide on.

    previousState = IsAutoHideTaskbarEnabled();
    SetAutoHideTaskbar(true);
    

    When the program exits, it restores the original setting.

    SetAutoHideTaskbar(previousState);
    

    This is a highly fragile solution for several reasons: What if the application crashes before it can restore the setting?

    What if two people did this?

    1. Initially, auto-hide is off.
    2. Program A remembers that auto-hide was off and sets it on.
    3. Program B remembers that auto-hide was on and sets it on.
    4. Program A exits and restores auto-hide to off.

    Oops, now we have a problem: Program B wants auto-hide on, but Program A just turned it off.

    1. Program B exits and restores auto-hide to on.

    Oops, the auto-hide setting was left in the 'on' state after everybody thought they had restored it.

    As a special case of What if two people did this?, the Program B could be the Taskbar Properties page itself. While your program is running, the user goes to Taskbar Properties and sees that the checkbox is set incorrectly. Maybe they go in and "fix it", and now Program A is running with a visible taskbar.

    What if the application tries to restore the state after Explorer has already saved its settings? When the user logs off, all processes are told to clean up their toys and to go bed. In response to WM_ENDSESSION, Explorer saves out its settings and calls it a night. What if this happens before the application programmatically unchecks the box? Explorer says, "Okay, I unchecked the box." But Explorer already saved out its settings; these updated settings aren't going to be saved again.

    This is what happens when you expose a global setting programmatically. People see the setting and think that twiddling it will solve their problem instead of looking for a local solution to their local problem, in this case creating a fullscreen window that covers the taskbar.

  • The Old New Thing

    The continuing battle between people who offer a service and others who want to hack into the service

    • 32 Comments

    In the history of the Internet, there have been many cases of one company providing a service, and others trying to piggyback off the service through a nonstandard client. The result is usually a back-and-forth where the provider changes the interface, the piggybacker reverse-engineers the interface, back and forth, until one side finally gives up.

    Once upon a time, there was one company with a well-known service, and another company that was piggybacking off it. (I first heard this story from somebody who worked at the piggybacking company.) The back-and-forth continued for several rounds, until the provider made a change to the interface that ended the game: They exploited a buffer overflow bug in their own client. The server sent an intentional buffer overflow to the client, resulting in the client being pwned by the server. I'm not sure what happened next, but presumably the server sent some exploit code to the client and waited for the client to respond in a manner that confirmed that the exploit had executed.

    With that discovery, the people from the piggybacking company gave up. They weren't going to introduce an intentional security flaw into their application. The service provider could send not only the exploit but also some code to detect and disable the rogue client.

    By an amazing stroke of good fortune, I happened to also hear the story of this battle from somebody who worked at the provider. He said that they had a lot of fun fighting this particular battle and particularly enjoyed timing the releases so they caused maximum inconvenience for their adversaries, like, for example, 2am on Saturday.

    Reminder: The ground rules prohibit "trying to guess the identity of a program whose name I did not reveal."

  • The Old New Thing

    Why doesn't RealGetWindowClass return the real window class for my superclass?

    • 41 Comments

    A customer was reporting that the Real­Get­Window­Class function was not reporting the base window class of their superclass. (Error checking has been elided for expository purposes.)

    // Get the static window class window procedure
    WNDCLASS wc;
    GetClassInfo(NULL, TEXT("static"), &wc);
    WNDPROC StaticWndProc = wc.lpfnWndProc;
    
    // Build our derived class
    wc.lpfnWndProc = AwesomeWndProc;
    wc.hInstance = g_hinst;
    wc.lpszClassName = TEXT("AwesomeWindow");
    RegisterClass(&wc);
    
    LRESULT CALLBACK AwesomeWndProc(HWND hwnd, UINT uMsg,
                                    WPARAM wParam, LPARAM lParam)
    {
        TCHAR szClass[128];
        RealGetWindowClass(hwnd, szClass, 128);
        ASSERT(strcmp(szClass, TEXT("static")) == 0);
    
        switch (uMsg) { ... }
    
        return CallWindowProc(StaticWndProc, hwnd, uMsg, wParam, lParam);
    }
    

    The customer found that the assertion fails, returning a window class name of "AwesomeWindow" instead of "static". "I thought the point of RealGetWindowClass was to dig through the superclassing to find the base class. But it's not returning the base class."

    That's right, because you haven't told it what the base class is yet!

    "What do you mean I haven't told it? It's right there at the end of my function: CallWindowProc(StaticWndProc)."

    Yeah, but that line of code hasn't executed yet. The external behavior of your program is like this:

    WNDCLASS wc;
    wc.style = (something);
    wc.lpfnWndProc = AwesomeWndProc;
    wc.cbClsExtra = (something);
    wc.cbWndExtra = (something);
    wc.hInstance = g_hinst;
    wc.hIcon = (something);
    wc.hCursor = (something);
    wc.hbrBackground = (something);
    wc.lpszMenuName = (something);
    wc.lpszClassName = TEXT("AwesomeWindow");
    RegisterClass(&wc);
    
    
    LRESULT CALLBACK AwesomeWndProc(HWND hwnd, UINT uMsg,
                                    WPARAM wParam, LPARAM lParam)
    {
        TCHAR szClass[128];
        RealGetWindowClass(hwnd, szClass, 128);
        ASSERT(strcmp(szClass, TEXT("static")) == 0);
    
        // ... it doesn't matter what goes here
        // because it hasn't executed yet ...
    

    The window manager isn't clairvoyant. It doesn't know that AwesomeWndProc is going to do a CallWindowProc(StaticWndProc) in the future. All it knows is that somebody registered a class, and then in response to its very first message, that class asked, "Hey, you're so smart, tell me what my base class is."

    The window manager says, "Dude, you haven't shown me any base class yet. So I'm just going to say that you are your own base class."

    Since anything can go into the "... it doesn't matter what goes here ...", we can demonstrate that the window manager cannot possibly know what you're going to pass to CallWindowProc by rewriting it like this:

    // Get the static window class window procedure
    WNDCLASS wc;
    GetClassInfo(NULL, TEXT("static"), &wc);
    WNDPROC StaticWndProc = wc.lpfnWndProc;
    
    // Build our class
    wc.lpfnWndProc = AwesomeWndProc;
    wc.hInstance = g_hinst;
    wc.lpszClassName = TEXT("AwesomeWindow");
    RegisterClass(&wc);
    
    LRESULT CALLBACK AwesomeWndProc(HWND hwnd, UINT uMsg,
                                    WPARAM wParam, LPARAM lParam)
    {
        TCHAR szClass[128];
        RealGetWindowClass(hwnd, szClass, 128);
        ASSERT(strcmp(szClass, TEXT("static")) == 0);
    
        switch (uMsg) { ... }
    
        // Psych! You thought that when I asked for StaticWndProc
        // I was going to be a superclass of "static", but in fact
        // I'm just a regular boring window class.
        return DefWindowProc(hwnd, uMsg, wParam, lParam);
    }
    

    If you felt really crazy, you could do this:

    // Get the button window class procedure
    WNDCLASS wcButton;
    GetClassInfo(NULL, TEXT("button"), &wcButton);
    WNDPROC ButtonWndProc = wcButton.lpfnWndProc;
    
    // Get the static window class window procedure
    WNDCLASS wc;
    GetClassInfo(NULL, TEXT("static"), &wc);
    WNDPROC StaticWndProc = wc.lpfnWndProc;
    
    // Build our class
    wc.lpfnWndProc = AwesomeWndProc;
    wc.hInstance = g_hinst;
    wc.lpszClassName = TEXT("AwesomeWindow");
    wc.cbWndExtra = max(wc.cbWndExtra, wcButton.cbWndExtra);
    wc.cbClsExtra = max(wc.cbClsExtra, wcButton.cbClsExtra);
    RegisterClass(&wc);
    
    LRESULT CALLBACK AwesomeWndProc(HWND hwnd, UINT uMsg,
                                    WPARAM wParam, LPARAM lParam)
    {
        TCHAR szClass[128];
        RealGetWindowClass(hwnd, szClass, 128);
        ASSERT(strcmp(szClass, TEXT("static")) == 0);
    
        switch (uMsg) { ... }
    
        // Decide at the last moment what we are.
        static WNDPROC BaseClass = nullptr;
        if (BaseClass == nullptr)
        {
            BaseClass = rand() % 2 ? StaticWndProc : ButtonWndProc;
            // Or if you are particularly perverse,
            // BaseClass = radioactive_decay_has_occurred() ?
            //                StaticWndProc : ButtonWndProc;
        }
        return CallWindowProc(BaseClass, hwnd, uMsg, wParam, lParam);
    }
    

    Since the code to decide the base class hasn't run yet, the window manager will have to use that time machine that the research division has been working on.

  • The Old New Thing

    One way to make sure you pass an array of the correct size

    • 61 Comments

    Another entry in the very sporadic series of "very strange code I've seen." The code has been changed to protect the guilty, but the essence has been preserved.

    class Store
    {
    public:
        // Retrieve "count" colors from item "itemId" into "values"
        bool GetItemColors(int itemId, int count, COLORREF *values);
    
        // Set "count" colors from "values" into item "itemId"
        bool SetItemColors(int itemId, int count, const COLORREF *values);
    };
    
    bool CopyUpToFourColors(Store *store1, Store *store2, int itemId, int count)
    {
        COLORREF size1[1];
        COLORREF size2[2];
        COLORREF size3[3];
        COLORREF size4[4];
    
        int *buffer = ((count == 1) ? size1 :
                      ((count == 2) ? size2 :
                      ((count == 3) ? size3 :
                      ((count == 4) ? size4 :
                                      nullptr))));
    
        if (buffer == nullptr)
            return false;
    
        if (!store1->GetItemColors(itemId, count, buffer))
            return false;
    
        if (!store2->SetItemColors(itemId, count, buffer))
            return false;
    
        return true;
    }
    
Page 1 of 3 (24 items) 123