August, 2012

  • The Old New Thing

    Exiting a batch file without exiting the command shell -and- batch file subroutines

    • 17 Comments

    Prepare your party hats: Batch File Week is almost over.

    In your batch file, you may want to exit batch file processing (say, you encountered an error and want to give up), but if you use the exit command, that will exit the entire command processor. Which is probably not what you intended.

    Batch file processing ends when execution reaches the end of the batch file. The trick therefore is to use the goto command to jump to a label right before the end of the file, so that execution "falls off the end".

    @echo off
    if "%1"=="" echo You must provide a file name.&goto end
    if NOT EXIST "\\server\backup\%USERNAME%\nul" mkdir "\\server\backup\%USERNAME%"
    if NOT EXIST "\\server\backup\%USERNAME%\nul" echo Unable to create output directory.&goto end
    copy "%1" "\\server\backup\%USERNAME%"
    :end
    

    Here, there are two places where we abandon batch file execution. One is on an invalid parameter, and another is if the output directory couldn't be created (or if it isn't a directory at all).

    The batch command interpreter provides a courtesy label to simply this technique: The special goto target goto :eof (with the colon) jumps to the end of the batch file. It's as if every batch file had a hidden goto label called :eof on the very last line.

    The goto :eof trick becomes even more handy when you start playing with batch file subroutines. Okay, let's back up: Batch file subroutines?

    By using the call command, a batch file can invoke another batch file and regain control after that other batch file returns. (If you forget the call, then control does not return. In other words, the default mode for batch file invocation is chain.) In other words, the call command lets you invoke another batch file as a subroutine. The command line parameters are received by the other batch file as the usual numbered parameters %1, %2, etc.

    It's annoying having to put every subroutine inside its own batch file, so the command interpreter folks added a way to call a subroutine inside the same batch file. The syntax for this is call :label parameter parameter parameter. This is logically equivalent to a batch file recursively calling itself, except that execution begins at the specified label instead of the first line of the file. (It's as if a secret goto label were added to the top of the file.)

    And since it is a batch file, execution of the called subroutine ends when execution falls off the end of the file. And that's where the special goto target comes in handy. At the end of your subroutine, you can jump to the end of the batch file (so that execution falls off the end) by doing a goto :eof.

    In other words, goto :eof is the return statement for batch file subroutines.

    Let's take it for a spin:

    @echo off
    call :subroutine a b c
    call :subroutine d e f
    goto :eof
    
    :subroutine
    echo My parameters are 1=%1, 2=%2, 3=%3
    goto :eof
    

    That final goto :eof is redundant, but it's probably a good habit to get into, like putting a break; at the end of your last case.

    The subroutine technique is handy even if you don't really care about the subroutine, because stashing the arguments into the %n parameters lets you use the tilde operators to process the inbound parameter.

    @echo off
    call :printfilesize "C:\Program Files\Windows NT\Accessories\wordpad.exe"
    goto :eof
    :printfilesize
    echo The size of %1 is %~z1
    goto :eof
    

    Okay, this isn't actually much of a handy trick because you can also do it without a subroutine:

    @echo off
    for %%i ^
    in ("C:\Program Files\Windows NT\Accessories\wordpad.exe") ^
    do echo The size of %%i is %%~zi
    

    On the other hand, the subroutine trick combines well with the FOR command, since it lets you put complex content in the loop body without having to mess with delayed expansion:

    @echo off
    setlocal
    set DISKSIZE=1474560
    set CLUSTER=512
    set DISKS=1
    set TOTAL=0
    for %%i in (*) do call :onefile "%%i"
    set /a DISKS=DISKS+1
    echo Total disks required: %DISKS%
    endlocal
    goto :eof
    
    :onefile
    set /a SIZE=((%~z1 + CLUSTER - 1) / CLUSTER) * CLUSTER
    
    if %SIZE% GEQ %DISKSIZE% (
        echo File %1 does not fit on a floppy - skipped
        goto :eof
    )
    
    set /a TOTAL=TOTAL+SIZE
    if %TOTAL% GEQ %DISKSIZE% (
        echo ---- need another disk
        set /a DISKS=DISKS+1
        set /a TOTAL=SIZE
    )
    echo copy %1
    goto :eof
    

    This program calculates the number of floppy disks it would take to copy the contents of the current directory without compression.

    The setlocal command takes a snapshot of the environment for restoration when we perform the endlocal at the end. That will clean up our temporary variables when we're done.

    The first two variables are parameters for the calculation, namely the disk capacity and the cluster size. (We're assuming that the root directory can hold all the files we may ultimately copy. Hey, this is just a demonstration, not a real program.)

    The next two variables are our running total of the number of disks we've used so far, and how many bytes we've used on the last disk.

    The for command iterates over all the files in the current directory. For each one, we call :onefile with the file name.

    The :onefile subroutine does all the real work. First, it takes the file size %~z1 and rounds it up to the nearest cluster. It then sees if that size is larger than a floppy disk; if so, then we're doomed, so we just skip the file. Otherwise, we add the file to the current disk and see if it fits. If not, then we declare the disk full and put the file on a brand new disk.

    After the loop is complete, we print the number of floppy disks we calculated.

    (This algorithm erroneously reports that no files require one disk. Fixing that is left as an exercise.)

    There's your quick introduction to the secret :eof label and batch file subroutines.

    [Raymond is currently away; this message was pre-recorded.]

  • The Old New Thing

    FORFILES, for your fancier batch file enumeration needs

    • 20 Comments

    Crack open open the champagne: Batch File Week is finally over!

    Variations on the for /f %%i in ('dir /b ...') will let you repeat an operation on the contents of a directory, possibly even recursively if you add the /s option, with some basic attribute-level filtering if you add the /a or /a- flags.

    For your fancy recursive file operations, there's a tool called FORFILES which iterates through the contents of a directory (recursively if requested), executing a command on each item it finds. It also has additional filtering capability, like selecting files based on their last-modified time. For example, you could copy all files in the current directory which were modified today:

    forfiles /D +0 /c "cmd /c copy @file \\server\today"
    

    Unfortuantely, the /D option is not as flexible as one might like. For example, while it can pick files modified today, it can't pick files modified in the last week, because the relative-date-picker knows only how to pick files modified on or before a date in the past or files modified on or after a date in the future. (Who the heck wants to operate on files modified in the future? Except perhaps the Microsoft Research folks who are working on that time machine.)

    You can type FORFILES /? for more information on what you can do (and by seeing what's omitted, what you can't do).

    If the command you want to execute is rather long, you can offload it back into the batch file being executed:

    @echo off
    if "%1"=="/callback" goto callback
    forfiles /D +0 /c "cmd /c call "%~f0" /callback @isdir @file @fsize"
    goto :eof
    :callback
    rem %2 = @isdir
    rem %3 = @file
    rem %4 = @fsize
    if %2==TRUE echo Skipping directory %3.&goto :eof
    echo Copying file %3 to \\server\today (%4 bytes)
    

    One gotcha here is that since each command runs in a sub-shell, it can read environment variables, but any modifications it makes to environment variables will be lost since the command is modifying only its local environment variables. A workaround for this is to use FORFILES to select the data to operate on, but use FOR to actually perform the operation. Since FOR runs inside the main command interpreter, it can modify environment variables.

    set TOTALSIZE=0
    for /f %%i in ('forfiles /d +0 /c "cmd /c if @isdir==FALSE echo @fsize"') ^
    do set /a TOTALSIZE=TOTALSIZE + %%i
    

    Here, we use FORFILES to enumerate all the files (not directories) modified today and print their sizes. We wrap this inside a FOR which reads the sizes and adds them up.

    If the operation you want to perform on each file is complex, you can of course offload it into a subroutine call.

    for /f %%i ^
    in ('forfiles /d +0 /c "cmd /c if @isdir==FALSE echo @fsize"') ^
    do call :subroutine %%i
    

    I'm cheating here because I know that @fsize doesn't contain spaces. If you are processing file names, then you need to be more careful.

    for /f "tokens=*" %%i ^
    in ('forfiles /d +0 /c "cmd /c if @isdir==FALSE echo @fname"') ^
    do call :subroutine %%i
    
  • The Old New Thing

    How do I find the most recently created file in a directory from a batch file?

    • 29 Comments

    We've reached Hump Day of Batch File Week. Remember, nobody actually likes batch programming. You merely tolerate it.

    Today, we'll find the most recently-created item in a directory. (For example, we have a server that holds our daily builds, and you might want to write a batch file that automatically installs the latest build.)

    There may be better ways, but what I do is ask for a list sorted oldest-to-newest, and then choose the last one.

    for /f %%i in ('dir /b/a-d/od/t:c') do set LAST=%%i
    echo The most recently created file is %LAST%
    

    This trick works by asking the dir command to list just the names (/b) of just the files /a-d, sorted by date (/od), based on the creation time (/t:c).

    Each time a new file is reported, its name is stored in the LAST variable, overwriting the previous one. When the loop finishes, the LAST variable contains the name of the newest file, since that's the one that didn't get overwritten.

    You can tweak the command line to perform other queries. For example, if you want the newest file, then just ask for a reverse sort (/o-d). If you want the file sorted by modified time rather than creation time, then use /t:w. You get the idea.

    Limitations: The implementation above assumes that no files contain spaces in their name. Removing this limitation is left as an exercise.

  • The Old New Thing

    What is SysFader and why is it always crashing?

    • 23 Comments

    If you type SysFader into your favorite search engine, you'll find lots of hits from people asking, "What is SysFader, and why is it always crashing Internet Explorer?"

    SysFader: iexplore.exe - Application Error
    The exception unknown software exception (0xe06d7363) occurred in the application at location 0x7c812afb.

    When a program encounters a fatal error, the system crash dialog appears. And it needs to put somebody's name in the title of the dialog to indicate which application crashed. Sure, it has the process name (iexplore.exe), but it has this nagging feeling that it can do better. After all, not everybody will know that "AcroRd32.exe" is "The menu for my favorite restaurant that I was looking at in Adobe Acrobat Reader". So it goes looking for a window that belongs to the thread so it can steal the window's title and use that to help the user understand what it was that crashed.

    And if can't find any visible windows, it will go for invisible ones, on the theory that, "Well maybe the application crashed before it could show the window."

    Now let's see what happens when we apply this logic to SysFader.

    SysFader is a helper window used by Internet Explorer to perform fade-out animations. It really doesn't do much, but it is a window, albeit an invisible one when there are no animations in progress.

    SysFader happens to run on a shared worker thread. If that worker thread is being borrowed by some other background task, and that background task crashes, then when the crash dialog appears and looks around for a window to put in the title, it says "Rats, I don't have any visible windows, but I do have this invisible one, so I'll go ahead and put that one in the title bar. Better than nothing."

    It's sort of the error-reporting version of the Politician's Fallacy:

    1. A window must be blamed.
    2. This is a window.
    3. Therefore, we must blame it.

    It's like your photo appearing in a newspaper article headlined Robbery at Woodgrove Bank, Suspect At Large, not because you're the suspect, but because you happen to have been in the building at the time of the robbery.

    Bonus chatter: You probably recognize the exception code as an unhandled C++ exception. Internet Explorer doesn't use C++ exceptions, so the exception most likely came from a plug-in.

    [Raymond is currently away; this message was pre-recorded.]

  • The Old New Thing

    Microspeak: planful

    • 20 Comments

    Every year, Microsoft invites its employees to fill out a poll which asks questions regarding all sorts of things. One of the things Microsoft employees are asked to evaluate is whether they think that their vice president is acting planfully.

    The tricky part about that question is that nobody knows exactly what the word planfully means.

    Merriam-Webster defines planful as "full of plans : RESOURCEFUL, SCHEMING." Is that what they're asking us? Whether our vice president is resourceful and scheming? Is that a good thing or a bad thing?

    The OED, on the other hand, defines it as "Full or rich in plans; characterized by planning; organized, systematic." It's not clear whether this means that the person goes into a situation with a plan ahead of time, or that once they get into a situation, they develop a plan for getting out. (Maybe both?)

    I caught a vice president using the word planful during a meeting, so I raised my hand and asked, "What does planful mean?" (Most people don't have the nerve to raise their hand during a meeting and ask, "Um, what does that word mean?")

    The vice president replied, "Actually, I don't think it's a real word. I think [name of a senior manager] made it up."

    I followed up: "Well, in that case, why are we asked to evaluate you every year on whether your actions are planful? How can we answer a question that uses a made-up word nobody knows the definition of?"

    Obligatory xpclient example: "Was it a planful decision to introduce the auto-sorting problem?"

    Reminder: Microspeak is not merely terms unique to Microsoft, but also terms used at Microsoft more often than in general.

  • The Old New Thing

    The shifting sands of "Run as different user"

    • 31 Comments

    A customer liaison asked the following question on behalf of his customer.

    When I do a shift-right-click on a shortcut to a program, I see the following:

    • On Windows Server 2008, shift-right-click does not offer the option Run as different user.
    • On Windows 7, shift-right-click does offer the option Run as different user.

      On Windows Server 2008 R2 (the server counterpart to Windows 7), shift-right-click does offer the option Run as different user.

    The option to run a program as another user (other than Administrator) was present in Windows XP, but it was lost in Windows Vista. It appears that we responded to those complaints by restoring the functionality in Windows 7.

    Is that right?

    The odd thing is that my customer has the Run as different user option available on their Windows 7 machines, but not on their Windows Server 2008 R2 machines. Does whether you have access to Run as different user depend on how you installed Windows Server 2008 R2? (If it matters, my customer installed it via the Microsoft Deployment Toolkit.)

    I found this question interesting for a variety of reasons.

    First of all, it comes dangerously close to being one of those Please tell me I'm not hallucinating type of requests. These support requests take the following peculiar form:

    We did X, then Y, then Z, and then Q happened. Is that right?

    "Um, if you say so. I wasn't there."

    But in this case, it started out looking like it was going to turn into a Please tell me I'm not hallucinating request, then veered into "Is X the reason the feature was added back?"

    This is a rather peculiar question, because knowing the answer one way or the other doesn't actually take you any closer to a solution to the problem. (And I don't know the answer, but fortunately, it wasn't relevant to solving the customer's problem.)

    Another interesting thing about the customer's question is that he never actually comes out and asks the question. He sort of says a few related things, and asks a tangential question, but never comes right out and asks, "How do I get the Run as different user option to work on my Windows Server 2008 R2 machine?"

    It's like a kid who pointedly hangs around a candy bowl, hoping that an adult will say, "Here, have a piece of candy."

    You're a grown-up now. You don't have to linger around the candy bowl hoping somebody will figure out that you want some candy. You should just ask, "May I have a piece of candy?"

    My psychic powers tell me that they have set the Require trusted path for credential entry policy. The Run as different user feature is disabled if you set this policy.

    The customer liaison replied, "It appears that the option for Require trusted path for credential entry is not enabled. The customer is going to do a clean install and test on a non-domain-joined machine to avoid any GPOs."

    Some time passed, and the customer liaison reported back with a resolution.

    The culprit was indeed the Require trusted path for credential entry policy. It didn't show up in a GPO search because they were setting the policy via a script rather than deploying a group policy object.

    It was very nice of the customer liaison to reply with confirmation that the problem was solved. This is, unfortunately, a notable event. Most of the time, people never report back if your suggestion solved their program; they only come back if your suggestion didn't help. Which means you're not sure if your suggestion solved the problem, or if it didn't solve the problem but they decided to continue the investigation somewhere else.

    Bonus chatter: This shows yet another reason why you should use Group Policy Objects to manage group policies rather than custom scripts that whack registry keys. In addition to the fact that registry key whacking may not work, there are tools for processing Group Policy Objects that make this sort of investigation much easier.

  • The Old New Thing

    Command line tool to manage Windows 7 Libraries, with source code

    • 14 Comments

    A customer asked if there was a command-line tool for managing Windows 7 Libraries so that they could create and deploy libraries across their organization.

    Not only is there such a tool, it even comes with source code.

    The Shell Library Command Line Sample shows how to manage Windows 7 libraries programmatically via the IShell­Library interface. And it's actually a useful program on its own.

    Usage: shlib.exe SUBCOMMAND
    
    Displays and modifies the attributes of Shell Libraries.
    
    Supported commands:
      create      Creates a library at the specified path.
      info        Prints info about the given library.
      enum        Enumerates the folders in the library.
      setattrib   Modifies the attributes of the library.
      add         Adds the specified folder to the specified library.
      remove      Removes the specified folder from the library.
      setsaveloc  Sets the default save location of the library.
      resolve     Resolves the specified folder in the library.
      resolveall  Resolves all locations in the library in bulk.
      manage      Displays the Manage Library Dialog for the library.
    

    For example, to see all the folders that are part of your Documents library, use the command

    shlib enum FOLDERID_DocumentsLibrary
    

    Each of the commands has further sub-options.

    Usage: create shlib.exe [OPTIONS]  [...]
    
    Creates a library at the specified path.
    
    Options:
     -create[:ARG]       Specifies that a new library should be created.
       <none>              Fail if the library already exists.
       overwrite           Overwrite any existing library.
       uniquename          Generate a unique name in case of conflict.
    
    The library may be specified by a file system path,
    or by a KNOWNFOLDERID (e.g. "FOLDERID_DocumentsLibrary").
    

    And the source code shows how to perform each of these actions programmatically.

    The IShellLibrary interface is also supported by the Windows API Code Pack for .NET, so you C# folks can use it too.

    (One thing I noticed that is missing from the sample is a seticon command. Shouldn't be hard to implement. You just call IShell­Library::Set­Icon.)

  • The Old New Thing

    What happened to the Windows 2000 "Language settings for the system" control panel?

    • 41 Comments

    In 2011, a customer had a question about migrating from Windows 2000 to Windows XP. (That's right, this customer was still using Windows 2000 in the year 2011.) Specifically, they noted that in Windows 2000, they can select multiple languages in the "Language settings for the system" portion of the Regional Options control panel, and they couldn't find the corresponding control panel setting in Windows XP.

    Regional Options
    General
    Numbers
    Currency
    Time
    Date
    Input Locales
     
    Settings for the current user
    Many programs support international settings for numbers, currencies, times, and dates. Set the locale in order to use the standard settings.
    Your locale (location):
    English (United States)
    Language settings for the system
    Your system is configured to read and write documents in multiple languages.
    Arabic
    Armenian
    Baltic
    Central Europe
    Cyrillic

    In Windows 2000, "Language settings for the system" provides the option to install support (such as code pages, keyboard layouts, and fonts) for various language groups. In Windows XP, the big list of language groups was reduced to three categories:

    • Basic (Baltic, Central Europe, Cyrillic, Greek, Turkish, Western Europe)
    • Complex (Arabic, Armenian, Georgian, Hebrew, Indic, Vietnamese, Thai)
    • East Asia (Chinese Simplified, Chinese Traditional, Japanese, Korean)

    The Basic category is always installed. To install the Complex or East Asia categories, use the "Supplemental language support" section of the Regional and Language Options control panel.

    Windows XP Regional and Language Options property sheet, with a section titled "Supplemental language support" with options "Install files for complex script and right-to-left languages (including Thai)" and "Install files for East Asian languages

    Someday, that customer might upgrade to Windows Vista, so I may as well answer the question right now. In Windows Vista and onward, things were simplified even more: All language groups are installed at all times. The dialog box went away completely since there were no options remaining.

    As it turns out, the customer's problem had nothing to do with language support. Of course, they didn't come out and describe the problem they were having; rather, they reduced the problem into multiple pieces, and then asked for help on one specific piece. They tried out a solution based on this new information, but it didn't solve the problem, because as it turns out, the Language settings for the system control panel was a red herring. If they had told us what their original problem was, we could have told them "But this setting will do nothing to solve your problem. What you really need is over there."

    Tomorrow, we'll look at the customer's actual problem. (So please don't try to guess or you'll ruin the surprise. I can't believe I had to write that.)

  • The Old New Thing

    Of what possible legitimate use are functions like CreateRemoteThread, WriteProcessMemory, and VirtualProtectEx?

    • 26 Comments

    There are a bunch of functions that allow you to manipulate the address space of other processes, like Write­Process­Memory and Virtual­Alloc­Ex. Of what possible legitimate use could they be? Why would one process need to go digging around inside the address space of another process, unless it was up to no good?

    These functions exist for debuggers. For example, when you ask the debugger to inspect the memory of the process being debugged, it uses Read­Process­Memory to do it. Similarly, when you ask the debugger to update the value of a variable in your process, it uses Write­Process­Memory to do it. And when you ask the debugger to set a breakpoint, it uses the Virtual­Protect­Ex function to change your code pages from read-execute to read-write-execute so that it can patch an int 3 into your program.

    If you ask the debugger to break into a process, it can use the Create­Remote­Thread function to inject a thread into the process that immediately calls Debug­Break. (The Debug­Break­Process was subsequently added to make this simpler.)

    But for general-purpose programming, these functions don't really have much valid use. They tend to be used for nefarious purposes like DLL injection and cheating at video games.

    [Raymond is currently away; this message was pre-recorded.]

  • The Old New Thing

    Adventures in undefined behavior: The premature downcast

    • 45 Comments

    A customer encountered the following problem:

    class Shape
    {
    public:
        virtual bool Is2D() { return false; }
    };
    
    class Shape2D : public Shape
    {
    public:
        virtual bool Is2D() { return true; }
    };
    
    Shape *FindShape(Cookie cookie);
    
    void BuyPaint(Cookie cookie)
    {
        Shape2D *shape = static_cast<Shape2D *>(FindShape(cookie));
        if (shape->Is2D()) {
           .. do all sorts of stuff ...
        }
    }
    

    The Buy­Paint function converts the cookie back to a Shape object, and then checks if the object is a Shape2D object by calling Is­2D. If so, then it does some more stuff to figure out what type of paint to buy.

    (Note to nitpickers: The actual scenario was not like this, but I presented it this way to illustrate the point. If you say "You should've used RTTI" or "You should've had a BuyPaint method on the Shape class", then you're missing the point.)

    The programmers figured they'd save some typing by casting the result of Find­Shape to a Shape2D right away, because after all, since Is­2D is a virtual method, it will call the right version of the function, either Shape::Is­2D or Shape2D::Is­2D, depending on the actual type of the underlying object.

    But when compiler optimizations were turned on, they discovered that the call to Is­2D was optimized away, and the Buy­Paint function merely assumed that it was always operating on a Shape2D object. It then ended up trying to buy paint even for one-dimensional objects like points and lines.

    Compiler optimization bug? Nope. Code bug due to reliance on undefined behavior.

    The C++ language says (9.3.1) "If a nonstatic member function of a class X is called for an object that is not of type X, or of a type derived from X, the behavior is undefined." In other words, if you are invoking a method on an object of type X, then you are promising that it really is of type X, or a class derived from it.

    The code above violates this rule, because it is invoking the Is­2D method on a Shape2D*, which therefore comes with the promise "This really is a Shape2D object (or something derived from it)." But this is a promise the code cannot deliver on, because the object returned by Find­Shape might be a simple Shape.

    The compiler ran with the (false) promise and said, "Well, since you are guaranteeing that the object is at least a Shape2D, and since I have studied your code and determined that no classes which further derive from Shape2D override the Is­2D method, I have therefore proved that the final overrider is Shape2D::Is­2D and can therefore inline that method."

    Result: The compiler inlines Shape2D::Is­2D, which returns true, so the "if" test can be optimized out. The compiler can assume that Buy­Paint is always called with cookies that represent two-dimensional objects.

    The fix is to do the annoying typing that the original authors were trying to avoid:

    void BuyPaint(Cookie cookie)
    {
        Shape *shape = FindShape(cookie);
        if (shape->Is2D()) {
          Shape2D *shape2d = static_cast<Shape2D *>(shape);
           .. do all sorts of stuff (with shape2d) ...
        }
    }
    
Page 1 of 3 (29 items) 123