• The Old New Thing

    Keep your eye on the code page: C# edition (warning about DllImport)


    Often, we receive problem reports from customers who failed to keep their eye on the code page.

    Does the SH­Get­File­Info function support files with non-ASCII characters in their names? We find that the function either fails outright or returns question marks when asked to provide information for files with non-ASCII characters in their name.

    using System;
    using System.Runtime.InteropServices;
    class Program
     static void Main(string[] args)
      string fileName = "BgṍRồ.txt";
      Console.WriteLine("File exists? {0}", System.IO.File.Exists(fileName));
      // assumes extensions are hidden
      string expected = "BgṍRồ";
      Test(fileName, SHGFI_DISPLAYNAME, expected);
     static void Test(string fileName, uint flags, string expected)
      var actual = GetNameViaSHGFI(fileName, flags);
      Console.WriteLine("{0} == {1} ? {2}", actual, expected, actual == expected);
     static string GetNameViaSHGFI(string fileName, uint flags)
      SHFILEINFO sfi = new SHFILEINFO();
      if (SHGetFileInfo(fileName, 0, ref sfi, Marshal.SizeOf(sfi),
                        flags) != IntPtr.Zero) {
       return sfi.szDisplayName;
      } else {
       return null;
     struct SHFILEINFO {
      public IntPtr hIcon;
      public int iIcon;
      public uint dwAttributes;
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
      public string szDisplayName;
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 80)]
      public string szTypeName;
     const uint SHGFI_USEFILEATTRIBUTES = 0x10;
     const uint SHGFI_DISPLAYNAME = 0x0200;
     static extern IntPtr SHGetFileInfo(
        string path, uint fileAttributes, ref SHFILEINFO info, int cbSize,
        uint flags);
    // Output:
    // File exists? True
    //  == Bg?R? ? False
    // Bg?R? == Bg?R? ? False

    If we ask for the display name, the function fails even though the file does exist. If we also pass the SHGFI_USE­FILE­ATTRIBUTES flag to force the system to act as if the file existed, then it returns the file name but with question marks where the non-ASCII characters should be.

    The SH­Get­File­Info function supports non-ASCII characters just fine, provided you call the version that supports non-ASCII characters!

    The customer here fell into the trap of not keeping their eye on the code page. It goes back to an unfortunate choice of defaults in the System.Runtime.Interop­Services namespace: At the time the CLR was originally being developed, Windows operating systems derived from Windows 95 were still in common use, so the CLR folks decided to default to Char­Set.Ansi. This made sense back in the day, since it meant that your program ran the same on Windows 98 as it did in Windows NT. In the passage of time, the Windows 95 series of operating systems became obsolete, so the need to be compatible with it gradually disappeared. But too late. The rules were already set, and the default of Char­Set.Ansi could not be changed.

    The solution is to specify Char­Set.Unicode explicitly in the Struct­Layout and Dll­Import attributes.

    FxCop catches this error, flagging it as Specify­Marshaling­For­PInvoke­String­Arguments. The error explanation talks about the security risks of unmapped characters, which is all well and good, but it is looking too much at the specific issue and not so much at the big picture. As a result, people may ignore the issue because it is flagged as a complicated security issue, and they will think, "Eh, this is just my unit test, I'm not concerned about security here." However, the big picture is

    This is almost certainly an oversight on your part. You didn't really mean to disable Unicode support here.

    Change the lines



     [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
     [DllImport("shell32.dll", CharSet=CharSet.Unicode)]

    and re-run the program. This time, it prints

    File exists? True
    Bg?R? == Bg?R? ? True
    Bg?R? == Bg?R? ? True

    Note that you have to do the string comparison in the program because the console itself has a troubled history with Unicode. At this point, I will simply cue a Michael Kaplan rant and link to an article explaining how to ask nicely.

  • The Old New Thing

    Why was the replacement installer for recognized 16-bit installers itself a 32-bit program instead of a 64-bit program?


    Even though 64-bit Windows does not support 16-bit applications, there is a special case for 16-bit installers for 32-bit applications. Windows detects this scenario and substitutes a 32-bit replacement installer which replicates the actions of the 16-bit installer. Commenter Karellen is horrified at the fact that the replacement installer is a 32-bit program. "You're writing a program that will run exclusively on 64-bit systems. Why not built it to run natively on the OS it's designed for? Why is this apparently not the "obvious" Right Thing(tm) to do? What am I missing?"

    Recall that a science project is a programming project that is technically impressive but ultimately impractical. For example it might be a project that nobody would actually use, or it attempts to add a Gee-Whiz feature that nobody is really clamoring for.

    But at least a science project is trying to solve a problem. This proposal doesn't even solve any problems! Indeed, this proposal creates problems. One argument in favor of doing it this way is that it is satisfies some obsessive-compulsive requirement that a 64-bit operating system have no 32-bit components beyond the 32-bit emulation environment itself.

    Because! Because you're running a 64-bit system, and running apps native to that system is just more elegant.

    Okay, it's not obsessive-compulsive behavior. It's some sort of aesthetic ideal, postulated for its own sake, devoid of practical considerations.

    Remember the problem space. We have a bunch of 32-bit applications that use a 16-bit installer. Our goal is to get those applications installed on 64-bit Windows. By making the replacement installer a 32-bit program, you get the emulator to do all the dirty work for you. Things like registry redirection, file system redirection, and 32-bit application compatibility.

    Suppose the original installer database says

    • Copy X.DLL file into the %Program­Files%\AppName directory.
    • Copy Y.DLL into the %windir%\System32 directory.
    • If the current version of C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\AcroPDF.dll is 7.5 or higher, then set this registry key.

    If you write the replacement installer as a 32-bit program, then other parts of the 32-bit emulation engine do the work for you.

    • The environment manager knows that 64-bit processes get the environment variable Program­Files pointing to C:\Program Files, whereas 32-bit processes get Program­Files pointing to C:\Program Files (x86).
    • The file system redirector knows that if a 32-bit process asks for %windir%\System32, it should really get %windir%\SysWOW64.
    • The registry redirector knows that if a 32-bit process tries to access certain parts of the registry, they should be sent to the Wow­64­32­Node instead.

    If you had written the replacement installer as a 64-bit program, you would have to replicate all of these rules and make sure your copy of the rules exactly matched the rules used by the real environment manager, file system redirector, and registry redirector.

    Now you have to keep two engines in sync: the 32-bit emulation engine and the 64-bit replacement installer for 32-bit applications. This introduces fragility, because any behavior change in the 32-bit emulation engine must be accompanied by a corresponding change in the 64-bit replacement installer.

    Suppose the application compatibility folks add a rule that says, "If a 32-bit installer tries to read the version string from C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\AcroPDF.dll, return the version string from C:\Program Files (x86)\Common Files\Adobe\Acrobat\ActiveX\AcroPDF.dll instead." And suppose that rule is not copied to the 64-bit replacement installer. Congratulations, your 64-bit replacement installer will incorrectly install any program that changes behavior based on the currently-installed version of AcroPDF.

    I don't know for sure, but I wouldn't be surprised if some of these installers support plug-ins, so that the application developer can run custom code during installation. It is possible for 16-bit applications to load 32-bit DLLs via a technique known as generic thunking, and the 16-bit stub installer would use a generic thunk to call into the 32-bit DLL to do whatever custom action was required. On the other hand, 64-bit applications cannot load 32-bit DLLs, so if the 64-bit replacement installer encountered a 32-bit DLL plug-in, it would have to run a 32-bit helper application to load the plug-in and call into it. So you didn't escape having a 32-bit component after all.

    And the original obsessive-compulsive reason for requiring the replacement installer to be 64-bit was flawed anyway. This is a replacement installer for a 32-bit application. Therefore, the replacement installer is part of the 32-bit emulation environment, so it is allowed to be written as a 32-bit component.

    Let's look at the other arguments given for why the replacement installer for a 32-bit application should be written as a 64-bit application.

    Because complexity is what will be our undoing in the end, and reducing it wherever we can is always a win.

    As we saw above, writing the replacement installer as a 64-bit application introduces complexity. Writing it as a 32-bit application reduces complexity. So this statement itself argues for writing the replacement installer as a 32-bit application.

    Because we can't rewrite everything from scratch at once, but we can create clean new code one small piece at a time, preventing an increase to our technical debt where we have the opportunity to do so at negligible incremental cost to just piling on more cruft.

    As noted above, the incremental cost is hardly negligible. Indeed, writing the replacement installer as a 64-bit application is not merely more complex, it creates an ongoing support obligation, because any time there is a change to the 32-bit emulation environment, that change needs to be replicated in the 64-bit replacement installer. This is a huge source of technical debt: Fragile coupling between two seemingly-unrelated components.

    And writing the replacement installer as a 32-bit application does not create a future obligation to port it to 64 bits when support for 32-bit applications is dropped in some future version of Windows. Because when support for 32-bit applications disappears (as it already has on Server Core), there will be no need to port the replacement installer to 64-bit because there's no point writing an installer for a program that cannot run!

    Writing the replacement installer as a 32-bit program was the right call.

  • The Old New Thing

    If you don't want to try to repair the data, then don't, but you should at least know that you have corrupted data


    When I wrote about understanding the consequences of WAIT_ABANDONED, I mentioned that one of the possible responses was to try to repair the damage, but some people are suspicious of this approach.

    Mind you, I'm suspicious of it, too. Repairing corruption is hard. You have to anticipate the possibility, create enough of a trail to be able to reconstruct the original data once the corruption is recognized, and then be able to restore the data to some semblance of consistency. I didn't say that this was mandatory; I didn't even say that it was recommended. I just listed it as one of the options, an option for the over-achievers out there.

    For most cases, attempting repair is overkill. But you still have to know that something went wrong; otherwise, one crashed program will lead to more crashed programs as they try to operate on inconsistent data. The purpose of the article was to raise awareness of the issue, based on my observation that most people blindly ignore the possibility that the mutex was abandoned.

  • The Old New Thing

    Why does GetCommandLine give me a corrupted command line?


    A customer had the following problem:

    We're calling GetCommandLine to retrieve the command line, and the documentation says that it returns a single null-terminated string. However, when we call it in our application, we find that it is actually a double-null-terminated string. The buffer returned consists of a series of null-terminated strings, one string per word on the command line, all stored one after the other, and with two null terminators at the end. How do I get the original string?

    Recall that the command line is just a conveniently-initialized variable in a process and once it's set up, the kernel doesn't really care about it any more.

    What is most likely happening is that somebody is taking the raw command line returned by GetCommandLine and writing to it. The customer can confirm this by dumping the command line just as the process starts, even before any DLLs get to run their DllMains, and then setting a write breakpoint on the command line to see who is writing to it.

    And in fact, the customer did find the culprit.

    It turns out it was some other part of the code (not written by me!) which was parsing the command line and writing into it in the process.
  • The Old New Thing

    Announcements on the ferry, and hills that grew while I was away


    This weekend is the annual Chilly Hilly bike ride, an early wake-up call to bicyclists in the Seattle area to get their act together and hit the road.

    Last year, I joined my 2999 closest friends up and down the hills of Bainbridge Island. Unfortunately, the hills had grown taller in the two years since my last ride.

    There was an announcement on the ferry. It went something like this: "Your attention please. There is a bicycle on the car deck that has fallen over onto its side. Repeat: A bicycle has fallen onto its side."

    It took a few seconds for the joke to sink in with all the people on the ferry, but after a short while, the passenger deck was chuckling.

  • The Old New Thing

    Researchers discover link between music and drinking

    A British scientific study shows that a bit of classical music can persuade diners to buy more fancy coffees, pricey wines and luxurious desserts. "North has shown that playing German or French music can persuade diners to buy wine from those countries." I found this to be true in my experience. If you get two thousand people in a tent and play live oom-pah music, they end up drinking lots of German beer.
  • The Old New Thing

    Welcome to Taiwan's premier English-only nightclub


    One of my friends is fluent in both Mandarin and English. When she lived in Taiwan, she paid a visit to a nightclub whose gimmick was that you had to speak English. The target audience was not foreigners but rather native Taiwanese who learned English as a second language. My friend didn't have any problems with this rule, but many of the guests appeared to be struggling to conform.

    My friend paid a visit to the ladies' room, and there she overheard a conversation between two other guests. (They were speaking in Mandarin. Apparently, the rules aren't enforced in the bathroom.)

    "There's this cute guy out on the dance floor, but I don't know what to say to him. My English is not very good."

    My friend told her, "That's okay. His English isn't very good either."

  • The Old New Thing

    The evolution of menu templates: 32-bit extended menus


    At last we reach the 32-bit extended menu template. Introduced in Windows 95, this remains the most advanced menu template format through Windows Vista. As you might expect, the 32-bit extended menu template is just a 32-bit version of the 16-bit extended menu template, so if you've been following along, you should find no real surprises here; all the pieces have been telegraphed far in advance.

    The header remains the same:

    struct MENUHEADER32 {
     WORD wVersion;
     WORD cbHeaderSize;
     BYTE rgbExtra[cbHeaderSize-4];

    The differences here from the 32-bit classic menu template header are analogous to the changes between the 16-bit classic menu template and the 16-bit extended menu template. The wVersion is set to one for extended templates, and the cbHeaderSize includes the wVersion and cbHeaderSize fields themselves, so the number of extra bytes is four less than the value specified in cbHeaderSize. There is one additional constraint: The cbHeaderSize must be a multiple of four because extended menu item templates must be aligned on DWORD boundaries. But as with 32-bit classic templates, the cbHeaderSize must be four in order to avoid a bug in the Windows 95 family of operating systems.

    After the header comes the menu itself, and like the 16-bit extended menu template, there is a prefix structure that comes before the items and which serves the same purpose as in the 16-bit extended menu template:

    struct MENUPREFIX32 {
     DWORD dwContextHelpID;

    The list of menu items is basically the same as the 16-bit version, just with some expanded fields.

    struct MENUITEMEX32 {
     DWORD dwType;
     DWORD dwState;
     DWORD dwID;
     WORD  wFlags;
     WCHAR szText[]; // null terminated UNICODE string

    As we saw before when we studied the 16-bit extended menu template, the big difference between classic and extended menu items is that classic menu items were designed for the InsertMenu function, whereas extended menu items were designed for the InsertMenuItem function. The dwType, dwState, and dwID members correspond to the fType, fState, and wID members of the MENUITEMINFO structure, and the the szText goes into the dwItemData if the item requires a string.

    One additional quirk of 32-bit extended menu item templates which the 16-bit version does not have is that 32-bit extended menu item templates must begin on a 32-bit boundary; therefore, you must insert a WORD of padding after the menu text if the text is an odd number of characters long. (Fourteen bytes of the fixed-length part of the MENUITEMEX32 plus an odd number of WCHARs plus the null terminator WCHAR leaves a value that is 2 mod 4; therefore, you need an additional WORD to return to a DWORD boundary.)

    The wFlags field has the same values as in the 16-bit extended menu item templates; the high byte is always zero. And, as before, if the bottom bit is set, then the menu item describes a pop-up submenu, which is inserted directly after the extended menu item template.

    That's all there is to it. Let's see how our example menu resource looks when converted to a 32-bit extended menu template:

    1 MENUEX 1000
      POPUP "&File", 200,,, 1001
        MENUITEM "&Open\tCtrl+O", 100
        MENUITEM "&Exit\tAlt+X",  101
      POPUP "&View", 201,,, 1002
        MENUITEM "&Status Bar", 102,, MFS_CHECKED

    First comes the header, whose contents are fixed:

    0000  01 00          // wVersion = 1
    0002  04 00          // cbHeaderSize = 4

    Before the list of extended menu item templates, we have the context help ID:

    0004  E8 03 00 00    // dwContextHelpID = 1000

    Since our first menu item is a pop-up submenu, the wFlags will have the bottom bit set:

    0008  00 00 00 00    // dwType = MFT_STRING
    000C  00 00 00 00    // dwState = 0
    0010  C8 00 00 00    // wID = 200
    0014  01 00          // wFlags = "pop-up submenu"
    0016  26 00 46 00 69 00 6C 00 65 00 00 00
                         // "&File" + null terminator
    0022  00 00          // Padding to restore alignment

    Notice the two bytes of padding so that we return to DWORD alignment.

    The wFlags promised a pop-up submenu, so here it is.

    0024  E9 03 00 00    // dwContextHelpID = 1001
    // First item
    0028  00 00 00 00    // dwType = MFT_STRING
    002C  00 00 00 00    // dwState = 0
    0030  64 00 00 00    // dwID = 100
    0034  00 00          // wFlags = 0
    0036  26 00 4F 00 70 00 65 00 6E 00 09 00
          43 00 74 00 72 00 6C 00 2B 00 4F 00 00 00
                         // "&Open\tCtrl+O" + null terminator
    // Second item
    0050  00 08 00 00     // dwType = MFT_SEPARATOR
    0054  00 00 00 00     // dwState = 0
    0058  FF FF FF FF     // dwID = -1
    005C  00 00           // wFlags = 0
    005E  00 00           // ""
    // Third (final) item
    0060  00 00 00 00     // dwType = MFT_STRING
    0064  00 00 00 00     // dwState = 0
    0068  65 00 00 00     // dwID = 101
    006C  80 00           // wFlags = "this is the last menu item"
    0070  26 00 45 00 78 00 69 00 74 00 09 00
          41 00 6C 00 74 00 2B 00 58 00 00 00
                          // "&Exit\tAlt+X" + null terminator
    0086  00 00          // Padding to restore alignment

    When we see the "end" marker, we pop one level back to the main menu.

    0088  00 00 00 00     // dwType = MFT_STRING
    008C  00 00 00 00     // dwState = 0
    0090  C9 00 00 00     // dwID = 201
    0094  81 00           // wFlags = "pop-up submenu" |
                          //          "this is the last menu item"
    0096  26 00 56 00 69 00 65 00 77 00 00 00
                          // "&View" + null terminator
    00A2  00 00          // Padding to restore alignment

    The set bottom bit in the wFlags indicates that another pop-up submenu is coming, and the "end" marker means that once the submenu is finished, we are done.

    00A4  EA 03 00 00    // dwContextHelpID = 1002
    00A8  00 00 00 00    // dwType = MFT_STRING
    00AC  08 00 00 00    // dwState = MFS_CHECKED
    00B0  66 00 00 00    // dwID = 102
    00B4  80 00          // wFlags = "this is the last menu item"
    00B6  26 00 53 00 74 00 61 00 74 00 75 00
          73 00 20 00 42 00 61 00 72 00 00 00
                         // "&Status Bar" + null terminator
    00CE  00 00          // Padding to restore alignment

    Since the pop-up submenu has only one item, the first item is also the last.

    That's it for the evolution of menu templates, starting from a series of calls to the ANSI version of InsertMenu to a series of calls to the Unicode version of InsertMenuItem. Menu templates get much less attention than dialog templates, but if you wanted to know how they work, well, there you have it.

  • The Old New Thing

    Reflections create Xbox logo on neighbor's roof


    In other parts of the world, religious images emerge from random patterns. Out here, we get Microsoft marketing.

    If I were on it, I could've charged admission.

  • The Old New Thing

    Menu item states are not reliable until they are shown because they aren't needed until then


    A question arrived from a customer (with the rather unhelpful subject line Question for Microsoft) wondering why, when they call Get­System­Menu and then ask for the states of the various menu items like SC_MINIMIZE, the menu item states don't reflect reality. The menu item states don't synchronize with reality until the user actually opens the system menu.

    There is no requirement that applications keep menu item states continuously in sync. After all, that's why we have messages like WM_INIT­MENU: To tell the application, "Whoa, we're about to show this menu, so you might want to comb its hair and pick the food out of its teeth so it can be seen by the user." Lazy evaluation is common, because maintaining states continuously can be expensive, and there's no point constantly turning items on and of and on and off if the user can't see them anyway.

    This is double-true for system menus, because maintaining the states continuously is not possible when the system menu is being shared across windows. The menu states are not synchronized to the window until the menu is about to be displayed.

    If you want to know whether the SC_MINIMIZE menu item would be enabled if the menu were shown, you can check the window styles: A window can be minimized if it has a WS_MINIMIZE­BOX and is not already WS_MINIMIZEd. Similar logic can be applied to the other menu items.

    Well, except for SC_CLOSE. While in most cases the window caption determines what is enabled on the menu, the Close button works backward: It is the state of the menu item that controls whether the Close button is enabled. So in the special case of SC_CLOSE, you can query the state at any time, because for that case, the menu controls the state rather than simply reflecting it.

    Why is SC_CLOSE so special? Here come da history.

    The Close button was added in Windows 95. Since versions of Windows prior to Windows 95 didn't have a Close button, they didn't need a style to specify whether the Close button should be enabled or not. (You don't need a style to control something that doesn't exist.) Windows 95 added the Close button and hooked it up to the only thing that it had available, namely, the SC_CLOSE item on the system menu. Sure, Windows 95 could have have invented a new window style, but since SC_CLOSE already existed and applications were already using it, using SC_CLOSE to control the Close button allowed old applications to reap the benefits of the new Close button automatically. It also meant that there was one less thing you had to change when porting your program to Windows 95.

    Bonus chatter: You can now answer Alex Cohn's question:

    I wonder if the EnableMenuItem method will work for minimize and maximize, too. After all, these buttons also have siblings in the Alt-space menu.
Page 382 of 446 (4,451 items) «380381382383384»