• The Old New Thing

    The sociology of the mobile phone


    It has become obvious by now that the mobile phone has changed the way people interact. These two papers were forwaded to me by a colleague, whose summary I am shamelessly lifting from heavily.

    First is a short paper titled Exploring the implications for social identity of the new sociology of the mobile phone.

    The much more fascinating (and much longer) one is The Effects of Mobile Telephones on Social and Individual Life [PDF]. Read about Flight, Suspension and Persistence—the three ways people deal with incoming calls. Learn how to tell an Innie from an Outie. Set up an Approximeeting. Are you a swift, an owl, a dove, a sparrow, a starling, or a peacock? Pick up some UK mobile phone lingo:

    • "Gooseberry call": calling someone during their date to get a progress report.
    • "Shagbile": the mobile phone your partner doesn't know about, used for affairs.
    • "Sad": derogatory term for people who show off their phones. Enormous ritualization surrounds this.

    "Shagbile" is the best. So Austin Powers.

  • The Old New Thing

    Importance of alignment even on x86 machines


    Sometimes unaligned memory access will hang the machine.

    Some video cards do not let you access all the video memory at one go. Instead, you are given a window into which you can select which subset of video memory ("bank") you want to see. For example, the EGA video card had 256K of memory, split into four 64K banks. If you wanted to access memory in the first 64K, you had to select bank zero into the window, but if you wanted to access memory in the second 64K, then you had to select bank one.

    Bank-switching makes memory access much more complicated, For example, if you want to copy a block of memory into bank-switched memory, you have to check when you are going to cross a bank boundary and break the copy up into pieces. If you are doing something that requires non-sequential access (say, drawing a diagonal line), you have to check when your line is going to cross into another bank.

    To simplify matters, Windows 95 had a driver called VFLATD that made bank-switched memory look flat to the rest of the system. Flattening the bank-switched memory model was also crucial for DirectDraw support; in particular, the IDirectDrawSurface::Lock method gave you direct access to a (seemingly) flat expanse of video memory. For example, if the application wanted to see a 256K surface and accessed memory in the first 64K of memory, the VFLATD driver would select bank zero and map the 64K physical memory window into the first 64K of the virtual 256K memory window.

    This worked great as long as everybody uses only aligned memory accesses. But if you access unaligned memory, you can send VFLATD into an infinite loop and hang the machine.

    Suppose you make an unaligned memory access that straddles two banks. This memory access can never be satisfied. A page fault is taken on the lower portion of the unaligned access, and VFLATD maps the lower bank into memory. Then a page fault is taken on the higher portion of the unaligned access, and VFLATD now has to map the upper bank; this unmaps the lower bank, since the video card is bank-switched and only one bank can be mapped ata time. Now a page fault is taken on the lower portion, and the infinite loop continues.

    Moral of the story: Keep those memory accesses aligned, even on the x86, which most people would consider to be one where it is "safe" to violate alignment rules.

    Next time, another example of how misaligned data access can create bugs x86.

  • The Old New Thing

    Why do some structures end with an array of size 1?


    Some Windows structures are variable-sized, beginning with a fixed header, followed by a variable-sized array. When these structures are declared, they often declare an array of size 1 where the variable-sized array should be. For example:

    typedef struct _TOKEN_GROUPS {
        DWORD GroupCount;

    If you look in the header files, you'll see that ANYSIZE_ARRAY is #define'd to 1, so this declares a structure with a trailing array of size one.

    With this declaration, you would allocate memory for one such variable-sized TOKEN_GROUPS structure like this:

    PTOKEN_GROUPS TokenGroups =
       malloc(FIELD_OFFSET(TOKEN_GROUPS, Groups[NumberOfGroups]));
    and you would initialize the structure like this:
    TokenGroups->GroupCount = NumberOfGroups;
    for (DWORD Index = 0; Index = NumberOfGroups; Index++) {
      TokenGroups->Groups[Index] = ...;

    Many people think it should have been declared like this:

    typedef struct _TOKEN_GROUPS {
        DWORD GroupCount;

    (In this article, code that is wrong or hypothetical will be italicized.)

    The code that does the allocation would then go like this:

    PTOKEN_GROUPS TokenGroups =
       malloc(sizeof(TOKEN_GROUPS) +
              NumberOfGroups * sizeof(SID_AND_ATTRIBUTES));

    This alternative has two disadvantages, one cosmetic and one fatal.

    First, the cosmetic disadvantage: It makes it harder to access the variable-sized data. Initializing the TOKEN_GROUPS just allocated would go like this:

    TokenGroups->GroupCount = NumberOfGroups;
    for (DWORD Index = 0; Index = NumberOfGroups; Index++) {
      ((SID_AND_ATTRIBUTES *)(TokenGroups + 1))[Index] = ...;

    The real disadvantage is fatal. The above code crashes on 64-bit Windows. The SID_AND_ATTRIBUTES structure looks like this:

    typedef struct _SID_AND_ATTRIBUTES {
        PSID Sid;
        DWORD Attributes;

    Observe that the first member of this structure is a pointer, PSID. The SID_AND_ATTRIBUTES structure requires pointer alignment, which on 64-bit Windows is 8-byte alignment. On the other hand, the proposed TOKEN_GROUPS structure consists of just a DWORD and therefore requires only 4-byte alignment. sizeof(TOKEN_GROUPS) is four.

    I hope you see where this is going.

    Under the proposed structure definition, the array of SID_AND_ATTRIBUTES structures will not be placed on an 8-byte boundary but only on a 4-byte boundary. The necessary padding between the GroupCount and the first SID_AND_ATTRIBUTES is missing. The attempt to access the elements of the array will crash with a STATUS_DATATYPE_MISALIGNMENT exception.

    Okay, you may say, then why not use a zero-length array instead of a 1-length array?

    Because time travel has yet to be perfected.

    Zero-length arrays did not become legal Standard C until 1999. Since Windows was around long before then, it could not take advantage of that functionality in the C language.

  • The Old New Thing

    Why can't you treat a FILETIME as an __int64?


    The FILETIME structure represents a 64-bit value in two parts:

    typedef struct _FILETIME {
      DWORD dwLowDateTime;
      DWORD dwHighDateTime;

    You may be tempted to take the entire FILETIME structure and access it directly as if it were an __int64. After all, its memory layout exactly matches that of a 64-bit (little-endian) integer. Some people have written sample code that does exactly this:

    pi = (__int64*)&ft; // WRONG
    (*pi) += (__int64)num*datepart; // WRONG

    Why is this wrong?


    Since a FILETIME is a structure containing two DWORDs, it requires only 4-byte alignment, since that is sufficient to put each DWORD on a valid DWORD boundary. There is no need for the first DWORD to reside on an 8-byte boundary. And in fact, you've probably already used a structure where it doesn't: The WIN32_FIND_DATA structure.

    typedef struct _WIN32_FIND_DATA {
        DWORD dwFileAttributes;
        FILETIME ftCreationTime;
        FILETIME ftLastAccessTime;
        FILETIME ftLastWriteTime;
        DWORD nFileSizeHigh;
        DWORD nFileSizeLow;
        DWORD dwReserved0;
        DWORD dwReserved1;
        TCHAR  cFileName[ MAX_PATH ];
        TCHAR  cAlternateFileName[ 14 ];

    Observe that the three FILETIME structures appear at offsets 4, 12, and 20 from the beginning of the structure. They have been thrown off 8-byte alignment by the dwFileAttributes member.

    Casting a FILETIME to an __int64 therefore can (and in the WIN32_FIND_DATA case, will) create a misaligned pointer. Accessing a misaligned pointer will raise a STATUS_DATATYPE_MISALIGNMENT exception on architectures which require alignment.

    Even if you are on a forgiving platform that performs automatic alignment fixups, you can still run into trouble. More on this and other consequences of alignment in the next few entries.

    Exercise: Why are the LARGE_INTEGER and ULARGE_INTEGER structures not affected?

  • The Old New Thing

    Beware of non-null-terminated registry strings


    Even though a value is stored in the registry as REG_SZ, this doesn't mean that the value actually ends with a proper null terminator. At the bottom, the registry is just a hierarchically-organized name/value database.

    And you can lie and get away with it.

    Lots of people lie about their registry data. You'll find lots of things that should be REG_DWORD stored as a four-byte REG_BINARY. (This is in part a holdover from Windows 95's registry, which didn't support REG_DWORD.)

    One of the most insidious lies is to lie about the length of a string you're writing to the registry. Consider the following program:

    #include <windows.h>
    #include <stdio.h>
    int __cdecl main(int argc, char **argv)
        RegSetValueExW(HKEY_CURRENT_USER, L"Scratch",
                       0, REG_SZ, (BYTE*)L"12", 2);
        DWORD cb = 0;
        RegQueryValueExW(HKEY_CURRENT_USER, L"Scratch",
                         NULL, NULL, NULL, &cb);
        printf("Size is %d bytes\n", cb);
        WCHAR sz[2];
        sz[0] = 0xFFFF;
        sz[1] = 0xFFFF;
        cb = sizeof(sz[0]);
        DWORD dwRc = RegQueryValueExW(HKEY_CURRENT_USER, L"Scratch",
                                      NULL, NULL, (BYTE*)sz, &cb);
        printf("RegQueryValueExW requesting %d bytes => %d\n",
               sizeof(sz), dwRc);
        printf("%d bytes required\n", cb);
        if (dwRc == ERROR_SUCCESS) {
            printf("sz[0] = %d\n", sz[0]);
            printf("sz[1] = %d\n", sz[1]);
        RegDeleteValueW(HKEY_CURRENT_USER, L"Scratch");
        return 0;

    If you run this program, you get this:

    Size is 2 bytes
    RegQueryValueExW requesting 4 bytes => 0
    2 bytes required
    sz[0] = 49
    sz[1] = 65535

    What happened?

    First, observe that the call to RegSetValueExW lies about the length of the string, claiming that it is two bytes long when in fact it is six! (Two WCHARs plus a terminator.)

    The registry dutifully records this lie and reports it back to subsequent callers.

    The first call to RegQueryValueExW asks how big the string is, and the registry reports the value 2, since that's the value it was given when the value was originally stored.

    To show that there really is no null terminator, we ask the registry to read those two bytes of data into our buffer, pre-filling the buffer with sentinel values so we can see what got updated and what didn't.

    Lo and behold, the values were read from the registry and only two bytes were read. sz[0] contains the character '1', and sz[1] remains uninitialized.

    This has security implications.

    If your program assumes that strings in the registry are always null-terminated, then you can be tricked into a buffer overflow if you happen across a non-null-terminated string. (For example, if you use strcpy to copy it around.)

    (Note: I'm not going to get into whether it should have been possible to get into this state in the first place. I didn't design the registry. Arguing about the past isn't going to change the present, and the present is that this is how it works so you'd better be ready for it.)

    Exercise: Change the last parameter of RegSetValueExW to 3 and run the program again. Explain the results and discuss its consequences.

  • The Old New Thing

    The kooky STRRET structure


    If you've messed with the shell namespace, you've no doubt run across the kooky STRRET structure, which is used by IShellFolder::GetDisplayNameOf to return names of shell items. As you can see from its documentation, a STRRET is sometimes an ANSI string buffer, sometimes a pointer to a UNICODE string, sometimes (and this is the kookiest bit) an offset into a pidl. What is going on here?

    The STRRET structure burst onto the scene during the Windows 95 era. Computers during this time were still comparatively slow and memory-constrained. (Windows 95's minimum hardware requirements were for 4MB of memory and a 386DX processor - which ran at a whopping 25MHz.) It was much faster to allocate memory off the stack (a simple "sub" instruction) than to allocate it from the heap (which might take thousands of instructions!), so the STRRET structure was designed so the common (for Windows 95) scenarios could be satisfied without needing a heap allocation.

    The STRRET_OFFSET flag took this to an even greater extreme. Often, you kept the name inside the pidl, and copying it into the STRRET structure would take, gosh, 200 clocks (!). To avoid this wasteful memory copying, STRRET_OFFSET allowed you to return just an offset into the pidl, which the caller could then copy out of directly.

    Woo-hoo, you saved a string copy.

    Of course, as time passed and computers got faster and memory became more readily available, these micro-optimizations have turned into annoyances. Saving 200 clock cycles on a string copy operation is hardly worth it any more. On a 1GHz processor, a single soft page fault costs you over a million cycles; a hard page fault costs you tens of millions.

    You can copy a lot of strings in twenty million cycles.

    What's more, the scenarios that were common in Windows 95 aren't quite so common any more, so the original scenario that the optimization was tailored for hardly occurs any more. It's an optimization that has outlived its usefulness.

    Fortunately, you don't have to think about the STRRET structure any more. There are several helper functions that take the STRRET structure and turn it into something much easier to manipulate.

    The kookiness of the STRRET structure has now been encapsulated away. Thank goodness.

  • The Old New Thing

    Finished competing in your event? Let the games begin!


    Ten thousand human beings in peak physical condition. All in one dormitory complex. With 130,000 free condoms. Let the games begin!

    And you have to tip your hat (tam?) to The Scotsman for finding an athlete named "Randy Jones" for this article.

  • The Old New Thing

    Summary of the recent spate of /3GB articles


    A table of contents now that the whole thing is over. I hope.

    I'm not sure how successful this series has been, though, for it appears that even people who have read the articles continue to confuse virtual address space with physical address space. (Or maybe this person is merely mocking a faulty argument? I can't tell for sure.)

  • The Old New Thing

    The curious interaction between PAE and NX


    Carmen Crincoli covered the interaction between PAE and NX on his own blog, so I'll merely incorporate his remarks by reference.

    (And notice again the concession to backwards compatibility. Without the backwards compatibility work, XP SP2 would have shipped with NX support and an asterisk, "* and those of you who have device drivers that are not PAE-ready will not be able to take advantage of these new security enhancements. We could've done something to make your systems secure, but we decided not to do it in order to teach you a lesson.")

  • The Old New Thing

    Writing your own menu-like window


    Hereby incorporating by reference the "FakeMenu" sample in the Platform SDK. It's in the winui\shell\fakemenu directory.

    For those who don't have the Platform SDK, what are you doing writing Win32 programs without the Platform SDK? Download it if it didn't come with your development tools.

    If for some reason you don't want the Platform SDK yet you want to write Win32 programs (I bet you're the sort of person who throws away the manual as soon as you buy something), you can look at the version that Chris Becke has stashed away on this page.

Page 380 of 430 (4,292 items) «378379380381382»