August, 2004

  • The Old New Thing

    Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory

    • 40 Comments

    Virtual memory is not virtual address space (part 2).

    This myth is being perpetuated even as I write this series of articles.

    The user-mode virtual address space is normally 2GB, but that doesn't limit you to 2GB of virtual memory. You can allocate memory without it being mapped into your virtual address space. (Those who grew up with Expanded Memory or other forms of bank-switched memory are well-familiar with this technique.)

    HANDLE h = CreateFileMapping(INVALID_HANDLE_VALUE, 0,
                                 PAGE_READWRITE, 1, 0, NULL);
    

    Provided you have enough physical memory and/or swap file space, that 4GB memory allocation will succeed.

    Of course, you can't map it all into memory at once on a 32-bit machine, but you can do it in pieces. Let's read a byte from this memory.

    BYTE ReadByte(HANDLE h, DWORD offset)
    {
     SYSTEM_INFO si;
     GetSystemInfo(&si);
     DWORD chunkOffset = offset % si.dwAllocationGranularity;
     DWORD chunkStart = offset - chunkOffset;
     LPBYTE pb = (LPBYTE*)MapViewOfFile(h, FILE_MAP_READ, 0,
          chunkStart, chunkOffset + sizeof(BYTE));
     BYTE b = pb[chunkOffset];
     UnmapViewOfFile(pb);
     return b;
    }
    

    Of course, in a real program, you would have error checking and probably a caching layer in order to avoid spending all your time mapping and unmapping instead of actually doing work.

    The point is that virtual address space is not virtual memory. As we have seen earlier, you can map the same memory to multiple addresses, so the one-to-one mapping between virtual memory and virtual address space has already been violated. Here we showed that just because you allocated memory doesn't mean that it has to occupy any space in your virtual address space at all.

    [Updated: 10:37am, fix minor typos reported in comments.]

  • The Old New Thing

    Myth: Without /3GB the total amount of memory that can be allocated across all programs is 2GB

    • 10 Comments

    Virtual memory is not virtual address space (part 1).

    I don't know where this myth comes from; it's a non sequitur.

    Virtual address space describes how addresses are resolved, but since each process has its own virtual address space, the amount consumed by one program has no effect on that consumed by another program.

    Say you have a program that allocates 1GB of memory. Run three copies of it. Now you have a total of 3GB of allocated memory. And none of the programs came even close to exhausting its 2GB virtual address space allotment.

    Tomorrow, the debunking of a variation on this myth.

  • The Old New Thing

    Kernel address space consequences of the /3GB switch

    • 22 Comments

    One of the adverse consequences of the /3GB switch is that it forces the kernel to operate inside a much smaller space.

    One of the biggest casualties of the limited address space is the video driver. To manage the memory on the video card, the driver needs to be able to address it, and the apertures required are typically quite large. When the video driver requests a 256MB aperture, the call is likely to fail since there simply isn't that much address space available to spare.

    All of kernel's bookkeeping needs to fit inside that one gigabyte. Page tables, page directories, bitmaps, video driver apertures. It's a very tight squeeze, but if you're willing to cut back (for example by not requiring such a large video aperture), you can barely squeak it through. (A later entry will discuss another casualty of the reduced address space.)

    It's like trying to change your clothes inside a small closet. You can do it, but it's a real struggle, you're going to have to make sacrifices, and the results aren't always very pretty.

  • The Old New Thing

    The oft-misunderstood /3GB switch

    • 32 Comments

    It's simple to explain what it does, but people often misunderstand.

    The /3GB switch changes the way the 4GB virtual address space is split up. Instead of splitting it as 2GB of user mode virtual address space and 2GB of kernel mode virtual address space, the split is 3GB of user mode virtual address space and 1GB of kernel mode virtual address space.

    That's all.

    And yet people think it does more than that.

    I think the problem is that people think that "virtual address space" means something other than just "virtual address space".

    The term "address space" refers to how a numerical value (known as an "address") is interpreted when it is used to access some type of resource. There is a physical address space; each address in the physical address space refers to a byte in a memory chip somewhere. (Note for pedants: Yes, it's actually spread out over several memory chips, but that's not important here.) There is an I/O address space; each address in the I/O address space allows the CPU to communicate with a hardware device.

    And then there is the virtual address space. When people say "address space", they usually mean "virtual address space".

    The virtual address space is the set of possible pointer values (addresses) that can be used at a single moment by the processor. In other words, if you have an address like 0x12345678, the virtual address space determines what you get if you try to access that memory. The contents of the virtual address space changes over time, for example, as you allocate and free memory. It also varies based on context: each process has its own virtual address space.

    Saying that 2GB (or 3GB) of virtual address space is available to user mode means that at any given moment in time, out of the 4 billion virtual addresses available in a 32-bit value, 2 billion (or 3 billion) of them are potentially usable by user-mode code.

    Over the next few entries, I'll talk about the various consequences and misinterpretations of the /3GB switch.

  • The Old New Thing

    Why .shared sections are a security hole

    • 30 Comments

    Many people will recommend using shared data sections as a way to share data between multiple instances of an application. This sounds like a great idea, but in fact it's a security hole.

    Proper shared memory objects created by the CreateFileMapping function can be secured. They have security descriptors that let you specify which users are allowed to have what level of access. By contrast, anybody who loads your EXE or DLL gets access to your shared memory section.

    Allow me to demonstrate with an intentionally insecure program.

    Take the scratch program and make the following changes:

    #pragma comment(linker, "/SECTION:.shared,RWS")
    #pragma data_seg(".shared")
    int g_iShared = 0;
    #pragma data_seg()
    
    void CALLBACK TimerProc(HWND hwnd, UINT, UINT_PTR, DWORD)
    {
      int iNew = g_iShared + 1;
      if (iNew == 10) iNew = 0;
      g_iShared = iNew;
      InvalidateRect(hwnd, NULL, TRUE);
    }
    
    BOOL
    OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)
    {
        SetTimer(hwnd, 1, 1000, TimerProc);
        return TRUE;
    }
    
    void
    PaintContent(HWND hwnd, PAINTSTRUCT *pps)
    {
      TCHAR sz[2];
      wsprintf(sz, TEXT("%d"), g_iShared);
      TextOut(pps->hdc, 0, 0, sz, 1);
    }
    

    Go ahead and run this program. It counts from 0 to 9 over and over again. Since the TimerProc function never lets g_iShared go above 9, the wsprintf is safe from buffer overflow.

    Or is it?

    Run this program. Then use the runas utility to run a second copy of this program under a different user. For extra fun, make one of the users an administrator and another a non-administrator.

    Notice that the counter counts up at double speed. That's to be expected since the counter is shared.

    Okay, now close one of the copies and relaunch it under a debugger. (It's more fun if you let the administrator's copy run free and run the non-administrator's copy run under a debugger.) Let both programs run, then break into the debugger and change the value of the variable g_iShared to something really big, say, 1000000.

    Now, depending on how intrusive your debugger is, you might or might not see the crash. Some debuggers are "helpful" and "unshare" shared memory sections when you change their values from the debugger. Helpful for debugging (maybe), bad for my demonstration (definitely).

    Here's how I did it with the built-in ntsd debugger. I opened a command prompt, which runs as myself (and I am not an administrator). I then used the runas utility to run the scratch program as administrator. It is the administrator's copy of the scratch program that I'm going to cause to crash even though I am just a boring normal non-administrative user.

    From the normal command prompt, I typed "ntsd scratch" to run the scratch program under the debugger. From the debugger prompt, I typed "u TimerProc" to disassemble the TimerProc function, looking for

    01001143 a300300001       mov     [scratch!g_iShared (01003000)],eax
    
    (note: your numbers may differ). I then typed "g 1001143" to instruct the debugger to execute normally until that instruction is reached. When the debugger broke, I typed "r eax=12341234;t" to change the value of the eax register to 0x12341324 and then trace one instruction. That one-instruction trace wrote the out-of-range value into shared memory, and one second later, the administrator version of the program crashed with a buffer overflow.

    What happened?

    Since the memory is shared, all running copies of the scratch program have access to it. ALl I did was use the debugger to run a copy of the scratch program and change the value of the shared memory variable. Since the variable is shared, the value also changes in the administrator's copy of the program, which then causes the wsprintf buffer to overflow, thereby crashing the administrator's copy of the program.

    A denial of service is bad enough, but you can really do fun things if a program keeps anything of value in shared memory. If there is a pointer, you can corrupt the pointer. If there is a string, you can remove the null terminator and cause it to become "impossibly" long, resulting in a potential buffer overflow if somebody copies it without checking the length.

    And if there is a C++ object with a vtable, then you have just hit the mother lode! What you do is redirect the vtable to a bogus vtable (which you construct in the shared memory section), and put a function pointer entry in that vtable that points into some code that you generated (also into the shared memory section) that takes over the machine. (If NX is enabled, then the attack is much harder but still possible in principle.)

    Even if you can't trigger a buffer overflow by messing with variables in shared memory, you can still cause the program to behave erratically. Just scribbling random numbers all over the shared memory section will certainly induce "interesting" behavior in the program under attack.

    Moral of the story: Avoid shared memory sections. Since you can't attach an ACL to the section, anybody who can load your EXE or DLL can modify your variables and cause havoc in another instance of the program that is running at a higher security level.

  • The Old New Thing

    Spammers look stupid when they don't read the blog they spam on

    • 8 Comments

    Yesterday, I got a 419 scam via the contact form. This is a new low for spam stupidity. You'd think people who blog are more likely to be aware of Internet scams because they're reading the news, are clearly more comfortable with technology, and often make fun of news stories about 419 scammers.

    But if you're for real, I have this to say to you: Be careful! I'm sure you are being completely truthful when you say you are the only child of Deposed Corrupt Dictator XYZ, but be aware that there are many unscrupulous people claiming the same thing! So watch out for Vivian, Pierre, Lama, Austine, Martin, Kabilla, Jean Paul, Farek, David, Curimbada, Cynthia, and of course, Dictator Junior. It's very disappointing that there are so many impostors trying to get their hands on the fortune that you stole and that by rights belongs to you.

    Also watch out for XYZ's former Finance Minister, accountant, widow, brother, and hair stylist. They're after the same thing.

    Good luck.

    (I also find it impressive that you were able to embezzle $100 billion U.S. dollars, since your country raises less than $2 billion per year in tax revenues. How you managed to amass that much money so quickly is truly an amazing accomplishment of economics. If you returned even just half of that money back to the country in exchange for amnesty, that would fund your government for a generation. Think about it. Do it for the children.)

    P.S. I was joking about the hair stylist. Please don't hurt him.

  • The Old New Thing

    Never leave focus on a disabled control

    • 16 Comments

    One of the big no-no's in dialog box management is disabling the control that has focus without first moving focus somewhere else. When you do this, the keyboard becomes dead to the dialog box, since disabled windows do not receive input. For users who don't have a mouse (say, because they have physical limitations that confine them to the keyboard), this kills your dialog box.

    (I've seen this happen even in Microsoft software. It's very frustrating.)

    Before you disable a control, check whether it has focus. If so, then move focus somewhere else before you disable it, so that the user isn't left stranded.

    If you don't know which control focus should go to, you can always let the dialog manager decide. The WM_NEXTDLGCTL message once again comes to the rescue.

    void DialogDisableWindow(HWND hdlg, HWND hwndControl)
    {
      if (hwndControl == GetFocus()) {
        SendMessage(hdlg, WM_NEXTDLGCTL, 0, FALSE);
      }
      EnableWindow(hwndControl, FALSE);
    }
    

    (And of course you should never disable the last control on a dialog. That would leave the user completely stranded with no hope of escape!)

    [This was supposed to go out yesterday but the autoblog tool had a bad day and forgot to post this. Sorry.]

  • The Old New Thing

    How to set focus in a dialog box

    • 22 Comments

    Setting focus in a dialog box is more than just calling SetFocus.

    A dialog box maintains the concept of a "default button" (which is always a pushbutton). The default button is typically drawn with a distinctive look (a heavy outline or a different color) and indicates what action the dialog box will take when you hit Enter. Note that this is not the same as the control that has the focus.

    For example, open the Run dialog from the Start menu. Observe that the OK button is the default button; it has a different look from the other buttons. But focus is on the edit control. Your typing goes to the edit control, until you hit Enter; the Enter activates the default button, which is OK.

    As you tab through the dialog, observe what happens to the default button. When the dialog box moves focus to a pushbutton, that pushbutton becomes the new default button. But when the dialog box moves focus to something that isn't a pushbutton at all, the OK button resumes its position as the default button.

    The dialog manager remebers which control was the default button when the dialog was initially created, and when it moves focus to something that isn't a button, it restores that original button as the default button.

    You can ask a dialog box what the default button is by sending it the DM_GETDEFID message; similarly, you can change it with the DM_SETDEFID message.

    (Notice that the return value of the DM_GETDEFID message packs the control ID in the low word and flags in the high word. Another place where expanding dialog control IDs to 32-bit values doesn't buy you anything.)

    As the remarks to the DM_SETDEFID function note, messing directly with the default ID carelessly can lead to odd cases like a dialog box with two default buttons. Fortunately, you rarely need to change the default ID for a dialog.

    A bigger problem is using SetFocus to shove focus around a dialog. If you do this, you are going directly to the window manager, bypassing the dialog manager. This means that you can create "impossible" situations like having focus on a pushbutton without that button being the default!

    To avoid this problem, don't use SetFocus to change focus on a dialog. Instead, use the WM_NEXTDLGCTL message.

    void SetDialogFocus(HWND hdlg, HWND hwndControl)
    {
     SendMessage(hdlg, WM_NEXTDLGCTL, (WPARAM)hwndControl, TRUE);
    }
    

    As the remarks for the WM_NEXTDLGCTL message observe, the DefDlgProc function handles the WM_NEXTDLGCTL message by updating all the internal dialog manager bookkeeping, deciding which button should be default, all that good stuff.

    Now you can update dialog boxes like the professionals, avoiding oddities like having no default button, or worse, multiple default buttons!

  • The Old New Thing

    The company picnic, sponsored by Microsoft

    • 11 Comments

    Robert Scoble blogs from the Microsoft company picnic. Although he writes, "Microsoft holds a picnic for its employees every year," this suggests that the picnic is organized by Microsoft. It isn't.

    The company picnic is run by a professional company picnic company. They also do the Alaska Airlines company picnic, and probably those of many other organizations who don't put their picnic information on the public Internet. Wander down to Mountain Meadows Farm any weekend during the summer and you're likely to see a picnic or other group event under way.

Page 3 of 3 (29 items) 123