• The Old New Thing

    Processes, commit, RAM, threads, and how high can you go?


    Back in 2008, Igor Levicki made a boatload of incorrect assumptions in an attempt to calculate the highest a process ID can go on Windows NT. Let's look at them one at a time.

    So if you can't create more than 2,028 threads in one process (because of 2GB per process limit) and each process needs at least one thread, that means you are capped by the amount of physical RAM available for stack.

    One assumption is that each process needs at least one thread. Really? What about a process that has exited? (Some people call these zombie processes.) There are no threads remaining in this process, but the process object hangs around until all handles are closed.

    Next, the claim is that you are capped by the amount of physical RAM available for stack. This assumes that stacks are non-pageable, which is an awfully strange assumption. User-mode stacks are most certainly pageable. In fact, everything in user-mode is pageable unless you take special steps to make it not pageable.

    Given that the smallest stack allocation is 4KB and assuming 32-bit address space:

    4,294,967,296 / 4,096 = 1,048,576 PIDs

    This assumes that all the stacks live in the same address space, but user mode stacks from different processes most certainly do not; that's the whole point of separate address spaces! (Okay, kernel stacks live in the same address space, but the discussion about "initial stack commit" later makes it clear he's talking about user-mode stacks.)

    Since they have to be a multiple of 4:

    1,048,576 / 4 = 262,144 PIDs

    It's not clear why we are dividing by four here. Yes, process IDs are a multiple of four (implementation detail, not contractual, do not rely on it), but that doesn't mean that three quarters of the stacks are no longer any good. It just means that we can't use more than 4,294,967,296/4 of them since we'll run out of names after 1,073,741,824 of them. In other words, this is not a division but rather a min operation. And we already dropped below 1 billion when we counted kernel stacks, so this min step has no effect.

    It's like saying, "This street is 80 meters long. The minimum building line is 4 meters, which means that you can have at most 20 houses on this side of the street. But house numbers on this side of the street must be even, so the maximum number of houses is half that, or 10." No, the requirement that house numbers be even doesn't cut the number of houses in half; it just means you have to be more careful how you assign the numbers.

    Having 262,144 processes would consume 1GB of RAM just for the initial stack commit assuming that all processes are single-threaded. If they commited 1MB of stack each you would need 256 GB of memory.

    Commit does not consume RAM. Commit is merely a promise from the memory manager that the RAM will there when you need it, but the memory manager doesn't have to produce it immediately (and certainly doesn't have to keep the RAM reserved for you until you free it). Indeed, that's the whole point of virtual memory, to decouple commit from RAM! (If commit consumed RAM, then what's the page file for?)

    This calculation also assumes that process IDs are allocated "smallest available first", but it's clear that it's not as simple as that: Fire up Task Manager and look at the highest process ID. (I've got one as high as 4040.) If process IDs are allocated smallest-available-first, then a process ID of 4040 implies that at some point there were 1010 processes in the system simultaneously! Unlikely.

    Here's a much simpler demonstration that process IDs are not allocated smallest-available-first: Fire up Task Manager, tell it to Show processes from all users, go to the Processes tab, and enable the PID column if you haven't already. Now launch Calc. Look for Calc in the process list and observe that it was not assigned the lowest available PID. If your system is like mine, you have PID zero assigned to the System Idle Process (not really a process but it gets a number anyway), and PID 4 assigned to the System process (again, not really a process but it gets a number anyway), and then you have a pretty big gap before the next process ID (for me, it's 372). And yet Calc was given a process ID in the 2000's. Proof by counterexample that the system does not assign PIDs smallest-available-first.

    So if they aren't assigned smallest-available-first, what's to prevent one from having a process ID of 4000000000?

    (Advanced readers may note that kernel stacks do all share a single address space, but even in that case, a thread that doesn't exist doesn't have a stack. And it's clear that Igor was referring to user-mode stacks since he talked about 1MB stack commits, a value which applies to user mode and not kernel mode.)

    Just for fun, I tried to see how high I could get my process ID.

    #include <windows.h>
    int __cdecl _tmain(int argc, TCHAR **argv)
     DWORD dwPid = 0;
     TCHAR szSelf[MAX_PATH];
     GetModuleFileName(NULL, szSelf, MAX_PATH);
     int i;
     for (i = 0; i < 10000; i++) {
      STARTUPINFO si = { 0 };
      if (!CreateProcess(szSelf, TEXT("Bogus"),
            &si, &pi)) break;
      TerminateProcess(pi.hProcess, 0);
      // intentionally leak the process handle so the
      // process object is not destroyed
      // CloseHandle(pi.hProcess); // leak
      if (dwPid < pi.dwProcessId) dwPid = pi.dwProcessId;
     _tprintf(_TEXT("\nCreated %d processes, ")
              _TEXT("highest pid seen was %d\n"), i, dwPid);
     _fgetts(szSelf, MAX_PATH, stdin);
     return 0;

    In order to get the program to complete before I got bored, I ran it on a Windows 2000 virtual machine with 128MB of memory. It finally conked out at 5245 processes with a PID high water mark of 21776. Along the way, it managed to consume 2328KB of non-paged pool, 36KB of paged pool, and 36,092KB of commit. If you divide this by the number of processes, you'll see that a terminated process consumes about 450 bytes of non-paged pool, a negligible amount of paged pool, and 6KB of commit. (The commit is probably left over page tables and other detritus.) I suspect commit is the limiting factor in the number of processes.

    I ran the same program on a Windows 7 machine with 1GB of RAM, and it managed to create all 10,000 processes with a high process ID of 44264. I cranked the loop limit up to 65535, and it still comfortably created 65535 processes with a high process Id of 266,232, easily exceeding the limit of 262,144 that Igor calculated.

    I later learned that the Windows NT folks do try to keep the numerical values of process ID from getting too big. Earlier this century, the kernel team experimented with letting the numbers get really huge, in order to reduce the rate at which process IDs get reused, but they had to go back to small numbers, not for any technical reasons, but because people complained that the large process IDs looked ugly in Task Manager. (One customer even asked if something was wrong with his computer.)

    That's not saying that the kernel folks won't go back and try the experiment again someday. After all, they managed to get rid of the dispatcher lock. Who knows what other crazy things will change next? (And once they get process IDs to go above 65535—like they were in Windows 95, by the way—or if they decided to make process IDs no longer multiples of 4 in order to keep process IDs low, this guy's program will stop working, and it'll be Microsoft's fault.)

  • The Old New Thing

    Why does SHGetSpecialFolderPath take such a long time before returning a network error?


    A customer reported that their program was failing to start up because the call to SHGet­Special­Folder­Path(CSIDL_PERSONAL) was taking a long time and then eventually returning with ERROR_BAD_NETPATH. The account that was experiencing this problem had a redirected network profile, "but even if he's redirecting, why would we get the bad net path error? Does calling SHGet­Folder­Path actually touch the folder/network? If so, we should probably stop calling this function on the UI thread since network problems could cause our program to hang."

    The SHGet­Folder­Path function will access the network if you pass the CSIDL_FLAG_CREATE flag, which says "Check if the folder is there, and if not, create it."

    The customer had been passing the flag. "We'll remove it. As if our program is going to dictate the creation of the user profile directory."

    The CSIDL_FLAG_CREATE flag has been implicated in some other unwanted behavior. For example, if you pass the CSIDL_FLAG_CREATE flag when asking for CSIDL_MYPICTURES, this will create a My Pictures directory if there wasn't one before. Generally speaking, you shouldn't be creating these directories as side-effects of other actions. Corporate administrators may suppress creation of folders like Pictures and Videos, but that doesn't do much good if your program casually creates them as part of its startup.

    Note that SHGet­Special­Folder­Path and CSIDL values have been superseded by SHGet­Known­Folder­Path and KNOWN­FOLDER­ID. The flag corresponding to CSIDL_FLAG_CREATE is KF_FLAG_CREATE. If you want to make things even faster, consider passing the KF_FLAG_DONT_VERIFY flag (formerly known as CSIDL_FLAG_DONT_VERIFY).

  • The Old New Thing

    From inside the Redmond Reality Distortion Field: Why publish documents in PDF?


    A few years ago, the Windows 7 team developed a document to introduce technology writers to the features of Windows 7. The document was released in PDF format, which created quite a stir among certain people trapped inside the Redmond Reality Distortion Field, who indignantly complained,

    Why are we releasing this document in PDF format? Shouldn't it be in docx or XPS? I would expect people interested in Windows 7 to be willing to use more Microsoft technology.

    Um, hello from the real world. It's the people who are critical of Windows 7 who are least likely to use Microsoft technology!

    "Okay, so Microsoft has this document telling me about their new product, but it's in some Microsoft proprietary file format that requires me to install a custom viewer that works only in Internet Explorer? You've gotta be kidding me."

    No wonder people hate Microsoft.

    It's like handing out brochures titled "Gründe, warum du Deutsch lernen solltest."

    Bonus plug: Stephen Toulouse bookified his blogerations. (Part 2: The Hardbackening.) I've read the softcopy of his book. Good stuff. And I would've endorsed his book even if he didn't promise me a personalized copy.

  • The Old New Thing

    Begin feeling


    A few years ago, the gas station near Microsoft's main campus (the one which had been run by my colleague in a previous stage of his career) appeared to have suffered some problems with the LCD unit on one of its pumps. Instead of "Please insert card", it said "@leace incebd cabd". As a geek, I quickly determined that bit 4 got wiped out in the ASCII codes for the characters in the message. Undaunted, I set about going through the usual steps for purchasing gasoline, though it took a little longer than normal because I first had to decode the corrupted strings. And then after I thought I had finished all the necessary preliminaries, the panel said, "Begin Feeling".

  • The Old New Thing

    2010 year-end link clearance


    Another round of the semi-annual link clearance.

    And, as always, the obligatory plug for my column in TechNet Magazine:

    • Beware the Balloon.
    • Hiding in Plain Sight.
    • History—the Long Way Through. In their zeal to make this article meet length, the editors cut what I consider to be the most important part of the article! Here's the penultimate paragraph in its full unedited version, with the important part underlined.
      But wait, there's still more. What if you want to access the real 64-bit system directory from a 32-bit process? File system redirection will take your attempt to access the C:\Windows\System32 directory and redirect it to the C:\Windows\SysWOW64 directory. Programmatically, you can use functions with unwieldy names like Wow64­Disable­Wow64­Fs­Redirection, but those disable redirection for all operations until re-enabled, which causes trouble if you're doing anything more complicated than opening a single file, because a complex operation may result in multiple files being accessed and possibly even worker threads being created. Instead of using a gross switch like disabling file system redirection, you can use the special C:\Windows\SysNative virtual directory. When a 32-bit process tries to access the C:\Windows\SysNative directory, the operations are redirected to the real C:\Windows\System32 directory. A local solution to a local problem.
    • Leftovers from Windows 3.0.
    • The Story of Restore.
    • The Tumultuous History of 'Up One Level'. The editors messed up the diagram in this article. The "1" is supposed to be an "open folder" icon, but due to the same error that results in that mysterious J, the Wingdings glyph turned into a plain "1". Here's what the diagram was supposed to look like. (Of course, if your browser is one who believes that Wingdings doesn't have a "1" glyph, then you'll just see a "1".)

      So for those of you looking for your Up One Level button, it's right there on the Address Bar. I've drawn a box around it so it's easier to see.

      Computer OS (C:) Windows Web Wallpaper
  • The Old New Thing

    What makes RealGetWindowClass so much more real than GetClassName?


    There's Get­Class­Name and then there's Real­Get­Window­Class. What makes Real­Get­Window­Class more real?

    Recall from last time that the Real... functions were added to support Windows accessibility. The goal with Real­Get­Window­Class is to help accessibility tools identify what kind of window it is working with, even if the application did a little disguising in the form of superclassing.

    If you ask Real­Get­Window­Class for the class name of a window, it digs through all the superclassing and returns the name of the base class (if the base class is one of the standard window manager classes). For example, if your application superclassed the button class, a call to Get­Class­Name would return Awesome­Button, but a call to Real­Get­Window­Class would return button. Returning the underlying window class allows accessibility tools to know that the user is interacting with some type of button control (albeit a customized one), so that it can adjust the interaction to something appropriate for buttons. Without Real­Get­Window­Class, the accessibility tool would just see Awesome­Button, and it would probably shrug and say, "I have no idea what a Awesome­Button is."

    (I guess you could have the accessibility tool do a strstr for button, but then it would be faked out by classes like Button­Bar or applications which superclass a button but call it something completely different like Awesome­Radio.)

    If you read the winuser.h header file, you can see a comment next to the Real­Get­Window­Class function:

     * This gets the name of the window TYPE, not class.  This allows us to
     * recognize ThunderButton32 et al.

    What is Thunder­Button32?

    Thunder was the code name for Visual Basic 1.0. Visual Basic superclassed all the standard Windows controls and called its superclassed version Thunder­Whatever.

  • The Old New Thing

    WindowFromPoint, ChildWindowFromPoint, RealChildWindowFromPoint, when will it all end?


    Oh wait, there's also ChildWindowFromPointEx.

    There are many ways of identifying the window that appears beneath a point. The documentation for each one describes how they work, but I figured I'd do a little compare/contrast to help you decide which one you want for your particular programming problem.

    The oldest functions are WindowFromPoint and ChildWindowFromPoint. The primary difference between them is that WindowFromPoint returns the deepest window beneath the point, whereas ChildWindowFromPoint returns the shallowest.

    What do I mean by deep and shallow?

    Suppose you have a top-level window P and a child window C. And suppose you ask one of the above functions, "What window is beneath this point?" when the point is squarely over window C. The WindowFromPoint function looks for the most heavily nested window that contains the point, which is window C. On the other hand ChildWindowFromPoint function looks for the least nested window that contains the point, which is window P, assuming you passed GetDesktopWindow as the starting point.

    That's the most important difference between the two functions, but there are others, primarily with how the functions treat hidden, disabled, and transparent windows. Some functions will pay attention to hidden, disabled, and/or transparent windows; others will skip them. Note that when a window is skipped, the entire window hierarchy starting from that window is skipped. For example, if you call a function that skips disabled windows, then all children of disabled windows will also be skipped (even if the children are enabled).

    Here we go in tabular form.

    Function Search Hidden? Disabled? Transparent?¹
    WindowFromPoint Deep Skip Skip It's Complicated²
    ChildWindowFromPoint Shallow Include Include Include
    ChildWindowFromPointEx Shallow Optional Optional Optional
    RealChildWindowFromPoint Shallow Skip Include Include³

    The return values for the various ...FromPoint... functions are the same:

    • Return the handle of the found window, if a window was found.
    • Return the handle of the parent window if the point is inside the parent window but not inside any of the children. (This rule obviously does not apply to WindowFromPoint since there is no parent window passed into the function.)
    • Otherwise, return NULL.

    The entries for ChildWindowFromPointEx are marked Optional because you, the caller, get to specify whether you want them to be skipped or included based on the CWP_* flags that you pass in.

    ¹There is a lot hiding behind the word Transparent because there are multiple ways a window can be determined transparent. The ...ChildWindowFromPoint... functions define transparent as has the WS_EX_TRANSPARENT extended window style.

    ²On the other hand, WindowFromPoint defines transparent as returns HTTRANSPARENT in response to WM_NCHITTEST. Actually, that's still not true. If the window belongs to a process thread different from the one calling WindowFromPoint, then WindowFromPoint will not send the message and will simply treat the window as opaque (i.e., not transparent).

    ³The RealChildWindowFromPoint includes transparent windows in the search, but has a special case for group boxes: The RealChildWindowFromPoint function skips over group boxes, unless the return value would have been the parent window, in which case it returns the group box after all.

    Why is RealChildWindowFromPoint so indecisive?

    The RealChildWindowFromPoint function was added as part of the changes to Windows to support accessibility. The intended audience for RealChildWindowFromPoint is accessibility tools which want to return a "reasonable" window beneath a specific point. Since group boxes usually enclose other controls, RealChildWindowFromPoint prefers to return one of the enclosed controls, but if the point belongs to the group box frame, then it'll return the group box.

    One place I see confusion over the various ...WindowFromPoint... functions is code which uses one of the functions, and then massages the result, unaware that there is already a function that returns the pre-massaged result for you. For example, I've seen code which calls WindowFromPoint followed by GetAncestor(GA_ROOT). This does a pointless down-and-up traversal of the window tree, searching for the deepest window that lies beneath the specified point, then walking back up the tree to convert it to a shallow window. This is the Rube Goldberg way of calling ChildWindowFromPointEx(GetDesktopWindow(), ...).

    Next time, a look at the mysterious RealGetWindowClass function. What makes this function more real?

  • The Old New Thing

    Psychic debugging: When I copy a file to the clipboard and then paste it, I get an old version of the file


    A customer reported the following strange problem:

    I tried to copy some text files from my computer to another computer on the network. After the copy completes, I looked at the network directory and found that while it did contain files with the same names as the ones I copied, they have completely wrong timestamps. Curious, I opened up the files and noticed that they don't even match the files I copied! Instead, they have yesterday's version of the files, not incorporating the changes that I made today. I still have both the source and destination folders open on my screen and can confirm that the files I copied really are the ones that I modified and not files from some backup directory.

    I tried copying it again but still an outdated version of the file gets copied. Curiously, the problem does not occur if I use drag/drop to copy the files. It happens only if I use Ctrl+C and Ctrl+V. Any ideas?

    This was indeed quite puzzling. One possibility was that the customer was mistakenly copying out of a Previous Versions folder. Before we could develop some additional theories, the customer provided additional information.

    I've narrowed down the problem. I've found that this has something to do with a clipboard tool I've installed. Without the tool running, everything is fine. How is it that with the tool running, Explorer is copying files through some sort of time machine? Those old versions of the files no longer exist on my computer; where is Explorer getting them from?

    Other people started investigation additional avenues, taking I/O trace logs, that sort of thing. Curiously, the I/O trace shows that while Explorer opened both the source and destination files and issued plenty of WriteFile calls to the destination, it never issued a ReadFile request against the source. An investigation of Previous Versions shows that there are no previous versions of the file recorded in the file system. It's as if the contents were being created from nothing.

    While the others were off doing their investigations, my head shuddered and I was sent into a trance state. A hollow voice emanated from my throat as my psychic powers manifested themselves. Shortly thereafter, my eyes closed and I snapped back to reality, at which point I frantically typed up the following note while I still remembered what had happened:

    My psychic powers tell me that this clipboard program is "virtualizing" the clipboard contents (replacing whatever is on the clipboard with its own data) and then trying (and failing) to regurgitate the original contents when the Paste operation asks for the file on the clipboard.

    A closer investigation of the clipboard enhancement utility showed that one of its features was the ability to record old clipboard contents and replay them (similar to the Microsoft Office Clipboard). Hidden inside the I/O operations was a query for the last-access time.

    And that's when things fell into place.

    Starting in Windows Vista, last access time is no longer updated by default. The program apparently saw that the file was never accessed and assumed that that meant that it also had never been modified, so it regenerated the file contents from its internal cache. (The quick fix for the program would be to switch to checking the last modified time instead of the last access time.)

    Upon learning that my psychic powers once again were correct, I realized that my prize for being correct was actually a penalty: Now even more people will ask me to help debug their mysterious problems.

  • The Old New Thing

    Windows 7 not only can make a wallpaper slide show from images on your computer, it can even pull them from an RSS feed


    Buried in the theme file documentation is a section called [Slideshow] which lets you control the source for images that are used when you put the desktop wallpaper in slideshow mode. And a bonus feature hidden in the [Slideshow] section is the ability to draw the images from an RSS feed. After creating the .theme file, double-click it and it will be added to the list of available themes.

    One thing about the RSS feed is that when you first set it up, it'll probably take a while for the initial images to download. You don't get any feedback that the images are still downloading; they just show up once they're ready. So don't freak out.

    Well, okay, if the images have already downloaded and you still don't see them, then maybe you can freak out. (Did you remember to select the theme after you added it to the list of available themes?)

  • The Old New Thing

    Why can't you use the space bar to select check box and radio button elements from a menu?


    Nektar wants to know why you can't use the space bar to select check box and radio button elements from a menu.

    The short answer is "Because it's a menu, not a dialog box."

    The check mark and radio button are just visual adornments provided by the menu manager as a courtesy; they do not affect the behavior of the menu itself. Notice, for example, that there is no way to specify "radio button groups" in a menu, so the menu manager wouldn't know which items needed to be deselected when you select a radio button menu item. (I guess it could infer them from separators, but then you would have people saying "I want my radio button group to exclude item number 4, but I don't want to put a separator in between; that looks ugly.")

    And then how would a program reject an auto-selected check box or radio button? E.g., the user pushes the space bar to turn on Show the Widget Bar and an error occurs trying to show the Widget Bar. If the program displays an error dialog, that would dismiss the menu. So maybe the program would just silently re-uncheck the box, which leaves the user puzzled as to why the space bar "doesn't work" for turning on the Widget Bar. Or worse, what if hiding the Widget Bar causes the menu to reconfigure itself? (Maybe there are some menu items that are visible only when the Widget Bar is selected; if the user hides the Widget Bar, those menu items need to be deleted.) Windows doesn't have a precedent for menus reconfiguring themselves while they're being displayed. What if one of the items that gets deleted when you hide the Widget Bar is the menu item that contains the Widget Bar checkbox? ("I turned off the Widget Bar, and my menu disappeared!")

    That said, there is no technical reason these design issues couldn't be overcome. You could have a style like MF_GROUP that behaves like WS_GROUP to establish the scope of menu radio buttons; you could have some way to mark a menu item as "this is an unselected check box" or "this is an unselected radio button"; you could come up with a way for a program to reject a user's attempt to change the check box status; you could design a way for menus to be dynamically reconfigured while they are open; you could even design a way for menus to respond in some vaguely reasonable way when the item the user just selected gets dynamically deleted! But all of these features take effort, and they detract from the simple design of a menu as "Here's a list of things you can do. Pick one. Once you pick one, the menu dismisses." Every feature starts out with minus 100 points and needs to beat out the other 200 items on the list.

Page 120 of 432 (4,312 items) «118119120121122»