• The Old New Thing

    Suggestion Box 3

    • 469 Comments

    Post suggestions for future topics here instead of posting off-topic comments. Note that the suggestion box is emptied and read periodically so don't be surprised if your suggestion vanishes. (Note also that I am under no obligation to accept any suggestion.)

    Topics I are more inclined to cover:

    • Windows history (particularly the Windows 95 era).
    • Windows user interface programming in Win32, and shell programming in particular.
    • General programming topics (selectively).
    • Issues of general interest.
    • My personal hobbies.

    Topics I am not inclined to cover:

    • The blog software itself. You can visit the Community Server home page and cruise their support forums.
    • Internet Explorer. You can try the IE folks.
    • Visual Studio. You can try one of the Visual Studio blogs.
    • Managed code. This is not a .NET blog. I do not work on .NET technologies. As far as .NET is concerned, I'm just another programmer like you. Occasionally I touch a .NET-related topic, but I do not bring any expertise to the subject.
    • Non-software Microsoft topics, such as product support policies, marketing tactics, jobs and careers, legal issues.
    • Microsoft software that isn't Windows. (Exchange, Office, ...)
    • Windows topics outside user interface programming. (Plug and Play, Terminal Services, Windows Messenger, Outlook Express, SQL, IIS, remoting, SOA...)
    • User interface programming in anything other than Win32. (Because I know nothing about it.)
    • Debugging a specific problem. (Not of general interest.)
    • Predictions for the future. (What's the title of this blog again?)
    • Participation in Internet memes.

    You can also send feedback on Microsoft products directly to Microsoft. All the feedback gets read, even the death threats.

    Suggestions should be between two and four sentences in length. As you can see, there are hundreds of them already, so you have three seconds to get your point across. Please also search the blog first because your suggestion may have already been covered. And remember, questions aren't suggestions.

    Note the enormous topic backlog. Consequently, the suggestion box has been closed temporarily and will reopen once the existing backlog has cleared, which I estimate will happen sometime in early 2010. If your suggestion is that important, I'm sure you'll remember it when the suggestion box reopens.

  • The Old New Thing

    Capturing the current directory from a batch file

    • 37 Comments

    Sometimes people go to great lengths to get information which is available in a much simpler way. We saw it a few days ago when we found a 200+-line C# program that could be replaced with a 90-byte batch file. Here's another example of a rather roundabout way of capturing the current directory from a batch file.

    The easy way is to use the %CD% pseudo-variable. It expands to the current working directory.

    set OLDDIR=%CD%
    .. do stuff ..
    chdir /d %OLDDIR% &rem restore current directory
    

    (Of course, directory save/restore could more easily have been done with pushd/popd, but that's not the point here.)

    The %CD% trick is handy even from the command line. For example, I often find myself in a directory where there's a file that I want to operate on but... oh, I need to chdir to some other directory in order to perform that operation.

    set _=%CD%\curfile.txt
    cd ... some other directory ...
    somecommand args %_% args
    

    (I like to use %_% as my scratch environment variable.)

    Type SET /? to see the other pseudo-variables provided by the command processor.

  • The Old New Thing

    What does the COM Surrogate do and why does it always stop working?

    • 35 Comments

    The dllhost.exe process goes by the name COM Surrogate and the only time you're likely even to notice its existence is when it crashes and you get the message COM Surrogate has stopped working. What is this COM Surrogate and why does it keep crashing?

    The COM Surrogate is a fancy name for Sacrificial process for a COM object that is run outside of the process that requested it. Explorer uses the COM Surrogate when extracting thumbnails, for example. If you go to a folder with thumbnails enabled, Explorer will fire off a COM Surrogate and use it to compute the thumbnails for the documents in the folder. It does this because Explorer has learned not to trust thumbnail extractors; they have a poor track record for stability. Explorer has decided to absorb the performance penalty in exchange for the improved reliability resulting in moving these dodgy bits of code out of the main Explorer process. When the thumbnail extractor crashes, the crash destroys the COM Surrogate process instead of Explorer.

    In other words, the COM Surrogate is the I don't feel good about this code, so I'm going to ask COM to host it in another process. That way, if it crashes, it's the COM Surrogate sacrificial process that crashes instead of me process. And when it crashes, it just means that Explorer's worst fears were realized.

    In practice, if you get these types of crashes when browsing folders containing video or media files, the problem is most likely a flaky codec.

    Now that you know what the COM Surrogate does, you can answer this question from a customer:

    I'm trying to delete a file, but I'm told that "The action can't be completed because the file is open in COM Surrogate." What is going on?
  • The Old New Thing

    How does the calculator percent key work?

    • 49 Comments

    The Windows calculator percent sign works the same way as those cheap pocket calculators (which are often called four-function calculators even though they have around six function nowadays). What you first have to understand is that the percent key on those pocket calculators was not designed for mathematicians and engineers. It was designed for your everyday person doing some simple calculations. Therefore, the behavior of the key to you, an engineer, seems bizarrely counter-intuitive and even buggy. But to an everyday person, it makes perfect sense. Or at least that's the theory.

    Let's look at it from the point of view of that everyday person. Suppose you want to compute how much a $72 sweater will cost after including 5% tax.¹ Pull out your handy pocket calculator² (or fire up Calc if you don't have a pocket calculator) and type

    72 + 5% =

    The result is 75.6, or $75.60, which is the correct answer, because 5% of 72 is 3.6. Add that to 72 and you get 75.6.

    Similarly, suppose that sweater was on sale at 20% off. What is the sale price?

    72 − 20% =

    The result is 57.6 or $57.60. This is the correct answer, because 20% of 72 is 14.4. Subtract that from 72 and you get 57.6.

    You can chain these percentage operations, too. For example, how much will you have to pay for that 20%-off sweater after adding 5% tax?

    72 − 20% + 5% =

    The result is 60.48. A mathematician or engineer would have calculated the same result via the equivalent computation:

    72 × 0.80 × 1.05 =

    Okay, now that we see how the calculator product designer intended the percent key to be used, let's look at what the calculator engineer it has to do in order to match the specification. When the user enters A + B % =, the result should be A × (1 + B/100) or A + (A × B/100) after you distribute the multiplication over the addition. Similarly, when the user enters A − B % =, the result should be A × (1 − B/100) or A − (A × B/100).

    Aha, the calculator engineer says, we can achieve this result by defining the percent key as follows:

    When the user enters a value, an operator, a second value, and then the percent key, the first two values are multiplied and the product divided by 100, and that result replaces the second value in the ongoing computation.

    Let's walk through that algorithm with our first example.

    You typeRemarks
    72First value is 72
    +Operation is addition
    5Second value is 5
    %72 × 5 ÷ 100 = 3.6
    3.6 becomes the new second value
    =72 + 3.6 = 75.6, the final result

    If you watch the display as you go through this exercise, you will even see the number 3.6 appear in the display once you press the % key. The percentage is calculated and replaces the original value in the ongoing computation.

    This algorithm also works for the chained percentages.

    You typeRemarks
    72First value is 72
    Operation is subtraction
    20Second value is 20
    %72 × 20 ÷ 100 = 14.4
    14.4 becomes the new second value
    +72 − 14.4 = 57.6, intermediate result
    57.6 is the new first value
    Operation is addition
    5Second value is 5
    %57.6 × 5 ÷ 100 = 2.88
    2.88 becomes the new second value
    =57.6 + 2.88 = 60.48, the final result

    This even works for multiplication and division, but there is much less call for multiplying or dividing a number by a percentage of itself.

    500 × 5 % =

    The result of this is 12,500 because you are multiplying 500 by 5% of 500 (which is 25). The result of 500 × 25 is 12,500. You aren't computing five percent of 500. You're multiplying 500 by 5% of 500. (It appears that the authors of this Knowledge Base article didn't consult with the calculator engineer before writing up their analysis. The percent key is behaving as designed. The problem is that the percent key is not designed for engineers.)

    What if you want to compute 5% of 500? Just pick a dummy operation and view the result when you press the percent key.

    500 + 5 %

    When you hit the percent key, the answer appears: 25. You could've used the minus key, multiplication key, or division key instead of the addition key. It doesn't matter since all you care about is the percentage, not the combined operation. Once you hit the % key, you get your answer, and then you can hit Clear to start a new calculation.

    Footnotes

    ¹In the United States, quoted prices typically do not include applicable taxes.

    ²In my limited experiments, it appears that no two manufacturers of pocket calculators handle the percent key in exactly the same way. Casio appears to handle it in a manner closest to the engineering way. TI is closer to the layman algorithm. And when you get into cases like 1 ÷ 2 %, calculators start wandering all over the map. Should the answer be 50, since 1/2 is equal to 50%? Or should it be 0.005 since that is the numeric value of 0.5%? Should that answer appear immediately or should it wait for you to hit the equals sign? I don't know what the intuitive result should be either.

  • The Old New Thing

    What's the deal with the System Volume Information folder?

    • 53 Comments

    In the root of every drive is a folder called "System Volume Information". If your drive is NTFS, the permissions on the folder are set so not even administrators can get in there. What's the big secret?

    The folder contains information that casual interference could cause problems with proper system functioning. Here are some of the things kept in that folder. (This list is not comprehensive.)

    • System Restore points. You can disable System Restore from the "System" control panel.
    • Distributed Link Tracking Service databases for repairing your shortcuts and linked documents.
    • Content Indexing Service databases for fast file searches. This is also the source of the cidaemon.exe process: That is the content indexer itself, busy scanning your files and building its database so you can search for them quickly. (If you created a lot of data in a short time, the content indexer service gets all excited trying to index it.)
    • Information used by the Volume Snapshot Service (also known as "Volume Shadow Copy") so you can back up files on a live system.
    • Longhorn systems keep WinFS databases here.
  • The Old New Thing

    Why does Windows not recognize my USB device as the same device if I plug it into a different port?

    • 66 Comments

    You may have noticed that if you take a USB device and plug it into your computer, Windows recognizes it and configures it. Then if you unplug it and replug it into a different USB port, Windows gets a bout of amnesia and thinks that it's a completely different device instead of using the settings that applied when you plugged it in last time. Why is that?

    The USB device people explained that this happens when the device lacks a USB serial number.

    Serial numbers are optional on USB devices. If the device has one, then Windows recognizes the device no matter which USB port you plug it into. But if it doesn't have a serial number, then Windows treats each appearance on a different USB port as if it were a new device.

    (I remember that one major manufacturer of USB devices didn't quite understand how serial numbers worked. They gave all of their devices serial numbers, that's great, but they all got the same serial number. Exciting things happened if you plugged two of their devices into a computer at the same time.)

    But why does Windows treat it as a different device if it lacks a serial number and shows up on a different port? Why can't it just say, "Oh, there you are, over there on another port."

    Because that creates random behavior once you plug in two such devices. Depending on the order in which the devices get enumerated by Plug and Play, the two sets of settings would get assigned seemingly randomly at each boot. Today the settings match up one way, but tomorrow when the devices are enumerated in the other order, the settings are swapped. (You get similarly baffling behavior if you plug in the devices in different order.)

    In other words: Things suck because (1) things were already in bad shape—this would not have been a problem if the device had a proper serial number—and (2) once you're in this bad state, the alternative sucks more. The USB stack is just trying to make the best of a bad situation without making it any worse.

  • The Old New Thing

    Everybody thinks about garbage collection the wrong way

    • 89 Comments

    Welcome to CLR Week 2010. This year, CLR Week is going to be more philosophical than usual.

    When you ask somebody what garbage collection is, the answer you get is probably going to be something along the lines of "Garbage collection is when the operating environment automatically reclaims memory that is no longer being used by the program. It does this by tracing memory starting from roots to identify which objects are accessible."

    This description confuses the mechanism with the goal. It's like saying the job of a firefighter is "driving a red truck and spraying water." That's a description of what a firefighter does, but it misses the point of the job (namely, putting out fires and, more generally, fire safety).

    Garbage collection is simulating a computer with an infinite amount of memory. The rest is mechanism. And naturally, the mechanism is "reclaiming memory that the program wouldn't notice went missing." It's one giant application of the as-if rule.¹

    Now, with this view of the true definition of garbage collection, one result immediately follows:

    If the amount of RAM available to the runtime is greater than the amount of memory required by a program, then a memory manager which employs the null garbage collector (which never collects anything) is a valid memory manager.

    This is true because the memory manager can just allocate more RAM whenever the program needs it, and by assumption, this allocation will always succeed. A computer with more RAM than the memory requirements of a program has effectively infinite RAM, and therefore no simulation is needed.

    Sure, the statement may be obvious, but it's also useful, because the null garbage collector is both very easy to analyze yet very different from garbage collectors you're more accustomed to seeing. You can therefore use it to produce results like this:

    A correctly-written program cannot assume that finalizers will ever run at any point prior to program termination.

    The proof of this is simple: Run the program on a machine with more RAM than the amount of memory required by program. Under these circumstances, the null garbage collector is a valid garbage collector, and the null garbage collector never runs finalizers since it never collects anything.

    Garbage collection simulates infinite memory, but there are things you can do even if you have infinite memory that have visible effects on other programs (and possibly even on your program). If you open a file in exclusive mode, then the file will not be accessible to other programs (or even to other parts of your own program) until you close it. A connection that you open to a SQL server consumes resources in the server until you close it. Have too many of these connections outstanding, and you may run into a connection limit which blocks further connections. If you don't explicitly close these resources, then when your program is run on a machine with "infinite" memory, those resources will accumulate and never be released.

    What this means for you: Your programs cannot rely on finalizers keeping things tidy. Finalizers are a safety net, not a primary means for resource reclamation. When you are finished with a resource, you need to release it by calling Close or Disconnect or whatever cleanup method is available on the object. (The IDisposable interface codifies this convention.)

    Furthermore, it turns out that not only can a correctly-written program not assume that finalizers will run during the execution of a program, it cannot even assume that finalizers will run when the program terminates: Although the .NET Framework will try to run them all, a bad finalizer will cause the .NET Framework to give up and abandon running finalizers. This can happen through no fault of your own: There might be a handle to a network resource that the finalizer is trying to release, but network connectivity problems result in the operation taking longer than two seconds, at which point the .NET Framework will just terminate the process. Therefore, the above result can be strengthened in the specific case of the .NET Framework:

    A correctly-written program cannot assume that finalizers will ever run.

    Armed with this knowledge, you can solve this customer's problem. (Confusing terminology is preserved from the original.)

    I have a class that uses Xml­Document. After the class is out of scope, I want to delete the file, but I get the exception System.IO.Exception: The process cannot access the file 'C:\path\to\file.xml' because it is being used by another process. Once the progam exits, then the lock goes away. Is there any way to avoid locking the file?

    This follow-up might or might not help:

    A colleague suggested setting the Xml­Document variables to null when we're done with them, but shouldn't leaving the class scope have the same behavior?

    Bonus chatter: Finalizers are weird, since they operate "behind the GC." There are also lots of classes which operate "at the GC level", such as Weak­Reference GC­Handle and of course System.GC itself. Using these classes properly requires understanding how they interact with the GC. We'll see more on this later.

    Related reading

    Unrelated reading: Precedence vs. Associativity Vs. Order.

    Footnote

    ¹ Note that by definition, the simulation extends only to garbage-collected resources. If your program allocates external resources those external resources continue to remain subject to whatever rules apply to them.

  • The Old New Thing

    Does Windows have a limit of 2000 threads per process?

    • 30 Comments

    Often I see people asking why they can't create more than around 2000 threads in a process. The reason is not that there is any particular limit inherent in Windows. Rather, the programmer failed to take into account the amount of address space each thread uses.

    A thread consists of some memory in kernel mode (kernel stacks and object management), some memory in user mode (the thread environment block, thread-local storage, that sort of thing), plus its stack. (Or stacks if you're on an Itanium system.)

    Usually, the limiting factor is the stack size.

    #include <stdio.h>
    #include <windows.h>
    
    DWORD CALLBACK ThreadProc(void*)
    {
     Sleep(INFINITE);
     return 0;
    }
    
    int __cdecl main(int argc, const char* argv[])
    {
    int i;
     for (i = 0; i < 100000; i++) {
      DWORD id;
      HANDLE h = CreateThread(NULL, 0, ThreadProc, NULL, 0, &id);
      if (!h) break;
      CloseHandle(h);
     }
     printf("Created %d threads\n", i);
     return 0;
    }
    

    This program will typically print a value around 2000 for the number of threads.

    Why does it give up at around 2000?

    Because the default stack size assigned by the linker is 1MB, and 2000 stacks times 1MB per stack equals around 2GB, which is how much address space is available to user-mode programs.

    You can try to squeeze more threads into your process by reducing your stack size, which can be done either by tweaking linker options or manually overriding the stack size passed to the CreateThread functions as described in MSDN.

      HANDLE h = CreateThread(NULL, 4096, ThreadProc, NULL,
                   STACK_SIZE_PARAM_IS_A_RESERVATION, &id);
    

    With this change, I was able to squeak in around 13000 threads. While that's certainly better than 2000, it's short of the naive expectation of 500,000 threads. (A thread is using 4KB of stack in 2GB address space.) But you're forgetting the other overhead. Address space allocation granularity is 64KB, so each thread's stack occupies 64KB of address space even if only 4KB of it is used. Plus of course you don't have free reign over all 2GB of the address space; there are system DLLs and other things occupying it.

    But the real question that is raised whenever somebody asks, "What's the maximum number of threads that a process can create?" is "Why are you creating so many threads that this even becomes an issue?"

    The "one thread per client" model is well-known not to scale beyond a dozen clients or so. If you're going to be handling more than that many clients simultaneously, you should move to a model where instead of dedicating a thread to a client, you instead allocate an object. (Someday I'll muse on the duality between threads and objects.) Windows provides I/O completion ports and a thread pool to help you convert from a thread-based model to a work-item-based model.

    Note that fibers do not help much here, because a fiber has a stack, and it is the address space required by the stack that is the limiting factor nearly all of the time.

  • The Old New Thing

    On 64-bit Windows, 32-bit programs run in an emulation layer, and if you don't like that, then don't use the emulator

    • 43 Comments

    On 64-bit Windows, 32-bit programs run in an emulation layer. This emulation layer simulates the x86 architecture, virtualizing the CPU, the file system, the registry, the environment variables, the system information functions, all that stuff. If a 32-bit program tries to look at the system, it will see a 32-bit system. For example, if the program calls the GetSystemInfo function to see what processor is running, it will be told that it's running on a 32-bit processor, with a 32-bit address space, in a world with a 32-bit sky and 32-bit birds in the 32-bit trees.

    And that's the point of the emulation: To keep the 32-bit program happy by simulating a 32-bit execution environment.

    Commenter Koro is writing an installer in the form of a 32-bit program that detects that it's running on a 64-bit system and wants to copy files (and presumably set registry entries and do other installery things) into the 64-bit directories, but the emulation layer redirects the operations into the 32-bit locations. The question is "What is the way of finding the x64 Program Files directory from a 32-bit application?"

    The answer is "It is better to work with the system than against it." If you're a 32-bit program, then you're going to be fighting against the emulator each time you try to interact with the outside world. Instead, just recompile your installer as a 64-bit program. Have the 32-bit installer detect that it's running on a 64-bit system and launch the 64-bit installer instead. The 64-bit installer will not run in the 32-bit emulation layer, so when it tries to copy a file or update a registry key, it will see the real 64-bit file system and the real 64-bit registry.

  • The Old New Thing

    The implementation of iterators in C# and its consequences (part 1)

    • 33 Comments

    Like anonymous methods, iterators in C# are very complex syntactic sugar. You could do it all yourself (after all, you did have to do it all yourself in earlier versions of C#), but the compiler transformation makes for much greater convenience.

    The idea behind iterators is that they take a function with yield return statements (and possible some yield break statements) and convert it into a state machine. When you yield return, the state of the function is recorded, and execution resumes from that state the next time the iterator is called upon to produce another object.

    Here's the basic idea: All the local variables of the iterator (treating iterator parameters as pre-initialized local variables, including the hidden this parameter) become member variables of a helper class. The helper class also has an internal state member that keeps track of where execution left off and an internal current member that holds the object most recently enumerated.

    class MyClass {
     int limit = 0;
     public MyClass(int limit) { this.limit = limit; }
    
     public IEnumerable<int> CountFrom(int start)
     {
      for (int i = start; i <= limit; i++) {
       yield return i;
      }
     }
    }
    

    The CountFrom method produces an integer enumerator that spits out the integers starting at start and continuing up to and including limit. The compiler internally converts this enumerator into something like this:

     class MyClass_Enumerator : IEnumerable<int> {
      int state$0 = 0;// internal member
      int current$0;  // internal member
      MyClass this$0; // implicit parameter to CountFrom
      int start;      // explicit parameter to CountFrom
      int i;          // local variable of CountFrom
    
      public int Current {
       get { return current$0; }
      }
    
      public bool MoveNext()
      {
       switch (state$0) {
       case 0: goto resume$0;
       case 1: goto resume$1;
       case 2: return false;
       }
    
     resume$0:;
       for (i = start; i <= this$0.limit; i++) {
        current$0 = i;
        state$0 = 1;
        return true;
     resume$1:;
       }
    
       state$0 = 2;
       return false;
      }
      ... other bookkeeping, not important here ...
     }
    
     public IEnumerable<int> CountFrom(int start)
     {
      MyClass_Enumerator e = new MyClass_Enumerator();
      e.this$0 = this;
      e.start = start;
      return e;
     }
    

    The enumerator class is auto-generated by the compiler and, as promised, it contains two internal members for the state and current object, plus a member for each parameter (including the hidden this parameter), plus a member for each local variable. The Current property merely returns the current object. All the real work happens in MoveNext.

    To generate the MoveNext method, the compiler takes the code you write and performs a few transformations. First, all the references to variables and parameters need to be adjusted since the code moved to a helper class.

    • this becomes this$0, because inside the rewritten function, this refers to the auto-generated class, not the original class.
    • m becomes this$0.m when m is a member of the original class (a member variable, member property, or member function). This rule is actually redundant with the previous rule, because writing the name of a class member m without a prefix is just shorthand for this.m.
    • v becomes this.v when v is a parameter or local variable. This rule is actually redundant, since writing v is the same as this.v, but I call it out explicitly so you'll notice that the storage for the variable has changed.

    The compiler also has to deal with all those yield return statements.

    • Each yield return x becomes
       current$0 = x;
       state$0 = n;
       return true;
      resume$n:;
      

      where n is an increasing number starting at 1.

    And then there are the yield break statements.

    • Each yield break becomes
       state$0 = n2;
       return false;
      
      where n2 is one greater than the highest state number used by all the yield return statements. Don't forget that there is also an implied yield break at the end of the function.

    Finally, the compiler puts the big state dispatcher at the top of the function.

    • At the start of the function, insert
      switch (state$0) {
      case 0: goto resume$0;
      case 1: goto resume$1;
      case 2: goto resume$2;
      ...
      case n: goto resume$n;
      case n2: return false;
      }
      

      with one case statement for each state, plus the initial zero state and the final n2 state.

    Notice that this transformation is quite different from the enumeration model we built based on coroutines and fibers. The C# method is far more efficient in terms of memory usage since it doesn't consume an entire stack (typically a megabyte in size) like the fiber approach does. Instead it just borrows the stack of the caller, and anything that it needs to save across calls to MoveNext are stored in a helper object (which goes on the heap rather than the stack). This fake-out is normally quite effective—most people don't even realize that it's happening—but there are places where the difference is significant, and we'll see that shortly.

    Exercise: Why do we need to write state$0 = n2; and add the case n2: return false;? Why can't we just transform each yield break into return false; and stop there?

Page 1 of 432 (4,318 items) 12345»