August, 2006

  • The Old New Thing

    Why can't I see all of the 4GB of RAM in my machine?

    • 34 Comments

    Here's a question that came from a customer:

    I have set my boot options to /3GB /NOPAE. Even though the server has 4GB of physical memory, we see only 3.5GB of physical memory on the server after it boots up. However, when we change the boot options to /PAE, we can observe 4GB of memory. Furthermore, when I enable only /PAE, even though the physical memory is 4GB, I can see 4GB. Without /PAE, even though the physical memory is 4GB, I can see 3.5GB. What is the reason for this?

    Remember that in the absence of the /PAE switch, the Windows memory manager is limited to a 4GB physical address space. Most of that address space is filled with RAM, but not all of it. Memory-mapped devices (such as your video card) will use some of that physical address space, as will the BIOS ROMs. After all the non-memory devices have had their say, there will be less than 4GB of address space available for RAM below the 4GB physical address boundary. The rest of it went above the boundary. Here's a picture:

    4.5GB
    4GB Potential RAM (0.5GB)   Requires PAE to see
    3.5GB Hardware/ROM (0.5GB)   Visible without PAE
    0GB Potential RAM (3.5GB)

    On this imaginary system, the motherboard assigned the ROMs and the hardware devices to the physical address space between 3.5GB and 4GB (occupying 0.5GB of address space). When you start plugging in your memory chips, then, they are assigned physical addresses starting at the bottom, and then skip over the address space that has already been assigned to the hardware and ROM, then resume.

    On this imaginary system, then, the 0.5GB of address space used for hardware and ROMs causes that much memory to get shoved upwards, and it ends up above the 4GB boundary. Without PAE, the processor is capable only of addressing memory below the 4GB boundary, which means that the memory above that boundary is inaccessible. It's consuming electricity but isn't doing anything.

    The solution is to enable /PAE so that the processor can access the physical address space above the 4GB boundary. Because that's what PAE stands for: Physical Address Extension.

  • The Old New Thing

    C# nested classes are like C++ nested classes, not Java inner classes

    • 24 Comments

    When you declare a class inside another class, the inner class still acts like a regular class. The nesting controls access and visibility, but not behavior. In other words, all the rules you learned about regular classes also apply to nested classes.

    The this keyword in an instance methods of a class (nested or not) can be used to access members of that class and only those members. It cannot be used to access members of other classes, at least not directly. (And the this can be omitted when it would not result in ambiguity.) You create an instance of a class (nested or not) by saying new ClassName(...) where ... are the parameters to an applicable class constructor.

    Java nested classes behave the same way, but Java also has the concept of inner classes. To construct an instance of an inner class in Java, you write new o.InnerClass(...) where ... as before are the parameters to an applicable class constructor. The o in front is an expression that evaluates to an object whose type is that of the outer class. The inner class can then use the this keyword to access its own members as well as those of the instance of the outer class to which it was bound.

    In C++ and C#, you will have to implement this effect manually. It's not hard, though:

    // Java
    class OuterClass {
     string s;
     // ...
     class InnerClass {
    
      public InnerClass() { }
      public string GetOuterString() { return s; }
     }
     void SomeFunction() {
      InnerClass i = new this.InnerClass();
      i.GetOuterString();
     }
    }
    
    // C#
    class OuterClass {
     string s;
     // ...
     class InnerClass {
      OuterClass o_;
      public InnerClass(OuterClass o) { o_ = o; }
      public string GetOuterString() { return o_.s; }
     }
     void SomeFunction() {
      InnerClass i = new InnerClass(this);
      i.GetOuterString();
     }
    }
    

    In Java, the inner class has a secret this$0 member which remembers the instance of the outer class to which it was bound. Creating an instance of an inner class via the new o.InnerClass(...) notation is treated as if you had written new InnerClass(o, ...), where o is automatically assigned to the secret this$0 member, and attempts to access members of the outer class are automatically treated as if they were written this$0.outermember. (This description of how inner classes are implemented is not just conceptual. It is spelled out in the language specification.)

    The C# equivalent to this code merely makes explicit the transformation that in Java was implicit. We give the inner class a reference to the outer class (here, we called it o_) and pass it as an explicit parameter to the inner class's constructor. And when we want to access a member of that outer class, we use o_ to do it.

    In other words, Java inner classes are syntactic sugar that is not available to C#. In C#, you have to do it manually.

    If you want, you can create your own sugar:

    class OuterClass {
     ...
     InnerClass NewInnerClass() {
      return new InnerClass(this);
     }
     void SomeFunction() {
      InnerClass i = this.NewInnerClass();
      i.GetOuterString();
     }
    }
    

    Where you would want to write in Java new o.InnerClass(...) you can write in C# either o.NewInnerClass(...) or new InnerClass(o, ...). Yes, it's just a bunch of moving the word new around. Like I said, it's just sugar.

    Now, I'm not saying that the Java way of representing inner classes isn't useful. It's a very nice piece of sugar if you access the outer class's members frequently from the inner class. However, it's not the type of transformation that makes you say, "Well, if a language doesn't support this, it's too hard for me to implement it manually, so I'll just give up." The conversion is not that complicated and consists entirely of local changes that can be performed without requiring a lot of thought.

    As a postscript, my colleague Eric Lippert points out that JScript.NET does have instance-bound inner classes.

    class Outer {
     var s;
     class Inner {
      function GetOuterString() {
       return s;
      }
     }
    }
    
    var o = new Outer();
    o.s = "hi";
    var i = new o.Inner();
    i.GetOuterString();
    
  • The Old New Thing

    The implementation of anonymous methods in C# and its consequences (part 1)

    • 50 Comments

    You may not even have realized that there are two types of anonymous methods. I'll call them the easy kind and the hard kind, not because they're actually easy and hard for you the programmer, but because they are easy and hard for the compiler.

    The easy kind is the anonymous method that doesn't use any local variables from its lexically-enclosing method. These are anonymous methods that could have been their own separate member functions; all the anonymization does is save you the trouble of coming up with names for them:

    class MyClass1 {
     int v = 0;
     delegate void MyDelegate(string s);
    
     MyDelegate MemberFunc()
     {
      int i = 1;
      return delegate(string s) {
              System.Console.WriteLine(s);
             };
      }
    }
    

    This particular anonymous method doesn't access any MyClass1 members, nor does it access the local variables of the MemberFunc function; therefore, it can be converted to a static method of the MyClass1 class:

    class MyClass1_converted {
     int v = 0;
     delegate void MyDelegate(string s);
    
     // Autogenerated by the compiler
     static void __AnonymousMethod$0(string s)
     {
      System.Console.WriteLine(s);
     }
    
     MyDelegate MemberFunc()
     {
      int i = 1;
      return __AnonymousMethod$0;
      // which is in turn shorthand for
      // return new MyDelegate(MyClass1.__AnonymousMethod$0);
      }
    }
    

    All the compiler did was give your anonymous methods a name and use that name in place of the "delegate (...) { ... }". (Note that all compiler-generated names I use here are purely illustrative. The actual compiler-generated name will be something different.)

    On the other hand, if your anonymous method used the this parameter, then that makes it an instance method instead of a static method:

    class MyClass2 {
     int v = 0;
     delegate void MyDelegate(string s);
    
     MyDelegate MemberFunc()
     {
      int i = 1;
      return delegate(string s) {
              System.Console.WriteLine("{0} {1}", v, s);
             };
      }
    }
    

    The anonymous method in MyClass2 uses the this keyword implicitly (to access the member variable v). Therefore, the conversion is to an instance member rather than to a static member.

    class MyClass2_converted {
     int v = 0;
     delegate void MyDelegate(string s);
    
     // Autogenerated by the compiler
     void __AnonymousMethod$0(string s)
     {
      System.Console.WriteLine("{0} {1}", v, s);
     }
    
     MyDelegate MemberFunc()
     {
      int i = 1;
      return this.__AnonymousMethod$0;
      // which is in turn shorthand for
      // return new MyDelegate(this.__AnonymousMethod$0);
      }
    }
    

    So far, we've only dealt with the easy cases. The transformation is local and not particularly complicated. These are the sorts of transformations you could make yourself without too much difficulty in the absence of anonymous methods.

    The hard case is where things get interesting. The body of an anonymous method is permitted to access the local variables of its lexically-enclosing method, in which case the compiler needs to keep those variables alive so that the body of your anonymous method can access them. Here's a sample anonymous method that accesses local variables from its lexically-enclosing method:

    class MyClass3 {
     int v = 0;
     delegate void MyDelegate(string s);
    
     MyDelegate MemberFunc()
     {
      int i = 1;
      return delegate(string s) {
              System.Console.WriteLine("{0} {1} {2}", i++, v, s);
             };
      }
    }
    

    In this example, the anonymous method prints "1 v s" the first time it is called, then "2 v s" the second time it is called, and so on, with the integer increasing by one. (And where v s are the current values of v and s, of course.) This happens because the i variable that the anonymous method is accessing is the same one each time, and it's the same i that the MemberFunc method was using, too. If the function were rewritten as

    class MyClass4 {
     int v = 0;
     delegate void MyDelegate(string s);
    
     MyDelegate MemberFunc()
     {
      int i = 0;
      MyDelegate d = delegate(string s) {
              System.Console.WriteLine("{0} {1} {2}", i++, v, s);
             };
      i = 1;
      return d;
      }
    }
    

    the behavior would be the same as in MyClass3. The creation of the delegate from the anonymous method does not make a copy of the i variable; changes to the i variable in the MemberFunc are visible to the anonymous method because both are accessing the same variable.

    When faced with this "hard" type of anonymous method, wherein variables are shared with the lexically-enclosing method, the compiler generates a helper class:

    class MyClass3_converted {
     int v = 0;
     delegate void MyDelegate(string s);
    
     // Autogenerated by the compiler
     class __AnonymousClass$0 {
      MyClass this$0;
      int i;
      public void __AnonymousMethod$0(string s)
      {
        System.Console.WriteLine("{0} {1} {2}", i++, this$0.v, s);
      }
     }
    
     MyDelegate MemberFunc()
     {
      __AnonymousClass$0 locals$ = new __AnonymousClass$0();
      locals$.this$0 = this;
      locals$.i = 0;
      return locals$.__AnonymousMethod$0;
      // which is in turn shorthand for
      // return new MyDelegate(locals$.__AnonymousMethod$0);
      }
    }
    

    Wow, there was a lot of rewriting this time. A helper class was created to contain the local variables that were shared between the MemberFunc function and the anonymous method (in this case, just the variable i), as well as the hidden this parameter (which I have called this$). In the MemberFunc function, access to that shared variable is done through this anonymous class, and the anonymous method that you wrote is an anonymous method on the anonymous class.

    Notice that the assignment to i in MemberFunc modifies the copy inside locals$, which is the same object that the anonymous method will be using when it runs. That's why it prints "1 v s" the first time: The value had already been changed to 1 by the time the delegate ran for the first time.

    Those who have done a good amount of C++ programming (or C# 1.0 programming) are well familiar with this technique, since C++ callbacks typically are given only one context variable; that context variable is usually a pointer to a larger structure that contains all the complex context you really want to operate on. C# 1.0 programmers went through a similar exercise. The "hard" type of anonymous method provides syntactic sugar that saves you the hassle of having to declare and manage the helper class.

    If you thought about it some, you'd have realized that the way it's done is pretty much the only way it could have been done. It turns out that most computer programming doesn't consist of being clever or making hard decisions. You just have one kernel of an idea ("hey let's have anonymous methods") and then the rest is just doing what has to be done, no actual decisions needed. You just do the obvious thing. Most programming consists of just doing the obvious thing.

    Okay, so that's a quick introduction to the implementation of anonymous methods in C#. Mind you, this information isn't just for your personal edification. It's actually important that you understand how these works (and not just treat it as "magic"), because lack of said understanding can lead to subtle programming errors. We'll look at those types of errors over the next few days.

  • The Old New Thing

    Environment variable expansion occurs when the command is read

    • 27 Comments

    On the command line (and in batch files), environment variable expansion occurs when the command is read. This sounds obvious at first, but it has its own consequences.

    In the online documentation for SET, one such consequence is spelled out:

        set VAR=before
        if "%VAR%" == "before" (
            set VAR=after
            if "%VAR%" == "after" @echo If you see this, it worked
        )
    

    would never display the message, since the %VAR% in both "if" statements is substituted when the first "if" statement is read, since it logically includes the body of the "if", which is a compound statement.

    In other words, the "if" command is not complete until the closing parenthesis is read. You can see this if you type the commands interactively:

    C:\>set VAR=before
    
    C:\>if "%VAR%" == "before" (
    More? set VAR=after
    More? if "%VAR%" == "after" @echo If you see this, it worked
    More? )
    
    C:\>
    

    Notice that the "if" command didn't execute until you closed the parenthesis; the command interpreter kept prompting "More?" to collect the body of the "if". This means that everything you type as the body of the "if" is evaluated before the "if" condition or any of the lines in the body are evaluated. It's as if you had typed

    C:\>if "before" == "before" (
    More? set VAR=after
    More? if "before" == "after" @echo If you see this, it worked
    More? )
    

    Note that this is different from most UNIX shells, which do not expand environment variables until the enclosing command is executed. For example,

    $ var=before
    $ var=after; echo $var
    after
    

    Notice that the $x is not expanded until the echo command's arguments are being computed. The analogous commands in the Windows command interpreter result in something quite different:

    C:\>set VAR=before
    
    C:\>set VAR=after & echo %VAR%
    before
    

    That's because the command interpreter expanded the environment variables at the time the line was read (not at the time the line is executed), yielding

    set VAR=after & echo before
    

    As a result, the old value of VAR is echoed. Some people treat this as a feature, allowing them to "restore" a variable without having to save it anywhere:

    set VAR=newvalue & call helper.cmd & set VAR=%VAR%
    

    This command sets the VAR variable to a new value, calls helper.cmd (which presumably uses the value of the %VAR% variable to control its behavior), then magically restores the variable to its original value since the %VAR% is expanded early, producing the old value.

    But what if you want the variable to be expanded at execution time rather than at parse time? For that, you use "delayed expansion", which is enabled by the /V command line option or by using the SETLOCAL ENABLEDELAYEDEXPANSION command in a batch file.

    C:\> copy con "%TEMP%\helper.cmd"
    SETLOCAL ENABLEDELAYEDEXPANSION
    set VAR=before
    set VAR=after & echo immediate:%VAR%, delayed:!VAR!
    ENDLOCAL
    ^Z
            1 file(s) copied.
    
    C:\> "%TEMP%\helper.cmd"
    
    C:\>SETLOCAL ENABLEDELAYEDEXPANSION
    
    C:\>set VAR=before
    
    C:\>set VAR=after   & echo immediate:before, delayed:!VAR!
    immediate:before, delayed:after
    
    C:\>ENDLOCAL
    

    Immediate expansion is performed with percent signs, whereas delayed expansion is performed with exclamation points.

    Why is immediate expansion the default? Because prior to Windows NT, that was the only type of expansion supported by the command interpreter. Retaining immediate expansion as the default preserved backwards compatibility with existing batch files. (The original command interpreter was written in assembly language. You really didn't want to be too clever or it would make your brain hurt trying to maintain the code. An interpreter loop of the form "Read a line, expand environment variables, evaluate" was therefore simple and effective.)

    Armed with this understanding of immediate versus delayed expansion, perhaps you can explain what is really going on here. (Hint: It has nothing to do with ERRORLEVEL.)

  • The Old New Thing

    The implementation of anonymous methods in C# and its consequences (part 3)

    • 38 Comments

    Last time we saw how the implementation details of anonymous methods can make themselves visible when you start taking a delegate apart by looking at its Target and Method. This time, we'll see how an innocuous code change can result in disaster due to anonymous methods.

    Occasionally, I see people arguing over where local variables should be declared. The "decentralists" believe that variables should be declared as close to their point of first use as possible:

    void MyFunc1()
    {
     ...
     for (int i = 0; i < 10; i++) {
      string s = i.ToString();
      ...
     }
     ...
    }
    

    On the other hand, the "consolidators" believe that local variables should be declared outside of loops.

    void MyFunc2()
    {
     ...
     string s;
     for (int i = 0; i < 10; i++) {
      s = i.ToString();
      ...
     }
     ...
    }
    

    The "consolidators" argue that hoisting the variable s means that the compiler only has to create the variable once, at function entry, rather than each time through the loop.

    As a result, you can find yourself caught in a struggle between the "decentralists" and the "consolidators" as members of each school touch a piece of code and "fix" the local variable declarations to suit their style.

    And then there are the "peacemakers" who step in and say, "Look, it doesn't matter. Can't we all just get along?"

    While I admire the desire to have everyone get along, the claim that it doesn't matter is unfortunately not always true. Let's stick some nasty code in where the dots are:

    delegate void MyDelegate();
    void MyFunc1()
    {
     MyDelegate d = null;
     for (int i = 0; i < 10; i++) {
      string s = i.ToString();
      d += delegate() {
       System.Console.WriteLine(s);
      };
     }
     d();
    }
    

    Since the s variable is declared inside the loop, each iteration of the loop gets its own copy of s, which means that each delegate gets its own copy of s. The first time through the loop, an s is created with the value "0" and that s is used by the first delegate. The second time through the loop, a new s is created with the value "1", and that new s is used by the second delegate. The result of this code fragment is ten delegates, each of which prints a different number from 0 to 9.

    Now, a "consolidator" looks at this code and says, "How inefficient, creating a new s each time through the loop. I shall hoist it and bask in the accolades of my countrymen."

    delegate void MyDelegate();
    void MyFunc2()
    {
     MyDelegate d = null;
     string s;
     for (int i = 0; i < 10; i++) {
      s = i.ToString();
      d += delegate() {
       System.Console.WriteLine(s);
      };
     }
     d();
    }
    

    If you run this fragment, you get different behavior. A single s variable is created for all the loop iterations to share. The first time through the loop, the value of s is "0", and then the first delegate is created. The second loop iteration changes the value of s to "1" before creating the second delegate. Repeat for the remaining eight delegates, and at the end of the loop, the value of s is "9", and ten delegates have been added to d. When d is invoked, all the delegates print the value of the s variable, which they are sharing and which has the value "9". The result: 9 is printed ten times.

    Now, I happen to have constructed this scenario to make the "consolidators" look bad, but I could also have written it to make the "decentralists" look bad for pushing a variable declaration into a loop scope when it should have remained outside. (All you have to do is read the above scenario in reverse.)

    The point of this little exercise is that when a "consolidator" or a "decentralist" goes through an entire program "tuning up" the declarations of local variables, the result can be a broken program, even though the person making the change was convinced that their change "had no effect; I was just making the code prettier / more efficient".

    What's the conclusion here?

    Write what you mean and mean what you write. If the precise scope of a variable is important, make sure to comment it as such so that somebody won't mess it up in a "clean-up" pass over your program. If there are two ways of writing the same thing, then write the one that is more maintainable. And if you feel that one method is superior from a performance point of view, then (1) make sure you're right, and (2) make sure it matters.

  • The Old New Thing

    Moving a file does not recalculate inherited permissions

    • 39 Comments

    Inherited permissions on an object are established when it is created. Once the object has been created, you can change the permissions of the parent and it won't have any effect unless you explicitly ask for the inheritable properties to be re-propagated to the child objects. (You may recall that the CREATOR_OWNER SID works in a similar way.) This rule applies to files, though that can lead to behavior that some people might consider non-intuitive.

    Files are strange from a security perspective because a single file can have multiple parent folders, thanks to hard links. Suppose you have two directories DirA and DirB. DirA has an inheritable permission that gives UserA full access and denies access to UserB, while DirB does the same with the roles of the two users reversed. It's easy to tell who the parent of a file is when it is created, since a file is created by giving a path, and you can extract the parent from the path. For example, if the file DirA\File is created, it will naturally inherit permissions from DirA.

    One the file is created, though, that's the end of it. Inheritable permissions don't have any effect any more.

    This simple rule wouldn't normally cause any problems, except that files have properties unusual among most named objects: They can go by multiple names (thanks to hard links) and can change their names (via renaming). The apparently counter-intuitive behavior stems from confusing the object with its name.

    For example, suppose we create a hard link to DirA\File under the name DirB\File. What effect does this have on the file's ACLs? Answer: None whatsoever. The inheritable ACLs on DirB aren't applied to the file since the file is not being created. Besides, you couldn't apply the inheritable ACLs from DirB even if you wanted to because it would create a paradox: The file resides in both DirA and DirB, but each of those directories contains contradictory inheritable permissions. What would the result be if you somehow managed to apply both of them?

    All right, then. We have no choice but to decide that a file's ACLs don't change when you create a hard link. Now delete the original link DirA\File, leaving just DirB\File. Does this change the ACLs now? If you believe that it should, then you're saying that inheritable ACLs can take effect even when nothing got created! After all, we didn't create a hard link; we deleted one.

    Okay, maybe you concede that deleting a hard link shouldn't affect the ACL. But what did we just do by creating a hard link and then deleting another one? The net effect is that we moved the file from DirA\File to DirB\File. Which brings us to our third example: Renaming/moving a file does not change its ACL.

    We've just rediscovered the simple rule that inheritable ACLs take effect only when a file is created. Nothing special happens when a new hard link is created or when the file is moved.

    Of course, that simple rule holds only when you look at the file system at a low level. Layers built on top of the low-level file system can end up complicating our simple rule.

    When you move a file across volumes with the MOVEFILE_COPY_ALLOWED flag, you're saying that "move the file if possible; if not, then convert it to a copy/delete operation". The copy operation creates a new file, which means that inheritable properties on the destination folder do take effect. But only if the file motion crosses volumes. If you're moving the file within the same volume, then the ACL remains unchanged. How confusing. When you pass the MOVEFILE_COPY_ALLOWED flag, you lose control of the ACL. (You actually lose control of much more than just the ACL. Since the file is being copied, none of the attributes from the original file are kept on the copy. The copy inherits its encryption and compression status from the new parent directory. The copy also gets a new owner, which has follow-on consequences for things like disk quota.)

    Another layer built on top of the low-level file system operations is the shell's copy engine. If you use SHFileOperation to move a file and pass the FOF_NOCOPYSECURITYATTRIBUTES flag, then it will not preserve the original ACLs on the moved files but will rather recalculate them from the destination's inheritable properties. (If you want to do the same thing in your own code, you can call the SetNamedSecurityInfo function, specifying that you want an empty, unprotected DACL.) In Windows XP, the shell decides whether or not to use this "naive mode" ACL management based on the following algorithm:

    • If the MoveSecurityAttributes policy is set, then the policy determines how ACLs are handled when files are moved. (ACLs are reset if set to "0" and are preserved if set to "1".)
    • Otherwise, the "Use simple file sharing" setting controls how ACLs are handled when files are moved. (ACLs are reset if simple file sharing is enabled and are preserved if simple file sharing is disabled.)
  • The Old New Thing

    Candidate for most obscure keyboard shortcut: Shift+F8

    • 28 Comments

    One of the most obscure keyboard shortcuts has got to be Shift+F8, which is used for listbox discontiguous extended selection. Man, what a mouthful. KB article Q301583 doesn't help matters by listing this keyboard shortcut under "Dialog box keyboard shortcuts" even though it isn't a dialog box keyboard shortcut. It's a listbox keyboard shortcut.

    If the listbox supports extended selections (via LBS_EXTENDEDSEL), then you can use the Shift+F8 shortcut to create discontiguous multiple selections from the keyboard. (Via the mouse, you can just Ctrl+Click to create a discontiguous multiple selection.) Type Shift+F8 once to enter extended selection mode, then use the arrow keys to select an item and press Ctrl+Space or Shift+Space to select (or deselect) it. When finished, type Shift+F8 again (or just move focus to some other window).

    And yes, this particular keyboard interface is pretty wacked out. A more natural mechanism would be to have Ctrl+Arrow move the focus without changing the selection, then using Ctrl+Space to select (or deselect) the focus item. Thankfully, the list view control went for that approach rather than emulating the crazy Shift+F8 keyboard shortcut.

  • The Old New Thing

    Sucking the exception pointers out of a stack trace

    • 20 Comments

    Often, you find yourself staring at a stack trace for a caught exception and want to see the original exception.

    ChildEBP RetAddr  Args to Child
    030c21d0 76df3448 00000000 030c6138 76db6b0d ntdll!DbgBreakPoint
    030c21dc 76db6b0d 030c2204 77b8d585 030c220c ole32!PeekMessageExceptionFilter+0x42
    030c21e4 77b8d585 030c220c 00000000 030c220c ole32!CCliModalLoop::MyPeekMessage+0x41
    030c220c 77f36992 030c25d0 030c6128 030c22e8 msvcrt!_except_handler3+0x61
    030c2230 77f36964 030c25d0 030c6128 030c22e8 ntdll!ExecuteHandler2+0x26
    030c22d8 77f36884 030c1000 030c22e8 00010007 ntdll!ExecuteHandler+0x24
    030c25b8 77f6e0dd 030c25d0 00000000 00000000 ntdll!RtlRaiseException+0x3d
    030c262c 77d3c239 77d4a4b6 77d3e2c5 030c3767 ntdll!RtlDeactivateActivationContextUnsafeFast+0x233
    030c2630 77d4a4b6 77d3e2c5 030c3767 030c26a0 USER32!UserCallWinProcCheckWow+0x167
    030c2634 77d3e2c5 030c3767 030c26a0 77d4a46f USER32!_NLG_Return2
    030c265c 77d3e288 030c57b4 ffffffff 030c2688 USER32!__local_unwind2+0x70
    030c2688 77f36992 030c26f8 030c57b4 030c27a4 USER32!_except_handler3+0xd5
    030c26ac 77f36964 030c26f8 030c57b4 030c27a4 ntdll!ExecuteHandler2+0x26
    030c2a74 77b8d36d 030c6128 77b8d36d 00000000 ntdll!ExecuteHandler+0x24
    030c2a9c 77b8d59d 030c6128 030c2ac0 00000000 msvcrt!__global_unwind2+0x18
    030c2ac0 77f36992 030c2ba4 030c6128 030c2bc0 msvcrt!_except_handler3+0x75
    030c2ae4 77f36964 030c2ba4 030c6128 030c2bc0 ntdll!ExecuteHandler2+0x26
    030c2b8c 77f36796 030c1000 030c2bc0 030c2ba4 ntdll!ExecuteHandler+0x24
    030c2b8c 77b7aa54 030c1000 030c2bc0 030c2ba4 ntdll!KiUserExceptionDispatcher+0xe
    030c3300 77b7b4dc 030c3324 7715b1b4 00000000 msvcrt!_woutput_l+0x18
    

    (You too can get symbols for operating system binaries, either by using the symbol server to get the symbols on-demand or, if you have a gigabyte of disk space, you can download symbol packages to get them all at one go. Even if you go for the symbol package, you still need the symbol server, since it gets updated with symbols for binaries that have been updated since the most recent service pack.)

    Here, we caught an exception in the PeekMessageExceptionFilter. What was the exception? Well, an exception filter receives a pointer to an EXCEPTION_POINTERS structure as its argument.

    typedef struct _EXCEPTION_POINTERS {
        PEXCEPTION_RECORD ExceptionRecord;
        PCONTEXT ContextRecord;
    } EXCEPTION_POINTERS, *PEXCEPTION_POINTERS;
    

    Here, we see that the parameter to PeekMessageExceptionFilter is

    030c21dc 76db6b0d 030c2204 77b8d585 030c220c ole32!PeekMessageExceptionFilter+0x42
    
    0:0000> dd 030c2204 l2
    030c2204 030c25d0 030c22e8
             -------- --------
             .exr     .cxr
    

    The first value points to the exception record and the second points to the context record. You can view the exception by typing .exr 030c25d0 and view the context for the exception (i.e., what code was executing when the exception occurred) by typing .cxr 030c22e8. Those two values also appear as the first and (go figure) third parameters to ExecuteHandler2.

    It so happens that doing the .exr on this particular exception record reports that the exception was c015000f which happens to be STATUS_SXS_EARLY_DEACTIVATION, and after setting the context to the exception context record, the stack goes

    ChildEBP RetAddr
    030c262c 77d3c239 77d4a4b6 77d3e2c5 030c3767 ntdll!RtlDeactivateActivationContextUnsafeFast+0x233
    030c2630 77d4a4b6 77d3e2c5 030c3767 030c26a0 USER32!UserCallWinProcCheckWow+0x167
    030c2634 77d3e2c5 030c3767 030c26a0 77d4a46f USER32!_NLG_Return2
    030c265c 77d3e288 030c57b4 ffffffff 030c2688 USER32!__local_unwind2+0x70
    030c2688 77f36992 030c26f8 030c57b4 030c27a4 USER32!_except_handler3+0xd5
    030c26ac 77f36964 030c26f8 030c57b4 030c27a4 ntdll!ExecuteHandler2+0x26
    030c2a74 77b8d36d 030c6128 77b8d36d 00000000 ntdll!ExecuteHandler+0x24
    030c2a9c 77b8d59d 030c6128 030c2ac0 00000000 msvcrt!__global_unwind2+0x18
    030c2ac0 77f36992 030c2ba4 030c6128 030c2bc0 msvcrt!_except_handler3+0x75
    030c2ae4 77f36964 030c2ba4 030c6128 030c2bc0 ntdll!ExecuteHandler2+0x26
    030c2b8c 77f36796 030c1000 030c2bc0 030c2ba4 ntdll!ExecuteHandler+0x24
    030c2b8c 77b7aa54 030c1000 030c2bc0 030c2ba4 ntdll!KiUserExceptionDispatcher+0xe
    030c3300 77b7b4dc 030c3324 7715b1b4 00000000 msvcrt!_woutput_l+0x18
    

    Wow, we took an exception while trying to handle another exception! (It so happens this was easy to spot in the original stack trace, but in the general case, the next outer exception may require digging.)

    Repeat the exercise with this next exception:

    0:000> .exr 030c2ba4
    ExceptionAddress: 77b7aa54 (msvcrt!_woutput_l+0x00000018)
       ExceptionCode: c00000fd (Stack overflow)
      ExceptionFlags: 00000000
    NumberParameters: 2
       Parameter[0]: 00000001
       Parameter[1]: 030c2e88
    
    0:000> .cxr 030c2bc0
    eax=030c33b0 ebx=00000000 ecx=0000005c edx=00000000 esi=030c33c4 edi=030c33c4
    eip=77b7aa54 esp=030c2e8c ebp=030c3300 iopl=0         nv up ei pl nz na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202
    msvcrt!_woutput_l+0x18:
    001b:77b7aa54 53          push    ebx
    

    Aha, so the SXS exception was triggered by a stack overflow. At this new context, you can use the "k" command to see how we got into this state.

    It so happens that this particular bug was, as predicted, a stack overflow caused by unintended recursion when a call to an off-thread COM object forced the calling thread to wait for the reply, during which time a new request came in. The precise details of the problem aren't important; the goal today was to show how you can suck the exception pointers out of the stack to see what Win32 exception was the source of the problem.

  • The Old New Thing

    Try to avoid having BOOL function parameters

    • 66 Comments

    Generally speaking, I believe that you should try to avoid giving functions a boolean parameter (BOOL, bool, etc.) unless the meaning of that boolean parameter is blatantly obvious. Examples of obvious meaning would be the second parameter to the EnableWindow function (TRUE obviously means the window is being enabled and FALSE means that it's being disabled) and the final parameter to ShowScrollBar (TRUE obviously means the scroll bar is being shown and FALSE means that it's being hidden). In both of these cases, the meaning of the boolean parameter is encoded in the name of the function itself.

    But for functions like CreateEvent, what does that first BOOL parameter mean? First, you have to realize that the first BOOL parameter controls whether you get an auto-reset event or a manual-reset one. Does FALSE create a manual-reset event? Or is that done by passing TRUE? I can never remember and I have to go looking it up each time. That first parameter should have been declared as, say, a DWORD or, even better, an enum with two legal values, EVENTTYPE_AUTORESET and EVENTTYPE_MANUALRESET.

    Even worse is that CreateEvent has two BOOL parameters. Like anybody reading the code later can remember which comes first.

    And the mystery bools keep coming. Consider, for example, StreamReader(Stream, bool). What does true mean? Or false? Heck if I know.

    Mind you, this is just my opinion. Others may disagree with me.

  • The Old New Thing

    The implementation of anonymous methods in C# and its consequences (part 2)

    • 15 Comments

    Last time we took a look at how anonymous methods are implemented. Today we'll look at a puzzle that can be solved with what we've learned. Consider the following program fragment:

    using System;
    
    class MyClass {
     delegate void DelegateA();
     delegate void DelegateB();
    
     static DelegateB ConvertDelegate(DelegateA d)
     {
      return (DelegateB)
       Delegate.CreateDelegate(typeof(DelegateB), d.Method);
     }
    
     static public void Main()
     {
      int i = 0;
      ConvertDelegate(delegate { Console.WriteLine(0); });
     }
    }
    

    The ConvertDelegate method merely converts a DelegateA to a DelegateB by creating a DelegateB with the same underlying method. Since the two delegate types use the same signature, this conversion goes off without a hitch.

    But now let's make a small change to that Main function:

     static public void Main()
     {
      int i = 0;
      // one character change - 0 becomes i
      ConvertDelegate(delegate { Console.WriteLine(i); });
     }
    

    Now the program crashes with a System.ArgumentException at the point where we try to create the DelegateB. What's going on?

    First, observe that the overload of Delegate.CreateDelegate that was used is one that can only be used to create delegates from static methods. Next, note that in Test1, the anonymous method references neither its own members nor any local variables from its lexically-enclosing method. Therefore, the resulting anonymous method is a "static anonymous method of the easy type". Since the anonymous method is a static member, the use of the "static members only" overload of Delegate.CreateDelegate succeeds.

    However, in Test2, the anonymous method dereferences the i variable from its lexically-enclosing method. This forces the anonymous method to be a "anonymous method of the hard type", and those anonymous methods use an anonymous instance member function of an anonymous helper class. As a result, d.Method is an instance method, and the chosen overload of Delegate.CreateDelegate throws an invalid parameter exception since it works only with static methods.

    The solution is to use a different overload of Delegate.CreateDelegate, one that work with either static or instance member functions.

     DelegateB ConvertDelegate(DelegateA d)
     {
      return (DelegateB)
       Delegate.CreateDelegate(typeof(DelegateB), d.Target, d.Method);
     }
    

    The Delegate.CreateDelegate(Type, Object, MethodInfo) overload creates a delegate for a static method if the Object parameter is null or a delegate for an instance method if the Object parameter is non-null. Hardly by coincidence, that is exactly what d.Target produces. If the original delegate is for a static method, then d.Target is null; otherwise, it is the object for which the instance method is to be invoked on.

    This fix, therefore, makes the ConvertDelegate function handle conversion of delegates for either static or instance methods. Which is a good thing, because it may now be called upon to convert delegates for instance methods as well as static ones.

    Okay, this time we were lucky that the hidden gotcha of anonymous methods resulted in an exception. Next time, we'll see a gotcha that merely results in incorrect behavior that will probably take you forever to track down.

Page 1 of 4 (39 items) 1234