August, 2009

  • The Old New Thing

    Why can't I declare a type that derives from a generic type parameter?

    • 18 Comments

    A lot of questions about C# generics come from the starting point that they are just a cutesy C# name for C++ templates. But while the two may look similar in the source code, they are actually quite different.

    C++ templates are macros on steroids. No code gets generated when a template is "compiled"; the compiler merely hangs onto the source code, and when you finally instantiate it, the actual type is inserted and code generation takes place.

    // C++ template
    template<class T>
    class Abel
    {
    public:
     int DoBloober(T t, int i) { return t.Bloober(i); }
    };
    

    This is a perfectly legal (if strange) C++ template class. But when the compiler encounters this template, there are a whole bunch of things left unknown. What is the return type of T::Bloober? Can it be converted to an int? Is T::Bloober a static method? An instance method? A virtual instance method? A method on a virtual base class? What is the calling convention? Does T::Bloober take an int argument? Or maybe it's a double? Even stranger, it might accept a Canoe which gets converted from an int by a converting constructor. Or maybe it's a function that takes two parameters, but the second parameter has a default value.

    Nobody knows the answers to these questions, not even the compiler. It's only when you decide to instantiate the template

    Abel<Baker> abel;
    

    that these burning questions can be answered, overloaded operators can be resolved, conversion operators can be hunted down, parameters can get pushed on the stack in the correct order, and the correct type of call instruction can be generated.

    In fact, the compiler doesn't even care whether or not Baker has a Bloober method, as long as you never call Abel<Baker>::DoBloober!

    void f()
    {
     Abel<int> a; // no error!
    }
    
    void g()
    {
     Abel<int> a;
     a.DoBloober(0, 1); // error here
    }
    
    Only if you actually call the method does the compiler start looking for how it can generate code for the DoBloober method.

    C# generics aren't like that.

    Unlike C++, where a non-instantiated template exists only in the imaginary world of potential code that could exist but doesn't, a C# generic results in code being generated, but with placeholders where the type parameter should be inserted.

    This is why you can use generics implemented in another assembly, even without the source code to that generic. This is why a generic can be recompiled without having to recompile all the assemblies that use that generic. The code for the generic is generated when the generic is compiled. By comparison no code is generated for C++ templates until the template is instantiated.

    What this means for C# generics is that if you want to do something with your type parameter, it has to be something that the compiler can figure out how to do without knowing what T is. Let's look at the example that generated today's question.

    class Foo<T>
    {
     class Bar : T
     { ... }
    }
    

    This is flagged as an error by the compiler:

    error CS0689: Cannot derive from 'T' because it is a type parameter
    

    Deriving from a generic type parameter is explicitly forbidden by 25.1.1 of the C# language specification. Consider:

    class Foo<T>
    {
     class Bar : T
     {
       public void FlubberMe()
       {
         Flubber(0);
       }
     }
    }
    

    The compiler doesn't have enough information to generate the IL for the FlubberMe method. One possibility is

    ldarg.0        // "this"
    ldc.i4.0    // integer 0 - is this right?
    call T.Flubber // is this the right type of call?
    

    The line ldc.i4.0 is a guess. If the method T.Flubber were actually void Flubber(long l), then the line would have to be ldc.i4.0; conv.i8 to load an 8-byte integer onto the stack instead of a 4-byte integer. Or perhaps it's void Flubber(object o), in which case the zero needs to be boxed.

    And what about that call instruction? Should it be a call or callvirt?

    And what if the method returned a value, say, string Flubber(int i)? Now the compiler also has to generate code to discard the return value from the top of the stack.

    Since the source code for a generic is not included in the assembly, all these questions have to be answered at the time the generic is compiled. Besides, you can write a generic in Managed C++ and use it from VB.NET. Even saving the source code won't be much help if the generic was implemented in a language you don't have the compiler for!

  • The Old New Thing

    Why doesn't String.Format throw a FormatException if you pass too many parameters?

    • 19 Comments

    Welcome to CLR Week 2009. As always, we start with a warm-up.

    The String.Format method doesn't throw a FormatException if you pass too many parameters, but it does if you pass too few. Why the asymmetry?

    Well, this is the type of asymmetry you see in the world a lot. You need a ticket for each person that attends a concert. If you have too few tickets, they won't let you in. If you have too many, well, that's a bit wasteful, but you can still get in; the extras are ignored. If you create an array with 10 elements and use only the first five, nobody is going to raise an ArrayBiggerThanNecessary exception. Similarly, the String.Format message doesn't mind if you pass too many parameters; it just ignores the extras. There's nothing harmful about it, just a bit wasteful.

    Besides, you probably don't want this to be an error:

    if (verbose) {
      format = "{0} is not {1} (because of {2})";
    } else {
      format = "{0} not {1}";
    }
    String.Format(format, "Zero", "One", "Two");
    

    Think of the format string as a SELECT clause from the dataset provided by the remaining parameters. If your table has fields ID and NAME and you select just the ID, there's nothing wrong with that. But if you ask for DATE, then you have an error.

  • The Old New Thing

    Why can't I pass a reference to a derived class to a function that takes a reference to a base class by reference?

    • 23 Comments

    "Why can't I pass a reference to a derived class to a function that takes a reference to a base class by reference?" That's a confusing question, but it's phrased that way because the simpler phrasing is wrong!

    Ths misleading simplified phrasing of the question is "Why can't I pass a reference to a derived class to a function that takes a base class by reference?" And in fact the answer is "You can!"

    class Base { }
    class Derived : Base { }
    
    class Program {
      static void f(Base b) { }
    
      public static void Main()
      {
          Derived d = new Derived();
          f(d);
      }
    }
    

    Our call to f passes a reference to the derived class to a function that takes a reference to the base class. This is perfectly fine.

    When people ask this question, they are typically wondering about passing a reference to the base class by reference. There is a double indirection here. You are passing a reference to a variable, and the variable is a reference to the base class. And it is this double reference that causes the problem.

    class Base { }
    class Derived : Base { }
    
    class Program {
      static void f(ref Base b) { }
    
      public static void Main()
      {
          Derived d = new Derived();
          f(ref d); // error
      }
    }
    

    Adding the ref keyword to the parameter results in a compiler error:

    error CS1503: Argument '1': cannot convert from 'ref Derived' to 'ref Base'
    

    The reason this is disallowed is that it would allow you to violate the type system. Consider:

      static void f(ref Base b) { b = new Base(); }
    

    Now things get interesting. Your call to f(ref d) passes a reference to a Derived by reference. When the f function modifies its formal parameter b, it's actually modifying your variable d. What's worse, it's putting a Base in it! When f returns, your variable d, which is declared as being a reference to a Derived is actually a reference to the base class Base.

    At this point everything falls apart. Your program calls some method like d.OnlyInDerived(), and the CLR ends up executing a method on an object that doesn't even support that method.

    You actually knew this; you just didn't know it. Let's start from the easier cases and work up. First, passing a reference into a function:

    void f(SomeClass s);
    
    ...
       T t = new T();
       f(t);
    

    The function f expects to receive a reference to a SomeClass, but you're passing a reference to a T. When is this legal?

    "Duh. T must be SomeClass or a class derived from SomeClass."

    What's good for the goose is good for the gander. When you pass a parameter as ref, it not only goes into the method, but it also comes out. (Not strictly true but close enough.) You can think of it as a bidirectional parameter to the function call. Therefore, the rule "If a function expects a reference to a class, you must provide a reference to that class or a derived class" applies in both directions. When the parameter goes in, you must provide a reference to that class or a derived class. And when the parameter comes out, it also must be a reference to that class or a derived class (because the function is "passing the parameter" back to you, the caller).

    But the only time that S can be T or a subclass, while simultaneously having T be S or a subclass is when S and T are the same thing. This is just the law of antisymmetry for partially-ordered sets: "if a ≤ b and b ≤ a, then a = b."

  • The Old New Thing

    Actually, FlagsAttribute can't do more; that's why it's an attribute

    • 27 Comments

    A few years ago, Abhinaba wondered why FlagsAttribute didn't also alter the way enumeration values are auto-assigned.

    Because attributes don't change the language. They are instructions to the runtime environment or (in rarer cases) to the compiler. An attribute can instruct the runtime environment to treat the function or class in a particular way. For example, you can use an attribute to tell the runtime environment that you want the program entry point to run in a single-threaded apartment, to tell the runtime environment how to look up your p/invoke function, or to tell the compiler to suppress a particular class of warnings.

    But changing how values for enumerations are assigned, well that actually changes the language. An attribute can't change the operator precedence tables. An attribute can't change the way overloaded functions are resolved. An attribute can't change the statement block tokens from curly braces to square braces. An attribute can't change the IL that gets generated. The code still compiles to the same IL; the attribute just controls the execution environment, such as how the JIT compiler chooses to lay out a structure in memory.

    Attribute or not, enumerations follow the same rule for automatic assignment: An enumeration symbol receives the value one greater than the previous enumeration symbol.

  • The Old New Thing

    Common gotchas when writing your own p/invoke

    • 21 Comments

    If you're looking to get into some p/invoke action, you'd be well-served to check out the pinvoke wiki to see if somebody else has done it too. If what you need isn't there, you may end up forced to write your own, and here are some gotchas I've seen people run into:

    • C++ bool and Win32 BOOLEAN are not the same as C# bool (aka System.Boolean). In Win32, BOOL is a 4-byte type, and BOOLEAN is a 1-byte type. [See also MadQ's remarks about VARIANT_BOOL.] Meanwhile, C++ bool is not standardized by Win32, so the size will vary based on your compiler, but most compilers use a 1-byte value. And then C# is even weirder: The bool is a 1-byte type, but it marshals as a 4-byte type by default.
    • Win32 char is not the same as C# char (aka System.Char). In C#, char is a Unicode character (two bytes), whereas in C/C++ under Win32 it is an ANSI character (one byte).
    • Win32 long is not the same as C# long (aka System.Int64). In C#, long is 64-bit value, whereas in C/C++ under Win32 it is a 32-bit value.
    • If memory is allocated and freed across the interop boundary, make sure both sides are using the same allocator. It is my understanding that the CLR uses CoTaskMemAlloc/CoTaskMemFree by default. If your Win32 function doesn't use CoTaskMemAlloc, you'll have to teach the CLR which allocator you really want.
    • When laying out structures, you have to watch out for alignment.

    That last one is particularly gnarly on 64-bit systems, where alignment requirements are less forgiving than on x86. The structure declarations on pinvoke.net tend to ignore 64-bit issues. For example, the declaration of the INPUT structure (as of this writing—it's a wiki so it's probably changed by the time you read this) reads as follows:

    [StructLayout(LayoutKind.Explicit)]struct INPUT {
      [FieldOffset(0)] int type;
      [FieldOffset(4)] MOUSEINPUT mi;
      [FieldOffset(4)] KEYBDINPUT ki;
      [FieldOffset(4)] HARDWAREINPUT hi;
    }
    

    This structure layout is correct for 32-bit Windows, but it's incorrect for 64-bit Windows.

    Let's take a look at that MOUSEINPUT structure, for starters.

    typedef struct tagMOUSEINPUT {
        LONG    dx;
        LONG    dy;
        DWORD   mouseData;
        DWORD   dwFlags;
        DWORD   time;
        ULONG_PTR dwExtraInfo;
    } MOUSEINPUT, *PMOUSEINPUT, FAR* LPMOUSEINPUT;
    

    In 64-bit Windows, the LONG and DWORD members are four bytes, but the dwExtraInfo is a ULONG_PTR, which is eight bytes on a 64-bit machine. Since Windows assumes /Zp8 packing, the dwExtraInfo must be aligned on an 8-byte boundary, which forces four bytes of padding to be inserted after the time to get the dwExtraInfo to align properly. And in order for all this to work, the MOUSEINPUT structure itself must be 8-byte aligned.

    Now let's look at that INPUT structure again. Since the MOUSEINPUT comes after the type, there also needs to be padding between the type and the MOUSEINPUT to get the MOUSEINPUT back to an 8-byte boundary. In other words, the offset of mi in the INPUT structure is 8 on 64-bit Windows, not 4.

    Here's how I would've written it:

    // This generates the anonymous union
    [StructLayout(LayoutKind.Explicit)] struct INPUT_UNION {
      [FieldOffset(0)] MOUSEINPUT mi;
      [FieldOffset(0)] KEYBDINPUT ki;
      [FieldOffset(0)] HARDWAREINPUT hi;
    };
    
    [StructLayout(LayoutKind.Sequential)] struct INPUT {
      int type;
      INPUT_UNION u;
    }
    

    I introduce a helper structure to represent the anonymous union that is the second half of the Win32 INPUT structure. By doing it this way, I let somebody else worry about the alignment, and it'll be correct for both 32-bit and 64-bit Windows.

    static public void Main()
    {
      Console.WriteLine(Marshal.OffsetOf(typeof(INPUT), "u"));
    }
    

    On a 32-bit system, this prints 4, and on a 64-bit system, it prints 8. The downside is that you have to type an extra u. when you access the mi, ki or hi members.

    input i;
    i.u.mi.dx = 0;
    

    (I haven't checked what the PInvoke Interop Assistant comes up with for the INPUT structure.)

  • The Old New Thing

    Why does Windows wait longer than my screen saver idle timeout before starting the screen saver?

    • 32 Comments

    You may find yourself in a situation where Windows takes longer than your specified screen saver idle timeout to start the screen saver.

    First of all, there are ways for programs to block the screen saver entirely. Calling SetThreadExecutionState(ES_DISPLAY_REQUIRED), is how a program says, "Even though there is no mouse or keyboard input, the screen is indeed in use, so don't blank it or start the screen saver." Media playback programs use this so the screen saver doesn't kick in while you're watching a movie on your DVD, and presentation programs use it so the screen saver doesn't start in the middle of your multi-million-dollar proposal.

    But even if no program has marked the screen as busy, Windows itself will delay activating the screen saver if it detects that you would prefer that it not run for a while.

    If you repeatedly dismiss the screen saver less than one minute after it starts on Windows Vista or later, the operating system says, "Oh, sorry. I thought there was nobody there, but obviously there is. You're probably reading an information-dense document or using your laptop as a flashlight or clock, and you want the screen to stay on even though you aren't generating any input. I'll hold off the screen saver for a little while for you."

    After the second time you do a "fast dismiss" of the screen saver, the screen saver idle time is temporarily incremented by its nominal value. For example, if you set your screen saver timeout to two minutes, then starting with the second fast dismiss, Windows will wait an additional two minutes before trying the screen saver again. Here's a timeline for people who like timelines:

    • At T = 0, you stop generating input.
    • At T = 2m, the screen saver starts. You dismiss it immediately.
    • At T = 4m, the screen saver starts. You dismiss it immediately. This is your second consecutive fast dismiss, so the screen saver timeout is temporarily increased to 4 minutes.
    • At T = 8m, the screen saver starts. You dismiss it immediately. This is your third consecutive fast dismiss, so the screen saver timeout is temporarily increased to 6 minutes.
    • At T = 14m, the screen saver starts. You dismiss it immediately. This is your fourth consecutive fast dismiss, so the screen saver timeout is temporarily increased to 8 minutes.

    As long as you keep fast-dismissing, the delay will increase. Of course, the delay won't grow indefinitely, because you'll eventually hit some other idle timeout, like the system sleep timeout, and then the system will sleep before it gets a chance to run the screen saver.

    If this feature offends you, then you can disable it by setting Adaptive Display to off in the Advanced Power Settings. There are also Group Policies for controlling this feature.

  • The Old New Thing

    Programming means that sometimes you have to snap two blocks together

    • 53 Comments

    Part of the challenge of programming (and for some people, the reason why programming is fun in the first place) is looking at the building blocks provided to you and deciding how to assemble them to build something new. After all, if everything you wanted a program to do already existed ready-made, it wouldn't be called programming any more. It would be called shopping.

    Is there an API or a quick way to find out which window the mouse is in?

    I replied, "The LEGO Group does not make a piece for every possible object. Sometimes you just have to take two LEGO blocks and click them together. Here are some interesting blocks: GetCursorPos, WindowFromPoint."

    Thanks for your reply. But WindowFromPoint gives me the window of the object at the location of the cursor. But I'm looking for the top level window containing the cursor.

    Fine, then use a different block.

    I wonder how it is these people manage to write programs at all. I get the impression they write code by asking a million tiny questions and cutting and pasting together all the replies.

    No wait, pasting together the replies counts as snapping blocks together. Maybe they just ask for completed programs.

  • The Old New Thing

    What is the maximum number of timers a program can create?

    • 12 Comments

    As MSDN correctly notes, "Timers are a limited global resource." How limited are they? (We're talking about timers created by SetTimer.)

    Not as limited as they once were.

    Originally, there were eight timers total for the entire system. When there are only eight timers, you have to be very careful not to waste them and to destroy them when they aren't needed.

    Windows 3.0 increased this to 32.

    Windows 95 increased this to around 2500, although the debug version of Windows got mad at you if a single program created more than 32.

    Windows NT 3.1 would create timers until you exhausted the desktop heap. The maximum number of timers depends on what other window manager objects had been created.

    Starting with Windows NT 4, the window manager treats timers as User Objects for bookkeeping purposes, so all your timers, windows, menus, and so on fight for space in the per-process 10,000 object limit and the 32,000-ish object limit for a desktop.

    As you can see, the precise limit changes over time, so you couldn't count on any particular limit being in effect. Who knows, maybe the limit will change again in a future version of Windows.

    These limits are pretty high, but that doesn't mean you can be lazy. Jessica pointed out that the WinForms Timer component is easily leaked "because most people don't know that they have to call Dispose on it." Another thing to check for when you find that you can't create any more timers.

  • The Old New Thing

    The way to stop people from copying files to a folder is to use NTFS security, not to block drag/drop

    • 19 Comments

    A customer wanted to prevent users from copying files to certain locations, and they did it by hooking functions like SHFileOperation and failing the operation if the parameters were not to its liking. The customer found that the hooks stopped working in Windows Vista because Explorer in Windows Vista uses the new IFileOperation COM interface instead of using the old SHFileOperation function. The customer wanted assistance in getting their hook working again so they could prevent users from copying files to directories they wanted to block.

    Well, first of all, arbitrary function hooking is not supported by any version of Windows, so the customer was already in unsupported territory right off the bat. (There are some components which have an infrastructure for hooks, such as file system filter drivers or Winsock Layered Service Providers.)

    Second, attempting to hook SHFileOperation to prevent the user from copying files into specific directories is looking at the problem at the wrong level, similar to the people who want to block drag/drop when what they really want to block is accidental drag/drop. If you block copying files via drag/drop in Explorer, that won't stop the user from copying files by other means, or by doing the "poor man's copy" by opening the document from the source location and doing a Save As to create a duplicate in the destination.

    If you want to prevent the user from copying files to a directory, use the NTFS security model. Withhold Create files permission in the folder, and users will be blocked from copying files into the directory in Explorer, Notepad, or any other program.

    Related: Shell policy is not the same as security.

  • The Old New Thing

    Restating the obvious about the WM_NOTIFY message

    • 5 Comments

    It appears that people seemed to appreciate restating the obvious about the WM_COMMAND message, so I'll try it again with the WM_NOTIFY message.

    The WM_NOTIFY message is typically used by a control to communicate with its parent, either to provide information, request it, or both. Although that is the typical use, there are exceptions. For example, property sheets send the WM_NOTIFY to their children. Property sheets are this sort-of backwards model, where the common controls provide the parent window (the property sheet) and applications provide the child windows (the property sheet pages). The window manager doesn't care who sends the message to whom, as long as the sender and recipient are in the same process.

    The message cannot cross a process boundary because WM_NOTIFY is basically a sender-defined version of WM_USER. Anybody can define a new notification code and associate it with any structure they want (as long as the structure begins with a NMHDR). The window manager can't marshal the structure between processes because the structure is defined by the control, not the window manager.

    A little elaboration of that "sender-defined version of WM_USER": As we saw, the meaning of WM_USER messages is determined by the implementor of the window class. In other words, the code receiving the message decides what WM_USER means. That works great if you're some external code that wants to send a message to a known window class. But what if you're external code that wants to send a message to an unknown window class? For example, you're a list view control and you want to tell your parent about some event. You want to send a message to the parent window, but which message? You can't send anything in the WM_USER range because each parent window defines independently what those messages mean, and it's highly unlikely that all the parent windows are going to agree that WM_USER+205 means the same thing. For similar reasons, the WM_APP range is no good. A registered message would work, but if you have hundreds of potential events, then a hundred registered messages is a bit heavy-handed. The old-school answer to this was the WM_COMMAND message, whose notification code is defined by the sending control. Unfortunately, the notification code is all you get; the other parameters are busy doing other things. Enter WM_NOTIFY, which is basically WM_COMMAND on steroids: The NMHDR structure contains everything that was in the WM_COMMAND message, and since it's a structure, you can embed the NMHDR inside a larger structure to provide (and possibly receive) more information.

    Okay, end of strange digression.

    The NMHDR structure itself is a convention, in the same way that the parameters to WM_COMMAND are a convention. The hwndFrom member is supposed to be the control that generated the notification, but there's no enforcement.

    First, there's no way to enforce it. A window doesn't send a message; code sends a message. You can check the thread that is executing the code that is sending a message, but you don't know which window that code is associated with.

    "Well, the window that is sending the message is the one that most recently received a message."

    That doesn't work because you can have code associated with one window call code associated with another window without actually sending a message. In fact, you probably do this all the time:

    class CFrame : public CWindow {
    ...
     LRESULT OnCommand(...);
    ...
     CGraphWindow *m_pwndGraph;
    };
    
    LRESULT CFrame::OnCommand(...)
    {
     switch (idFrom) {
     case IDC_CPU: // user clicked the "CPU" button
      m_pwndGraph->ChangeMode(CPU); // change to a CPU graph
      ...
    }
    

    Suppose that CGraphWindow::ChangeMode function calls SendMessage as part of its processing. Which window "sent" this message? Since you have the power to read code, the message was conceptually sent by CGraphWindow, but the most recently received message is a WM_COMMAND sent to the frame window.

    Your method call is just a transfer of control inside your program. The window manager doesn't know what's going on. All it knows is that it delivered a WM_COMMAND message to the frame window, and then some mystery code executed, and the next thing you know, somebody is sending a message. It doesn't have the source code to your program to know that "Oh, that's coming from CGraphWindow::ChangeMode, and to get the window handle for CGraphWindow, I should call CGraphWindow::operator HWND()." (And even if it did, imagine your surprise when your breakpoint on CGraphWindow::operator HWND() gets hit because SendMessage called it!)

    Second, even if there were some psychic way for the window manager to figure out which window is sending the message, you still wouldn't want that. It is common for WM_NOTIFY handlers of complex controls to forward the message to another window. For example, the list view control in report mode receives WM_NOTIFY messages from the header control and forwards them back out to its own parent, so that the list view parent can respond to header notifications. (The parent normally should just let the list view handle it, but the operation is performed in case you're one of those special cases that needs it.)

    Okay, back to what the fields of NMHDR mean. There are only three fixed fields to NMHDR and they pretty much match up with the parameters to WM_COMMAND:

    • hwndFrom is the handle to the window that is the logical source of the notification.
    • idFrom is the control ID corresponding to the window specified by hwndFrom. In other words, idFrom = GetDlgCtrlID(hwndFrom).
    • code is the notification code. The meaning of this notification code depends on the window class of hwndFrom.

    It is an unwritten convention that the notification codes for the common controls are all negative numbers. This leaves positive numbers for applications to use for their own purposes. Not that applications strictly speaking needed the help, because the meaning of the notification code depends on the window that generated the notification, so if you want a brand new 32-bit message number namespace, just register a new window class, and boom, a whole new range of codes becomes available just to you. (Even though the notification code values do not need to be unique across window classes, the common controls team tries to keep the system-defined notification codes from overlapping, just to make debugging easier.)

    The idFrom member is provided as a convenience to the window receiving the message so that it can use a simple switch statement to figure out who is sending the notification.

    Once you figure out which notification you're receiving, you can use the documentation for that notification to see which structure is associated with the notification. This answers Norman Diamond's complaint that he couldn't figure out what to cast it to. For example, if the notification is LVN_ITEMCHANGING, well, let's see, the documentation for LVN_ITEMCHANGING says,

    LVN_ITEMCHANGING
    pnmv = (LPNMLISTVIEW) lParam;

    pnmv: Pointer to an NMLISTVIEW structure that identifies the item and specifies which of its attributes are changing.

    In other words, your code goes something like this:

    case LVN_ITEMCHANGING:
     pnmv = (LPNMLISTVIEW) lParam;
     ... do stuff with pnmv ...
    

    I'm not sure how much more explicit the documentation could be made to be. All it was missing was the word case in front.

Page 1 of 4 (35 items) 1234