• The Old New Thing

    Everybody thinks about CLR objects the wrong way (well not everybody)

    • 34 Comments

    Many people responded to Everybody thinks about garbage collection the wrong way by proposing variations on auto-disposal based on scope:

    What these people fail to recognize is that they are dealing with object references, not objects. (I'm restricting the discussion to reference types, naturally.) In C++, you can put an object in a local variable. In the CLR, you can only put an object reference in a local variable.

    For those who think in terms of C++, imagine if it were impossible to declare instances of C++ classes as local variables on the stack. Instead, you had to declare a local variable that was a pointer to your C++ class, and put the object in the pointer.

    C#C++
    void Function(OtherClass o)
    {
     // No longer possible to declare objects
     // with automatic storage duration
     Color c(0,0,0);
     Brush b(c);
     o.SetBackground(b);
    }
    void Function(OtherClass o)
    {
     Color c = new Color(0,0,0);
     Brush b = new Brush(c);
     o.SetBackground(b);
    }
    void Function(OtherClass* o)
    {
     Color* c = new Color(0,0,0);
     Brush* b = new Brush(c);
     o->SetBackground(b);
    }

    This world where you can only use pointers to refer to objects is the world of the CLR.

    In the CLR, objects never go out of scope because objects don't have scope.¹ Object references have scope. Objects are alive from the point of construction to the point that the last reference goes out of scope or is otherwise destroyed.

    If objects were auto-disposed when references went out of scope, you'd have all sorts of problems. I will use C++ notation instead of CLR notation to emphasize that we are working with references, not objects. (I can't use actual C++ references since you cannot change the referent of a C++ reference, something that is permitted by the CLR.)

    C#C++
    void Function(OtherClass o)
    {
     Color c = new Color(0,0,0);
     Brush b = new Brush(c);
     Brush b2 = b;
     o.SetBackground(b2);
    
    
    
    
    
    }
    void Function(OtherClass* o)
    {
     Color* c = new Color(0,0,0);
     Brush* b = new Brush(c);
     Brush* b2 = b;
     o->SetBackground(b2);
     // automatic disposal when variables go out of scope
     dispose b2;
     dispose b;
     dispose c;
     dispose o;
    }

    Oops, we just double-disposed the Brush object and probably prematurely disposed the OtherClass object. Fortunately, disposal is idempotent, so the double-disposal is harmless (assuming you actually meant disposal and not destruction). The introduction of b2 was artificial in this example, but you can imagine b2 being, say, the leftover value in a variable at the end of a loop, in which case we just accidentally disposed the last object in an array.

    Let's say there's some attribute you can put on a local variable or parameter to say that you don't want it auto-disposed on scope exit.

    C#C++
    void Function([NoAutoDispose] OtherClass o)
    {
     Color c = new Color(0,0,0);
     Brush b = new Brush(c);
     [NoAutoDispose] Brush b2 = b;
     o.SetBackground(b2);
    
    
    }
    void Function([NoAutoDispose] OtherClass* o)
    {
     Color* c = new Color(0,0,0);
     Brush* b = new Brush(c);
     [NoAutoDispose] Brush* b2 = b;
     o->SetBackground(b2);
     // automatic disposal when variables go out of scope
     dispose b;
     dispose c;
    }

    Okay, that looks good. We disposed the Brush object exactly once and didn't prematurely dispose the OtherClass object that we received as a parameter. (Maybe we could make [NoAutoDispose] the default for parameters to save people a lot of typing.) We're good, right?

    Let's do some trivial code cleanup, like inlining the Color parameter.

    C#C++
    void Function([NoAutoDispose] OtherClass o)
    {
     Brush b = new Brush(new Color(0,0,0));
     [NoAutoDispose] Brush b2 = b;
     o.SetBackground(b2);
    
    
    }
    void Function([NoAutoDispose] OtherClass* o)
    {
     Brush* b = new Brush(new Color(0,0,0));
     [NoAutoDispose] Brush* b2 = b;
     o->SetBackground(b2);
     // automatic disposal when variables go out of scope
     dispose b;
    }

    Whoa, we just introduced a semantic change by what seemed like a harmless transformation: The Color object is no longer auto-disposed. This is even more insidious than the scope of a variable affecting its treatment by anonymous closures, for introduction of temporary variables to break up a complex expression (or removal of one-time temporary variables) are common transformations that people expect to be harmless, especially since many language transformations are expressed in terms of temporary variables. Now you have to remember to tag all of your temporary variables with [NoAutoDospose].

    Wait, we're not done yet. What does SetBackground do?

    C#C++
    void OtherClass.SetBackground([NoAutoDispose] Brush b)
    {
     this.background = b;
    }
    void OtherClass::SetBackground([NoAutoDispose] Brush* b)
    {
     this->background = b;
    }

    Oops, there is still a reference to that Brush in the o.background member. We disposed an object while there were still outstanding references to it. Now when the OtherClass object tries to use the reference, it will find itself operating on a disposed object.

    Working backward, this means that we should have put a [NoAutoDispose] attribute on the b variable. At this point, it's six of one, a half dozen of the other. Either you put using around all the things that you want auto-disposed or you put [NoAutoDispose] on all the things that you don't.²

    The C++ solution to this problem is to use something like shared_ptr and reference-counted objects, with the assistance of weak_ptr to avoid reference cycles, and being very selective about which objects are allocated with automatic storage duration. Sure, you could try to bring this model of programming to the CLR, but now you're just trying to pick all the cheese off your cheeseburger and intentionally going against the automatic memory management design principles of the CLR.

    I was sort of assuming that since you're here for CLR Week, you're one of those people who actively chose to use the CLR and want to use it in the manner in which it was intended, rather than somebody who wants it to work like C++. If you want C++, you know where to find it.

    Footnote

    ¹ Or at least don't have scope in the sense we're discussing here.

    ² As for an attribute for specific classes to have auto-dispose behavior, that works only if all references to auto-dispose objects are in the context of a create/dispose pattern. References to auto-dispose objects outside of the create/dispose pattern would need to be tagged with the [NoAutoDispose] attribute.

    [AutoDispose] class Stream { ... };
    
    Stream MyClass.GetSaveStream()
    {
     [NoAutoDispose] Stream stm;
     if (saveToFile) {
      stm = ...;
     } else {
      stm = ...;
     }
     return stm;
    }
    
    void MyClass Save()
    {
     // NB! do not combine into one line
     Stream stm = GetSaveStream();
     SaveToStream(stm);
    }
    
  • The Old New Thing

    VirtualLock only locks your memory into the working set

    • 60 Comments

    When you lock memory with VirtualLock it locks the memory into your process's working set. It doesn't mean that the memory will never be paged out. It just means that the memory won't be paged out as long as there is a thread executing in your process, because a process's working set need be present in memory only when the process is actually executing.

    (Earlier versions of the MSDN documentation used to say this more clearly. At some point, the text changed to say that it locked the memory physically rather than locking it into the working set. I don't know who changed it, but it was a step backwards.)

    The working set is the set of pages that the memory manager will keep in memory while your program is running, because it is the set of pages that the memory manager predicts your program accesses frequently, so keeping those pages in memory when your program is executing keeps your program running without taking a ridiculous number of page faults. (Of course, if the prediction is wrong, then you get a ridiculous number of page faults anyway.)

    Now look at the contrapositive: If all the threads in your process are blocked, then the working set rules do not apply since the working set is needed only when your process is executing, which it isn't. If all your threads are blocked, then the entire working set is eligible for being paged out.

    I've seen people use VirtualLock expecting that it prevents the memory from being written to the page file. But as you see from the discussion above, there is no such guarantee. (Besides, if the user hibernates the computer, all the pages that aren't in the page file are going to get written to the hibernation file, so they'll end up on disk one way or another.)

    Even if you've magically managed to prevent the data from being written to the page file, you're still vulnerable to another process calling ReadProcessMemory to suck the data out of your address space.

    If you really want to lock memory, you can grant your process the SeLockMemoryPrivilege privilege and use the AWE functions to allocate non-pageable memory. Mind you, this is generally considered to be an anti-social thing to do on a paging system. The AWE functions were designed for large database programs that want to manage their paging manually. And they still won't prevent somebody from using ReadProcessMemory to suck the data out of your address space.

    If you have relatively small chunks of sensitive data, the solution I've seen recommended is to use CryptProtectData and CryptUnprotectData. The encryption keys used by these functions are generated pseudo-randomly at boot time and are kept in kernel mode. (Therefore, nobody can ReadProcessMemory them, and they won't get captured by a user-mode crash dump.) Indeed, this is the mechanism that many components in Windows 2003 Server to reduce the exposure of sensitive information in the page file and hibernation file.

    Follow-up: I've been informed by the memory manager folks that the working set interpretation was overly conservative and that in practice, the memory that has been virtually locked won't be written to the pagefile. Of course, the other concerns still apply, so you still have to worry about the hibernation file and another process sucking the data out via ReadProcessMemory.

  • The Old New Thing

    Psychic debugging: The first step in diagnosing a deadlock is a simple matter of following the money

    • 26 Comments

    Somebody asked our team for help because they believed they hit a deadlock in their program's UI. (It's unclear why they asked our team, but I guess since our team uses the window manager, and their program uses the window manager, we're all in the same boat. You'd think they'd ask the window manager team for help.)

    But it turns out that solving the problem required no special expertise. In fact, you probably know enough to solve it, too.

    Here are the interesting threads:

      0  Id: 980.d30 Suspend: 1 Teb: 7ffdf000 Unfrozen
    ChildEBP RetAddr  
    0023dc90 7745dd8c ntdll!KiFastSystemCallRet 
    0023dc94 774619e0 ntdll!ZwWaitForSingleObject+0xc 
    0023dcf8 774618fb ntdll!RtlpWaitOnCriticalSection+0x154 
    0023dd20 00cd03f2 ntdll!RtlEnterCriticalSection+0x152 
    0023dd38 00cd0635 myapp!LogMsg+0x15 
    0023dd58 00cd0c6a myapp!LogRawIndirect+0x27 
    0023fcb8 00cb64a7 myapp!Log+0x62 
    0023fce8 00cd7598 myapp!SimpleClientConfiguration::Cleanup+0x17 
    0023fcf8 00cd8ffe myapp!MsgProc+0x1a9 
    0023fd10 00cda1a9 myapp!Close+0x43 
    0023fd24 761636d2 myapp!WndProc+0x62 
    0023fd50 7616330c USER32!InternalCallWinProc+0x23 
    0023fdc8 76164030 USER32!UserCallWinProcCheckWow+0x14b 
    0023fe2c 76164088 USER32!DispatchMessageWorker+0x322 
    0023fe3c 00cda3ba USER32!DispatchMessageW+0xf 
    0023fe9c 00cd0273 myapp!GuiMain+0xe8 
    0023feb4 00ccdeca myapp!wWinMain+0x87 
    0023ff48 7735c6fc myapp!__wmainCRTStartup+0x150 
    0023ff54 7742e33f kernel32!BaseThreadInitThunk+0xe 
    0023ff94 00000000 ntdll!_RtlUserThreadStart+0x23 
     
       1  Id: 980.ce8 Suspend: 1 Teb: 7ffdd000 Unfrozen
    ChildEBP RetAddr  
    00f8d550 76162f81 ntdll!KiFastSystemCallRet 
    00f8d554 76162fc4 USER32!NtUserSetWindowLong+0xc 
    00f8d578 76162fe5 USER32!_SetWindowLong+0x131 
    00f8d590 74aa5c2b USER32!SetWindowLongW+0x15 
    00f8d5a4 74aa5b65 comctl32_74a70000!ClearWindowStyle+0x23 
    00f8d5cc 74ca568f comctl32_74a70000!CCSetScrollInfo+0x103 
    00f8d618 76164ea2 uxtheme!ThemeSetScrollInfoProc+0x10e 
    00f8d660 00cdd913 USER32!SetScrollInfo+0x57 
    00f8d694 00cdf0a4 myapp!SetScrollRange+0x3b 
    00f8d6d4 00cdd777 myapp!TextOutputStringColor+0x134 
    00f8d93c 00cd04c4 myapp!TextLogMsgProc+0x3db 
    00f8d960 00cd0635 myapp!LogMsg+0xe7 
    00f8d980 00cd0c6a myapp!LogRawIndirect+0x27 
    00f8f8e0 00cd6367 myapp!Log+0x62 
    00f8faf0 7735c6fc myapp!remote_ext::ServerListenerThread+0x45c 
    00f8fafc 7742e33f kernel32!BaseThreadInitThunk+0xe 
    00f8fb3c 00000000 ntdll!_RtlUserThreadStart+0x23 
    

    The thing about debugging deadlocks is that you usually don't need to understand what's going on. The diagnosis is largely mechanical once you get your foot in the door. (Though sometimes it's hard to get your initial footing.)

    Let's look at thread 0. It is waiting for a critical section. The owner of that critical section is thread 1. How do I know that? Well, I could've debugged it, or I could've used my psychic powers to say, "Gosh, that function is called LogMsg, and look there's another thread that is inside the function LogMsg. I bet that function is using a critical section to ensure that only one thread uses it at a time."

    Okay, so thread 0 is waiting for thread 1. What is thread 1 doing? Well, it entered the critical section back in the LogMsg function, and then it did some text processing and, oh look, it's doing a SetScrollInfo. The SetScrollInfo went into comctl32 and ultimately resulted in a SetWindowLong. The window that the application passed to SetScrollInfo is owned by thread 0. How do I know that? Well, I could've debugged it, or I could've used my psychic powers to say, "Gosh, the change in the scroll info has led to a change in window styles, and the thread is trying to notify the window of the change in style. The window clearly belongs to another thread; otherwise we wouldn't be stuck in the first place, and given that we see only two threads, there isn't much choice as to what other thread it could be!"

    At this point, I think you see the deadlock. Thread 0 is waiting for thread 1 to exit the critical section, but thread 1 is waiting for thread 0 to process the style change message.

    What happened here is that the program sent a message while holding a critical section. Since message handling can trigger hooks and cross-thread activity, you cannot hold any resources when you send a message because the hook or the message recipient might want to acquire that resource that you own, resulting in a deadlock.

  • The Old New Thing

    What is the default version of a header file?

    • 34 Comments

    The general rule with Windows header files is that if you don't specify which version of the header file you want, you get the latest version. For example, if you have the Windows XP Platform SDK header files and you #include <windows.h>, you're going to get the Windows XP function prototypes, the Windows XP structures, the the Windows XP flags, all that stuff. And unless you're careful, the program you get as a result will most likely run only on Windows XP.

    If you call a function that is new for Windows XP, then your program won't run on earlier versions of Windows because the import can't be resolved.†

    If you use a structure that changed for Windows XP, then your program won't run on earlier versions of Windows because the structure size will be wrong.

    Even if the structure size didn't change, using a flag that was introduced in Windows XP will create difficulties for your program when run on earlier versions of Windows because those earlier versions don't support the flag you're passing. Depending on how the function in question was written, it may ignore the "flag from the future" or it may reject it as invalid.

    If you want your program to run on older versions of Windows, you have a few options. First, you can explicitly "downgrade" your header file by defining an appropriate symbol or symbols before including the windows.h header file.

    #define WINVER         0x0400
    #define _WIN32_WINNT   0x0400
    #define _WIN32_WINDOWS 0x0400
    #define _WIN32_IE      0x0400
    
    #include <windows.h>
    #include <commctrl.h>
    #include <shlobj.h>
    ...
    

    Oh yuck, now we have the messy world of "So what's the difference between _WIN32_WINNT, _WIN32_WINDOWS, _WIN32_IE, and WINVER?" We'll pick up this topic next time, but you're not going to like the answer.

    Nitpicker's corner

    †That statement is from the operating system's‡ point of view. You can of course use techniques like Visual Studio linker's delay-load feature to avoid creating an import dependency, but that's outside the operating system.‡

    ‡s/operating system/Windows operating system/

  • The Old New Thing

    How do I mark a shortcut file as requiring elevation?

    • 25 Comments

    Specifying whether elevation is required is typically something that is the responsibility of the program. This is done by adding a requestedExecutionLevel element to your manifest. (Bart De Smet shows you how. Calvin Hsia does the same for your Visual FoxPro programs.) But if the program you're running doesn't have such a manifest—maybe it's an old program that you don't have any control over—you can create a shortcut to the program and mark the shortcut as requiring elevation.

    To do this, you set the SLDF_RUNAS_USER flag in the shortcut attributes. Here's a skeleton program that sets the flag on the shortcut whose path is passed on the command line. For expository purposes, I've skimped on the error reporting, and just to shake things up, I've used ATL smart pointers.

    #include <windows.h>
    #include <shlobj.h>
    #include <atlbase.h>
    
    void MarkShortcutRunAs(LPCWSTR pszShortcut)
    {
     CComPtr<IPersistFile> sppf;
     if (FAILED(sppf.CoCreateInstance(CLSID_ShellLink))) return;
     if (FAILED(sppf->Load(pszShortcut, STGM_READWRITE))) return;
     CComQIPtr<IShellLinkDataList> spdl(sppf);
     if (!spdl) return;
     DWORD dwFlags;
     if (FAILED(spdl->GetFlags(&dwFlags))) return;
     dwFlags |= SLDF_RUNAS_USER;
     if (FAILED(spdl->SetFlags(dwFlags))) return;
     if (FAILED(sppf->Save(NULL, TRUE))) return;
     wprintf(L"Succeeded\n");
    }
    
    int __cdecl wmain(int argc, wchar_t *argv[])
    {
     if (argc == 2 && SUCCEEDED(CoInitialize(NULL))) {
      MarkShortcutRunAs(argv[1]);
      CoUninitialize();
     }
     return 0;
    }
    

    There's not really much to this program. It creates a shell link object (CLSID_ShellLink) and asks it to load from the file whose path is given on the command line. It then uses IShellLinkDataList::GetFlags and IShellLinkDataList::SetFlags to fetch the old flags and set new flags that include SLDF_RUNAS_USER. Once that's done, it saves the result back out.

    The hard part was knowing that the SLDF_RUNAS_USER flag existed in the first place.

    (I fear that most people will read this article and say, "Awesome! My program requires elevation, and this is how I can mark my Start menu shortcut to prompt for elevation. Thanks, Raymond!" These people will have completely ignored the opening paragraph, which explains that that is the wrong thing to do.)

  • The Old New Thing

    The history of the Windows XP common controls

    • 63 Comments

    In the beginning, there was one control library, namely USER, the window manager itself, which provided buttons, static controls, edit controls, scroll bars, list boxes, and combo boxes. These controls were under the purview of the window manager team.

    In Windows 3.1 a second control library known as the shell common controls was added, but the library really didn't come into it own until Windows 95, where it consisted of the list view, header, tree view, tooltip, toolbar, status bar, track bar, tab, updown, progress, hotkey, and animation controls. These controls were originally custom controls written by the shell team for use in Explorer, but since they seemed to be generally useful, time was budged to do the extra work to make the controls more suitable for general use, testing the controls in combinations and scenarios that Explorer itself didn't use, and putting together formal documentation.

    The shell common controls library underwent many changes over the years, whereas the core intrinsic controls in the window manager changed much more conservatively.

    With Windows XP, the visual design team wanted to give the look of Windows a new life. Changing the non-client area (such as the window frame) was comparatively straightforward, since programs didn't have much control over that part of the window anyway. As a result, they get the Windows XP look "for free".

    The client area is another story. Programs are in control of their client area, where they can place controls that come with Windows, their own custom controls, or controls obtained from a third party library. Making major changes to the core controls or the common controls would be a high-risk endeavor since there are thousands upon thousands of Windows program that not only use them, but use them in all sorts of crazy ways.

    The initial stab at resolving these two conflicting goals (making major changes to these controls to increase visual appeal and functionality while simultaneously not changing them to maintain compatibility) was to create a new DLL that would contain the "fancy new version" of the core controls as well as the shell common controls. The old DLLs USER32 and COMCTL32 stayed where they were, so that old programs continued to get the behavior they were expecting, and the new XP-style controls were placed in a DLL named UXCTRL.DLL. UX stands for user experience, which was the hot new buzzword at the time. To avoid name collision with the old style controls, the new controls got new names beginning with Ux. For example, the UXCTRL version of the button control was called UxButton.

    New features could be added to these new Ux controls with wild abandon without heed for backward compatibility since they were brand new controls. There was nothing they had to be compatible with. Explorer was changed to use these new controls instead of the old stodgy controls, and everything worked great.

    Or so it seemed.

    We thought we had cleverly sidestepped the backward compatibility problem by creating entirely new controls, but doing that created a whole new category of compatibility bugs. Even though it's completely undocumented and unsupported, programs like to grovel into the internal data structures of other programs or otherwise manipulate those programs' windows. In Explorer's case, it turns out that a lot of programs like to go spelunking around Explorer's window hierarchy and use functions like FindWindow and EnumChildWindows to find the object of their affections. For example, a program might use EnumChildWindows to enumerate all the child windows of an Explorer browser, and then use GetClassName and lstrcmpi(szClassName, TEXT("button")) to look for a specific control. In this example, the target was a button, but it could have been a list view or a tool bar. Since all the new XP-style controls were named things like UxButton and UxListView, these programs which looked for a button by comparing against the string "button" stopped working.

    Of course, there was no guarantee that Explorer would even use buttons at all; Explorer was completely within its rights to revamp its user interface. But that's not much consolation to the customers who paid good money for these programs, especially since magazine columnists are the types of people most likely to be running (or indeed even writing!) strange off-the-wall programs that pull these sorts of nasty stunts in the first place.

    Okay, so it is now a compatibility requirement that all the new window classes have the same names as their old counterparts. This created an impasse, since these controls needed to be created by dialog boxes, and therefore they had to be globally-registered window classes. But you can't have two global window classes with the same name, because that would create ambiguity over which one the caller was asking for.

    More brainstorming ensued, and a Plan C emerged. The common controls library would take advantage of side-by-side assemblies and use the application manifest to control which DLL a given window class name would resolve to. Thus was born a new DLL also called COMCTL32, but with a new version number—version 6. Old programs would get version 5.82 just like they did in Windows 2000. New programs would have to use a manifest to specify that they wanted version 6.

    Once again, the solution came with a new problem. Since the entire COMCTL32 library got split into two versions, this meant that there were two versions of the image list code. Whole new scenarios emerged, such as putting a version 5 image list in a version 6 tree view, or vice versa. (As the linked thread notes illustrates, not all of the problems with cross-version scenarios were caught in the initial release and had to wait for a service pack for the fix to become available.)

  • The Old New Thing

    Why do operating system files still adhere to the old 8.3 naming convention?

    • 44 Comments

    Commenter Brian Reiter asks a duplicate of a question that was already submitted to the Suggestion Box: Darren asks why operating system† files still (for the most part) adhere to the old 8.3 naming convention.

    There are a few reasons I can think of. I'm not saying that these are the reasons; I'm just brainstorming.

    First, of course, the name of a DLL cannot change once it has been chosen, because that would break programs which linked to that DLL by its old name. Windows 95 did not require the system volume and user profile volume to support long file names, although that was certainly the case by default. Companies which used roaming profiles or redirected folders may have had a heavy investment in servers which did not support long file names. Therefore, all system files on Windows 95 had to conform to the 8.3 naming convention.

    I believe that Windows NT permitted the system volume to be a short-file-names-only FAT partition as late as Windows 2000. Therefore, any DLL that existed in the Windows 2000 era had to conform to the 8.3 naming convention.

    Starting in Windows XP, long file names became mandatory, and a few system files such as shellstyle.dll waded tentatively into the long file name world. (The .NET Framework folks jumped in with both feet with their managed DLLs, but notice that their unmanaged DLLs like mscoree.dll still conform to 8.3.) But the waters in this world can be treacherous for operating system components.

    First of all, you have to worry about the automatically-generated short name. Suppose the operating system setup program is copying the shellstyle.dll file, but there is already a file called shellstuff.dll. The short name for shellstuff.dll will probably be SHELLS~1.DLL, and therefore the short name for shellstyle.dll will likely be SHELLS~2.DLL. Now, this may not be a big deal, except that some programs like to hard-code a file's short name. (There are a lot of programs that assume that the Program Files directory is C:\PROGRA~1, for example.)

    Furthermore, you can create confusion if the same DLL is loaded by both its short and long names, since the loader treats them as distinct:

    #include <stdio.h>
    #include <windows.h>
    
    int __cdecl main(int argc, char **argv)
    {
     printf("%p\n", LoadLibrary("SHELLS~1.DLL"));
     printf("%p\n", LoadLibrary("SHELLSTYLE.DLL"));
     return 0;
    }
    

    If you run this program, you will get something like this:

    6F2C0000
    00340000
    

    Even though the two paths refer to the same DLL, the loader treats them as different, and you end up with two copies of the same DLL loaded into memory. Now things get confusing, since you now have two sets of global variables, and if two components both use SHELLSTYLE.DLL but one used the short name and the other the long name, things get exciting when those two components try to talk about what they think is the same thing.

    It's like that time when I was a child and our family took a trip to Disneyland. Our parents put my brother and me on the gondola ride, and upon arrival at the other end, we were to go to the Autopia ride which was right next door. The plan was that our parents would meet us at the exit to Autopia. When my brother and I exited Autopia, we expected our parents to be waiting there for us, but they were nowhere to be seen. Sticking to the plan, we waited patiently for our parents to arrive. We sat there for what seemed like two hours (but which was probably much less), until eventually we decided that my brother would stay put and I would go looking around, at which point it didn't take long for me to find my father, who was walking around looking for us.

    What went wrong? Well, the problem was that the map of Disneyland showed Autopia, but what the map didn't say was that there were two Autopia rides (and therefore two Autopia exits) right next to each other. My brother and I were waiting by one exit, and our parents were waiting by the other. Each of us thought the other party was simply late.

    Similarly, if a DLL goes by multiple names, you can end up with two copies of it loaded into the process, with different components talking about different copies, unaware that they are talking about different things.

    And one final reason I can think of for sticking with 8.3 file names for operating system DLLs is simply, "Well, that's the way we've always done it. All the problems with 8.3 names are well-understood and under control. If we switched to long file names, we'd end up discovering a whole new set of problems. Why mess with something that works if it isn't broken?"

    Better the devil you know.

    Exercise: Why is it okay for the .NET Framework to use long file names for their managed DLLs?

    Nitpicker's Corner

    †s/operating system/Windows operating system/. Apparently nothing is obvious from context any more.

  • The Old New Thing

    Changes to power management in Windows Vista

    • 70 Comments

    As I noted some time ago, it's hard to get programmers excited about power management. What's worse, programs that do take the effort to care about power management often do it rudely, hijacking your computer and preventing you from going into standby or hibernation, instead preferring to drain your battery until the computer just dies from lack of power.

    The Power Management folks decided that they've had enough with these selfish programs and redesigned the way power management works. In addition to exposing finer control over various aspects of power management to the user, they also decided to "take the ball and go home" when it comes to programs rejecting suspend or hibernation requests. When the user asks for the computer to go into a low power state, programs will get notified of the state change but are not allowed to veto it. Programs get two seconds to clean up, and then that's it. The laptop is going into standby, ready or not.

    If you ordered a copy of the PDC DVDs, you can go back and watch session FUN319 to learn more. (Here are some follow-up remarks to that session.) I'm not the expert in this area; I'm just repeating what I've heard. If you have questions about the future of power management, you probably should ask the power management folks over on the Windows Mobile PC Team Blog.

  • The Old New Thing

    When does an object become available for garbage collection?

    • 13 Comments

    As we saw last time, garbage collection is a method for simulating an infinite amount of memory in a finite amount of memory. This simulation is performed by reclaiming memory once the environment can determine that the program wouldn't notice that the memory was reclaimed. There are a variety of mechanism for determining this. In a basic tracing collector, this determination is made by taking the objects which the program has definite references to, then tracing references from those objects, contining transitively until all accessible objects are found. But what looks like a definite reference in your code may not actually be a definite reference in the virtual machine: Just because a variable is in scope doesn't mean that it is live.

    class SomeClass {
     ...
     string SomeMethod(string s, bool reformulate)
     {
      OtherClass o = new OtherClass(s);
      string result = Frob(o);
      if (reformulate) Reformulate();
      return result;
     }
    }
    

    For the purpose of this discussion, assume that the Frob method does not retain a reference to the object o passed as a parameter. When does the OtherClass object o become eligible for collection? A naïve answer would be that it becomes eligible for collection at the closing-brace of the SomeMethod method, since that's when the last reference (in the variable o) goes out of scope.

    A less naïve answer would be that it become eligible for collection after the return value from Frob is stored to the local variable result, because that's the last line of code which uses the variable o.

    A closer study would show that it becomes eligible for collection even sooner: Once the call to Frob returns, the variable o is no longer accessed, so the object could be collected even before the result of the call to Frob is stored into the local variable result. Optimizing compilers have known this for quite some time, and there is a strong likelihood that the variables o and result will occupy the same memory since their lifetimes do not overlap. Under such conditions, the code generation for the statement could very well be something like this:

      mov ecx, esi        ; load "this" pointer into ecx register
      mov edx, [ebp-8]    ; load parameter ("o") into edx register
      call SomeClass.Frob ; call method
      mov [ebp-8], eax    ; re-use memory for "o" as "result"
    

    But this closer study wasn't close enough. The OtherClass object o becomes eligible for collection even before the call to Frob returns! It is certainly eligible for collection at the point of the ret instruction which ends the Frob function: At that point, the Frob has finished using the object and won't access it again. Although somewhat of a technicality, it does illustrate that

    An object in a block of code can become eligible for collection during execution of a function it called.

    But let's dig deeper. Suppose that Frob looked like this:

    string Frob(OtherClass o)
    {
     string result = FrobColor(o.GetEffectiveColor());
    }
    

    When does the OtherClass object become eligible for collection? We saw above that it is certainly eligible for collection as soon as FrobColor returns, because the Frob method doesn't use o any more after that point. But in fact it is eligible for collection when the call to GetEffectiveColor returns—even before the FrobColor method is called—because the Frob method doesn't use it once it gets the effective color. This illustrates that

    A parameter to a method can become eligible for collection while the method is still executing.

    But wait, is that the earliest the OtherClass object becomes eligible for collection? Suppose that the OtherClass.GetEffectiveColor method went like this:

    Color GetEffectiveColor()
    {
     Color color = this.Color;
     for (OtherClass o = this.Parent; o != null; o = o.Parent) {
      color = BlendColors(color, o.Color);
     }
     return color;
    }
    

    Notice that the method doesn't access any members from its this pointer after the assignment o = this.Parent. As soon as the method retrieves the object's parent, the object isn't used any more.

      push ebp                    ; establish stack frame
      mov ebp, esp
      push esi
      push edi
      mov esi, ecx                ; enregister "this"
      mov edi, [ecx].color        ; color = this.Color // inlined
      jmp looptest
    loop:
      mov ecx, edi                ; load first parameter ("color")
      mov edx, [esi].color        ; load second parameter ("o.Color")
      call OtherClass.BlendColors ; BlendColors(color, o.Color)
      mov edi, eax
    looptest:
      mov esi, [esi].parent       ; o = this.Parent (or o.Parent) // inlined
      // "this" is now eligible for collection
      test esi, esi               ; if o == null
      jnz loop                    ; then rsetart loop
      mov eax, edi                ; return value
      pop edi
      pop esi
      pop ebp
      ret
    

    The last thing we ever do with the Other­Class object (presented in the Get­Effective­Color function by the keyword this) is fetch its parent. As soon that's done (indicated at the point of the comment, when the line is reached for the first time), the object becomes eligible for collection. This illustrates the perhaps-surprising result that

    An object can become eligible for collection during execution of a method on that very object.

    In other words, it is possible for a method to have its this object collected out from under it!

    A crazy way of thinking of when an object becomes eligible for collection is that it happens once memory corruption in the object would have no effect on the program. (Or, if the object has a finalizer, that memory corruption would affect only the finalizer.) Because if memory corruption would have no effect, then that means you never use the values any more, which means that the memory may as well have been reclaimed out from under you for all you know.

    A weird real-world analogy: The garbage collector can collect your diving board as soon as the diver touches it for the last time—even if the diver is still in the air!

    A customer encountered the Call­GC­Keep­Alive­When­Using­Native­Resources FxCop rule and didn't understand how it was possible for the GC to collect an object while one of its methods was still running. "Isn't this part of the root set?"

    Asking whether any particular value is or is not part of the root set is confusing the definition of garbage collection with its implementation. "Don't think of GC as tracing roots. Think of GC as removing things you aren't using any more."

    The customer responded, "Yes, I understand conceptually that it becomes eligible for collection, but how does the garbage collector know that? How does it know that the this object is not used any more? Isn't that determined by tracing from the root set?"

    Remember, the GC is in cahoots with the JIT. The JIT might decide to "help out" the GC by reusing the stack slot which previously held an object reference, leaving no reference on the stack and therefore no reference in the root set. Even if the reference is left on the stack, the JIT can leave some metadata behind that tells the GC, "If you see the instruction pointer in this range, then ignore the reference in this slot since it's a dead variable," similar to how in unmanaged code on non-x86 platforms, metadata is used to guide structured exception handling. (And besides, the this parameter isn't even passed on the stack in the first place.)

    The customer replied, "Gotcha. Very cool."

    Another customer asked, "Is there a way to get a reference to the instance being called for each frame in the stack? (Static methods excepted, of course.)" A different customer asked roughly the same question, but in a different context: "I want my method to walk up the stack, and if its caller is OtherClass.Foo, I want to get the this object for OtherClass.Foo so I can query additional properties from it." You now know enough to answer these questions yourself.

    Bonus: A different customer asked, "The Stack­Frame object lets me get the method that is executing in the stack frame, but how do I get the parameters passed to that method?"

  • The Old New Thing

    User interface code + multi-threaded apartment = death

    • 17 Comments

    There are single-threaded apartments and multi-threaded apartments. Well, first there were only single-threaded apartments. No wait, let's try that again.

    First, applications had only one thread. Remember, 16-bit Windows didn't have threads. Each process had one of what we today call a thread, end of story. Compatibility with this ancient model still exists today, thanks to the dreaded "main" threading model. The less said about that threading model the better.

    OLE was developed back in the 16-bit days, so it used window messages to pass information between processes, there being no other inter-process communication mechanism available. When you initialized OLE, it created a secret OleMainThreadWnd window, and those secret windows were used to communicate between processes (and in Win32, threads). As we learned some time ago, window handles have thread affinity, which means that these communication windows have thread affinity, which means that OLE has thread affinity. When you made a call to an object that belonged to another apartment, OLE posted a message to the owner thread's secret OleMainThreadWnd window to tell it what needs to be done, and then it went into a private message loop waiting for the owner thread to do the work and post the results back.

    Meanwhile, the OLE team realized that there were really two parts to what they were doing. There was the low-level object and interface management stuff (IUnknown, CoMarshalInterThreadInterfaceInStream) and the high-level "object linking and embedding" stuff (IOleWindow, IOleDocument) that was the impetus for the OLE effort in the first place. The low-level stuff got broken out into a functional layer known as COM; the high-level stuff kept the name OLE.

    Breaking the low-level and high-level stuff apart allowed the low-level stuff to be used by non-GUI programs, which for quite some time were eyeing that object management functionality with some jealousy. As a result, COM grew two personalities, one focused on the GUI customers and another focused on the non-GUI customers. For the non-GUI customers, additional functionality such as multi-threaded apartments were added, and since the customers didn't do GUI stuff, multi-threaded apartments weren't burdened by the GUI rules. They didn't post messages to communicate with each other; they used kernel objects and WaitForSingleObject. Everybody wins, right?

    Well, yes, everybody wins, but you have to know what side your bread is buttered on. If you initialize a GUI thread as a multi-threaded apartment, you have violated the assumptions under which multi-threaded apartments were invented! Multi-threaded apartments assume that they are not running on GUI threads since they don't pump messages; they just use WaitForSingleObject. This not only clogs up broadcasts, but it can also deadlock your program. The thread that owns the object might try to send a message to your thread, but your thread can't receive the message since it isn't pumping messages.

    That's why COM objects involved with user interface programming nearly always require a single-threaded apartment and why OleInitialize initializes a single-threaded apartment. Because multi-threaded apartments were designed on the assumption that there was no user interface. Once you're doing user interface work, you have to use a single-threaded apartment.

Page 4 of 419 (4,184 items) «23456»