The High Altitude Developer

Beware of Auto-Reset Events (They Don't Behave the Way You Think They Do)

2008-08-19T02:53:00Z

So I just spent several days debugging an intermittent crash in some test code. Part of the reason it took so long was because whoever wrote this code introduced a dependency on a legacy test component that was apparently so old that sources and symbols had disappeared. So the first part of the diagnosis involved a bit of reverse engineering, which is always a bit time consuming. Another reason it took so long was because the crash couldn't reliably be reproduced in the debugger, so I had to start with post-mortem analysis. Fortunately, windbg is a great tool for doing that sort of thing.

So I reproduce the crash, attach windbg and get started (interestingly enough, Dr. Watson couldn't grab a crash dump...):

*** wait with pending attach
Symbol search path is: *******************
Executable search path is:
ModLoad: 09320000 0932b000   c:\dd\OfficeClient_2\binaries\x86chk\SuiteBin\sdmquery.dll
eax=00000000 ebx=00000001 ecx=00000000 edx=00000000 esi=00000000 edi=00000000
eip=77170bc5 esp=0046e770 ebp=0046e808 iopl=0         nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b             efl=00000206
ntdll!ZwWaitForMultipleObjects+0x15:
77170bc5 c21400          ret     14h

The first thing to do is dump the stacks of all threads so we can figure out which one crashed.
0:000> ~*k

. 0 Id: c80.ef0 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0046e76c 753ddcea ntdll!ZwWaitForMultipleObjects+0x15
0046e808 712b549e KERNEL32!WaitForMultipleObjectsEx+0x11d
0046e86c 712b5529 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x56
0046e88c 712b5645 mscorwks!Thread::DoAppropriateAptStateWait+0x4d
0046e918 712b51ed mscorwks!Thread::DoAppropriateWaitWorker+0x14d
0046e984 714337cd mscorwks!Thread::DoAppropriateWait+0x60
*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.10722_32\mscorlib\d3ebaf905742c816f27f122da299ef5f\mscorlib.ni.dll
0046ea80 10699cef mscorwks!WaitHandleNative::CorWaitOneNative+0x14f
0046ea98 10699bc2 mscorlib_ni!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)+0x1f
0046eab4 05c280c0 mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int32, Boolean)+0x2a
0046eb7c 05c27a40 Motif!MS.Internal.Motif.Runtime.TestElementWorkerThread.Join()+0x30
0046eb7c 026fdd9a Motif!MS.Internal.Motif.Runtime.WorkerThreadInvokeHandler.ElementInvokeHandler(System.Object, MS.Internal.Motif.Runtime.InvokedEventArgs)+0x1f8
0046eb7c 026fdb3e Motif!MS.Internal.Motif.Runtime.TestElement.OnInvoked(MS.Internal.Motif.Runtime.InvokedEventArgs)+0x1d2
0046ecc8 05c2c472 Motif!MS.Internal.Motif.Runtime.TestElement.Invoke()+0x17e
0046ecc8 002312d4 Motif!MS.Internal.Motif.Runtime.TestMethodAction.Execute()+0x212
*** ERROR: Module load completed but symbols could not be loaded for c:\dd\OfficeClient_2\binaries\x86chk\SuiteBin\sth_x86.exe
0046f064 713c2ee9 sth_x86!MS.Internal.Motif.TestHarness.StandaloneTestHarness.Main(System.String[])+0x10d4
0046f074 71256056 mscorwks!CallDescrWorker+0x33
0046f0f0 7125721e mscorwks!CallDescrWorkerWithHandler+0x8e
0046f22c 71257253 mscorwks!MethodDesc::CallDescr+0x19e
0046f248 712572ac mscorwks!MethodDesc::CallTargetWorker+0x21
0046f260 71305db5 mscorwks!MethodDescCallSite::Call_RetArgSlot+0x1c

...

8 Id: c80.98c Suspend: 1 Teb: 7ef40000 Unfrozen
ChildEBP RetAddr
06b3bc34 753d1270 ntdll!NtWaitForSingleObject+0x15
06b3bca4 753d11d8 KERNEL32!WaitForSingleObjectEx+0xbe
06b3bcb8 713acb41 KERNEL32!WaitForSingleObject+0x12
06b3bce8 713ad1ed mscorwks!ClrWaitForSingleObject+0x24
06b3c1a4 713afbcd mscorwks!RunWatson+0x1e3
06b3c8f4 713b01d1 mscorwks!DoFaultReportWorker+0xa46
06b3c934 714130a8 mscorwks!DoFaultReport+0x40
06b3c958 71411f04 mscorwks!WatsonLastChance+0x42
06b3d5f4 71411f7c mscorwks!EEPolicy::LogFatalError+0x420
06b3d60c 71413758 mscorwks!EEPolicy::HandleFatalError+0x38
06b3d630 71413c44 mscorwks!CLRVectoredExceptionHandlerPhase3+0xa6
06b3d664 714145a3 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
06b3d690 7141466c mscorwks!CLRVectoredExceptionHandler+0xad
06b3d6bc 771aa357 mscorwks!CLRVectoredExceptionHandlerShim+0x6d
06b3d6e8 7718c6c2 ntdll!RtlpCallVectoredHandlers+0x7a
06b3d6fc 7718c754 ntdll!RtlCallVectoredExceptionHandlers+0x12
06b3d774 77172eff ntdll!RtlDispatchException+0x19
06b3d774 712a0364 ntdll!KiUserExceptionDispatcher+0xf
06b3dad4 712a46e5 mscorwks!WKS::CFinalize::ScanForFinalization+0xc8
06b3db14 712a371f mscorwks!WKS::gc_heap::mark_phase+0x2c3

...

17 Id: c80.980 Suspend: 1 Teb: 7ef22000 Unfrozen
ChildEBP RetAddr
0e24fb2c 77217489 ntdll!ZwWaitForMultipleObjects+0x15
0e24fcc8 7544e3f3 ntdll!TppWaiterpThread+0x294
0e24fcd4 771ccfed KERNEL32!BaseThreadInitThunk+0xe
0e24fd14 771cd1ff ntdll!__RtlUserThreadStart+0x23
0e24fd2c 00000000 ntdll!_RtlUserThreadStart+0x1b

Thread #8 is clearly the problem, so we switch the debugger to that thread.

0:000> ~8s
eax=00000000 ebx=753d1661 ecx=0cd965ac edx=00400000 esi=06b3bc7c edi=00000000
eip=7717039d esp=06b3bc38 ebp=06b3bca4 iopl=0         nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b             efl=00010246
ntdll!NtWaitForSingleObject+0x15:
7717039d c20c00          ret     0Ch

Now we dump the full mixed stack so we can get the full picture of what is going on.

0:008> !dumpstack
OS Thread Id: 0x98c (8)
Current frame: ntdll!NtWaitForSingleObject+0x15
ChildEBP RetAddr Caller,Callee
06b3bc34 753d1270 KERNEL32!WaitForSingleObjectEx+0xbe
...
06b3df6c 0905e3fa (MethodDesc 0x127fe4 +0x52 MS.Internal.Mita.Logging.Log.Fail(System.Object, System.String, System.String))
06b3df94 0905e38d (MethodDesc 0x127fdc +0x2d MS.Internal.Mita.Logging.Log.Fail(System.Object, System.String)), calling (MethodDesc 0x127fe4 +0 MS.Internal.Mita.Logging.Log.Fail(System.Object, System.String, System.String))
06b3dfa4 0905e346 (MethodDesc 0x5cf2c78 +0x6 Microsoft.TrinityQA.Logging.MitaGlobalLog.Fail(System.String)), calling (MethodDesc 0x127fdc +0 MS.Internal.Mita.Logging.Log.Fail(System.Object, System.String))
06b3dfa8 0905e31b (MethodDesc 0x5cf46c0 +0x1b Microsoft.TrinityQA.Logging.SharedLog.Fail(System.String)), calling 09180582
06b3dfb8 0905e29e (MethodDesc 0x5cf30f0 +0x3e Microsoft.TrinityQA.Logging.TrinityLog.Fail(System.String)), calling 09180572
06b3dfec 0905e245 (MethodDesc 0x5cf30f8 +0x2d Microsoft.TrinityQA.Logging.TrinityLog.Fail(System.Exception)), calling (MethodDesc 0x5cf30f0 +0 Microsoft.TrinityQA.Logging.TrinityLog.Fail(System.String))

...
06b3e87c 77173099 ntdll!ExecuteHandler2+0x26
06b3e8a0 7717306b ntdll!ExecuteHandler+0x24 calling ntdll!ExecuteHandler2
06b3e8c4 7718c7ba ntdll!RtlDispatchException+0x124 , calling ntdll!RtlpExecuteHandlerForException
06b3e950 77172eff ntdll!KiUserExceptionDispatcher+0xf, calling ntdll!RtlDispatchException
06b3ec84 09321260 *** WARNING: Unable to verify checksum for c:\dd\OfficeClient_2\binaries\x86chk\SuiteBin\sdmquery.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for c:\dd\OfficeClient_2\binaries\x86chk\SuiteBin\sdmquery.dll -
sdmquery!SDMCALL+0x1c0 ====> Exception Code 255cc2d cxr@6b3e9b8 exr@b3e968
06b3ecb8 76cf7deb USER32!IsWindow+0x54 calling USER32!__SEH_epilog4
06b3ecbc 0011aa60 0011aa60, calling KERNEL32!GetLastError
06b3ecc8 0029874f 0029874f
06b3ecf0 0905e0ef (MethodDesc 0x6f3bf7c +0x1df Maui.Core.SdmWindow.GetDialogInfo()), calling 0034832c
06b3ed10 0905e0ef (MethodDesc 0x6f3bf7c +0x1df Maui.Core.SdmWindow.GetDialogInfo()), calling 0034832c
06b3ed80 0905dead (MethodDesc 0x6f3bf04 +0x35 Maui.Core.SdmWindow.Init()), calling 003482b8
06b3ed94 0905de5d (MethodDesc 0x6f3beec +0x25 Maui.Core.SdmWindow..ctor(System.String, Maui.Core.Utilities.StringMatchSyntax, System.String, Maui.Core.Utilities.StringMatchSyntax, Maui.Core.App, Int32)), calling 00348274
06b3eda0 0905de18 (MethodDesc 0x6f3c2d0 +0x30 Maui.VisualStudio.Current.Vsto.Office12.Dialogs.SdmAddNamedRangeDialog..ctor(Maui.Core.App)), calling 00348264
06b3edc4 0905ddc7 (MethodDesc 0x6f38bf0 +0x27 Maui.VisualStudio.Current.Vsto.Office12.DesignerControls.Excel.NamedRange.OnComponentInitialize()), calling 00348244
06b3edd4 09059799 (MethodDesc 0x92c4a60 +0x2a9 Maui.VisualStudio.Current.Vsto.Designers.VstoDesigner.AddToolBoxComponent(Maui.VisualStudio.Current.Ide.Designers.IToolBoxComponent, Maui.VisualStudio.Current.Ide.Designers.AddMethod, Int32, Int32, Int32, Int32)), calling 09180432
06b3ee08 0905945d (MethodDesc 0x92c4cbc +0x2d Maui.VisualStudio.Current.Vsto.Office12.Designers.WorksheetDesigner.AddToolBoxComponent(Maui.VisualStudio.Current.Ide.Designers.IToolBoxComponent, Maui.VisualStudio.Current.Ide.Designers.AddMethod, Int32, Int32, Int32, Int32)), calling (MethodDesc 0x92c4a60 +0 Maui.VisualStudio.Current.Vsto.Designers.VstoDesigner.AddToolBoxComponent(Maui.VisualStudio.Current.Ide.Designers.IToolBoxComponent, Maui.VisualStudio.Current.Ide.Designers.AddMethod, Int32, Int32, Int32, Int32))
06b3ee2c 09059361 (MethodDesc 0x92c4a50 +0xa9 Maui.VisualStudio.Current.Vsto.Designers.VstoDesigner.AddToolBoxComponent(Maui.VisualStudio.Current.Ide.Designers.IToolBoxComponent, Maui.VisualStudio.Current.Ide.Designers.AddMethod))
06b3ee54 0905927b (MethodDesc 0x6f38ba8 +0x43 Maui.VisualStudio.Current.Vsto.Office12.DesignerControls.Excel.NamedRange..ctor(Maui.VisualStudio.Current.Vsto.Office12.Designers.WorksheetDesigner, Maui.VisualStudio.Current.Ide.Designers.AddMethod))
06b3ee68 09059206 (MethodDesc 0x92c4784 +0x1e Microsoft.TrinityQA.Vsto.Helpers.Project.Excel12DesignerHelper.AddNamedRange(Maui.VisualStudio.Current.Ide.Designers.AddMethod)), calling 0034767c
06b3ee7c 09053dbd (MethodDesc 0x5cfba34 +0x5d Microsoft.TrinityQA.FeatureTesters.Vsto.ExcelDocProjectTester.BF_Debug_Windows()), calling 09180182
06b3ee84 06807dc0 (MethodDesc 0x5cfaedc +0xb0 Microsoft.TrinityQA.FeatureTesters.Vsto.DocProjectTester.RunTest(Microsoft.TrinityQA.FeatureTesters.TestSession.TestCase))
06b3eea4 06807dc0 (MethodDesc 0x5cfaedc +0xb0 Microsoft.TrinityQA.FeatureTesters.Vsto.DocProjectTester.RunTest(Microsoft.TrinityQA.FeatureTesters.TestSession.TestCase))
06b3eedc 06807cf7 (MethodDesc 0x48ed118 +0x4f Microsoft.TrinityQA.FeatureTesters.Vsto.DocProjectEntryPoints.BF_Debug_Windows()), calling (MethodDesc 0x5cfaedc +0 Microsoft.TrinityQA.FeatureTesters.Vsto.DocProjectTester.RunTest(Microsoft.TrinityQA.FeatureTesters.TestSession.TestCase))
06b3eef4 713c2ee9 mscorwks!CallDescrWorker+0x33
06b3ef04 71256056 mscorwks!CallDescrWorkerWithHandler+0x8e calling mscorwks!CallDescrWorker
...

06b3ff40 771cd1ff ntdll!_RtlUserThreadStart+0x1b, calling ntdll!__RtlUserThreadStart

From the stack, we can see that sdmquery!SDMCALL is throwing and there is an exception record (exr). However, the exception code isn't something I recognize. Luckily, there is a managed exception on the stack, so if we dump that, we should get a string that explains the problem.

0:008> !dso
OS Thread Id: 0x98c (8)
ESP/REG Object   Name
06b3c890 028310b4 System.ExecutionEngineException
06b3c8dc 028310b4 System.ExecutionEngineException
06b3db90 0cda8628 System.String    cat="INFO/TEST/FAIL"
06b3dc24 0cf8467c System.String    2
06b3dc40 0cf83ba8 System.String    System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Maui.Core.NativeMethods.BldSdmCall(IntPtr HWnd, Int32 wMsg, Int32 wp, CTRLDEF& lp)

...

06b3f824 029042ac System.Threading.ThreadStart

Ok, now we can see we're looking at an access violation. WinDbg has a very useful dot command that allows us to switch the context to reflect system state at the time of the exception. So we'll use the .exr command to do that and see what our state was when we crashed.

0:008> .cxr 6b3e9b8
eax=09530000 ebx=000000a0 ecx=00000249 edx=00001900 esi=09531000 edi=0cd97460
eip=09321260 esp=06b3ec84 ebp=06b3ecc8 iopl=0         nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b             efl=00010202
sdmquery!SDMCALL+0x1c0:
09321260 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]

Now we're getting somewhere. The access violation occured during a memory copy of some sort. We can dissasamble from here to get the address of the next instruction. We'll use that to disassemble backwards so we can see what the code was doing prior to the memory copy.

0:008> u 09321260
sdmquery!SDMCALL+0x1c0:
09321260 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]
09321262 8bca            mov     ecx,edx
09321264 83e103          and     ecx,3
09321267 f3a4            rep movs byte ptr es:[edi],byte ptr [esi]
09321269 eb4d            jmp     sdmquery!SDMCALL+0x218 (093212b8)
0932126b 81f9fd7f0000    cmp     ecx,7FFDh
09321271 7408            je      sdmquery!SDMCALL+0x1db (0932127b)
09321273 81f9fc7f0000    cmp     ecx,7FFCh

Now that we have the next instruction, we can disassemble backwards to see what the function does.

0:008> ub 09321262 l 100
sdmquery+0xf78:
09320f78 0000            add     byte ptr [eax],al
...
0932123c a174873209      mov     eax,dword ptr [sdmquery!SDMCALL+0x76d4 (09328774)]
09321241 8b4d0c          mov     ecx,dword ptr [ebp+0Ch]
09321244 81f9f77f0000    cmp     ecx,7FF7h
0932124a 8b5804          mov     ebx,dword ptr [eax+4]
0932124d 751c            jne     sdmquery!SDMCALL+0x1cb (0932126b)
0932124f 8b7d14          mov     edi,dword ptr [ebp+14h]
09321252 8d0c9b          lea     ecx,[ebx+ebx*4]
09321255 c1e103          shl     ecx,3
09321258 8bd1            mov     edx,ecx
0932125a 8d7024          lea     esi,[eax+24h]
0932125d c1e902          shr     ecx,2
09321260 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]

Ok, we have an idea of what the function is doing, so the next step is to figure out what kind of an acess violation we're getting. We'll start be checking the address that edi points to. If we get garbage there, we're looking at a buffer overrun. However, it looks like edi is pointing to a valid address.

0:008> dd edi
0cd97460 0074006e 0032005f 0073005c 00630072
0cd97470 0073005c 00690075 00650074 00720073
0cd97480 005c0063 00720074 006e0069 00740069
0cd97490 005c0079 00750061 006f0074 0061006d
0cd974a0 00690074 006e006f 0043005c 006d006f
0cd974b0 006f006d 005c006e 0069004c 005c0062
0cd974c0 0061004d 00690075 0056005c 00730069
0cd974d0 00610075 0053006c 00750074 00690064

Next, we'll dump the source pointer. Clearly, this address is bad. So at a minimum, we've read off the end of our source buffer while doing the memory copy. The 64k question is why.

0:008> dd esi
09531000 ???????? ???????? ???????? ????????
09531010 ???????? ???????? ???????? ????????
09531020 ???????? ???????? ???????? ????????
09531030 ???????? ???????? ???????? ????????
09531040 ???????? ???????? ???????? ????????
09531050 ???????? ???????? ???????? ????????
09531060 ???????? ???????? ???????? ????????
09531070 ???????? ???????? ???????? ????????

Let's refer back to the exception context and look at some registers. ECX is definitely suspect. We're trying to copy at least 585 bytes (and probably more since we don't yet know what ECX started at). In any event, the overrun may be caused by a bug in the size calculation, so lets explore that.

eax=09530000 ebx=000000a0 ecx=00000249 edx=00001900 esi=09531000 edi=0cd97460
eip=09321260 esp=06b3ec84 ebp=06b3ecc8 iopl=0         nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b             efl=00010202
sdmquery!SDMCALL+0x1c0:
09321260 f3a5            rep movs dword ptr es:[edi],dword ptr [esi]

Looking at the disassembly, we see that ECX's value is determined by some EBX calculations. EBX contains an offset of an EAX address. Luckily we can see that none of these registers have been overwritten so we should be able to examine them. We'll start with eax+4--which should be the value that ebx got.

0:008> dd eax+4
09530004 00000004 00120ab2 00007ff7 00000002
09530014 0cd96484 76cf7deb 0011aa60 00000001
09530024 00010002 00060000 00000000 00000086
09530034 00000035 000000d2 0000004d 0000011a
09530044 0000ffff ffffffff 00020002 00070000
09530054 00000000 000000da 00000035 00000126
09530064 0000004d 0000011a 0000ffff ffffffff
09530074 00120001 00400000 00000000 00000006

Strange; eax+4 is 0x4, yet ebx is 0x0a. The only possible way that could happen is if eax+4 got overwritten with a new value *after* it got read into EBX. This code certainly isn't doing that. It is worth mentioning though that when this bug does not occur, *both* EAX+4 and EBX are set to 0x4.
0:008> ?ebx
Evaluate expression: 160 = 000000a0

We also see that edx holds the original value that ECX was set to. IOW it contains the original count. If we follow the assembly, we can see it jibes with an EBX value of 0xa0. IOW 0xa0 + (0xa0*4) = 0x320 0x320 * 8 (the shl instruction multiplies by 2^#bits shifted) = 0x1900 which is what we see in edx.

0:008> ?edx
Evaluate expression: 6400 = 00001900

And just for grins we can verify that the starting address for the memory copy is valid. ESI was originally set to eax+24, which is still valid since EAX hasn't been overwritten. EAX+24 looks like its valid.

Luckily, before I had to try to reverse engineer the entire function, a collegue came to the rescue with some old source files for the problem component. With that I was able to figure out the bug--but only after several days of painful debugging. The problem code seemed to make sense. It was a mechanism for sending dialog messages cross-process in a synchronous fashion. To do that, the code would wrap up the message in some shared memory (using a memory mapped file), install a GetMessage hook in the target process, post a custom message to the target process, and then wait for an event to be signalled. Once the event was signalled, the returned data would be read out of the shared memory buffer. On the target side, the hook proc would intercept the custom message, unwrap the real message from shared memory, send it to the dialog, write the return values to the shared memory and then signal the event.

Since I couldn't see the problem from just looking at the code, I decided to try to build the module from the sources I had and see if it would still reproduce the bug. Fortunately it did, so I started by adding a bunch of asserts and trace statements to try to gather more information. What I could see was that in some cases, the value of the shared memory was changing after the WaitForSingleObject call had returned. The other interesting thing is that 0x0a is a legitmate value returned by the first call into the offending code.

The first thing that occured to me was that perhaps I was seeing latency in the memory mapped file--i.e. perhaps we were missing a FlushViewOfFile call. However, closer examination of the code revealed that the memory mapped file was backed by the page file, so there could be no latency. My next thought was that perhaps there was some re-entrancy occuring, but eventually I was able to rule that out. Finally, I started looking at the WaitForSingleObject call. What had to be happening was that somehow the event was getting incorrectly signalled. Unfortunately, there isn't a way to determine the state of the event, but from looking at the code, it didn't seem possible for the state to ever be wrong. Finally, in desperation, I decided to change the Event to be a manual reset event. My hope was that by adding explicit state changes, I might be able to pin-point where the state was being incorrectly set. Lo and behold, moving to manual reset events fixed the bug.

Now I was really scratching my head, so I started a careful review of the SetEvent and ResetEvent documentation. And that was where I found the cause of the bug. According to msdn documentation (http://msdn.microsoft.com/en-us/library/ms686211.aspx), calling SetEvent has the following behavior:

"The state of an auto-reset event object remains signaled until a single waiting thread is released, at which time the system automatically sets the state to nonsignaled. If no threads are waiting, the event object's state remains signaled."

Whoa! My understanding of auto-reset events was that the reset occured when the wait was satisified. Clearly this is not the case!

Suddenly, the cause of the problem became crystal clear. What was happening when the bug occured was this: When the offending code called PostMessage, the hook proc would grab the message in the target process. In some cases, the timing would be such that that hook proc would finish its work and signal the event (via SetEvent) before the code in the calling process had reached the WaitForSingleObject call. In that case, since no thread was (yet) waiting on the event, the SetEvent call would *not* reset the event. This resulted in the event staying signalled which meant that the next call to WaitForSingleObject would immediately (and incorrectly) return. As a result, the value in the shared memory block would be read before the hook proc had actually performed the update. In the bug case, this value would be sufficiently wrong to produce the read access violation on the memory copy.

So the moral of the story is that auto-reset events can be very dangerous. I think many of us tend to use auto-reset events by default--with an incorrect understanding of how they actually behave. Because it is virtually impossible to guarantee that the thread to be synchronized is actually waiting when the event is signalled, auto-reset events aren't particularly useful as a synchronization mechanism. Manual-reset events are much safer and are a much better choice for all-around use. Really, now that I understand the behavior of auto-reset events, I can't think of a good reason to use them. What little overhead they save you in terms of code seems hardly worth the risk of having a broken synchronization model.

Why Stepping Through Code Trumps Unit Testing

2008-08-04T16:51:00Z

Over the past few years, there has been a big buzz about how writing unit tests (preferably via Test Driven Development) combined with code coverage is the silver bullet for keeping bugs out of the product. While the benefits of unit testing are undeniable, there are quite a few folks out there who advocate unit testing as a replacement for stepping through code. In some cases I have even heard stepping through code described as "old school". I've always been uncomfortable with that position and I finally encountered a bug that exposes it for the utter nonsense that it is. Consider the following code:

const string RegValue1 = "SomeValue";

const string RegValue2 = "SomeOtherValue";

int bufferSize = 255 * Marshal.SizeOf(char);

StringBuilder buffer = new StringBuilder(bufferSize);

result = NativeMethods.RegQueryValueEx(hkey, RegValue1, IntPtr.Zero, IntPtr.Zero, buffer, ref bufferSize);

string value1= buffer.ToString();

buffer = new StringBuilder(bufferSize);

result = NativeMethods.RegQueryValueEx(hkey, RegValue2, IntPtr.Zero, IntPtr.Zero, buffer, ref bufferSize);

string value2= buffer.ToString();

buffer = new StringBuilder(bufferSize);

Were you able to spot the bug? I'll give you a hint: it will only occur if the string stored in the second registry value is larger than the first. If you still don't see it, don't worry. This bug is extremely difficult to spot if you are just looking at code. The basic problem stems from the design of the RegQueryValueEx API. The last parameter (lpcbData) ist he problem; it requires you to pass the byte count of the buffer in, but it then overwrites that value with the number of bytes actually written on return. In the code above, the buffer size is reused in multiple calls. Although it starts out as 1020 bytes, the first call to RegQueryValueEx will result in the value of bufferSize being changed to the number of bytes written to the first buffer. So on the second call to RegQueryValueEx, the buffer size being passed in will likely end up being less than the intended size of the buffer. If it happens that the second registry value is larger than the first, then the call will fail with ERROR_MORE_DATA because the passed in buffer size will be too small.

As a side note, that is why desgning an API with in/out parameters for anything but buffers is generally a bad design. Buffers are about the only place where the developer mind-set expects the output will be different than the input. In just about any other case, such a design practically guarantees that the caller will introduce this type of bug in their code because they will forget that their value got overwritten.

Back to topic: the truly scary thing about this bug is that there is no reason to think that the relative sizes of the registry values would have an effect on how this code behaves. As a result, it is highly unlikely that anyone would ever think to test the behavior where the length of the second registry value was greater than the first. Moreover, even if the developer did decide to write that test, they would have to correctly mock the behavior in RegQueryValueEx that causes the problem. Given that the developer mindset was clearly that they didn't care about the bytes written, it is unlikely that they would include that behavior in the mock--which means that their test results would be invalid. Which is why mocking system APIs is generally not a particularly good use of time. It should also be pointed out that if tests were written to exercise this code, we would see 100 percent code coverage even with inputs that would not find this bug. Which is a great reminder as to why code coverage numbers aren't as meaningful as some would make them.

The bottom line is that this bug is unlikely to be discovered through intentional testing. The best case scenario would be to find this bug accidentally by simply being lucky in the test values chosen. While accidents can happen, the most likely scenario for a developer relying solely on testing (unit or otherwise) is that this bug ships to customers. In our case, that is exactly what happened. Fortunately, the bug involved in-house code so we were able to deploy the fix immediately. It should be noted that it took a year for this bug to surface, and when it did, it completely broke us. Which makes this type of bug the ultimate nightmare scenario.

So how do we prevent a bug like this from shipping? Stepping through code is the ultimate answer, but before we go there, let's talk about how the compiler is the first line of defense. In this case, bufferSize should have been declared as a constant because it is clearly not intended to be changed. If that had been done, the developer would have seen compiler errors because RegQueryValueEx doesn't accept a constant type for the by reference lpcbData parameter. Assuming the developer paid attention to the error (rather than just removing the const to get their code to compile), they would have quickly realized the problem and the bug would have never been coded in the first place.

But we're all human. Forgetting to declare buffSize as const would be a common enough mistake, as would removing it in response to the compiler error. The latter would result from the superficial analysis of "right, of course I can't pass a const by ref" while ignoring the implications of what by ref actually means. So that leaves stepping through code as the best line of defense against a bug like this. The problem with stepping through code is that developers either don't do it, or they don't understand how to do it correctly. Many developers do attempt to step through every line of code that they write, but they do so in a superficial way--whereby they only look at return values. With this bug, that doesn't cut it. So let's talk about how to correctly step through code.

The first issue with stepping through code is simply keeping track of where we have been so we can ensure we've stepped through every line. I've found that the best way to do this is to place break points on the first line of every code block. This includes not only functions, but every conditional or flow control statement. As we complete stepping through each block, we remove the breakpoint. When no breakpoints remain, we know we have stepped through each line of code. Unit tests can be helpful in this regard because they can enable us to enter code paths that we might not otherwise be able to enter normally. Alternatively, we can explicitly manipulate the instruction pointer and/or change values in the debugger to direct flow into a code path that we wish to step through.

While visiting every line of code is a good start, it is only half the battle. The other half is verifying that what each line actually does matches our original intent. Certainly, checking return values of functions is important, but checking inputs is equally important. If we only look at outputs, we are just guessing that the function behaved correctly. We are guessing because if we don't know what the inputs are, we have no basis for saying that the function did what we expected it to do. Checking inputs would have resulted in this bug being caught because we would have seen that bufferSize was not what we expected on the second call to RegQueryValueEx. The important thing to call out here is that proper stepping through code would have caught this bug *regardless* of whether the inputs were sufficient to actually cause the bug. This is the critical advantage with stepping through code--it allows us to find bugs that we didn't even know could exist (and therefore didn't think to test for).

As a side note, this is why embedding function calls as parameters makes life harder. When it comes time to step through code, the only way to verify the input is to evaluate the embedded function. This will usually require stepping in or (particularly if no sources are available for the embedded code) stepping through some disassembly. Storing the results of the embedded function in a variable and then passing that as a parameter is much easier to debug. In retail builds the compiler will optimize the generated assembly anyway, so we don't gain anything by attempting to optimize higher level code.

In addition to checking inputs it is also important to verify proper loop behavior. At a minimum, we should step through a few iterations of any loop to make sure our variables are behaving properly (such as loop counters). We should also step through code executing at the boundaries of our loop; i.e. step through execution where we are at the maximum possible iteration minus one, the maximum possible iteration, and then verify that the loop exits. Conditional break-points are very useful here because they allow us to do things like break on the 1000th iteration of a loop.

Conditionals are another place where we have to be extra diligent. It is very important to verify both sides of a conditional. This is something that is easy to miss when "if" conditions lack an "else". In that case, I've found it helpful to add a breakpoint to the line following the conditional block and remove it when stepping through the case where the conditional is not satisfied.

The bottom line with stepping through code is that the goal ist to prove that each line behaves as intended. There is more to it than what I have mentioned above, but hopefully this will help get folks in the right mind-set to actually step through code in a meaningful way. When done properly, stepping through code is the best defense against shipping bugs. It is the only way to get the complete picture of how a given block of code actually behaves. Stepping through code is not "old school" and it is not optional. Unit tests are great, but by themselves they are not sufficient to keep bugs at bay.

EDIT:

I got an e-mail from a customer suggesting that the "Why" in the title should be changed to "When". My decision to use "Why" was based on my point that having unit tests doesn't excuse us from stepping through code. However, I think the comment raises an important point. That is, why do we write unit tests? Those who claim that unit tests are a substitute for stepping through code are missing the boat on unit testing. Unit tests are not about finding bugs, they are about preventing them. A unit test is just an active form of an assertion. Assertions verify design assumptions in code and fire an alert when those design assumptions are violated. However, assertions are completely passive; they lie dormant until the code path in which they reside is actually executed. The benefit of assertions is that they are not tied to any specific set of actions, so they will verify design assumptions regardless of how the code path was entered. Assertions are powerful and should be used liberally. However, we can also benefit from a more deterministic approach.

Unit tests fufill the same function of validating design assumptions, but they do so in an active way. Unit tests actually force a code path to execute and verify the outcome. So by incorporating unit tests into our code we are able to define how the code should behave and prove definitively that it behaves that way. If changes are made that violate the design assumptions behind the code, a unit test will fail and alert us to the problem. Because unit tests are active, we can guarantee that our design assumptions are actually validated--rather than hoping we do the right things to exercise our assertions. So the primary reason for writing unit tests is to ensure that future changes to the code don't violate the original design assumptions and introduce new bugs.

One of the benefits of writing unit tests is that because we are thinking about testing, it forces us to consider scenarios that our code must handle that we might have ignored when we were writing it. So unit tests can help us find bugs in our existing code, but that is a secondary function. In fact, unit tests tend to give us a false sense of security--we think "hey, my code passes all of these tests, so it must be good." But what unit tests don't address are the unknowns--those tests that we didn't know we needed.

So I'm going to leave my title the way it is, but also acknowledge that in reality unit testing and stepping through code have fundamentally different (yet complementary) purposes. So in that sense, stepping through code only trumps unit testing when unit testing is being done for the wrong reasons.

Wither White-Box Testing?

2008-07-24T20:17:00Z

I was recently reading JW's blog post on Prevention vs. Cure: http://blogs.msdn.com/james_whittaker/archive/2008/07/24/prevention-v-cure-part-1.aspx and it set me off a bit. The blog post talks about "developer testing" which got me thinking about one of my biggest gripes: the "Software Design Engineer In Test" title. I hate this title because it implies that someone who does test development has a different skill-set then someone who does product development.

So what does this have to do with JW's blog? Well, JW draws a distinction between "developer testing" and "tester testing" where developer testing involves things like design reviews, code reviews, unit testing, and presumably stepping through code. In other words, its what we traditionally have referred to as "white box" or sometimes "glass box" testing. White-box testing is extremely valuable. Getting chest-deep in the code, crawling memory and operating systems primitives, using tools like perfmon and process explorer is a really good way to find nasty bugs. Sometimes its the only way. For example, consider threading bugs. To borrow from a well-known gun slogan, IMO "Testers don't find threading bugs, customers do." The variations in timing between thread execution are infinite and testing all cases is impossible. You can't guarantee you don't have deadlock bugs through black-box testing--you have to analyze the code itself.

So who is going to do this work? There is no question that the buck stops with the developer who wrote the code. Good developers have their designs and their code reviewed. They write unit tests and step through their code. In effect, they do white-box testing every day. However, there are two problems with this. The first is that developers are psychologically invested in their code. No developer wants to think they write buggy code; it would mean they are a bad developer. Therefore they have a vested interest in not finding bugs. Which isn't to suggest most developers just close their eyes and check in. Far from it--many developers are very dilligent and make every effort to test their code. Still the psychological disadvantage cannot be ignored; its a bit akin to letting a student grade their own papers. At the end of the day, developers have to believe their code is great and at some level, that prevents them from being able to take an unbiased view of their code.

The second problem is that the degree to which the most well-intentioned developer is able to to test their code tends to be dictated by the schedule. You can "Wide Band Delphi" all you want, but if reliable estimating were possible, my landscaping would have been done last week and the plumber would have been at my house on Tuesday instead of Friday. Good developers try to provide good estimates and leave time for testing, but the reality is that estimating is almost impossible to get right--unless it happens to be something that you have done recently and repetitively (which isn't typically the case in a software project). And even if you get your estimate right, other things interfere like customer issues and special projects which somehow never seem to affect ship dates. So at the end of the day, developers do the level of testing that they have time to do which is often severely curtailed by the schedule.

Schedule pressures come into play when it comes to spec, design, and code reviews. Everybody is under the gun to get their own stuff checked in, so who has time to review anybody else's work? Sure, many orgs require code reviews prior to checking in, but usually what that means is two developers sitting in front of a machine where the author rapidly scrolls through their changes while the review sits and listens to the author explain how great their code is. The whole thing is over in minutes. A good code review should take a significant percentage of the time that it took to write the code in the first place and should be done without the author present, but that never happens because nobody has the time.

Well, almost nobody. SDETs (who should theoretically have the same skills as developers, or else what is the 'D' doing in their title) are paid to test, so and they should be fully capable of white-box testing. By employing "test developers" in a dedicated capacity as white-box testers, the two core problems that limit the effectiveness of white-box testing are removed. First, the fact that they are dedicated ensures that whatever time is available in the schedule will be solely dedicated to white-box testing. Second, unleashing developers on someone else's code not only removes the psychological barriers to finding bugs, but it ties their ego to doing an extremely thorough review of the code. In addition, when experienced people do white-box testing, it becomes a learning experience for the less experienced people.

Which brings me back to why I hate the SDET title. Development is development, product or otherwise. Developers benefit from doing white-box testing as well as from having it done on their code. Skilled developers may not wish to be forced to choose between product development or development-level testing. Some developers should be forced to do some of the latter for their own personal growth. But as it stands, the SDET title gets in the way; if you want to be a dedicated tester, you must choose a different career path. IMO, we should just have an SDE title where it is well understood that part of being a developer means spending time being solely dedicated to white-box testing.

Finally, I should mention that none of this is meant to diminish the importance of black-box testing. Its just that most organizations get black box testing right because it is what we tend to associate with traditional testing.

The Three Laws of Software Development

2008-03-31T17:12:00Z

If you haven't read The Pragmatic Programmer, by Andrew Hunt and David Thomas, I'd highly recommend it. It puts a fresh coat of paint on many of the concepts discussed in Steve Maguire's classic Writing Solid Code and Steve McConnell's CodeComplete (both of which are must reads for any software developer). Anyway, one of the things I really like about Hunt and Thomas' book is that they distill software development into three fundamental laws. A recent code review got me thinking about them again, so I decided that this might be a good time to revisit them in my blog.

Law #1: Do Not Repeat Yourself (DRY)

This law is very simple to understand: do not duplicate code, ever. Duplicating code causes two problems. The first is that if there is a bug in the duplicated code, it must be fixed in every copy of the code--which is time consuming. The second problem is that in order to fix the bug in every copy of the code, the developer must be aware that the code has been duplicated in the first place. This latter issue is a fairly serious one. Although good developers will try to evaluate any bug in the context of the system and consider if there are other manifestations of the bug that could occur, there is no way to account for duplicate code. Typically, when a bug is found in a piece of code that has been duplicated, the developer will fix the bug where it was found, but the fix will not be made to the other areas of duplication. This is often the case even when the developer who caused the duplication in the first place is fixing the bug; he or she will often not remember that the duplication exists and needs to be addressed. The end result is that the same fundamental bug may turn up several times more in different contexts--in each case wasting time and effort to investigate the issue and redevelop the same fix.

The solution to this problem is to simply avoid it. However, in order to do that, we must recognize when we are duplicating in the first place. Some forms of duplication are easier to recognize than others. Hunt and Thomas mention several types of duplication:

Imposed duplication--where duplication occurs intentionally because developers don't think they have a choice.
Inadvertant duplication--where duplication happens without developers realizing it.
Impatient duplication--where developers duplicate code because it is the easiest thing to do.
Interdeveloper duplication--where duplication occurs because two different developers are writing the same code without being aware of what the other is doing.

The first three forms of duplication are what I'm interested in. The bottom line with all of them is to quit using copy and paste when you are writing code. Seriously. If you find yourself copying and pasting code, stop and ask yourself what you are doing and whether you are introducing duplication into your code. If it turns out the answer to the question is yes, rewrite your code to remove the duplication.

As it turns out, some duplication is easier to detect and remove than others. I was recently reviewing some code where the author was generating file names and directory names by appending a number to the base name. The logic was to simply create a name by appending a number and check if the name existed on disk. If it did, the number was incremented and the name tested again. The first available name was used. The problem that the developer faced was that they were dealing with both directory and file names. So they needed to use File.Exists in one case and Directory.Exists in the other. As a result, they created two different methods which differed only by the mechanism used to check the existence of a filename. Although the two methods were not strictly identical, the key problem is that the algorithms were duplicated. If a bug were to turn up in the algorithm, it would need to get fixed in multiple places.

In this case, the duplication was intentional--the developer was aware of it, but didn't see how to avoid it. As it turns out, the solution is fairly simple. The duplication could be avoided by simply passing the name checking function as a delegate to a single method that implemented the algorithm. If necessary, top level methods could be created to call the implementation method with the correct context. In any event, intentional duplication is almost never acceptable. If you find yourself in a position where you feel you cannot avoid duplication, get help. Keep pulling people in until a solution is reached that doesn't involve duplication.

The only places where duplication is (in my opinion) at least tolerable is in generated code and in test cases. In the former, duplication is acceptable because generated code is never modified directly. If there is a bug in generated code, it must be fixed in the generator itself. So as long as the generator itself contains no duplication, the bug can be fixed in one place and then the corrected code can be regenerated.

I believe duplication in test cases is acceptable for a few reasons. First, and foremost, attempting to eliminate duplication in test cases can make it difficult to understand what the test case does. To me, this somewhat trumps maintenance. More importantly, test cases generally are not (and should not be) heavy-weight. They generally do not contain much logic, so typically they will not contain bugs and will generally be easy to refactor. Even if readability weren't an issue, at the end of the day I'm still willing to risk some extra maintenance cost simply so I don't have to spend time trying to figure out how to avoid duplication. To me, rapid development of test cases also trumps maintenance. Again, though, test cases should be extremely light weight. I'm a firm believer in enforcing the law of DRY in the framework that the test cases are built on.

Law #2: Preserve Orthogonality

Two things are orthogonal if they are independent of one another. Essentially, the Hunt and Thomas' concept of orthogonality is just a new-school way of talking about the concepts of cohesion and to a lesser extent, coupling. The bottom line here is that we should always strive to limit methods, classes, and modules to a singular purpose as well as limit the dependencies between pieces. If you need more of a refresher on these topics, Wikipedia has a good overview on cohesion here: http://en.wikipedia.org/wiki/Cohesion_%28computer_science%29 and coupling here: http://en.wikipedia.org/wiki/Coupling_%28computer_science%29.

The idea behind preservation of orthogonality is to make maintenance simpler. As long as two methods or classes are orthogonal, there is no chance that a change in one could affect the other. In other words, it is less risky to make changes when orthogonality is preserved since unintended side-effects are less likely.

While Hunt and Thomas address many aspects of preservation of orthogonality, I want to touch on one aspect that they don't. Namely, that if statements are often an indication that orthogonality is breaking down. While there are certainly many valid uses for if statements, I personally tend to treat them with a great deal of skepticism because they generally have the effect of reducing cohesion. One warning sign that an if statement is not a good approach is when you see the same conditional spread throughout a class or method. For example, I was recently looking at a class that was responsible for automating the creation of project items in Visual Studio. One of the methods in the class consisted of a switch statement, which depending on whether the project item was created new or added from an existing object would do the right thing to create the project item. As written, the method would create the project items via automating the user interface. However, a change was being made to allow project items to be created via object model automation. The proposed fix was to simply add an if statement at every level of the switch statement that would use the object model if the caller required it, otherwise it would fall through to the original implementation.

There are a couple of problems with this approach. The first is just the duplication of the if statement. Remember DRY? If the conditional expression needed to change (or there was a bug in it), every instance of the expression would have to get fixed. This problem could be mitigated by abstracting the conditional into its own function; i.e. if (UseObjectModelAutomation).

The second problem is that code can be added both at the top and the bottom of the method outside of the conditional expression. This makes it easy to introduce bugs. If someone isn't aware (or thinking about) the fact that there are two code paths here, they could easily add code that either depends on one or the other, or interferes with one or the other. Its a common scenario; the developer gets a bug that says if I do A then B is broken. The developer fixes the bug (and tests the fix), but never realizes they have broken C. And their response will invariably be, "I didn't realize I had to test C".

There are some mitigations here. The first would be to ensure the conditional can't be missed. That means using else clauses and making sure all code within the method is within the if or the else. While doing this can't prevent the above scenario, it will make it a whole lot less likely. In this case though, the problem is the switch statement. The only way to have a clean if-else statement would be to duplicate the switch--which really isn't a good option (that DRY thing again).

As it turns out, there is a better option: use an object-oriented solution. The trick is to refactor the project item class. Start by extracting a class that contains all of the project item creation code from the switch statement (which may require doing an extract method first). This gives you a class that knows how to create project items using UI automation. Then create a second class that contains the implementation for creating project items via object model automation. Depending on what you are trying to accomplish, this class can either derive from the UI Automation project item creator class, or it can derive from a base class that is common to both classes. In any event, the original method implementation would simply spin up the correct project item creator as an instance of the base type. The switch statement would call the appropriate method via the interface defined by the common base class. In other words, the details of which automation mechanism will be used are no longer known to the switch statement.

The other thing I look for with if statements, is just the size of the enclosed block. Generally, if it is anything more than a function call, I'll look for an opportunity to do an extract method. This isn't really an orthogonality issue per se, but it helps keep things readable and helps cut down on duplication.

Law #3: All Decisions Must Be Reversible

Reversibility is simply the ability to easily undo any decision you make. What Hunt and Thomas are really talking about here are the larger decisions that comprise architecture. Software must be able to adapt to change. For example, the database system in use today, may be suplanted by a better faster version from a different vendor. Can your system easily handle the switch? If not, it will probably be discarded.

One of the best ways to ensure reversibillity is to build a good abstraction layer around all dependencies. Doing this ensures that not only could you change your dependencies at the drop of a hat, but it also makes it easier to handle the case when your dependencies break you. With an abstraction layer in place, all changes are local to the layer. There is no need to search your code base for all of the places where code uses the dependency. In addition, the layer defines your requirements. The layer's interface tells you exactly what you would need from any potential replacement technology in order to make a switch.

I was guilty of violating this rule recently. I was trying to overhaul our system of running tests which was extremely complicated and and involved several diferent files. I wasn't fond of the test harness we were using, but since I was told we were committed to it, I figured, "why not leverage it?" Since that test harness required a data file of its own, I decided that things could be greatly simplfiied if we just moved all of our test data into that file--which we had to use anyway. That approach certainly streamlined our system, but at the cost of reversibility. Recently, our testing requirements changed and it turned out we need to use a different harness. Unfortunately, my decision to move our test data into the original harness's file left us with no simple way to get our data into the new harness. The ironic thing about this is that I had taken great pains to ensure that our system would allow you to easily swap in whatever test harness you needed. I just got blinded by the other problem I was trying to solve and didn't think about how my solution would impact reversibility.

Well, there you have it--software development distilled into three simple laws. As complex as software development is, I'm a big believer in these laws. Think about them when you are writing and/or reviewing code and they will help make you a better developer.

Using a Custom Proxy For Interception

2007-11-08T01:21:00Z

I've been working on a system that required intercepting the method calls in an object model; basically I was trying to use Aspect Oriented Programming to weave in some logging and verification code. My first attempt was to simply add an object sink to a ContextBoundObject as discussed in the following msdn article: http://msdn.microsoft.com/msdnmag/issues/03/03/ContextsinNET/default.aspx.

While this was relatively simple to implement, I quickly ran into a problem: I needed to reference the context bound objects from a client in a different context. Unfortunately, the context bound objects contained private fields (which cannot be marshalled across context boundaries). As a result, I would get exceptions in the transparent proxy whenever I tried to reference these objects from a client in a different context.

The solution to this seemed to be clear. The only reason for proxying objects at all was for interception, so I could solve my problem by handing out the real objects (instead of a proxy) from the interceptor. However, as I started digging into this, it appeared that obtaining the real object might be a problem. While I could get the RealProxy, I couldn't figure out how to get at the underlying object. RealProxy.GetUnwrappedServer was clearly the method I needed to call, but it was protected. So it became clear that the solution would be to implement a custom proxy.

The problem with that is that there is a dearth of documentation on the subject. Aside from the following short article (http://msdn2.microsoft.com/en-us/library/scx1w94y(VS.71).aspx) and a host of samples that don't demonstrate how to actually proxy a call to the real object, MSDN had very little to say on the subject. A few folks have blogged about this but the few examples out there were all explicitly creating the proxy. I wanted my interception to be completely seamless such that object activation would return a proxy. While it was clear that ProxyAttribute would allow this, there were no examples of how use the ProxyAttribute to do anything more than intercept activation. So, in this blog post, I'm going to put the pieces together and demonstrate how to implement a completely transparent (no pun intended) interception system using a custom proxy.

The first step is to create the custom proxy itself. The custom proxy is simply a class that derives from RealProxy. The class should provide an implementation of the parameterized constructor which does nothing more than delegate to the base. The interesting code is in the Invoke method (which is the only required override). Here is an implementation:

public class InterceptorProxy : RealProxy

{

private MarshalByRefObject m_realObject;

public InterceptorProxy()

{

}

public InterceptorProxy(Type classToProxy) : base(classToProxy)

{

}

public override IMessage Invoke(IMessage msg)

{

IMessage returnMessage = null;

if (msg is IConstructionCallMessage)

{

IConstructionCallMessage ctorCallMessage = msg as IConstructionCallMessage;

returnMessage = InitializeServerObject(ctorCallMessage);

m_realObject = GetUnwrappedServer();

SetStubData(this, m_realObject);

}

else if (msg is IMethodCallMessage)

{

IMethodCallMessage methodCallMessage = msg as IMethodCallMessage;

Preprocess(methodCallMessage);

IMethodReturnMessage rawReturnMessage = RemotingServices.ExecuteMessage(m_realObject, methodCallMessage);

returnMessage = PostProcess(methodCallMessage, rawReturnMessage);

}

else

{

throw new NotSupportedException();

}

return returnMessage;

}

private void Preprocess(IMessage msg)

{

// Do interception work here.

}

private IMessage PostProcess(IMessage msg, IMessage msgReturn)

{

// Do interception work here.

}

There are two pieces to the Invoke implementation. The first is how the IConstructionCall message is handled--we'll go through each call as it happens. The first call is to RealProxy.InitializeServerObject. This call simply instantiates the real object that we are proxying. The next call is to RealProxy.GetUnwrappedServer. This call actually returns a reference to the object that we just created. We need this so we can marshal calls (among other things). Finally, we call RealProxy.SetStubData. This last call performs the magic that ensures that calls to the Transparent Proxy are routed to the RealProxy. If SetStubData is not called, no further messages will be routed to the RealProxy.

The second piece of the Invoke implementation deals with handling IMethodCallMessages. Assuming you hooked up your proxy correctly, you will get IMethodCallMessages whenever a method is invoked on your proxied object. This implementation provides hooks before and after the method invocation to allow whatever pre and postprocessing you might require. In my case I do some logging and verification actions (not shown). The interesting call is RemotingServices.ExecuteMessage. This call performs the execution of a message on the real object.

In this case, we simply delegate what we were passed in since we aren't interested in modifying the message. However, if we needed to, we could provide our own message in lieu of the one we received to do anything we liked. For example, we could create a custom proxy implementation that would provide a thunking layer to allow existing code to call what would otherwise be an incompatible API. Another thing to point out is that Invoke's return IMessage can also be modified. For example, the interceptor can eat exceptions by simply replacing the IMethodReturnMessage it receives from the method call with one that does not contain any exceptions.

Now that we've got our custom proxy created, we need to hook into activation. What we want to accomplish is to ensure that when we "new" up our object we always get a transparent proxy instead. In other words, we want to make sure that our interceptor is installed automatically and without any special code required.

To do this, we need to create a ProxyAttribute. It looks like this:

[AttributeUsage(AttributeTargets.Class)]

[SecurityPermissionAttribute(SecurityAction.Demand, Flags = SecurityPermissionFlag.Infrastructure)]

public class InterceptorProxyAttribute : ProxyAttribute

{

public InterceptorProxyAttribute()

{

}

public override MarshalByRefObject CreateInstance(Type serverType)

{

RealProxy proxy = new InterceptorProxy(serverType);

MarshalByRefObject transparentProxy = (MarshalByRefObject)proxy.GetTransparentProxy();

return transparentProxy;

}

The interesting method is the only override we need: ProxyAttribute.CreateInstance. CreateInstance gets called during non-remote activation. Normally, in that case you would get the real object back. However, we want to return a TransparentProxy that delegates to our custom interceptor proxy. So instead of activating our real object, we activate our InterceptorProxy. We then call RealProxy.GetTransparentProxy to provide a TransparentProxy that gets returned in lieu of the real object.

Since CreateInstance doesn't actually create the real object, you might wonder where that happens. If you recall, the actual object gets created when RealProxy.Invoke handles the IConstructionCallMessage. RealProxy.InitializeServerObject instantiates the real object that is being backed by the proxy.

That's it. The only remaining step is to apply the InterceptorProxyAttribute to any class which we want to set up interception on. Once this is done, activation of that class will result in our interceptor being automatically installed. Note that in order for this technique to work, the class must derive from ContextBoundObject--otherwise it cannot be proxied.

As a final note, the following blog entry from Chris Brumme provides some good background on proxies and contexts: http://blogs.msdn.com/cbrumme/archive/2003/07/14/51495.aspx

Why UI Automation Is Not All That And A Bag Of Chips

2007-09-22T02:00:00Z

I spent some time today looking at an issue on a Korean system where a test was searching for a specific item in a combo box. The test would iteratively select each item in the combo box, examine it and continue looping until it found the right entry. The only problem was, this particular test was failing. Oddly enough, when I watched the test run, I could see it selecting the right item in the combo box. Upon debugging, I found that even though the right item was selected, the call to get the selected item was returning the text of the previously selected item. So what was going on?

It turns out that the test was interacting with the combo box by sending key strokes. At first glance, this might seem like a good approach--after all that is what users are going to do, right? The problem is that there is a fundamental difference between users and test automation. Users can take advantage of visual feedback to know when the system has actually processed their input. Automation cannot do this.

The authors of the SendKeys code clearly realized this and they attempted to ascertain when the input had actually been processed. Their strategy was to make a SendMessageCallback call immediately after sending the keystrokes. In case you are not familiar with it, the behavior of SendMessageCallback is to call the WndProc of the specified window and then return immediately. After the WndProc handles the message, the specified callback function is called and the results are passed back. At first glance, this seems like a sound strategy. Since the call is cross-process (and therefore cross-thread), the end result of the SendMessageCallback call is that a message is placed in the thread's message queue after the keystroke messages. Since messages are processed serially, in order for the callback to be called, the keystroke processing must have completed, right? Wrong. The oversight here is re-entrancy. If the key stroke handling code itself pumps messages, then the callback message will be completely processed before the key stroke handling has completed. In other words, the callback message gets processed in the middle of the key stroke handling. This is what was happening with the bug I was investigating; the code to get the currently selected item in the combo box was getting called before the combo box had a chance to update the selection.

The other very serious problem with this mechanism is that the SendKeys implementation uses the Win32 SendInput function. All SendInput does is place input primitives on the system event queue for processing by the Raw Input Thread. Once it has done that, it immediately returns. On single-procecessor machines this will work since the RIT will be running with higher priority and context will switch. On multi-processor machines, however, there is the possibility that the scheduling will work out such that the RIT ends up running concurrently on another processor and no context switch away from the SendKeys thread occurs. In that case, it is possible for the SendMessageCallback to be executed prior to the RIT posting the keyboard message to the application thread's queue. In that case, the result will be the same as the re-entrancy scenario; SendKeys will return prior to the application's keystroke handler actually executing.

This very issue is the fundamental problem with all test code that attempts to automate user input using keyboard and mouse inputs. While it is often trivial to generate the inputs, it is very difficult to reliably determine when the input has actually been completely processed. It is even more difficult to do this in a generic fashion (such as what SendKeys was attempting). The end result is that ui automation tends to be flaky because it so timing dependent. Inevitably, the inherent timing problems lead to a proliferation of Sleep statements strewn through out test cases and the automation will gradually get slower and slower over time.

In addition to the timing problems, UI automation turns into a nightmare when localized builds of the product must be tested. The localized versions of all of those UI strings must somehow be located if you want to test a different language. Typically, this will involve grovelling the binaries you are testing for string resources and trying to map resource identifiers to strings. The problem gets more difficult if you have duplicate strings. For example, you might be several "OK" strings that differ only by context. Unfortunately, it turns out that these strings are only duplicated in English. In other languages, the context results in completely different strings. So if you are working with English, you cannot assume that the first OK you find is necessarily the right string.

So all of these issues beg the question: why are we trying to automate UI at all? The typical answer is "well we have to test what our users will actually do." However, I would argue that such a view is overly simplistic. For example, let's take the case of the combo box. The combo box is a self-contained Win32 control. User input has already been tested by the people that developed the control. The combo box is just a state machine. There is no question what will happen with respect to any of the inputs a user can throw at a combo box.

The part that has not been tested is how the application using the combo box will react to the state changes. In fact, the code we are responsible for testing is completely independent of the combo box UI. All we need to do to fully test it is put the combo box into a state and/or cause the combo box to generate the events that we might be listening to. We can do this by bypassing the user interface and coding directly against the combo box API. To put it another way, we can theoretically achieve 100 percent code coverage without ever automating the UI. Assuming that to be true, how could it be argued that we have not met our burden as testers?

Certainly, there are those who would cry "but what if the combo box is broken and clicks don't do what they are supposed to do?" While that may be true, the reality is that nobody relies solely on automation to ship a product. The rare bug that is actually truly a UI issue will get discovered through manual testing.

The other thing to consider is what we are trying to accomplish through automation. We automate tests because we expect to have to execute them many times before the product ships. Why is this? So we can ensure that changes to the product do not errode its quality--that is, we use automation to catch regressions. So in the context of UI, how often do we really expect regressions in UI behavior? The answer is, not often. Moreover, such bugs would generally be simple and low risk fixes; i.e. adding some event hookup code that was incorrectly removed. These are the kind of fixes that can be applied as a patch to a daily build if they do actually happen.

At the end of the day, testing is about trade-offs. This is particularly true with respect to automation. We simply cannot automate everything, so we must make smart choices about where choose to invest our resources. In my mind, the costs of UI automation are tremendous, yet the benefits are marginal when compared to approaches that enable testing immediately below the UI layer. The amount of time spent developing, maintaining, and most importantly, analyzing failed test cases is far greater than the time it would take just test the UI manually. In the face of such costs and difficulties, it would seem that the only time full UI automation may be appropriate when the team is directly responsible for testing UI primitives that they have developed. Even then, it may still be cheaper to manually test the keyboard and mouse interaction and restrict automation to more direct calls.

The Designer Process That Would Not Terminate (Part 2)

2007-09-07T17:43:00Z

In my previous post (http://blogs.msdn.com/geoffda/archive/2007/08/31/the-designer-process-that-would-not-terminate.aspx) I talked about an issue where RCWs (Runtime Callable Wrappers) awaiting garbage collection were holding references to a COM object and preventing it from being deterministically shut down. In this post, we'll continue the discussion--this time focusing on how to use the debugger to track down such a problem.

To facilitate this blog entry, I started to code up a simple example of the problem. What I discovered is that the insidious nature of RCW leaks can be reproduced in code that is even simpler than I had originally intended to write. Hopefully, this example will hammer home just how careful you have to be when using RCWs.

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Office.Interop.Excel;
using System.Runtime.InteropServices;

namespace RCWLeak
{
    class Program
    {
        static void Main(string[] args)
        {
            ApplicationClass excel = new ApplicationClass();
            Workbook workbook = excel.Workbooks.Add(Type.Missing);
            Marshal.FinalReleaseComObject(workbook);
            excel.Quit();
            Marshal.FinalReleaseComObject(excel);

            Console.ReadKey();
        }
    }
}

So here is a very simple routine that does nothing more than start Excel, add a workbook, and then shut down. It looks correct right? Unfortunately it is not. When we run this code, execution pauses at the end and waits for input. What we can see by viewing task manager (or tasklist) that Excel is still running despite the fact that we thought we had taken all of the necessary steps to ensure a clean shut down.

Clearly there is an RCW that is still holding a reference to Excel somewhere, but it is non-obvious from looking at the code. To get to the bottom of this we'll need to take a look at the managed heap. Fortunately, we can do this with the SOS ("Son Of Strike") debugging extension. If you have installed WinDBG, you already have SOS. If you haven't installed, WinDBG, you can get it free from Microsoft here: http://www.microsoft.com/whdc/devtools/debugging/default.mspx.

If you are familiar with WinDBG, you are probably aware that it has an open interface that allows for the development of custom, pluggable debugging tools to extend the base debugging functionality. As I mentioned, SOS is one such extension. To load SOS in WinDBG, you would type the following in the command window:

.loadby sos mscorwks

Once SOS is loaded, you can see what it does by typing "!help". To get help for individual commands, you can simply type "!help <command>".

What we want to do is figure out what is holding on to Excel. To start with, we can find out all of the objects on the managed heap by using the !dumpheap command.

0:005> !dumpheap -stat
total 6901 objects
Statistics:
      MT    Count    TotalSize Class Name
79132f9c        1           12 System.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib]]
79116b3c        1           12 System.Security.Permissions.ReflectionPermission
79116a1c        1           12 System.Security.Permissions.FileDialogPermission
79116998        1           12 System.Security.PolicyManager
791135c8        1           12 System.Resources.FastResourceComparer
7910a9e8        1           12 System.RuntimeTypeHandle
7910746c        1           12 System.DBNull
791073ac        1           12 System.Empty
7910556c        1           12 System.__Filters
7910551c        1           12 System.Reflection.Missing
791045e4        1           12 System.RuntimeType+TypeCacheQueue
7912ed84        1           16 System.SByte[]
7912d8dc        1           16 System.Collections.ObjectModel.ReadOnlyCollection`1[[System.Reflection.CustomAttributeData, mscorlib]]
7912d274        1           16 System.Collections.ObjectModel.ReadOnlyCollection`1[[System.Reflection.CustomAttributeTypedArgument, mscorlib]]
7911baf8        1           16 System.Enum+HashEntry
79113ea0        1           16 System.Security.Permissions.FileIOAccess
79113744        1           16 System.Resources.ResourceReader+TypeLimitingDeserializationBinder
79112510        1           16 System.Globalization.GlobalizationAssembly
791106e4        1           16 System.Security.Permissions.UIPermission
79108934        1           16 System.Reflection.Cache.InternalCache
790fa4f8        1           16 System.__ComObject
008cdbfc        1           16 Microsoft.Office.Interop.Excel.WorkbookClass
008c5e84        1           16 Microsoft.Office.Interop.Excel.ApplicationClass

...

790fcb30 2466 217980 System.String
Total 6901 objects

The "stat" flag limits the output to just the objects themselves--very handy. If we put our cursor on the first line of the output, we can use the find window to search for "Microsoft.Office.Interop". This shows us the typed RCWs that are on the heap. However, all we see are:

008cdbfc 1 16 Microsoft.Office.Interop.Excel.WorkbookClass
008c5e84 1 16 Microsoft.Office.Interop.Excel.ApplicationClass

We know from checking our code that we are calling Marshal.FinalReleaseComObject on these, so they can't be the problem. (As a side note, if we wanted to dump these objects, we could call !dumpheap -mt <method table> and it would give us the address that we could then use to call !dumpobj.)

What else could be going on? Well, we have accounted for all of the typed RCWs, but we haven't checked for System.__ComObject. To look for a specific type, we can make use of the !dumpheap type flag:

0:005> !dumpheap -type System.__ComObject
Address       MT     Size
02893294 790fa4f8       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
790fa4f8        1           16 System.__ComObject
Total 1 objects

Aha! This must be the RCW that is causing problems. Now we just need to figure out where it is coming from. Let's start by trying to figure out who owns it. We'll do that by looking for the root using the !gcroot command:

0:005> !gcroot 02893294
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread f48
Scan Thread 2 OSTHread c3c

Uh oh, the object is already eligible for collection. We know this because there are no roots shown. It looks like we need to start debugging earlier when this object is still alive. The problem is that managed source level debugging doesn't work very well in WinDBG at the moment.

What to do? Take advantage of one of the best kept secrets out there: the Visual Studio 2005 debugger can load debugger extensions. To load SOS in the Visual Studio Debugger, enable native debugging in your project, start debugging and then simply go to the immediate window and type ".load sos". Subsequent SOS commands can be entered in the immediate window exactly as they would be entered in the WinDBG command window.

So we'll open our project in Visual Studio, turn on native debugging, hit F10 (single step) and load SOS:

.load sos

extension C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll loaded

Now that we have SOS loaded, we'll try setting a breakpoint on the first Marshal.FinalReleaseComObject call. After running to that location, we'll take a look at the ComObject again to see if we've managed to catch it while it is still rooted.

!dumpheap -type System.__ComObject

Address MT Size

02a83294 790fa4f8 16

02a83494 790fa4f8 16

total 2 objects

Statistics:

MT Count TotalSize Class Name

790fa4f8 2 32 System.__ComObject

Total 2 objects

Well, we see two ComObjects now. Lets find out who they belong to.

!gcroot 02a83294

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Error during command: Warning. Extension is using a callback which Visual Studio does not implement.

Scan Thread 5988 OSTHread 1764

Scan Thread 3964 OSTHread f7c

!gcroot 02a83494

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Error during command: Warning. Extension is using a callback which Visual Studio does not implement.

Scan Thread 5988 OSTHread 1764

Scan Thread 3964 OSTHread f7c

DOMAIN(004514F0):HANDLE(Strong):18118c:Root:02a83494(System.__ComObject)

The second ComObject is rooted to a GCHandle, which means that it was probably created by some internal implementation and we can ignore it for now. The first ComObject is not rooted--which means that it is probably the one we are looking for, but we still don't know who is allocating it. To find out, we'll try stopping at each line of code and dumping ComObjects.

What we discover with this approach is that the ComObject does not appear until we step over the following line of code.

Workbook workbook = excel.Workbooks.Add(Type.Missing);

However, it is still unrooted. That can only mean one thing--this line of code is creating an RCW on the heap but not referencing it. Looking carefully at the code, we finally see what we have been missing. The Workbooks accessor (of the ApplicationClass) creates a Workbooks interface instance and returns it. Since we don't assign the return value to anything, there is nothing to reference it beyond the scope of the call. Presumably, the object backing the Workbooks interface does't implement IProvideClassInfo, so the RCW is created as a System.__ComObject.

We can actually confirm this with a little creative stepping in the dissembly window:

Workbook workbook = excel.Workbooks.Add(Type.Missing);

0000003e mov ecx,esi

00000040 mov eax,dword ptr [ecx]

00000042 call dword ptr [eax+000000F8h]

00000048 mov edi,eax

After stepping over the first call (which invokes the Workbooks accessor), we dump the ComObjects again:

!dumpheap -type System.__ComObject

Address MT Size

02763294 790fa4f8 16

027632a4 790fa4f8 16

027632b4 790fa4f8 16

total 3 objects

Statistics:

MT Count TotalSize Class Name

790fa4f8 3 48 System.__ComObject

Total 3 objects

The last one proves our hypothesis:

!gcroot 027632b4

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Error during command: Warning. Extension is using a callback which Visual Studio does not implement.

Scan Thread 3924 OSTHread f54

ESP:47ef64:Root:027632b4(System.__ComObject)

Scan Thread 4672 OSTHread 1240

What we see is that the ComObject is currently rooted to the stack. Now after we step over the remainder of the code for that line, we can dump the heap again and we will see that object has lost its root (which is what we expect).

!gcroot 027632b4

Note: Roots found on stacks may be false positives. Run "!help gcroot" for

more info.

Error during command: Warning. Extension is using a callback which Visual Studio does not implement.

Scan Thread 3924 OSTHread f54

Scan Thread 4672 OSTHread 1240

In order to effect a clean shutdown, it would seem that we need to refactor our code so that we can call Marshal.FinalReleaseComObject on the Workbooks object. We'll need to assign the return value of the Workbooks accessor to a variable. Here is the corrected code:

        static void Main(string[] args)
        {
            ApplicationClass excel = new ApplicationClass();
            Workbooks workbooks = excel.Workbooks;
            Workbook workbook = workbooks.Add(Type.Missing);
            Marshal.FinalReleaseComObject(workbook);
            Marshal.FinalReleaseComObject(workbooks);
            excel.Quit();
            Marshal.FinalReleaseComObject(excel);

            Console.ReadKey();
        }

Upon running it, we see that we have solved the problem and Excel now shuts down cleanly when we expect it to.

Hopefully, this demonstration has reinforced the perils of RCW usage. When writing interop code, it is important to avoid calls that tunnel into the object model because they will orphan RCWs on the heap that you will not be able to access in order to call Marshal.ReleaseComObject.

The Designer Process that Would Not Terminate

2007-09-01T00:00:00Z

I recently was asked to take a look at some VSTO test automation that wasn't behaving correctly on lab machines. The test was fairly simple; it created a VSTO Excel project, dirtied the document, closed the designer window without saving, and then reopened the document to verify that it was not dirty. However, it turned out that the Excel process was never terminating as expected after the designer window was closed--though Excel would terminate after the test case had completed.

My initial assumption was that there was an RCW (Runtime Callable Wrapper) somewhere that was holding a reference to Excel so I started digging into the code. However, it appeared that all test code holding onto RCWs had an IDisposable implementation that was calling Marshal.ReleaseComObject on said RCWs. I stepped through all of the Marshal.ReleaseComObject calls to check the return values by examining EAX. Sure enough in a few cases, Marshal.ReleaseComObject was returning 1. So, I reset EIP in the debugger to make additional ReleaseComObject calls to ensure the referenced IUnknown was fully released. However, that still did not solve the problem.

So what was going on? I finally started stepping deeper into various methods and realized that a somewhat lower layer of the test automation code was allocating RCW's on the heap and those RCWs were never explicitly released before they were discarded. As a result, the RCW's would hold a reference that would keep Excel open until they were collected--which wasn't happening prior to the designer being closed.

Since the methods in question weren't caching the RCW's that were being allocated on the heap, there was no possibility for implementing a Dispose/Marshal.ReleaseComObject pattern without doing a major rewrite of the code. Instead, I was able to solve the problem by calling GC.Collect to force a garbage collection to occur. The specific pattern to use (which is documented by Andrew Whitechapel here: http://msdn2.microsoft.com/en-us/library/aa679805(office.11).aspx) is as follows:

GC.Collect();

GC.WaitForPendingFinalizers();

// Only call if you care about reclaiming RCW memory right now.

GC.Collect();

This deserves some additional explanation. What we are really trying to accomplish here is to get the RCW's finalizer executed. The RCW's finalizer is where the actual IUnkown::Release call will occur--which is what we need to happen in order to destroy the underlying COM object. The actual reclamation of the RCW memory is secondary and probably not important). In any event, in the initial GC.Collect call, the garbage collector will see that there are pointers to the unrooted RCWs in the finalizer queue and will duly move them to the freachable queue for processing by the finalizer thread. The subsequent WaitForPending finalizers call blocks the calling thread until all finalizers in the freachable queue have been executed--which ensures that all RCW's will have released their wrapped COM objects. When the finalizer thread executes a finalizer on an object, it removes the pointer to that object from the finalizer queue. At that point the object will be eligible for collection, which means that the memory will not be recovered until the next time a garbage collection occurs. For this reason, if you actually care about recovering the RCW memory, you would need to call GC.Collect a second time. Andrew Whitechapel actually recommends a second call to GC.WaitForPendingFinalizers as well, but I can't see how this would be useful since any unrooted objects would have been finalized as a result of the first set of calls.

The real issue is that using RCW's is very tricky and if you aren't careful, you can end up putting yourself in a box that you can only escape with the scorched earth approach outlined above. Anyone using RCWs should read Chris Brumme's blog entry that explains how they work here: http://blogs.msdn.com/cbrumme/archive/2003/04/16/51355.aspx.

For certain, utilizing the GC.Collect pattern will guarantee proper shutdown of a COM server (assuming all RCWs are eligible for collection). The advantage of the pattern lies in its simplicity. By forcing a GC, it sidesteps all of the issues associated with prematurely calling Marshal.ReleaseComObject--since only those RCWs that are not rooted will release. The disadvantage to this pattern is that degrades performance. Scott Holden details these issues in his blog entry: http://blogs.msdn.com/scottholden/archive/2004/12/28/339733.aspx.

It should also be pointed out that when an AppDomain is torn down, all finalizable objects get finalized regardless of whether they are rooted. So (excepting the rare case where the finalizer thread gets blocked or the process gets summarily terminated) all RCWs will eventually release as part of a normal shutdown. Depending on your Com object usage, it might be reasonable to simply let this happen. The question to be answered will be whether an unacceptable level of memory pressure is created by waiting. The issue is that RCWs are small objects so a great many can be created before the generation 0 heap fills up and a collection is performed by the system. If the underlying Com objects are large, you may end up with a great deal of memory pressure from the native heap. Obviously, the situation to be avoided is where so much memory is used that hard page faults start to occur and system performance degrades substantially.

For anyone wanting a deeper understanding of garbage collection, I'd recommend the following KB article which provides a great reading list: http://support.microsoft.com/default.aspx/kb/317866/.

Once you've read up on garbage collection, try taking Tess Ferrandez's garbage collection quiz: http://blogs.msdn.com/tess/archive/2007/04/02/net-garbage-collection-popquiz.aspx

The answers can be found here: http://blogs.msdn.com/tess/archive/2007/04/10/net-garbage-collector-popquiz-followup.aspx

In my next blog entry I'll talk about how you can use the debugger to diagnose a problem like this.