• Ntdebugging Blog

    Next Week: Windows NT Debugging Blog Live Chat

    • 0 Comments

    Windows NT Debugging Blog Live Chat

    Microsoft Platform Global Escalation Services is hosting our second live group debug chat session for the debugging community on March 17, 2009 at 11 AM Pacific Time.  We will be focusing on debugging techniques and any questions you may have about anything we’ve previously blogged about. 

    Details about the “PGES-Windows NT Debugging Blog Live Chat” can be found here: http://www.microsoft.com/communities/chats/default.mspx

    Share this post :
  • Ntdebugging Blog

    It’s not my fault! – A case of remote code injection gone bad

    • 5 Comments

    Today we’ll examine a case where a crash is occurring in a Microsoft process, in core Windows code, but the culprit isn’t the crashing code.  In fact, the culprit isn’t even running in the process that crashed!  But before I get ahead of myself, let’s start by examining a crash dump that shows the problem...

     

    // The crashing process is the Windows Sidebar in Vista...

    0:000> |

    .  0    id: 1b3c        create  name: sidebar.exe

     

    // The exception record shows that we have hit an access violation in RtlInitUnicodeString while reading from 00080000

    0:001> .exr -1

    ExceptionAddress: 77837e8b (ntdll!RtlInitUnicodeString+0x0000001b)

       ExceptionCode: c0000005 (Access violation)

      ExceptionFlags: 00000000

    NumberParameters: 2

       Parameter[0]: 00000000

       Parameter[1]: 00080000

    Attempt to read from address 00080000

     

    // Here’s the register context at the time of failure.

    // Looks like we are trying to find the end of a unicode string

    0:001> r

    Last set context:

    eax=00000000 ebx=00000000 ecx=ffffffff edx=0088fa88 esi=00000000 edi=00080000

    eip=77837e8b esp=0088fa60 ebp=0088fabc iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

    ntdll!RtlInitUnicodeString+0x1b:

    77837e8b 66f2af          repne scas word ptr es:[edi]

     

    // Here’s the call stack that the debugger has tried to assemble for us...

    0:001> kb

      *** Stack trace for last set context - .thread/.cxr resets it

    ChildEBP RetAddr  Args to Child             

    0088fad0 77154911 00080000 0088fb1c 7782e4b6 ntdll!RtlInitUnicodeString+0x1b

    0088fadc 7782e4b6 00080000 77030f7b 00000000 kernel32!BaseThreadInitThunk+0xe

    0088fb1c 7782e489 7713361f 00080000 00000000 ntdll!__RtlUserThreadStart+0x23

    0088fb34 00000000 7713361f 00080000 00000000 ntdll!_RtlUserThreadStart+0x1b

     

    // The address we are trying to read is freed memory...

    0:003> du 00080000

    00080000 "????????????????????????????????"

    00080040 "????????????????????????????????"

    00080080 "????????????????????????????????"

    000800c0 "????????????????????????????????"

    00080100 "????????????????????????????????"

     

     

    So we crash because RtlInitUnicodeString attempted to deference an invalid pointer (address 00080000).  Where did RtlInitUnicodeString get this value?  Let’s unassemble the function and see...

     

    0:012> uf ntdll!RtlInitUnicodeString

    ntdll!RtlInitUnicodeString:

    77567e70 57              push    edi

    77567e71 8b7c240c        mov     edi,dword ptr [esp+0Ch]  // edi comes from here

    77567e75 8b542408        mov     edx,dword ptr [esp+8]

    77567e79 c70200000000    mov     dword ptr [edx],0

    77567e7f 897a04          mov     dword ptr [edx+4],edi

    77567e82 0bff            or      edi,edi

    77567e84 7422            je      ntdll!RtlInitUnicodeString+0x38 (77567ea8)

    77567e86 83c9ff          or      ecx,0FFFFFFFFh

    77567e89 33c0            xor     eax,eax

    77567e8b 66f2af          repne scas word ptr es:[edi]  // We crash here

    <snip>

     

     

    We can see from the above assembly that edi came from esp+c, which is the second parameter to this function (MSDN tells us that this is the SourceString parameter).  So naturally we’ll want to examine the caller of RtlInitUnicodeString to see if it is at fault.  If we are to believe the call stack that the debugger gave us, then caller of RtlInitUnicodeString is BaseThreadInitThunk...but that doesn’t make sense.  BaseThreadInitThunk normally is the function that calls the thread’s start function.  Why would anyone try to make RtlInitUnicodeString the start function for a thread?  Let’s assume that the debugger isn’t showing us the real call stack for a moment, and look at the raw stack values...

     

    0:001> dds esp

    0088fa60  77837e70 ntdll!RtlInitUnicodeString

    0088fa64  7713312c kernel32!LoadLibraryExW+0x6f

    0088fa68  0088fa88

    0088fa6c  00080000

    0088fa70  06255e62

    0088fa74  00000000

    0088fa78  00000000

    0088fa7c  00080000

    0088fa80  00000000

    0088fa84  00000000

    0088fa88  00000000

    0088fa8c  00080000

    0088fa90  00000000

    0088fa94  00000000

    0088fa98  00000000

    0088fa9c  00000000

    0088faa0  00000000

    0088faa4  0088fa70

    0088faa8  0088f678

    0088faac  0088fb0c

    0088fab0  7711e289 kernel32!_except_handler4

    0088fab4  71be953e

    0088fab8  fffffffe

    0088fabc  0088fad0

    0088fac0  77133630 kernel32!LoadLibraryW+0x11

    0088fac4  00080000

    0088fac8  00000000

    0088facc  00000000

    0088fad0  0088fadc

    0088fad4  77154911 kernel32!BaseThreadInitThunk+0xe

    0088fad8  00080000

    0088fadc  0088fb1c

    0088fae0  7782e4b6 ntdll!__RtlUserThreadStart+0x23

    0088fae4  00080000

    0088fae8  77030f7b urlmon!g_StaticLiteralTreeCode <PERF> (urlmon+0xe0f7b)

    0088faec  00000000

    0088faf0  00000000

    0088faf4  00080000

    0088faf8  c0000005

    0088fafc  771af389 kernel32!UnhandledExceptionFilter

    0088fb00  771af389 kernel32!UnhandledExceptionFilter

    0088fb04  0088fae8

    0088fb08  0088f678

    0088fb0c  ffffffff

    0088fb10  777f9834 ntdll!_except_handler4

    0088fb14  000ecb9f

    0088fb18  00000000

    0088fb1c  0088fb34

    0088fb20  7782e489 ntdll!_RtlUserThreadStart+0x1b

    0088fb24  7713361f kernel32!LoadLibraryW

    0088fb28  00080000

    0088fb2c  00000000

    0088fb30  00000000

    0088fb34  00000000

    0088fb38  00000000

    0088fb3c  7713361f kernel32!LoadLibraryW

    0088fb40  00080000

    0088fb44  00000000

    <snip - all zeros to base of stack at 00890000>

     

    We can see from the information above that there is more going on here than the “kb” output would lead us to believe.  Notice that the second non-zero value on the stack (working from the bottom up) is the address of the LoadLibraryW function.  This is not a return address, it is the start of the function.  This is a clue that the start address of this thread, specified by some other thread that called CreateThread, is actually LoadLibraryW.   This is a parameter to _RtlUserThreadStart, which is the bottommost function on this thread’s call stack.  So we can examine the raw data above, and reconstruct the call stack like so...

     

    ntdll!RtlInitUnicodeString

    kernel32!LoadLibraryExW

    kernel32!LoadLibraryW

    kernel32!BaseThreadInitThunk

    ntdll!__RtlUserThreadStart

    ntdll!_RtlUserThreadStart

     

    Does this make sense?  Let’s think it through.  In Vista, it is typical to see the bottom 3 functions at the start of a thread.  Following this is the start address of the thread, which in this case is LoadLibraryW.  It also makes sense that LoadLibraryW would call LoadLibraryExW.  That leaves us with the question of whether LoadLibraryW actually calls RtlInitUnicodeString.  Let’s find out...

     

    0:011> uf kernel32!LoadLibraryExW

    <snip>

    kernel32!LoadLibraryExW+0x60:

    76f7311d ff7508          push    dword ptr [ebp+8]

    76f73120 8d45cc          lea     eax,[ebp-34h]

    76f73123 50              push    eax

    76f73124 8b3d6c10f576    mov     edi,dword ptr [kernel32!_imp__RtlInitUnicodeString (76f5106c)]

    76f7312a ffd7            call    edi

    76f7312c a14cd50177      mov     eax,dword ptr [kernel32!BasepExeLdrEntry (7701d54c)]

    76f73131 3bc3            cmp     eax,ebx

    76f73133 0f8498dafeff    je      kernel32!LoadLibraryExW+0x78 (76f60bd1)

    <snip>

     

    We can see from the above assembly that LoadLibraryExW does indeed call RtlInitUnicodeString, so our re-assembled call stack makes sense. 

     

    Now we know how we got to RtlInitUnicodeString, but where did the bad pointer value come from?  Note that the bad value, 00080000, is actually the very first non-zero value on the stack, right before the address of LoadLibraryExW.  When a call is made to CreateThread, not only is the thread start address specified, but also an optional parameter to start function is specified.  That is what we are seeing here.  Both the thread start address and the parameter end up as parameters to _RtlUserThreadStart.

     

    Let’s sum up what we know so far.   Some other thread is calling CreateThread with LoadLibraryW as the start function address, and 00080000 as the parameter.  This leads LoadLibraryExW to call RtlInitUnicodeString with 00080000 as the SourceString parameter, and when RtlInitUnicodeString attempts to read from this address we crash because 00080000 is the address of freed memory.  Incidentally, what should 00080000 point to?  It is being used as the parameter to LoadLibrary, so it should be a pointer to a null terminated string that specifies the name of the library file to load.

     

    Why would one want to start a thread with LoadLibraryW as the start address?  The most common reason is code injection.  Typically a thread outside of the target process will use the CreateRemoteThread function to invoke LoadLibrary against a particular DLL, thus loading their code into the target process.  Assuming that this is what is going on, we’ll need to move beyond the crash dump and debug a live system that is having the problem.  Fortunately we noticed that the crashing systems we observed all had a particular third party application installed.   We set up a test system with the third party application, and we were able to sometimes reproduce the crash.  Looks like possibly a timing-related crash, since we didn’t have a 100% repro.

     

    So let’s move forward with the debug of the live system.  Note that the memory addresses in the live debug notes will differ from the above.  Also, addresses and module names that could identify the application vendor have been changed.  First things first: setting a breakpoint on ntdll!NtCreateThreadEx and running through the code showed that the LoadLibraryW thread was starting without any in-process thread starting the thread.  It also revealed that the LoadLibraryW thread started without crashing this time, and the library that it was loading was a DLL that belonged to the third party app in question....

     

    0:003> du 00b90000

    00b90000 "C:\Program Files\AppVendor\App N"

    00b90040 "ame\applib.dll"

     

    We now know that the thread that is creating the LoadLibraryW thread isn’t in the sidebar.exe process, and we know that the third party application is definitely using LoadLibraryW to remotely inject their code into sidebar.exe.  Sometimes this works, and sometimes it crashes because the DLL name buffer points to freed memory.  Next step, let’s debug the third party process to confirm our suspicions, and hopefully figure out why the memory is freed sometimes.  We’ll set breakpoints on VirtualAllocEx, WriteProcessMemory, CreateRemoteThread, and VirtualFreeEx in the third party process...

     

    //

    // Third party app calls calls VirtualAllocEx against the sidebar.exe process

    //

    0:003> kb

    ChildEBP RetAddr Args to Child

    0200f8c0 4c9864db 000002b0 00000000 00000090 kernel32!VirtualAllocEx

    WARNING: Stack unwind information not available. Following frames may be wrong.

    0200f914 4c986af9 000aada0 0200f938 000adc20 AppName!Ordinal1+0x411c0

    0200f96c 4c986d1f 00000000 0200f9a0 00000003 AppName!Ordinal1+0x417de

     

    0:003> kP

    ChildEBP RetAddr

    0200f8c0 6c9884db kernel32!VirtualAllocEx(

    void * hProcess = 0x000002b0,

    void * lpAddress = 0x00000000,

    unsigned long dwSize = 0x90,

    unsigned long flAllocationType = 0x1000,

    unsigned long flProtect = 4)

     

    // PID 2836 == sidebar.exe

    0:003> !handle 0x000002b0 f

    Handle 2b0

    Type     Process

    Attributes     0

    GrantedAccess    0x10143a:

    Synch

    CreateThread,VMOp,VMRead,VMWrite,QueryInfo

    HandleCount     8

    PointerCount     122

    Name     <none>

    Object Specific Information

    Process Id 2836

    Parent Process 2944

    Base Priority 8

     

    // 00b90000 is the base address of the allocation in this run

    0:003> gu

    eax=00b90000 ebx=0200f9ac ecx=0200f89c edx=77249a94 esi=0200f904 edi=00000090

    eip=4c9844db esp=0200f8dc ebp=0200f914 iopl=0 nv up ei pl zr na pe nc

    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246

     

    //

    // Now the app calls WriteProcessMemory writing the following string:

    // "C:\Program Files\AppVendor\App Name\applib.dll" to the new virtual alloc

    //

    0:003> g

    Breakpoint 5 hit

    eax=00000000 ebx=0200f9ac ecx=0200f904 edx=00b90000 esi=00000090 edi=00000090

    eip=77371cc6 esp=0200f8bc ebp=0200f8d8 iopl=0 nv up ei pl nz na pe nc

    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206

    kernel32!WriteProcessMemory:

    77371cc6 8bff mov edi,edi

     

    0:003> kP

    ChildEBP RetAddr

    0200f8b8 6c9885ba kernel32!WriteProcessMemory(

    void * hProcess = 0x000002b0,

    void * lpBaseAddress = 0x00b90000,

    void * lpBuffer = 0x000aada0,

    unsigned long nSize = 0x90,

    unsigned long * lpNumberOfBytesWritten = 0x0200f8e0)

    0200f8d8 4c986895 AppName!Ordinal1+0x4129f

     

     

    0:003> du 0x000aada0

    000aada0 "C:\Program Files\AppVendor\App Name\applib.dll"

     

     

    // Within the sidebar.exe process, we can see that after WriteProcessMemory in the app returns,

    // The address has the expected string….

    0:003> du 00b90000

    00b90000 "C:\Program Files\AppVendor\App N"

    00b90040 "ame\applib.dll"

     

    //

    // The app then calls CreateRemoteThread against the sidebar.exe process,

    // using LoadLibraryW as the start address, and the address of the virtual alloc as the parameter.

    //

     

    0:003> g

    Breakpoint 0 hit

    eax=000002b0 ebx=77370000 ecx=7739361f edx=77249a94 esi=0200f9ac edi=00000000

    eip=773b46ef esp=0200f8a0 ebp=0200f8d0 iopl=0 ov up ei pl nz na po nc

    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000a02

    kernel32!CreateRemoteThread:

    773b46ef 6858010000 push 158h

     

    0:003> kb

    ChildEBP RetAddr Args to Child

    0200f89c 6c988714 000002b0 00000000 00000000 kernel32!CreateRemoteThread

    0200f8d0 4c9688ea 0200f8f8 00000000 6cade78c AppName!Ordinal1+0x413f9

    0200f914 4c968af9 000aada0 0200f938 000adc20 AppName!Ordinal1+0x415cf

     

    0:003> kP L1

    ChildEBP RetAddr

    0200f89c 6c988714 kernel32!CreateRemoteThread(

    void * hProcess = 0x000002b0,

    struct _SECURITY_ATTRIBUTES * lpThreadAttributes = 0x00000000,

    unsigned long dwStackSize = 0,

    <function> * lpStartAddress = 0x7739361f,

    void * lpParameter = 0x00b90000,

    unsigned long dwCreationFlags = 0,

    unsigned long * lpThreadId = 0x00000000)

     

    0:003> ln 0x7739361f

    Exact matches:

    kernel32!LoadLibraryW (wchar_t *)

     

    //

    // The application then calls VirtualFreeEx on the virtual alloc.

    //

    0:003> kb

    ChildEBP RetAddr Args to Child

    0200f8cc 6c98851c 000002b0 00b90000 00000000 kernel32!VirtualFreeEx

    WARNING: Stack unwind information not available. Following frames may be wrong.

    0200f914 4c988af9 000aada0 0200f938 000adc20 AppName!Ordinal1+0x41201

    0200f96c 4c988d1f 00000000 0200f9a0 00000003 AppName!Ordinal1+0x417de

     

    // Within the sidebar.exe process, the thread for LoadLibrary starts...

    0:003> kb

    ChildEBP RetAddr Args to Child

    0412f814 7722e489 7739361f 00b90000 00000000 ntdll!__RtlUserThreadStart

    0412f82c 00000000 7739361f 00b90000 00000000 ntdll!_RtlUserThreadStart+0x1b

     

    // But the memory has already been freed by the third party application...

    0:003> du 00b90000

    00b90000 "????????????????????????????????"

    00b90040 "????????????????????????????????"

    00b90080 "????????????????????????????????"

    00b900c0 "????????????????????????????????"

    00b90100 "????????????????????????????????"

     

    //

    // ...and sidebar.exe crashes once the address is read

    //

    (b14.4d8): Access violation - code c0000005 (first chance)

    First chance exceptions are reported before any exception handling.

    This exception may be expected and handled.

     

    eax=00000000 ebx=00000000 ecx=ffffffff edx=0412f780 esi=00000000 edi=00b90000

    eip=77237e8b esp=0412f758 ebp=0412f7b4 iopl=0 nv up ei pl zr na pe nc

    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246

    ntdll!RtlInitUnicodeString+0x1b:

    77237e8b 66f2af repne scas word ptr es:[edi]

     


    We’ve now come full circle, back to the original crash, at the same instruction in RtlInitUnicodeString.  So now we know that the problem is that the third party vendor frees the memory that it allocated in sidebar.exe, before it is read.  Sometimes the call to VirtualFreeEx happens after the memory is read, and in that scenario the crash doesn’t occur.  But if VirtualFreeEx completes before RtlInitUnicodeString has a chance to read it, then sidebar.exe crashes. 

     

    This information was passed on to the third party vendor, and hopefully they’ll make a change to their code to avoid this problem.  There are multiple ways that this could be addressed, but one potential fix would be for the third party app to wait on the thread handle returned from CreateRemoteThread before freeing the memory.

     

    -          Matthew Justice

     

     

    Share this post :
  • Ntdebugging Blog

    Global Engineering Conference

    • 3 Comments

    The Global Escalation Services team at Microsoft hosted the 1st annual Global Engineering Conference on February 25th & 26th. This year’s theme of “Building the Engineering Community” was centered around our efforts to increase knowledge sharing, collaboration, and best practices throughout the engineering community.  We hope you enjoy these sessions and look forward to your attendance next year.

     

    The conference was held virtually via Live Meeting with Q&A sessions following every presentation.

    Each recorded presentation will be accompanied with a link to the corresponding slide deck.

     

    Day 1:

     

    T. Roy founder of www.codemachine.com

    T. Roy presented on instrumentation techniques for developers and engineers who instrument binaries to help resolve complex operating system problems.

    http://blogs.msdn.com/ntdebugging/archive/2009/03/31/t-roy-from-codemachine-presents-supporting-support-at-the-2009-microsoft-gec.aspx

     

    Mark Russinovich

    Mark discussed new features and tools in the Sysinternals Tools suite.

    No video

     

    Best Practices for engaging Microsoft Support.

    Members from the Escalation Services team share best practices when engaging Microsoft support, included in this video is a demo of the new MPSReports tool and a demo of using Hyper-V to package up problems that can be reproduced and send to Microsoft.

    No video

     

    Day 2:

     

    Jeff Dailey - Principal Escalation Engineer

    Jeff discusses the different locations where you can find great content from the Escalation Services team to help solve common and difficult problems.

    http://blogs.msdn.com/ntdebugging/archive/2009/03/24/microsoft-platforms-global-escalation-services-community-presentation-at-the-2009-microsoft-gec.aspx

     

    Tate Calhoun - Escalation Engineer

    Tate walks through the use of the new XPERF tool and how to analyze the data for specific scenario's.

    http://blogs.msdn.com/ntdebugging/archive/2009/03/31/part-1-tate-calhoun-presents-xperf-at-the-2009-microsoft-gec.aspx 

    http://blogs.msdn.com/ntdebugging/archive/2009/03/31/part-2-tate-calhoun-presents-xperf-at-the-2009-microsoft-gec.aspx

     

    Citrix - Escalation Engineers

    Citrix engineers Nicholas Vasile, Dmitry Vostokov, and Kapil Ramlal, share tools they have created to take advantage of the ETW tracing infrastructure in Windows, and debugging scripts, and best practices for engaging issues that require multi-vendor support.

    http://blogs.msdn.com/ntdebugging/archive/2009/08/26/video-citrix-engineers-nicholas-vasile-dmitry-vostokov-and-kapil-ramlal-at-the-gec-conferences.aspx

     

    Dennis Smeltzer - Executive Director TSANet

    Dennis discusses a proposed new TSANet offering which would allow Escalation engineers to contact each other to collaborate on issues that require multi-vendor support.

    http://blogs.msdn.com/ntdebugging/archive/2009/03/24/tsanet-presentation-at-the-2009-microsoft-gec.aspx

     

    Share this post :
  • Ntdebugging Blog

    Reminder: Windows NT Debugging Blog Live Chat 2 Coming Soon!

    • 1 Comments

    Windows NT Debugging Blog Live Chat

    Microsoft Platform Global Escalation Services is hosting our second live group debug chat session for the debugging community on March 17, 2009 at 11 AM Pacific Time.  We will be focusing on debugging techniques and any questions you may have about anything we’ve previously blogged about. 

    Details about the “PGES-Windows NT Debugging Blog Live Chat” can be found here: http://www.microsoft.com/communities/chats/default.mspx

    Share this post :
  • Ntdebugging Blog

    Debug Fundamentals Exercise 4: The GAMEOVER exercise

    • 8 Comments

     

    We are back with another addition to the debugging fundamentals series here in 2009! Regardless of your debugging experience we hope you enjoy this one. Don’t worry too much if you can’t answer them all but give them your best shot.

     

    The problem:

     

    While debugging an application, a developer observes a crash and is puzzled. The developer saves a memory dump of the process using the command “.dump /ma dbgex4.dmp” and sends it to you for debug assistance.

     

    0:000> r

    eax=53454750 ebx=00000000 ecx=00000100 edx=000001b0 esi=001bfe20 edi=00ac36bc

    eip=47414d45 esp=001bfe00 ebp=4f564552 iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

    47414d45 ??              ???

     

    The attached zip, dbgex4.zip, includes three files. The program itself, the program’s symbols file and the saved memory dump file. (NOTE: You do not need to execute the program for the exercise however you may want to in order to solve the bonus’. You will need to rename the file dbgex4.ex_ to dbgex4.exe in order to run it).

     

    Using the accompanying files, answer the following questions:

     

    Part One – Debugging

     

    1.       What caused the access violation?

    2.       Examine the registers at the time of the crash.  Is there anything interesting about the contents?

    3.       How did the registers get into this state?

    4.       Find the offending data structure which caused this state to occur. What are its contents?

    5.       Are there security concerns with this access violation? Why?

    6.        Why is this class of crashes not seen in the wild much anymore?

    Part Two – Reverse Engineering

     

    Examine the functions main(), snap(), crack(), pop(), and boom().

    1.       Describe with a sentence or two what each one is doing.  (There is no need to comment on every assembly instruction or re-write the code in C here unless you really feel you need to.)

    Bonus:

    1.       If this access violation occurred while the program was running without a debugger present, would there be anything different about the crashing register state or the data structure which caused it? If so what is it and why? (HINT: If you would like to observe the crashing state like this, you can register windbg.exe to be the Just-In-Time debugger using the –I switch like so “windbg.exe -I”. When you run the dbgex4.exe program, it will crash and windbg.exe will launch automatically and attach to it.)

    2.       The functions from “Part Two – Reverse Engineering” perform operations on a set of data. During this manipulation some data is lost. Find this data and decode its “secret” message.

     

    Share this post :
  • Ntdebugging Blog

    Windows NT Debugging Blog Live Chat 2

    • 0 Comments

    Windows NT Debugging Blog Live Chat

    Microsoft Platform Global Escalation Services is hosting our second live group debug chat session for the debugging community on March 17, 2009 at 11 AM Pacific Time.  We will be focusing on debugging techniques and any questions you may have about anything we’ve previously blogged about. 

    Details about the “PGES-Windows NT Debugging Blog Live Chat” can be found here: http://www.microsoft.com/communities/chats/default.mspx

  • Ntdebugging Blog

    Easily Resolving an Event Viewer Error using a Process Memory Dump

    • 11 Comments

    My name is Ryan Mangipano (ryanman) and I am a Sr. Support Escalation Engineer at Microsoft.  Today I will be blogging about how I used the SOS  .Net Framework debugging extension (and !analyze -v) to easily troubleshoot a .Net Framework exception. This exception was preventing Event Viewer from displaying properly. Event Viewer was returning an error that provided very little information about what was actually causing the issue. I will demonstrate how, in this case, it was very easy to use windbg to obtain information about what went wrong. I did not have to perform a live debug on the issue.  Instead, I used a process dump to obtain very exact information, which was returned by the debugger, relating to the root cause. I was then able to use Process Monitor to identify the file that needed to be examined. These actions led me to the source of the problem, which was easily corrected. Also, the issue discussed in this blog was easily reproduced on Windows Server 2008. This means that you should be able to practice this debug exercise, on your non-production Windows 2008 SP1 Server, for learning purposes if you are interested in doing so.

     

    Issue Reported:  The following error was encountered when opening eventvwr.msc (Event Viewer) on a Windows 2008 Server system:

     

                           "MMC could not create the snap-in."

    MMC could not create the snap-in.

    The snap-in might not have been installed correctly

    Name: Event Viewer

                   CLSID: FX:{b05566ad-fe9c-4363-be05-7a4cbb7cb510}

    clip_image001

    First Step- Research & Data Gathering: After ensuring I first understood the problem reported, I searched for known issues. I found out that we have seen this error before.  It may occur when the following registry key gets deleted or corrupted:

     

                    HKLM\software\Microsoft\MMC\SnapIns\FX:{ b05566ad-fe9c-4363-be05-7a4cbb7cb510}

     

    I had the customer export this key and found that it was not corrupted in any way. I verified that all data was as expected

     

    Next,  a memory dump of the mmc.exe process was collected.  The mmc.exe process is used to host the eventvwr.msc snap-in. This was easily obtained using the built in Windows 2008 Server "Windows Task Manager" feature: "Create Dump File" . If you have several mmc console instances executing on your system, you can use the Task Manager context menu shortcuts "Switch To" and "Go To Process" to help you to identify the correct instance.

    clip_image002clip_image003

    clip_image004

     

    Note: We also collected a process monitor logfile during the startup of eventvwr.msc. This log file later proved very helpful in resolving the issue (as I will show below). Process monitor can be obtained at the following URL:  http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

     

     

     Now let's have a look at the debug.

     

    1.          First, I navigated Windows Explorer to the location of the dump file and double-clicked it to open it in windbg.exe.

    clip_image005

    It opened in windbg because I had previously run the command windbg -IA, which associates .dmp files with windbg. You can read more about the command line options in windbg in the help file that is included with the debugging tools.

    clip_image002[5]

    clip_image002[9]

    2.            I noticed the following output from the debugger after it loaded the dump file:

     

    This dump file has an exception of interest stored in it.

    The stored exception information can be accessed via .ecxr.

    (ff8.a2c): CLR exception - code e0434f4d (first/second chance not available)

     

     

    3.            Next, I wanted to ensure my symbol path was set correctly. I could have set it using the .sympath command:

     

    0:011> .sympath SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols

    0:011> .sympath

    Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols

     

    However, when your goal is to simply point to the default symbol server, .symfix is a very nice shortcut. It prevents one from having to try to remember the URL.  Here’s the syntax: 

     

    0:011> .symfix c:\websymbols

    0:011> .sympath

    Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

     

    4.            To ensure that I didn't waste time reviewing the wrong data, I performed a quick check to ensure that we collected a dump of the requested snap-in.

     

    0:005> !peb

    PEB at 000007fffffdb000

    ....

       CommandLine:  '"C:\Windows\system32\mmc.exe" "C:\Windows\system32\eventvwr.msc" '

     

    You could alternatively dump the  CommandLine from the nt!_PEB using the dt command

     

    0:005> dt nt!_PEB ProcessParameters->CommandLine 000007fffffdb000

    ntdll!_PEB

       +0x020 ProcessParameters              :

          +0x070 CommandLine                    : _UNICODE_STRING ""C:\Windows\system32\mmc.exe" "C:\Windows\system32\eventvwr.msc" "

     

    5.            Next, I then dumped out all of the threads in this process and found the following thread contained a stack that was raising a .Net Framework exception

     

    0:011> ~* kL

    ... (ommitted the non-relevent threads)

     

    # 11  Id: ff8.a2c Suspend: 1 Teb: 7ffd3000 Unfrozen

    ChildEBP RetAddr 

    0691f03c 7343a91c kernel32!RaiseException+0x58

    0691f09c 7343d81a mscorwks!RaiseTheExceptionInternalOnly+0x2a8

    *** WARNING: Unable to verify checksum for MMCEx.ni.dll

    0691f140 6bfe0b5a mscorwks!JIT_Rethrow+0xbf

    *** WARNING: Unable to verify checksum for mscorlib.ni.dll

    0691f1e8 69926cf6 MMCEx_ni+0xd0b5a

    0691f1f4 6993019f mscorlib_ni+0x216cf6

    0691f208 69926c74 mscorlib_ni+0x22019f

    0691f220 733d1b4c mscorlib_ni+0x216c74

    0691f230 733e21b1 mscorwks!CallDescrWorker+0x33

    0691f2b0 733f6501 mscorwks!CallDescrWorkerWithHandler+0xa3

    0691f3e8 733f6534 mscorwks!MethodDesc::CallDescr+0x19c

    0691f404 733f6552 mscorwks!MethodDesc::CallTargetWorker+0x1f

    0691f41c 7349d803 mscorwks!MethodDescCallSite::CallWithValueTypes+0x1a

    0691f604 733f845f mscorwks!ThreadNative::KickOffThread_Worker+0x192

    0691f618 733f83fb mscorwks!Thread::DoADCallBack+0x32a

    0691f6ac 733f8321 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3

    0691f6e8 733f84ad mscorwks!Thread::ShouldChangeAbortToUnload+0x30a

    0691f710 7349d5d4 mscorwks!Thread::ShouldChangeAbortToUnload+0x33e

     

    6.            Out of curiosity, I also ran the Get Last Error command

     

    0:011> !gle

    LastErrorValue: (Win32) 0 (0) - The operation completed successfully.

    LastStatusValue: (NTSTATUS) 0 - STATUS_WAIT_0

     

    7.            After this, I ran analyze -v to see what helpful information the debugger would provide. The debugger did output exception information but informed me that I needed to use the x86 debugger instead.

     

    0:011> !analyze -v

    ...

                            EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)

                            ExceptionAddress: 771a42eb (kernel32!RaiseException+0x00000058)

                              ExceptionCode: e0434f4d (CLR exception)

                              ExceptionFlags: 00000001

                            NumberParameters: 1

                            Parameter[0]: 80131604

                            MANAGED_BITNESS_MISMATCH:

                                                    Managed code needs matching platform of sos.dll for proper analysis. Use 'x86' debugger.

                            FAULTING_THREAD:  00000a2c

                                                    PRIMARY_PROBLEM_CLASS:  CLR_EXCEPTION

     

    8.            I fired up the x86 debugger and loaded the appropriate version of the SOS .Net Framework debugger extension.  This extension ships in the Operating System along with the .Net Framework.  On most occasions, I would have initiated the loading of the extension through the use of the following syntax:

     

     0:011> .load C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll

     

    OR

     

    0:011> .load  c:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.dll

     

    However, once you realize that managed debugging will be necessary and that you need the services of the SOS extension, it’s best to use the .loadby command rather than .load.  This is due to the fact that the version of SOS must match the version of the CLR loaded into that process.  Here’s the recommended syntax: 

     

                            0:011>.loadby sos mscorwks

     

    I always verify that my extensions are loaded properly by using the  .chain command.

     

    0:011> .chain

    ...   Extension DLL chain:

                            C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll: image 2.0.50727.1434, API 1.0.0, built Wed Dec 05 22:42:38 2007

     

    9.            Running !help printed out the following helpful information about the SOS extension since sos.dll was at the top of the .chain output:

     

    0:011> !help

    -------------------------------------------------------------------------------

    SOS is a debugger extension DLL designed to aid in the debugging of managed

    programs. Functions are listed by category, then roughly in order of

    importance. Shortcut names for popular functions are listed in parenthesis.

    Type "!help <functionname>" for detailed info on that function.

     

    Object Inspection                  Examining code and stacks

    -----------------------------      -----------------------------

    DumpObj (do)                       Threads

    DumpArray (da)                     CLRStack

    DumpStackObjects (dso)             IP2MD

    DumpHeap                           U

    DumpVC                             DumpStack

    GCRoot                             EEStack

    ObjSize                            GCInfo

    FinalizeQueue                      EHInfo

    PrintException (pe)                COMState

    TraverseHeap                       BPMD

     

    10.            Using the exception address, displayed by the debugger when opening the dump, and the !pe command listed above, I obtained more information about the exception:

                   

    0:011> !pe 771a42eb

    Invalid object

    There are nested exceptions on this thread. Run with -nested for details

     

    0:011> !pe -nested 771a42eb

    Invalid object

     

    Nested exception -------------------------------------------------------------

    Exception object: 040a676c

    Exception type: System.Reflection.TargetInvocationException

    Message: Exception has been thrown by the target of an invocation.

    InnerException: System.Reflection.TargetInvocationException, use !PrintException 040a6a20 to see more

    StackTrace (generated):

        SP       IP       Function

     

    StackTraceString: <none>

    HResult: 80131604

     

    0:011> !PrintException 040a6a20

    Exception object: 040a6a20

    Exception type: System.Reflection.TargetInvocationException

    Message: Exception has been thrown by the target of an invocation.

    InnerException: System.Configuration.ConfigurationErrorsException, use !PrintException 040a6cf8 to see more

    StackTrace (generated):

    <none>

    StackTraceString: <none>

    HResult: 80131604

    There are nested exceptions on this thread. Run with -nested for details

     

    0:011> !PrintException 040a6cf8

    Exception object: 040a6cf8

    Exception type: System.Configuration.ConfigurationErrorsException

    Message: Configuration system failed to initialize

    InnerException: System.Configuration.ConfigurationErrorsException, use !PrintException 040a7174 to see more

    StackTrace (generated):

    <none>

    StackTraceString: <none>

    HResult: 80131902

    There are nested exceptions on this thread. Run with -nested for details

     

    0:011> !PrintException 040a7174

    Exception object: 040a7174

    Exception type: System.Configuration.ConfigurationErrorsException

    Message: Unrecognized configuration section system.web/myInvalidData  

    InnerException: <none>

    StackTrace (generated):

    <none>

    StackTraceString: <none>

    HResult: 80131902

    There are nested exceptions on this thread. Run with -nested for details

     

    11.            Based on the exception information listed above, it appeared that a .Net Framework configuration section, system.web, contained an invalid configuration section named myInvalidData  inside of it. I then re-ran !analyze -v against the dump again (now that I had loaded the x86 debugger) and found that !analyze -v will load the sos.dll extension and even run the !pe extension automatically. It then automatically displayed the exception record information for me as well. Also, notice that the thread listed by !analyze -v matches the thread I examined earlier.

     

                0:011> !analyze -v

                    ...

                            EXCEPTION_MESSAGE:  Unrecognized configuration section system.web/myInvalidData.

                                                    MANAGED_OBJECT_NAME:  System.Configuration.ConfigurationErrorsException

                                                    FAULTING_THREAD:  00000a2c

     

    0:011> ~

       0  Id: ff8.c84 Suspend: 1 Teb: 7ffdf000 Unfrozen

       1  Id: ff8.96c Suspend: 1 Teb: 7ffde000 Unfrozen

       2  Id: ff8.d10 Suspend: 1 Teb: 7ffdd000 Unfrozen

       3  Id: ff8.d94 Suspend: 1 Teb: 7ffdc000 Unfrozen

       4  Id: ff8.a14 Suspend: 1 Teb: 7ffda000 Unfrozen

       5  Id: ff8.fbc Suspend: 1 Teb: 7ffd9000 Unfrozen

       6  Id: ff8.f88 Suspend: 1 Teb: 7ffd8000 Unfrozen

       7  Id: ff8.a64 Suspend: 1 Teb: 7ffd6000 Unfrozen

       8  Id: ff8.bf8 Suspend: 1 Teb: 7ffd5000 Unfrozen

       9  Id: ff8.d24 Suspend: 1 Teb: 7ffd4000 Unfrozen

      10  Id: ff8.ff0 Suspend: 1 Teb: 7ffd7000 Unfrozen

    . 11  Id: ff8.a2c Suspend: 1 Teb: 7ffd3000 Unfrozen

     

    12. At this point I was interested in identifying the source of this unrecognized configuration. Instead of engaging our .Net support team, I started with a quick search using www.live.com for     

     

                "unrecognized configuration section" system.web site:microsoft.com

                 

    This returned the following results http://search.live.com/results.aspx?q=%22unrecognized+configuration+section%22+system.web+site%3Amicrosoft.com&form=QBRE

     

    By quickly reviewing some of the hits returned, I found that others had encountered this exception in their own applications.  This is due to invalid entries in the various .config  files used in .Net. Looking through the posts, different configuration file names and paths were observed.

     

    So, I opened up the process monitor logfile to see which configuration files we were reading data from. I added filter criterion to match entries from the mmc.exe process, the TID from the FAULTING_THREAD listed in the exception data, path data containing .config,  and a successful status result. It's best to be as specific as possible.

     

    clip_image002[7]

    I found that we were reading in a large amount of settings over and over again from the .net Framework global configuration file:             

                                    c:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config

                    (on x64 this would be     C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\machine.config)

    clip_image002[11]

    Final Step- Putting it all together, Reproducing the issue, & confirming resolution : Using notepad, a quick search of the suspect xml file (C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\machine.config) on my system revealed a <system.web> section. At this point, I suspected that this section contained an invalid section which may have been related to the problem. To verify this, and since I like to break things, I added an invalid configuration setting <myInvalidData/> to my global configuration file. Doing so, I successfully reproduced the issue on my system. I then contacted the customer and asked if they had by any chance added any settings under the <system.web> in the configuration file: c:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config.

     

    The customer informed me that, per the request of their ASP.net developer, they had in fact added settings to that section of the file.  By researching http://msdn.microsoft.com/en-us/library/system.web.aspx and the schema documentation at http://msdn.microsoft.com/en-us/library/dayb112d.aspx, we were able to determine that the settings that were present in this file should not have been present inside of <system.web> . The settings were moved to the proper location per the developer and the issue was resolved.

     

    Here are the steps I used to reproduce the issue in case you are attempting to replicate this at home-

     

    A. Using notepad, open the following configuration file on a non-production Windows Server 2008 SP1 system:

         (please make a backup copy first in case you make a mistake)  

       c:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config

         OR   (Open the version that matches the architecture of your platform )

       C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\machine.config

     

    clip_image012

    B. Find the section <system.web> in this file (you can use the find function in notepad):

    clip_image013

    C.  Add the following line directly after <system.web> as shown in the example below:

                         <myInvalidData/>   

    clip_image014

    D. Save the file and then open eventvwr.msc and verify that the following error is displayed:

    clip_image015

    Conclusion

    Hopefully this blog has demonstrated an example of how you can use the "create dump file" feature of Windows 2008, windbg, and other related tools in an attempt to gain more specific data when your error message is not revealing the source of the problem. Feel free to post any questions, comments, or concerns.

     

  • Ntdebugging Blog

    Debugging a Hang at “Applying Computer Settings”

    • 2 Comments

    Hi Everyone - My name is Aman. I'm an Escalation Engineer on the Microsoft GES (Global Escalation Services) team. We recently came across a critical situation where-in the following issues were reported by the customer:

     

    1.       DC login hangs at “applying computer settings” regardless of RDP or local console.

    2.       Connecting to the server using computer management console fails with "RPC Server is too busy to complete the operation"

    3.       Server responds to ping in the problem state but cannot be accessed through ports 445/139.

    4.       Reboot will fix the issue for a couple of days but the problem will return again.

     

    The case had already been troubleshot many times in the past without any results. So, to get to the root cause, it was escalated to GES and we took a full memory dump. In this blog I want to show you how to navigate between processes via LPC messages to drill down to root cause.

     

    Disclaimer – Not every hang at “Applying Computer Settings” requires a debug. Standard troubleshooting typically resolves this type of thing. 

     

    Now let's have a look at the Complete Memory Dump (note: a Kernel dump is not enough). Since the console session hangs at "applying computer settings", the target starting point of my debug will be the process called winlogon. Winlogon.exe is the executable responsible for initiating the logon process for users. If the console session is hung at "applying computer settings", winlogon has to have an explanation as to why it's unable to process the user logon successfully.

     

    So, let’s get the address of the winlogon process:

     

    kd> ! process 0 0 winlogon.exe

    PROCESS 85c63830  SessionId: 0  Cid: 01a4    Peb: 7ffdf000  ParentCid: 0158

        DirBase: 39f6a000  ObjectTable: e13d7928  HandleCount: 535.

        Image: winlogon.exe

     

    PROCESS 84f02020  SessionId: 1  Cid: 0344    Peb: 7ffdf000  ParentCid: 0158

        DirBase: 2d675000  ObjectTable: e35abc78  HandleCount: 313.

        Image: winlogon.exe

     

     

    So, why we have two winlogon processes executing? Simple, the customer tried to log into the domain controller via RDP and local console. Both methods resulted in the hang, hence creating two winlogon sessions. Let’s check the first one and see what's happening in there. I picked up only questionable threads for the sake of this session.

     

     

    kd> .process /p /r 85c63830

    Implicit process is now 85c63830

    Loading User Symbols

    ................................................................

    .........

    kd> !process 85c63830 7

    PROCESS 85c63830  SessionId: 0  Cid: 01a4    Peb: 7ffdf000  ParentCid: 0158

        DirBase: 39f6a000  ObjectTable: e13d7928  HandleCount: 535.

        Image: winlogon.exe

        VadRoot 85c599c8 Vads 157 Clone 0 Private 1266. Modified 1081. Locked 0.

        DeviceMap e1001150

        Token                             e1743a38

        ElapsedTime                       15:50:33.974

        UserTime                          00:00:00.187

        KernelTime                        00:00:02.125

        QuotaPoolUsage[PagedPool]         53444

        QuotaPoolUsage[NonPagedPool]      63864

        Working Set Sizes (now,min,max)  (1161, 50, 345) (4644KB, 200KB, 1380KB)

        PeakWorkingSetSize                2834

        VirtualSize                       49 Mb

        PeakVirtualSize                   52 Mb

        PageFaultCount                    10861

        MemoryPriority                    BACKGROUND

        BasePriority                      13

        CommitCharge                      1979

     

     

     

            THREAD 847318d0  Cid 01a4.58e4  Teb: 7ffdd000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                84731abc  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 00213bd7:

            Current LPC port e125d4e0

            Not impersonating

            DeviceMap                 e1001150

            Owning Process            85c63830       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3650157        Ticks: 525 (0:00:00:08.203)

            Context Switch Count      2            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init f5c98000 Current f5c97c20 Base f5c98000 Limit f5c95000 Call 0

            Priority 14 BasePriority 13 PriorityDecrement 0

            ChildEBP RetAddr  Args to Child             

            f5c97c38 808202b6 847318d0 84731978 00000000 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            f5c97c50 8081fb6e 84731abc 84731a90 847318d0 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            f5c97c94 80915355 84731abc 00000011 f5c97d01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            f5c97d50 8082350b 0000080c 00087ba0 00087ba0 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            f5c97d50 7c8285ec 0000080c 00087ba0 00087ba0 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f5c97d64)

            0075fab0 7c82787b 77c80a6e 0000080c 00087ba0 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            0075fab4 77c80a6e 0000080c 00087ba0 00087ba0 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            0075fb00 77c7fcf0 0075fb3c 0075fb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            0075fb0c 77c80673 0075fb3c 75822c38 0075ff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            0075fb20 77ce315a 0075fb68 00087c18 001125a8 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            0075ff08 75822b74 75822c38 75822b8a 0075ff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            0075ff20 75822b4f 00b81460 0075ff3c 0075ff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            0075ff30 75824e7a 0075ff3c 00100000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            0075ff60 0103917b 0075ff78 00000000 00000000 WlNotify!SensStopScreenSaverEvent+0x58 (FPO: [1,9,0])

            0075ffb8 77e64829 001125a8 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            0075ffec 00000000 010390b7 001125a8 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

    We found somewhere around 15 threads in the process above. Only the above thread looked suspicious since it has issued a LPC call and hasn't received a reply yet. Before we investigate further, let’s check the other winlogon process and find out about the threads in there as well....just in case if there is a commonality or not amongst both the processes...

     

    Note the long tick counts in these threads!

     

     

    kd> .process /p /r 84f02020

    Implicit process is now 84f02020

    Loading User Symbols

    ................................................................

    ................

    kd> !process 84f02020 7

    PROCESS 84f02020  SessionId: 1  Cid: 0344    Peb: 7ffdf000  ParentCid: 0158

        DirBase: 2d675000  ObjectTable: e35abc78  HandleCount: 313.

        Image: winlogon.exe

        VadRoot 85e86dd0 Vads 184 Clone 0 Private 656. Modified 722. Locked 0.

        DeviceMap e1001150

        Token                             e3a09030

        ElapsedTime                       04:51:33.657

        UserTime                          00:00:00.140

        KernelTime                        00:00:00.796

        QuotaPoolUsage[PagedPool]         49464

        QuotaPoolUsage[NonPagedPool]      10080

        Working Set Sizes (now,min,max)  (1099, 50, 345) (4396KB, 200KB, 1380KB)

        PeakWorkingSetSize                2193

        VirtualSize                       48 Mb

        PeakVirtualSize                   50 Mb

        PageFaultCount                    4312

        MemoryPriority                    BACKGROUND

        BasePriority                      13

        CommitCharge                      1329

     

            THREAD 847a93f0  Cid 0344.4ab0  Teb: 7ffd7000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                847a95dc  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 0020b318:

            Current LPC port e3761290

            Impersonation token:  e3a92250 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3500419        Ticks: 150263 (0:00:39:07.859)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9d00000 Current b9cffc20 Base b9d00000 Limit b9cfd000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b9cffc38 808202b6 847a93f0 847a9498 00000310 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b9cffc50 8081fb6e 847a95dc 847a95b0 847a93f0 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b9cffc94 80915355 847a95dc 00000011 00000001 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b9cffd50 8082350b 00000364 000b9760 000b9760 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b9cffd50 7c8285ec 00000364 000b9760 000b9760 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9cffd64)

            0087fab0 7c82787b 77c80a6e 00000364 000b9760 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            0087fab4 77c80a6e 00000364 000b9760 000b9760 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            0087fb00 77c7fcf0 0087fb3c 0087fb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            0087fb0c 77c80673 0087fb3c 75822c38 0087ff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            0087fb20 77ce315a 0087fb68 000b9820 000a8bd8 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            0087ff08 75822b74 75822c38 75822b8a 0087ff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            0087ff20 75822b4f 000c1e30 0087ff3c 0087ff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            0087ff30 75824e19 0087ff3c 00080000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            0087ff60 0103917b 0087ff78 00000000 00000000 WlNotify!SensStartScreenSaverEvent+0x58 (FPO: [1,9,0])

            0087ffb8 77e64829 000a8bd8 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            0087ffec 00000000 010390b7 000a8bd8 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 847ad8d0  Cid 0344.4c7c  Teb: 7ffac000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                847adabc  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 0020bf59:

            Current LPC port e3761290

            Impersonation token:  e39b1358 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3513766        Ticks: 136916 (0:00:35:39.312)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9dc0000 Current b9dbfc20 Base b9dc0000 Limit b9dbd000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b9dbfc38 808202b6 847ad8d0 847ad978 00000341 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b9dbfc50 8081fb6e 847adabc 847ada90 847ad8d0 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b9dbfc94 80915355 847adabc 00000011 b9dbfd01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b9dbfd50 8082350b 00000364 000ba340 000ba340 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b9dbfd50 7c8285ec 00000364 000ba340 000ba340 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9dbfd64)

            00b7fab0 7c82787b 77c80a6e 00000364 000ba340 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            00b7fab4 77c80a6e 00000364 000ba340 000ba340 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            00b7fb00 77c7fcf0 00b7fb3c 00b7fb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            00b7fb0c 77c80673 00b7fb3c 75822c38 00b7ff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            00b7fb20 77ce315a 00b7fb68 000ba400 000a8b80 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            00b7ff08 75822b74 75822c38 75822b8a 00b7ff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            00b7ff20 75822b4f 000c1e30 00b7ff3c 00b7ff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            00b7ff30 75824e7a 00b7ff3c 00100000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            00b7ff60 0103917b 00b7ff78 00000000 00000000 WlNotify!SensStopScreenSaverEvent+0x58 (FPO: [1,9,0])

            00b7ffb8 77e64829 000a8b80 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            00b7ffec 00000000 010390b7 000a8b80 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 84989b40  Cid 0344.4c88  Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                84989d2c  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 0020bf60:

            Current LPC port e3761290

            Impersonation token:  e3a48030 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3513766        Ticks: 136916 (0:00:35:39.312)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9c44000 Current b9c43c20 Base b9c44000 Limit b9c41000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b9c43c38 808202b6 84989b40 84989be8 00000741 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b9c43c50 8081fb6e 84989d2c 84989d00 84989b40 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b9c43c94 80915355 84989d2c 00000011 b9c43d01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b9c43d50 8082350b 00000364 000c18a8 000c18a8 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b9c43d50 7c8285ec 00000364 000c18a8 000c18a8 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9c43d64)

            00dffab0 7c82787b 77c80a6e 00000364 000c18a8 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            00dffab4 77c80a6e 00000364 000c18a8 000c18a8 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            00dffb00 77c7fcf0 00dffb3c 00dffb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            00dffb0c 77c80673 00dffb3c 75822c38 00dfff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            00dffb20 77ce315a 00dffb68 000c1968 000bcd80 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            00dfff08 75822b74 75822c38 75822b8a 00dfff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            00dfff20 75822b4f 000c1e30 00dfff3c 00dfff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            00dfff30 75824d57 00dfff3c 00020000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            00dfff60 0103917b 00dfff78 00000000 00000000 WlNotify!SensLockEvent+0x58 (FPO: [1,9,0])

            00dfffb8 77e64829 000bcd80 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            00dfffec 00000000 010390b7 000bcd80 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 847d7b18  Cid 0344.4c9c  Teb: 7ffa6000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                847d7d04  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 0020bfb4:

            Current LPC port e3761290

            Impersonation token:  e423a358 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3514154        Ticks: 136528 (0:00:35:33.250)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9c00000 Current b9bffc20 Base b9c00000 Limit b9bfd000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b9bffc38 808202b6 847d7b18 847d7bc0 00002601 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b9bffc50 8081fb6e 847d7d04 847d7cd8 847d7b18 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b9bffc94 80915355 847d7d04 00000011 b9bffd01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b9bffd50 8082350b 00000364 000ab970 000ab970 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b9bffd50 7c8285ec 00000364 000ab970 000ab970 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9bffd64)

            00fcfab0 7c82787b 77c80a6e 00000364 000ab970 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            00fcfab4 77c80a6e 00000364 000ab970 000ab970 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            00fcfb00 77c7fcf0 00fcfb3c 00fcfb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            00fcfb0c 77c80673 00fcfb3c 75822c38 00fcff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            00fcfb20 77ce315a 00fcfb68 000aba30 000bd858 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            00fcff08 75822b74 75822c38 75822b8a 00fcff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            00fcff20 75822b4f 000c1e30 00fcff3c 00fcff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            00fcff30 75824db8 00fcff3c 00040000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            00fcff60 0103917b 00fcff78 00000000 00000000 WlNotify!SensUnlockEvent+0x58 (FPO: [1,9,0])

            00fcffb8 77e64829 000bd858 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            00fcffec 00000000 010390b7 000bd858 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 8478e8d0  Cid 0344.53d0  Teb: 7ffab000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                8478eabc  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 0020f7bb:

            Current LPC port e3761290

            Impersonation token:  e3811358 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3572296        Ticks: 78386 (0:00:20:24.781)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b94bc000 Current b94bbc20 Base b94bc000 Limit b94b9000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b94bbc38 808202b6 8478e8d0 8478e978 00000310 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b94bbc50 8081fb6e 8478eabc 8478ea90 8478e8d0 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b94bbc94 80915355 8478eabc 00000011 b94bbd01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b94bbd50 8082350b 00000364 000880a8 000880a8 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b94bbd50 7c8285ec 00000364 000880a8 000880a8 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b94bbd64)

            00dbfab0 7c82787b 77c80a6e 00000364 000880a8 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            00dbfab4 77c80a6e 00000364 000880a8 000880a8 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            00dbfb00 77c7fcf0 00dbfb3c 00dbfb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            00dbfb0c 77c80673 00dbfb3c 75822c38 00dbff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            00dbfb20 77ce315a 00dbfb68 00088168 0008bd80 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            00dbff08 75822b74 75822c38 75822b8a 00dbff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            00dbff20 75822b4f 000c1e30 00dbff3c 00dbff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            00dbff30 75824e19 00dbff3c 00080000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            00dbff60 0103917b 00dbff78 00000000 00000000 WlNotify!SensStartScreenSaverEvent+0x58 (FPO: [1,9,0])

            00dbffb8 77e64829 0008bd80 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            00dbffec 00000000 010390b7 0008bd80 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 846e6660  Cid 0344.56a8  Teb: 7ffa7000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                846e684c  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 0021117f:

            Current LPC port e3761290

            Impersonation token:  e37a5358 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3602966        Ticks: 47716 (0:00:12:25.562)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9500000 Current b94ffc20 Base b9500000 Limit b94fd000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b94ffc38 808202b6 846e6660 846e6708 00000241 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b94ffc50 8081fb6e 846e684c 846e6820 846e6660 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b94ffc94 80915355 846e684c 00000011 b94ffd01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b94ffd50 8082350b 00000364 0008ba78 0008ba78 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b94ffd50 7c8285ec 00000364 0008ba78 0008ba78 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b94ffd64)

            00f8fab0 7c82787b 77c80a6e 00000364 0008ba78 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            00f8fab4 77c80a6e 00000364 0008ba78 0008ba78 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            00f8fb00 77c7fcf0 00f8fb3c 00f8fb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            00f8fb0c 77c80673 00f8fb3c 75822c38 00f8ff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            00f8fb20 77ce315a 00f8fb68 0008bb38 0009bbf0 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            00f8ff08 75822b74 75822c38 75822b8a 00f8ff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            00f8ff20 75822b4f 000c1e30 00f8ff3c 00f8ff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            00f8ff30 75824e7a 00f8ff3c 00100000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            00f8ff60 0103917b 00f8ff78 00000000 00000000 WlNotify!SensStopScreenSaverEvent+0x58 (FPO: [1,9,0])

            00f8ffb8 77e64829 0009bbf0 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            00f8ffec 00000000 010390b7 0009bbf0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 8470d428  Cid 0344.56b4  Teb: 7ffa4000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                8470d614  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 00211186:

            Current LPC port e3761290

            Impersonation token:  e4167358 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3602966        Ticks: 47716 (0:00:12:25.562)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9378000 Current b9377c20 Base b9378000 Limit b9375000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b9377c38 808202b6 8470d428 8470d4d0 00002241 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b9377c50 8081fb6e 8470d614 8470d5e8 8470d428 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b9377c94 80915355 8470d614 00000011 b9377d01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b9377d50 8082350b 00000364 0013b340 0013b340 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b9377d50 7c8285ec 00000364 0013b340 0013b340 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9377d64)

            016dfab0 7c82787b 77c80a6e 00000364 0013b340 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            016dfab4 77c80a6e 00000364 0013b340 0013b340 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            016dfb00 77c7fcf0 016dfb3c 016dfb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            016dfb0c 77c80673 016dfb3c 75822c38 016dff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            016dfb20 77ce315a 016dfb68 0013b400 0007f808 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            016dff08 75822b74 75822c38 75822b8a 016dff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            016dff20 75822b4f 000c1e30 016dff3c 016dff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            016dff30 75824d57 016dff3c 00020000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            016dff60 0103917b 016dff78 00000000 00000000 WlNotify!SensLockEvent+0x58 (FPO: [1,9,0])

            016dffb8 77e64829 0007f808 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            016dffec 00000000 010390b7 0007f808 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

            THREAD 846c4b40  Cid 0344.56d0  Teb: 7ffa2000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

                846c4d2c  Semaphore Limit 0x1

            Waiting for reply to LPC MessageId 00211223:

            Current LPC port e3761290

            Impersonation token:  e3826358 (Level Impersonation)

            DeviceMap                 e1bca430

            Owning Process            84f02020       Image:         winlogon.exe

            Attached Process          N/A            Image:         N/A

            Wait Start TickCount      3603490        Ticks: 47192 (0:00:12:17.375)

            Context Switch Count      1            

            UserTime                  00:00:00.000

            KernelTime                00:00:00.000

            Win32 Start Address winlogon!WlpExecuteNotify (0x010390b7)

            Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

            Stack Init b9310000 Current b930fc20 Base b9310000 Limit b930d000 Call 0

            Priority 13 BasePriority 13 PriorityDecrement 0

            Kernel stack not resident.

            ChildEBP RetAddr  Args to Child             

            b930fc38 808202b6 846c4b40 846c4be8 00002301 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

            b930fc50 8081fb6e 846c4d2c 846c4d00 846c4b40 nt!KiSwapThread+0x83 (FPO: [0,2,0])

            b930fc94 80915355 846c4d2c 00000011 b930fd01 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

            b930fd50 8082350b 00000364 0013c440 0013c440 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

            b930fd50 7c8285ec 00000364 0013c440 0013c440 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b930fd64)

            0175fab0 7c82787b 77c80a6e 00000364 0013c440 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

            0175fab4 77c80a6e 00000364 0013c440 0013c440 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

            0175fb00 77c7fcf0 0175fb3c 0175fb20 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230 (FPO: [1,11,0])

            0175fb0c 77c80673 0175fb3c 75822c38 0175ff28 RPCRT4!I_RpcSendReceive+0x24 (FPO: [1,0,0])

            0175fb20 77ce315a 0175fb68 0013c500 00098ef8 RPCRT4!NdrSendReceive+0x2b (FPO: [2,0,0])

            0175ff08 75822b74 75822c38 75822b8a 0175ff28 RPCRT4!NdrClientCall2+0x22e (FPO: [SEH])

            0175ff20 75822b4f 000c1e30 0175ff3c 0175ff60 WlNotify!SensApip_RPC_SensNotifyWinlogonEvent+0x1c (FPO: [2,1,0])

            0175ff30 75824db8 0175ff3c 00040000 00000024 WlNotify!SensNotifyWinlogonEvent+0x1c (FPO: [1,0,0])

            0175ff60 0103917b 0175ff78 00000000 00000000 WlNotify!SensUnlockEvent+0x58 (FPO: [1,9,0])

            0175ffb8 77e64829 00098ef8 00000000 00000000 winlogon!WlpExecuteNotify+0xc4 (FPO: [SEH])

            0175ffec 00000000 010390b7 00098ef8 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

    Wow!....look at that...around 8 threads, and all waiting for replies to the LPC calls issued. So, both the winlogon processes are stuck waiting for a reply for the LPC calls they made.

     

    The next part is to find out what's happening at the server part of these LPC calls since winlogon is stuck until the processing of the LPC calls are finished.

     

    To find the server part of the LPC call, we pick the message ID from each thread. Look for (Waiting for reply to LPC MessageId "message_id":) in the threads above. Let's start with the first winlogon process to find out the server end of the LPC call it had in there. If you remember, we did find one thread in the first winlogon process waiting for a reply to the LPC call and message id of that LPC call was 00213bd7.

     

    kd> !lpc message 00213bd7

    Searching message 213bd7 in threads ...

    Client thread 847318d0 waiting a reply from 213bd7                         

    Searching thread 847318d0 in port rundown queues ...

     

    Server communication port 0xe365aa90

        Handles: 1   References: 1

        The LpcDataInfoChainHead queue is empty

            Connected port: 0xe125d4e0      Server connection port: 0xe1c37920

     

    Client communication port 0xe125d4e0

        Handles: 1   References: 2

        The LpcDataInfoChainHead queue is empty

     

    Server connection port e1c37920  Name: senssvc

        Handles: 1   References: 17

        Server process  : 85e021e0 (svchost.exe)

        Queue semaphore : 850fa890

        Semaphore state 16 (0x10)

            Messages in queue:

            0000 e3791a88 - Busy  Id=001fbe4f  From: 0f4c.335c  Context=80060002  [e1c37930 . e1281378]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000028 00000005 00010050 00d0fdac

            0000 e1281378 - Busy  Id=001fbe56  From: 0f4c.336c  Context=80060002  [e3791a88 . e15c5350]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00490065 0074006e 006c0065 00500000

            0000 e15c5350 - Busy  Id=00200326  From: 0f4c.3840  Context=80060002  [e1281378 . e14ba008]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 0008c080 00000000 0049004d 00440048

            0000 e14ba008 - Busy  Id=0020032a  From: 0f4c.3848  Context=80060002  [e15c5350 . e1120290]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000000 00000000 0000000c 00000000

            0000 e1120290 - Busy  Id=002049b3  From: 0f4c.3b08  Context=80060002  [e14ba008 . e40a48c8]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00600040 00600050 00900050 00600050

            0000 e40a48c8 - Busy  Id=00204e8d  From: 0f4c.09c8  Context=80060002  [e1120290 . e1ab41d0]

                       Length=00200008  Type=00000005 (LPC_PORT_CLOSED)

                       Data: fd48cde3 01c9700a

            0000 e1ab41d0 - Busy  Id=0020b318  From: 0344.4ab0  Context=80050001  [e40a48c8 . e1b35378]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000028 00000005 00010050 00d0fdac

            0000 e1b35378 - Busy  Id=0020bf59  From: 0344.4c7c  Context=80050001  [e1ab41d0 . e1bdc918]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000000 00000000 00000000 00000000

            0000 e1bdc918 - Busy  Id=0020bf60  From: 0344.4c88  Context=80050001  [e1b35378 . e1657ca0]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 0037002d 00350032 00340033 00350035

            0000 e1657ca0 - Busy  Id=0020bfb4  From: 0344.4c9c  Context=80050001  [e1bdc918 . e3a463e8]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 4ed07378 cf0ee4b3 5faeb176 0000000c

            0000 e3a463e8 - Busy  Id=0020f7bb  From: 0344.53d0  Context=80050001  [e1657ca0 . e1bcab08]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000000 00000000 00000000 00000000

            0000 e1bcab08 - Busy  Id=0021117f  From: 0344.56a8  Context=80050001  [e3a463e8 . e4190008]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000000 813a8198 00001601 00027011

            0000 e4190008 - Busy  Id=00211186  From: 0344.56b4  Context=80050001  [e1bcab08 . e40afca8]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 002e0070 0061006d 00720072 006f0069

            0000 e40afca8 - Busy  Id=00211223  From: 0344.56d0  Context=80050001  [e4190008 . e37de340]

                       Length=00c000a8  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 4bdd485c 5d18bba4 d4bb211b 0000000c

            0000 e37de340 - Busy  Id=00213bd7  From: 01a4.58e4  Context=80030000  [e40afca8 . e1c37930]

                       Length=00780060  Type=00000001 (LPC_REQUEST)

                       Data: 00000001 00000241 00000000 811fc308 00001601 000152cb

        The message queue contains 15 messages

        The LpcDataInfoChainHead queue is empty

     

    Done!                                            

     

    Interesting, along with the message id, 00213bd7 , in the queue, we also found all other message ID's seen in the second winlogon process.

    Take a look at the messages in queue and you will see all of them in there.

     

    So, now the question arises as to what is this process doing and why its unable to process these LPC call's successfully.

     

    Let's switch context to the Server process  :  85e021e0 (svchost.exe)

     

    kd> .process /p /r 85e021e0

    Implicit process is now 85e021e0 

    Loading User Symbols

    ................................................................

    ...........................................

     

    When I dumped the threads in this process, the thread count turned out to be an alarming 4900 which is extremely bizarre. Around 1325 of those threads were waiting to acquire a common critical section. This became a very solid point of interest and almost all of them (around 4870) were in a wait state pending a single common SynchronizationEvent i.e. 85d1eeb0.

     

    These threads wait for the message to come through via the LPC port to the server process. That also explains why the server service (user mode component: srvsvc.dll) is unable to process any remote requests.

     

    Few of those threads, waiting on the common critical section, looked like this: (Don’t get bothered by the “Kernel stack not resident” as we know memory gets swapped to disk)

     

    kd> !thread 84832af8

    THREAD 84832af8  Cid 03d8.4760  Teb: 7ef49000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

        85d1eeb0  SynchronizationEvent

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3474348        Ticks: 176334 (0:00:45:55.218)

    Context Switch Count      1            

    UserTime                  00:00:00.000

    KernelTime                00:00:00.000

    Win32 Start Address srvsvc!XsProcessApisWrapper (0x74ed39a9)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init b9fd4000 Current b9fd3c78 Base b9fd4000 Limit b9fd1000 Call 0

    Priority 8 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident.

    ChildEBP RetAddr  Args to Child             

    b9fd3c90 808202b6 84832af8 84832ba0 00000000 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

    b9fd3ca8 8081fb6e 00000000 00000000 00000000 nt!KiSwapThread+0x83 (FPO: [0,2,0])

    b9fd3cec 8090e64e 85d1eeb0 00000006 00000001 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

    b9fd3d50 8082350b 00000314 00000000 00000000 nt!NtWaitForSingleObject+0x9a (FPO: [SEH])

    b9fd3d50 7c8285ec 00000314 00000000 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9fd3d64)

    3d6bfbc8 7c827d0b 7c83d236 00000314 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    3d6bfbcc 7c83d236 00000314 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])

    3d6bfc08 7c83d281 00000314 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3 (FPO: [2,7,4])

    3d6bfc28 7c82ee3b 7c8877a0 00000000 7ffd8000 ntdll!RtlEnterCriticalSection+0xa8 (FPO: [1,1,0])

    3d6bfcb8 7c82ec9f 3d6bfd28 3d6bfd28 00000000 ntdll!LdrpInitializeThread+0x68 (FPO: [SEH])

    3d6bfd14 7c8284c5 3d6bfd28 7c800000 00000000 ntdll!_LdrpInitialize+0x16f (FPO: [SEH])

    00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x25

     

    kd> !thread 847d8b40

    THREAD 847d8b40  Cid 03d8.4be0  Teb: 7ed63000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

        85d1eeb0  SynchronizationEvent

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3549550        Ticks: 101132 (0:00:26:20.187)

    Context Switch Count      1            

    UserTime                  00:00:00.000

    KernelTime                00:00:00.000

    Win32 Start Address srvsvc!XsProcessApisWrapper (0x74ed39a9)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init b98a4000 Current b98a3c78 Base b98a4000 Limit b98a1000 Call 0

    Priority 8 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident.

    ChildEBP RetAddr  Args to Child             

    b98a3c90 808202b6 847d8b40 847d8be8 00000000 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

    b98a3ca8 8081fb6e 00000000 00000000 00000000 nt!KiSwapThread+0x83 (FPO: [0,2,0])

    b98a3cec 8090e64e 85d1eeb0 00000006 00000001 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

    b98a3d50 8082350b 00000314 00000000 00000000 nt!NtWaitForSingleObject+0x9a (FPO: [SEH])

    b98a3d50 7c8285ec 00000314 00000000 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b98a3d64)

    4503fbc8 7c827d0b 7c83d236 00000314 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    4503fbcc 7c83d236 00000314 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])

    4503fc08 7c83d281 00000314 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3 (FPO: [2,7,4])

    4503fc28 7c82ee3b 7c8877a0 00000000 7ffd8000 ntdll!RtlEnterCriticalSection+0xa8 (FPO: [1,1,0])

    4503fcb8 7c82ec9f 4503fd28 4503fd28 00000000 ntdll!LdrpInitializeThread+0x68 (FPO: [SEH])

    4503fd14 7c8284c5 4503fd28 7c800000 00000000 ntdll!_LdrpInitialize+0x16f (FPO: [SEH])

    00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x25

     

    kd> !thread 847a8db0

    THREAD 847a8db0  Cid 03d8.4998  Teb: 7eed1000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

        85d1eeb0  SynchronizationEvent

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3487819        Ticks: 162863 (0:00:42:24.734)

    Context Switch Count      1            

    UserTime                  00:00:00.000

    KernelTime                00:00:00.000

    Win32 Start Address srvsvc!XsProcessApisWrapper (0x74ed39a9)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init b9dc4000 Current b9dc3c78 Base b9dc4000 Limit b9dc1000 Call 0

    Priority 8 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident.

    ChildEBP RetAddr  Args to Child              

    b9dc3c90 808202b6 847a8db0 847a8e58 00000000 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

    b9dc3ca8 8081fb6e 00000000 00000000 00000000 nt!KiSwapThread+0x83 (FPO: [0,2,0])

    b9dc3cec 8090e64e 85d1eeb0 00000006 00000001 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

    b9dc3d50 8082350b 00000314 00000000 00000000 nt!NtWaitForSingleObject+0x9a (FPO: [SEH])

    b9dc3d50 7c8285ec 00000314 00000000 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ b9dc3d64)

    3f4bfbc8 7c827d0b 7c83d236 00000314 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    3f4bfbcc 7c83d236 00000314 00000000 00000000 ntdll!NtWaitForSingleObject+0xc (FPO: [3,0,0])

    3f4bfc08 7c83d281 00000314 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3 (FPO: [2,7,4])

    3f4bfc28 7c82ee3b 7c8877a0 00000000 7ffd8000 ntdll!RtlEnterCriticalSection+0xa8 (FPO: [1,1,0])

    3f4bfcb8 7c82ec9f 3f4bfd28 3f4bfd28 00000000 ntdll!LdrpInitializeThread+0x68 (FPO: [SEH])

    3f4bfd14 7c8284c5 3f4bfd28 7c800000 00000000 ntdll!_LdrpInitialize+0x16f (FPO: [SEH])

    00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x25

     

    kd> !thread 848e1db0

    THREAD 848e1db0  Cid 03d8.2c10  Teb: 7f67a000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

        85d1eeb0  SynchronizationEvent

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3183097        Ticks: 467585 (0:02:01:46.015)

    Context Switch Count      1             

    UserTime                  00:00:00.000

    KernelTime                00:00:00.000

    Win32 Start Address srvsvc!XsProcessApisWrapper (0x74ed39a9)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init f3570000 Current f356fc78 Base f3570000 Limit f356d000 Call 0

    Priority 8 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident

     

    kd> !thread 84890db0

    THREAD 84890db0  Cid 03d8.31cc  Teb: 7f4e3000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

        85d1eeb0  SynchronizationEvent

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3244800        Ticks: 405882 (0:01:45:41.906)

    Context Switch Count      1            

    UserTime                  00:00:00.000

    KernelTime                00:00:00.000

    Win32 Start Address srvsvc!XsProcessApisWrapper (0x74ed39a9)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init f2ee8000 Current f2ee7c78 Base f2ee8000 Limit f2ee5000 Call 0

    Priority 8 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident.

     

    kd> !thread 848223f0

    THREAD 848223f0  Cid 03d8.3714  Teb: 7f3d8000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable

        85d1eeb0  SynchronizationEvent

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3283235        Ticks: 367447 (0:01:35:41.359)

    Context Switch Count      1            

    UserTime                  00:00:00.000

    KernelTime                00:00:00.000

    Win32 Start Address srvsvc!XsProcessApisWrapper (0x74ed39a9)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init f2ac4000 Current f2ac3c78 Base f2ac4000 Limit f2ac1000 Call 0

    Priority 8 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident.

     

    kd> dt ntdll!_RTL_CRITICAL_SECTION 7c8877a0

       +0x000 DebugInfo        : 0x7c8877c0 _RTL_CRITICAL_SECTION_DEBUG

       +0x004 LockCount        : -19514

       +0x008 RecursionCount   : 2

       +0x00c OwningThread     : 0x00001634

       +0x010 LockSemaphore    : 0x00000314

       +0x014 SpinCount        : 0

     

    In Microsoft Windows Server 2003 Service Pack 1 and later versions of Windows, the LockCount field is parsed as follows: (also at link http://msdn.microsoft.com/en-us/library/cc267476.aspx)

    • The lowest bit shows the lock status. If this bit is 0, the critical section is locked; if it is 1, the critical section is not locked.
    • The next bit shows whether a thread has been woken for this lock. If this bit is 0, then a thread has been woken for this lock; if it is 1, no thread has been woken.
    • The remaining bits are the ones-complement of the number of threads waiting for the lock.

    In our case, the LockCount is -19514. The lowest bit can be determined in this way:

    kd> ? 0x1 & (-0n19514)

    Evaluate expression: 0 = 00000000

     

    kd>  (0x2 & (-0n19514)) >> 1

         ^ Syntax error in ' (0x2 & (-0n19514)) >> 1'

     

    kd> ?(0x2 & (-0n19514)) >> 1

    Evaluate expression: 1 = 00000001

     

    kd> ? ((-1) - (-0n19514)) >> 2

    Evaluate expression: 4878 = 0000130e

     

     

    So, there are 4878 threads waiting for this particular critical section. Seeing the number of above threads in the waiting list, it makes lot of sense.

    In RED above, we see the offender (0x00001634) who is not ready to leave the critical section.

     

    So now, at least we know which thread is causing the server to be unresponsive. Let’s go further and find out what’s happening in that particular thread.

     

    kd> !thread 85125020

    THREAD 85125020  Cid 03d8.1634  Teb: 7ffdb000 Win32Thread: e1754ea8 WAIT: (Unknown) UserMode Non-Alertable

        8512520c  Semaphore Limit 0x1

    Waiting for reply to LPC MessageId 001e295b:

    Current LPC port e38140c8

    IRP List:

        850eff68: (0006,0094) Flags: 00000800  Mdl: 00000000

    Not impersonating

    DeviceMap                 e1001150

    Owning Process            85e021e0       Image:         svchost.exe

    Attached Process          N/A            Image:         N/A

    Wait Start TickCount      3470768        Ticks: 179914 (0:00:46:51.156)

    Context Switch Count      93                 LargeStack

    UserTime                  00:00:00.000

    KernelTime                00:00:00.015

    Win32 Start Address RPCRT4!ThreadStartRoutine (0x77c7b0f5)

    Start Address kernel32!BaseThreadStartThunk (0x77e617ec)

    Stack Init f5424000 Current f5423c20 Base f5424000 Limit f5421000 Call 0

    Priority 10 BasePriority 8 PriorityDecrement 0

    Kernel stack not resident.

    ChildEBP RetAddr  Args to Child             

    f5423c38 808202b6 85125020 851250c8 00000500 nt!KiSwapContext+0x25 (FPO: [Uses EBP] [0,0,4])

    f5423c50 8081fb6e 8512520c 851251e0 85125020 nt!KiSwapThread+0x83 (FPO: [0,2,0])

    f5423c94 80915355 8512520c 00000011 80912701 nt!KeWaitForSingleObject+0x2e0 (FPO: [5,12,4])

    f5423d50 8082350b 00000a40 000e9dd8 000de270 nt!NtRequestWaitReplyPort+0x776 (FPO: [SEH])

    f5423d50 7c8285ec 00000a40 000e9dd8 000de270 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f5423d64)

    00a0c580 7c82787b 77e77342 00000a40 000e9dd8 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    00a0c584 77e77342 00000a40 000e9dd8 000de270 ntdll!NtRequestWaitReplyPort+0xc (FPO: [3,0,0])

    00a0c5bc 77e77214 01b1f978 7c840936 00a0c604 kernel32!BasepQueryModuleData+0x121 (FPO: [5,6,4])

    00a0c5dc 7c84086e 01b1f978 7c840936 00a0c604 kernel32!BaseQueryModuleData+0x3b (FPO: [5,1,0])

    00a0c82c 7c8343b7 0009ec88 0009ec88 00000000 ntdll!LdrpCheckAppDatabase+0x9c (FPO: [1,138,4])

    00a0c840 7c8333ba 0009ec88 0016ecb0 00a0ce98 ntdll!LdrpCheckNXCompatibility+0x27 (FPO: [1,1,4])

    00a0c93c 7c834311 00000000 00000000 777964d0 ntdll!LdrpRunInitializeRoutines+0x20f (FPO: [SEH])

    00a0cbd0 7c834065 00000000 0016ecb0 00a0ce98 ntdll!LdrpLoadDll+0x3cd (FPO: [SEH])

    00a0ce4c 77e41bf3 0016ecb0 00a0ce98 00a0ce78 ntdll!LdrLoadDll+0x198 (FPO: [SEH])

    00a0ceb4 776adc89 00a0cf30 00000000 00000008 kernel32!LoadLibraryExW+0x1b2 (FPO: [SEH])

    00a0ced8 776adb95 00a0cf30 00a0cefc 00a0cf00 ole32!CClassCache::CDllPathEntry::LoadDll+0xb0 (FPO: [4,1,4])

    00a0cf08 776ad2da 00a0cf30 00a0d208 00a0cf28 ole32!CClassCache::CDllPathEntry::Create_rl+0x37 (FPO: [3,3,4])

    00a0d154 776ad1fd 00000001 00a0d208 00a0d184 ole32!CClassCache::CClassEntry::CreateDllClassEntry_rl+0xd8 (FPO: [3,139,4])

    00a0d19c 776ad112 00000001 00105624 00a0d1c8 ole32!CClassCache::GetClassObjectActivator+0x1fe (FPO: [3,10,4])

    00a0d1cc 776a1b7f 00a0d208 00a0dce8 00a0d7a0 ole32!CClassCache::GetClassObject+0x24 (FPO: [1,1,0])

    00a0d240 776a19b5 77794964 00a0d7a0 00a0dce8 ole32!CServerContextActivator::GetClassObject+0xfd (FPO: [3,23,4])

    00a0d278 776a1c4e 00a0d7a0 00a0dce8 00a0d7a0 ole32!ActivationPropertiesIn::DelegateGetClassObject+0xf3 (FPO: [2,7,0])

    00a0d298 776a1bfa 77794978 00a0d7a0 00a0dce8 ole32!CApartmentActivator::GetClassObject+0x4d (FPO: [3,2,0])

    00a0d2b0 776acb27 77794978 00000001 00000000 ole32!CProcessActivator::GCOCallback+0x2b (FPO: [6,0,4])

    00a0d2d0 776acad8 77794960 00a0d5fc 00000000 ole32!CProcessActivator::AttemptActivation+0x2c (FPO: [7,0,0])

    00a0d30c 776a1ca1 77794960 00a0d5fc 00000000 ole32!CProcessActivator::ActivateByContext+0x4f (FPO: [6,3,0])

    00a0d334 776a19b5 77794960 00a0d7a0 00a0dce8 ole32!CProcessActivator::GetClassObject+0x48 (FPO: [3,1,4])

    00a0d36c 776a1925 00a0d7a0 00a0dce8 0073006d ole32!ActivationPropertiesIn::DelegateGetClassObject+0xf3 (FPO: [2,7,0])

    00a0d5b4 776a19b5 7779487c 00000001 00a0dce8 ole32!CClientContextActivator::GetClassObject+0x88 (FPO: [3,3,0])

    00a0d5ec 776a1a07 00a0d7a0 00a0dce8 7767c9b0 ole32!ActivationPropertiesIn::DelegateGetClassObject+0xf3 (FPO: [2,7,0])

    00a0dda4 776a187e 00a0ded8 00000001 00000000 ole32!ICoGetClassObject+0x3a5 (FPO: [7,52,0])

    00a0ddd4 776a1841 00a0ded8 00000002 00000000 ole32!CComActivator::DoGetClassObject+0xe0 (FPO: [6,0,4])

    00a0ddf4 772d7218 00a0ded8 00000001 00000000 ole32!CoGetClassObject+0x20 (FPO: [5,0,0])

    00a0de60 772d65ce 01ab48d8 01aeba70 00a0de80 urlmon!CProtMgr::FindFirstCF+0x11a (FPO: [4,17,0])

    00a0de84 772c4408 01ab4870 01aeba70 00000000 urlmon!COInetSession::CreateHandler+0x2d (FPO: [6,1,0])

    00a0deec 772d6b6c 00083ff8 01aeba70 000841b0 urlmon!CTransaction::LoadHandler+0xbd (FPO: [5,15,4])

    00a0df18 772c8302 00083ff8 0000000d 00000001 urlmon!CTransaction::DispatchReport+0x203 (FPO: [8,1,4])

    00a0df44 772c874b 00083ff8 01b092e8 00000000 urlmon!CTransaction::DispatchPacket+0x31 (FPO: [2,0,4])

    00a0df64 772c6681 00083ff8 00000000 00a0e014 urlmon!CTransaction::OnINetCallback+0x92 (FPO: [2,1,0])

    00a0df9c 7739b6e3 00c60048 00000465 00000000 urlmon!TransactionWndProc+0x33 (FPO: [SEH])

    00a0dfc8 7739b874 772c664e 00c60048 00000465 USER32!InternalCallWinProc+0x28

    00a0e040 7739ba92 00000000 772c664e 00c60048 USER32!UserCallWinProcCheckWow+0x151 (FPO: [SEH])

    00a0e0a8 773a16e5 00a0e0d0 00000001 00a0e10c USER32!DispatchMessageWorker+0x327 (FPO: [SEH])

    00a0e0b8 772de54a 00a0e0d0 00084100 00000000 USER32!DispatchMessageA+0xf (FPO: [1,0,0])

    00a0e10c 772ca8f5 00000000 00000000 01ad6588 urlmon!CTransaction::CompleteOperation+0x15a (FPO: [1,15,0])

    00a0e148 772c77ee 00083ff8 01ac6930 01ad64e0 urlmon!CTransaction::Start+0x52c (FPO: [6,9,4])

    00a0e1cc 772c744f 00a0e214 01aa2b30 772c9c00 urlmon!CBinding::StartBinding+0x4d8 (FPO: [6,22,4])

    00a0f264 772c7319 01aa7230 00000000 01aa2b30 urlmon!CUrlMon::StartBinding+0x1d8 (FPO: [6,1051,4])

    00a0f29c 772eb8f5 01aa7230 01aa2b30 00000000 urlmon!CUrlMon::BindToStorage+0x67 (FPO: [5,3,4])

    00a0f2e0 7732238b 001422e0 00a0f344 001422e0 urlmon!CBaseBSCB::KickOffDownload+0x191 (FPO: [2,8,0])

    00a0f2f4 7732296f 001422e0 00a0f344 0000000c urlmon!CFileDownload::KickOffDownload+0x2d (FPO: [2,0,4])

    00a0f30c 77322ce6 00000000 00a0f344 00a0f3c4 urlmon!URLDownloadToFileW+0x51 (FPO: [5,0,0])

    00a0f444 00a0f59d 00000000 00a0f63e 00a0f632 urlmon!URLDownloadToFileA+0xd8 (FPO: [5,68,4])

    WARNING: Frame IP not in any known module. Following frames may be wrong.

    00a0f474 71bf396b 48544950 71bf3969 554a4649 0xa0f59d

     77430b8c 77e40000 00000000 00000004 0000ffff 000000b8 WS2HELP!WahOpenHandleHelper+0xf7 (FPO: [1,114,0])

     

    Now, looking at the thread, lets list down the observations and our interpretation from it:

     

    1)      So, by looking at API URLDownloadToFileW(), it seems this thread is busy downloading a file from the internet. The API and parameters are documented at  http://msdn.microsoft.com/en-us/library/ms836094.aspx Let’s find out which file are we trying to download here:

     

    kd> du 00a0f344

    00a0f344  http://10.162.13.115:31797/x

     

    kd> du 00a0f3c4

    00a0f3c4  "x.exe"

     

    So we are trying to download an executable here? It’s worth having the customer investigate this binary?

     

    2)      This thread is waiting for a reply to a LPC call issued (LPC MessageId 001e295b), so let’s find out who do we have at the server end of this LPC call:

     

    kd> !lpc message 001e295b

    Searching message 1e295b in threads ...

    Client thread 85125020 waiting a reply from 1e295b                         

    Searching thread 85125020 in port rundown queues ...

     

    Server connection port e1395030  Name: AELPort

        Handles: 1   References: 22

        Server process  : 85e021e0 (svchost.exe)

        Queue semaphore : 85c593f8

        Semaphore state 0 (0x0)

        The message queue is empty

     

    So, as you see above, the server piece of this LPC call is this process itself hence a straightforward deadlock situation. Also, please remember that critical section is process specific.

     

    3)      The stack below URLMON looks messed up so we shall have to look at the raw stack and reconstruct the correct stack. 

     

    kd> kv =00a0f8cc 0x00a0f8cc 71c49573

     

    ChildEBP RetAddr  Args to Child             

    00a0f8cc 71c49578 000c9620 000c93ac 00104100 NETAPI32!NetpwPathCanonicalize+0x76 (FPO: [6,0,4])

    00a0f8f4 74edc916 000c93ac 00104100 0000019a NETAPI32!NetpwPathCanonicalize+0x7b (FPO: [6,0,4])

    00a0f914 77c80193 000c9398 000c93ac 00104100 srvsvc!NetprPathCanonicalize+0x1c (FPO: [7,0,0])

    00a0f944 77ce33e1 74edc8fa 00a0fb28 00000007 RPCRT4!Invoke+0x30

    00a0fd44 77ce35c4 00000000 00000000 000b9fd8 RPCRT4!NdrStubCall2+0x299 (FPO: [SEH])

    00a0fd60 77c7ff7a 000b9fd8 000a80d8 000b9fd8 RPCRT4!NdrServerCall2+0x19 (FPO: [1,1,0])

    00a0fd94 77c8042d 74ed31d7 000b9fd8 00a0fe34 RPCRT4!DispatchToStubInCNoAvrf+0x38 (FPO: [SEH])

    00a0fde8 77c80353 0000001f 00000000 74ee3f0c RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x11f (FPO: [4,13,4])

    00a0fe0c 77c68e0d 000b9fd8 00000000 74ee3f0c RPCRT4!RPC_INTERFACE::DispatchToStub+0xa3 (FPO: [4,0,4])

    00a0fe40 77c68cb3 000b9f98 000002a4 000b9f60 RPCRT4!OSF_SCALL::DispatchHelper+0x149 (FPO: [0,4,4])

    00a0fe54 77c68c2b 00000000 0000001f 000c9370 RPCRT4!OSF_SCALL::DispatchRPCCall+0x10d (FPO: [2,0,4])

    00a0fe84 77c68b5e 000c9370 030002bc 0000001f RPCRT4!OSF_SCALL::ProcessReceivedPDU+0x57f (FPO: [3,5,4])

    00a0fea4 77c6e8db 00000000 000002bc 00000000 RPCRT4!OSF_SCALL::BeginRpcCall+0x194 (FPO: [2,0,4])

    00a0ff04 77c6e7b4 00000000 000c9370 000002bc RPCRT4!OSF_SCONNECTION::ProcessReceiveComplete+0x435 (FPO: [3,17,0])

    00a0ff18 77c7b799 0008cd68 0000000c 00000000 RPCRT4!ProcessConnectionServerReceivedEvent+0x21 (FPO: [7,0,0])

    00a0ff84 77c7b9b5 00a0ffac 77c8872d 0008cd68 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0x1b8 (FPO: [0,15,0])

    00a0ff8c 77c8872d 0008cd68 00000000 00000000 RPCRT4!ProcessIOEventsWrapper+0xd (FPO: [1,0,0])

    00a0ffac 77c7b110 0008b120 00a0ffec 77e64829 RPCRT4!BaseCachedThreadRoutine+0x9d (FPO: [1,2,4])

    00a0ffb8 77e64829 001262f0 00000000 00000000 RPCRT4!ThreadStartRoutine+0x1b (FPO: [1,0,0])

    00a0ffec 00000000 77c7b0f5 001262f0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

     

    Our first course of action was to install the latest and greatest netapi32.dll binary which cleared up the issue.

     

     

    I hope that helps!!

     

    Thanks and regards,

    Aman

  • Ntdebugging Blog

    Remote Debugging connecting to a Remote Stub using the Microsoft Debugging Tools for Windows

    • 3 Comments

    The Microsoft Debugging Tools for Windows provide a couple ways to create a remote debugging connection including "Connecting to a remote session" and "Connecting to a remote stub". Connecting to a remote session is the most commonly used of the two, and is all you typically need, but it does not allow the client to access debugging symbols missing on the remote.

    Jeff’s blog post here has more information on some of the intricacies of debugging a remote session. Often a remote debugging session will be done within a corporate network where all users have access to symbol information, but if we are debugging outside of the secure environment where we cannot share symbols freely, debugging becomes more difficult .

    The Microsoft debugging tools package has two server side command line applications to help us do this by connecting to a remote stub: Dbgsrv.exe for user mode debugging and Kdsrv.exe for kernel mode debugging. They provide a thin server layer for the debugger to attach with full use of local symbols and source code.

    Conventional remote debug session

    Let’s illustrate the benefits of using remote stubs by showing the problems encountered when connecting to a remote session. 

     image_thumb[1][5]

    This diagram shows the conventional remote debug session. Both the debugging host and target are running windbg, though we could use KD or CDB. The goal is to debug a buggy application called Getif.exe which runs on the debugging target. We have symbols for the application on the debugging host however for security reasons the debugging target does not have symbol access to Getif.exe. On both machines we have Windows internet symbols loaded from the http://msdl.microsoft.com/download/symbols.

     

    Here is the symbol path on the debugging target machine. Because this system is outside our secure environment it does not have symbol access for the Getif.exe application. This will be problematic for debugging.

    clip_image008[4]_thumb[1][5]

     

    Here is the symbol path on the debugging host which includes a path to the symbols for the application because it’s inside the secure location.

    clip_image010[4]_thumb[1]

     

    To create the remote debugging server on the target machine, I specify the local port to use for the connection by using the .server command:

    .server tcp:port=<localport>

    clip_image012[4]_thumb[1]

     

    Next I select “Connect to Remote Session” on the debugging host.

    clip_image014[4]_thumb[1]

     

    Then enter the port and server name:

    clip_image016[4]_thumb[1]

     

    And this is where we run into the headache! Because we don’t have Getif.exe symbol access on the machine located outside of the secure environment, the stacks for getif.exe don’t display symbolic information. It makes it hard to debug the application because we can’t dump data types or view other symbolic info.

    clip_image018[4]_thumb[1]

    If we want to debug Getif.exe with symbols, we need to copy the symbol files to the debug target machine and add the location to the target’s symbol path. In our scenario this may not be possible because symbols are often restricted to in-house use and not allowed to be copied freely.

     

     

    Dbgsrv to the rescue!!

    image_thumb[3]

    By activating Dbgsrv on the debugging target machine, the debugger on the debugging host connects with full symbol access to the user mode process on the target. The diagram above illustrates the process.

    Let's run through an example using dbgsrv on the debugging target. On the debugging target enter

     <path to debugging tools>\dbgsrv -t tcp:port=<localport>

    Dbgsrv returns to a command prompt with no output in the command window and the debug port can be any open port on the system. It’s a good idea to confirm we are listening on the port we selected by using netstat. Here’s an example -  

    clip_image020[4]_thumb[1]

    Now let’s move our focus to the debugging host computer. Instead of choosing the typical “Connect a Remote Session”, we use the option, “Connect to a Remote Stub” from the File menu.

    clip_image022[4]_thumb[1]

    Next enter the port and server name.

    clip_image024[4]_thumb[1]

    After selecting OK, we are back to the empty Command window as expected. We won’t see any text until connected to the target process:

    clip_image026[4]_thumb[1]

    Next select File-Attach to a Process option.

    clip_image028[4]_thumb[1]

    It is important to note the “Attach to Process” dialog shows the list of processes running on the debugging target machine.

    clip_image030[4]_thumb[1]

    Now let’s attach to the process. The title bar confirms that we connected to the process on the kdsrv target:

    clip_image032[4]_thumb[1]

    Here you can see the benefit of connecting with a remote stub. The symbols for getif.exe are working for the application running on a machine located outside of the secure environment.

    clip_image034[4]_thumb[1]

     

     

    But what if I need to Kernel debug?

    You can do the same thing with Kernel debugging using Kdsrv.

    image_thumb[5]

    If we need to attach to a kernel debugger we can use kdsrv. Much like the previous example we enter the command:

    kdsrv -t tcp:port=<localport>

    We can see that we are listening on the local port we selected:

    clip_image036[4]_thumb[1]

    We must now start our debugger on the debugging host machine with a command line that includes the information needed to connect to the KdSrv host. As of Windbg 6.10.3.233 we cannot attach to the kdsrv remote stub from the Windbg graphical interface and then make the kernel connection to the debugging target .

    clip_image038[4]_thumb[1]

    Here is the string I used in the graphic above-

    windbg -k kdsrv:server=@{tcp:server=rodolpho,port=5004},trans=@{com:pipe,port=\\mimi\pipe\2008target}

    Note that we are debugging to a named pipe on a kernel debug target running under Microsoft Hyper-V! This is a great way to debug without the hassle of setting up a second machine and attaching with a null modem cable.

    If we wanted to attach to COM1 with a null-modem cable we would use:

    windbg -k kdsrv:server=@{tcp:server=rodolpho,port=5004},trans=@{com:port=com1,baud=11500}

    For more information on client and server syntax see http://msdn.microsoft.com/en-us/library/cc266440.aspx.

    After Windbg starts the command window will look just like if we attached with a local debugger. The title bar will confirm that we are attaching through the debug server:

    clip_image040[4]_thumb[1]

    We can also change context to the Getif.exe process after connecting to the remote stub. When we look at the application stack we have symbols for our application:

    clip_image042[4]_thumb[1]

    After we have our remote stub connection, we can then share our client as a remote session, but we will always handle the symbols on the remote stub client.

    Once you setup a remote stub a couple of times it’s as simple as setting up a remote session, and can make a remote debug go a lot smoother. For more information check out the debugging tools help, available outside of windbg as debugger.chm in the debugger directory, as well as http://msdn.microsoft.com/en-us/library/cc266434.aspx

    Security Note: These examples use TCP/IP or named pipes and are not password protected. The debugger remotes will allow SSL and Secure pipe connections with password and certificate authentication if you require greater security. See the online help or this link for more information.

  • Ntdebugging Blog

    Microsoft Windows Dynamic Cache Service

    • 14 Comments

    Excessive cached read I/O is a growing problem.  For over one year we have been working on this problem with several companies.  You can read more about it in the original blog post:

    http://blogs.msdn.com/ntdebugging/archive/2007/11/27/too-much-cache.aspx

    On 32 bit systems, the kernel could address at most 2GB of virtual memory.  This address range is shared and divided up for the many resources that the system needs; one of which is the System File Cache's working set.  On 32 bit systems the theoretical limit is almost 1GB for the cache’s working set; however, when a page is removed from the working set it will end up on the standby page list.  Therefore the system can cache more than the 1 GB limit if there is available memory. The working set; however, is just limited to what can be allocated within the Kernel's 2GB virtual address range.  Since most modern systems have more than 1 GB of physical RAM, the System File Cache's working set's size on a 32 bit system typically isn't a problem.

    With 64 bit systems, the kernel virtual address space is very large and is typically larger than physical RAM on most systems.  On these systems the System File Cache's working set can be very large and is typically about equal to the size of physical RAM.  If applications or file sharing performs a lot of sustained cached read I/O, the System File Cache's working set can grow to take over all of physical RAM.  If this happens, then process working sets are paged out and everyone starts fighting for physical pages and performance suffers.

    The only way to mitigate this problem is to use the provided APIs of GetSystemFileCacheSize() and SetSystemFileCacheSize().  The previous blog post "Too Much Cache" contains sample code and a compiled utility that can be used to manually set the System File Cache's working set size.

    The provided APIs, while offering one mitigation strategy, has a couple of limitations:

    1) There is no conflict resolution between multiple applications.  If you have two applications trying to set the System File Cache's working set size, the last one to call SetSystemFileCacheSize() will win.  There is no centralized control of the System File Cache's working set size.

    2) There is no guidance on what to set the System File Cache's working set size to.  There is no one size fits all solution.  A high cache working set size is good for file servers, but bad for large memory application and a low working set size could hurt everyone's I/O performance.  It is essentially up to 3rd party developers or IT administrators to determine what is best for their server and often times, the limits are determined by a best guesstimate backed by some testing.

    We fully understand that while we provide one way to mitigate this problem, the solution is not ideal. We spent a considerable amount of time reviewing and testing other options.  The problem is that there are so many varied scenarios on how users and applications rely on the System File Cache.  Some strategies worked well for the majority of usage scenarios, but ended up negatively impacting others.  We could not release any code change that would knowingly hurt several applications.

    We also investigated changing some memory manager architecture and algorithms to address these issues with a more elegant solution; however the necessary code changes are too extensive.  We are experimenting with these changes in Windows 7 and there is no way that we could back port them to the current operating systems.  If we did, we would be changing the underlying infrastructure that everyone has been accustomed to.  Such a change would require stress tests of all applications that run on Windows. The test matrix and the chance of regression are far too large.

    So that brings us back to the only provided solution - use the provided APIs.  While this isn't an ideal solution, it does work, but with the limitations mentioned above.  In order to help address these limitations, I've updated the SetCache utility to the Microsoft Windows Dynamic Cache Service.  While this service does not completely address the limitations above, it does provide some additional relief.

    The Microsoft Windows Dynamic Cache Service uses the provided APIs and centralizes the management of the System File Cache's working set size.  With this service, you can define a list of processes that you want to prioritize over the System File Cache by monitoring the working set sizes of your defined processes and back off the System File Cache's working set size accordingly.  It is always running in the background monitoring and dynamically adjusting the System File Cache's working set size.  The service provides you with many options such as adding additional slack space for each process' working set or to back off during a low memory event.

    Please note that this service is experimental and includes sample source code and a compiled binary.  Anyone is free to re-use this code in their own solution.  Please note that you may experience some performance side effects while using this service as it cannot possibly address all usage scenarios. There may be some edge usage scenarios that are negatively impacted.  The service only attempts to improve the situation given the current limitations.  Please report any bugs or observations here to this blog post.  While we may not be able to fix every usage problem, we will try to offer a best effort support.

    Side Effects may include:

    Cache page churn - If the System File Cache's working set is too low and there is sustained cached read I/O, the memory manager may not be able to properly age pages.  When forced to remove some pages in order to make room for new cache pages, the memory manager may inadvertently remove the wrong pages.  This could result in cached page churn and decreased disk performance for all applications.

    Version 1.0.0 - Initial Release

    http://www.microsoft.com/downloads/details.aspx?FamilyID=e24ade0a-5efe-43c8-b9c3-5d0ecb2f39af

    NOTE: The memory management algorithms in Windows 7 and Windows Server 2008 R2 operating systems were updated to address many file caching problems found in previous versions of Windows. There are only certain unique situations when you need to implement the Dynamic Cache service on computers that are running Windows 7 or Windows Server 2008 R2.  For more information on how to determine if you are experiencing this issue and how to resolve it, please see the More Information section of Microsoft Knowledge Base article 976618 - You experience performance issues in applications and services when the system file cache consumes most of the physical RAM.

Page 15 of 24 (237 items) «1314151617»