LPC part 2 Kernel Debugger Extensions

LPC part 2 Kernel Debugger Extensions

Rate This
  • Comments 1

Hello my name is Roy, I’m an EE on the Microsoft global escalation services / CPR team.   This blog is a follow on to my first LPC blog.  We will be discussing debugger extensions that allow you to look at LPC related issues.

 

Disclaimer: The purpose of this blog is to illustrate debugging techniques with LPC. Please do not rely on this information as documentation for the APIs when writing your own code. As always, this is subject to change in the future.  

 

LPC Kernel Debugger Extensions

 

 

Command

Description

!lpc                    

Display the list and description of all the !lpc commands

!lpc message [MessageId]

Display the message with a given ID and all related information by attempting to match the given Message ID to the EHTREAD->LpcReceivedMessageId and to the ETHREAD->LpcReplyMessageId of all threads in the system.

If the MessageId parameter is omitted then it attempts to display all the outstanding messages in the system by searching for the tag ‘LpcM’ in the pools.

!lpc port [PortAddress] 

Displays port information. If a server connection port address is specified then only information about that port is displayed. If either a client or server communication port is specified it prints information about the specified communication port, the peer communication port and the server connection port.

If the PortAddress parameter is omitted then it attempts to walk the list of all objects of type “Port” and “WaitablePort” and display them. Note that for this feature to work the GFlags option “+otl” i.e. “Maintain a list of objects for each type” must be enabled.

!lpc scan PortAddress   

Displays port information. It attempts to walk the list of all objects of type “Port” and “WaitablePort” and display the one matching the specified port address. Note that for this feature to work the GFlags option “+otl” i.e. “Maintain a list of objects for each type” must be enabled.

!lpc thread [ThreadAddr]

If  ThreadAddr is specified it walks the list of threads in the ETHREAD-> LpcReplyChain to locate the list head i.e. a “Port” or “WaitablePort” object on which the thread is waiting for a reply.

If the ThreadAddr parameter is omitted then it attempts to find all LPC server threads by looking for threads with a non-NULL EHTREAD->LpcReceivedMessageId and all client threads by looking for threads with a non-NULL ETHREAD->LpcReplyMessageId and displays them.

!lpc PoolSearch

Toggles a setting that controls whether the “lpc message” command will search for LPC message tag (‘LpcM’) in the kernel pools or not.

 

 

 

LPC Kernel Debugger Extension Usage

 

 

LPC Connection Port information from call stack

 

On the call stack of any client or server threads blocked on LPC data transfer or LPC connection there will be a frame containing either one of the functions  NtRequestWaitReplyPort() or NtReplyWaitReceivePortEx() . The first parameter to either one of these functions is the handle to the port they are blocked on.

 

kd> !thread 810de2a8 

THREAD 810de2a8  Cid 01dc.01f4  Teb: 7ffde000 Win32Thread: 00000000 WAIT: (WrLpcReceive) UserMode Non-Alertable

    81131188  Semaphore Limit 0x7fffffff

    810de398  NotificationTimer

Not impersonating

DeviceMap                 e196c460

Owning Process            810ddda0       Image:         rpclpcs.exe

Wait Start TickCount      64555          Ticks: 402 (0:00:00:04.025)

Context Switch Count      2            

UserTime                  00:00:00.000

KernelTime                00:00:00.000

Win32 Start Address 0x77e76bf0

Start Address 0x7c810856

Stack Init f8e28000 Current f8e27c4c Base f8e28000 Limit f8e25000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 0

ChildEBP RetAddr  Args to Child             

f8e27c64 804dc6a6 810de318 810de2a8 804dc6f2 nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])

f8e27c70 804dc6f2 e1084108 8055a540 e1084108 nt!KiSwapThread+0x46 (FPO: [0,0,0])

f8e27c98 8056a50a 00000001 00000010 00000001 nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])

f8e27d48 804df06b 000007c4 002bff70 00000000 nt!NtReplyWaitReceivePortEx+0x3dc (FPO: [Non-Fpo])

f8e27d48 7c90eb94 000007c4 002bff70 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f8e27d64)

002bff80 00000000 00000000 00000000 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

 

kd> !handle 7c4 3 810ddda0       

processor number 0, process 810ddda0

PROCESS 810ddda0  SessionId: 0  Cid: 01dc    Peb: 7ffd9000  ParentCid: 01b4

    DirBase: 058cd000  ObjectTable: e1a13278  HandleCount:  18.

    Image: rpclpcs.exe

 

Handle table at e107d000 with 18 Entries in use

07c4: Object: e1084108  GrantedAccess: 001f0001 Entry: e107df88

Object: e1084108  Type: (812b5c80) Port

    ObjectHeader: e10840f0 (old version)

        HandleCount: 1  PointerCount: 4

        Directory Object: e14c72c8  Name: rpclpc

 

kd> !lpc port e1084108

 

Server connection port e1084108  Name: rpclpc

    Handles: 1   References: 4

    Server process  : 810ddda0 (rpclpcs.exe)

    Queue semaphore : 81131188

    Semaphore state 0 (0x0)

    The message queue is empty

    The LpcDataInfoChainHead queue is empty

 

 

LPC Messages that are waiting to be picked up by the server thread

 

When a message is queued to the server connection port and the server thread has not yet not been signaled to pull out the message the ‘!lpc port <Port>” displays the following output that includes the message.

 

kd> !lpc port e10d8388

 

Client communication port 0xe10d8388

    Handles: 1   References: 2

    The LpcDataInfoChainHead queue is empty

        Connected port: 0xe106dc00      Server connection port: 0xe15512e0

 

Server communication port 0xe106dc00

    Handles: 1   References: 1

    The LpcDataInfoChainHead queue is empty

 

Server connection port e15512e0  Name: rpclpc

    Handles: 1   References: 9

    Server process  : ffbbebe8 (rpclpcs.exe)

    Queue semaphore : 81250848

    Semaphore state 0 (0x0)

        Messages in queue:

        0000 e10513f8 - Busy  Id=000048d8  From: 05f4.0108  Context=80020000  [e15512f0 . e15512f0]

                   Length=0054003c  Type=00000001 (LPC_REQUEST)

                   Data: 00000001 00000241 00000000 00000000 f877bc04 804ec10e

    The message queue contains 1 messages

    The LpcDataInfoChainHead queue is empty

 

 

LPC Messages being processed by the server

 

When an LPC message has been queued by the client, dequeued by the sever and being worked upon by the server both the client and the server thread have the message ID associated with them

 

kd> !lpc message 16fa

Searching message 16fa in threads ...

    Server thread 810de2a8 is working on message 16fa                        

Client thread 810dc930 waiting a reply from 16fa                         

Searching thread 810dc930 in port rundown queues ...

 

Server connection port e1084108  Name: rpclpc

    Handles: 1   References: 3

    Server process  : 810ddda0 (rpclpcs.exe)

    Queue semaphore : 81131188

    Semaphore state 0 (0x0)

    The message queue is empty

    The LpcDataInfoChainHead queue is empty

Done.                                             

 

kd> !thread 810de2a8

THREAD 810de2a8  Cid 01dc.01f4  Teb: 7ffde000 Win32Thread: 00000000 RUNNING on processor 0

Not impersonating

DeviceMap                 e196c460

Owning Process            810ddda0       Image:         rpclpcs.exe

Wait Start TickCount      65732          Ticks: 10 (0:00:00:00.100)

Context Switch Count      4            

UserTime                  00:00:00.000

KernelTime                00:00:00.010

Win32 Start Address 0x000016fa

LPC Server thread working on message Id 16fa

Start Address kernel32!BaseThreadStartThunk (0x7c810856)

Stack Init f8e28000 Current f8e27728 Base f8e28000 Limit f8e25000 Call 0

Priority 9 BasePriority 8 PriorityDecrement 0 DecrementCount 0

ChildEBP RetAddr  Args to Child             

f8e27d64 7c90eb94 badb0d00 002bfe24 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ f8e27d64)

002bfe18 7c90e399 77e76703 000007c4 002bff70 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

002bfe1c 77e76703 000007c4 002bff70 00000000 ntdll!NtReplyWaitReceivePortEx+0xc (FPO: [5,0,0])

002bff80 77e76c22 002bffa8 77e76a3b 00084540 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0xf4 (FPO: [Non-Fpo])

002bff88 77e76a3b 00084540 7c90ee18 0006fb10 RPCRT4!RecvLotsaCallsWrapper+0xd (FPO: [Non-Fpo])

002bffa8 77e76c0a 00083f20 002bffec 7c80b50b RPCRT4!BaseCachedThreadRoutine+0x79 (FPO: [Non-Fpo])

002bffb4 7c80b50b 00084718 7c90ee18 0006fb10 RPCRT4!ThreadStartRoutine+0x1a (FPO: [Non-Fpo])

002bffec 00000000 77e76bf0 00084718 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

 

kd> !thread 810dc930

THREAD 810dc930  Cid 01e4.01ec  Teb: 7ffde000 Win32Thread: 00000000 READY

Waiting for reply to LPC MessageId 000016fa:

Pending LPC Reply Message:

    e16b8538: [e16b8538,e16b8538]

Not impersonating

DeviceMap                 e196c460

Owning Process            810e9860       Image:         rpclpcc.exe

Wait Start TickCount      65731          Ticks: 11 (0:00:00:00.110)

Context Switch Count      26            

UserTime                  00:00:00.000

KernelTime                00:00:00.020

Win32 Start Address rpclpcs!pre_c_init (0x01001a91)

Start Address kernel32!BaseProcessStartThunk (0x7c810867)

Stack Init fbe0e000 Current fbe0dc28 Base fbe0e000 Limit fbe0b000 Call 0

Priority 8 BasePriority 8 PriorityDecrement 0 DecrementCount 16

ChildEBP RetAddr  Args to Child             

fbe0dc40 804ea3a4 00000000 810dc930 e107f388 nt!KiUnlockDispatcherDatabase+0x77 (FPO: [Uses EBP] [0,0,4])

fbe0dc58 80586255 81131188 00000001 00000001 nt!KeReleaseSemaphore+0x70 (FPO: [Non-Fpo])

fbe0dd10 80598c58 000847b0 0006f8c0 0006f8c8 nt!NtSecureConnectPort+0x635 (FPO: [Non-Fpo])

fbe0dd3c 804df06b 000847b0 0006f8c0 0006f8c8 nt!NtConnectPort+0x24 (FPO: [Non-Fpo])

fbe0dd3c 7c90eb94 000847b0 0006f8c0 0006f8c8 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame @ fbe0dd64)

0006f8d0 7c93040b ffffffff 00000000 7c859e48 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

0006f9b0 7c8107fd 00000000 00084718 7c90e642 ntdll!DbgPrint+0x1b (FPO: [Non-Fpo])

0006fe68 77ea11a8 00000022 00084540 00000000 kernel32!CreateRemoteThread+0x284 (FPO: [Non-Fpo])

000847a8 00000000 00000001 00000000 00000000 RPCRT4!LRPC_ADDRESS::CompleteListen+0x84 (FPO: [Non-Fpo])

 

 

Finding all threads in the system involved in LPC communication

 

To get a consolidated list of all client threads in the system waiting on LPC replies and a list of server threads working on received LPC requests use the “!lpc threads” command without any parameters

 

0: kd> !lpc thread

Searching message 0 in threads ...

Client thread 89a4cdb0 waiting a reply from 15f6                         

    Server thread 898c6a50 is working on message 63f09                        

    Server thread 89671020 is working on message 63f75                        

    Server thread 89904a48 is working on message 63f10                        

    Server thread 88d6bdb0 is working on message 3d6b1                        

    Server thread 89973db0 is working on message 63f32                         

    Server thread 896d32b8 is working on message 3454                        

    Server thread 8995d020 is working on message 63f76                        

    Server thread 8960f020 is working on message 63f77                         

    Server thread 898cd350 is working on message 61bd9                        

    Server thread 8900edb0 is working on message 3832f                        

    Server thread 8900fbf0 is working on message 33c2e                        

    Server thread 88d539a8 is working on message 3343f                        

Client thread 89be4020 waiting a reply from 6077e                         

Client thread 89012990 waiting a reply from 39239                         

    Server thread 89012990 is working on message 39232                        

Client thread 89531020 waiting a reply from 3923d                         

    Server thread 89531020 is working on message 39236                        

Client thread 88d13b40 waiting a reply from 3c96e                            

 

 

 

Quick Tips

 

8 No thread in the system is doing a NtListenPort( ) does that mean no LPC connection  port is open for connections?

 

!stacks 2 NtListenPort does not return any thread because NtListenPort( ) calls NtRequestWaitReplyPort( ) to wait for an incoming LPC_CONNECTION_REQUEST message. Instead of calling NtListenPort( ) for listening for incoming connections most LPC connection threads directly call NtRequestWaitReplyPort( ).

 

8 !lpc message does not show any message?

 

!poolfind LpcM 1 should do the trick.

 

 

0: kd> !lpc message

 

Scanning large pool allocation table for Tag: LpcM (8974e000 : 8976e000)

 

Searching Paged pool (e1000000 : f7000000) for Tag: LpcM

 

 

0: kd> !poolfind LpcM 1

 

Scanning large pool allocation table for Tag: LpcM (8974e000 : 8976e000)

 

 

Searching Paged pool (e1000000 : f7000000) for Tag: LpcM

 

e1061000 size:  168 previous size:    0  (Allocated) LpcM

e1090758 size:  168 previous size:    8  (Allocated) LpcM

e10a7a40 size:  168 previous size:    8  (Allocated) LpcM

e1188468 size:    8 previous size:   20  (Free)      LpcM

e1261000 size:  168 previous size:    0  (Allocated) LpcM

e1297428 size:  168 previous size:    8  (Allocated) LpcM

 

Once you have the pool blocks listed with the “LpcM” tag dump the ones that are “Allocated”. On a 32-bit machine add 8bytes to the pool address listed on the left most column. On a 64-bit machine add 16bytes i.e. 0x10. Dump the message as below to retrieve the message id.

 

0: kd> dt nt!_LPCP_MESSAGE e1061000+8

   +0x000 Entry            : _LIST_ENTRY [ 0xe295fa80 - 0xe29279e8 ]

   +0x000 FreeEntry        : _SINGLE_LIST_ENTRY

   +0x004 Reserved0        : 0xe29279e8

   +0x008 SenderPort       : 0xe2327020

   +0x00c RepliedToThread  : (null)

   +0x010 PortContext      : 0x8021000d

   +0x018 Request          : _PORT_MESSAGE

 

0: kd> dt nt!_LPCP_MESSAGE e1061000+8 Request.

   +0x018 Request  :

      +0x000 u1       : __unnamed

      +0x004 u2       : __unnamed

      +0x008 ClientId : _CLIENT_ID

      +0x008 DoNotUseThisField : 9.8460604703041575e-311

      +0x010 MessageId : 0x4617f

      +0x014 ClientViewSize : 0

      +0x014 CallbackId : 0

 

0: kd> !lpc message 0x4617f

Searching message 4617f in threads ...

    Server thread 88f91c08 is working on message 4617f                        

Client thread 88f0f5f8 waiting a reply from 4617f                         

Searching thread 88f0f5f8 in port rundown queues ...

 

Server communication port 0xe2300a38

    Handles: 1   References: 1

    The LpcDataInfoChainHead queue is empty

        Connected port: 0xe2327020      Server connection port: 0xe13734b0

 

Client communication port 0xe2327020

    Handles: 1   References: 9

    The LpcDataInfoChainHead queue is empty

 

Server connection port e13734b0  Name: epmapper

    Handles: 1   References: 93

    Server process  : 896e1450 (svchost1.exe)

    Queue semaphore : 8995ac10

    Semaphore state 0 (0x0)

    The message queue is empty

 

 

8   The number of server threads shown by !lpc thread processing requests is much more than the number of client threads waiting for reply.

 

!lpc thread interprets stale data in ETHREAD structure as valid LPC information. An indication of the stale data is the ETHREAD.LpcExitThreadCalled will be set. This field is only set when the thread is exiting.

 

0: kd> !lpc thread

Searching message 0 in threads ...

Client thread 89a4cdb0 waiting a reply from 15f6                         

    Server thread 898c6a50 is working on message 63f09                        

    Server thread 89671020 is working on message 63f75                         

    Server thread 89904a48 is working on message 63f10                        

    Server thread 88d6bdb0 is working on message 3d6b1                        

    Server thread 89973db0 is working on message 63f32                        

    Server thread 896d32b8 is working on message 3454                        

    Server thread 8995d020 is working on message 63f76                        

    Server thread 8960f020 is working on message 63f77                        

    Server thread 898cd350 is working on message 61bd9                        

    Server thread 8900edb0 is working on message 3832f                        

    Server thread 8900fbf0 is working on message 33c2e                        

    Server thread 88d539a8 is working on message 3343f     

 

0: kd> !lpc message 3454

Searching message 3454 in threads ...

    Server thread 896d32b8 is working on message 3454                        

Done. 

 

 

0: kd> dt nt!_ETHREAD 896d32b8 Lpc.

   +0x1c0 LpcReplyChain :  [ 0x896d3478 - 0x896d3478 ]

      +0x000 Flink : 0x896d3478 _LIST_ENTRY [ 0x896d3478 - 0x896d3478 ]

      +0x004 Blink : 0x896d3478 _LIST_ENTRY [ 0x896d3478 - 0x896d3478 ]

   +0x1ec LpcReplySemaphore :

      +0x000 Header : _DISPATCHER_HEADER

      +0x010 Limit : 1

   +0x200 LpcReplyMessage :

   +0x200 LpcWaitingOnPort :

   +0x220 LpcReceivedMessageId : 0x3454

   +0x234 LpcReplyMessageId : 0

   +0x248 LpcReceivedMsgIdValid : 0x1 ''

   +0x248 LpcExitThreadCalled : 0x1 ''

 

8 Finding all client threads in the system waiting for LPC reply.

 

!lpc thread

Leave a Comment
  • Please add 8 and 1 and type the answer here:
  • Post