Hi, welcome back,
I've been dealing these days with an issue about a Custom Authentication Package which was crashing LSASS.exe process even before we had the opportunity to log on the machine. So, how can I debug the package/LSASS process with my favorite debugger, WinDbg (Debugging Tools for Windows), to know what's going on there?
To make things easier I use two machines: a Windows XP machine running on Virtual PC and my Windows Vista machine. Virtual PC makes my life much easier as I can recover the machine very easily if I "break" it. It also makes kernel debugging easier as I don't need any cables to connect the machines involved. The target package/LSASS process will run on WinXP, and WinDbg will run on Vista.
1) Let's configure LSASS on WinXP to load and use the problematic package (i.e. MyPackage.dll):
1.1) Copy MyPackage.dll to %SystemRoot%\system32.
1.2) Add "MyPackage" to the Authentication Packages list in the \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa registry key.
2) Let's configure WinXP to run LSASS under a user mode debugger like NTSD.exe (Debugging Tools for Windows). We won't be able to use this debugger directly, but we will be able to use it through WinDbg working as kernel mode debugger on Vista.
2.a) Set Debugger value to REG_SZ "ntsd -d" in HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\lsass.exe.
2.b) Instead of manipulating the registry directly, we may use GFlags.exe (Debugging Tools for Windows) to make these changes: go to Image File tab, enter "LSASS.exe" in Image field, press TAB to refresh, click on Debugger checkbox and enter "ntsd -d" as the debugger.
After these changes NTSD will break in LSASS execution just when LSASS is about to start (thx to -d parameter), so we can debug it from the very beginning of its execution. Now we'll need a kernel debugger (WinDbg in our case) to control NTSD remotely.
3) Let's configure WinXP to start on debug mode and be able to attach a kernel debugger to it:
3.1) Run msconfig.exe (System Configuration utility), go to BOOT.INI tab, click on Advanced Options... button, and select /DEBUG, select & set /DEBUGPORT= to COM1:, and select & set /BAUDRATE= to 115200.
3.2) On Virtual PC go to Edit > Settings menu option, select COM1 and set it to a Named pipe called i.e. \\.\pipe\machinename.
Now our WinXP is prepared to be debugged. We can restart it now. We will see that WinXP hangs before showing the logon screen. NTSD has broken in LSASS, and because LSASS is not executing, WinXP won't start either. NTSD is now waiting for our debugging commands, but we will need WinDbg to send those commands to it.
4) Let's configure WinDbg on Vista to do kernel debugging on WinXP:
On WinDbg go to File > Kernel Debug... menu option, select COM tab, check Pipe checkbox, set Baud Rate to 115200 and Port to \\.\pipe\machinename (the same we used on Virtual PC). When we click OK WinDbg will connect to WinXP as its kernel debugger.
When WinDbg connects to WinXP and WinLogon is about to start, WinDbg shows the prompt of NTSD ("Input>"). We will now be able to send commands to NTSD, like bp to set a breakpoint, etc.
Please take a look to "Controlling the User-Mode Debugger from the Kernel Debugger" topic on WinDbg's help for more info on how to control NTSD from WinDbg.
We will now be able to do user mode (thx to NTSD) and kernel mode debugging at the same time with WinDbg.
Note: I had some issues to load the right symbols (.pdb files), because I couldn't set .sympath on NTSD to c:\symbols, for instance. So I did the following: I took a dump of LSASS process with .dump command, I opened the dump and loaded all the symbols I needed (i.e. lsasrv.pdb, lsass.pdb, kernel32.pdb, ntdll.pdb and MyPackage.pdb). I copied all those symbols to System32 folder on XP, and after that NTSD was able to find them without setting its .sympath.
Note: We may debug CSRSS and WinLogon processes following the same principles. Take a look to "Debugging CSRSS with NTSD" and "Debugging WinLogon with NTSD" topics on WinDbg's help.
I hope this helps.
Good information, thanks Alex. I ran through this because I am debugging LSASS for unexpected access violations. The interesting thing I found was that when I run the .dump command, the dumps are saved to the local machine being debugged. Is there a way to save those to the debugger machine instead?
That makes sense. The debugger in the problematic machine is acting as a debugging server. The debugger that is controlling the session from a distance is called the debugging client. The client just transmits commands to the server and shows the resulting output. The server is actually doing the debugging. So a .dump command will take effect in the server.
Nice article. However, I am trying to debug LSASS.exe on Vista within a VMware machine from either: a) XP in another VMWare machine (on same physical box - i.e. VM machine to VM machine) - or b) Vista as the VM host machine. I cannot get either scenario to work. I have tried at least five different options (that I won't list here) of setting virtual serial\com\pipe ports, configuring bcdedit, etc. with no success. Do you have any suggestions? I also found some documentation that says ntsd -d won't work for remote debugging - is this applicable to this scenario? From ntsd /?:
"-d cannot be used with debugger remoting"
I am using debug tools v. 6.9.0003.113
Thanks in advance!
Never mind! I realized that ntsd -d has nothing to do with remote debugging. I was confusing the remote/server concept since I was debugging across machines. But the virtual serial port is considered local. Doh! All is good now.
I really enjoy reading your posts.
This one is helpful for lsass, but could you perhaps expand on debugging csrss? Windbg help doesn't do the trick. Can't pop input> with NTSD instructions. :(
Hi Kj, did you follow "Debugging CSRSS" topic on WinDbg then? Did you enable CSRSS Debugging as the article says? If that doesn't work, I don't know what could be going on...