Written by Jeff Dailey.
When it comes to live debugging, the breakpoint is king. Oftentimes solving a very complex problem in a production environment involves doing a local, non-production debug one of my own test machines. I’ll typically debug the process or code in question to get a good idea of how it works and what data I need to collect when I break in. This normally involves some reverse engineering and doing code review of the process or module in question. We always want to minimize impact on production environments when it’s time to do the live debug. I call it designing the perfect breakpoint.
There are several types of breakpoints we typically use.
The most common breakpoint is bp, and setting it in windbg is simple: bp 0XADDRESS. Note that when you break into a user mode process it halts execution of the process. When you break into the kernel debugger it halts the entire machine. This being the case, we like to figure out what commands we would issue once the breakpoint hits. You can then set the breakpoint to automatically issue the commands at the time of break in.
The syntax for running commands upon hitting a breakpoint is straightforward: bp 0XADDRESS “comand1;command2;…”
As an example - bp 0xMYADDRESS “kv;dd g_myGlobal;g” This command will break on 0xMYADDRESS, dump the callstack via kv; dump memory as DWORDs starting at the g_MyGlobal address; and then continue execution with the g command.
Another cool trick is to have a breakpoint enable or disable another breakpoint or disable itself. Say you have a breakpoint that is hit far too often to break in EVERY single time that code executes. However, once a particular state or a condition is met, you may want every call stack associated with that break point. All you need do is set a breakpoint that is waiting on your key conditional code to execute. This breakpoint can then enable the more expensive breakpoint.
Here is the example:
bp ntdll!RtlAllocateHeap “kv;g” << This will be breakpoint 1, set on a very common call.
bd 1 << Disable break point 1, as we don’t want to see its output until breakpoint 2 executes.
bp notepad!OpenFile “be 1;g” << This will enable breakpoint 1 when NotePad!OpenFile is executed.
The most expensive type of break point is a ba - Break on access. This break point allows you to break on access to either memory or I/O ports. There are several variations : ba w (break on write), ba r (break on read), ba e (break on execute), and ba i (for I/O ports).
Let’s use break on write for our sample. The syntax is simple. ba w4 0xNNNNnnnn. This means break on access, access type WRITE, with a width of 4 bytes, followed by the target address. However, because the processor has to watch for access to this memory, it generally slows down execution time. This might be a breakpoint that you enable conditionally using BE during another condition check.
Ba type breakpoints are very handy when you have a resource changing and you don’t know what code path is touching it. Imagine you had a corrupted critical section. It’s not likely one of the critical section APIs is causing the corruption. It’s more likely that something overwrote that memory. That being the case you could simply do a ba w4 on the critical section address followed by a “kv;g” and let the process run. You will see all the places the critical section APIs touches the critical section; however, you will also see any offender that writes over that memory.
Refer to the debugger.chm file in the debugger tools directory for details on the BA variations.
The conditional break
You can also do conditional breaks or checks during the breakpoint execution in the debugger by using the j command in the section following the breakpoint: bp Address “j EXPRESSION ‘IfTrueCommands1;IfTruecommand2’ ; ‘ElseCommand1;ElseCommand2’”
The following is an example of checking a call to a file open operation. The breakpoint is set on the instruction following a CALL.
bp notepad!SomeNotePadCodeOpeningAFile+0x132 "j @eax >0 '.echo all is good file opened ok;gc';'!gle;kv;lsa eip;gc'“
If eax (the default register used to store function return values) is nonzero, we echo ‘all is good, file opened ok’ to the command output window. If eax is zero (indicating failure), we display the last error with !gle, dump the call stack via kv, and list the source at the eip address using LSA EIP.
Good luck and Happy debugging.
Often you need to use conditional breakpoints
(see later example on this blog)
However for live kernel debugging the conditional evaluation is done on the debug workstation (running the debugger).
This can be slow for frequently hit breakpoints
(especially over serial, probably ok over fastwire).
One technique I have devised to speed this up involves on the fly patching of the code on the target system, eg jump to a spare area, do a conditional jump, then have both paths jmp back and continue in the original code. Can them set a debug breakpoint on the condition path of interest.
Another, more advanced techinque (that I might write up), actually patches the debugger trap code on the target and does the condition code there, especially for coniditional break on memory accesses.
Update to my previous 'patching code on the target' technique
Should also work on x64 as patchguard is disabled if running under the debugger.
ba i, BTW, requires that the processor have the Debugging Extension (DE).
If you have a Intel Pentium or AMD K5 or later, you should have it.