NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

re: NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

Tal Rosen — Sun, 25 May 2008 11:21:44 GMT

Run application verifier, and enable "handle tracing". Then attach a debugger to the application, and reproduce the crash.

App Verifier should raise a debug break when attempting to use the invalid handle (you can use Adplus to auto generate a full dump).

If the handle is NULL or some strange value, it’s probably a memory corruption (needs memory breakpoints monitoring).

If the handle value is reasonable, it’s probably a closed handle (closed by some other thread?), you can watch the handle trace call stacks (using !htrace) to figure out who closed our handle.

re: NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

molotov — Fri, 23 May 2008 04:15:01 GMT

Scenario 1:

I suppose, I would have the customer download and install Application Verifier and the Debugging Tools for Windows. I'd instruct the customer to set up the application in verifier for Basics -> Handles, and then have the customer launch the app in WinDbg and reproduce the problem. If the problem was a "double close" (an attempt to close a handle twice), verifier would cause a break at the time of the second attempt to close the handle. If the problem was that a function was presented with an invalid handle (say, WriteFile being passed a file handle that is not valid, or has already been closed), verifier would cause a break at the time of the attempted use of that handle. Both situations can be caused by faulty / non-existent thread synchronization. Another cause may be not setting the HANDLE to NULL after closing it, and then attempting to use the same HANDLE variable. Memory corruption could also cause the value held by a HANDLE to be invalid. If memory corruption is suspected, I'd suggest also turning on full page heap (Basics -> Heaps) and running the app in WinDbg. Anyways, once there is a verifier stop, the best case would probably be that the stack immediately points out what's wrong, but exploration with !htrace may be desirable once the HANDLE value is determined by looking at the parameters to whatever function is on the stack at the time of the stop, that is attempting to use an invalid handle (!htrace [handle value]).

Scenario 2:

I guess I'd do about the same as in scenario 1, for starters. Once I have more info, I might see if I can replicate the problem in the debugger, and set breakpoints at various locations if looking at the code didn't turn up anything immediately obvious. If the code just happened to have trace logging that covered activities relating to usage of the handles, all the better...

re: NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

JM — Wed, 21 May 2008 13:21:08 GMT

This isn't a puzzler, it's a quiz, so it's not half as stimulating. This would be an excellent interview question for those people you want to hire, though. :-)

That said, 1. procmon, 2. appverifier/windbg. Details are left as an exercise.

re: NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

Infro — Wed, 21 May 2008 10:54:04 GMT

Most Likely Options:

#1. Invalid Handle Returned but not checked for. (I do this all the bleedin time, ahh bad coding practices when you make too many programs for yourself.)

#2. Handle Closed and then used. (Hurm, bad thread safety or very bad programming habits, (worse than mine!))

#3. Handle is of wrong type. (2nd most common for myself, WaitForMultipileObjects with a screw up somewhere)

#4. Handle being released by some other method. (Uncommon, and probably hardest to track down)

Anyhow, we want to be able to narrow it down from these first off. To do that we know that we have unhandled exceptions and app crashing, first we'll back track.

So #1 We'll want a record of events leading to the crash, as its an invalid handle error lets first use API Spy which is going to be very very handy in this situtation.

We'll want to have it Looking for Creation, Loading, and Deletion of

Process, Thread, Section/FileMapping, File, Event, Semaphore, Mutex, Timer, Pipe, Socket, GDI Objects, Resource Objects

Also we'll want a Dump File with the crashed file Open.

#1 Use API Spy and Mark the following to be Recorded, then open the application that crashes. (Assuming that the handle will get closed by this application, it can provide us with details as to when it was closed)

DuplicateHandle

WaitForMultipleObjects (Check Array Size)

CloseHandle

CloseObject

CloseWindow

DeleteObject

DeleteTimerQueue

DestroyWindow

TerminateJobOject

TerminateProcess

TerminateThread

Save the Log from API Spy and send it to .....

#2 Use WinDBG or Adplus to get a dump, Adplus -crash or use WinDBG, open the process that is to crash, wait until it crashes, then run .dump /ma filename, and send the file to ......

#3 Get better description of situtation (Does it crash when you do something specific? Random Crash? Repeatable? If Something Specific try and get more information)

Now if I'm going to be activly debugging this with source in WinDBG I'd let it run until it crashes, while looking through/DebugOutputting the source for the CloseHandle-TerminateThread functions. DebugOutput would have a stack trace at each one of these functions as to see where/how this function was called, or at least get an idea. Once it crashes, try and find as much info about the handle that was invalid and how it came to be invalid (!Handle, !Object, Find if its valid, was valid, what not, what released it/modified the value).

As this is getting a bit long I think that its pretty good, anyhow, looking forward to some more puzzlers, so long as I can find time ;).

re: NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

Skywing — Tue, 20 May 2008 02:47:27 GMT

I'd enable handle tracing in gflags and have the process be run under the debugger. Take a dump when the invalid handle exception comes up.

Then, use !htrace to find when the handle was opened/closed, and analyze from there along with the stack trace of where the exception occurred.