NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

NTDebugging Puzzler 0x00000006: Invalid Handle - can you handle it?

  • Comments 5

Hi NTDebuggers, this week’s puzzler just so happens to match its number: 0x000000006 = ERROR_INVALID_HANDLE.   That said, let me give you a scenario and the challenge will be to provide the best action plan to isolate the problem.  This should include an explanation of what types of code problems cause invalid handles.  

 

Scenario 1 :  You have a customer or client that is getting invalid handle errors.  This is causing unhandled exceptions and the application is crashing.   What action plan do you give the customer?  In scenario 1, you don’t have access to the remote machine.  Your customer will execute any action plan you give them, gather the data, and provide you with the results. 

 

Scenario 2: You have full access to the process and you can debug it live on your own personal machine; you even have source access.  What do you do?

 

Good luck and happy debugging.

 


 

[Update: our answer. Posted 5/27/2008]

 

Hi NTDebuggers, if you’ve been debugging for a while I’m sure you’ve run into an Invalid Handle Error 6 (aka ERROR_INVALID_HANDLE or “The handle is invalid”).  If you see it in your debugger output it indicates you have tried to pass an invalid handle to a call that is either expecting a different handle type or a the value for the handle its self is not valid.   To understand this error in more detail you have to understand what handles are and where they live in the operating system.  Each process has a handle table in the kernel.  Think of this as an array or table of objects in kernel mode.   The handle values are actually multiples of four.  Review http://blogs.msdn.com/oldnewthing/archive/2005/01/21/358109.aspx to see for what the bottom 3 bits are used for.  For this illustration we will be numbering them 1,2,3 etc.

 

 

When you open a file you will get back a simple pointer-sized value that in the handle table points to a file object.  In this case let’s say you opened a file and you received a handle value of 2.  If you envision the handle table you might see something like this.

 

------------- USER ADDRESS SPACE FOR YOUR PROCESS -------------

 

Your handle for your open file is 2

 

------------- KERNEL ADDRESS SPACE FOR YOUR PROCESS -------------

 

Handle table

 

Handle [1]=Pointer to ->Reg

Handle [2]=Pointer to ->File      <<<<<< Your handle is 2  It points to a file object

Handle [3]=Pointer to ->File

Handle [4]=Pointer to ->Reg

Handle [5]=Pointer to ->Process

Handle [6]=Pointer to ->Thread

Handle [7]=Pointer to ->Thread

Handle [8]=Pointer to ->File

Handle [9]=Pointer to ->File

 

In the debugger, you can see the handle table is referenced from the EPROCESS object for each process.  In the !process output it is called the ObjectTable.

 

0: kd> !process 85df8da0 

PROCESS 85df8da0  SessionId: 0  Cid: 0114    Peb: 7ffda000  ParentCid: 0268

    DirBase: 0acc07e0  ObjectTable: e4b6a4c8  HandleCount:  49

 

 

Before we get into troubleshooting let’s look at a couple of scenarios for bad handles.

 

In the first bad handle scenario, let’s say your application closes handle 2 using CloseHandle(), however your application keeps the value 2 in a variable somewhere. In this scenario we will assume no other code has come along and opened a new object that is occupying element 2 of the handle table.  If you go to make a file I/O call using handle 2 the object manager will catch this and your app will get an invalid handle exception.  This is somewhat harmless but your app my crash if you don’t catch the exception and your I/O will obviously not complete.

 

In the second scenario we’ll say that your file I/O code closes the handle, and holds on to that handle value.  Meanwhile another part of your application opens a registry key or some other object that is not a file object.  Now your file I/O code goes to use handle 2 again.  The problem is it’s not a handle to a file now.  When the registry code opened the registry key element 2 was available so we now have a registry object in that location.  If you go to do a file I/O on handle 2 you will get an invalid handle message.

 

Now in our final scenario, we close a file handle in our file I/O code and keep the handle around.  In this case though we have different code that is also doing file I/O, It opens a file and gets handle value 2.  Now you have big trouble because if the first file I/O code now writes to handle 2, and it should not be because it closed it!  This handle now belongs to a different file.  This means you have code basically doing a write into an unknown region for that code’s context and file format.   This will result in file corruption and no invalid handle error.

 

So this really comes down to best practice.  If you’re writing code and close a handle NULL IT OUT!  That way you guarantee it will not be reused accidently at some other point.

 

CloseHandle(MyHandle);

MyHandle = NULL;

 

 

On to debugging:  So how do you track this down? For the remote scenario you could give your customer the following action plan:

 

 

1.       Start gflags and under the image file tab, enter your application name in the image edit control.

 

2.       Check Create User mode stack trace database.

 

3.       Start the process under windbg.   “WinDBG processname.exe”

 

4.       In windbg Run !htrace -enable

 

5.       Do a sxe ch.  This will cause windbg to break on an invalid handle exception.

 

6.       Run sx to confirm  “ch - Invalid handle – break” should be set.

 

7.       Enter g for go.

 

8.       Let the process run until you get an invalid handle message.  The debugger should break in.

 

9.       Now all you have to do is run !htrace

 

 

Htrace will dump out all the call stacks for each handle that was opened or closed.

You need to take the invalid handle value and search backward to see where it was last opened, and where it was closed before that.  It’s likely that the close before the last open is your culprit code path.  Make sure that suspect close nulls that handle value out so it does not reuse it. 

 

In the live debug scenario following the same procedure.  In this case you have the luxury of going back and setting a breakpoint in the offending call stack that freed the handle before the reuse.  You can then figure out why it was not zeroed out.

 

Good luck and happy debugging.

 

 





Leave a Comment
  • Please add 6 and 1 and type the answer here:
  • Post
  • I'd enable handle tracing in gflags and have the process be run under the debugger.  Take a dump when the invalid handle exception comes up.

    Then, use !htrace to find when the handle was opened/closed, and analyze from there along with the stack trace of where the exception occurred.

  • Most Likely Options:

    #1. Invalid Handle Returned but not checked for. (I do this all the bleedin time, ahh bad coding practices when you make too many programs for yourself.)

    #2. Handle Closed and then used. (Hurm, bad thread safety or very bad programming habits, (worse than mine!))

    #3. Handle is of wrong type. (2nd most common for myself, WaitForMultipileObjects with a screw up somewhere)

    #4. Handle being released by some other method. (Uncommon, and probably hardest to track down)

    Anyhow, we want to be able to narrow it down from these first off. To do that we know that we have unhandled exceptions and app crashing, first we'll back track.

    So #1 We'll want a record of events leading to the crash, as its an invalid handle error lets first use API Spy which is going to be very very handy in this situtation.

    We'll want to have it Looking for Creation, Loading, and Deletion of

    Process, Thread, Section/FileMapping, File, Event, Semaphore, Mutex, Timer, Pipe, Socket, GDI Objects, Resource Objects

    Also we'll want a Dump File with the crashed file Open.

    #1 Use API Spy and Mark the following to be Recorded, then open the application that crashes. (Assuming that the handle will get closed by this application, it can provide us with details as to when it was closed)

    DuplicateHandle

    WaitForMultipleObjects (Check Array Size)

    CloseHandle

    CloseObject

    CloseWindow

    DeleteObject

    DeleteTimerQueue

    DestroyWindow

    TerminateJobOject

    TerminateProcess

    TerminateThread

    Save the Log from API Spy and send it to .....

    #2 Use WinDBG or Adplus to get a dump, Adplus -crash or use WinDBG, open the process that is to crash, wait until it crashes, then run .dump /ma filename, and send the file to ......

    #3 Get better description of situtation (Does it crash when you do something specific? Random Crash? Repeatable? If Something Specific try and get more information)

    Now if I'm going to be activly debugging this with source in WinDBG I'd let it run until it crashes, while looking through/DebugOutputting the source for the CloseHandle-TerminateThread functions. DebugOutput would have a stack trace at each one of these functions as to see where/how this function was called, or at least get an idea. Once it crashes, try and find as much info about the handle that was invalid and how it came to be invalid (!Handle, !Object, Find if its valid, was valid, what not, what released it/modified the value).

    As this is getting a bit long I think that its pretty good, anyhow, looking forward to some more puzzlers, so long as I can find time ;).

  • This isn't a puzzler, it's a quiz, so it's not half as stimulating. This would be an excellent interview question for those people you want to hire, though. :-)

    That said, 1. procmon, 2. appverifier/windbg. Details are left as an exercise.

  • Scenario 1:

    I suppose, I would have the customer download and install Application Verifier and the Debugging Tools for Windows.  I'd instruct the customer to set up the application in verifier for Basics -> Handles, and then have the customer launch the app in WinDbg and reproduce the problem.  If the problem was a "double close" (an attempt to close a handle twice), verifier would cause a break at the time of the second attempt to close the handle.  If the problem was that a function was presented with an invalid handle (say, WriteFile being passed a file handle that is not valid, or has already been closed), verifier would cause a break at the time of the attempted use of that handle.  Both situations can be caused by faulty / non-existent thread synchronization.  Another cause may be not setting the HANDLE to NULL after closing it, and then attempting to use the same HANDLE variable. Memory corruption could also cause the value held by a HANDLE to be invalid.  If memory corruption is suspected, I'd suggest also turning on full page heap (Basics -> Heaps) and running the app in WinDbg.  Anyways, once there is a verifier stop, the best case would probably be that the stack immediately points out what's wrong, but exploration with !htrace may be desirable once the HANDLE value is determined by looking at the parameters to whatever function is on the stack at the time of the stop, that is attempting to use an invalid handle (!htrace [handle value]).

    Scenario 2:

    I guess I'd do about the same as in scenario 1, for starters.  Once I have more info, I might see if I can replicate the problem in the debugger, and set breakpoints at various locations if looking at the code didn't turn up anything immediately obvious.  If the code just happened to have trace logging that covered activities relating to usage of the handles, all the better...

  • Run application verifier, and enable "handle tracing". Then attach a debugger to the application, and reproduce the crash.

    App Verifier should raise a debug break when attempting to use the invalid handle (you can use Adplus to auto generate a full dump).

    If the handle is NULL or some strange value, it’s probably a memory corruption (needs memory breakpoints monitoring).

    If the handle value is reasonable, it’s probably a closed handle (closed by some other thread?), you can watch the handle trace call stacks (using !htrace) to figure out who closed our handle.

Page 1 of 1 (5 items)