Welcome to MSDN Blogs Sign in | Join | Help

What possible use are those extra bits in kernel handles? Part 1: Sentinels

Kernel handles are always a multiple of four; the bottom two bits are available for applications to use. But why would an application need those bits anyway?

The short answer is extending the handle namespace. The long answer will take a few days to play out. (This series was written in response to Igor Levicki being unable to imagine "how this can save anything (in terms of performance)". Then again, who said that it had anything to do with performance? Actually, I'm surprised that my dear readers weren't familiar with the techniques described in this series. Perhaps I shouldn't have written this series and merely replied, "If you can't think of how this could be useful, then you are not my target audience." On the other hand, reader Aaargh! believes that whoever thought to make the bottom two bits of handles available to applications should receive an asswhooping.)

But we'll start with a warm-up. If you need some sentinel values for a HANDLE, you need to make sure your chosen sentinel value will never conflict with a valid HANDLE value. If you decide that your sentinel value is something like

// code in italics is wrong
#define DEBUGWINDOW_HANDLE ((HANDLE)0x1234)

then your program is going to start acting really strange if the kernel ever gave you handle value of 0x1234. Knowing that kernel handles are always multiples of four means that you can choose a value that isn't a multiple of four and use it as your sentinel value.

#define DEBUGWINDOW_HANDLE ((HANDLE)0x1233)

Since 0x1233 is not a multiple of four, you can rest assured that no actual kernel handle will have this value, and you can write your logging function like this:

void LogOutput(HANDLE hOutput, LPCVOID pv, DWORD cb)
{
 if (hOutput == NULL) {
   // logging disabled
 } else if (hOutput == DEBUGWINDOW_HANDLE) {
  AddToDebugWindow(pv, cb);
 } else {
  DWORD cbWritten;
  WriteFile(hOutput, pv, cb, NULL, &cbWritten);
 }
}

Since you can't WriteFile to a window handle, your logging function has to do something special if somebody decided that their output should go to the debug window. But if they chose to log to a normal kernel object (a file, the console, a serial port, whatever) then you can just write the data to that kernel object.

You've already seen this before; you just didn't realize it. The special values for INVALID_HANDLE_VALUE and kernel pseudo-handles such as GetCurrentProcess are not multiples of four for exactly this reason.

Now, sure, you could have defined your own LogHandle type and have all the logging go through that type instead of just logging to HANDLEs:

class LogHandle {
public:
  static LogHandle *GetDebugLogHandle();
  BOOL IsDebugWindow();
  HANDLE GetKernelHandle();
  static LogHandle *CreateFromKernelHandle(HANDLE KernelHandle);
  ~LogHandle() { }

private:
  LogHandle(BOOL IsDebugWindow, HANDLE KernelHandle);
  static LogHandle DebugWindow;

  BOOL IsLogToDebugWindow;
  HANDLE RegularKernelHandle;
};

Throughout, your program would use pointers to LogHandles instead of actual handles, using functions like these to convert between them:

// Does not take ownership of the handle
LogHandle::LogHandle(BOOL IsDebugWindow, HANDLE KernelHandle)
    : IsLogToDebugWindow(IsDebugWindow)
    , RegularKernelHandle(KernelHandle)
{
}

LogHandle LogHandle::DebugWindow(TRUE, NULL);

LogHandle* LogHandle::GetDebugWindowLogHandle()
{
  return &DebugWindow;
}

BOOL LogHandle::IsDebugWindow()
{
  return IsLogToDebugWindow;
}

HANDLE LogHandle::GetKernelHandle()
{
  assert(!IsDebugWindow());
  return RegularKernelHandle;
}

LogHandle *LogHandle::CreateFromKernelHandle(HANDLE KernelHandle)
{
  return new LogHandle(FALSE, KernelHandle);
}

Or you could make everybody pass two parameters instead of one. For example, a class that went

class SomeObject {
public:
  SomeObject(int SomeParameter, BOOL SomeParameter,
             HANDLE LogHandle);
...
private:
  ...
  HANDLE LogHandle; // log to this handle
};

would have to change to

class SomeObject {
public:
  SomeObject(int SomeParameter, BOOL SomeParameter,
             BOOL LogToDebugWindow, HANDLE LogHandle);
...
private:
  ...
  BOOL LogToDebugWindow; // if TRUE, log to window
  HANDLE LogHandle; // if not logging to window, then log to here
};

Either way is an awful lot of work just to define a sentinel value. But still, at least you can avoid the need for a sentinel value by just passing more parameters. But sometimes that option isn't available. We'll look at that next time.

Published Wednesday, August 27, 2008 7:00 AM by oldnewthing
Filed under:

Comments

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 10:31 AM by steven

Hmmm... 1234 isn't a multiple of 4, unless you mean 0x1234, of course.

[Thanks. Fixed. -Raymond]

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 10:31 AM by keithmo

I wonder if "next time" will concern the disgusting "set the low bit of the event handle in an OVERLAPPED structure to suppress the I/O completion indication" hack...

[Um, that topic was the inspiration for this one. Try clicking that first link. -Raymond]

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 11:10 AM by John

This whole thing just seems like too much of a hack for me.  I much prefer adding a second parameter to a function.  My personal feeling is that an opaque handle should mean one thing and one thing only.  If you want to associate some extra information with a handle perhaps you should implement SetHandleLong() or something.

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 11:34 AM by Koro

But one must still be careful. "Console" pseudo-handles returned by GetStdHandle have some lower bits set, that WriteFile uses to determine if it should redirect the call to WriteConsoleA.

All this because console handle are not real handles.

# heh...

Wednesday, August 27, 2008 7:03 PM by Igor Levicki

I am glad that my "ignorance" on the subject has inspired Raymond to write an educational article.

As for my imagination shortage -- still there is nothing here to convince me that such implementation is actually faster. Why not write two test cases along with some timing code and post them in one of the future blog posts?

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 7:10 PM by Dean Harding

"still there is nothing here to convince me that such implementation is actually faster"

Quote from Raymond: "Then again, who said that it had anything to do with performance?"

# Boost Optional

Wednesday, August 27, 2008 7:59 PM by Bruno Martínez

You can use boost::optional to do this cleanly.

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 8:22 PM by Maurits

Of course, just because you /can/ doesn't mean you /should/.

// works, but ew

switch (h) {

// technically NULL % 4 == 0 but, hey, it's special

case NULL: ...; break;

case INVALID_HANDLE_VALUE: ...; break;

// probably need to unwrap this

case GetCurrentProcess(): ...; break;

case YOUR_FAVORITE_NON_MULTIPLE_OF_FOUR_SENTINEL1: ...; break;

case YOUR_FAVORITE_NON_MULTIPLE_OF_FOUR_SENTINEL2: ...; break;

...

case YOUR_FAVORITE_NON_MULTIPLE_OF_FOUR_SENTINEL/n/: ...; break;

default: assert((h % 4) == 0); WriteFile(h, ...);

}

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 8:31 PM by MadQ

I've taken advantage of those lower two bits in a different way when I was writing a little debugging tool. It needed to keep track of process IDs (which aren't handles, but also have the lower two buts set to zero), and it was important to be able to look up process handles by process ID very fast. It used a hash table with the simplest hash function ever: right-shift the process ID by two.

Nitpickery: "But if the lower two bits weren't always zero, you wouldn't need a hash function at all!"

Yeah, but then I wouldn't have had a possibly mildly interesting anecdote to tell.

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Wednesday, August 27, 2008 10:35 PM by Erzengel

I can see what Raymond is getting at, but I personally would have done:

class LogHandle

{

public:

virtual ~LogHandle(){}

virtual void Write(LPCVOID ToWrite, DWORD Len)=0;

};

class LogFileHandle : public LogHandle

{

HANDLE regularHandle;

public:

LogFileHandle(HANDLE);

~LogFileHandle();

virtual void Write(LPCVOID ToWrite, DWORD Len)

{

DWORD WrittenLen = 0;

WriteFile(regularHandle, ToWrite, Len, 0, &WrittenLen);

}

};

class LogDebugHandle : public LogHandle

{

public:

virtual void Write(LPCVOID ToWrite, DWORD Len)

{

AddToDebugWindow(ToWrite, Len);

}

};

__declspec(dllexport) LogHandle* NewLogHandle(HANDLE WriteTo)

{

return new LogFileHandle(WriteTo);

}

__declspec(dllexport) void FreeHandle(LogHandle* ToDelete)

{

delete ToDelete;

}

__declspec(dllexport) LogHandle* GetDebugHandle()

{

//Yes, I could use a singleton, but I'd need a

//way to allow the "free" function to recognize

//the static variable, and that's beyond

//the scope of this example.

return new LogDebugHandle();

}

__declspec(dllexport) void LogOutput(LogHandle* LogTo, LPCVOID ToLog, DWORD Len)

{

LogTo->Write(ToLog, Len);

}

Yes, this may be slower (function pointers rather than if branches), but really, do you need to be fast when outputting to a file, debug window, etc? Wouldn't the actual writing be orders of magnitude slower than a function pointer? Plus, this is extensible. What if I want to output via WM_COPYDATA? Or across a network? I could create COPYDATA_HANDLE and add an if, but how would I know what window I'm sending to? Using a class allows me a great deal of flexibility.

So I honestly, sincerely want to get your opinions: how would using a HANDLE sentinal be better than the above?

[It's a lot easier if you're retrofitting an existing code base that originally used HANDLEs. -Raymond]

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Thursday, August 28, 2008 12:19 AM by Anon

For some reason this reminds of that Mistakes Demotivator with the caption "It could be that the purpose of your life is only to serve as a warning to others"

# re: What possible use are those extra bits in kernel handles? Part 1: Sentinels

Thursday, August 28, 2008 3:26 AM by Me

John wrote on Wednesday, August 27, 2008 11:10 AM:

> My personal feeling is that an opaque handle should mean one thing and one thing only.

But if this were always followed, we would not know the FILENOTFOUND value for a boolean enum:

http://thedailywtf.com/Articles/What_Is_Truth_0x3f_.aspx

But I think we all know this same from some other circumstance, don't we?

# What possible use are those extra bits in kernel handles? Part 3: New object types

Sunday, August 31, 2008 11:05 PM by The Old New Thing

Extending the namespace to cover new object types.

New Comments to this post are disabled
 
Page view tracker