Holy cow, I wrote a book!
Some people have noticed that if you load a DLL with the
you sometimes get strange behavior if you then pass that HINSTANCE
to a dialog box function.
The problem here is that since the bottom 16 bits of a proper
HINSTANCE are always zero, different components have "borrowed"
those bits for different purposes.
The kernel uses the bottom bit to distinguish modules loaded
by having been mapped into memory as sections (i.e., loaded
normally) from those who have been mapped as one giant block
(loaded as a datafile).
It needs to know this so that the various resource-management
functions such as
the FindResource function
know how to interpret the data in order to locate the
resource in question.
Although everybody now knows that the HINSTANCE is the base
address of the DLL,
in principle, it is an opaque value
(and in the 16-bit world, the value was indeed opaque).
Meanwhile, the window manager has its own problems.
In order to support 16-bit applications running seamlessly
on the desktop
(not in a virtual machine, as discussed earlier),
as well as thunking between 16-bit and 32-bit code,
it needs to accept both 32-bit HINSTANCE values
as well as 16-bit HINSTANCE values.
memory allocation granularity is 64KB,
the window manager knows that valid 32-bit HINSTANCEs
have zero in the bottom 16 bits,
whereas 16-bit HINSTANCE values are nonzero there.
Perhaps you see the conflict now.
If you pass the instance handle of a DLL loaded as a datafile,
the kernel will set the bottom bit as a signal to itself to
locate its resources in the flat datafile manner rather than
in the mapped DLL manner.
But the window manager sees that the bottom 16 bits are not all
zero and assumes that it has been given a 16-bit HINSTANCE value.
Amazingly, this doesn't cause a problem most of the time
because the things that need to be handled differently
between 32-bit and 16-bit HINSTANCEs are relatively minor.
The one that is most likely to bite you is the dialog instance data
In 16-bit Windows, a dialog box came with its own data segment,
which was used as the local data segment for controls hosted by that
Most controls didn't need a lot of storage in the local data segment,
so the issue of where it came from wasn't really important.
The big exception was edit controls, since they can contain
multiple kilobytes of text.
a dozen kilobytes of text may very well not
fit in the application's data segment.
Therefore, creating a new data segment gave the edit controls
on the dialog a new 64KB block of memory to store their data in.
Programs were expected to extract the data from the edit control
via mechanisms such as
the GetWindowText function
and store the result someplace that had the capacity
to handle it (outside the cramped local data segment).
In order to maintain compatibility with 16-bit programs who
are expecting this behavior to continue, the window manager,
when it sees a 16-bit HINSTANCE, dutifully creates a 16-bit
data segment in which to store the data for the edit controls,
using a helper function provided by the 16-bit emulation layer.
But if you aren't really a 16-bit program, then the
16-bit emulation layer is not active, and consequently
it never got a chance to tell the window manager how to create
one of these compatibility segments.
The solution is to add the DS_LOCALEDIT style to your dialog
This flag means "Do not create a dialog box data segment;
just keep using the data segment of the caller."
Therefore, when your LOAD_LIBRARY_AS_DATAFILE
is mistaken for a 16-bit dialog template,
the dialog manager won't try to create a dialog box data segment and
therefore won't call that function that doesn't exist.
I believe this issue has been resolved in Windows XP SP 2.
The window manager uses a different mechanism to detect that it is
being asked to create a dialog box on behalf of a 16-bit program
and is no longer faked out by the faux-HINSTANCEs produced by
the LoadLibraryEx function.