Welcome to MSDN Blogs Sign in | Join | Help

The NT DLL Loader: reentrancy - play along at home!

Anyone care to hazard a guess about what happens if you have the following code in your DllMain()?  Ignore the leak and the lack of error checking; focus on the what the loader's behavior has to be...

BOOL WINAPI DllMain(HINSTANCE hInstance, DWORD fdwReason, LPVOID lpvReserved) {
    switch (fdwReason) {
    case DLL_PROCESS_ATTACH:
        SOME_FUNCTION_PTR_T pfn = (SOME_FUNCTION_PTR_T) GetProcAddress(LoadLibraryW(L"SomeOther.DLL"), "SomeFunction");
        if (pfn != NULL) (*pfn)();
        break;
    }
    return TRUE;
}

The answer is actually relatively straightforward (the tricky bit is what is GetProcAddress()'s implied contract).  I'll do exposition tomorrow.  For extra credit, consider what the effect of the leak and lack of error checking is on the loader's internal data tables...

Published Tuesday, June 21, 2005 11:06 PM by mgrier

Comments

# re: The NT DLL Loader: reentrancy - play along at home!

Wednesday, June 22, 2005 4:05 AM by Graham Harper
My guess would be that the loader deduces a list of dependencies to load for "SomeOther.DLL" which aren't already imported in the process and aren't already scheduled to be loaded in the loaders list.

Interestingly if the loader encounters an import in "SomeOther.DLL" that is already in the list to be loaded but has yet to be visited by the loader you could have all sorts of fun 'n' games when calling (*pfn)() (or even before if "SomeOther.DLL"'s initialization section assumes it to be present).

Am I right in thinking that there will be no deadlock induced by the attempted reacquisition of the loader lock? Simply because it is the same thread acquiring the critical section. (That’s not to say that a deadlock would not happen by some shenanigans in the initialization of "SomeOther.DLL" or one of its dependents)

Presumably failing to free the library means "SomeOther.DLL" and all it's dependents will never be unloaded from the process until process termination as the loader will always maintain the references.

I hope I’m not way off the mark :(

# re: The NT DLL Loader: reentrancy - play along at home!

Wednesday, June 22, 2005 9:38 AM by Skywing
The thing there is that SomeOther.DLL's DllMain may not have already been executed depending on whether it was already loaded or not before our DllMain was called, and depending on the initialization list order.

So, if SomeFunction depends on something that was initialized from SomeOther.DLL's DllMain then it may break.

Additionally, if SomeFunction tries to load another DLL, the same problem may occur.

The leak will cause the SomeOther.DLL to remain pinned in memory (unless something else has tries to unload it too many times).

The lack of error checking may cause a crash if pfn is null. I believe LDR wraps all DllMain calls in an SEH frame though, so it should recover and fail the DLL load if I remember correctly.

(Incidentally, LDR has some hacks to support broken DllMains that do stupid things like mess up the stack pointer or change some of the nonvolatile registers...).

# re: The NT DLL Loader: reentrancy - play along at home!

Wednesday, June 22, 2005 2:20 PM by mgrier
You're very close. Imagine it worked the way you describe and someone wrote the code above. What bug would they file and how would the loader developer in the early 80s address it? Remember that at that time the NT/win32 community was small and well contained and so people felt that they could always trust their clients.

Hint: in the normal case, DLL_PROCESS_ATTACH would be issued to SomeOther.DLL. But can you just call it and no other initializers?

# re: The NT DLL Loader: reentrancy - play along at home!

Wednesday, June 22, 2005 9:39 PM by Norman Diamond
Sorry for going off on a tangent, but I'm trying to figure out this:

> GetProcAddress(LoadLibraryW(L"SomeOther.DLL"), "SomeFunction")

Regardless of whether the compilation environment is ANSI (multibyte) or Unicode (wide), you're calling the Unicode version of LoadLibrary with a wide string, fine. Then you're calling either the ANSI or Unicode version of GetProcAddress but with a multibyte string "SomeFunction" instead of wrapping it in _T(), so it looks rather unsafe. And you're doing these right next to each other in the same line of code, so I wonder why.

Wondering if you had some reason to specify the Unicode version of LoadLibraryW instead of just LoadLibrary and wrapping its argument in _T(), I looked in MSDN. The MSDN page is oddly worded. It says to take care in the pathname to use yen symbols instead of slashes, but then the clarification of a yen symbol in parentheses has a backslash instead of a yen symbol. The MSDN page was coded carefully to change fonts and character encoding just to put in that inappropriate backslash, since Japanese character sets don't have a single-byte backslash, they have a yen symbol which could have been put there most simply and appropriately. Very odd. Anyway, while reading the page I didn't find any difference in functionality between the Unicode version and the ANSI version, so again I wonder why you specified which version just for that call.

# re: The NT DLL Loader: reentrancy - play along at home!

Wednesday, June 22, 2005 10:00 PM by mgrier
Unless you're a library like MFC, there is zero value for building TCHAR clean. New code should be Unicode only and in that case you should not be dependent on exactly what -D switches are passed in on the compiler's command line.

# re: The NT DLL Loader: reentrancy - play along at home!

Thursday, June 23, 2005 2:27 AM by Norman Diamond
> New code should be Unicode only

That sounds like a religious argument on which I'm more or less agnostic[*], but in this particular case there's no need to decide whether or not to argue it. Look again at the code you posted:

> GetProcAddress(LoadLibraryW(L"SomeOther.DLL"), "SomeFunction")

If you're compiling in a Unicode environment then that calls GetProcAddressW and the pointer to the multibyte string "SomeFunction" will kill you.

[* The marketplace and the mountains of existing databases and existing installations aren't so agnostic, and there's no such thing as telling them to convert to Unicode only. That doesn't make me less agnostic, it just makes me pragmatic.]

# re: The NT DLL Loader: reentrancy - play along at home!

Thursday, June 23, 2005 4:49 AM by Jeroen Frijters
Norman, there is no unicode version of GetProcAddress.

# re: The NT DLL Loader: reentrancy - play along at home!

Thursday, June 23, 2005 1:43 PM by mgrier
Right - the PE export/import tables are all byte oriented strings. It wouldn't even be correct to say that they're MBCS since there's no code page anywhere and it's also incorrect to say that they're CP_ACP since CP_ACP varies user to user and obviously the DLL does not.

Exports pretty much have to be 7-bit clean. There have been discussions about saying that they're actually utf-8 encoded Unicode/UCS but this isn't compatible with what the existing tool sets do and you have to imagine that there are people dependent on the current implementation.

# re: The NT DLL Loader: reentrancy - play along at home!

Thursday, June 23, 2005 8:20 PM by Norman Diamond
Thursday, June 23, 2005 4:49 AM by Jeroen Frijters
> Norman, there is no unicode version of
> GetProcAddress.

Ouch. When an API takes a string as a parameter, I think it would be necessary to look it up to see if there are two versions. Oh boy was I wrong. Sorry.

Thursday, June 23, 2005 1:43 PM by mgrier
> Exports pretty much have to be 7-bit clean.

Yup. No other ___W functions take UTF8 format. Furthermore, since the calling thread's ANSI code page can't be assumed to match the ANSI code page that was used on the development machine where the DLL was compiled, we can't convert it to UTF8 for the same reason we can't convert it to UTF16.

Now wondering why the filename of the DLL doesn't have to be 7-bit clean...

# re: The NT DLL Loader: reentrancy - play along at home!

Thursday, June 23, 2005 10:07 PM by mgrier
If the name of the DLL in the import table isn't 7-bit clean you have similar deployment problems.

Nowadays, people tend to use COM or explicit LoadLibrary() calls more often for new components rather than static imports so the problem tends to be ignored...

# mgrier s WebLog The NT DLL Loader reentrancy play along at home | Cellulite Creams

# mgrier s WebLog The NT DLL Loader reentrancy play along at home | work from home

New Comments to this post are disabled
 
Page view tracker