In theory, there is no difference between theory and practice. But, in practice, there is. : Platform

DllMain : a horror story

oleglv — Tue, 28 Oct 2003 12:10:00 GMT

Last time I was talking about DllMain and what nasty things can occur if you misuse it. I have also mentioned that it may not be one of those "I'm always careful it can never happen to me" situations - things can get out of hand very quickly.

Keep in mind that OS loader has been evolving over time. OS creators know that not all DLLs are well-behaved and they have been trying to do their best to minimize the impact of poorly-written DllMains, however it is still more than possible to shot oneself in the foot.

Let me give you a very simple example as to how easy this can be (behavior may vary on different OS's, I'm running this on Windows XP SP1).

Consider the following:

/////////////////////////////////////////////////////////////////////

Dll2.cpp

/////////////////////////////////////////////////////////////////////

HMODULE g_Module;

TCHAR g_tclpszFileName[256];

BOOL APIENTRY DllMain( HINSTANCE hModule,

DWORD ul_reason_for_call,

LPVOID lpReserved

)

{

if ( DLL_PROCESS_ATTACH == ul_reason_for_call )

{

printf("Dll2:DllMain\r\n");

g_Module = hModule;

::GetModuleFileName( g_Module, g_tclpszFileName, 255 );

}

return TRUE;

}

extern "C" __declspec(dllexport) void WINAPIV OutputModuleInfo2(void)

{

printf("Enetering Dll2::OutputModuleInfo2\r\n");

printf("Name: %s\r\nHandle 0x%x\r\n", g_tclpszFileName, g_Module );

}

/////////////////////////////////////////////////////////////////////

// Dll1.cpp - NEVER do this

/////////////////////////////////////////////////////////////////////

typedef void (WINAPIV *LPFOUTPUTMODULEINFOFUNC) (void);

HMODULE g_Module;

TCHAR g_tclpszFileName[256];

BOOL APIENTRY DllMain( HINSTANCE hModule,

DWORD ul_reason_for_call,

LPVOID lpReserved

)

{

if ( DLL_PROCESS_ATTACH == ul_reason_for_call )

{

printf("Dll1:DllMain\r\n");

g_Module = hModule;

::GetModuleFileName( g_Module, g_tclpszFileName, 255 );

// Load Dll2 - never EVER do this

HMODULE hModule1 = ::LoadLibrary("Dll2.dll");

LPFOUTPUTMODULEINFOFUNC lpOutputModuleInfo1Func = (LPFOUTPUTMODULEINFOFUNC)::GetProcAddress( hModule1,"OutputModuleInfo2");

lpOutputModuleInfo1Func();

}

return TRUE;

}

extern "C" __declspec(dllexport) void WINAPIV OutputModuleInfo1(void)

{

printf("Enetering Dll1::OutputModuleInfo1\r\n");

printf("Name: %s\r\nHandle 0x%x\r\n", g_tclpszFileName, g_Module );

}

/////////////////////////////////////////////////////////////////////

// Main.cpp

/////////////////////////////////////////////////////////////////////

extern "C" __declspec(dllimport) void WINAPIV OutputModuleInfo1(void);

extern "C" __declspec(dllimport) void WINAPIV OutputModuleInfo2(void);

int _tmain(int argc, _TCHAR* argv[])

{

OutputModuleInfo1();

OutputModuleInfo2();

return 0;

}

What's wrong with this? Let me count the ways. On top of non-existent error-handling and the fact that we have an un-paired LoadLibrary() call, this code has a very fundamental problem. Let's just say that depending on how this code is compiled, it may

Run and produce results you expect
Run and produce results you don't expect
Blow up with AV

That's right.

Let's dig into it.

First let's see what this code is supposed to do in the first place. You'll have to bear with me here - it's after midnight and I haven't been able to come up with something brilliantly meaningful, but this will just have to do for now.

As you can see, we are dealing with two DLLs and one EXE that uses those DLLs. First DLL (inventively called Dll2) - gets its own HMODULE in DLLMain(DLL_PROCESS_ATTACH), gets its name based on that and stores them away in global variables. Exported function OutputModuleInfo2 simply prints that out using printf.

Dll1 is almost identical, except it dynamically calls into Dll2 right after collecting its own information. It's a little weird, but this is just a primitive example after all.

Main is a console appliccation that is statically bruit against export libraries produced by the build of the first two DLLs and calls both OutputModuleInfo1 and OutputModuleInfo2.

Simple enough? Let's roll.

It works! It works! Let's compile everything, but make sure that all three binaries use CRT(C/C++ runtime) dynamically (/MD compiler option) and that Dll2.lib appears before Dll1.lib in linker options pertaining to additional input libraries for our console app (something like link.exe main.obj /out:main.exe dll2.lib dll1.lib). Turns out that is important - we'll see why in a little bit. When you run the application, it outputs:

Dll2:DllMain

Dll1:DllMain

Enetering Dll2::OutputModuleInfo2

Name: c:\Temp\KillDllMain\Dll2.dll

Handle 0x10000000

Enetering Dll1::OutputModuleInfo1

Name: c:\Temp\KillDllMain\Dll1.dll

Handle 0x320000

Enetering Dll2::OutputModuleInfo2

Name: c:\Temp\KillDllMain\Dll2.dll

Handle 0x10000000

As you see, things seem to be working fine. One thing that is worth pointing out is that - as you can see from the output - DllMain for Dll2.dll was called before DllMain of Dll1.dll. Why? Well, technically, there's no explicit guarantee as to the order of these things - it is all in the loader's hands. As I mentioned before, the loader looks at static dependencies and builds a list of DllMains to be called based on that. But what happens if the order really doesn't matter? From loader's perspective, Main.exe depends on Dll1 and Dll2 and there's no reason to choose one over the other (remember, the fact that Dll1 does in fact load Dll2 is our little dirty secret).
Well, turns out that the loader seems to be preserving the order in which the imported DLLs are listed in the Imports Section of the loading executable. You can read all about the low-level details in Matt Pietrek's article, but for the purpose of this discussion let's just say that each PE file (EXE or DLL) knows what binaries it "references" - that is what external functions it imports - and that a list of those binaries, together with referenced functions is linked into its PE header. Microsoft Linker seems to build that header based on the order in which export libraries are supplied, which is why we built our app the way we did.

What happens if change that? Let's see.

Huh? But... So now let's build the same code, only this time supply export libraries in the opposite order (something like link.exe main.obj /out:main.exe dll1.lib dll2.lib). Let's run it:

Dll1:DllMain

Enetering Dll2::OutputModuleInfo2

Name:

Handle 0x0

Dll2:DllMain

Enetering Dll1::OutputModuleInfo1

Name: c:\Temp\KillDllMain\Dll1.dll

Handle 0x10000000

Enetering Dll2::OutputModuleInfo2

Name: c:\Temp\KillDllMain\Dll2.dll

Handle 0x320000

Interesting... As you see, this time DllMain from Dll1 got called first. That loaded Dll2 and its OutputModuleInfo2 got called ... before its DllMain! No wonder it printed what it did. Note that the second call into OutputModuleInfo2 went through just fine because Dll2's DllMain was called already.

So why in the world is OS loader acting so dumb? Doesn't it know we are loading Dll2? We have explicitly called LoadLibrary after all, which loaded it from disk, laid it out in memory, resolved its exports etc. Why wasn't DllMain called? If you experiment a little, you will find out that in most cases DllMain of dynamically loaded libraries will be called, even if the "illegal" LoadLibrary is used to load it. The only case that will not take place is when OS loader already "knows" about that DLL but hasn't yet called DllMain on it, which is exactly what happened here.

Main.exe statically depends on Dll2.dll, so it's already in the loader's plan. It turns out, the loader is not so willing to change its original plan created based on static dependencies. If new binaries get thrown in, the loader will stop and dutifully load them; but if the binary is in fact the "old" one - that is already in the plan - the loader will just skip it.

Why? My guess is that this works pretty well for most scenarios. The loader is still trying to be nice and compensate for our bad behavior. Once we attempt to load something it already knows about, it simply preserves its current plan - I suspect doing otherwise would cause all kinds of nasty consequences. Mind you, we are on no position to complain - we are not supposed to call LoadLibrary from DllMain in the first place. Keep in mind, these are my speculations - I'm not trying to give a precise recipe as to how the OS loader can be mistreated, I'm just saying that it can be done.

So...there you go. In this particular situation you "just" got the wrong value printed out, but you can imagine that this can easily cause a wide range of nastiness - AVs for instance. Speaking of which...

What??? How did that happen?... Let's build the whole thing again, only this time let's use static CRT (/MT or /ML compiler options). Why should it matter, right?
Now let's run it:

Dll1:DllMain

... and then... whoa...

First-chance exception at 0x77f57bd2 (ntdll.dll) in MainApp.exe: 0xC0000005: Access violation reading location 0x00000010.

But why? If you look at the stack, you will see the following:

ntdll.dll!_RtlAllocateHeap@12() + 0x24

Dll2.dll!_heap_alloc(unsigned int size=0x00000018) Line 212 C

Dll2.dll!_nh_malloc(unsigned int size=0x00000018, int nhFlag=0x00000000) Line 113 C

Dll2.dll!malloc(unsigned int size=0x00000018) Line 54 + 0xf C

Dll2.dll!_mtinitlocknum(int locknum=0x00000011) Line 251 + 0x7 C

Dll2.dll!_lock(int locknum=0x00000011) Line 311 + 0x6 C

Dll2.dll!_lock_file2(int i=0x00000001, void * s=0x00346b68) Line 267 + 0x9 C

Dll2.dll!printf(const char * format=0x0034204c, ...) Line 57 + 0xd C

Dll2.dll!OutputModuleInfo2() Line 30 + 0xa C++

> Dll1.dll!DllMain(HINSTANCE__ * hModule=0x10000000, unsigned long ul_reason_for_call=0x00000001, void * lpReserved=0x0012fd30) Line 30 + 0x5 C++

Dll1.dll!_DllMainCRTStartup(void * hDllHandle=0x10000000, unsigned long dwReason=0x00000001, void * lpreserved=0x0012fd30) Line 297 + 0xd C

So this is caused by calling "printf" from Dll2's OuputModuleInfo2, which is sort of strange. If you look some more, you will find that the CRT internal global _crtheap is NULL, which means that CRT has no heap. Why? You guessed it - static CRT allocates its heap in DllMain of the owning DLL! If our case DllMain wasn't called yet, so naturally - no heap.

Ouch. (Incidentally, this means that just about any CRT call will AV - it's awfully difficult to do anything without allocating any memory...)

Moral

OK, this is much longer than I intended... but here's the moral: be careful. OS loader is not dumb, and it is as forgiving as it gets, but sometimes it won't be there to help - simply because it has no idea what your intentions are.

OK, I think I'm officially done with the topic - I'm feeling much better now :)

DllMain and life before birth

oleglv — Sat, 25 Oct 2003 03:47:00 GMT

Preface

OS loader has always intrigued me - probably because it works behind the scenes and no-one normally bothers to understand what is that is does exactly, until strange or funny things start happening. And they do. And then we read through the documentation and we are forced to remember that there's more to loading a binary than just slapping it into process address space. In fact there's a wonderful article by Matt Pietrek that discusses those matters. I strongly encourage every person who deals with native code to go and read it - it may be quite enlightening for you - I know it was for me. When you know how things get loaded, you are less likely to forget to re-base your binary, consider early binary binding etc.

Every now and then another piece of information or a great summary on the subject comes up and I find myself mystified with the whole loader topic all over again. This time it was a very lengthy post in Chris Brumme's blog. As many people have mentioned, the post in question is very long and very dense with technical information well, what else did you expect from Chris's blog? :) Anyway, in order to absorb the topic better and in hopes of getting the whole thing out of my system I decided to write things down.

DllMain and OS loader

As we are all well aware now, things are not as easy as they seem. In fact are they ever? DllMain which used to be briefly discussed in most books on Win32 as a reasonably innocent initialization routine may now look like a vicious monster which obeys no rules and causes nasty side-effects. But let's get to the source - MSDN reference

It all starts innocently enough. The article defines DllMain as an optional entry point into a DLL, called by the system when the DLL gets attached to a process or a thread; outlines the somewhat tricky but reasonable rules that govern the calls (for instance, calls may be unmatched for a thread if it's a main thread of the process or if it was already running when LoadLibrary was called), discusses abnormal termination and then

...whoa...

Without missing a heart-beat, it carries on describing what you can do there. That is pretty startling as of itself since when should you be limited in that regard? - but as you keep reading, things just get worse. It turns out, you can do pretty much nothing at all. Calls to LoadLibrary/LoadLibraryEx are explicitly prohibited. Other calls into kernel32 are OK. But you can't call into User32. And don't use CRT memory management (unless you are linked statically) - use HeapAlloc instead. Oh, and of course don't call anything that would do any such nasty things: that would be bad. One last thing - don't read the registry either. Have a nice day.

The fact that none of this is written is big, bold, maybe even red print is truly unfortunate - it really ought to be, because most people simply miss that part. So let's say, you have read it all now the question is: why?

The thing is, as far as your binary is concerned, DllMain gets called at a truly unique moment. By that time OS loader has found, mapped and bound the file from disk, but - depending on the circumstances - in some sense your binary may not have been "fully born". Things can be tricky.

In a nutshell, when DllMain is called, OS loader is in a rather fragile state. First off, it has applied a lock on its structures to prevent internal corruption while inside that call, and secondly, some of your dependencies may not be in a fully loaded state. Before a binary gets loaded, OS Loader looks at its static dependencies. If those require additional dependencies, it looks at them as well. As a result of this analysis, it comes up with a sequence in which DllMains of those binaries need to be called. It's pretty smart about things and in most cases you can even get away with not following most of the rules described in MSDN - but not always.

The thing is, the loading order is unknown to you, but more importantly, it's built based on the static import information. If some dynamic loading occurs in your DllMain during DLL_PROCESS_ATTACH and you're making an outbound call, all bets are off. There is no guarantee that DllMain of that binary will be called and therefore if you then attempt to GetProcAddress into a function inside that binary, results are completely unpredictable as global variables may not have been initialized. Most likely you will get an AV.

Another scenario is when you start spinning a new thread on DLL_THREAD_ATTACH and wait for it to finish initialization via some syncronization technique. This blocks your thread in DllMain, while still keeping OS lock. This can lead to deadlocks.

Overall, if anything - anything - goes wrong in DllMain of one of the binaries, the whole process may be doomed.

The trouble is, definition of "wrong" is very, very vague in this case. For instance, developers using MC++ know that you shouldn't even dream of having DllMain in your library. And if you do you do, you may be very, very sorry. I think CLR folks want to fix this for the "Whidbey" release.

Chris Brumme lists the following things that should never, ever be done in DllMain.

· Dynamic binds. That includes LoadLibrary/UnloadLibrary calls or anything that may call implicitly call them

· Locking of any kind. If you are trying to acquire a lock that is currently help by a thread that needs OS loader lock (which you may be holding), you'll deadlock.

· Cross-binary calls. As been discussed the binary youre calling into may not have been initialized or have already been unutilized.

· Starting new threads and then wait for completion. As discussed, thread in question may need to acquire OS lock that you are holding.

So, what does this tell us?

DllMain is that gun you can easily shoot yourself with

How many people do you know that did stupid things like calling CoInitialize() in DllMain? I know of cases when that was done on DLL_THREAD_ATTACH, which not only means that we were risking to hit a deadlock, but also that any thread in that process will have COM initialized. What's worse, it may be initialized with the wrong threading model. And then people will be wondering how the heck they ended up with STA threads in thread pools. Or something much more subtle like calling a system function that starts a worker thread as part of its execution? How many times did you do all those things?

Another problem with this is that all these horrors can present themselves under very limited circumstances. In most cases things do work fine, but a race condition, a slightly modified DLL load order or other factors may change everything. Which means you may not even know it until your ship. This may be fine for a user application (well, things like that are never fine, it's just that the damage may not be substabtial), but this is always bad for servers - especially if you are talking enterprise availability. I don't think this can ever become a security threat - one you can fight anyway - but random crashes are just not nice.

So let's get back to what we can do in DllMain. According to MSDN, "The entry-point function should perform only simple initialization or termination tasks."

These tasks can only include calls to Kernel32 (excluding LoadLibrary/LoadLibraryEx). If you look at what this means for you, you will find that this is extremely liming. Further, CRT functions, including memory allocations are not safe unless you are statically linked. This means that seemingly innocent things something like g_pMyGlobalObject = new CMyGlobalObject() can theoretically cause all kinds of nasty stuff because they will use malloc that is dynamically linked from msvcr*.dll.

This leaves us with primitive types, synchronization objects initialization ... that's about it. And definitely - definitely - no managed code.

So what am I saying? There aren't too many things that are legal there; it's extremely easy to do illegal stuff - you have to always know if what you're calling really does, which is extremely difficult if you use something defined elsewhere - C/C++ LIB for instance; the compiler won't tell you that you are doing the wrong thing; and the code is likely to run fine in most cases... but not all of them.

Where options does this leave us with?

Just say no. Avoid the darn thing altogether and link with /noentry. Reconsider the way you deal with globals. Do lazy TLS initialization.
Be very careful. Sometimes you simply have to use it. It's just too ugly not to. Have a full code review. See what's being done and what OS does. Make sure that everyone understands that DllMain is just different. Read and memorize horror stories about people who didn't know better.
One thing you can do here to minimize the damage is disabling calls to your DllMain when new threads join/leave the process - this can be done with DisableThreadLibraryCalls. This is generally a good idea in all cases where you don't need thread-level initialization because OS loader doesn't need to call into your binary every time a new thread is born
Be afraid. Be very afraid. Well, just leave things where they are. Things don't crash right now and you have other things to do. Good plan.

Silver lining : DllMain and resource leaks diagnostics

There's one piece of information that gets provided through DllMain which you can't possibly get any other way. If you review the signature of DllMain, youll notice that the last argument passed in despite being called lpReserved actually has some meaning:

If fdwReason is DLL_PROCESS_ATTACH, lpvReserved is NULL for dynamic loads and non-NULL for static loads.

If fdwReason is DLL_PROCESS_DETACH, lpvReserved is NULL if DllMain has been called by using FreeLibrary and non-NULL if DllMain has been called during process termination.

As you see, lpvReserved does tell you something. Although I can't see why you would be interested in knowing whether your DLL has been statically or dynamically loaded - there may be uses there, I just don't see them - but knowing how you are being unloaded could be interesting.

For one, if you're managing some kind of resource in DllMain, which only lives within process context, you can possibly skip some clean-up if you knew that the process is dying as it is. This is not too valuable because the very nature of DllMain does not make it a very good entry point for resource management.

There are cases, however, when you expect your DLL to be unloaded in a specific way and you can use DllMain to verify that it is indeed being unloaded as you expect. For instance, if:

· your DLL is in fact a COM server (and has no other uses), and

· the COM host is well-behaved and

· all of your COM objects have been properly released,

then you should expect that you will get lpvReserved=NULL - that is unloaded via FreeLibrary.

Heres what seems to be happening. Every well-behaved COM process should call CoUnintialize() on each thread when it gets shut down. Internally that calls DllCanUnloadNow on your binary which returns TRUE if all outstanding references are closed. If that's the case, COM will call FreeLibrary, which - unless there are other LoadLibrary references outstanding - will unload your DLL. That will pass lpvReserved=NULL. If any of these conditions is not satisfied, your DLL will reside in the process until it terminates and you'll get lpvReserved!=NULL( I'd like to thank Michael Entin - who really ought to start blogging - for helping me to get all the pieces together).

So if - and that's a big if - your application is well-behaved, and no-one ever messed up loading your DLL with LoadLibrary and forgetting to unload it, then lpvReserved!=NULL means that some of your COM objects have not been released. There's nothing your code can do about that - except maybe asserting - and you will then have to look into that further.

This approach is not limited to only COM leaks - theoretically you should expect that when your binary is leaving this world, it's not taking anything with it. You can look through the list of globally-managed resources and see of they have been disposed if. Be very, very careful there - you shouldn't be doing any stuff that may compromise OS loader: see the four bullets above.