New NX APIs added to Windows Vista SP1, Windows XP SP3 and Windows Server 2008
In the interests of helping secure the platform, we want more people to opt-in to using Data Execution Prevention (aka DEP aka NX), and we have lowered the barrier to entry for application developers in Windows Vista SP1, Windows XP SP3 and Windows Server 2008.
We've added some new APIs that allow a developer to set DEP on their process at runtime rather than using linker options. The new APIs also give developers some more flexibility if your application uses an older version of the Active Template Library (ATL.) Before I explain the new APIs, let me give you a little history behind ATL and NX.
Some ATL History
ATL has been around for a long time; it's reasonably light-weight and allows developers to build COM components rapidly. It also includes classes for manipulating security descriptors and such; to be honest, it makes working with Windows security objects open to mere mortals.
Older versions of ATL, and by older I mean pre-Visual C++ 2005, used dynamically generated code in small isolated cases. Obviously, without the appropriate APIs this is going to cause problems on a DEP-enabled computer, because you can't execute data. This code is referred to as a "thunk" and versions of ATL in VC++ 2005 and later work correctly with DEP.
The APIs
The most important API added is SetProcessDEPPolicy, which sets the DEP policy for the running process. You would normally use this function pretty early in main.
The function takes only one argument: the policy setting. The possible values are:
- 0x00000000 Turn off DEP for this process (Why are you doing this?)
- PROCESS_DEP_ENABLE Enable DEP for the process.
- PROCESS_DEP_ENABLE | PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION Enable DEP for the process, and disallow ATL thunks.
The last option is the killer argument - if you build an application that hosts components that might not be DEP compatible because they were built using an older version of ATL, you can still use DEP for your process.
There are two other functions: GetSystemDEPPolicy and GetProcessDEPPolicy; I'm not going to insult your intelligence and explain what they do.
The only negative to these APIs is they must be dynamically loaded because they don't exist on all supported versions of Windows. The following code shows how you can use the functions regardless of Windows version:
#define PROCESS_DEP_ENABLE 0x00000001
#define PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION 0x00000002
BOOL SetDEP(__in DWORD dwFlags = PROCESS_DEP_ENABLE) {
HMODULE hMod = GetModuleHandleW(L"Kernel32.dll");
if (!hMod) return FALSE;
typedef BOOL (WINAPI *PSETDEP)(DWORD);
PSETDEP procSet = (PSETDEP)GetProcAddress(hMod,"SetProcessDEPPolicy");
if (!procSet) return FALSE;
return procSet(dwFlags);
}
If you OR the two flags together, it's virtually the same as linking with /NXCOMPAT.
When to use the NX APIs
There are three main reasons to use these new APIs:
- If your application has some form of in-process extensibility mechanism, and some applications might use older ATL, then you can enable DEP for your process, and the extensibility mechanisms using ATL will function correctly.
- If you support DEP but want to allow customers to disable DEP if there are serious compatibility issues, then this is the API to use because the argument can be a configuration option.
- If your application uses an old version of ATL, and you still want to do the right thing by DEP, then use this function. Of course, you really ought to use an updated version of ATL!
One Caveat
I'm only telling you this because it bit me.
There is one caveat that you should know; SetPRocessDEPPolicy often returns error 5 (Access Denied) but this error does not mean the operating system is denying access, it means you are attempt to change DEP policy in a way that is not appropriate. For example, if you link with /NXCOMPAT, and then use this API, you'll get the error. Or, if the operating system is configured to use DEP for all processes all the time no matter what, then you'll see the same error. Finally, you'll get an access denied error if you attempt to call SetPRocessDEPPolicy twice in one application; once the policy is set, it's set for the process lifetime.
In short, don't be overly alarmed if you see this error.
