The C# compiler in Visual Studio 2008 and the .NET 3.5 Framework (csc.exe) is now generating PE files with the NXCOMPAT bit set. What is that bit and who cares, you ask? You may very well care if your application interops with native binaries or exposes a plugin model to 3rd parties. First, some background...
DEP is short for Data Execution Prevention. It is a technology that exists in Microsoft operating systems which prevents execution of code from memory pages which are not marked as executable. DEP exists to reduce the attack surface available to malicious software that is trying to hijack a process, and it has been acknowledged to be very helpful in that regard. In Windows Vista, the set of processes and applications to which DEP is applied is configurable by administrators, but there is also a role for application developers.
In the header of a PE file there is a flag called IMAGE_DLLCHARACTERISTICS_NX_COMPAT. This flag affects whether or not the OS enables DEP for a process. Setting this flag tells the OS that the image is compatible with DEP. For executable images, if this flag is set, the process is run with DEP enabled unless the machine is configured with the DEP policy set to AlwaysOff. If the image is a DLL and the flag is set, the OS skips checking the DLL against a compatibility database which results in a small performance improvement. All of this applies to x86, 32-bit processes only. On a 64-bit OS, DEP is always enabled for 64-bit processes, but 32-bit processes are configured by the PE flag and system policy as described above. So how does one control the flag in the PE header?
Since the C# compiler emits PE files which are MSIL only and therefore compatible with DEP, the output binaries from the VS 2008 and .NET 3.5 C# compilers have this flag set. Our expectation is that the vast majority of C# executables produced by these compilers will be part of a DEP-compatible application. For that reason we did not surface a compiler switch to configure the NXCOMPAT setting. Of course you can write a C# application that uses a native or mixed binary which is not compatible with DEP. Some ATL types in 7.1 and earlier used to do simple code generation into data pages which is a DEP no-no. If your application is generating IP_ON_HEAP exceptions, then you may need to clear the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this you can use EDITBIN.EXE from the VC toolset like so:
editbin.exe /NXCOMPAT:NO <your binary>
If you're using Visual Studio, you can add a post build step to your executable's project. You'll need to setup the environment so that EDITBIN's dependencies can be resolved. Since the post build steps you author in Visual Studio's properties page are written into a batch file that is launched by the build process, you can use Visual Studio's VSVARS32.BAT to establish the right environment. My post build step looks like this:
call $(DevEnvDir)..\tools\vsvars32.bateditbin.exe /NXCOMPAT:NO $(TargetPath)
If you sign the binary in Visual Studio, flipping the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag in the post build step after the binary has been signed will result in an assembly that will fail strong name validation. To work around this you'll need to begin signing your binary as part of the post build steps. To do this, use SN.EXE from the Windows SDK.
We like DEP so much that when you install .NET Framework 2.0 SP1 your C# compilers in VS 2005 and .NET Framework 2.0 will also begin to emit binaries with the IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit set. This will undoubtedly surprise a few developers...download a framework service pack, recompile, run your app, and you're now getting IP_ON_HEAP exceptions. Obviously this is not ideal, but aggressively building a computing ecosystem filled with DEP-enabled applications and their accompanying security benefits is very beneficial to Windows users. If you begin to encounter IP_ON_HEAP exceptions after installing .NET Framework 2.0 SP1, you can use the same technique described above to clear the IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit in VS 2005. The only difference I'm aware of is that the SDK (and therefore the location of SN.EXE) has moved.