EXE Custom Actions are Bad
Windows Installer custom actions that launch executables (base custom action type msidbCustomActionTypeExe, 0x2) are typically bad and should be avoided. How are they bad? Let the Windows Installer team count the ways.
During internal audits we question any EXE custom actions (CAs) but inevitably some make it into the product. Typically these EXE CAs are standalone configuration utility applications that are installed with the product or are managed EXEs where developing them as EXEs yields the benefit that the required CLR version can be loaded. You can read more about managed CAs from Rob Mensching and find more guidelines on my blog, which is a summary of what we use internally.
The analysis which the Windows Installer team posted isn't complete when it comes to mitigations, however. In several places it describes writing a DLL to launch the EXE and do whatever is required to mitigate a problem. It fails to document a common mitigation, exposed in WiX as CAQuietExec for example, of using a DLL CA to capture output from the process and putting that output into the Windows Installer log.
The UILevel could also be passed directly to the command line to the EXE rather than requiring a DLL that alters the command line depending on what arguments your application accepts. Because the UILevel values of 2 through 5 probably aren't the typical verbosity levels an application might accept, a CA might be necessary but you could just as easily use type 51 CAs to set a common property, conditioned on the UILevel, and use that common property in the command line arguments. No need to further complicate an installer package that uses binary CAs by adding more binary CAs.
Rollback CAs can also be scheduled as EXEs to run, and might typically be the uninstall application or a variant of the same application with different command line arguments.
Finally, missing dependencies are also a problem for DLL custom actions. A common mistake is linking to the side-by-side CRT libraries, 8.0 and newer. Aaron Stebner goes into detail, but the basic issue is if you're installing the CRT with your application in the same transaction, the dependent DLLs are not available until after the InstallFinalize custom action returns. Perhaps equally common during development is linking to the debug CRT libraries but not installing the debug runtime on the target machine.
Ultimately, avoiding CAs is the best approach but, when necessary, you should develop DLL CAs. Immediate DLL CAs can read from the installer package database and should be data driven, making it easier to service that data and providing perhaps some transparency to what your CA does. Those immediate CAs can then schedule deferred, rollback, and commit CAs as necessary based on that data and the current state of the machine.
