About Windows Installer, the .NET Framework, and Visual Studio.
When problems occur when installing, repairing or patching, or uninstalling a product using Windows Installer technology, you need to be able to figure out what happened and why it happened. Dialogs that occur when the user interface is displayed can be helpful sometimes, but many times you need to look at a log to determine the exact cause of the problem.
The first step in diagnosing a problem using the log is to, of course, enable logging. When performing any actions with an installation package or patch always be sure to pass /L*v filename.log on the command line. This enables verbose logging. You can also use /L*vx which outputs debugging information as well, but this isn't always helpful and can lead to much larger log file sizes than even verbose logs. This option is also only available on Windows Server 2003 and newer. You can also set logging options using the logging policy that will apply to all Windows Installer operations on the machine.
Now that verbose logging is enabled when an error occurs you can search for "value 3" and in most cases find the return code from a failed standard or custom action. Note that this is not the only indication that a standard or custom action did not run perhaps as expected. An installation that returned prematurely but that did not return an error code from the process would have returned IDIGNORE, so you would have to search for "value 5". In Return Values of JScript and VBScript Custom Actions you can find the return values that can be returned from actions and what they indicate. Of course, it is recommended that you write custom actions in native C++ but this page does list both the #define constants and their values. Another good source of return values for custom actions is Logging of Action Return Values that lists a few more possible return values, as well as what value is returned from the custom action server depending on what values you return from your custom action.
You should see a line like the following listing the action that returned the error value:
Action ended 12:00:00: MyCustomAction. Return value 3.
Now that you've located the error in the log you must determine what really happened since the return value isn't really adequate for determining the cause of the problem. If your custom action failed hopefully you've logged enough information to determine the cause right away. If you used the MsiProcessMessage function to log the error from the Error table, or a standard action failed, you will find the error number in field 1 of the log message a little above where you found "value 3":
MSI (s) (AC:14) [12:00:00:000]: Note: 1: 1601 2: C: 3: 4096 4: 1024MSI (s) (AC:14) [12:00:00:000]: Product: My Product -- Out of disk space -- Volume: 'C:'; required space: 4096 KB; available space: 1024 KB
The first line lists the error in field 1 as 1601. If you look up the error in Windows Installer Error Messages you will see Windows Installer error 1601 has the English text of "Out of disk space -- Volume: ''; required space:  KB; available space:  KB". Plus the values of the fields in the first line into that message and you'll end up with the second line in the log. This will give you a much better indication of what error occurred. If the error value isn't in the Windows Installer Error Messages table, be sure to look in the Error table itself since this may be a custom error. Custom errors should be in the range from 25000 to 30000 according to the documentation for the Error table, but this isn't always the case.
In Logging of Action Return Values return codes from msiexec.exe are paired with error codes returned from custom actions, but msiexec.exe may return other error codes as documented in Error Codes, such as ERROR_UNKNOWN_PRODUCT (1605) if you try uninstalling a product by its ProductCode that is not installed, or ERROR_PATCH_TARGET_NOT_FOUND (1642) if you try applying a patch to a product that is not installed.
Rob Mensching, father of Windows Installer XML (WiX), blogs about why managed custom actions are a bad
How to determine the source of Windows Installer error 2262.
Another reason a custom action may not run is because the custom action function isn't exported. This