There is a trick I use very often when trying to figure out why an MSI-based setup is failing that I wanted to share with everyone.  I believe it is commonly known among setup developers and people who have to troubleshoot failed setups, but I could not find any "official" documentation for it.  This trick helps narrow down the root cause of error code 1603, which is a generic catch-all error code that means "fatal error during installation".  The 1603 error code is returned when any action fails during an installation, and most commonly it indicates that one of the custom actions in the MSI failed.

When I encounter a failed setup with return code 1603, here are the steps that I follow:

1. Re-run the setup with verbose logging enabled using steps similar to those that I listed here (if there is not already a verbose log file available).  Those steps will generate a verbose log file named msi*.log in the %temp% directory the next time the setup package is executed.
   
   Important note - some MSI-based setups, including the .NET Framework 2.0, 3.0, 3.5 and higher and Visual Studio, will not create log files named %temp%\msi*.log even if using the instructions listed below.  Please see this blog post for more details about why that is the case and also for a list of some products that I know of that use different log file creation logic and the locations of the log files that they create.
   
   
2. Open the verbose log in a text editor such as notepad and search for the string "return value 3".  In nearly all cases, this takes me to the section in the verbose log that lists the action that failed that initially caused setup to rollback.
3. Review the contents of the log file immediately above the "return value 3" string to determine which custom action or standard action failed.
4. Depending on which action is failing, I will proceed to more detailed debugging from here

I find that the biggest hurdle to debugging a failed setup is often zeroing in on which part of the setup is actually failing, and this trick of searching for "return value 3" ends up helping speed this process up in nearly all cases.  Of course, it does not work in 100% of scenarios.  Notably, if you are running setup on a non-English version of Windows, the string "return value 3" is written to the log file in the language of the operating system instead of in English, so string searches will not work.

Also note that there is an MSI verbose log parsing tool in the Windows Installer PSDK that is also very useful in locating errors inside verbose log files.  You can read more about this parsing tool (called wilogutl.exe) by clicking here.  This tool is more thorough in identifying errors, but most often I end up not using it because it is faster to open the log in notepad and do a string search than it is to load up the parsing tool, browse to the log file, wait for it to parse the whole log and then read the output it produces.

<update date="1/21/2009"> Added a caveat to these instructions indicating that some setups create their own verbose logs and enabling verbose logging using the Windows Installer logging registry keys will not work as expected for those setups. </update>