It is crucial that you test your packages thoroughly before deployment. A classic mistake is for developers and package authors to test only on their own systems where they have full administrator rights and then discover that normal users cannot use their applications. You may want to divide your testing into these phases:

Installation Testing
Test your package under different installation scenarios, including:

• Full user input and silent installation
• Different combinations of selected features
• Different user roles (e.g. normal user, administrator, etc.)
• Different deployment methods (e.g. manual, SMS, GPO, etc.)
• Administrative and advertised installations

Enable installer logging for each test and examine the system event logs for errors after each run. Make sure that all installation initialisation and run-time errors are resolved.

An important, but often overlooked, part of installation testing is to ensure that your setup UI is accessible and can handle different screen resolutions and font sizes.

Patching and Repair
As mentioned in the Patching section, it is very important that you think about patching before deployment. In fact you should not only think about it but test it thoroughly. You should include these scenarios:

• Small update
• Minor upgrade
• Major upgrade
• Product Repair
• Multiple simultaneous patch installs and uninstalls
• Installing and uninstalling patches "out of order"

Uninstall Testing
Many operating system related errors caused by a repackaged application happen during an uninstall session. Therefore it is extremely important to verify that a generated package does not uninstall “too much” and only uninstalls data that belongs to the package itself and not to the operating system or other applications.

Test uninstall as thoroughly as install, including:

• Uninstall under different user contexts (e.g. normal user and administrator)
• Uninstall via different methods (e.g. Control Panel, command line, etc.)
• Make sure that Operating System does not report any integrity problems
• Reboot a test machine after uninstall (even if the uninstall did not require a reboot)
• Verify integrity of common system tools and other standard applications

As with install, use Installer logs and event logs to look for errors, and make sure you resolve all those you find.

Product Functionality Test
A successful installation and uninstall sessions can be insufficient for an application to work correctly. Therefore you need to ensure that the application works as expected. Some things to check:

• Verify each shortcut in the package starts the corresponding application correctly
• Walk through main menu items in the application
• Follow the application functionality test provided by the vendor, if any
• Verify that file associations by launching a file for each registered extension.
• Make sure that application opens, save and prints the files correctly

It is recommended that corporations provide application functionality tests allowing a repackaging engineer to quickly verify major application functionality right away, and make sure that a developed MSI package correctly installs an application, and an application is ready to work properly.

User Acceptance Test (UAT)
Just as important, but often overlooked, is user input into the testing strategy. User Acceptance Testing (UAT) is especially important when re-packaging legacy applications to MSI format.

Some points to note for UAT:

• Application should be deployed for UAT in exactly same manner as it will be deployed in the production environment
• Pilot the deployment with a selection of “typical” users for the application
• Make it easy for users to report problems and provide feedback
• For internal applications, the owning Business Unit must ensure each application passes the UAT to be defined by that Business Unit. Where the package is processed externally an amount of time must be agreed upon, after which it is reasonable to assume the package has been accepted into production
• It is recommended that product functionality tests be provided for all internal applications. Such tests will allow a packaging engineer to verify major application functionality before submitting a migrated package for UAT

Validation is a means by which you can run certain standard checks (called ICEs) against your package to check for inconsistencies and non-compliance with Installer rules.

Many ICEs ship with the Installer SDK in .CUB files and you can write your own. They can be run against your packages using the ORCA or MsiVal2.exe tools from the SDK.

You should always fully validate your packages and, much like when working with application development, correct any reported errors and treat all warnings seriously.

Validation is also a useful troubleshooting technique, particularly if you are working with 3-rd party packages.

As mentioned previously, the Installer supports extensive logging. This is by far the best tool for troubleshooting Installer issues and the time spent understanding the log files will be repaid many times over when problems arise later. At the least logging should be used throughout testing.

You are certain to be asked to provide verbose logs if you contact Microsoft support, so either have these ready or be prepared to reproduce your problem to gather some.

Using virtual machines (VMs) for testing rather than real systems has a couple of advantages:

• You can build a VM for each of the different systems in your environment and store these on a single server or workstation to be brought on-line when needed
• With “undo”-style features of VMs you can easily revert to a clean, known state after each test
• The same VM images can be used by testing and helpdesk staff

However, do note that if you think you have discovered a bug in the Installer you may be required to reproduce the problem on real hardware when you contact Microsoft support:

Support policy for Microsoft software running in non-Microsoft hardware virtualization software