Software product installation is a critical part of the product usability and experience. Here at the Windows Embedded Team, we strive to make the process of installing the Windows Embedded suite of tools as seamless as possible for the product’s users. The purpose of this blog is to give a quick overview of the testing methodology that is used to ensure that the Windows Embedded tools installer meets our high quality standards.

Windows Installer has been extensively used internally at Microsoft by different teams and externally by different software developers and vendors. It has many advantages such as “reducing the Total Cost of Ownership for the customers by enabling them to efficiently install and configure the products and applications” - [Reference 1]. The Windows Embedded Team is leveraging those advantages by using the Windows Installer framework to create installation packages for the Windows Embedded tools.

Testing Areas

The test procedure covers many user scenarios as well as other Windows Installer package tests. The following sections provide more information on each test category.

Package Validation Testing

Windows Installer framework provides means for validating a certain installation package (msi). ICE (Internal Consistency Evaluators) are run on every build as a way to ensure that the package is valid and that it complies with Windows Installer rules. Consult [Reference 2] for more information about Internal Consistency Evaluators.

Installer Logs check testing

Windows Installer logs are regularly checked to ensure that all irregularities and warnings that might be caught by the installer are resolved. This also simplifies our communication internally with the Windows Installer team when we have concerns or suggestions regarding the installer.

Installation Testing

Several installation options and features are currently being provided with our product. During our installation testing many user scenarios are covered. These scenarios include:

• Full installation with user input or with silent mode (using “msiexec –quiet”).
• Custom installation with different combinations of the supplied features.
• Repair/Modify installation.
• Administrative installation (using “msiexec –a”).
• Installation under different user profiles (administrator , regular user, etc..)
• Installation with different target locations (network paths, local paths, etc..)
• Cancelling an installation results in a complete rollback where the system is back to the state that was in before the installation.

After each installation some checks are performed on the target machine to ensure that the installation process is consistent with our specifications. Such as:

• File/folder structure and naming are correct.
• Product versioning is correct.
• Files are digitally signed correctly.
• Timestamps and file sizes are correct.
• No files/folders should be missing and no additional un-required files/folders exist.
• Registry Keys are installed correctly in the registry when required.
• EULA (End User License Agreement) is tested to make sure that it is displayed correctly to the users.
• Any shared dlls are updated when required.

An important point to mention here is that our product shares common resources (such as dlls) with Windows especially when dealing with the Windows OPK tools (OEM preinstallation Kit). We consistently collaborate with the Windows team to make sure that these common resources are installed/uninstalled correctly.

Uninstallation Testing

With uninstallation it is extremely important to ensure that the uninstallation process doesn’t remove any data that might be shared with other programs. As with installation, different scenarios are tested and these include:

• Uninstalling using different paths (Control Panel, command line, etc..)
• Uninstalling using different user profiles (administrator, regular user, etc..)

The list of checks that are applied after uninstallation includes:

• All files/folders that are not used by other programs are removed, and that user-specific files are not removed.
• All product specific registry keys are removed except the ones that are required for saving user-specific variables. (which normally allows for re-applying some of the settings for the user in case of a re-installation)

Product Basic Functionality Testing

After every installation basic functionality tests are run to verify that the product got installed correctly on the target system. These tests include:

• Shortcut tests to ensure that all created shortcuts are functioning correctly.
• Make sure that the tools open, create and save configurations correctly.

Further Testing

This blog has presented the main concepts in testing installer that are currently being used to ensure high quality for the Windows Embedded product. I will be posting more blogs regarding our testing process in the near future. The current list of future blogs that I might write for Windows Embedded Studio Installer include:

• Test Automation framework for Windows Installer.
• Quality Indicators used in Windows Embedded Team and how they are being enforced.
• Preparing the Windows Installer package for localization.
• Validating a Windows Installer patch and testing it.
• Security Testing for Windows Installer.

[Reference 1] http://msdn.microsoft.com/en-us/library/aa370566(VS.85).aspx

[Reference 2] http://msdn.microsoft.com/en-us/library/aa369554(VS.85).aspx

-Sami