The first post in the Windows Embedded Installer Testing series discussed a detailed level of the testing process that the installer undergoes. The purpose of this second post is to give a higher-level view of this process by introducing the automation test framework used in testing the installer along with many features of Windows Embedded Tools. Implementing automated tests has many advantages which include:

• Removing the risk of human errors in testing.
• Providing consistency with respect to which tests are run on a frequent basis.
• Reducing time and cost of running the test and thus allowing the test engineers to have more allocated time to develop and run more tests.
• Allowing the ability to run tests simultaneously on multiple platforms in an efficient way. This includes overnight stress and other types of tests.
• An important point mention here is that as much as it sounds great to have all of our existing planned tests automated, it is rarely the case where all the test cases for a specific feature are automated. In some cases, developing and maintaining test automation can lead to more effort being put on the automation process than what is actually required to run the tests, and there lies the responsibility of each individual test engineer in determining which way would be the best way to proceed with when she/he is creating the test plan of a specific feature.

The framework:

At Microsoft, there is a test automation framework that is internally developed and heavily used throughout different teams across the company. Much like any other test automation framework, this tool functions as follows:

• A controller machine controls and monitors the testing process and gives an in-time updates on the status of the network and the test cases being run.
• One or more client machines which can have different platforms and which will run the assigned test cases and reply to the controller with the test results.

Fig. 1

Fig. 1 gives a visual representation of the process mentioned above. In the case of running test cases for the Windows Embedded installer, the test client machines are usually running the installer specific test cases such as installing/file checking/uninstalling simultaneously and reporting back the result to the test controller PC where the test engineer monitors those test to ensure that the installer’s quality is meeting the expected high quality metrics.

Finally, for readers who are interested in learning more about test automation frameworks and to have some hands-on experience in it, I recommend trying out some of the open source automation frameworks. STAF (Software Testing Automation Framework) is one of the many open source frameworks that you can find online. More information about STAF can be found here http://staf.sourceforge.net/

-Sami

Technorati Tags: XPe,Windows Installer,Standard 2009,Testing