The Basics of the IE Testing Matrix

As we recently shipped security update MS04-025, I thought it would be good to talk about the testing coverage that we do for each security update.  Testing is about risk management, and we have to make judgments about where to invest heavily in testing and where to invest less in testing to achieve the highest quality IE when we release.  Code change is one main driver of this (test where the code is changing), but another important part of the process is to understand the larger landscape of what you are testing and how it fits into its overall software ecosystem.  

If you take a look at the security bulletin for MS04-25 (http://www.microsoft.com/technet/security/bulletin/MS04-025.mspx), you will the see two of the main variables for our testing - IE version and Windows version.  I have copied those lists here for reference.  These lists are generated from the Microsoft Lifecycle policies (http://support.microsoft.com/common/international.aspx?rdpath=fh;en-us;lifecycle).  

Internet Explorer Version (for MS04-025)
Internet Explorer 5.01 Service Pack 2
Internet Explorer 5.01 Service Pack 3
Internet Explorer 5.01 Service Pack 4
Internet Explorer 5.5 Service Pack 2
Internet Explorer 6
Internet Explorer 6 Service Pack 1
Internet Explorer 6 Service Pack 1 (64-Bit Edition)
Internet Explorer 6 for Windows Server 2003
Internet Explorer 6 for Windows Server 2003 (64-Bit Edition)
 
Windows Version (for MS04-025)
Microsoft Windows NT® Workstation 4.0 Service Pack 6a
Microsoft Windows NT Server 4.0 Service Pack 6a
Microsoft Windows NT Server 4.0 Terminal Server Edition Service Pack 6
Microsoft Windows 2000 Service Pack 2
Microsoft Windows 2000 Service Pack 3
Microsoft Windows 2000 Service Pack 4
Microsoft Windows XP and Microsoft Windows XP Service Pack 1
Microsoft Windows XP 64-Bit Edition Service Pack 1
Microsoft Windows XP 64-Bit Edition Version 2003
Microsoft Windows Server® 2003
Microsoft Windows Server 2003 64-Bit Edition
Microsoft Windows 98
Microsoft Windows 98 Second Edition (SE)
Microsoft Windows Millennium Edition (ME)

Not all versions of IE run on all versions of Windows.  For example, IE 5.01 SP2 is specifically produced for Microsoft Windows 2000 Service Pack 2.

There are more variables than just IE version and Windows version, and some like language create a multiplier on IE and Windows version.  For IE 5.5 Service Pack 2 there are 26 language specific binaries.  With the general assumption of 26 languages for each IE version, we have at minimum 234 installs to test for each security update we release.  My test team has to make sure that each one of these install packages installs correctly and works as expected - that is a lot of setups! It is definitely the type of testing where we utilize automation and tools to reduce the need to manually do all those setups.
 
We also include other test variables in our coverage, like applications and sites, which can range from a variety of client based applications that host the IE browser to web sites that leverage different scripting technologies.  Application and site compatibility is very important to IE, and in order to get good testing coverage we have to anticipate it will impact the Internet as well as corporate Intranets.  In order to address this, we classify apps, sites and technologies into buckets, which allows us to test specific sites or apps that are representative of the overall technology bucket.   We also look at a variety of measures that tell us what the top sites are, and then test IE against those sites.  

This is just a quick snapshot of some of the test coverage issues we grapple with in testing IE.  More testing tidbits later.