Engineering Windows 7

Welcome to our blog dedicated to the engineering of Microsoft Windows 7

Application Compatibility Testing -- Overview

Application Compatibility Testing -- Overview

  • Comments 50

Delivering a new release of Windows includes a major effort to insure that applications continue to function as well on the new release as they have on the previous release. At the PDC we talked about some of the new areas of Windows Vista that reduced this level of compatibility, such as changes we made around the OS security model. With Windows 7 we renewed our engineering efforts to maintain compatibility. As with device testing, compatibility testing is an effort that spans the entire engineering organization, though we also have a group that is dedicated to this effort. This post was authored by a set of folks and coordinated by Grant George, the corporate vice president for testing in the Windows Experience team. --Steven

We have taken a very proactive approach to Application Compatibility and our process starts from the moment we first plan our product schedule and design and check in code for Windows and runs through all of our engineering processes and disciplines leading up to our final release (and beyond).

Application Compatibility Goals for Windows 7

Our main Application Compatibility goal for Windows 7 is to make sure that most all applications which work on Windows Vista will continue to work seamlessly on Windows 7. We do have to be careful about making this claim to be universal because there is a class of applications that are always updated in tandem with a new Windows release. These applications are primarily system utilities, diagnostics, and security software—the common thread is that they make assumptions about the underlying implementation of Windows internals and thus require updates. We carefully coordinate with a large set of ISVs who provide these applications. This was talked about earlier this month as we announced our Ecosystem Readiness Program, which we will discuss more below.

At the start of our product cycle we review our new features and changed designs to ensure that every element of Windows 7 has Application Compatibility in mind. Our engineering process includes automated quality checks to assure public APIs don’t change, and our test engineers have the right tools, engineering time and information that is used to find application issues as early as possible in our development cycle. Telemetry information is collected to assess and prioritize the breadth of applications our users depend on, paired with market data and install base information, across a wide variety of software categories to make sure they work as expected in our new OS version.

Below we expand on how we deliver on this goal.

Engineering Application Compatibility

Rich telemetry from external usage and the broad software marketplace helps us to provide a list of the most popular and critical applications, updated frequently throughout our development cycle. Our engineers then acquire these applications and build automated tests that verify each one works as expected on Windows 7.

Changes that could impact application compatibility are followed up closely, for example Legacy Code Removal which involves removing code that existed in the previous release of the product is strictly guarded and tightly managed and should not happen without proper documentation and engagement with the impacted parties. For example if we need to deprecate an API call, the documentation for this API will be updated and we will wait until the impact of the removal is minimal as indicated by telemetry data unless it is required sooner to fix a security issue.

Throughout the development process we are running tests in the background creating an ongoing validation of new code relative to application compatibility. As code is getting ready to be checked into the main build, if a compatibility failure is detected in an automated regression test the checkin is halted. At that point the code is scanned for known compatibility issues and if an issue is detected the developer is asked to fix the problem. Of course we also develop new tests throughout the course of developing Windows 7 in order to broaden our coverage of third party software.

Engineering & Testing Strategy

We have several teams dedicated to application compatibility as part of the Windows development effort. These teams provides guidance on how to build in application compatibility, provides data on application dependencies, and information on what applications may be impacted for any particular change we make in the Windows platform. These teams also reviews new feature designs, as well as other planned changes, to ensure that the engineering team has fully taken application compatibility into account so that we achieve the tenet of "keeping applications working on Windows 7".

In addition to working with the internal Windows engineering teams, we also reach out to 3rd party developers writing Windows applications to ensure these partners have all the information they need to make their solutions fully compatible with Windows Vista and Windows 7. Furthermore if we do uncover any issues that may need to be resolved by the 3rd party developers, we collect all the information, resources and guidance and engage in a conversation with those external developers to help them understand and fix these issues.

We recently announced the Windows 7 Ecosystem Readiness Program. This program provides partners with access to Windows 7 builds and tools they need to test solutions for Windows 7, Windows Vista, Windows Server 2008 and Windows Server 2008 R2. The Ecosystem Readiness Program also facilitates testing multiple components of the ecosystem together to improve the overall user experience. Rather than just focusing on getting a specific OEM product, software application, or hardware device certified, we will be bringing multiple components together to verify a rich user experience that delivers quality, reliability, and performance as well as innovation through new feature adoption.

As mentioned earlier we use telemetry data and market share information to choose the applications we directly test on as part of our compatibility efforts, below you can find the a sample of the Consumer Scenarios we focus our testing on:

  • Communications
  • Gaming
  • Fundamentals like Setup, Security and Performance
  • Memories
  • Music
  • Productivity
  • TV/Movies
  • Data Backup/Security
  • Mobility
  • Financial Management

To view a full list of Application Categories covered please take a look at the Appendix section at the end of this post. While we would love to provide the detailed list of products, several of the sources of this information are based on proprietary research from third parties.

Another very important set of technologies we test on are the middle tier technologies like Java, the .Net Framework, etc. to make sure the applications that use these technologies continue working as expected

In addition to 3rd party stand alone applications we test a subset of OEM pre-installed software and their inbox applications for compatibility. The software tested come from the engagements we have with our OEM partners and their submitted installation images. These images are tested on clean installations of Windows 7 and upgrades from Windows Vista on OPEM standard hardware. This level of coverage allows us to best replicate the initial experience with Windows 7 for many of our customers. Because many of these applications are closely aligned with the OS, hardware and drivers, it is not unusual for an OEM to provide updates to this software with a new OS release.

In addition to the above mentioned testing approaches, Microsoft IT maintains a software portfolio of approximately 1,500 applications. These applications must be tested prior to software deployments inside Microsoft.

Microsoft IT developed an application-tracking method that simplified the process of selecting applications for sample-based testing. By identifying groups of applications that have similar data processing, controls, underlying technology, and methods, Microsoft IT is able to test approximately 4 to 6 percent of the total applications and gain a reasonable assurance of compatibility for all.

For more information you can read the LOB Application Compatibility Technical White Paper.

Validation Strategy

Part of our testing process includes the creation of scenarios that we validate on 3rd party applications. We approach this by verifying the intended functionality of the application while focusing some of our attention to changes in the OS, new functionality and risky integration areas. Manual and automated test passes are scheduled to cover the identified scenarios and to verify the user experience. We cover the applications on different sets of hardware and make sure that that we cover a lot of different configurations, x86, x64, Intel, AMD, touch and multi -touch machines, etc.

We use specific categories when measuring the compatibility of the applications we test on:

  • Excellent - No compatibility problems
  • Poor - Minor functionality is not working
  • Failure (regression) - It used to work in Windows Vista now it is not working properly
  • Failures (non-regression) - The application does not work on Windows 7 but it did not work on Windows Vista either
  • Accepted Regression - We know of changes in the OS that may affect the applications and we have/are working with ISVs on mitigations for these issues. A simple example is version number. An application that is hard coded to only work on specific versions and it doesn’t know what our new operating system version is would fail.

We will cover these categories in more detail in subsequent blog posts about how we manage application compatibility.

As part of our testing we do find issues that may need to be resolved by the 3rd party developers at the companies that develop and sell these applications. We collect all of the relevant technical information, resources and guidance and engage in a conversation with those external partners to help them fix the issues. Of course we also engage them in our technical beta programs so they can test Windows 7 compatibility with their products at the same time we do.

Final Word

Application compatibility is very important to the Windows team. We are constantly working to improve your experience with applications as you move from one release of Windows to the next. We encourage you to try our Windows 7 beta release to experience the improvements in the application compatibility space and we want to hear your feedback.

It is worth mentioning the work we have done from an end-user perspective to assist in application compatibility. Many failures of application compatibility happen at install time and to assist with this, in Windows 7 we have improved the detection of failed installations and provided a step by step wizard which will help to get an application’s “compatibility mode” correctly set. We also provide real-time problem reports and solutions that can help you locate an updated version or patch. Many of you might have experienced this when trying to run Skype after an upgrade or install the current version, and in both cases you were automatically referred to the Beta download site.

Appendix

Microsoft Covered Consumer Scenarios & Application Categories

Table of Microsoft Covered Consumer Scenarios and Application Categories

Our list includes the top 50% best selling applications in the past 24 months, some of this data is collected and aggregated from several well-known third party research information providers.

Leave a Comment
  • Please add 8 and 2 and type the answer here:
  • Post
  • this is my feedback and my forum frends

    Beta 7000  (Italian Feedback)

    http://tinyurl.com/bqz8u5  

  • All my WDM softsynths fail to work on Windows 7 but work flawlessly on Windows Vista. In Vista the UI to select the softsynth was gone but with community replacements like "MIDI Device Picker" (search a bit), it was fixed. Now after installing my favorite Roland Virtual Sound Canvas VSC-MP1 or WinGroove or Yamaha XG Softsynth on Windows 7, I find they no longer show up. My sound card's FM synthesizer also does not work. Tried the 32-bit PDC build, beta everything. Has this been fixed in RC1?

  • I'll be happy if I don't have to add a line of code to say my app is W7 compatible. I had to add a meta tag to my sites for IE8 compatibility.... :-(

  • Won't removing Microsoft Agent break some apps? Moreover installing Agent from the 2000-ish redistributable succeeds however the Agent characters lose their transparency (looks very ugly). The Vista release of the Agent binaries doesn't have this problem but it isn't redistributable.

  • "A simple example is version number. An application that is hard coded to only work on specific versions and it doesn’t know what our new operating system version is would fail."

    I still find it very strange that Windows 7 is internally 6.1. I agree that it is indeed "6.1", but if marketing decides that the next version is called "Windows 7" the version number should reflect that.

    But you guys do a good job. This is one of the most difficult things to work on. And you're doing great.

    Is it getting easier with each new Windows release? Is Windows 7 an "easy" release for you since there aren't many fundamental changes?

  • The VPN client of my university by Cisco didn't work under Windows 7. I wasn't able to boot into Windows 7 because right before the login screen the VPN client caused a bluescreen. There wasn't anything I could do except using one of the recovery tools Windows provides to recover an earlier state of the OS.

  • The Vista version of Inkball is not compatible with Windows 7.  Inkball was excluded from the Windows 7 beta.  Inkball is a great game that should be playable on future versions of windows.  Would you reconsider including it in the final release.  Alot of people will be disappointed by it's omission.

    Perhaps you can release downloadable non-drm x86 and x64 versions.  Thanks.

  • Um - what about compatibility with apps that work in XP? A smooth upgrade from Vista won't be what MS has to sell us in the corporate world, but a hassle-free move from XP. MS has to convince us to move from XP. Merely releasing 7 as an upgrade to Vista isn't the deal.

  • We said we just one SKU that can be modularized but here comes Microsoft with so many SKUs again. Can you guys either merge or remove Pro edition of Windows 7 and make Ultimate edition available on retail stores because like I said before we don't wanna be a part of that PAY MORE AND GET MORE--Ultimate scheme. I am not gonna pay AGAIN by taking out my credit card again after purchasing Windows 7 Pro just to take advantages of so called enterprise features like BitLocker, AppLocker, booting from VHD when those features should be present in the consumer/business edition of Windows 7. I have Vista Ultimate x64, so upgrading to Pro means I would lose BitLocker? that really sucks. Shouldn't AppLocker and booting from VHD be present in both the home and pro editions of Windows 7? it should be...Please do listen to us.

    There should be an option to turn the Superbar and the window border to black just like in Vista when a window is maximized as opposed to manually changing the color of glass (Aero) because I often find it very distracting when using programs like Visual Studio etc.

  • Among apps, I haven't found anything that is compatible with Vista but not compatible with Windows 7. Inkball, talked about above, is the only thing I've heard about.

  • here comes the "Why doesn't my 12 year old 32-bit application not work on Windows 7" comments. Their are people still lamenting the death of 16-bit application support in Windows.

  • I was trying to install the latest version of ISOrecorder, but came across the "this product must be installed on Vista"-type error message.  How is Microsoft classifying errors such as these?  The product works under Win7, I'm sure, but won't install because it detects the wrong OS.

  • i'am sorry if this dont belongs here but i fond no other place

    @solaris:

    1.  Uninstall any previous version of Cisco VPN client that you may have installed

    2.  Reboot

    3.  Download the Citrix DNE Update and install it.

    or

    3.  Download the Cisco DNE update and install it.

    4.  Reboot

    5.  Take ownership and delete ndis.sys (in c:\windows\system32\drivers).

    6.  Take ownership and delete ndis.sys.mui (in c:\windows\system32\drivers\en-us).

    7.  Install Cisco VPN Client 5.0.04.0300.

    8.  Reboot

    9.  Windows 7 will repair itself (should take a few seconds) and automatically reboot.

    10.  Cisco VPN Client should work without any other tweaks.

    this worked for me...but no guarantee ;)

  • "someone" wrote:  

    "All my WDM softsynths fail to work on Windows 7 but work flawlessly on Windows Vista. In Vista the UI to select the softsynth was gone but with community replacements like "MIDI Device Picker" (search a bit), it was fixed. Now after installing my favorite Roland Virtual Sound Canvas VSC-MP1 or WinGroove or Yamaha XG Softsynth on Windows 7, I find they no longer show up. My sound card's FM synthesizer also does not work. Tried the 32-bit PDC build, beta everything. Has this been fixed in RC1?

    Tuesday, March 03, 2009 9:09 AM by someone"

    Hey "someone" -- we've been trying to find a repro for this issue that we first heard about in Larry Osterman's blog.  I'd love to find out more about your specific HW and config to see if we can get to the root cause?  

    -Steve Ball from the Windows Sound team.

  • @gormed

    Thanks for the note. Please use the Send Feedback button to submit your report with the details above.

Page 1 of 4 (50 items) 1234