Building Windows 8 blog
Windows Store for developers blog
Visual Studio blog
The Windows blog
Inside SkyDrive blog
Download Windows 8 Release Preview
Windows Dev Center
Follow us @WinDevs
The //build/ conference
In the world of Metro style apps, your apps are the focus of the whole experience. For this experience to be delightful to your users, you want to ensure that your apps are well tested and represent a high level of quality. Producing a Metro style app involves four main stages: design, development, testing and finally distribution.
A number of previous blog posts focus on the design and development aspects of Metro style apps. In this post, we cover a few high level verification areas related to testing your apps.
The term testing is overloaded, meaning different things to different people, especially in the context of the software development lifecycle. It also involves a diverse set of activities that can be executed at different stages of software development. These activities include unit testing, functional testing, usability testing, scenario testing, and a few others. Any one of these activities represents only one piece of the testing puzzle and doesn’t cover all the necessary verification that you need to put an app through to make sure it is of high quality. In this post, we don’t distinguish between the specific types of verification. Instead, we provide a broad set of high-level verification areas and exercises that will help you ensure that your app represents a high level of quality. In addition to these verification areas, you can also verify the functionality of your app during development using Visual Studio for debugging and testing, and the Visual Studio Unit Test Tools. In the coming weeks, we plan to do more blog posts that will dive deeper into specifics of testing Metro style apps.
As we saw in the “managing app lifecycle” blog post, a Metro style app can transition between various runtime states. Based on this app lifecycle, we like to think of these key areas of verification:
Let’s dive deeper into the key verification areas during each of these areas.
To test activation we need to go through the various contexts in which users can launch your app. It is important to verify each activation scenario your app supports. Here are the common contexts in which users can activate your app:
Verifying the functionality of a Metro style app is a combination of verifying the common characteristics that apply to all apps, and verifying the app specific functionality. In addition to the key verification scenarios we talk about in a moment, you must also verify the app-specific functionality. This involves understanding the key scenarios targeted by your app and verifying that these scenarios work as expected.
The app tile is a core part of your app, and possibly the most frequently seen part of your app. The following aspects of your tile behavior are important to verify:
If your app supports toast notifications, verify that the toasts trigger and display correctly. Also verify that clicking the toast notification results in the app launching and taking you to the expected context within the app.
We looked at a few contracts earlier as part of verifying the activation behavior of your app. Let’s look at a few additional areas related to verifying contracts at runtime:
The Windows App Certification Kit allows you to verify that your app meets the Windows Store submission criteria. This kit includes automated tests to verify the usage of supported APIs, testing for crashes/hangs, verification of the application manifest, and launch/suspension time performance. We recommend that you run this kit early and often. We’ve experienced a number of situations where the app didn’t make it to the Store by the deadline because the app owner waited till the last minute to run the certification kit against the app. If you run the kit early and often you can discover and fix any app issues quickly and submit your app to the Store. The Windows App Certification Kit documentation has details on the types of tests the kit does and additional tips on the configurations under which to run the kit.
Users can run your Metro style app under various view states. In addition to the full screen view, users can use your app in the snapped and fill views. It is important to verify that your app handles these view state transitions correctly.
Users can rotate and flip their tablets and monitors. If your app supports the portrait orientation in addition to the landscape orientation, verify that your app can handle the orientation changes. You can use the Visual Studio simulator for Metro style apps to test your app in both modes. So, you don’t necessarily need a tablet device to test this scenario.
Windows 8 enables apps to have a number of new touch experiences. You can use the Visual Studio simulator for Metro style apps to test the touch interactions that your app supports. So, you don’t necessarily need dedicated touch hardware to verify these interactions.
Windows 8 supports a model for Metro style apps to play audio, including playing in the background. Here’s what we recommend for testing audio:
Windows 8 runs on a variety of screen sizes, from a small screen on a tablet, to a medium laptop screen all the way up to a large desktop screen. Therefore it is important to test your app on different screen sizes. You can use the Visual Studio simulator for Metro style apps to test your app on a variety of screen sizes and pixel densities.
Windows 8 enables roaming settings across multiple PCs, including the roaming state persisted by your Metro style apps. To test this:
If your Metro style app supports devices or sensors, you need to verify these areas.
Users expect apps to respond immediately to inputs. So it is critical to verify the performance characteristics of your app. As Windows 8 can run on a variety of hardware, consider installing your app on a low end machine to measure its performance characteristics. The post How to improve performance in your Metro style app has details on how to prepare a machine for measurements.
As the post How to improve performance in your Metro style app explains, it is important to verify that your app is a good citizen in the app ecosystem. This involves contributing positively to the user’s overall experience (whether your app is in the foreground or not). In addition to verifying the areas that post lists, here are a few additional areas to verify:
As the saying goes, anything that can go wrong, will :-). Seriously, when a condition your app depends on changes, it can impact your app’s behavior. For example, your app needs to handle network connectivity not being available and decide on how to behave in this situation.
Users expect apps to work reliably and consistently. It is important to verify that your app is highly reliable.
After you publish your app to the Windows Store, you can use the Windows Store developer portal to review your app’s usage and quality data. For example, you can view the data about exceptions, crashes, and unresponsiveness issues that your users encountered. Consider using this data to fix the top crashes and hangs in your app.
Windows 8 is designed for a broad spectrum of people and is accessible to everyone regardless of their abilities. If your Metro style app meets a baseline of accessibility, it can contribute tremendously to improving the accessibility ecosystem. If your app supports accessibility, verify that it meets the criteria described in Testing your app for accessibility.
If your Metro style app is designed to be global, it can be localized for a variety of regions and markets. Metro style apps can use the configured language and display formats (date, time, currency, and numbers) to tailor their content. For example, if a user configures Japanese as the primary language on an English installation of Windows, and then launches a Metro style Stocks app, the app can display all the related news articles in Japanese and additionally show info about the stocks trading in the selected region (e.g. Tokyo Stock Exchange).
If your app is localized, verify that it works for different locales and display languages.
A Metro style app can be architecture-neutral or it could target a specific architecture.
If your app is architecture neutral, it can be tested on any chosen architecture.
If it is architecture specific, it needs to be tested on all the supported architectures.
Windows suspends your app when it is not in the foreground. In this way, Windows gives the foreground app better use of system resources and ensures that apps in the background can’t drain the user’s battery. For example, during suspension, a Stocks app can persist info about the last stock that the user was viewing and the time range for a stock chart.
You can use Visual Studio to simulate the suspension/resumption of an app. To do this, select View > Toolbars > Debug Location.
When your app comes back to the foreground, Windows resumes your app, and it continues from the state that it was in when Windows suspended it. It is important to verify that your app resumes gracefully.
Windows can terminate a suspended app when the system is running low on resources. Additionally, your app can also get terminated when the user manually closes it, or signs out of Windows. Depending on how the app gets terminated, it is important to verify that your app restores the right app data when it is launched after a termination.
For more details, see Managing app lifecycle so your apps feel "always alive".
A Metro style app can surface info to the user even when it is not running. So, it is important to verify that the below scenarios work as expected.
Here are the main areas to verify when your app is not running:
If your app supports toast notifications even when it’s not running (using scheduled notifications or push notifications), verify that the toast notifications trigger and display as expected even when the app is not running. Examples of this are a calendar app that schedules meeting reminders as toast notifications and a news app that displays breaking news as toast notifications.
If your app supports background tasks, verify the corresponding behavior.
For details, see the Background tasks blog post.
Testing your Metro style app and verifying its quality is a critical step to ensure that your users have a great experience with your app. In this blog post, we covered a few key areas related to verifying your Metro style apps including verifying the behavior of your app when it is launched in different contexts, verifying the functionality of your app, and verifying the behavior of your app as it transitions through various states of the Metro style app lifecycle.
In the real world, a user could combine many interactions that impact your app. For example, a user could, in quick succession, snap your app to the Fill view, rotate the device, and adjust the screen resolution. So, in addition to testing the app behavior for a specific scenario in isolation, bring out your creative juices and exercise your app to account for such complex interactions.
We hope you find this to be a good starting point to think about how to verify the functionality of your apps and have them represent a high level of quality. In the coming weeks, we plan to do more blog posts that will dive deeper into specific topics pertaining to testing Metro style apps.
-- J. Kalyana Sundaram, Senior SDET, Windows
-- Ashwin Needamangala, Principal Test Lead, Windows
-- Mete Goktepe, Principal Test Manager, Windows
Special thanks to Tom White, Patrik Lundberg, Tom Baxter, Vik Kumar, Rui Sun and Yves Neyrand for their help and contributions to this post.