In the world of Metro style apps, your apps are the focus of the whole experience. For this experience to be delightful to your users, you want to ensure that your apps are well tested and represent a high level of quality. Producing a Metro style app involves four main stages: design, development, testing and finally distribution.

A number of previous blog posts focus on the design and development aspects of Metro style apps. In this post, we cover a few high level verification areas related to testing your apps.

The term testing is overloaded, meaning different things to different people, especially in the context of the software development lifecycle. It also involves a diverse set of activities that can be executed at different stages of software development. These activities include unit testing, functional testing, usability testing, scenario testing, and a few others. Any one of these activities represents only one piece of the testing puzzle and doesn’t cover all the necessary verification that you need to put an app through to make sure it is of high quality. In this post, we don’t distinguish between the specific types of verification. Instead, we provide a broad set of high-level verification areas and exercises that will help you ensure that your app represents a high level of quality. In addition to these verification areas, you can also verify the functionality of your app during development using Visual Studio for debugging and testing, and the Visual Studio Unit Test Tools. In the coming weeks, we plan to do more blog posts that will dive deeper into specifics of testing Metro style apps.

As we saw in the “managing app lifecycle” blog post, a Metro style app can transition between various runtime states. Based on this app lifecycle, we like to think of these key areas of verification:

  • Activated: verify the app through several activation paths
  • Running: verify the app at runtime
  • Suspended: verify the app suspends correctly
  • Resumed: verify the app resumes correctly
  • Terminated: verify the app terminates
  • Not running: verify behavior when your app is not running

Let’s dive deeper into the key verification areas during each of these areas.

Verify activation behavior of the app

To test activation we need to go through the various contexts in which users can launch your app. It is important to verify each activation scenario your app supports. Here are the common contexts in which users can activate your app:

  • Tile: The launch of your app through the tile is very likely the first interaction that users have with your app. Verify that your app follows the appropriate app launch design pattern.
  • Toast notifications: If your app supports toast notifications, verify that your app activates functions as expected when you click an incoming notification that your app is meant to handle.
  • Search: If your app supports Search, verify that your app activates and that you can complete a search in your app. If your app supports displaying search suggestions, verify that it functions as expected.
  • Share target: If your app supports being a Share target, verify that your app activates and posts the content shared from another app for the supported types of content.
  • Protocol: If your app registers for a protocol, verify that you can activate your app by launching various types of URLs corresponding to this protocol.
  • File association: If your app registers to handle a file type, verify that you can activate your app by launching files of the corresponding file type.
  • File pickers/contact picker: If your app is registered as a file open/save picker or as a contact picker, verify that you can activate your app in those contexts.

Verify runtime behavior of the app

Verifying the functionality of a Metro style app is a combination of verifying the common characteristics that apply to all apps, and verifying the app specific functionality. In addition to the key verification scenarios we talk about in a moment, you must also verify the app-specific functionality. This involves understanding the key scenarios targeted by your app and verifying that these scenarios work as expected.

Tiles

The app tile is a core part of your app, and possibly the most frequently seen part of your app. The following aspects of your tile behavior are important to verify:

  • If your app supports live tiles, verify its behavior while your app is running. Verify that the tile updates are consistent with the state displayed within your app. For example, a photo app’s tile could show photos from the most recently added album.
  • If your app supports secondary tiles, verify that they display the expected info. For example, a weather app can support pinning specific cities as secondary tiles. You need to also verify that clicking the secondary tile results in the app launching and taking you to the expected context within the app.
  • If your app supports notification badges in the tiles, verify that any badge update shows as expected, and that it is consistent with the corresponding state displayed within your app. For example, a badge related to the count of unread mail in a mail app must get cleared when the app runs to let the user know that they no longer have any unread mail.

Toast notifications

If your app supports toast notifications, verify that the toasts trigger and display correctly. Also verify that clicking the toast notification results in the app launching and taking you to the expected context within the app.

Contracts and extensions

We looked at a few contracts earlier as part of verifying the activation behavior of your app. Let’s look at a few additional areas related to verifying contracts at runtime:

  • Share source: If your app supports being a sharing source, verify that you can use the Share charm to share content from your app (for the supported types of content) to other target apps on the machine.
  • Play to: If your app supports the Play To contract, verify that you can use the Devices charm to play media from your app on a remote device.
  • Settings: If your app supports the Settings contract, verify that you can view and modify your app specific settings. Also, verify that the app persists the modified settings across termination/re-launch.

Windows App Certification Kit

The Windows App Certification Kit allows you to verify that your app meets the Windows Store submission criteria. This kit includes automated tests to verify the usage of supported APIs, testing for crashes/hangs, verification of the application manifest, and launch/suspension time performance. We recommend that you run this kit early and often. We’ve experienced a number of situations where the app didn’t make it to the Store by the deadline because the app owner waited till the last minute to run the certification kit against the app. If you run the kit early and often you can discover and fix any app issues quickly and submit your app to the Store. The Windows App Certification Kit documentation has details on the types of tests the kit does and additional tips on the configurations under which to run the kit.

Snapped and fill views

Users can run your Metro style app under various view states. In addition to the full screen view, users can use your app in the snapped and fill views. It is important to verify that your app handles these view state transitions correctly.

  • Verify that your app supports the snapped and fill views. Verify that your app maintains the context when it gets snapped or unsnapped.
  • Verify that snapping and unsnapping doesn’t destroy the user's work or state.
  • Verify that the user can interact with your app while it is snapped.

Rotation

Users can rotate and flip their tablets and monitors. If your app supports the portrait orientation in addition to the landscape orientation, verify that your app can handle the orientation changes. You can use the Visual Studio simulator for Metro style apps to test your app in both modes. So, you don’t necessarily need a tablet device to test this scenario.

Touch

Windows 8 enables apps to have a number of new touch experiences. You can use the Visual Studio simulator for Metro style apps to test the touch interactions that your app supports. So, you don’t necessarily need dedicated touch hardware to verify these interactions.

  • Verify that all the controls used in the app are standard basic controls that are touch-friendly. Depending on your programming language, use either the XAML controls or the JavaScript controls.
  • Verify that your Metro style app follows the standard Touch interaction design.
  • Verify that your app provides visual feedback for the user actions.
  • Verify that your app avoids dragging actions from the edges of the screen to avoid conflicting with the Windows OS gestures.
  • If the app supports any text input controls, verify that the soft keyboard works as expected. Verify that the area with the keyboard focus shifts up into view when the soft keyboard is displayed.

Playing audio

Windows 8 supports a model for Metro style apps to play audio, including playing in the background. Here’s what we recommend for testing audio:

  • If your app plays audio, verify that it handles the sound level notifications appropriately when a user switches between your app and other types of audio apps. For example, if your app is a game, you might want to pause the game when it receives a “Muted” notification. For details, see Audio playback in a Metro style app.
  • If your app supports playing audio in the background, verify that your app continues playing audio in the background without getting suspended. Verify that your app continues to play audio when the user locks the machine.
  • If your app registers for the transport controls to control playback of music, verify that it works as expected.

Different screen sizes

Windows 8 runs on a variety of screen sizes, from a small screen on a tablet, to a medium laptop screen all the way up to a large desktop screen. Therefore it is important to test your app on different screen sizes. You can use the Visual Studio simulator for Metro style apps to test your app on a variety of screen sizes and pixel densities.

  • On resolutions between 1024x768 and 2560x1440, verify that your app’s layouts are not truncated and that content appears where expected.
  • Verify that images are crisp when scaled to pixel density.

Data roaming

Windows 8 enables roaming settings across multiple PCs, including the roaming state persisted by your Metro style apps. To test this:

  • Verify that the app context roams as expected across multiple PCs. For example, this could be the last completed level in a game, the most recently read page in a book, or the last watched position of a video in your app. Verify that the context the app set on one PC is reflected when you use the app on another PC.
  • Verify that the applicable user preferences for your app roam as expected across multiple PCs. For example, this could be the user’s unit preference for the temperature in a Weather app. In this example, verify that the user’s unit preference set on one PC is reflected when you use the app on another PC.

Devices and sensors

If your Metro style app supports devices or sensors, you need to verify these areas.

  • If your app supports tailoring the view based on the geolocation, verify that it works as expected. You can use the Visual Studio simulator for Metro style apps to test this as it supports an option to set the current location.
  • If your app supports responding to user-initiated device connections like a camera or USB drive, verify that your app correctly performs the action corresponding to that event. For example, this could be verifying that users can use your app to copy all the images from a camera. Also, verify that your app handles device disconnects gracefully. For example, this involves verifying how your app behaves if the user disconnects the camera while your app is copying images from the camera.
  • Users can dock certain types of machines or devices to a docking station. Verify that your app behaves as expected both when the machine is docked and undocked.

Performance

Users expect apps to respond immediately to inputs. So it is critical to verify the performance characteristics of your app. As Windows 8 can run on a variety of hardware, consider installing your app on a low end machine to measure its performance characteristics. The post How to improve performance in your Metro style app has details on how to prepare a machine for measurements.

Responsiveness
  • For the key scenarios supported by your app, understand the expected interaction classes and verify that the app meets the goals for that particular interaction class.
  • Verify that your app remains responsive to the user.
  • Verify that your app provides feedback to the user during any long-running activities, such as an asynchronous network call.
Being a good citizen in the app ecosystem

As the post How to improve performance in your Metro style app explains, it is important to verify that your app is a good citizen in the app ecosystem. This involves contributing positively to the user’s overall experience (whether your app is in the foreground or not). In addition to verifying the areas that post lists, here are a few additional areas to verify:

  • When in idle state, if you expect your app to not be using CPU, verify that the app doesn’t consume any CPU.
  • Users could be on various types of networks including metered networks. If your app connects to the Internet, verify that it adapts its behavior for metered networks when sending and receiving data.

Error handling

As the saying goes, anything that can go wrong, will :-). Seriously, when a condition your app depends on changes, it can impact your app’s behavior. For example, your app needs to handle network connectivity not being available and decide on how to behave in this situation.

  • Verify that your app handles error conditions appropriately. As in our example, network and/or device connectivity is a great way to trigger error paths in your app. Verify that your app follows the guidelines for managing connection state changes.
  • Verify that the error handling is not intrusive and that your app displays only the critical errors to the user.
  • Verify that your app provides actionable error messages to the user. See the guidelines for actionable error messages.

Reliability

Users expect apps to work reliably and consistently. It is important to verify that your app is highly reliable.

  • Verify that your app runs reliably across varied conditions and that it doesn’t crash because of unhandled exceptions.
  • Verify that your app doesn’t leak memory or any other resources. To detect memory leaks in a Metro style app, you can use the same methods that you normally use with other Windows applications.

After you publish your app to the Windows Store, you can use the Windows Store developer portal to review your app’s usage and quality data.  For example, you can view the data about exceptions, crashes, and unresponsiveness issues that your users encountered.  Consider using this data to fix the top crashes and hangs in your app.

Accessibility

Windows 8 is designed for a broad spectrum of people and is accessible to everyone regardless of their abilities. If your Metro style app meets a baseline of accessibility, it can contribute tremendously to improving the accessibility ecosystem. If your app supports accessibility, verify that it meets the criteria described in Testing your app for accessibility.

Globalization and localization

If your Metro style app is designed to be global, it can be localized for a variety of regions and markets. Metro style apps can use the configured language and display formats (date, time, currency, and numbers) to tailor their content. For example, if a user configures Japanese as the primary language on an English installation of Windows, and then launches a Metro style Stocks app, the app can display all the related news articles in Japanese and additionally show info about the stocks trading in the selected region (e.g. Tokyo Stock Exchange).

If your app is localized, verify that it works for different locales and display languages.

  • For the languages your app supports, use the Language settings in the Control Panel to add each of the specific languages as a primary language, and verify that your app tailors the content appropriate for that language.
  • Verify that changing the language when your app is running or suspended works as expected.
  • For the languages your app supports, verify that the user interface strings appear as expected and don’t get truncated.
  • Verify that any text shown in the app’s images is in the selected display language.

Processor architectures

A Metro style app can be architecture-neutral or it could target a specific architecture.

If your app is architecture neutral, it can be tested on any chosen architecture.

If it is architecture specific, it needs to be tested on all the supported architectures.

  • If your app is targeted specifically for Windows RT (ARM), verify its functionality on a Windows RT PC. You can find additional info on developing and testing on ARM in the blog post What you need to know about developing for Windows on ARM (WOA).
  • If your app targets x86 and doesn’t separately provide a x64 package, verify that your app functions correctly on both x86 and x64 systems. This is because x86 apps can run on x86 (32-bit Windows) or x64 (64-bit Windows) systems. Windows allows apps targeted for x86 to run on x64 systems.

Verify suspension behavior of the app

Windows suspends your app when it is not in the foreground. In this way, Windows gives the foreground app better use of system resources and ensures that apps in the background can’t drain the user’s battery. For example, during suspension, a Stocks app can persist info about the last stock that the user was viewing and the time range for a stock chart.

  • Verify that your app reduces memory usage while it is in a suspended state. To view the suspension status in Task Manager, click More details in Task Manager to see all available options, and then click View > Status values > Show suspended status.
  • If your app supports uploading or downloading of data in the background, verify that data continues to be transferred even when the app is suspended.

You can use Visual Studio to simulate the suspension/resumption of an app. To do this, select View > Toolbars > Debug Location.

Verify resumption behavior of the app

When your app comes back to the foreground, Windows resumes your app, and it continues from the state that it was in when Windows suspended it. It is important to verify that your app resumes gracefully.

  • Your app could be resuming after being suspended for a long time. During this time, the displayed content and any network connections could have become stale. Verify that your app resumes and that it provides a visual indication about the state of its content.
  • After your app resumes, verify that it can refresh its content and can reconnect to any network resources, as required. For example, after a Stocks app resumes, it can provide a visual indication of the time the data was last updated, and then go on to update the cached stock data so that the user sees the latest stock prices.

Verify termination behavior of the app

Windows can terminate a suspended app when the system is running low on resources. Additionally, your app can also get terminated when the user manually closes it, or signs out of Windows. Depending on how the app gets terminated, it is important to verify that your app restores the right app data when it is launched after a termination.

  • When launching an app after Windows terminated it, verify that your app restores the context and state as it was before getting suspended and/or terminated.  For example, this could be the last viewed stock info and the selected duration of the chart that was saved when the app was suspended.
  • When launching an app after manually closing it (by swiping from the top to the bottom of the screen), verify that your app starts with the default data.
  • When launching an app after an unexpected termination such as a crash, verify that your app is still able to launch and that it loads the default data. To cause an unexpected termination for your app, in the Task Manager right click your app > Go to details > right click > end task.

For more details, see Managing app lifecycle so your apps feel "always alive".

Verify app behavior when not running

A Metro style app can surface info to the user even when it is not running. So, it is important to verify that the below scenarios work as expected.

Tiles

Here are the main areas to verify when your app is not running:

  • Verify that the default tile displays as expected. Verify that the tile size, usage, and notifications follow the guidelines in Guidelines and checklist for tiles.
  • If your app supports scheduled notifications or periodic/push notifications from the cloud, verify that its tile gets updated even when your app is not running. Examples of this are a weather app that periodically updates its tile with the current temperature, and a sports app that updates its tile with the latest scores pushed from the cloud.
  • A live tile is one of the best ways to entice users back to your app. Validate that the info in your live tile shows the best of what’s going on inside of your app.

Toast notifications

If your app supports toast notifications even when it’s not running (using scheduled notifications or push notifications), verify that the toast notifications trigger and display as expected even when the app is not running. Examples of this are a calendar app that schedules meeting reminders as toast notifications and a news app that displays breaking news as toast notifications.

Background tasks

If your app supports background tasks, verify the corresponding behavior.

  • If your app is capable of showing lock screen notifications (e.g. a mail app or a VoIP app that always has to stay up to date), verify that the info surfaced by your app stays up to date even when the PC is not used.
  • If your app can perform background tasks on certain triggers (e.g. a Photos app that processes pictures from the Photos library when on AC power), verify that your app performs that task correctly when the appropriate conditions are met.

For details, see the Background tasks blog post.

Conclusion

Testing your Metro style app and verifying its quality is a critical step to ensure that your users have a great experience with your app. In this blog post, we covered a few key areas related to verifying your Metro style apps including verifying the behavior of your app when it is launched in different contexts, verifying the functionality of your app, and verifying the behavior of your app as it transitions through various states of the Metro style app lifecycle.

In the real world, a user could combine many interactions that impact your app.  For example, a user could, in quick succession, snap your app to the Fill view, rotate the device, and adjust the screen resolution.  So, in addition to testing the app behavior for a specific scenario in isolation, bring out your creative juices and exercise your app to account for such complex interactions.

We hope you find this to be a good starting point to think about how to verify the functionality of your apps and have them represent a high level of quality. In the coming weeks, we plan to do more blog posts that will dive deeper into specific topics pertaining to testing Metro style apps.

Thanks!

-- J. Kalyana Sundaram, Senior SDET, Windows

-- Ashwin Needamangala, Principal Test Lead, Windows

-- Mete Goktepe, Principal Test Manager, Windows

Special thanks to Tom White, Patrik Lundberg, Tom Baxter, Vik Kumar, Rui Sun and Yves Neyrand for their help and contributions to this post.