In the blog post for testing Windows Store apps, we’ve primarily covered what to test your apps for. While you can manually execute the verification areas we’ve discussed in the previous post, it is more efficient to automate some of these verification areas. Automating the verification of your app has many benefits. Automated testing saves time and money that you’d otherwise spend doing manual testing. Once created, automated tests can be run over and over again at minimal cost and are much faster than manual tests. This helps ensure that the quality of your app stays high and the cost stays low with every new release of the app. Automated testing also improves the accuracy of the verification because even the most conscientious tester might make mistakes during tedious manual testing.

In this post, we provide some tips and techniques on automating the testing of Windows 8 apps. Keep in mind that automated testing is a powerful technique and it requires some initial investment to reap the full rewards. The advice and examples in this post are meant to get you started, but you’ll need to build and maintain your own infrastructure on top of them. If you are looking for some lighter weight testing techniques, check out the recent blog on testing apps with Visual Studio.

A typical app test automation workflow involves these steps:

  • Installation: Automate the installation of your app.
  • Activation: Automate the activation of your app.
  • Runtime: Automate your app when it is running.
  • Lifecycle state: Automate your app being suspended, resumed, and terminated.
  • Uninstallation: Automate the uninstallation of your app.

Let’s cover each of these steps in detail and the automation tools/technologies to use for each of them.

Note about testing on Windows RT

Before we dive into the automation topics, we wanted to make a quick note about testing on Windows RT. You can’t build or port x86/64 processes to run on Windows RT. Thus, all the tools and technologies discussed in the rest of this post do not apply for testing on Windows RT. We recommend that you use Visual Studio to test on Windows RT.

Automating the installation of your app

To test your app, you first need to install it on your test machine. Using Visual Studio to share an app package locally is the recommended approach for creating your app package and installing it on the test machine. In this option Visual Studio creates a folder that contains all the relevant files along with a PowerShell script that installs the appropriate certificate and license, the dependency packages, and the app package itself. You have to package your app manually, but the installation is PowerShell based and can be automated. Here is how the installation goes.

Step 1:

Enable PowerShell script execution. For security reasons, the default PowerShell execution policy restricts the execution of PowerShell scripts, so you need to override this policy. This step is manual because it requires user interaction. Luckily, you need to do this one time per machine. Execute this command from an elevated PowerShell window to enable PowerShell script execution.

PS C:\> Set-ExecutionPolicy AllSigned
 Powershell_admin
Figure 1: Enable PowerShell script execution

Step 2:

Copy the app package folder created by Visual Studio to the test machine and execute the Add-AppDevPackage PowerShell script from a PowerShell window. Use this command:

PS C:\JSGrid1_1.0.0.0_AnyCPU_Debug_Test> .\Add-AppDevPackage.ps1
Windows Powershell
Figure 2: Run the Add-AppDevPackage script

Step 3:

Acquire a developer license. This step is manual because it requires user interaction. But you need to do this one time per machine for the validity period of the license. If your test machine already has the developer license, you can skip this step.

Accept the UAC prompt, and follow the instructions for acquiring a developer license. The below screenshots illustrate what you will see.

This first screen shot requires that you agree to the licensing terms for installing a developer license. Click I Agree if you do to proceed.

Powershell_devlic

Figure 3: Accept the developer license acquisition prompt

In the Microsoft account dialog box, type your Microsoft account. If you don’t have a Microsoft account, click Sign up to create one.

msaccount_signin

Figure 4: Sign in with Microsoft account credentials

You receive a confirmation that your developer license has been created and the date that it expires.

devlic_approve

Figure 5: Developer License acquired successfully

The next screen shot indicates that you got your developer license and your package successfully installed.

PS_packagesuccess

Figure 6: Installation Complete

Automating the activation of your app

Your app package is now installed on the test machine and you are ready to launch your app. You can automate the activation of your app with the IApplicationActivationManager interface. This API is available as part of the Windows SDK which is installed by default with Visual Studio 2012. Use the IApplicationActivationManager::ActivateApplication method to launch your app. The code snippet here illustrates the usage of this method.

#include "stdafx.h"
#include <shlobj.h>
#include <stdio.h>
#include <shobjidl.h>
#include <objbase.h>
#include <atlbase.h>
#include <string>

/*++

Routine Description:

This routine launches your app using IApplicationActivationManager.

Arguments:

strAppUserModelID - AppUserModelID of the app to launch.
pdwProcessId - Output argument that receives the process id of the launched app.

Return value:

HRESULT indicating success/failure

--*/
HRESULT LaunchApp(const std::wstring& strAppUserModelId, PDWORD pdwProcessId)
{
CComPtr<IApplicationActivationManager> spAppActivationManager;
HRESULT hrResult = E_INVALIDARG;
if (!strAppUserModelId.empty())
{
// Instantiate IApplicationActivationManager
hrResult = CoCreateInstance(CLSID_ApplicationActivationManager,
NULL,
CLSCTX_LOCAL_SERVER,
IID_IApplicationActivationManager,
(LPVOID*)&spAppActivationManager);

if (SUCCEEDED(hrResult))
{
// This call ensures that the app is launched as the foreground window
hrResult = CoAllowSetForegroundWindow(spAppActivationManager, NULL);

// Launch the app
if (SUCCEEDED(hrResult))
{
hrResult = spAppActivationManager->ActivateApplication(strAppUserModelId.c_str(),
NULL,
AO_NONE,
pdwProcessId);
}
}
}

return hrResult;
}

int _tmain(int argc, _TCHAR* argv[])
{
HRESULT hrResult = S_OK;
if (SUCCEEDED(CoInitializeEx(NULL, COINIT_APARTMENTTHREADED)))
{
if (argc == 2)
{
DWORD dwProcessId = 0;
++argv;
hrResult = LaunchApp(*argv, &dwProcessId);
}
else
{
hrResult = E_INVALIDARG;
}

CoUninitialize();
}

return hrResult;
}

You can compile this snippet and use it like this:

C:\>Win8AppLaunch.exe Microsoft.BingNews_8wekyb3d8bbwe!AppexNews

In this code snippet, the call to CoAllowSetForegroundWindow is crucial. Without this call, the app will be launched but will not be brought to the foreground. From experience, this has tripped up a number of people trying to write a launch tool.

Finally, a quick note about AppUserModelId. In this approach, the input required to launch your app is the AppUserModelId of your app. The AppUserModelId functions as a unique identifier of your app. To retrieve it, we recommend that you use PowerShell. Here’s a PowerShell script that illustrates how you can retrieve the AppUserModelId for all installed apps on your machine.

$installedapps = get-AppxPackage
foreach ($app in $installedapps)
{
foreach ($id in (Get-AppxPackageManifest $app).package.applications.application.id)
{
$app.packagefamilyname + "!" + $id
}
}

You can also use IAppxManifestReader to enumerate the apps in the package and obtain the AppUserModelId using the IAppxManifestApplication::GetAppUserModelId method. But if your primary focus is a single app that you are testing, using the registry is far simpler than writing a tool that reads your app’s manifest.

Automating the core functionality of your app

Your app has now been installed on the test machine and your automation can launch it. The next step involves automating the testing of the core functionality of your app. You can achieve this by a combination of unit testing and automating the app via its user interface using UI automation.

Unit testing and UI automation are complementary techniques that when used together can result in a broadly tested, higher quality app. Unit testing allows you to automate the core business logic within your app. UI automated testing allows you to verify your app’s functionality by simulating the usage of its user interface. A combination of these approaches enables you to get broader test coverage of your app.

Let’s go over some tools and techniques for each of these approaches.

Unit testing

Unit testing is a powerful technique for verifying the core functionality of your app. Visual Studio 2012 supports building unit tests for apps written using C# or C++. For more info on creating and running unit tests from Visual Studio 2012 see Creating and running unit tests. If you are already familiar with other unit testing frameworks, you can also continue to use those to unit test your Windows Store app.

UI automation

Unit testing can be useful for testing the inner workings of your app, but it doesn’t exercise the user interface of your app. We recommend UI automation (UIA) for verifying an app’s functionality through its user interface.

In the Windows 8 security model, apps don’t have the privileges required to be a UI automation client. But you can write a desktop app that acts as an automation client with your app as the target. To do this, your desktop automation client app needs to be built with UIAccess permissions as outlined in UI Automation Security Overview.

The Windows SDK includes some tools that serve as good examples of UI automation clients: Inspect.exe and AccEvent.exe. Inspect allows you to inspect the UI automation tree of both kinds of apps. AccEvent listens for UIA events. On a machine that has the Windows SDK installed, these tools are usually under %ProgramFiles(x86)%\Windows Kits\8.0\bin\<architecture>. Here are both these tools in action.

inspect.exe

Figure 7: Inspect tool running against the Bing News app

accevent_tool

Figure 8: AccEvent tool running against the Bing Sports app

Similar to the approach used by these tools, you can use UIA to write a desktop client app that automates your app. The Building UI Automation client applications in C++ and C# blog post is a great starting point for learning about how to build a UIA client. This blog post speaks about traditional UI automation, but the same techniques will work for any app.

The UI Automation Document Content Client sample illustrates how you can use UIA control patterns to retrieve various types of content from the target app’s window, such as headers, comments, or the current selection. The Touch Injection sample illustrates how you can use the Touch Injection APIs to simulate touch for the verification of your app.

As mentioned earlier, Unit testing and UI automation are complementary techniques that allow you to automate both the core business logic within your app and its functionality when exercised from the user interface. Use both these techniques to debug your app.

Automating lifecycle state of your app

As outlined in the managing app lifecycle blog post, an app can transition between various runtime states. From a testing perspective, it is important to verify your app in all these states. Automating the transition into these states is possible using a tool (PLMDebug) that ships with the Debugging Tools for Windows, a ready-to-use command-line tool that you can use to automate the transition of your app between lifecycle states.

If PLMDebug is not sufficient for your needs, you can use the IPackageDebugSettings interface to implement your own tool that can change the lifecycle state of your app. This API is available as part of the Windows SDK, which is installed by default with Visual Studio. The code illustrates how to use the IPackageDebugSettings API to change the lifecycle state of your app to verify that your app works as expected during these state transitions.

#include "stdafx.h"
#include <stdio.h>
#include <shobjidl.h>
#include <objbase.h>
#include <atlbase.h>
#include <string>

/*++

Routine Description:

This routine changes the lifecycle state of a Windows Store app depending on the input argument.

Arguments:

strPackageFullName - Package Full Name of the Windows Store app
strOperation - Operation to take (/enabledebug, /suspend, /resume, /terminate, /cleanTerminate,
/disabledebug)

Return Value:

HRESULT indicating success/failure

--*/
HRESULT ChangeLifecycleState(const std::wstring& strPackageFullName, const std::wstring& strOperation)
{
CComPtr<IPackageDebugSettings> spPackageDebugSettings;
HRESULT hrResult = E_INVALIDARG;

if (!strPackageFullName.empty() && !strOperation.empty())
{
// Instantiate IPackageDebugSettings
hrResult = CoCreateInstance(CLSID_PackageDebugSettings,
NULL,
CLSCTX_INPROC_SERVER,
IID_PPV_ARGS(&spPackageDebugSettings));

// Depending on the operation specified as the command line arg, change the lifecycle state
// of the app
if (SUCCEEDED(hrResult))
{
if (_wcsicmp(strOperation.c_str(), L"/enableDebug") == 0)
{
// Increment debug ref count on the app package - you must do this before you can
// suspend/resume/terminate
hrResult = spPackageDebugSettings->EnableDebugging(strPackageFullName.c_str(),
NULL,
NULL);
}
else if (_wcsicmp(strOperation.c_str(), L"/suspend") == 0)
{
// Asynchronously suspend the app
hrResult = spPackageDebugSettings->Suspend(strPackageFullName.c_str());
}
else if (_wcsicmp(strOperation.c_str(), L"/resume") == 0)
{
// Resume the app
hrResult = spPackageDebugSettings->Resume(strPackageFullName.c_str());
}
else if (_wcsicmp(strOperation.c_str(), L"/terminate") == 0)
{
// Terminate the app
hrResult = spPackageDebugSettings->TerminateAllProcesses(strPackageFullName.c_str());
}
else if (_wcsicmp(strOperation.c_str(), L"/cleanTerminate") == 0)
{
// Clean terminate the app - suspend, then terminate
hrResult = spPackageDebugSettings->StartServicing(strPackageFullName.c_str());
if (SUCCEEDED(hrResult))
{
hrResult = spPackageDebugSettings->StopServicing(strPackageFullName.c_str());
}
}
else if (_wcsicmp(strOperation.c_str(), L"/disableDebug") == 0)
{
// Decrement debug ref count on the app package
hrResult = spPackageDebugSettings->DisableDebugging(strPackageFullName.c_str());
}
else
{
hrResult = E_INVALIDARG;
}
}
}

return hrResult;
}

int _tmain(int argc, _TCHAR* argv[])
{
HRESULT hrResult = S_OK;
if (SUCCEEDED(CoInitializeEx(NULL, COINIT_APARTMENTTHREADED)))
{
if (argc == 3)
{
std::wstring strOperation(argv[1]);
std::wstring strPackageFullName(argv[2]);
hrResult = ChangeLifecycleState(strPackageFullName, strOperation);
}
else
{
hrResult = E_INVALIDARG;
}

CoUninitialize();
}

return hrResult;
}
If you compile this snippet as LifecycleManager.exe, here’s how to use it.
C:\>Win8AppLifecycleManager.exe /enableDebug Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe

C:\> Win8AppLifecycleManager.exe /suspend Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe

C:\> Win8AppLifecycleManager.exe /resume Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe

C:\> Win8AppLifecycleManager.exe /terminate Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe

C:\> Win8AppLifecycleManager.exe /cleanTerminate Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe

C:\> Win8AppLifecycleManager.exe /disableDebug Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe
 

Finally, a quick note about PackageFullName. All approaches we looked at for managing lifecycle state require the PackageFullName as the input argument by which to identify the app. To retrieve the PackageFullName we recommend that you use the Get-AppxPackage PowerShell cmdlet as the next example demonstrates (output trimmed for readability).

PS C:\> Get-AppxPackage

Name : Microsoft.BingNews
Publisher : CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Architecture : X64
ResourceId :
Version : 1.2.0.98
PackageFullName : Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe ←
InstallLocation : C:\Program Files\WindowsApps\Microsoft.BingNews_1.2.0.98_x64__8wekyb3d8bbwe
IsFramework : False
PackageFamilyName : Microsoft.BingNews_8wekyb3d8bbwe
PublisherId : 8wekyb3d8bbwe

Automating uninstallation of your app

For uninstalling your app, we recommend that you use the PowerShell cmdlets, specifically, Remove-AppxPackage. Here is an example (output trimmed for readability).

PS C:\> Get-AppxPackage

Name : Microsoft.SDKSamples.ListViewEssentials.JS
Publisher : CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Architecture : Neutral
ResourceId :
Version : 1.0.0.0
PackageFullName : Microsoft.SDKSamples.ListViewEssentials.JS_1.0.0.0_neutral__8wekyb3d8bbwe ←
InstallLocation : C:\Users\user1\Downloads\Samples\Controls_ListViewBasic\JS\bin\Debug\AppX
IsFramework : False
PackageFamilyName : Microsoft.SDKSamples.ListViewEssentials.JS_8wekyb3d8bbwe
PublisherId : 8wekyb3d8bbwe

PS C:\> Remove-AppxPackage Microsoft.SDKSamples.ListViewEssentials.JS_1.0.0.0_neutral__8wekyb3d8bbwe

Conclusion

In this blog post, we’ve covered a number of tips, tools and techniques for automating the testing of your apps. Automated testing can be a cost effective way to increase the level of verification of your app and ensure that the quality never drops. That said, it is important to remember that manual testing also plays a key role in verification. It provides the human element to the verification of your app. As such, a combination of automated and manual testing will give you a well-rounded approach to testing your app.

-- Ashwin Needamangala, Principal Test Lead, Windows

Special thanks to Mete Goktepe, J. Kalyana Sundaram, Ben Betz, Will Wei, Chris Edmonds, Craig Campbell and Jake Sabulsky for their help and contributions to this post.