Learn to use Visual Studio, Visual Studio Online, Application Insights and Team
Foundation Server to decrease rework, increase transparency into your application and increase the rate at which
you can ship high quality software throughout the application lifecycle
Unit Testing is an important part of the application development life cycle for any platform. Visual Studio users no longer need to depend on 3rd party frameworks for unit tests on Windows Phone. The Windows Phone Unit Test framework delivered in CTP2 of Visual Studio Update 2 provides a consistent experience for developers to author, run and debug tests for Windows Phone 8 platform. This article walks you through the steps to start unit testing on Windows Phone.
Note: You need a 64 bit Windows 8 physical machine (with Hyper-V enabled) to run unit tests on Windows Phone Emulator.
One Framework – Multiple Platforms
You can now write Unit Tests for Windows Phone in C# by creating a new “Windows Phone Unit Test App” project under Visual C#/Windows Phone. Tests are executed on the target device by installing the Unit Test app on the phone.
Figure 1: New Windows Phone Unit Test Template
Whether you are writing classic unit tests, Windows Store unit tests, or Windows Phone unit tests – Visual Studio automatically adds all the necessary references and you are ready to start writing tests using the familiar MS Test framework. You no longer need to learn multiple frameworks to author unit tests for different platforms.
Figure 2: Author tests using MSTest framework
Running Tests from Visual Studio
The Phone UT framework is designed as an adapter on top of the extensible Visual Studio 2012 unit testing platform (similar to other adapters like nUnit, xUnit). With this, the whole Visual Studio Test Driven Development experience is now extended to Windows Phone. Tests are discovered and displayed in the familiar Test Explorer window. Right from within the IDE, users can group/filter tests using traits, turn on continuous test runner to view test results after each build, and debug or run selected (or failed) set of tests.
Figure 3: Phone Unit Tests discovered
Figure 4:Set Target Device
Visual Studio uses the target device set on the Run palette to install the unit test app, run tests, and return results. Supported targets include Emulators (only available when Hyper-V is enabled) and Windows Phone 8 devices physically connected to the machine. Developers can author, run, and debug tests on any of the target devices without leaving the IDE. While unit tests are running on the device, a 'Tests Running' screen will be shown.
Figure 5: Unit Tests running on device
Running Tests from Command Line
As I mentioned earlier, since Phone UT is designed as an adapter to unit testing platform, you can use vstest.console.exe to execute unit tests from command line. You can also choose to export the results to Visual Studio Test Results File using /Logger:trx.
Figure 6: Run tests from command line
Figure 7: Test Results from TRX logger
By default Emulator WVGA 512MB is the target device to execute tests, use run settings to change target device (Sample run settings attached).
Figure 8: Run Settings to set target device
FAQs and Known Issues
Any plans to bring this to WP7? FYI, this doesn't send a very positive message...
What's the point in creating and deploying test app to run unit tests? Usually tests cover some business logic or algorithms and are fine to run as any other test. Can you share some scenarios where this feature may be useful?
Thanks for bringing this much-needed feature to Visual Studio!
@andreycha: Theoretically you are right, but in reality just too many methods/code paths of a phone application depend in some way or another on namespaces that are only available on the phone/emulator. Except for the smallest algorithmic methods none of this code would be runnable and testable in VS directly.
Great post Vinay!
Congrats on getting your first one out!
@friederbluemle I see. So creating of test app is just a way to run that code in proper environment?
@Richard Szalay: Owing to architectural constraints, this remains a WP8 only feature.
@andreycha: Yes - ideal unit tests should mock all Phone APIs to be able to run tests in VS. In absence of that, the next best thing is to run tests on the device itself.
@Chuck: Thanks! :)
Trying this out... and couldn't get the test to run (after a long pause, the test runner stops and then leave the tests in "Not Run Tests" group). So I went to try the command line instead (what's up with it saying Windows Store apps instead of Phone, btw? :)), and it gives following error (am trying on device - I don't currently have Hyper-V to use emulators):
Starting test execution, please wait...
Warning: Using Isolation mode to run unit tests for Windows Store apps. Use /InI
solation switch to suppress this warning.
Warning: Diagnostic data adapters are not supported when executing unit tests fo
r Windows Store apps. Remove diagnostic data adapters settings from settings.
Error: Failed to initialize client proxy: could not connect to .
Error: The socket connection was aborted. This could be caused by an error proce
ssing your message or a receive timeout being exceeded by the remote host, or an
underlying network resource issue. Local socket timeout was '00:00:59.9990004'.
Can one let the windows phone unit tests run in a Team Foundation Build Server (maybe inside of vstest.console.exe)?
@Vinay.Krishna Any chance you could elaborate on those "architectural constraints"? Since the tests run on the phone, I assume you're just hosting an HTTP service to communicate the results. I've done this for the WP7 (wp7ci on Codeplex) so I know it's possible, albeit without the VS integration.
@Wiyono: Since you are running tests on a processor without SLAT capability, the Target defaults to Device. Can you set Platform to ARM in Build -> Configuration Manager and give it a try? Also, while running tests from CLI, provide the arm xap and test settings with Target set to device. (I'm assuming your device is already unlocked with a developer license). And we are fixing up the names in messages.
@Alexander Soutchilin: You cannot run Phone Unit Tests as part of Team Build yet - we have this in our backlog. However when you execute tests from CLI, you can use /logger:TfsPublisher to publish results to Build Summary Page.
@Richard Szalay: More than the communication channel, it's the .NET CLR for Windows Phone 8 that made it possible for us to port our existing unit testing solution to WP8 platform.
Thanks Vinay, turns out I overlooked to set Platform to ARM (or use the ARM xap for CLI) instead, indeed. Nice one! (although perhaps, error experience could've been more helpful...) Now that it works on device I have found it to be more reliable also it seems (I have since tried it on emulators on another machine and it seems flaky that way in that sometimes not all tests are run and I have to manually select and run each of those, despite having selected Run All - note they are not skipped). But thanks, I have a better workflow now that this works!
Regarding the known issue : "Phone Unit Tests that have a native dependency cannot be run from command line". Will this issue be fixed when update 2 is released?
Does the unit test framework provide a way to collection code coverage numbers?
@RobN75- Yes, referencing native dependency has been addressed in upcoming release.
@Julius Ang- No, WP8 Unit Test framework doesn't support code coverage collection.
Are UITestMethodAttribute and async Assert.ThrowsException coming to this WP8 test framework too? Perhaps CTP3 (please)? :)