The process of creating an application does not only revolve around its development. There is another important aspect that we tend to avoid or not take seriously enough: user testing. Beta phase, as I like to call it. No matter which platform you build your app for, you need to test your app at some point. Yes, you worked really hard for many weeks or months on your app. Yes, you know your app by heart. Yes, it is bug free. I don’t want to disappoint you, but there is always a little something that you didn’t think of, or a special usage scenario you didn’t pay attention to.
I speak from experience. I have submitted apps without doing a beta phase. I have also submitted an app where a crash could be easily reproduced. It is nothing to be proud of, but at least now, I do a beta phase.
Before releasing DualShot, we ran a beta phase.
So, how do you attract testers?
Believe me, you’ll get testers pretty easily. People are curious. Depending on how well known you are, you can keep track of the emails manually like I do or you can share an Excel sheet or use MailChimp like one of my friends does.
Unless you pay the testers, don’t expect to get feedback from everyone. This is normal, so don’t take it personally. Sometimes it is just not the tester’s type of app, or perhaps they are just too busy.
PRO TIP I use BugSense to keep track of crashes and I use Flurry for the analytic events. I highly recommend using these services (or any equivalent services) during the beta phase. As I previously mentioned, you might not get written feedback from all the testers, but at least you’ll receive traces of what they did with your app.
During the beta phase, the tester can provide feedback about the user experience (UX). As a concrete example, in DualShot, the app desigmer, Vincent, designed the following page:
I really liked this page and I didn’t see a problem using it. When I submitted the beta to my testers, in the same evening, three users complained that the UX was really bad. It was clear to us that we needed to put more work into that view. We ended up with the following design:
Unless you have the complete Windows Phone collection at home, you are most likely going to have only one or two devices to test with. When you have testers, you increase the chances that the app will be tested with many different Windows Phones. Don’t assume that all Windows Phone 8s from different manufactures behave the same. In DualShot, the image capture with some HTC 8XT (only this model) does not work. We didn’t have a tester with this rare device and Murphy’s Law struck again.
Not only will the testers find bugs/crashes and give you feedback, they will often give you two thumbs up. It will give you the extra energy to polish your app before certification.
Beta in the Windows Phone Developer Portal
When developing a Windows Phone app, you are lucky because the Windows Phone team developed a feature in the portal where it is easy to run a beta phase. Essentially, you publish an app as if you were submitting it for certification. However, the app is kept private for the testers that you have specified and the certification passes automatically in the subsequent two hours.
In the Windows Phone developer portal, you need to select Beta as the Distribution channel, then you enter the list of tester email addresses (using a ‘;’ between addresses).
After that you can fill out all the info and screenshots.
PRO TIP Avoid flooding your testers with daily emails if you submit a new beta every day. If you submit an update, the tester will be notified via the Store Live Tile. Take advantage of this automatic notification to enter all the new features/bugs into the app description. The first time that you send the deep link of the beta app, you can tell your tester to check the app description when a new update is available.
Unfortunately, in the current Windows developer portal, it is not possible to easily distribute an app to testers. You need to create a package in Visual Studio and send the package to your testers. The testers then need to manually install the app.
I know the feeling when an app is finally complete and ready to be submitted for certification. It is so tempting to submit it right away in order to see it live in the store, ready to be downloaded. Resist the temptation and do yourself a favour – do a beta test phase. It is better that your friends find any problems than strangers.
Remember that when your apps are out there in the store and people are downloading, they have the ability to rate your app. You want them to rate your app as many good things come out of that. BUT, strangers will most likely give you a 1-star review if they find a bug or a crash, so it’s in your best interest to work all of those out before the real “raters” come in and take a look at your app.
Also, as much as I focused on Windows Phone above, all of this is just as important to your Windows 8 and Windows 8.1 apps – just that there’s no Store way of doing it. After all, an app is an app regardless of what platform you develop on.
Develop, TEST, and then release!
Actually, there is a way to do it without sending the Visual Studio project, but it involves PowerShell scripts. I don't have a link handy, but one of the Twitter clients did it during the early 8.0 days.
When I was referring to "create package", it is the PowerShell script that you are talking about. I should haved elaborated more.
Here is a blog post that describes how to do a package: geoffwebbercross.blogspot.ca/.../creating-windows-8-metro-app-package.html
Thanks for reading my post!
Good points! Definitely a good practice, especially for developers starting on the platform.
Your app will be as successful as you decide it will be. Make use of all the great tools and resources and your apps will shine.
Great Article !!!