HoppeRx - the cure for your ailing device : ISV

Improving the Cat Parade (Part 3)

JeCahill — Fri, 05 Oct 2007 21:44:00 GMT

I was recently running hopper on a device that supported screen rotation and I realized that my test coverage was completely missing the rotation scenario. The device would switch between portrait and landscape mode if the user took a specific action,...(read more)

Improving the Cat Parade (Part 2)

JeCahill — Thu, 27 Sep 2007 23:21:00 GMT

There have been a few blogs that talk about the benefit of focusing hopper runs on individual applications to work out the stability bugs one application at a time. But, there comes a time when testing one by one may not be the best use of resources....(read more)

The Hopper Cookbook (Step-by-step instructions to improve application stability)

shende — Wed, 06 Jun 2007 19:53:00 GMT

The intent of this document is to introduce Hopper to the application developer who is targeting applications for the Windows Mobile platform. The purpose of running Hopper against your application is to find exceptions, hangs and deadlocks that may be lurking in your application that may go unnoticed during testing and may cause device crashes during customer acceptance or field trials. This introduction will include all the necessary materials to begin Stability testing of your application using Hopper as well as pointers for advanced techniques.

Goals

The goal of this document is to educate ISV development community on Hopper so they can improve application quality and stability. Developing with Hopper, ISV’s will reduce customer acceptance time as well as find more application bugs that will contribute to the overall quality of their application. The expected steps from the ISV community are:

Consistently run Hopper during the development cycle and sort the bugs into three categories:

Category #1: ISV application dependent bugs (to be fixed by the ISV)
Category #2: OEM device dependent bugs (to be reported to the respective OEM for fixes, if applicable)
Category #3: WM dependent bugs (to be reported to Microsoft for evaluation and possible fixes)

Coordinate with both the respective OEM and Microsoft (if applicable) to secure bug resolutions and proper application stability
Run Hopper during WM device development by coordinating with their respective OEMs (if applicable)

The ISV development community following the above process will enjoy improved application quality and stability. Hopper takes the guesswork out of quality measurement, thus reducing application time to market.

Prerequisites for this document

This document builds upon existing materials published for the OEM and the ISV development community. It is highly recommended that the reader review the following topics before reading this document:

Hopper for ISV’s – part I (http://blogs.msdn.com/hopperx/archive/2006/06/05/618018.aspx)
The Cat Parade (http://blogs.msdn.com/hopperx/archive/2005/11/30/498113.aspx)
Improving the Cat Parade (http://blogs.msdn.com/hopperx/archive/2006/11/21/improving-the-cat-parade-part-1.aspx)

The Hopper Doctor blog, which contains the above materials is updated frequently and often contains ISV-specific information, so it is recommended that ISVs check the main blog page (http://blogs.msdn.com/hoppeRx/) frequently. This document also assumes the reader has Visual Studio 2005 and the Windows Mobile 6 SDK installed.

Review – and overview of the Hopper system

As mentioned in the Hopper for ISV’s blog, Hopper is a stability test tool that randomly sends keystrokes and screen taps to the device. Hopper is designed to stress the entire device and not remain in a single application. However, application developers are not interested in stability of the entire device– they want to locate and fix bugs only in THEIR application. To achieve this, ISVs need a way to focus Hopper on their application and to minimize the time Hopper spends away from their application.

To successfully perform stability testing with hopper, three elements are needed:

The Hopper component is available under your SDK tools directory. A sample focusApp.exe is provided in the same Hopper tools directory and will need to be customized for each application you are testing. To begin testing, you need to perform the following steps:

Using Visual Studio, deploy your application to a device or the device emulator and attach the debugger. This will enable you to debug any exceptions or hangs that your application encounters. Step-by-step instructions on how to do this are located below in the section labeled Demo.

Customize the focusApp.cpp source to keep your application in the foreground. Refer to “The Cat Parade” blog entry for more information as well as the end of this document for more ideas. Once customized, this tool needs to be deployed to device and your application

Deploy and run Hopper while your application is running. The focusApp will force Hopper to remain “focused” on your application and avoid the rest of the system.

By combining these three elements, we are able to remove platform stability from the equation and allow Hopper’s effort to be focused on your application and away from other potential problems. For example, if you are running this test on a device that has known stability issues, it is unlikely that Hopper will encounter those issue because your application is processing all the key strokes. This is very convenient since it is the application bugs that you are interested in, not the bugs elsewhere in the system.

Demo - Watch Hopper in action and find real bugs!

Below represents a step-by-step procedure for running Hopper on a sample application that is included in the SDK. Ultimately you will want to proceed through the following steps using the application you are developing and not CECamera.

Step 1: Launch Visual Studio. Launch the SDK tools folder from the start menu and navigate into the Hopper Folder and then again into focusApp folder. Double click on this solution to bring up VS.

Step 2: Customize focusApp. We need to customize focusApp to point to the application under test. For this sample, open the file FocusApp.cpp and replace the “\\Windows\\WMPlayer.exe” with “\\Program Files\\CECamera\\CECamera.exe”. Rebuild the solution. When testing your own application, you will replace the CECamera path and filename with the path and filename of your application

Once this application has been rebuilt – you can close this solution.

Step 3: Launch the CECamera solution provided under the Samples folder – this is the application we will torture under hopper.

Navigate to the CECamera directory under the following directory: Samples\Common\CPP\Win32\CECamera – double click on this CECamera solution to re-launch Visual Studio with this solution active.

Step 4: Add Hopper dependencies to your deployment.

Change the Project properties to automatically deploy your focsuApp.exe and Hopper.exe. Under the menus Project->Properties, select Configuration Properties -> Deployment and add the following lines to the existing:

focusApp.exe|C:\Program Files\Windows Mobile 6 SDK\tools\hopper\focusApp\FocusApp\$(OutDir)||\|0

hopper.exe|C:\Program Files\Windows Mobile 6 SDK\tools\hopper||\|0

Please note that your Path locations may be different from the example, and care must be taken to copy & paste the correct paths. When this is complete, your dialog should look similar to:

Test your deployment. Using the menu action Build -> Deploy solution, check the results of your deployment, you should see:

1>------ Deploy started: Project: CECamera, Configuration: Debug Windows Mobile 6 Professional SDK (ARMV4I) ------

1>C:\Program Files\Microsoft Visual Studio 8\VC\ce\dll\ARMV4I\msvcr80.dll

1>C:\Program Files\Microsoft Visual Studio 8\VC\ce\dll\ARMV4I\msvcr80d.dll

1>C:\Program Files\Windows Mobile 6 SDK\tools\hopper\focusApp\FocusApp\Windows Mobile 6 Professional SDK (ARMV4I)\Debug\focusApp.exe

1>C:\Program Files\Windows Mobile 6 SDK\tools\hopper\hopper.exe

1>c:\Program Files\Windows Mobile 6 SDK\Samples\Common\CPP\Win32\CECamera\Windows Mobile 6 Professional SDK (ARMV4I)\Debug\CECamera.exe

========== Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

========== Deploy: 1 succeeded, 0 failed, 0 skipped ==========

You will note that the Device Emulator will launch in the background and fully boot before copies can begin. If there are any errors during deployment, please check your pathnames above to make sure you are pointing to the correct locations. You must be able to copy & paste the full paths and reference the binaries. Please correct any mistakes before continuing.

Step 5: Close off exit points.

If your application responds to WM_CLOSE to exit, we need to ignore this command so Hopper does not close the application. CECamera sample contains both a menu item and an “X” box that will sent a WM_CLOSE to the app – so we need to comment out this code. Locate the CECamera.cpp file and search for WM_CLOSE like below:

…

case IDCANCEL:

// Fall through

case IDM_EXIT:

// PostMessage(hwndDlg, WM_CLOSE, 0, 0); ** Comment out this line **

break;

…

Step 6: Start debugging the CECamera.exe application.

Before we begin debugging we need to verify the debugger will detect and break when an exception is hit. Menu Debug -> Exceptions will bring up the following dialog – make sure all thrown check boxes are checked and your settings saved.

Next, Select the menu Debug -> Start Debugging to deploy and debug CECamera. This will launch the application under test and if you switch back to Device Emulator, you will be presented with the sample application.

Step 7: Launch the Hopper and focusApp. In this step, we need to access the File Explorer so we can launch focusApp and then Hopper. Click on the Start Menu, select Programs and then File Explorer.
Navigate UP to the root directory and select focusApp.exe – this will launch focusApp.exe which will quickly bring CECamera back to the foreground.
Bring the File Explorer back by clicking the Start Menu -> File Explorer. Use this opportunity to launch Hopper.exe. If you wait too long, focusApp will bring the CECamera application back to the foreground and obscure File Explorer. Simply Bring the File Explorer back by clicking the Start Menu -> File Explorer and click on Hopper.exe. This tight-timing is expected and part of the design. In a moment, Hopper will take over your system and begin pressing keys and tapping on the screen. Once Hopper encounters an exception in your application, Visual Studio will break and you will be notified.

How to debug breaks

Having Hopper focus on your application is only useful if it finds exceptions, hangs or deadlocks. Hopper’s job is to identify these bugs and it is left to the developer to actually find and fix the root cause for any break The Hopper Doctor Blog (http://blogs.msdn.com/hoppeRx/) is a great resource for how to debugging exceptions, hangs and deadlocks found by hopper.

Since Visual Studio is an application debugger you will notice that Hopper will continue stressing the device after your application has broken into the debugger. You will also note that focusApp.exe continues to run and will continue trying to bring your application to the foreground.

This is normal and should not interfere with debugging the break that stopped Visual Studio.

Improving the Cat Parade (Part 1)

JeCahill — Wed, 22 Nov 2006 02:30:00 GMT

The focusapp described in a previous blog entry The Cat Parade is a great way for ISVs to get additional test coverage on their applications with hopper. I recently had the opportunity to work with an ISV that was developing a suite of apps and we made use of focusapp, but I noticed it didn’t work quite as well as I expected. After a hopper run of 10 hours, we looked at the hopper summery and found that hopper only spent ~30% of its time on the target app. This is pretty good, but I expected better.

After watching hopper for about a minute, I realized that hopper frequently brings up the Dialer or the Start menu after just a few seconds in the target app, and from the Start menu hopper quickly finds its way into other applications. We tried decreased the period for focusapp, but this didn’t seem to increase the hopper coverage significantly and would have prevented hopper from exploring the deeper levels of the sub-menus in the target app. However, we found that we could get a huge improvement in our hopper coverage if we used ActiveSync’s File Explorer to delete all of the shortcuts in the Start Menu except for the target app and the Settings menu. We also got some improvement by changing the home screen layout to “Large Font”, which doesn’t have application shortcuts on it. These two changes took the hopper time from 30% to 90% for the target app. We left the Setting menu on the Start menu because we wanted to make sure hopper had the ability the change the system and hardware settings. In our target app, we wanted to verify none of the settings affected our app, but this may not be interesting for all apps.

Hopper for ISV’s – part I

shende — Mon, 05 Jun 2006 19:47:00 GMT

I have been reflecting on the things Javier and I learned from our Hopper talk at MEDC and I wanted to thank everyone that attended our talk. I also wanted to give special kudos to those who took the time to fill out the evaluations – this post is your voice in action. The one theme that rang louder than all others is we did not properly address or set expectations for our ISV customers and how they can benefit from Hopper.

Most readers already know all Windows Mobile OEM devices must pass Hopper testing to receive their logo certification – this includes all inrom-ISV applications. However, applications installed after Logo are not covered by this requirement and can introduce stability risks to the end user.

The answer to this dilemma is to evangelize Hopper testing to all applications destined for the Windows Mobile platform – not just for those shipping on the device.

Why should ISV’s care about Hopper?

Hopper is good - Hopper will find bugs in your application that can affect the stability and usability of your device. Hopper is an input stress tool that randomly sends keystrokes to the device and will put your application through scenarios you likely haven’t though of. Who wouldn’t want to find bugs in the application they are trying sell?

How can ISV’s get Hopper?

If you are an ISV, this tool is provided automatically when you install the Windows Mobile 6 SDK.

How to run ISV-Hopper

Hopper for ISV’s is just like Hopper for OEM’s, the only different being the addition of a “focus app” customized to keep the intended application in the foreground. When your application is in the foreground, Hopper will test only it and not the rest of the system – the “focus app” is needed to bring your application back to foreground if Hopper tries to navigate away. The “focus app” blog and source code is described in The Cat Parade.

The specifics of running Hopper.exe on your device are documented in the readme.exe from the Power Toys directory and will depend on if you are running connected to a debugger (recommended) or not. The easiest way to run Hopper.exe is to simply copy the binary to your device and execute directly.

How can I debug application bugs found with Hopper?

This is really a loaded question and will depend on many factors, including whether or not you are connected to a debugger (recommended). If you have a debugger connected, exceptions and hangs should be easy to diagnose if you have correct symbols. If you are not connected (NOT recommended), then having current symbols and some way of capturing the debug output is a good start.

Luckily this question is the exact intent of the blog you are reading so each entry located on the HoppeRx site will help with this.

How many hours should my application survive?

It is entirely up to you and it depends on the complexity of your application, but 20 hours with a “focused Hopper” is a good goal. Anything longer than that will likely give you decreasing returns as the same code paths are likely being re-executed. But don’t feel limited by just “focused Hopper” – can you add additional stress elements to create interesting user scenarios?