3rd Party Software Integration: What Quality Gates are Needed?
One of the key phases for a mobile device project is 3rd party software integration, which is usually done after initial software baseline has been established (BSP migration complete, driver stabilized, hopper & battery life pass). According to device software product definition, a number of 3rd party applications and modules (for example, some DShow filters) have been identified and made available for integration at this phase. Some of these software may not be the final release as the vendors may also need to conduct technical evaluation and testing with a sample device on their side before releasing a final release. Vendors will provide in-ROM integration instructions, and the ROM build team just needs to follow the instructions to generate test images.
Nonetheless, a well-defined strategy and process for 3rd party software quality gating turn out to be highly critical for device software stability. At minimum, for each 3rd party application, the following quality gates must be used:
- Detecting leaks using Application Verifier: Make sure the application does no have significant memory leaks, handle leaks, GDI leaks. For modules/drivers, develop test tools to run within Application Verifier. This can be done within 3 days for a single application or module.
- MTTF using hopper: Any application should not take down hopper number by more than 10%. Some applications were not designed with hopper testing in mind (for example, an application runs full-screen but does not enable "Start" button). These will bring down hopper numbers so they must be fixed. Having a focused hopper tool handy such that you can stress a specific application. Also useful is to enable debug messages in those 3rd party applications such that you can always check CElogs for a failed hopper run. This will usually take 3-6 days depending on the degradation of hopper numbers caused by the application.
- Battery life/DoU: Windows Mobile application design guideline suggests that an application should follow device power management (by registering for power notification events) and react to system power state changes accordingly. No polling or animations when the application is in background. Use power monitor to check power consumption when the application is running and the device is in various power states. DoU test can be performed to ensure an application does not bring down more than 10% of the DoU number. This will take one or two weeks if the application needs to be modified and re-tested.
- No device crashes/hang: this is quite obvious. A 3rd party application often comes with an acceptance test plan, which identifies test cases in different priorities. Perform all the tests and make sure there is no crash (we're sorry messages) or device hang. If there are crashes, the root cause must be found and fixed (if it is in the application) by looking into the dump files (kdmp) with symbols. A consistent repro of a crash is ship-blocking and must be fixed. Depending on the complexity of the test plan, this step may take 2-3 weeks and even longer if the 3rd party software does not come with good quality.
In summary, for a single 3rd party application, it may take several weeks to perform the integration and quality gating, during which effective communication with the vendors is important and both sides understand the device will not ship unless these quality gates are passed (according to the license agreement with the vendors). Issues uncovered should be properly prioritized such that key blocking/critical issues are resolved in the new builds from the vendor, whereas low-user-impact issues can be punted to maintenance releases later.
To ensure high quality of the 3rd party software and overall device stability, and importantly device software will be delivered as planned, an earlier planning of integration and quality gating for each 3rd party software is a must-do for overall device software engineering management, along with required QA/Dev resources pre-allocated.
