My name is Martha Wieczorek and I’m a Software Design Engineer in Test on the Visual C++ IDE team. I would like to share with you some different testing approaches that we use on the team and talk about the advantages/disadvantages of each approach based on my experience.
Testing effectively and efficiently becomes more and more challenging as the size and complexity of our software grows, especially when the software involves a lot of UI interaction. The maintenance costs of test automation have historically been very high. Therefore, the importance of developing a test automation infrastructure that takes advantage of different testing approaches is very important.
First some definitions:
UI level tests: Feature functionality is tested by manipulating UI elements such as windows and controls, i.e.: menu items, buttons or dialogs.
Object model level tests: Feature functionality is tested programmatically by bypassing the UI.
Component level tests: Multiple components are tested together but without the entire system being present.
Unit level tests: Individual APIs are tested in isolation. This approach is mainly used by Developers, and I am not covering this here.
Object Model/Component level and UI level approaches to test automation are very different. Each has its own focus and goals and one cannot replace another. Replacing UI-approach by Object Model/Component approach guarantees that you will miss bugs in the target product. Therefore, it is important to have a good balance between Object Model /Component Level testing and UI Level testing.
Creating Object Model/Component Level tests that bypass the UI completely helps my team to develop robust and cost-effective tests suites. We have test hooks in the product that exercise the same functionality as that driven by the UI. This makes writing test automation much more efficient, improves feature coverage, speeds test runs (~70% faster than UI level tests), and most important of all it increases the interaction with developers. This collaboration helps testers understand the components/features better. In turn, this results in better testing methodology.
While Object Model/Component Level tests are very beneficial, it is important to keep in mind that UI testing doesn’t go away completely. It is necessary for a real simulation on how the customer is going to use the feature. However, building UI tests is far from trivial. UI changes that occur late in the development cycle (ex: button, menu item, or dialogs may change location or appearance), UI synchronization issues (i.e. issues related to machine speed), localized strings, and many other factors increase the probability of false failures. This makes UI test automation a high maintenance activity. As a result, the QA team spends a lot of time investigating non-product related failures.
Designing, implementing, and running automated tests is critically important, but there are many bugs that test automation cannot catch. Taking the time to check features visually, at the end of the product cycle, is very critical. I have seen bugs when UI elements such as dialogs/controls are cut-off the screen and these situations are not caught through automation. The importance of real world usage of the feature is very important and should be considered in every product cycle.
PingBack from http://msdnrss.thecoderblogs.com/2008/03/06/approaches-to-testing/
I agree that a lot of bugs cannot be caught by automation, but I wonder about this one:
"I have seen bugs when UI elements such as dialogs/controls are cut-off the screen and these situations are not caught through automation."
A few months ago I reinstalled a Windows 2000 Server and reinstalled Office 2000 on top of it. Either during the installation of Office 2000 or one of its service packs, a log file was created which diagnosed exactly that kind of bug. Several lines in the log file reported detection of controls or text being truncated by 1 pixel due to missizing. I was impressed by the effort that had gone into automating that test and equally unimpressed that the bugs remained unfixed.
(Of course 1 pixel isn't much of a problem though. It's more serious when an entire line of text is lost or an entire button is lost so users don't even know they're missing anything. Anyway, it seems that maybe this particular bug detection can be automated.)
Thanks for the interesting post. What software do you use for your UI Level Tests? I have used Mercury QuickTest Pro for this, and overall I've been pretty dissatisfied.
Hello,I am a chinese student,my pc sysytem is vista basic,I find that visual c++6.0 is not compatiable with my vista system.I am puzzled,I don't know how to deal with it.can you help me? thank you.my email:email@example.com
To Wang Sheng Lei:
It is true that VC++ 6 is not compatible with Vista. Either change to Windows XP or change to Visual Studio 2005 or 2008.
If you use Vista and run Visual Studio 2005 SP1 + Visual Studio 2005 hotfix for Vista, then you will find yourself still testing Visual Studio 2005.
163.com is a very famous spamming ISP. If you want to use e-mail with people outside of China, look for a less spammy ISP.
This blog post is not a support channel for Visual Studio -- Try the MSDN forums, or Connect.
Your comment is out of place here.
Our team is using a internal tool based on the Accessibility framework for Microsoft Windows : “Microsoft® Windows® User Interface (UI) Automation” as well as “Microsoft Active Accessibility (MSAA)”. If you want to read more about these technologies the following article is a good start:
Thanks for your comments,
A simple program with all the following:
UI level tests:
Object model level tests:
Component level tests:
Unit level tests:
The idea is to have a starting place best practice/example code/environment.
Does such an application exist as a total environment set.