I. M. Testy : The Professional Tester

Localization Testing Part III

I.M.Testy — Tue, 03 Nov 2009 19:36:00 GMT

Part 1 provided an overview of localization class issues, and Part II discussed issues with non-translated strings in a localized product and gave some helpful hints to manage that problem during the software development lifecycle. In Part III I will cover various issues with access key mnemonics. An access key mnemonic is the underlined letter on a menu or control that corresponds to a key on the native keyboard layout for a particular language.

Missing & duplicate access key mnemonics

Interestingly enough, most localization tools have built in tools to test for duplicate key mnemonics; however, missing or duplicate access key mnemonics is another significant issue in localization testing, and also affects the English language version as well. Duplicate or missing key mnemonics can adversely affect the usability of software because it impacts the ability of the user to easily access or invoke commonly repeated functions using the keyboard. Duplicate or missing duplicate key mnemonics can also negatively impact the software’s ability to meet certain accessibility requirements.

Although missing or duplicate access key mnemonics are sometimes caused by poorly designed dialogs with an overabundance of controls, there are other factors that can cause duplicate key mnemonics. For example, some controls may dynamically appear in some dialogs in specific machine states. These dynamically generated controls may also come from a file that is different than the file which generated the dialog. Another reason for duplicate key mnemonics could also be dynamically generated key mnemonic assignments, which are especially problematic in situations where a dialog contains a mixture of statically assigned key mnemonics and dynamically generated key mnemonics.

Manual testing for missing or duplicate key mnemonics is especially labor intensive, and finding ways to automate this testing will save countless hours of time sitting in front of a computer checking menus and dialogs. There is also a large probability of missing duplicate key mnemonic assignment problems using manual testing methods because eyes get tired, people get bored, and some keys are grayed out (as in the illustration below) or may not be present in certain machine states. Fortunately, there are several automation tools that detect duplicate key mnemonic problems and automated detection is more effective than manual test approaches. For example, Automation.Element.AccessKeyProperty in the UIAutomation class library in C# is one approach to more efficiently test access key mnemonics.

Access key mnemonic assignments

As a general rule of thumb (heuristic), key mnemonics should be assigned to characters mapped to the default state of the keyboard for each particular language. For the Latin 1 family of languages, access key mnemonics should generally not be assigned to non-ASCII characters; even if that particular character is accessible on the default state keyboard layout for a particular language. Certain, access key mnemonics should never be assigned to a character glyph that is formed through combining characters used in languages such as Thai and Arabic. Also, punctuation characters should never be assigned as access key mnemonics.

Of course, the default keyboard layout for many non-Latin 1 languages only contain characters in the native script for that language, and assigning non-ASCII characters as access key mnemonics may be the only choice. However,

Japanese hiragana and katakana glyphs, Korean Hangeul glyphs, and all East Asian ideographic glyphs are invalid character assignments for access key mnemonics. The default keyboard layout for most East Asian languages (Japanese, Korean, Simplified Chinese, and Traditional Chinese) is the standard keyboard layout similar to the US English keyboard. In the above example, their is no way for a Japanese user to access the ‘My Computer’ (マイコンピユータ）menu item because it is using a katakana character as an access key mnemonic (which violates several guidelines for access key mnemonic assignment). Also, the standard key mnemonic guidelines described below should be used for all East Asian language versions for consistency and backwards compatibility.

Another general guideline to follow for access key mnemonic assignments is to avoid the lower case Latin letters ‘g’ ‘y’ ‘p’ ‘q’ and ‘j’ because there is a high probability of confusion especially with high display resolutions especially with the letters i and l, and q and g. These letters could also be hard to discern on high resolution desktop settings as well. If the number of controls on a single dialog or in a menu list require usage of inappropriate key mnemonics, then perhaps the problem is the design of the dialog.

East Asian language versions should use the identical key mnemonics as the English language version. The characters assigned as key mnemonics in the East Asian language are capitalized, enclosed within parenthesis, and positioned at the end of the translated string. Even when a key mnemonic appears within words or acronyms which are not translated or transliterated into the target the key mnemonic should be relocated to the end of the string and enclosed within parenthesis for consistency.

The character assigned as the key mnemonic should be capitalized because many East Asian computer users use an English key keyboard, and for users whose native language does not frequently employ Latin characters it is much easier for those users to visually identify key mnemonics which are capitalized with keys on the keyboard which are also capital case characters.

Accelerator Keys

Accelerator keys are commonly refered to as shortcut keys. Accelerator keys are keys (such as F1 - F12 and Esc) or key combinations (Ctrl + Shift + B, or Ctrl C) that allow users to evoke certain functions without navigating the software menus via access keys or using the mouse to click button controls on a toolbar. Here is good source for common Windows accelerator keys, and here is one for common accelerator keys for Office products.

Shortcut key combinations are common throughout all language versions. Contrary to the Wikipedia entry on the subject some language versions localize the letter key (not a mnemonic...it is not underlined). For example, in German the Ctrl key is localized as "Strg" and, dispite it is generally frowned upon to change the ASCII upper case letter assigned to an accelerator key combination the Spanish versions of Office use Ctrl+G (Guardar) for Save instead of Ctrl+S, and Ctrl+N (Negrita) for Bold instead of Ctrl+B. Also, letter keys used as part of an accelerator key combination are capitalized. East Asian language versions use Ctrl to designate the Control key. Also, accelerator key combinations do include an elipsis after the letter.

In Part IV I will discuss common layout issues such as clipping and truncation.

Localization Testing – Part II

I.M.Testy — Fri, 30 Oct 2009 19:34:00 GMT

I should be of no surprise to anyone that localization testing generally focuses on changes in the user interface, although as mentioned in the previous post these are not the only changes necessary to adapt a product to a specific target market. But, the most common category of localization class bug are usability or behavioral type issues that do involve the user interface. Bugs in this category generally include un-localized or un-translated text, key mnemonics, clipped or truncated text labels and other user interface controls, incorrect tab order, and layout issues. Fortunately, the majority of problems in this category do not require a fix in the software’s functional or business layer. Also, the majority of problems in this category do not require any special linguistic skills in order to identify, and in some cases, an automated approach can be even more effective than the human eye (more on that later).

Perhaps the most commonly reported issue in this category is “un-localized” or un-translated textual string. Unfortunately, in many cases un-translated strings is also an over-reported problem that only serves to flood the the defect tracking database with unnecessary bugs. Translating textual strings is a demanding task, and made even more difficult when there are constant changes in the user interface or contextual meaning of messages early in the product life cycle. Over-reporting of un-translated text too early in the product cycle only serves to artificially inflate the bug count, and causes undue pressure and creates extra work for the localization team.

Identifying this type of bugs is actually pretty easy. Here’s a simple heuristic; if you are testing a non-English language version in a language you are not familiar with and you can clearly read the textual string in English it is probably not localized or translated into the target language. The illustration below provides a pretty good example of this general rule of thumb. A tester doesn’t have to read German to realize that the text in the label control under the first radio button is not German.

There are several causes of un-localized text strings to appear in dialogs and other areas of the user interface. For example:

Worse case scenario is that the string is hard-coded into the source files
Perhaps localizers did not have enough time to completely process all strings in a particular file
Perhaps this is a new string in a file localizers thought was 100% localized
Strings displayed in some dialogs come from files other than the file that generates the dialog, and the localization team has not process that file
And, sometimes (usually not often), a string may simply be overlooked during the localization process

Testing for un-localized text is often a manually intensive process of enumerating menus, dialogs, and other user interface dialogs, message boxes and form, and form elements. But, if the textual strings are located in a separate resource file (as they should be), a quick scan of resource files might more quickly reveal un-translated textual strings. Of course, there is little context in the resource file, and I also hope the localization team is reviewing their own work as well prior to handing it over to test.

Also, here are a few suggestions that might help focus localization testing efforts early in the project milestone and reduce the number of ‘known’ or false-positive un-translated text bugs being reported:

Ask the localization team to report the percentage of translation completion by file or module for each test build. Early in the development lifecycle only modules that are reported to be 100% complete which appear to have un-translated text should be reported as valid bugs. Of course, sometimes some strings are used in multiple modules, or may be coming from external resources. But, especially early in the development lifecycle reporting a gaggle of un-translated text bugs is simply “make work.” As the life cycle starts winding down…all strings are fair game for bug hunters!
Testers should use tools such a Spy++ or Reflector to help identify the module or other resources, and the unique resource ID for the problematic string or resource. This is much better then than simply attaching an image of the offending dialog to a defect report. Identifying the module and the specific resource ID number allows the localization team to affect a quick fix instead of having to search for the dialog through repro steps and track down the problem.
Also remember that not all textual strings are translated into a specific target language. Registered or trademarked product names are often not translated into different languages. In case of doubt, ask the localization team if a string that appears un-localized is a ‘true’ problem or not.

Unlocalized strings usually due to hard coded strings also tend to occur in menu items. This is especially true in the Windows Start menu or sub-menu items hard-coded in the INF or other installation/setup files. For example, the image on the right shows a common problem on European versions of Windows. Many European language versions localize the name of the Program Files folder, and the menu item in the start menu. But, often times when we install an English language version of software to Windows it creates a new "Programs" menu item (and even a new Program Files directory, rather than detecting the default folder to install to. In the example on the left, the string Accessories is a hard-coded folder name. But, there is another issue as well. This illustrates not only a problem with the non-translated string "Accesssories," but also shows one full-width Katakana string for 'Accessories' and another half-width string.

In part 3 I will discuss another often problematic area in localization….key mnemonics.

Localization Testing: Part 1

I.M.Testy — Tue, 27 Oct 2009 06:00:00 GMT

When I first joined Microsoft 15 years ago I was on the Windows 95 International team. Our team was responsible for reducing the delta between the release of the English version and the Japanese version to 90 days, and I am very proud to say that we achieved that goal and Windows 95 took Japan by storm. It was so amazing that even people without computers were lined up outside of sales outlets waiting to purchase a copy of Windows 95. The growth of personal computers in Japan shot through the roof over the next few years. Today the Chinese market is exploding, and eastern European nations are experiencing unprecedented growth as well. While the demand for the English language versions of our software still remains high, many of our customers are demanding software that is ‘localized’ to accommodate the customers national conventions, language, and even locally available hardware. Although much of the Internets content is in English, non-English sites on the web are growing, and even ICANN is considering allowing international domain names that contain non-ASCII characters this week in Seoul, Korea.

But, a lot has changed in how we develop software to support international markets. International versions of Windows 95 were developed on a forked code base. Basically, this means the source code contained #ifdefs to instruct the compiler to compile different parts of the source code depending on the language family. From a testing perspective this is a nightmare, because if the underlying code base of a localized version is fundamentally different than the base (US English) version then the testing problem is magnified because there is a lot of functionality that must be retested. Fortunately today, much software being produced is based on a single-worldwide binary model. (I briefly explained the single world wide binary concept at a talk in 1991, and Michael Kaplan talks about the advantages here.) In a nutshell, a single worldwide binary model is a development approach in which any functionality any user anywhere in the world might need is codified in the core source code so we don’t need to modify the core code once it is compiled to include some language/locale specific functionality. For example, it was impossible to input Japanese text into Notepad on an English version of Windows 95 using an Input Method Editor (IME); I needed the localized Japanese version. But, on the English version of Windows Xp, Vista, or Windows 7 all I have to do is install the appropriate keyboard drivers and font files and expose the IME functionality. In fact, these days I can map my keyboard to over 150 different layouts and install fonts for all defined Unicode characters on any language version of the Windows operating system.

The big advantage of the single worldwide binary development model is that it allows us to differentiate between globalization testing and localization testing. At Microsoft we define globalization as “the process of designing and implementing a product and/or content (including text and non-text elements) so that it can accommodate any locale market (locale).” And, we define localization as “the process of adapting a product and/or content to meet the language, cultural and political expectations and/or requirements of a specific target market.” This means we can better focus on the specific types of issues that each testing approach is most effective at identifying. For localization testing, this means we can focus on the specific things that change in the software during the “adaptation processes” to localize a product for each specific target market.

The most obvious adaptation process is the ‘localization’ or actually the translation of the user interface textual elements such as menu labels, text in label controls, and other string resources that are commonly exposed to the end user. However, the translation of string resources is not the only thing that occurs during the localization process. The localization processes that are required to adapt a software product to a specific local may also include other changes such as font files and drivers installed by default, registry keys set differently, drivers to support locale specific hardware devices, etc.

3 Categories of Localization Class Bugs

I am a big fan of developing a bug taxonomic hierarchy as part of my defect tracking database as a best practice because it better enables me to analyze bug data more efficiently. If I see a particular category of bug or a type of bug in a category that is being reported a lot, then perhaps we should find ways to prevent or at least minimize the problem from occurring later in the development lifecycle. After years of analyzing different bugs, I classified all localization class bugs into 3 categories; functionality, behavioral/usability, and linguistic quality.

Functionality type bugs exposed in localized software affect the functionality of the software and require a fix in the core source code. Fortunately, with a single worldwide binary development model where the core functionality is separated from the user interface the number of bugs in this category of localization class bugs is relatively small. Checking the appropriate registry keys are set and files are installed in a new build is reasonably straight forward and should be built into the build verification test (BVT) suite. Other types of things that should be checked include application and hardware compatibility. It is important to identify these types of problems early because they are quite costly to correct, and can have a pretty large ripple effect on the testing effort.

Behavioral and usability issues primarily impact the usefulness or aesthetic quality of the user interface elements. Many of the problems in this category do not require a fix in the core functional layer of the source code. The types of bugs in this category include layout issues, un-translated text, key mnemonic issues, and other problems that are often fixed in the user interface form design, form class, or form element properties. This category of problems often accounts for more than 90% of all localization class bugs. Fortunately, the majority of problems in this category do not require any special linguistic skills; a careful eye for detail is all that is required to expose even the most discrete bugs in this category.

The final category of localization class bug is linguistic quality. This category of bugs are primarily mis-translations. Obviously, the ability to read the language being tested is required to identify most problems in this category of errors. We found testers spent a lot of time looking for this type of bug, but later found the majority of linguistic quality type issues reported were resolved as won’t fix. There are many reasons for this, but here is my position on this type of testing….Stop wasting the time of your test team to validate the linguistic quality of the product. If this is a problem then hire a new localizer, hire a team of ‘editors’ to review the work of the localizer, or hire a professional linguistic specialist from the target locale as an auditor. Certainly, if testers see an obvious gaff in a translation then we must report it; however, testers are generally not linguistic experts (even in their native tongue), and I would never advocate hiring testers simply based on linguistic skills nor as a manager would I want to dedicate my valuable testing resources on validating linguistic quality…that’s usually not where their expertise lies, and it probably shouldn’t be.

What’s Next

Since behavioral /usability category issues are the most prevalent type of localization class bug this series of posts will focus on localization testing designed to evaluate user interface elements and resources. The next post will expose the often single most reported bug in this category.

Adding Variability in Test Case Design

I.M.Testy — Tue, 20 Oct 2009 02:03:00 GMT

I love autumn! Yes, I am definitely a boy of summer and very much prefer warmer weather; however, there is something special about autumn. This past weekend my daughter, and my 2 friends Dongyi and her husband Yuning and I participated in the Rum Run sailboat fun race with an overnight raft up at Bainbridge Island’s Port Madison. Saturday morning was quite rainy, but the wind was blowing 15 knots with gusts to 25 knots and NOAA weather radio announcing gale force warnings in Puget Sound. Wow…what a ride! But, it was actually the rather relaxing sail back to my marina on Sunday morning that rekindled the beauty of autumn in my mind. The bright reds, golden yellows, and pastel browns of the foliage seemed to blend into a collage framed by the darkness of the waters of Puget Sound and the snow covered peaks of the Olympic mountains. The beauty of autumn reminds me about change. A sloughing of the old, the cleansing brought about by the pure white snows, eventually followed by the new and fresh growth that blossoms in spring.

Just as the earth goes through variable cycles of rejuvenation, we must also continually update our tests, and (more importantly) the test data we use in our test cases to prevent them from becoming stale. Trees shed their leaves in the autumn and new leaves emerge in the spring, but the tree is fundamentally still the same tree. Similarly, a well-designed test case has a unique fundamental purpose and by changing the variables we can grow the value of that test case. Of course, the cycle of change in test data should be dramatically shorter in duration as compared to the seasonal changes of mother earth.

Here is a simple example of how a well-designed test case using variable test data can increase the value of the information each test iteration provides through increased confidence and also potentially reduce overall risk. In my role at Microsoft I am in a unique position to not only conduct controlled studies, but I can also implement ideas into practice on enterprise level software projects. One experiment I started about 2 years ago involved multiple groups of testers (sessions) located around the world divided into 3 separate control groups. Each control group tested the identical web page that would display the stock price if the user input a valid stock ticker symbol into a single textbox on the page and pressed the OK button. The only difference in the control groups was the instructions to perform single positive test case with the specific purpose of “ensure any valid stock ticker symbol displays the current stock price for the publicly traded stock specified by its symbol.” The purpose of the study was to determine if different cultural and experiential backgrounds impacted the test data used in a test based on the instructions for a test case. The study collected demographic information on the participants as well as specific inputs applied to the web page. Information on the oracle used by the students was collected anecdotally. Step one in each test was identical because we were not interested in how the tester launched the browser. (Of course this assumes there are other tests that test the multitude of ways to launch a browser and navigate to a URL. Also, if the browser failed to launch the test case is blocked.)

Group 1 was given the most vague instructions for the test case. The instruction was simply:

Launch browser and navigate to [url address]
Enter a valid stock ticker symbol and press the OK button and verify the accuracy of the returned stock price.

The instructions in the test case given to Group 2 were also somewhat vague, but provided a little guidance both on input and oracle.

Launch browser and navigate to [url address]
Enter a valid stock ticker symbol (e.g. “MSFT”)
Press the OK button
Verify the returned stock price is identical to the current stock price listed on the appropriate exchange

Group 3 had similar instructions to Group 2, but the group was given additional guidance as indicated below.

Launch browser and navigate to [url address]

Enter a valid stock ticker symbol from a publicly traded stock listed on any public stock exchange

Listings of valid stock ticker symbols are on stock exchange web sites such as:
http://www.nyse.com
http://www.eoddata.com/Symbols.aspx
http://www.nasdaq.com
http://www.londonstockexchange.com

Press the OK Button

Verify the returned stock price is identical to the current stock price listed on the appropriate exchange

Results

The results were mostly not surprising, but rather reinforcing. For example, we expected Group 1 to be rather random, but mostly aligned with ticker symbols they were familiar with. Of course, the majority (90%) of stock ticker symbols entered was MSFT and there was no significant difference in cultural background, locale, experience or educational background. (As this study was conducted at Microsoft I am sure there was some bias as to the symbol entered.) What was most interesting was that testers with no formal training (no previous courses in testing, no CS degree, and read less than one discipline specific book) and with more than 2 years of test experience were approximately more likely (25%) to violate the purpose of the test and enter random or completely invalid data as their first action. In other words, instead of executing the required test their initial reaction was to immediately go on a bug hunt.

In group 2 99% of the participants simply entered the stock ticker symbol “MSFT.” But, what was even more surprising was the fact that one the next day, the same people in that group were given the same exact test, and 95% of them simply reentered MSFT. Perhaps this is laziness, perhaps this is related to the superficial nature of the study, or perhaps this is due to individuals taking the path of least resistance. The percentage of people who entered identical stock ticker symbols on consecutive days was not significantly different between group 1 and group 2.

It should be no surprise that group 3 had the greatest distribution of variable test data applied to the web page. Demographics had no impact on any of the people who were in group 3. The majority of people in group 3 (78%) would select the first stock exchange listed (regardless of what link it was) but there was no significant overlap in the selected stock ticker symbols. When asked to repeat the test on the next day 83% of the participants selected a different link and and a different symbol. Of those who selected the same link 97% selected a different stock ticker symbol. On the down side, approximately 4% of the people simply took the path of least resistance and input MSFT as the test data on both days of the experiment.

Conclusion

One of the most common problems I hear about ‘scripted,’ or pre-defined test cases is that they are too prescriptive and not flexible enough to allow the tester to try things. Of course, a well-designed test case is not simply a prescriptive set of steps inputting the same hard coded test data they run over and over. So, in this study we made the assumption that a scripted test case that specified “Enter MSFT in the textbox” would simply result in the tester entering “MSFT” without any thinking on the part of the tester. Hard-coding variable test data is often times the worse possible way to design a test case.

Vaguely written test cases added some level of variability, but also seemed to increase the probability of the tester executing context free tests outside the scope of the purpose of the test. In fact, what we found was some testers (approx 2%) simply went on a bug hunt and never actually input a valid stock ticker symbol at all during the session.

A test case that provided only one example that is representative of the type of test data required for the test case produced the least desirable results in this study. I am not sure this would be the case in practice. However, based on this study if I were to outsource execution of a test case similar to that used by group 2 the only thing I could guarantee is that MSFT would definitely be tested numerous times, and the variability of other test data would be extremely limited regardless of the number of testers executing that test or the number of iterations.

When faced with a virtually infinite number of possibilities for input variables as test data used in either positive or negative tests we need to test as many possibilities as possible given the available resources in order to increase test coverage and reduce overall risk. So, one way increase the coverage of test data while still achieving the specific purpose of the test case is to provide useful resources that help guide the tester while relying on the tester’s creative thinking skills and curiosity to expand the test coverage.

Of course, we can also increase variability of test data and capture the essence of the tester’s creativity using a similar approach in a well-designed automated test case as well. In fact, a similarly designed automated test case enables us to significantly increase the amount of variable test data that is exercised in order to expand test coverage and increase overall confidence.

Randomizing static test data in automated tests

I.M.Testy — Sun, 11 Oct 2009 06:19:01 GMT

A significant percentage of static test data is stored in tabular comma delimited or tab-delimited formats and saved in Excel spreadsheets. Reading in comma or tab-delimited static test data into an automated test is pretty straight forward and there are numerous examples in many programming languages illustrating how to read in these types of test data repositories. Reading in rows of data is the foundation of data-driven automation and definitely has its place in any automation project.

I am a big proponent of stochastic (random) test data generation that is customized to the context, but I also know that sometimes static test data is useful for establishing baselines and more exact emulation of ‘real-world’ customer-like inputs. But, if the automated test is simply passing the same variable arguments to the same input parameters in the same order over and over again the value of subsequent iterations of that automated test using that static data set diminishes rather quickly. So how can we more effectively utilize static test data in our automated tests?

One possible solution is to randomly select an argument from a collection of static variables that is passed to the specific input parameter. The advantage of this approach is that it effectively increases the test data permutations in each iteration of the test case. For example, let’s consider 2 input parameters; one for a given name and one for a surname. In a traditional data-driven approach in which the static test data is read in by rows our test data file might be similar to:

Bob,Smith
John,Johnson
Roger,Williams
Steve,Abbot

This static data file would give us 4 sets of test data, but each time the test data is read into the test case the given and surnames are always the same.

However, if we read in the given names and surnames into 2 collections, and then randomly select a given name and surname from the appropriate collection to pass to the respective parameter we effectively have 16 possible combinations of static test data to work with. An advantage of this approach is that our ‘collections’ of given names and surnames can contain differing numbers of elements (in which case the number of possible combinations of test data is the Cartesian product of the number of elements in each collection).

Of course there are many ways to accomplish this. For example, one approach is to continue to use a comma or tab-delimited file format and list given names in one row and surnames in a second row. Another approach is to list the given names and surnames in columns in a spreadsheet and read in each column into a collection of some sort. The latter is the approach I used in developing my PseudoName test data generator tool. I chose this approach for 2 reasons; first an Excel spreadsheet is a simple yet powerful file format for storing static test data, and secondly because lists of test data are sometimes better represented in columns rather than rows.

The following code shows one way to read in test data by columns from an Excel spreadsheet.

Code Snippet
  // <copyright file="datareader.cs" company="TestingMentor">
// Copyright © 2009 by Bj Rollison. All rights reserved.
// </copyright>
 
namespace TestingMentor.TestTool
{
  using System;
  using System.Collections;
  using System.Globalization;
  using System.Runtime.InteropServices;
  using System.Threading;
  using Excel = Microsoft.Office.Interop.Excel;
 
  /// <summary>
  /// This class contains methods for reading test data from Excel spreadsheets
  /// </summary>
  public class TestDataReader
  {
    /// <summary>
    /// This method reads all the data elements in the specified number of
    /// columns in the specified Excel spreadsheet containing the test data
    /// and copies the data into a multi-dimensional array
    /// </summary>
    /// <param name="dataFileName">The filename containing the test data</param>
    /// <param name="columnCount">The number of columns in the Excel
    /// spreadsheet to read</param>
    /// <returns>A multi-dimensional array containing the data eleements for
    /// each column </returns>
    public static string[][] ExcelColumnReader(string dataFileName, uint columnCount)
    {
      CultureInfo originalCulture = null;
      Excel.Application excelApp = null;
      Excel.Workbook excelWorkbook = null;
      Excel.Worksheet excelActiveWorksheet = null;
      string[][] testData = new string[columnCount][];
 
      originalCulture = Thread.CurrentThread.CurrentCulture;
      Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");
 
      excelApp = new Excel.Application();
      excelWorkbook = excelApp.Workbooks.Open(
        dataFileName,
        0,
        false,
        5,
        String.Empty,
        String.Empty,
        false,
        Type.Missing,
        String.Empty,
        true,
        false,
        0,
        true,
        false,
        false);
      excelActiveWorksheet = (Excel.Worksheet)excelWorkbook.ActiveSheet;
 
      for (int i = 0; i < columnCount; i++)
      {
        // Start at column 1
        object columnIndex = i + 1;
 
        // Row 1 is the column title; test data starts on Row 2
        object rowIndex = 2;
        ArrayList tempCollection = new ArrayList();
        while (
          ((Excel.Range)
          excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2 != null)
        {
          tempCollection.Add(
            ((Excel.Range)
            excelActiveWorksheet.Cells[rowIndex, columnIndex]).Value2);
          rowIndex = (int)rowIndex + 1;
        }
 
        testData[i] = new string[tempCollection.Count];
        testData[i] = (string[])tempCollection.ToArray(typeof(string));
      }
 
      // Clean up 
      excelWorkbook.Close(false, Type.Missing, Type.Missing);
      excelWorkbook = null;
      excelApp.Quit();
      excelApp = null;
 
      // Garbage collection is not pretty, but necessary to release Excel proc
      System.GC.Collect();
      System.GC.WaitForPendingFinalizers();
 
      if (originalCulture != null)
      {
        Thread.CurrentThread.CurrentCulture = originalCulture;
      }
 
      return testData;
    }
  }
}
 

I must tell you that performance can be an issue especially if the columns contain a lot of data. For example, to read in approximately 700 elements of test data in 3 separate columns took slightly less than 1 second, and reading in 1800 elements in 3 columns required just over 4 seconds. Unfortunately, I didn’t compare total byte counts, but it is pretty obvious the greater the number of test data elements being read the longer the read operation will take and you certainly will have to take the read time into consideration in your automated test case.

Reading static test data line by line from a data file while looping through a data-driven automated test case is a useful test design approach in some situations, this is another useful approach that will allow the test designer to randomize the combinations of static test data values applied to multiple input parameters in multiple iterations of an automated test case.

Measuring Test Automation ROI

I.M.Testy — Tue, 25 Aug 2009 09:40:00 GMT

I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent points such as:

Automated software testing “is software development.”
Automated software testing “and manual testing are intertwined and complement each other.”
And, “The overall objective of AST (automated software testing) is to design, develop, and deliver an automated test and retest capability that increases testing efficiencies.”

Of course, I was also pleased to read the section on test data generation since I design and develop test data generation tools as a hobby. The authors correctly note that random test data increases flexibility, improve functional testing, and reduce limited in scope and error prone manually produced test data.

There is also a chapter on presenting the business case for an automation project by calculating a return on investment (ROI) measure via various worksheets. I have 2 essential problems with ROI calculations within the context of test automation. First, if the business manager doesn’t understand the value of automation within a complex software project (especially one which will have multiple iterations) they should read a book on managing software development projects. I really think most managers understand that test automation would benefit their business (in most cases). I suspect many managers have experienced less than successful automation projects but don’t understand how to establish a more successful automation effort. I also suspect really bright business managers are not overly impressed with magic beans.

Magic beans pimped by a zealous huckster are the second essential problem with automation ROI calculations. Let’s be honest, the numbers produced by these worksheets or other automation ROI calculators are simply magic beans. Now, why do I make this statement? Because the numbers that are plugged into the calculators or worksheets are ROMA data. I mean really, how many of us can realistically predict the number of atomic tests for any complex project? Also, do all tests take the same amount of time, or will all tests be executed the same number of iterations? Does it take the same amount of time to develop all automated tests, and how does one go about predicting a realistic time for all automated tests to run? And of course, how many of those tests will be automated? (Actually, that answer is easy….the number of automated tests should be 100% of the tests that should be automated.)

Personally, I think test managers should not waste their time trying to convince their business manager of the value of a test automation project; especially with magic beans produced from ROMA data. Instead test managers should start helping their team members think about ROI at the test level itself. In other words, teach your team how to make smart decisions about what tests to automate and what tests should not be automated because they can be more effectively tested via other approaches.

In my next post I will outline some factors that testers, and test managers can use to help decide which tests you might consider automating. Basically, the bottom line here is that an automated test should provide significant value to the tester and the organization, and should help free up the testers time in order to increase the breadth and/or scope of testing.

Testing is Sampling

I.M.Testy — Thu, 16 Jul 2009 09:11:21 GMT

It seems it is about this time of year that I need to detach a bit from the world to reflect back on the past year and reevaluate my personal and professional goals moving forward. Perhaps I am just getting older or perhaps just a bit wiser (that is synonymous with 'sapient' for the C-D crowd), but I find it refreshing to break away this time of year to tend to my gardens, work on my boat, read some novels, and contemplate life's joys. Now, the major work projects are (almost) finished on my boat, the garden is planted and we are harvesting the early produce, and I reset both personal and professional development objectives for the next year and beyond. So, let me get back to sharing some of my ideas about testing.

Many of you who read this blog also know of my website Testing Mentor where I post a few job aids and random test data generation tools I've created. I am a big proponent of random test data using an approach I refer to as probabilistic stochastic test data. In May I was in Dusseldorf, Germany at the Software & Systems Quality Conference to present a talk on my approach. I especially enjoy these SQS conferences (now igniteQ) because the attendees are a mix of industry experts and academia, and I was looking for feedback on my approach. I call my approach probabilistic stochastic test generation because the process is a bit more complex than simple random data generation. Similar to random data generation we cannot absolutely predict a probabilistic system, but we can control the feasibility of specified behaviors. And the adjective stochastic simply means "pertaining to a process involving a randomly determined sequence of observations each of which is considered as a sample of one element from a probability distribution." In a nutshell, my approach involves segregating the population into equivalence partitions, then randomly selects elements from specified parameterized equivalence partitions (which is how we know the probability of specific behaviors), finally the data may be mutated until the test data satisfies the defined fitness criteria. By combining equivalence partitioning and basic evolutionary computation (EA) concepts it is possible to generate large amounts of random test data that is representative from a virtually infinite population of possible data.

One of the questions that came up during the presentation was how many random samples are required for confidence in any given test case; in other words how to we determine the number of tests using randomly generated test data? This is not an easy question to answer because the sample size of any given population depends on several factors such as:

variability of data
precision of measurement
population size
risk factors
allowable sampling error
purpose of experiment or test
probability of selecting "bad" or uninteresting data

Using sampling for equivalence class partition testing

But, the question also brought to mind a parallel discussion regarding how we go about selecting elements from equivalence class partition subsets. I am adamantly opposed to hard-coding test data in a test case (automated or manual), but a colleague challenged me and said that since any element in an equivalent partition is representative of all elements in that partition then why can't we simple choose a few values from that equivalence subset. I realize this approach is done all the time by many testers; which is perhaps why we sometimes miss problems. But, hard-coding some small subset of values from a relatively large population of possible values is rarely a good idea, and is generally not the most effective approach for robust test design. One problem with hard-coding a variable is that the hard-coded value becomes static, and we know that static test data loses its effectiveness over time in subsequent tests using the same exact test data. Also, by hard-coding specific values in range of values means that we have absolutely 0% probability of including any other values in that range that are not specified. Another problem with hard-coded values stems from the selection criteria used to choose the values from a set of possible values. Typically we select values from a set based on based historical failure indicators, customer data, and our own biased judgment or intuition of ‘interesting’ values.

However, the problem is that any equivalence class partition is a hypothesis that all elements are equal. Of course, the only way to validate or affirm that hypothesis is to test the entire population of the given equivalence class partition. Using customer-like values, or values based on failure indicators, and especially values we select based on our intuition are biased samples of the population, and may only represent a small portion of the entire population. Also, the number of values selected from any given equivalence partition set is usually fewer than the number required for some reasonable level of statistical confidence. So, while we definitely want to include values representative of our customers, values derived from historical failure indicators, and even our own intuition, we should also apply scientific sampling methods and include unbiased, randomly sampled values or elements from our set of values or population to help reduce uncertainty and increase confidence.

For example, lets say that we are testing font size in Microsoft Word. Most font sizes range from 1pt through 1638pt and include half-sized fonts as well within that range. That is a population size of 3273 possible values. If we suspected that any value in the population had an equal probability of causing an error the standard deviation would be 50%. In this example, we would need a sample size of 343 statistically unbiased randomly selected values from the population to assert a 95% confidence level with a sampling error or precision of ±5%. Even in this situation, the number of values may appear to be quite large if the tests are manually executed which is perhaps one reason why extremely small subsets of hard-coded values fail to find problems that are exposed by other values within that equivalent partition (all too often after the software is released). Fortunately, statistical sampling is much easier and less costly with automated test cases and probabilistic random test data generation.

Testing is Sampling

Statistical sampling is commonly used for experimentation in natural sciences as well as studies in social sciences (where I first learned it while studying sociology an anthropology). And, if we really stop to think about it; any testing effort is simply a sample of tests of the virtually impossible infinite population of possible tests. Of course, there is always the probability that sampling misses or overlooks something interesting. But, this is true of any approach to testing and explained by B. Beizer's Pesticide Paradox. The question we must ask ourselves is will statistical sampling of values in equivalence partitions or other test data help improve my confidence when used in conjunction with customer representative data, historical data, and data we intuit based on experience and knowledge? Will scientifically quantified empirical evidence help increase the confidence of the decision makers?

In my opinion anything that helps improve confidence and provides empirical evidence is valuable, and statistical sampling is a tool we should understand put into our professional testing toolbox. There are several well established formulas for calculating sample size that can help us establish a baseline for a desired confidence level. But, rather than belabor you with formulas, I decided to whip together a Statistical Sample Size Calculator that I posted to CodePlex and also on my Testing Mentor site to help testers determine the minimum number of samples of statistically unbiased randomly generated test data from a given equivalence partition to use in a test case to help establish a statistically reliable level of confidence.

Cockamamie chaos causes confusion; controlled chaos cultivates confidence!

Better Bug Reports

I.M.Testy — Wed, 20 May 2009 20:58:52 GMT

When we report a bug our hope is that bug is fixed. But, of course we know that isn’t always the case which is why there are usually several alternative resolutions developers, project managers, or managers may choose for resolving a bug such as postponed, won’t fix, and by design. It is unfortunately quite common to see a tester metaphorically explode into passionate fits of outrage when one of their bugs is resolved as postponed, won’t fix, or by design. It is unfortunate because these tantrums often involve the tester hurling personal insults (e.g. “How can the developer be so stupid not to fix this bug"?”), decrying product quality (e.g. “If we don’t fix this bug this product will totally suck!”), and playing the whiny customer card (e.g. “We will loose customers if we don’t fix this bug.”). Yes, in my early years I was also guilty of these sorts of irrational outbursts of hyperbole when a bug that I thought was important was resolved not fixed. But, of course, I quickly learned that such sophistical speculations rarely resulted in the bug being fixed, and mostly lessened my credibility with developers and managers.

The other day I was speaking with a tester who was a bit miffed because the developer had resolved a few of her bugs as by design and won’t fix and she asked how she could ‘fight’ these resolutions. “Well,” I began, “Getting people to change their minds usually involves negotiation and the logical presentation of facts in a non-judgmental approach. Sometimes you will succeed, and sometimes you will not succeed. As testers surely we want all our bugs to be fixed; however, from a practical standpoint that may not always be the case especially if the bug is subjective.” I previously wrote about 10 common problems with bug reporting, but, in this case I proceeded to discuss a few strategies I use to advocate bugs.

Make it easy for the developer to fix the bug

As a minimum a tester must provide a description of the problem, the environmental conditions in which the problem occurred (if localized to a specific environment), the shortest number of exact steps to reproduce the bug, and the actual results versus the expected results. Occasionally a screen shot may be beneficial, but mostly if there is a contrasting example. But, I will also point the developer to my test; especially if it is automated. Providing the developer an automated mechanism to reproduce a problem reduces a lot of overhead. Of course, in this case I am talking about an automated test case that runs in a few seconds, or an automated script that even assists the developer reproduce the problem quickly.

Provide specific contradictions to specified and/or implied requirements or standards

Of course, if the product design or functionality deviates from stated requirements pointing this out in a non-confrontational way is a no-brainer. The key here is our argument must be non-confrontational because sometimes we may misinterpret the requirements, and sometimes the requirements may change without us being aware of those changes. There are also occasionally deviations from implied requirements such a UI design guidelines as a result of the introduction of new technologies, or changes in how customers use the product based on usability studies. Other implied standards include competing products or previous versions of the product. In any case, when arguing for a bug fix based on specified or implied requirements I recommend using a compare and contrast type of approach to better illustrate the problem as I perceive it.

Provide concrete examples of customer impact

This is really important! Providing a real world scenario that clearly illustrates not only how this bug will manifest itself to the customer, but also providing corroborating evidence from customers presents a strong case in favor of a bug fix. There are several useful repositories of customer feedback testers can use to bolster their point of view such as newsgroups, popular blogs, trade journal reviews of past or similar products, at Microsoft we also have Watson and SQM data, and product support reports. Using ‘real-world’ constructive feedback is often more meaningful than an internal mutiny by a portion of the test team.

Know your primary target customer profile

Testers often like to think we are representative of our customers. However, this may not always be the case. (It has always puzzled me as to why testers seem to think they have some greater affinity to the end user customer as compared to others on the product team.) Yes, it is important that testers understand who the primary target customer is for the current project or release and that is why many teams have detailed personas of primary, secondary, and sometimes even tertiary customer audiences. Of course, if we are in the commercial software business we want our customer base to be as large as possible. But, as the number of customers increase so does the diversity of value, and as they say…you can never please everyone! So, when defending your position to fix a particular bug it is always better to frame the discussion from the point of view of the primary customer persona as compared to your own personal bias.

Use your brain, not your emotions

Passion has long been an admired trait in software testers. However, unbridled passion fraught with antagonistic accusations can be detrimental to a successful bug resolution (and sometimes even a career). Some bugs obviously need to be fixed, while others may be more dependent on several mitigating (and competing) factors such as where you are in the software lifecycle, business impact, primary customer impact, risk, etc. I think it is largely agreed that perhaps the primary role of testers is to provide information, but that means we must also gather the pertinent information and represent that information logically within the relevant context to the management team (or decision makers). Remember…reckless rants rarely render reasonable results!

Programming Paradigms in Test Automation

I.M.Testy — Thu, 14 May 2009 06:26:00 GMT

Regardless of the personal opinions of a few people, the simple fact is that the demand for software testers who can design and develop effective test automation is increasing. Perhaps one reason for the distain by some folks in the industry is due to limitations of the test automation approach they are most familiar with, and they sometimes assume those limitations apply to all types of test automation. However, not all test automation approaches are equal, and there are advantages and disadvantages for any approach.

At its core an automated test case is software code, and similar to the various approaches used in developing product software there are different programming paradigms used to develop test automation such as:

Record and playback automation
Keyword or action-word driven automation
Scripted automation
Procedural automation
Model based automation

Record and playback automation

The record and playback paradigm simply records sequences of keyboard and mouse events, auto-magically codifies them usually into some proprietary scripting language which can then be replayed (executed) over and over again. There are usually severe limitations to this type of automation and it tends to be extremely fragile requiring constant massaging (re-recording). Although many record/playback paradigm allows 'test developers' to modify the scripted actions to some extent, and possibly even incorporate simple yes/no oracles I think many people view the record/playback paradigm as being slightly more useful than trained monkeys in limited situations.

Keyword or action-word driven automation

Keyword or action-words are simple scripts usually in some tabular format that 'describe' a sequence of 'actions' for the computer to perform. Of course, the key to keywords is the underlying architecture of the tool that interprets the keywords and executes the sequence of events. The beauty of keyword driven testing is that it hides the actual code, and similar to record and playback can be more easily used by business analysts or 'user domain experts' hired to into testing roles to automate something. I do see the benefit of keyword driven testing in some limited contexts (especially for companies who rely on business analysts/user domain experts for testing software), but let's be real…these people aren't automating anything…they are simply filling out a form that is then feed into a tool that performs the actions as prescribed by the listed instructions. The keyword form does nothing by itself, and the only thing a 'tester' has to think about is using the correct key words to sequentially get from point A to point Z for a 'test.'

Scripted automation (imperative programming)

The primary difference between keyword and scripted automation is the tester actually develops the test in a programming language rather than filling in a form with abstracted key words that drive some automation engine. However, similar to keywords, scripted automation tends to use rudimentary statements of basic instructions that manipulate the software to perform a pre-determined sequence of events as illustrated below.

   1: def test_b_googlenews   2:      3:   #-------------------------------------------------------------------------   4:   # Test to demonstrate WATIR select from drop-down box functionality   5:   #   6:      7:   #variables   8:   test_site = 'http://news.google.com'   9:    10:   puts '## Beginning of test: google news use drop-down box'  11:   puts '  '  12:    13:   puts 'Step 1: go to the google news site: news.google.com'  14:   $browser.goto(test_site)  15:   puts '  Action: entered ' + test_site + ' in the address bar.'  16:    17:   puts 'Step 2: Select Canada from the Top Stories drop-down list'  18:   $browser.select_list( :index , 1).select("Canada English")  19:   puts '  Action: selected Canada from the drop-down list.'  20:    21:   puts 'Step 3: click the "Go" button'  22:   $browser.button(:caption, "Go").click  23:   puts '  Action: clicked the Go button.'  24:    25:   puts 'Expected Result: '  26:   puts ' - The Google News Canada site should be displayed'  27:    28:   puts 'Actual Result: Check that "Canada" appears on the page by using an assertion'  29:   assert($browser.text.include?("Canada") )  30:    31:   puts '  '  32:   puts '## End of test: google news selection'  33:    34: end # end of test_googlenews  35:    36:    37: def test_c_googleradio  38:   

Most examples of scripted automation appear as codified versions of a set of steps listed in a less-than-adequately designed manual test case using hard-coded arguments for variables, mindless progression between steps, and simple deterministic oracles. Scripted automation is probably most beneficial for automating specific sub-tasks in "computer assisted testing." However, scripted automation is usually too prescriptive, and rely heavily on nothing going wrong during the execution of the test case.

Procedural automation (procedural programming)

In procedural automation the testers also develops a test by writing a series of computational steps to achieve a desired purpose. However unlike scripted automation the procedural automation paradigm generally provides better control flow options during the execution of the automated test case, allows for greater complexity in the design, improves reuse and reduces maintenance through modularity, and can employ both deterministic and heuristic oracles.

   1: // Procedural programming example    2:     3: static void Main(string[] args)   4: {   5:   string logResult = string.Empty;   6:     7:   // Path to the data file passed as a string argument to the test case   8:   string pictTestData = args[0];   9:    10:   //Stopwatch to measure test case duration  11:   Stopwatch sw = new Stopwatch();  12:   sw.Start();  13:    14:   // Launch the AUT  15:   AutomationElement desktop = AutomationElement.RootElement;  16:   AutomationElement myAutForm = null;  17:   Process myProc = new Process();  18:   myProc.StartInfo.FileName = myConstantAutFileName;  19:   if (myProc.Start())  20:   {  21:     // Polling loop to find AUT window by window property  22:     int pollCount = 0;  23:     do  24:     {  25:       myAutForm = desktop.FindFirst(TreeScope.Children,  26:         new PropertyCondition(AutomationElement.AutomationIdProperty,  27:         myConstantAUTPropertyID));  28:       pollCount++;  29:       System.Threading.Thread.Sleep(100);  30:     }  31:     while (myAutForm == null && pollCount < 50);  32:    33:     if (myAutForm == null)  34:     {  35:       throw new Exception("Failed to find dialog");  36:     }  37:    38:     // Get UI element collection here...  39:    40:     // Call method to read in test data  41:     string[] testData = ReadTabDelimitedFile(pictTestData);  42:    43:     // iterate through each set of test data (data-driven test example)  44:     foreach (string test in testData)  45:     {  46:       // Call method to execute each set of test data and assign the return  47:       // value to the logResult variable; Oracle is separate method called   48:       // from the test method  49:       LogResultMethod(ExecuteCombinatorialTestMethod(test));  50:     }  51:    52:     // close AUT and clean-up  53:     TimeSpan ts1 = sw.Elapsed;  54:     // log test case duration...  55:    56:   // Deal with situation if AUT failed to launch  57: }

Procedural automation can be used for anything from API to GUI automated test cases designed to evaluate functionality (computational logic), non-functional areas such as stress, performance, and security, and also behavioral . Using a language similar to the programming language removes abstraction layers, and also enables other members of the team (developers) to easily review test cases.

Model based automation

Model based automation is a relatively new automation paradigm, and its complexity is beyond the scope of this single post. Basically, model based automation involves codifying abstracted state machines and state traversals and couples these parts with an automation engine that uses some form of graph traversal logic to drive the system under test between the various state machines identified in the model. In some sense model based automation is similar to exploratory testing because tests are generally not pre-determined or pre-scripted, what constitutes a single test is really hard to describe, and the oracles generally detect errant behavior (or being in an unexpected state). Personally, I think there is tremendous potential in model based automation, but the industry has just begun to scratch the surface of this automation paradigm and it is still largely misunderstood. This automation paradigm requires more complex skill sets of the person designing the test automation such as the ability to abstract important machine states as a model, and encode system behaviors. For more information about model based automation I recommend taking a look at http://research.microsoft.com/en-us/projects/specexplorer.

So, approach which is best?

In my opinion, there may be some limited value in record/playback, keyword, and scripted automation in specific contexts; however, a robust automated test case that will run on multiple environments, multiple languages, and distributed across multiple platforms without rewriting the test for each variation requires well designed tests developed using procedural automation or model based automation approach.

Assessing Tester Performance

I.M.Testy — Tue, 28 Apr 2009 19:59:00 GMT

Using context-free software product measures as personal performance indicators (KPI) is about as silly as pet rocks!

Periodically a discussion of assessing tester performance surfaces on various discussion groups. Some people offer advice such as counting bugs (or some derivation thereof), number of tests written in x amount of time, number of tests executed, % of automated tests compared to manual tests, and (my one of my least favorite measures of individual performance) % of code coverage.

The problem with all these measures is they lack context, and tend to ignore dependent variables. It is also highly likely that an astute tester can easily game the system and potentially cause detrimental problems. For example, if my manager considered one measure my performance on the number of bugs found per week, I would ask how many I had to find per week to satisfy the 'expected' criteria. Then each week I would report 2 or 3 more bugs than the 'expected' or 'average' number (in order to 'exceed' expectations), and any additional bugs I found that week, I would sit on and hold in case I was below my quota the following week. Of course, this means that bug reports are being artificially delayed which may negatively impact the overall product schedule.

The issue at hand is this bizarre desire by some simple-minded people who want an easy solution to a difficult problem. But, there is no simple formula for measuring the performance of an individual. Individual performance assessments are often somewhat subjective, and influenced by external factors identified through Human Performance Technology (HPT) research such as motivation, tools, inherent ability, processes, and even the physical environment.

A common problem I often see is unrealistic goals such as "Find the majority of bugs in my feature area." (How do we know what the majority is? What if the majority doesn't include the most important issues? etc.) Another problem I commonly see is for individuals to over-promise and under-deliver relative to their capabilities. I also see managers who dictate the same identical set of performance goals to all individuals. While there may be a few common goals, as a manager I would want to tap into the potential strengths of each individual on my team. I also have different expectations and levels of contributions from individuals depending on where they are in their career, and also based on their career aspirations.

So, as testers we must learn to establish SMART goals with our managers that include:

goals that align with my manager's goals
goals that align with the immediate goals of the product team or company
and stretch goals that illustrate continued growth and personal improvement relative to the team, group, or company goals

(This last one may be controversial; however, we shouldn't be surprised to know individual performance is never constant in relation to your peer group. )

But, (fair or not) for a variety of reasons most software companies do (at least periodically) evaluate their employee performance in some manner, the key to success is in HPT and agreeing on SMARTer goals upfront.

"Good enough" is not good enough!

I.M.Testy — Sat, 18 Apr 2009 00:02:00 GMT

This week I came across a discussion [regarding test design] in which a tester wrote, "…the main goal is having something that is 'good enough'." Every time I hear a tester utter the phrase "good enough" my head wants to explode!

Wrapping duct tape around a splint on the broken handle on my hoe is good enough to finish the job until I can go buy a new handle. While I may sometimes temporarily improvise a "good enough" solution; I am never truly satisfied with good enough, and I personally aspire to be better than good enough. My father always told me if something was worth doing, I should do it right! He also raised me to always put forth my best effort, and constantly strive to improve myself.

I seriously can't think of any professional (in any discipline) who seriously considers good enough to ever really be good enough? The "good enough" argument is the ultimate cop out! In my opinion "good enough" epitomizes an unprofessional, apathetic attitude sanctioning mediocrity.

From a job performance perspective I suspect that if we told our employers that we were going to simply design and execute tests that are "good enough" we probably wouldn't be in a job very long. I certainly would not want people on my team who are satisfied with "good enough;" I want people who want to do their best, and to strive for better!

I spent some time in the US Air Force and we often used the phrase "it's good enough for government work" to describe slop-shoddy work. So, it amazes me that some people seem to be satisfied by consciously condoning ignominious practices. But, I guess some people are taught to expend just enough effort to be good enough!

In my opinion, good enough may be "good enough for government work" or for individuals who don't have a vested interested in helping organizations improve, or who don't really care about improving themselves; but, there is no room for the slovenly "good enough" mentality among professional testers.

The quality quandary

I.M.Testy — Fri, 27 Mar 2009 18:27:00 GMT

I often find discussions about quality to be hypothetical, and in fact unless you define your specific context the word itself is nebulous, vague, or simply meaningless philosophical psycho-babble. For a while now, I previously posted my opposition to the simplistic notion that quality is value to some person. Sure, most thesaurus' equate the two words as synonyms, and the context-driven posse constantly regurgitate Weinberg's quote about "quality is value to some person." Yes, that is one perspective of quality, but only one.

In the past I have taught that quality is not value in the context as one of the goals of software testing. My definition of value is the purpose or usefulness of a product in satisfying a customer's needs or wants. My definition of quality was based on tangible aspects of the attributes or capabilities of a product from an engineering perspective. Our definition of software testing as any task designed to evaluate or assess the attributes and capabilities of a software project in relation to implicit or explicit guidelines in order to provide information to the management team (the people who make the business decisions). Those evaluations result in measurements we call quality measurements or criteria (the "essential or distinctive characteristics, properties, or attributes" that are critical for the success of our project) and that is part of the information we present to the decision makers to help them make more informed business decisions. Remember, Weinberg also stated "Thinking about measurability from the beginning is an essential part of creating a well-formed effort."

The other day I met with members of one of our research teams and they were talking about quality in terms of tenet and non-tenet quality; and I clarified that basically they were essentially talking about the customer perceptions of quality versus the engineering aspects of quality. Surely, from a holistic point the observation and reliable measurements of both perspectives of quality are important for the success of any organization. Customers usually buy/download software because it helps them satisfy some need or want. The decision of which software product to buy made be based on personal bias, or perhaps the herding instinct, or it may be more rational based on the comparison of features (measureable attributes and capabilities). Once the customer begins to use the software they form their own opinions based on expectations and/or previous experiences that provides the company with information regarding customer satisfaction.

So, the next time someone starts talking about quality stop them and ask them if they are talking about the engineering aspects of quality, or the customer perceptions of quality. They are closely related, but different perspectives of the same topic.

Exploratory testing inside the box

I.M.Testy — Fri, 20 Mar 2009 11:24:39 GMT

Much of the information about exploratory testing focuses on testing from an end-user perspective. Pundits of exploratory testing claim the approach is also useful from a white box test design approach, but I have yet to see any practical discussion or examples. But, professional testers use exploratory testing approaches all the time from a white box perspective to explore the code for untested paths. Professional testers learn about areas of the code that are at risk, and reactively design effective tests to evaluate previously untested or under-tested areas of the code.

Let's use a simple example to get started. Suppose we had to drive from Lynnwood, Washington to Puyallup, Washington without a map or (GPS auto navigation system). Just as we have 'clues' to point us in the various directions while performing exploratory testing at the user interface we have the numerous highway signs to help us navigate various routes to complete our journey. And, it is up to us to decide which route to take. The shortest route is I-405 south to SR-167. But, I-405 is always at a stand-still, so another popular route is I-5 to SR-18 east then SR-167 south. Of course, after traversing those routes a couple of times the scenery (and crawling in traffic) gets a bit boring, so we might find additional less travelled routes. But, regardless of how many times we make the journey or how many different drivers we choose to complete this journey it is highly unlikely that we will traverse every possible route in any reasonable amount of time. Some routes may not be obvious such as I-5 south to Seattle, then taking the ferry from Seattle to Bremerton and continuing to SR-310 south to SR-16, then I-5 north to SR-167. And, of course some routes are impossible (or at least so convoluted they would be improbable).

Fortunately, control flow through even complex algorithms is not as labyrinthine as the state roadways in western Washington. And, just as the department of transportation uses various tools to measure traffic volumes testers can use path profiling tools to measure frequently traversed paths through the code. We can also use code coverage tools to see what paths have or have not been traversed, and which decisions are made at branching statements. Using code coverage and profiling tools to map control flow through the algorithm we are able to more thoroughly explore the code. Using our 'map' we can learn what paths have not been traversed and even whether or not certain paths through the code are even possible. After we explore the 'map' we can more effectively design additional tests to traverse un-tested paths through an algorithm. Common structural test design techniques include to evaluate code statements, code blocks, simple decisions or branches, or multiple Boolean conditional clauses in a single predicate statement. Then, using those test designs we can execute those tests either using stubs or mock objects at the unit or component level, or through the user interface to traverse those paths to reduce overall susceptibility to risk.

I discuss the various techniques commonly used in structural testing in Chapter 6 of our book How We Test Software At Microsoft, and also address the subject here, and here. Of course, the application of structural techniques is usually referred to as code coverage analysis. But, using this simple analogy hopefully other testers can begin to understand how exploratory testing approaches are used not only from the user interface, but also below the GUI at the code level. As Boris Beizer initially stated, "all testing is essentially exploratory in nature," and code coverage analysis (analyzing code coverage results to learn about, design additional tests, then execute those tests) also makes great use of exploratory approaches inside the box.

GUI test automation is not child's play

I.M.Testy — Fri, 13 Mar 2009 01:29:28 GMT

There are many approaches to test automation from unit testing to system level testing through the GUI. Of course, the most often discussed approach is the automation approach that drives the GUI to perform some action; or GUI automation. This also happens to be the most controversial approach to test automation, and is perhaps the hardest type of automation to design and develop. One reason why an automated GUI test fails or doesn't achieve its potential is due to a lack of understanding of the "system" by the tester, which in turn leads to a poorly designed test from the outset.

This problem is especially obvious when people who may have specialized business knowledge but lack a in-depth understanding of the systems they are working on (non-technical testers) are asked to 'automate' something. The automation in this case is usually in the form of using record/playback tools or perhaps creating a rote script to drive a keyword-driven framework, and the 'test' is usually consists of nothing more than merely mimicking some contrived behavior by the 'tester.' In fact, this over-simplified view of test automation is sometimes perpetuated by tool vendors. A manager at one tool vendor said, "By automatically capturing the tester's process and documenting their keystrokes and mouse clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating."

There are 2 fundamental problems with the above quote, and this approach to 'automation' (and I use that term loosely in this context). First, while I don't totally discount the value of record/playback, and in the right context it could very well be the best approach in a specific situation, the general consensus in the industry is that due to the limited capabilities of record/playback type tools this type of automation is simply one level above using a hoard of monkeys trained to repeat a set of sequences using the keyboard and mouse. In fact, some might suggest trained monkeys may be a better alternative because bananas are much cheaper than the costs of licensing a tool and then realizing that you have to hire someone to try to patch together some proprietary script in an often vain attempt to build an automation test suite that is beyond the reasonable capabilities of a record/playback automation approach. Unfortunately, in either case the organization is usually left with a horrible mess that nobody wants to clean up. Secondly, it makes a ridiculous assumption that testers are too stupid to automate complex scenarios or test automation is a brain-dead, non-interesting, zero-value-add activity.

If simply recording or 'documenting' rudimentary scripts that essentially repeat a sequence of 'hard-coded' steps over and over again is someone's idea of well-designed test automation then I would agree that automation is a brain-dead activity. When you automate poorly designed tests, you simply get poorly designed automated tests! And, since I don't mind calling the kettle black, I will say it…recording a set of actions, or documenting a sequence of steps with hard-coded values to feed into a keyword driven architecture is not test automation! Surely it automates tasks, but well designed automated tests are much more powerful than the production of some crude script that automates the actions performed by someone sitting in front of a computer.

Of course, well designed automated tests require highly skilled professional testers with in-depth knowledge of the systems they are working on, as well as proficiency in programming concepts and languages. Similar to how doctors study anatomy, physiology, pharmacology, immunology, biochemistry, etc., professional testers need to constantly study the various systems they are tasked to test. Developing a well designed automated test is very different than simply using a tool to automatically repeat a sequence of actions. Designing robust tests (automated or not) requires not only incredible creativity and problem-solving skills, but an in-depth knowledge of the system and an understanding of how to manipulate the system programmatically.

But, if you buy into the idea that automation is simple and merely rote recording or documenting some sequence of steps performed by some person then you get exactly that; simplistic repetitive automated actions. Simple automation is simply automated simplicity.

Basic Blocks Aren't So Basic

I.M.Testy — Fri, 06 Mar 2009 09:51:08 GMT

In the book How We Test Software at Microsoft I discuss structural testing techniques. Structural testing techniques are systematic procedures designed to analyze and evaluate control flow through a program. These are classic white box test design techniques, although my friend and respected colleague Alan Richardson states in his review of the book that he also employs similar techniques on models and I have to agree with him on that point.

Also, Peter M. sent me mail pointing out a reasonably obvious bug in the code chunks on pages 118 and 119. Both functions are declared as static void, but each has a return statement. Somehow this oversight made it through the review process, but of course a return statement in a function declared as static void would cause a compiler error. (Thanks for discovering that bug Peter and letting us know so we can fix it for the 2nd edition!)

Peter also asked for further clarification of how blocks are counted, and why a test that evaluated both conditional clauses in the compound expression as true in the below example (and on page 119) results in 85.71% coverage. Unfortunately, the answer for that is not simple.

Some surprising details…

        1: public static int BlockExample1(bool cond_1, bool cond_2)

       2: {

       3:   int x = 0, y = 0, z = 0;

       4:   if (cond_1 && cond_2)

       5:   {

       6:     x = 1;

       7:     y = 2;

       8:     z = 3;

       9:   }

      10:   return x + y + z;

      11: }

The above code can be re-written as:

       1: public static int BlockExample2(bool cond_1, bool cond_2)

       2: {

       3:   int x = 0, y = 0, z = 0;

       4:   if (cond_1)

       5:   {

       6:     if (cond_2)

       7:     {

       8:       x = 1;

       9:       y = 2;

      10:       z = 3;

      11:     }

      12:   }

      13:   return x + y + z;

      14: }

First, a 'basic block' is defined as a set of contiguous executable statements with no logical branches which seems pretty straight forward. So, based on our definition of basic blocks it appears there are 4 blocks of contiguous statements. However, the conditional clauses on line 4 and line 6 in the BlockExample2 method introduce logical branches which theoretically introduce 2 implicit blocks (e.g. one block when control flow follows the true path, and another block when control flow follows the false path). So, that is essentially how the 6 blocks are determined. But, that's not the end of the story.

If we pass a Boolean true to both cond_1 and cond_2 conditional clauses the block coverage measure in BlockExample1 results in 85.71% coverage; however, the block coverage measure for BlockExample2 actually results in 100% coverage as illustrated below.

What? How can this be? Both BlockExample1 and BlockExample2 are syntactically identical. Well, to understand this we would really need to dig deeper into compilers and coverage tools. That is well beyond the boundaries of this blog, but the IL does provide some insight.

The MSIL for BlockExample1 is on the left and BlockExample2 is on the right. Now, I don't want to do a deep dive into MSIL, but those who are really observant can see that for some reason the Visual Studio compiler evaluated a branch in BlockExample1 to false (instruction IL_0008), and then instruction IL_000c compares the 2 values for equality and instruction IL_0015 appears to evaluate the optimized compound conditional expression to true. Compare that to BlockExample2 MSIL which shows the first comparison of 2 values occurs at IL_0009 and the branch is evaluated as true (IL_000f) and the second comparison of 2 values occurs at IL_0014 and again evaluates to true at instruction IL_001a.

But wait…it gets even more confusing. We typically measure structural coverage using the debug build. So, imagine my surprise when I recompiled the code using the retail build settings and again passed true arguments to the cond_1 and cond_2 parameters for BlockExample1 and BlockExample2 and the coverage tool in Visual Studio indicated these methods now only had 4 blocks, and the block coverage measure for both methods was 100% as illustrated below.

Also, interestingly enough the compiler optimized the code so both methods had identical MSIL op code instructions as illustrated below.Steve Carroll (a senior developer in Visual Studio) wrote we "shouldn't be too concerned if you can't exactly identify where all the blocks are. When you turn the optimizer on your binary, block counts are fairly unpredictable. Don't worry though, the source line coloring will almost always lead you to the parts of the code that you need to worry about targeting to get your coverage stats up."

I agree with Steve when he states block counts are unpredictable when the code is optimized (and different tools that measure block coverage may provide different results). However, I only partially with his statement that source line coloring leading us to parts of the code we need to test. Maybe it will, maybe it won't. But, professional testers performing an in-depth analysis of code coverage results will help us identify important parts of the code that require further investigation and testing.

So, what does it all mean?

Block testing is useful for unit testing and designing white box tests for switch statements and exception handlers (based on how we can track control flow through source code using a debugger as opposed to through the IL Disassembler). But, as I stated in How We Test Software at Microsoft block testing is the weakest form of structural testing. But, it does provide a different perspective as compared to other structural approaches or techniques and is useful when used by a professional tester in the right context.

But, the important point here is that just as we wouldn't rely on only one tool to tune the carburetor on an automobile, we certainly would rely on only one technique or approach for designing structural tests; and we certainly wouldn't only rely on structural testing as a single approach to testing. This example further reinforces another important point that I make in the book; code coverage is not directly related to quality. Any professional tester can clearly see that although we are able to achieve high levels of coverage with one test, these methods are not at all well tested.

Only a fool would use code coverage metrics to derive some measure of quality, or suggest the implication that high coverage measures equal greater quality. In truth, the value of code coverage is in its ability to help professional testers identify areas of the code that have not been previously exercised and to design tests to evaluate those areas of the code more effectively to help reduce overall risk.

If we don't execute an area of code then we have zero probability of exposing errors in that code if they exist. However, just because we do execute a code statement doesn't mean we expose all potential errors. But, it at least increases the probability from 0% and helps reduce risk.