I. M. Testy

Better Bug Reports

I.M.Testy — Wed, 20 May 2009 17:58:52 GMT

When we report a bug our hope is that bug is fixed. But, of course we know that isn’t always the case which is why there are usually several alternative resolutions developers, project managers, or managers may choose for resolving a bug such as postponed, won’t fix, and by design. It is unfortunately quite common to see a tester metaphorically explode into passionate fits of outrage when one of their bugs is resolved as postponed, won’t fix, or by design. It is unfortunate because these tantrums often involve the tester hurling personal insults (e.g. “How can the developer be so stupid not to fix this bug"?”), decrying product quality (e.g. “If we don’t fix this bug this product will totally suck!”), and playing the whiny customer card (e.g. “We will loose customers if we don’t fix this bug.”). Yes, in my early years I was also guilty of these sorts of irrational outbursts of hyperbole when a bug that I thought was important was resolved not fixed. But, of course, I quickly learned that such sophistical speculations rarely resulted in the bug being fixed, and mostly lessened my credibility with developers and managers.

The other day I was speaking with a tester who was a bit miffed because the developer had resolved a few of her bugs as by design and won’t fix and she asked how she could ‘fight’ these resolutions. “Well,” I began, “Getting people to change their minds usually involves negotiation and the logical presentation of facts in a non-judgmental approach. Sometimes you will succeed, and sometimes you will not succeed. As testers surely we want all our bugs to be fixed; however, from a practical standpoint that may not always be the case especially if the bug is subjective.” I previously wrote about 10 common problems with bug reporting, but, in this case I proceeded to discuss a few strategies I use to advocate bugs.

Make it easy for the developer to fix the bug

As a minimum a tester must provide a description of the problem, the environmental conditions in which the problem occurred (if localized to a specific environment), the shortest number of exact steps to reproduce the bug, and the actual results versus the expected results. Occasionally a screen shot may be beneficial, but mostly if there is a contrasting example. But, I will also point the developer to my test; especially if it is automated. Providing the developer an automated mechanism to reproduce a problem reduces a lot of overhead. Of course, in this case I am talking about an automated test case that runs in a few seconds, or an automated script that even assists the developer reproduce the problem quickly.

Provide specific contradictions to specified and/or implied requirements or standards

Of course, if the product design or functionality deviates from stated requirements pointing this out in a non-confrontational way is a no-brainer. The key here is our argument must be non-confrontational because sometimes we may misinterpret the requirements, and sometimes the requirements may change without us being aware of those changes. There are also occasionally deviations from implied requirements such a UI design guidelines as a result of the introduction of new technologies, or changes in how customers use the product based on usability studies. Other implied standards include competing products or previous versions of the product. In any case, when arguing for a bug fix based on specified or implied requirements I recommend using a compare and contrast type of approach to better illustrate the problem as I perceive it.

Provide concrete examples of customer impact

This is really important! Providing a real world scenario that clearly illustrates not only how this bug will manifest itself to the customer, but also providing corroborating evidence from customers presents a strong case in favor of a bug fix. There are several useful repositories of customer feedback testers can use to bolster their point of view such as newsgroups, popular blogs, trade journal reviews of past or similar products, at Microsoft we also have Watson and SQM data, and product support reports. Using ‘real-world’ constructive feedback is often more meaningful than an internal mutiny by a portion of the test team.

Know your primary target customer profile

Testers often like to think we are representative of our customers. However, this may not always be the case. (It has always puzzled me as to why testers seem to think they have some greater affinity to the end user customer as compared to others on the product team.) Yes, it is important that testers understand who the primary target customer is for the current project or release and that is why many teams have detailed personas of primary, secondary, and sometimes even tertiary customer audiences. Of course, if we are in the commercial software business we want our customer base to be as large as possible. But, as the number of customers increase so does the diversity of value, and as they say…you can never please everyone! So, when defending your position to fix a particular bug it is always better to frame the discussion from the point of view of the primary customer persona as compared to your own personal bias.

Use your brain, not your emotions

Passion has long been an admired trait in software testers. However, unbridled passion fraught with antagonistic accusations can be detrimental to a successful bug resolution (and sometimes even a career). Some bugs obviously need to be fixed, while others may be more dependent on several mitigating (and competing) factors such as where you are in the software lifecycle, business impact, primary customer impact, risk, etc. I think it is largely agreed that perhaps the primary role of testers is to provide information, but that means we must also gather the pertinent information and represent that information logically within the relevant context to the management team (or decision makers). Remember…reckless rants rarely render reasonable results!

Programming Paradigms in Test Automation

I.M.Testy — Thu, 14 May 2009 03:26:00 GMT

Regardless of the personal opinions of a few people, the simple fact is that the demand for software testers who can design and develop effective test automation is increasing. Perhaps one reason for the distain by some folks in the industry is due to limitations of the test automation approach they are most familiar with, and they sometimes assume those limitations apply to all types of test automation. However, not all test automation approaches are equal, and there are advantages and disadvantages for any approach.

At its core an automated test case is software code, and similar to the various approaches used in developing product software there are different programming paradigms used to develop test automation such as:

Record and playback automation
Keyword or action-word driven automation
Scripted automation
Procedural automation
Model based automation

Record and playback automation

The record and playback paradigm simply records sequences of keyboard and mouse events, auto-magically codifies them usually into some proprietary scripting language which can then be replayed (executed) over and over again. There are usually severe limitations to this type of automation and it tends to be extremely fragile requiring constant massaging (re-recording). Although many record/playback paradigm allows 'test developers' to modify the scripted actions to some extent, and possibly even incorporate simple yes/no oracles I think many people view the record/playback paradigm as being slightly more useful than trained monkeys in limited situations.

Keyword or action-word driven automation

Keyword or action-words are simple scripts usually in some tabular format that 'describe' a sequence of 'actions' for the computer to perform. Of course, the key to keywords is the underlying architecture of the tool that interprets the keywords and executes the sequence of events. The beauty of keyword driven testing is that it hides the actual code, and similar to record and playback can be more easily used by business analysts or 'user domain experts' hired to into testing roles to automate something. I do see the benefit of keyword driven testing in some limited contexts (especially for companies who rely on business analysts/user domain experts for testing software), but let's be real…these people aren't automating anything…they are simply filling out a form that is then feed into a tool that performs the actions as prescribed by the listed instructions. The keyword form does nothing by itself, and the only thing a 'tester' has to think about is using the correct key words to sequentially get from point A to point Z for a 'test.'

Scripted automation (imperative programming)

The primary difference between keyword and scripted automation is the tester actually develops the test in a programming language rather than filling in a form with abstracted key words that drive some automation engine. However, similar to keywords, scripted automation tends to use rudimentary statements of basic instructions that manipulate the software to perform a pre-determined sequence of events as illustrated below.

   1: def test_b_googlenews   2:      3:   #-------------------------------------------------------------------------   4:   # Test to demonstrate WATIR select from drop-down box functionality   5:   #   6:      7:   #variables   8:   test_site = 'http://news.google.com'   9:    10:   puts '## Beginning of test: google news use drop-down box'  11:   puts '  '  12:    13:   puts 'Step 1: go to the google news site: news.google.com'  14:   $browser.goto(test_site)  15:   puts '  Action: entered ' + test_site + ' in the address bar.'  16:    17:   puts 'Step 2: Select Canada from the Top Stories drop-down list'  18:   $browser.select_list( :index , 1).select("Canada English")  19:   puts '  Action: selected Canada from the drop-down list.'  20:    21:   puts 'Step 3: click the "Go" button'  22:   $browser.button(:caption, "Go").click  23:   puts '  Action: clicked the Go button.'  24:    25:   puts 'Expected Result: '  26:   puts ' - The Google News Canada site should be displayed'  27:    28:   puts 'Actual Result: Check that "Canada" appears on the page by using an assertion'  29:   assert($browser.text.include?("Canada") )  30:    31:   puts '  '  32:   puts '## End of test: google news selection'  33:    34: end # end of test_googlenews  35:    36:    37: def test_c_googleradio  38:   

Most examples of scripted automation appear as codified versions of a set of steps listed in a less-than-adequately designed manual test case using hard-coded arguments for variables, mindless progression between steps, and simple deterministic oracles. Scripted automation is probably most beneficial for automating specific sub-tasks in "computer assisted testing." However, scripted automation is usually too prescriptive, and rely heavily on nothing going wrong during the execution of the test case.

Procedural automation (procedural programming)

In procedural automation the testers also develops a test by writing a series of computational steps to achieve a desired purpose. However unlike scripted automation the procedural automation paradigm generally provides better control flow options during the execution of the automated test case, allows for greater complexity in the design, improves reuse and reduces maintenance through modularity, and can employ both deterministic and heuristic oracles.

   1: // Procedural programming example    2:     3: static void Main(string[] args)   4: {   5:   string logResult = string.Empty;   6:     7:   // Path to the data file passed as a string argument to the test case   8:   string pictTestData = args[0];   9:    10:   //Stopwatch to measure test case duration  11:   Stopwatch sw = new Stopwatch();  12:   sw.Start();  13:    14:   // Launch the AUT  15:   AutomationElement desktop = AutomationElement.RootElement;  16:   AutomationElement myAutForm = null;  17:   Process myProc = new Process();  18:   myProc.StartInfo.FileName = myConstantAutFileName;  19:   if (myProc.Start())  20:   {  21:     // Polling loop to find AUT window by window property  22:     int pollCount = 0;  23:     do  24:     {  25:       myAutForm = desktop.FindFirst(TreeScope.Children,  26:         new PropertyCondition(AutomationElement.AutomationIdProperty,  27:         myConstantAUTPropertyID));  28:       pollCount++;  29:       System.Threading.Thread.Sleep(100);  30:     }  31:     while (myAutForm == null && pollCount < 50);  32:    33:     if (myAutForm == null)  34:     {  35:       throw new Exception("Failed to find dialog");  36:     }  37:    38:     // Get UI element collection here...  39:    40:     // Call method to read in test data  41:     string[] testData = ReadTabDelimitedFile(pictTestData);  42:    43:     // iterate through each set of test data (data-driven test example)  44:     foreach (string test in testData)  45:     {  46:       // Call method to execute each set of test data and assign the return  47:       // value to the logResult variable; Oracle is separate method called   48:       // from the test method  49:       LogResultMethod(ExecuteCombinatorialTestMethod(test));  50:     }  51:    52:     // close AUT and clean-up  53:     TimeSpan ts1 = sw.Elapsed;  54:     // log test case duration...  55:    56:   // Deal with situation if AUT failed to launch  57: }

Procedural automation can be used for anything from API to GUI automated test cases designed to evaluate functionality (computational logic), non-functional areas such as stress, performance, and security, and also behavioral . Using a language similar to the programming language removes abstraction layers, and also enables other members of the team (developers) to easily review test cases.

Model based automation

Model based automation is a relatively new automation paradigm, and its complexity is beyond the scope of this single post. Basically, model based automation involves codifying abstracted state machines and state traversals and couples these parts with an automation engine that uses some form of graph traversal logic to drive the system under test between the various state machines identified in the model. In some sense model based automation is similar to exploratory testing because tests are generally not pre-determined or pre-scripted, what constitutes a single test is really hard to describe, and the oracles generally detect errant behavior (or being in an unexpected state). Personally, I think there is tremendous potential in model based automation, but the industry has just begun to scratch the surface of this automation paradigm and it is still largely misunderstood. This automation paradigm requires more complex skill sets of the person designing the test automation such as the ability to abstract important machine states as a model, and encode system behaviors. For more information about model based automation I recommend taking a look at http://research.microsoft.com/en-us/projects/specexplorer.

So, approach which is best?

In my opinion, there may be some limited value in record/playback, keyword, and scripted automation in specific contexts; however, a robust automated test case that will run on multiple environments, multiple languages, and distributed across multiple platforms without rewriting the test for each variation requires well designed tests developed using procedural automation or model based automation approach.

Assessing Tester Performance

I.M.Testy — Tue, 28 Apr 2009 16:59:00 GMT

Using context-free software product measures as personal performance indicators (KPI) is about as silly as pet rocks!

Periodically a discussion of assessing tester performance surfaces on various discussion groups. Some people offer advice such as counting bugs (or some derivation thereof), number of tests written in x amount of time, number of tests executed, % of automated tests compared to manual tests, and (my one of my least favorite measures of individual performance) % of code coverage.

The problem with all these measures is they lack context, and tend to ignore dependent variables. It is also highly likely that an astute tester can easily game the system and potentially cause detrimental problems. For example, if my manager considered one measure my performance on the number of bugs found per week, I would ask how many I had to find per week to satisfy the 'expected' criteria. Then each week I would report 2 or 3 more bugs than the 'expected' or 'average' number (in order to 'exceed' expectations), and any additional bugs I found that week, I would sit on and hold in case I was below my quota the following week. Of course, this means that bug reports are being artificially delayed which may negatively impact the overall product schedule.

The issue at hand is this bizarre desire by some simple-minded people who want an easy solution to a difficult problem. But, there is no simple formula for measuring the performance of an individual. Individual performance assessments are often somewhat subjective, and influenced by external factors identified through Human Performance Technology (HPT) research such as motivation, tools, inherent ability, processes, and even the physical environment.

A common problem I often see is unrealistic goals such as "Find the majority of bugs in my feature area." (How do we know what the majority is? What if the majority doesn't include the most important issues? etc.) Another problem I commonly see is for individuals to over-promise and under-deliver relative to their capabilities. I also see managers who dictate the same identical set of performance goals to all individuals. While there may be a few common goals, as a manager I would want to tap into the potential strengths of each individual on my team. I also have different expectations and levels of contributions from individuals depending on where they are in their career, and also based on their career aspirations.

So, as testers we must learn to establish SMART goals with our managers that include:

goals that align with my manager's goals
goals that align with the immediate goals of the product team or company
and stretch goals that illustrate continued growth and personal improvement relative to the team, group, or company goals

(This last one may be controversial; however, we shouldn't be surprised to know individual performance is never constant in relation to your peer group. )

But, (fair or not) for a variety of reasons most software companies do (at least periodically) evaluate their employee performance in some manner, the key to success is in HPT and agreeing on SMARTer goals upfront.

"Good enough" is not good enough!

I.M.Testy — Fri, 17 Apr 2009 21:02:00 GMT

This week I came across a discussion [regarding test design] in which a tester wrote, "…the main goal is having something that is 'good enough'." Every time I hear a tester utter the phrase "good enough" my head wants to explode!

Wrapping duct tape around a splint on the broken handle on my hoe is good enough to finish the job until I can go buy a new handle. While I may sometimes temporarily improvise a "good enough" solution; I am never truly satisfied with good enough, and I personally aspire to be better than good enough. My father always told me if something was worth doing, I should do it right! He also raised me to always put forth my best effort, and constantly strive to improve myself.

I seriously can't think of any professional (in any discipline) who seriously considers good enough to ever really be good enough? The "good enough" argument is the ultimate cop out! In my opinion "good enough" epitomizes an unprofessional, apathetic attitude sanctioning mediocrity.

From a job performance perspective I suspect that if we told our employers that we were going to simply design and execute tests that are "good enough" we probably wouldn't be in a job very long. I certainly would not want people on my team who are satisfied with "good enough;" I want people who want to do their best, and to strive for better!

I spent some time in the US Air Force and we often used the phrase "it's good enough for government work" to describe slop-shoddy work. So, it amazes me that some people seem to be satisfied by consciously condoning ignominious practices. But, I guess some people are taught to expend just enough effort to be good enough!

In my opinion, good enough may be "good enough for government work" or for individuals who don't have a vested interested in helping organizations improve, or who don't really care about improving themselves; but, there is no room for the slovenly "good enough" mentality among professional testers.

Look below the UI for more effective and robust UI automated test case designs

I.M.Testy — Tue, 14 Apr 2009 06:51:00 GMT

Last month I wrote about simplistic views of UI test automation in which some people want to pretend that recording for playback or scripting hard-coded actions and data to mimic some human’s interactions at the keyboard is an automated test. Balderdash! Automating a set of sequences or preconceived steps simply for the sake of automating or preparing an environment is perhaps what Kaner, et. el. mean when they refer to computer assisted testing; however, computer assisted testing is not the same as a well designed automated test. (And yes, computers are very good tools for completely automating some types of tests quite effectively; including the oracle.) We see a lot of computer assisted testing in UI automation projects. I suspect this occurs because people are focused on trying to automate a test the same way they or an end-user would interact with the computer rather than design the automated test to evaluate an important attribute or capability of the software in order to provide significant information to the project team and add value to the testing process.

Personally, I am not a big fan of UI automation because it is usually done poorly, and it is usually very fragile and needs constant massaging; more so than test automation that runs below the UI layer. Also, I see a lot of misuse of UI automation. For example, I recently came across a comment by one fellow that wrote, “UI Automation is not necessarily meant for testing the UI (though, we use it for that also).” What??? I do understand the need for UI automation in the testing process, and done well it can provide tremendous benefit and free up my time to actually design new tests and think more critically about what has and has not been tested. But, when I automate through the UI my test cases are primarily testing behavioral aspects of the software (end-user scenarios for example) and that UI elements call the appropriate event handlers. While UI automation can be used to test functional capabilities also, it is generally not the best approach for robust functional testing. This is especially true when the automated UI test is over-loaded with excess baggage (manipulating UI elements not directly associated with the purpose of a test). The more baggage a UI test carries, the greater the potential for maintenance nightmares.

For example, not too long ago a tester was performing international sufficiency testing of his component to ensure his feature supported multiple national conventions and custom formats supported by Windows National Language Support (NLS) APIs. He knew the steps to manipulate the national conventions and custom formats required the user to click the Start menu, select Control Panel, then click on the Regional and Language Options control panel applet, click the Customize button, select the appropriate property sheet for the national convention he wanted to customize the setting for, and finally click the OK button the the Customize dialog and the Regional Settings dialog, and verify the results. Lather, rinse, and repeat as necessary!

To make matters more complicated the sequence of steps to change these settings are slightly different between Windows Xp and Windows Vista and we certainly don’t want to write 2 separate test cases, or branch the test code depending on the operating system in this case. Complexity cultivates complication; especially with UI automation! Fortunately, this fellow also knew that essentially all underlying functionality can be accessed via Windows APIs, and that is exactly the information he was looking for. In this situation I suggested he look at the SetLocaleInfo function and within minutes he incorporated that function to efficiently resolve his problem, and his automated test was capable of testing his application on any currently supported version of the Windows operating system.

In C# automation, we can use Process Invocation Services to PInvoke this Win32 API function from Kernel32.DLL as illustrated below

   1: using System;   2: using System.Runtime.InteropServices;   3:     4: namespace TestingMentor.PInvoke   5: {   6:   class NativeMethod   7:   {   8:     [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]   9:     public static extern bool SetLocaleInfo(  10:       uint locale,   11:       uint localeType,  12:       string localeData);  13:    14:     [DllImport("kernel32.dll", SetLastError = true)]  15:     public static extern bool SetLocaleInfo(  16:       uint locale,  17:       uint localeType,  18:       int localeData);  19:   }  20:    21:   class NlsConstant  22:   {  23:     public enum Locale : uint  24:     {  25:       Invariant = 0x007F,  26:       SystemDefault = 0x0800, // use system default for setlocaleinfo  27:       UserDefault = 0x0400,  28:       Neutral = 0x0000,  29:       CustomDefault = 0X0C00, // Vista and later  30:       CustomUiDefault = 0x1400, // Vista and later  31:       CustomUnspecified = 0X1000  // Vista and later  32:     };  33:    34:     public enum LocaleType : uint  35:     {  36:       // VALUE LCDATA TYPES  37:       CalendarType = 0x00001009,  // type of calendar specifier  38:       CurrencyDigits = 0x00000019,  // local monetary fractional digits  39:       CurrencySymbol = 0x0000001B,  // position of positive currency symbol   40:       FractionalDigits = 0x00000011,  // number of fractional digits  41:       NativeDigitSubstitution = 0x00001014,  // native digit substitution  42:       FirstDayOfWeek = 0x0000100C,  // first day of week specifier  43:       FirstWeekOfYear = 0x0000100D,  // first week of year specifier  44:       LeadingZeros = 0x00000012,  // leading zeros for decimal  45:       Measure = 0x0000000D,  // 0 = metric, 1 = US  46:       NegativeCurrency = 0x0000001C,  // negative currency mode  47:       NegativeNumber = 0x00001010,  // negative number mode  48:       PaperSize = 0x0000100A,  // paper size  49:       TimeFormat = 0x00000023,  // time format specifier  50:         51:       // STRING LCDATA TYPES  52:       // Valid Unicode characters   53:       AM = 0X00000028,  // AM designator  54:       PM = 0x00000029,  // PM designator  55:       CurrencySymbol = 0x00000014,  // local monetary symbol  56:       DecimalSeparator = 0x0000000E,  // decimal separator  57:       DigitGrouping = 0x00000010,  // digit grouping  58:       ListSeparator = 0x0000000C,  // list item separator  59:       LongDate = 0x00000020,  // long date format string  60:       MonetaryDecimalSeparator = 0x00000016,  // monetary decimal separator  61:       MonetaryGrouping = 0x00000018,  // monetary grouping  62:       MonetaryThousandSeparator = 0x00000017,  // monetary thousand separator  63:       NativeDigits = 0x00000013,  // native ascii 0-9  64:       NegativeSign = 0x00000051,  // negative sign  65:       PositiveSign = 0x00000050,  // positive sign  66:       ShortDate = 0x0000001D,  // short date format string  67:       ThousandSeparator = 0x0000000F,  // thousand separator  68:       TimeSeparator = 0x0000001E,  // time separator  69:       TimeFormat = 0x00001003,  // time format string  70:       YearMonthFormat = 0x00001006   // year month format string  71:     };  72:    73:    74:     public enum LocaleData : int  75:     {  76:       // LOCALE_ICALENDARTYPE VALUES  77:       Gregorian = 1, // Gregorian (localized)  78:       GregorianUS = 2, // Gregorian(Always English)  79:       GregorianMEFrench = 9, // Middle East French  80:       GregorianArabic = 10,  81:       GregorianXlitEnglish = 11, // transliterated English  82:       GregorianXlitFrench = 12, // transliterated French  83:       Japan = 3,  84:       Taiwan = 4,  85:       Korea = 5,  86:       Hijri = 6,  87:       Thai = 7,  88:       Hebrew = 8,  89:       Umalqura = 23, // Um Al Qura (Arabic lunar) Vista or later  90:    91:       // LOCALE_ICURRENCY  92:       PositiveCurrencyPrefixNoSeparation = 0,  93:       PositiveCurrencySuffixNoSeparation = 1,  94:       PositiveCurrencyPrefixSeparation = 2, // one character separation  95:       PositiveCurrencySuffixSeparation = 3, // one character separation  96:    97:       // LOCALE_IDIGITSUBSTITUTION  98:       DigitSubstitutionContextBased = 0,  99:       DigitSubstitutionNone = 1, // use this setting for full unicode support 100:       DigitSubstitutionNative = 2, // uses digits based on national conventions 101:                                     // according to LOCALE_SNATIVEDIGITS 102:   103:       //LOCALE_IFIRSTDAYOFWEEK 104:       Monday = 0, // LOCALE_SDAYNAME1 105:       Tuesday = 1, // LOCALE_SDAYNAME2 106:       Wednesday = 2, // LOCALE_SDAYNAME3 107:       Thursday = 3, // LOCALE_SDAYNAME4 108:       Friday = 4, // LOCALE_SDAYNAME5 109:       Saturday = 5, // LOCALE_SDAYNAME6 110:       Sunday = 6, // LOCALE_SDAYNAME7 111:   112:       // LOCALE_IFRISTWEEKOFYEAR 113:       FirstDay = 0, // Week containing 1/1 even if single day 114:       FirstFullWeek = 1, // first full week following 1/1 115:       FirstWeek = 2, // first week with at least 4 days after 1/1 116:   117:       // LOCALE_ILZERO 118:       NoLeadingZero = 0,  // .975 119:       LeadingZero = 1,     // 0.975 120:   121:       // LOCALE_IMEASURE 122:       Metric = 0, 123:       US = 1, 124:   125:       // LOCALE_INEGCURR 126:       ParenthesisSymbolNumber = 0,   // ($1.1) 127:       NegativeSignSymbolNumber = 1,  // -$1.1 128:       SymbolNegativeSignNumber = 2,  // $-1.1 129:       SymbolNumberNegativeSign = 3,  // $1.1- 130:       ParenthesisNumberSymbol = 4,   // (1.1$) 131:       NegativeSignNumberSymbol = 5,  // -1.1$ 132:       NumberNegativeSignSymbol = 6,  // 1.1-$ 133:       NumberSymbolNegativeSign = 7,  // 1.1$- 134:       NegativeSignNumberSpaceSymbol = 8,  // -1.1 $ 135:       NegativeSignSymbolSpaceNumber = 9,  // -$ 1.1 136:       NumberSpaceSymbolNegativeSign = 10,  // 1.1 $- 137:       SymbolSpaceNumberNegativeSign = 11,  // $ 1.1- 138:       SymbolSpaceNegativeSignNumber = 12,  // $ -1.1 139:       NumberNegativeSignSpaceSymbol = 13,  // 1.1- $ 140:       ParenthesisSymbolSpaceNumber = 14,  // ($ 1.1) 141:       ParenthesisNumberSpaceSymbol = 15,  // (1.1 $) 142:   143:       // LOCALE_INEGNUMBER 144:       Parenthesis = 0,  // (1) 145:       NegativeSignNumber = 1,  // -1 146:       NegativeSignSpaceNumber = 2, // - 1 147:       NumberNegativeSign = 3,  // 1- 148:       NumberSpaceNegativeSign = 4,  // 1 - 149:   150:       // LOCALE_PAPERSIZE 151:       USLetter = 1, 152:       USLegal = 5, 153:       A3 = 8, 154:       A4 = 9, 155:   156:       // LOCALE_ITIME 157:       FormatAM_PM = 0, 158:       Format24Hour = 1 160:     }; 161:   } 162: }

You see, manipulating the Regional Options settings through the user interface had nothing to do with the purpose of his test; it was whether or not those changes in the NLS settings were propagated to the application under test, and whether the resultant output displayed correctly. The oracle to verify the output in this case was simply reading the string from the appropriate control in the application and comparing each character code point value with the expected character. For example, one test changed the date format from dd/mm/yyyy to yyyy-MM-dd. The automated oracle verified the year, month and day values in the correct format and order, and also checked whether the date separator characters in the 4th and 7th position in the string were Unicode values U+002D in this example (or other randomly generated Unicode character value(s)). This automated test was able to test and verify 31 different customizable NLS settings with multiple variables per setting to satisfy basic international sufficiency of this tester’s feature in a fraction of the time it would require a human, and with greater precision. Of course, this assumes that as a tester you have an in-depth understanding of the “system” on which you are tasked to test, and capable of designing effective tests from perspectives other than that of the end-user.

I try to constantly emphasize the emerging role of a software tester primarily focuses on analysis and design; analysis of the “system”, the tests, and the results of tests, and the design of effective tests with reasoned purpose and well defined goals. Professional testers provide value by enriching their organization’s intellectual knowledge repository and ultimately resolving hard problems. But, we can’t start to resolve the hard problem of effective UI test automation by perpetuating the medieval mentality that UI automation is merely mindlessly mimicking the clicks and keystrokes through the user interface because we don’t understand how the system works below the surface, or we can’t think intelligently about effective oracles capable of interpreting the results for some of our automated tests. The persistent prophets of pestilence will perpetually pule, but fortunately I see more and more professional software testers stepping up to meet increasingly complex technological challenges head on with increasing success. As I have said before, the only problems we can’t solve are those which we have not yet devised a solution.

The quality quandary

I.M.Testy — Fri, 27 Mar 2009 15:27:00 GMT

I often find discussions about quality to be hypothetical, and in fact unless you define your specific context the word itself is nebulous, vague, or simply meaningless philosophical psycho-babble. For a while now, I previously posted my opposition to the simplistic notion that quality is value to some person. Sure, most thesaurus' equate the two words as synonyms, and the context-driven posse constantly regurgitate Weinberg's quote about "quality is value to some person." Yes, that is one perspective of quality, but only one.

In the past I have taught that quality is not value in the context as one of the goals of software testing. My definition of value is the purpose or usefulness of a product in satisfying a customer's needs or wants. My definition of quality was based on tangible aspects of the attributes or capabilities of a product from an engineering perspective. Our definition of software testing as any task designed to evaluate or assess the attributes and capabilities of a software project in relation to implicit or explicit guidelines in order to provide information to the management team (the people who make the business decisions). Those evaluations result in measurements we call quality measurements or criteria (the "essential or distinctive characteristics, properties, or attributes" that are critical for the success of our project) and that is part of the information we present to the decision makers to help them make more informed business decisions. Remember, Weinberg also stated "Thinking about measurability from the beginning is an essential part of creating a well-formed effort."

The other day I met with members of one of our research teams and they were talking about quality in terms of tenet and non-tenet quality; and I clarified that basically they were essentially talking about the customer perceptions of quality versus the engineering aspects of quality. Surely, from a holistic point the observation and reliable measurements of both perspectives of quality are important for the success of any organization. Customers usually buy/download software because it helps them satisfy some need or want. The decision of which software product to buy made be based on personal bias, or perhaps the herding instinct, or it may be more rational based on the comparison of features (measureable attributes and capabilities). Once the customer begins to use the software they form their own opinions based on expectations and/or previous experiences that provides the company with information regarding customer satisfaction.

So, the next time someone starts talking about quality stop them and ask them if they are talking about the engineering aspects of quality, or the customer perceptions of quality. They are closely related, but different perspectives of the same topic.

Exploratory testing inside the box

I.M.Testy — Fri, 20 Mar 2009 08:24:39 GMT

Much of the information about exploratory testing focuses on testing from an end-user perspective. Pundits of exploratory testing claim the approach is also useful from a white box test design approach, but I have yet to see any practical discussion or examples. But, professional testers use exploratory testing approaches all the time from a white box perspective to explore the code for untested paths. Professional testers learn about areas of the code that are at risk, and reactively design effective tests to evaluate previously untested or under-tested areas of the code.

Let's use a simple example to get started. Suppose we had to drive from Lynnwood, Washington to Puyallup, Washington without a map or (GPS auto navigation system). Just as we have 'clues' to point us in the various directions while performing exploratory testing at the user interface we have the numerous highway signs to help us navigate various routes to complete our journey. And, it is up to us to decide which route to take. The shortest route is I-405 south to SR-167. But, I-405 is always at a stand-still, so another popular route is I-5 to SR-18 east then SR-167 south. Of course, after traversing those routes a couple of times the scenery (and crawling in traffic) gets a bit boring, so we might find additional less travelled routes. But, regardless of how many times we make the journey or how many different drivers we choose to complete this journey it is highly unlikely that we will traverse every possible route in any reasonable amount of time. Some routes may not be obvious such as I-5 south to Seattle, then taking the ferry from Seattle to Bremerton and continuing to SR-310 south to SR-16, then I-5 north to SR-167. And, of course some routes are impossible (or at least so convoluted they would be improbable).

Fortunately, control flow through even complex algorithms is not as labyrinthine as the state roadways in western Washington. And, just as the department of transportation uses various tools to measure traffic volumes testers can use path profiling tools to measure frequently traversed paths through the code. We can also use code coverage tools to see what paths have or have not been traversed, and which decisions are made at branching statements. Using code coverage and profiling tools to map control flow through the algorithm we are able to more thoroughly explore the code. Using our 'map' we can learn what paths have not been traversed and even whether or not certain paths through the code are even possible. After we explore the 'map' we can more effectively design additional tests to traverse un-tested paths through an algorithm. Common structural test design techniques include to evaluate code statements, code blocks, simple decisions or branches, or multiple Boolean conditional clauses in a single predicate statement. Then, using those test designs we can execute those tests either using stubs or mock objects at the unit or component level, or through the user interface to traverse those paths to reduce overall susceptibility to risk.

I discuss the various techniques commonly used in structural testing in Chapter 6 of our book How We Test Software At Microsoft, and also address the subject here, and here. Of course, the application of structural techniques is usually referred to as code coverage analysis. But, using this simple analogy hopefully other testers can begin to understand how exploratory testing approaches are used not only from the user interface, but also below the GUI at the code level. As Boris Beizer initially stated, "all testing is essentially exploratory in nature," and code coverage analysis (analyzing code coverage results to learn about, design additional tests, then execute those tests) also makes great use of exploratory approaches inside the box.

GUI test automation is not child's play

I.M.Testy — Thu, 12 Mar 2009 22:29:28 GMT

There are many approaches to test automation from unit testing to system level testing through the GUI. Of course, the most often discussed approach is the automation approach that drives the GUI to perform some action; or GUI automation. This also happens to be the most controversial approach to test automation, and is perhaps the hardest type of automation to design and develop. One reason why an automated GUI test fails or doesn't achieve its potential is due to a lack of understanding of the "system" by the tester, which in turn leads to a poorly designed test from the outset.

This problem is especially obvious when people who may have specialized business knowledge but lack a in-depth understanding of the systems they are working on (non-technical testers) are asked to 'automate' something. The automation in this case is usually in the form of using record/playback tools or perhaps creating a rote script to drive a keyword-driven framework, and the 'test' is usually consists of nothing more than merely mimicking some contrived behavior by the 'tester.' In fact, this over-simplified view of test automation is sometimes perpetuated by tool vendors. A manager at one tool vendor said, "By automatically capturing the tester's process and documenting their keystrokes and mouse clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating."

There are 2 fundamental problems with the above quote, and this approach to 'automation' (and I use that term loosely in this context). First, while I don't totally discount the value of record/playback, and in the right context it could very well be the best approach in a specific situation, the general consensus in the industry is that due to the limited capabilities of record/playback type tools this type of automation is simply one level above using a hoard of monkeys trained to repeat a set of sequences using the keyboard and mouse. In fact, some might suggest trained monkeys may be a better alternative because bananas are much cheaper than the costs of licensing a tool and then realizing that you have to hire someone to try to patch together some proprietary script in an often vain attempt to build an automation test suite that is beyond the reasonable capabilities of a record/playback automation approach. Unfortunately, in either case the organization is usually left with a horrible mess that nobody wants to clean up. Secondly, it makes a ridiculous assumption that testers are too stupid to automate complex scenarios or test automation is a brain-dead, non-interesting, zero-value-add activity.

If simply recording or 'documenting' rudimentary scripts that essentially repeat a sequence of 'hard-coded' steps over and over again is someone's idea of well-designed test automation then I would agree that automation is a brain-dead activity. When you automate poorly designed tests, you simply get poorly designed automated tests! And, since I don't mind calling the kettle black, I will say it…recording a set of actions, or documenting a sequence of steps with hard-coded values to feed into a keyword driven architecture is not test automation! Surely it automates tasks, but well designed automated tests are much more powerful than the production of some crude script that automates the actions performed by someone sitting in front of a computer.

Of course, well designed automated tests require highly skilled professional testers with in-depth knowledge of the systems they are working on, as well as proficiency in programming concepts and languages. Similar to how doctors study anatomy, physiology, pharmacology, immunology, biochemistry, etc., professional testers need to constantly study the various systems they are tasked to test. Developing a well designed automated test is very different than simply using a tool to automatically repeat a sequence of actions. Designing robust tests (automated or not) requires not only incredible creativity and problem-solving skills, but an in-depth knowledge of the system and an understanding of how to manipulate the system programmatically.

But, if you buy into the idea that automation is simple and merely rote recording or documenting some sequence of steps performed by some person then you get exactly that; simplistic repetitive automated actions. Simple automation is simply automated simplicity.

Basic Blocks Aren't So Basic

I.M.Testy — Fri, 06 Mar 2009 06:51:08 GMT

In the book How We Test Software at Microsoft I discuss structural testing techniques. Structural testing techniques are systematic procedures designed to analyze and evaluate control flow through a program. These are classic white box test design techniques, although my friend and respected colleague Alan Richardson states in his review of the book that he also employs similar techniques on models and I have to agree with him on that point.

Also, Peter M. sent me mail pointing out a reasonably obvious bug in the code chunks on pages 118 and 119. Both functions are declared as static void, but each has a return statement. Somehow this oversight made it through the review process, but of course a return statement in a function declared as static void would cause a compiler error. (Thanks for discovering that bug Peter and letting us know so we can fix it for the 2nd edition!)

Peter also asked for further clarification of how blocks are counted, and why a test that evaluated both conditional clauses in the compound expression as true in the below example (and on page 119) results in 85.71% coverage. Unfortunately, the answer for that is not simple.

Some surprising details…

        1: public static int BlockExample1(bool cond_1, bool cond_2)

       2: {

       3:   int x = 0, y = 0, z = 0;

       4:   if (cond_1 && cond_2)

       5:   {

       6:     x = 1;

       7:     y = 2;

       8:     z = 3;

       9:   }

      10:   return x + y + z;

      11: }

The above code can be re-written as:

       1: public static int BlockExample2(bool cond_1, bool cond_2)

       2: {

       3:   int x = 0, y = 0, z = 0;

       4:   if (cond_1)

       5:   {

       6:     if (cond_2)

       7:     {

       8:       x = 1;

       9:       y = 2;

      10:       z = 3;

      11:     }

      12:   }

      13:   return x + y + z;

      14: }

First, a 'basic block' is defined as a set of contiguous executable statements with no logical branches which seems pretty straight forward. So, based on our definition of basic blocks it appears there are 4 blocks of contiguous statements. However, the conditional clauses on line 4 and line 6 in the BlockExample2 method introduce logical branches which theoretically introduce 2 implicit blocks (e.g. one block when control flow follows the true path, and another block when control flow follows the false path). So, that is essentially how the 6 blocks are determined. But, that's not the end of the story.

If we pass a Boolean true to both cond_1 and cond_2 conditional clauses the block coverage measure in BlockExample1 results in 85.71% coverage; however, the block coverage measure for BlockExample2 actually results in 100% coverage as illustrated below.

What? How can this be? Both BlockExample1 and BlockExample2 are syntactically identical. Well, to understand this we would really need to dig deeper into compilers and coverage tools. That is well beyond the boundaries of this blog, but the IL does provide some insight.

The MSIL for BlockExample1 is on the left and BlockExample2 is on the right. Now, I don't want to do a deep dive into MSIL, but those who are really observant can see that for some reason the Visual Studio compiler evaluated a branch in BlockExample1 to false (instruction IL_0008), and then instruction IL_000c compares the 2 values for equality and instruction IL_0015 appears to evaluate the optimized compound conditional expression to true. Compare that to BlockExample2 MSIL which shows the first comparison of 2 values occurs at IL_0009 and the branch is evaluated as true (IL_000f) and the second comparison of 2 values occurs at IL_0014 and again evaluates to true at instruction IL_001a.

But wait…it gets even more confusing. We typically measure structural coverage using the debug build. So, imagine my surprise when I recompiled the code using the retail build settings and again passed true arguments to the cond_1 and cond_2 parameters for BlockExample1 and BlockExample2 and the coverage tool in Visual Studio indicated these methods now only had 4 blocks, and the block coverage measure for both methods was 100% as illustrated below.

Also, interestingly enough the compiler optimized the code so both methods had identical MSIL op code instructions as illustrated below.Steve Carroll (a senior developer in Visual Studio) wrote we "shouldn't be too concerned if you can't exactly identify where all the blocks are. When you turn the optimizer on your binary, block counts are fairly unpredictable. Don't worry though, the source line coloring will almost always lead you to the parts of the code that you need to worry about targeting to get your coverage stats up."

I agree with Steve when he states block counts are unpredictable when the code is optimized (and different tools that measure block coverage may provide different results). However, I only partially with his statement that source line coloring leading us to parts of the code we need to test. Maybe it will, maybe it won't. But, professional testers performing an in-depth analysis of code coverage results will help us identify important parts of the code that require further investigation and testing.

So, what does it all mean?

Block testing is useful for unit testing and designing white box tests for switch statements and exception handlers (based on how we can track control flow through source code using a debugger as opposed to through the IL Disassembler). But, as I stated in How We Test Software at Microsoft block testing is the weakest form of structural testing. But, it does provide a different perspective as compared to other structural approaches or techniques and is useful when used by a professional tester in the right context.

But, the important point here is that just as we wouldn't rely on only one tool to tune the carburetor on an automobile, we certainly would rely on only one technique or approach for designing structural tests; and we certainly wouldn't only rely on structural testing as a single approach to testing. This example further reinforces another important point that I make in the book; code coverage is not directly related to quality. Any professional tester can clearly see that although we are able to achieve high levels of coverage with one test, these methods are not at all well tested.

Only a fool would use code coverage metrics to derive some measure of quality, or suggest the implication that high coverage measures equal greater quality. In truth, the value of code coverage is in its ability to help professional testers identify areas of the code that have not been previously exercised and to design tests to evaluate those areas of the code more effectively to help reduce overall risk.

If we don't execute an area of code then we have zero probability of exposing errors in that code if they exist. However, just because we do execute a code statement doesn't mean we expose all potential errors. But, it at least increases the probability from 0% and helps reduce risk.

Troubleshooting Test Data with String Decoder

I.M.Testy — Wed, 25 Feb 2009 10:12:51 GMT

I value static test data that is derived from historical failure indicators, or representative of typical end-users. But, of course a problem with static test data is that it only provides a limited set of all possible data, and becomes stale or provides little new information after multiple iterations of the test. So, I am a proponent of using random data in well-designed tests. Of course, recklessly generating random data is just plain dumb and potentially results in numerous false positives. But, when the data set is well defined and decomposed into equivalence class subsets then it is possible to generate random data that is representative of all possible data elements; probabilistic stochastic test data!

Last week I released an update to the test tool Babel for generating random strings of Unicode characters. Babel is a useful tool for comprehensive positive or negative testing of a textbox and other edit controls, and API parameters that take string arguments. Using probabilistic stochastic test data significantly increases the breadth of data coverage during a test cycle which increases the probability of exposing anomalies in string parsing and other string manipulation algorithms. But, when using characters from across the Unicode spectrum anomalies are usually caused by a specific character code point (or code points for surrogate pair characters), or combinations of characters.

Of course, telling a developer that a string composed of the characters ꁲᱚRבּ䍳㄁܁쭤࿦ኳ causes an unexpected error would most likely be met with that classic deer in headlights look followed by some muttering such as "That's not a real string" and "nobody would ever enter such a string." Often times developers are likely to shun random strings as test data, and managers might claim it is not representative of 'real' customer scenarios. So, the professional tester knows that instead of simply arguing in favor of random string testing we must troubleshoot the string to identify the specific character code point or code point combination causing the error. Because while a 'real' customer may not likely enter a string of random characters from multiple language scripts, the problem is likely caused by a single character (and sometimes the combination of character code points), and there is some probability of a customer somewhere in the world entering that problematic character! So, as professional's we must find that specific problematic character.

To help professional testers decode each character in a string to its code point value I recently completed a new tool called String Decoder. This test tool is an updated version of my old Str2Val tool (which had some serious problems when converting strings with surrogate pair characters). String Decoder will decode Unicode characters (including surrogate pairs) to their hexadecimal UTF-16 (Big or Little Endian), UTF-8, UTF-7 encoding values, or an integer value (UTF-32).

For example the characters in the string んޏ᠘㎝Xᔲ뉞ဵ have UTF-16 Big Endian encoding values displayed in the Results list in the image.

Once the specific character code point or combination is identified, the tester can now tell the developer exactly what Unicode character or integer value is causing the anomaly. For example, it is much better to state a Unicode value of U+13BD is causing unexpected functionality as compared to trying to explain how to input the Cherokee letter MU or saying "just enter this character Ꮍ."

String Decoder can also be used to compare different Unicode transformation format encodings, or convert between Unicode hex values and 32-bit integer values of characters.

Let me know what you think!

Random string generation…Update!

I.M.Testy — Tue, 17 Feb 2009 06:58:49 GMT

One of the biggest challenges in input testing is the sheer amount of potential characters and the virtually infinite number of permutations of those characters in different character positions in a string. Even if we know about the myriad of language scripts used throughout the world, manually generating characters from multiple language groups would be excruciatingly inefficient.

Since any modern application should support Unicode character we can assert the strings “abcdefg” and “ڄƥ藖꼩昨”are equivalent for most input testing requiring a Unicode string. So, random string test data generation is useful for easily increasing the breadth of test data tested, and also for testing the robustness of the applications ability to process complex data streams.

Babel 2.0 is a free test tool, and one of the few random string generators that can generate a string of character across the entire Unicode spectrum, since its initial release in 2006 it has been widely popular. So, I am happy to announce that an updated Babel 2.0 is released! I know this constitutes a shameless plug…but, sometimes it helps to plug tools we’ve made that can benefit other testers or developers.

Unlike many string generators that only produce a string of random ASCII characters, Babel can produce a string of random Unicode characters defined in the Unicode 5.1 specification, including surrogate pair characters (which often expose problems in various text boxes…hint, hint). Additional updates to Babel 2.0 include:

Updated to the Unicode 5.1 spec (including new script groups and character code points)
Ability to include/exclude combining character code points
Ability to include/exclude reserved NetBIOS characters
Custom range allows character generation from 0x01 through 0xFFFF.
Ability to generate strings with a max length of 100,000 characters
Improved distribution of characters from the selected language script groups

The following illustration provides a basic flow diagram of how Babel generates random strings. Essentially, one script group is randomly selected from all selected script group nodes, and all code points assigned to that script group are put into a collection. Next, one character is randomly selected from that collection and is appended to a string. This process continues until the string length equals a specified number of characters.

Better distribution of character selection across multiple script groups occurs by preventing the same script group from being selected before at least ½ of the other specified groups are selected. This means that as long as more than one script group node is selected the selected group of characters will be removed from the random selection process until at least half of the other script groups are chosen. This provides a greater distribution as compared to simple random generation.

The download also includes the Babel.DLL (and the dependent UnicodeData.DLL) for test automation. The older methods are deprecated and no longer supported. The new methods have been simplified and now include:

public static string Polyglot (int, int, bool, bool, bool, bool, bool)
Returns a string of random Unicode characters in all Unicode script groups based on a specified seed value.

public static string Polyglot (int, bool, bool, bool, bool, bool, out int)
Generates a random seed value and returns a string of random Unicode string of characters in all Unicode script groups, and passes a reference to the seed value.

public static string Polyglot ( int, int, bool, bool, bool, bool, bool, char, char)
Returns a string of random Unicode string of characters in all Unicode script groups based on a specified seed value

public static string Polyglot (int, bool, bool, bool, bool, bool, char, char, out int)
Generates a random seed value and returns a string of random Unicode string of characters in all Unicode script groups, and passes a reference to the seed value.

Get the new release of Babel 2.0 !

Thinking about fly fishing…

I.M.Testy — Fri, 06 Feb 2009 23:57:45 GMT

I am an avid fly-fisherman, and I am spending a few of these last winter evenings tying flies in preparation for the new year. The lakes are still too cold so the trout are deep and lethargic, and many of the rivers are closed and too damn cold and swollen anyway. So, now is a good time restock my fly boxes, reflect on past years, and dream of the up-coming season. While fly fishing is an enjoyable escape from the day to day torrent of technology, when I can’t stand in a river and wave a stick I can still relax with a good book that conjures memories of years past or engulfs my mind in an adventurous narrative as if I were there. I don’t really enjoy reading most fiction books, but I do enjoy reading the memoirs of people such as John Gierach. (Perhaps it is the sailor in me that loves a good yarn, or the fact that I can relate personally to the stories.) Anyway, this weekend I acquired a book entitled Fishless Days, Angling Nights by Sparse Gray Hackle that introduced a legend in American fly-fishing by the name of Theodore Gordon. I had never heard of Mr. Gordon before this (perhaps that is because I tend to fish a lot more soft hackle flies rather than dry flies), but I found a quote by him quite interesting. He said, “The great charm of fly fishing is that we are always learning.”

This morning I thought how apropos this statement is to software testing. But, there of course, as it pertains to the practice of software testing I would rephrase Mr. Gordon’s statement to state, “The great demand of software testing is that we must always learn.”

There are different types of people who fly fish. There are those who buy a fly rod and a pocket full of store bought flies and head to the water just to catch fish. This group of people don’t seem to care much about the techniques of casting, learning how to read water, or understand patterns of fish or aquatic entomology; they are simply there to try to catch fish using whatever slop-shod mooching approach seems to work. Yes, they still catch fish…usually the small, dumb farm raised trout stocked into regional lakes by the state’s department of wildlife who will bite at anything. On the other end of the spectrum are true purists who fish with cane poles, tie their own flies to match the hatch (sometimes right beside the river), and study trout, regional entomological lifecycles, and the geological formations of a river bed to better pin-point where the big, smart trout are hiding. Then there are the group in between these extremes with varying degrees of skill and knowledge. Depending on how much time a person devotes to both practice and learn (and their capacity to learn) about the sport will often make a huge difference in both their enjoyment and their effectiveness in catching large, persnickety trout.

From my observations of the testing community over the past few decades, I can see a similar pattern regarding the spectrum of skills and knowledge of people who participate in the practice. In the past, it was not uncommon for some companies to hire ‘clever’ people who were simply good at finding bugs into testing roles. Some companies hired developers who would (often times begrudgingly) take the job as a stepping stone into a developer position. Unfortunately, some people at both extremes of this spectrum often stagnated because they did not learn more about software testing or the technological advances that were happening around them. At one extreme, I suspect that some people thought as long as they were finding behavioral type issues they were providing a benefit to the company because they were ‘good at representing the customer.’ At the other end of the spectrum the developer’s in testing roles who failed to realize the challenges in software testing. In both extremes complacency and stagnation usually occurs. Of course, there are many other testers between these extremes; some who will go on to become professional testers and have significant impact, and others who will simply belly-ache and whine about how unfair it is or claim how wrong any change is and why change won’t work.

As professional testers we must constantly strive to improve our knowledge of testing, technology, and the systems we are working on. We must also increase our skills and abilities as the demands of the role expand beyond the traditional comfort zone of behavioral testing and ‘playing customer advocate’ by executing ‘tests’ to find bugs at the end of a cycle. The challenges of testing complex systems built around advancing technologies significantly raises the aptitude bar for testers. Emerging practices such as TDD and agile lifecycles designed to drive engineering quality upstream and form closer customer connections is also impacting the role of testing and how testing adds value in the lifecycle (and I don’t think the role of testing in an agile lifecycle is trying to wedge testing between the end of a sprint cycle and the release to customers in order to provide a pseudo-proxy customer buffer…that’s a bottleneck.) Reinstituting best practices such as design and code reviews and inspections (when warranted), or developing new approaches or tools to help increase testing effectiveness and reduce costs also require greater skills and knowledge among testers.

The formidable challenges of testing software that lie ahead will require highly intelligent critical thinkers who also have an in-depth understanding of the systems they are working on, and who possess the technical aptitude to provide valued input in throughout the product cycle. Indeed we work in a very dynamic industry filled with diverse challenges that demand continued learning and greater proficiency of the skills used in our profession.

The Minefield Myth (Part 2) – The value of regression testing

I.M.Testy — Fri, 30 Jan 2009 02:57:14 GMT

Last week I discussed the fallacy of the minefield analogy misrepresented by some people to suggest regression testing as uninteresting or unlikely to reveal new or important information. Their premise is that executing the same test is similar to walking in someone’s footsteps through a minefield. While this argument seems logical on the surface, this interpretation of the minefield analogy trivializes the importance of regression testing. However, there may be specific instances where regression testing (especially the execution of redundant, poorly designed, basic scripted test cases that simply reuse hard-coded data or mindlessly follow a prescriptive steps to retest unchanged or unaffected areas of code) may not provide great value to the organization. Some of those situations include:

retesting scripted, or simple procedural programs
retesting programs that are relatively static (unchanging),
retesting programs with no internal or external dependencies
rapid testing approaches where the test strategy is to simply sample the program behavior in order to provide a quick assessment and find some bugs

So, everything being equal, I would tend to agree that walking through someone else’s footsteps in a minefield may be somewhat redundant. But, there is a huge difference between redundant testing and effective regression testing. Redundant testing has zero probability of revealing new or valuable information. But, well-designed tests are not all equal, and in iterative and agile software development lifecycles some portion of effective regression test suites have some reasonable probability in exposing new information. For example, regression testing is typically useful where:

the underlying code base is changing (new features, refactoring, or bug fixes)
poor or incorrectly used revision control processes
a change in one module affects other modules in software developed using object oriented and procedural programming paradigms
the complex system has other internal and external dependencies
the software is regulated by some governing authority or law (Sarbanes-Oxley, FDA, FAA, etc)
the system is highly critical
an established baseline is required or important
legacy code bases where new functionality can unintentionally destabilize older code
code bases contain lots of ‘spaghetti’ code that is simply patched together
the results of a well-designed regression test suite helps instill a sense of confidence in the decision makers

In these contexts, an effective regression testing strategy can not only help identify important and potentially destabilizing changes in expected functionality quicker, but can also increase overall confidence in the product; especially in key areas.

Overcoming some common misconceptions of regression testing

One common misconception of regression testing strategies is that all tests are equal and all documented (manual or automated) tests go into the regression test suite. But, the simple fact is that not all tests are equal, and not all tests need to be re-executed repeatedly (every build) throughout the development lifecycle. For example, a test to check for proper tab order on a dialog or property sheet doesn’t need to be re-executed on each new build unless the UI elements have changed on that dialog or property sheet. So, in most situations this type of test is probably a poor candidate for inclusion in a regression test suite. (As a side note, if the UI elements are changing every build, I wouldn’t even test tab order until there was at least some semblance of UI stabilization.) When redundant test cases or tests that don’t provide great value to the overall testing effort are slovenly added to the regression test suite to simply increase code coverage or to artificially inflate the number of tests for ‘feel-good’ generally only results in an overloaded test suite that rapidly grows out of control and becomes a management nightmare.

Another common misconception of regression testing strategies is the suggestion that unit tests are sufficient for retesting code churn. Of course, this is simply a fool-hearty assumption and idealistic. Unit tests are a type of smoke tests performed by developers. The primary purpose of unit testing is to verify a method or function does what it is supposed to do, and may include ancillary negative tests as well. However, unit tests generally do not include comprehensive data coverage, data permutations, combinatorial analysis of parameters, etc. (which is why API testing is important in a well-rounded test strategy). Also, unit tests are usually executed in a “clean-room” environment using stubs or mock-objects, although in some cases unit tests are reran on private builds before the code is checked into the main build tree as a form of low level regression tests. While there have been significant advances in both static and dynamic analysis tools to increase the robustness of unit tests, if unit testing were the answer for effectively evaluating code churn I suspect a lot of testers would be out of a job and testing would simply be a rudimentary process of behavioral-type acceptance testing performed by non-technical ‘end-user-like’ individuals poking about the UI looking for errant anomalies and subjectively evaluating their perception of ‘usefulness’ of a product.

Another common misconception of regression testing strategies is that regression tests are less valuable because of their ineffectiveness in identifying or exposing ‘new' bugs. This is a myopic view of testing that focuses on the current version of the product, and on ‘bug-finding’ as the main purpose of testing. Of course, value is very subjective, and industry reports suggest that less than 15% of the bugs reported during a product development lifecycle are detected via regression testing. But, if even half that number are critical issues that would cause the company to pay upwards of $100K to release a patch (or even worse, the issue leads to a major calamity) then regression testing (especially highly automated regression testing) is most probably worth the time and effort. Of course, the cost of designing effective regression test cases is quite high and executing those tests requires valuable resources. So, we should not just consider the legal or financial liability issues to the company. We should also consider the product shelf-life as a factor. An effective suite of well-designed regression test cases will not only help provide a baseline assessment of the current version, but a significant portion of the test suite will/should be reused during maintenance or sustained engineering of that product for upwards of 10 years, and some subset of those tests in that suite will also be reused on the next release or version of the product. There are additional benefits to well-designed regression test cases (or any test case for that matter) in that it preserves knowledge and helps eliminate or at least reduce tribal knowledge and hero worship mentalities.

Some effective regression testing strategies

The regression test suite is essentially a set of test cases that provide a baseline measure of expected, or important functionality, and to verify previously fixed bugs do not reoccur. Based on those assertions, test cases included in the regression test suite must be carefully selected to prevent overloading or rerunning unimportant tests during a regression test pass. I generally recommend 4 types of test cases for inclusion in a regression test suite:

Test cases designed to verify critical functional attributes and capabilities of the software program
Test cases designed to validate baseline functionality/behavior specified in project requirements or user acceptance criteria
Test cases designed to verify functional anomalies that are found and fixed during the software development lifecycle
Test cases designed to evaluate collateral or dependent areas associated with a code fix

Even with these targeted tests the regression test suite can grow quite large. Within the regression test suite categorize the test cases by the feature or component that test directly targets, and sub-categorized to identify dependent modules or features. Associating test cases in the regression test suite with the primary features or components they are designed to evaluate allow the tester to prioritize the regression test pass to focus initially on the modules in which code churn occurred, followed by dependent or interrelated modules, and finally the other remaining tests in the suite. Also, prioritize test cases within each category based on criticality. Prioritization of tests is another mechanism we can use to help us better identify an appropriate set of tests based on the constraints of available time and resources.

Finally, automate, automate, automate! In order to gain the full benefit of an effective regression test suite (similar to the build verification/acceptance (BVT/BVA) test suite ) automate as many regression test cases as is reasonable possible. But, I am referring to well-designed automated test cases; not just a bunch of simple, elementary UI script monkeys. While some regression test cases will be very targeted prescriptive tests designed to verify specific functionality, most well-designed test cases include variability in the execution while still evaluating the hypothesis, or purpose of the test case. Also, some test cases in the regression test suite will be executed through the UI layer, but in applications where the functional code is separated from the UI layer some regression test cases are more effectively executed below the UI or through an abstraction layer.

Similar to other testing approaches, regression testing is an organizational investment, and its value to the team must be considered as an investment and not done simply because it seems like the right thing to do. In some cases it may not make much sense to invest heavily in regression testing, but in cases where a missed regression in functionality results in fines, legal costs, or potential loss of hundreds of thousands of dollars in post-production costs I suspect regression testing is a critical component of the overall testing strategy to validate a baseline and instill confidence.

The Minefield Myth (Part 1)

I.M.Testy — Mon, 19 Jan 2009 19:04:00 GMT

In my studies at university I studied anthropology. Several courses I took surveyed folklore and its relevance in modern society. Many people mistakenly believe that most folklore (folktales, legends, myths, ballads, etc.) are purely fictional and simply fanciful tales. However, folklore is usually based on some grain of truth, or is used to instill societal or religious mores and values. For example, social scientists have found that many ancient civilizations have folklore regarding a massive “flood” in the distant past which wiped out huge populations of people. Did this actually occur? Well, we don’t know for certain, but geological evidence does suggests is that at one time coastal waters did rise significantly. Was this caused by cyclical change in the earth’s temperatures or by a series of earthquakes causing tsunami’s to ravage coastal villages? We don’t know; but the folklore may indicate that at some point many societies suffered a devastating travesty caused by rising waters. Was the story embellished over time…certainly. Another example is the “Cinderella” story. There are over 450 versions of the “Cinderella” story around the world. The story is about over-coming adversity and oppression, and avoiding self-pity and selfishness. Basically, it is much more than a Disney animation, in traditional folklore it has been passed down through the generations to reinforce societal values.

The first time I read about a minefield analogy was in the context of sampling. Later, Brian Marick used a similar analogy to suggest repeating tests (regression testing) is not likely to reveal new bugs. Marick’s analogy is perpetuated by Bach, Kaner, and others who tend to diminish the value of regression testing (especially automated testing) because we are simply traversing a minefield by following a previously cleared path.

The Marick minefield analogy is simply an alternate perspective of Beizer’s pesticide paradox which states “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. Basically, no single approach to software testing is effective in identifying all categories of defects, and we must use many approaches in software testing and vary our tests. In that context I absolutely agree with the analogy.

However, a basic problem of Marick’s minefield analogy as it is often misrepresented is that it seems to treat the software under test as a static, unchanging field of easily exposed mines.

If you were hired as a consultant to come in a perform a rapid evaluation of a software product using a sampling approach such as exploratory testing, then Marick’s minefield analogy is a wonderful strategy. In that context re-running a test provides no new value and has little probability of exposing new information. However, for the rest of us who work in iterative software development lifecycles (including agile lifecycles) building complex systems with interdependent components the minefield analogy may not be as useful.

For example, in complex systems with interdependent modules we know that a change in one module can adversely affect other modules that have some dependence on that module. So, a change in one module can impact the functional behavior of other modules. In layman’s terms, activating a mine while traversing one path through the minefield may reactivate an already cleared mine in another part of the minefield, or even plant a new mine in a previously traversed path.

In iterative development lifecycles, the minefield is in constant flux (at least until the code complete stage, but even then the code is changing as issues are being addressed.) In iterative lifecycles features are being added, changed, and possibly removed during the process. Depending on the length of your product lifecycle the changes can be massive. The PDC release of Windows 95 ‘looked’ very different as compared to the final release. The build verification/acceptance test suite for Windows 95 was a relatively static baseline regression test suite that continued to find ‘regression’ problems up to the final weeks of the project due to code churn.

Also, not all mines are equal! Some mines are quite easy to detect while others are very hard to find which is why systematic probing is still used by professional’s to clear latent minefields. Similarly, an exploratory approach to testing software will easily reveal some bugs very quickly, but without ‘systematic probing’ we could just as easily overlook other types of issues.

There are also different types of mines which may be activated differently, so traversing a minefield with a size 10 boot may not activate the mine, but someone with a size 12 boot, or who weighs more than the previous person may in fact activate the mine. Likewise, traversing the same path through software using different data or applying a more systematic analysis of a path may reveal interesting information or expose anomalies that were not previously discovered. For example, throwing simple ASCII characters at a text input control is not likely to expose any bugs (or restated it is likely to show us a clear path through the minefield). However, when we take that same exact path using Unicode characters, or Unicode surrogate pair characters we are very likely to expose problems not revealed previously.

In part 2 I will discuss regression testing and specific situations where regression testing is very valuable.

Data-Driven Testing

I.M.Testy — Sun, 04 Jan 2009 07:27:00 GMT

I am generally not a big fan of static data in test automation, but being a pragmatic person, I know there are clearly times when using data-driven testing is just plain common-sense. For example, data-driven testing is an effective automation approach when designing ‘black-box’ tests for testing an API.

Data-driven testing is a common approach to test automation where static test data is passed to application parameters and the expected result (which is usually also read from static data) is compared against the actual result. This automation approach is effective when the actual result compared to some expected result can be resolved as a Boolean outcome. In other words, if the actual and expected results match the outcome is true and the test passes; otherwise the outcome is false and the test fails. (Of course, if something occurs during the test where there is no actual result then that particular test is usually logged as indeterminate.)

Of course, the key to effective data-driven testing is the data! If we don’t identify the most appropriate data to use in the test then the test case may have holes and we might overlook important information or miss critical anomalies. If we have too much redundant data then we may be simply running unnecessary tests (yes, even with test automation redundant testing is not an efficient use of resources).

Let’s say we had to test an API method such as:

public bool IsValidNetBiosName(string name)

where the return value is true if the string argument passed to the name parameter is a valid NetBIOS name on the Windows operating environment; otherwise it returns false.

With a data-driven testing approach we could use a simple CSV file that contained the string arguments and the expected result for each string passed to the name parameter. A partial sample of the CSV data file would be:

a,true
validname,true
validnamexxxxxx,true
invalidnamexxxxx,false
invalidnamexxxxxxxxxxxxxxxxxxxxxxx,false
,false
null,false,
xx\x,false
xx/xx,false
x:x,false
xxx*xx,false
x?xx,false
xxxx",false
;,false
xxx|xxx,false

(NOTE: null is a special case in which we need to convert the string “null” to a null in the test code, and the test above null is an empty string. An empty string and null are two different things and both must be tested in this case.)

Next, we need to read in the CSV file into our automated test, and perhaps the easiest way I found to read in a text or CSV file in C# is with the File.ReadAllLines method. The ReadAllLines method opens a text file, reads each line of text as an element in a string array, and then closes the file. Once we have a array of all lines in our data file, we simply need to parse each element in the string array into test data and/or expected result, and then compare the actual result against the expected result as illustrated in this example.

// Read each line in the entire CSV file into a string array
string[] testDataArray = System.IO.File.ReadAllLines("myTestData.csv");

// Iterate through each element in the test data file
foreach(string test in testDataArray)
{
    // Split each element in each line into an array where the elements are the
    // test data and the expected result
    string[] testElement = test.Split(',');
    string testData = testElement[0];
    string expectedResult = testElement[1];
    
    // Special case for passing a null to the API parameter
    if (string.Equals(testData, "null", 
        StringComparison.OrdinalIgnoreCase))
    {
        if (string.Equals(api.IsValidNetBiosName(null).ToString(), expectedResult, 
            StringComparison.OrdinalIgnoreCase))
        {
            result = "Pass";
        }
        else
       { 
           result = "Fail";
       } 


    // Compare the return value against the expected result
    else if(string.Equals(api.IsValidNetBiosName(testData).ToString(), expectedResult,
        StringComparison.OrdinalIgnoreCase))
    {
        result = "Pass";
    }
    else
    {
        result = "Fail";
    }
}

This is rather simple example, but data-driven testing is effective for unit testing, API testing, and can even be used in automated GUI testing (although data-driven automation may only have limited applicability in GUI automation). I am a firm believer in the KISS principle when it comes to developing automated tests, and the ReadAllLines method is perhaps the easiest and most efficient way to read in data file for data-driven development. Of course, data-driven testing doesn’t solve all problems. Chan Chaiyochlarb has a good post on some pitfalls to watch out for. But, in the right context, data-driven testing can be one approach used in automated testing.