Welcome to MSDN Blogs Sign in | Join | Help

Thoughts on Professionalism

As a young lad growing up on the shores of the Chesapeake Bay I would often spend part of my summer vacation from grade school helping my grandfather work the crab pots on the north shore. Now, don't think "Dangerous Catch," crabbing in the Chesapeake is much different than crabbing in Alaskan waters, although we did have our share of squalls and nor'easters that will humble any man. Crabbing is hard work, especially for a young boy. On one particular day working the pots with my grandfather I recall something he said which has stuck with me my entire life. The water was rough that day and we were being tossed around a bit. It was raining...not like the wimpy Seattle rain, I mean hard East coast rain that comes down in sheets with drops so large they hurt when they hit you. Even in the driving rain the smell of dead fish, rotten chicken pieces and turkey necks that we used for bait permeated everything, the slime of fish guts coated the decks making footing somewhat precarious, and the brine of the Chesapeake seeped into every little cut and nick on my hands. I wasn't having a lot of fun that day, and although my grandfather knew it he didn't say anything to me until I complained aloud. Now my grandfather was a tall, barrel-chested man with a stern face, and he never put up with any senseless, petty whining. But, instead of scolding me he simply said, "Son, we are not rich and there is no inheritance coming your way, so you better find a job you like doing." At first, I was a bit puzzled. What did that have to do with how I was feeling at the moment? But, I soon realized what he meant, and my grandfather's sage advice has stuck with me ever since and guided me in my career pursuits.

My father, who was also every bit as practical and pragmatic as my grandfather taught me the value of working hard, and the virtues of responsibility. He taught me to take pride in the things I did well, and take responsibility for things that didn't work out so well. My father also taught me never to quit, to always look at alternative options, and that the path of least resistance or the easy path through life is not always the best path to take. He also said if something was worth doing, do the very best you can do, and never sit back and rest on my laurels. My father was not the type of man to do something half-way, or 'good-enough.' One summer we built a new barn on our property. This wasn't the typical barn, instead my father got some telephone poles and even got his friend at the telephone company to bring out a truck with an auger to drill the holes 10 feet deep! Next we used oak planks between the stalls. Now hammering nails through oak is not easy business and the entire project took about 2 weeks. But, my father was adamant about not having to rebuild the walls of the stall if a horse tried to kick it down.

Practicality in practice

Now perhaps it was my upbringing, or perhaps it is my Myers-Briggs ISTP personality traits, but it is hard for me to understand how some people embark on a career, or pursue a job opportunity without knowing (or perhaps worse..completely ignoring the fact) that they will be challenged to continually improve their knowledge and skills everyday. This is especially true of many of our chosen profession of software testing. We work in a highly dynamic industry and we must continue to learn and develop new skills to meet the growing diversity and increasingly complex challenges that will potential advance the discipline as well as provide benefit to our employers.

Sure, we can whine and complain about the changes. We can point fingers, and otherwise express negative emotions and utter baseless, counter-productive propaganda slogans such as, "automation can't do such and such," or "I know non-coding testers who find more bugs than (fill in the blank here)" or the virtually inverse argument "tester's  who write code are biased and don't know how to think like an end-user." I learned a long time ago that although misery loves company, it is usually not those who whine the most or put forth pointless and petty gripes, or who only see the world through bi-polar, black and white lenses who drive meaningful change. Sure, it is easy to empathize with some people in some situations; but empathy without practical solutions to resolve the situation effectively is simply a pathetic play of the irresponsible victim card.

For example, we unfortunately often get mired down in a this vs. that, exploratory vs. scripted, or STE vs SDET debate that is generally too myopic (on my project… blah, blah, blah) and often counter-productive (meaningless comparisons, ridiculous measures such as raw bug count, and baseless assertions that lack empirical evidence to substantiate the claim, etc). There seems to be an opinion that if someone writes automation, or comes with a coding background then they are not good ‘exploratory’ or behavioral, or (and I hate using the phrase) “black-box” testers.  (All testers are “black-box testers!). Yet, some of the most talented testers I know in the industry began with very little in-depth knowledge of the ‘system’ or an understanding of programming concepts and applied themselves to  grow their knowledge and skills about the overall system and technologies to advance their careers, their teams, and the industry by taking on greater challenges. I also know highly talented testers with great industry experience and very strong coding backgrounds who are masters at explorative, behavioral,  or “black-box” testing.

So, for those of you who have been living in a cave, or have your head permanently implanted in the posterior end of your alimentary canal it is obvious the industry and the profession of software testing is changing and there is an increasing demand for greater system knowledge, experience, and technical skills for many testing positions in the industry. There are many reasons for these changes, and the simple fact is that the industry’s needs and strategic direction primarily dictate what skills and knowledge are required to remain competitive in the industry and advance the business.

Are ya' gonna lay there and bleed, or ya' gonna' cowboy up!?

Now, many people mistakenly assume that I only promote technical knowledge or advancement though increased technical skills. (Of course, many of these people are so narrow minded and biased they only equate 'technical knowledge and skill with 'coding' skills and programming knowledge.) I do talk about these subjects a lot because...well, they are directly related to our professional growth, and because I studied and interviewed thousands of testers to perform a skills gap analysis (delta between the disciplinary needs of the business in relation to the actual skills and knowledge of the existing workforce at the time). The skills gap analysis indicated that many people in the testing industry at the time were well-educated, intelligent people who were very capable of thinking critically about problems within the appropriate context. But, it also illustrated that a significant population in the testing profession at that time lacked in-depth knowledge of the system they were working and the technical skills to expand their own productivity or become more influential in driving product quality. So, as a manager and mentor it was (and is) my responsibility to help people understand changes based on the maturation of the industry and the profession, and to help them grow professionally, and that is the role I took on with great passion. I decided to focus on growing people's technical skills and knowledge, and their understanding of established, and contextually effective software testing processes.

One of my first public talks revolved around the idea that good-enough was simply not good enough any longer based on changing customer views and increasing maintenance costs. That was in 2003! In 2005, I presented empirical evidence suggesting that exploratory testing by 'presumable power-users' and 'domain experts' is not as effective as some claims, and I advised testers to begin thinking about the future and the increasing demand for testers with more in-depth system knowledge and technical skills to not only remain effective and productive in the industry, but to help drive it forward. Some people jeered me, and one respondent from the StarWest 2005 conference wrote "Bj is a fool, and his talk shows why Microsoft products suck and why Microsoft will fail" on the feedback forms. A few others said they agreed with the message, but they really didn't like hearing it out loud. One person told me "Yes, we know we need to improve, but your talk was like hitting us in the face with a frying pan full of hot grease."

I mention this not to "toot my own horn;" that is simply not my style, for I am yet a simple man who is still learning to improve myself. I only mention this because here we are 5 years later still arguing over petty things that we sometimes do not have the knowledge, skill, experience, business insight, scope of influence, or credibility to change.  (And in case you don't know, whining. griping, and emotional conjecture is simply not viewed as being very credible in trying to influence change in a room full of decision makers or business leaders.) Right now, the industry, and the testers in the industry need leaders to help them adjust to changes, provide strategic vision, and guide career growth that enables each person to be successful in their professional development and compete with their peers. Responsible leaders are those who face challenges head on, who put forth logical and rational arguments based on analysis and empirical evidence, but who continue to search for ways to solve not only the immediate problems, but also have the ability to envision potential side-effects and deal with those as well before they become problems.

Our profession is multi-dimensional, and as testers we must continually strive to increase our knowledge and our skills in order to take on new and exciting challenges in this dynamic and agile world of software testing. The path is not always easy, but it does provide many rewards to those who work in a job they truly enjoy.

Equivalence class partitioning: is it real or is it a figment in our imagination?

Last week I attended the Software Testing and Performance conference in Boston. I presented a workshop on Systematic Testing Techniques, as well as a talk on random test data generation, and combinatorial analysis. One way I continue to learn about our profession and increase my own knowledge is by going to conferences to hear different points of view from practitioners from around the world. So, I also attended several talks during the conference, but there was one talk in particular that was particularly entertaining (and I don't mean that in a good way).

When I listen to other testers sometimes I hear something that is new to me and I desire to learn more about it. Sometimes I hear something prophetic that makes me think, contemplate alternatives, or reflect more deeply on my own personal perspectives. Sometimes I hear something revolutionary that causes me to reevaluate my position. And, sometimes I hear something so irrational I almost barf up a lung!

In this case the speaker opened his talk with an attack on a quote from the ISTQB foundation syllabus used to describe boundary testing which states, "Behavior at the edge of each equivalence partition is more likely to be incorrect..." Now I know the speaker a bit, and I know he disdains the ISTQB and other certification organizations, but what surprised me was his initial rebuttal by emphatically stating, "equivalence class partitions are figments of our imaginations!"

These days I usually just try to shake off wild and baseless comments as bombastic bloviations used to generate controversy. But, in this case what caught my attention was when the speaker later said that he and another well-known person defined boundaries as "a dividing point between two otherwise contiguous regions of behavior; or a principle or mechanism by which things are classified into different sets." What!? I couldn't believe what I heard, so I had to stop reading email and look up at the presentation. As I visually processed the words I thought my head was going to explode from the so seemingly obvious contradiction.

Now, I am not a linguistic expert, but I am pretty sure that "otherwise contiguous regions of behavior" and "classifying things into different sets" are just overly simplistic ways of describing equivalence class partitions. But, I could be wrong. So, I began thinking that since most people start learning about sets in elementary schools they probably understand the foundation of equivalence class partitioning is set theory which basically states "a set is an aggregate, class, or collection of objects," and the collection of objects or 'classification of things' in different sets is based on an equivalent relation between the elements in each set. The application of equivalence class partitions in our profession is elegantly explained by Lee Copeland in his excellent book A Practitioner's Guide to Software Test Design by stating "An equivalence class consists of a set of data that is treated the same by the module or that should produce the same result." Equivalence class partitioning is also discussed in-depth in books by noted experts in the industry such as Beizer, Binder, Myers, Jorgensen, Perry, and Marick just to name a few.

In fact, the concept of sets and equivalence almost seems instinctive in most humans and is generally expressed at a young age. I remember my young daughter at age 2 or so separating beads by color into "different sets" on the carpet. The red beads in one group, blue in another, and so on. She was diligent to make sure the different sets of beads did not touch as she put them into the appropriate piles. If a pile of beads got to close to another pile she would run the edge of her hand between the "contiguous regions" to clearly delineate the "dividing point."  When I asked her to get me a red bead, she would randomly grab one from the pile, because all the red beads were...red, and there were no significant differences among the red beads (elements) in the set she created that were relevant in that context of that game.

Perhaps the majority of the industry's experts are wrong and I wasted my time reading books on software testing principles and practices because this person is right and equivalence class partitioning is really only a figment of our imagination.

However, on the other hand, although I certainly have never claimed to be an expert, I am still pretty darn sure the underlying foundation of computers and computer software is somewhat influenced by mathematical principles, and as a tester I might be able to use those same principles to help me design effective tests that might help me better evaluate discrete functional capabilities and attributes of software components and more efficiently expose certain categories or patterns of errors.

But, why should we get mired down and confused with facts (especially all that boring math stuff) when it is much easier to appeal to some peoples' emotions. So,forget everything you just read...and if anyone asks why testing is so hard just tell them testing is an art with no practical foundation in logic because software is...well, it's just magic!

Think before re-inventing the wheel

There are rare occasions when an exceptionally bright and innovative person comes along and actually builds a better 'mousetrap.'  My friend Ken Smith is one such person. Ken is the inventor of SmartPlug. Anyone with a boat that connects to shore power has experienced the aggravation of lining up the 3 keyed prongs in the dark, over-heated and burnt out receptacles, and cross threading the sealing ring on those circular Marinco power inlets. Finally, Ken came up with a better design that offers a safer solution for shore power. But, Ken has owned boats for much of his life, and he has been in the marine industry for 38 years. His invention was based on years of personal and industry experience coupled with the knowledge and experience of several other boat owners and electrical experts.

So, I am often amused (not in a funny way) by people who are relatively new to the industry who decide to build yet another automation testing framework or test tool. Unfortunately, many of these 'new' and 'innovative solutions are simply rewrites of existing frameworks or tools. Sometimes they may introduce a new and useful feature, but mostly it is just another 'mousetrap.' Often these "new" mousetraps are designed and developed to solve problems that are unique to a specific team or product. But, rather than building on existing frameworks or tools some people seem compelled to completely re-invent the wheel and then want to promote 'their' solution as the holy grail (which, of course, are not compatible with any existing solution).

Upon recently seeing another such proposal for yet another test automation framework I decided that I would also join the re-invention foray and provide a "new, and innovative" solution myself. Now, I need to ballyhoo my revolutionary project in order to gain greater adoption and win political points with the not-so-clued-in managers who actually think completely redesigning a mousetrap over and over again by inexperienced people working in isolation is a good idea as a way to motivate people who are bored.

So, without further ado, here is my announcement for the new, exciting, and revolutionary Wheeleze!

“I am wasting precious time and mindless resources working on yet another needless re-invention for which I don’t yet have the breadth of experience or span of influence to succeed, but it is really cool and fun and I don't have to work on other stuff that I think is boring. I call my senseless re-invention “Wheeleze.” Let me tell you about my new innovative solution that solves the problem of transporting various cargo over land.  Wheels are already in common use and provide effective solutions to the transportation problem, but there are several different types of wheels in existance and everyone knows there should only be one wheel because having several different types of wheels is expensive and it is mind-numbing to keep track of which wheel should be used in which specific context. In contrast, Wheeleze will replace all of these existing wheels with one easily adaptable and extensible solution. And, although there are some other wheel type solutions being developed in isolated pockets around the world that are also claiming to optimize rolling, Wheeleze is really new and different. (Really, it is, and if I get enough people to say it enough everyone else will believe it). As the name suggests, Wheeleze is intended to make rolling easier by providing a vehicle agnostic, intuitive, simple to use, modular solution to roll public and private conveyances down paved, cement, gravel, dirt, and even sand roadways. Best of all, Wheeleze can be recycled as fenders on tug boats, can be buried in playgrounds for children to climb on, or have one side turned inside out to form red-neck flower planters. This innovate and unique solution optimizes reuse and minimizes landfill waste. Wheeleze is destined to revolutionize the wheel industry.  Wheeleze is fully globalized and I even have a localized version called “Wheelese” for non-US English markets (such as Canada, Ireland, England, and Australia). Wheeleze is an attempt to eliminate duplication of effort because nobody would ever dream of recreating another wheel after using Wheeleze. My new, unique, and innovate Wheeleze is designed for unicycles, bicycles, oxen-carts, wagons, cars, sports-utility vehicles, light trucks, tractor trailers, combines, and even wheelbarrows. I am also planning to extend the Wheeleze to support motorboats, canoes, and surfboards. (NOTE: Wheeleze on a surfboard appears to be similar to a skateboard, but they really are different...you can't surf on a skateboard!) Unfortunately there is no plan to support Wheeleze on sailboats (the keels present too hard of a problem that I don't know how to solve, and of course I don't want to ask anyone else for help because this solution is mine) because sailboats are tool slow and "old tech" and most people will probably not want wheels on a sailboat anyway. Please visit http://www.anothercompletelystupidwasteoftime.com for more information.”

Test Automation: Simple Automated Test Case Templates

Templates can be useful tools to help increase individual efficiency and team consistency. Yet, I am sometimes quite surprised that some organizations lack standard templates for such daily routines such as bug reporting, status reporting, and even test cases. There are some in the industry who shun templates because they claim they are too restrictive and the person using the template is 'forced' to do things one way and not encouraged to think beyond the scope of the template. Of course, any person who has matured to the point of being able to think rationally for themselves and has the cognitive ability to for logical reasoning (as opposed to those who base decisions on pure emotion and volitional impulse) realize this narrow-minded viewpoint is pure nonsense!

A template is simply a model or reference that provides a starting point. Frankly, if a person does not understand that a template simply provides a starting framework from which he must rely on his knowledge, skill, and experience to customize to fit the specific context, and how to use a template to help him accomplish a task more efficiently then I would recommend that person not be allowed beyond the reach of their mother's apron strings because they are probably a danger to themselves and people around them and they are incapable of rational constructive thinking.

An example of where templates have added value for me is in the test development process. When developing test cases using Visual Studio I frequently find myself adding the references and using directives repeatedly to each new test case. For example, when developing a UI automated test case I need to include references for Windows.Forms, UIAutomationClient, UIAutomationType, UIAutomationClientsideProviders, and UIAutomationProvider. That's a lot of clicking for each new test case.

Creating a template in Visual Studio is easy and straight forward. In this case, I started with a standard console application project. Next, I added the following references to the project:

  • System.Windows.Forms
  • UIAutomationClient
  • UIAutomationClientsideProviders
  • UIAutomationProvider
  • UIAutomationTypes

Then I customize the template as illustrated in the code below by adding common namespace directives, including XML comments, and commonly used code snippets.

using System;
using System.IO;                       // For file I/O operations
using System.Text;                     // For StringBuilder and encoder/decoder
using System.Diagnostics;              // Process, stopwatch, trace/debug 
using System.Globalization;            // For Unicode support
using System.Windows.Forms;            // For sendkeys and clipboard
using System.Windows.Automation;       // For UIAutomation

namespace CompanyName.ProjectName.FeatureArea
{
    class TestCase
    {
        /// <summary>TEST PURPOSE: (specific functional capability or end-2-end user scenario test is verifying)</summary>
        /// <status>TEST STATUS: (Active, NotInMilestone, Obsolete)</status>
        /// <category>TEST TYPE: (category or test suite bucket such as BVT/BAT, Regression, etc.)</category>
        /// <priority>TEST PRIORITY: (importance of test in relation to critical functionality or requirements)</priority>
        /// <time>TEST TIME: (approximate duration of test)</time>
        /// <steps>TEST STEPS: (outline high level steps or procedures)</steps>
        /// <expected>EXPECTED RESULTS: (specific verification criteria for oracle to determine pass, fail, or indeterminate)</expected>
        /// <dependencies>CONFIGURATION: (required expected settings or dependencies)</dependencies>
        /// <remarks>OTHER INFO:</remarks>

        [STAThread]
        static void Main(string[] args)
        {
            // Time the duration of test in order to better predict time to
            // run each test suite
            Stopwatch TestTimer = new Stopwatch();
            TestTimer.Start();

            // TODO: Design and develop the code to achieve the specific purpose
            //       of the test. 

            // Don't forget to add comments to the code!

            // Don't forget to clean up or restore the test bed to pre-test
            // conditions in cases other than end-to-end user scenarios.

            TestTimer.Stop();
            // Don't forget to log or record test time
        }
    }
}

I also like to include valuable information about the test case using XML comments to jog my memory or provide some background for anyone who views the code. For example it is a best practice to clearly define the specific purpose of each test so people don't have to guess what the test is proving/disproving (or at least what it is supposed to be doing). (If someone doesn't have a clearly defined purpose and goal of what the automated test is going to prove or disprove and they are simply writing automated test just to automate tests than that person is probably a mindless dolt!) The information that adds value to the organization may be subjective, so determine what information is important for the consumers of the completed template. Essentially, in this case, the comments captured in each test should provide value to the management and maintainability of the test and the testing process. If a template tries to collect too much information that is not being consumed or is not important to reduce long term costs of maintenance then that information should be suspect and is probably not adding value to the overall team. A big advantage of Visual Studio and C# is that it allows comments and important information to be easily captured via XML tags documented in a XML file at build time so I don't have to worry about performing a separate task to document each automated test.

The STAThread attribute is included above the Main method to support the Clipboard class members. Without this attribute set the test will throw an un-handled exception. Finally, I put in any code that I commonly reuse. That doesn't mean that I have to use it...I can easily remove it if I don't need it. (That's the beauty of being able to think logically for one's self!) But, if it is code that is commonly used then I don't have to retype it (meaning...it increases my efficiency even by a little bit).

To complete the process I simply select the File --> Export Template... menu item and complete the Export Template Wizards.

image

I can create several custom templates depending on the context of each type of test. For example, if the automated test case is not going to drive the UI then I don't need all those additional references in my template. Rather than remove them each time, I can simply create another template for non-UI automated test cases. Now, each time I start a new project I can simply select the appropriate template. The templates are saved in a ZIP file in the Visual Studio\My Exported Templates directory, and those ZIP files can be sent around to other members of the team.

Also, here are some important key concepts for creating standard templates (in general):

  • Don't screw the majority just to make the minority happy! Only include key elements that provide value to the majority of consumers of the template the majority of the time. A common problem with many templates is that they try to include everything for everybody. When templates are over-laden with things that only a handful of people use, or if people have to spend too much time trimming out things the majority of people don't use then the templates are a bottleneck. Remember, templates can be customized by those few who might need a few additional things occasionally; majority rules!
  • Don't over-document useless tidbits. Notes and additional comments must be useful, but brief. If someone has to spend more time learning how to fill out a template than it takes to fill out the template then that template provides no value in terms of making the process or the team more efficient. If the template is so complicated that it requires a user manual...then it is not a useful template.
  • Intuition is important. If the template is not easy to use, if the consumer of the template has to search around for items, or even worse, forces the user to use parts of the template because they are on the template, then the template is not intuitive and using the template becomes a chore rather than a tool to improve efficiency.
  • Get buy in! Don't develop templates in isolation. Talk to various consumers and users of the templates to better define the template and what is captured in the template. For example, if you are developing a template for reporting anomalies in software then developers should be included in that conversation as should program managers and any other consumers of the completed templates.
  • Templates are not commandments! Templates provide a starting point; they are not engraved in stone. Templates should be designed to include things that are commonly reused by various users and consumers of the completed template. But, sometimes they can include things that are unnecessary and do not add value in a specific context and in those cases the user of the template should be able to remove those parts of the template or at least ignore them. Conversely, other times additional things may be necessary in a template and the user of the template should be able to easily add those things that provide important value to the purpose of the completed template.

Bottom line, templates are tools to provide a common reference, or central starting point...they are not hand-cuffs designed to constrain rational and logical thought or constrict a person's creativity. In fact, good templates are tools that enable and support more efficient creativity!

La rentrée

It has been some time since I posted. Quite frankly I was burnt out, and I finally realized that I needed a vacation. So, I decided to unplug and take the month of August off to catch up on chores around the house, tend to my gardens, and finally wrapped it up with a week of single-handed sailing down to Olympia and back. I love gardening because it gets my mind off computers and I love to work with my hands. But, the last week of my vacation I decided to sail alone and completely unplug from all electronics and get back to the basics of sailing and navigation.

This was my first voyage to Olympia (I usually head north from Seattle) so I was heading into unfamiliar waters, and decided to do so without the conveniences of GPS, the depth finder, autopilot, wind instruments, and other modern conveniences to refresh my basic sailing and navigation skills. I even kept my cell phone and VHF turned off, and used oil lamps inside the cabin at night as I re-read the first edition of Robinson Crusoe. The only luxury I would afford myself was using propane for cooking and taking a hot shower once securely anchored (I even used an oil lamp as an anchor light, but only because my masthead anchor light was burnt out and I didn't feel like climbing the stick to change the bulb).

So, I studied the charts and the tide and current tables, read through the Coast Pilot and the Waggoner Cruising Guide, and plotted my course. There are several channels where timing the currents are critical so, if my calculations were wrong I might not make it through a pass for at least 6 hours. The land masses also cause the wind to shift quite a bit, so tending to the sheets and trimming the sails meant there was little free time once the sails were raised. Despite the gray skies, pulling a shoulder muscle, ripping off part of a fingernail, and sustaining a rather deep gash near my right eye while reefing my mainsail the trip restored confidence in my abilities, provided me with peace and solitude to contemplate life, and prepare me for la rentrée (the return) back to the real world.

So, you are probably wondering what the relevance of my vacation is to software testing?

It's all about the basics. Without a solid foundation of basic skills and knowledge of the discipline then a person's ability to fully engage in all the tasks associated with professional software testing are limited. There are some who mistakenly assume that testing from an end-user's perspective is sufficient; and that may be the case in some limited situations. But, in highly complex or mission critical systems, or software products that must be maintained over several years require professional testers who possess a rich understanding the system and who are also proficient using basic skills and tools of the trade. Understanding the basics enables a person to better understand the how and why, and provides a base for approaching a problem from multiple rational perspectives given a specific context.

Posted by I.M.Testy | 0 Comments

Functional techniques are more than black box techniques

Too often many tester's mistakenly assume that functional techniques such as equivalence class partitioning, boundary value analysis, combinatorial analysis, etc. are simply "black-box" testing techniques. I suspect this rather narrow perspective of functional testing techniques is due to a lack of in-depth understanding of testing throughout the product lifecycle, and confusion between test design and test execution. I also suspect this incorrect assumption is perpetuated by 'testers' whose only approach to testing is interacting with the software via the user interface with the intent to find bugs. Of course when a person can only 'test' via the user interface then everything is a "black-box."

When I teach software testing courses I reinforce the concepts of black, white, and gray boxes as perspectives for test design; not for test execution. Using these concepts as test design approaches designing tests from black box test design makes no assumptions about the code and focuses on inputs and outputs from the end user interface, white box test design explicitly uses the code to help the tester design more effective tests, and gray box principles use an in-depth understanding of the entire system to design effective tests within the context of the software solution.

Functional testing techniques were derived years ago by industry experts and the founding fathers of software testing to provide systematic procedures that help identify specific categories of problems based on fault models that are commonly encountered during the development process. Certainly, functional techniques such as combinatorial analysis are very effective when applied from the end-user interface; however, we all know the fault model which combinatorial testing is effective in identifying involves semi-coupled or directly dependent unordered parameters. So, in order for this functional technique to be applied correctly the tester must have an in-depth understanding of the domain space and know which unordered parameters are semi-coupled or directly dependent. Boundary testing is very effective in evaluating how specific linear variable conditions of independent parameters are handled and is useful for identifying problems with overflowing intrinsic data type ranges, casting between data types, incorrect usage of relational operators, etc. Effective boundary testing means the tester must have some knowledge of data types and programming concepts and insight into the code is helpful in order to design the most effective tests.

Many of the functional techniques we apply at the end-user interface actually require a solid understanding of the underlying system to be most effective. So, designing tests using many functional testing techniques actually approaches test design from a gray-box perspective. These functional testing techniques can also be effectively applied during unit testing to drive quality upstream and prevent problems from getting into the product. Unit tests, of course, that include comprehensive boundary value analysis are tests designed from a white box perspective.

This doesn't mean that functional testing techniques shouldn't be executed from the end-user interface. But, the purpose of functional testing techniques is to help testers design tests from multiple perspectives in order to perform a more in-depth, systematic analysis of software, and potentially expose very specific categories of problems based on empirical fault models. The execution of those tests may be applied either at the code level or from the end-user interface. But, effective application of functional techniques requires in-depth system level knowledge and cognitive skills that extends below the end-user interface.

Testers must embrace and extend the role of testing beyond the simple ability to find bugs at the end-user interface. Testers must start testing earlier as a partner in the process rather than an adversary of the development team. Testers must become proficient in various approaches necessary to provide a wide variety of information to the key decision makers throughout the development lifecycle instead of at the end of it. Quite simply, professional testers need to break beyond the "black-box "barrier!

Posted by I.M.Testy | 0 Comments

Test Automation: Programmatic Platform Profiling

Occasionally, the execution or outcome of a test depends on the operating system version on which the test is executed. Platform profiling is important because subtle differences in operating system platforms can affect how certain tasks are carried out, inclusion or exclusion of specific features or capabilities, work-arounds, or even changes in deterministic or declarative oracles that determines pass, fail, or indeterminate results. For example, accessing certain features on the Vista platform will trigger the User Account Control (UAC) dialog, whereas this feature does not exist on previous versions of Windows.

It is easy for a tester to select the appropriate OS version when executing tests manually and determine the correct steps or settings and determine the correct results of a test based on their cognitive expectations for an outcome, But, in an automated test system where tests are distributed across the wire to various machine configurations and operating system platforms the test designer must programmatically detect the operating system version in automated tests that need to be profiled for the appropriate platform. Programmatic platform profiling negates the need to write several different tests for a single test case that is dependent on variations in the operating system versions. Detecting the operating system version at runtime allows the automation test designer to handle subtle differences between the operating systems within one one test case.

Detecting the major and minor OS versions with C# is relatively straight forward using the System.PlatformID enumeration and the System.Environment.OSVersion property and OperatingSystem class members.

The System.PlatformID enumeration detects the major operating platforms including:

  • Win32Windows - versions of Windows 9x, including Windows ME
  • Win32NT - versions of Windows 2000, Xp, and Vista
  • WinCE - versions of Windows CE
  • Unix - versions of the Unix OS

Once the platform is determined, the test designer can drill down further using the Environment.OSVersion.Version.Major and Environment.OSVerions.Version.Minor members to find the specific major and minor operating system version number. If tests are dependent on specific revisions or service packs of the major operating system then the integer or string values can be obtained using the OSVersion.Version.Revision or OSVersion.Version.Build members to determine  specific versions of an operating system. Some versions of Windows used alphabetic characters in the revision number, so in those cases the revision should be converted to and compared using a string object as represented by the constant values for Windows 95 retail, Windows 95 OSR 2.1 and Windows 95 OSR 2.5 and Windows 98 Second Edition. Also, the OperatingSystem.ServicePack property can be used to determine if, and what service pack has been installed on the operating system.

        // Major NT Kernel Versions
        private const int WINDOWS_NT6_KERNEL    = 6;
        private const int WINDOWS_NT4_KERNEL    = 4;
        private const int WINDOWS_NT5_KERNEL    = 5;
        
        // Minor NT Kernel Versions
        private const int WINDOWS_2000          = 0;
        private const int WINDOWS_XP            = 1;
        private const int WINDOWS_XP2003        = 2;

        // Service Packs
        private const string SP1                = "Service Pack 1";

        // Minor Win9x Kernel Versions
        private const int WINDOWS_95            = 0;
        private const string WINDOWS_95_SP1     = "950A";
        private const int WINDOWS_95_OSR        = 3;
        private const string WINDOWS_95_OSR2X   = "950B";
        private const string WINDOWS_95_OSR25   = "950C";
        private const int WINDOWS_98            = 10;
        private const string WINDOWS_98SE       = "2222A";
        private const int WINDOWS_ME            = 90;

        private static string GetOSVersion()
        {
            OperatingSystem osVersionInfo = Environment.OSVersion;
            string osVersion = string.Empty;

            switch (osVersionInfo.Platform)
            {
                case PlatformID.Win32NT:
                    switch (osVersionInfo.Version.Major)
                    {
                        // Windows NT 4.0
                        case WINDOWS_NT4_KERNEL:
                            osVersion = "NT4";
                            break;
                        case WINDOWS_NT5_KERNEL:
                            switch (osVersionInfo.Version.Minor)
                            {
                                // Windows 2000
                                case WINDOWS_2000:
                                    osVersion = "Win2K";
                                    break;
                                // Windows Xp
                                case WINDOWS_XP:
                                    osVersion = "WinXp";
                                    break;
                                // Windows Xp 2003 Server
                                case WINDOWS_XP2003:
                                    osVersion = "WinXp2003";
                                    break;
                            }
                            break;
                        // Windows Vista and Server 2008
                        case WINDOWS_NT6_KERNEL:
                            if (osVersionInfo.ServicePack == SP1)
                            {
                                osVersion = "Vista SP1";
                            }
                            else
                            {
                                osVersion = "Vista";
                            }
                            break;
                    }
                    break;
                case PlatformID.Win32Windows:
                    switch (osVersionInfo.Version.Minor)
                    {
                        // Windows 95, OSR 1.0, OSR 2.0
                        case WINDOWS_95:
                            if (osVersionInfo.Version.Revision.ToString() == WINDOWS_95_SP1)
                            {
                                osVersion = "Windows 95 SP1";
                            }
                            else if (osVersionInfo.Version.Revision.ToString() == WINDOWS_95_OSR2X)
                            {
                                osVersion = "Windows 95 OSR 2.0";
                            }
                            else
                            {
                                osVersion = "Win95";
                            }
                            break;
                        // Windows 95 OSR 2.1, OSR 2.5
                        case WINDOWS_95_OSR:
                            if (osVersionInfo.Version.Revision.ToString() == WINDOWS_95_OSR2X)
                            {
                                osVersion = "Win95_OSR2.1";
                            }
                            else if (osVersionInfo.Version.Revision.ToString() == WINDOWS_95_OSR25)
                            {
                                osVersion = "Win95_OSR2.5";
                            }
                            break;
                        // Windows 98 and Windows 98 Second Edition
                        case WINDOWS_98:
                            if (osVersionInfo.Version.Revision.ToString() == WINDOWS_98SE)
                            {
                                osVersion = "Win98SE";
                            }
                            else
                            {
                                osVersion = "Win98";
                            }
                            break;
                        // Windows ME
                        case WINDOWS_ME:
                            osVersion = "WinME";
                            break;
                    }
                    break;
            }
            return osVersion;
        } 

Currently, C# does not have a property or method to determine the specific edition of an operating system. If a test is dependent on a specific edition of Windows 2000, Windows Xp, or Windows 2003 operating system edition then we need to invoke the Win32 GetVersionEx() function. If a test is dependent on a specific edition of Windows Vista or Windows Server 2008 then we can invoke the Win32 GetProductInfo() function.

Programmatically detecting the operating system version at runtime enables the test designer to design one test that can execute on a different operating system versions by allowing for specific control flow of a test based on the operating system the test is running. Programmatic platform profiling also negates the need to rewrite the same fundamental test over and over again to handle differences in behavior or expectations of operating system versions. It can also reduce some long term costs of test case maintenance, give the test designer greater control over the execution of an automated test, and increases the reusability and the maintainability of a test case that must be ran on multiple operating system versions.

Quite simply, programmatic platform profiling is a best practice in software test automation when a given test is dependent on the profile of the platform on which it is executing!

Certification Wars

I started diving in the late 70's, and in1985 I became a PADI certified open water scuba instructor. In those days, scuba instruction involved a lot of classroom time discussing various concepts that impacted human physiology such as Boyle's Law, Dalton's Law, Haldane's Principle, and also immediate responder first aid for scuba diving related maladies (most doctors are not trained in hyperbaric medicine and divers were generally more aware of potential symptoms of DCS as compared to most doctors). The training at the time emphasized in-depth knowledge of the theory of diving as well as repetitive practicing of fundamental and critical skills in controlled environments (usually a swimming pool) and 'real-world' environments (the open water). Certification entailed successfully completing multiple written examinations as well as demonstrated competence in the water.

I remember those early days when PADI and NAUI instructors would compete for students, and many instructors would ridicule other certification agencies while claiming the agency they supported to be superior as compared to the others. Some instructors would completely denounce other certifications as inadequate or claim that those certifications were not accepted worldwide, or make some other malicious attacks based on ignorance and their own personal motives.

Of course, we know that some people are incapable of thinking on their own, or they are easily persuaded with scare tactics, and some simply blindly follow the bloviated jabberwocky of  charismatic people. But, most intelligent people who are capable of processing cognitive rational thoughts are able to see through the prevarication and realize the hypocrisy of people who lambaste some certifications while selflessly pander their own certification.

My personal views on certifications in software testing haven't changed. I can see both the perceived benefits of certifications by employers as well as the limitations of certifications. For example, certifications in software testing or other professional disciplines are generally based on knowledge of the discipline rather than on the ability of a person to perform a particular task.

But this is really no different than other professional organizations. For example, it is possible to get a certification (Juris Doctor) to practice law in California without ever having gone to law school or presenting a case before a judge under the tutelage of a mentor. And a person only has to drive Interstate 5 through Seattle to know that the Washington state certified engineers who completed their requirements for certification succeeded in concocting a major transportation boondoggle. Perhaps the hundreds of successful major corporations in Europe and elsewhere around the world see the value of a well-established testing certifications such as the ISTQB and ASQ for their ability to help professional testers effectively communicate using a common discipline jargon rather then constantly coming up with confusing neologisms. (Of course, all these successful organizations could be wrong...but I tend to think they are successful because the  influential decision makers in those companies make the right decisions most of the time regardless of what some external person with a limited perspective or an idealistic neophyte thinks. Perhaps that's why they are successful!)

Of course, it would be ideal for certifications in our industry to also include practical skill-building exercises that taught testers how to test a product using various techniques and approaches, and certification required both practical demonstration and in-depth knowledge of the discipline. This doesn't simply mean that we teach people to find bugs by banging on the GUI, or asking 'probing' questions such as should a button control enlarge when a user mouse's over it. (Please...finding bugs is really not that hard, and if I wanted to know if a button control should enlarge or can enlarge I would look at the button properties for that control rather than sit there and ponder the question for 5 minutes.)

While I would like to see certifications include skill based learning along with teaching in-depth knowledge, we should also realize there are potential limitations with practical skill-building exercises. For example, in our own training we can assess if an individual learns to correctly apply a systematic procedure to design effective tests after in-depth analysis and logically decomposing a feature area or data set for a simulation used in our training. The concepts and application of some approaches and techniques such as combinatorial analysis are applicable across multiple software projects within appropriate contexts. So, we  instruct our SDETs to identify the well-defined contexts in which certain approaches or techniques are appropriate and when they are not, and we also explain how they are sometimes misused so they don't also fall into the same traps that untrained people fall into. However, current certification schemas can not yet accurately assess how well a person will perform on a real project until that person is put into that situation. This is true of software testing just as much as it is true of scuba diving.

Fortunately in scuba diving the certification wars are mostly in the past. But, it seems the testing certification wars are heating up. Today, certifications are valued in some business sectors for various reasons. I suspect some new certifications will come along and claim some great benefit beyond the others. And perhaps they will, or perhaps they will be isolated communities of zealots who want to simply be different. I just find it rather Pecksniffish for any person to claim all certifications are bogus; of course except for the one in which he or she has some personal vested interest.

Automation Foibles Unveiled: Saving random data

Now, many of you probably know that I am a big fan of computer generated random test data that is a represents a reasonable sample data set from the total population of possible test data. (I refer to this a probabilistic stochastic test data.) So, why would I argue against preserving randomly generated test data?

I just returned from STAREast, where for the second time in a month I heard someone suggest storing randomly generated test data in a file. Many people will site the inability to recreate random test data as a drawback to using randomly generated test data in a test. So, the reason these people suggested storing the random data in a file is so they can easily repeat a test with the same data should some randomly generated test data expose an anomaly. I absolutely concur that if we generate random test data, and that test data exposes a problem we need a way to recreate the data. But, isn't there a better way than to save random test data in a file?

Saving randomly generated test data to a file creates a test artifact. Depending on how much randomly generated data is generated, this file could become quite large. Also, saving data to a file impacts the performance of an automated test and certainly slows down manual execution of tests. Then consider the number of tests that generate random test data are executed numerous times throughout the lifecycle, and it doesn't take long until we have countless test artifacts simply storing more static test data that quickly loses its value (especially if no problems were detected). Of course, we can easily delete the files after the test if no anomaly was detected, but I suspect that most testers will delete those files upon the completion of the test if no problems were detected.

So, the question is how can we reproduce computer generated probabilistic stochastic test data if we don't save that randomly generated data to a file?

Planting Seeds

In computing, a seed is simply an integer value that is used by a random generator as the starting value. If we pass a seed value as an argument to a given random generator then we will consistently get the same random value each and every time. Essentially, a seed allows us to replicate computer generated probabilistic stochastic test data anytime as long as we use the same seed and the same random generator algorithm. So, instead of saving each and every piece of randomly generated test data used in any given test, we can simply log the seed value used by that test in the test results log file.

But, if we use the same seed all the time, then we are simply generating the same data over and over again. And, manually inputting a seed for each test that generates probabilistic stochastic test data is not an ideal situation, especially for automated tests. So, to solve that problem we can randomly generate a seed value that is then passed to the random generator algorithm!  Again, logging the randomly generated seed allows us to accurately reproduce the probabilistic stochastic test data at any later time.

The example below illustrates a simple method in C# that will either generate a random seed or return a user specified seed value.

        public static int GetSeedValue(string seedValue)
        {
            // check if user specified seed value is passed as an arguement to 
            // the seedValue parameter
            if (seedValue == string.Empty)
            {
                // Create a new random object
                Random randomObject = new Random();
                // Generate a random integer value between 0 and 2,147,483,647
                return randomObject.Next();
            }
            else
            {
                // convert the seedValue to an integer value
                // NOTE: This example method does not include exception handling
                return int.Parse(seedValue);
            }
        }

The following example illustrates how to use this method to get a random seed value to generate random strings and numbers that increase the breadth of test data coverage in each subsequent iteration of a test.

        static void Main(string[] args)
        {
            // These variables declare the range of characters used for the
            // string test data. In this case the strings are composed of upper
            // case ASCII characters 'A' through 'Z'
            char minChar = '\u0041';
            char maxChar = '\u005A';
            
            // This reads the user specified seed value from the console window
            // If no seed value is specified an empty string is passed to the 
            // GetRandomSeed method which will cause it to generate a random 
            // seed value.
            string mySeed = Console.ReadLine();
            
            // Declare a seed variable and initialize it to either the user
            // specified seed or to a computer generated random seed value
            int seed = GetSeedValue(mySeed);

            // The seed value should be permenently recorded in the logged
            // results for this test
            Console.WriteLine("The seed value for this test is {0}\n", seed);

            // Create a new random object based on the seed
            Random randomGeneratorObject = new Random(seed);

            // Generate 10 random strings
            for (int count = 0; count < 10; count++)
            {
                // Declare and initialize a string variable for our test data
                string testString = string.Empty;
                // Generate random length strings between 1 and 10 characters
                for (int length = 0; length < randomGeneratorObject.Next(1, 11); length++)
                {
                    // Generate a random character within the defined range and
                    // concatenate it to the testString variable until the 
                    // random string length has been reached
                    testString += Convert.ToChar(randomGeneratorObject.Next(
                        minChar, maxChar + 1)).ToString();
                }
                // Write the test string to the console window
                Console.WriteLine("Test String {0}: {1}", count + 1, testString);
            }

            Console.WriteLine("\nRandom numbers");
            // Generate 5 random numbers
            for (int numberCount = 0; numberCount < 5; numberCount++)
            {    
                Console.WriteLine("{0} ", randomGeneratorObject.Next());
            }
        }

Calling the Main method and passing an integer value between 0 and 2,147,483,647 will generate 10 random length strings composed of random upper case characters between 'A' and 'Z' and 5 random numbers. If no user specified seed is passed to the Main method then the code will call the GetGenerateSeed method and generate a random seed value for use in the test. Of course, passing the same integer value will produce the same strings and numbers each and every time.

Using probabilistic stochastic test data is valuable because it efficiently increases the breadth of data coverage, and significantly augments 'typical' static test data, user-generated test data, or static test data derived from historical failure indicators. But, instead of storing randomly generated test data in a file, it is a best practice to simply record the seed value of each test. With a seed value we can easily recreate the computer generated random test data should any of the random data used in a test exposes an anomaly.

Posted by I.M.Testy | 4 Comments

Email - the curse of productivity

It has been quite some time since I have posted. Part of that is due to personal distractions (getting my garden planted and my sailboat ready for the upcoming season), and part of that is being 'in the zone' working on some special projects at work. DeMarco and Lister began talking about being "in the zone" in their famous book Peopleware written in 1987. (In my opinion, this is a must read for anyone in the software business, especially for managers who should reread it yearly.) The Croatian psychologist Mihaly Csikszentmihalyi also discusses a similar concept he calls "flow," and identifies 9 characteristics of 'flow.'

Being in the "zone" for me is a myopic mental state where we are so focused on completing a task the world whizzes by and time becomes irrelevant. For many it is sort of a magical place; a momentary escape from reality. For example, when I sit down to write code in the evening the time passes so quickly that I soon discover it is 1 am in the morning. But, I want to complete one more class, and before I realize it is 4 am. (I am a slow coder.) People who are addicted to computer games know  all too well about the 'zone.'

This past month I had to come out of my zone and fly down to San Mateo, California to present a workshop and 2 talks at the Software Testing and Performance conference. It was a welcome break, and was nice running into old friends and meeting new acquaintances. I had the pleasure of meeting Karen Johnson and reconnecting with Doug Hoffman (who I had only met once previously) for lunch one day, and interestingly enough the conversation found its way to being in the zone. Karen brought up the fact that email is a constant distraction that often times impedes productivity.

Often times when we are in our zone our productivity increases. But, every time something changes our focus, such as responding to an email on a completely unrelated or tangential topic, or answering a phone call we are sucked out of the zone, and according to Lister and DeMarco it takes approximately 30 minutes to get back into the zone or back to our peak point of productivity.

For example, when I sit down to write, or code, or meditate I don't want to be disturbed and will usually retreat to my boat or some quite place where I know I won't be disturbed. I turn off my cell phone, no radios, no newspapers, magazines, no instant messaging (which I abhor) , and certainly no email. If I must stay at the office, then I block of my calendar for at least 4 hours and will sometimes disappear into some nook or cranny on campus.

I know that email is the life blood of many tech-companies. Unfortunately, I suspect that many of us could use a good bloodletting and need to relearn the art of verbal communication. I also suspect that we could better optimize our time by not being tethered to some email client during every single minute our waking hours. Let's face reality. For most of us 80% of the email we get is noise that we will forget about within the next 4 hours or so, 15% is good to know stuff (but not necessarily critical to our success), and I suspect that approximately 5% is really important stuff that we must respond to immediately. Of course, these percentages depend on our primary job. For example a manager or a consultant probably gets a larger percentage of email the requires immediate response.

So, I wonder if we managed our own use of email (and instant messaging) a bit more effectively how our own personal productivity might increase?

Posted by I.M.Testy | 2 Comments

GUI Automation and ROI

It seems that many test automation efforts around the industry tend to focus on GUI automation, or automating functional tests primarily by manipulating GUI objects. In general, GUI automation tends to be a very expensive approach to test automation, and the automation efforts often end in failure or achieve less than satisfactory results.

The majority of automated tests at Microsoft are below the GUI; however, automated tests that manipulate GUI objects are quite useful within specific contexts. Unfortunately, many testers attempt to develop automated GUI tests way too early in the project cycle while the user interface design is still unstable. I guess the assumption is that constantly maintaining automated tests is somehow better then executing manual tests. But, in general, when the UI is in flux it is usually counter-productive and a loss of return to automate GUI level tests too soon in the development lifecycle.

Dan Mosley and Bruce Posey (Just Enough Software Test Automation) suggest that on average an automated test must run approximately 17 times in order to break even. But, this doesn’t imply that we break even if we simply run the same test on a daily build for the next 17 days. The presumption underlying the ROI after 17 runs of a test is that something changed in the build that is covered by that particular test, and so by executing that (regression) test we are providing important information (changes in the build did not destabilize that area) to the team. Steve Rowe also has an excellent blog post on Too Much Test Automation that you should consider reading, and Dustin Andrews also has an excellent blog post on getting great results from test automation with an 8-minute video.

The cost of test automation is never easy to figure out and is certainly not a straight forward comparison of automated time versus manual time. Comparing automation time versus manual execution time is an overly simplistic measure that rarely takes into consideration the design and the development time of the initial tests (or the overall costs of building and enhancing test frameworks or drivers), or the time required to identify false negatives, troubleshoot the cause, fix the problem, verify the fix (manually), and then check the source back in for the next run. (And, there are other tangible costs, and intangible costs such as loss of confidence that must also be considered in any cost model.)

So, in general any automation that requires constant maintenance is usually not cost effective, and the more our test automation throws false negatives the less our management team views automation as a viable resource to provide us with valuable and reliable information for improved risk analysis. Ultimately the decision comes down to how much perceived and measurable value a test has in providing important information for improved risk assessment and quality measurement.

Just because we can automate something, doesn’t always mean we should!

Posted by I.M.Testy | 16 Comments

Customer expectations

Last October after presenting a keynote at the Conquest software testing conference I was invited to speak at an internal quality conference at SAP. At first it may seem a bit odd because Microsoft and SAP do compete within one market segment; however we also do collaborative work on other projects. Regardless of the company we work for, I do believe that most software engineers are intent on doing the best possible job they can to produce the best possible product they can that will ultimately provide a high value solution to the end user customer.

The common theme throughout the conference was the need to improve quality. In my keynote I suggested that customers demand higher quality because the end user customers of today are very different than the customers of yesterday. Today, software permeates virtually all aspects of our life. Today, software is found in children's toys, in our automobiles, and even in toothbrushes. A decade ago computer users were accustomed to periodic anomalies and  assumed it was the price of technology; however the end user customers today have much higher expectations of software and  presume it will simply work and provide an easy solution that offers some perceived value in their lives!

In my keynote address to the engineers at SAP I described how I would sometimes say "my Mom wouldn't understand how to do such and such" when describing ambiguous functionality, but in order to remain successful and competitive in today's market we need to consider designing and developing software for our children and future generations. Of course we want our existing customers (and my mom) to feel comfortable using our software, but it is readily apparent that our children want a very different experience and have higher expectations from software.

Posted by I.M.Testy | 2 Comments

Do testers do code reviews?

This weekend on the flight from Seattle to Ireland I finally got to catch up on some reading. One of the books I grabbed off the shelf that I hadn't gotten around to reading yet was Best Kept Secrets of Peer Code Review by Jason Cohen. The book is 160 pages packed with great information, and I highly recommend it for some fresh perspectives of the value or ideas on improving code reviews. For decades the industry collected mounds of empirical data that pretty clearly illustrates the value of code reviews in a software development lifecycle in the early detection and removal of issues and anomalies. With the industry wide push to drive quality upstream via approaches such as agile programming and test driven development there is a resurgence of effort by developers to engage in more frequent code reviews.

Many people seem assume the primary reason why Microsoft (and other companies) is recruiting more technically skills testers is simply to automate tests. But, in fact, many testers at the company engaged in writing test automation long before our efforts towards engineering excellence. The skills and knowledge that people who have a much deeper understanding of the 'system' bring to an organization extends well beyond their ability to write automation.

But, should testers be performing code reviews? In my opinion, code reviews are simply another approach in a myriad of approaches to testing a software project, and professional testers should be able to engage the testing effort from a variety of testing perspectives.

Testers who participate in project code reviews are part of the team effort to drive quality upstream, reduce certain classes of issues before they are checked into the build, improve overall long-term maintainability of the code base, and effectively reduce long term costs. Just as testers participate in the early stages of the design and requirements phase of a project to provide valuable input, some testers on every team should engage in code reviews either with other developers, or perhaps as the primary reviewers of project code. Some people will argue that testers may be biased by participating in code reviews and lose the 'customer' perspective. This may be true in some cases, but I suspect that most professional testers clearly understand the different classes of issues that are more easily found in a code review versus other approaches of testing. But, testers who engage in code reviews tend to have a much deeper understanding of the overall project, and they are also able to identify potential areas of the code that may be more problematic and focus additional testing approaches in those areas.

Vista Rant #3 - And yet another boundary issue in Explorer's listview?

This morning I installed Vista SP1 onto my laptop. I was pretty excited about this release of Vista SP1 because it includes some pretty significant performance enhancements. But, as I was preparing to teach an internal course I came across a new boundary issue. I thought, how fitting this comes as I prepare to teach another class on systematic testing techniques (including boundary value analysis) that I find yet another classic boundary issue in Explorer's list view (albeit the boundary condition is not readily apparent) . Interestingly enough, the previous boundary issue I found appears to be fixed with Vista's SP1; however, I previously did not run across this issue which is also connected with how listview repaints itself after an event.

To reproduce this issue, open a folder with several dozen files in Explorer and select Views -> List and resize the Explorer window so there are several columns of files.

Select a file in the list and press the up or down arrow key so a dotted line appears around the file name

Resize the Explorer window so that the highlighted area is about 1 pixel away from the right most edge of the Explorer window as in the example below

LV1

Press the down (or up depending on which file you have selected) arrow key, and notice how the file highlight jumps to the next file in the list as expected.

Next, resize the the Explorer window so the dotted line touches the inner boundary of the window as in the example below

image

Now, press the down (or up depending on which file you have selected) and...

 image

Notice, in the image above that just to the left of the frogger.def file the right most edge of the file highlight is visible, and also notice the file image has changed, and the scroll bar has jumped over to the next column.

Press the down (or up) arrow key again, and....

image

Seeing this behavior the first time made me think I was losing my mind! (OK...I probably am, but that is a completely different topic). And yes, reducing the size of the window beyond where the maximum boundary of the highlight window will also cause the same behavior, but the issue occurs when the width of the file highlight window >= the maximum x boundary size of the Explorer listview window.

The listview in Windows Vista has some very cool features, and overall I like Vista. What I dislike are some of the simple anomolies that could be easily exposed via more detailed, systematic testing approaches and analysis of the system under test rather than simply assuming more bodies banging on keyboards for some arbitrary period of time equates to better testing.

So, some things to consider when boundary testing...

  • Not all boundary conditions are easily identified from the GUI by numbers, but all boundaries conditions have linear physical values that can be measured at some level
  • Boundaries can change, similar to way the boundaries changed around Kosovo. Somebody moved the lines. The same thing is true in software, a boundary condition may change with human interaction, but a boundary condition is a linear physical range between a minimum and a maximum value (in this case the x coordinates of the Explorer changed with human interaction, but once focus moved away from the Explorer window the size of x established a fixed boundary (at least until a user again resized the window or the window is closed).
  • Boundary conditions for one parameter are usually independent. For example, resizing the y-axis in the above scenario has no impact on this defect (unless of course the y-axis is large enough to accommodate all files in the listview without having to scroll horizontally). Boundary testing is based on the single fault assumption theory which states that a boundary issue is most likely to occur with independent parameters where the boundary variables for one parameter are analyzed while holding other parameters to nominal values. (Note: if we suspect that parameters are dependent or semi-coupled, then we should also perform combinatorial analysis testing.)
  • There is a difference between boundary conditions (at least momentarily fixed, linear, measures) and threshold values. Threshold values can be altered by various influences. For example, in performance testing the point of degradation in performance can often be changed by several external influences such as increasing physical memory, cleaning and defragmenting the hard disk, modifying the software, etc.
  • A detailed analysis of the system under test will reveal issues that other approaches to testing do not expose...that's the pesticide paradox!

Contextual blindness: or How to take things completely out of context

Many testers are familiar with the concept of inattentional blindness (or at least should be in my opinion). Basically inattentional blindness occurs when we are so visually focused on a task  or object that we completely fail to see something out of the ordinary.

But, I am going to introduce my own neologism that I will refer to as contextual blindness. Contextual blindness occurs when someone is so restrictive in their thinking or so biased by their own opinion that they take references to a document or study completely out of context to support their own biased argument. In essence, they are ignoring the original context in which the statements are made, and perverting the sentence or using a statement out of its original context to support an oppositional point of view.

I too have been guilty of this when I made statements without completely researching available data or carefully reviewing empirical, or factually substantiated evidence. (These days, if I don't have sufficient data or information to support a strong argument for or against something then I try to preface my statement with "I suspect..." or "in my opinion..."). However, some people seem to make a habit out of making wild, and often fallacious statements often in seemingly juvenile attempts of one-upsmanship. I suspect that sometimes people do this because they think they are beyond reproach; that they assume to know more than others, or that they consider themselves to be such an expert that nobody should question anything they say.

As I have gotten older and a bit more wiser, I have learned to question things and reassess my position or my ideas from time to time. I often speak with recognized industry experts, read several books and studies (often presenting  contradictory approaches or perspectives), review empirical data (and just to be clear, IMHO bug count as the only data point offered as empirical data is about as useful as nipples on men), and when I can I try to experiment or experience new things in order to draw my own conclusions. By now your probably asking where I am going with all this...there is a point...read on!

This evening a person sent me mail asking me if my keynote at EuroStar last year was "an attack on certification programs." She knew I gave one of the keynotes at EuroStar, but was a little shocked that I would attack certification programs. I told her that I gave one of 5 keynote addresses at EuroStar 2007 and my talk was entitled The Path to Professionalism: Skills of star performers, and gave her