Welcome to MSDN Blogs Sign in | Join | Help

Contextual blindness: or How to take things completely out of context

Many testers are familiar with the concept of inattentional blindness (or at least should be in my opinion). Basically inattentional blindness occurs when we are so visually focused on a task  or object that we completely fail to see something out of the ordinary.

But, I am going to introduce my own neologism that I will refer to as contextual blindness. Contextual blindness occurs when someone is so restrictive in their thinking or so biased by their own opinion that they take references to a document or study completely out of context to support their own biased argument. In essence, they are ignoring the original context in which the statements are made, and perverting the sentence or using a statement out of its original context to support an oppositional point of view.

I too have been guilty of this when I made statements without completely researching available data or carefully reviewing empirical, or factually substantiated evidence. (These days, if I don't have sufficient data or information to support a strong argument for or against something then I try to preface my statement with "I suspect..." or "in my opinion..."). However, some people seem to make a habit out of making wild, and often fallacious statements often in seemingly juvenile attempts of one-upsmanship. I suspect that sometimes people do this because they think they are beyond reproach; that they assume to know more than others, or that they consider themselves to be such an expert that nobody should question anything they say.

As I have gotten older and a bit more wiser, I have learned to question things and reassess my position or my ideas from time to time. I often speak with recognized industry experts, read several books and studies (often presenting  contradictory approaches or perspectives), review empirical data (and just to be clear, IMHO bug count as the only data point offered as empirical data is about as useful as nipples on men), and when I can I try to experiment or experience new things in order to draw my own conclusions. By now your probably asking where I am going with all this...there is a point...read on!

This evening a person sent me mail asking me if my keynote at EuroStar last year was "an attack on certification programs." She knew I gave one of the keynotes at EuroStar, but was a little shocked that I would attack certification programs. I told her that I gave one of 5 keynote addresses at EuroStar 2007 and my talk was entitled The Path to Professionalism: Skills of star performers, and gave her my perspective of certifications. In her response she forwarded me some mail from a distribution list which included excerpts from a rather lively debate between two other members of the list in which one  of the participants in the debate stated,

"The Department of Defense has identified the failure of traditional testing processes, including the problem of over-documentation, as one of the top five problems in IT. The keynote speech at Eurostar, in December was an attack on certification programs such as the CSQE."

So, I reminded her that the second sentence should more appropriately read, "One of the keynote speeches at Eurostar...". I told her that Michael Bolton gave that talk. (I must say that it was the first time I had seen him speak and I his stage presence is excellent, but I wasn't overly impressed with his arguments against certification because he simply denigrated the existing programs without offering any other solutions. Several delegates at the conference later asked me about his comments and I deferred by stating, "when in Rome." (Certifications in Europe are highly valued by employers for a variety of reasons.)  But, the above mentioned statement was merely misleading; it is actually the first statement that is most fallacious, and taken completely out of context.

The statement "The Department of Defense has identified the failure of traditional testing processes, including the problem of over-documentation, as one of the top five problems in IT" used the following study as a reference. So, let's take a critical look at that statement and question its truth or validity. (Because, in our jobs as professional testers understanding the correct context, critical thinking and logical questioning are important skills!)

"All we want are the facts, ma'am"

Fact #1. The DoD did not identify anything. The study was conducted by the National Defense Industrial Association (NDIA), which is not in anyway shape or form a part of the Department of Defense (DoD). NDIA members include "individuals from academia, government, the military services, small businesses, prime contractors, and the international community, the opportunity to network effectively with the government."  There were 26 participants in this study and one participant represented the DoD.

Fact #2. The study did not identify "the failure of traditional testing processes."  The purpose of the study conducted by 26 participants was to "Identify the top 5 software engineering problems or issues  prevalent within the defense industry." The participants actually came up with 7 issues, and they determined that one of the top issues in software engineering included "Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems."

Fact #3. The problem of over-documentation was in the context of a discussion related to tests with a "disproportionate effort on detailed procedures." This was not one of the 5 (actually 7) problems identified, the participants concluded "Tests are over-documented with disproportionate effort on detailed procedures." as a possible reason to explain the 5th top issue of "Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems." I suspect this statement speaks to the fact that many documented tests that I have seen by under-trained testers are simply prescriptive, regimented scripts based on some interpretation of an ambiguous requirements document rather than well-formed tests designed from an in-depth analysis of the system under test.

What the study really said...

But, not only did this person take parts of several statements in the report, munge them together in an attempt to use the findings completely out of context; this person also completely ignored (or purposefully omitted)  other points discussed in relation to this specific problem such as:

  • Over-reliance on testing alone rather than robust SW verification techniques.
  • Manual testing techniques are labor-intensive, scale poorly, and are unproductive relative to the large investment of resources."
  • Compliance based tools do not adequately cover risks or failure conditions
  • Tests are over-documented with disproportionate effort on detailed procedures
  • Education, training, certifications are inadequate to develop effective test skills.

The person also ignored (or purposefully omitted) the recommendations by the participants which included:

  • Sponsor a study of state-of-the-practice verification and testing approaches.
  • Review/update testing policies and guidance to emphasize robust, productive approaches that maximize ROI.
  • Review adequacy of verification plans/approaches early in the acq. life cycle.
  • Emphasize skilled investigation throughout the life cycle, based on coverage, risk mitigation, high volume automation.
  • Strengthen curricula, training, certifications, career incentives for testing roles.

Now, I really don't understand how someone can read this report and misconstrue the information to imply that traditional testing processes (and it is not clear what they are referring to here), or that over-documentation is one of the top 5 problems identified by the DoD; especially when the report also recommends strengthening policies or guidelines for full requirements traceability.

I have my suspicions as to why the person who made the fallacious remark above might omit all the facts and additional counterpoints in their argument. I suspect those details were not revealed because those points do not support (in fact, they completely dispute) the context-driven ideology that emphasizes manual GUI testing and the vehement opposition to documentation, test automation, coverage analysis, robust verification techniques, strengthening of certifications, or essentially anything that involves the need for greater technical know-how, logical analysis, or measurable advancement of the testing profession.

Published Wednesday, February 13, 2008 8:56 AM by I.M.Testy

Comments

# re: Contextual blindness: or How to take things completely out of context

>>>I suspect those details were not revealed because those points do not support (in fact, they completely dispute) the context-driven ideology that emphasizes manual GUI testing and the vehement opposition to documentation, test automation, coverage analysis, robust verification techniques, strengthening of certifications, or essentially anything that involves the need for greater technical know-how, logical analysis, or measurable advancement of the testing profession.

This sentense is bit tough to read and I suggest you rephrase this in 2-3 more sentenses so that we can understand your viewpoint

I assume that the word "I suspect" or "In my opinon" is applicable the eitnre sentence above.

Meaning -

1. You can not very sure if context driven community opposes test automation and other things you mentioned.

2. You are not likely to be sure if context driven community opposes anything that involves the need for greater technical know-how, logical analysis, or measurable advancement of the testing profession.

Can you share few examples to demonstrate above view point of yours?

As a "self proclaimed" member of context driven testing community, I can say for sure that our community does not oppose test automation and anything that involves the need for greater technical know-how, logical analysis, or measurable advancement of the testing profession.

Please do check the principles of context driven school .. nothing there suggests that we oppose test automation or a need for technical know-how.

http://www.context-driven-testing.com/

Please do let me know if I have mis-interpreted your words "out of context"

Wednesday, February 13, 2008 4:50 AM by Shrini

# BioSensorAB » Contextual blindness: or How to take things completely out of context

PingBack from http://www.biosensorab.org/2008/02/13/contextual-blindness-or-how-to-take-things-completely-out-of-context/

Wednesday, February 13, 2008 4:55 AM by BioSensorAB » Contextual blindness: or How to take things completely out of context

# Not sure Blindness is the word I would use

I totally agree with your article; this is a problem that, frankly, occurs everywhere (politics comes to mind -- let me form an opinion, then massage/twist the facts to justify that opinion). Really, though, this is only one step removed from the pure dogmatic approach of just having an opinion and ignoring any facts.

That said, though, I don't know if 'blindness' is quite the word I would use. Obstinateness (yes, that's a word) or just dogged ignorance mixed with hubris. Although that last one doesn't really make for a catchy phrase: "Contextual ignorance through hubris".

Personally, I have found over the years that there are many people happy to say "This is the way we do it," but only a few people willing to ask "Is there a better way to do it?"

Wednesday, February 13, 2008 11:31 AM by trainer_erich

# re: Contextual blindness: or How to take things completely out of context

Hi Shrini,

Hmmm...OK...let me break this down since you seem to be having a hard time grasping the point.

I suspect the points were not revealed for the following reasons:

- Context-driven ideology heavily emphasizes manual "human" test execution

- People who associate themselves with that ideology tend to have negative views of test automation

- People who associate themselves with that ideology tend to discount robust verification techniques or logical, in-depth, technical analysis of the system

- People who associate themselves with that ideology tend to have a distain for documentation

I really don't have time to do an in-depth search and report on all the instances to back up my supposition. But, here are a few quotes from people who self-identify themselves as 'context-driven' testers:

"How do you propose to get automation to do those things?"

"I think most documentation in most test projects is a complete waste."

"I'm not the one presuming to tell others that good testing requires that you spend your time writing on paper instead of ACTUALLY TESTING."

"That testing should be context-driven rather than practice-driven, standards-driven, or specification-driven."

"The RUP test strategy template is crap. The IEEE-829 template is crap."

"Pairwise testing...it's dressed up in fancy mathematical clothing, and it looks like it might reduce our workload. Does it provide the kind of coverage the kind that's most important to the project?"

(Actually, the answer to that question will be revealed at StarEast 2008 with empirical data and not just emotional feel-good and numbers of bugs.)

I know what the so-called principles state. But, what is stated and what is preached and practiced can sometimes be very different things. For example, before our govenor was elected she said she would not raise taxes. Yet the first thing she did was raise taxes.

But, I think you miss the point of this post. The point of this article is about taking things out of context, spinning them, and then simply making stuff up to suit one's own personal needs.

This post was really not a bash against people who think that some special school exists, or that their ideology of testing is better than the holy grail. However, the person who made the fallicious statement that I critically analyzed associates himself with that ideolgy. And I must say that some self-annointed context-driven ideologs are more driven to take things out of context and spin them into perverse meaningless arguments against automation, documentation, etc. without ever offering any other viable or meaningful alternative to significantly advance the profession (with empirical data to prove or disprove a claim beyond "yippie....I found another bug!").

Wednesday, February 13, 2008 1:48 PM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

Some more views ...

>>>> Context-driven ideology heavily emphasizes manual "human" test execution

I would add that "using tools as appropriate to supplement human capabilities". So if you read the whole sentence - "Context-driven ideology heavily emphasizes manual "human" test execution using tools as appropriate to supplement human capabilities" - Do you find any issue or concern? Is it good or bad? Even otherwise what is bad in the emphasis on manual human execution?

>>>> People who associate themselves with that ideology tend to have negative views of test automation

I strongly disagree ... I have not heard any such arguments against test automation. Any opposition to test automation might have been with notion of implementing automation as replacement to human testing.

>>>>> People who associate themselves with that ideology tend to discount robust verification techniques or logical, in-depth, technical analysis of the system

Context driven people are against blind beliefs on techniques (we have discussed this lot before - but could not conclude in any way). When it comes to techniques, you seem to put techniques first then a professional tester to choose relevant techniques where as I and others context driven community put human tester first and let her choose techniques that are relevant. So we are hitting a dead end ..

>>> People who associate themselves with that ideology tend to have a distain for documentation

That kind of documentation that distracts from good testing. There are several instances where documentation becomes main requirement than testing. In that case we respect documentation. In fact, context driven testers always have "testing mission" in the mind at priority. If testing missing is to produce documentation - we do it with at most care and passion.

>>>>>>I really don't have time to do an in-depth search and report on all the instances to back up my supposition. But, here are a few quotes from people who self-identify themselves as 'context-driven' testers:

>>> "How do you propose to get automation to do those things?"

This is incomplete ... I am not sure what kinds of things being talked about? I am sure, there are several important things as part of any useful testing that automation can not do.

And you can not use this statement as an indication that context driven people have negative opinion on automation.

>>>"That testing should be context-driven rather than practice-driven, standards-driven, or specification-driven."

What is wrong with this statement ... this is true and I agree with this... Any problems?

>>>>"Pairwise testing...it's dressed up in fancy mathematical clothing, and it looks like it might reduce our workload. Does it provide the kind of coverage the kind that's most important to the project?"

Personally, I agree with this statement. This boils down to our discussion on techniques. Pairwise technique is a mathematical one and best usage of this especially depends upon human tester. Morever, pair wise technique is a heuristic ... based on the fact that only pairwise interactions matter ... it can fail and terribly fail.

>>>But, I think you miss the point of this post.

No I have not. Apart from the points I have raised here, I agree with rest of the post. No issues.

>>>>> The point of this article is about taking things out of context, spinning them, and then simply making stuff up to suit one's own personal needs.

If you were to do that you could have just stopped at your view points on certification. Why attack context driven community? I would say - by quoting comments made by context driven people (such as on automation), without specifying the context in which those statements were made, you are committing the same mistake as the person who sent you mail about certification. Comments made against documentation had

>>>>I must say that some self-annointed context-driven ideologs are more driven to take things out of context and spin them into perverse meaningless arguments against automation, documentation, etc. without ever offering any other viable or meaningful alternative to significantly advance the profession (with empirical data to prove or disprove a claim beyond "yippie....I found another bug!").

I don’t think those are meaningless arguments. But they can be made to look meaningless if you remove the context in which those statements made (as you have done here). All those the statements you referred here (allegedly made by context driven people) are standing bare without context - add the context they might look just right.

Significant alternatives ... for all you know there may not be any ...

Shrini

Thursday, February 14, 2008 1:12 AM by Shrini

# re: Contextual blindness: or How to take things completely out of context

BJ,

My earlier comment had an incomplete sentence

"Comments made against documentation had .."

should be read as

"Comments made against documentation had some context to it. Under those circumstances, documentation was waste of an effort. I believe it is not an universal statement that all test document is bad and worthlesss"

Shrini

Thursday, February 14, 2008 2:01 PM by Shrini

# re: Contextual blindness: or How to take things completely out of context

Some of these statements are without full dialog, but they convey the intent of that conversation, and are within the context of the perception that many people seem to have of both exploratory testing and the context-driven ideology.

I will say some common misconceptions of exploratory testing is unfortunate because exploration can be a valuable tool in the toolbox of a professional tester when used correctly and within its limitations. Knowing when to use certain tools, how to use certain tools within an appropriate context, understanding the capabilities and limitations of tools in various contexts, and when NOT to use tools in inappropriate contexts is the difference between an amateur and a professional.

For example in support of a statement regarding pair-wise testing you stated: <quote>"Personally, I agree with this statement. This boils down to our discussion on techniques. Pairwise technique is a mathematical one and best usage of this especially depends upon human tester. Morever [sic], pair wise technique is a heuristic ... based on the fact that only pairwise interactions matter ... it can fail and terribly fail." </quote>

I have said before that the successful application of any technique, method, or approach to testing relies on a tester's in-depth knowledge of the system. The pair-wise technique is not based on the fact that only pair-wise interactions matter. Combinatorial analysis (which includes t=2 or testing pair-wise combinations) is one technique among several that is useful in systematically analyzing parameters with direct or semi-coupled interaction, and there are multiple detailed studies with lots of empirical data to demonstrate it's overall effectiveness and value add to the project when used correctly in the right circumstances. (But, of course, the highly educated scientists and testers at NASA, the DoD, Boeing, AETG, Siemens, Microsoft, Google, Daimler-Chrysler,  etc., could all be wrong.)

Of course, in the hands of an amateur who tries to apply this technique to parameters that are not at least semi-coupled, or involve sequential operations, or mathematical calculations then I agree pair-wise testing can "fail and terribly fail."

Here are a couple of things to think about.

  • Do you really believe that testing occurs without context?
  • Why do only a handful of testers (relative to the total population) claim to be of a specific mindset while everyone else adopts the appropriate approach to achieve the goals necessary to reach the final objective without claiming some allegiance to some 'school' or ideology?
  • Why is it that many people who lack in-depth technical knowledge of the system they are working on, or claim they don’t need to know even basic programming concepts to be a good tester seem to identify with the so-called "context-driven school" and extol the virtues of exploratory testing as superior to anything else?

I understand the value of exploratory testing and I believe all testing occurs within some implicit or explicit context (because without context in our lives we would simply be walking around bumping into trees). But, I also know that people often have an incorrect perception of exploratory testing and its values and limitations. I wonder why that is given all the information there is on the subject?

Thursday, February 14, 2008 6:23 PM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

>>>Some of these statements are without full dialog, but they convey the intent of that conversation

I don’t think so ... Not some, all the statements are without full dialog.

>>..many people seem to have of both exploratory testing and the context-driven ideology.

How did exploratory testing get into this discussion?

>>>For example in support of a statement regarding pair-wise testing you stated

No .. My statement was NOT in favor of pairwise testing but in favor of the statement that is attributed to a context driven tester. I am of the opinion that pairwise testing is a simple mathematical model that helps us to comprehend multi variable scenario. Beyond that it is up to "sapient" tester to use it appropriately.

>>> The pair-wise technique is not based on the fact that only pair-wise interactions matter.

Is it so? Then why it is called pairwise technique? Is the technique wrongly named?

>>>But, of course, the highly educated scientists and testers at NASA, the DoD, Boeing, AETG, Siemens, Microsoft, Google, Daimler-Chrysler,  etc., could all be wrong.)

100% all of these people could be wrong ... Newton was proved wrong so some might prove Einstein wrong .... We all are learning. Nothing is static in this world .. No knowledge is ultimate truth.

>>>Of course, in the hands of an amateur who tries to apply this technique to parameters that are not at least semi-coupled, or involve sequential operations, or mathematical calculations then I agree pair-wise testing can "fail and terribly fail."

Well, I propose a rephrase pairwise technique fails when the person who applies to gets blinded by empirical data, great white papers from highly educated scientists and testers at NASA, the DoD, Boeing, AETG, Siemens, Microsoft, Google, Daimler-Chrysler,  etc. and stops thinking "how this technique can fail me". It fails when the person fails to model the problems space appropriately. It fails when tester's perspective is narrowly focused on mathematics behind the technique not on other possibilities ....

>>>Do you really believe that testing occurs without context?

Yes, in those circumstances, where people talk about empty words like "process", "best practices", "systematic methodology", "Technical depth" and so on. If you hear some one using these terms to describe their testing ... you suspect "out of context" there ....

>>>>Why do only a handful of testers (relative to the total population) claim to be of a specific mindset ...

It is not claim but a real mind set of thinking differently. These people acknowledge the "high human" content in software engineering - these people are systems thinkers. Only few in today’s world can accept that software testing is much more than code, techniques, automation. Very few in today’s world acknowledge software (a system) operates a set of complex systems (human, society, and cultures, psychological, economical) and software tester requires systems thinking concepts. There are very few who can relate software testing to philosophy, epistemology, critical thinking, lateral thinking, general sytems theory, scientific thinking ... while rest of the world is after techniques, code, automation. Only handful of tester have this kind of thinking.... that is true. Is such kind of thinking is useful ...? only time can tell.

>>>while everyone else adopts the appropriate approach to achieve the goals necessary to reach the final objective

I am not sure what does this mean ... words like "appropriate", "necessary” can mean nothing or everything.

>>> without claiming some allegiance to some 'school' or ideology?

Discussion on schools can be endless. Let us agree to disagree. First set of people (having specific mind set) might identify themselves to a school as it helps them find common ground with others that think alike. It helps to form a community. It helps to reconcile the fundamental difference about software testing. Second set of people feel that there are no differences about the way testing is performed hence prefer to live in a world where every one agrees to everything that is being said. It is the choice one makes.. I prefer to belong to a school as it helps me to develop my thinking, ideology, skills in testing.

>>>Why is it that many people who lack in-depth technical knowledge of the system they are working on, or claim they don’t need to know even basic programming concepts to be a good tester seem to identify with the so-called "context-driven school" and extol the virtues of exploratory testing as superior to anything else?

Who are they ... do you know them? Can you name them? What is community opinion about them? What is level of acceptance of these folks in overall testing community? Are they influential? Do they have followers? Can they do good testing? Can they demonstrate good testing "on demand"?

From general systems theory, I model software as one system operating among other systems such as human beings, their society, cultural background, organizations, businesses, economic system, political system, nature, related biological systems and list is endless... Hence a sapient tester (or a context driven tester) can see and appreciate this big picture. So in this complex system of systems - software code, technical knowledge is one small subset. Without knowing "much" (not that it is difficult to know) in terms of programming language, operating systems, systematic techniques etc, a tester can still contribute in several useful ways. Out of 100 odd things that a tester needs to know, technical (technique oriented and technology oriented) is one list item. They are like few stars that I can see in whole sky with several other stars.

Technical knowledge, its depth and the concept of system can mean many things to many people. Moreover, such people can learn and improve the depth. No big deal ...

Whether a tester need to have fundamental programming concepts - is debatable in above context. A non programming tester - can either learn programming (no big deal !!) or pair up with a "programming" tester or simply use other skills or critical thinking, systems approach, questioning, modeling can fill the gap and still be a "great" tester. I repeat, there are 100 odd things that tester requires to be effective, programming skills, technical (or technical) knowledge is one of them. Without that too .... a tester can be effective. For a sharp thinker and quick learner it might take few days to few weeks to pick up programming concepts while it would take ones entire life or several life's to be good at testing in this complex world.

why context driven testers seem to have such opinion about programming skills and technical knowledge ... this is because

1. They see the big picture - the systems view where software is just another system.

2. They can see the sky and spot programming/technical skills as 1-2 specific stars and there are other millions of stars.

3. Few of them are great thinkers and quick learners - for them learning a programming language or a technique does not take much ... they constantly attempt to be good at still lower level skills - thinking, modeling, analyzing, creating and using heuristics, general systems theory, scientific thinking etc that are "tougher" to master ...

Exploratory testing (your favorite term for bashing context driven people - a surprise entry for this discussion) is not a superior thing in context driven methodology. In fact, if you understand the philosophy of context driven thinking, there is no superior thing. There are some good and some not so good things/approaches to testing GIVEN a CONTEXT.

Context driven thinkers do not sell ET as Holy Grail or "the superior" thing. I would like to mention two other things that are specific to our community.

1. We view all testing as "heuristic" - a fallible method to solve problem. Whatever approach or technique we use, we are acknowledge that it can fail - after all it is fallible. Hence there are no universal methods of testing in our context driven ideology. Please, please note this ...

2. For every approach or technique we use, we would ask "Can I think of 3 situations or scenarios in which this can fail me". So while applying ET to any testing, it goes through same screening method as this. We would ask, in what 3 contexts, ET would be inappropriate. This is what we strongly believe in.

So much for what context driven people stand for ...

Shrini

Friday, February 15, 2008 8:03 AM by Shrini

# re: Contextual blindness: or How to take things completely out of context

Hi Shrini,

Your right...all of the statements were without the full dialog. You asked for examples, and I replied "here are a few quotes" as examples to illustrate how I reached my conclusions.

I stated "For example in support of a statement regarding pair-wise testing you stated..." I was not suggesting that you support pair-wise testing, beause I know you do not. This is the complexity of the English language. In between the word 'in' and the word 'support' is an implicit pronoun 'your'. When I said that you supported a statement regarding pair-wise testing implied that you supported an argument against pair-wise testing; not that you supported pair-wise testing.

Actually, your statement, "pairwise testing is a simple mathematical model that helps us to comprehend multi variable scenario" is an illustration of exactly how it can be misused. I highly suspect that no one familiar with the algorithm would say the formula is simple, and this technique doesn't apply to all multi-variable scenarios.

You asked, "...why it is called pairwise technique? Is the technique wrongly named?" Actually, the technique is more properly called combinatorial analysis; pair-wise testing or analysis (t=2) provides only one level of testing combinations in 2 pairs. A lot of studies suggest that t=3, t=4, t=5, and t=6 combinations may be necessary in highly complex systems in which parameters are dependent or semi-coupled.

When I asked you if you believed that testing that occurs out of context you responded, "Yes, in those circumstances, where people talk about empty words like "process", "best practices", "systematic methodology", "Technical depth" and so on. If you hear some one using these terms to describe their testing ... you suspect "out of context" there ...." The words are not empty, they certainly convey meaning in the appropriate context. I certainly don't suspect "out of context" when I analyze something systematically. Perhaps doctors have no context when analyzing lab test results, or perhaps architects have no context when they test the design of bridges or buildings, but, when I employ a process to systematically analyze a feature I am very clear of the context before I embark on such a task because I understand that it takes a lot of skill, in-depth knowledge and technical depth in order to be highly successful. Of course, I could be wrong and you have some empirical data to clearly demonstrate that systematic methods, or processes are simply the wild notions of crazy people.

One final note (since I really don't have time) a heurisitc (noun) is a general rule of thumb or a set of rules to guide investigation and/or problem solving, but  it may be fallible in some situations. Many 'rules of thumb'  (heuristics) have a great probability of being successful in guiding investigation or problem solving, but certainly not always. If all heuristics were always fallible, or had a high probability of fallibility then I suspect that most intelligent individuals would not rely upon them.

 But, our conversation has again strayed from the original context of the post.

Edited Feb 17,

Shrini, if you are still watching this thread I would appreciate it if you could tell me three different contexts in which ET would not be appropriate and also point me to references in the available literature or published talks by individuals whom self-identify as 'context-driven' that illustrate these weaknesses of the approach?

Friday, February 15, 2008 12:08 PM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

<i>I must say that it was the first time I had seen him speak and I his stage presence is excellent, but I wasn't overly impressed with his arguments against certification because he simply denigrated the existing programs without offering any other solutions.</i>

I'd like to thank you for the kind words, but I'd also like to suggest that there may have been more to the talk than "simply denigrating the existing programs without offering any other solutions".  You can review the notes for the talk at http://www.developsense.com/presentations/notyetcertified.pdf.

On pages 3 and 4, I give examples of ways in which I certify myself; self-certification is form of certification.  On pages 5 and 6, I note personal endorsement as another way of thinking about of certification.  On pages 14, 15, and 16, I offer several suggestions on elements of a certification program that I could believe in, with an example question or answer for each.

On page 17, I suggest that a body of knowledge for certification schemes should be built upon a descriptive, Oxford English Dictionary model (and in the talk, I spoke of the process by which this was done; refer to <i>The Meaning of Everything</i> by Simon Winchester for a very accessible history of the project).  I suggested that foundation-level certifications based on multiple choice tests should be free, to remove the perception of the conflict of interest between those who provide training and those who provide certification.  I further suggest on page 19 that we could easily assess a bevy of additional skills at the foundation level.  On page 19, I have specific proposals for skills to be examined at the advanced level, and I allude to an existing model (Cisco's) that takes that kind of approach.

With respect to your comments on context-driven testing, I think that if you looked, you would find much support in the community for the things that you claim that we denigrate.  We're not opposed to documentation, though I think you'll find that most of us are opposed to wasteful documentation.  We're not opposed to automation (especially in the broader sense of automation as "any use of tools to support testing"); we more likely to be opposed to the inefficient, wasteful, and gratuitous use of automation.  We're not single-mindedly focused on testing the GUI using hands as the input mechanism, but we often try to emphasize its value and the risk associated with <i>not</i> using human testing.  We're not opposed to coverage analysis; far from it--we're in favour of far more expansive notions of <i>test</i> coverage than that afforded by mere <i>code</i> coverage.  I'm not opposed to strengthening of certifications if those certifications are relevant and meaningful, but in the models being delivered by QAI and the ISTQB, I don't think they are.  Still, see the references to my talk above.

One more thing:  as a good empiricist, you'll notice that men have nipples, apparently useless or not.  Can you think of at least three reasons why nipples on men might exist?

Cheers,

---Michael B.

Wednesday, February 20, 2008 5:41 PM by Michael A Bolton

# re: Contextual blindness: or How to take things completely out of context

Hi Michael,

In my opinion, self certification simply does not hold credible value in many businesses and some social circles. Respect from others is also not a form of certification because some people may not necessarily respect the same people as you. So, with regards to an industry recognized certification, self selection or respect from peers simply doesn't cut it.

The Oxford dictionary is an interesting example, but I think all professions rely upon commonly understood jargon. Mutate the jargon used to convey consistent meaning within an established profession and effective communication breaks down.

I agree with some of your other examples, and apologize for not highlighting some of them. I agree tt would be nice if exams not only included ticking checkboxes on a piece of paper, but also required individuals to sit down and analyze the application under test to discover what was tested and what wasn't tested, and understand why it wasn't tested.(Finding bugs is only one component of testing, but it really doesn't tell me anything about how well something is tested.)

With respect to my comments about the context-driven school, I have read much about the ideology, and I actually agree with what is written; who couldn't agree with it, it is really mostly common sense. Personally, I am sure you know that I don't think it really makes much sense to segregate testing into 'schools' and I think my assessment is correct in that there are those who self-identify with one particular school and then there is everyone else. But, if some people want to self segregate that is fine.  Unfortunately, it seems to me that the perception of some people who like to also identify with the context-driven ideology is different than what is stated.

At the end of the day, I really don't suspect that you and I disagree as much as we agree. For example, I suspect we both agree that code coverage by itself is a poor (and misleading) measure (similarly bug count by itself is a poor measure of anything other than we found another bug), and test coverage is also important and we are discovering ways to evaluate and measure effective test coverage.

Our focuses may be different, but we both strive to enrich the skills and knowledge of testers, and I suspect to our respective audiences we are both considered successful; otherwise I doubt we would continually be invited back to conferences.

I don't consider myself an empiricist (I personally disdain all types of needless segregationist type labels), but yes, being a man, and seeing other men without shirts on, I did happen to notice that the vast majority of human men have nipples (and I have not seen or heard of an instance in which a human man (homo-sapiens) was born without nipples). There is actually only one reason why nipples exist on men. The reason why men have nipples is because it is encoded in our genetic autosomes.

Now, there might be various reasons why men derive pleasure (or not) from the fact they have nipples (which I am not at all interested in discussing), but there is really only one scientifically established reason for the existence of nipples in men (and women) of the homo-sapiens species...and the answer is genetic encoding.

Wednesday, February 20, 2008 9:22 PM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

<i>In my opinion, self certification simply does not hold credible value in many businesses and some social circles.</i>

Quite true--but it does in others, and that's why it's so useful to me, both as a positive and negative test of my potential relationship with a potential employer.  I like to belong to clubs that would have me as a member, and I'm not interested in clubs that don't want me.  If  someone is impressed by my actions, my writing,  my speech, my experiences, and so on, then we're more likely to get along.  If I find that someone isn't interested in my <i>own</i> credentials, but is instead interested in the fact that I've taken a less-than-trivial certification exam, that tells us both very useful information:  neither of us is likely to be interested in the other.  That's cool.

<i>Respect from others is also not a form of certification because some people may not necessarily respect the same people as you. So, with regards to an industry recognized certification, self selection or respect from peers simply doesn't cut it.</i>

I think you might mean "in comparison to" rather than "with regards to".  Certification is simply the granting of respect to an individual by a group--largely anonymous, from the perspective of most hiring managers--that has appointed itself worthy of granting respect.  Some people buy into that; others don't.  Some organizations that I'm aware of actively reject "certified" testers.  I think that's extreme, but I can understand why they might make that choice:  trivial certification is inconsistent with their values.  

Again, if someone doesn't respect the same people that I do, that gives me an important hint.  If we engage, maybe one or both of us will learn something, or maybe we'll simply not engage and avoid wasting each other's time.

<i>I personally disdain all types of needless segregationist type labels.</i>

Hmmm.  You frequently refer to the "professional tester".  Is that a label that someone could interpret as segregationist?

Oh... and since Shrini has apparently stopped playing, I'll give you three instances in which an exploratory approach might not be appropriate.

1) Mandate.  Client mandates against it and issues an edict.  "You shall not perform exploratory tests; you shall perform these tests and no others."

2) Sufficiency.  A very deterministic, already well-tested, and essentially closed system where inputs and outputs are highly constrained and the test space has been rigidly modelled; scripted tests have been deemed sufficient to reveal the information we need.  Components in embedded systems might provide a lot of contextual examples here.

3) Benchmarking.  When it's important for the results of the test to be entirely consistent with previous iterations of the same test, exploration is the wrong thing to do.

Here's another one for free:

4) Avoidance:  if we want to actively avoid testing or learning new information, avoiding exploration is a great way to do it.  For example, when a tester receives a request from the marketing department, "Please give us a suite of tests that have been run before lots of times and that don't crash, so that we can give them to Bill for tomorrow's trade show."

James Bach and I teach at least six more reasons to repeat tests, rather than to explore, as an exercise in our Rapid Software Testing course.   Text associated with this exercise is published on James' site:  Reasons to Repeat Tests (http://www.satisfice.com/repeatable.shtml)

---Michael B.

Wednesday, February 20, 2008 11:44 PM by Michael A Bolton

# re: Contextual blindness: or How to take things completely out of context

I am glad that you find self-certification useful and that you are successful in selling that to your employers.

In my experience, the companies I have worked for review my past accomplishments and then hire me for my demonstrable knowledge and skills, my passion for personal growth, and my potential to benefit the organization.

I think we (myself and readers of this thread) understand that you do not hold certifications in high regard. I am not sure why or what you are trying to convince me of here. I am neutral on the subject. I understand the value some businesses perceive, and I also understand the limitations.

Regarding the phrase professional tester, I suppose if someone were ignorant of the term professional, then I suppose they could misinterpret it anyway they saw fit. But, as I have said previously, I consider a professional anyone "who is expert at his or her work" or "having or showing great skill."

Notice there is no mention of some sacred doctrine to adhere to, nor do I state anything regarding a philosopy the way Shrini and others talk about "the philosophy of context driven thinking." There is no special club, and there is no special forum where moderators remove people who disagree with some pre-established principles. No secret hand-shakes, and no us versus them mentality.

I refer to professionalism as merely the state of mind of any person who strives to constantly improve his or her in-depth knowledge and broaden the diverse set of skills required to be considered an expert in a myriad of tasks used in our discipline to provide the best possible service to our organization.

Thanks for your reasons where exploratory would not be appropriate.

Mandate seems rather narrow minded of management. Personally, I have never seen this happen, but I guess it could in a shop where the only thing the organization wanted was to perform strict verification tests.

I would think that even in closed systems a professional tester should think critically about the system and question the model from different perspectives, perhaps analyze parameter interaction and not only perform combinatorial analysis of semi-coupled parameters from t=2 through t=6, but also randomize the combinatorial output for breadth of coverage.

I agree with benchmarking, except I would say the the desired results are consistent with previous iterations of the same test when ran against a new build, because of course each new build could introduce new defects even in areas of the product that have been previously tested; that's why we refer to them regression tests.

I am not sure I follow your example for avoidance. I call that setting up a demo and most presenters do this because they don't want the demo to fail unexpectedly; it wastes people's time.

But, if a tester actively avoids testing or learning in the commission of their job then I wouldn't consider that person a professional tester, would you?

Personally, I can think of other areas where a tester might consider exploratory testing inappropriate such as:

- efficient structural testing

- perfoming an analysis of untested code and either designing tests to exercise those areas or understanding why those tests may not be cost effective

- performance, stress, and load testing

- concurrency testing

On the other hand, I would suggest that debugging is highly exploratory in nature with logical deduction sprinkled in.

I am glad that you and James teach reasons to repeat tests in your course. BVT/BAT testing, regression testing, compliance testing, and other repeatable test suites are important for many reasons. See, common ground!

Thursday, February 21, 2008 3:34 AM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

>>>Shrini, if you are still watching this thread I would appreciate it if you could tell me three different contexts in which ET would not be appropriate

<MB>

Oh... and since Shrini has apparently stopped playing, I'll give you three instances in which an exploratory approach might not be appropriate.

I am back ... (was on vacation) ... BJ, Here are three contexts that *I* think ET would be inappropriate.

1. Demos - In demos, where one wants demonstrate a software application, tried and tested paths are preffered - scripted tests are good. You would not want to explore when you are demonstrating ...

2. Audits/Compliance Testing - Where the mission of testing is to show confirmance to some regulatory standard (pretty clearly known in advance). Exploration is not appropriate when you are auditing or checking if a confirmance test passes. Exploration in that case would mean "doing something" else other than the confirmance.

3. Any other forms or types of testing when you do not look for any information other than the one provided by scripted tests.

So, you have heard Michael Bolton, me ... Please please make a note ... we, in context driven school do not *sell* exploratory testing as "holy grail".

Also I would not agree with the cases you mentioned where ET is inappropriate ....

>>> - efficient structural testing

I am not sure if "efficient structural testing" exists in this world. It is mythical and ellusive. Irrespective of state of structural testing - ET would always be appropriate - if you are looking for problems, issues and bugs (any kind of new information that could come out of a sapient human testing)

>>>>- perfoming an analysis of untested code and either designing tests to exercise those areas or understanding why those tests may not be cost effective

Analysys, understanding - are highly exploratory human process. You can not have a static script for these things. These are exploratory by definition ....

>>>- performance, stress, and load testing

These too, involve great deal of exploratory elements - especially performance tuning work

>>> - concurrency testing

Setting up concurrency conditions, observing results, learning from the results and re-doing things - see exploration at every point. A scripted concurrency testing would be pretty narrow and limited utility as one can not (even a professional tester) can not predict some interesting outcomes and develop a good scripted concurrency test ...

Shrini

Tuesday, February 26, 2008 7:51 AM by Shrini

# re: Contextual blindness: or How to take things completely out of context

Hi Shrini,

Thanks for reiterating the demo point Michael made above. I am not sure why staged demos would be considered testing in anyone's mind; perhaps that is why they call them demos instead of testing?  

The audit/compliance makes perfect sense, and I agree, but see the point below because I think tester's 'explore' the requirement to understand what to audit or how to audit something?

I find it quite interesting that you only list problems, issues and bugs as the "kind of new information that would come out of a sapient human testing." A professional tester who is truly wise and uses sound judgement would of course realize that defects, problems and issues are only a fraction of the valuable information testing should provide to the organization to reduce exposure to risk.

So based on the above, I would suspect that you would consider structural testing mythical and ellusive, especially if the only information one can provide is bug count, time, and 'feel-good' observations.

In essence (according to your definition), everything which provides new information is exploratory. I guess we could say that even scripted tests are exploratory because they are derived from a 'sapient' person 'exploring' the requirements and designing tests!

Unless of course, you do not consider that a person who applies critical thinking and use of their cognitive skills to analyze (since you state analysis is highly exploratory) the requirements and use sound judgement (sapience) to design a set of both positive and negative tests (scripted tests) from the requirements to be exploratory testing.

Tuesday, February 26, 2008 12:18 PM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

>>> I am not sure why staged demos would be considered testing in anyone's mind; perhaps that is why they call them demos instead of testing?

Doing demos (or showing) is not testing (exploratory or otherwise). You could be demo'ing a software to your family, or business group or to a customer.

Before one could demo a peice of software, it has to be configured and operated (dry run) or rehearsal. That part of demo cycle where you prepare the software, setup the data and scenarios, do a dry run (few scenarios) is what I am referring as "Non explopratory".

Depending upon the length of the demo, it could involve a great amount of scripted testing to make sure that specified scripts "pass" and demo will be successful.

>>> but see the point below because I think tester's 'explore' the requirement to understand what to audit or how to audit something?

That portion of actual execution of the audit or regulatory confirmation that involves execution of specific scripts (created via some sort of exploration) is "Non exploratory".

You know *exactly* what to do (execute) and what exactly to expect (results). I believe most of the audits and compliance tests happen as per pre-defined scripts - no room for exploration.

>>>I guess we could say that even scripted tests are exploratory because they are derived from a 'sapient' person 'exploring' the requirements and designing tests!

Right. But I would make a distiction between "design" and "execution". Design of scripted tests is often "exploratory" (some engage in mechanial ways of "deriving" test cases from specification)

Execution of scripted is Non exploratory. You will do exactly as mentioned in script and *observe* only what test tells you to observe. Nothing more or nothing less (deviating from script is considered as indescipline according to QA/Process department)

>>> you do not consider that a person who applies critical thinking and use of their cognitive skills to analyze (since you state analysis is highly exploratory) the requirements and use sound judgement (sapience) to design a set of both positive and negative tests (scripted tests) from the requirements to be exploratory testing.

Did I say that ...? No. Design scripted tests is exploration. Execution of scripted tests with sole intention of checking if those tests pass is not.

If you seperate  the creation and execution activities, you might see the difference. IT is important to note that often, since the person who designed tests is not the one who will be executing them - line of seperation between exploratory and scripted testing is clear.

Shrini

Wednesday, February 27, 2008 12:14 AM by Shrini

# re: Contextual blindness: or How to take things completely out of context

Shrini,

Thank you for bringing the conversation back to my original point about taking things completely out of context.

I do understand the difference between design and execution, and I only spoke about designing tests and I did not even refer to or infer execution of tests in my response. So, your arguments about design vs. execution are tangential.

Also, my statement was "Unless of course, you do not consider..." not simply "you do not consider..."; which changes the context of my statement.

Wednesday, February 27, 2008 12:57 AM by I.M.Testy

# re: Contextual blindness: or How to take things completely out of context

That's quite an out-of-the-box way of looking at things I.M Testy. This begs the question, are we talking about contextual "blindness" or just a sad case of laziness or apathy?

Wednesday, March 05, 2008 4:44 AM by puretesting

# re: Contextual blindness: or How to take things completely out of context

I am not sure what to call it. I don't really suspect it was laziness, apathy, or even ignorance. I think some people have a habit of avoiding facts or taking things completely out of context to further some personal agenda.

Friday, March 07, 2008 1:03 PM by I.M.Testy
Anonymous comments are disabled
 
Page view tracker