Welcome to MSDN Blogs Sign in | Join | Help

The purpose of software testing

For the past few years I have been teaching new testers at Microsoft. Before beginning the Microsoft ‘assimilation’ process I always ask each new group to define testing in one sentence and list the primary objectives of software testing. Invariably, the majority of folks new to software testing believe the purpose of software testing is to ensure quality and the primary objective is to find bugs. Unfortunately, this is an antiquated view of testing, and perpetuating these limited perspectives of testing simply retards the role of testing from maturing into a professional discipline in the field of computer science.

 

Testing doesn’t ensure quality

 The purpose of software testing is to assess or evaluate the capabilities or attributes of a software program’s ability to adequately meet the applicable standards and customer needs. So, when I hear a tester say the purpose of testing is to ensure quality I immediately ask them, “how can testing ensure quality?” Think about this rhyme for a moment:

 

I don’t get to drive the train,

I don’t get to ring the bell,

But, if the train jumps the tracks

See who catches hell!

 

I don’t get to discuss the needs of the customer, I don’t get to translate those needs into requirements, I don’t get to design or code the solution, and when I find a bug in the software I don’t even get to fix it. I can’t even demand that every bug I find gets fixed. In fact, I know that a number of reported bugs (including ones I think should get fixed) simply won’t get fixed. So, how in the hell can I ensure quality?

 

What I can do as a professional tester is to design the appropriate tests to evaluate the attributes and capabilities of a software program’s ability to handle valid and invalid inputs under various conditions. This approach not only has the potential to demonstrate a program’s ability to meet customer needs and expectations, but also includes identification of flaws or deficiencies that could negatively impact the customer (or long term maintenance of the product).

 

The purpose of testing is not to find bugs

 The primary objectives of testing include:

  • Qualifying a software program’s quality by measuring it’s attributes and capabilities against expectations and applicable standards
  • And (probably the most important role of testing is simply to) provide information

 

Although I vehemently disagree the primary objective of software testing is to find bugs, I am not surprised when new testers state bug-finding as a primary purpose of testing because it is so heavily emphasized in many books and proselytized by so-called ‘experts’ in the field. Even Cem Kaner states in his “best selling” book on software testing, “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” Bugs are simply a by-product of testing.

 

So riddle me this, if testers are spending all day beating on the product in attempts to find bugs, who in the hell is verifying the product does what it is supposed to do? I guess performance testing is a waste of time if the performance measurements never fall below acceptable levels. But wait, how would we know if we didn’t “waste the time” to measure performance to begin with? I guess we would have to waste time to actually measure the capabilities against the expectations or applicable standards, or say…”yep, in my ‘expert’ (biased and subjective opinion) it’s good enough!

 

My teammate Alan also recently wrote an excellent post entitled Testing != bugs, where he talks about pushing quality upstream through defect prevention. This is a common theme we discuss in our training for new software testers. We advocate the expansion of the testing role beyond bug-finding. We encourage testers to explore opportunities, seek out solutions to difficult challenges, and unlock their potential to become a greater asset to the organization throughout the entire software process. Fortunately, other companies besides Microsoft are beginning to understand that a highly trained and competent test team can provide much greater value to the organization beyond the simple finding bug job.

 

But, personally, I think even many experienced testers have yet to realize their full potential or the significant impact the testing role can provide beyond finding bugs. But, I do think within the next few years we will see a transition in the industry away from hiring a plethora of “expert bug-finders” towards the maturation of a respected discipline of professional software testers.

Published Thursday, July 13, 2006 6:02 AM by I.M.Testy

Comments

# re: The purpose of software testing

After several years out of testing, I thought I'd take the opportunity to go back and read Kaner, et al's testing textbook.  The first chapter gives 100% contrary advice to what you are saying.  Testing is about finding bugs.  Tests that don't find bugs are wastes of time.  Etc.

It's amazing that this book is so popular, though I think that its style really resonates with beginning testers.  It tells the neophytes that their job is to break things.  It's much easier and fun to go around with a hammer smashing things.  It's not as much fun to become a cog in a large "professional team" where the primary purpose is to provide documentation about the performance of the SUT.

[Bj] I agree the Kaner, et. el. book resonates well with neophytes because it is an easy read and does not deep dive into technical issues which is unfortunate because software testing is a technical trade. But, I don't think that tester's who design good quality tests, or tester's who focus on defect prevention simply become another cog in the wheel of a large team. In fact, senior professional testers who possess in-depth "system" knowledge can have much greater influence on the development team (teaching them how to write better unit tests), the project managers (evaluating initial product designs), and the managers (factual quality measures and risk analysis information).

Hopefully you're right.  QA has a reputation as a holding area and training grounds before making the jump to dev rather than as a discipline unto itself.  This is one bad rap that I think that the profession would do well to shake off.

Thursday, July 13, 2006 1:40 AM by Lauren Smith

# re: The purpose of software testing

How do you define a bug?
You only know something is a bug by "measuring it’s attributes and capabilities against expectations and applicable standards". "Bug" is just a short way of saying it.

Ian

[Bj] Actually, I particularly don't like using the term bug. I much prefer defect, error, or fault. Defects or errors are usually faults of commission, or problems resulting from poorly crafted code. There are also faults of omission resulting from missing requirements in vague or incomplete documentation which translates the customers needs into a functional specification.

One of the greatest values a tester brings to the game is his or her ability to design tests to verify the proper functionality of software, and to also design tests to validate the software meets the customer's expectations. In some cases, those expectations may not be well defined, and professional testers can also look beyond the "documentation" in their pursuit of evaluating the capabilities and attributes of a product.

Friday, July 14, 2006 2:09 PM by eebster

# The purpose of a good test case

Many experts have written articles and devoted chapters of books on the attributes of what constitutes...
Friday, September 15, 2006 6:03 PM by I. M. Testy

# re: The purpose of software testing

Hi BJ,

It was good to catch up at STARwest.

To quote from my testing spot, "In September 1997, in an online discussion on the purpose of testing, Boris Beizer said that software was tested 1] to find bugs and 2] to check quality. David Gelperin said a higher priority purpose (when achievable) was 3] to prevent bugs being born. None of these areas need be the exclusive domain of software testers, particularly the last 2. ".  I think I wrote that back in 2002.   I spoke about some of this in my Back to the Beginning talk at STARwest last week, as Myers has an axiom that says a good test is one that finds bugs.

My current view, which I think aligns with Cem's current view and yours, is that the key purpose of testing is to provide information on the current state of software quality, and the relevant qualifications on that (what have we tested and how, and by implication what we haven't tested, and ways we havent tested it).

And I agree we need to broaden this to include the communication of a quality assessment.  I also think we need to go down a level.  There are two views of this quality (a ying and yang) which complement each other - risk and confidence.  We want confidence in the parts that we need to work (e.g the most important parts for our users), but at the same time we need to understand the risks in things not working (e.g through discovering bugs).  The other important part of this is coverage, we need to understand this across our application.  At the same time we need to focus on key areas, the sweet spots where users spend most of their time, and the sweet spots where most of the defects are located.

While we need to have confidence that the software works, if we have open high severity bugs our testing cannot be considered finished. So in a very brutal software environment, we could say that we can replace not-quite-there functionality with manual workarounds (to get past our confidence gates), but we typically cannot work around high severity defects (to get past our risk gates).

While any sort of upstream activity is great at improving the delivered quality (and avoiding bugs), the natural focus on bug hunting (particularly on new software development) provides sanity checks on the size of the testing task.  If we bug hunt for half a day and find some high severity bugs, we can extrapolate this against the allotted time and see if it is reasonable.  Typically estimates on testing for confidence are much easier to create than testing for risk.  We need feedback from the process to qualify our risk analysis.  Our defect detection replaces (or reaffirms) our risk analysis, and should also help us find our bug clusters to focus on, particularly any high severity bugs.

So while bug hunting may not be a primary objective of testing in all cases, it is more effective in providing information about what is being tested.  As a metaphor, we want to get a team of people to swim across a harbour.  We have to have the confidence that they can all swim the distance (and to ensure coverage they all need to make it), but we also need to look at the risk of shark or jellyfish attack! It is pointless to look at just risk or just confidence, we need to consider both.

cheers,

Erik

Melbourne, Australia

Tuesday, October 24, 2006 11:20 PM by ecp

# re: The purpose of software testing

Hi Erik,

Yes, it was my pleasure to meet you at StarWest and I enjoyed your talk.

Everyone in the product team is responsible for producing the highest 'quality' deliverable for which they are responsible. I have never worked on a project where the outcome was in the exclusive domain of a single group or person.

"Checking quality" (I would restate this as (evaluating the capabilites and attributes of software against applicable guidelines)and defect prevention are becoming more within the domain of highly skilled testers, and perhaps rightly so because a well-trained, knowledgeable tester understands testing much better than other people on the team and can help them expose potential defects earlier.

For example, I can't think of a single reason why 99% of boundary class defects cannot be detected at the unit testing level. But, we know that many boundary issues are still found at the integration or system level. (I will probably get flamed for this but...) this class of defects still escapes unit testing because we know that most developers have very little formal education in testing, and thus their unit tests are marginal. (I will also say that this is changing, and with Agile and extreme programming approahes I am meeting more and more developers who are interested in testing techniques and methods.) As a professional tester who understands my discipline more than my developer counterparts can I best help my team (and the company's bottom line) by partnering with my developers and teaching them how to write better unit tests that will detect certain classes of defects earlier.

I have a slightly different take on confidence and risk. When a test passes then I have increased confidence the capability or attribute adheres to (or exceeds) the applicable guidelines or standards against which it is being compared. If the test fails then my confidence level is low (actually non-existent). But, as testers we provided the information that a test has failed, and management can analyze it and make a conscious business decision whether to correct the defect or not. In this case, if the defect is not corrected, my confidence in the overall success of the project is deminished.

However, risk not only involves failed tests, but more importantly areas that are unknown. We cannot test everything and what isn't tested is pure risk. For example, we can't test every combination of complex dependent parameters; however, we can mitigate risk by using techniques such as combinatorial analysis to significantly and effectively reduce our risk by n%. However, if 30% of the code is untested, then that 30% of code is at 100% risk!

I agree the majority of users actually use very little of an applications available features, and this is why past approaches to testing (banging away on the keyboard to find bugs) and the 'good-enough' quality mindset has been relatively successful. However, Microsoft and other companies have realized that our biggest weakness is not necessarily in the 'sweet spots' or areas of greatest use by the majority of customers, but in areas that are under-tested or untested.

I don't buy into the assertion that 80% of the defects are in 20% of the code. In fact, I think it is a dangerous assumption and a fallacious use of Pareto principle. We know that developers tend to make similar mistakes over and over, thus the theory of equitable distribution of defects is also quite popular. (The truth is probably somewhere inbetween.)

I do agree with the stated Pareto principle that "80% of the consequences stem from 20% of the causes." In other words, a single defect can manifest itself in many ways. Thus what may appear to be several 'bugs' are simply different symptoms or consequences of a single defect.

While we may differ slightly on terminology (and perhaps defect distribution theories), I think we both agree (and as I have said in a previous post) that exposing defects and errors is one of the explicit purposes of a test. I don't encompass 'bug finding' as a primary objective of testing becuase I think defects are artifacts or results of effective testing. The defects exposed by tests are testing artifacts, or results of the testing process and are used to qualify the products quality and provides information to the product stakeholders.

My contention is that as the testing discipline matures we must expand the role of the discipline in various ways and move beyond the misconception that software testing is primarily bug hunting. The future of software testing as a credible and respected profession is only limited by the inability to see beyond the responsibilities and roles typically relegated to (mostly) non-technical 'business expert' type testers. 

Wednesday, October 25, 2006 1:36 PM by I.M.Testy

# Software Information » I. M. Testy : The purpose of software testing

PingBack from http://softwareinformation.247blogging.info/i-m-testy-the-purpose-of-software-testing/

Sunday, January 20, 2008 6:43 PM by Software Information » I. M. Testy : The purpose of software testing

# re: The purpose of software testing

'i.m.testy' i agree with you that purpose of testing as a whole is not just bug hunting...

to me bug hunting (finding) is just a (inherent)part of software testing when i am talking about a software testing process... because... if v have software testing process in place defined on the guidelines of any testing model like TMM, at level 2(basic maturity level of software testing process) we incorporate areas like test policy , test planning and testing techniques + methods and testing environment  ...

after saying this i doesnt mean that we are doing so many things just to find the bugs.... rather emphasis is on objective that software satisfies the specified requirements which is first step to reach highest QC level(near to zero defect)...

Friday, May 02, 2008 7:03 AM by Mr. I A N

# re: Testing doesn’t ensure quality <HOW it could b??>

'i.m.test' you said that """ The purpose of software testing is to assess or evaluate the capabilities or attributes of a software program’s ability to adequately meet the applicable standards and customer needs"""

when u say this u are actually negating your own statement " Testing doesn’t ensure quality  "

...

any step you are taking to ensure that customer needs+expectation are met , u are actually ensuring the customer satisfaction... and customer satisfaction is linked with the quality attribute of any product or service...

its similar to a mathematical rule of equality that if x=y and y=z then x=z.

No doubt simple testing (part of software testing process) plays some part of ensuring the quality since its a part of Quality Control area ... where as Quality Control + Quality Assurance covers major part of Total Quality Management ... which on the basis of continual improvement ensure the customer satisfaction..

in short what i am trying to say is that...

< if somebody says that when we do software testing we are not ensuring the quality rather simply collecting information, infact he is ensuring quality because

like implementing best practices in other area of SDLC we can ensure quality of product and service similarly by implementing best practices of software testing we no doubt impart quality attribute in product or service and ensure the customer satisfaction...

this is what TQM says...

Friday, May 02, 2008 8:19 AM by Mr. I A N

# re: The purpose of software testing

Hi Mr. IAN,

Thank you for you feedback. We are in absolute agreement that detecting anomolies in software is one possible result of executing tests.

But, I would also say that testing conformance to requirements is a part of the whole testing picture, but too much emphasis on 'specific requirements' could cause us to overlook other important tests. The specifications are often vague and ambiguous.

I don't think my statements are contradictory. By evaluating capabilities and attributes the information of those tests may influence changes in the functional or behavioral aspects of the product that may impact engineering quality or customer quality or value.

But, there are not too many testers who can acually ensure (def. - to secure or guarantee, to make certain) quality. For example, I can't make a developer fix all defects, as a test manager I generally don't have the ability by myself to stop the release of a product from releasing even if it fails to meet pre-determined quality objectives.

In my opinion, customer satisifaction is primarily achieved by those who do the marketing analysis and best understand the customer needs, and who propose a product design the business decision makers agree will appeal to customers and satify their needs.

- Bj -

Friday, May 02, 2008 11:38 PM by I.M.Testy

# re: The purpose of software testing

hi i.m.testy

its nice to read ur good comments....

No doubt testing conformance to requirement is a part of the whole testing picture ...

other activities of Software testing process are helping to minimize the cause of any NC to req.. + help manage the Testing part of SDLC...

imagine if we have processes to manage different parts/phases of SDLC in more effective way dont u think we will be able to fulfill the requirements of users with confidence.. this is quality ...

yes.. i object on this statement that "Testing doesn’t ensure quality"" .. some part of reason  lies in above written lines :)

secondly, i would like to refer ISO 9001-2000 in this regard... in clause 7 (product realization) ISO demand for VERIFICATION (i.e. specifically 7.3.5 -Design & dev verification and 7.3.6- design & dev validation...) similarly CMMI have one process area at ML3 - Verification and Validation... one can say that... in short verification and Validation (testing) is important for quality and it ensures the quality attribute of any product and service....

lastly... i agree with u that mkt analyst, product designer, decision maker , product maker (developer) play their vital role in achievement of customer satisfaction (if they are following the process approach)...and u cannot eliminate/neglect testers from this chain of quality producers if they are performing their activities as per any well defined testing process..

because.. effective system approach (in any org) without critical testing process is incomplete ... as result Customer satisfaction will b incomplete / uncertain...

its similar to say that Quality of product/service is not only responsibility of QA department rather every one is and has to play his or her role in producing quality product.

Monday, May 05, 2008 5:17 AM by Mr. I A N

# re: The purpose of software testing

Quality is perhaps the most misused word in the English language (with the exception of perhaps one other word which has come to be used as a noun, adverb, verb, and interjection).

Yes, I agree that everyone in any company the manufacturers some thing does their part and has (or should have) a vested interested in wanting the company to succeed.

I am not discounting what ISO and CMMI suggest. In my opinion, functional or engineering 'quality' attributes or capabilities of a thing are not directly synonymous with customer satisfaction, but are indirectly related.

Additionally, during the SDLC testers do not usually measure customer satisfaction (nor should they), and customer satisfaction is a subjective measure at best and is only realized after the product is released.

I think we essentially agree that testing is necessary and critical to the success of any software project

Monday, May 05, 2008 4:04 PM by I.M.Testy

# I M Testy The purpose of software testing | Paid Surveys

PingBack from http://paidsurveyshub.info/story.php?title=i-m-testy-the-purpose-of-software-testing

Friday, May 29, 2009 1:02 PM by I M Testy The purpose of software testing | Paid Surveys

# I M Testy The purpose of software testing | debt solutions

PingBack from http://debtsolutionsnow.info/story.php?id=11728

Friday, June 19, 2009 1:23 PM by I M Testy The purpose of software testing | debt solutions
Anonymous comments are disabled
 
Page view tracker