I. M. Testy : Test Management

Measuring Test Automation ROI

I.M.Testy — Tue, 25 Aug 2009 09:40:00 GMT

I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent points such as:

Automated software testing “is software development.”
Automated software testing “and manual testing are intertwined and complement each other.”
And, “The overall objective of AST (automated software testing) is to design, develop, and deliver an automated test and retest capability that increases testing efficiencies.”

Of course, I was also pleased to read the section on test data generation since I design and develop test data generation tools as a hobby. The authors correctly note that random test data increases flexibility, improve functional testing, and reduce limited in scope and error prone manually produced test data.

There is also a chapter on presenting the business case for an automation project by calculating a return on investment (ROI) measure via various worksheets. I have 2 essential problems with ROI calculations within the context of test automation. First, if the business manager doesn’t understand the value of automation within a complex software project (especially one which will have multiple iterations) they should read a book on managing software development projects. I really think most managers understand that test automation would benefit their business (in most cases). I suspect many managers have experienced less than successful automation projects but don’t understand how to establish a more successful automation effort. I also suspect really bright business managers are not overly impressed with magic beans.

Magic beans pimped by a zealous huckster are the second essential problem with automation ROI calculations. Let’s be honest, the numbers produced by these worksheets or other automation ROI calculators are simply magic beans. Now, why do I make this statement? Because the numbers that are plugged into the calculators or worksheets are ROMA data. I mean really, how many of us can realistically predict the number of atomic tests for any complex project? Also, do all tests take the same amount of time, or will all tests be executed the same number of iterations? Does it take the same amount of time to develop all automated tests, and how does one go about predicting a realistic time for all automated tests to run? And of course, how many of those tests will be automated? (Actually, that answer is easy….the number of automated tests should be 100% of the tests that should be automated.)

Personally, I think test managers should not waste their time trying to convince their business manager of the value of a test automation project; especially with magic beans produced from ROMA data. Instead test managers should start helping their team members think about ROI at the test level itself. In other words, teach your team how to make smart decisions about what tests to automate and what tests should not be automated because they can be more effectively tested via other approaches.

In my next post I will outline some factors that testers, and test managers can use to help decide which tests you might consider automating. Basically, the bottom line here is that an automated test should provide significant value to the tester and the organization, and should help free up the testers time in order to increase the breadth and/or scope of testing.

Assessing Tester Performance

I.M.Testy — Tue, 28 Apr 2009 19:59:00 GMT

Using context-free software product measures as personal performance indicators (KPI) is about as silly as pet rocks!

Periodically a discussion of assessing tester performance surfaces on various discussion groups. Some people offer advice such as counting bugs (or some derivation thereof), number of tests written in x amount of time, number of tests executed, % of automated tests compared to manual tests, and (my one of my least favorite measures of individual performance) % of code coverage.

The problem with all these measures is they lack context, and tend to ignore dependent variables. It is also highly likely that an astute tester can easily game the system and potentially cause detrimental problems. For example, if my manager considered one measure my performance on the number of bugs found per week, I would ask how many I had to find per week to satisfy the 'expected' criteria. Then each week I would report 2 or 3 more bugs than the 'expected' or 'average' number (in order to 'exceed' expectations), and any additional bugs I found that week, I would sit on and hold in case I was below my quota the following week. Of course, this means that bug reports are being artificially delayed which may negatively impact the overall product schedule.

The issue at hand is this bizarre desire by some simple-minded people who want an easy solution to a difficult problem. But, there is no simple formula for measuring the performance of an individual. Individual performance assessments are often somewhat subjective, and influenced by external factors identified through Human Performance Technology (HPT) research such as motivation, tools, inherent ability, processes, and even the physical environment.

A common problem I often see is unrealistic goals such as "Find the majority of bugs in my feature area." (How do we know what the majority is? What if the majority doesn't include the most important issues? etc.) Another problem I commonly see is for individuals to over-promise and under-deliver relative to their capabilities. I also see managers who dictate the same identical set of performance goals to all individuals. While there may be a few common goals, as a manager I would want to tap into the potential strengths of each individual on my team. I also have different expectations and levels of contributions from individuals depending on where they are in their career, and also based on their career aspirations.

So, as testers we must learn to establish SMART goals with our managers that include:

goals that align with my manager's goals
goals that align with the immediate goals of the product team or company
and stretch goals that illustrate continued growth and personal improvement relative to the team, group, or company goals

(This last one may be controversial; however, we shouldn't be surprised to know individual performance is never constant in relation to your peer group. )

But, (fair or not) for a variety of reasons most software companies do (at least periodically) evaluate their employee performance in some manner, the key to success is in HPT and agreeing on SMARTer goals upfront.

The quality quandary

I.M.Testy — Fri, 27 Mar 2009 18:27:00 GMT

I often find discussions about quality to be hypothetical, and in fact unless you define your specific context the word itself is nebulous, vague, or simply meaningless philosophical psycho-babble. For a while now, I previously posted my opposition to the simplistic notion that quality is value to some person. Sure, most thesaurus' equate the two words as synonyms, and the context-driven posse constantly regurgitate Weinberg's quote about "quality is value to some person." Yes, that is one perspective of quality, but only one.

In the past I have taught that quality is not value in the context as one of the goals of software testing. My definition of value is the purpose or usefulness of a product in satisfying a customer's needs or wants. My definition of quality was based on tangible aspects of the attributes or capabilities of a product from an engineering perspective. Our definition of software testing as any task designed to evaluate or assess the attributes and capabilities of a software project in relation to implicit or explicit guidelines in order to provide information to the management team (the people who make the business decisions). Those evaluations result in measurements we call quality measurements or criteria (the "essential or distinctive characteristics, properties, or attributes" that are critical for the success of our project) and that is part of the information we present to the decision makers to help them make more informed business decisions. Remember, Weinberg also stated "Thinking about measurability from the beginning is an essential part of creating a well-formed effort."

The other day I met with members of one of our research teams and they were talking about quality in terms of tenet and non-tenet quality; and I clarified that basically they were essentially talking about the customer perceptions of quality versus the engineering aspects of quality. Surely, from a holistic point the observation and reliable measurements of both perspectives of quality are important for the success of any organization. Customers usually buy/download software because it helps them satisfy some need or want. The decision of which software product to buy made be based on personal bias, or perhaps the herding instinct, or it may be more rational based on the comparison of features (measureable attributes and capabilities). Once the customer begins to use the software they form their own opinions based on expectations and/or previous experiences that provides the company with information regarding customer satisfaction.

So, the next time someone starts talking about quality stop them and ask them if they are talking about the engineering aspects of quality, or the customer perceptions of quality. They are closely related, but different perspectives of the same topic.

Thoughts on leadership

I.M.Testy — Fri, 24 Oct 2008 12:27:00 GMT

Last week I was at the Test2008 conference in India. The organizers from PureTesting planned a grand event with workshops in Hyderabad, Delhi, Bangalore, and Pune. Then the main conference was then held outside of New Delhi. When I arrived in Delhi at the conference I was told I would be on a discussion panel. Surprise!

Although the conference organizers thought the topic would be controversial, in retrospect it turned out to be a non-issue to the majority of the audience. But, during the discussion one person asked the most important question of the session. He essentially said that new people coming into the industry and specifically the testing discipline are sometimes confused because there is sometimes contradictory information. "So," he asked, "how do new people know who the leaders in testing are?"

Rather than drone on forever, here is a list of traits of leaders whom I respect, and attributes I try to follow when I lead a team or mentor people.

Leaders are able to foresee technological changes and changes in business practices on the horizon and predict how those changes will influence the careers of the people they manage or mentor.
Leaders don't let the people they are managing or mentoring to become stagnant.
Leaders constantly seek opportunities for the people the manage or mentor to flourish.
Leaders constantly help the people they manage or mentor develop their careers even if that means moving to a different role or team.
Leaders connect with the people they manage or mentor and develop a nurturing bond.
Leaders delegate responsibility because they explicitly trust in the people they manage or mentor to do the best they can do. Similarly, people respect leaders who they know grew to become a leader and were not merely placed in some position of management.
Leaders understand the challenges in the industry and they unleash the potential of the people they manage or mentor to take on and tackle those challenges. If the team fails the leaders accept the responsibility and support their people for giving it their best effort. Then, they rethink the problem, and try again.
Leaders don't say that something can't be done, or you can't do such and such; they continuously search for alternative solutions to problems.
Leaders identify hard problems and point the people they manage or mentor in the right direction and say, let's figure out how to solve this together as a team!
Leaders don't whine about changes in the industry. We work in one of the most dynamic industries in the world, and leaders can successfully lead their teams to face new challenges head on.
Leaders don't shamelessly ridicule other people or hurl personal insults.
Leaders challenge the ideas and statements of others, but they do so in a professional manner.
Leaders present compelling points of view based on rational logic and empirical analysis. Not everyone may agree with a point of view, but they comprehend the results, and may sometimes present conflicting data which is repeatable in unbiased studies.
Leaders must also occasionally make hard decisions that may be unpopular with the people they manage or mentor or even their own managers.
Leaders don't attempt to segregate the discipline or mislead neophytes with reckless statements based on emotional or philosophical ideals.
Leaders have a strong personal constitution and are not swayed by emotional opinions or baseless peer-pressure.
Leaders not only strive to improve the people around them, but they also continually strive to be the best they can be.
Leaders never become apathetic or dispassionate. (If a person is apathetic or dispassionate then it is way past time for them to leave and pursue other directions.)
Leaders are often recognized as technical experts in their fields.
Leaders are respected by other leaders in their field.
Leaders don't refute challenges to ideas or statements with hypothetical or philosophical multi-syllabic hyperbole, they present substantiated facts or logical and rational points of view within the context of the discussion.
Leaders know to criticize in private and promote in public.
Leaders are also competent contributors.
Leaders know the difference between 'big-bang' one-time dog-and-pony shows, and achievements that provide lasting results, and they reward accordingly.
Leaders figure out how to permanently fix small problems so they can tackle larger and larger issues.
Leaders drive themselves and others around them to be the best they can be because they know that being good enough is simply not good enough in the long run.
Most importantly, leaders provide strategic direction and help guide and grow the people they manage or mentor to face new and exciting challenges.

I rattled off some of these traits and in the end I told the person that I have come across a lot managers in this industry (people who have people reporting to them in some capacity), but in my opinion there are too few real leaders. Fortunately I personally see many new highly knowledgeable and technically skilled people coming into the discipline again along with many experienced people who are reemerging. These people represent the potential for the type of leaders we need to help drive the discipline forward. Are you one of those people?

Thoughts on Professionalism

I.M.Testy — Wed, 08 Oct 2008 21:36:00 GMT

As a young lad growing up on the shores of the Chesapeake Bay I would often spend part of my summer vacation from grade school helping my grandfather work the crab pots on the north shore. Now, don't think "Dangerous Catch," crabbing in the Chesapeake is much different than crabbing in Alaskan waters, although we did have our share of squalls and nor'easters that will humble any man. Crabbing is hard work, especially for a young boy. On one particular day working the pots with my grandfather I recall something he said which has stuck with me my entire life. The water was rough that day and we were being tossed around a bit. It was raining...not like the wimpy Seattle rain, I mean hard East coast rain that comes down in sheets with drops so large they hurt when they hit you. Even in the driving rain the smell of dead fish, rotten chicken pieces and turkey necks that we used for bait permeated everything, the slime of fish guts coated the decks making footing somewhat precarious, and the brine of the Chesapeake seeped into every little cut and nick on my hands. I wasn't having a lot of fun that day, and although my grandfather knew it he didn't say anything to me until I complained aloud. Now my grandfather was a tall, barrel-chested man with a stern face, and he never put up with any senseless, petty whining. But, instead of scolding me he simply said, "Son, we are not rich and there is no inheritance coming your way, so you better find a job you like doing." At first, I was a bit puzzled. What did that have to do with how I was feeling at the moment? But, I soon realized what he meant, and my grandfather's sage advice has stuck with me ever since and guided me in my career pursuits.

My father, who was also every bit as practical and pragmatic as my grandfather taught me the value of working hard, and the virtues of responsibility. He taught me to take pride in the things I did well, and take responsibility for things that didn't work out so well. My father also taught me never to quit, to always look at alternative options, and that the path of least resistance or the easy path through life is not always the best path to take. He also said if something was worth doing, do the very best you can do, and never sit back and rest on my laurels. My father was not the type of man to do something half-way, or 'good-enough.' One summer we built a new barn on our property. This wasn't the typical barn, instead my father got some telephone poles and even got his friend at the telephone company to bring out a truck with an auger to drill the holes 10 feet deep! Next we used oak planks between the stalls. Now hammering nails through oak is not easy business and the entire project took about 2 weeks. But, my father was adamant about not having to rebuild the walls of the stall if a horse tried to kick it down.

Practicality in practice

Now perhaps it was my upbringing, or perhaps it is my Myers-Briggs ISTP personality traits, but it is hard for me to understand how some people embark on a career, or pursue a job opportunity without knowing (or perhaps worse..completely ignoring the fact) that they will be challenged to continually improve their knowledge and skills everyday. This is especially true of many of our chosen profession of software testing. We work in a highly dynamic industry and we must continue to learn and develop new skills to meet the growing diversity and increasingly complex challenges that will potential advance the discipline as well as provide benefit to our employers.

Sure, we can whine and complain about the changes. We can point fingers, and otherwise express negative emotions and utter baseless, counter-productive propaganda slogans such as, "automation can't do such and such," or "I know non-coding testers who find more bugs than (fill in the blank here)" or the virtually inverse argument "tester's who write code are biased and don't know how to think like an end-user." I learned a long time ago that although misery loves company, it is usually not those who whine the most or put forth pointless and petty gripes, or who only see the world through bi-polar, black and white lenses who drive meaningful change. Sure, it is easy to empathize with some people in some situations; but empathy without practical solutions to resolve the situation effectively is simply a pathetic play of the irresponsible victim card.

For example, we unfortunately often get mired down in a this vs. that, exploratory vs. scripted, or STE vs SDET debate that is generally too myopic (on my project… blah, blah, blah) and often counter-productive (meaningless comparisons, ridiculous measures such as raw bug count, and baseless assertions that lack empirical evidence to substantiate the claim, etc). There seems to be an opinion that if someone writes automation, or comes with a coding background then they are not good ‘exploratory’ or behavioral, or (and I hate using the phrase) “black-box” testers. (All testers are “black-box testers!). Yet, some of the most talented testers I know in the industry began with very little in-depth knowledge of the ‘system’ or an understanding of programming concepts and applied themselves to grow their knowledge and skills about the overall system and technologies to advance their careers, their teams, and the industry by taking on greater challenges. I also know highly talented testers with great industry experience and very strong coding backgrounds who are masters at explorative, behavioral, or “black-box” testing.

So, for those of you who have been living in a cave, or have your head permanently implanted in the posterior end of your alimentary canal it is obvious the industry and the profession of software testing is changing and there is an increasing demand for greater system knowledge, experience, and technical skills for many testing positions in the industry. There are many reasons for these changes, and the simple fact is that the industry’s needs and strategic direction primarily dictate what skills and knowledge are required to remain competitive in the industry and advance the business.

Are ya' gonna lay there and bleed, or ya' gonna' cowboy up!?

Now, many people mistakenly assume that I only promote technical knowledge or advancement though increased technical skills. (Of course, many of these people are so narrow minded and biased they only equate 'technical knowledge and skill with 'coding' skills and programming knowledge.) I do talk about these subjects a lot because...well, they are directly related to our professional growth, and because I studied and interviewed thousands of testers to perform a skills gap analysis (delta between the disciplinary needs of the business in relation to the actual skills and knowledge of the existing workforce at the time). The skills gap analysis indicated that many people in the testing industry at the time were well-educated, intelligent people who were very capable of thinking critically about problems within the appropriate context. But, it also illustrated that a significant population in the testing profession at that time lacked in-depth knowledge of the system they were working and the technical skills to expand their own productivity or become more influential in driving product quality. So, as a manager and mentor it was (and is) my responsibility to help people understand changes based on the maturation of the industry and the profession, and to help them grow professionally, and that is the role I took on with great passion. I decided to focus on growing people's technical skills and knowledge, and their understanding of established, and contextually effective software testing processes.

One of my first public talks revolved around the idea that good-enough was simply not good enough any longer based on changing customer views and increasing maintenance costs. That was in 2003! In 2005, I presented empirical evidence suggesting that exploratory testing by 'presumable power-users' and 'domain experts' is not as effective as some claims, and I advised testers to begin thinking about the future and the increasing demand for testers with more in-depth system knowledge and technical skills to not only remain effective and productive in the industry, but to help drive it forward. Some people jeered me, and one respondent from the StarWest 2005 conference wrote "Bj is a fool, and his talk shows why Microsoft products suck and why Microsoft will fail" on the feedback forms. A few others said they agreed with the message, but they really didn't like hearing it out loud. One person told me "Yes, we know we need to improve, but your talk was like hitting us in the face with a frying pan full of hot grease."

I mention this not to "toot my own horn;" that is simply not my style, for I am yet a simple man who is still learning to improve myself. I only mention this because here we are 5 years later still arguing over petty things that we sometimes do not have the knowledge, skill, experience, business insight, scope of influence, or credibility to change. (And in case you don't know, whining. griping, and emotional conjecture is simply not viewed as being very credible in trying to influence change in a room full of decision makers or business leaders.) Right now, the industry, and the testers in the industry need leaders to help them adjust to changes, provide strategic vision, and guide career growth that enables each person to be successful in their professional development and compete with their peers. Responsible leaders are those who face challenges head on, who put forth logical and rational arguments based on analysis and empirical evidence, but who continue to search for ways to solve not only the immediate problems, but also have the ability to envision potential side-effects and deal with those as well before they become problems.

Our profession is multi-dimensional, and as testers we must continually strive to increase our knowledge and our skills in order to take on new and exciting challenges in this dynamic and agile world of software testing. The path is not always easy, but it does provide many rewards to those who work in a job they truly enjoy.

Quality is not Value!

I.M.Testy — Mon, 16 Jul 2007 19:11:00 GMT

I previously blogged about quality and value , but after giving it more thought I determined that quality and value are indirectly related, but quality is not value! I define testing as any activity designed to evaluate an attribute or capability of a...(read more)

Testing is NOT responsible for quality!

I.M.Testy — Wed, 04 Apr 2007 20:01:00 GMT

The value of testing is important and critical to the success of many projects. However, testing is NOT solely responsible for the quality of a project! When we discuss the role of testing and quality in our internal training we emphasize the primary...(read more)

The future of testing

I.M.Testy — Sun, 03 Dec 2006 03:33:00 GMT

I spent the last week in beautiful Denmark teaching at our offices in Vedbeak. I really like Denmark . The people are absolutely wonderful, the city of Copenhagen is magnificently rich with history, and of course the beer is delicious. Of all the places...(read more)

Peopleware: A must read for everyone (especially managers)

I.M.Testy — Tue, 24 Oct 2006 21:22:00 GMT

I recently went to Portland, and when I am there I make it a point to always stop by Powells Book Store. They have a whole building about the size of a typical Barnes & Noble dedicated to technical books. It had been several years since I had read...(read more)

Bug counts as key performance indicators (KPI) for testers

I.M.Testy — Mon, 26 Jun 2006 13:16:00 GMT

Every once in awhile I meet testers who say their manager rates individual performance based on bug metrics. It is no secret that management is constantly looking at bug metrics. But, bug numbers are generally a poor indication of any direct meaningful...(read more)