I. M. Testy

Automation Foibles Unveiled: Saving random data

Now, many of you probably know that I am a big fan of computer generated random test data that is a represents a reasonable sample data set from the total population of possible test data. (I refer to this a probabilistic stochastic test data.) So, why would I argue against preserving randomly generated test data?

I just returned from STAREast, where for the second time in a month I heard someone suggest storing randomly generated test data in a file. Many people will site the inability to recreate random test data as a drawback to using randomly generated test data in a test. So, the reason these people suggested storing the random data in a file is so they can easily repeat a test with the same data should some randomly generated test data expose an anomaly. I absolutely concur that if we generate random test data, and that test data exposes a problem we need a way to recreate the data. But, isn't there a better way than to save random test data in a file?

Saving randomly generated test data to a file creates a test artifact. Depending on how much randomly generated data is generated, this file could become quite large. Also, saving data to a file impacts the performance of an automated test and certainly slows down manual execution of tests. Then consider the number of tests that generate random test data are executed numerous times throughout the lifecycle, and it doesn't take long until we have countless test artifacts simply storing more static test data that quickly loses its value (especially if no problems were detected). Of course, we can easily delete the files after the test if no anomaly was detected, but I suspect that most testers will delete those files upon the completion of the test if no problems were detected.

So, the question is how can we reproduce computer generated probabilistic stochastic test data if we don't save that randomly generated data to a file?

Planting Seeds

In computing, a seed is simply an integer value that is used by a random generator as the starting value. If we pass a seed value as an argument to a given random generator then we will consistently get the same random value each and every time. Essentially, a seed allows us to replicate computer generated probabilistic stochastic test data anytime as long as we use the same seed and the same random generator algorithm. So, instead of saving each and every piece of randomly generated test data used in any given test, we can simply log the seed value used by that test in the test results log file.

But, if we use the same seed all the time, then we are simply generating the same data over and over again. And, manually inputting a seed for each test that generates probabilistic stochastic test data is not an ideal situation, especially for automated tests. So, to solve that problem we can randomly generate a seed value that is then passed to the random generator algorithm!  Again, logging the randomly generated seed allows us to accurately reproduce the probabilistic stochastic test data at any later time.

The example below illustrates a simple method in C# that will either generate a random seed or return a user specified seed value.

        public static int GetSeedValue(string seedValue)
        {
            // check if user specified seed value is passed as an arguement to 
            // the seedValue parameter
            if (seedValue == string.Empty)
            {
                // Create a new random object
                Random randomObject = new Random();
                // Generate a random integer value between 0 and 2,147,483,647
                return randomObject.Next();
            }
            else
            {
                // convert the seedValue to an integer value
                // NOTE: This example method does not include exception handling
                return int.Parse(seedValue);
            }
        }

The following example illustrates how to use this method to get a random seed value to generate random strings and numbers that increase the breadth of test data coverage in each subsequent iteration of a test.

        static void Main(string[] args)
        {
            // These variables declare the range of characters used for the
            // string test data. In this case the strings are composed of upper
            // case ASCII characters 'A' through 'Z'
            char minChar = '\u0041';
            char maxChar = '\u005A';
            
            // This reads the user specified seed value from the console window
            // If no seed value is specified an empty string is passed to the 
            // GetRandomSeed method which will cause it to generate a random 
            // seed value.
            string mySeed = Console.ReadLine();
            
            // Declare a seed variable and initialize it to either the user
            // specified seed or to a computer generated random seed value
            int seed = GetSeedValue(mySeed);

            // The seed value should be permenently recorded in the logged
            // results for this test
            Console.WriteLine("The seed value for this test is {0}\n", seed);

            // Create a new random object based on the seed
            Random randomGeneratorObject = new Random(seed);

            // Generate 10 random strings
            for (int count = 0; count < 10; count++)
            {
                // Declare and initialize a string variable for our test data
                string testString = string.Empty;
                // Generate random length strings between 1 and 10 characters
                for (int length = 0; length < randomGeneratorObject.Next(1, 11); length++)
                {
                    // Generate a random character within the defined range and
                    // concatenate it to the testString variable until the 
                    // random string length has been reached
                    testString += Convert.ToChar(randomGeneratorObject.Next(
                        minChar, maxChar + 1)).ToString();
                }
                // Write the test string to the console window
                Console.WriteLine("Test String {0}: {1}", count + 1, testString);
            }

            Console.WriteLine("\nRandom numbers");
            // Generate 5 random numbers
            for (int numberCount = 0; numberCount < 5; numberCount++)
            {    
                Console.WriteLine("{0} ", randomGeneratorObject.Next());
            }
        }

Calling the Main method and passing an integer value between 0 and 2,147,483,647 will generate 10 random length strings composed of random upper case characters between 'A' and 'Z' and 5 random numbers. If no user specified seed is passed to the Main method then the code will call the GetGenerateSeed method and generate a random seed value for use in the test. Of course, passing the same integer value will produce the same strings and numbers each and every time.

Using probabilistic stochastic test data is valuable because it efficiently increases the breadth of data coverage, and significantly augments 'typical' static test data, user-generated test data, or static test data derived from historical failure indicators. But, instead of storing randomly generated test data in a file, it is a best practice to simply record the seed value of each test. With a seed value we can easily recreate the computer generated random test data should any of the random data used in a test exposes an anomaly.

Email - the curse of productivity

It has been quite some time since I have posted. Part of that is due to personal distractions (getting my garden planted and my sailboat ready for the upcoming season), and part of that is being 'in the zone' working on some special projects at work. DeMarco and Lister began talking about being "in the zone" in their famous book Peopleware written in 1987. (In my opinion, this is a must read for anyone in the software business, especially for managers who should reread it yearly.) The Croatian psychologist Mihaly Csikszentmihalyi also discusses a similar concept he calls "flow," and identifies 9 characteristics of 'flow.'

Being in the "zone" for me is a myopic mental state where we are so focused on completing a task the world whizzes by and time becomes irrelevant. For many it is sort of a magical place; a momentary escape from reality. For example, when I sit down to write code in the evening the time passes so quickly that I soon discover it is 1 am in the morning. But, I want to complete one more class, and before I realize it is 4 am. (I am a slow coder.) People who are addicted to computer games know  all too well about the 'zone.'

This past month I had to come out of my zone and fly down to San Mateo, California to present a workshop and 2 talks at the Software Testing and Performance conference. It was a welcome break, and was nice running into old friends and meeting new acquaintances. I had the pleasure of meeting Karen Johnson and reconnecting with Doug Hoffman (who I had only met once previously) for lunch one day, and interestingly enough the conversation found its way to being in the zone. Karen brought up the fact that email is a constant distraction that often times impedes productivity.

Often times when we are in our zone our productivity increases. But, every time something changes our focus, such as responding to an email on a completely unrelated or tangential topic, or answering a phone call we are sucked out of the zone, and according to Lister and DeMarco it takes approximately 30 minutes to get back into the zone or back to our peak point of productivity.

For example, when I sit down to write, or code, or meditate I don't want to be disturbed and will usually retreat to my boat or some quite place where I know I won't be disturbed. I turn off my cell phone, no radios, no newspapers, magazines, no instant messaging (which I abhor) , and certainly no email. If I must stay at the office, then I block of my calendar for at least 4 hours and will sometimes disappear into some nook or cranny on campus.

I know that email is the life blood of many tech-companies. Unfortunately, I suspect that many of us could use a good bloodletting and need to relearn the art of verbal communication. I also suspect that we could better optimize our time by not being tethered to some email client during every single minute our waking hours. Let's face reality. For most of us 80% of the email we get is noise that we will forget about within the next 4 hours or so, 15% is good to know stuff (but not necessarily critical to our success), and I suspect that approximately 5% is really important stuff that we must respond to immediately. Of course, these percentages depend on our primary job. For example a manager or a consultant probably gets a larger percentage of email the requires immediate response.

So, I wonder if we managed our own use of email (and instant messaging) a bit more effectively how our own personal productivity might increase?

GUI Automation and ROI

It seems that many test automation efforts around the industry tend to focus on GUI automation, or automating functional tests primarily by manipulating GUI objects. In general, GUI automation tends to be a very expensive approach to test automation, and the automation efforts often end in failure or achieve less than satisfactory results.

The majority of automated tests at Microsoft are below the GUI; however, automated tests that manipulate GUI objects are quite useful within specific contexts. Unfortunately, many testers attempt to develop automated GUI tests way too early in the project cycle while the user interface design is still unstable. I guess the assumption is that constantly maintaining automated tests is somehow better then executing manual tests. But, in general, when the UI is in flux it is usually counter-productive and a loss of return to automate GUI level tests too soon in the development lifecycle.

Dan Mosley and Bruce Posey (Just Enough Software Test Automation) suggest that on average an automated test must run approximately 17 times in order to break even. But, this doesn’t imply that we break even if we simply run the same test on a daily build for the next 17 days. The presumption underlying the ROI after 17 runs of a test is that something changed in the build that is covered by that particular test, and so by executing that (regression) test we are providing important information (changes in the build did not destabilize that area) to the team. Steve Rowe also has an excellent blog post on Too Much Test Automation that you should consider reading, and Dustin Andrews also has an excellent blog post on getting great results from test automation with an 8-minute video.

The cost of test automation is never easy to figure out and is certainly not a straight forward comparison of automated time versus manual time. Comparing automation time versus manual execution time is an overly simplistic measure that rarely takes into consideration the design and the development time of the initial tests (or the overall costs of building and enhancing test frameworks or drivers), or the time required to identify false negatives, troubleshoot the cause, fix the problem, verify the fix (manually), and then check the source back in for the next run. (And, there are other tangible costs, and intangible costs such as loss of confidence that must also be considered in any cost model.)

So, in general any automation that requires constant maintenance is usually not cost effective, and the more our test automation throws false negatives the less our management team views automation as a viable resource to provide us with valuable and reliable information for improved risk analysis. Ultimately the decision comes down to how much perceived and measurable value a test has in providing important information for improved risk assessment and quality measurement.

Just because we can automate something, doesn’t always mean we should!

Customer expectations

Last October after presenting a keynote at the Conquest software testing conference I was invited to speak at an internal quality conference at SAP. At first it may seem a bit odd because Microsoft and SAP do compete within one market segment; however we also do collaborative work on other projects. Regardless of the company we work for, I do believe that most software engineers are intent on doing the best possible job they can to produce the best possible product they can that will ultimately provide a high value solution to the end user customer.

The common theme throughout the conference was the need to improve quality. In my keynote I suggested that customers demand higher quality because the end user customers of today are very different than the customers of yesterday. Today, software permeates virtually all aspects of our life. Today, software is found in children's toys, in our automobiles, and even in toothbrushes. A decade ago computer users were accustomed to periodic anomalies and  assumed it was the price of technology; however the end user customers today have much higher expectations of software and  presume it will simply work and provide an easy solution that offers some perceived value in their lives!

In my keynote address to the engineers at SAP I described how I would sometimes say "my Mom wouldn't understand how to do such and such" when describing ambiguous functionality, but in order to remain successful and competitive in today's market we need to consider designing and developing software for our children and future generations. Of course we want our existing customers (and my mom) to feel comfortable using our software, but it is readily apparent that our children want a very different experience and have higher expectations from software.

Do testers do code reviews?

This weekend on the flight from Seattle to Ireland I finally got to catch up on some reading. One of the books I grabbed off the shelf that I hadn't gotten around to reading yet was Best Kept Secrets of Peer Code Review by Jason Cohen. The book is 160 pages packed with great information, and I highly recommend it for some fresh perspectives of the value or ideas on improving code reviews. For decades the industry collected mounds of empirical data that pretty clearly illustrates the value of code reviews in a software development lifecycle in the early detection and removal of issues and anomalies. With the industry wide push to drive quality upstream via approaches such as agile programming and test driven development there is a resurgence of effort by developers to engage in more frequent code reviews.

Many people seem assume the primary reason why Microsoft (and other companies) is recruiting more technically skills testers is simply to automate tests. But, in fact, many testers at the company engaged in writing test automation long before our efforts towards engineering excellence. The skills and knowledge that people who have a much deeper understanding of the 'system' bring to an organization extends well beyond their ability to write automation.

But, should testers be performing code reviews? In my opinion, code reviews are simply another approach in a myriad of approaches to testing a software project, and professional testers should be able to engage the testing effort from a variety of testing perspectives.

Testers who participate in project code reviews are part of the team effort to drive quality upstream, reduce certain classes of issues before they are checked into the build, improve overall long-term maintainability of the code base, and effectively reduce long term costs. Just as testers participate in the early stages of the design and requirements phase of a project to provide valuable input, some testers on every team should engage in code reviews either with other developers, or perhaps as the primary reviewers of project code. Some people will argue that testers may be biased by participating in code reviews and lose the 'customer' perspective. This may be true in some cases, but I suspect that most professional testers clearly understand the different classes of issues that are more easily found in a code review versus other approaches of testing. But, testers who engage in code reviews tend to have a much deeper understanding of the overall project, and they are also able to identify potential areas of the code that may be more problematic and focus additional testing approaches in those areas.

Vista Rant #3 - And yet another boundary issue in Explorer's listview?

This morning I installed Vista SP1 onto my laptop. I was pretty excited about this release of Vista SP1 because it includes some pretty significant performance enhancements. But, as I was preparing to teach an internal course I came across a new boundary issue. I thought, how fitting this comes as I prepare to teach another class on systematic testing techniques (including boundary value analysis) that I find yet another classic boundary issue in Explorer's list view (albeit the boundary condition is not readily apparent) . Interestingly enough, the previous boundary issue I found appears to be fixed with Vista's SP1; however, I previously did not run across this issue which is also connected with how listview repaints itself after an event.

To reproduce this issue, open a folder with several dozen files in Explorer and select Views -> List and resize the Explorer window so there are several columns of files.

Select a file in the list and press the up or down arrow key so a dotted line appears around the file name

Resize the Explorer window so that the highlighted area is about 1 pixel away from the right most edge of the Explorer window as in the example below

Press the down (or up depending on which file you have selected) arrow key, and notice how the file highlight jumps to the next file in the list as expected.

Next, resize the the Explorer window so the dotted line touches the inner boundary of the window as in the example below

Now, press the down (or up depending on which file you have selected) and...

Notice, in the image above that just to the left of the frogger.def file the right most edge of the file highlight is visible, and also notice the file image has changed, and the scroll bar has jumped over to the next column.

Press the down (or up) arrow key again, and....

Seeing this behavior the first time made me think I was losing my mind! (OK...I probably am, but that is a completely different topic). And yes, reducing the size of the window beyond where the maximum boundary of the highlight window will also cause the same behavior, but the issue occurs when the width of the file highlight window >= the maximum x boundary size of the Explorer listview window.

The listview in Windows Vista has some very cool features, and overall I like Vista. What I dislike are some of the simple anomolies that could be easily exposed via more detailed, systematic testing approaches and analysis of the system under test rather than simply assuming more bodies banging on keyboards for some arbitrary period of time equates to better testing.

So, some things to consider when boundary testing...

• Not all boundary conditions are easily identified from the GUI by numbers, but all boundaries conditions have linear physical values that can be measured at some level
• Boundaries can change, similar to way the boundaries changed around Kosovo. Somebody moved the lines. The same thing is true in software, a boundary condition may change with human interaction, but a boundary condition is a linear physical range between a minimum and a maximum value (in this case the x coordinates of the Explorer changed with human interaction, but once focus moved away from the Explorer window the size of x established a fixed boundary (at least until a user again resized the window or the window is closed).
• Boundary conditions for one parameter are usually independent. For example, resizing the y-axis in the above scenario has no impact on this defect (unless of course the y-axis is large enough to accommodate all files in the listview without having to scroll horizontally). Boundary testing is based on the single fault assumption theory which states that a boundary issue is most likely to occur with independent parameters where the boundary variables for one parameter are analyzed while holding other parameters to nominal values. (Note: if we suspect that parameters are dependent or semi-coupled, then we should also perform combinatorial analysis testing.)
• There is a difference between boundary conditions (at least momentarily fixed, linear, measures) and threshold values. Threshold values can be altered by various influences. For example, in performance testing the point of degradation in performance can often be changed by several external influences such as increasing physical memory, cleaning and defragmenting the hard disk, modifying the software, etc.
• A detailed analysis of the system under test will reveal issues that other approaches to testing do not expose...that's the pesticide paradox!

Contextual blindness: or How to take things completely out of context

Many testers are familiar with the concept of inattentional blindness (or at least should be in my opinion). Basically inattentional blindness occurs when we are so visually focused on a task  or object that we completely fail to see something out of the ordinary.

But, I am going to introduce my own neologism that I will refer to as contextual blindness. Contextual blindness occurs when someone is so restrictive in their thinking or so biased by their own opinion that they take references to a document or study completely out of context to support their own biased argument. In essence, they are ignoring the original context in which the statements are made, and perverting the sentence or using a statement out of its original context to support an oppositional point of view.

I too have been guilty of this when I made statements without completely researching available data or carefully reviewing empirical, or factually substantiated evidence. (These days, if I don't have sufficient data or information to support a strong argument for or against something then I try to preface my statement with "I suspect..." or "in my opinion..."). However, some people seem to make a habit out of making wild, and often fallacious statements often in seemingly juvenile attempts of one-upsmanship. I suspect that sometimes people do this because they think they are beyond reproach; that they assume to know more than others, or that they consider themselves to be such an expert that nobody should question anything they say.

As I have gotten older and a bit more wiser, I have learned to question things and reassess my position or my ideas from time to time. I often speak with recognized industry experts, read several books and studies (often presenting  contradictory approaches or perspectives), review empirical data (and just to be clear, IMHO bug count as the only data point offered as empirical data is about as useful as nipples on men), and when I can I try to experiment or experience new things in order to draw my own conclusions. By now your probably asking where I am going with all this...there is a point...read on!

This evening a person sent me mail asking me if my keynote at EuroStar last year was "an attack on certification programs." She knew I gave one of the keynotes at EuroStar, but was a little shocked that I would attack certification programs. I told her that I gave one of 5 keynote addresses at EuroStar 2007 and my talk was entitled The Path to Professionalism: Skills of star performers, and gave her my perspective of certifications. In her response she forwarded me some mail from a distribution list which included excerpts from a rather lively debate between two other members of the list in which one  of the participants in the debate stated,

"The Department of Defense has identified the failure of traditional testing processes, including the problem of over-documentation, as one of the top five problems in IT. The keynote speech at Eurostar, in December was an attack on certification programs such as the CSQE."

So, I reminded her that the second sentence should more appropriately read, "One of the keynote speeches at Eurostar...". I told her that Michael Bolton gave that talk. (I must say that it was the first time I had seen him speak and I his stage presence is excellent, but I wasn't overly impressed with his arguments against certification because he simply denigrated the existing programs without offering any other solutions. Several delegates at the conference later asked me about his comments and I deferred by stating, "when in Rome." (Certifications in Europe are highly valued by employers for a variety of reasons.)  But, the above mentioned statement was merely misleading; it is actually the first statement that is most fallacious, and taken completely out of context.

The statement "The Department of Defense has identified the failure of traditional testing processes, including the problem of over-documentation, as one of the top five problems in IT" used the following study as a reference. So, let's take a critical look at that statement and question its truth or validity. (Because, in our jobs as professional testers understanding the correct context, critical thinking and logical questioning are important skills!)

"All we want are the facts, ma'am"

Fact #1. The DoD did not identify anything. The study was conducted by the National Defense Industrial Association (NDIA), which is not in anyway shape or form a part of the Department of Defense (DoD). NDIA members include "individuals from academia, government, the military services, small businesses, prime contractors, and the international community, the opportunity to network effectively with the government."  There were 26 participants in this study and one participant represented the DoD.

Fact #2. The study did not identify "the failure of traditional testing processes."  The purpose of the study conducted by 26 participants was to "Identify the top 5 software engineering problems or issues  prevalent within the defense industry." The participants actually came up with 7 issues, and they determined that one of the top issues in software engineering included "Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems."

Fact #3. The problem of over-documentation was in the context of a discussion related to tests with a "disproportionate effort on detailed procedures." This was not one of the 5 (actually 7) problems identified, the participants concluded "Tests are over-documented with disproportionate effort on detailed procedures." as a possible reason to explain the 5th top issue of "Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems." I suspect this statement speaks to the fact that many documented tests that I have seen by under-trained testers are simply prescriptive, regimented scripts based on some interpretation of an ambiguous requirements document rather than well-formed tests designed from an in-depth analysis of the system under test.

What the study really said...

But, not only did this person take parts of several statements in the report, munge them together in an attempt to use the findings completely out of context; this person also completely ignored (or purposefully omitted)  other points discussed in relation to this specific problem such as:

• Over-reliance on testing alone rather than robust SW verification techniques.
• Manual testing techniques are labor-intensive, scale poorly, and are unproductive relative to the large investment of resources."
• Compliance based tools do not adequately cover risks or failure conditions
• Tests are over-documented with disproportionate effort on detailed procedures
• Education, training, certifications are inadequate to develop effective test skills.

The person also ignored (or purposefully omitted) the recommendations by the participants which included:

• Sponsor a study of state-of-the-practice verification and testing approaches.
• Review/update testing policies and guidance to emphasize robust, productive approaches that maximize ROI.
• Review adequacy of verification plans/approaches early in the acq. life cycle.
• Emphasize skilled investigation throughout the life cycle, based on coverage, risk mitigation, high volume automation.
• Strengthen curricula, training, certifications, career incentives for testing roles.

Now, I really don't understand how someone can read this report and misconstrue the information to imply that traditional testing processes (and it is not clear what they are referring to here), or that over-documentation is one of the top 5 problems identified by the DoD; especially when the report also recommends strengthening policies or guidelines for full requirements traceability.

I have my suspicions as to why the person who made the fallacious remark above might omit all the facts and additional counterpoints in their argument. I suspect those details were not revealed because those points do not support (in fact, they completely dispute) the context-driven ideology that emphasizes manual GUI testing and the vehement opposition to documentation, test automation, coverage analysis, robust verification techniques, strengthening of certifications, or essentially anything that involves the need for greater technical know-how, logical analysis, or measurable advancement of the testing profession.

Microsoft ranked in top 10 training organizations

I have been tardy in my writing...too many irons in the fire these days. This week I have been teaching new SDETs in the morning, then rushing downtown to teach evening classes on software testing at University of Washington. These days I spend a great deal of my time teaching (mostly about systematic testing approaches and test automation), consulting on testing practices, designing and developing hands-on technical training, and talking about the future of the discipline. This role is personally satisfying, and it is certainly one of the most challenging positions I have ever had. So, as a member of the Engineering Excellence group at Microsoft I am very proud to announce that we are in the top 10 of the world's most successful learning and development organizations. When Microsoft embarked on its journey in 2004 to refocus our product teams on engineering practices including the drive to increase the technical skills and acumen of our software testers our training was not even listed in Training Magazines Top 125 awards. In 2005 we were listed in 38th place. In 2006 we moved up to 23rd place, and 19th place in 2007. Last week, Microsoft was ranked 9th among the top 125 rated training organizations around the world. Companies are ranked based on various factors including number of training hours per employee, detailed formal programs, use of various learning technologies, and quantitative and qualitative case studies demonstrating strategic business impact. I must admit, I consider myself very lucky to be part of such a great team of professionals who are incredibly passionate about their discipline and strive to have a positive impact on how we design, develop and test our products.

Do testers need programming skills?

The debate over whether testers need to at least understand programming concepts is still raging within the discipline. To me this debate is puzzling because it seems to suggest that as a professional, I don't have to really understand or be completely proficient in critical aspects of my trade. Even Cem Kaner noted, "I think that the next generation of testers will have to have programming skills." Actually, there was a time not so long ago when testers had to have programming skills, so it is nice that Cem now acknowledges that skill as useful in testing.

Unfortunately, occasionally even within Microsoft a few people still want to differentiate between STE and SDET by blindly assuming that STE meant non-programming testers. The fact is, that the old STE ladder level guidelines clearly stated skills such as debugging production code, and design and develop effective automation as required skills for Microsoft testers. Unfortunately, some managers chose to selectively ignore these skill requirements and some groups chose to differentiate between GUI testers and  any tester who could write code by labeling them as STE and SDET respectively.  (This was a horrible abomination of job titles in my opinion.) The new SDET competencies at Microsoft are designed, and supposed to be implemented in a manner, to reinforce the essential skills we expect from our testers so a tester at a certain level in their career stage in one business unit essentially has equitable skills of any other tester at the same level in their career stage in any group in the company.

But, people are often resistant to change, and as I wrote in my last post some people choose to wallow in self-pity, pretend they are a victim of some evil plot, hypercriticize change with dogmatic arrogance, and incessantly bemoan dubiously negative aspects of change from an often overly emotional narrow-minded perspective. A person who moved from a testing role to program management stated, "I was a tester because I understand how users think and how they use products and I wanted to use that knowledge to make our software better." Really? We make software better by beating quality into it? Does this demonstrate a good understanding of software processes and good business logic? I only ask these questions because it is pretty well-known that it is much cheaper to prevent defects, and that many defects can be found in the design process. So, I am asking myself why in the world didn't this person start as a Program Manager (responsible for interpreting marketing analysis and customer feedback into requirements and product design) or become one before now? What is even more amazing about this statement is that it doesn't even acknowledge the fact that as a program manager this person is now in a role that should have a direct connection to the customer and a greater impact on making our software better. A development strategy or process that emphasizes customer advocacy primarily in the testing phases is ridiculously immature and a gross waste of resources since it is widely known through empirical studies that it is cheaper to prevent defects by better designs and clear requirements as opposed to finding them during a testing cycle.

The same person stated, "I wanted to keep breaking software in the incredibly fun, very effective way I had been doing." (Personally, I find API testing (which can also use a black-box approach), and white box test design extremely fun and intellectually challenging, and is also very effective when used appropriately.) Unfortunately, this comment seems to perpetuate a myth that testers make software better by finding bugs, and it also demonstrates an extremely limited view of the role and overall potential value of software testing to an organization. This is indeed a very narrow, antiquated (in technology time), and immature view of software testing that emphasizes testing as primarily a bug finding endeavor. However, Beizer wrote that black box testing exercises approximately 35 - 65% of the product, and Marne Hutcheson and I have empirical data that demonstrates that GUI testing (which is one type of black box testing most people are familiar with, and the type of testing most non-technical testers are limited to perform) is not as effective as most people want to believe, and it is often more costly as compared to using a variety of approaches to software testing. Again, even Kaner notes, "Programmer productivity has grown dramatically over the years, a result of paradigmatic shifts in software development practice. Testing practice has evolved less dramatically and our productivity has grown less spectacularly. This divergence in productivity has profound implications—every year, testers impact less of the product. If we continue on this trajectory, our work will become irrelevant because its impact will be insignificant." (I highly suspect the 'testers' Kaner is referring to in this context are primarily non-technical, GUI testers since that is the type of testing emphasized in his BBST course.)

There is no doubt that a person who does not at least understand programming concepts or have an in-depth technical understanding of the system they are testing is unable to perform various activities that may be required in the role of a professional software tester. That person cannot perform code reviews (which have been proven to find certain classes of defects more effectively than any other type of testing); they cannot analyze code to determine which areas of the code have not been tested and design tests from a white-box approach to increase testing effectiveness and reduce risk, they cannot debug errors and identify root causes of defects, they cannot automate tests to free up their time or reduce costs during the maintenance phase of the product lifecycle, they may not be able to adequately analyze and decompose test data, etc. While some companies don't rely on their testers to do this type of work, these are certainly tasks that any professional tester should be able to perform.

I guess there are some software companies that are not interested in actually maturing their processes, reducing long term costs, have no interest in the intellectual property value of testing artifacts, or simply want to continue to rely primarily on GUI testing to get a 'gut-feel' of their product before releasing it. However, many large companies that produce software (Microsoft, Cisco, Google, Siemens, etc.) understand the value add proposition that professional testers provide to the organization health and specifically hire people into testing roles who have both broad technical skills as well as the common traits we tend to associate with good testers.

This post is not to question the need for non-technical people who have in-depth and current domain or business knowledge of the application space, or who understand the market expectations and customer demands/needs in the software engineering process. The question I ask is whether the value these individuals bring to the software development process is misplaced, or would their contribution be more cost effective and provide greater overall value to the customer if they were in a role (other than testing) that better utilized their knowledge by contributing to defining requirements and designing high quality software rather than trying beat in quality through bug finding?

Thoughts on becoming a professional tester

"If a man is called to be a streetsweeper, he should sweep streets even as Michelangelo painted, or Beethoven composed music or Shakespeare wrote poetry. He should sweep streets so well that the hosts of heaven and earth will pause to say, here lived a great streetsweeper who did his job well."- Rev. Martin Luther King Jr.

As I was growing up my parents taught me the value of working for things I wanted, and to how to bear the burdens associated with responsibility. When I was a teen we moved to the country because my mother had become very active in horseback riding and my father wanted to stop boarding the horses. So, we moved out of the suburbs and away from the upper Chesapeake. I was not really happy about this because I love the water. As a young boy I spent a great deal of time with my friends hanging out in the marinas where I learned a lot about boats, hard work (from the brightwork to the bilge), and life in general. As I young lad I remember listening intently to the trials and tribulations of old blue crabbers, fishermen, and sailors. I loved swimming in the tributaries of the upper Chesapeake, fishing for yellow perch, sunfish, and going crabbing and walking the riverbeds at low tide for soft-shell crabs. I also was not really looking forward to the change because it meant getting up very early in the morning to feed horses, and cleaning stalls after coming home from school. But, since I also opted to get a horse, that is burden of responsibility I had to bear.

Throughout life our environment changes. That's just the way it is. New 'doors' open, and some 'doors' close. I often recall my first manager at Microsoft telling me that it is up to me to control my own destiny at the company. But, I had been around long enough to realize that life also sometime throws us curves or unexpectedly changes direction, and Microsoft is certainly a dynamic company. It is often during times of change where new and exciting opportunities present themselves. Even during times of change I believe we still control our own destiny (at least somewhat). When our environment changes (as it always does) we generally have many choices. For example, often we can embrace the change and dynamically adapt and/or rise up to new challenges that life presents. Or, we can choose to wallow in self-pity, pretend we are a victim of some evil plot, hypercriticize the change with dogmatic arrogance, and incessantly bemoan the dubiously negative aspects of the change from an often overly emotional narrow-minded perspective. This latter choice is usually not an especially productive one (personally or professionally), and generally only demonstrates one's extremely biased, and limited view and their incapacity to grasp the "big picture."

Let's face it; we have chosen to work in one of the most dynamic industries in the world. Change is all around us, although some things remain relatively the same. For example, the techniques medical doctors use in the initial screening of certain maladies have remained relatively constant for decades, but at the same time advancements medical imagery has made tremendous technological leaps forward. Likewise, the practice of exploratory testing has been used extensively in software testing for decades, but the effective application of combinatorial analysis (pair-wise, triplet, n-wise) of interdependent or semi-coupled parameters has only recently become a mainstream best practice.

The testing discipline is undergoing tremendous change these days. There are many people around the world who are serious about maturing and advancing our profession. Some ideas are great, some still need to be refined, but at least they are seriously investigating at ways to advance the profession. As a tester working in a rapidly changing industry we must constantly re-evaluate our skills, and increase our professional knowledge of software testing and computer systems in general in order to provide the best possible service to our employer.

I think if someone chooses testing as a profession, then they should strive to become a professional in the discipline, and develop and refine the skills and knowledge that entails.

UTF What?

Years ago life was pretty simple with regard to data input. Most computer programs were limited to ASCII characters and a set of character glyphs mapped into the code points between 0x80 and 0xFF (high or extended ASCII) depending on the language. The set of characters was limited to 256 code points (0x00 through 0xFF) primarily due to the CPU architecture. Multiple languages were made available via ANSI code pages. Modifying the glyphs in the upper 127 character code points between 0x80 and 0xFF worked pretty well expect for East Asian language versions. So, someone came up with the brilliant idea of encoding a character glyph with 2 bytes instead of just one. This double byte encoding worked quite well except that many developers were unaware that a lead byte could be an 0xE5 character and a trail byte could be a reserved character such as 0x5C (backslash). So, an unknowledgeable developer who stepped incrementally though a string byte by byte would often encounter all sorts of defects in their code. Fortunately today, most of us no longer have to deal with ANSI based character streams on a daily basis. Today most operating system platforms, the Internet, and many of our applications implement Unicode for data input, manipulation, data interchange, and data storage.

Unicode was designed to solve a lot of the problems with data interchange between computers, especially between computer systems using different language version platforms. For example, using a Windows 95 operating system there was virtually no way to view a file containing double byte encoded Chinese ideographic characters using Notepad on an English version of Windows 95. But, on Windows Xp or Vista not only can we view the correct character glyph we can also enter Chinese characters by simply installing the appropriate keyboard drivers and fonts. No special language version or language pack necessary! So, if we created a Unicode document using Russian characters those same character glyphs would appear no matter what language version operating system or application I used as long as the OS and application were 100% Unicode compliant.

However, Unicode of course has its own unique problems. Unicode was originally based on the UCS-2 Universal Multiple Octet Coded Character Set defined by ISO/IEC 10646. Essentially, UCS-2 provided an encoding schema in which each character glyph is encoded with 16-bits (or 32-bits for UCS-4). A pure 16-bit or 32-bit encoding format didn't really appeal to a lot of people due to various problems that would arise in string parsing. Most data around the world up to that point (with the exception of East Asian language files) were encoded with 8-bit characters. So, some really creative folks came up with ingenious ways to encode characters that more or less captured the essence of UCS (i.e., one code point == one character) using UCS transformation formats (UTF).

Another problem with UCS-2 and a pure 16-bit encoding was the limitation of 65,635 character code points. It wasn't very long before most people realized this set of code points was not adequate for our data needs. But, instead of adopting a UCS-4 encoding schema the Unicode Consortium redefined a range of character code points in the private use area as surrogates. These surrogate pairs would reference 16-bit character code points in different UCS-4 planes.

A while back I designed a tool called Str2Val to help developers and testers troubleshoot problematic strings. For example, lets assume the following string ṙｭϑӈɅ䩲Ẩլ｡ḩ»ﾓNJĬջḰǝĦ涃ᾬよㇳლȝỄ caused an error in a text box control that accepted a string of Unicode characters. A professional tester would isolate the problematic character or combination of characters causing the error and reference the exact character code point(s) by encoding format in the defect report. I recently upgraded the Str2Val tool to show the same string by various encoding formats such as UTF 16 (big and little endian), UTF-8, UTF-7, and decimal. Not only is this a good tool for trouble shooting problematic strings, it is also a useful training tool to explain the differences in the various common UCS Transformation Formats or UTF encoding methods.

Why is this important as a tester? Well, if you think you represent your customers yet the only characters you use in your testing are the ones labeled on the keyboard that is currently staring you in the face then you are only dealing with a small fraction of the data used by customers around the world (assuming that your software is used outside the country where it is developed, and most English language versions of software are used around the world if they are available on the open market.) If you don't know how the characters are encoded or which types of problems can arise from the various encoding methods then do you really know how to devise good tests, or are you just guessing? Do you know how to design robust tests with stochastic test data, or are you stuck with stale static data strings in flat files that you simply use over and over again? When a defect occurs in a string of characters (since string data is quite common in testing) can you troubleshoot the cause or isolate the code point, or do you simply just say "yea!...I found another bug!" and throw it back at the developer to figure out?

Automation Foibles Unveiled: Coding Guidelines Part 1 - Basic Layout

It has been awhile since I have written about test automation, so I thought I would start the new year off with a post about test automation. More specifically, I wanted to start talking about coding guidelines. Just as many development teams have adopted coding standards and guidelines the test team should also adopt a set of guidelines and standards for all testers to follow when developing test automation. Some teams have coding standards and guidelines, but some teams just sort of wing it. Of course, those teams that just sort of wing it and allow developers the freedom to develop their own style of coding soon realize the cost of reviews and maintenance increase. Sloppy coding also increases the probability of subtle defects that may go unnoticed. Coding standards or guidelines are not (or should not) restrict a test developers creativity; but consistent coding standards and guidelines improve the efficiency and effectiveness of code reviews and sustained maintenance (which is often performed by someone other than the person who wrote the original code).

This week I want to start by talking about problems with various coding styles in source files such as indentation and general layout issues that make code more difficult to read. Readability is important for efficient code reviews, and when the code is later maintained by someone other than the person who originally wrote the code to begin with. There are many coding standards and guidelines that a team can and should adopt and I plan to discuss various guidelines over time, but for this week I thought I would introduce some commonly accepted guidelines that improve the readability of code by adopting a consistent layout.

Indentation

• Set the indentation or tab size to 4 characters in width
  I have seen people set the tab size to as small as 2 and in some rare cases to 3 for some bizarre reason. But, 4 characters in width seems to be the standard tab size, and an additional 2 characters per line is really not going to make that much of a difference, and I think it improves readability of the code by clearly indenting blocks. (I really don't want to get into a long debate about 2 or 4 spaces; the point is that the team should use a consistent indentation size.) 
• Use spaces instead of tabs
  This is often controversial, but spaces are recommended over tabs for a variety of reasons. One reason why one should use spaces instead of tabs is for consistent layout and readability in a variety of editors. Many text editors interpret tab stops differently, and occasionally the editor a reviewer uses to look at source code may be different than the source code editor used by the developer causing the line formatting to change radically. For example, when I need to take a quick look at a .cs file I will often open that file using Notepad instead of launching Visual Studio. Sure, I don't get the pretty colors and such, but it's a lot quicker! (Again, I don't really want to get into a debate about tabs vs. spaces. That subject has been beaten to death on the Internet and there are pundits on both sides of the fence. If file size is a really critical factor than tabs will reduce the overall physical file size in bytes. If readability in different editors is important, or if restricting the use of the 0x09 character code point control code in your source code is important the use of spaces is preferential. Again the important point would be consistency within the team.)

To setup these settings if you are using Visual Studio (including the freely available Visual Studio Express editions)

1. Select Tools -> Options... menu item
2. Select Text Editor -> All Languages -> Tabs item in the Options tree control
3. Set the indentation to Smart.
4. Set the tab size and indent size to 4
5. Select Insert spaces (or Keep tabs; whichever standard your team adopts)

Line length

• Do not extend lines beyond 80 characters in length
  Occasionally, developers will simply write a statement that extends well beyond the viewing area of the editor window forcing reviewers to scroll. Or worse yet, when opening a file in Notepad or some other simple text editor the line wraps (if word wrapping is selected) around causing the layout to become quite confusing to the eyes. Word wrapping also becomes a problem when printing source files for review. Either way extending lines beyond 80 columns looks sloppy and makes readability more difficult.

Notice lines 76 and 80 extend well beyond 90 characters. If this source file was printed lines 76, 80, and 83 would wrap to the next lines on the page starting from the left margin impacting the readability of the file.

The column (or character count) per line is usually displayed in the status bar of an IDE, but for Visual Studio (including Visual Studio Express editions) a simple registry setting provides a visual guide in the IDE. The visual guide in the IDE means that I don't have to look at the status bar to see where my column count is, or go back later and reformat lines after I am finished coding a method or class.

Notice the red dotted line on the right hand side of the code window set at a column width of 80 characters.

To setup these settings if you are using Visual Studio (including the freely available Visual Studio Express editions)

1. Select Start -> Run menu item
2. Type regedit in the Open control on the Run dialog and press OK
3. In the Key tree control select HKEY_CURRENT_USER | Software | Microsoft | Visual Studio | [version] | Text Editor
4. Right click in the value pane and select New -> String Value from the pop up menu
5. Rename the new value Guides
6. Right click on the Guides value item and select the Modify... menu item
7. Type RGB(255, 0, 0) 80 in the Value data textbox on the Edit String dialog

I also prefer to display line numbers. This makes it a bit more convenient when doing things such as Finding All References to variables.

In subsequent posts I will discuss additional coding guidelines, but these are just a few ideas to get the discussion started. If you have additional ideas on layout then please let us know!

How Professional Testers Think: Why Microsoft primarily hires testers with a Computer Science, Math or Engineering background?

The easiest thing to criticize is that which one does not fully comprehend.

There has been a lot of discussion lately about Jerome Groopman's book How Doctor's Think and a correlation between doctors in the medical profession and software testers. The book is an excellent read, and provides readers with valuable insights not only with medicine, but we can certainly abstract many of the fundamental lessons to testing software. Indeed there are many similarities between doctors practicing medicine, and software testers testing software. I won't belabor this point since many others have discussed the similarities, and one only has to read the book to discover the obvious.

However, there are several aspects that some people seemingly blindly overlook when comparing medical doctors or others in the medical profession to software testers. For example, medical doctors are highly educated in biology, chemistry, human anatomy, physiology, etc. In other words, doctors spend a great deal of time being formally educated in subjects directly related to their profession. Compare this with many testers working in the field of software testing. This is not to imply that in order to be a professional tester that one must study computer science at a university; however, I know a great number of testers who have worked in the industry for several years and still do not understand basic programming concepts, lack the ability to perform an in-depth analysis of test data, and are incapable of adequately decomposing or modeling features below the user interface. In other words, testers who lack the basic knowledge of computer systems and software are utterly blind to the internal workings of the system on which they are working. Doctors also spend a great deal of time post graduation reading medical journals and attending medical conferences and workshops to continue their education and build their knowledge and skills. Dorothy Graham, Marne Hutcheson, Steve McConnell, and I have collected a lot of empirical evidence to suggest that very few testers have ever received any formal training in software testing methods, and less than 1% of testers polled have read more than one book on software testing. Lee Copeland presented an excellent talk at a conference entitled "The Nine Forgettings" in which he asks, "How can we call ourselves professional testers if we are ignorant of the fundamental techniques of our craft?"

Doctors also realize there are practical techniques (systematic procedures) commonly used in their profession that are extremely useful in the correct situation and valuable in either diagnosing an ailment or identifying contraindications, and competent doctors know when and how to apply those techniques. Conversely, I often hear testers whom I suspect have a superficial understanding of the overall system or fail to adequately perform an in-depth investigation of the system on which they are working refer to testing techniques as folklore. Who needs techniques...if we pound on the keyboard or touch screen long enough we are surely bound to find a bug! (The only folklore I've ran across in the industry is a belief that only a handful of people really understand, or are even capable of understanding exploratory testing.)

Medical science also has an common body of knowledge and professional jargon that is universally accepted throughout the trade. When doctors refer to exploratory surgery there is a common understanding of what is being discussed. Doctor's don't pointlessly argue how one person's practice of exploratory surgery is different than another person's approach and therefore it is not really the 'exploratory' surgery that "I" preach. Reputable doctors also don't needlessly make up words to make themselves sound like they offer something new or revolutionary. When they discuss medical conditions they discuss things within rational contexts. For example, when someone complains about numbness in the fingers they may perform tests to check for adequate blood flow to the hand, or they may ask the patient about neck injuries or pain in the neck or upper shoulder region to see if the problem might be caused by a pinched nerve. I don't suspect that too many doctors would ask the patient what they had for breakfast, or if they recently stubbed their toe to diagnose numbness in the fingers (but, I am not a medical doctor so that is just guessing on my part). Basically doctor's understand the head bone is connected to the neck bone and don't waste a lot of time hypothesizing "what if the head bone was connected to the knee bone?"

Finally, a really big difference about doctors and many testers is in how they fundamentally approach their job. Doctors are constantly striving to help people stay healthy. In other words, they are trying to prevent the spread of illnesses, or searching for ways to help people from becoming sick to begin with. Of course, this means that doctors must have an in-depth understanding of the 'system,' the types of 'bugs' a system might be exposed to, and how those 'bugs' can act on the 'system' or how the 'system' reacts to those foreign agents.

As Groopman discussed in his book, "There are primary care physicians in every hospital who speak with great sensitivity and concern, and their longtime patients love them, but clinically they are incompetent..." Likewise, I have met many people, some at MS and some in the industry, who still think software testing is simply about finding bugs, and that additional skills or knowledge of the profession is unnecessary if one is a good bug finder. I often listen with amusement at talks or when I read articles by people who profess that testing is about providing information, but the only 'information' they seem to provide is how to expose yet another obscure bug, or denigrate practices by incorrectly applying techniques out of context (such as attempting to apply pairwise analysis on a function requiring sequential inputs then claiming pairwise analysis is not a best practice because it is not good at finding bugs while failing to mention where and when the technique is best applied and other types of information that the technique provides such as increased code coverage). As I said in the opening, it is easy to criticize those things we least understand (unless of course we are purposefully obfuscating facts for personal motives). And the uselessness of any critique is only exacerbated by irrational arguments, illogical alternatives, or personal attacks.

So, how does all this relate to Microsoft's hiring practices and why we primarily hire people with computer science backgrounds?

Easy, graduate coming from university with a computer science background have a strong understanding of the 'system' and the 'system internals' much like the doctor understands physiology and human anatomy. This doesn't mean that we only hire testers with a computer science degree. Myself, and many other testers at Microsoft do not have a computer science degree; however, we have also not stagnated or simply rested on our ability to find bugs. We have constantly strived to improve our technical skills and overall understanding of computer systems, software testing practices, and testing knowledge. Interestingly enough, Microsoft career job guidelines have always required testers to be able to automate tests, debug production code, and design tests from both a white box and black box perspective, etc. The difference now is that those skill requirements are universally applied across the company.

Also, much of our internal training can now be based on a common baseline of expectations. For example, when we teach testers how to design tests from a white box perspective, or when I talk about the best practices of code reviews I expect that the testers already know how to read code. Our automation courses require that testers already know how to write code using procedural programming and object oriented programming approaches. Our training simply exposes testers to namespaces, classes, and methods commonly used in test automation (not in application development), and mentor them on how to design tests that will provide value to the organization from an automated testing perspective. (Also, I can attest that our testers are very good critical thinkers and not simply mindless droids who bang on keyboards or pump out simple rote, prescriptive scripts (similar to what is described here) and label them automated tests.

Finally, one of the things we reinforce in our training and throughout a person's career growth at Microsoft is driving quality upstream and defect prevention. We have senior testers who are mentoring developers in how to design and write better unit tests. Our testers are engaged in design and code reviews. Many teams now rely on testers to debug problems to the line of code in order to identify patterns of problems. We have testers who are doing root cause analysis of specific classes or types of defects and building tools or enabling processes to identify those types of defects earlier in the project or development cycle. And yes, our testers still design and execute a lot of manual tests. They use the product in development everyday by dog-fooding. And most testers even participate in newsgroups and sit with or listen in on product support calls to indirectly hear from end users.

So, much like you probably seek out the best possible medical doctor as a primary care provider or when struck with an illness, at Microsoft we want highly competent individuals in our software testing roles who understand that testing is a huge challenge and requires a great breadth of skills and knowledge in order to test software from various perspectives and approaches, and who possess a great deal of in-depth knowledge of the system they are working on. We seek professionals who realize that constantly bemoaning the inadequacies or drawbacks of an approach or perspective solve nothing, and instead actively engage in finding great, scalable solutions to problems in order to reduce testing costs and help drive overall quality upstream.

In a dynamic industry that changes rapidly as professional testers we must constantly reevaluate our skills and knowledge to stay abreast of changes in technology, and learn more about our chosen profession. We must be vigilant and strive unceasingly to improve our skills and knowledge of computers, computer software, and our profession in order to remain competitive with our peers and remain a valuable team member in our organization or to our employer.

Blindly buying into rumor and innuendo: or how to lose stock in your credibility

It never ceases to amaze me that every time we see a calamity involving software the immediate reaction of the sensationalist media types and other people who are generally misinformed is to blame inadequate testing. Recently, it seems that Joel Spolsky not only fell victim to rumors and misinformation, but also to the lunacy of blaming the testing organization for the rather lack luster adoption by individual consumers to move to Windows Vista.

Sales of Windows Vista are slow for a myriad of reasons. The UI is a radical departure (which most design experts applaud), security features are annoying (which most security experts applaud, yet many individual consumers dislike), machine requirements (means many would have to purchase new hardware), performance issues, etc. But inadequate testing or too much automation? Pah...leaze! I usually like reading Spolsky's blog, but must admit this is the first time I can recall in which he has used unsubstantiated rumors and innuendo to reach such a faulty conclusion as suggested in his post. Too bad...

Joel's attack was not about testing per se, but more about his assumption that Windows Vista suffered because of too much test automation. He based his conclusion on the information from a blog noted for its petty bitching about things inside of Microsoft. (Let me say this...there are a lot of things about MS and corporate policy that I don't like. Bitching about those things is easy and usually non-productive. But, working with the diversity of people who make up one of the most influential software companies of our time to effect change is difficult and challenging. I mostly choose the later route.) And the non-productive bitch-fest blog based its disinformation on the observations of Chris Pirillo's blog post written in May of 2006 exposing several critical issues such as:

• Shouldn't the phrases used during the setup state (Copying Windows files, Expanding files, etc) be intercapped? i.e., Copying Windows Files, Expanding Files, Installing Features, etc.?
  [Uh....no. Check with the terminologist and documentation experts]
• The "You're Ready to Start" graphic flickered and jumped when I pressed the Windows button.
  [Does it happen every time, or was it an anomaly, or did was it perhaps an electrical or wind storm...because here in Washington when we get wind storms my lights flicker all the time.]
• Control Panel>Additional Options displays...get this..."There are no items to display." Well, if there are no items to display, then why did you show me the link in the first place?
  [Perhaps because when user's install Additional options such as the Java Control Panel, they have a common place to access them]
• The Undo and Redo button for the Photo Gallery Viewer are at the bottom of the window, whereas I would've expected them to be at the top. Thought they weren't there until I looked hard.
  [ Is this inconsistent with other Task Panes? Or is this based on your personal preference? BTW...I happen to like them at the bottom no cluttering up other stuff.]
• What if I don't like the red "X" close button? Why can't I change that to...purple?
  [Great suggestion that I am sure millions of user's will appreciate. I think they have that scheduled as a hotfix somewhere around the turn of the next century]
• Microsoft Windows Mail splash screen needs an overhaul
  [Great...I am sure the customers who really want less functional defects or better performance will be more than happy to know we instead decided to spend time overhauling the splash screen based on your valuable input.]

This is not to say that Chris' feedback was ignored or not valued (I also enjoy reading Chris' blog from time to time). Personally, I would like to think we investigate a lot of the feedback we get from our customers (we simply don't act on all of it because...we'll we have bigger problems to solve other than allowing users to change the color of the close button.)

Spolsky stated, " The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one. And, None of those things could be checked by automated scripts."

Unfortunately, I guess Joel hasn't remained abreast of advances in test automation and perhaps assumes that automation is still primarily record/playback or simple rote scripts in some proprietary or limited scripting language. As software becomes more complex, and as the capabilities of test automation improve the fact is that quite a lot can be done with automation.

For example, checking the consistency of fonts in each dialog throughout a product is more effectively and efficiently accomplished via automation than by human eyes. Automation can also detect differences in the alignment of controls on dialog boxes and also detect clipping and truncation of controls or text in controls more efficiently than humans (and we actually have empirical data to prove it). Spelling and politically sensitive words are more efficiently detected via automation. Most testers and developers that I know are not experts in English grammar and I would never hire an English language major as a tester just to test grammar. We made our documentation/content experts responsible for messages and other 'string' content thus pushing quality upstream!

(Also, as a side note...the majority of the testers that worked on Windows Vista were the "old testers at Microsoft" doing they same work they did shipping Windows Xp.)

Isn't if funny that when certain things don't meet our personal expectations (Vista sucks), or we are fearful of change (test automation is evil and we should continue to hire hoards of non-technical people to bang on keyboards) it is so easy to jump to faulty conclusions, base assumptions on unfounded hearsay and rumor, constantly utter petty unproductive complaints, and generally play the victim. Of course, being a victim provides you with attention from others, and useless bitching and wild speculation is way more fun than facts and research...besides...it makes people laugh, and when people laugh they are happy, and the role of the tester is to make people happy...right?

Putting the Context back in Context-driven testing

Tonight I was having dinner at the superb Italian restaurant L'Olivo along the canal in Nyhaven, Denmark with my 5 year old daughter. She was slightly upset because they didn't have chocolate mousse on the dessert menu so when the waiter came by to take our order she neglected to say 'please.' I reminded her that we should always say 'please' and 'thank you.' She asked, "Daddy, why do we need to say please and thank you?" I told her that is was the polite thing to do, and that whenever she asks someone for something she should say 'please', and when someone presents her with something she should say 'thank you' or 'no thank you' if she doesn't want it. She paused for a moment, then said "But, if it is an emergency then I don't have to say 'please'...right?" Amazing! Not only does she understand the context in which it is polite to say please and thank you (and she usually is polite) she also understands and was able to clearly articulate one specific situation that it is not necessary to say that because of a different context. I, of course, had to agree with her that in that context she did not have to say please or thank you.

Unfortunately, not all conversations with some people go as well from my perspective. For example I had the following discussion recently with a reader from a self-proclaimed member of the so-called context-driven school of testing:

[Reader] "We never know what will happen if we do "X" with the software and can never be sure about all possible outcomes."

[Bj] "In your example above "We never know what will happen if we do "X" with the software..." is true. Of course we won't. Because there is NO CONTEXT!"

[Reader] "Because there are so many contexts possible, hence the answer would depend upon who is asking and in what context. Context driven software testers are aware of this and constantly practice probing around context."

[Bj] "Since you made the statement "We never know what will happen if we do "X" with the software...", I am asking for you to define the context of "X" and the software."

Unfortunately, this is where the conversation ended. So, without providing a context (or when taking words out of context) then I agree there are "so many contexts possible" and there are also contexts that are completely impossible or improbable. I have nothing personally against people who wish to label themselves, or identify themselves with a group "whose thought, work, or style demonstrates a common origin or influence or unifying belief." I personally don't believe in the so-called schools of testing, and previously explained my positions in the following posts, End the segregation of the four schools of testing, and Schools of Testing Revisited. I think software testing is difficult and challenging and one must constantly view the problem from multiple perspectives within the set (all realistic possibilities) of circumstances or facts that surround a particular event or situation.

For example, I know that a 0x5C character code point (the backslash) on a Windows operating system is a reserved character code point and cannot be used as a character in a file name. In a completely different context such as using a 0x5C character code point in a UNC path then that character code point is a valid character but only within specific limitations. So, I won't waste my time or my employers time with wild speculation and "practice probing around" searching for bizarre unrelated contexts. For example, I won't attempt to see if the 0x5C character code point can be used in the 5th and 19th character positions of a file name on a Texas Instruments 99-4A computer, or design a test to check whether it can be used as part of a file name if I set the computer internal clock to December 13, 1901. I could try these things (assuming I could find a working TI 99-4A) but, these are random and meaningless contexts.

At a recent conference a speaker asked "what is an integer?" I thought...hmm...an integer is a positive or negative whole number or zero. Then the speaker said, an integer is different depending on whether we are talking about the C# programming language, the Ruby scripting language, or an embedded device. Then I thought...well, an integer is still a postive or negative whole number or zero, but given more specific context I suspect the speaker was referring to data types in programming languages for the C# and Ruby example. In the C# programming language an integer is an intrinsic data type. In Ruby it is a class, and on an embedded device...well it is a positive or negative whole number or zero. (I must admit, the embedded device threw me for a moment because including embedded devices in a comparison set of programming languages seemed analogous to comparing a rhinoceros to apples and oranges (the only similarity is that they all have skin). So, in this particular situation in which we are given a more specific context to clarify the key word 'integer' based on "the parts of a written or spoken statement that precede or follow a specific word or passage, usually influencing its meaning or effect" we know that the speaker is referring to integers as data types on a programming language, a class object on a scripting language, and well...an integer on an embedded device.

Context denotes "the parts of a written or spoken statement that precede or follow a specific word or passage, usually influencing its meaning or effect" or  "the set of circumstances or facts that surround a particular event, situation". This is important because the English language is composed of words that have various definitions depending on how the word is used in a written or spoken statement. Unfortunately, when I have discussions with people who identify themselves with a singular way of thinking they often take one of the key words from the statement,  apply an alternate definition of the key word or element of speech, and then attempt to rephrase my original statement resulting in a tangential interpretation of the original statement or discussion because it is completely out of context!

For example, someone suggested in an article that the phrase 'best-practice' is an idiom (an expression whose meaning is not predictable from the usual meanings of its constituent elements, as kick the bucket or hang one's head, or from the general grammatical rules of a language, as the table round for the round table, and that is not a constituent of a larger expression of like characteristics). I challenged their use of the word idiom to describe the phrase "best practice" and suggested that it is actually an adjectival phrase. They replied that an adjectival phrase can be an idiom and and idiom can be an adjectival phrase. Yes, that is true when there is no context! For example, let's take the phrase "lame duck." In one context it is an adjectival phrase that may refer to a duck with a broken leg. In a completely different context the idiom "lame duck" refers to someone who is incompetent. I happen to think a "best practice" is an adjectival phrase that describes a practice which is better than similar comparable practices in a specific situation; I am still waiting for the commonly understood idiomatic inference of "best practice."

Dissecting every word in every statement for the sake of questioning the various connotation of words, or attempting to rephrase sentences based on limited knowledge of the domain space in which words are used in specific contexts (which I perceive is what is meant by "who is asking"), or thinking tangentially by removing or applying unrelated contexts is simply playing with words, and fun for debate from a philosophical perspective. Philosophy involves a rational investigation of 'things', and is extremely useful in that it helps us think of 'things' from various perspectives. However, the statement "...there are so many context possible (for what will happen if we do "x" with the software), hence the answer would depend on who is asking and in what context" is completely irrational because the person has removed the context from their own statement. Thus, it is impossible to perform any rational investigation of "x" because the speaker is unable to provide context for "x" or for the software which they are applying "x." Philosophy is really about asking questions; but philosophical discussion rarely provides any facts or empirical evidence that decision makers rely on to make critical business decisions.

But, as a professional tester I need to put things in realistic and meaningful contexts which are appropriate for the task. I also need to think of other possible rational contexts that I can clearly articulate. I need to not only investigate software and the capabilities, attributes, and limitations of that software from various rational perspectives, I also need to provide my managers with valuable and substantiated information which they can use to analyze risk and make better qualified business decisions.

I happen to think we have always viewed software testing in various appropriate contexts (hence context-driven is nothing new or revolutionary). So, instead of playing with words and offering alternate and unrelated contexts of a sentence element or situation, I think we need to drive the context of context-driven to mean focusing on the appropriate conte