wltester

Overcoming UI Automation Challenges

2012-03-06T02:37:52Z

UI automation isn’t an easy task, as any UI feature test owner who is responsible for automating their areas can tell you. It is quite challenging in terms of getting more test coverage and higher test reliability from automation. Let's discuss these challenges in more detail.

From the test coverage perspective:

Each automated test consists of two parts: the code to drive the product and the code to validate the test result. Many product teams have a coverage goal of 70% or higher. Without help from developers' unit tests, it is actually quite difficult to achieve this goal by UI automation tests alone. Here are 3 main reasons: test hooks, result validation, and cost/complexity.

1) Some UI controls can't be seen (or found) by the UI automation framework being used. This is either due to some technical limitations of the UI automation framework or the accessibility support for the UI controls not implemented properly in the product code. Either way, this often prevents common tests from being automated until the problems are resolved.

2) Test validation often is not easy, especially for UI tests or UI changes that require manual verification from testers. Many teams are trying to solve this problem by developing tools or using techniques like screenshot comparison, video comparison, etc. But these tools can't replace human testers very well because they usually cause higher rates of false positive or false negative results in automation tests. As a result, many UI tests like these won't be automated.

3) Complex UI test scenarios have more steps involved, which potentially adds more cost to the automation effort and may introduce more instability to the tests. From an efficiency perspective, sometimes it is much cheaper to run the tests manually rather than spending time automating them, especially when there is a short product cycle. Also, it is difficult to manually trigger most error messages and difficult to automate them. For this reason, we are not able to achieve the test coverage we want from our automated tests.

From the pass rate (reliability) perspective:

Ideally, tests fail when the product breaks. But this is not always the case for complex projects. Sometimes no matter how hard they try, test owners still need to spend time fixing their tests as part of their daily routine. This is, of course, a headache that they try to avoid. Unfortunately, there isn't a quick or easy solution to remedy this. Here is why:

1) Timing issues are one of the most common and annoying problems that we encounter in UI automation. The product code and the test code usually run in different processes, which sometimes results in their getting out of sync. It isn't easy to handle or debug a timing issue. Some testers run into situations where their test continues to fail although it has been fixed over and over again. Every time there is a UI state change in a product, it creates a potential timing issue for UI automation. To handle a timing issue properly in test code, waiters (UI event listeners), ETW events, polling, or even "sleeps" are often used. Since timing issues can be difficult to reproduce, it might take a few tries before the root cause is determined and the proper fix is applied.

2) Each UI automation test is designed and written based on the expected product behaviors, UI tree structure and control properties. So if a developer changes a product behavior or UI layout, or modifies a control property, that could easily break automated tests. When that happens, testers will need to spend time to debug and fix their test failures. If there is an intermittent issue in the product, debugging the test failure to figure out the root cause is even tougher. In this case, test logs, product logs, screenshots and videos for the failing test state are needed for debugging. Sometimes, additional test logging is required to help debug the problem. In the early product development stage where the product and test automation are not yet stabilized, the test code is always assumed "guilty" by default until proven "innocent" when there is a test failure.

3) We all know that there is no bug-free software out there. So it shouldn't be a surprise that issues from the operating system, network, backend services, lab settings, external tools, or other layers account for many automation test failures. Since those are external issues, testers have to implement less than ideal workarounds to fix their automation failures, which is very unfortunate. Please keep in mind that workarounds for external issues don't always work well because they often band aid problems only.

UI automation remains one of the biggest challenges and a significant cost for test teams in product groups in the upcoming years. Below are some proposed ideas on how to mitigate the UI automation challenges discussed above:

1) Choose a UI automation framework that fits your automation requirements. Four major criteria can be used when testers write prototype tests during the framework evaluation.

- Performance: how fast your tests can be run

- API Supports: how easily you can build your own OMs and tests and how many major test scenarios can be automated with the framework

- Reliability: how stable your tests can be (are your tests flaky because of the framework?)

- Maintainability: how easily you can update your existing test OMs when there is a UI/behavior change in the product

2) Work closely with your developers to add testability (such as accessibility supports and automation IDs for UI controls) needed by your automation. Searching for UI controls by their automation IDs is usually the preferred way because that can reduce some localization support for non-English builds.

3) Make sure to use a waiter or other waiting logic every time there is a UI state change or a timing delay in the UI after triggered by a test step.

4) Understand your product behaviors and pay attention to detail. Again, UI automation tests are written based on expected product behaviors. In other words, any unexpected product behavior could potentially break your tests. Your tests will become more reliable if you can properly handle product behaviors in various situations.

5) The cost for UI automation is actually a long term investment and isn't cheap. So do not try to automate every test. We can't afford to do that anyway since we always have other tasks to do. Before you decide to automate a test, you should ask yourself a couple of questions first: is it worth the effort to automate this test? What is the gain vs. the cost? Basically, almost everything can be automated, the difference being the cost. Here are 3 areas that we can start first: a) BVTs (Build Verification Tests) and performance tests; b) common P1 test cases; c) test cases that provide high coverage with reasonable cost (under 2 days). After that, then we can continue to automate more test cases if time allows.

6) Find and leverage (maybe develop) more tools to help validate results for UI tests.

7) Identify the root cause of a test failure and fix it completely in the right place. Do not just band aid the problem. For example, try to remove or not use SLEEPs!

8) If a test failure is caused by an unexpected product behavior, push the development team to fix the issue. Do not try to work around the issue in test code as that will hide a product bug.

9) Write less automated tests and focus on achieving a higher test pass rate in the early stages of product development. That will help reduce the cost for test case maintenance. Once the product is stabilized, then write more test automation and focus on the code coverage goal.

10) Abstract test cases from UI changes to reduce the test maintenance cost. Please see the article on Object Model Design.

Wei X. Feng, Microsoft

Service Upgrade Testing

2012-01-06T01:17:46Z

To bring the windows live experience to life, we host a variety of interconnected services. These are large scale services which run on thousands of servers, and due to their scale cannot be upgraded atomically. One of the challenges is making sure that when one of these services is upgraded, the rest of the system works properly. Upgrades to the full site require a coordinated test effort that often involves running one service against multiple versions of another. To ensure that upgrades go smoothly, Windows Live has several strategies to make sure that all the versions to which our users will be exposed during a release are covered.

Keeping the state throughout upgrades

When testing an upgrade in a stateful system like ours, we need to think about how the new version of the software will interact with preexisting state and data. It is a common mistake to only test a service with data created using the new version, when existing customer data has been around for much longer and has gone through multiple upgrades and migrations. The question is how we replicate this state in our labs, particularly using our automation. Some of our approaches are the following:

Know your data scenarios: One strategy is having a list of the relevant user data scenarios and having multiple instances of each of them provisioned in our lab. We can then check on the state of the data throughout the upgrade. Without knowing the states that the user can be in, it is hard to make sure that they will be working on the new version.
Build automation for upgrades: Generally test automation has a setup, execution and verification stage. However, when defining tests for upgrade scenarios, we can think about pre-upgrade setup steps and post-upgrade validation steps. We can then structure tests so that, before the upgrade begins, we run the setup step, and then on each step of the upgrade we run any number of validation actions. Building tests designed for upgrades gives us the ability to automate the process of testing an upgrade.
Run a practice upgrade: It is always good to have an environment in the labs which has gone through the same upgrade steps as production has, with accounts that have gone through the same migration process. This often lets us detect unexpected issues that we did not catch through our automation.
Test throughout the full upgrade: Beyond having tests that run after each upgrade step, another strategy is to have tests that are continuously running during the upgrade. These tests start with an existing user state, and execute operations with corresponding verification steps as the upgrade runs through. This lets us find issues that may only happen when the upgrade is actually running.

Another interesting issue that comes when upgrading stateful systems is when old versions of client software cannot support state created with new versions of the service. Testing an upgrade that covers both client and server scenarios requires us to consider the version matrix and test each combination. Often this matrix is reduced by having a way to enforce that a new version of the software client cannot run against an old version of the service code, which means that the transition that we need to test is:

Service: Vcurrent
Client: Vcurrent

Service: Vnext
Client: Vcurrent

Service: Vnext
Client: Vnext

Testing strategies when upgrading multiple services

Upgrading a single service in the cloud has its own level of complexity, but in practice we are often upgrading multiple services at a time. Common issues during upgrades happen when one service is rolled out sooner or later than expected. It is thus important to be able to build a timeline of the full upgrade to understand the dependencies that one service will have on another. The best would be to be able to replicate this upgrade procedure on a production like environment, then running end to end application tests. However, this is not always easy to coordinate and execute. It is generally easier to list the different versions of the services that we are planning to run against during the upgrade. With this information, we can build a test matrix to ensure that we are clear when running against any of those versions.

One thing to keep in mind is that upgrade test procedures often rely on moving a test lab back and forth from one version to the other. How easy it is to execute upgrade testing depends entirely on how easy it is to set up a lab environment to a particular state. If you work on a system that is constantly running through upgrades, or has a complex upgrade matrix, investing in easy and quick deployments will be crucial in being empowered to test and find upgrade related issues.

In conclusion, upgrade testing requires thinking in terms on versions of services and clients, and thinking about the different states that a user may be when the process begins, executes and completes. Upgrade tests are often more involved and sometimes hard to execute; However, they are crucial to ensure that our customers will have a good experience when we offer them the new version of our service. Even if customers may be exposed to the system in upgrade for a small period of time, a small issue can lead to issues like corruption which can cause long term issues. Upgrade testing is often overlooked, because it is hard and because it is transient. However our customers always deserve the highest quality service, and that includes the periods of time when we are upgrading.

Federico Gomez Suarez, Microsoft

You write too many tests and open too many bugs

2011-12-20T18:42:25Z

In my experience, many testers tend to identify with certain types of measurements and metrics which on the surface may seem to be desirable to maximize. These statistics can provide feedback to the tester to indicate that they are making progress and getting something done through the course of their work. However, when one looks below the surface, “overachieving” within these statistics may in fact be an indicator of an unhealthy software development system. Let’s take a look at a couple of tester-related statistics which I believe are worthwhile to scrutinize: Number of Tests, and Number of Bugs Found.

Number of Tests

I think many agree it is good to provide regression coverage by authoring stable automated tests – it provides a way to repeatedly check that the system has not deviated from a well-known “good” state. As part of the daily job of the SDET, writing test automation is something that most of us do on a regular basis. And heck, if writing one test case to provide coverage is good, then writing 100 cases must be better, right? This line of thought unfortunately seems to be quite common. I have seen multiple instances of testers being proud of the sheer volume of tests they added to the automation system:

“As part of my feature, I added 450 tests to the BVTs, isn’t it great?”

Not necessarily :) One should consider the ramifications before claiming that “more is always better”:

Tests written as part of automation suites are usually long-lived and passed down to subsequent owners, and the TCO of each test case is usually (much) greater than the initial cost to author the case. Each test written has maintenance costs that are compounded over the entire lifetime of the test, this includes such items as:

Triage of test failures
Changes of test code to accommodate design changes

Fixing unreliable/unstable test variations

Reporting test results

Etc...

When multiple tests provide duplicate coverage, the time spent on maintenance of duplicated tests is a sunk cost which prevents that time being spent on other areas with low (or zero) coverage

What then should be the tester’s goal when writing automation? In my opinion, testers should shoot for the “Minimal Maximal Set”, which is coined to describe the minimal set of test cases which provide the maximum amount of coverage. If you can generate a set of 2 robust test cases which provides the same amount of coverage as another set of 100, the TCO for the minimal set will likely be far less over the lifetime of the cases and will be net positive in terms of team efficiency.

Number of Bugs Opened

Testers find bugs; it is part of their DNA. On the surface, it is great to see testers find many defects that would otherwise make their way to the customer. It is also great to see testers to carry a certain pride in being able to uncover bugs in the process of their work. However, this is again a case of “more is [not always] better”. Some types of bugs found again and again can indicate intrinsic problems in the way that we develop and test the system. For example, consider the following:

A large quantity of BVT/Pri0 types of defects being found repeatedly can be an indicator that proper defect prevention procedures are not sufficient to catch low hanging fruit before developers check in their new code. In essence, teams should never be in a situation where BVT blockers are happening that could have been easily prevented by a check-in test or code review procedure. We should push to find/prevent issues as early as possible in the software development process as it is the most efficient way to ship a quality product
A pattern of low quality bugs (for example, a preponderance of "won't fix" or "not repro" bugs) may indicate that the test team is wasting the feature team's time with non-issues. Sometimes this can also indicate that the test team has the wrong priorities in mind when testing the product
Large numbers of base-case functional bugs found late in the milestone may indicate a disconnect between the functional requirements and assumptions that the test team is making
Patterns of similar defects being found milestone after milestone can indicate that the feature team is not learning from past mistakes. It can also indicate that proper analysis is not being performed against the defect patterns at the end of the individual milestones in order to prevent the problems from happening again in subsequent milestones
Categories of bugs being missed may indicate an overall lack of attention to fundamentals like security, stress, performance, globalization, etc.
Large numbers of integration bugs being found late in the cycle can hint at design problems between intersecting components

Again, what should be the goal here? While it is a fact that good testers can find lots of bugs, I believe that we should look far beyond the numbers and ask ourselves "What do these bugs tell us about the product?" and "What do these bugs tell us about the way we test the product?" Many folks tend to gravitate towards simple bug statistics because they are easy to measure, but unfortunately they don't tell us much in isolation.

Only when we look at what the statistics are really telling us can we begin to make meaningful progress...

-Liam Price, Microsoft

Just because you can test it doesn’t mean you should

2011-12-13T17:59:00Z

A wise test lead once said to me, “do as little as possible while still ensuring quality”. He wasn’t giving me tips on how to be a slacker =) I think what he wanted to say was there always seems to be more work than there are people so it pays to be as efficient as possible, especially when prolonged testing can hold up shipping a product.

One thing I’ve seen at Microsoft is sometimes the tendency to want to test everything, which isn’t a bad thing as long as you don’t have to ship =) Hence the title – “just because you can test it doesn’t mean you should.” If say you have several components on a stack calling each other, it makes sense to just test at the highest layer to automatically get coverage for lower components. Some people also call this end to end testing. When you test each component separately, you may have the following issues:

Duplication of effort. Tests of lower level components doing the same testing as end to end tests.
You may not be testing a component the way it will be used in the product. For example, your tests call an API passing it X, but the caller in the product will only ever call the API with Y.
Holes in coverage. Because you’re testing components in isolation, you may miss main stream customer scenarios.

So my recommendation?

Test at the highest layer whenever possible.

If the top layer is not yet available, then temporarily test the lower layer with the intention that they will be replaced with end to end tests.
Focus testing on main customer scenarios first taking a top down approach. This maximizes coverage and at the same time focuses on scenarios customers will hit first.
Use code coverage tools to drill down on coverage of components. If code is not being used by top-level components then question whether the code should be there at all.
Push devs to do unit testing of their components.

Of course there are times when component testing is called for.

Component code cannot be easily hit by end to end testing. An example is negative or error testing.
A component is consumed by multiple clients or it's a public API that anybody can consume. In this case you should treat this component as the highest layer.

Performance tests should ideally go through the top layer and each component logs their own performance numbers. But if you want to test things like throughput it might make sense to go through the component directly.

Any testing where going through the top layer would just be too slow or not possible like security testing, stress testing, fault injection, etc. For example, you may need to ensure an app can open 100K different files for fuzz testing but going through the UI would be painfully slow.

What about MVVM (Model View ViewModel) you say? This is where you have a layer (ViewModel) that encapsulates data and functionality needed by the UI allowing you have to have a very thin UI layer, even making it easy to replace the UI layer if needed. Some folks prefer testing against the ViewModel as opposed to going through the UI. As somebody who has done both, I can tell you that testing against the ViewModel is much easier and UI testing can be a pain. But in my opinion, the easy way is not worth the risk. We saw several bugs slip through because the thin UI layer that supposedly didn’t have any bugs of course did. Going through the ViewModel has its uses like expediting certain operations but I don’t recommend exclusively testing against it.

So to close, test end to end first, and component test only if you need to. Lazy testers are efficient testers, unless they’re trying to outsource their work on craigslist =) Feel free to comment if you have any thoughts on this topic.

-Samson Tanrena

Creativity in Testing

2011-12-07T21:55:00Z

One of the wonderful parts of testing software is that beyond the pure technical aspects of the discipline lie many opportunities for the tester to exercise their creativity in potentially game-changing ways. Although we work in a field where some of the quality assurance tasks are handed down to us from previous owners (legacy test suites, existing test cases, etc.), it is important to note that it is possible to break from the norm and provide real value by using approaches that have never been used before in the areas which we are testing. Sometimes we can see greater gains not from working harder, but by simply thinking differently. Some of this takes courage, as it is hard to differ from some of the time honored approaches. However, as my experience grows in the field, time and time again I find that the testers who distinguish themselves are not afraid to try new techniques and approaches, and are always looking for creative ways to break the product and new ways to improve. Sometimes these new techniques end in failure, and that is ok. Sometimes the new techniques spur on measurable improvements in the ways that we look at and validate the quality of the software.

There are times when testers are viewed simply as “test execution machines”, but great value can come out of looking beyond what is scripted or specified and going “beyond the tests”. What we do is more of an art than a science, and that is what keeps it interesting to me personally. Sometimes it can feel great to hear the words from your teammates "How did you think of doing that?" J

What are some ways that you are able to spark your testing creativity and/or find inspiration? Here are a few ideas that we have used in teams that I have worked in.

Randomly pick two features in the same product and look for all intersections between them; test those intersections
Use the software without a mouse
Test on extremely low-end hardware or extremely low bandwidth network connection
Learn from all of your predecessors. Look at old bug reports, you will find a wealth of untapped information and ideas which can help you with new approaches. Also look at reports from unrelated areas…
Use the software with your "off" hand
Take your work outside J Test outside, on your laptop
Gain inspiration from other disciplines. Sometimes use of creativity in art, music, business, athletics, etc. can help influence creativity and provide a spark in how you go about testing. Research your favorite musicians, athletes, carpenters, humanitarians, and see what inspires them, see what makes them tick. As an example: Jazz musicians will frequently engage in improvisational music in which the instrumentalists feed off of the spontaneous melodies of their band-mates. This same technique can be applied to testing…get a small group of folks in a room together and have them “group-test” a product, supplying little structure/guidance to the testing. Allow the testing to bounce from one area to another and see where it takes the group…sometimes the results can be very interesting…
Test a feature side-by-side with a competitor's offering

Go for a quick jog, and brainstorm test cases along the way. You will remember some of them when you get back J

Use the software for a day with a very large font setting
Write a test suite which generates data based off of customer interaction patterns, gathered by telemetry
Artificially constrain your time – role play the scenario where you only have a single hour to ensure the quality of your feature. What would you test? What would you try? Where would you spend your time?

Have your children and/or parents try to break the product. Watch them as they use it, and note any usability or functional concerns

Use the wisdom of others to help test your software. Have a large population of engineers vote on whether particular features work correctly or not. Use this data to target testing deep-dives

Share with others – sometimes just talking through some of the more interesting bugs you have encountered with a few folks informally can start important testing collaborations

Have someone else give you a list of things that are important to them in the software, and then use it as a starting place for your own exploration
Try model-based testing in an area which has never been tested that way before
Without looking at specifications, come up with a list of "Ten things that must work" for your feature/area; verify that these things indeed work
Test someone else's area while they test yours
Step through the code in the debugger while you are testing a scenario

Read the code flow itself while you are testing a scenario

Tape a golf ball to your keyboard (yes, this was done J )
Test while listening to mariachi (or speed metal, classical, jazz, etc) music
Etc.

Comments/Questions are always welcome. Please contribute/share your own examples of inspiration and/or creativity. Thanks for reading!

-Liam Price, Microsoft

Testing In Production

2011-11-28T23:48:18Z

One of the most useful resources for a test engineer dealing with web services is the production environment. This is the live environment that exposes the product to end users. Some of the challenges that the production environment provides us are the following:

How do we know that software that works in a developer box or on a test lab will work in a production environment?

What information can we gather from production that will help us release a higher quality product?

How do we detect and react to issues found after a software upgrade?

In this blog post, we will look at some of the strategies that can be used to improve quality by incorporating the production environment and production data into our testing.

Smoke Testing in production

Some bugs appear more readily in production due to discrepancies between the test and live environments. For example, the network configuration in a test environment might be slightly different from the live site, causing calls between datacenters to fail unexpectedly. One possible way to identify issues like this would be to perform a full test pass on the production environment for every change that we want to make. However, we don't want to require running a full suite of tests before every upgrade in the live environments since this would be prohibitively time-consuming. Smoke tests are a good compromise, as they give us confidence that the core features of the product are working without incurring too high of a cost.

A smoke test is a type of test that performs a broad and shallow validation of the product. The term comes from the electronics field, where after plugging in a new board, if smoke comes out, we cannot really do any more testing. During daily testing, we can use smoke tests as a first validation that the product is functional and ready for further testing. Smoke tests also provide a quick way to determine if the site is working properly after deploying an update. When we release an update to our production environment we generally perform the following steps to validate that everything went as planned:

Prior to updating the site, we run some tests against the current version. Our goal is to make sure that the system is healthy and our tests are valid before starting the upgrade.

We then update a subset of the production site. Preferably, this portion will not be available to end users until we complete the smoke test.

Next, we run the tests against the updated portion of the site. It is important to have clarity on which version the tests are running against. We should have a clean pass of the smoke tests before we proceed. If we encounter problems, we can compare the results pre- and post- upgrade to help focus the troubleshooting investigation.
Continue the rollout to the rest of the production environment.

Finally, run the tests again to validate the entire site is working as expected.

The tests used for smoke testing should have the following qualities:

Smoke tests need to be very reliable, as a false positive may cause either unnecessary false alarms or a loss of trust in the smoke test suite.
Smoke tests need to be very fast. The main point of smoke testing is quickly identify problems, and long running tests can either delay updates to the site, or potentially allow users to access buggy code before the tests catch it.
Smoke tests need to be good at cleaning up after running. We need to avoid having test data mixed in with real customer data since tests can potentially create data that isn’t intended to be processed in production.

Windows Live uses an automated smoke test tool which is able to do a validation of the service within a few minutes. The same utility is used in developer boxes, test environments and production, and is consistently updated as new features are added to the system.

Reacting to issues through data collection and monitoring

Even though we may have done thorough functional validation, shipping a new feature to production always implies a risk that things may not work as intended. Logging and real-time monitoring are tools that help us in this front. Before shipping a new feature to production, try to answer the following questions. This will give you a sense of readiness for handling issues:

How will you know that users are having issues with the feature? Will you mostly rely on user feedback, or will you be able to detect and measure failures yourself? Will the people running the site be able to tell something is wrong?

If a user raises an issue, what are the resources that you will have available to investigate? Will you require the user to collect detailed logs?

In the event of an issue, are your test libraries prepared for quickly building up a test scenario based on a user's feedback? The ability to craft tests based on logs and the user’s repro steps generally indicates how long it will take for someone to reproduce the issue and validate a fix, which has a direct impact on the time to resolution.

Some of the strategies that windows live uses for allowing quicker reaction to issues are the following:

We allow user initiated log collection, both on the client and on the server side. Taking the product group out of the critical path when collecting data saves the team a significant amount of time and effort.
We support using the user logs to craft tests for reproducing an issue. Our tools take logs, remove any actual reference of the users data contents, and replay the traffic.

Using production data as input for tests

The involvement of Test in production should be limited to releasing a new feature or investigating an issue. Production contains a wealth of data that helps us better define what to test. The higher priority tests are those that map to the core customer calling patterns, and for existing scenarios, production data is the best source. Some of the interesting questions that production data analysis is able to answer are the following:

What are the different kinds of users in the environment? What are the characteristics that identify them?
What are the most common calling patterns? Which ones most frequently cause errors?
Do the site’s traffic patterns indicate changes in user behavior?

Gathering and analyzing data to answer the above and other questions is often non-trivial, but the resulting data is invaluable, particularly when deciding which areas should have a bigger focus when testing.

Within Windows Live, we have used this approach to understand both user scenarios and calling patterns. We measure some of the characteristics of the data (like how many folders a SkyDrive has, or how many comments photos typically have) to identify both common scenarios and outliers. This data lets us focus efforts like performance testing and stress on the most common scenarios, while ensuring that we have coverage on the edge cases.

When using production data in testing, the approach to privacy is extremely important and needs to be figured out before starting the work. Our tools only interact with abstractions of user data, with all actual user content and identity removed. We care about what the data looks like, not specifically what the data is.

In conclusion, the effectiveness of a test engineer can be enhanced by using production as a source of information. It may be by making sure that all the core scenarios work as expected through smoke testing, by creating a quick mechanism for reacting to issues, or by harvesting data to feed into test tools and plans.

References

[1] http://en.wikipedia.org/wiki/Smoke_testing

Object Model Design

2011-11-14T18:34:29Z

Recently I was asked my opinion on creating object models for automated UI software testing – do I prefer LFM or POM? There is no right or wrong answer here – because I own a framework for automating Web UI, I’ll speak to this topic from that perspective. POM (Physical Object Model) style models expose the controls on the page and abstract away the details of how those controls are implemented. LFM (Logical Functional Model) style models expose methods that automate the actions the user can take and abstract away all of the details of how those actions are performed. Here is a concrete example that shows how one would write a test case to initiate a search on Bing’s home page:

LFM Style:

Logical.Bing.Search(“Michael Cartwright”);

POM Style:

Physical.Bing HomePage.SearchTextBox.Text = “Michael Cartwright”

Physical.Bing HomePage.SearchButton.Click();

I feel that the “best” object models expose both – what do I mean by “best?” The notion of best must start with the idea that the best object models are the ones that maximize the number of real product bugs found while minimizing the “cost.” What are the major “costs” of an object model? There is:

Cost of an Object Model

1. Up-front cost of writing the model

2. Updating the model as the product changes

3. Writing test cases that use the model

The object models that minimize cost over time minimized all three costs - it is cheap to create them, keep them updated as the product changes, and its also cheap to write tests. One approach to achieving this end is to first expose POM style controls classes that encapsulate all of the controls on a page. Once all controls are automatable, take any actions that are taken in more than 1 test case and create a LFM style action-based method for that action. The method will internally use the POM style controls so that there is not code duplication. So what does this look like?

// A class to enable automating the BING home page

public class BingHomePage

{

// A method that will execute a search. Multiple test cases perform

// this action, so we wrap it into a re-usable method

public void Search(string searchString)

{

this.SearchTextBox.Text = searchString;

this.SearchButton.Click();

}

// All of the controls on the Bing home page

public HtmlTextBox SearchTextBox { get; set; }

public HtmlButton SearchButton { get; set; }

}

How does this minimize cost over time?

2 ways:

1. No duplication of code

a. The two steps required to search are in one place - inside of the Search() method in the BingHomePage class

b. The code for finding the Search text box occurs in one place - its inside of the SearchTextBox property in the class BingHomePageControls

2. Resilient to product changes

a. If the search text box location changes, the only object model change will be to the controls class where the text box is defined. The test cases and action methods in the model will remain unchanged.

b. If the steps required to perform a search change, but the controls on the page do not change, then the tester will only need to update the body of the “Search” method, and all test cases using this method will remain unchanged.

Conclusion

While I know that there is no “right way” to write object models, I’ve used this style of model in the past and it served us well. It was our approach to eliminate duplication of code. The rule of thumb we followed was to use the reusable LFM style methods for any actions occurring more than once, and use the POM controls for 1-off actions that only occur once. There is significant cost in creating and maintaining this style of model but in my experience it minimizes long term maintenance cost my minimizing code duplication. Recently teams have been exploring ways to generate the controls classes in an automated way, saving tons of time. Please leave comments on this post as I would love to hear what style of object model your team uses, and what you think about this style.

Michael

Lots of good info in this post came from http://www.teknologika.com/blog/category/watin/

Playback Testing Using Log files

2011-11-07T22:36:19Z

Author

Vijay Upadya, vijayu@microsoft.com

Abstract

Most testers are familiar with record/playback testing. A variation of this approach is log playback testing. Instead of explicitly recording the user actions using a recorder, this technique simply relies on the information in the log file produced by the system under test to play back the scenario. This is extremely useful for reproducing issues reported by customers which are typically hard to reproduce otherwise due to lack of detailed repro steps. This approach also lends itself well to creating data driven tests and can be used to augment the existing test automation.

Testing and investigating failures in a non-deterministic software system can be very costly and painful. In most cases the only source of information to know what code path got exercised are the log files produced by the application. This paper talks about how the information in log files can be used to playback the sequence of actions to test specific code paths and also reproduce bugs. We will look at exploring an extension of this approach and how log files can be used as a source for data driven testing by simply varying the sequencing of actions that are played back. This paper illustrates the technique using real a world example. Finally the paper discusses the benefits and how it can be applied to different test domains.

1. Introduction

Log files are typically used for diagnosing issues reported by users of the product. In most cases log analysis is done manually where developers read the log and try to understand the sequence of events happening during the failure and where it went wrong. For catastrophic failures like crashes the call stack will mostly suffice in figuring out the issue in the product. But when there are functional failures, having a detailed log will go a long way in helping diagnose the issue quickly.

Now we can take it further and design logging with the intent of generating logs for easy auto analysis and use it to actually playback the scenario as it happened in user’s machine. In order for this to be successful, the logging format needs to be designed right from the beginning of the product cycle with playback in mind.

Log files are also a great source of information that shed light on how customers are using the product and information contained in them can be sourced as test cases that test teams can then run regularly during their test passes.

In the next sections we will discuss how log files can be used for playback and more exhaustive testing. We will use an example of file synchronization software ^[i] for the discussion.

2. Information needed for playback

In general log files for the application contain sequence of important events that happen along with information about data. Let us explore characteristics of a good log that is ‘playback ready’.

There are two main pieces of information needed for playback

User intent (i.e., scenario user intended to accomplish)
Path taken by product in fulfilling this intent (i.e., product's execution flow)

User Intent

User intent is about capturing what a user intended to do while using the product. It could be a simple task or a series of tasks as part of a larger scenario.

Examples:

User performing mathematical operations on a series of numbers in a calculator application
User copying photos from camera to local folder on his PC and syncing it to other PCs using file synchronization software.

A log file needs to provide enough information to make it possible to interpret the user intent. There are two items of interest for capturing user intent

Timeline of actions performed by user
Data passed to each of these actions

For simple applications like a calculator, just capturing user intent will probably suffice as the component behaves in a deterministic manner for a given input. For example, user presses 2 + 4= and calculator produces 6. The calculation logic is synchronous and execution flow deterministic. However for non-deterministic software like file synchronization where there are a number of threads executing in an asynchronous manner, and where the order of operations are not guaranteed, just capturing user intent is not enough. We need an additional piece of data – the path taken by the product or execution flow.

Product Path

Path taken by the product: When a user performs a specific task, the product itself will perform a series of tasks to accomplish the user intent. These tasks are typically implemented in multiple components and some of these could be executed in parallel.

There are two items of interest for capturing product path

Timeline of states and their transitions while executing tasks
Information on any concurrent execution of tasks
How to capture user intent

In general, in order to capture user intent the product needs to understand general user actions or entry points to the application. This would be happening in any functional product. For example, when a user presses ‘+’ in a calculator application, the application will ‘interpret’ it as an addition (user intent). In most cases it’s just a matter of logging this information at appropriate locations in the product where it ‘understands’ the user intent. The timeline is also an important piece of data that enables determining the ordering of operations happening in the product and helps identify any concurrent operations happening. The product needs to log the timestamp information indicating when this action occurred along with any relevant data passed.

4. How to capture the product path

This involves thinking through control flow of the product while realizing the user intent and would involve logging different state changes and their transitions. Again here timestamp information needs to be logged along with data. More on this is discussed in the next section with an example.

5. How to playback

There are two things that need to be played back - one is the user intent and the other is the control flow of the application. For simple applications, just playing back the user intent (using the same data and in the same order) would help in replaying the scenario and reproducing the bugs. However for more complex products that are non-deterministic in nature (meaning, for the same scenario it could take a slightly different code path due to its asynchronous nature) the product path also needs to be 'played back’ so that it converges on the path that caused failure in the user scenario.

Let’s take an example of file synchronization software. There are three main operations that users of this application typically perform – Creation, Updates and Deletion (CUD) of files and folders. Now these can happen in any order at any time on any number of files on any number of devices that are enabled for sync. The product need to understand each of these main user gestures (CUD) and it’s easy for it to log when it happens.

Looking into the control flow of typical file synchronization software while performing these CUD operations reveals that it comprises of the following sub-operations

Scanning for detecting changes,
Uploading (file to cloud storage),
Downloading (file on another machine) and
Realizing (copying) the file downloaded to right folder location so that the file appears on disk on other machine.

Some of these operations can happen in parallel asynchronously (for example, uploading and downloading).

Let’s say a user reported a ‘failure to sync’ issue and shared the log file which looks something like this (only relevant portion shown for illustration purposes) with the log format ‘TIME_STAMP’ ‘TASK’ ‘Data’

06-14-2011 00:08:46.507 SCANNER: thread for 'C:\Users\vijayu\SyncFolder' started

06-14-2011 00:08:46.617 SCANNER: FILE_ACTION_ADDED for 'C:\Users\vijayu\SkyDrive\ myPhoto1.jpg’

06-14-2011 00:08:48.703 SCANNER: thread finished scanning

06-14-2011 00:08:49.826 DOWNLOADIN file ‘myPhoto2.jpg’

06-14-2011 00:08:49.827 UPLOADING file 'C:\Users\vijayu\SyncFolder\myPhoto1.jpg’

06-14-2011 00:08:51.626 DOWNLOADIN file ‘myPhoto2.jpg’

06-14-2011 00:08:51.629 UPLOADING file 'C:\Users\vijayu\SyncFolder\myPhoto1.jpg’

06-14-2011 00:08:53.322 DOWNLOADIN file ‘myPhoto2.jpg’

06-14-2011 00:08:53.327 UPLOADING file 'C:\Users\vijayu\SyncFolder\myPhoto1.jpg’

From the above log snippet it’s easy to figure out that a new file myPhoto1.JPG was added, picked up by scanner and it’s trying to upload it. Also it’s trying to download a different file myPhoto2.JPG (on a different thread) at around the same time. Also this looks like the point of failure as log contains repeated section of the above uploading/downloading sequences. This might be an indication of the application getting into some kind of infinite loop and the issue surfaces only when upload and download happens simultaneously. This can happen when user has added files on both machine1 and machine2 and each is making its way onto the other machine. On the machine where failure is happening, the upload and download are occurring simultaneously resulting in some kind of a failure.

So to summarize we have following information:

User intent: Addition of file on two machines and syncing to all machines
Product path: Scanning completed, Simultaneous download and upload task started

Now for playing it back, the playback tool needs to detect these two pieces of information. From the line on scanning along with file name it’s straightforward to interpret that the user added a new file myPhoto1.jpg and this condition can easily be reproduced by adding a new file to the path specified.

The lines in log on downloading tasks along with file names indicate that there was newly added file on the other machine that it’s trying to download to this machine (where failure occurred). So to simulate this condition, the tool needs to add a file to a different machine to the path specified.

Examining the timestamp for upload and download indicates that these two tasks are happening (nearly) simultaneously. Putting it all together, these are the steps playback tool need to perform

Create file myPhoto1.jpg in machine1
Create file myPhoto2.jpg in machine2
Wait for download to start for file myPhoto2.jpg
As soon as download starts, also start uploading myPhoto1.jpg

User intent actions can easily be reproduced by adding a new file to the path specified on machine1 and machine2. However controlling the flow to ensure download and upload occurs simultaneously is a little tricky.

In order to accomplish this control over flow of execution our team used the detours ^[ii] library. In a nutshell the detours library enables hijacking function calls in the product and gives test code a chance to control its execution timings. The way this is used in the example case for file synchronization mentioned above is:

Create a test binary that detours upload and download functions by providing a detour function
Load this test binary to the product process
In the upload detour function wait until download is ready to begin
In the download detour function wait until upload is ready to begin
When the condition is met, continue the execution

How it can be used for test generation

The test team could use this playback technique to pro-actively identify issues before releasing the product. This can be achieved by taking an existing log for a given scenario (say produced by functional tests), interpreting the log to identify different states in the product and simulating all possible control flow conditions.

For example, in the above file sync case, on creating a file to sync folder we noted that product goes through four states. From this we can come up with the following three cases that can occur simultaneously:

Download during Scanning
Download during Uploading
Download during Realizing

The same playback tool can then be used to simulate the above three conditions and test the products behavior.

Conclusion

The log playback has the potential to make reproduction of issues less painful and cheaper. It does require initial investment to come up with the infrastructure for playback, but once it’s in place, it can rapidly enable test teams to test different conditions/code paths the product could take and also to reproduce issues reported by the users.

8. Reference

^[i] http://en.wikipedia.org/wiki/File_synchronization

^[ii] Galen Hunt and Doug Brubacher. Detours: Binary Interception of Win32 Functions,

Old World vs. New World

2011-10-31T16:58:06Z

As software has become increasingly more complex and the demands placed on the test discipline have escalated, it is clear that the legacy approaches for testing software are not always adequate. Sometimes test teams get caught up in performing the same types of tests and/or taking the same types of approaches not because those things keep providing great value, but oftentimes simply because “that’s the way it has always been done” J.

Years ago, as part of thinking about this phenomenon with a colleague, we drew up a small playbook which described the values that our test team “aspired to”. As a small part of this was the below “Old World, New World” chart which tried to summarize some of the tenets of what we were trying to move towards (and some of the stuff we were trying to move away from…). We wanted to take a hard look at some of the things that we were investing a good deal of time into and see if we could make any tweaks to get more bang for the buck. Looking at the chart now it is interesting for me to see that many of the points are still relevant and we still haven’t met all of our goals…but some incremental progress has been made J

Let’s look at the chart and then briefly comment on some of the bullet points:

Old World	New World
Doing same old testing that we have always done (grandfathered in)	Killing old tests, procedures, efforts, etc. that have little value or whose value does not measure up to the cost Expanding /adding tests with creative additions Exploratory/Scenario testing investments
Test spending time running regression tests on private binaries Test adding coverage on new functionality	Having Developers run regressions using centralized system Test adding coverage on new functionality
Spending time triaging regression results because of test issues	Spending time getting tests to 100% in all configurations
3-4 week Full Test Pass	1 week Full Test Pass
Adding, Adding, Adding, Adding	Removing, Removing, Innovating, Adding
“Classical SDET” deliverables are rewarded	Radical thinking and results are rewarded
Finding bugs after check-in	Finding bugs before check-in
Test serving Development	Test serving Quality

“Doing same old testing that we have always done (grandfathered in)/Killing old tests, procedures, efforts….”: The idea behind this row is as was described above, i.e. test teams frequently keep doing the same old things again and again without analysis of the return on investment for some of these tasks. As an example one of my test teams maintained an extremely large and complex test suite which tested a particular type of authentication behavior. This test suite had thousands upon thousands of variations and was not particularly stable. Analysis showed that the maintenance costs for the test suite were quite high and in fact the suite was not catching many regressions and/or finding new issues (certainly not enough to justify its existence). The decision was made to retire the old test suite (at least from regular automated runs) and replace it with a much more simplified version which was stable and which provided similar coverage. Although changing something like this was scary, in the long run it was the right thing to do for the team. The end result was that due to the reduction in maintenance costs, the testers who previously owned the suite were able to spend more time on end-to-end scenario testing and ad-hoc testing and as a result were able to catch more defects from reaching the customer.

“Test spending time running regression tests on private binaries…/Having Developers run regressions using centralized system.”: When at all possible, centralized systems should be put into place in such a way that the test team is not a bottleneck for development. Instead of developers tossing the binaries to test over the fence and waiting for the test team’s “blessing”, a centralized system which can be leveraged by everyone across the disciplines can enable developers to verify their private binaries and protect against regressions. In the meantime, test can be off tracking down the tough bugs J

“Spending time triaging regression results because of test issues/Spending time getting tests to 100% in all configurations”: In my estimation, there is no greater drain in an SDET’s work life than unstable automation. When SDETs spend too much time triaging test automation bugs, it takes away from high value work such as moving quality upstream or finding hard-to-track-down bugs in end to end flows. The “Golden Standard” that our test teams try to evangelize is that every time a test fails, it should be an indicator of an actual product bug. While this is sometimes difficult to achieve in practice, it is certainly a good result to aspire to. Another problem with unstable automation is that it causes a loss of credibility within the work group. Unstable tests and/or false alarms reduce our clout as testers and may cause the “Boy who cried wolf” syndrome in which people simply stop being alarmed when tests fail. Apathy like this in the team is dangerous.

“3-4 week Full Test Pass/1 week Full Test Pass”: This speaks to the results that a test team can realize when they tighten up their battery of automation by increasing the stability of their test automation and removing tests/procedures which no longer provide benefit. These particular numbers came from a test team which worked on a component which had a huge number of system interdependencies, so please treat the particular example with a grain of salt J.

“Adding, Adding, Adding, Adding/Removing, Removing, Innovating, Adding”: The trap of continuously adding is something that I have encountered in multiple test teams that I have worked in and is a fairly common occurrence. When a new feature comes along, tests are added. When a test hole is uncovered, tests are added. When a new test technique is researched, tests are added. The problem with this approach is that every new test/process/procedure that is added has a cost, and that cost is compounded over its lifetime. This creates a maintenance burden that becomes unmanageable. Thankfully there are many ways out of the trap. One example is to enforce review and removal of old test variations which are no longer relevant at the same time that new tests are added. Another example is to figure out ways to combine/consolidate test cases such that coverage is maximized with a minimal set of tests.

“ ’Classical SDET’ deliverables are rewarded/Radical thinking and results are rewarded”: Along the same lines as the comments above about questioning the same old testing that was grandfathered in, it is important for SDETs to feel safe that they can try new techniques to augment or potentially replace some of these old approaches. This requires management to be “on board” with SDETs who are able to see past the cloudy veil of the past, and take risks (with the expectation that some of these risks will fail). We will never escape from the past if we are not receptive to innovation and the risk that comes with it.

“Finding bugs after check-in/Finding bugs before check-in”: Although testers are frequently characterized by the issues that they raise while testing the actual product (i.e., after the code has made its way into the build after check-in), the most efficient time to find the bugs is before the code makes its way into the build and ideally before a single line of code has been written. This is all part of the “Push Quality Upstream” war cry. There are many ways to go about this and each method probably deserves its own series of blog posts J:

Ensuring test is plugged into the design and spec phase
Test actively involved in pre check-in code inspection
Creating a check-in battery of tests which are run automatically before the new code is committed to the source control, and whose failure will prevent the code from being checked in
Etc.

“Test serving Development/Test serving Quality”: Sometimes the test discipline is seen as a henchman for the development discipline, with test waiting ready for development to throw the code over the fence. It is important for test to assert that the master we are serving is quality and that may not always include being the safety net for development. For example, sometimes it means teaching development to be their own safety net J

Future blog posts will dive deeper into the above shifts in behavior as well as describing some of the processes that can be used to facilitate the transitions.

Comments/Questions are always welcome. Thanks for reading!

-Liam Price, Microsoft

Code Inspection for SDETs, Part II

2011-10-25T00:59:00Z

In part I of this series, we looked at some of the benefits of the SDET discipline being fully plugged into the code inspection pipeline. These benefits include such outcomes as pushing quality upstream and raising SDET awareness of the code that is being tested. Plus, it’s fun! J

Now we will share the general process and approach that we take in one of our SDET Code Inspection virtual teams. Please note that the format and style of the code inspection process described below may differ from what you are familiar with (especially if you have used a formal code review process in the past). We participate in a type of lightweight group review, which takes bits and pieces from several formal inspection processes. Also, please note that the approach being outlined below is for post-check-in inspections in which we are able to review code on a larger scale across multiple components (as opposed to pre-check-in inspections which tend to be narrower and focused more around particular isolated changes).

Ownership

As often as possible, we push to have the SDET who owns the feature area also own and drive the end-to-end code inspection process for that area. This is helpful in several ways, such as helping reinforce ownership of the area, introducing more SDETs to the process, and also in getting the SDET owner to think about questions like:

What materials/specs/information should I provide to the participants of the review in order for them to gain an understanding of the feature that is being reviewed? Am I able to distill the essence of the feature and describe this to the participants?
What are the most critical pieces of code to review? Oftentimes, we must scale back the review in order to meet our maximum lines-of-code guidelines. It is a worthwhile exercise for SDETs to be able to make qualitative judgments about what code area are riskier and what code requires more attention

Setting Up the Meeting

In general, the feature test owner will do the following prior to the review meeting:

1. Decide what code will be reviewed

Usually 1500-2000 lines-of-code is optimal for a one hour review meeting. Anything more than that is counter-productive, and is unlikely to be finished in a single review meeting. Sometimes multiple review meetings are needed depending on the size/complexity of the feature. Smaller features can sometimes be reviewed completely in an hour. Other features may require multiple meetings (3, 4, 5+) to cover all of the critical code
Areas of code to review are decided upon at least 3-4 days in advance and are clearly stated (included files(s), line numbers, etc) in the meeting invite. To allow for SDETs to work at their own pace, it is important to give them several days to complete the review
It is oftentimes useful for the code selection process to be a collaborative effort between the disciplines to figure out which are the most critical areas to review

2. Decide upon supporting materials

Additional materials can/should be provided in order to inform the reviewers about the feature(s) under review. This can include functional/design/test specifications or a non-technical synopsis of what the code is trying to do. It is amazing how useful this can be during the inspection process, as we have seen numerous bugs come out of reviews where the only thing wrong with the code was that it wasn’t doing what the spec said it should be! J
It is important that all reviewers provide feedback using the same process so that folks are on the same page (both literally and figuratively) during the actual review meeting. In some cases, this means that physical printouts using a common tool are made for all members who will participate in the review. In other cases, this means that an online tool will be leveraged to provide feedback. Whatever the process is, it should be consistent for all reviewers and designated by the test owner. Otherwise, you run into confusion over exactly what lines-of-code are being discussed at particular times during the review meeting.

3. Send out the meeting request

The meeting request goes out to all members of the code inspection virtual team as well as other members of the feature area’s crew
The meeting request includes the code to be reviewed as well as any supporting materials

Reviewing the Code

In this post, I won’t dive into the mechanics of reviewing the code (this will be covered in its own future post). In our inspection process, the actual heads-down inspection happens offline at the reviewer’s pace, and this happens before the code review meeting occurs. Reasons for this approach include:

This allows for different styles/paces of review to take place
This provides more time during the actual meeting to discuss the defects and other potential problems

The Meeting

As mentioned above, in our process the meeting itself is not where the inspection takes place, but rather where the defects are tabulated and discussed. In general this is accomplished during the meeting by moving through the code, page by page, and recording the defects and which participants found them. Since one of our main goals in this process is education, in many cases we try to take the time to talk about particular patterns of defects found in the code and how these might be avoided altogether with a holistic approach. We also spend some time talking about the design decisions that were made as well as interdependencies between features, as this is a common area where defects tend to crop up.

During the review meeting, the feature test owner owns ensuring that:

1. The meeting keeps moving and does not stall over incidental discussions

2. All defects/comments are recorded

3. Any open issues or unclear areas are noted

The Aftermath

After the meeting is completed, the feature test owner owns:

1. Opening bugs and/or work items to ensure that the issues are tracked to completion

2. Following up on any open issues uncovered during the review. Sometimes particular areas of the code need more discussion and/or research before the issue is considered completely understood. All issues marked for follow-up during the review should be driven to closure by the feature test owner

3. Providing a summary of the review to the Code Inspection virtual team, including data about the area reviewed and the issues found

Tracking Statistics

We keep running totals of various types of data from the review sessions. In general, we look at the following statistics/metrics across multiple reviews and multiple areas

Areas of code reviewed
Total lines-of-code reviewed per area
Number of issues found (total yield)
Issue efficiency (#bugs resolved fixed)/(total yield)
Special “High Impact“ bucket bugs – such as security bug yield

At the end of particular milestones, we look at the overall statistics and may alter the approach based on the data. In addition, we will frequently provide “post mortem” feedback to the feature team of particular areas if there are coding/design patterns that the code review team deems to be pervasive enough to warrant special attention. For example, if memory leaks due to a particular coding style/convention are running rampant in a given area, it is helpful to provide feedback to the feature team about the ways in which these types of leaks could be addressed via a slight coding practice adjustment. These suggestions can then be rolled up into development coding guidelines, and in some cases incorporated into static analysis tools.

In a future post in this series, we will discuss some of the guidelines that we use during the review of the code in order to make for an efficient and “high-yield” inspection. Thanks for reading!

-Liam Price, Microsoft

Code Inspection for SDETs, part I

2011-10-04T22:23:07Z

(Note for those unfamiliar with the acronym, SDET stands for Software Design Engineer in Test. In simple terms an SDET is an engineer who develops code to test software)

Code inspection has historically been a “high-yield for low cost” bug-prevention or bug-detection approach. This is especially true when used as part of formal code review process and integrated into the software development lifecycle. In the multiple teams and companies I have worked at it has been very common for the developers to practice a peer code review system before every check-in, and periodically review entire features post code-complete. However, it has not been common for the test discipline to be fully plugged into the code inspection pipeline, which is less than ideal.

Here are some examples of the benefits realized with the test discipline participating in the code inspection of the code that they test. Let’s walk through a few of them.

Motivation

1) Knowledge Growth: Code inspection is an excellent way for testers to become more familiar with the code base that they own from a quality perspective. As we move into a world which requires more technical horsepower and understanding from SDETs, there is really no substitute for having deep code knowledge of the components that they own. In addition, it is a great way for SDETs to uncover test holes and lesser-known functionality in their components. A strategy to grow SDETs into senior roles has to include some level of in-depth code knowledge improvement, and code inspection is a great way to attain that knowledge

One common concern about testers using code inspection techniques is that the tests that they author will somehow become extremely reliant on implementation details and become fragile and brittle when confronted with product code implementation changes. This is a valid concern and is something to consider. Testers should be careful that they do not rely too much on underlying implementation when designing automation for regression testing for example. On the other hand, knowing current interdependencies and contracts between portions of the product can teach SDETs about what portions of the code may warrant additional testing attention

2) Efficient Defect Prevention: Root Cause Analysis of various defects has shown that many high priority bugs could have been effectively found and/or prevented through code review or static analysis. In some cases, the best test tool for the job is going through the code itself. Certain classes of bugs can be found quite easily through inspection, for example:

Cleanup and logic errors in failure conditions
Misuse of certain APIs
Broken API contracts
Some types of security issues
Stylistic coding issues which have long term time drain in terms
of maintainability
Etc.

3) Time Savings: A wonderful side effect of having a low cost bug finding approach is that it frees SDETs up to do other more time-consuming testing. After all, we are not being paid to be test execution machines. When we find certain classes of bugs more quickly and with less effort, we can move on to tracking down the really difficult bugs through other sophisticated techniques

4) Constant Feedback Loop: After the code inspection process has been followed for some time on a codebase, certain types of patterns emerge from the various reviews. Many of these defect patterns could be avoided by simply evangelizing how particular defects could be avoided at the code level. As part of the process, these defect patterns can be communicated to the development team as part of coding guidelines which will also prevent future occurrences. Participating in this feedback loop is empowering and provides great value

5) Eliminating Classes of Defects via Tools: Concentrated effort at code inspection pushes quality upstream and can help multiple teams, division, or company-wide. Working with static analysis tool teams to help identify certain classes of defects and being able to feed that logic into the tools themselves can have a huge impact downstream by preventing the bugs from ever seeing the light of day in the first place

6) Beginner Impact: Even new code reviewers can make statistically significant impact. Some engineers shy away from doing code inspection because they feel that they need to be a seasoned expert in order to get anything out of it. This couldn’t be further from the truth – studies have shown that although experts at code review do find more bugs than novices, the delta between these two groups is not large and should not discourage folks from participating

Goals

While building our SDET Code Inspection virtual team in my group, we wanted to avoid having a single-minded emphasis on finding defects (although we do encounter numbers of defects). We wanted to also think about the education aspect and the defect prevention aspect as well. Along those lines we concentrate on the following goals:

1) Find bugs earlier in the development cycle and push quality upstream

2) Increase SDET awareness of the code that is being tested so that more educated decisions can be made with respect to test coverage, test investments, test prioritization, etc.

3) Lower the bar of entry into code inspection. Learn techniques and approaches from others

4) Provide general code quality feedback per area and feature in order to raise the quality bar

After every milestone I review our progress towards these goals and tweak the process if needed.

Upcoming blog entries will cover the exact process that we follow in my code inspection team and some examples of the types of defects that can be found through code inspection techniques. Thanks for reading!

-Liam Price, Microsoft

Hello world!

2011-09-13T18:39:00Z

We are the senior SDETs (Software Engineer in Test) at Windows Live in Microsoft. We plan to use this blog to make our voices heard regarding anything and everything test related. So stay tuned for some interesting articles coming your way.