Unit testing, component level testing and UI testing, what to use and when

This is a topic that anyone doing software testing faces every time they are writing new tests. What are the advantages and disadvantages of each? When should I use them? What’s a healthy mix?

Let’s start with the basic definition of each of these scenarios:

Unit Testing – Unit testing is testing directly at the most granular level. If given a method that takes two values and return a positive result. Does the method fails (crashes, throws an exeption, etc) if either of the values is null or invalid? Does it return valid results given a specific set of values? Does it fails if given an incorrect set of values?

Component Level Testing – Similar to unit testing but with a higher level of integration. The big difference here is that the testing is done in the context of the application instead of just directly testing the method in question. Take the same example as the method above, say that the method is invoked through a keyboard shortcut after your application opened a document and after it executes something is displayed on the application. While testing at the component level you would have the application open and the document displayed but you would be testing the method and evaluating it’s result (without taking into account the shortcut or what gets displayed on the application afterwards). For the most part this is the kind of testing we do when there are test hooks inside your product code.

UI Testing – This tests the whole end to end scenario (which simulates exactly what the user would be doing). This (for the most part) goes through all the mouse clicks and keyboard presses the user goes through to get an action done. If we extend the previous example this would be invoking the same action except that this time instead of using any test hooks it would be using the keyboard shortcut and instead verify the actual displayed results on the application.

What method to use?

The question that first comes to mind for everyone, since UI testing is the most complete and closest to the user why not simple do UI testing exclusively and be done with it? Going through the UI will execute the action at every layer and that in turn will call the individual methods so why not simplify things? The issue here is that for extensive scenarios you have to pay the cost of loading the application and going through all the actions every single time (which depending on your application it can take several minutes per test).

After hearing the opposite comes to mind, so if doing UI testing is so expensive why not simply do Unit Testing which is pretty cheap since it only needs to invoke the functions for the multiple scenarios? With this approach you’ll know that all your methods are working properly but you’re not testing the integrations of the features and the product as a whole.

That leaves us with the third option; component testing seems to be a pretty good happy medium? Why not just do component level testing then? While this would cover most of the scenarios we’re still missing one layer of testing that still needs to be tested

The table below summarizes the integration level versus the speed/cost of each method:

There’s no reason to be completely constrained by any of the methods so the answer to the question is you want to use all of them. The question then becomes what do I test with each method and what amount of testing, per method, is enough. The answer here is going to be different for everyone and it’s going to be constrained by the product and features you’re testing.

These are the general rules I follow:

Unit Testing – Sanity tests at the lowest level. Make sure the methods can handle valid and invalid data and that the basic functionality works. No real test case scenario testing here. (Note: depending on how your organization is defined this might fall solely on the feature owner rather than the testers).

Component Level Testing – The bulk the work is here; test all the define test case scenarios here since it’s fast and gives a good amount of coverage.

UI Testing –This covers the things you can’t test through component level testing such as verifying that actual UI elements are displayed correctly. In addition to that end to end scenarios, this means running a very small subset of the scenarios testing through the component level to verify that the feature as a whole, as the user will be experiencing it, will work.

With that mix you can have full coverage of the whole feature with a major emphasis on correctness with an extra layer of testing for the UI integration. The huge advantage here is that since your correctness testing is relatively fast you can run and investigate a full test run in a fraction of the time that it would take to do a full UI test pass but without actually neglecting the UI scenarios.

One thing here that I want to make some emphasis on is that people have a preference for one type of testing over the other (which is natural) however some people end up with tunnel vision and dismiss the other methods which as we’ll see from the next example can either hurt efficiency or coverage (or both).

Case Study

Let’s take those 3 concepts on more concrete example, a simplified IntelliSense scenario. When a call is invoked the compiler calculates the raw result data and passes it to the IDE which does some formatting and then that result is passed to the Editor to be displayed. Our three components here are the compiler, the IDE and the editor.

Now let’s go in a deeper layer and define an AutoComplete scenario with these methods defined on the components mentioned above:

· string[] Compiler.CompleteStatement(string file, int position)

· bool IDE.CompleteStatement(int position)

· bool Editor.DisplayCompletions(string[] completionList)

· bool Editor.InsertText(string text)

The action workflow would be this:

1) IDE.CompleteStatement is called

2) Compiler.CompleteStatement is called

3) If there’s any results

a. If there’s just one result call Editor.InsertText

b. If there’s multiple results call Editor.DisplayCompletions

Finally let’s define a simple test case in which we’re invoking autocomplete on line 3, column 5 of a.cpp and I'm expecting have a list displayed with 3 results. With the specs for our action and our test case defined here’s testing divided on those 3 areas:

Unit Testing (verify that the methods can work independently, this also doesn’t take into account our scenario):

- Verify that each of the 4 methods return the expected results given specific input

o Compiler.CompleteStatement return results with the raw data

o IDE.CompleteStatement invoke InsertText when there’s just one result and DisplayCompletions when there’s more than one

o IDE.CompleteStatement formats the raw data

o Editor.InsertText inserts the given text

o Editor.DisplayCompletions invokes the event to display the list and passes it the relevant data? (ie which data to display, tooltip data, etc)

- Verify that the methods don’t fail for valid inputs

- Verify that the methods can handle invalid data gracefully

- Verify negative cases (ie: the results should be empty)

Component Testing (at the main layer through test hooks verify the correctness of the results):

- The IDE gets the 3 results from the compiler and formats that data correctly before passing it to the editor

UI Testing (verify that the action as a whole is successful)

- After invoking the action in the same way the user would does the list displays with the 3 expected items.

- Verify that it works for every way to invoke the action when there’s multiple ways defined (menus, context menu, mouse, keyboard shortcuts).

One thing you might be wondering here is wherever you would want to add component tests at the other two layers (the compiler and the editor). In this example it doesn’t makes much sense since the IDE is calling the Compiler at the component level anyway (not to mention that most likely there’s a full test suite dedicated to those scenarios at the compiler level already anyway). As for the editor it just consumes whatever the IDE feeds it so the UI tests would cover that integration.

The final result of all this is that for a more realistic test plan in which you had more than just one scenario you would only have to add tests at the component level as testing those scenarios either with unit tests or UI tests wouldn’t give you any extra coverage.