Jose's Blog : Testing

How To: Extend the Network Types for a Visual Studio Load Test

josealmeida — Wed, 20 Aug 2008 16:14:00 GMT

Out-of-the-box Visual Studio allows testers to define a network mix for load tests with the following network profiles:

LAN
T3 6.0 Mps
T1
Cable/DSL 1.5Mbps
Cable/DSL 768k
Cable/DSL 384k
Dial-up 56k
Dial-up 33.6k
Dial-up 28.8k

If we want to add an additional entry, say "Dial-up 128k", we'll have to add a new network profile definition file.

These files can be found in the following folder:

C:\Program Files\Microsoft Visual Studio 9.0\Common7\ide\Templates\LoadTest\Networks

Just select an appropriate file (say "Dial-up 56k.network"), copy and rename it (to "Dial-up 128k.network") and edit it's contents accordingly:

<Network Name="Dial-up 128k" BandwidthInKbps="128">
</Network>

You'll have to close all Visual Studio instances and re-open them so that it can reload all network profiles.

Unit testing session

josealmeida — Mon, 05 Jul 2004 19:06:00 GMT

During TechEd'04 in Amsterdam I attended a Birds Of a Feather session on how to integrate unit testing into the Software Development Life-Cycle. For those who are unfamiliar with the format of these sessions, it's a speaker moderated discussion, where we can ask questions and have them answered and commented by the speaker and other attendees. I enjoyed this session format very much as it's highly interactive.

This session was moderated by James Newkirk. The audience placed a few questions that led to various discussions on unit testing and the correct way to do them (or not to do them), unit testing techniques and personal experiences both positive and negative when using these techniques.

Below are the topics discussed during this session and I thought of placing them here, along with some of my personal impressions on the various subjects.

Testing abstract classes with different implementations

How to test a virtual implementation of a class, when different implementations can spawn at any time ?

James Newkirk was of the opinion that you should abstract the tests when a new implementation appears.

My opinion on the subject follows along the same lines. If I'm doing TDD, I want to write test code for all my implementation (this is my opinion on testing privates as well, BTW). So, I'm a firm believer in testing the implementation not the contract, because that's where my logic will be. It's also my opinion that I should code my unit tests just like I do my production code. So, when there's the need to do some refactoring in my unit test code, I'll do it. This includes doing an Extract Interface, or Extract Class, or any other abstraction refactoring.

How much to test / testing privates

This is a topic that has always had strong arguments pro and against as was also evident in this session. While James Newkirk's opinion is not to test privates, I'm not so sure we shouldn't. James doesn't like to test the private methods, because it hinders refactoring. But if you don't have a test for a piece of logic, how can you refactor it (this seems like a catch 22 to me) ?

As I've stated above I'm not so sure we shouldn't test private members. I guess that this depends on how I do TDD. There are two different approaches here. Write code test-first only considering the publicly visible members (or the public interface), or write tests for all your logic and during refactoring, if a given method turns out not to be used by any other class other than the test class, use the Hide Method refactoring. I use this last approach, for two reasons:

If I'm only coding tests for the public interface, I'll have no idea how much logic I'll be having "under the hood" or beneath the public interface. But typically, as someone pointed out during this discussion, this is were most of the logic will be;
The other reason is that this allows me to code this (soon to be) "internal" logic, using a test-driven approach, which gives me confidence in the implementation and how I refactor it.

This approach, however, leaves me with a bunch of orphaned test methods, since they can no longer access the methods they should be exercising. This is when I usually refactor the test code as well, replacing method invocation with reflective method discovery and invocation.

Legacy code w/o tests

The scenario is the following. You have a large code-base for which you don't have any unit tests. You'd like to add unit tests, but where do you start ?

James answered this question in a very pragmatic way IMO. "Don't add unit tests for anything you aren't about to change." He said.

Well this seems a very sensible decision. He then elaborated on this by stating that it could be done by establishing a virtual "boundary" of unit tests defined by the things about to be changed. Tests can be written against this "boundary" and the changes made on the inside of the boundary.

How to convince people to code test-first?

I see this issue as a very hot topic. Primarily because, I have been trying to convince people to adopt this way of developing. And I know from personal experience that it's hard to find the right arguments. Strangely enough, I've found that direct management (PMs or Dev Leads) are more receptive to the message of TDD than the actual developers.

James Newkirk stated that developing software "...is not about the individual developer, it's about the team and the continuous maintainability of the code". Someone also pointed out that "blinking lights changing from red to green help a lot!" I couldn't agree more.

Additionally, James pointed out a "technique" that I've also used with some success, which is to ask someone to test their own code. Most of the times people find it hard to write tests against their own code, because they usually don't write their code to be tested.

This debate spawned another thread of discussion. If the developers write unit tests, which was usually a big chunk of the work that was done by QA teams (white-box testing), the QA teams must start testing at a different (higher) level.

Mock Objects

Just like Benjamin posted, "James described mock objects as the situation where object A interacts with objects B and you need some way of 'switching out' Object B to create a 'dummy response' from the calls of Object A."

True, but it's necessary to determine why would we want to 'switch out' Object B. I usually 'switch out' object B if it isn't part of my domain code (maybe it's an object belonging to a third-party API). If my Object B is, for instance a DAL component and it uses ADO.NET to connect to the database, I would use (dynamic) mock objects to replace the ADO.NET's data access components in object B using the "Inversion of Control" pattern. It may be somewhat complex, but it promotes decoupling and testability, which rate very high in my trade-off scale, so I usually think it's worth it and end up implementing it.

Testing against a DB.

Sometimes it's hard to mock out the data access layer components, someone pointed out. This is true and there's usually a tendency to end up testing against the database.

Actually, James Newkirk says that there are situations where he prefers to test against the database. "When writing unit tests for DAL components use the actual database. When writing unit tests for components that use DAL components, mock out the DAL components." James said.

I have a different point of view. If the DAL components where written by me, I'd like to test not only the components that use the DAL components, but also the interaction between them. On the other hand, data connectivity components supplied by the underlying API (system or third-party) are not part of my domain code and I can (and probably should) mock them out.

How long should tests run ?

This was an interesting point, as people have very different experiences. Someone said they had test that took 6 hours to complete, others have them built into the build process and run the whole suite nightly for as long as they take. Another delegate's tests took about 2 hours to complete.

In James' opinion, 1 hour is too long, because tests should provide rapid feedback to the developer and promote a very fast turn around in the development lifecycle. The recommendation that came out of this debate was that in large projects we should separate developer unit tests and run all the tests (all test suites from all developers) during the daily build cycle. Some integration problems will arise only during build, but it's a trade-off.

Automatic test generation.

Commercial products don't take into account the actual intention of the methods being tested and so usually create 'poor' tests. On the other hand, Visual Studio Team System has a built-in functionality to implement a boiler plate test for any given method.

How do you unit test GUIs ?

In any discussion on unit testing this topic usually comes up. In this session people mentioned some commercial products that they've used with some success, although they couldn't survive for more than a few builds without breaking them. A pretty "standard" response to this issue is to use the Model View Controller design pattern to separate presentation and presentation control logic and eventually test them individually.

The problem is that when we're coding unit tests against an API, the interface is programmatic, whilst when we are coding unit tests for a GUI, the interface is visual and designed for human and not machine interaction.

I've used NUnitForms and NUnitASP with some success to get around this situation and exercise the visual elements of the GUI individually. The main drawback with both of these extensions to NUnit is the relatively small number of controls supported (granted that they are the most used).

Well, a few other questions and discussions came up, but were marginal to the previous ones that were the main thread of the session.

Unit Testing User Interfaces (Updated!)

josealmeida — Tue, 15 Jun 2004 23:13:00 GMT

Usually one of the major difficulties a developer faces when writing unit tests is how to write test code for User Interfaces. This is particularly important when we're doing Test-Driven Development....(read more)

Loading .config files in NUnit

josealmeida — Mon, 31 May 2004 23:46:00 GMT

Last week I gave a presentation at Microsoft DevDays 2004 in Lisbon on Test-Driven Development. A day later I got this interesting question:

"(...) if I'm testing a DLL assembly that uses a .config file to read in data, when I run the tests, NUnit loads it's own .config file and all my tests fail. How can I solve this ?"

I also had this same problem a while back. There are two separate answers to this question as far as I could tell:

If we're using the NUnit GUI than it's just a question to specify which .config file to load (Project -> Edit)
If we're running NUnit from the console, copy the config file to the directory where the test assembly is located (tipically bin\Debug) and rename the config file to <test assembly>.config (ex: UniTests.dll.config)

This will cause NUnit to load the required .config file.

Testable code

josealmeida — Tue, 27 Apr 2004 19:31:00 GMT

Previously I mentioned that one of the most important benefits I get from using TDD is that it drives me to write more testable code. I've been thinking quite a lot about this, and another recent post from Chris Dickens made we even more aware of the importance behind writing testable code.

Chris states:

"As you might expect, before we can start serious development we have to write specs. Those aren't my job (that's what Program Managers do) but developers and testers are actively involved in critiquing the specs. Technically my purpose in those meetings is to flag any design issues that would make the product untestable, but in reality I'm there to provide my input on what the product should do."

I've found (the hard way) that trying to test code that wasn't written with testing in mind, is extremely hard, especially when you find tightly-coupled code.

TDD promotes decoupling and this, by itself, is very helpful in producing testable code, because there's some way to plug into the production logic you want to test. If the code you're testing depends on other parts of the system you can find yourself writing a huge amount of code into the setup methods of the unit tests. This, unfortunately, introduces a high degree of coupling into the testing logic, particularly if the code you're dependant on belongs to another domain (i.e. it's an external system)

Using mock objects can help us develop loosely-coupled unit tests. Considering this, some questions seem to arise almost instantly... "When should mock objects be used?", "Is it recommended to use mock objects to represent every external system that our production code interacts with?", etc.

Here's a definition of a Mock Object:

is easily created
is easily set up
is quick
is deterministic
has easily caused behaviour
has no direct user interface
is directly queriable

From the above definition, it's possible to identify a couple of situations where using mock objects can be valuable in coding unit tests:

the real object doesn't yet exist
the real object is difficult to setup
the real object is slow (if it takes too long to run the tests, you might feel discouraged to run them)
the real object produces unpredictable results which may cause the test to behave in a nondeterministic way;
the real object is a user interface
the test needs to ask the real object how it was used (for instance, the test might need to verify if an event was raised and caught correctly)

Developing unit tests with mock objects, improves domain code by preserving encapsulation, reducing global dependencies, and clarifying the interactions between classes [Mackinnon, 2000].

TDD Adoption Numbers

josealmeida — Fri, 16 Apr 2004 12:19:00 GMT

Matt Hawley posted a few numbers in order to justify/support the adoption of TDD.

I've found an interesting research paper on the subject, from Matthias Müller and Frank Padberg from the Karlsruhe University in Germany.

Some of my personal views on the subject are:

I do find there's a slight overhead to developing Test-First. However all developers unit-test their code, to one degree or another. All developers write some form of test code. I believe the overhead comes, not from writing test code, but from using a testing framework.
IMO one of the greatest benefits I've experienced by coding test-first is that the code I write is testable (i.e. If someone else needs to write/extend test-code, for instance to test some wierd situation, it can easily be done);
Coding flows with much more confidence (more immediate feedback on what's wrong);

I do believe that TDD is a very helpful practice and from my experience, together with other practices like Refactoring, I do believe that it helps to write better code.