As I talk to customers I find that there are many different work flow models. I use the term “work flow” to describe all the things you do to get complete your objectives. That work flow can be documented and shared with others at the office. Having a positive and efficient work flow is important, otherwise we get cranky.
Meeting the needs of the different work flows can be difficult. In this post, I’ve outlined three common work flow situations with steps to successfully collect code coverage data. These scenarios were built around different models of access to source code because that seems to be the underlying issue customers experience.
Every scenario minimally requires product binaries, product PDBs, test binaries, and Visual Studio Team System. For better, richer, and more flexible experiences you’ll need more, like access to test and/or product source code. In the end, you just want code coverage data so it doesn’t matter how you go about it. Pick a method that works for your setup and enjoy what VS Team System has to offer. J
For more information about how you can go “off-roading” with code coverage, I’d recommend looking at John Cunningham’s blog.
This is the most ideal scenario. One has the product and test projects loaded in VS. One has full ability to build both which provides the most flexible experience.
This is probably a developer or a tester who work directly on the product. They are trusted to and it is logistically possible to have full access.
This is probably the most likely scenario. One has product binaries (and PDB files!) from a daily build. One has access to the test sources and can build test binaries. This nearly provides the same flexibility as Scenario 1.
In this model, the user may be the same developer or tester as in Scenario 1, but they may be encouraged to only test against “official” builds from a common source. This could also be a person testing offsite from core development who receives regular drops of builds from the development team. They may be interested in gauging the comprehensiveness of their testing as measured by code coverage, or this activity may have been requested by the development team. In the latter case, the code coverage results file will be sent to the development team for analysis.
This scenario requires the most work and provides the least amount of benefit, but unfortunately it is all too common. All the great work VS does for you when you have your own test project is gone because you only have test binaries now. Additionally, this could include running manual tests so there might be no automation to run at all.
That said the scenario still works. You can get code coverage data. Also, if that data is shared with someone who has access to source code, then they will get full benefit from the results as they can see code coverage detail and line coloring.
Note: There are alternatives to manually executing these commands, but they are slightly more complex.
This will instrument and execute the tests, while correctly starting & shutting down the coverage collector. If you still wish to instrument the binaries yourself, then you still need a dummy-dll to be instrumented. The tools do not start the coverage monitor unless the tool itself has instrumented a DLL.