Being Cellfish

Stuff I wished I've found in some blog (and sometimes did)

Merging coverage reports with Bullseye

Change of Address
This blog has moved to blog.cellfish.se.

Merging coverage reports with Bullseye

  • Comments 22

I've previously recommended Bullseye. And there is another nifty feature with Bullseye you should know; the ability to merge reports. This is pretty useful when you have one report from your unit tests and one from some other type of test run. Use this command to merge reports with Bullseye:

covmerge.exe -c -fMergedData.cov file1.cov file2.cov
  • I have merged 2 .cov files, how to identify that the generated .cov file is correctly merged.

    Thanks

  • Why would you like to verify this? Why not trust the tool? Why would it not be correct?

    But the way to verify is to look at the data and the merged data will be max coverage in the two original files. Let's assume that your first coverage file tells you that function A has 25% coverage and function B has 50% coverage. The second coverage file then has function B at 75% and function C at 100% coverage. Then the merged file should show: A-25%, B-75%, C-100%.

  • re: Merging coverage reports with Bullseye

    >Then the merged file should show: A-25%, B-75%, C-100%.

    This doesnt always happen. If function B has modified, bullseye retains the cov for the one which is more recent.

    So if the second coverage file has an modified function B , and has coverage B- 25% and Func C at 100%, then the merged file would show

    A-25%, B-25% , C-100%

    Is there anyway to get around this?

  • @ram:

    I didn't know that because I run my coverage with the same build. Does not make sense to me to run coverage runs with different versions of the application. The only thing that makes sense is that if a function is changed you get the latest coverage rather than the merged because whatever value you had before is no longer valid since you changed the function. Hence I don't think there is anything that needs to be fixed...

  • >>Hence I don't think there is anything that needs to be fixed...

    True. But consider a case where only function B is modified, (with A and C remain unchanged) and the file has a newer version. In that case merged coverage report would only be the one which corresponds to the recent version of the source.

    In which case the merged coverage would have

    A-0%, B-25% , C-100%

    and earlier obtained coverage for A is discarded. There are pros and cons for this approach, but one would expect Bullseye to leave it to the user to a pass on an parameter to covmerge to do either.

  • @ram: I disagree that it is up to the user to decide what to do if two merge files being merged have been using different versions of the code because if there are different versions basically all coverage from the old merg file is no longer up to date. I think it is much more productive to focus on getting your different runs using the same version and doing it automatically. It just makes more sense to me.

  • Guys, how about merging 2 coverage files build with different path. For ex: covfile1.cov has project directory c:\x and covfile2.cov has project directory c:\x\y\z. In this case whether Bullseye merger will merge the files(if the same function is not modifed; instead only new files are added in the recent covfile2.cov)?

  • @Rajan: Never tried that. Don't see why it should not work since the two sets of code are different. But also, since the two sets are completely different there is no benefit from joining the files other than the ability to look in one file instead of two...

  • Let's take one example. If more than one developer started working on one feature then as per the BullsEye's behavior, they both have to MAINTAIN the same directory structure of their project(in their dev machines), in order to merge their .cov files from Unit Testing.

    Am I right? My question here is, why BullsEye do this much check and why don't it identify the project/file/func names and do the merge of .cov files?

  • Two developers working on the same project/feature should have the same folde structure because they both chech their code into some version controlsysyem, right? The root may be different, but from the rot down it should be the same. Don't even understand why you would run coverage seperatly since there is a merged version of the code; in the version control system...

  • Thing is, our directory structure is same but root is different. When root is different, BullsEye merge is not merging the .cov files at all.

    Fine. What i want to convey is, let's take, Dev1 writes Func1 in File.cpp and Dev2 writes Func2 in File.cpp, both took code from same SCM. Here, Dev1 could run unit test cases to check the coverage of his func and same with Dev2. Once they are done, their manager wants to know the code coverage % of the new module. Obviously they got to merge to show the result? this is what my use case is.

    Whether they take code from SCM or not, as per the BullsEye, they have to maintain the same directory structure(from root) to build the code for code coverage and to create .cov file. when the result is asked they got to merge the .cov files.

  • @Rajan:

    They don't obviously have to merge their seperate results. I would say you get the latest from SCM and run the tests and get a single coverage file. I think that is the only thing that makes sense.

    I know having different roots is not a problem. Other than that I donät understand if you've experienced a problem and something does not work for you, in which case you need to talk to Bullseye support. Or you're asking if this will work in which case I think there is something missing since the type of testing you describe; testing one module using two different versions does not make any sense to me.

  • how to get separate coverage for each solution files separately.

    Example : A.sln -> coveraheA.cov

                     B.sln -> coveraheB.cov

  • @Mithun: Before running any tests, copy the coverage file generated when compiling to the new desired names. Then set the COVFILE environment variable to different values for each solution. Or just build and run each solution seperatly deleteing (or probably moving it away for later analysis) the coverage file in between

  • Thanks for your reply cellfish.

    I need one more help. When I'm using covc in command line with vcbuild. I get  exception like this

    Example:

    covc -fC:\ProjectTest\Coveragetest.cov vcbuild C:\Project\A.sln

    "Exception: unknown compiler c:/Program Files/Microsoft Visual Studio 9.0/VC/vcpackages/vcbuild.exe"

    Waiting for Ur reply

    Thanks in advance.

Page 1 of 2 (22 items) 12