Using a little Ruby to automate mstest code coverage runs

RobertHorvick — Mon, 08 Dec 2008 15:29:00 GMT

I hate repetitive tasks - especially ones that are easily automated. Generating code coverage runs using mstest is an example of that.

The goal was to create a script that could do that following:

1) Validate that the input binaries were available

2) Validate that the necessary runtime components were available

3) Instrument the input binaries

4) Build up the test sandbox

5) Start the coverage service

6) Perform the tests using the instrumented binaries

7) Stop the coverage service

Being able to start the process at any point in there was bonus points.

I choose to use ruby, and specifically Rake, to automate this process. Rake is perfect for this type of task. I can easily break the tasks down into their core parts and build up the automation one bit at a time. Creating dependencies between the tasks is trivial. It reads well (even with my poor ruby skills) and has a very nice command line experience.

What I ended up with was a process where I could do this:

C:\ruby\coverage>rake --tasks

(in C:/ruby/coverage)

coverage coverage:all # Perform a complete code coverage pass...

coverage coverage:clean # Clean the coverage temp directory and sandbox

coverage coverage:cover # Instrument the assemblies for code coverage

coverage coverage:merge # Merge the test results

coverage coverage:populate # Populate the sandbox

coverage coverage:restore # Restores the backed up files if they exist

coverage coverage:run # Run the test suite

coverage coverage:start # Start the coverage service

coverage coverage:stop # Stop the coverage service

I can do all the things I set out to do (using “coverage:all”) or onesy-twosy as I need them (e.g. if rake aborts midway through the “run” step the coverage service will need to be manually stopped before the next run – I could modify a task to handle this more gracefully but I haven’t yet since the payoff isn’t there yet).

But you came for code ...

Rakefile

require 'rake'

require "win32/dir"

require 'fileutils'

require 'yaml'

def assert_config(config)

['vsperfcmd',

'prodbin',

'root',

'sandbox',

'covtemp',

'covorig',

'vsinstr',

'suitebin']. each { |setting|

assert_config_setting(config, setting)

}

end

def assert_config_setting(config, setting)

fail "Setting \"#{setting}\" not found in coverage.yml" if config[setting].nil?

end

Rake.application.init('coverage')

config = YAML::load(File.open('coverage.yml'))

assert_config(config)

covfiles = FileList.new(

'Microsoft.TeamFoundation.Admin.dll',

'Microsoft.TeamFoundation.Management.Core.dll',

'Microsoft.TeamFoundation.Management.SnapIn.dll',

'Microsoft.TeamFoundation.Management.Controls.dll',

'tfsconfig.exe',

'tfsmgmt.exe'

)

namespace :coverage do

desc "Instrument the assemblies for code coverage"

task :cover => [:clean_coverage, :clean_orig ] do

Dir.chdir(config['prodbin']) do

mkdir config['covtemp']

covfiles.each do |f|

fail "Input file not found: #{f}" if not File.exists?(f)

puts "Instrumenting #{f}"

cmd = "\"#{config['vsinstr']}\" \"#{f}\" /COVERAGE"

sh cmd

assert_file_exists(File.join(config['covtemp'], f), cmd)

copy_coverage(f, config['covtemp'])

end

desc "Restores the backed up files if they exist"

task :restore do

Dir.chdir(config['prodbin']) do

covfiles.each do |f|

origfile = "#{f}.orig"

if File.exists?(origfile)

rm f if File.exists?(f)

mv origfile, f

instrpdb = f.gsub(/(.*)\.(dll|exe)$/, '\1.instr.pdb')

rm instrpdb if File.exists?(instrpdb)

end

task :clean_backup do

Dir.chdir(config['prodbin']) do

covfiles.each do |f|

origfile = "#{f}.orig"

rm origfile if File.exists?(origfile)

instrpdb = f.gsub(/(.*)\.(dll|exe)$/, '\1.instr.pdb')

rm instrpdb if File.exists?(instrpdb)

end

task :clean_coverage do

FileUtils.rm_rf(config['covtemp'])

end

task :clean_orig do

FileUtils.rm_rf(config['covorig'])

end

task :clean_sandbox do

FileUtils.rm_rf(config['sandbox'])

end

desc "Start the coverage service"

task :start do

mkdir config['sandbox'] unless File.exists?(config['sandbox'])

sh "\"#{config['vsperfcmd']}\" /start:coverage \"/output:#{File.join(config['sandbox'], 'adminops.coverage')}\" /user:redmond\\vseqa1 /user:redmond\\tfssvc /user:#{ENV['USERDOMAIN']}\\#{ENV['USERNAME']} /waitstart"

end

desc "Stop the coverage service"

task :stop do

sh "#{config['vsperfcmd']} /shutdown"

end

desc "Run the test suite"

task :run do

Dir.chdir(config['sandbox']) do

puts "Running Unit Tests"

sh "aotest.exe /assembly:AdminOps.UnitTests.dll /test * /out:logs"

end

desc "Merge the test results"

task :merge do

end

desc "Populate the sandbox"

task :populate=> [:clean_sandbox] do

mkdir config['sandbox']

puts "Copying Product binaries..."

FileUtils.cp_r(Dir.glob(File.join(config['prodbin'], 'Microsoft.TeamFoundation.*')), config['sandbox'])

cp File.join(config['prodbin'], 'NetFwTypeLib.dll'), config['sandbox']

puts "Copying Test binaries..."

FileUtils.cp_r(File.join(config['suitebin'], '.'), config['sandbox'])

puts "Copying Instrumented binaries..."

FileUtils.cp_r(Dir.glob(File.join(config['covtemp'], '.')), config['sandbox'])

end

desc "Perform a complete code coverage pass from scratch"

task :all => [:clean, :cover, :populate, :start, :run, :stop, :merge]

desc "Clean the coverage temp directory and sandbox"

task :clean => [ :clean_coverage, :clean_sandbox, :restore, :clean_backup ]

end

def assert_file_exists(file, cmd)

if not File.exists?(file)

puts "COVERAGE ERROR: file not found: #{file}"

puts "Executed: #{cmd}"

end

def copy_coverage(file, dir)

if File.exists?(file)

cp file, dir

else

die "Assembly not found for assembly #{file}"

end

pdbfile = file.gsub(/(.*)\.(dll|exe)$/, '\1.pdb')

if File.exists?(pdbfile)

cp pdbfile, dir

else

puts "WARNING: PDB file not found for assembly #{pdbfile}"

end

instrpdbfile = file.gsub(/(.*)\.(dll|exe)$/, '\1.instr.pdb')

if File.exists?(instrpdbfile)

cp instrpdbfile, dir

else

puts "WARNING: INSTR PDB file not found for assembly #{instrpdbfile}"

end

coverage.yml

vsperfcmd: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsperfcmd.exe

prodbin: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386

root: C:/VMDEV/TfsArch1/binaries/x86chk

sandbox: C:/SANDBOX

covtemp: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage

covorig: C:/VMDEV/TfsArch1/binaries/x86chk/bin/i386/coverage/orig

vsinstr: E:/Program Files/Microsoft Visual Studio 10.0/Team Tools/Performance Tools/vsinstr.exe

suitebin: C:/VMDEV/TfsArch1/binaries/x86chk/SuiteBin/i386/tfs/AdminOps/bin

So there you have it…

I go to my ruby command line (which is just a normal command line with the ruby bin directory in the path - e.g. “set PATH=%PATH%;c:\ruby\bin”) and perform:

C:\ruby\coverage>rake coverage:all

When the tests are done the coverage file I created is in c:\sandbox ready to be imported into VS or merged with other files or dumped to excel or whatever.

And one note about the rakefile – you may notice that I call “aotest.exe” not “mstest.exe” – aotest is just a little wrapper around mstest that one of our test devs wrote. I could have just as easily used mstest.

A link to my Ruby and Rails blog

RobertHorvick — Mon, 19 Nov 2007 21:58:00 GMT

Since life can't be all Team Foundation Server and rainbows I've started getting serious about learning more about Ruby and the Rails framework. To keep the TFS focus on this blog I've started a new blog dedicated to non-TFS tech topics http://www.GhostOnThird.com.

Robert Horvick's Weblog : ruby

Using a little Ruby to automate mstest code coverage runs

But you came for code ...

Rakefile

So there you have it…

A link to my Ruby and Rails blog