Record code coverage for MSTest unit tests, with opencover (nant enabled)

I’ve been working on some unit tests lately. The tools provided with Visual Studio 2015 are pretty nice, but I wanted a report that shows, which classes are tested and what lines / branches are covered. This report should be generated every time our buildserver (Jenkins) completes a build. With this information it is possible to decide, which functions need more or maybe better tests.

The big picture

For automating the build process, I choose Jenkins as our buildserver. I scripted the whole build process with nant. That enables me to store the build script next to my code files in the same repository. If I want to change the way Jenkins builds my current source, I only have to change the nant script and Jenkins executes the build always matching the source.

The tests are run, with help of vstest.console.exe, which is part of visual studio. OpenCover calls vstest.console.exe and loads the corresponding .pdb files to gather information about which test touches which part of your code. The result is a coverage report, which can be passed on to the next tool.

ReportGenerator takes the coverage report, generated by OpenCover and generates a nice HTML report from it. You can store this historical to the changes in your test coverage.

Details, please!

First of all, we need to figure out, how MSTest tests are executed from the command line, which is quite self-explanatory:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe “MSTestReport.Tests\bin\Release\MSTestReport.Tests.dll” /Logger:trx

This will run our tests. No, let’s see how OpenCover is called from the command-line. OpenCover will be installed as a nutget Package, so the command is as follows:

Packages\OpenCover.4.6.166\tools\OpenCover.Console.exe \
-register:user \
-returntargetcode \
-target:”C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe” \
-targetargs:“MSTestReport.Tests\bin\Release\MSTestReport.Tests.dll” /Logger:trx \
-filter:+[*]* -[Tests*]* \
-hideskipped:All \
-output:artifact\report\coverage.xml \
-log:Info

As you can see, OpenCover calls vstest.console.exe with its corresponding arguments with the –target and –targetargs parameter.
Execution of this command will result in a file called coverage.xml, which is stored in the folder “artifact\report\”. This file is used by ReportGenerator to generate the report.
ReportGenerator is again very easy to use:

Packages\ReportGenerator.2.3.2.0\tools\ReportGenerator.exe –reports:artifact\report\coverage.xml –targetdir:artifact\report

That’s all. You’ll find a very nice Report in the folder artifact\report

Summary of report from Open Cover

Report Summary from OpenCover

Putting it all together

Now it’s time to put everything together in a nant script.

I created three targets. The first target is called “test”. It will wrap the execution of multiple tests on multiple test assemblies.

<!-- tests every configured assembly -->
<target name="test" description="tests every configured assembly"> 
 <!-- the reports will be stored in this directory -->
 <property name="output.dir" value="${reports.dir}\testcoverage" />
 <!-- clean up -->
 <delete dir="${output.dir}" if="${directory::exists('${output.dir}')}"/>
 <mkdir dir="${output.dir}" />
 <!-- testing the first dll --> 
 <property name="unittestdll" value="MSTestReport.Tests\bin\${build.configuration}\MSTestReport.Tests.dll" />
 <echo message="Testing: ${unittestdll}" />
 <call target="test.dll" />
 <!-- testing the second dll -->
 <property name="unittestdll" value="MSTestReport.Additional.Tests\bin\${build.configuration}\MSTestReport.Additional.Tests.dll" />
 <echo message="Testing: ${unittestdll}" />
 <call target="test.dll" />
 <call target="generate.report" />
</target>

The script cleans up the output directory, and than it executes the target test.dll for each assembly to test.
The second target will be „test.dll“ – this executes OpenCover, with then executes the vstest.console.exe to do the testing. Please note the “–mergeoutput” parameter. This parameter will merge all results in one file so you can create one big report, combining all tested assemblies.

<!-- calls opencover and merges all results in one file: coverage.xml -->
<target name="test.dll">
 <exec program="${opencover.dir}/tools/OpenCover.Console.exe">
 <arg value="-register:user" />
 <arg value="-returntargetcode" />
 <arg value="-target:${mstestpath}" />
 <arg value="-targetargs:&quot;${unittestdll}&quot; /Logger:trx" />
 <arg value="-filter:+[*]* -[Tests*]*" />
 <arg value="-hideskipped:All" />
 <arg value="-output:${output.dir}/coverage.xml" />
 <arg value="-mergeoutput" />
 <arg value="-log:Info" />
 </exec>
</target>

The last target is „generate.report“. This target tagkes the coverage.xml file and generates a report from it.

<!-- generates the report with help of ReportGenerator.exe -->
<target name="generate.report">
 <exec program="${reportgenerator.dir}/tools/ReportGenerator.exe" verbose="true">
 <arg value="-reports:${output.dir}/coverage.xml" />
 <arg value="-targetdir:${output.dir}" />
 </exec>
</target>

That’s all. You can now add a call of the „test“ target in your Jenkins Config or call the tests manually from the command line:

tools/nant/NAnt.exe test

XUnit, NUnit?

This solution works with most of the available testing tools. The only thing you have to change is the command to run the tests, that OpenCover uses!

Please use my test solution I uploaded to github, to see everything in detail.

Sources:

Share