Test Impact Analysis

What is Test Impact Analysis?

Test Impact Analysis (TIA) is an optimization method used to determine which tests exercise a specific code change. The goal of TIA is to improve the efficiency of the software testing process, a goal critically important in development projects where the amount of test cases grows rapidly in tandem with changes to production code. 

A Test Impact Analysis is conducted with a code coverage tool. Utilizing a code coverage tool, engineers are able to prioritize which subset of their test suite(s) are run, through the tool’s determination of which changes are hit by a certain test. TIA with a code coverage tool can also uncover duplicate tests; that is, multiple tests which hit the same sources. Employing this analysis will enable your development team to streamline test automation, no matter who wrote the source code or how long ago it was written.

This article will walk you through how to use Squish Coco for Test Impact Analysis.

Coco’s Integration of TIA

Consider the following development scenario where you’d want to conduct a TIA. In a large project, a last-minute patch has to be evaluated. There is not enough time to run the entire test suite, but some risk assessment needs to performed. With Coco’s Patch Analysis feature, you can specifically display the code coverage for the changed lines of code, and find the tests in a large suite that cover them. You can then see how risky the changes are.

You can try this feature yourself by following these instructions:

Prepare

If you have not already done so, adjust Coco to map your test cases to executions by name. Coco’s documentation lists the steps for some specific frameworks, including CppUnit and QTestLib. 

Instrument and run the complete test suite for your application. This will result in a csmes file and a csexe file. Import the data from the csexe file into the csmes file via the CoverageBrowser or the cmcsexeimport tool.

Develop

Develop your project: write new code and add new features. 

Run the git command diff with the current changes:

$ git diff --staged > diff.txt

Analyze

Now analyze the difference with the cmreport tool:

$ cmreport -m report.csmes -p diff.txt \
--csv-excel=patchanalysis.csv \
--section=patch-execution 

This will result in a .csv file that holds all of the executions that are covering the changed parts of your code.

We can now print out the test case names:

$ sed '1,/^"Execution Name.*$/d' patchanalysis.csv \
| grep '^"\K([^,"]+)' -oP

You can now feed your test framework with this information in order to execute only the mentioned test cases. This will greatly reduce the time spent on testing of future incremental changes to production code. 

For more info on Coco’s Patch Analysis feature, visit our documentation.