What is incoherent instrumentation?

Introduction 

There are two related phenomena in Coco that have a common cause: 

• In CoverageBrowser or in the HTML report, one can see a file multiple times with file names distinguished by a numerical prefix. A file named “header.h” appears as “header.h#1” and “header.h#2” or in even more versions. 
• During compilation, Coco prints a warning that begins with 

Warning (Squish Coco): Instrumentation of source file 'myproject/header.h' is different. 

This line is followed by several other warning lines that describe the cause of the problem in greater detail. 

Both are examples of incoherent instrumentation, where a file is compiled twice or more, but in different ways. 

In this article, we will explain what this means, how this happens, and how it can be prevented. 

What does the warning mean? 

The ultimate cause of this warning is the C/C++ preprocessor. This is a component of the compiler that replaces parts of the source code at the beginning of the compilation before the translation into machine code has even started. This is necessary because C and C++ code often should run on several different platforms. With the preprocessor, it is possible to have platform-specific constants in the source code and set their values only at compile time, or to have several versions of the same code but for different platforms and to select the right one only at compile time. 

#ifndef HEADER_H 

#define HEADER_H 

static inline char letter() 

{ 

#ifdef A 

    return 'a'; 

#else 

    return 'b'; 

#endif 

} 

#endif 

A header file called header.h with a function that behaves differently depending on whether the preprocessor macro A is defined. 

The file header.h is included by two different source files, a.cpp and b.cpp. If these files are then compiled with different command line options and later merged with a common main program, incoherent instrumentation occurs. 

It is even possible to compile the same source file twice and with different preprocessor options in the same program, whether intentionally or not. And here the problem for Coco begins because the result are two different subprograms that usually need different instrumentation. Coco then has to consider these subprograms as if they were the results of compiling two distinct source files—files that are called, for example, “header.h#1” and “header.h #2”, if the original file was “header.h”. These numbered file names then appear in CoverageBrowser and in the HTML reports. 

The file header.h appears twice in CoverageBrowser. 

When does the warning appear? 

One would think that a file needs to be compiled only once and that, therefore, this kind of problem would be rare. But the compilation of the same file several times actually happens in every realistic project. The main cause is the header files. A header file is meant to be included by several different source files, and it may contain instrumentable code, for example, in inline functions. So if a header file is read twice in different compiler runs with different preprocessor options, it may be instrumented differently. 

Ordinary source files (those that end in “.cpp”) are more rarely compiled twice, but it also happens with them. Apart from multi-platform compilation, it could be that a file is used in different executables and compiled for each of them separately. In most cases, this could be avoided by putting the file into a library, but sometimes two compilations with different parameters are exactly what is intended. 

Why does incoherent instrumentation occur in a project? 

In a larger project, it can easily happen that parts of the build process have different command line options. Or if you use an IDE, it may set many of the compilation options on its own and it may not provide an overview over the command line options for all the different source files. The same might be true for automated build systems like CMake if the project is large enough. 

Another possible cause is that debug and release builds are mixed. (This is possible under Linux and similar systems but is usually a bad idea.) It has, for example, an influence on the ASSERT macro, which compiles to nothing in a typical release build. There may also be inconspicuous changes in the compilation of the system header files when changing between debug and release builds. 

But there are also reasons to have different compiler settings for different parts of a project, especially if it is very big and must comply with the build requirements of the external libraries that it uses. 

What disadvantages does incoherent instrumentation bring? 

If a file is compiled and instrumented more than once but differently, Coco treats these instrumentations as different files. This then means that each version of the file needs to be covered separately and full coverage is more difficult to reach—which may or may not be intended. 

The code in each version of the header file is only used by a subset of the program code. To reach full coverage for one version of this file, it is, therefore, necessary to write tests that exercise exactly that subset of the compiled code that uses this specific version of the header. Writing such tests will be difficult if the incoherence is caused by “random” differences in the compiler options. If, however, the different versions of a header file belong to different submodules or to code that is run on different platforms, then there will be no real problem. 

An incoherently instrumented header file also points to a possible problem with the code, at least if it is an unexpected incoherency: Header files provide an interface between different parts of a program, and when such a file is translated differently at different places in the code, the communication between them may fail in a way that is difficult to find out afterwards—another reason why incoherent instrumentation should be avoided. 

What can be done to prevent incoherent instrumentation? 

Coco offers several methods to find incoherencies. First, one can search in the compiler output of an instrumented build for the warnings that arise when two incoherent instrumentations of the same file are merged. They also report the command line options of the different builds of the same file. This may make it possible to find and correct the part of the build script that caused the incoherence. Changing the actual build script requires knowledge of the specific build system being used, but Coco provides information on inconsistent compilation flags that may have resulted in incoherent instrumentation. 

If you actually want a totally coherent build, you may also set the Coco command line option --cs-warnings=error. It lets the compilation fail when Coco issues a warning so that you are automatically notified when there is a problem. A possible disadvantage is that the build will also fail after any other warning. 

Another method is to have a more local approach and to decide for each file separately whether incoherence is a problem. In this case, you can use CoverageBrowser to detect incoherently instrumented files: They appear with a numbered suffix in the “Sources” window file list. It is then also possible to detect the command line parameters for their compilation, namely by clicking on the file name, whereupon the command line parameters of their build become visible in the “Explanation” window. 

Finding the compiler options for the file header.h#1. 

Uses for code quality 

While there are some cases, where incoherent instrumentation is expected, for example when a program is built on more than one platform, in other cases it points to problems with the code. It may mean that a header file has been read in two different ways, so that the code that calls a function and the code of the function cannot interact correctly; or it just means that full code coverage can only be reached with unnecessary many tests: In any case, regular checking for incoherencies will help to improve code quality.