Code Coverage with QTest

Today’s tip will show how to measure code coverage for unit tests written using the Qt Test framework (or short QTest).

The example

A simple example for code that uses Qt which is tested via the Qt Test framework can look like the following. To keep it short the application logic is a function that does only slightly more than the famous “Hello World” example:

// applogic.h
#include <QString>

bool doSomething(int val, const QString &msg);

// applogic.cpp
#include "applogic.h"
#include <QTextStream>

bool doSomething(int val, const QString &msg) {
  QTextStream str(stderr);
  str << "Doing something; " << msg << "; val " << val << endl;
  if (val > 21) {
    str << "Doing something more" << endl;
    return true;
  }
  return false;
}

To verify that this function does something different depending on the arguments we’ll also write a QTest-based unit test.

// test_applogic.cpp
#include <QtTest/QtTest>
#include "applogic.h"

class TestLogic : public QObject {
  Q_OBJECT
 private:
  Q_SLOT void testNothing() {
    QVERIFY(doSomething(0, "nothing") == false);
  }
  Q_SLOT void testAnswerToEverything() {
    QVERIFY(doSomething(42, "answer to everything") == true);
  }
};

QTEST_GUILESS_MAIN(TestLogic)

#include "test_applogic.moc"

And to make the example complete we’ll also need a qmake project that can build and run the unit test application.

# test_applogic.pro"
CONFIG += testcase
QT = core testlib
SOURCES = applogic.cpp test_applogic.cpp

Adding Coverage Information

The first step for getting coverage information for the above test is to instrument all code using Squish Coco. One way to archieve that in a Qt project is to replace and extend the compiler and linker configuration in qmake:

coco {
    message(Building with Squish Coco)
    # Exclude unit test code from coverage
    COVERAGE_OPTIONS = --cs-exclude-file-abs-wildcard=*/test_*
    # Exclude Qt inline code from coverage
    COVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=$[QT_INSTALL_HEADERS]/*
    # Append excludes to compile/link steps
    QMAKE_CFLAGS += $COVERAGE_OPTIONS
    QMAKE_CXXFLAGS += $COVERAGE_OPTIONS
    QMAKE_LFLAGS += $COVERAGE_OPTIONS
    # Replace toolchain with Coco wrappers
    QMAKE_CC=cs$QMAKE_CC
    QMAKE_CXX=cs$QMAKE_CXX
    QMAKE_LINK=cs$QMAKE_LINK
    QMAKE_LINK_SHLIB=cs$QMAKE_LINK_SHLIB
    QMAKE_AR=cs$QMAKE_AR
    QMAKE_LIB=cs$QMAKE_LIB
}

The block-syntax allows to enable/disable coverage instrumentation by passing or omitting ‘CONFIG+=coco’ on the qmake commandline. The custom COVERAGE_OPTIONS variable further configures Squish Coco. It omits instrumentation of the actual test code as well as any inline code that gets expanded from Qt include files.

To enable code coverage instrumentation we run ‘qmake test_applogic.pro CONFIG+=coco’. Afterwards, rebuilding and executing the unit test can be triggered by executing ‘make clean check’.

Finally, the resulting coverage information can be loaded into Coco’s CoverageBrowser. This can be done by opening the Instrumentation Database ‘test_applogic.csmes’ followed by loading the Execution Report from ‘test_applogic.csexe’.

The result in CoverageBrowser should look similar to this:

As can be seen from the function list, the application logic in ‘doSomething()’ has been fully covered. However, there’s room for improvement because it is not visible which test actually covered which part of the code. Coco only shows that both possible outcomes of the ‘if()’ decision have been covered by some test.

Improving coverage information for QTest

To have more context for coverage based on what test is currently being executed we can extend the unit test code a bit. Coco can be informed, where a test is started and where it ends as well as what test is currently being executed. For QTest this information can be provided by implementing slots that are called automatically before and after each testcase:

// Extension to test_applogic.cpp"
// New base class to inherit test classes from
class TestCoverageObject : public QObject {
  Q_OBJECT
 public:
  virtual void initTest() {}
  virtual void cleanupTest() {}

 protected:
  Q_SLOT void init() {
#ifdef __COVERAGESCANNER__
    __coveragescanner_clear();
#endif
    initTest();
  }

  Q_SLOT void cleanup() {
    cleanupTest();
#ifdef __COVERAGESCANNER__
    QByteArray testName("test_applogic");
    testName += '/';
    testName += metaObject()->className();
    testName += '/';
    testName += QTest::currentTestFunction();
    __coveragescanner_testname(testName.constData());
    __coveragescanner_teststate(QTest::currentTestFailed() ? "FAILED" : "PASSED");
    __coveragescanner_save();
#endif
  }
};

// Inherit from TestCoverageObject instead of QObject
class TestLogic : public TestCoverageObject {
   ...
};

After rebuilding and executing the test and loading the new Execution Report into CoverageBrowser, the list of Executions will now be named automatically. There is no need to provide a name when loading an Execution Report anymore.

Additionally the Explanation for the ‘if()’ decision now explains which part of the decision was covered by which testcase.

Final thoughts

To keep the example short, the application logic was built as part of the unit test. For more complex projects the application logic might be part of an extra library. In that case, one would have to instrument that library for code coverage as well. It will also make sense to share the qmake code that enables coverage information by moving it into a qmake include file.