Coco Test Engine – The New Era of Code Coverage Analysis

Our Coco 7 major release brought a highly anticipated test data generation feature (or as we call it: Coco Test Engine) to code coverage analysis tool Coco. We interviewed Sébastien Fricker, Lead Developer of Coco, to get to the bottom of the update and explore how Coco Test Engine improves the world of code coverage.

In short, the test data generation feature improves the code analysis process and makes code quality assurance easier by:

• Helping you reach the highest feasible coverage level faster
• Sorting out redundancies in code coverage analysis
• Providing automatic collections of test data that includes edge cases

Introducing Coco Test Engine - Overview Video

Interview with Sébastien Fricker, Lead Developer of Coco

What was the challenge in code coverage analysis that you set out to solve?

Previously, software developers have had to come up with their own unit tests and test data. This is an arduous task in the first place, and not optimal at all, considering that the data you come up with does not even cover all potential edge cases that you should test for.

As an example, let’s say you are developing a calculator app. To test it, you would write unit tests for short expressions. Then you would input test data manually: 1+2, 3*4, 3/2, 3-2 and so on – to see what output the calculator generates.

Even simple cases like this used to require a lot of manual effort in coming up with the test cases and running the analyses. In general, the test code can be much bigger than the production code. It is not rare to see that for a small source code snippet, a big set of unit tests needs to be written to achieve a coverage of 100%.

That’s why, in most cases, developers concentrate their effort on the usual test cases (typical usages, normal workflows, correct data…) and omit a lot of edge cases (error cases, illegal input, interrupted workflows, etc.). For our example, it would be a division through zero (1/0), empty expression or illegal like a number with several dots (1.2.3). 

In the end, you spend a lot of time finding test cases to increase your confidence in the quality of the code, but you still wouldn’t achieve high coverage after forgetting or missing some cases in your tests. There’s a real conflict between feature quality and time spent on developing the feature.

How does test data generation help overcome the problem?

Test data generation is exactly what it sounds like. Coco Test Engine automatically generates test data to improve the quality of your testing and helps cover various edge or error cases. Just put the algorithm to work and let it find cases that you don’t normally cover, or complete the existing test suite.

Coco’s test data generation reduces the complexity of writing data-driven tests in three ways:

1. It separates test code and test data. For many test frameworks, the concept of data- driven testing is not provided, which means that the developer needs to write a separate function per test data. If supported, test data generation and the unit test code is put on the same file, which imposes that the data conforms to the C++ syntax and increases the size of the unit test code.
2. The data editor makes it easy to review and edit tests in spreadsheet form.
3. The set of tools helps discover new test cases but also streamlines the processes of updating test data with code changes, and data validation by the tester.

“Coco Test Engine helps refactor code with more confidence. You can focus on the normal cases and let Coco cover the rest.”

Describe a situation where the test data generation feature is particularly useful?

Let’s say you want to rewrite a piece of code – this refactoring happens all the time. If you go ahead and make modifications without having proper unit tests in place, your work can cause many issues. 

Coco with test data generation automates unit tests and the verification of your edits to the code. This works because we have a reference code: the code being refactored is supposed to work. So the test case generation will produce valid test data that can be used as a testing reference.

It essentially helps in refactoring code with more confidence. You just let it generate a test suite automatically and wait until you get a high coverage. Then you can focus on the code modifications and let the Test Engine algorithm cover various “trash” data that need to be tested.

Which industries can get the most value out of improved code coverage analysis?

The application or software you’re working on doesn’t need to be complicated for this to be useful. Simple code runs many critically important systems in for instance medical devices, aviation, and vehicles. In general, high code coverage, and therefore also the test data generation feature, are a must in safety-intensive industries and software.

Let’s take aviation as an example. The industry has high standards that demand code coverage of 100% in some cases. If achieving this isn’t possible, each uncovered line of code needs to be justified.

You can get from 10% to 11% coverage pretty easily, but as the coverage grows, getting to a higher level gets increasingly more difficult. If 50% to 51% requires a few minutes, going all the way up to 90% can take an hour. The time to test grows exponentially when you reach 100% coverage. Coco gets you to the needed coverage level faster, because it uses the data provided by the tester and completes it automatically by finding new test cases.

But finding test data is not enough because the quality of the data also needs to be reviewed. Let’s continue with the calculator example. The user can easily provide the input data for a division “4/2” and the result is “2”. By letting Coco discover new tests, it would find for instance the division “1/0”. But what is the correct result of this output? “Inf”? “Illegal expression”? “Error:Division through zero”? 

The output needs to conform to what the user expects, and only a human being can decide it – the application cannot know on its own. This is why a validation flag is added for each instance of data. It lets you mark data as verified and not just good because it was a previous output of a function call.

“Going all the way up to 100% code coverage can take several months for some products. Coco gets you to the needed coverage level faster, and helps to give justification for the remaining code.”

How does Coco Test Engine work in practice?

Here’s a simplified outline of the process:

1.  Write a small unit test code using 2 macros:
   1. FETCH to get new input data
   2. CHECK to compare the output to the expected result
2. Compile the unit test
3. Use the data editor to create some test cases and debug it like a normal application
4. Execute test discovery to find new cases and increase the coverage
5. Review the generated data to see if the output corresponds to what is expected 

One thing to consider with Coco’s test data generation: The initial setup can take a bit of time because you need to validate more test data given by the algorithm. But I can assure you that over time the benefit will be greater than that.

Tutorial: How to set up Coco Test Engine

Try it yourself

Experience faster and more complete code coverage analysis. Start a free trial with Coco →

Or find out more about our code coverage analysis tool →