Measuring Code Coverage on Devices with Limited Memory

In this article, we would like to explore the possibilities that are available to a Squish Coco user if he or she wants to use the tool on devices with limited memory.

It is relatively clear that things don't get better if we have less memory. Therefore, the whole discussion is centered around the question of: what can we do to "suffer less" because of the absence of additional (to whatever memory we have) memory. Let's start then...

Optimization of Memory Consumption: Main Considerations

From the instrumentation of the code comes an additional memory overhead. This fact is rather unfortunate, but the instrumentation is an essential part of Squish Coco, which makes possible more precise results compared to other (i.e., not involving instrumentation) techniques. And, of course, by definition, an instrumentation cannot avoid use of certain additional memory.

We can, however, choose, by fine-tuning the CoverageScanner (the part of the Squish Coco suite responsible for instrumentation), which features to use and to what extent they are necessary for us, and which features are more of a luxury. Of course, we are probably not willing to get rid of some luxury for free. Thus, it is also important to understand the impact which every discussed feature has on memory consumption.

There are three main things we can do to reduce the memory consumption drastically:

1. Use a more basic coverage level (more on this later)
2. Measure ‘hits' instead of ‘counts'
3. Reduce the range of the counter used for counts

We start with the second and third options:

Using ‘Hits' Instead of ‘Counts'

By default, the CoverageScanner uses counters to make it possible to count how many times a specific part of the code was executed.

A counter is a variable (which is inserted into the code during instrumentation), where we keep a current count. So, if we want to be able to track up to 255 executions of one particular piece of code, we have to have a counter of a size of at least 1 byte.

On the other hand, it may be quite satisfactory just to know that the piece of code was executed at least once, meaning we had a ‘hit'. Of course, we still have to store this into a memory using a variable, but a hit requires less memory to store information (because it is only binary) and (if you think about that) less memory to keep track of, because we don't need to ‘count', so we do not need additional operations in the code.

Now, you can imagine how many such variables (for ‘hits' or for ‘counts') we have to have in an average program. It is approximately number of blocks (functions, loop bodies, etc.) + number of branches (e.g. if ... then ... else ... branches).

We use --cs-hit as an option when compiling with CoverageScanner and get ‘hits' instead of ‘counts'.

Reduce the Range of the Counter Used for ‘Counts'

As of the date of this article, Squish Coco supports 1, 4 and 8 byte counters, meaning they have ranges of 2⁸, 2³² and 2⁶⁴, respectively. Furthermore, the default value is 8 (range 2⁶⁴). As you may see, 2³², for instance, is approximately 4 billion executions and 2⁶⁴, obviously, 4x4 billion billion executions. So, without much of a doubt, we may reduce 4x4 billion billion to "only" 4 billion which gives us 4 bytes of memory for every counter variable.

We use an option --cs-counter-size=<number>, where <number> is either 1 or 4 to adjust the size of the counter used. Of course, it is only useful if you do not already have the --cs-hit option.

Use a More Basic Coverage Level

We can list coverage levels provided by Squish Coco in the ascending order from ‘basic' to ‘advanced' as follows:

1. Statement Block Coverage
2. Decision Coverage (includes Statement Block Coverage)
3. Condition Coverage (includes Decision Coverage)
4. Modified Condition/Decision Coverage, or MC/DC
5. Multiple Condition Coverage, or MCC

These levels have a direct impact on memory consumption: a higher level means greater memory consumption.

Please
note that every level includes Statement Block coverage, so it is an
absolute minimum. Other levels add some quality together with some
overhead, including, of course, memory overhead.

• Statement Block Coverage: we count blocks of code statements that are executed.
• Decision Coverage: every decision regardless of how many conditions were involved in ‘making' a decision.
• Condition Coverage: Decision Coverage + every condition, separately. ‘Separately' means that we are not taking into account combinations of the conditions, we are just interested in knowing if every condition was TRUE and FALSE and (possibly) how many times. We cannot guarantee here for any particular condition which was tested that it influenced a decision.
• MC/DC: for every condition at least minimum number of combinations with other conditions that can affect a decision.
• MCC: every possible combination of conditions is taken into account.

To track all combinations, the CoverageScanner uses, for every decision, a truth table formed with lines consisting of particular combinations of conditions.

Now, it is clear that the use of a more primitive coverage level will help us to reduce the size of the memory used. For example, instead of using MC/DC, we may use a Condition Coverage.

However, there is an even better option:

As mentioned before, the CoverageScanner uses tables to track conditions for MC/DC and MCC coverage levels. Default number of lines in the tables is 8000. This can be changed with the help of the options --cs-mcc-max=<size> and --cs-mcdc-max=<size>. They set maximum number of lines in the table for MCC and MC/DC, respectively, but the most important part is the fact that if a particular decision requires a table with the <size> greater than the set maximum, CoverageScanner falls back to a condition coverage for that decision. In other words by using this option we can (partly) preserve a quality level, but, at the same time, reduce memory consumption.

There is another option which may be considered as separate, but we think it is best to mention it here while discussing coverage levels. It is called a partial instrumentation.

Consider a typical if-statement: if (a == true) doSmth(). We have an ‘action' for ‘condition is true', but since we have no ‘else', there is in fact nothing for ‘condition is false'. In some situations, it is not a great evil to think of ‘no action' as effectively non-existent. Thus, we can spare some counting (hence, reduce memory consumption) for such statements because we will count only ‘condition is true'.

Please be aware that the justification of partial instrumentation strictly depends on code because it is very much possible that the absence of some ‘action' is crucial.

Please use the following options for CoverageScanner

• --cs-statement-block: enable statement block coverage
• --cs-decision: enable decision coverage
• --cs-condition: enable condition coverage (Default)
• --cs-mcc-max=<size>: set maximum size of the instrumentation tables used for MCC if enabled via --cs-mcc
• --cs-mcdc-max=<size>: set maximum size of the instrumentation tables used for MC/DC if enabled via --cs-mcdc
• --cs-partial-instrumentation: enable partial instrumentation

More Options to Reduce Memory Consumption

The three options described above are not the only possible ways to reduce memory consumption when working with Squish Coco.

a) Disable performance counters which are ENABLED by default. If no performance measurement is needed, some counters can be saved.

Please use --cs-no-execution-time option for that.

b) Adjust the output buffer size. There are files with the name of the form <toolname>.cspro. They are responsible for profiles of the building tools ("compilers" -- please see Squish Coco Documentation for further details). This file is very useful for adjusting Squish Coco to our needs and to our limitations. In case of reducing memory consumption, we recommend to adjust in the file OUTPUT_BUFFER_SIZE=<size>, where size is the size of the internal buffer used for generating an execution report. A higher value means potentially better performance, but at the cost of memory.

We
typically recommend a size value of 64 for platforms with limited
memory.

c) Use the --cs-minimal-api CoverageScanner option. In this case, the CoverageScanner API has fewer dependencies to external libraries than the default API. Among other things, it should help to reduce memory consumption.

d) Use --cs-combine-switch-cases. This option allows the CoverageScanner not to distinguish between case labels that lead to the same code (fall-through). So, if it is enabled, we do not need some counters.

Options to Be Avoided on Devices with Limited Memory

A few words about what not to use or enable. The latter implies that we are talking about features which are not available by default.

DO NOT use the option --cs-record-empty-functions which allows CoverageScanner to instrument functions with an empty body.

Summary

USE

(in order of descending importance, i.e., impact on memory consumption):

1)

FIRST ALTERNATIVE:

either --cs-condition (Default)

or --cs-decision

or --cs-statement-block (not recommended)

to set a coverage
level with less than a maximum memory consumption.

OR

SECOND ALTERNATIVE:

either --cs-mcc-max=<size> or --cs-mcdc-max=<size>

to set the maximum size of the instrumentation tables used for MCC or MC/DC, respectively (with potential fallback to condition coverage)

2) either--cs-hit

or--cs-counter-size=<number>, where <number> is one of 1 or 4 to reduce size of the counter used.

3) --cs-partial-instrumentation

to enable partial
instrumentation

4) --cs-no-execution-time

to disable
performance counters

5) OUTPUT_BUFFER_SIZE=<size> (e.g. OUTPUT_BUFFER_SIZE=64) in ‘.cspro'-file to adjust (reduce) the size of the internal buffer used for generating an execution report.

6) --cs-minimal-api

7) --cs-combine-switch-cases

DO NOT USE

(in order of descending importance i.e. impact on memory consumption):

1) --cs-mcc or --cs-mcdc

UNLESS accompanied by --cs-mcc-max=<size> or --cs-mcdc-max=<size>, respectively.

2) --cs-keep-instrumentation-files

3) --cs-record-empty-functions