Measure code coverage

Use code coverage to measure how much code a process runs during a test or benchmark. After you run code coverage, you can use the analysis to create additional test cases that increase coverage. You can also use the analysis to determine a quantitative measure of code coverage, which is a direct measure of the quality of your tests.

Code coverage uses a signal to tell the application to send information back to the Momentics IDE for BlackBerry. Depending on the application design, there are several possible risks that can result from running code coverage from the Momentics IDE:

  • It can change or break the behavior of applications that it monitors.
  • It can cause code to run that a test suite does not test.
  • It can result in data not being collected.

  1. On the toolbar, select the Launch configuration for your app.
  2. In the Launch mode drop-down list, select Coverage.
  3. In the Launch target drop-down list, select a device or simulator.

    To resolve any device connection issues, click Edit, and follow the instructions on the Device Properties dialog box.

    Screen showing the Launch Target drop-down list with the Edit icon, which opens the Device Properties dialog box.

  4. Click Momentics build button. The Console view displays the output.
  5. Click Momentics Launch button.

Block coverage

The code coverage tool uses the gcov metrics that the gcc compiler produces. The Momentics IDE presents these metrics as line coverage, and shows which lines are fully covered, partially covered, and not covered at all. The Momentics IDE also presents percentages of coverage in terms of the actual code covered, and not just lines.

The coverage metrics provide basic block coverage, or line coverage, which describes whether a block of code is run. A block of code does not have any branch point within it, so that the path of execution enters at the beginning and exits at the end.

The Momentics IDE tracks the number of times that the block of code has been run. It uses this information to determine the total coverage for a particular file or function. It also uses this information to analyze the blocks on each line and to determine the level of coverage for each line.

Although the gcc compiler produces metrics for branch coverage, the Momentics IDE doesn't provide this information.

How the coverage tool works

The code coverage tool works with the compiler (gcc), the QNX C library (libc), and optionally the remote target agent (qconn). When you enable code coverage for an application, the compiler instruments the code to count each branch execution to a basic block at runtime. During the build, the Momentics IDE produces data files to recreate the application's flow graph, and to provide the line locations of each block.

Since the Momentics IDE creates secondary data files at compilation time, be careful when building your applications in a multitargeted build environment, such as the BlackBerry 10 OS.

You must either:
  • ensure that the last compiled binary is the one you're collecting coverage data on,

    or:

  • enable only one architecture and one variant (debug or release).

The compiler's optimizations could produce unexpected results, so you should perform coverage tests on an unoptimized, debug-enabled build.

To instrument a static library with code coverage, you must instrument your binary with code coverage, or link with the code coverage library using the following option in the linker command:
-lgcov

This option links in the ${QNX_HOST}/usr/lib/gcc/target/version/libcov.a library.

When a coverage session starts, you can immediately view the data. The QNX Code Coverage perspective contains a Code Coverage Sessions view that lists previous as well as currently active sessions. You can explore each session and browse the corresponding source files that have received coverage data.

Code coverage might not work as expected. The code coverage data for C++ projects includes other functions that are also in the source file, such as static initializer and global constructor functions. In addition, the files included by include statements aren't included in the overall coverage total; only those functions that are in the original source are included for code coverage.

Edit the code coverage settings

  1. On the toolbar, in the Launch mode drop-down list, select Coverage.
  2. Select the Launch configuration.
  3. In the Launch target drop-down list, select the device or simulator.
  4. Click Momentics build button.
  5. In the Launch configuration drop-down list, click Edit next to your launch configuration.

    Screen showing the Edit button in the Launch Configuration drop-down list on the toolbar.

  6. In the Edit Configuration dialog box, click the Coverage tab.
  7. Configure the settings that you need:
    Code Coverage data scan interval (sec)
    Sets how often the Code Coverage tool polls for data. A low setting can cause continuous network traffic. The default setting of 5 seconds should be sufficient.
    Comments for this Code Coverage session
    Your notes about the session, for your own personal use. The comments appear at the top of the generated reports.
    Collect data for
    By default, all code coverage data built with code coverage in a project is included in the current code coverage session. To include referenced projects or to collect data only from certain sources, clear the All Sources in this application compiled with code coverage option. To select the projects or files that you want to collect code coverage data for, click Select.
    Select
    Opens the Projects to include Code Coverage data from dialog box, where you select the projects to include your coverage data (projects and files). Select any project from this list that you want to gather code coverage data for. Projects must be built with code coverage enabled to capture data.
    Advanced
    Defines a signal to enable the dynamic collection of code coverage data. The Momentics IDE sends a signal to suspend the application thread so that it can perform data collection.

Enable code coverage for make projects

If you're using your own custom build environment, rather than the project build environment, you'll have to manually pass the coverage option to the compiler.
To enable code coverage for non-QNX projects:
If you're using qcc/gcc, compile and link with the following options:
-fprofile-arcs -ftest-coverage

For example, your Makefile might look something like the Makefile below, which belongs to the Code Coverage example project included with the Momentics IDE (although, this example includes additional comments):

DEBUG = -g 
CC = qcc 
LD = qcc 
CFLAGS += -Vgcc_ntox86 $(DEBUG) -c -Wc,-Wall -I. -O0 -Wc,-ftest-coverage 
-Wc,-fprofile-arcs 
LDFLAGS+= -Vgcc_ntox86 $(DEBUG) -ftest-coverage -fprofile-arcs  

# CC refers to the program for compiling C programs (the default is 
# qcc. Use 
# CXX as the program for compiling C++ programs. 

# CFLAGS are additional flags to give to the C compiler. Use CFLAGS  
# for the C++ compiler. 

# -c compiles or assemble the source files, but doesn't link, and the 
# -Wc captures the warning messages. The linking stage isn't done. 
# The ultimate output is in the form of an object file for each
# source file. 

# -Wall turns on all optional warnings that are desirable for normal  
# code. -I. adds the current directory to the list of directories to  
# search for header files. Directories named by -I are searched before  
# the standard system include directories. 

# -O0 is an optimization flag that indicates 'Do not optimize.' 

# LDFLAGS are additional flags to give to compilers when they invoke 
# the ld linker. 

# -ftest-coverage -Wc means that Code Coverage is enabled for your  
# project, and the data is used for test coverage analysis. 

# -fprofile-arcs adds code so that program flow arcs are instrumented.  
# During execution, the program records how many times each branch and 
# call is executed and how many times it is taken or returns, and it 
# saves this data to a file with the extension .gcda for each source  
# file.
# 
# For Code Coverage, you'll need the -fprofile-arcs -ftest-coverage  
# options in both the compile and link lines.

dir := $(shell pwd) 

BINS = rbt_client rbt_server 

# This next line is the rule for <cmd>all</cmd> that  
# incrementally builds your system by performing a <cmd>make</cmd>  
# of all the top-level targets the Makefile knows about. It does this by 
# expressing a dependency on the results of that system, which in turn  
# have their own rules and dependencies. 

all: $(BINS) 

# The following line shows a simple rule for cleaning your build  
# environment. It cleans your build environment by deleting all files  
# that are normally created by running make. 

# It has a Target named <cmd>clean</cmd> to left of the colon,  
# no dependencies (to the right of the colon), and two commands that are 
# indented by tabs on the lines that follow. 

clean: 
rm -f *.o *.img *.gcno *.gcda $(BINS) 

# The following lines are Dependency Rules, which are rules without any  
# command. If any file to the right of the colon changes, the target to  
# the left of the colon is no longer considered current (out of date). 
# Dependency Rules are often used to capture header file dependencies. 

rbt_server: rbt_server.o 

# Alternatively, to manually capture dependencies, several automated 
# dependency generators exist. 

rbt_server.o : rbt_server.c rbt_server.h 
$(CC) $(CFLAGS) $(dir)/$< 

rbt_client: rbt_client.o 

rbt_client.o: rbt_client.c rbt_server.h 
$(CC) $(CFLAGS) $(dir)/$< 

To enable Code Coverage for your project, you must use the options -fprofile-arcs -ftest-coverage when compiling and linking.

For example, in the Makefile, you'll have the following gcc options set for Code Coverage:

CFLAGS += -g -fprofile-arcs -ftest-coverage 
LDFLAGS+= -g -fprofile-arcs -ftest-coverage 

Import gcc code coverage data from a project


Previously, and for older compilers, if you launched a code coverage-enabled build process and chose to disable code coverage in the launch configuration, the process wrote the coverage information to a data file (.gcda) at run time, rather than read it from the process's data space. This meant that you could choose to import this data into the Code Coverage tool at a later time. The newer gcc compiler doesn't stream the data coverage; the Momentics IDE waits for the generation of the data file before it copies it back to host machine.

In addition, the Momentics IDE generates notes files (.gcno) when it compiles projects that have enabled code coverage.

There are multiple ways to import a file.

It isn't necessary to move the file you want to import into the Workspace location.

By default, the .gcda files for gcc are located in a folder structure created under /tmp.

When copying a project_name.gcda file into your workspace, you must copy it to the top level of the directory structure. In this case, it is the variant_name/o_g directory.

To import gcc code coverage data from a project:
  1. Create and build a project with code coverage selected.
  2. Create a launch configuration where code coverage is disabled.
  3. Run this configuration.
  4. SelectFile > Import > QNX > GCC Coverage Data and click Next.
  5. Specify the name of the session, project, and platform used to generate the code coverage data. Click Next.
    Now, you'll browse on the remote target to the folder that contains the data file.
  6. Optional: If you want to browse the remote file system for the Coverage protocol type (i.e. .gdca), browse to the location where the data files are located (such as on the remote target, within the workspace, or on the filesystem).
  7. Optional: If there are referenced projects to include data for, select the referenced projects to import code coverage data from. Also specify a comment about the import session, if desired.
  8. Optional: To select protocol type and coverage data location, click Next, deselect the Look up in the project option, and then select one of Remote target, Workspace or File System to browse for the coverage data location.
  9. Click Finish.
    Now, the Code Coverage tab shows the session name and imported gcc code coverage data for the selected project.

After you run the configuration in Step 3, you can choose to do the following:
  1. Optional: Observe the target's directory using the Target File System Navigator tab in the Tasks view (bottom of the Workbench window) in the location where the file project_name.gcda resides.

    By default, you won't have the Target File System Navigator tab in your Tasks view. To add this tab to your view:

    1. Select Window > Show View > Other.
    2. Expand QNX Targets.
    3. Select Target File System Navigator.
    4. Click OK.

    For a QNX project, if a project is built using gcc version 4.6, the files are created under the variant_name/o_g directory.

  2. Optional: For the target, right-click on the file project_name.gcda and select Copy to > Workspace.
  3. Optional: In the Select Target Folder window, specify a folder location to copy the file, and click OK.

    The project_name.gcda will be visible under the C/C++ tab for the corresponding project.

Last modified: 2014-06-24



Got questions about leaving a comment? Get answers from our Disqus FAQ.

comments powered by Disqus