Measuring code coverage

You can use code coverage to measure how much code runs during a test or benchmark. After you run code coverage, you can analyze the results to create additional test cases that increase your code coverage.

Code coverage uses a signal to tell the app to send information back to the Momentics IDE. Depending on your app design, there are several possible risks that can result from running code coverage from the Momentics IDE:

  • It can change or break the behavior of apps that it monitors.
  • It can cause code to run that a test suite does not test.
  • It can result in data not being collected.

  1. In the Launch configuration drop-down list, select the launch configuration for the project.
  2. In the Launch mode drop-down list, select Coverage.
  3. In the Launch target drop-down list, select a device or simulator. To resolve any device connection issues, click Momentics IDE edit icon, and follow the instructions on the Device Properties dialog box.
  4. Click Momentics IDE build button. The Console view displays the output.
  5. Click Momentics IDE run button.

What is code coverage?

The code coverage tool uses the GNU Code Coverage (Gcov) tool that the GCC compiler produces. The Momentics IDE presents metrics as line coverage, and shows which lines are fully covered, partially covered, and not covered at all. The Momentics IDE also presents percentages of coverage in terms of the actual code covered and not just lines.

The coverage metrics provide basic block coverage, or line coverage, which describes whether a block of code is run. A block of code doesn't have any branch points within it, so that the path of execution enters at the beginning and exits at the end.

The Momentics IDE tracks the number of times that the block of code has been run. It uses this information to determine the total coverage for a particular file or function. It also uses this information to analyze the blocks on each line and to determine the level of coverage for each line.

Although the gcc compiler produces metrics for branch coverage, the Momentics IDE doesn't provide this information.

How does code coverage work?

The code coverage tool works with the compiler (gcc), the QNX C library (libc), and optionally the remote target agent (qconn). When you enable code coverage for an app, the compiler instruments the code to count each branch execution to a basic block at runtime. During the build, the Momentics IDE produces data files to recreate the app's flow graph, and to provide the line locations of each block.

Because the Momentics IDE creates secondary data files at compilation time, be careful when building your apps in a multitargeted build environment, such as the BlackBerry 10 OS.

Do one of the following:
  • Ensure that the last compiled binary is the one you're collecting coverage data on.
  • Enable only one architecture and one variant (debug or release).

The compiler's optimizations could produce unexpected results, so you should perform coverage tests on an unoptimized, debug-enabled build.

To instrument a static library with code coverage, you must instrument your binary with code coverage, or link with the code coverage library using the following option in the linker command:

-lgcov

This option links the ${QNX_HOST}/usr/lib/gcc/<target>/<version>/libcov.a library.

When a coverage session starts, you can view the data immediately. The QNX Code Coverage perspective contains a Code Coverage Sessions view that lists previous as well as currently active sessions. You can explore each session and browse the corresponding source files that have received coverage data.

Code coverage might not work as expected. The code coverage data for C++ projects includes other functions that are also in the source file, such as static initializer and global constructor functions. In addition, the files included by include statements aren't included in the overall coverage total; only those functions that are in the original source code are included for code coverage.

Configure code coverage settings

  1. On the toolbar, in the Launch mode drop-down list, select Coverage.
  2. In the Launch configuration drop-down list, select the launch configuration for the project.
  3. In the Launch target drop-down list, select the device or simulator.
  4. Click Momentics IDE build button.
  5. In the Launch configuration drop-down list, click Momentics IDE edit icon next to your launch configuration.
  6. In the Edit Configuration dialog box, click the Coverage tab.
  7. Configure the settings that you need.
Code Coverage data scan interval (sec)
Sets how often the Code Coverage tool polls for data. A low setting can cause continuous network traffic. The default setting of 5 seconds should be sufficient.
Comments for this Code Coverage session
Contains your notes about the session, for your own personal use. The comments appear at the top of the generated reports.
Collect data for
Determines what data to include in your code coverage session. By default, all code coverage data built with code coverage in a project is included in the current code coverage session. To include referenced projects or to collect data only from certain sources, clear the All Sources in this application compiled with code coverage option. To select the projects or files that you want to collect code coverage data for, click Select.
Select
Opens the Projects to include Code Coverage data from dialog box, where you select the projects to include your coverage data (projects and files). Select any project from this list that you want to gather code coverage data for. Projects must be built with code coverage enabled to capture data.
Advanced
Defines a signal to enable the dynamic collection of code coverage data. The Momentics IDE sends a signal to suspend the app thread so that it can perform data collection.

Enable code coverage for Makefile projects

To enable code coverage for a Makefile project, if you're using qcc or gcc, compile and link with the following options:

-fprofile-arcs -ftest-coverage

For example, in the Makefile, set the following gcc options for code coverage:

CFLAGS += -g -fprofile-arcs -ftest-coverage 
LDFLAGS+= -g -fprofile-arcs -ftest-coverage 

Your Makefile might look similar to the Makefile below, which is part of the Code Coverage example project included with the Momentics IDE:

DEBUG = -g 
CC = qcc 
LD = qcc 
CFLAGS += -Vgcc_ntox86 $(DEBUG) -c -Wc,-Wall -I. -O0 -Wc,-ftest-coverage 
-Wc,-fprofile-arcs 
LDFLAGS+= -Vgcc_ntox86 $(DEBUG) -ftest-coverage -fprofile-arcs  

# CC refers to the program for compiling C programs (the default is 
# qcc. Use CXX as the program for compiling C++ programs. 

# CFLAGS are additional flags to give to the C compiler. Use CFLAGS  
# for the C++ compiler. 

# -c compiles or assemble the source files, but doesn't link, and the 
# -Wc captures the warning messages. The linking stage isn't done. 
# The ultimate output is in the form of an object file for each
# source file. 

# -Wall turns on all optional warnings that are desirable for normal  
# code. -I. adds the current directory to the list of directories to  
# search for header files. Directories named by -I are searched before  
# the standard system include directories. 

# -O0 is an optimization flag that indicates 'Do not optimize.' 

# LDFLAGS are additional flags to give to compilers when they invoke 
# the ld linker. 

# -ftest-coverage -Wc means that Code Coverage is enabled for your  
# project, and the data is used for test coverage analysis. 

# -fprofile-arcs adds code so that program flow arcs are instrumented.  
# During execution, the program records how many times each branch and 
# call is executed and how many times it is taken or returns, and it 
# saves this data to a file with the extension .gcda for each source  
# file.
# 
# For Code Coverage, you'll need the -fprofile-arcs -ftest-coverage  
# options in both the compile and link lines.

dir := $(shell pwd) 

BINS = rbt_client rbt_server 

# This next line is the rule for <cmd>all</cmd> that  
# incrementally builds your system by performing a <cmd>make</cmd>  
# of all the top-level targets the Makefile knows about. It does this by 
# expressing a dependency on the results of that system, which in turn  
# have their own rules and dependencies. 

all: $(BINS) 

# The following line shows a simple rule for cleaning your build  
# environment. It cleans your build environment by deleting all files  
# that are normally created by running make. 

# It has a Target named <cmd>clean</cmd> to left of the colon,  
# no dependencies (to the right of the colon), and two commands that are 
# indented by tabs on the lines that follow. 

clean: 
rm -f *.o *.img *.gcno *.gcda $(BINS) 
menu
# The following lines are Dependency Rules, which are rules without any  
# command. If any file to the right of the colon changes, the target to  
# the left of the colon is no longer considered current (out of date). 
# Dependency Rules are often used to capture header file dependencies. 

rbt_server: rbt_server.o 

# Alternatively, to manually capture dependencies, several automated 
# dependency generators exist. 

rbt_server.o : rbt_server.c rbt_server.h 
$(CC) $(CFLAGS) $(dir)/$< 

rbt_client: rbt_client.o 

rbt_client.o: rbt_client.c rbt_server.h 
$(CC) $(CFLAGS) $(dir)/$< 

Import gcc code coverage data

With older compilers, if you built your app with code coverage enabled and chose to disable code coverage in the launch configuration, the process wrote the coverage information to a data file (for example, .gcda) at runtime, rather than read it from the data space. You could choose to import this data into the Code Coverage tool later. The new gcc compiler doesn't stream the data coverage; the Momentics IDE waits for the generation of the data file before it copies it back to your computer.

In addition, the Momentics IDE generates notes files (.gcno) when it compiles projects that have enabled code coverage.

There are multiple ways to import a file.

You don't need to move the file that you want to import into the Workspace location.

By default, the .gcda files for gcc are located in a folder structure created under /tmp.

When you copy a <project_name>.gcda file into your workspace, you must copy it to the top level of the folder structure. For example, the <variant_name>/o_g folder.

To import gcc code coverage data from a project:

  1. Create and build a project with code coverage selected.
  2. Create a launch configuration where code coverage is disabled.
  3. Run this configuration.
  4. On the File menu, clickImport > QNX > GCC Coverage Data and click Next.
  5. Specify the name of the session, project, and platform used to generate the code coverage data. Click Next. Now, you can browse on the remote target to the folder that contains the data file.
  6. To browse the remote file system for the Coverage protocol type (that is, .gcda), browse to the location where the data files are located (such as on the remote target, within the workspace, or on the filesystem).
  7. To include data for referenced projects, select the referenced projects to import code coverage data from. You can specify a comment about the import session.
  8. To select the protocol type and coverage data location, click Next, clear the Look up in the project option, and then select either Remote target, Workspace or File System to browse for the coverage data location.
  9. Click Finish. The Code Coverage tab shows the session name and imported gcc code coverage data for the selected project.
After you run the configuration, you can do the following:
  1. To add the Target File System Navigator tab to your Tasks view:
    1. On the Window menu, click Show View > Other.
    2. Expand QNX Targets.
    3. Select Target File System Navigator.
    4. Click OK.
  2. In the Target File System Navigator tab, find the target's folder, which is where the file <project_name>.gcda is located.
  3. Right-click the file <project_name>.gcda and select Copy to > Workspace.
  4. In the Select Target Folder window, specify a folder location, and click OK. The <project_name>.gcda file is visible under the C/C++ tab for the corresponding project.

Examine data line-by-line

The Momentics IDE can show the line-by-line coverage information for your source code by showing green check marks for fully covered code, a red cross for each line not covered, and a yellow ball icon for each partially-covered block of collapsed code. In the image below, the left margin of the editor shows a summary of the coverage and the right margin shows color-coded bars.

Screen showing the Code Coverage Editor view.

Open a file in the Code Coverage perspective

  • In the Code Coverage Sessions view, expand a session and double-click a file or function. Code coverage markers are added to the left pane of the opened file.

Show coverage information for a specific session

  • In the Code Coverage Sessions view, select a session. The Momentics IDE shows all of the markers.

Show coverage information when you open a file

  1. On the Window menu, click Preferences.
  2. Expand QNX > Code Coverage.
  3. Select the desired markers in the Coverage markup when file is opened field.
  4. Click OK. The next time you open a file, the markers appear automatically.

Remove coverage markers

  • In the title bar of the Code Coverage Sessions view, click The Remove All Coverage Markers icon.

Combine code coverage sessions

  1. In the Code Coverage Sessions view, select the sessions you want to combine.
  2. Right-click your selections and select Combine/Copy Sessions. The Momentics IDE prompts you for a session name and creates a combined session.

Last modified: 2014-11-17



Got questions about leaving a comment? Get answers from our Disqus FAQ.

comments powered by Disqus