November 2016
35
t
ools
& s
oftware
Code coverage in automated
test of embedded systems
By Jens Braunes,
PLS
By means of code coverage,
it is not complicated to determine
test quality in parallel to software and
system tests on real hardware.
In any case, a continuous and
seamless workflow for the software
and system testing, which logically
requires open interfaces for
efficient tool coupling, is essential
for correct and reliable results.
n
Today the complexity of SoCs presents
many new challenges for developers, espe-
cially with regard to software quality and
software security. This means for effective
software testing, it is not only automatic gen-
eration of test results and their evaluation that
are essential. To reliably determine the quality
of the tests, it is also necessary to measure and
document the achieved code coverage.
Software errors can never be completely
avoided during the development of complex
applications with megabytes of source code. It
is therefore all the more important, especially
with safety-critical applications, to detect and
eliminate these bugs as early as possible using
modern methods. Testing is one of the most
crucial parts of this process. With good rea-
son, a great deal of attention is given to testing
in relevant standards such as ISO 26262 for
the automotive industry, EN/IEC 62061 for
safety of machinery or DO-178 for the avia-
tion industry.
Whether a software solution is finally released
for use by the customer or needs reworking by
the development department, however, does
not only depend on the successful completion
of testing. In addition, the quality of the testing
performed has to be right. The decisive factor
here is test coverage. What is the proportion
of the application, module or individual func-
tions, which the respective test has stressed,
compared to the proportion, which could not
be stressed at all due to an unfavourable choice
of test cases? To explore this question more
thoroughly, which is extremely important for
software quality and security nowadays, in the
area of software development code coverage is
generally used to assess test quality. Since the
standards mentioned at the beginning not only
demand the documentation of test results but
also the degree of coverage achieved, it is of
course sensible to determine the code coverage
for the function or module under test in paral-
lel to the test execution.
To calculate code coverage, for example of a
function, information is required that can
only be gathered from executing the code. In
the simplest case, only those pieces of code
that are actually executed and those which are
not must be recorded. The latter is of course
implicitly given if the sources or at least the
binary are available. In turn, the statement
coverage - meaning how much code is tested
by the test cases in relation to the total code
- can be directly derived from this informa-
tion. Program code that is not executed (dead
code) can therefore be detected with this
comparatively simple method; however, the
current test quality requirements of the stan-
dards ISO 26262, EN/IEC 62061 or DO-178
are generally not fulfilled.
Branch coverage is significantly more mean-
ingful with regard to test quality. The execu-
tion of all possible program branches is used
for this. In the case of a simple IF statement,
both the TRUE and the FALSE branch have
to be executed. Because all pieces of code are
implicitly reached, the statement coverage
can be directly derived from the result of the
branch coverage.
Modified condition/decision coverage (MC/
DC) goes another step further in looking at
branches. In simple terms, for MC/DC, all
individual conditions contained in each com-
posed condition must take every possible out-
come in order to be regarded as tested. This
ensures that each individual condition affects
the overall result, independently of the other
individual conditions involved. However, this
method proves to be extremely cumbersome
in practical use, because an extremely high
number of paths have to be considered, espe-
cially within loops containing conditional
code sequences. In practice, there are now a
number of possibilities to obtain code cover-
age, as follows.
Simulation of executable code on a virtual
platform and determining the necessary
information for the calculation. Fairly simple
data about runtime behaviour can of course
be determined from simulations. For this
Figure 1. The UDE object model
provides access to almost all
debugger functions.
