November 2016
36
t
ools
& s
oftware
reason, this method is also very widespread
with many test tools. A disadvantage, how-
ever, is that the actual test is not carried out on
the real embedded system; the actual timing
behaviour is therefore not taken into consid-
eration.
Instrumentation of the code to be tested and
execution on real hardware. Additional test
code, which collects information necessary
for calculating the code coverage and stores
this in the target memory, is inserted in the
software for this. However, as a consequence
the test code not only influences the runtime
behaviour and code size but possibly even
the memory layout. This is certainly a crucial
point in safety-critical applications or systems
with hard real-time requirements.
Execution of original code on real hardware
with simultaneous program trace. The target
system must provide suitable trace hardware
for this including a trace interface to the cov-
erage tool. This method requires absolutely
no instrumentation and the timing behaviour
of the application to be tested also remains
unchanged.
However, suitable high-performance debug-
ger solutions, such as for example the Uni-
versal Debug Engine (UDE) from PLS, are
crucial for successful test execution and deter-
mining the code coverage on real target hard-
ware by means of instrumentation or trace.
This is because in the instrumented case, the
target memory, which contains the data for
the coverage calculation, must at least be read
out by the debugger. For the trace-based solu-
tion, the debugger even uses the entire trace
infrastructure on the chip.
For tests on real hardware, tight coupling
between the debugger, which of course
enables the actual access to the target system,
and the test tool, which takes over test case
management and the test documentation, is
of essential importance. Only in rare cases do
debuggers also offer complete project man-
agement for the test. Test tools, on the other
hand, often lack suitable possibilities to com-
municate directly with the target system via
the debug interface. Modern high-end debug-
gers, like the UDE, therefore provide an auto-
mation interface for the tool coupling. That
interface allows test tools to utilize the debug-
ger functions for controlling the target system
and for reading out the target state as well as
manipulating it.
In the case of the UDE, this interface is based
on the Component Object Model (COM)
from Microsoft. For a long period of time,
COM has been established as the de facto
standard in the world of Windows. Even
Microsoft itself offers a large part of its newly
added Windows functions via COM interface.
The object model of the UDE encompasses
almost all functions of the debugger such as
flash programming, run control, reading and
writing of the target system memory contents,
trace data acquisition and analysis, and of
course also code coverage. Therefore, via the
said interface, with the UDE it is possible to
establish a tight coupling to test systems of
different vendors and thus build a complete
tool chain for testing on real hardware. Fur-
thermore, COM offers the major advantage
that it can be used with a very large number of
different languages. These include C, C++, C#
and other .NET languages as well as scripting
languages such as JavaScript, Python, Perl and
VB Script or Windows PowerShell. Therefore,
the UDE can also be automated by the users
very easily and controlled remotely via own
scripts without the aid of a test tool.
As is already known, instrumentation of
the executed code is not really advisable for
obtaining code coverage in safety-critical
applications or systems with high real-time
requirements. For most of these cases already
mentioned trace-based solutions are inevita-
bly used. Also for this the automation inter-
face of UDE offers appropriate functions. For
example, if UDE is controlled by a test system
the recording of trace for code coverage can
be enabled already during the configuration
of test tasks within the test tool itself. The
other settings for code coverage are subse-
quently made in the debugger itself, as follows.
Used coverage level: here it is possible to
choose between statement coverage and
branch coverage. For both levels, the cov-
erage can be calculated solely based on pro-
gram trace and from debug information of the
application.
Accumulation of sequentially performed
coverage measurements: if multiple runs are
necessary for testing all functions of an appli-
cation, for example because the trace memory
is not sufficiently large for the coverage, an
accumulated calculation of the code cover-
age for all runs makes sense. In this case, the
results of the individual runs are then com-
bined to form an overall result.
Settings for report generation: depending on
the user needs, separate reports for all func-
tions tested or an overall report can be gen-
erated. Furthermore, it is possible to define
basic things such as file name conventions or
locations for the respective reports.
All settings can be done by the user directly
via the user interface of the debugger. In addi-
tion, they are also accessible via functions of
the COM-based automation interface. The
code coverage can thus also be used by third
party tools or by own scripts. Modern high-
end debuggers, like the Universal Debug
Engine (UDE) from PLS, play an increasingly
key role in this close interaction of different
components and tools. Thanks to a wide vari-
ety of COM-based internal functions of UDE
for example, virtually all settings can option-
ally be made via third party tools or via own
scripts and there is nothing else standing in
the way of fully automated testing including
documented test quality.
n
Figure 2. By means of trace, the code coverage achieved can be determined in parallel with the
actual testing. The reports generated from this serve as proof of the test quality.
