21
October 2013
S
OFTWARE
D
EVELOPMENT
adds challenges. Most SOUP modules are pro-
vided by third-party vendors, who do not fol-
low any software process, software standards
or even document the code. And while it ad-
dresses platform challenges, SOUP is developed
under stringent time schedules where the em-
phasis is on productivity, not standards com-
pliance. When subjected to system tests that
check functionality, SOUP projects typically
show very poor code coverage, leaving many
code paths unexercised. In figure 1 the blue
curve represents functionally tested code.When
that code is deployed, the different data and
circumstances are likely to use many untested
paths for the first time, potentially creating
unexpected results. The red dotted curve in
figure 1 illustrates the part of the code used
when the application is run infield.
The European System and Software Initiative
“Performance of Errors through experience-
driven Test” (PET) investigated this phenome-
non and agreed that deployed code often pos-
sesses many errors. PET aimed to reduce the
number of bugs reported after release by 50%
and to reduce the hours of test effort per bug
found by 40%. Interestingly, PET exceeded
this achieving 75% fewer reported bugs and
46% improvement in test efficiency. PET find-
ings demonstrated that the number of bugs
that can be found by newer methods of testing,
such as static and dynamic analysis, is large,
even if that code has been through functional
system testing and subsequently been released.
The SOUP previously subjected to functional
test is then deployed for further tests. Even if
it worked well, some part of the code may not
be exercised, even when the product is infield.
When SOUP code needs ongoing development
for later revisions or new applications, previ-
ously unexercised code paths are likely to be
called into use by combinations of data never
previously encountered, potentially creating
unexpected results. The green dotted curve in
figure 1 illustrates the part of the code used
when enhancement are made to SOUP code.
To counteract this potential weakness, a detailed
structural coverage analysis needs to be done
to ensure that there is no unexercised code in
the legacy software. IEC 62304 mandates func-
tional coverage (i.e. ensuring the software func-
tions per system design requirements) and
structural coverage (i.e. all code sections are
exercised and shown to work properly) of the
legacy software along with a detailed analysis
of the risk that could be introduced by the ad-
dition of such software. IEC 62304 requires
that all SOUP items to be incorporated in the
medical device design are identified along with
the specification of their functional and per-
Figure 1. Traditional functional testing can leave many parts of the code unproven.
Figure 2. The requirements traceability
matrix (RTM) plays a central role in a devel-
opment lifecycle model even when SOUP
items are part of the system.
