IEC Certification Kit

Simulink Test ISO 26262

Tool Qualification Package
R2015b

How to Contact MathWorks

Latest news:

www.mathworks.com

Sales and services:

www.mathworks.com/sales_and_services

User community:

www.mathworks.com/matlabcentral

Technical support:

www.mathworks.com/support/contact_us

Phone:

508-647-7000 (Phone)

The MathWorks, Inc.

3 Apple Hill Drive
Natick, MA 01760-2098
For contact information about worldwide offices, see the MathWorks Web site.

IEC Certification Kit: Simulink Test ISO 26262 Tool Qualification Package

COPYRIGHT 2015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under
the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written
consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the
federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees
that this software or documentation qualifies as commercial computer software or commercial computer software documentation
as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,
reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or
other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.
If this License fails to meet the governments needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of
additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more information.

Revision History
September 2015

New for IEC Certification Kit Version 3.6 (Applies to Release 2015b)

Contents
1 Introduction ...................................................................................................................................... 1-1
1.1 Application Identification........................................................................................................ 1-2
1.2 Tool Overview and Identification ........................................................................................... 1-3
1.3 Tool Qualification Artifacts Summary .................................................................................... 1-4
2 Software Tool Criteria Evaluation Report........................................................................................ 2-1
2.1 Tool Environment ................................................................................................................... 2-2
2.2 Tool Configuration .................................................................................................................. 2-3
2.3 Reference Workflow ............................................................................................................... 2-4
2.4 Tool Use Cases ........................................................................................................................ 2-5
[SLTEST_UC1] Development and execution of tests for Simulink models ................................ 2-5
[SLTEST_UC2] Development and execution of tests for back-to-back testing between model and
code .............................................................................................................................................. 2-5
[SLTEST_UC3] Assessment of test results ................................................................................. 2-5
[SLTEST_UC4] Generation of test reports .................................................................................. 2-5
[SLTEST_UC5] Identification of traceability between requirements and tests cases.................. 2-6
2.5 Generic Tool Classification ..................................................................................................... 2-7
2.5.1
Potential Malfunctions and Erroneous Output ............................................................... 2-7
[SLTEST_E1] Incorrect behavior of test harness ........................................................................ 2-7
[SLTEST_E2] Incorrect run of test procedure ............................................................................. 2-7
[SLTEST_E3] Erroneous assessment of test results false negative .......................................... 2-7
[SLTEST_E4] Erroneous assessment of test results false positive ........................................... 2-7
[SLTEST_E5] Generation of erroneous test report ...................................................................... 2-7
[SLTEST_E6] Usage of incorrect input data ............................................................................... 2-7
[SLTEST_E7] Incorrect Tool Usage............................................................................................ 2-8
[SLTEST_E8] Incorrect or Modified or Incompatible with Environment Tool Installation........ 2-8
2.5.2
Error prevention and Detection Measures ..................................................................... 2-8
[SLTEST_M1] Requirements-based testing ................................................................................ 2-8
[SLTEST_M2] Tool installation integrity checks ........................................................................ 2-8
[SLTEST_M3] Configuration management ................................................................................. 2-8
[SLTEST_M4] Input data integrity checks .................................................................................. 2-8
[SLTEST_M5] Competency of project team ............................................................................... 2-8
[SLTEST_M7] Manual comparison of test results to expected results ........................................ 2-9
[SLTEST_M8] Manual review of test report content .................................................................. 2-9
2.6 Tool Classification Summary ................................................................................................ 2-10
3 Software Tool Qualification Report ................................................................................................. 3-1
3.1 Requirement for Tool Qualification ........................................................................................ 3-2
3.2 Tool Qualification Documentation .......................................................................................... 3-3
4 Confirmation Review of Tool Classification and Qualification ....................................................... 4-1
4.1 Requirement for Confirmation Review ................................................................................... 4-2
4.2 Validity of Generic Tool Classification .................................................................................. 4-3

4.3
4.4

Validity of Generic Tool Qualification ................................................................................... 4-4

Conformance with Reference Workflow ................................................................................. 4-5

vi

vii

1 Introduction
This document constitutes the ISO 26262 Tool Qualification Package for the Simulink Test
product. This document is intended for use in the ISO 26262 tool classification and qualification
process for software tools. It contains templates for the ISO 26262 tool qualification work
products (see ISO 26262-8, Clause 11).
The applicant shall review this template for applicability to the application under consideration,
and tailor and complete the information.
See also:

IEC Certification Kit: Users Guide, R2015b

ISO 26262-8, Clause 11

ISO 26262-8, Clause 11 provides provisions for software tools that are used to tailor activities or
tasks required by ISO 26262. The standard outlines a two-step approach to establish the
required confidence in the tools:
Tool classification determines the required level of confidence in the software tool.
Depending on the result of the tool classification, you might need to carry out a formal
tool qualification.
The following work products need to be created when applying this approach to a software tool
(see ISO 26262-8, 11.5):

A software tool criteria evaluation report documenting the tool classification.

A software tool qualification report documenting the tool qualification, if required.

Note The applicant needs to review this template for applicability to the project under
consideration and insert missing information.

1.1 Application Identification

Applicant:

<Insert information>

Application under consideration:

<List application under consideration>

1-2

1.2 Tool Overview and Identification

Simulink Test is a tool for authoring, managing, and executing systematic, simulation-based
tests of the Simulink models. You can create nonintrusive test harnesses to test models and
subsystems. Simulink Test includes a test sequence block that lets you construct complex test
sequences and assessments, and a test manager that lets you manage and execute tests. It enables
functional, baseline, equivalence, and back-to-back testing, including software-in-the-loop (SIL)
and processor-in-the-loop (PIL). You can generate reports, archive and review test results, rerun
failed tests, and debug the component or system under test.
Software Tool

Version (Release)

Tool Vendor

Simulink

Version 1.1 (R2015b)

The MathWorks, Inc.

3 Apple Hill Drive
Natick, MA, 01760-2098
USA

Test

IEC Certification Kit

Version 3.6 (R2015b)

1-3

1.3 Tool Qualification Artifacts Summary

The following table lists:

Prerequisites (see ISO 26262-8, 11.3.1)

Supporting information (see ISO 26262-8, 11.3.2)
Tool qualification work products (see ISO 26262-8, 11.5)

for the Simulink Test product. The table also maps these tool qualification artifacts to sections in
this document and artifacts found elsewhere.
Tool Certification Artifact

Corresponding Documents / Artifacts

Safety plan

<Insert document title, version, and filename / link>

Applicable prerequisites of the <Insert software lifecycle phase(s)>

lifecycle phases where software
tool is used
<Insert prerequisite(s)>
Predetermined maximum ASIL

<Insert ASIL>

Software tool documentation

Simulink Test Getting Started Guide

R2015b
sltest_gs.pdf
Simulink Test Users Guide
R2015b
sltest_ug.pdf
Simulink Test Reference
R2015b
sltest_ref.pdf
Simulink Test Release Notes
R2015b
rn.pdf

Environment and constraints of

the software tool

MathWorks bug report system at

www.mathworks.com/support/bugreports/
<Insert information>

1-4

Tool Certification Artifact

Corresponding Documents / Artifacts

Software tool criteria evaluation Customized and completed section Software Tool Criteria Evaluation
report
Report in the Simulink Test ISO 26262 Tool Qualification Package
(this document)
certkitiec_sltest_tqp.docx
Simulink Test Reference Workflow
R2015b
certkitiec_sltest_workflow.pdf
Certificate Z10 15 06 67052 016
June 2015
certkitiec_sltest_certificate.pdf
Report to the Certificate Z10 15 06 67052 016
June 2015
certkitiec_sltest_certreport.pdf
Software tool qualification
report

Customized and completed Software Tool Qualification Report in the

Simulink Test ISO 26262 Tool Qualification Package (this document)
certkitiec_sltest_tqp.docx
Customized and completed Simulink Test Conformance Demonstration
Template
certkitiec_sltest_cdt.docx
Certificate Z10 15 06 67052 016
June 2015
certkitiec_sltest_certificate.pdf
Report to the Certificate Z10 15 06 67052 016
June 2015
certkitiec_sltest_certreport.pdf

Confirmation review of
qualification of a software tool

Customized and completed Confirmation Review of Tool Classification

and Qualification in the Simulink Test ISO 26262 Tool Qualification
Package (this document)
certkitiec_sltest_tqp.docx

1-5

1-6

2 Software Tool Criteria Evaluation

Report

2.1 Tool Environment

It is assumed that Simulink Test will be used in the following environment (see ISO 26262-8,
11.4.4.1d):
<Insert operating system and other pertinent environment information>

2-2

2.2 Tool Configuration

It is assumed that Simulink Test will be used in the following tool configuration (see ISO
26262-8, 11.4.4.1b):
Configuration Parameter

Setting

Test Result Report Pane

Include MATLAB version

Yes

Include in Report controls

<Insert relevant configuration parameter names>
File Format

<Insert application-specific setting>

<Insert application-specific setting>

2-3

2.3 Reference Workflow

It is assumed that Simulink Test will be used as described in the reference workflow
documented in Simulink Test Reference Workflow. To access the reference workflow document,
on the MATLAB command line, type certkitiec . The reference workflow document is in
Simulink Test.
Simulink test features integrated into the generic Model-Based Design workflow are shown in
Figure 1.

Figure 1: Simulink Test Workflow Overview

2-4

2.4 Tool Use Cases

It is assumed that Simulink Test will be used as described by the following use cases (see ISO
26262-8, 11.4.4.1c). Additional information about the assumed usage of Simulink Test can be
found in the Simulink Test Reference Workflow document and Simulink Test Users Guide
document.

[SLTEST_UC1] Development and execution of tests for

Simulink models
Simulink Test is used to create and execute tests for Simulink models. Testing of Simulink
models can be leveraged to implement the following verification and testing methods:

Simulation of dynamic parts of the software architectural design including mechanisms

for error detection and handling at the architecture level (ISO 26262-6 Table 6 method
1c).

Verification of software unit design (ISO 26262-6 Table 9 method 1c).

Implementation model testing (ISO 26262-6 Table 10 methods 1a 1c).

[SLTEST_UC2] Development and execution of tests for

back-to-back testing between model and code
Simulink Test is used to create and execute tests for back-to-back testing between model and
code using equivalence test capability (ISO 26262-6 Table 10 method 1e and Table 13 method
1e).

[SLTEST_UC3] Assessment of test results

Simulink Test is used to evaluate test results comparing them with expected results. Applicable
for all testing activities identified in the use cases SLTEST_UC1 and SLTEST_UC2.

[SLTEST_UC4] Generation of test reports

Simulink Test is used to generate test reports. Applicable for all testing activities identified in
the use cases SLTEST_UC1 and SLTEST_UC2.

2-5

[SLTEST_UC5] Identification of traceability between

requirements and tests cases
Simulink Test is used to establish bidirectional links between textual requirements and test cases
(ISO 26262-6 Tables 11 and 14 methods 1a and 1b).

2-6

2.5 Generic Tool Classification

The tool classification for Simulink Test was performed in a generic manner, independently
from the development of a particular safety-related item or element.
For the generic tool classification, the reference use cases listed in the section Tool Use Cases
have been taken into account.

2.5.1 Potential Malfunctions and Erroneous Output

[SLTEST_E1] Incorrect behavior of test harness
Test harness developed using Simulink Test produces erroneous input test stimulus or corrupt
simulation outputs of model under test.

[SLTEST_E2] Incorrect run of test procedure

Test procedure developed using Simulink Test run erroneously, e.g. invoke simulation of wrong
model or skips test cases.

[SLTEST_E3] Erroneous assessment of test results

false negative
Comparison of model simulation results to expected results incorrectly marks test case as FAIL.

[SLTEST_E4] Erroneous assessment of test results

false positive
Comparison of model simulation results to expected results incorrectly marks test case as PASS.

[SLTEST_E5] Generation of erroneous test report

Simulink Test produces erroneous test report which does not correspond to the actual test
results.

[SLTEST_E6] Usage of incorrect input data

Incorrect input data is used, resulting in tool malfunction and erroneous output.

2-7

[SLTEST_E7] Incorrect Tool Usage

User does not follow established procedures when using the tool, resulting in tool malfunction
and erroneous output.

[SLTEST_E8] Incorrect or Modified or Incompatible with

Environment Tool Installation
User does not follow established procedures when installing the tool, installs the tool in an
incorrect operational environment, or modifies a valid installation. This might result in tool
malfunction and erroneous output.

2.5.2 Error prevention and Detection Measures

To mitigate potential malfunctions and corresponding erroneous outputs of the Simulink Test
product, the following measures are provided. Additional considerations are described in
Simulink Test Reference Workflow.

[SLTEST_M1] Requirements-based testing

The test cases and expected results are derived from requirements independent of the model
under test and the test environment. The independence provides a high degree of confidence that
errors will be detected using the actual results from the model under test in the test environment.

[SLTEST_M2] Tool installation integrity checks

Integrity of tool installation can be insured by re-running the validation test suite provided with
Simulink Test in the IEC Certification Kit.

[SLTEST_M3] Configuration management

Configuration of the life cycle data shall be managed by applicant in accordance with Clause 7
of ISO 26262.

[SLTEST_M4] Input data integrity checks

Simulink Test verifies the integrity of input files using checksum.

[SLTEST_M5] Competency of project team

Training of users can be performed to ensure correct usage of tool.

2-8

[SLTEST_M6] Analysis of Available Bug Report

Information
Assess and analyze bug report information for Simulink Test and comply with the
recommendations and workarounds, if applicable.

[SLTEST_M7] Manual comparison of test results to

expected results
Test results are manually compared to expected results to determine whether test passed or
failed.

[SLTEST_M8] Manual review of test report content

Test report content is manually reviewed to verify that it corresponds to the actual test results.

2-9

2.6 Tool Classification Summary

Potential
malfunction or
erroneous
output

Use cases

Prevention /
detection
measures

TD

[SLTEST_E1]
[SLTEST _UC1] TI2 Incorrect behavior of test
Incorrect behaviour [SLTEST _UC2]
harness could prevent
of test harness
errors in an object under
test from being detected.

[SLTEST _M1]
Requirements-based
testing

[SLTEST_E2]
[SLTEST _UC1] TI2 Incorrect run of test
Incorrect run of test [SLTEST _UC2]
procedure could prevent
procedure
errors in an object under
test from being detected.
[SLTEST_E3]
[SLTEST _UC3] TI1 Nuisance only, failed tests
Erroneous
have to be manually
assessment of test
reviewed and explained by
results passed test
user
indicated as failed
[SLTEST_E4]
[SLTEST _UC3] TI2 Incorrect assessment of
Erroneous
test results could prevent
assessment of test
errors in an object under
results failed test
test from being detected.
indicated as passed

[SLTEST _M1]
Requirements-based
testing

TD1 The test cases and expected results are TCL1

derived from requirements
independent of the model under test
and the test environment. The
independence provides a high degree
of confidence that errors will be
detected using the actual results from
the model under test in the test
environment
TD1 Requirements-based testing will
TCL1
detect incorrect run of test procedure,
see TD justification for [SLTEST_E1]

TCL1

None

TD3 -

TCL3

[SLTEST_E5]
Simulink Test
produces erroneous
test report which
doesnt correspond
to the actual test
data
[SLTEST_E6]
Usage of incorrect
input data

None

Justification for TI

[SLTEST _UC4] TI2 Invalid test report could

[SLTEST _UC5]
prevent errors in an object
under test from being
detected.

Justification for TD

TCL

[SLTEST _M7]
TD1 Manual comparison of test results to TCL1
Manual comparison of
expected results can verify that results
test results to expected
have been correctly assessed by the
results
tool.
TD3 -

TCL3

[SLTEST _M8]
TD1 Manual review of test report content
Manual review of test
can verify that report has been
report content
correctly generated by the tool.

[SLTEST _UC1] TI2 Incorrect input data may

[SLTEST _UC2]
lead to incorrect test run
and could prevent errors in
an object under test from
being detected.

[SLTEST_E7]
All
Incorrect tool usage

[SLTEST _E8]
Incorrect or
Modified or

TI

All

TCL1

[SLTEST _M3]
TD1 Revision control and configuration
TCL1
Configuration
management facilitate integrity of the
management
input data. Using checksums allows
[SLTEST _M4]
the unique identification the input
Input data integrity
data.
checks
TI2 Incorrect tool usage could [SLTEST _M5]
TD1 Training of tool users can prevent
TCL1
prevent errors in an object Competency of project
these issues.
under test from being
team
detected.
TI2 Incorrect tool installation [SLTEST _M2]
TD1 Verification of the installed tool
TCL1
may lead to incorrect test Tool installation
version will detect invalid tool
run could prevent errors in integrity checks
installation.

2-10

Potential
malfunction or
erroneous
output
Incompatible with
Environment Tool
Installation

Use cases

TI

Justification for TI

Prevention /
detection
measures

TD

Justification for TD

TCL

an object under test from

being detected.

Based on the preceding analysis, the maximum tool impact of the Simulink Test use cases taken
into account is TI2.
Subsequent use of error detection measures [SLTEST _M7] and [SLTEST _M8] provides high
degree of confidence that tool malfunctions SLTEST_E4 and SLTEST_E5 will be detected.
Therefore the tool confidence level for the capabilities implementing the corresponding use
cases SLTEST_UC3, SLTEST_UC4 and SLTEST _UC5 is TCL1. If no measures are applied
the tool confidence level is TCL3.
For the capabilities implementing use cases SLTEST_UC1 and SLTEST_UC2 the tool
confidence level is TCL1 provided the prevention/detection measures identified in the table
above are taken.
TV SD reviewed the generic tool classification and confirmed the results in Report to the
Certificate Z10 15 06 67052 016.

2-11

3 Software Tool Qualification Report

3.1 Requirement for Tool Qualification

TCL1 can be claimed for the Simulink Test capabilities implementing use cases SLTEST_UC1
and SLTEST_UC2 given the workflow and error detection measure specified in the document
are applied. Therefore additional tool qualification methods are not necessary according to ISO
26262-8, clause 11.4.6.1.
Given the required tool confidence level TCL3 for the Simulink Test capabilities
SLTEST_UC3, SLTEST_UC4 and SLTEST_UC5 without manual comparison and review (see
Generic Tool Classification), these capabilities need to be qualified up to TCL3. Permissible
tool qualifications for TCL3 are listed in ISO 26262-8 Table 4.

3-2

3.2 Tool Qualification Documentation

MathWorks carried out an application-independent prequalification of the Simulink Test
product. The Simulink Test capabilities SLTEST_UC3 (Assessment of test results) and
SLTEST_UC4 (Generation of test reports) and SLTEST_UC5 (Identification of traceability
between requirements and tests cases) were prequalified for all ASILs according to ISO 262628, up to and including TCL3. These capabilities of Simulink Test were prequalified using a
combination of the following methods:

Evaluation of the tool development process (ISO 26262-8, Table 4, Method 1b).
Validation of the software tool (ISO 26262-8, Table 4, Method 1c).

According to ISO 26262-8, table 4, these two methods are permissible for all ASILs. Method 1b
is highly recommended for ASILs A, and B. Methods 1c and 1d are highly recommended for
ASIL D.
Tool qualification for the corresponding capabilities of the Simulink Test product can be
claimed for TCL1 and TCL3 by referencing the certification report and corresponding
certificate.
TV SD carried out an independent tool qualification assessment. MathWorks submitted the
results of the methods applied to pre-qualify Simulink Test to TV SD.
TV SD reviewed the generic tool qualification artifacts for Simulink Test and confirmed the
results in Report to the Certificate Z10 15 06 67052 016.

3-3

3-4

4 Confirmation Review of Tool

Classification and Qualification

4.1 Requirement for Confirmation Review

The tool classification (see Software Tool Criteria Evaluation Report) was carried out
independently from the development of the application under consideration. Therefore, the
resulting, predetermined tool confidence level shall be confirmed by the applicant prior to
Simulink Test being used for the development of a particular safety-related item or element for
the application under consideration (see ISO 26262-8, 11.4.2, 11.4.10).
If TCL3 is confirmed, the prequalification shall be confirmed prior to Simulink Test being used
for the development of a particular safety-related item or element for the application under
consideration (see ISO 26262-8, 11.4.2, 11.4.10).. The confirmation is required, because the
prequalification was carried out independently from the development of the application under
consideration.
If TCL1 is confirmed, tool qualification and hence confirmation of the tool qualification are not
required.
The generic tool classification is based on the assumption that Simulink Test is being used as
described in the reference workflow documented in Simulink Test Reference Workflow.
Therefore, conformance with the entire reference workflow (for TCL1) or the suitable subset
(for TCL3) in the application under consideration shall be confirmed by the applicant.

4-2

4.2 Validity of Generic Tool Classification

Applicable Tool Confidence Level: < Select TCL1 or TCL3>
<Insert results of confirmation review or reference to confirmation review documentation>

4-3

4.3 Validity of Generic Tool Qualification

Applicable Tool Confidence Level: < Select TCL1 or TCL3>
< Insert results of confirmation review or reference to confirmation review documentation in
case of TCL3>

4-4

4.4 Conformance with Reference Workflow

Applicable Tool Confidence Level: < Select TCL1 or TCL3>
< Insert reference to customized and completed Conformance Demonstration Template >

4-5