Software Verification and Validation (VIMMD052)

Model based testing (part 1)

Istvan Majzik
[email protected]

Budapest University of Technology and Economics

Dept. of Artificial Intelligence and Systems Engineering

Budapest University of Technology and Economics

Dept. of Artificial Intelligence and Systems Engineering 1
 Typical development steps and V&V tasks
Requirement
analysis

System
specification

Architecture
design

Component
design
• Source code analysis
• Proof of program correctness by theorem proving
Component
implementation • Software model checking with abstraction
• Component testing: classic techniques Model-based
System • Component testing: code-based testing techniques
integration for test case
• Integration testing generation
System • System testing
delivery

Operation,
maintenance

2
 Overview of part 1 and part 2
 Introduction
o The role of models in testing
o Use cases for model based testing
 Test case generation for test coverage criteria
o Using graph-based (direct) algorithms
o Using model checkers
o Using bounded model checkers
 Test case generation on the basis of mutations
o Model mutations
 Conformance and refinement relations for testing
o May and must preorder
o IOCO
+ Tools for model based test case generation
3
 Introduction

Role of models in testing

Use cases for model based testing

4
 Common practice: UML models in manual testing
 Component testing
o Class and object diagrams: Identifying components,
interfaces
o State machine and activity diagrams: Reference for
structure based testing
 Integration testing
o Sequence and collaboration diagrams: Identifying
scenarios
 System testing
o Component diagram: Identifying physical components
o Deployment diagram: Designing test configuration
 Validation (acceptance) testing
o Use case diagrams: Covering „happy path”, „mainstream
use”, extreme uses
5
Model based test case generation: Typical approach

Formal verification
Automated
(e.g., model checking)
code generation

Design model
Manual coding Implementation
(specification)

Test case generation Test cases

Test cases on the basis of the model (specification)

Verifying/validating the implementation w.r.t. the model (specification)

6
 Use cases for model based testing
 In case of manual coding: Conformance checking
test generation
Model Abstract test cases

manual
coding mapping
automated
testing
Implementation Concrete test cases

 In case of automated code generation: Validation

test generation
Model Abstract test cases

code
generation mapping
manual
evaluation
Implementation Concrete test cases
validation

7
 Abstract and concrete test cases

State, path,
requirement
coverage

Abstract
test cases:
events, actions

Concrete test
cases: inputs,
function calls

Source: M. Utting, A. Pretschner, B. Legeard. „A taxonomy of model-based testing approaches”, STVR 2012; 22:297–312

8
 The role of models in testing

Using the design models

as specifications:
Testing the conformance
of the model and the
implementation

Using separate
test models:
Specifying what to
test, how to test

A. Pretschner, J. Philipps. „Methodological Issues in Model-Based Testing”, Model-Based Testing of Reactive Systems, 2005.

9
 Basic tasks for model based testing (MBT)
 Based on the model and the test criteria:
o Test case generation (for coverage or behavior conformance)
o Test oracle generation (synthesis)
o Test coverage analysis (for the model)
o Conformance verdict (between model and implementation)
Test criteria,
test case spec.

Test
Model Implementation
generation

Test cases
Testing
Test oracle
Coverage Conformance

10
 Example: Open source MBT tool GraphWalker

Source: GraphWalker

 Input: Finite state machine modell (graph) + simple guards

 Output: Test cases for state and transition coverage
+ Generating JUnit test stubs (adapter)
 Traversing the graph: random walk, graph based search, shortest path
11
 Example: Industrial MBT tool Conformiq

 Input: State machine models + Java action code

 Output: Tests for state, transition, requirement coverage
 Integration with other tools for testing
12
 Overview of algorithms for model based test generation
 Graph-based algorithms
o Model represented as a graph + traversal/search in this graph
 Application of model checkers
o Diagnostic traces are used as test sequences for specified coverage
o Symbolic or bounded model checkers
 Mutation based test generation algorithms
o Test goal: Detect model mutations  detect code bugs
 Planner based methods
o The planner constructs an operation sequence for a test goal
 Evolutionary algorithms (e.g., genetic algorithms)
o Modifying an initial test suite generated by random walk
o Optimization: increase coverage, reduce test length, …
 Symbolic execution
o Using program graph (control flow graph) models

14
Graph-based algorithms for test
generation

15
 Typical applications of graph-based algorithms
 Graph model: Represents state based, event driven behavior
o States + transitions triggered by input events
o Actions are given as outputs
 Background formalisms:
o Finite state automata (FSM; Mealy, Moore, …)
o Higher level formalisms mapped to automata (UML statecharts, SCADE Safe
Statechart, Simulink Stateflow, …)
 Typical applications
o Embedded controllers processing signals
o Communication protocols, user interfaces, web based applications
 Using graph based algorithms
o Test case generation tasks are mapped to existing graph based problems,
thus the related graph algorithms are used to generate test cases
o Generating optimal test suite: Typically NP-complete

16
 Graph-based algorithm for transition coverage
 Mapping the problem
o Testing problem: Coverage of transitions
• All transitions shall be covered by a test sequence
• The test sequence shall go back to the initial state
o Graph-based problem: ”Street sweeper” problem
• In a directed graph, find the (shortest) path that covers all transitions and goes
back to the initial state
• (Similar problem in undirected graphs: ”Chinese postman” problem)
 Basic idea for the algorithm: Euler-graph  Euler-circuit generation
o Computing the polarity of vertices: nr. of incoming minus outgoing edges
o Forming Euler-graph: Duplicating edges that lead from a vertex with positive
polarity to vertex with negative polarity, until all vertices have zero polarity
o Finding an Euler-circuit in the resulting graph (linear algorithm)
• Euler-circuit: All edges are covered; it can always be constructed in such graph
o The traversal of the Euler-circuit defines the test sequence

17
 Example: Transition coverage

Original graph with Graph with duplicated edges and zero polarities
polarities of vertices (this way resulting in an Euler-graph)

Generated sequence for traversing all edges (Euler-circuit):

abcbfegdeg
(here some original edges are covered twice)

18
Graph-based algorithm for covering transition pairs
 Mapping the problem
o Testing problem: Coverage of transition sequences
• All possible sequences of n=2, 3,… subsequent transitions
shall be covered by a test sequence
• The test sequence shall go back to the initial state
• Simplest case: n=2, covering all transition pairs
o Graph-based problem: “Safecracker” sequence
• Find the (shortest) edge traversal that includes all possible sequences of
subsequent n edges (simplest case: n=2, all possible pairs of edges)
 Basic idea of the algorithm for n=2 (de Bruijn algorithm):
o Constructing a dual graph
• Edges of the original graph are mapped to vertices of the dual graph
• If there is a pair of subsequent edges in the original graph then we draw
an edge in the dual graph between the vertices that represent these edges
o Forming an Euler-graph (by duplicating edges) from the dual graph
o Finding an Euler-circuit that defines the test sequence
19
 Example: Covering transition pairs

Original graph Dual graph: vertices represent edges,

edges represent edge pairs in the original graph.
An Euler-graph can be constructed from the dual graph.

Euler-circuit in the Euler-graph constructed from the dual graph:

Edges of the dual graph - and thus edge pairs of the original graph are covered
Resulting sequence in the original graph:
abcbfecbgdefeg

20
 Graph-based algorithm for concurrent testing
 Mapping the problem
o Testing problem: Covering all transitions by concurrent testers
• Goal is complete transition coverage
• There are several testers who share (preferably equally)
the testing task to finish it in the shortest time
• All testers start in the initial state
• Condition: The tested system shall be resetable to the initial state
o Graph-based problems: ”Street sweepers brigade” problem
 Solution with heuristics (not an optimal solution)
o Giving an upper limit k of the length of the test sequence for each tester
o Forming Euler-graph and generating edge sequences such that each
• contains the highest number of edges that were not covered yet,
• and consists of at most k edges
o Generating additional test sequences until uncovered edge exists
o Trying to lower the limit k until the number of testers can be increased

21
 Example for concurrent transition coverage

Original graph Graph with some

duplicated edges
(Euler-graph)

Original test sequence (Euler-circuit, for 1 tester):

abcbfegdeg
A potential set of concurrent test sequences (k=7, 2 testers):
o Tester 1: abcbfeg
o Tester 2: deg
A better set of concurrent test sequences (k=5, 2 testers):
o Tester 1: abcbf
o Tester 2: deg

22
Test generation by model checking

23
 Basic idea
 Goal: Covering predefined elements of the model
o Control flow based coverage criteria:
• States, transitions, transition pairs in the model
o Data flow based coverage criteria:
• Variable definition and usage coverage (for all variables)
 Basic approach:
o For each element to be covered, the related test case is a
sequence of steps to reach that element in the state space of
the model
o Idea: Let the model checker find this sequence of steps
o Proper temporal logic properties are needed as inputs of the
model checker
• If the model checker is able to generate a witness trace (diagnostic trace
that demonstrates reachability): Reachability property (EF) used as input
• If the model checker is able to generate a counterexample for
reachability: Negated reachability property (EF) used as input
24
Basic idea: Using a model checker for test generation
1. Test sequence to be generated: 3. The witness trace generated by
Coverage of the state LineWeak the model checker demonstrates
that the given state can be reached

keyOn keyRdy
LineOk

Ready
PowerOff Error

LineWeak keyAck
keyOff

2.Specifying property for the 4. The witness trace is a test sequence

model checker: Reaching the covering the state LineWeak
state LineWeak
EF LineWeak
If the model checker supports only the
generation of counterexamples:
 EF LineWeak has to be specified

25
 Framework for automated test generation

Engineering
Formal model
model

Model Witness traces as

checker test sequences

Test coverage
Set of TL formula
criteria

26
 A possible implementation of the framework

UML PROMELA
statechart model

SPIN model XML based

checker test sequences

Test coverage
LTL formula
criteria

27
 Representing test coverage criteria by TL formula
 Identifying states and transitions (with characteristic functions)
o s: being in state s
o t: executing transition t (reaching its target state from its source state)
 Identifying specific transitions in the model for variable v:
o def(v) Using the variable in condition for an
o c-use(v) implicit transition
Implicit transition: The state does not
o p-use(v) change if the condition of the implicit
o implicit-use(v) transition holds

 Identifying sets (represented by characteristic functions):

o d(v): all def(v) transitions
o u(v): all c-use(v) or p-use(v) transitions
o im-u(v): all implicit-use(v) transitions
o start: state for starting a new test (e.g., initial state)
28
 Formula for control flow based coverage criteria
 State coverage: Set of formula is defined
Witness traces cover all
{EF s |  s basic state} basic states

If a predefined start state shall also be reached

for starting the subsequent test:
{EF (s  EF start) |  s basic state}
(EF start is omitted from the next formula)

 Weak transition coverage: Strong coverage: Implicit

transitions are also tested
{EF t |  t transition} (these shall not leave the state)
 Strong transition coverage:
{EF t |  t transition}  {EF it | it implicit transition}

29
 Recap: Data flow based test coverage criteria
 All-defs: For all v, from all def v: def v

at least one
def-clear path:

to at least
one use v: use v use v use v

 All-uses: For all v, from all def v: def v

at least one
def-clear path:

to all use v:
use v use v use v

30
 Formula for data flow based test coverage criteria
One def-clear path traversed from all
 Weak all-defs coverage: def(v) to one use(v) transition

{EF (t  EX E(d(v) U u(v))) |  v variable, td(v)}

One def-clear path traversed from all
 Weak all-uses coverage: def(v) to all use(v) transitions

{EF (t  EX E(d(v) U t’)) | v variable, td(v), t’u(v)}

Also covering implicit uses (on implicit
 Strong all-defs coverage: transitions)

{EF (t  EX E(d(v) U (u(v)  im-u(v))))

|  v variable, td(v)}
 Strong all-uses coverage:
{EF (t  EX E(d(v) U t’))
|  v variable, td(v), t’ u(v)  im-u(v)}
31
 Features of model checker based test generation
 Limitations of model checkers
o Generate typically only a single witness trace
• In case of some test goals (e.g., all-du-paths criterion) all witness traces
would be needed, which is difficult to achieve
o Generate typically some witness trace; special settings are
needed to find a shortest trace
• Breadth first search (BFS) in the state space
• Depth first search, but with limited depth (limited DFS)
• Finding shorter test sequences in an iterative way
o Generating the shortest test sequences: NP-complete problem
 Abstract test sequences are generated
o Defining the sequence of steps in the model
o Mapping is needed to executable test sequences:
actions (e.g., function calls) in a concrete test environment
32
 Example: Results for generating test sequences
Options Time required Length of
(compile time for test all test
or run-time) generation sequences Iterative search:
-i 22m 32.46s 17 Very slow
-dBFS 11m 48.83s 17
Breadth-first search:
-i -m1000 4m 47.23s 17 Shortest test sequences
-I 2m 48.78s 25 but slow
default 2m 04.86s 385
-I -m1000 1m 46.64s 22
Default: Depth-first search
Very long test sequences
-m1000 1m 25.48s 97
are possible
-m200 –w24 46.7s 17

SPIN settings (dfs is the default):

• -i iterative, -I approx. iterative State machine model of the
• -dBFS breadth first search behavior of a mobile phone
• -m limit for depth first search (10 states, 11 transitions)
• -w hash table size

33
Extension of MBT to testing time-dependent behavior
• Timed automata models with clock variables
• Specific model checker: UPPAAL
Clock variables:
Modelling time dependency
(conditions, state invariants)

To reach this state waiting is needed

at the PowerOn control location

The generated witness trace includes

this in the form of “Delay: 5”
between the steps

36
Test generation by bounded model
checking

38
 Recap: Bounded model checking
 Using SAT solvers for checking reachability of “bad” states
o SAT solver: Given a Boolean formula (Boolean function), it generates a
variable assignment (substitution) that makes the formula true
 Mapping the state space to a Boolean function:
Subsequent
o Characteristic function for initial states: I(s) states are
o Characteristic function for specified “bad” states: p(s) identified by
o Characteristic function of the transition relation: CR(s, s’) different
variable sets si
o “Stepping forward” along the transitions: CR(si, si+1)
 The characterization of reaching a “bad” state (with conjunction):
o Starting from the initial state: I(s)
o „Stepping” along the transition relation: CR(si, si+1)
o Specifying that p(si) holds somewhere along the path

39
 Recap: Encoding a model
(0,0) Initial state:
s1
I(x,y) = (xy)
(0,1)
s2
Transition relation:
s3 CR(x,y, x’,y’) = (xy   x’ y’) 
(1,1)
 (x y  x’ y’) 
 ( x y   x’ y’) 
 ( x y   x’y’)
(0,0) (0,1) (1,1)
s0:
Paths with 3 steps from the initial state:
I(s0)  path(s0,s1,s2,s3) =
s1: = I(x0,y0) 
CR(x0,y0, x1,y1) 
s2:
CR(x1,y1, x2,y2) 
CR(x2,y2, x3,y3)
s3:

40
 SAT based test generation for given test goal
 Constructing the Boolean function for the SAT solver:
o Encoding paths with k steps from the initial state
o Specifying test criterion: In general, a TG formula (test goal)
• Reaching (covering) a state
• Executing (covering) a transition
• Traversing (covering) a part of the model, …

 C (s , s
k 1
i 1
I (s ) 
0
R
i
)  TG
i 0

Model paths Test

of length k goal

 If this formula can be satisfied, then the substitution gives a test

o This test is according to TG and limited to k steps
o If the formula cannot be satisfied then there is no test for TG in k steps

41
 Features of BMC based test generation
 Limitations for test generation
o Test of maximum k steps can be generated
o The length of potential paths can be increased iteratively
o If a state sequence is found then it is a valid test case
o If there is no test found then a longer test sequence may exist
 Mapping the test generation problem to SAT problem can
be made automatically
 Higher-level specification of test goals is possible
o For C programs: FQL language for test goals (FSHELL tool)
in /code.c/ cover @line(6),@call(f1) passing @file(code.c) \ @call(f2)
o Specifying pre- and postconditions: Is there a test when the
postcondition is not satisfied (although the precondition holds)?

42
Test generation based on mutations

43
 Using fault sets for test generation
 Experience in software testing:
o Coupling effect: Test cases that are efficient to find simple faults are also
efficient for finding more complex faults (more efficient than random testing)
o Competent programmer hypothesis: The programs are usually good, and the
majority of faults are often occurring typical faults
 Basic idea:
o Generating “mutant” models that contain typical simple faults,
and generate tests for detecting these faults
o These tests are expected to be efficient also for detecting complex faults
 Typical “mutations”:
o Changing arithmetic operations in conditions, …
o Changing the ordering of actions, omission of actions, messages, calls, …
 Mutant is detected: different behavior in original/mutant model
o Equivalence relation is needed to capture different behavior

44
 An equivalence relation for BMC based test generation
 Inputs and outputs are distinguished in the model
o in(s) – inputs (events) in state s
o out(s) – observable outputs (actions) in state s
o  action: lack of observable output
 Definition of the k-equivalence for the behaviour of two models:
For the first k steps, providing identical input sequences,
the outputs of the two models are the same
 Notation:
Original model M: Mutated model M’:
Predicate for initial state: I(s0) I’(s’0)
State transition relation: CR(si, si+1) CR’(s’i, s’i+1)

 C (s , s
k 1
i 1
Paths of length k from the initial state: I (s ) 
0
R
i
)
i 0

45
Mutation based test generation using k-equivalence
 Construction of a SAT formula for detecting a mutation:
o Providing the same input sequence for the two models
o Traversing paths of length k in the original model
o Traversing paths of length k in the mutant model
o At least one output shall be different in the two models

(in(s )  in(s ' ))  I (s )   C (s , s  C ' (s ' , s ' (out(s )  out(s ' ))

k k 1 k 1 k
i 1 i 1
i i 0
R
i
)  I '( s ' ) 
0
R
i
) i i
i 0 i 0 i 0 i 0

The same k steps in the k steps in the at least one

inputs original model mutant model output is different

 If this formula can be satisfied then the substitution defines a test

o The test detects the mutation: At least one output is different from
the normal one if the mutation is included in the model
o If there is no substitution: there is no test for detecting the mutant in k steps

46
 Summary for mutation-based testing
 Inserting mutations into the model
 Test generation for mutations
o Construction of test input sequences that result in different
behavior in the original (fault-free) and in the mutated model
o Mutation can be detected if such input sequence exists
 Test execution on the implementation
o Mutation detected: Output is different from the fault-free case
o “Negative test”: No difference means that there is no mutation
 Mutation-based tests are expected to be efficient for the
detection of more complex faults than the mutations

47