Part IV Introduction to Software Testing

H el` ene WAESELYNCK

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verication, and Validation 163

Testing

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

164

Software: rst failure cause of computing systems

Size: from some (tens) of thousands of code lines to some millions of code
lines

Development effort:
0,1-0,5 person.year / KLOC (large software) 5-10 person.year / KLOC (critical software)

Share of the effort devoted to fault removal:

45-75%

Fault density:
10-200 faults / KLOC created during development - static analysis - proof - model-checking

- testing
0,01-10 faults / KLOC residual in operation
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

165

Test inputs

Program Under test

Test outputs

Oracle

Verdict
(pass/fail)

! Issues of controllability and observability

" Examples :
??? ???

! Oracle problem = how to decide about the correctness of the observed outputs?
" Manual computation of expected results, executable specication, back-toback testing of different versions, output plausibility checks, ...

! To reveal a fault, the following chain of conditions must be met:

" At least one test input activates the fault and creates an error " The error is propagated until an observable output is affected " The erroneous output violates an oracle check

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

166

If x>0: output (x+1)/3 + 1 Else: output 0 Example_function (int x) BEGIN int y, z ; IF x ! 0 THEN z = 0 ELSE y = x-1 ; /* y = x+1 */ z = (y/3) +1 ; ENDIF Print(z) ; END ! Activation of the fault if x > 0 ! Error propagation: incorrect output if (x mod 3) ! 1 ! Violation of an oracle check:
" Expected result correctly determined by the operator # fault revealed " Back-to-back testing of 2 versions, V2 does not contain this fault # fault revealed " Plausibility check 0 < 3z-x < 6 # fault revealed if (x mod 3) = 0
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 167

Explicit consideration of testing in the software life cycle

Example: V life cycle
BUSINESS REQUIREMENTS ACCEPTANCE TESTS

REQUIREMENTS

SYSTEM TESTS

HIGH-LEVEL DESIGN

INTEGRATION TESTS

DETAILED DESIGN

UNIT TESTS

CODING
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

168

 Whatever the adopted life-cycle model, it denes a testing process, in interaction with the software development process
Planning test phases associated with development phases Progressive integration strategy (e.g., top-down design, bottom-up testing) Tester/developer independence rules (according to software phase and criticality) Rules guiding choice of test methods to employ (according to software phase and criticality) Procedures for coordinating processes

Documents are produced at each testing step

Employed test methods Test sets + oracle Test platform: host machine, target machine emulator, target machine, external environment simulator Other tools: compiler, test tools, drivers and stubs specically developed Test reports

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

169

 Unit testing = testing of an isolated component

Test inputs Test driver Outputs Oracle Unit under test
calls

Stub

Stub

Integration testing = gradual aggregation of components

E.g.: bottom-up strategy
Test {C}, test {D} Test {B, C, D} Test {A, B, C, D} A B C D

! no stub to develop " High-level components are tested late, while it may be important to test them early # because they are major components of the system (e.g., GUI)
# to reveal high-level faults (e.g., inadequate functional decomposition)

! Other strategies : top-down, sandwich

ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 170

Test design methods: problem

Exhaustive testing is impractical!
Very large, or even innite, input domain
Testing a simple program processing three 32 bits integers: 296 possible inputs Testing a compiler: innite input domain

Relevance of the very notion of exhaustiveness?

Elusive faults (Heisenbugs): activation conditions depend on complex combinations of internal state x external requests

Partial verication using a (small) sample of the input domain

Adequate selection?

No model of all possible software faults

ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

171

Classication of test methods

% $
STRUCTURAL MODEL deterministic structural statistical structural FUNCTIONAL MODEL deterministic functional statistical functional

SELECTIVE CHOICE

RANDOM CHOICE

criterion

The model synthesizes information about the program to be tested. The criterion indicates how to exploit the information for selecting test data: it denes a set of model elements to be exercised during testing.

input generation process

Deterministic: selective choice of inputs to satisfy (cover) the criterion. Probabilistic: random generation according to a probabilistic distribution over the input domain; the distribution and number of test data are determined by the criterion.
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 172

Purposes of testing
Fault nding test: test aimed at uncovering fault

Conformance test:

functional test aimed at checking whether a software complies with its specication (integrated testing level, required traceability between test data and specication) test aimed at checking the ability of a software to work acceptably in the presence of faults or of stressing environmental conditions

Robustness test:

Regression test :

after software modication, test aimed at checking that the modication has no undesirable consequence

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

173

& & & & &

Introduction Structural testing Functional testing Mutation analysis Probabilistic generation of test inputs

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

174

Control ow graph
Oriented graph giving a compact view of the program control structure:
built from the program source code A node = a maximal block of consecutive statements i1, in
i1 is the unique access point to the block the statements are always executed in the order i1, in the block is exited after the execution of in

edges between nodes = conditional or unconditional branching

Structural criteria

Based on: - control ow graph - control ow graph + data ow annotations

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

175

POWER function:
computes Z = XY, where X and Y are two integers (X! 0 )

BEGIN read (X,Y) ; W = abs (Y) ; Z = 1; WHILE (W <> 0) DO Z = Z * X ; W = W-1 ; END IF (Y<O) THEN Z = 1/Z ; END print (Z) ; END

read (X,Y) W=abs(Y) Z=1

2
W! 0 Z=Z.X W =W-1 W= 0

3 4
Y"0 print(Z) Y<0

5 Z=1/Z 6

Program execution = activation of a path in the graph Structural Criterion: guides the selection of paths
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 176

! All Paths
" Non-executable path: 1!2!4!5!6 " Innite (or very large) number of paths: number of loop iterations 2 ! (3 !2)* determined by |Y|

read (X,Y) W=abs(Y) Z=1

2
W! 0 W= 0

! All Branches

" Two executions are sufcient Y < 0 : 1 ! 2 ! (3 ! 2)+ ! 4 ! 5 ! 6 Y " 0 : 1 ! 2 ! (3 ! 2)* ! 4 ! 6

Z=Z.X W =W-1

3 4
Y"0 print(Z) Y<0

5 Z=1/Z 6

! All Statements
" Covered by a single execution Y < 0 : 1 ! 2 ! (3 ! 2)+ ! 4 ! 5 ! 6

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

177

Other criteria
Criteria for covering loops
Intermediate between all paths and all branches E.g., pick paths that induce 0, 1 and n>1 loop iterations (you may, or not, consider all subpaths for the loop body at each iteration)

Criteria for covering branch predicates

Renement of all branches (and also possibly all paths) when the branch predicate is a compound expression with Boolean operators. Example: A " (B # C) Test every possible combination of truth values for conditions A, B et C ' 23 cases Test a subset of combinations such that each condition independently affects the outcome of the decision to False and True /...
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 178

 Test a subset of combinations such that each condition independently affects the outcome of the decision to F and T

MC/DC criterion = Modied Condition / Decision Coverage

Principle
A A A"B
AB 0 1 2 3 FF FT TF TT

2 test cases F T 3 test cases FT TF TT

Dec. F F F T Cond. Affect. Dec. A (3) B (3) A (1), B (2) 0 1 2 3

Very much used in the avionics domain: required by DO-178B for Level A software

A#B
AB FF FT TF TT

3 test cases FF FT TF
Dec. F T T T Cond. Affect. Dec. A (2), B (1) B (0) A (0)

A1 " A2 " An $ n+1 test cases single Ai is F (n cases) + case TTT

ReSIST Courseware

A1 # A2 # An $ n+1 test cases single Ai is T (n cases) + case FFF

Testing, Verification and Validation 179

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Ex : A " (B # C)
ABC 1 2 3 4 5 6 7 8 TTT TTF TFT TFF FTT FTF FFT FFF Res. T T T F F F F F Oper. Affect. Res. A (5) A (6), B (4) A (7), C (4) B (2), C (3) A (1) A (2) A (3)

Take a pair for each operand A: (1,5) or (2,6) or (3,7) B: (2,4) C: (3,4) Hence two minimal sets for covering the criterion {2, 3, 4, 6} ou {2, 3, 4, 7} i.e., 4 cases to test (instead of 8) Generally: [n+1, 2n] instead of 2n

Remark: MC/DC can be applied to instructions involving Boolean expressions, in addition to branching conditions If (A and (B or C))
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

res := A and (B or C);

Testing, Verification and Validation 180

MC /DC and coupled conditions

$If a condition appears more than once in a decision, each occurrence is a distinct condition$ Example: (A " B) # (A " C)
2 occurrences of A: 2 pairs for A Occurrence 1 -> pair (6,2) A B C res 6 TTF T 2 FTF F Occurrence 2 -> pair (5,1) A B C res 5 TFT T 1 FFT F

Some cases may be impossible to cover when conditions are not independent
Example 1: (A " B) # A 1st occurrence of A cannot affect the decision outcome F Example 2 : (A " x # y) # (x > y-10 ) x # y cannot affect the decision outcome F
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 181

MC / DC in practice
A priori approach via complete truth table often infeasible (e.g., number of operands > 4)

A posteriori evaluation of an existing (functional) test set

Coverage analysis tools: Rational Test RealTime, IPL Cantata, LDRA Testbed

According to results, complement, or not, the test set

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

182

Data ow
Annotating the control graph:
Node i
Def (i) = set of variables dened at node i, which can be used externally to i C-use (i) = set of variables used in a calculus at node i, not dened in i

Edge (i,j)
P-use (i, j) = set of variables appearing in the predicate conditioning transfer of control from i to j

Associated criteria

For each variable v dened in node i, selection of subpaths between this denition and one or several subsequent uses of v

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

183

1
node i 1 def(i) X, Y, W, Z c-use (i) arc (i,j) (1,2) p-use (i,j)

read (X,Y) W=abs(Y) Z=1

2
(2,3) (2,4) W W

2 3 4 5 6 Z Z Z W, Z X, W, Z

W! 0 Z=Z.X W =W-1

W= 0

(3,2) (4,5) (4,6) (5,6) Y Y

3 4
Y"0 print(Z) Y<0

5 Z=1/Z 6

Example : denition of Z at node 1 --> covering all uses? use at node 3: test with |y| > 0 use at node 6: test with y = 0 use at node 5 impossible, as path 1 ! 2 ! 4 ! 5 ! 6 is infeasible
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

184

criteria:
All denitions
Selection of a subpath for each variable denition, for some use (equally in a calculus or predicate)

All C-uses / some P-Uses (resp. all P-uses / some C-Uses)

Use in calculation (resp. in predicate) is favored

All Uses
Selection of a subpath for each use

All DU paths
Selection of all possible subpaths without iteration between denition and each use

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

185

All paths

Ordering of structural criteria (subsumption)

All DU paths

All uses

All C-Uses / some P-Uses

All P-Uses / some C-Uses All P-Uses MC/DC

All C-Uses

All definitions Branches Instructions

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

186

Structural Criteria conclusion

Criteria dened in a homogeneous framework
Model = control ow graph (+ possibly data ow)

Mainly applicable to the rst test phases (unit testing, integration of small sub-systems)
Complexity of analysis rapidly grows Note: for integration testing, a more abstract graph may be used (call graph)

Tool support (except for data-ow)

Automated extraction of control-ow graph, coverage analysis

Structural coverage is required by standards

Typically: 100% branch coverage Can be more stringent: MC/DC for DO-178B
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

187

& & & & &

Introduction Structural testing Functional testing Mutation analysis Probabilistic generation of test inputs

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

188

Equivalence classes + boundary values

Principle
Partition the input domain into equivalence classes to be covered
Classes determined from the functional requirements (set of values for which functional behavior is the same), and/or from the data types (e.g., for int, positive, negative)

Consider both valid and invalid classes (robustness testing) Identify boundary values for dedicated tests
E.g., -1, 0, 1, +/- MAXINT

Example
The price entered by the operator must be a strictly positive integer. If price # 99%, then From 100% and above, the processing Valid classes: 1 # price # 99 100 # price + possibly:
ReSIST Courseware

Invalid class: price # 0 price = real number price = string

Boundary values: price = -1, 0, 1, 98, 99, 100, 101, +/- MAXINT nothing entered ($)
Testing, Verification and Validation 189

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

 Informal approach, as based on natural language specication

Differing partitions and limit values can be obtained from same specication analysis

Analysis granularity?
Separate analysis of input variables ( considering all case combinations?
For price input variable For in_stock input variable 1 # price # 99 in_stock = TRUE 100 # price in_stock = FALSE

Class dened by composite predicate ( breaking up in disjoint cases?

discount = 10% or discount = 20%

Classes with non-empty intersection ( breaking up in disjoint cases?

1 # price # 99 prix * (1-discount) # 50

"

Rapid explosion of the number of created classes!

Examples of selective choice strategies aimed at covering the classes

Selecting one test input for each class, without considering possible intersections Minimizing the number of test inputs while favoring the patterns covering several valid classes; contrarily, testing separately each invalid class Pairwise strategies for covering combinations of values from classes

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

190

Decision Table
rule 1 rule 2
LIST OF CONDITIONS C1 C2 C3 LIST OF ACTIONS A1 A2 TRUE X TRUE YES YES TRUE TRUE

rule 3

rule 4

FALSE FALSE FALSE FALSE TRUE NO YES

FALSE FALSE YES NO NO YES

For a combinatorial function

Ci : input conditions Ai : disjoint actions, order in which they are executed does not matter completeness and consistency of the rules (exactly one rule is eligible) ( document the impossible cases

& Rule coverage = 1 test case for each rule

ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

191

Finite State Machine

For a sequential function
S : nite set of states S0 : initial state % : nite input alphabet & : nite output alphabet ' : transition relation, ' : S x % ! S ( : output relation, ( : S x % ! &

)/r 1 2 */r 1 )/s Graphical representation Preliminaries: 2 ) 2/r 1/s * ? 2/r

Tabular representation

- Check completeness. E.g., add self-loop &*/-' on State 1

- Deterministic? Minimal? Strongly connected?

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

192

)/r ) 1 */)/s Graphical representation 2 */r 1 2 2/r 1/s * 1/2/r

Tabular representation

) coverage of states, transitions, switches (pairs of transitions)

Testing wrt original machine or completed one
Impact on the input domain + oracle

controllability: reach State i to test transition i->j?

Availability of a &reset' function in the test driver

) Other strategies based on fault models

/
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 193

spec

)/r

impl1

)/s

impl2

)/r

faults leading to output errors

faults leading to transfer errors

) Testing all transitions + checking the arrival state

observability: observing state j?
Existence of a &status' function?
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 194

Generally, direct observation of state i is impossible Selective choice strategy often used:

Ti,j = preamblei eij SCj

With: Tij = test sequence for transition i!j preamblei = input sequence starting at initial state, and arriving at i eij = input activating transition i!j SCj = caracterising sequence for state j

)/r 1 */)/s 2

Input ) enabling to distinguish between ! and ": ! --> r output " --> s output */r Testing transition " )/s ! via sequence:

Reset . ) . ) . )
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 195

Generally, direct observation of state j is impossible Selective choice strategy often used:

Ti,j = preamblei eij SCj

With: Tij = test sequence for transition i!j preamblei = input sequence strating at initial state, and arriving at i eij = imput activating transition i!j SCj = caracterising sequence for state j

Do not always exist { Exists for each minimal and complete MEF

- Distinguishing sequence DS (same + j) - Sequence UIO (unique for each j) - Set W (set of sequences enabling to distinguish states 2 by 2)

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

196

Example: Cho's W Method

)/r [12] 1 */)/s 2 */r % [1] [2] W = {)}
(Here, DS)

General case [ 1 2 n] % W = {)} [1 3] [2, 4 n] % W = {), **} [1] [3] [2, ...] [4] [5,]

1
W

)
W 2

* 1

)
W

*
W

Reset ) Reset * ) Reset ) ) Reset ) ) ) Reset ) * )

Test tree
ReSIST Courseware

Test sequences
Testing, Verification and Validation 197

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

W method )/r 1 */)/s 2 */r

Transition tour

Reset ) Reset ) * ) * * Reset * ) Reset ) ) * Reset ) ) ) * Reset ) * )

Rapid explosion of the test size

W method: For a completely specied, minimal, deterministic FSM of n states If implementation is also deterministic, with n states at most, the driver implements correctly the Reset, Then the W method guarantees to reveal faults producing output errors (according to the output alphabet) and transfer errors
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Strong assumptions ...

Testing, Verification and Validation

198

Transition systems
LTS (Labelled transition system) = low level formalism for describing process systems / communicating automata
Example: specications in LOTOS, SDL, Estelle, can be translated in LTS

Basic LTS: input and output events are not distinguished $ IOLTS (Input/Output LTS) for testing

?a !z ,1 !x ,2 !y Input event: a Output event: x, y, z Internal actions: ,1, ,2 non-determinism, quiescence

Test approaches referring to relations between LTSs

Chosen relation = conformance relation between spec and impl

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

199

Example: ioco relation

Implementation not in conformity if it can exhibit outputs or quiescences not present in specication Spec
?a !z ,1 !x ,2 !y !z ,3 !x

Impl1 not in conformity

?a ,4 !z

Impl2 not in conformity

?a !z ,3 !x ,5
ReSIST Courseware

Impl3 in conformity
?b ?a !z !x !c

,4 !y

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

200

Test cases
Test case = IOLTS with specic trap states (Pass, Fail, Inconclusive)
--> incorporates inputs, expected outputs, and verdict

Example:
?a !x ! other !y ! other FAIL

enables non conformity of impl1 to be uncovered

Observation of !z after ?a enables non conformity of impl2 to be uncovered Observation of a quiescence after ?a !x

! z PASS

FAIL PASS
(Actually, the mirror image ?- ! is taken, because inputs/outputs of the system are outputs/inputs of the tester)

Test case automatically synthesized from a test purpose Test purpose =IOLTS X Spec =IOLTS Test case =IOLTS

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

201

Test purpose
handwritten automatically generated from a specication coverage criterion
Example: transition coverage of the SDL model

Example: purpose = produce !z

other ?a

!z

?a !z ,1 !x ,2 ! other !y

!x !y

! other FAIL

ACCEPT (= covered purpose)

! z INCONCL.

FAIL PASS
(Actually, the mirror image of this)

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

202

Functional Testing
No homogeneous framework, methods depend on the used model
Equivalence classes Decision tables Finite state machines Statecharts SDL Logic-based specications (algebraic, model-oriented)

Model abstraction level & criterion stringency depend on the complexity (integration level) of the tested software Formal models used for specication and design $ makes it possible to (partially) automate testing = input generation, oracle

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

203

& & & & &

Introduction Structural testing Functional testing Mutation analysis Probabilistic generation of test inputs

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

204

Assessing the fault-revealing power of test methods?

Feedback from past projects
Assessment wrt real faults But reduced sample of faults

Mutation analysis = controlled experiments

Assessment wrt seeded faults Large sample of faults

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

205

Mutation analysis (1)

& introduction of a simple fault (= mutation) C ! C +1 true ! false + ! - & execution of each mutant with test set T killed (fault revealed by the test set) alive (fault not revealed) & measurement = mutation score Number of mutants killed by T Total number of mutants & mutations are syntactically not representative of development faults but produce similar errors
(it is as difcult to reveal mutations as to reveal real faults)

< ! #

low ! mid

& Tools Mothra (Georgia Inst. of Techn.): Fortran Sesame (LAAS): C, Pascal, Assembly, Lustre JavaMut (LAAS): Java and many others
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 206

Mutation analysis (2)

! Comparing test methods (academic research) ! Identifying imperfections of a structural or functional test set --> complement test set with additional cases ! Evaluating software propensity to mask errors --> locally more stringent test But " Explosion of the number of mutants " Identication of equivalent mutants partly by hand equivalent mutant = mutant which cannot be distinguished from the original programme by any test input

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

207

Effectiveness of structural criteria: some experimental results

Mutation score
1 0.9 0.8 0.7 0.6 0.5 paths All uses, P-uses, branches All defs

Mutation score FCT4: 587 mutations

1 0.96 0.92 0.88 0.84 0.80 All uses P-uses branches All defs

FCT3: 1416 mutations

! All paths: score < 100% ! For a given criterion, strong impact of the chosen input values ! Criterion stringency + no guarantee
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation 208

Imperfection of test criteria

& Deterministic approach: criterion renement
Examples : instructions ! branches ! paths ! paths + decomposition of branch conditions states ! transitions ! W method ! W+1 method (number of implemented states # number of specied states +1) ! ! W+n method

! exhaustiveness according to fault assumptions " explosion of test size in order to account for weaker assumptions

& Probabilistic approach

Random, less focused, selection: exhaustiveness according to fault assumptions not searched for, reasoning in terms of failure rate according to a sampling prole. /...

Remark: increase of the test size in both cases # Necessity to automate input selection and oracle procedure
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

209

& & & & &

Introduction Structural testing Functional testing Mutation analysis Probabilistic generation of test inputs

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

210

Probabilistic approaches
Random testing
Uniform distribution over the input domain ! Easy to implement " blind selection, usually inefcient usually not recommended!

Operational testing
Input distribution is representative of operational usage ! Removal of faults that will have the greatest impact on reliability + software reliability assessment " inadequate for revealing, or assessing the consequences of, faults that induce small failure rates (e.g., < 10-4/h)

Statistical testing = criterion + random selection

Imperfect but relevant information

ReSIST Courseware

compensates imperfection of criterion

Testing, Verification and Validation

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

211

Operational testing
Population of users $ test according to several operational proles Denition of operational prole(s): functional modeling
Identifying operating modes, functions which can be activated in a mode, input classes Quantifying probabilities associated to the model according to operating data for similar systems, or to projected estimates $ introducing measurement capabilities in the system ($log$ les)

Some gures (Source: AT&T)

Operational prole(s) denition = 1 person.month for a project involving 10 developers, 100 KLOC, 18 months Denity project: duration of system test %2, maintenance costs %10

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

212

Criterion-based statistical testing

(i) Search for an input distribution criterion ! {elements to activate}
(structural or fonctional) maximizing P, prob. of least probable element ! balance of element weight according to selected prole

(ii) Calculation of the number of inputs to generate

N " ln (1-QN) / ln (1-P) QN = test quality wrt criterion Rq : QN = 0,999 ! 7 activations on average QN = 0,9999 ! 9 activations on average

Structural statistical testing Probabilitic analysis of control graph and data flow Functional statistical testing Probabilitic analysis of behavior models
Finite state machine, decision table, Statechart

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

213

Example
&
Analytical techniques B0 0 ! Y ! 99 path k1 p1 = Prob[0 # Y # 99] p2 = Prob[99 < Y # 9999] B4 B1 0 ! Y ! 9999 99 < Y ! 9999 B2 path k2

C = {paths} + SC = {k1, k2}

p1 = p2 = 0.5

Structural distribution / C QN = 0,999 ! N = 10 inputs to generate

&

Empirical techniques Successive renements of an initial distribution

ReSIST Courseware

v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

214

results for 4 unit functions and 2816 mutations :

TEST TYPE Structural deterministic Uniform random Structural statistical LIVE MUTANTS from 312 to 405 from 278 to 687 6 MUTATION SCORE [85,6% 88,9%] [75,6% 90,1%] 99,8%

Typical cases for limit value testing Compensates imperfection of structural criteria Adequate test proles

Contribution of probabilistic approach? Contribution of criterion to efciency of random inputs?

ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Testing, Verification and Validation

215

Functional statistical testing: mastering complexity?

! adoption of weak criteria (e.g., states,...) and possibly denition of several proles ! high test quality (e.g., qN = 0.9999)

Example : component from a nuclear control system

Functional modeling: hierarchy of Statecharts $ Statistical testing according to two complementary proles N = 85 + 356 = 441 Efciency wrt actual faults?
$Student$ version: 12 faults (A, , L) $industrial$ version: 1 fault Z (hardware
compensation)

+ programming: + understanding: + initialisation:

A, G, J B, , F, I H, K, L, Z

Efciency wrt injected faults?

$Student$ version: mutation score = 99.8 - 100% $industrial$ version: mutation score = 93 - 96.1%
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck

Necessity of initialisation process specic test

Testing, Verification and Validation 216

General conclusion
Numerous test methods
General approach: take a (structural or functional) model and dene model elements to be covered

Choice
Depends on available models and their complexity Complementarity of global coverage and testing of specic cases (boundary values, transient modes, ) Deterministic or probabilistic selection

(Semi-)automation of testing is highly recommended

B. Beizer : About 5% of the software development budget should be allocated to test tool building and/or buying$

Beyond this introductive lecture...

Not covered here: specicities wrt testing OO, concurrent, distributed, real-time software systems
ReSIST Courseware v. Henke, Bernardeschi, Masci, Pfeifer, Waeselynck Testing, Verification and Validation

217