Software Testing

How do you test a system?

 Input test data to the

system.
 Observe the output:
 Check if the system behaved as

expected.
How do you test a system?
 How do you test a system?

• If the program does not behave as

expected:
•  note the conditions under which it
failed.
•  later debug and correct.
Errors and Failures
 A failure is a manifestation of an
error (aka defect or bug).

 presence of an error may

not lead to a failure.
Test cases and Test suite

 Test a software using a set of

carefully designed test cases:

the set of all test cases is

called the test suite
Test cases and Test suite

 A test case is a triplet [I,S,O]:

• I is the data to be input to the
system,
• S is the state of the system at
which the data is input,
• O is the expected output from the
system.
Verification versus Validation

 Verification is the
process of determining:
 whether the output of one
phase of development
conforms to its previous phase.


whether a fully developed system
Validation
conforms to is
itsthe
SRSprocess of
document.
determining
Verification versus Validation

 Aim of Verification:
 phase containment of errors
 Aim of validation:
 final product is error free.
Verification versus Validation

 Verification:
 are we doing right?
 Validation:
 have we done right?
 Design of Test Cases
 Exhaustive testing of any
system is impractical:
 input data domain is
extremely large.
 Design an optimal test suite:

 of reasonable size
 to uncover as many errors
as
possible.
Design of Test Cases
 If test cases are selected randomly:
 many test cases do not contribute
to the significance of the test
suite,
 do not detect errors not already
detected by other test cases in the
suite.
 The number of test cases in a
randomly selected test suite:
 not an indication of the effectiveness
of the testing.
Design of Test Cases

 Testing a system using a large

number
of randomly selected test cases:
 does not mean that many errors
in the system will be uncovered.

 Consider an example:

 finding the maximum of two

Design of Test Cases

 If (x>y) max = x;
else max = x;
 The code has a simple error:
 test suite {(x=3,y=2);
(x=2,y=3)} can detect the error,
 a larger test suite
{(x=3,y=2); (x=4,y=3);
(x=5,y=1)} does not detect
the error.
 Design of Test Cases

 Systematic approaches are required

to design an optimal test suite:
 each test case in the suite

should detect different errors.

Design of Test Cases
 Two main approaches to
design test cases:
 Black-box approach
 White-box (or glass-box)
approach
Black-box Testing
 Test cases are designed using
only functional specification
of the software:
 without any knowledge of the
internal structure of the
software.

 For this reason, black-box

testing is also known as
 Black-box Testing
Two main approaches to design


black box test cases:

 Equivalence class partitioning

 Boundary value analysis

 White-box Testing
 Designing white-box test cases:
 requires knowledge about the

internal structure of software.

 white-box testing is also

called structural testing.

White-Box Testing
 There exist several popular
white-box testing
methodologies:
 Statement coverage
 branch coverage
 path coverage
 condition coverage
 Statement Coverage
 Statement coverage
methodology:
 design test cases so that
 every statement in a program

is executed at least once.

 Statement Coverage
 The principal idea:
 unless a statement is executed,

 we have no way of knowing if

an error exists in that statement.
Example
 int f1(int x, int
y){
 if (x>y)
 1 whilethen
(x != y)
2
{
 x=x-y;
 5 }else y=y-x;
3
 6 return x; }
4
do suit
Test 100% is:statement
(3,3), (4,3) and (3,4) which
Branch Coverage
 Test cases are designed such
that:
 different branch conditions
 given true and false values in

turn.
 Branch Coverage
 Branch testing guarantees
statement coverage:
 a stronger testing compared to
the statement coverage-based
testing.
Example
 int f1(int x,int
y){
 if (x>y)
 1 whilethen
(x !=
2
y){
 x=x-y;
 5 }else y=y-x;
3
 6 return x;
}
4
Example
 Test cases for branch
coverage can be:
{(x=3,y=3),(x=3,y=2),
(x=4,y=3), (x=3,y=4)}
Condition Coverage
 Test cases are designed such
that:
 each component of a

composite conditional
expression
 given both true and false

values.
 Example
Consider the conditional


expression
 ((c1.and.c2).or.c3):

Each of c1, c2, and c3 are

exercised at least once,
 i.e. given true and false values.
 Branch testing
 Condition testing
 stronger testing than branch

testing:
 Branch testing

 stronger than statement

coverage testing.
 Condition coverage

 Consider a Boolean
expression having n
components:
 for condition coverage, we
n
require 2 test cases. (n is no. of
condtions)
 Path Coverage
 Design test cases such that:
 all linearly independent paths
in the program are executed
at least once.
 Linearly independent
paths
 Defined in terms of
 control flow graph (CFG) of a
program.
 Path coverage-based
testing
 To understand the path
coverage-based testing:
 we need to learn how to draw

control flow graph of a program.

Control flow graph (CFG)

A control flow graph (CFG)



describes:
 the sequence in which

different
instructions of a program get
executed.
 the way control flows through

the program.
 How to draw Control flow
graph?

 Number all the statements of a

program.
 Numbered statements:
 represent nodes of the control

flow graph.
How to draw Control flow
graph?

 1

Sequence: 2

 1 a=5;

2
b=a*b-1;
How to draw Control flow
graph?

 Selection:

 1 if(a>b)
c=3
2
then ;
3
4 c=5
else ;
c=c*c;
How to draw Control flow
graph?

 Iteration:

 1 while(a>b){
 b=b*a;
2 b=b-

4 1;}
c=b+d;
3
Example
 int f1(int x,int
1
y){
 if (x>y) 2
 1 whilethen
(x !=
2 3 4
y){
 x=x-y;
5
 5 }else y=y-x;
3 6
 6 return x;
}
4
 Path
 A path through a program:
 a node and edge sequence from

the starting node to a terminal

node of the control flow graph.
 There may be several
terminal nodes for the program.
 Independent path
 It is straight forward:
 to identify linearly independent

paths of simple programs.

 For complicated programs:

 it is not so easy to determine the

number of independent paths.

McCabe's cyclomatic
metric
Given a control flow graph G,


cyclomatic complexity V(G):

 V(G)= E-N+2
 N is the number of nodes in
G
 E is the number of edges in G
 Example Control Flow
Graph
1
2
3 4
5
6

 Cyclomatic complexity = 7-6+2 = 3.

 Cyclomatic complexity

 Another way of computing

cyclomatic complexity:
 inspect control flow graph
 determine number of bounded

areas in the graph

 V(G) = Total number of bounded

areas + 1
Bounded area
 Any region enclosed by nodes
and edge sequence.
Example Control Flow
Graph
1
2
3 4
5
6
 Example
 From a visual examination of the
CFG:
 the number of bounded areas is

2.
 cyclomatic complexity =

2+1=3.
 Cyclomatic complexity

 The cyclomatic complexity of a

program provides:
 a lower bound on the number of

test cases to be designed

 to guarantee coverage of all

linearly independent paths.

 Cyclomatic complexity

 Defines the number of

independent paths in a
program.
 Provides a lower bound:
 for the number of test cases for path
coverage.
 Cyclomatic complexity

 Knowing the number of test cases

required:
 does not make it any easier

to derive the test cases,

 only gives an indication of the

minimum number of test

cases required.
 Derivation of Test
Cases
 Draw control flow graph.
 Determine V(G).

 Determine the set of linearly

independent paths.
 Prepare test cases:

 to force execution along each

path.
Example
 int f1(int x,int
y){
 2 if (x>y)
 1 whilethen
(x !=
y){
3 x=x-y;
 4 else y=y-x;
 6 return x;

}
5}
Example Control Flow
Diagram
1
2
3 4
5
6
 Derivation of Test Cases

Number of independent paths: 3



 1,6 test case (x=1, y=1)

 1,2,3,5,1,6 test case(x=2, y=1)

 1,2,4,5,1,6 test case(x=1, y=2)

 Cyclomatic complexity

 Cyclomatic complexity of a
program:
 also indicates the psychological

complexity of a program.
 difficulty level of understanding

the program.
 Cyclomatic complexity

 From maintenance perspective,

 limit cyclomatic complexity

 of modules to some reasonable

value.
 Good software development

organizations:
 restrict cyclomatic complexity of
functions to a maximum of ten.
Types of failure

1. Transient: only for certain input values

while invoking a function
2. Permanent: for all inputs values while
invoking a function
3. Recoverable: when it occurs,
system can recover without having
to shutdown the system
4. Unrecoverable: need to restart the
system