Software Verification and
Validation
Verification and Validation
• Verification
– Have you built the product right?
– Does the product meet system specifications?
• Validation
– Have you built the right product?
– Does the product meet user expectations?
Defect
• A defect is a variance from a desired
product attribute.
• These attributes may involve system
specifications well as user expectations.
• Anything that may cause customer
dissatisfaction is a defect.
• Whether these defects are in system
specifications or in the software products,
it is essential to point them out and fix.
• “Death and taxes are inevitable.”
-- Haliburton
• “Death, taxes, and bugs are the only
certainties in the life of a programmer.”
-- Kernighan
Software Testing
• Software testing is the process of examining the
software product against its requirements.
• It is a process that involves verification of
product with respect to its written requirements
and conformance of requirements with user
needs.
• Software testing is the process of executing
software product on test data and examining its
output vis-à-vis the documented behavior.
Software Testing Objective
• The correct approach to testing a scientific theory is not to
try to verify it, but to seek to refute the theory. That is to
prove that it has errors. (Popper 1965)
• The goal of testing is to expose latent defects in a software
system before it is put to use.
• A software tester tries to break the system. The objective is
to show the presence of a defect not the absence of it.
• Testing cannot show the absence of a defect. It only
increases your confidence in the software.
• exhaustive testing of software is not possible
Successful Test
• “If you think your task is to find problems
then you will look harder for them than if
you think your task is to verify that the
program has none” – Myers 1979.
• “A test is said to be successful if it
discovers an error”.
Limitations of Testing
• In order to prove that a formula or hypothesis is incorrect all you
have to do to show only one example in which you prove that the
formula or theorem is not working.
• On the other hand, million of examples can be developed to support
the hypothesis but this will not prove that it is correct.
• You cannot test a program completely because:
– The domain of the possible inputs is too large to test.
– There are too many possible paths through the program to test.
• According to Discrete Mathematics
– To prove that a formula or hypothesis is incorrect you have to show only
one example.
– To prove that it is correct any numbers of examples are insufficient. You
have to give a formal proof of its correctness.
Example
• Test a function that compares two strings
of characters stored in an array for
equality.
Test Cases
A B Expected
result
“cat” “dog” False
“” “” True
“hen” “hen” True
“hen” “heN” False
“” “ball” False
“cat” “” False
“HEN” “hen” False
“rat” “door” False
“ ” “ ” True
bool isStringsEqual(char a[], char b[])
{
bool result;
if (strlen(a) != strlen(b))
result = false;
else {
for (int i =0; i < strlen(a); i++)
if (a[i] == b[i])
result = true;
else
result = false;
}
return result;
}
Test Cases and Test Data
• In order to test a software application, it is necessary to
generate test cases and test data that is used in the
application.
• Test cases correspond to application functionality such
that the tester writes down steps that should be followed
to achieve certain functionality.
• Thus a test case involves
– Input and output specification plus a statement of the function
under test.
– Steps to perform the function
– Expected results that the software application produces
• Test data includes inputs that have been devised to test
the system.
Testing vs. Development
The Developer and Tester
Development Testing
Development is Testing is a
a creative destructive
activity activity
Objective of Objective of
development is testing is to
to show that the show that the
program works program does
not work
Testing and Software Phases
Description of Testing Phases
• Unit Testing: testing individual components independent
of other components.
• Module Testing: testing a collection of dependent
components – a module encapsulates related
components so it can be tested independently.
• Subsystem Testing: testing of collection of modules to
discover interfacing problems among interacting
modules.
• System Testing: Integrating subsystems into a system
and testing this system as a whole.
Description of Testing Phases
• Acceptance Testing: validation against user
expectations. Usually it is done at the client
premises.
• Alpha Testing: acceptance testing for customized
projects, in-house testing for products.
• Beta Testing: field-testing of product with potential
customers who agree to use it and report problem
before system is released for general use
Testing Techniques
• Black Box or Functional Testing
• White Box of Structured Testing
Black Box Testing
• a component or system is treated as a black box and it is
tested for the required behavior.
• This type of testing is not concerned with how the inputs
are transformed into outputs.
• As the system’s internal implementation details are not
visible to the tester. He gives inputs using an interface
that the system provides and tests the output. If the
outputs match with the expected results, system is fine
otherwise a defect is found.
Structural Testing (White box)
• we look inside the system and evaluate
what it consists of and how is it
implemented.
• in white box testing the internal structures
of the program are analyzed and test
cases are devised that can test these
structures.
Effective Testing
• The objective of testing is to discover the maximum number of
defects with a minimum number of resources before the system is
delivered to the next stage.
• Question is: how to increase the probability of finding a defect?
• As, good testing involves much more than just running the program
a few times to see whether it works or not. A good tester carries out
a thorough analysis of the program to devise test cases that can be
used to test the system systematically and effectively. Problem here
is how to develop a representative set of test cases that could test a
complete program. That is, selection of a few test cases from a huge
set of possibilities. What should the sets of inputs be used to test
the system effectively and efficiently?
Example, String Equal
• For how many equal strings do I have to
test to be in the comfortable zone?
• For how many unequal strings do I have
to test to be in the comfortable zone?
• When should I say that further testing is
unlikely to discover another error?
Equivalence Classes or
Equivalence Partitioning
• Two tests are considered to be equivalent if it is
believed that:
– If one discovers a defect, the other probably will too,
and
– If one does not discover a defect, the other probably
won’t either.
• Equivalence classes help you in designing test
cases to test the system effectively and
efficiently.
Equivalence Partitioning Guidelines
• Organize your equivalence classes. Write them in some
order, use some template, sequence, or group them
based on their similarities or distinctions. These
partitions can be hierarchical or organized in any other
manner.
• Boundary conditions: determine boundary conditions.
For example, adding in an empty linked list, adding after
the last element, adding before the first element, etc.
• You should not forget invalid inputs that a user can give
to a system. For example, widgets on a GUI, numeric
instead of alphabets, etc.
Example, String Match
• Organization
– For equivalence partitions, we divide the
problem in two obvious categories of equal
strings and the other one for unequal strings.
Test Cases for Equivalence
Partitions - Equal
• Two equal strings of arbitrary length
– All lower case “cat” “cat”
– All upper case “CAT” “CAT”
– Mixed case “Cat” “Cat”
– Numeric values “123” “123”
– Two strings with blanks only “ ” “ ”
– Numeric and character mixed “Cat1” “Cat1”
– Strings with special characters “Cat#1” “Cat#1”
• Two NULL strings “” “”
Unequal Strings
• Two different unequal strings of arbitrary length
– Two strings with different length “cat” “mouse”
– Two strings of same length “cat” “dog”
• Check for case sensitivity
– Same strings different capitalization “Cat” “caT”
• One string is empty
– First is NULL “” “cat”
– Second is NULL “cat” “”
White Box Testing
• White box testing tests the structure of a
program
Basis Structures
• Basic Structures
– Sequence
– If
– While
– Case
• Flow chart vs flow graph
Structural Testing
Basic Code Structures and Flow
Graph Notation
Sequence
If
Case While
Example – Bubble Sort
sorted = false;
while (! sorted)
{
sorted = true;
i = 0;
while (i < size -1)
{
if (a[i] > a[i+1])
{
sorted = false;
swap(a[i], a[i+1]);
}
i++;
}
}
Path
• A sequence of edges in the flow graph
form input to output
White Box Testing
• White box testing tests the structure of a
program
• Statement Coverage
• Branch Coverage
– a sequence of statement following a decision
• Path Coverage
– a sequence of statements from input to output
• statement coverage
– test cases
Input: a = 2, b = 2, c = 3
expected result: a = 3; b = 2; c = 3; d = 3
• branch coverage
– test cases
Input: a = 2; b = 2; c = 3; d = 4
expected result: a = 3; b = 2; c = 3; d = 3
input: a = 1, b = 2; c = 3; d = 4
expected result: a = 2; b = 2; c = 3; d = 4
• path coverage
– test cases
Input: a = 2; b = 2; c = 3; d = 4
expected result: a = 3; b = 2; c = 3; d = 3
input: a = 1, b = 2; c = 3; d = 4
expected result: a = 2; b = 2; c = 3; d = 4
Path Coverage
• N=0
– 1-5
• N=1
– 1-2-4-1-5, 1-3-4-1-5
1
• N=2
– 1-2-4-1-2-4-1-5, 1-2-4-1-3-4-1-5 2 3
– 1-3-4-1-2-4-1-5, 1-3-4-1-3-4-1-5
4
• 2N paths
– N = 20 > 1 million paths 5
Path Coverage
• The number of
paths in
programs
containing loops
tend to infinity
• Simple graph
contains 1852
paths with each
loop not iterated
more than twice
Path Coverage
• Complete path coverage is not possible
• Unfortunately there are many faults which
are associated with more complex patterns
of loop iterations such as traverse the outer
loop 3 times and inner loop twice in each
iteration of the outer loop.
• Personal experience
Cyclomatic Complexity
(McCabe)
• Independent Path - any path through the program (from
start to end) that introduces at least one new set of
processing statements or a new condition
– covers statements and branches
• Quantitative measure of the logical complexity of a
program.
• Defines number of independent paths in the basis set of a
program.
• Provides an upper bound for the number of tests that must
be conducted to ensure that all statements and branches
have been executed at least once.
Cyclomatic Complexity
• Cyclomatic Complexity, V(G), for a flow graph G
is defined as:
V(G) = E - N + 2
where E is the number of edges and N is the
number of nodes in the flow graph G.
• Cyclomatic complexity provides us with an upper
bound for the number of independent paths that
comprise the basis set.
Infeasible Paths
• A path through a
1
program which is
never traversed for
any input data 2
1-2-3-4-5 is infeasible
3
if (a == b) //1 1-3-5 is also infeasible
c = c-1; //2
if (a != b) //3 4
c = c+1;//4
//5 5
Infeasible Paths
• Infeasible paths can
be analyzed and fixed.
if (a == b) //1 1
c = c-1; //2
else
2 3
c = c+1;//3
//4
4
• There are no
infeasible paths
now!
Limitation of White Box Testing
• Path coverage
• missing paths
• infeasible paths
• Use black box and white box testing
together
– pure coverage
– http://wiki.c2.com/?CodeCoverageTools
Unit Testing
• Software should be tested more like hardware,
with
– Built-in self testing: such that each unit can be tested
independently
– Internal diagnostics: diagnostics for program units
should be defined.
– Test harness
• The emphasis is on built in testability of the
program units from the very beginning where
each piece should be tested thoroughly before
trying to wire them together.
Unit Testing Principles
• In unit testing, developers test their own
code units (modules, classes, etc.) during
implementation.
• Normal and boundary inputs against
expected results are tested.
• Thus unit testing is a great way to test an
API.
Benefits
• Quantitative Benefits
– Repeatable: Unit test cases can be repeated to verify that no
unintended side effects have occurred due to some
modification in the code.
– Bounded: Narrow focus simplifies finding and fixing defects.
– Cheaper: Find and fix defects early
• Qualitative Benefits
– Assessment-oriented: Writing the unit test forces us to deal
with design issues - cohesion, coupling.
– Confidence building: We know what works at an early stage.
Also it's easier to change when it’s easy to re-test.
Example, Testing against the
Contract – Square Root Routine
result = squareRoot(argument);
assert (abs (result * result – argument) < epsilon);
• The above contract tells us what to test:
– Pass in a negative argument and ensure that it is rejected
– Pass in an argument of zero to ensure that it is accepted (this is
a boundary value)
– Pass in values between zero and the maximum expressible
argument and verify that the difference between the square of
the result and the original argument is less than some value
epsilon.
Unit Testing Tips
• Unit test should be conveniently located
– For small projects you can embed the unit test for a
module in the module itself
– For larger projects you should keep the tests in the
package directory or a /test subdirectory of the
package
• By making the code accessible to developers
you provide them with:
– Examples of how to use all the functionality of your
module
– A means to build regression tests to validate any
future changes to the code
Defect Removal Efficiency
Defect Origination
• In inspections the emphasis is on early detection
and fixing of defects from the program.
– Requirements
– Design
– Coding
– User documentation
– Testing itself can cause defects due to bad fixes
– Change requests at the maintenance or initial usage time
Inspection and Chaotic Zone
Inspection versus Testing
• Inspections and testing are complementary and not
opposing verification techniques.
• Both should be used during the verification and
validation process.
• Inspection does not require execution of program and
they maybe used before implementation.
• Inspections may be applied to any representation of the
system (requirements, design, test data, etc.)
• Inspections can check conformance with a specification
but not conformance with the customer’s real
requirements.
• Inspections cannot check non-functional characteristics
such as performance, usability etc.
Inspection versus Testing
• Many different defects may be discovered in a
single inspection.
• In testing, one defect may mask another so
several executions are required for inspections,
checklists are prepared that contain information
regarding defects.
• Reuse domain and programming knowledge of
the viewers likely to help in preparing these
checklists.
• Inspections involve people examining the source
representation with the aim of discovering
anomalies and defects.
Inspection pre-Conditions
• A precise specification must be available
before inspections.
• Team members must be familiar with the
organization standards.
• In addition to it, syntactically correct code
must be available to the inspectors.
• Inspectors should prepare a checklist that
can help them during the inspection
process.
Inspection Checklists
• Checklist of common errors in a program should be
developed and used to drive the inspection process.
• checklists are programming language dependent
• For example, in a language of weak type checking, one
can expect a number of peculiarities in code that should
be verified. So the corresponding checklist can be larger.
• Other examples of programming language dependent
defects are defects in variable initialization, constant
naming, loop termination, array bounds, etc.
Interface faults Do all function and procedure calls
have correct number of parameters?
Do formal and actual parameters
types match?
Are the parameters in right order?
If components access shared memory,
do they have the same model of
shared memory structure?
Storage If a linked structure is modified, have
management all links been correctly assigned?
faults If dynamic storage is used, has space
been allocated correctly?
Is space explicitly de-allocated after it
is no longer required?
