Unit 5

Testing
 5.1 Concepts
• Testing is an essential activity in software life cycle.
• The goal of testing is to uncover as many errors as possible.
• The software testing is an important activity carried out in order to improve the quality of
software.
• For finding out all possible errors the testing must be conducted systematically and test
cases must be designed using disciplined techniques.
• “Software testing is an activity performed to uncover errors. It is a critical element of
software quality assurance and represents the ultimate review of specification, design and
coding.”
• The purpose of software testing is to ensure whether the software functions appear to be
working according to specification and performance requirements.
 5.1.1 Testing Objectives
• According to Glen Myers, the testing objectives are
1. Testing is a process of executing a program with the intend of finding an error.
2. A good test case is one that has high probability of finding an undiscovered error.
3. A successful test is one that uncovers an as-yet undiscovered error.
• The major testing objective is to design tests that systematically uncover types of errors
with minimum time and effort.
 5.1.2 Testing Principles

• Every software engineer must apply following testing principles while

performing the software testing.

1. All tests should be traceable to customer requirements.

2. Tests should be planned long before testing begins.
3. The Pareto principle can be applied to software testing – 80% of all errors uncovered
during testing will likely be traceable to 20% of all program modules.
4. Testing should begin “in the small” and progress toward testing “in the large”.
5. Exhaustive testing is not possible.
6. To be most effective, testing should be conducted by an independent third party.
 5.1.3 Why Testing is Important?

• Generally, testing is a process that requires more efforts than any other software
engineering activity.

• Testing is a set of activities that can be planned in advance and conducted systematically.

• If it is conducted haphazardly, then only time will be wasted and more even worse errors
may get introduced.

• This may lead to have many undetected errors in the system being developed. Hence
performing testing by adopting systematic strategies is very much essential in during
development of software.
 5.2 Error, Fault and Failure
• Error
– Error refers to the difference between the actual output of the software and correct output of the software.
Hence error is the difference between actual and ideal.
• Fault
– Fault is a condition which causes a system to fail in performing the required functionality. Any software has only
one kind of fault and that is design fault.
• Failure
– Failure is inability of the system of performing the required functionality as per the specification. If the software
behaves differently than specified in the specification then it is said that the failure has occurred in the system.
– Failure occurs when there is fault in the system. But presence of fault does not guarantee a failure.
– On observing failure we can say that faults may exist in the system. But on observing failure we can not say that
the faults are not present in the system.
– Similarly it hard to decide how long should we test the system without observing the failure. Hence “when to stop
the testing?” is hard to decide.
– During testing if faults occur, then there is a separate activity conducted in order to find out the faults in the
system. This activity is referred as debugging.
 5.3 Test Oracles
• It is a mechanism which is used to check the correctness of the output of the program for
the given test case.
• Suppose there are two test cases: One test case is for the program which is to be tested and
other for the test oracle.
• If the output of both is same then that means program behaves correctly otherwise there is
some fault in the program.
Test Case
Program under testing
Output

Test oracle
Test Case
Fig. Testing with Test Oracle
 Continued…
• A test oracle can be either an automated testing tool which checks the expected output of
the program under testing or it can be human being itself.
• If there occurs some difference in test results of program and test oracle then we must
verify the result produced by test oracle.
• If test oracle is a human being then that means human checks the specification of the
program to endure proper working of it.
• But if the program specification itself contain some error or ambiguities then we may get
wrong feedback from test oracle.
• There are some systems in which test oracles are automatically generated from program
specification.
• And output of such oracles are consistent and less error prone.
• Thus correct program specification is necessary to get correct results from test oracle.
 5.4 Test Criteria and Test Case Design
• Test cases are used to determine the presence of fault in the program. Sometimes even if
there is some fault in our program the correct output can be obtained for some inputs.
Hence it is necessary to exercise those set of inputs for which faults can be exposed off.
• Executing test cases require money because –
I. Machine time is required to execute test cases
II. Human efforts are involved in executing test cases
• Hence in project testing minimum number of test cases should be there as far as possible.
• The testing activity should involve two goals –
I. Maximize the number of errors detected
II. Minimize the number of test cases
• The selection of test case should be such that faulty module or program segment must be
exercised by at least one test case.
 Continued…
• Selection of test cases is determined by some criteria which is called test selection criterion.
Hence the test selection criterion T can be defined as the set of conditions that must be
satisfied by the set of test cases.
• Testing criterion are based on two fundamental properties like reliability and validity.
• A test criterion is reliable if all the test cases detect same set of errors.
• A test criterion is valid for any error in the program there is some set which causes error in
the program.
• For testing criteria there is an important theorem –
– “If testing criterion is valid and reliable if a set satisfying testing criterion succeeds then that means
program contains no errors.”
• Generating test cases to satisfy criteria is complex task.
 5.5 Psychology of Testing
• Test cases are designed to detect the errors. However the test cases do not guarantee that
all possible errors will get detected. Moreover, there is no standard method of selecting the
test cases.
• However, there are certain criteria or rule of thumb for the selection of test cases but the
selection of test case is an art and it entirely depends upon the intelligence of the tester.
Hence, psychology of the tester is an important aspect in testing.
• The basic goal of testing is to uncover as much error as possible. Hence the intent of testing
is to show how the program does not work.
• A tester should find out the possibilities wherein the program does not work. Hence testing
should be carried out with the intension of finding out as much error as possible.
• Testing is an destructive process in which the tester has to treat the program as adversary
and should find out presence of errors.
•
 Continued…
• One of the reasons why organisations do not select the developer as tester is
dependent upon the human psychology. It is quite natural that the man who
creates something does not easily dare to find something wrong in it.
• Hence the person who develops the code, would design such test cases that will
show that the code works correctly; he will not select the test cases that shows
errors in his creativity. Hence an independent team is recruited as a tester.
• Another reason for testing the program by independent person is that
sometimes the programmer does not understand the specification clearly, and
then testing the code by independent group will find out the bug easily.
• This approach towards testing will reveal as much error as possible.
 5.6 Levels of Testing
• The testing can be typically carried out in levels.
• In software development process, at each phase some faults may get
introduced.
• These faults are eliminated in the next software development phase but at the
same time some new faults may get introduced.
• Each level of testing performs some typical activity.
• Various levels of testing are –
– Unit testing
– Integration testing
– System testing
– Acceptance testing
 Continued…
• These testing can be conducted at various stages of software development. The levels of
testing along with corresponding software development phase is shown by following figure.

Acceptance
Client Demands
Testing

Fig.
Requirement Levels of Testing
System Testing
Gathering

Integration
Design
Testing

Coding Unit Testing

 Continued…
• Unit testing
– In this type of testing errors are detected from each software component individually.
• Integration testing
– In this testing technique interacting components are verified and the interface errors are
detected.
• System testing
– In system testing all the system elements forming the system is tested as whole.
• Acceptance testing
– Acceptance testing is a kind of testing conducted to ensure that the software works
correctly in user’s working environment.
 5.7 Testing Process
• The main objective of software testing is to uncover as many errors as possible.

• The testing can be carried out in different phases and in each phase the verification is
conducted by taking a review.

• Testing is normally considered as a last activity in software development.

• It is essential to conduct testing before the product gets delivered to the customer. Even
after delivering the product to the customer, customer may suggest some changes and
ultimately the many changes occurs in the software product.

• Hence testing is again necessary even after delivery of the product. That is why testing is
considered as the costliest activity in software development and it should be done
efficiently.
 5.8 Test Case Design
• Test plan specifies how to carry out the testing process for the project, which units need to be tested and
what are those tools that can be used for testing. But it does not specify about the testing unit.
• The specification about the testing unit is given by test cases.
• The test case specification records the results of the testing, the conditions used for testing particular unit.
It also specifies the expected test results. It records the outcome of test cases (Pass/Fail).
• Test case specification is the major activity in the testing process. Careful selection of test cases will help in
conducting proper testing.
• There are two basic reasons why test case specification should be before using them for testing – Firstly it
will assist the tester to reveal as many errors as possible from the program and secondly the high quality
code can be produced.
• The sample structure of a test case specification is as given below –
Test Steps Test Case
Test Test Case Test Case Test Defect
Status
Case Id Name Description Step Expected Actual Priority Severity
(Pass/Fail)
1
n
 5.9 Execution
• After test case specification the next step is to execute the test cases.
• Execution of test cases requires driver and stubs.
• The test case execution procedure is sometimes called as test procedure specification. This
specification consists of special requirements required for setting up the test environment. It also
describes the format of reports of testing result.
• The most common report produced during testing is test summary report and error report.
• The test summary report describes the execution of entire test cases and is used in project
management. The error report specifies the errors and defects encountered during testing.
• During execution of test cases it is important to estimate the test efforts. Testing effort is the total
effort actually spent by the team in testing activities, and is an indicator of whether or not
sufficient testing is being performed.
• Another measure used during testing is the computer time. This measure is used during project
management. At the beginning of the project the computer time consumption is very low then
gradually as the project progresses it increases and reaches to peak. Afterwards at the completion
of the project tis consumption decreases again and becomes low.
 5.10 Taxonomy of Testing
• There are two general approaches for the software testing.
1. Black box testing
– The black box testing is used to demonstrate that the software functions are operational.
– As the name suggests in black box testing it is tested whether the input is accepted properly an output
is correctly produced.
– The major focus of black box testing is on functions, operations, external interfaces, external data and
information.
2. White box testing
– In white box testing the procedural details are closely examined.
– In this testing the internals of software are tested to make sure that they operate according to
specifications and designs.
– Thus major focus of white box testing is on internal structures, logic paths, control flows. Data flows,
internal data structures, conditions, loops etc.
 5.10.1 Testing Strategy
• It begins by “testing-in-the-small” and move toward
“testing-in-the-large”.
• Various testing strategies for conventional software are –
i. Unit Testing: In this type of testing techniques are applied
to detect errors from each software component
individually.
ii. Integration Testing: It focuses on issues associated with
verification and program construction as components
begin interacting with one another.
iii. Validation Testing: It provides assurance that the software
validation criteria meets all functional, behavioural and
performance requirements.
iv. System Testing: In system testing all system elements
forming the system is tested as a whole.
 5.11 Black Box Testing
• The black box testing is also called “Behavioural Testing”.
• Black box testing methods focus on the functional requirements of the software. Test sets are
derived that fully exercise all functional requirements.
• The black box testing is not an alternative to white box testing and it uncovers different class of
errors than white box testing.

Why to perform black box testing?

• Black box testing uncovers following types of errors:
– Incorrect or missing functions
– Interface errors
– Errors in data structures
– Performance errors
– Initialization or termination errors
 5.11.1 Boundary Value Analysis (BVA)

• Boundary value analysis is done to check boundary conditions.

• A boundary value analysis is a testing technique in which the elements at the edge of the
domain are selected and tested.

• Using boundary value analysis, instead of focusing on input conditions only, the test cases
from output domain are also derived.

• Boundary value analysis is a test case design technique that complements equivalence
partitioning technique.
 Continued…
• Guidelines for boundary value analysis technique are:
1. If the input condition specified the range bounded by values x & y, then test cases should
be design with values x & y. Also test cases should be with values above and below x & y.
2. If input condition specifies number of values then the test cases should be designed with
minimum and maximum values as well as with the values that are just above and below
maximum and minimum should be tested.
3. If the output condition specified the range bounded by values x and y, then test cases
should be designed with the values x and y, Also test cases should be with the values
above and below x and y.
4. If output condition specifies the number of values then the test cases should be designed
with minimum and maximum values as well as the values that are just above and below
the maximum and minimum should be tested.
5. If the internal program data structures specify such boundaries then test cases must be
designed such that the values at the boundaries of data structure can be tested.
 Example
• Integer D with input condition [-2, 10] then test values for this input are:
▪ 11, 10, 0, - 1, - 2

• If input condition specifies a number values, test cases should be developed to exercise the
minimum and maximum numbers. Values just above and below this minimum and
maximum value should be tested.

• Enumerate data E with input condition: [2, 7, 100, 102] then test values are:
▪ 2, 102, - 1, 200, 7
 5.11.2 Pair Wise Testing
• There are many parameters that determine the behaviour of the software product.
• For some values this product behaves correctly for some another values this product does
not behave correctly.
• Testing of such product must be done using different values in different conditions.
• That is if there are n parameters each having m different values then the test cases must be
generated for the nm combinations.
• Testing all these combinations is complex. Hence a new approach of testing is designed,
called pair wise testing.
• In this type of testing all the pair values must be exercised and tested.
• The objective of pair-wise testing is to have a set of test cases that cover all the pairs.
 Continued…
• E.g. suppose we have three different set of values
(x1,x2,x3), (y1,y2,y3), (z1,z2,z3) then the test cases x1 y1 z1 (x1,y1),(x1,z1),(y1,z1)
for pair wise testing are shown in table. x1 y2 z2 (x1,y2),(x1,z2),(y2,z2)

• The total number of pair wise combination will be x1 y3 z3 (x1,y3),(x1,z3),(y3,z3)

9*3 = 27. x2 y1 z1 (x2,y1),(x2,z1),(y1,z1)
• In pair wise testing obtaining test cases that will x2 y2 z2 (x2,y2),(x2,z2),(y2,z2)
cover all pairs is not simple task. Sometimes it is not x2 y3 z3 (x2,y3),(x2,z3),(y3,z3)
possible to generate minimum number of test cases x3 y1 z1 (x3,y1),(x3,z1),(y1,z1)
fro different values and different parameters. x3 y2 z2 (x3,y2),(x3,z2),(y2,z2)
• Pair wise testing is a practical method of testing x3 y3 z3 (x3,y3),(x3,z3),(y3,z3)
large software systems which consists of different
parameters with distinct functioning as an example
would be a telephone billing system can be tested
using pair wise testing.
 5.11.3 State Based Testing
• Sometimes the system behaves different based on the input and conditions.
• The different behaviour of the system can be represented by states.
• Any software that saves the states can be modelled as state machines. Every state model has four
components.
1. States: Any event which has some impact of the input.
2. Transition: Represents how one state changes to another on occurrence of some event.
3. Events: Input to the system.
4. Actions: The output for the events.
• The state model is typically created from the specification of the software design.
• Once the state model is built we can select the test cases. However, for the use of state model
detailed system information is needed.
• For instance classes, method and attributes of the entities in the state model.
• This type of testing is called grey box testing as it is between black box and white box testing.
 Example
• Following state model represents the scenario of transaction made by the customer for
using the ATM system.
Invalid Card
Reading Card
Card read
successfully
Invalid Pin
Reading Pin

Getting valid pin Ejecting Card

Choosing Transaction

Cancel transaction

Finished
Another Transaction chosen
transaction
transaction Performing
Transaction
 5.12 White Box Testing
• The white box testing is a testing method which is based on close examination of procedural
details. Hence it is called as glass box testing.
• In white box testing the test cases are derived for
i. Examining all the independent paths within module.
ii. Exercising all the logical paths within module.
iii. Executing all the loops within their boundaries and within operational bounds.
iv. Exercising internal data structures to ensure their validity.
• Why to perform white box testing?
• There are three main objectives behind performing the white box testing.
• Why to perform white box testing?
• There are three main objectives behind performing the white box testing.
i. Programmers may have some incorrect assumptions while designing or implementing
some functions. Due to this there are chances of having logical errors in the program.
To detect and correct such logical errors procedural details need to be examined.
ii. Certain assumptions on flow of control and data may lead programmer to make design
errors. To uncover the errors on logical path, white box testing is must.
iii. There may be certain typographical errors that remain undetected even after syntax &
type checking mechanisms. Such errors can be uncovered during white box testing.
• The structural testing is called white box testing.
• In structural testing the derivation of all the test cases is according to program structure.
Hence knowledge of program is used to identify the additional test cases.
• Objective of structural testing is to exercise all program statement.
 5.12.1 Control Flow Based Testing
• Path testing is a structural testing strategy. This method is intended to exercise every
independent execution path of a program at least once.
• Following are the steps that are carried out while performing path testing.
• Step 1: Design the flow graph for the program or a component.
• Step 2: Calculate the cyclomatic complexity.
• Step 3: Generate test cases for these paths.
• Step 1: Design the flow graph for the program or a component
• Flow graph is a graphical representation of logical control flow of the program.
• Such a graph consists of circle called flow graph node which basically represents one or
more procedural statements, and arrow called edges or links which basically represent
control flow. In this flow graph the areas bounded by nodes and edges are called regions.
• Various notations used in flow graph are.

Sequence If - else
While Case
• E.g. following program is for searching number using binary search method. Draw a flow graph for the
same. (Step 1)
Void search (int key, int n, int a[ ]) Flow Graph
{ int mid; 1
1) int bottom = 0;
2) int top = n – 1; 2
3) while (bottom <= top) { bottom > top while bottom < = top
4) mid = (top + bottom)/2; 3
5) if (a[mid] == key){
6) printf(“Element is present”); 4 III
7) return;} I a[mid] != key
else{ 5 8 a[mid] < key
8) if (a[mid] < key) a[mid] a[mid] > key
9) bottom = mid + 1; == key
9 II 10
6
else
10) top = mid – 1; 7
}} IV
11) } Regions
11
• Step 2: Calculate cyclomatic complexity
• The cyclomatic complexity can be computed by three ways:
1. Cyclomatic complexity = Total number of regions in the flow graph = 4
2. Cyclomatic complexity = E – N + 2 = 13(edges) – 11(nodes) + 2 = 4
3. Cyclomatic complexity = P(Predicates) + 1 = 3 + 1 = 4 (Predicate nodes are 3, 5 and 8)

• Step 3: Select a basis set of path

• The basis paths are:
– Path 1: 1, 2, 3, 4, 5, 6, 7, 11
– Path 2: 1, 2, 3, 11
– Path 3: 1, 2, 3, 4, 5, 8, 9, 3, …
– Path 4: 1, 2, 3, 4, 5, 8, 10, 3, …
• Step 4: Generate test cases for these paths
• After computing cyclomatic complexity and finding independent basis paths, the test cases has to be
executed for these paths. The format for test case is –
• The test case for binary search can be written as
• Preconditions: There should be list of elements arranged in ascending order. The element to be searched
from the list, its value should be entered and will be stored in variable ‘key’.

Test case id Test case Test case Test steps Test case status Test priority Defect
name description Step Expected Actual (Pass / Fail) Severity

1
2
3
…
n
 Test Test case Test case Test steps Test case Test status Test Defect
case id name description Step Expected Actual status (P/F) priority Severity

1. Validating Checking Set bottom = 0 top = Initially bottom <= top Design
the list the bottom n – 1 check if bottom will be true. But during
boundary and top <= top by while loop. iterations list’s length
values for This condition defines will be reduced and if
the list of the length of the list entire list gets scanned
elements from which the key is at one point (bottom >=
searched. top) will be reached
then return to main.
2. Checking Checking if Set mid = (top + if a[mid] = key value Design
list middle bottom) / 2 then print message
element element of Then separate if “Element is present”
with key array is a[mid] is equal to key and return to main.
equal to
key value
if a[mid] is Set bottom = mid + 1 The right sublist will be Design
< key value and then goto “while” searched. The left
if a[mid] is set top = mid + 1 and sublist will be searched
> key value go back to “while” for key element
loop
 5.12.2 Data Flow Based Test Coverage Criteria
• The testing based on data flow mechanism performs testing on definition and uses of
variables in the program.
• In this method of testing, definition and use chain (DU-chain)is required. The DU-chain is
obtained by identifying the def and use pairs from the program structure. This strategy of
testing is also called as DU testing strategy.
• Set DEF(n) contains variables that are defined at node n.
• Set USE(n) contains variables that are read or used at node n.
• E.g. Flow Graph
1
1) s := 0;
2) a := 0; 2
3) while (a<b)
while
4) a := a + 2; 3
5) b := b - 4;
6) if ((a + b) < 20) 4

7) s := s + a + b;
else 5

8) s := s + a - b; if

} T 6 F
7 8

9 10
• For the given programming lines the DU chain will be as
shown in table: DEF(1) = {s} USE(1) = {ø}
DEF(2) = {a} USE(2) = {ø}
DEF(3) = {ø} USE(3) = {a, b}
• Identify all DU pairs and construct test cases that cover
DEF(4) = {a} USE(4) = {a}
these pairs.
DEF(5) = {b} USE(5) = {b}
• Identify all DU paths: That means for each DU pair
DEF(6) = {ø} USE(6) = {a, b}
(n1,n2) for variable a, exercise all possible paths n1…n2
DEF(7) = {s} USE(7) = {s, a, b}
that are clear of definitions of a.
DEF(8) = {s} USE(8) = {s, a, b}
• Identify all users: That means for each DU pair (n1,n2)
DEF(9) = {ø} USE(9) = {ø}
for a, exercise at least one path n1…n2 that is clear of
DEF(10) = {ø} USE(10) = {ø}
definitions of a.
 5.13 Test Case Generation and Tool Support
• After deciding the coverage criteria there are two problems that need to be solved –
1) Deciding whether the test case will satisfy the coverage criteria or not.
2) Generating the test cases for the decided criteria.

• For deciding whether the test case will satisfy the coverage criteria we need to use some
tool. In fact, for all the structural testing methods the use of some tool is must. Generally
such tools give the feedback about what needs to be tested in order to fulfil the criteria.

• But such tools are not easily available, hence fully automated tool for selecting the test
cases that satisfy the criteria is not at all available. Hence we must select the test case
randomly, then using tool get the feedback which will be used for selecting the next
possible test case. This process is continued iteratively until the desired criteria is not
satisfied.
 Continued…
• There are many tools available for statement and branch coverage. Both commercial and
freeware tools are available for different source languages. To get coverage data the
execution of the program during testing has to be monitored. The coverage data can then
be obtained by inserting some statements in the program. Such insertion is called probing.
Thus the purpose of probing is to collect the coverage data required for selection of test
cases. There are three phases of obtaining the coverage data –
– Phase 1: Prepare the program with probes
– Phase 2: Execute the program with test case
– Phase 3: Analyse the output from probed data

• The probe insertion can be done automatically by pre-processor.

• Complex data flow testing tools are also available.
 5.14 Integration Testing
• A group of dependent components are tested
together to ensure their quality of their
integration unit. Integration Testing approach
• The objective is to take unit tested components
and build a program structure that has been
dictated by software design.
• The focus of integration testing is to uncover Non-incremental Incremental
errors in: Integration Integration
– Design and construction of software architecture. Top down
– Integrated functions or operations at subsystem level. Testing
– Interfaces and interactions between them. Regression
Testing
– Resource integration and/or environment integration.
Smoke Testing
• The integration can be carried out using two Big
Bottom up
approaches as shown. Bang
integration
 Continued…
• The non-incremental integration is given by the “big bang” approach. All components are
combined in advance. The entire program is tested as a whole. And chaos usually results. A
set of errors is tested as a whole. Correction is difficult because isolation of causes is
complicated by the size of entire program. Once these errors are corrected new ones
appear. This process continues infinitely.
• Advantages of big-bang is that it is a simple approach.
• Disadvantages of big-bang are:
– It is hard to debug.
– It is not easy to isolate errors while testing.
– In this approach it is not easy to validate test results.
– After performing testing, it is impossible to form an integrated system.
• An incremental construction strategy includes Top down integration, Bottom up integration,
Regression testing and Smoke testing.
 5.14.1 Top Down Integration Testing
• Top down testing is an incremental approach in which modules are integrated by moving
down through the control structure.
• Modules subordinate to the main control module are incorporated into the system in either
a depth first or breadth first manner.
• Integration process can be performed using following steps:
1. The main control module is used as a test driver and stubs are substituted for all modules directly
subordinate to the main control module.
2. Subordinate stubs are replaced one at a time with actual modules using either depth first or breadth
first method.
3. Tests are conducted as each set of tests, another stub is replaced with the real module.
4. Regression testing is conducted to prevent the instruction of new errors.
 Continued…
M1

M2 M6 Program Structure

M3 M4 M5 M7

• In top down integration if the depth first approach is adopted then we will start integration
from module M1 then we will integrate M2 then M3, M4, M5,M6 and then M7.
• If breadth first approach is adopted then we will integrate module M1 first then M2, M6.
Then we will integrate module M3, M4, M5 and finally M7.
 5.14.2 Bottom Up Integration Testing
• In bottom up integration the modules at the lowest levels are integrated at first, then
integration is done by moving upward through the control structure.

• The bottom up integration process can be carried out using following steps:
1. Low-level modules are combined into clusters that perform a specific software sub-
function.
2. A driver program is written to co-ordinate test case input and output.
3. The whole cluster is tested.
4. Drivers are removed and clusters are combined moving upward in the program
structure.
 Continued… M1

D1 D2

Bottom Up Integration
Testing

Cluster 1
Cluster 2

• First components are collected together to form cluster 1 and cluster 2. Then each cluster is
tested using driver program. The clusters subordinate the driver module. After testing the
driver is removed and clusters are directly interfaced to the modules.
 5.14.3 Regression Testing
• Regression testing is used to check fro defects propagated to other modules by changes
made to existing program. Thus regression testing is used to reduce the side effects of the
changes.
• There are three different classes of test cases involved in regression testing:
– Representative sample of existing test cases is used to exercise all software functions.
– Additional test cases focusing software functions likely to be affected by the change.
– Test cases that focus on the changed software components.
• After product had been deployed, regression testing would be necessary because after a
change has been made to the product an error that can be discovered and it should be
corrected.
• Similarly, for deployed product addition of new feature may be requested and
implemented. Fro that reason regression testing is essential.
 5.14.4 Smoke Testing
• The smoke testing is a kind of integration testing technique used for time critical projects
wherein the project needs to be assessed on frequent basis.
• Following activities need to be carried out in smoke testing –
1. Software components already translated into code are integrated into a “build”. The “build” can be
data files, libraries, reusable modules or program components.
2. A series of tests are designed to expose errors from build so that the “build” performs its functioning
correctly.
3. The “build” is integrated with the other builds and the entire product is smoke tested daily.
• Smoke testing benefits are:
1. Integration risk is minimized.
2. The quality of the end product is improved.
3. Error diagnosis and correction are simplified.
4. Assessment of progress is easy.
 5.15 Validation Testing
• The integrated software is tested based on requirements to ensure that the desired product is
obtained.
• In validation testing the main focus is to uncover errors in –
– System input/output
– System functions and information data
– System interfaces with external parts
– User interfaces
– System behaviour and performance
• Software validation can be performed through a series of black box tests.
• After performing the validation tests there exist two condition.
1. The function or performance characteristics are according to the specification and are accepted.
2. The requirement specifications are divided and the deficiency list is created. The deficiencies then can be
resolved by establishing the proper communication with the customer.
• Finally in validation testing a review is taken to ensure that all the elements of software
configuration are developed as per requirements. This review is called configuration review or
audit.
 5.15.1 Acceptance Testing
• The acceptance testing is the kind of testing conducted to ensure that the software works correctly
in the user work environment.
• The acceptance testing can be conducted over a period of weeks or months.
• The types of acceptance testing are:

1. Alpha Test: The alpha testing is a testing in which the version of complete software is tested by
the customer under the supervision of developer. This testing is performed at developer site.
The software is used in natural settings in presence of developer. This test is conducted in
controlled environment.
2. Beta Test: The beta test is a testing in which the version of software is tested by the customer
without the developer being present. This testing is performed at customers’ site. As there is
no presence of developer during testing, it is not controlled by developer. The end user record
the problems and report them to developer. The developer than makes appropriate
modification.
 5.16 System Testing
• The system test is a series of tests conducted to fully the computer based system.
• Various types of system tests are:
1. Recovery testing
2. Security testing
3. Stress testing
4. Performance testing
• The main focus of such testing is to set:
– System functions and performance
– System reliability and recoverability (Recovery test)
– System installation (Installation test)
– System behaviour in the special conditions (Stress test)
– System user operations (Acceptance test/Alpha test)
– Hardware and software integration and collaboration
– Integration of external software and the system
 5.16.1 Recovery Testing

• Recovery testing is intended to check the systems’ ability to recover from failure.

• In this type of testing the software is forced to fail and then it is verified whether the system
recovers properly or not.

• For automated recovery then re-initialization, checkpoint mechanism, data recovery and
restart are verified.
 5.16.2 Security Testing

• Security testing verifies that system protection mechanism prevent improper penetration or
data alteration.

• It also verifies that protection mechanism built into the system prevent intrusion such as
unauthorised internal or external access or wilful damage.

• System design goal is to make the penetration attempt more costly then the value of the
information that will be obtained.
 5.16.3 Stress Testing

• Determines break point of the system to establish maximum service level.

• In stress testing the system is executed in a manner that demands resources in abnormal
quantity, frequency or volume.

• A variation of stress testing is a technique called sensitivity testing.

• The sensitivity testing is a testing in which it is tried to uncover data from a large class of
valid data that may cause instability or improper processing.
 5.16.4 Performance Testing

• Performance testing evaluates the runtime performance of the software, especially real
time software.

• In performance testing, resource utilization such as CPU load, throughput, response time,
memory usage can be measured.

• For big system (E.g. Banking system) involve many users connecting to servers (E.g. using
internet) performance testing is very difficult.

• Beta testing is useful for performance testing.

 5.17 Metrics

• Metrics used in testing is:

– Coverage analysis
– Reliability
 5.17.1 Coverage Analysis
• While developing the software project many work products such as SRS, design document,
source code are being created. Along with these work products many errors may get
generated. Project manager has to identify all these errors to bring quality software.
• Error tracking is a process of assessing the status of the software project.
• The software team performs the formal technical review to test the software developed. In
this review various errors are identified and corrected. Any errors that remain uncovered
and are found in later tasks are called defects.
• The defect removal efficiency can be defined as,
DRE = E / (E + D)
Where, DRE is the defect removal efficiency,
E is the error and
D is defect
• The DRE represents effectiveness of quality assurance activities. The DRE also helps project
manager to assess the progress of software project as it gets developed through its
scheduled work task.
• During error tracking activity following metrics are computed:
1. Errors per requirements specification page: denoted by Ereq
2. Errors per components – design level: denoted by Edesign
3. Errors per component – code level: denoted by Ecode
4. DRE – requirement analysis
5. DRE – architectural design
6. DRE – component level design
7. DRE – coding
The project manager calculates current values of Ereq, Edesign and Ecode .
These values are compared with past projects. If the current result differs more than 20% from the
average, then there may be cause of concern and investigation needs to be made in this regard.

• These error tracking metrics can also be used for better target review and testing resources.
 5.17.2 Reliability

• Software reliability is defined as the probability of failure free operation of a computer

program in specified environment for a specified time.

• The software reliability can be measured, directed and estimated.

 [Link] Measures of Reliability and Availability
• Normally there are two measures of software reliability
1. MTBF:
– Mean-Time-Between-Failure is a simple measure of a software reliability which can be calculated as,
MTBF = MTTF + MTTR
Where, MTTF is Mean-Time-To-Failure
and MTTR stands for Mean-Time-To-Repair
– Many software researchers feel that MTBF is more useful measure of software reliability then defects/
KLOC or defect/FP.
2. Availability
– It’s another measure of software reliability software availability is defined as the probability that the
program is working according to the requirements at a given points in time. It is measured as,
Availability = (MTTF / (MTTF + MTTR)) * 100 %
– MTBF is equally sensitive to MTTF and MTTR but availability is more sensitive to MTTR.