Software Quality Engineering
Lecture # 2
Instructor: Rabia Khan
SOFTWARE ERRORS, DEFECTS, AND FAILURES
The system crashed during production.
The designer made an error.
After a review, we found a defect in the test plan.
I found a bug in a program today.
The system broke down.
The client complained about a problem with a calculation in the
payment
report.
A failure was reported in the monitoring subsystem.
Defects: Definitions, Properties
and Measurements
The term “defect” generally refers to some
problem with the software, either with its
external behavior or with its internal
characteristics.
The IEEE Standard 610.12 (IEEE, 1990)
defines the following terms related to
defects:
Failure: The inability of a system or component to
perform its required functions
Fault: An incorrect step, process, or data definition in a
computer program.
Error: A human action that produces an incorrect result.
Error, Fault and Failure
Error, Fault and Failure
Error
Error is a missing or incorrect human
action. It can be a mistake, misconception
or misunderstanding on part of the
developer.
E.g., defective programming by a developer
Logical mistakes: divide by zero, access bad
memory, reusing a delete pointer etc.
E.g., erroneous requirements due to
unexpected wishes, expectations and needs
of different user groups.
Error, Fault and Failure
Fault/Bug/Defect
An incorrect process defect and data
definition in a computer program that
causes the program to perform in an
unintended or unanticipated manner.
A fault is introduced into the software as
the result of an error.
E.g., faults can be incorrect or forgotten
statements in the program OR state of
software caused by an error.
Error, Fault and Failure
Failure
Deviation of a software from its expected
result.
OR failure means that the given
requirements not fulfilled. It is a
discrepancy between an actual result and
the expected result or behavior.
The failure becomes visible to the tester or
user
E.g., an output is wrong or program crashes
• Failure is an event
• Fault is the state of the system caused by an error
Example to differentiate
between Error, Fault and
Failure - 2
Suppose a software company is developing a mobile banking
application. During testing, the QA team discovers a scenario
where users are able to transfer funds between accounts
without entering a valid PIN code.
Error - ?
Fault - ?
Failure - ?
Error - ? Developer doesn’t implement a check of PIN
verification in code
Fault - ? The software allows unauthenticated access
Failure - ? The software fails to prove authenticated access in
funds transfer
Example to differentiate
between Error, Fault and
Failure
Imagine a team of developers is working on an e-commerce
website. They recently implemented a new feature that
allows users to add items to their shopping cart and proceed
to checkout. However, after the feature is deployed, users
report an issue where their selected items disappear from
the cart when they try to checkout.
Error - Developer wrote code to manage the shopping cart
feature. Error in code that mistakenly clears the cart
contents before completing the checkout process (human
error)
Fault – Fault occurring in software (empty cart)
Failure - Deviation of a software from its expected
result
Defect Related Concepts and Relations
f4: dormant or latent faults
Case of Errors, Defects, and
Failures
QA Dealing with Defects
QA activities to prevent, eliminate, reduce, or
contain various specific problems associated with
different aspects of defects. There are 3 generic
categories:
Defect prevention through error blocking or error
source removal.
Defect reduction through fault detection and removal
Defect containment through failure prevention and
containment
Quality Engineering includes:
Quality planning before specific QA activities are
carried out
Measurement, analysis and feedback to monitor and
control the feedback activities.
Generic ways to deal with
defects
Defect prevention through error blocking or error source
removal
To avoid defects from ENTERING into the system
Example for a medical diagnostic application, check the
format of images being uploaded before them being
uploaded
Defect reduction through fault detection and removal
Removal of errors once they have ENTERED the system
Example: code reviews can pinpoint memory leaks
Defect containment through failure prevention and
containment
strategies aim to prevent defects from causing widespread failures
and contain their impact when they occur
Example: In the case of shipping defects in an ecommerce
application, customers can easily reach out for assistance, and
support teams can provide timely updates
Generic ways to deal with defects
Static
Testing
QA Activities
Related to Defect
Defect Prevention Defect
Containment
Defect Reduction
Error Software Fault
Error Source Tolerance
Blocking Removal
Inspection
Safety
Assurance
Improved
Processes Root-
cause Testing Hazard
Analysis Elimination
Use of
Tools/Technologi and Reduction
es Education/
Training Hazard Control
Defect Prevention
The strategy applied to the SDLC that identifies root
causes of defects and prevents them from recurring
Defect prevention activities assume that there are
known error sources or missing incorrect actions that
result in fault injections
Defect prevention can be done in two generic ways:
Error source removal
Error blocking
These QA activities prevent certain types of faults from
being injected into the software
Since errors are missing/incorrect actions that lead to
the injection of faults into the software, we can correctly
correct or block these actions or remove the underlying
causes of them
Defect Prevention – Error Source Removal
Error source removal
Eliminating certain error sources such as
eliminating ambiguities, or correcting human
misconceptions, which are the root causes for
errors
Root cause analysis – identifying error
sources
Remove through education training etc
Error Source Removal Techniques
Root cause analysis
Root cause analysis is basically the analysis of the
main cause of the defect.
It simply analyzes what triggered the defect to occur.
After analyzing the main cause of the defect one can
find the best way to simply avoid the occurrence of
such types of defects next time.
There are two main approaches of root cause analysis:
I) Pareto Analysis
II) Fishbone Analysis
Error Source Removal –
Rootcause Analysis
Root cause analysis
Pareto Analysis
Pareto analysis is a formal and simple technique that helps prioritize
the order of problem resolution for maximum impact. It states that
80% of the problem arises due to 20% reasons.
Therefore, the problems once identified are prioritized according to
frequency, and detailed-statistics based analysis is performed to
find 20% of the reasons attributed to the 80% of problems. By
simply focusing on those 20% reasons and eliminating those, results
are guaranteed while optimizing the extent of work involved.
Rootcause Analysis – Pareto
Analysis
Root-cause Analysis – Fishbone
Analysis
Root cause analysis technique
Fishbone Analysis
This method is the more visual root cause analysis technique. There
are no statistics involved as this method is based on team-wide
brainstorming.
The problem is first written on the rightmost side and on the horizontal
line that passes through it, the various causes are listed. The branch
that has the most cause subclass bones (lines/branches) is the problem
that is more serious and that is to be worked towards elimination. This
technique is also sometimes called cause and effect analysis.
Fishbone Analysis – Website
Crash Example
This is a root-cause example of a website
crash. It doesn’t include statistics, rather
classifies each problem into main categories
Defect Prevention – Error
Blocking
Error blocking
Fault prevention or blocking by directly
correcting or blocking these missing or
incorrect human actions
This group of techniques breaks the
casual relation between error sources
and faults using certain tools and
technologies, enforcement of certain
process and product standards.
QA Activities
Related to Defect
Defect Prevention Defect
Containment
Defect Reduction
Error Software Fault
Error Source Tolerance
Blocking Removal
Review
Safety
Assurance
Improved
Processes Root-
cause Testing Hazard
Analysis Elimination
Use of
Tools/Technologi and Reduction
es Education/
Training Hazard Control
Defect Prevention Activities
Most defect prevention activities assume that
there are known error sources or missing
incorrect actions that result in fault injection
as follows:
If human misconceptions are the error
sources, education and training can help
us remove these error sources.
It should be focused on following areas:
Product and domain specific knowledge
Software development knowledge and expertise
Knowledge about Development methodology
Development process knowledge
Defect Prevention
If imprecise designs and implementations
that deviate from product specifications or
design intentions are the causes for faults,
formal methods can help us prevent such
deviations.
QA Activities
Related to Defect
Defect Prevention Defect
Containment
Defect Reduction
Error Software Fault
Error Source Tolerance
Blocking Removal
Review
Safety
Assurance
Improved
Processes Root-
cause Testing Hazard
Analysis Elimination
Use of
Tools/Technologi and Reduction
es Education/
Training Hazard Control
Defect Reduction
For most large systems, it is unrealistic to
expect to prevent 100% of defects.
Need techniques to reduce the faults
injected as a result of errors
Review
Testing
Reviews
Widely used static technique for defect
reduction (defect detection and defect
removal)
STATIC TECHNIQUES
Informal reviews
Inspection is critical reading and analysis of
Form
software code, design, product specifications,
test plans etc
al
Walkthroughs
Formal Inspections
Fagan Inspection
Review Techniques
Informal reviews (casually between 2 or 3 team
members)
Inspection is critical reading and analysis of
software code, design, product specifications, test
plans etc.
Done by external reviewer and not the developer
Walkthroughs (an author leads the team through a
manual or simulated execution of the product)
Form
Used in high level documents i.e. SRS, design
document, code
al
Formal Inspections
Fagan Inspection
Formal technique with entry and exit points. Planning &
Scheduling, Documenting and logging the defects,
rework in the development process, Follow-up
Review Techniques
QA Activities
Related to Defect
Defect Prevention Defect
Containment
Defect Reduction
Error Software Fault
Error Source Tolerance
Blocking Removal
Review
Safety
Assurance
Improved
Processes Root-
cause Testing Hazard
Analysis Elimination
Use of
Tools/Technologi and Reduction
es Education/
Training Hazard Control
Testing
Execution of software and observation of
program/software outcome
Most common QA activity
Dynamic in nature
When to test?
Depends on the software development
methodology
However, it can start much earlier
What to test
White box Testing (Structural Testing)
Correct Implementation of Internal Units
Black Box Testing (Functional Testing)
Correct Handling of External Functions
White-box Testing vs Black Box
Testing
White box Black box
Testing Testing
• The tester • The tester
needs to have doesn’t need
coding/ to have
programming coding/progra
knowledge to mming
test internal knowledge.
working of the Only checks
product the
• Testing done functionality
from of the system
developer’s • Focus on Input
perspective and Output
• Testing done
from user
QA Activities
Related to Defect
Defect Prevention Defect
Containment
Defect Reduction
Error Software Fault
Error Source Tolerance
Blocking Removal
Inspection
Safety
Assurance
Improved
Processes Root-
cause Testing Hazard
Analysis Elimination
Use of
Tools/Technologi and Reduction
es Education/
Training Hazard Control
Defect Containment
Defect Reduction can only remove certain
faults to a fairly low level but cannot
completely eliminate them
“Containment” refers to containing the
effect of a defect to a local fault rather than
a global failure thus “tolerating” these local
faults
Limiting the damage caused by the defect
Achieved by :
Fault Tolerance
Break the causal relationship between
faults and failures
Safety Assurance
Defect Containment- Fault
Tolerance
Have system backups
For example, a server can be made fault
tolerant by using an identical server running
in parallel, with all operations mirrored to the
backup server.
2 Ways to achieve fault tolerance :
Recovery : Rollback and Redo
Recovery is a process that restores an erroneous
state to an error-free state. (after a failure, restoring
system to its "normal" state)
NVP (N-version Programming)
Have multiple versions of same specs running in
parallel
Seamlessly switch to another version
Defect Containment- Safety
Assurance
Avoid catastrophic consequences
Prevent accidents from happening
Works on the same principle as fault
tolerance but is more critical in terms of
severity
Hazard Elimination
Eliminate Hazard sources
Hazard Reduction
Reduce the probability of hazard
Hazard Control
Reduce the severity of failures
QA in Waterfall Model
QA Activities in Waterfall
Defect Prevention in
Waterfall
Requirement Specification Review
After understanding customer’s requirements prepare your
requirement’s gist.
A review is important at this step- the First level of review should be
within the team, followed by another level of external review (by a
dev or BA or client) to make sure that all the perspectives are in sync.
Design Review
Design stage can be considered a strategy stage of sorts and going
through it will ensure that the QA team understands the pros and
cons of each strategy.
This kind of critical walkthrough will help unearth any problems with
the said strategies and fix them before going further. This can be
considered a feasibility study for the strategy (or strategies).
Code Review
There is not a lot for testers to directly get involved in this phase, but
the review does go on here too. Developers carry out code
inspections, walkthroughs and reviews before they unit and
integration test the application.
Review- What we covered
Review – What we
covered
References
Chapter 3 : software quality engineering by
JEFF TIAN
Chapter 13 : software quality engineering
by JEFF TIAN
Chapter 15 : software quality engineering
by JEFF TIAN
