ISYS6338– Testing and System

Implementation

Session 1- Foundation of Testing

 Learning Outcomes

• LO 1: Explain the fundamentals of testing

 References

• Dorothy Graham, Rex Black, Erik van Veenendaal. (2019).

Foundations of Software Testing: ISTQB
Certification, 4th Edition. Cengage Learning.
Hampshire, UK. ISBN: 9781473764798.
• Black, Rex. (2009). Managing the testing process :
practical tools and techniques for managing hardware
and software testing. 03. Wiley. Indianapolis. ISBN:
9780470404157.
 Sub Topics

• Definition of Testing
• Test Objectives
• Quality Assurance
• Errors, Defects & Failures
• Principles in Testing
What is Testing?
 What is testing?

• Software systems, in the sense that we know them, are a

recent innovation, less than 70 years old, but have
already transformed daily life around the world.
Thomas Watson, the one-time head of IBM, famously
predicted that only about five computers would be
needed in the whole world.
• This vastly inaccurate prediction was based on the
idea that information technology was useful only for
business and government applications, such as banking,
insurance and con-ducting a census.
• Now, everyone who drives a car is using a machine not
only designed with the help of computers, but which also
contains more computing power than the
computers used by NASA to get Apollo missions to
and from the Moon.
 What is testing?
(cont…)
• Mobile phones are now essentially handheld
computers that get smarter with every new model. The
Internet of Things (IoT) now gives us the ability to
see who is at our door or turn on the lights when we are
nowhere near our home.
 What is testing?
(cont…)
• However, in the software world, the technological
triumph has not been perfect. Almost every living person
has been touched by information technology, and most
of us have dealt with the frustration and wasted time that
occurs when software fails and exhibits unexpected
behaviors.
• Some unfortunate individuals and companies have
experienced financial loss or damage to their personal or
business reputations because of defective software. A
highly unlucky few have even been injured or killed by
software failures, including by self-driving cars.
• One way to help overcome such problems is software
testing, when it is done well. Testing covers activities
throughout the life cycle and can have a few
different objectives
 Testing is a …

The process consisting of all life cycle activities, both static and
dynamic, concerned with planning, preparation and evaluation
of software products and related work products to determine
that they satisfy specified requirements, to demonstrate that
they are fit for purpose and to detect defects.
 Validation & Verification

• Testing is more that a verification

– Another common misconception about testing is
that it is only about checking correctness; that
is, that the system corresponds to its
requirements, user stories or other
specifications.
• Verification
– Confirmation by examination and through provision
of objective evidence that specified requirements
have been fulfilled.
• Validation
– Confirmation by examination and through provision of
objective evidence that the requirements for a
specific intended use or application have been
fulfilled.
 Validation & Verification
(cont..)
• In every development life cycle, a part of testing is
focused on verification testing and a part is focused on
validation testing. Verification is concerned with
evaluating a work product, component or system to
determine whether it meets the requirements set.
– In fact, verification focuses on the question, ‘Is the
deliverable built according to the specification?’
• Validation is concerned with evaluating a work
product, com-ponent or system to determine whether
it meets the user needs and requirements.
– Validation focuses on the question, ‘Is the deliverable
fit for purpose; for example, does it provide a solution
to the problem?’
Why is Testing Necessary?
 Why is Testing
Necessary?
• Testing can help to reduce the risk of failures occurring
during operation, provided it is carried out in a rigorous
way, including reviews of documents and other work
products.

• Testing both verifies that a system is correctly built and

validates that it will meet users’ and stakeholders’
needs, even though no testing is ever exhaustive.

• In some situations, testing may not only be helpful,

but may be necessary to meet contractual or legal
requirements or to conform to industry-
specific standards, such as automotive or safety-critical
systems.
 Objectives for Testing

• Test objective
– Is a reason or purpose for designing and executing a
test.
• The test objective based on the foundation
syllabus are: (but these objectives are not
universal. Different test viewpoints, test levels and
test stakeholders can have different objectives.)
– To evaluate work products such as requirements,
user stories, design and code by using static testing
techniques, such as reviews.
– To verify whether all specified requirements have
been fulfilled, for example, in the resulting system.
– To validate whether the test object is complete and
works as the users and other stakeholders expect – for
example, together with user or stakeholder groups.
 Objectives for Testing
(cont.)
• To build confidence in the level of quality of the test
object, such as when those tests considered highest risk
pass, and when the failures that are observed in the
other tests are considered acceptable.
• To prevent defects
• To find failures and defects; this is typically a prime
focus for software testing.
• To provide sufficient information to stakeholders to
allow them to make informed decisions, especially
regarding the level of quality of the test object
• To reduce the level of risk of inadequate software
quality
• To comply with contractual, legal or regulatory
requirements or standards, and/or to verify the test
object’s compliance with such requirements or standards.
 Debugging

• Debugging is
– A process of finding, analyzing and removing the
causes of failures in software.
• Testing is not debugging.
– While dynamic testing often locates failures which are
caused by defects, and static testing often locates
defects themselves, testing does not fix defects.
– It is during debugging, a development activity, that
a member of the project team finds, analyzes and
removes the defect, the underlying cause of the
failure.
– After debugging, there is a further testing activity
associated with the defect, which is called
confirmation testing. This activity ensures that the
fix does indeed resolve the failure.
Quality Assurance
 Quality Assurance

• Many people refer to ‘doing QA’ when they are

actually doing testing, and some job titles refer to
QA when they really mean testing.
• The two are not the same. Quality assurance is
actually one part of a larger concept, quality
management, which refers to all activities that direct and
control an organization with regard to quality in all
aspects.
• Quality affects not only software development but also
human resources (HR) procedures, delivery processes
and even the way people answer the company’s
telephones.
• Quality Assurance
– Part of quality management focused on providing
confidence that quality requirements will be fulfilled.
 Quality Assurance
(cont..)
• Quality management consists of a number of
activities, including quality assurance and quality
control (as well as setting quality objectives, quality
planning and quality improvement).
• Quality assurance is associated with ensuring that a
company’s standard ways of performing various tasks are
carried out correctly. Such procedures may be written in a
quality handbook that everyone is supposed to follow.
• The idea is that if processes are carried out correctly,
then the products produced will be of higher quality.
• Root cause analysis and retrospectives are used to help
to improve processes for more effective quality
assurance.
 Quality Assurance
(cont..)
• Testing plays an essential supporting role in delivering
quality software.
• However, testing by itself is not sufficient. Testing should
be integrated into a complete, team-wide and
development process-wide set of activities for quality
assurance.
• Proper application of standards, training of staff, the use
of retrospectives to learn lessons from defects and other
important elements of previous projects, rigorous and
appropriate software testing
• All these activities above and more should be deployed
by organizations to ensure acceptable levels of
quality and quality risk upon release.
Error, Defects & Failures
 Errors, Defects and
Failures
• Why does software fail? Part of the problem is that,
ironically, while computerization has allowed dramatic
automation of many professions, software engineering
remains a human-intensive activity.
• Humans are fallible beings. So, software is fallible
because humans are fallible.
• Errors (mistake)
– A human action that produces an incorrect result.
• Defects (bug, fault)
– An imperfection or deficiency in a work product where
it does not meet its requirements or specifications.
• Failures
– An event in which a component or system does not
perform a required function within specified limits.
 Errors, Defects and
Failures
• The precise chain
(cont..)
of events goes something like this. A
developer makes an error (or mistake), such as forgetting
about the possibility of inputting an excessively long
string into a field on a screen.
• The developer thus puts a defect (or fault or bug) into the
program, such as omitting a check on input strings for
length prior to processing them.
• When the program is executed, if the right conditions
exist (or the wrong conditions, depending on how you
look at it), the defect may result in unexpected
behavior; that is, the system exhibits a failure, such
as accepting an over-long input that it should reject, with
subsequent corruption of other data.
 Errors, Defects and
Failures
• Figure 1.1
(cont..)
shows four typical scenarios, the upper
stream being correct requirements, design and
implementation, the lower three streams showing defect
introduction at some phase in the software life cycle.

Source: Graham (2019)

 Errors, Defects and
Failures
• The extent to
(cont..)
which defects are removed in the phase
of introduction is called phase containment.
• Phase containment is important because the cost of
finding and removing a defect increases each time that
defect escapes to a later life cycle phase. Multiplicative
increases in cost, of the sort seen in Figure 1.2,
are not unusual.

Source: Graham (2019)

 Defects, root causes and
effects
• If root cause analysis is carried out for the defects
found on each project, the team can improve its software
development processes to avoid the introduction
of similar defects in future systems.
• Through this simple process of learning from past
mistakes, organizations can continuously improve the
quality of their processes and their software.
• A root cause is generally an organizational issue,
whereas a cause for a defect is an individual
action.
• Root cause
– A source of a defect such that if it is removed, the
occurrence of the defect type is decreased or
removed.
 Defects, root causes and
effects (cont..)
• The root cause can be addressed by providing additional
training in interest rate calculations to the product owner,
and possibly additional reviews of user stories by interest
calculation experts.
• If this is done, then incorrect interest calculations
due to ambiguous user stories should be a thing of
the past.
Principles in Software Testing
 Testing Principles

1
• Testing shows the presence of defects, not their absence

2
• Exhaustive testing is impossible

3
• Early testing saves time and money

4
• Defects cluster together

5
• Beware of the pesticide paradox

6
• Testing is context dependent

7
• Absence-of-errors is a fallacy
 Testing Principles
(cont.)

Source: Graham (2019)

Thank