Fundamentals of Testing
Fundamentals of Testing
Fundamentals of 1
Testing
1.1 What is 2
Testing?
Why testing is necessary?
A few questions:
1. Will we trust a medical device that is not
tested?
2. Will we use a car where the speed index is not
working properly?
3. Are we going to read e-books with a lot of
typos?
4. Are we going to live in an intelligent house 3
DO WE NEED TESTING?
What is Testing?
Testing
The process consisting of all lifecycle activities, both static and dynamic,
concerned with:
• planning,
• preparation,
Check that the application is according to the Check that the application is correct 7
Testing
Objectives of testing
Objectives of testing:
For example:
Acceptance testing • Validating that the system is complete and will work as expected.
• Verifying that functional and non-functional behaviors of the system are as speci ed.
1.1.2 Testing
and 11
Debugging
Testing it is not debugging
Debugging
The process of nding, analyzing and removing the causes of failures in a component or system.
Debugging is a development activity, that nds, analyzes, and xes such defects. Subsequent
con rmation testing checks whether the xes resolved the defects.
12
13
1.2 Why is
Testing 14
Necessary?
Why is Testing Necessary?
Rigorous testing of components and systems, and their associated documentation, can help reduce
the risk of failures occurring during operation. When defects are detected, and subsequently xed,
this contributes to the quality of the components or systems.
In addition, software testing may also be required to meet contractual or legal requirements or
industry-speci c standards.
15
1.2.1 Testing
Contributions 16
to Success
Testing Contributions to Success
Ariane 5 received a navigation platform from Ariane 4 on a copy-paste basis.
Nobody cared that the overloads and ight path di ered signi cantly from those in
Ariane 4.
During the rst phases of ight (30 seconds after taking o from the Kourou ELA-3
platform), the autopilot began to rapidly adjust the position of the afterburner
nozzles to do the same with the main engine nozzle. The reason for this action was
incorrect data supplied by the navigation computer to the control module.
17
The remote control software was written in Ada. The main cause of the catastrophe was the integer
over ow.
The error occurred when an unprotected number conversion from 32 bits to 16 bits was performed.
Testing Contributions to Success
For over 50 million people, these were three days of fear, panic
and a real life lesson. All through a certain server in a small control
room of the FirstEnergy corporation. Due to a minor error in the
software, a minor and harmless failure turned into an uncontrolled
energy disaster. As many as 265 power plants failed!
All of New York was plunged into darkness. Within moments,
millions of people lost access to electricity.
18
After the job queue was full, the primary server crashed, followed by the failover server. The
operators could not react because the error caused the delay in displaying the images on the control
screens from 2 seconds to 59! After a few alarming phone calls and the rst breakdowns, the
technicians, looking at the monitors, calmed the callers and informed that everything looked OK.
Testing Contributions to Success
Using appropriate test techniques can reduce the frequency of such problematic deliveries, when
those techniques are applied with the appropriate level of test expertise, in the appropriate test
levels, and at the appropriate points in the software development lifecycle.
Examples include:
• Having testers involved in requirements reviews or user story re nement could detect 19
defects in these work products.
• Having testers work closely with system designers while the system is being designed can
increase each party’s understanding of the design and how to test it.
• Having testers work closely with developers while the code is under development can increase
each party’s understanding of the code and how to test it.
• Having testers verify and validate the software prior to release can detect failures that might
otherwise have been missed, and support the process of removing the defects that caused the
failures (i.e., debugging).
1.2.2 Quality
Assurance 20
and Testing
Quality Assurance and Testing
Quality
The degree to which a component or system satis es the stated and implied needs of its various stakeholders.
Testing
Testing contributes to the achievement of quality in a variety of ways.
Test results provide information about software quality and give the possibility to measure 21
software quality..
Test results helps build con dence in the software quality,if detected few or no defects found.
Quality Assurance and Testing
22
1.2.3 Errors,
Defects, 23
and Failures
Errors / Defects / Failures
Error
A human action that produces an incorrect result.
Defect
An imperfection or de ciency in a work product where it does not meet its requirements or speci cations.
24
Failure
An event in which a component or system does not perform a required function within speci ed limits.
Errors / Defects / Failures
ERRORS may occur for many reasons, such as: FAILURES caused due to:
Time pressure, defects in the code,
Human fallibility, environmental conditions (radiation,
Inexperienced or insu ciently skilled project electromagnetic elds, and pollution)
participants,
Miscommunication between project
participants, including miscommunication
about requirements and design, 25
False negatives are tests that do not detect defects that they should have detected
1.2.4 Defects,
Root Causes 27
and E ects
Defects, Root Causes and E ects
AT&T network failure.
On January 15, 1990, there was a major failure of the telecommunications network in the United
States. As of 2:25 AM, the AT&T operations center in Bedminster began receiving warning messages
from various parts of the network. The failure proceeded rapidly. The problem turned out to be a
defect in the code intended to improve the speed of message processing.
The cost of this failure is estimated at $ 60 million.
28
E ects of AT&T failure are complaints from customers who could not make calls and use the
Internet.
Defects are BTS stations that are not working properly
Root causes was due to a problem in the code and incorrect use of the break instruction.
1.3 Seven
Testing 29
Principles
Seven Testing Principles
1. Testing shows the presence of defects, not their absence
Testing can show that defects are present, but cannot prove that there are no defects. Testing
reduces the probability of undiscovered defects remaining in the software but, even if no defects are
found, testing is not a proof of correctness.
Exhaustive testing
A test approach in which the test suite comprises all combinations of input values and preconditions.
Seven Testing Principles
We want to test the requirement for two-step veri cation for our bank account.
we need to enter the phone number (9 - digits) and the PIN code (4 - digits) received from the bank
The login form looks like this
31
Try to calculate how long it would take for the test script that checks
one combination in 1 millisecond ;-).
Seven Testing Principles
3. Early testing saves time and money
ASPA ... as soon as possible
Testing early in the software development lifecycle helps reduce or eliminate costly changes .
33
34
Predicted defect clusters, and the actual observed defect clusters in test or operation, are an
important input into a risk analysis used to focus the test e ort
Seven Testing Principles
5. Beware of the pesticide paradox
If the same tests are repeated over and over again, eventually these tests no longer nd any new
defects.
35
In some cases, such as automated regression testing, the pesticide paradox has a bene cial
outcome, which is the relatively low number of regression defects.
Seven Testing Principles
Let's go back to the client's login form to the bank using the phone number and PIN code.
The login form looks like this:
36
When can the pesticide paradox occur with such simple testing?
Seven Testing Principles
The pesticide paradox in this case can be associated with test data.
If we use the same phone number (694 733 000) and the same PIN code all the time
(1234) we ignore many of the conditions that may contribute to the failure.
• What if the phone number of another operator will not be recognized in system?
• We will also not check whether another (default) PIN code prevents us from getting to system?
Seven Testing Principles
6. Testing is context dependent
Testing is done di erently in di erent contexts. For example, safety-critical industrial control
software is tested di erently from an e-commerce mobile app..
There will be a di erent approach to testing for each of the above systems.
38
Seven Testing Principles
How we approach the tested system depends on many factors:
• business requirements,
• risk level,
• lifecycle project (np. SCRUM, Model V),
• system criticality.
39
Thoroughly testing all speci ed requirements and xing all defects found could still produce a
system that is di cult to use, that does not ful ll the users’ needs and expectations, or that is
inferior compared to other competing systems.
40
1.4 Test 41
Process
1.4.1 Test
Process in 42
Context
Test Process in Context
Test process
The set of interrelated activities comprising of test planning, test monitoring and control, test analysis, test design, test
implementation, test execution, and test completion.
There is no one universal software test process, but there are common sets of test activities 43
without which testing will be less likely to achieve its established objectives
Test Process in Context
Contextual factors that in uence the test process for an organization, include, but are not limited to:
47
Tasks
Test Process
Test Activities:
Test planning,
Test monitoring and control,
Test analysis,
49
Test design,
Test implementation,
Test execution,
Test completion.
Test planning
RESPONSIBLE: Test manager
Requirement
A provision that contains criteria to be ful lled.
Test condition
53
A testable aspect of a component or system identi ed as a basis for testing.
Test basis
The body of knowledge used as the basis for test analysis and design.
Test analysis - requirements
Requirement (IREB de nition)
1. A need perceived by a stakeholder
2. A capability or property that a system shall have
3. A documented representation of a need, capability or property.
The requirement is a description of a solution - a set of features required by the user, customer or 54
other stakeholder to achieve the objective.
User stories
is a convenient form of expressing the expected business value. User stories are written in such a
way that they can be understood by both people from the business side of the project, as well as by
engineers. They are simple in structure and provide a good platform for conversation.
55
Źródło
Test analysis - User stories template
A correct high-level story (not very detailed) should contain:
• User Stories
As a ......< type of user >
I want ......< some goal >
56
so that ...... < some reason >.
Test analysis - User stories example
User registration
As a an unregistered user
I want register on the site,
so that be able to log in.
Site navigation
57
As a user
I want access to the top menu of application from anywhere in the application,
so that navigate the site would be easily.
Test analysis - Test condition
An example is a properly functioning ATM.
The identi cation of defects during test analysis is an important potential bene t, especially where
no other review process is being used and/or the test process is closely connected with the review
process. Such test analysis activities not only verify whether the requirements are consistent,
properly expressed, and complete, but also validate whether the requirements properly capture
customer, user, and other stakeholder needs.
Test design
RESPONSIBLE Tester
During test design, the test conditions are elaborated into high-level test cases, sets of high-level test
cases, and other testware.
Test case
A set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on test
conditions.
62
Test design - test case
ID TC1
TC Name Successful cash withdrawal from an ATM
Input • Correct debit card
• Withdrawal amount = 50 $ 64
Precondition A withdrawal can only be made for a functioning ATM that displays the welcome screen as well using a good
card and entering the correct PIN.
Expected
result
• Successfully withdrawal 50 $
The test cases can be divided into: high-level test cases & low-level test cases.
65
Test design - test case
It is more general. It leaves space for the Describes the test process in detail. It has
tester to interpret. carefully written steps and often test data.
Easier to maintain. It guarantees repeatability. Each test case
They provide greater test coverage, because run will be the same.
each time the performance may be slightly It does not require a lot of experience from 66
di erent (e.g. used other test data). the tester and in-depth knowledge of the
A better choice if we do not have a well- application.
described requirement, or the functionality is They can be di cult to maintain. Change in
not implemented. the app can make us x many test cases.
Poor repeatability.
May require good application knowledge or
testing experience.
Test design - test case
High-level test case Przypadek testowy niskiego poziomu
ID: CG001 ID: CG002
TC name: Successfully login into the system. TC name:Successfully login into the system.
Steps: Precondition: Użytkownik jest
Enter the correct login information in the zarejestrowany w aplikacji.
required elds. Steps:
Expected result: The user has been logged 1. Type "John" into text eld "Login".
into the application. 67
Test procedure
A sequence of test cases in execution order, and any associated actions that may be required to set up the initial
preconditions and any wrap up activities post execution.
(wg. ISO/IEC/IEEE 29119-2 (2013) Software and systems engineering — Software testing — Part 2: Test processes)
Test suite 69
Test Suite
• Test suite 1 (TS_1) - Manual testing of ATM operation according to test procedures: TPr_1 and 71
TPr_2, the sequence of tests performed is in accordance with the procedure description.
Test execution
RESPONSIBLE: Tester
Test execution includes the following major
activities:
Recording the IDs and versions of the test
item(s) or test object, test tool(s), and
testware,
Executing tests either manually or by using
test execution tools, 72
Analyzing lessons learned from the completed test activities to determine changes needed for
future iterations, releases, and projects,
Using the information gathered to improve test process maturity.
Test monitoring and control
RESPONSIBLE: Test manager
Test monitoring involves the on-going comparison of actual progress against planned progress
using any test monitoring metrics de ned in the test plan.
Test control involves taking actions necessary to meet the objectives of the test plan (which may be
updated over time).
Products
Test Work Products
Test work products are created as part of the test process. Just as there is signi cant variation in the
way that organizations implement the test process, there is also signi cant variation in the types of
work products created during that process, in the ways those work products are organized and
managed, and in the names used for those work products.
Test analysis
• de ned and prioritized test conditions
77
Test design
• the design high-level test cases
• Test suites
Test execution
• Documentation of the status of individual test cases or test procedures
• Defect reports
• Documentation about which test item(s), test object(s), test tools, and testware were involved in the testing
• Report via bi-directional traceability with the associated the test procedure(s)
Test completion
• test summary reports
78
• nalized testware
• summary of project management tasks, such as task completion, resource allocation and usage, and e ort.
1.4.4 Traceability between the
Test Basis and Test Work 79
Products
Traceability between the Test Basis and Test Work Products
Example of traceability.
Traceability between the Test Basis and Test Work Products
Traceability between the Test Cases and Requirements (created by JIRA tool).
82
1.5 The
Psychology of 83
Testing
1.5.1 Human
Psychology 84
and Testing
Human Psychology and Testing
Tester and project teams:/p>
Finding bugs during testing can be viewed as criticizing the product or its author,
Testing seen as a destructive action for the team, despite the vast supply of knowledge from the
point of view of the Risk Management,
Constructive communication of errors, faults and failures can help to avoid short-circuits,,
The tester and test leader need good interpersonal skills,
The tester has features that contrast with those a developer needs. Testers need the knowledge 85
end users of the system . This allows the product to be validated in the way the client does
instead of what the developer would like,
Human Psychology and Testing
Ways to communicate well include the following examples:
Start with collaboration rather than battles. Remind everyone of the common goal of better
quality systems.
Emphasize the bene ts of testing. For example, for the authors, defect information can help
them improve their work products and their skills.
Communicate test results and other ndings in a neutral, fact-focused way without criticizing
the person who created the defective item. 86
Developer’s
Mindsets
Tester’s and Developer’s Mindsets
I DO NOT MAKE BUGS ...
Whose words could these be?
Independent Tester
No independent testers; the only form of testing available is developers testing their own code,
Independent developers or testers within the development teams or the project team; this could
be developers testing their colleagues’ products, 88
Independent test team or group within the organization, reporting to project management or
executive management,
Independent testers from the business organization or user community, or with specializations in
speci c test types such as usability, security, performance, regulatory/compliance, or portability.
Independent testers external to the organization, either working on-site (in-house) or o -site
(outsourcing).
89
Fundamentals of Testing
The end 95
Go to next chapter