Software

Testing

Instructor: Syed Areeb Jafri

AGENDA

01 Seven Testing Principles

02 Overview of the Testing Process

03 Test Planning, Monitoring, and Control

Test Analysis, Design, Implementation,

04
Execution and Closure
Testing shows presence of defects

• Testing is done to find as many defects as possible

• Testing is not done to prove that the software is bug-free
• Testing helps us to identify potential undiscovered defects in the software
• Even after testing is complete with no bugs found, we can’t say that the product is 100% defect free
Example:
• A software tester is testing an android mobile application. He should be working hard to identify potential defects
rather than providing defect free product. This will help him to identify more bugs rather than achieving
correctness.
Exhaustive Testing is impossible

• Testing all combinations of input data values and scenarios are impossible
• Testing all combinations would cost more time and money
• Smarter ways of testing should be adopted in order to complete the project timelines
• Prioritize important tests based on risk
Example:
• Assume you are testing the email subscription feature of an application. There are 1000 users subscribed.
Testing email subscription is working for 1000 users consumes a lot of time and cost. Instead, identify
sample or prioritize the important test based on the requirement.
Early Testing
• Software testing starts from requirement gathering
• Testing should be done parallelly along with other product development phases.
• Finding defects early on will save a lot of money and rework time
Example:
• Consider product requirement gathering is completed and developer started coding. Once the development
is done, tester identifies a defect in the requirement gathering and found that the requirement should be
changed. This costs lots of money and time loss for the company and also finally end up in rework. Instead, if
it’s found early on that would have saved the company a lot of money and time.

Defect Clustering

• Defects are not necessarily spread even

• They may be clustered in one or few modules
• Place where one bug found, likely to find some more
• It’s more like 80-20 Pareto principle that is 20% of the defects cause 80% of the problem
Example:
• Assume a new programmer developed a complex module. So, it is very likely to find more defects in that
particular module.
Testing is context dependent
• Testing is always context dependent
• Software should be tested based on the purpose of the product that has been built
• Different software are tested differently
• Banking site cannot be tested same as the E-commerce website
• Functionally both are different, so obviously they cannot be tested same
Example:
• Banking site will have functionality features like transactions, interests, savings, etc., the E-Commerce site will have some
features like shopping cart, order, payments, etc. So both are completely different in their functionality. Test scenarios and
tests conducted on the software should also be different and depend on the context.

Pesticide Paradox
• Executing same test cases, again and again, will not help us to identify new defects
• To overcome this, it is really important to review test cases regularly and modify them if required
• Different and new test cases should be added to find more defects in the software
Example:
• Executing the functional test cases, again and again, will not help to identify defects regarding the
performance of the software or the defects regarding the security threat. Our test cases should be
different to potentially identify new defects
Absence-of-errors fallacy

• Finding and fixing defects doesn’t help if the built software fail to meet user’s requirements
• Test designed should catch most defects and also should cover all client’s requirements and specifications before shipping the
product
• Software tester completely tested the product and the software is 100% defect free. Is that mean the product is ready to be
shipped? No! Software tester should verify the test design covers the customer’s requirements and specifications. If the built
software fails to meet requirements, then finding and fixing bugs doesn’t really help.

Example:
• Zero defects do not mean the software solves end-user problems successfully. Linux always had very few bugs, while
Microsoft Windows was (is?) notorious for its bugs. However, most people used Microsoft Windows as their operating
system because they found it easier to use and solved their problems better. Linux is becoming more and more
mainstream today as it started focusing on end-user experience.
Planning and Control
Test Planning
The first phase in the test process involves creating a robust test plan. During this,
we plan for the test by creating a document highlighting the overall test approach
and its objective.

Here, creating a document is important as it serves as the guide for the complete
test process. You may question why to make a test plan before proceeding with the
test process. Well, the following is the main purpose of to test plan that makes it a
crucial step in the test process:

• To define test scope, risks, and test objectives.

• To find resources required for testing, like a test environment.
• To schedule tasks for test designing and evaluation.
The crucial activities in the test planning are as mentioned below:
• Determining the test scope and risk: The scope of testing defines what aspects of the software or system will be tested. It
helps identify the boundaries and areas to be covered. Risks associated with the project are also assessed to prioritize
testing efforts and allocate resources accordingly.
• Identifying test objectives: Test objectives define what we want to achieve through testing. They can include goals such as
verifying specific functionalities, validating performance requirements, or ensuring compatibility with different platforms.
• Identifying the test basis: Here, you can understand the test basis as all the included information and documents that
form the test process's main base or foundation. This includes requirements, design specifications, and risk analysis related
to software application development. Knowing and documenting the above will help understand the software application’s
expected behavior and testing needs.
• Test condition and test cases: You can identify the test condition by considering the test basis and its evaluation. The
specified test condition shows diverse test scenarios and circumstances for testing a software application.
Following this, test cases are created in such a way that includes all test conditions. Such test cases specify each test
scenario's inputs, actions, and expected outcomes.
• Preparing the test environment: In the test plan, designing the test environment is crucial as it specifies the need for the
hardware, software, and network configuration during test execution and ensures its availability. This process helps ensure
the team has all the required test infrastructure and tools for initiating the test process.
• Specifying completion criteria: Completion criteria define the conditions that indicate when testing, at any stage, is
considered complete. These criteria may include factors such as the number of test cases executed, the pass percentage,
or specific goals achieved. Specifying completion criteria helps track progress and determine when testing can be
concluded.
Control
Control is an essential aspect of the test process and involves monitoring and reporting the
progress of testing activities. The main tasks involved in control are:
• Progress Monitoring: Progress is monitored by comparing the actual testing activities and
results against the planned objectives and schedule. It helps identify deviations or delays and
enables corrective actions to be taken.
• Status Reporting: Regular status reports are generated to inform stakeholders about the current
testing state. The reports include details about the progress, any issues or risks encountered, and
the overall quality of the software or system under test.
• Deviation Management: If deviations from the planned objectives or schedule are identified,
appropriate actions are taken to bring the testing back on track. It may involve reallocating
resources, adjusting timelines, or reassessing priorities.
• Risk Management: Risks identified during the planning phase are continually monitored, and
mitigation strategies are implemented to address them effectively. Risk management helps
minimize potential negative impacts on the testing process and the overall project.
• Stakeholder Communication: Communication with stakeholders, including project managers,
developers, and other team members, is crucial for effective control. Regular meetings and
discussions ensure everyone is informed about the testing progress and any challenges faced.
Analysis and Design
In this phase of the testing process, the information that forms the main foundation of the test is
reviewed. Such information includes design specifications, system architecture, and software
requirements. When the review is completed, the test condition is further identified, which requires it
to be tested.
Here is a detailed explanation of the key task involved in this phase:
• As in the test plan, we identify the test basis; further review is done in this phase. It involves
studying the project requirements, understanding the design specifications, analyzing the identified
risks, and gaining insights into the overall system architecture.
• Based on the review of the test basis, the identified test conditions are then analyzed and reviewed.
It shows different situations, functionalities, or behaviors that the software or system should exhibit.
• After analyzing the test conditions, it is important to design the actual test. This task involves
creating detailed test cases that specify each test scenario's inputs, actions, and expected outcomes.
• Another crucial task of this phase is to determine the test environment setup. It involves analysis of
the required hardware, software, and network configurations to create an environment that closely
resembles the actual production environment.
Thoroughly analyzing the test basis and designing effective tests lay the foundation for comprehensive
testing. This phase identifies the key areas to focus on, ensuring a well-defined testing approach.
Further, determining the necessary test environment setup and infrastructure allows you to create an
Implementation and Execution
In this phase, the actual test of the software application occurs. It involves test implementation and test execution. In test implementation, the development and
prioritizing of test cases are done along with creating test data for such tests. In addition, test suites are used for the execution of the test process.
In test execution, the specified test is run on the computer system either through a manual or automation approach. You can log the outcomes of the tests,
comparing actual results with expected results.
Here is a detailed explanation of the key activities involved in this phase:
• Test Case Implementation
The designed test case is developed in the previous phase of the testing process. It means that required code and scripts are written to execute the test cases to test software
applications. However, there might be many test cases that define each functionality of the software application. It is very time-consuming to execute each of those. Hence, test
cases should be prioritized based on risk, end-user requirements, and criticality.

• Test Data Preparation

Preparation of test data is essential to test software applications. The test data are created or gathered to mimic the real device scenario. Thus, it must include possible scenarios
and edge cases to ensure detailed testing.

• Test Suite Creation

A test suite is a collection or grouping of related test cases. It helps organize and streamline the execution of tests. Test suites are created during this phase based on the
implemented test cases. Grouping test cases into suites allows for efficient execution, as it is possible to execute multiple test cases simultaneously, saving a lot of time and effort.
Test suites can be organized based on specific modules, functionalities, or test objectives.

• Test Execution
Tests are executed once the test cases are implemented, and the test data is prepared. Test execution involves running the specified test cases on the target system or software. It
can be done manually by following predefined steps or automated using specialized test execution tools. During test execution, you can observe and record the actual outcomes or
results of the tests.

• Outcome Comparison
After executing each test case, you can compare the actual results obtained during the test execution with the expected results specified in the test cases. This comparison helps
identify any discrepancies or deviations from the expected behavior. You can log the outcomes of each test, noting whether the test passed or failed based on the comparison. This
information is valuable for further analysis and troubleshooting.
Evaluating Test Exit Criteria and
Reporting
The evaluating exit criteria and reporting phase in the testing process are crucial for determining when to conclude the testing activities and
communicating the results to stakeholders. This phase consists of two key components: evaluating exit criteria and generating test reports.

• Evaluating Exit Criteria

Exit criteria are predefined conditions or benchmarks indicating when testing can be completed. These criteria are established based on various factors,
including code coverage, functionality coverage, risk assessment, business priorities, and project constraints such as cost and time.
When these criteria are met, the testing objectives have been achieved, and further testing may not be necessary. During the evaluating exit criteria stage,
the following tasks are typically performed.
• Assessing the Completion of Testing
The testing team evaluates whether the predetermined exit criteria have been satisfied. This assessment involves reviewing the test results, analyzing the
coverage achieved, and considering the identified risks. For example, if a certain level of code coverage or a specific number of critical functionalities have
been tested successfully, it may indicate that the exit criteria have been met.
• Modifying or Refining Exit Criteria
In some cases, it may be necessary to reassess or modify the exit criteria based on the testing outcomes or changing project requirements. If the original
exit criteria are deemed inadequate or impractical, adjustments can be made to ensure they align with the current project context. It helps ensure that the
exit criteria remain relevant and meaningful.
Exit criteria can be considered in the following cases:
• When maximum test cases are executed with a pass percentage.
• When the bug rate falls below a certain level.
• When the deadline of the software project is reached.
• Reporting
During this stage, you can assess if more tests are needed or if changes should be made to the exit criteria. It includes
writing a test summary report for stakeholders and providing an overview of the testing process and outcomes. During
this stage, the following activities are typically undertaken:

• Test summary report:

You have to prepare the test report that gives an overview of the test activities and results by summarizing the test
objective, scope of the test, test coverage, and quality of the software applications. This gives details of the identified
bugs or errors during the test and their effect on the software project.
• Test completion recommendation:
Based on the evaluation of exit criteria and the overall testing outcomes, the testing team may provide
recommendations on whether testing should be concluded or if additional testing is required. These recommendations
can guide project managers and decision-makers in determining the next steps for the software or system being tested.

The evaluating exit criteria and reporting phase helps ensure that testing activities are well-managed and decisions
regarding test completion are based on predefined criteria and project considerations.

The test reports facilitate effective communication with stakeholders, providing them with the necessary information to
assess the quality of the software or system and make informed decisions regarding its release or further testing.
Test Closure Activities
These activities are performed when the software applications are ready to be released or when testing needs to be closed for other reasons, like project
cancellation or achievement of targets. During this phase, completed planned deliverables and resolved incident reports are checked.
Following this, finalize and archive the test artifacts, such as test scripts and environments, for future reuse is done. You must also evaluate the testing process
and identify lessons learned for future projects.
The test closure activities phase marks the conclusion of the testing process and involves several important tasks. Let's explore each of these activities in detail:
• Checking Planned Deliverables and Incident Reports
During this phase, the testing team verifies whether all planned deliverables have been completed per the initial test plan. It includes ensuring that all test cases have been
executed, test environments have been set up and utilized, and any other testing-related tasks or milestones have been achieved.
Additionally, any open incident reports or defects identified during testing are reviewed to ensure they have been resolved or appropriately addressed.

• Finalizing and Archiving Test Artifacts

In the test closure activities phase, it is important to consolidate and preserve the test artifacts generated during the testing process. It includes finalizing and archiving test scripts,
test plans, test data sets, and any other relevant documentation or tools used during testing.
Organizing and storing these artifacts can be readily accessible for future reference, reusability, or audits. It ensures that valuable knowledge and resources from the testing phase
are retained for future projects.

• Evaluation of the Testing Process

In the test closure activity phase, evaluation and reflection on the entire test process are key aspects. Here, the team performs a detailed analysis of the complete efforts and
verifies factors like the effectiveness of test strategies, the efficiency of test execution, and the quality of the software application under test. It will help you identify the strengths
and weaknesses of the test process and give insight to process improvement in future software projects.

• Lessons Learned
One of the important aspects of the test closure activities phase is capturing lessons learned from the testing experience. It involves gathering feedback from the testing team,
stakeholders, and project members to identify areas of improvement, best practices, and potential pitfalls encountered during testing. Lessons learned can be documented in a
formal report or knowledge base, ensuring valuable insights gained from the testing process are utilized to enhance future testing endeavors.
Following these phases ensures a well-planned and controlled testing process, with thorough analysis, design, implementation, execution, and evaluation, leading to effective and
reliable software application delivery.