Unit 5: Testing Tools & Measurements

5.1 Introduction
1. Today, rigorous application testing is a critical part of virtually all software
development projects. As more organizations develop mission- critical systems
to support their business activities, the need is greatly increased for testing
methods that support business objectives.
2. It is necessary to ensure that these systems are reliable, built according to
specification and have the ability to spoon business processes. Many internal and
external factors are forcing organisations to ensure a High level of software
quality and reliability.

5.2 Manual Testing

1. There are two ways that software can be tested; manually by a human and
automatically by a computer.
2. Looking from the business point of view, testing is the only way of knowing
whether the newly created product delivers the intended functionality.
3. Manual testing is a testing process that is carried out manually in order to find
defects without the usage of tools or automation scripting
4. "Manual testing is the process of manually testing software for defects. It
requires a tester to play the role of an end user and use most of all features of the
application to ensure correct behaviour."

5.2.1 Advantages of Manual Testing

1. Manual testing can be done on all kinds of applications
2. It is preferable for short life cycle products
3. It is cheaper In terms of initial investment compared to automation testing
4. It requires less time and expense to begin productive manual testing
5. There is no necessity to the tester to have knowledge on Automation Tools.

Disadvantages or limitations of Manual Testing

1. Time Consuming Process: Manual testing consumes much time because most of
the test operations or test cases are repeated again and again.

2. Limited Support for Regression Testing: When one portion of software is changed
then regression testing is performed to ensure that other portions are not infected by the
recent change. In this process entire testing is repeated which takes more time.

3. Error Prone Testing: As test process is repeated several times test engineers become
bored which may results in missing some important test cases and leads to release
defective software.
4. Impractical Performance Testing: In doing performance testing of any client server
application resources like people and computers are required.

5. Non consistent: Manual testing process is not consistent as testing methods and
approaches applied by every person are not remaining same. This produces different
results on same test.

6. Limited Scope: Considering large scale projects, scope of manual testing is very
limited compare to automated testing. As the size of software increases, its complexity
also increases thus the problems in testing occurs exponentially that is why the Manual
Testing doesn't scale well. This increases the time of testing and cost of testing to.

5.2.2 Automated Testing Tools

Q. What is software test automation? State types of test automation tools.

What is automated testing?

1. As we discussed in previous section that manual testing is time consuming and

error prone method of testing which demand more manpower and infrastructure
2. All these limitations can be overcome by using automated testing method
3. Automation Testing is the process to reduce manual testing work by using of
tools and strategies that reduce involvement or interaction in unskilled,
repetitive or redundant tasks.
4. Test automation uses of special software (separate from the software being
tested) to control the execution of tests and the comparison of actual outcomes
with predicted outcomes.
5. Test automation can automate some repetitive but necessary tasks in a
formalized testing process or perform additional testing that would be difficult
to do manually.

Types of test automation tools

Two types of test automation tools are as follows

1 Static test tools

2 Dynamic test tools

1 Static Testing Tool

Q. Explain static testing tools in details.

Which are features for selecting static test tools? Also list any two available test tools (static).
What are different static and dynamic Testing tools? Explain any one tool in detail

1. Static analysis tools are generally used by developers as part of the development
and component testing process. The key aspect is that the code is not executed
or run but the tool itself is executed and the source code we are interested in is
the input data to the tool
2. These tools are mostly used by developers.
3. Static analysis tools for code can help the developers to understand the structure
of the code and can also be used to enforce coding standards

Features or characteristics of static analysis tools

1. To calculate metrics such as cyclomatic complexity or nesting levels (which can

help to identify where more testing may be needed due to increased risk).
2. To enforce coding standards
3. To analyze structures and dependencies
4. Help in code understanding.
5. To identify anomalies or defects in the code.

2 Dynamic Testing Tool

Q. Explain dynamic testing tools in details

1. Dynamic analysis tools are 'dynamic because they require the code to be in a
running state. They are 'analysis rather than testing tools because they analyze
what is happening "behind the scenes that is in the code while the software is
running (whether being executed with test cases or being used in operation),
2. Let us take an example of a car to understand it in a better way. If you go to a
showroom of a car to buy it, you might sit in the car to see if is comfortable and
see what sound the doors make this would be static analysis because the car is
not being driven
3. If you take a test drive, then you would check that how the car performs when it
is in the running mode eg the car turns right when you turn the steering wheel
clockwise or when you press the break then how the car will respond and can
also check the oil pressure or the brake fluid, this would be dynamic analysis, it
can only be done while the engine is running

Features or characteristics of dynamic analysis tools

1. To detect memory leaks,

 2. To identify pointer arithmetic errors such as null pointers
3. To identify time dependencies.

Advantages of Test Automation Tools

1. Once the software is ready for testing the functionality of the software can be
tested repeatedly to improve the quality and reliability.
2. Testing can be done unattended, for example, during nighttime and during
holidays
3. When the software has to be tested in different environments (different hardware
platform, different operating systems, using different browsers etc.), the labour
involved can be reduced.
4. Performance testing can be done without the need for many computers and many
test engineers. The test tools simulate multiple users on a single machine. As
compared to manual testing, finding out transaction response times when
multiple users access the same application will be very easy
5. Testing process can be planned and managed effectively using these tools.
6. Test reports can be generated automatically for later analysis and corrective
action
7. Hence, using automated test tools results in
i. Improvement in the quality and reliability of the software.
ii. Drastic reduction in time, effort, and money spent on testing
iii. A systematic approach to the testing process
iv. Efficient management of the testing process even if the teams are at different
geographical locations.

Benefits of Automation Testing

Q. Write any four benefits of Automation testing

List the benefits of Automation Testing

Epinin the benefits of automation.

Explain any four benefits of automation and tools

What are the benefits of automation tools and explain

Enlist any four benefits of automation testing.

Which different benefits help to recommend automated testing? Write advantages of switching
to automated

State and explain any four benefits of automation in testing

 i. Reduce time of Testing
ii. Improve the bugs finding
iii. Deliver the quality product.
iv. Allow to run tests many time with different data.
v. Save resources or requires less resources,
vi. Automation never tires

Advantages of switching to automated testing

i. Efficient Testing
ii. Consistency in testing
iii. Better Quality Software
iv. Automated testing is faster
v. Automated testing is more reliable
vi. Automated testing reduces human and technical risks
vii. Automated testing is more powerful and versatile

Approaches of Automation Test

Code-driven testing : This approach is used to test public interfaces to objects, classes,
modules or libraries are tested using different types of inputs to validate the result with
return results

2 Graphical user interface testing: This approach is used testing frameworks for
generating testing different user Interface events like keystrokes and mouse clicks and
notices the changes in that user interface. To validates the noticed change behavior of
the system.

Differentiate between Manual Testing and Automation Testing

Manual Testing Automation Testing

It is carried out manually where a tester It is carried out with the help of various
himself executes all the steps, eg. in a test automation tools QTP, Selenium etc. There
case are many available in the marie Based on
factors like requirement and budget, one
needs to choose one

Manual testing is the most basic step. Automation testing can be said to be an
Without carrying out manual testing, one integral and continuous part of manual
cannot proceed towards automation testing testing
 Time consumed is more in this case Time consumed is less as expected

Manual testing is generally carried out in a Automation testing is carried out on a

sequential manner number of machines at one time

Programming is not involved in manual Programming is the heart of automation

testing testing as scripts need to be written in
various languages like Perl, python etc

More testers are required in Manual Testing Fewer testers are required in Automation
on the grounds that in this test cases should Testing on the grounds that in this test cases
be executed manually should be executed by utilizing Automation
Tools

Manual Testing is not accurate at all times

Due to human error, hence it is less reliable Automated testing is more reliable, as it is
performed by tools and/or scripts

It is time consuming taking up human

Resources It is executed by software tools, so it is
significantly faster than manual approach

investment is required for human Resources Investment is required for testing tools.

Need for Automated Testing Tools

1) Using testing effectively

By definition, testing is a repetitive activity. The methods that we employed to carry

out testing (manual or automated) remain repetitious throughout the development life
cycle.

Automation of testing processes allows machines to complete the tedious, repetitive

work while human personnel perform other tasks.

Automation eliminates the required "think time" or "read time” necessary for the
manual interpretation of when or where to click the Mouse.

An automated test allows test to be completed many times faster than the fastest
individual. Automated test also perform load/stress testing very effectively.

2) Reducing testing costs

Computers can execute instructions many times faster and with fewer errors than
individuals
Many automated testing tools can replicate the activity of a large number of using a
single computer.

Therefore, load/stress testing using automated methods requires only a fraction of the
computer hardware that would be necessary to complete a manual test.

3) Replicating testing across different platforms

Automation allows the testing organization to perform consistent and repeatable test.

When applications need to be deployed across different hardware or software

platforms and repeated on target platforms to ensure that new platforms operate
consistently.

4) Greater application coverage

The productivity gains delivered by automated testing allow and encourage

organization to test more often and more completely.

5) Results reporting

Full-featured automated testing systems also produce convenient test reporting and
analysis.

These reports provide a standardized measure of test status and results, thus allowing
more accurate interpretation of testing outcomes. Manual methods require the user to
self-document test procedures and test results.

Advantages of Using Tools

There are many benefits that can be gained by using tools to support testing. They are:

1. Reduction of repetitive work

Repetitive work is very boring if it is done manually. People tend to make mistakes
when doing the same task over and over.

Examples of this type of repetitive work Include running regression tests, entering the
same test data again and again [can be done by a test execution tool), checking against
coding standards (which can be done by a static analysis tool or creating a specific test
database (which can be done by a test data preparation tool).

2. Greater consistency and repeatability

People have tendency to do the same task in a slightly different way even when they
think they are repeating something exactly. A tool will exactly reproduce what it did
before, so each time it is run the result is consistent

3. Objective assessment

If a person calculates a value from the software or incident reports, by mistake they
may omit something, or their own one-sided preconceived judgments or convictions
may lead them to interpret that data incorrectly. Using a towel means that subjective
preconceived notion is removed and the assessment is more repeatable and
consistently calculated.

4. Ease of access to Information about tests or testing

Information presented visually is much easier for the human mind to understand and
interpret. For example, a chart or graph is a better way to show information than a
long list of numbers - this is why charts and graphs in spreadsheets are so useful.
Special purpose tools give these features directly for the information they process.

Disadvantages of Using Tools

Although there are many benefits that can be achieved by using tools to support
testing activities, but there are also many risks that are associated with it when tool
support for testing is introduced and used which includes:

1. Unrealistic expectations from the tool:

Unrealistic expectations may be one of the greatest risks to success with tools. The
tools are just software and we all know that there are many problems associated with
any kind of software. It is very important to have clear and realistic objectives for
what the tool can do.

2. People often make mistakes by underestimating the time, cost and effort
for the initial introduction of a tool:

Introducing something new into an organization is hardly straightforward. Once you

purchase a tool, you want to have a number of people being able to use the tool in a
way that will be beneficial. There will be some technical issues to overcome, but there
will also be resistance from other people - both need to be handled in such a way that
the tool will be of great success.

3. People frequently miscalculate the time and effort needed to achieve

significant and continuing benefits from the tool:
Mostly in the initial phase when the tool is new to the people, they miscalculate the
time and effort needed to achieve significant and continuing benefits from the tool.
Using a testing tool for the first time will not be your best use of the tool either. It
takes time to develop ways of using the tool in order to achieve what is expected

4. Mostly people underestimate the effort required to maintain the test

assets generated by the tool:

Generally people underestimate the effort required to maintain the test assets
generated by the tool. Because of the insufficient planning for maintenance of the
assets that the tool produces there are chances that the tool might end up as 'shelf
ware', along with the previously listed risks.

5. People depend on the tool a lot (over-reliance on the tool)

Since there are many benefits that can be gained by using tools to support testing like
reduction of repetitive work, greater consistency and repeatability, etc. people started
to depend on the tool a lot. But the tools are just a software they can do only what they
have been designed to do (at least a good quality tool can, but they cannot do
everything. A tool can definitely help, but it cannot replace the intelligence needed to
know how best to use it, and how to evaluate current and future uses of the tool.

5.4 Selecting a Testing Tool

While introducing the tool in the organization it must match a need within the
organization and solve that need in a way that is both effective and efficient.

The tool should help in building the strengths of the organization and should also
address its weaknesses. The organization needs to be ready for the changes that will
come along with the new tool.

The following factors are Important during tool selection:

1. Assessment of the organization's maturity (e.g. readiness for change).

2. Identification of the areas within the organization where tool support will help
to improve testing processes
3. Evaluation of tools against clear requirements and objective criteria:
4. Proof-of-concept to see whether the product works as desired and meets the
requirements and objectives defined for it:
5. Evaluation of the vendor (training, support and other commercial aspects) or
open-source network of support:
6. Identifying and planning Internal implementation (including coaching and
mentoring for those new to the use of the tool)
Automation testing success largely depends on the selection of right testing tools. It
takes lot of time to evaluate relevant automation tools available in the market But this
is a must one time exercise that will benefit your project in long run.

Here are the criteria you need to consider before selecting any testing tool

1. Do you have necessary skilled resource to allocate for automation tasks?

2. What is your budget?
3. Does the tool satisfy your testing needs? is it suitable for the project
environment and technology you are using? Does it support all tools and
objects used in the code? Sometime you may get stuck for small tests due to
inabilities of the tool to identify the objects used in the application.
4. Does the tool provide you the free trial version so that you can evaluate it
before making a decision? Also does the tool have all features available in trial
version?
5. Is the current tool version stable? is the vendor company established with good
customer support as well as online help resources and user manual?
6. Does the tool support easy interface to create and maintain test scripts? Record
and playback tool with abilities to edit recorded scripts could be a good
solution.
7. Does It provide simple interface yet powerful features to accomplish complex
tasks?
8. Does it integrate well with your other testing tools like project planning and
test management tools?
9. Tool vendor refund policy
10. . Existing customer reviews for the tool
11. Is the vendor providing initial training?

5.5 Automated Test Tools

5.5.1 When to use Automated Test Tools

Which types of test are the first candidates for test automation ? Why?

Adtomated tests are a very useful and impressive tool used ta make testing more
efficient. However, automated tests are not suited to all projects - this may be due to
lack of time available, or due to technical limitations Automated tests take time to
create. Depending on the testers, it can take 3-10 times the amount of time to create an
automated test as it takes to run the same test manually. Therefore, automated tests
will only start to be valuable when they are run more than 3-10 times.

Automated tests are suitable for the following purposen:

Regression testing for a stable system that will be run ana regular basis

Fast data creation in test systems where the database must be wiped on a regular bass

So regression testing is the first candidates for test automation

Automated tests are NOT suitable for the following purposes:

Testing new functionality this should be done manually before automated tests are
created,

Regression testing systems that are expected to have significant user interface
changes. Large changes to the user interface require a lot of maintenance for
automated tests When automating tests, it is wise to only automate as many tests as
your team can easily maintain. If some tests are becoming difficult to maintain, it may
be worth considering retiring those tests,

Above all, remember that automated tests will never detect as many bugs as a human
tester executing the same steps This is because the human tester's eyes can pick up
many things, whereas the test will only notice what it is programmed to notice.

Testing Using Automated Tools (Test Automation)

Software Test automation makes use of specialized tools to control the execution of
tests and compares the actual results against the expected result. Usually regression
tests, which are repetitive actions are automated.
Testing Tools not only help us to perform regression tests but also helps us to
automate data set up generation

product installation, GUI Interaction, defect logging etc.

Criteria for tool selection

For automating any application, the following parameters should be considered,

Data driven capabilities

Debugging and logging capabilities

Platform independence

Extensibility and Customizability

E-mail Notifications

Version control friendly

Support unattended test runs

Types of frameworks
Typically, there are 4 test automation frameworks that are adopted while automating
the applications

Data Driven Automation Framework

Keyword Driven Automation Framework

Modular Automation Framework

Hybrid Automation Framework

Popular Tools that are used for Functional automation:

Table 5.5.1: Functional Automation Tools

Product

Quick Test Professional

Rational Robot

Coded UI

Selenium

Auto IT
Popular Tools that are used for Non-Functional automation

Vendor

HP

IBM

Microsoft

Open Source

Open Source

Table 5.5.2: Non-Functional Automation Tools

Product

Load Runner

Imeter

Burp Suite

Acunetix
Vendor

HP

Apache

PortSwigger

Acunetix

Metrics and Measurement of Software Testing

Testing Tools and Measurements

Q. Define metrico and measurements

Q. Detine software metrics

What do you mean by Software Matrica ?

Software Testing Metric is defined as a quantitative measure that helps to estimate the
progress, quality, and health of a software testing effort.

A Metric defines in quantitative terms the degree to which a system, system

component, or process possesses a given attribute

Suppose, in general, "Kilogram" is a metric for measuring the attribute Weight

Similarly, In software, "How many Issues are found in thousand lines of code?", here
No. of issues is one measurement and No. of lines of code is another measurement.
Metric is defined from these two measurements

Test metrics example: How many defects exist within the module? How many test
cases are executed per person?

What is the Test coverage ?

- Measurement is the quantitative indication of extent, amount, dimension, capacity,

or size of some attribute of a product or process
Test measurement example: Total number of defects

Software metric and measure offer moet into the effectiveness of the software process
and the projects that are conducted using the process as a framework

The goal in to determine whether quality and productivity improvements have

occurred

F 5.61 shows metrics and Measurement

Mese

Seconde

Meme

Kmh

Fig. 5.6.1: General metrics and measurement

Need of Metrics and Measurements

mis

Q. Explain need of software measurement

Q. Give any two advantages and any two limitations of Function Test matrix
Gain an understanding of processes, products. resources, and environments

Establish baselines for comparisons with future assessments.

To evaluate in order to : Determine status with respect to plans

To predict in order to Gain understanding of relationships among processes and

products.

Build models of these relationships

To improve in order to identify roadblocks, roat causes, inefficiencies, and other

opportunities for improving product

quality and process performance.

Take the decision for next phase of activities such as estimate the cost and schedule of
future projects.

Understand the kind of improvement required to success the project

Take a decision on process or technology to be modified etc.

Importance of metrics

"We cannot improve what we cannot measure. We cannot control what we cannot
measure
Metrics is used to improve the quality and productivity of products and services thus
achieving Customer Satisfaction

Easy for management to digest one number and drill down, if required.

Different Metrics) trend act an meinitor when the process is going out-of-control.

Metrics provides improvement for current process

Take decision for next phase of activities

- Evidence of the claim or prediction,

Understand the type of improvement required.

Take decision on process or technology change.

Metrics Lifecyole

The process involved in setting up the metrics shown in Fig 5.6.2,

Analysis

Idently matic(s) to use.

Define matics) Identified Dolina paramotor(s) for evaluating the matric(s) identified.
Explain the need of metric lo stakeholder and the testing team. Educate testing team
about the data points need to De caplured lor processing the metric

Capture the data

Verily the data - Calculating the metric(s) value using the data captured

• Develope the report with effective conclusion Distribution report to the stakeholder
and their representative Taka to back trom the stak holder

Fig. 5.6.2: Metrics Lifecycle

Advantages and disadvantages of Test Matrix

Advantages

1.

it increases the function traceability

2 It helps to improve the serialization of processes

3.

it is able to identify/locate different tests involved in processes

Disadvantages or Limitations

1 it becomes time consuming in many of the cases where requirements are more.

2. Poor or unknown test coverage, more defects found in production

3 it will lead to miss some bugs in earlier test cycles which may arise in later test
cycles. Then a lot of discussion arguments with other teams and managers before
release

3 Difficult project planning and tracking, misunderstandings between different

teams over project dependencie delays, etc.

Types of Metrics

State the different matric types with its lasification

List any three of metrics

Software metrics can be classified Into three categories

1. Process metrics

These characteristics can be used to improve the development and maintenance

activities of the software
The process metrics are those metrics which are concerned with Soltware
Development Life Cycle (SDLC). They can be used to improve the process
efficiency of the SDLC.

For example, the process metric 'Defect Rate- the amount of defects reaching the
customer can be used to improve the efficiency of the Development and the
Testing team if Defect rate is less, we know that the efficiency of the Development
and the Testing team is improving

2 Product metrics

Describes the characteristics of the product such as size, complexity, design

features, performance, and quality

level.

The product metrics are those metrics which has more meaning in the perspective
of the software product being developed. One of the example is, quality of the
developed product. Considering the fact that quality is a subjective attribute, one
needs a very good understanding of the measures based on which the quality
metrics are computed

3. Project metrics

This metrics describe the project characteristics and execution Examples include
the number of software developers, the staffing pattern over the life cycle of the
software, cost, schedule, and productivity

The project metrics are those metrics which are more relevant to a project team.
They can be used to measur
the efficiency of a project team or any other tools being used by the team
members. One example of a projec metric is 'Errors found per engineer month'
which was mentioned before. This is a relevant project metric for

test team

Project metrics enable a software project manager to:

Assess the status of an ongoing project

Track potential risks

Uncover problem areas before their status becomes critical

Adjust work flow or tasks.

Evaluate the project team's ability to control quality of software work products:

Many of the same metrics are used in both the process and project domain. Project
metrics are used for mak/ tactical decisions. They are used to adapt project
workflow and technical activities

Use of project metrics

- - The Rest aNication ef project metri curs during estimation

Metrics from past projects are used as a basis for estimating time and effort
As a project proceed the amount of time and effort expended are compared to
original estimates

As technical work commences other project metrics become important

Production rates are measured (represented in terms of models created, review

hours, function points and delivered

source lines of code).

Error uncovered during each generic framework activity lle, communication,

planning modeling, construction,

deployment) are measured.

Project metrics are used to

Minimize the development schedule by making the adjustments necessary to avoid

delays and mitigate potential

problems and risks

Assess product quality on an ongoing basis and when necessary modify the
technical approach to Improve quality

in summary
As quality improves, defects are minimized.

As defects go down, the amount of rework required during the project is also
reduced.

As rework goes down, the overall project cost is reduced

5.6.3

Categories of Testing Metrics

Describe product Vs procesa metrics,

There are two broad categories of software testing metrics, namely

Product metrics

Process metrics

Product metrics

Product metrics describe the characteristics of the product such as size,

complexity, design features, performance,

efficiency, reliability, portability, etc

They focus on the quality of deliverables.

1. Product metrics are combined across several projects to produce process

metrics.

2. Product metrics help software engineers to better understand the attributes of

models and assess the quality of

the software,

4. They help software engineers to gain insight into the design and construction of
the software

MSBTE Questions and Answers

How to select a testing Tools? Explain in details

Enlist any four benefits of automation testing

Define metrics and measurements Explain need of software measurement (Section

Explain static and dynamic testing tools in details (Section 5.2.2.1)

Enlist factors considered for selecting a testing tool for test automation (Section

Which are features for selecting static test tools ? Also list any two available test
tools (static),
State Limitations of manual testing. Write any tour. (Section 5.2.1)

Which different benefits help to recommend automated testing ? Write advantages

of switching to automated testing
What is software test automation State types of test automation tools. (Section
State the different metrics types with its classification. (Section 5.6.2)
Whwh types od test are the first anti ty test n y etion 55.
Estimate the ages nyt) of ughe t nation too ention 53.
What are the advantages and disadvantages of g materd wrangLer ey w
List what are the different guidelines to be followed while selecting dynamic test
tools (Section S4 Explain three types of product metrics (Section 5.6.3.1)

Debe any four limitations of manni testing, (Section 5.2.1)

State and explain any four benefits of automation in testing (Section 5.23)

What is the difference between static and dynamic testing tool (Section 5.2.2.1)
State and explain any four benefits of automation in testing (Section 5.2.3)

Define software metrics. Describe product Vs process metrics (Sections 5.6 and
5.6.3)
How to select a testing tool ? Explain in detail (Section 5.4)

Define Metrics and Measurements. Explain need of Software measurement

(Section 5.5) Write down any four limitations of manual testing (Section 5.2.1)

What is automated testing ? Write down advantages of using automated testing

tools in satwa (Sections 5.2.2 and 5.3.1)

Ο Describe how to select testing tool (Section 5.4)

Give any four difference between Manual and Automated testing. (Section 5.2.6)

Ο Give any two advantages and any two disadvantages of using testing tools.
(Sections 5.3.1 and 5.3.2) What do you mean by Software Metrics ? List any three
types of metrics (Sections 5.6 and 5.6.2)
Q. State any eight limitations of manual testing. (Section 5.2.1)

What are different static and dynamic testing tools ? Explain any one tool in detail.
(Section 5.2.2.1)

Enlist factors considered for selecting a testing tool for test automation. (Section
5.4) Define the term metrics and measurements Explain need of software
measurement