World

Quality
Report In association with:
 Regional and country reports are available
online from www.worldqualityreport.com
or from Capgemini, Sogeti, and Micro Focus
local offices covering: North America, United
Kingdom, France, Germany, Benelux, Southern
Europe (covering Italy, Spain, and Portugal)
Eastern Europe (covering Poland, Romania,
Hungary, and the Czech Republic) China,
Australia and New Zealand, Japan, and the
Nordic region.

World
Quality
Report
2020-21
Twelfth Edition 2

2 World Quality Report I 2020-21

Contents

Introduction 04

Executive Summary 06

Current Trends in Quality Assurance & Testing 14

QA orchestration in agile & DevOps 16

Artificial intelligence and 21
machine learning

Test automation 27

Test data management and 33

test environment management

Budgets and cost containment 39

The impact of COVID-19 and its implications on 45

quality assurance activities in a post-pandemic world

Sector Analysis 50
Automotive 52
Consumer products, retail, and distribution 54
Energy, utilities, and chemicals 56
Financial services 58
Healthcare and life sciences 60
High-tech 62
Government and public sector 64
Telecoms, media, and entertainment 66

About the study 68

About the sponsors 72

3
 Introduction
Welcome to the 12th edition of the World Quality Report. It’s
a source of great pride to us here at Capgemini to know that
its publication has for many people become a calendar event –
something they look forward to receiving, and to reading.
Our purpose in these pages is not only to assess the current climate of
global quality assurance (QA), but also to provide you with best practices
and expert recommendations on how you and your teams can keep on
improving quality and test activities. This year’s survey reconfirms that
quality of IT remains a critical success factor. The most important aspect
of this is the degree to which organizations remain committed to the very
principle of quality. That commitment is, if anything, higher than ever.
We also see continuity in specific trends. For instance, year on year, there
is a general shift towards agile and DevOps development environments;
the use of test automation is growing, as for most organizations it is still
not at the level required; so, too, is interest in the use of smart tools and
techniques; and there is steady progress towards embedding QA practices
in every part of the software development lifecycle. It is reassuring that
69% of the organizations interviewed in this survey feel they always or
virtually always meet their quality goals.
The other constant is speed. Achieving not just quality, but value, at speed
Mark Buenen remains the focus of quality transformation, and that’s why we’re seeing
growth in test automation, and in shift-left, and more. Speeding up the
Global Leader
development cycles and achieving the right levels of quality is not easy.
Digital Assurance and Quality
One of the main challenges noted this year is the lack of the right testing
Engineering, Capgemini Group
methodologies for teams, as reported by 55% of respondents.
This report provides various recommendations and best practices for how
organizations can overcome these challenges. In addition, I would like
to point out that at Capgemini and Sogeti, we published earlier this year
“Quality for DevOps Teams” as a complete guide for agile and DevOps
teams on how to achieve quality in their endeavors.
For more information, please visit www.tmap.net and click on the ‘Our
books’ tab.
Not everything is a constant, though. There’s one major thing in this report
that is completely new. I refer, of course, to COVID-19, and its effects not
just on the disciplines covered in our analysis, but also on business as a
whole and indeed, on the entire world. In this survey, we also analyzed how
the new reality is likely to speed up the further transformation of quality
and test activities.
Finally, I thank the many people who contributed to our report this year.
In addition to the analysis of our own experts and of those from our
partner Micro Focus, we are grateful to the many senior figures from client
organizations who provided insights and best practices. We thank, too,
all the 1,750 people worldwide whose responses have helped shape our
understanding of the worldwide trends in quality and testing.
I hope that this 12th World Quality Report is a source of inspiration for you
to further enhance the quality and test activities within and across all your
teams and programs.

4 World Quality Report I 2020-21

 Introduction

This past year has seen unprecedented challenges across every

industry and in every country. The COVID-19 pandemic has severely
impacted QA and IT strategy, forcing teams to become more agile,
to do even more with even less, and to adjust to new ways of working with
their colleagues. However, as we noted in previous editions of the World
Quality Report, organizations have been steadily improving and accelerating
their ability to deliver value with high quality and security, and this
momentum has equipped them to deal with the changes we are all facing.
This year’s World Quality Report shows that QA has transitioned from
being an independent function in a separate team, towards becoming
an integrated part of the software delivery team, with responsibilities
reaching beyond testing and finding defects. QA engineers are now charged
with enabling the entire team to achieve their quality objectives, and
incorporating better engineering practices and state-of-the-art techniques
such as Artificial Intelligence and Machine Learning to achieve these aims.
The World Quality Report notes that organizations that had already begun
their digital transformation prior to the onset of the pandemic were in a
better position to adjust than those still working with legacy practices,
who have been prompted to overhaul their approach to software delivery
in general, and to QA and automation in particular. Much more than an
account of respondents’ answers to the questions in the survey, the World
Raffi Margaliot Quality Report includes a wealth of key action items across all aspects of
Senior Vice President and General QA, enabling the reader to identify areas in their organization that could be
Manager, Application Delivery improved, and make the changes necessary to become even more effective.
Management
With AI and ML now an integral part of our continuous testing and quality
Micro Focus
management tools, Micro Focus helps customers to advance their software
modernization initiatives and meet demand for increased test coverage,
better performance, and high security across an unparalleled range of web
and mobile devices and environments. We enable organizations to make the
cultural shift demanded by today’s reality, helping them to succeed in the
face of the global mandate to be even more productive with less budget
and fewer resources.
Finally, I’d like to convey my appreciation and thanks to our friends and
partners at Capgemini and Sogeti, and everyone who has worked tirelessly
to produce this edition of the World Quality Report.

5
Executive Growing expectations
from QA.
Summary Can we meet them?
WORLD QUALITY
REPORT 2020-21
Yes. We
can.

This has been a year of steady and encouraging progress.

Sathish Natarajan Acceleration in QA transformation is visible, and we think that
Group Vice President also contributed to a great extent in the resilience shown by
Head of Digital Assurance the QA and information technology (IT) teams in the face of
and Quality Engineering the COVID crisis.
Capgemini North America For instance, and as you’ll see in more detail in the rest of
this executive summary, quality is now being assured by
following better engineering practices, yet the testing
mindset remains as relevant as before. The adoption of
agile and DevOps continues to grow. The uptake of artificial
intelligence (AI) and machine learning (ML) in QA has been
growing gradually, with some of the use cases maturing and
Dhiraj Sinha others getting evaluated. Test automation is moving forward
Vice President too, and is becoming more intelligent and comprehensive.
Financial Services, In some cases, the pace of change has been hampered by
Digital Assurance and Quality legacy practices, budget constraints, and skills gaps, but
Engineering, APAC Region nonetheless, the momentum has been encouraging.
Expectations of QA have been steadily increasing. There’s
an upward trend in almost every case for the objectives for
which we have previous data. These include the need to
support business growth and the importance of ensuring
end-user satisfaction. Both of these indicate that QA is no
longer seen as a backroom discipline, divorced from the rest
of the organization. If you hear fewer soundbites about QA
and more about digital transformation, it’s not because QA

6 World Quality Report I 2020-21

 Executive Summary

has ceased to be relevant, but because the contribution of independent function to an integrated function, and now to
QA is implicit in the success of the digital transformation. an inclusive function. Also, the role of the QA practitioner is
transforming from testing and finding defects, to ensuring
The graph below (see Fig 01) shows responses to new criteria
that other engineering team members inculcate quality in
that were introduced this year. Time to market and test
their way of working. They need to do this by enabling them
automation are almost as significant as other objectives,
and by removing any impediments on their way to achieving
and it will be interesting to see how they change over the
quality objectives. QA is not only shifting left but also
next year.
moving right. We see more and more enterprises talk about
However, it’s the bottom bar that’s likely to prove the most exploratory testing, chaos engineering, and ensuring that the
intriguing. It won’t be an exaggeration if we say that out of product is experienced the way end users will experience it in
all the software disciplines, QA has witnessed the most rapid real life before releasing it to the market.
transformation. QA has been steadily evolving – from an

Q1. On a scale of 1 – 7 (where 1 = Not at all important at all and 7 = Essential) how important are each of
Fig 1
Fig 01 Objectives
the of objectives
following Quality Assurance
when it and Testing
comes in the and
to Testing organization
Quality Assurance?

Top 2 box summary: 7 Essential + 6 2020 2019 2018

74
Business Assurance: Contribute to
74
business growth and business outcomes
68

72
Custodian of Quality: Detect software
72
defects before go-live
70

70
Digital Happiness: Ensure end-user 68
satisfaction and customer experience
69

64
Brand Ambassador: Protect the
65
corporate image and branding
64

63
Quality at Speed: Speed up software
releases with good quality

62
Automate: Make QA and Testing a
smarter automated process

60
Quality Enablement: Support everybody
in the team to achieve higher quality

Single code per option 0 10 20 30 40 50 60 70

7
 Q4. On a scale of 1 to 7 (1 – never achieve to 7 – always achieve) how often do your teams achieve the following
Fig X
Fig 02 targets when
How often do testing
your QAkey applications.
teams achieve the following targets when testing key applications?

Top 2 box summary: 7 Always Achieve + 6 2020

We meet our quality goals 69

We have the right QA and Test expertise 65

We have the right level of test automation

58

Enough time is available for testing 58

We have the required in-house

testing environments 54

We have the right testing strategy/ process/

methodology
45

0 10 20 30 40 50 60 70 80

An entirely new question for this year gives further grounds So far, then, so promising. Except, of course, that this year
for optimism (see Fig 02). More than two-thirds (69%) of our brought an entirely new factor into the QA picture – a factor
respondents felt they always meet their quality goals, and that no one in this field predicted.
almost as many (65%) felt they had the right expertise.
That, of course, was the arrival of COVID-19. The coronavirus
Figures for test automation and for the time available for pandemic has been a major challenge, but as we shall see,
testing were also quite high, although we know, first, that both in this executive summary and elsewhere in the report,
automation is still in a growth phase, and second, that testing it has also acted as an accelerant, encouraging organizations
teams in the field often feel under more time pressure than and teams to do more things – and to do them better.
is indicated here. Next year, we’ll perhaps be able to see if
there is a reality check, or whether there is indeed consistent
confidence about the extent to which targets are being met.

8 World Quality Report I 2020-21

 Executive Summary

Key Findings
World Quality Report 2020-21

QA orchestration in agile Nonetheless, enthusiasm hasn’t diminished: organizations

are putting AI high among their selection criteria for new
and DevOps QA solutions and tools, and almost 90% of respondents to
this year’s survey said AI now formed the biggest growth
The adoption of agile and DevOps is steadily increasing, area in their test activities. It seems pretty clear they feel
resulting in QA teams becoming orchestrators of quality. The smart technologies will increase cost-efficiency, reduce the
risk and complexity associated with software development need for manual testing, shorten time to market – and, most
has gone up significantly, creating even more pressure to importantly of all, help to create and sustain a virtuous circle
assure high quality. At the same time, the need to do it faster of continuous quality improvements.
and cheaper is more important than ever. So, QA teams
need to use the right tool sets and technologies, the right
skilled professionals, and the fit-for-purpose processes to
get it right. We see a trend towards wanting QA engineers Budgets and cost containment
who have developer type skills, who yet retain their quality
mindset and business-cum-user centricity. Quality assurance budgets have always been challenging, but
the pressures have increased this year. This is partly because
Is this expecting too much? Yes, we think so. Only a few QA of the economic impact of the COVID-19 pandemic, but it’s
professionals can have all these skills in their repertoire. also because digital transformation has increased the online
That’s why organizations are experimenting with the QA real estate that QA teams have to manage.
operational structure, with the way QA teams work, and with
the skill acquisition and training of QA professionals. As a result, organizations are being forced to innovate and
become more efficient. The main focus remains manpower
Quality assurance is no longer a metric that is only of cost reduction, but some organizations are also looking
technical interest. It absolutely has to ensure that software at deriving the best out of their tool investments, moving
works for employees, suppliers, and customers, and this test environments to the cloud, and looking for efficiencies
year’s report demonstrates the continuing growth in from technologies including AI, machine learning, and test
acceptance of this truth. automation.
The other aspect is increasing difficulty in accounting
for the QA budget as it becomes an integral part of the
Artificial intelligence and machine end-to-end lifecycle. However, if the net outcomes are a
universally shared commitment to quality and an efficient QA
learning organization, that will be a great thing.
As last year, so this year. Expectations of the benefits that
AI and ML can bring to quality assurance remain high, but
while adoption is on the increase, and some organizations Test automation
are blazing trails, there are few signs of significant general
progress. Partly, this is because relevant skill sets still aren’t Automation has become core to QA transformation.
in place; and partly, the arrival of the COVID-19 pandemic has The good thing we saw this year is that more and more
disrupted schedules, budgets, and plans. practitioners are talking about in-sprint automation, about

9
automation in all parts of QA lifecycle and not just in
execution, and also about doing it smartly.
Impact of COVID-19
While the momentum could have been higher, the The effect of the pandemic on testing and quality assurance
automation scores have mostly risen since last year. Also, the has varied by sector, and those with legacy working
capability of automation tools being used seems to satisfy practices have been particularly badly hit. Circumstances
many organizations, but the signs are that benefits aren’t have also forced the pace of change, with faster adoption
being fully realized: only around a third of respondents (37%) of distributed agile, more widespread use of collaboration
felt they were currently getting a return on their investment. tools, greater use of data masking, and more real-time status
It really depends on how that return is being measured and monitoring across the whole QA function.
communicated to the relevant stakeholders. Another factor At a macro level, the pressures of COVID-19 have
may be that the tools are getting smarter, but the teams are accelerated digital transformation programs, with all the QA
not yet sufficiently skilled to take full advantage of them. consequences this implies. For instance, the growth of online
Overall, however, we are happy to note the progress. business has increased the demand for security testing.
Organizations are working their way forwards, in spite of In general, it’s heartening to note that, while the challenges
the increasing complexity of applications and the business have been many, there are signs that organizations are taking
processes that they support – and they are achieving greater advantage of the upheaval to improve their entire approach
control and transparency of their QA activities. to QA, with greater shared commitment, and greater
determination to succeed.

Test environment management

(TEM) and test data management
(TDM)
Approaches to TEM and to TDM are still evolving, but
there is reason for optimism. In our survey responses, we
observed an increase in overall satisfaction with TEM and
TDM outcomes, though we think that on the ground these
remain challenging. More organizations are adopting cloud
and cloud-based technologies for managing their TEM and
TDM needs. They are also implementing tools for test data
management, though most still use them only for data
masking purposes. Service and data virtualization techniques
are also becoming mainstream, and are being used for TEM
and TDM. We also witnessed more organizations moving to
create a specialized shared services organization for TEM and
TDM.
It was also interesting to note that process and governance
came out as a bigger challenge than technology in this area.

10 World Quality Report I 2020-21

 Executive Summary

Key Recommendations
World Quality Report 2020-21

QA orchestration in agile Keep learning.

There’s always something new to learn. To get the most out
and DevOps of AI in quality assurance, teams need to strengthen their
knowledge and experience of the tools, of overall QA and IT
Don’t silo responsibility
strategy, and of the commercial objectives of the business as
for QA. Share it.
a whole.
Quality practices should be integrated into every part of the
Have a toolkit.
lifecycle – and should be part of everyone’s mindset.
Encourage teams to explore the continuous adoption
Spread the word. of toolsets involving AI functionality. Invest in the cloud
Software quality isn’t just an IT thing, either. People solutions that can provide the needed GPUs for teams to fail
should spread the word, because the rest of the business fast and learn.
needs to understand that if it helps IT to get things right, It’s also worth considering building portfolios of common
everyone wins. toolsets and data libraries. This makes the development of
AI-based test strategies much easier to pursue.
Be part of the business.
IT can also help itself by getting closer to the business, Testing AI systems: have a strategy.
learning its skills, and taking time to understand Testing AI itself is a particular challenge – and it’s a moving
its objectives. target. Learning from best practice – from the automotive
sector, for instance – is a good start, especially when we are
Make room for dashboards. going to continue witnessing new innovations and strategies.
Quality dashboards make everything more visible
and straightforward.
Budgets and cost containment
Listen more to users.
Greater savings can be achieved by using test
“Is this working for you? Tell us what you think.” Asking
infrastructure smartly.
and listening will make things better, and will make
customers happier. Maturity and the adoption of cloud-based technologies
provide an excellent opportunity for organizations to
optimize their test environment and test tool licensing costs.
Artificial intelligence and machine Use advances in analytics, AI, and machine learning
learning to make testing smarter.
Organizations should have a solid plan in place to improve
Focus on what matters.
the efficiency and effectiveness of QA using advances in
If you identify and prioritize use cases, you’ll be able to analytics, AI, and machine learning.
increase the overall capacity of your QA function, and enable
your teams to concentrate their efforts. The area of greatest Similarly, organizations should plan their investment in test
need is likely to deliver the greatest benefit. automation up-front, so as to realize future cost savings in
addition to the time-to-market benefits.

11
Be prepared to pay well for smarter talent. Get as much value as you can out of your
Hiring people on price rather than on quality is a false tool investment.
economy. Talented people are more likely to have broader Teams should make every effort to plan and implement the
development and testing skill sets, and greater business various features that their TDM and TEM tools provide. They
aptitude – so they’re likely to help increase efficiency and should also train their team members and measure and
boost business, too. report regularly on the benefits.
Don’t put all key initiatives on hold. Strive to be more Have strong governance in place.
efficient instead. It’s essential to have a mature approach to the governance
Aim to identify QA initiatives with the greatest potential, and of managing the complete lifecycle, from raising the demand
pool your best thinking to find the smartest, shortest routes to de-commissioning and archiving the test environment and
to success. If that means thinking outside the box – and test data. QA teams need to work jointly with the TEM/TDM
outside the team – so be it. center of excellence to forecast demand and achieve visibility
of what is available.

Test automation
Getting ready to succeed in a
Change the status quo.
Testing will always be squeezed in the software development
post-COVID world
lifecycle. Introducing more automation – and pursuing it Be better prepared for business continuity.
vigorously – is the only answer.
Run disaster planning sessions more often, factoring in
Think ahead. not just workforce issues but systems availability. Test the
Choosing tools that will keep pace with dynamic changes to application architecture, and ensure it will be able to adapt in
applications is very important. line with any changes to business models.

Choose the right framework. Focus more on security.

Opt for design automation frameworks that are intuitive, More business is online, and more people are working
dynamic, have self-spinning environments and algorithm- remotely – which means there is greater vulnerability.
based prioritized scoping, and that can self-provision Prevention is better than cure, but make sure you also have a
test data. good Plan B for any breaches.

Balance automation against skills needs. Don’t look at COVID-19 as a way to cut costs, but as
an opportunity to transform.
Organizations need either to lessen dependency on skills or
to make automation inclusive. Cost-cutting in a crisis may only bring short-term gains.
Investing now in digital transformation and in changes to
Don’t think one size fits all. the business model will deliver greater benefits, and over
A multi-tool approach, instead of one-tool-for-everything, a longer timeframe. It’s also a good idea to ensure the QA
can be beneficial if applied correctly. strategy stays in step with these developments.

Get smart. Continue to use the best practices adopted during

the pandemic.
Take the heat off: let AI and machine learning tackle the
technical challenges in automation. The old ways aren’t always the best. If you’ve been doing
things differently in 2020, keep what works. For example,
ensure applications remain responsive to business changes;
Test environment management keep cutting back on unnecessary physical infrastructure and
travel needs; and sustain a new, more collaborative, and more
(TEM) and test data management remote way of working, in ways that suit your circumstances.
(TDM)
Create a shared center of excellence for TEM/TDM.
A shared center of excellence (CoE) should be created
to service the TEM and TDM requirements within an
organization. This team should create a strategic framework
which each of the project teams can use.

12 World Quality Report I 2020-21

 Executive Summary

Energy,
enthusiasm,
and
determination

As we’ve seen, COVID-19 has created significant encouraging to see how continuous testing is becoming
problems for business, but it has also sharpened minds mainstream and AI use cases for QA are maturing.
and accelerated the pace of change – from specific Indeed, there are signs that QA will, in time, become
areas of QA such as the use of crowd testing, to the a specialized quality-enabling function. This will be a
faster implementation of enterprise-wide digital good thing.
transformation initiatives.
The second one is related to the first. It’s the move
These pressures are mostly being greeted with towards multi-skilled team members – people who
optimism and enthusiasm. For example, organizations bring a mix of development, data, testing, and business
are finding out smarter ways to tackle the cost skills. Without these talented individuals, end-to-end
efficiency issues this year. They are more open to quality assurance will be much slower to arrive.
optimize tests by applying AI in QA, improving usage of
The final factor relates to the mindset. Partly because
test automation and test tools and maturing the test
of the pandemic, and partly because the gradual
environment and test data management practices.
improvements of recent years are beginning to bear
Several factors are increasing the likelihood of success, fruit, we sense a greater willingness to succeed. It’s a
and you’ll find plenty of detail in these pages. For now, sense that many people share, both within and across
though, we’ll highlight three of them. teams, and across disciplines, geographies, and sectors.
The first is that the trend to embed quality into every Can QA teams meet the growing expectations?
stage of the development process is continuing. It’s
Yes. We can.

13
 Current Trends in
Quality Assurance
& Testing

14 World Quality Report I 2020-21

QA orchestration in agile & DevOps 16

Artificial intelligence and machine learning 21

Test automation 27

Test data management and 33

test environment management

Budgets and cost containment 39

The impact of COVID-19 and 45
its implications on quality assurance
activities in a post-pandemic world

15
QA orchestration Régis Curien
Head of Digital Assurance and
Quality Engineering, Sogeti France

in agile & DevOps Albert Tort

Chief Technology Officer, Sogeti Spain
Tal Levi Joseph
A steady evolution – and what’s Head of Product Management
needed to accelerate the pace Application Lifecycle Management Portfolio
Micro Focus
Maheshwar Kanitkar
Vice President, Global Sales Leader
for Digital Assurance & Quality Engineering
Europe

In general, it seems there are no major changes either of pace

Narrowing The gap between or of direction this year in the adoption of agile and DevOps
the skills gap ‘black box’ testers in quality assurance (QA). It’s a case of steady evolution,
rather than of major revolution, as organizations integrate
and developers is these approaches into their testing practices. Our experience
narrowing. We’re finding people is that many of those who aren’t yet moving in this direction
are at least declaring their intention to do so.
in coding are growing more comfortable
It’s worth reminding ourselves of what’s prompting
with testing, and vice versa. In fact, over this steady transition. The world of business in general,
the last five years, I’d say that testers are and of technology in particular, is prone to regard new
developments as intrinsically positive – but as in these many
close to becoming developers in their cases, so here, the adoption of agile and DevOps should be
own right.” seen not as a destination in itself, but as a means to a much
more important end – and that objective is better overall
Tribikram Rath quality. These development methodologies can help to
Director, QA Engineering connect the execution of the QA process more directly to the
General Electric needs of the business – and, in the new climate created by
the COVID-19 pandemic, the rate at which this is happening
is increasing.
It’s interesting to note how mixed the QA picture is. This
year and also in previous years, the World Quality Report
has noted the extent to which functions are merging, such
that testing is becoming a concurrent part of the software
development lifecycle, practiced by people with hybrid skills.
But at the same time, we are seeing the introduction of more
centers of excellence, providing specific services as well as
overall software development governance.
Indeed, quality assurance now addresses many more
attributes, including functional, performance, security, and
usability. What’s more, it does this at several different levels,
such as at the unit level, the interface level, and end-to-end.
All this creates a need for multi-disciplinary skills on the one
hand, and specialization on the other.
Another perennial topic in these pages is the need for
training – but this year, we’re also seeing greater calls for a
change in organizational culture. As we shall shortly see in
this section, it’s clear from this year’s report that respondents
from different countries seem to interpret that word ‘culture’

16 World Quality Report I 2020-21

 QA orchestration in agile & DevOps

in different ways. However, in addition, it’s our view that, to

a great extent, the culture or mindset of a business emerges
from something far more tangible – and that is its org chart. Agile is being We’ve been an agile
pervasive organization, especially
The blending of testing and development roles, and the
retention of specialist areas of excellence – all of this helps
to define an organization, and to create the environment in in development, for the
which it moves. last few years now, and we’ve been
Incidentally, it’s heartening to see the extent to which people working that way in other areas of
both inside and outside the software development function
are recognizing the growing importance of data, and of the organization, too – for instance, in
interpreting it using smart analytics and intelligent insight. governance and finance.”
Increasing levels of adoption Anders Lemon
And so, to the survey data. The evolution of agile and DevOps Head of Architecture, Application infrastructure,
adoption is illustrated in this year’s figures by the degree to Development Platform and Quality Process,
which these environments are being used for testing. On Försäkringskassen
average, we see both agile and DevOps being used more by
a percentage point over last year – but this average doesn’t
tell the whole story. For example, in agile and DevOps
models, 40% of our respondents said 30% of their overall
project effort is allocated to testing. In general, it is clear so for more than 30% of their overall test effort, whereas
that the adoption of agile or DevOps is part of a process of only 11% of Italian respondents and 4% of UK respondents
continuous improvement. said the same. For another thing, we note that overall,
around one in five respondents using Waterfall said they are
There is still some way to go. For one thing, there is quite a still doing so for more than half of their overall test effort.
wide variation between individual countries: for instance, Nonetheless, it seems clear to us that the center of gravity
almost half of US respondents using agile (47%) said they do is shifting.

Q9. On a scale of 1 – 7 (where 1 = never likely to use and 7 = always use), how likely you are to use each of the
Fig
Fig03
X Approaches to speed up and optimize testing in agile and DevOps development
following special approaches to speed up and optimize testing in Agile/DevOps developments?

% of respondents saying 'always use' or 'almost always use' 2020

We prepare and execute test early

as possible (shift left)
52

We maximize the automation of test 51

We test less during development and focus 45

more on quality monitoring/production test

Teams are asked to find and remove 42

redundant test cases

We integrate test as automatic

quality gates in the CI/CD pipeline
39

We implement automated quality dashboards 35

to enable continuous quality monitoring

We use tools to generate or 29

update tests automatically

0 10 20 30 40 50
17
 We also asked respondents about the challenges they face in
applying testing to agile development. Only 1% of them said
Focus on Over the last they faced no real difficulties in this respect. It was a little
surprising to find that the most common challenge (50%)
innovation three years, we’ve was a reported difficulty in aligning appropriate tools for
increased our focus automated testing. However, this may perhaps be explained
on innovation. As part of this, we’ve by the fact that 42% of respondents reported a lack of
professional test expertise in agile teams – and so this lack
invested in DevOps, which helps us of skills may explain the uncertainty about identifying and
to deliver faster. There is a company- applying the right tools.

wide DevOps focus group of which we Different approaches to the acceleration and optimization
of testing are possible in agile and DevOps developments,
in QA are part, alongside colleagues and we asked respondents about this (see Fig 04). We
in other areas including infrastructure were a little surprised not to see dashboards feature more
and business analysis. ” highly: visibility is one of the important concepts of agile,
and the implementation of quality dashboards can make
a considerable difference to implementation. Indeed, we
Hemant Anugonda think their use is becoming a must-have and not nice-to-have

Q12
Assistant Vice President component of projects. Sharing is very important, and quality
TMNAS must be everybody’s concern. However, the emphasis given
to early testing and to automation are to be expected.

Fig
Fig04
X To what
Q12. extend
To what the skills
extent do theneeded fromneed
skills you the QA and
from QAtest
andprofessionals need to
Test Professionals change?
need to change?

Skills are less relevant to current Skills are OK - No change needed Skills are lacking and
QA and Test Professionals required more

Knowledge of [new] Test

automation skills 16% 55% 29%

Test case design skills 17% 54% 29%

Knowledge of CI/CD pipeline tools 11% 61% 28%

Collaboration skills 17% 55% 27%

Development skills 16% 58% 26%

Build and Deployment tool knowledge 15% 61% 25%

Knowledge of Business Process 19% 56% 26%

Data analytics and AI skills 11% 65% 24%

Test environment, containerization 17% 59% 23%

and test data skills

18 World Quality Report I 2020-21

 QA orchestration in agile & DevOps

survey was the response to a question about the importance

of various criteria for successful agile and DevOps adoption
Discipline and But whatever (see Fig 05). The technology stack was rated as essential
or almost essential by 65% of respondents, while skill sets
determination the case, agile is and organizational culture came at the bottom, with 34%
pretty much the and 28% respectively rating these highly. Operational and
go-to approach. It enables us to find business priorities were rated highly by 41% of respondents.

fast, and fix fast. For us, with agile, the How can this be? Maybe some respondents thought these
criteria were a given, or maybe they interpreted these
key word is discipline. It’s about the options differently. That’s certainly a possibility: we noted
determination to do it right.” wide variations between countries in response to this
question. For instance, the highest figure for skills needs
Geoff Meyer was Poland, with 63%, while the lowest was Brazil, with just
Validation Architect 5%. Similarly, for culture, the highest figure was Sweden,
Dell EMC with 69%, and the lowest was once again Brazil, with only
2%. (Brazil’s perceived technology stack need was very
high, at 98%.) The concept of culture could mean different
things in different countries, and may therefore be weighted
differently for people.
User experience is key in quality assurance, which is why
it’s essential to find any anomalies while still in production.
We asked respondents whether they proactively monitor Q13. On a scale of 1-7 (1= not at all important, 7
and review production logs for incidents, and performance = essential)
How rateare
important how important
the followingthe following
aspects for
Fig05
Fig X
trends, in order to identify application issues and potential aspects areagile
successful for successful
and DevOpsAgile and DevOps
adoption?
defects even before end-users might notice them. The vast adoption?
majority (89%) said they do this always, usually, or frequently;
only 11% said they do this only sometimes or occasionally.
We believe the 89% figure is rather high: in our experience, % of respondents saying 2020
production monitoring is not happening as often as it might. 'essential' or 'very important'
In the case of the 28% who specifically said they always
do this, it may be an indication that the process has been
automated here. 65
Technology stack
Which metrics are teams using to track applications quality?
Code coverage by test was the most important indicator, with
53% of respondents saying they always or almost always use
it. This is compliant with agile test pyramid good practices, Executive support 55
although it can be argued that this is more of a development
indicator, since it measures unit tests. Almost as high in
response, with 51% saying always or almost always, was risk Operational and
covered by test. This is particularly significant: if test strategy 41
business priorities
is based on risk, it’s a very good thing.
Our survey also highlights the continuing need, driven by
agile and DevOps environments, to extend team skills, and Talent /skill set 34
in a variety of areas, including technical and development,
operations, and also softer skills such as collaboration.
However, the areas of greatest skills needs, mentioned
by 28% to 29% of respondents, were, in growing order of Culture 28
importance, knowledge of CI/CD pipelines tools; test case
design skills; and, at the top, test automation skills. That
0 10 20 30 40 50 60
said, though, respondents mentioning test automation skills
needs have dropped from 31% last year to 29% this time, Regardless of what may have prompted some of these
which reflects our own experience: organizations do now responses, we remain resolute in our own view that success
seem to have more skills on board in this area. in agile and DevOps adoption is predicated on the extent
to which developments are business-driven. We derive this
Technology – or business? opinion not just from our own experience in the field, but
Out of all the insights relating to the agile and DevOps from the pervasive sense of a commercial imperative that
theme, perhaps the greatest surprise for us in this year’s emerges from many other areas of this year’s report.

19
 QA orchestration in Don’t silo responsibility for QA.
Share it.
agile & DevOps: Implicit in the adoption of agile and DevOps is a more

call to action collaborative approach to business processes, including

quality assurance and testing. Organizations should aim
as much as possible to make QA a shared responsibility,
integrating quality practices into every part of the
lifecycle. Centers of excellence should be retained
only for specialist areas of QA, such as emerging
AI-based routines, where it would be too costly and
time-consuming to ensure all team members had the
requisite skills.

Spread the word.

If quality isn’t only the responsibility of QA teams, it isn’t
only the responsibility of the IT function, either. The
whole organization needs to understand its importance.
In particular, IT should educate the business on good
practice in gathering, codifying and maintaining the data
on which future software is built, and from which future
insights will be derived.

Be part of the business.

IT and QA don’t only need to educate the business:
they also need to learn from it. It’s true that the IT
function is more business-driven these days, but there
is still a degree of them-and-us. Those barriers need
to go. Temporary secondments to non-IT functions,
and enrolment of IT and QA people on business skills
development programs, are both possibilities.

Make room for dashboards.

It’s easier to plan a direction of travel, to identify gaps
and errors, and to make adjustments along the way,
when everything is easily visible. Quality dashboards
are key.

Listen more to users.

Production monitoring is not as common as claimed.
That needs to change. Addressing this will lower risk and
improve customer experience.

20 World Quality Report I 2020-21

 Artificial intelligence and machine learning

Artificial intelligence Andrew Fullen

Head of Innovation and Technology,
Sogeti UK

and machine learning Michel Burini

National Manager, Sogeti France
Prabakaran Karuppiah
Vice President, Financial Services
Aspiration – and adoption by degrees Capgemini
Rik Marselis
Principal Quality Consultant
Digital Assurance & Testing, Sogeti NL
Shiva Agolla
Regional Practice Leader,
Digital Assurance and Quality Engineering
Sogeti USA

Learning We’re seeing real When we’ve asked about the greatest challenges in testing
key applications, the top answers for years have been not
lessons, benefits now from AI. having enough time to test, and not having the right test

feeding back For instance, over the processes and methods. This year’s pandemic has only
exacerbated the challenge, with digital as a prominent asset
last year we’ve been on which businesses are counting for survival.
using machine learning for testing Can one produce high-quality digital assets such as
analysis on customer usage, seeing e-commerce, supply chain systems, and engineering and
workforce management solutions, without spending time
which features are working best and money assuring quality? In other words, can a system be
for people – and we’re feeding tested without testing it? That may sound like a pipedream,
but the industry has already started talking about developing
that knowledge back into our systems and processes with intelligent quality engineering
development strategy.” capabilities. Typically, innovations are greeted with a mix of
cynicism and excitement. When the status quo is challenged,
Suresh Dwadasi the expectations run high. In our survey responses, that
Delivery head, multinational bank seems to be the case with the transformational capabilities
of the application of artificial intelligence (AI) to digital
assurance and quality engineering. While only time will tell
the extent to which self-testing systems become a reality, it
is evident that significant efficiency and speed can be derived
by applying these smart technologies in business assurance,
driving growth and improving outcomes across sectors.
So, while there are high expectations and some evidence of
application of supervised learning as a core part of machine
learning (ML) in making quality engineering (QE) smarter,
adoption and application have still not reached the required
maturity to show visible results. In fact, we saw that in several
cases, test automation is being given higher priority, in
order to find an immediate answer to the perennial need to
speed up the software development lifecycle and to lower
costs. This is not surprising, though, especially in a year in
which schedules have been shot to pieces by the COVID-19
pandemic, and in a time when experience, speed and quality
are key to the resilience of organizations.
Another reason for relatively less progress seems to be
lack of required knowledge and relevant skills within
organizations.

21
 In fact, it was the
There is World Quality Report
immense that catalyzed our
potential use of AI in testing. In
the reports of around
2016 and 2017, you
were calling it cognitive QA, and it gave
us ideas. We started with test planning
– including requirements analysis and
traceability, impact assessments, and
configuration planning. Bots can really
help in these areas. We’d already seen
that DevOps could enable the provisioning
of test environments, and we moved on
from there to test triage analysis. Until
that point, we’d needed a lot of subject
matter expert knowledge, and we needed
those SMEs to mine through loads of data.
But now, AI offers us the capability to
streamline the triage process by mining
failure artifacts, such as logs, past defects,
past failures, source code, etc., and quickly
determine potential sources of the problem
– the environment, or the test case/script,
or the code itself. In addition, it can be
augmented to provide a recommendation
as to whether it’s likely a duplicate or
“related to” issue which helps the order of
priority in dealing with them.”
Geoff Meyer
Validation Architect, Dell EMC

22 World Quality Report I 2020-21

 Testing of AI systems remains
We’ve been using at low maturity
Identifying
and NLP to build a test factory In addition to the application of AI and ML in QA, there is also
the matter of how to test the AI systems. Do organizations
prioritizing framework for data have a strategy for that? It’s clearly an emerging art: how do
use cases comparison. We’ve also you gauge something that is learning and morphing on its
own terms? Organizations are realizing they need to figure
been looking at other AI this out, and that new skills will be needed to test AI systems
tools such as visual UI automation, with a lot of when it is still at its current nascent stage.
dynamic content. The execution is very fast.” At the same time, we have also witnessed that some sectors
have made a start in this area. We are seeing medical device
Hemant Anugonda manufacturers develop standards for the process of verifying
Assistant Vice President and validating AI-based algorithms in their product lines. We
TMNAS are also seeing the automotive industry virtually validate
AI algorithms in their advanced driver-assistance systems
(ADAS) and autonomous systems.

New use cases are emerging

The good news is that some companies are now working hard
to change this, and are blazing the trail in the application to
QE of unsupervised learning, natural language processing Retaining Of course, our team
(NLP), and computer vision technologies. We have witnessed members are learning new
new use cases emerging since last year. For example, running the quality
analytics on production incidents and run-time application mindset skills, in areas such as AI,
logs is helping not only to conduct a deep intelligent what-if analytics, and machine
analysis, but also to predict future quality, and to prescribe
necessary planning in development and testing activities. learning. But they still need to retain a quality
This helps to improve testing by incorporating actual mindset, that’s essential.”
usage patterns into tests in a smart way. Another use case
that seems to have gained traction is the use of AI for the Andy Armstrong
generation and management of test data. For instance, it Head of Quality Assurance & Testing
is being used to identify test coverage gaps compared to Nordea Bank
real user experience patterns. The same is also successfully
applied to the creation of synthetic data, for example to
comply with GDPR rules.
The aspirational nature of some responses to questions
about AI and ML is borne out when we consider the smart
QA options that organizations deem to be relevant for them How AI impacts the test strategy –
this year. Well over half our total respondents (58%) said and where skills are needed
that automated root cause analysis was extremely or highly
When asked about how AI impacts quality and test
relevant, even though we suspect that many of them may not
strategy, respondents were broadly sanguine. They seem
actually have applied it yet.
to demonstrate they have a good grasp of how to test, and
The provision of fit-for-purpose test data was also highly are confident their strategies will cope. That said, more than
rated, at 54%. We find this less surprising, because it’s likely two-thirds of them (68%) said new test strategies would be
that organizations do see the benefit here. It would be useful, needed to test AI itself. A full three-quarters of respondents
in our view, if they saw fit-for-purpose test environments (75%) said that overall, only small changes to their current
and defect prediction in the same way: these two options test strategy were needed, possibly indicating that some
came in at 39% and 29% of respondents respectively. In fact, of them have yet to appreciate the genuine potential of AI
defect prediction is a big, obvious application for artificial beyond its headline glamor.
intelligence. It’s possible that in answering this question,
people were thinking principally of prediction in terms of how Respondents were also asked about where AI has changed
many defects, when what is far more important and useful is the QA skills they need. In almost every case (see Fig 06), we
predicting where they will be. see that perceived skills needs are lower than in previous
years, but these overall figures disguise some marked
country-specific differences in the response.

23
 Q18. To what extent does artificial intelligence change the skills you need from QA
Fig06
Fig 1 Extent to Professionals?
which AI and ML change the skills needed from QA and test professionals
and Test

“Skills are lacking” summary 2020 2019 2018

34
Software development
36
engineer testing skills
35

33
Understanding of AI implications
36
on business processes
36

32
Development and coding skills 34
34

31
Test strategy and test design skills 31
32

27
Data science skills 29
31

25
Test data set up expertise
28
31

19
Non-functional testing skills 28
(performance, security)
28

Single code per option 0 5 10 15 20 25 30 35

For instance, the greatest overall area of need this year was Relevance of AI and ML in QA tool selection
identified as software development engineering testing skills
(S-DET), mentioned by over a third (34%) of respondents. Is AI now a key criterion for selecting new QA solutions,
However, in the Netherlands, it was an issue for only 5% of products or tools? Overall, 86% of our respondents said yes.
respondents, while in the UK, Belgium and Luxembourg, the This is a figure we might have expected to be higher, and
figures were over 70%. It could be that these latter countries once again, it is a universal figure that hides some country-
are genuinely lagging in this regard, or it may simply be specific differences. For example, only 60% of Japanese
that category has been interpreted differently in different respondents agreed. Perhaps this is because Japan has a
countries and organizations. long-established history in QA practice, and has a broader
range of criteria than most when it comes to tools selection.
The country-specific differences were fewer for non- Indeed, many Japanese engineering companies have spent
functional testing skills, the bottom bar on the grid, where much time building in-house solutions for most of their
the global result of 19% of respondents shows a marked tooling needs, and they are averse to changing tools with
drop on figures for the previous two years. It could mean which they are familiar.
that organizations feel they have things pretty well covered
– or they might think that AI doesn’t really alter skills We also asked about the current usage of smart options in
requirements in these areas. QA. Two-thirds of respondents (66%) said they use smart
tools for the automatic design and the execution of test
cases from code changes, and 56% of respondents said they
use them for identifying the most important user aspects
and configurations to be covered during testing. It would be

24 World Quality Report I 2020-21

 Artificial intelligence and machine learning

interesting to know why people gave these answers: here, as

is so often the case with artificial intelligence and machine
learning, the words that are used to describe applications are
Making AI Some of the tools
capable of several interpretations.
and ML part we use have AI built into
Perhaps, as time passes, and as AI and ML move beyond early
excitement and into more widespread everyday acceptance, of tools them – in mobile and
we will begin to see more meaningful usage patterns.
evaluation web-based testing, for
Looking ahead instance. The AI learns
In a similar vein, we also asked people to look ahead (Fig 07):
the app, and adapts
what were their plans for AI and testing? itself within the automated test
Of all the responses to this question, the popularity of the routines.”
first statement is of course the least surprising. Almost
nine out of ten respondents (88%) said that AI was now the Leon van Niekerk
strongest growth area of their test activities. The bottom Head of Test Center of Excellence
position for test execution makes sense: current tools already Pick n Pay
address these activities very well. However, we might expect
to see smart QA dashboards increase in importance in years
to come. As we have just noted, it may well be the case that
some QA teams are exploiting AI hype for general budget
purposes – but that, we feel, will be only a small part of the
picture.
potential not just in cost-efficiency, in zero-touch testing,
Even though the benefits may not yet be fully in reach, the and in time to market, but in the most important way of
vast majority of people are genuinely enthusiastic about the all – and that is in helping to achieve continuous quality
prospects for AI and ML. These smart technologies have real improvements.

Q20. On a scale of 1-5 (1 – Strongly disagree to 5 – Strongly agree) rate to what degree following statements
Fig X reflect your plans for artificial intelligence and testing
Fig 07 To what degree the following statements reflect your plans to utilize AI in QA activities?

Top 2 box summary: 5 Strongly agree + 4 2020

AI is the strongest growth part of our test

activities (testing with AI and testing of AI) 88

We will have more AI trials/Proofs of

Concepts in place 80

AI will be used to generate test

environments and test data 80

We will use more smart QA dashboards

powered with AI 75

AI systems are going to be used to store

and reuse important domain knowledge 75

We will use more AI in our test

execution activities 71

25
 call to action Artificial intelligence and machine learning

Focus on what matters. In short, for organizations to derive the greatest benefits
If we were to summarize what artificial intelligence from AI in quality engineering, they’ll need their teams to
can do for digital assurance and quality engineering, it strengthen their knowledge and experience of the tools, of
would be the ability to bring higher efficiency and faster overall QA and IT strategy, and of the commercial objectives
outcome. of the business as a whole. It’s a great opportunity, not just
for companies, but for the QA people they employ.
It’s a good idea, therefore, to put these technologies
in the hands of smart people, and to ask them to focus QA teams need to have few team members having data
on what’s important. The area of greatest need is likely science, analytics and AI skills. If required, they should
to deliver the greatest benefit. What’s more, any early collaborate with other parts of the organization to acquire
successes won’t just be good for morale: they’ll also be a such skills.
platform for future success.
Testing AI systems: have a strategy.
Adopt smart ways of working. Using artificial intelligence to test is one thing. Testing AI
Focus on value added activities by leveraging the itself is another, particularly when it’s constantly evolving
insights gained through implementation of analytics, and becoming an integral part of your business survival.
AI and ML in QA. Encourage teams to adopt the tools There are challenges in the holistic coverage of AI systems
having AI capabilities – for instance, bias in AI and ethical AI. This is especially
the case when business solutions venture into supervised,
Look beyond conventional unsupervised, natural language processing (NLP) and deep
learning.
QA skills in the team.
Organizations can start to approach this challenge by gauging
Skills gaps feature in the World Quality Report every
the approach being taken in most common use cases of AI,
year. It demonstrates that there’s always something new
such as self-driving cars, ride-sharing apps, or speech and face
to learn, and in the case of AI, it’s not just about what the
recognition software. Can any general principles be derived
technology can do, but about how it can be incorporated
for testing other types of AI systems? We think yes, and this
in the overall software development lifecycle.
field is going to continue witnessing new innovations and
strategies.

26 World Quality Report I 2020-21

 Artificial intelligence and machine learning

Test automation Archie Roboostoff

Head of Product Management, Functional
and Performance Testing, Micro Focus
Parinita Patankar
Steady determination brings Vice President, Digital Assurance and
Quality Engineering, Sogeti India
progress towards increased speed
Rajesh Natarajan
and flexibility Director, Digital Assurance and
Quality Engineering, Sogeti USA
Sai Grandhi
Director, Digital Assurance and
Quality Engineering, Sogeti USA
Wouter Ruigrok
Agile Quality Coach
Digital Assurance & Testing, Sogeti, NL

Tracking We’ve started Test automation has become such a major area of focus in
quality assurance (QA) in recent years that, before we dive
progress to to automate some into what this year’s survey data tells us, it might be useful

a zero-touch of our testing, but to articulate some of the trends we have recently been
witnessing in the field.
future we need to do more.
We’ve been seeing new machine learning techniques being
We’re measuring the applied to object recognition, and also to determine the
automation levels we’re achieving scope of optimal automation tests to reduce the exponential
growth in test scripts. AI-based self-healing scripts to
right across our software development automatically modify scripts during run time, in case of object
function. The goal is to be as little and page element changes, are also an interesting addition to
hands-on as possible. It’s going to be the arsenal of new automation techniques.

part of the platform: it will happen We’ve also seen what we might term the automation of
automation – for instance, automating testing within an
without the developer even having to RPA routine in order to identify the types and the relative
think about it.” performance of the most common transaction each day.
Another good example of this is where intelligence is being
Anders Lemon brought into production, such that zero-touch automation
Head of Architecture, Application infrastructure, can identify missing test cases, generate them, and run them,
Development Platform and Quality Process, in a highly automated, highly intelligent, and highly targeted
way.
Försäkringskassen
However, it’s only some organizations we see that have
reached levels of maturity that are akin to all this. Many
others are still feeling their way.

Re-assessing challenges
Difficulties like this are revealing, which is why it has also
been interesting for us to revisit a question from the 2019
survey that was not asked this year. That question explored
the challenges organizations faced in achieving their desired
level of test automation, and among the most common
responses, in 2019 and also in previous years, was the
frequency with which applications were changing. That
challenge hasn’t gone away, and it is part of the reason why
interest is growing in machine-based testing. We’re going to

27
see more instances of the example provided above, where
an automated process can run software, identify needs, spin
out test cases, run those tests, and then report on them – all Balancing Our CTO is highly
without intervention.
speed focused on resilience
Another abiding challenge in previous years has been the lack against and stability. We have
of people with sufficient specialist skills and experience. It’s
why we see organizations having difficulty with formulating accuracy zero tolerance for
overarching strategies, and applying automation to areas downtime, as does
such as regression testing, rather than to anything broader in
scope and more ambitious. our regulator. So we invest a lot of time
A third of respondents last year said they had too many in production analysis, and in assessing
different automation tools. The good news is that this seems the operational performance of our
to be less the case now. It’s also worth noting that with so
many organizations moving to a continuous integration /
teams. There’s always a balance required
continuous delivery (CI/CD) platform, it’s difficult to work between quality, risk and delivery time.
with only a handful of tools. What’s needed here is not
necessarily a smaller toolkit, but better orchestration. This,
Test automation helps in this regard but
we feel, should be a goal for 2020. it’s about utilising the experience of our
Where is automation happening? subject matter experts too.”
Several of these themes are corroborated by this year’s Barry Blake
survey findings. The volatility in applications – in other Senior Manager, QA & Testing, Permanent TSB
words, the frequency with which they change – has not
gone away, and this is perhaps one reason why we see only
Q24. On a scale of 1 – 7 (1 – Never true 7 - Always true) please rate each of the following statements based on the
Fig X degree that they are true for your organisation.

Fig 08 Test Automation: Level of adoption

Top 2 box summary: 7 Always true + 6 2020

We have the required automation

tools
68

We have enough time to build/maintain the

automated tests
63

We have the right automation strategy

59

Test data and environment are available at 58

right time

We have the right skilled and experience 48

test automation resources

Applications have achieved the desired level 38

of stability required for test automation

We get ROI from our 37

automation efforts

28 World Quality Report I 2020-21

 Test automation

While the promises are big, we understand these automation

tools aren’t yet sufficiently mature, based on the challenges
Making Wherever possible, mentioned by our survey participants.
automation we focus on reusability
Perceived benefits…
part of the in performance testing,
It’s also interesting to note that while as many as 68% of
CoE toolkit so we need to do more respondents said they have the required automation tools,
R&D in this area. We only 37% of them felt they get a return on that investment.
may also look at introducing automation This may be because of increasing maintenance efforts that
are required to maintain larger automated suites. We feel
skills to the applications testing and moving towards scriptless automation tools may provide
performance testing teams in our Center better return on investment in the long term.

of Excellence.” We asked our survey participants to consider the benefits

question in a little more detail (see Fig 09). The bars on the
Tribikram Rath left show the percentages of respondents who felt they were
Director, QA Engineering seeing those benefits, while the boxes on the right show the
General Electric average value of that benefit for these respondents.
In the bars on the left, the perceived benefits for 2020 are,
as we would expect, mostly higher than in previous years,
although we are surprised to see fewer respondents seeing
slight increases in the degree to which automation is being better test coverage. It is interesting that the greatest
used in different areas of QA. Overall, our respondents told perceived benefit was better control and transparency of
us that around 15% of all testing was automated. Only 3% testing activities.
of them said they were automating 21% or more of their The boxes on the right indicate a lower perceived value of
test activities. these benefits year on year. The reduction of test costs is a
Percentages for the automation of individual test activity case in point. What may be behind this is the extent to which
types have mostly risen since last year, although we would respondents are factoring in the same criteria when they
have expected them to be higher still. User acceptance and consider this question. Are they each including investment in
unit testing each came in at 18% (both of them were at 15%
last year), and automated system integration testing and
automated test data generation each came in at 17% (up
from 14% each last year). A new survey option this year was
API testing, and respondents told us 19% of their activities in Incorporating We started
this area were automated. physical automating customer
Given these percentage levels, it’s perhaps surprising to see automation payment processes
the degree of confidence organizations expressed when
they were asked for their views on various aspects of test end-to-end in 2017.
automation (see Fig 08). There is a physical element to this
Some of these views seem aspirational to us. For example, automation: for instance, we have a
the frequency with which apps are now changing is hard
to square with the 63% of respondents who say they have
robotic arm that inserts into the machine
enough time to build and maintain their automated tests. It reader one of the bank cards we keep for
could demonstrate an unwillingness to admit that their teams
are time-squeezed: if those team members had been asked
testing, and that then enters the PIN. It’s
the same question, the answers may have been different. an example of how we focus on front-
Our survey also indicates the willingness of the QA end, functional business process testing
community to consider newer natural language processing – in other words, on what really matters
(NLP) based automation tools, which provide key benefits
such as scriptless automation, model-based testing, to our customers.”
the use of plain English statements to generate scripts,
and a shallower learning curve, which enables all project Leon van Niekerk
stakeholders to contribute to automation efforts. Options Head of Test Center of Excellence
such as self-healing capabilities are going to increase Pick n Pay
gradually, but there is no doubt that these are the future
of automation.

29
tooling, in infrastructure, in maintenance, and in building the development skills (down to 41% from 51% last year), and
suite, or are some of them only considering some of these? test automation architecture skills (down to 30% from 38%
last year). Robotic process automation (RPA) skills were also
Besides, it might be argued that test cost reduction is not as
down, to 35% from 43%, which we found surprising – but
useful a metric for measuring automation benefits as, say,
overall, we feel these drops can be explained not by the fact
the reduction of test cycle times – and it’s good to see a rise
that skills gaps are shrinking, but by the presence among the
this year to 65% of respondents reporting this benefit, from
responses of a CI-CD skills option for the first time this year,
54% last year. It’s the biggest increase on the graph.
which may have skewed the figures. The CI-CD approach
… and perceived skills needs is a hot topic right now, but strengths in this area are not
really needed by the whole team. Team members would
The perceived presence of the requisite test automation more broadly benefit from skills in other areas, including
skills is always interesting, and this year we see skills needs development, machine learning, and API and microservices.
dropping year on year in several categories, including

Fig
Fig09
X Q25. What
Benefits benefits
realized have
from you
test realised from Test Automation?
automation

Mean average 2020 2019 2018 2017 2016

2020 2019 2018 2017 2016

Benefit- Benefit- Benefit- Benefit- Benefit-
value % value % value % value % value %

69
Better control and 63 **Benefit **Benefit
transparency of 66 value % value %
not asked** not asked**
18% 19% 19%
test activities 43
38

65
Reduction of 54
test cycle time
64 19% 21% 18% 19% 21%
54
39

62
Reduction of 56
test costs
61 19% 21% 16% 20% 20%
53
39

62
Reduction of overall 53 **Benefit
security risk and security 18% value %
not asked**
related issues in code

58
**Benefit
Better test coverage
51
68 20% value %
not asked**
18% 19% 19%
40

57
56
Better detection
of defects
64 20% 21% 18% 19% 22%
60
42

Better reuse of 57
test cases
65 20% 18% 19% 20%
57
40

30 World Quality Report I 2020-21

 Test automation

Looking ahead
In conclusion, we asked survey respondents to look ahead, Moving Over the next
and to rate the automation techniques they would be
extremely likely to use in the year to come (see Fig 10). The automation two or three years, I
greatest response was for test environment virtualization at into think we’re going to
16%, while model-based testing, which was the highest rated
last year, has dropped from 17% to 9%. In our experience, few
mainstream QA see automation play a
companies have any kind of uptick in this area, partly because greater role – artificial
there are significant set-up and management overheads.
intelligence and DevOps, too. Already,
Like last year, test design automation came at the bottom end
of the results. It’s dropped even further this time, though,
we’re seeing more in-sprint automation,
with only 5% of respondents saying they will be extremely earlier in the development cycle, so
likely to use it in the coming year, against 8% last year. This, issues are being identified earlier, and
we feel, may be a good sign. It could be an indication that
measures in this area are already in place. faster, too. This helps to create better
Taking stock, we feel that positive progress is being made quality metrics.”
here overall. Organizations are working their way forwards,
in spite of the increasing complexity of applications, devices, Suvo Ghatak
and browsers. Senior Manager
Quality Engineering, Wabtec
We feel that the tools are getting smarter – and that people
need to be sufficiently skilled to get the most out of them.
Last of all, it’s worth noting that technological innovations
in general are a means to an end. They are not an end in
themselves, and test automation is no different in this
respect. It has a job to do – and that job is to bring new levels
of speed and flexibility to quality assurance.

Q27. On a scale of 1 – 7 (1 = Not at all likely and 7 = Extremely likely) rate how likely it is you (your team) will be
Fig
Fig10
X How likely you are to use the following techniques in the coming year?
using any of the following Automation Techniques within the coming year.

Top box summary: 7 Extremely likely 2020 2019

14
Test environment virtualization 16

Robotics automation (task-based 11

automation) for test activities 13

Headless automation 11
(Non-GUI based automation) 12

Model-Based testing tools 9

(automated test cases design) 17

Test data automation 9

(test data generation) 11

Test design automation 5

(test case generation) 8

31
 Test automation Change the status quo.
We need to accept that for various reasons, the testing
call to action phase will continue to be squeezed. Introducing more
automation – and pursuing it vigorously – is the only way
out of this evergreen challenge.

Think ahead.
Modern applications to which test automation might be
applied will continue to undergo rapid changes. That’s
why choosing a tool or framework that will keep pace
with dynamic changes is very important.

Choose the right framework.

Opt for design automation frameworks that are
intuitive, dynamic, that have self-spinning environments,
that have algorithm-based prioritized scoping, and that
can self-provision test data.

Balance automation against skills needs.

Gaps between advanced automation tools and required
skillsets are here to stay. The way forward is to develop
a strategy either to lessen dependency on skills or to
make automation inclusive.

Get smart.
Offload the technical challenges in automation for
artificial intelligence and machine learning to take care
of.

32 World Quality Report I 2020-21

 Test automation

Test data Kirthy Chennaian

IT Transformation Director
Digital Assurance and

management and Quality Engineering

Capgemini North America

test environment Ramesh Mahadevan

Senior Director

management
Digital Assurance and
Quality Engineering, Sogeti India
Kyle Abraham
Regional Practice Leader,
Digital Assurance and
Slow but steady progress in the Quality Engineering, Sogeti USA
complex world of test data and
test environment

Some things in business take a long time to change, and in

Focus is on We have created quality assurance (QA), many of the challenges we saw last

efficiency and a QA enablers team

year in the field of test data management (TDM) and test
environment management (TEM) are just as much in evidence
quality that focuses on this year.
test environments, Organizations still need to figure out the best way to spin up
test data, tooling and service test data and test environments on demand. While adoption
of cloud and tool usage has shown a positive trend for TEM
virtualization. Bringing this QA and TDM, more maturity is needed to make effective use
enablers team closer to the test of the benefits of the cloud and of TDM tools. The overall
execution has resulted in improved governance of test environment and test data came out as a
bigger challenge than the technology itself. The holy grail of
efficiency and quality.” test environment and test data is to strike the right balance
between the cost and the timely availability of the fit-for-
Andy Armstrong purpose environment and data. These challenges are further
Head of Quality Assurance & Testing exacerbated as teams adopt an automation-first, continuous
Nordea Bank testing approach to QA. Integrating with diverse systems,
including third-party systems, to simulate real scenarios
and orchestrate end-to-end, continuous testing remains a
challenging endeavor.
Addressing these challenges is far from straightforward. It
requires an organization to undertake a long, well planned
journey. The good news is that we see several organizations
ready with a roadmap, and making steady progress.

Evolution, not revolution

In our survey, we asked respondents about the extent to
which they are using different types of test environment
(see Fig 11), and we see that 29% of testing still occurs in a
traditional on-premises test environment.

33
However, it is worth noting a few things:
• The proportion of survey participants using a traditional
approach has been slowly declining for the last two years.
Effective QA As part of our
• The proportion using cloud-based, on-demand disposable
is integral overall push towards
test environments has been increasing over the
same period. to digital digital transformation,
• There has also been an increase since last year of testing
transformation we expect our test
in containerized environments, using Docker or a
similar technology.
environments to move
While the changes indicated here are small, two points are
from the data center
worth making. The first is that, in our experience, while to the cloud.”
differences may be incremental, some organizations are
building real momentum in the move to the cloud and in Tribikram Rath
using virtualized environment and data. Director, QA Engineering
General Electric
The second point is that the general pace of change is likely
to accelerate. The new business environment precipitated
by COVID-19 has created a genuine sense of urgency about
achieving digital transformation, and one of the many
beneficiaries of this new impetus is likely to be TEM. This is a
trend worth watching, and we anticipate real change over the
next couple of years.

Fig11
Fig X Q30. What %ofoftesting
Percentage all your testing occurs
occurring in each
in various testtype of test environment?
environment types

Mean average 2020 2019 2018 2017 2016

29
% of our testing occurs in a traditional 30
31
permanent test environment 26
28

21
% of our testing occurs in a cloud-based 20
19
temporary test environments 21
27

17
% of our testing occurs in a 17
17
virtualized test environment 19
24

17
% of our testing occurs in a temporary 17
17
test environment but not cloud-based 18
22

16
% of our testing occurs in containerized 15
(Docker or similar) test environment 16
15

34 World Quality Report I 2020-21

 TDM & TEM

Achieving benefits with TEM and TDM

How comfortable were organizations about their QA Centers of We’re
teams’ ability to succeed in achieving targets related to test
environments? High levels of satisfaction were reported excellence continuously
for the robust configuration of test environments (49% of can catalyze innovating on TEM
respondents); for the timely availability of the right test
environments (47%); and for the modernization of test innovation and TDM. We have a
environments, for example with the cloud and containers center of excellence
(also 47%).
in place that assesses priorities,
These technology-related responses scored higher than
those related to governance – for instance, sufficient facilities
oversees and directs processes, and
for teams to book and manage their test environments (33%), implements the right tools for TEM
and good visibility of available test environments (29%). But and TDM.”
on the whole, these figures aren’t too bad – and the timely
availability figure of 47% is a marked improvement on last Hemant Anugonda
year’s response. We think that IT teams need to do more to
Assistant Vice President
maximize the benefit from the cloud. While organizations
TMNAS
have moved in this direction, the right governance is
lacking in many cases. For example, we witnessed that the
environment is not released in a timely manner when it is not
needed, thereby resulting in a higher cost.
On the TDM front, we were little surprised at the satisfaction
score with teams’ ability to deliver benefits. As many as 61%
of respondents said their teams are able to manage the size
and complexity of test data sets. While this looks good, we team, rather than being derived from production data – but
think that several respondents still equate test data with even so, it’s a very high figure. We suspect teams are creating
copying and masking the production data. With continuous specific test data sets in large volume to address the needs of
testing and a push for more performance and security, that in continuous automation and performance. The complexity of
itself may not be sufficient. Also, if we compare this with last testing has increased. Adapted TDM solutions partially solve
year’s survey, we see that some of the challenges didn’t come the problem since we observed that clients buy TDM tools
out that strongly this year, such as the ability to manage test to mask production data. Masked production data doesn’t
data consistency across several disparate systems. So, based provide testers or developers the breadth of the volume of
on our experience with several customers, and also making test data needs.
comparisons with past surveys, we think that while progress
is being made related to TDM, the desired maturity will take
Test environment monitoring
more time to realize. Organizations are using different approaches to identify
environment-related issues encountered during testing. The
That said, well over half of our survey participants (58%)
figures weren’t especially high, but they were interesting:
said their teams create and maintain synthetic test data for
38% said they use infrastructure monitoring to notify the
testing, and we do indeed see this ourselves in the field.
test environment server level outage; 32% said they have
The proportion of respondents who claimed to provision and effective test environment monitoring solutions, enabling
generate test data for multiple iterations of testing in various outage notifications, trend analysis and self-healing features
ways was interesting. As many as 90% of them said they for all test environments; and 27% said they have detailed
maintain specifically created test data sets for their tests. It’s environment monitoring enabled only for higher test
true that in some industries – healthcare, for instance – there environments, such as acceptance tests, or integration tests.
are very stringent data requirements, but this figure seemed
This shows that only about one third of the enterprises
high to us, even though the trend is indeed in this direction.
have adopted a mature approach towards test environment
Even more surprising was how many survey participants said monitoring and management. However, enterprises are
they create test data manually with each run. Here, the figure still not using monitoring tools on test environments to
was 79% – a big jump up from 59% and 58% in the previous compare and contrast that of production to ensure that
two years. We suspect that this year, respondents have taken both environment configuration and performance issues are
“manually” to mean test data that has been generated by the correctly addressed.

35
Agile and test data provisioning and the graph opposite (see Fig 13) shows their highest
responses. Almost two-thirds (64%) of them said they use a
How are organizations provisioning and generating test data dedicated test data support team on an as-a-Service basis.
for multiple iterations of testing, and to what extent are This is to be expected: as organizations move toward agile,
they using each approach? In fact (see Fig 12), the range and TEM and TDM are indeed likely to become more of a shared
extent of responses were fairly evenly matched. This question service function, serving multiple and distinct agile teams.
was updated to include new answer options this year, which

Q33. How do you (your teams) a) provision and generate test data for multiple iterations of testing? and B)
Fig X
Please express as a percentage how much each method is used.
Fig 12 Provisioning and generating test data for multiple iterations of testing

% use of services

27% 90% 31% 82%

We maintain 30% 87% We use 30% 79%

specifically automated data
created test generation tools
data sets for for each test run
our tests
We use We create test
anonymized data manually
copies of with each run
production

% of respondents using test

% of each method utilised
data creation techniques

Q33
invalidates trend comparisons with previous reports. It will
be interesting to see if there is any movement here in years
to come.
We also note the extent to which teams are creating small,
dedicated test data sets for each sprint (39%), and the degree
to which they are generating bulk data sets that are re-used
for every sprint (31%). These approaches are becoming more
Finally, in a new question for this year, we asked our survey
popular in certain sectors such as healthcare, and we expect
participants to rate how frequently various test data
to see these numbers grow.
provisioning approaches are being used in their agile teams,

36 World Quality Report I 2020-21

 TDM & TEM

Q35. Using a scale of 1 and 7 (1 = never use, 7 = always use) please rate how frequently the following
Fig13
X Howdata
frequently the following
Fig
test provisions are being test
useddata provisions
in your are used by agile test teams?
agile teams.

% of respondents saying 'always use' or 'almost always use' 2020

Use a dedicated test data support team

to provide test data as a service 64
64%

Teams create small sets of test data

39
39%
dedicated for each sprint

Teams have generated a bulk data set 31

31%
and re-use it for every sprint

Teams use test data tools to create test 28

28%
data for each sprint

Summary
It’s clear to us – not just from this year’s survey responses, but
also from our collective experience – that TDM and TEM still Cloud-based Over the
represent a highly complex space. Test environments are still
in a hybrid form and are evolving slowly. We’re seeing some environments next two years,
clients adapting TDM solutions and TEM monitoring, but increase I expect to see
we have yet to see how their adaption is fully benefiting the
quality of their test data or test environment. availability test environments
We can also be cautiously positive about progress on test
become completely
data management. There are plenty of expensive TDM cloud-based. Access will be via
tools out there, but they are not silver bullets; and also,
organizations are still learning how best to take advantage
one click, and they will always be
of them so their benefits can be fully felt. For instance, some available.”
organizations use such tools simply to mask the production
data, when they could instead be used to create test data Suresh Dwadasi
sets with infinite permutations to help business leaders, Delivery Head, multinational bank
developers, and testers in the design and test of business
rules.
One last point. Many QA teams think of TEM and TDM as two
distinct problems. For instance, they might seek to solve TEM
by moving it to the cloud, but their TDM is still a mess. In our
view, the transition to the cloud offers a great opportunity
to build the test data management strategy so as to get the
best possible return on their investment.

37
 TDM & TEM Create a shared center of
excellence for TEM/TDM.
call to action A shared center of excellence (CoE) should be created
to service the TEM and TDM requirements within
an organization. This team should create a strategic
framework which each of the project teams can use. The
team should define governance and the process, the
TEM hosting strategy, and the tools strategy. Depending
on the specific organization dynamics, this team can
either service all the TEM and TDM requirements, or
it could be de-centralized. But, if maturity is low, we
suggest starting with a centralized approach.

Get as much value as you can out of

your tool investment.
There have been significant enhancements of TDM and
TEM tool capabilities, but there are very few teams
that are realizing their full potential. We suggest that
concentrated effort is made to plan and implement the
various features that these tools provide, to train the
team members, and to measure and report regularly on
the benefits.

Have strong governance in place.

It’s essential to have a mature approach to the
governance of managing the complete lifecycle, from
raising the demand to de-commissioning and archiving
the test environment and test data. The QA teams
should jointly work with the TEM and TDM center of
excellence to forecast demand and achieve visibility of
what is available. It should be able to use the available
resources with ease.

38 World Quality Report I 2020-21

 TDM & TEM

Budgets and Chaitanya Joshi

Director, Financial Services
Digital Assurance and

cost containment Quality Engineering, Capgemini UK

Hitesh Patel
Regional Practice Leader
Digital Assurance and
Managing QA budgets in Quality Engineering, Sogeti USA
challenging times
Jeba Abraham
Regional Practice Leader
Digital Assurance and
Quality Engineering, Sogeti USA
Shiva Balasubramanian
Vice President, Financial Services
Digital Assurance and
Quality Engineering, USA

Continuous Quality is
When Benjamin Franklin said, “In this world nothing can be
testing pays imperative to the bank, said to be certain, except death and taxes,” he should have
dividends and to its customers added budget control to the list.
and regulator, which The certainty of it was true then, and it’s true now.
is why we focus on consistently high It’s especially true for a modern business. Successful
organizations focus on being more efficient by doing more
quality across everything we do. As part with less, rather than cutting down on their key initiatives and
of this, we’re now building testing back saving money. However, situations such as this year’s COVID-
19 pandemic have resulted in even more focus on budget
into the very early phases of design and control. IT budgets have seen tighter scrutiny, and this has
development, and we’re taking it all the had a cascading effect on the quality assurance (QA) budgets.

way through implementation and post- There has been greater emphasis on being more efficient
production too.” as the digital business grows rapidly, especially in retail,
e-commerce, media and entertainment, and healthcare. New
platforming such as CRM, ERP, and native cloud and mobile
Barry Blake app development in these sectors has meant that QA teams
Senior Manager, QA & Testing, have to do more.
Permanent TSB
At the same time, new initiatives are being put on hold in
several sectors. The twin effect of being more efficient and
of cutting down on new initiatives has been to reduce IT
spend, and QA budgets have been proportionally impacted.

39
 Q14. According to your estimate, what percentage of your total IT budget is allocated to the Testing &
Fig X
QA function (including testing processes, tools, and resource costs)?
Fig 14 Percentage of IT budget allocated to QA activities

35%

31%

26% 26% 26%

23% 23%
22%

2020 2019 2018 2017 2016 2015 2014 2013

Responding to the challenge

– learning from best practice
Smarter organizations have continued to invest in new Skills: We’re seeing an
products and services, and in the QA function that goes benefiting evolution in the skill
with them. Digital banking startups are a case in point, and
from breadth sets of our testers. We
traditional banks, recognizing this threat, have followed
suit. These organizations are investing more in their QA and say they’re becoming
engineering transformation programs in order to increase T-shaped: they have a broad
efficiency, improve their upstream product quality, and
accelerate their time to market. competence in coding and testing,
Some of the cost containment measures we have seen in
but they also have skills in the
the field include the adoption of open source tools or more specifics of the development area
efficient usage of the commercial tools; increased cloud
adoption, to help with streamlining environment challenges;
of which they are part. The goal
and the application of AI and ML-based techniques to make for us is that they must be able to
QA more efficient. perform in various tasks.”
We have also seen a focus on the end-to-end automation
of critical business process testing, although a significant Anders Lemon
investment of time and effort is needed in this case to make Head of Architecture
it work seamlessly across disparate platforms. Application infrastructure,
However, others have been caught on the wrong foot by
Development Platform and Quality Process
the crisis, notably the organizations that are still at a low Försäkringskassen
maturity with respect to their digital transformation. For
example, we heard how some insurance companies, who had
not yet implemented digital transformation measures when
lockdown began, were unable to do claim processing because
their workforce wasn’t physically able to process it.

40 World Quality Report I 2020-21

 Budgets and cost containment

Centralized The testing

testing resources we’ve
function helps embedded in
in increasing development teams
efficiency can sometimes be
overloaded, so our
centralized testing function can
provide more bandwidth. Automation can
help greatly in this respect, too, of course.
The extent to which it’s used varies by
product – I’d say from 40% all the way up
to 80%. Some of it is full automation, and
some of it is at what might be termed an
assistance level – but even on this reduced
basis, it increases efficiency and saves a
great deal of time.”
Chris Trimper
Enterprise Test Automation Architect,
Independent Health

The It’s more difficult

boundaries these days to track
are blurring the movement of QA
budgets specifically as
an individual component. This is because
the budget supports the overall team: the
boundaries are blurring, and there is less
delineation between different activities
performed and the people who perform
them in the agile environment.”
Andy Armstrong
Head of Quality Assurance & Testing
Nordea Bank

41
Budget breakdown
– and efficiency considerations COVID has
Cost-cutting
Every year, we ask organizations to estimate the proportion
in QA can be significantly increased
of their total IT budget that is allocated to testing and QA
– and the downward trend that has continued in this year’s risky our online business.
responses too (see Fig 14). However, we also need to factor in I expect to see an
the effect of COVID-19. It’s likely that, while overall budgets
are down because of the pandemic, the testing and QA increase in our budgets for
budget has actually flattened out this year. performance testing and for
As in previous years, we asked the respondents how much security testing.”
QA budget is spent on hardware and infrastructure, on
software tools, and on QA staffing, and how the budgets Leon van Niekerk
are moving overall in each of these three areas (see Fig 15). Head of Test Center of Excellence,
The proportion of budget allocation hasn’t changed much, Pick n Pay
with hardware and infrastructure taking almost 45% of
the QA cost, followed by software tools, leaving about a
quarter of the budget being spent on QA staffing. However,
we see greater emphasis being placed on reducing the
human resources budget rather than the hardware and
infrastructure budget.

Q15. What percentage of the Testing & QA budget is used for hardware & infrastructure,
Fig15
Fig X Breakdown of QAresources?
budget
tools and human

Mean average 2020 2019 2018 2017 2016 2015 2014 2013

45
44
44
Hardware and infrastructure 46
40
37
33
40

31
30
31
33
Tools (software licences) 29
30
32
28

25
26
26
21
Human resources 31
33
35
23

0 10 20 30 40
42 World Quality Report I 2020-21
 Budgets and cost containment

In a new question this year, we asked organizations to rate Summary

the importance of different aspects of potential testing
efficiency (see Fig 16). The high levels of emphasis given to The QA budget continues to be flat, with a slight downward
shift-left testing, to intelligent autonomous solutions, and to bias. COVID-19 has put further pressure on IT budgets in
the enhanced provision of test environments and test data, general, and QA budgets are not immune. At the same time,
are all consistent with what we have observed in the field. the proportionate allocation between key cost components
of the IT budget hasn’t changed much. A large portion

Q16. On a scale of 1 – 7 (where 1 = not at all important and 7 = essential) rate how important the
Fig16
Fig X Mechanisms
following used are
aspects to make testing
towards more
making efficient
testing more efficient

% of respondents saying 'essential' or 'very important' Total

Shift test left (test earlier in the process) 54

Implement intelligent autonomous

test solutions
54

Enhance Test environment provisioning

solutions for teams 52

Enhance Test data generation and provisioning

solutions for teams 41

Focus on increasing quality of

software development 40

Shift test right (test less during development

and focus more on quality monitoring) 40

Remove redundant tests 34

Increase the level of test automation 31

Be more critical in early stage which test

are really required 30

Blurring boundaries continues to be spent on test infrastructure and tools.

As we have seen in previous years with the high adoption At the same time, the pace of digital transformation
of DevOps, the boundaries between development, QA, and has accelerated, and that has made quality even more
operations are blurring. There are instances where in-sprint important. Smarter organizations are using this opportunity
functional testing is being done by the developers, and to bring in higher efficiency in their QA practices. Also,
only the specialized testing such as performance testing, or while it is difficult now to measure the QA budget exactly,
security testing is conducted by dedicated QA teams. There because of the blurring boundaries between development,
is a trend towards keeping a dedicated shared team for QA, and operations, the good thing is that this blurring
activities such as test environment management, test data constitutes a recognition that the principle of quality is not
management, common framework and re-usable component an afterthought. It’s absolutely integral to every part of the
creation. So, while the dedicated QA budget may show a software development process.
downward trend, it’s difficult to ascertain how much of that
budget is now consumed by the developers doing the testing.

43
 call to action Budgets and cost containment

Greater savings can be achieved by area up front, so as to realize future cost savings in addition
using test infrastructure smartly. to the time-to-market benefits.
Maturity and adoption of cloud-based technologies,
whether these are using cloud as infrastructure, or Focus on quality not cost when it comes
virtualization, or software-as-a-service, provide an to having right skilled QA teams.
excellent opportunity for organizations to optimize Organizations are moving towards quality assurance as an
their test environment and test tool licensing costs. integral part of the software development lifecycle, and
Organizations should make an effort to have right skilled this requires testers to have development skills in addition
team members focusing on optimal utilization of these to having good domain knowledge. Traditionally the costs
resources to stay within budget. associated with testing skills have been lower than those for
development skills. But now there is an increasing need for
Use advances in analytics, AI and ML full-stack software development engineers in test, this needs
to make testing smarter. to change. Hiring people or outsourcing QA on price rather
than on quality is a false economy.
Organizations should have a solid plan in place to
improve the efficiency and effectiveness of QA using
advances in analytics, AI, and ML. We have seen several Don’t put key initiatives on hold.
use cases maturing over last year in this area. For Strive to be more efficient instead.
example, risk-based testing based on AI rather than on When times are tough – and for many businesses, that’s
the domain experience of a tester has the potential to especially the case this year – the temptation to shelve new
make testing faster and less of an effort. projects is strong.
Similarly, rather than taking a brute force approach to It may not be economically viable to do everything that
automation and spending money in automating 100% of was planned pre-COVID, but it’s still important not to hold
a regression test, it is more prudent first to optimize the back on transformation initiatives with sustainable long-
regression suite, and then automate the high impact test term returns. If you stand still, you may as well be moving
cases. Hence, it would be good to plan investment in this backward.

44 World Quality Report I 2020-21

 The impact of COVID-19...

The impact of Dhiraj Sinha

Vice President Financial Services
Digital Assurance and
COVID-19 and its Quality Engineering, APAC Region
Manish Goyal
implications on Program Manager
Digital Assurance and

quality assurance Quality Engineering, Capgemini Group

Sathish Natarajan
activities in a post- Group Vice President
Head of Digital Assurance and

pandemic world Quality Engineering

Capgemini North America

QA teams and the pandemic: strong in

will to strive, to seek, to find, and not
to yield

Agile working We were lucky to The World Quality Report has always asked some forward-
provides a solid adopt agile practices looking questions – but this time last year, no one could have
predicted that in this latest edition, we would be devoting
foundation in early on, and they lend a chapter to the effects of COVID-19 on testing and quality
tough times themselves as much assurance (QA).
It’s not just about the effects of one on the other. It’s also
to remote working as about the parallels between the pandemic and the discipline
they do to conventional environments, of quality assurance. The similarities are really quite striking.
which really helped when lockdown In both cases, we can see the importance of:

came. It helped to fortify a lot of future • The very principle of testing

successes.” • Testing early

• Automating testing, in order to achieve scale
Chris Trimper • Adopting a test strategy that is risk-based
Enterprise Test Automation Architect • Prioritizing in line with category-based needs, whether
Independent Health that be population demographics or lines of business.

Changes of direction
– and changes by degree
The changes brought by the pandemic vary substantially
by industry segment. Some sectors have been impacted
more due to their legacy working practices. For instance,
several organizations in the manufacturing sector and in
the financial services have been slow to move to cloud-
based environments, which has made it difficult for them to
maintain business operations while working from home. Also,
many industry sectors had to change their business model
almost overnight, such as major retailers whose business was

45
 built on an in-store experience, and who have had to move to
online sales simply in order to keep going.
Flexible As we emerge In other ways, however, things have changed not so much
working is from lockdown, I think in nature, as by degree. For example, working from home

here to stay… working from home is has accelerated the adoption of distributed agile, thereby
increasing the need for collaboration tools. This new working
going to be a big part of environment has also highlighted the need to achieve a
the new normal. We’ve already comprehensive perspective, with real-time status monitoring
and reporting in order to provide better visibility of QA
shown ourselves it can work. I activities.
imagine we’re only going to be We’re also seeing more crowd testing: the pressure of
having people in the building at circumstances is making it more acceptable. We’re seeing
greater need and use of data masking services. And we are,
something like 25% to 30% of of course, seeing greater pressures on costs, especially in
previous capacities, and they’ll large enterprises, even though some overheads such as office
energy costs and business travel costs have been lower. The
be given access in groups, on a pandemic has made commercial prospects harder to predict
roster basis.” – even for just one or two quarters ahead – and this has made
it hard to budget, not just for QA, but for IT as a whole. It
Geoff Meyer explains why we have seen increased interest in external
Validation Architect, Dell EMC service provision.

Changes of focus
Our survey responses give us additional and quantitative
insight. We asked VPs of applications and QA/testing
managers how they expected their focus to change in testing
and quality assurance post-pandemic (see Fig 17). Almost half
… and it’s We’ve found (47%) of them predictably said there would be significantly

productive. working from home has greater focus on customer experience validation and on
usability testing. This is certainly more relevant for any
been very productive, business moving from a business-to-business (B2B) model
both for the company and for to business-to-consumer (B2C) or to B2B2C. Accelerated
digital transformation is emerging clearly as an unintended
the team. People are saving on consequence of the COVID-19 pandemic.
their commute times, and they One of the greatest areas of impact of COVID-19 has
can also work flexibly – taking been on the supply chain. If ever an enterprise needed a
supply chain management system that was fast, flexible,
time out in the afternoon, for and comprehensive, that time is now – which is why we’re
instance, and making it up in the seeing even more intensive activity in digital transformation
programs in general, and in the transition to enterprise
evenings. They only come into solutions such as SAP S/4 HANA in particular.
the office if they want to, and Also, as expected, a high number (43%) of respondents
we’ve put measures in place for said that the focus on more and better collaboration tools
will go up, and over a third (34%) said they will need more
everyone there. remote access to test systems and test environments. While
collaboration tools were useful just a few months ago too,
We’ve seen an increase in the use they’re absolutely vital now. Usage has spiked: for instance,
of online collaboration tools such during the pandemic, Microsoft Teams grew by 894% during
a week in June, compared with its base usage in a week
as Microsoft Teams. It’s worked in February. Organizations now need to ensure that their
really well for us.” application of these tools will continue to be robust at scale.

Suvo Ghatak
Senior Manager, Quality Engineering
Wabtec

46 World Quality Report I 2020-21

 The impact of COVID-19...

Q36. On a scale of 1 – 7 (where 1 = No change in focus and 7 = Significant change in focus) how has (or will) your focus
Fig X
Fig 17 change
How hasfor each
your of the
focus following
changed when QA
towards it comes to testing
activities andthe
to meet quality assurance
challenges in the
caused bypost-COVID-19
COVID-19? scenario?

Top 2 box summary: 7 Significant change in focus + 6 2020

We focus more on customer experience

validation and usability testing
47

We need to improve the productivity

monitoring of our teams 44

We require more/better
collaboration tools for our teams 43

We focus more on performance

validation 38

We require more remote access to

test systems and test environments 34
(for example using SaaS and Cloud)

We focus more on security validation of

our applications. 31

We need to improve the orchestration

of QA activities in our teams 29

We need to focus more on disaster

recovery and resilience testing 22

Security… … and budgets

There have also been COVID-19-related QA implications CIOs and IT directors also anticipate budgetary
in the area of security. While the nature of security issues consequences. They reported an expected 13% average drop
as far as consumers are concerned hasn’t changed, the in post-pandemic testing and QA budgets.
scale certainly has. The incidents of data breach have gone
This cut may turn out to be a short-term, reactive measure,
up significantly as organizations have shifted to remote
though. While COVID-19 has indeed put pressure on costs,
operations. In some ways, as more transactions and
it has also put more of people’s lives online, which can
work happen online, the risk of cybercriminals launching
only mean that the need for robust quality assurance and
sophisticated attacks proportionately increases.
data security will increase. Current circumstances present
This, perhaps, is why 83% of CIOs and IT directors have told QA teams with the opportunity to make a strong case for
us their application security concerns have increased over greater funding.
the last 12 months. It’s why we now see a higher demand for
security testing.

47
The post-pandemic QA landscape
Has the pandemic changed perceived quality goals? It would
People People have
seem so: more than half (52%) of our respondents said that
those goals had changed a great deal, or that they needed a can turn been using lockdown
major overhaul.
challenges as an opportunity
The watchwords for QA have always been speed, quality, and
into to round out their
cost, and the trade-offs between them are ever-present. As
business begins to emerge from the pandemic, these goals opportunities skills. They’re seeking
will need to be reassessed for every individual system and additional certification
application test project.
in complementary disciplines,
There is certainly great determination to use the upheavals of
2020 as an opportunity to make positive changes in practice
so they will be ready to accept
and in mindset as far as quality assurance is concerned any challenge. Full-stack testing
(see Fig 18). Even the lowest-rated statement in the graph
shows that almost three-quarters (74%) of respondents said
isn’t just a concept – it’s a reality
more automation is needed, and the other four featured now.”
statements are at 80% or higher.
The interesting and heartening thing we witnessed is the
Suresh Dwadasi
positivity and willingness to change – to strengthen the
Delivery head, multinational bank
community and collaboration and to empower our teams
with greater responsibility. What emerges here is a more
collective, more can-do culture. It shows a determination, as
Alfred, Lord Tennyson put it, to strive, to seek, to find – and
not to yield.
Q40. Please rate between 1 and 5 (1 – strongly disagree, 5 – strongly agree) your agreement with the
Fig X
following potential changes for QA and testing as a result of the changes brought by COVID-19?

Fig 18 QA in the post-pandemic world

Top 2 box summary: 5 Strongly agree + 4 2020

We need to build a stronger

QA community culture
84

We need to measure the quality

of work of teams differently 82

We need to empower teams with

more responsibility for quality
80

We need to improve our team

collaboration tools for QA and test
80

We need to automate more 74

of QA and testing

48 World Quality Report I 2020-21

 The impact of COVID-19...

call to action Looking beyond the pandemic

Be better prepared for Don’t look at COVID-19 as a way to cut cost,

business continuity. but as an opportunity to transform.
Frequent testing for business continuity and disaster Organizations should recognize that instinctive cost-cutting
planning needs to be conducted. We can no longer in response to this pandemic carries an inherent risk. They
take anything for granted, be it working from a should instead look at re-purposing their spend to drive
secure workplace, having a co-located workforce, or accelerated transformation – not just for QA, but for IT
the availability of all systems within the enterprise as a whole. They should use this time to accelerate digital
boundaries. Being ready for any eventuality is the key. transformation and changes to the business model, making
Also, the application architecture has to be tested, and the applications flexible and more responsive to the business
to be shown to be sufficiently flexible to adapt to quick changes, strengthening security and business continuity
change in business models. practices, and promoting a new, more collaborative way
of working.
Focus more on security.
With the inherent security vulnerability of remote Continue to use the best practices
operations, and rapid changes in the way of working, it is adopted during the pandemic.
even more imperative that organizations should become Falling back on older habits is easy. We recommend that
paranoid about security. Ideally the focus should be on organizations don’t whittle away the best practices they have
preventing such incidents, but also preparedness to been adopting to respond effectively during these tough
respond to any security breaches quickly and effectively times. For example, they should continue to strengthen the
in order to minimize the adverse impact. processes adopted around remote working and operations,
making the business more digital, and making cost savings on
avoidable physical infrastructure and travel needs.

49
 Sector
Analysis

50 World Quality Report I 2020-21

Automotive 52

Consumer products, retail, 54
and distribution

Energy, utilities, and chemicals 56

Financial services 58

Healthcare and life sciences 60

High-tech 62

Government and public sector 64

Telecoms, media, and entertainment 66

51
Automotive
It’s difficult to overstate the transition that has been
taking place in the automotive industry in recent years.
What’s especially interesting about it is that many of its
leading players are beginning to think of themselves
The automotive industry continues to in an entirely new way. Just a few years ago, they were
blaze trails in the adoption of new manufacturers – but now, they regard themselves as
QA technologies – and is reinventing mobility services providers.
itself at the same time This isn’t merely playing with words. A car is no longer
purely a mechanical device. The on-board IT plays a
role that is just as important, which means that all
car-makers, from recent entrants such as Tesla to
established companies such as Volkswagen, are as
Axel Schoenwald much software companies as they are anything else. As
Head of Sector,
a result, and as we noted in last year’s report, what was
Automotive, Sogeti
once a monolithic business model is now built around a
number of different specialist participants, both inside
and outside the organization.
In-car functionality and the QA implications
We see the significance of in-car systems in this
year’s survey data. Almost three-quarters (71%) of
automotive industry respondents said that it was an
essential objective for testing and quality assurance to
contribute to business outcomes. Similarly, two-thirds
of automotive respondents (66%) said that ensuring
end-user satisfaction was essential. These objectives
are influenced considerably by the extent and usability
of on-board services. Drivers, like smartphone users,
tend to use only some of the functionality available
to them, which is why software development and test
teams need constantly to enhance the offer, including
making cloud-based functions available on demand.
The shift here is from the basic ‘Does the car work?’ to
‘Does the car do what the user expects?’ In the past,
manufacturers could sell a vehicle and, apart from
standard after-sales commitments, pretty much forget
it. Now, however, they need to remain invested in it.
A large part of customer satisfaction is wrapped up in
continuing digital happiness.
Cost-efficiency is also a big part of the picture. In
this year’s data, we see almost three-quarters of
the industry’s survey respondents (73%) said cost
optimization was an essential element of their IT
strategy. Of course, this is true for everyone – but the
automotive sector has particular challenges. Many
manufacturers have legacy systems, on which they are
holding new telematics data, and in some cases, the
worlds of factory IT and of in-car IT are set apart from
one another, making cohesion and hence efficiency
more difficult.
In general, automotive factory IT has a more mature
approach to quality than does in-car IT, simply because

52 World Quality Report I 2020-21

 Automotive

it’s better-established. This year’s survey shows a high When an industry is as keen as this one to progress in
degree of confidence about the extent to which targets customer satisfaction, in performance, in safety, and
are reached when testing key applications, and we indeed, in everything, it is hungry to learn as much
suspect that as in-car IT quality teams catch up, these as possible.
figures will be higher still. In particular, we expect to see
a growing proportion of people saying they have the COVID-19: a factor in business model disruption
necessary in-house testing environments. With so many Until early this year, we were seeing the emergence
specialist organizations now contributing to the design of new attitudes to cars in many parts of the world. In
and development of what we now need to call mobility cities especially, people were starting to think of them
services, overall test functionality will increasingly need as a service rather than as a product. In other words,
to be kept in-house, where everything can be brought instead of owning one, they were beginning to book
together and assessed in the round. one online, and use it only when needed.

Blazing the trail in new QA technologies COVID-19 has disrupted this emerging trend. Health
and approaches concerns are increasing interest in ownership once
more. For instance, this summer, Citroën launched the
It’s no surprise to note that this is a forward-thinking Ami in France, an electric vehicle just 2.4 meters long,
industry. More than half of its respondents (54%) said and 500 of them were sold in the first two weeks.
they always shift left when testing, and even more of
them (62%) said they always automate testing as much The pandemic has also encouraged the industry to think
as possible. about its future testing and quality assurance practices.
Around half of automotive respondents (49%) said
Confidence is also fairly high as far as the use of they would be needing more and better collaboration
artificial intelligence (AI) and machine learning in tools, and a higher-than-average number (37%) said
testing is concerned. In fact, it’s not just the use of AI they would be focusing more on the security validation
to test that’s a factor here – it’s also the actual use of of their applications. It’s likely that both these results
AI within the vehicle, which itself needs to be tested. have been influenced by the extent to which external
For example, on-board systems now include picture consultants, working remotely, are becoming the norm.
recognition, which is used among other things to
interpret road signs and identify potential hazards. A significant proportion of the industry (84%) also
We see over two-thirds of automotive industry strongly agreed that teams need to be empowered
respondents (69%) telling us that a new strategy is with more responsibility for quality. Implicit in this, we
needed to test AI. We also observe that a significantly believe, is an acknowledgement of the gap to which we
higher-than-average proportion of automotive industry referred earlier – that is, the gap between the factory
respondents (58%) reported that they use smart systems management and in-car developments. As
tools that predict user behaviors to help them decide automotive organizations move further into agile
what to test. This makes sense: the industry needs practices, we can expect to see the integration of
to understand how drivers are interacting with in-car these two testing environments, and the emergence
functionality so as to improve the offer. of a shared and growing commitment to quality. It will
encompass everything from in-car microservices to
Similarly, there are signs of significant investment
Industry 4.0 factory floor developments, and everyone
in test automation. High proportions of automotive
– manufacturers, suppliers and customers – will feel
industry respondents reported that they have the
the benefit.
right automation strategy, that test data and test
environments are available at the right time, and Since 1913, when Henry Ford introduced the moving
that they have the necessary test automation skills assembly line, the automotive industry has always been
and experience. a trailblazer. We see no signs of change at any time
soon.
However, the industry isn’t complacent about its skill
set. Far from it. The many test automation benefits it
reports, including shorter test cycle times and better
defect detection, seem to be maintaining an appetite
to learn more, in order to do more. Areas of especial
interest to the automotive industry in our survey
included skills in robotic process automation, in test
data management, and in test automation architecture.

53
Consumer In last year’s report, we pointed to the particular
development and testing challenges facing the
consumer products, retail, and distribution (CPRD)

products, sector. People were spending more online, which meant

businesses had to meet demands in volume while still

retail, and
enhancing the customer experience to address specific
individual needs. More function-rich applications and
more transactions meant, in turn, more pressure on

distribution quality assurance (QA) teams.

If anything, the advent of COVID-19 this year has turned
up the dial on all these challenges. It’s not just that
The COVID-19 pandemic has turned up online shopping has scaled even further and faster; it’s
the dial on quality assurance and testing also that new factors have come into play.
trends in the CPRD sector New mechanisms and models
The consumer packaged goods (CPG) market is a
case in point. Before the pandemic, companies in
this subsector often had little idea of who their
ultimate customers were. This is because the end-user
Ajith Madhavan relationship was owned not by them, but by the retailer.
Vice President, Capgemini USA But now, with retail outlets in lockdown over many
Rakesh Thaploo months, these manufacturers have realized they need
to get closer to their customers. There has been a major
Vice President, Capgemini USA
reassessment of delivery mechanisms, not just as a way
of coping during the crisis, but also for the post-COVID
future.
It’s an assessment that doesn’t provide any easy
answers. Do CPG companies continue to work with
retailers, remaining distanced from their user base?
Or do they invest in their own direct online channels,
thereby competing with their own supply chain? Either
way, and especially in the latter case, they will need
to keep retailers’ data separate from one another and
from their own. It seems to us that, even now, they are
not fully aware of how many hills they have yet to climb,
nor of how steep some of those hills will be.
Implicit in all this are highly integrated cloud-based
business models that are sufficiently flexible and
scalable to satisfy rapidly growing and changing needs.
They need to be optimized to perform during periods
of high demand – and they of course need to be secure.
All this is an area in which mainstream online retailers
are a little further ahead, and into which the rest of the
sector is moving.
Boundaries are beginning to blur. Retailers and
manufacturers are starting to interact with
consumers on common platforms, thereby creating
new ecosystems in which the end-to-end consumer
experience will be even more important. When more
consumer data is being collected and stored, there will
be a greater need for penetration testing, data security
testing, and regulatory alignment testing.

54 World Quality Report I 2020-21

 Consumer products, retail, and distribution

Quality assurance (QA) is a corollary to these general whole. This outcome could reflect the mixed picture
trends. While we have seen no major QA developments in the sector. In the fashion sector, costs are down
this year, we do anticipate great levels of test because the shops are closed – but revenue is down
automation, as well as an increasing need for customer too, of course. In the white goods market, the need
experience functionality, which of course also has for cost optimization is greater – and in food retail,
QA implications. cost issues have intensified because, while revenues
Also, the as-yet unseen global impact of COVID-19 will have shot up, basket sizes and hence profitability have
become apparent in the next year. Consumer sensitivity gone down.
towards sustainability, ethical sourcing and awareness Evolving attitudes
of natural resources will increase. Manufacturers
We’ve seen that cost optimization is significantly higher
and CPG companies will have another increased
than the norm for the CPRD sector, and in a market of
demand in front of them, which will be traceability.
heightened volume, speed, and competition, we can
Testing models will have to tie into new technology
therefore also expect to see efficiency measures much
drivers such as blockchain, which is then filtered for
in evidence. That is indeed the case. For instance, a
consumer transparency.
higher-than-average 56% of CPRD respondents rated
Customer experience – and sales shift-left testing as an essential step in this direction.
team experience Other highly rated efficiency measures include the
Indeed, and as we might expect, ensuring end-user implementation of smart autonomous test solutions,
satisfaction and a good customer experience (CX) and the enhanced provisioning of test environments
was rated as an essential objective of testing and QA and test data generation.
in this year’s survey by 44% of CPRD respondents – a All of these ratings need to be seen in the context
significantly higher figure than for survey respondents of working in an agile development environment.
as a whole, and a more popular “essential” response This year’s survey data seems to indicate that most
for the CPRD cohort than any other option, including companies in this sector have not yet fully embraced
contributing to business growth and protecting the this approach. For example, a high proportion (56%)
corporate image, both at 36%. of respondents said they have difficulty in aligning
In fact, since the pandemic took hold, we have noted the tools they should use for automated testing.
the frustration of retail teams in their keenness to Although cost optimization is important in this sector,
ensure high CX levels. In fashion and in footwear, in there is some way to go before many companies fully
particular, it has been hard for sellers to know how understand the extent to which, in an agile context,
to adapt to a changing market – a market in which testing can form an integral part of efforts to improve
customers have themselves been uncertain. We suspect returns on investment in software development.
that if the option in this question were rephrased to Similarly, our survey this year shows high levels of
ensure a good experience not just for customers, but confidence among consumer goods respondents about
for retail sales teams, the response would have been the use of artificial intelligence in testing. We see this
much, much higher. as aspirational rather than actual: there is, once more, a
lack of understanding about the need to move from a
Strategic imperatives
legacy mindset, to resolve their technology debt, and to
We would also have expected customer experience to bring more modern competencies into the mix.
be the most important aspect of IT strategy overall,
but surprisingly, the option rated as essential most Looking ahead
often in this case was enhancing security, at 42%, with Finally, looking at how testing and quality assurance
enhancing CX at 38%. This was significantly lower might change in this sector in a post-COVID world,
than the global average CX response of 43%. Even we see much determination in the responses to make
more bizarrely, 29% of consumer goods respondents improvements in areas such as building a stronger
rated faster time to market as essential, against community culture, and empowering teams with more
an overall average response of just 23%. There is a responsibility for quality.
puzzling dissonance here between lower-than-average There is much that is implicit in these aspirations.
CX ratings and higher-than-average time-to- Work still needs to be done in the adoption of tools,
market ratings, because these two factors are of technologies, and development environments,
course interconnected. and once again, it seems this needs to be better
Also curious was the fact that just 23% of CPRD understood. Service providers could perhaps
respondents rated cost optimization as an essential help companies in the sector to see these things
factor in IT strategy, against 30% for the survey as a more clearly.

55
Energy, This year, the use of the word unprecedented has
been, well, unprecedented; but in the case of the
energy, utilities, and chemicals sector (EUC), it has been

utilities, and entirely justifiable. As a result of COVID-19 and near-

global lockdown, demand for oil, gas, and energy fell

chemicals
significantly. At the height of the global lockdown, the
worldwide oil demand reduced by over 30%, and the
outlook for 2020 is likely to be 5–10%.
At the same time, Saudi Arabia and Russia flooded
In spite of challenging circumstances, the market with oil and gas, driving down prices, and
there are encouraging signs of progress creating further pressures on margins. So, massive drop
and determination among energy, utilities, in demand simultaneous with an over-supply in the
and chemicals organizations this year market. In order to deal with all this, EUC organizations
have been taking swift action on both operating
expenses and capital expenditures.
In this new climate, it’s no surprise that businesses in
this sector have realized the need to accelerate their
Randall Cozzens digital transformation efforts, in a bid to increase their
Head of North America Energy efficiency, their competitiveness, and their ability to
Utilities, Chemicals Market Unit, Capgemini adapt rapidly to changing circumstances. A key part of
these digital transformation efforts is, of course, quality
assurance (QA).

Assessing objectives…
All of this provides a good context in which to consider
the industry’s opinions, as revealed in this year’s World
Quality Report survey data. For EUC organizations,
the most important testing and QA objectives were
contributions to business growth (rated as essential
by 42% of EUC respondents); ensuring end-user
satisfaction (37%); the detection of software defects
before go-live (42%); and protecting the corporate
image and branding (33%). In fact, the corporate
image criterion is also implicit in the other three:
organizations in this sector set great store by the
degree to which their brand is trusted, which is why, in a
separate question, enhancing the customer experience
(business or consumer) and enhancing security are
rated as essential by 38% and 35% of EUC respondents
respectively. It’s also worth noting that the cost
optimization of IT was the only aspect of IT strategy not
to drop on last year’s figures: it was rated as essential by
a full third (33%) of survey participants from this sector.
The implicit caution in this market is also reflected
in responses to a question about methods of testing
digital applications. Considerably more than our survey
sample as a whole, 34% of EUC respondents said an
independent validation team checks quality before
go-live. We see this in the field, too. It’s a figure that
is probably driven by the obligations of the more
regulated industries in this sector.

56 World Quality Report I 2020-21

 Energy, utilities,and chemicals

… and agile and DevOps sense. So, too, does defect prediction, which at 40%
It was interesting to see the extent to which agile and of EUC respondents is much higher than the figure of
DevOps methodologies have been adopted in the 29% for the survey as a whole. Businesses in this sector
EUC sector. As we might expect, the use of waterfall have high duties of care with respect to safety and
has dropped from 30% last year to 19% this year – and the environment.
while agile usage is up, from 24% to 29%, DevOps This might explain why the need for data science skills,
has dropped, from 29% last year to 26% this year. mentioned by 37% of EUC respondents last year, has
This is curious: it may be that the line between these dropped to 27% this year: the industry seems to have
two development approaches is blurring for some recognized that need, and has been addressing it.
organizations. That said, it’s clear that agile is the
preferred direction of travel. Among other advantages, In fact, generally speaking, organizations in this sector
agile lends itself to the more flexible working seem fairly upbeat about their progress in AI and ML
environment of 2020. as far as quality assurance is concerned. For example,
as many as 83% of them feel their current test strategy
In a new question for this year, we asked about the will work for AI as well, and 88% enthusiastically agree
various approaches organizations use to accelerate and that AI is the strongest part of their testing activities.
optimize testing in agile and DevOps environments. As We feel these are things they would not have been able
we might expect, the most popular responses for EUC to say even as recently as two years ago.
respondents were shift-left testing, and maximizing the
use of test automation – although the figure of 52%
of respondents for this latter category seems unlikely. Coping with COVID-19
We feel test automation is not being used to its full
The global pandemic has obliged all businesses to take
potential by as many organizations as that, so it will be
stock, and many respondents to this year’s survey said
interesting to see if that figure changes next year, as
their testing and QA focus would be changing as a
people take stock of their position.
result. The most marked response from the EUC cohort
What was not at all surprising was the proportion of related to an increased focus on the security validation
EUC respondents who said they always proactively of applications: 45% of EUC respondents placed high
monitor and review production logs for incidents, and emphasis on this, against just 31% as a global average.
performance trends, in order to identify application This is to be expected: EUC cyber incidents have risen
issues and potential defects even before end-users considerably during this crisis.
might notice them. At 35%, the EUC response was
The impact on QA and testing budgets is not
substantially higher than the 28% average for our
anticipated to be as high as in other industries: an
survey as a whole. This is likely to be because of the
average budget decrease of 9% is suggested by our EUC
critical nature of this market, where safety is such a
respondents, against a survey average decrease of 13%.
high priority.
This is likely to be because of the nature of this market,
As we would expect, the desire for more skills in and in particular because of its regulatory obligations,
QA and test is high. For instance, 37% of EUC survey and also because of the importance of maintaining
participants – considerably more than the survey brand profile in a sector that is so under the spotlight.
average of 29% – said they need greater knowledge
Finally, what’s heartening about the reflections of
of test automation skills. We do see higher progress in
energy, utilities, and chemicals businesses on the
the pursuit of QA skills than in many other sectors, but
ramifications of COVID-19 is the extent to which they
to some degree, we feel aspirations are running ahead
indicate new levels of optimism. Almost nine in ten EUC
of reality.
respondents (88%) forcefully agreed they need to build
a stronger QA community culture, and almost as many
(82%) said they need to empower teams with more
The move to smartness responsibility for quality.
Energy, utilities, and chemicals businesses are
interested in the QA potential of artificial intelligence We see this as part of a bigger picture. Industry-wide,
(AI) and machine learning (ML), and we see that we sense a determination to turn the pandemic into
interest reflected in this year’s survey. Smart QA an opportunity, and to accelerate into scale in the
options highlighted by EUC respondents as highly transformation of the digital landscape.
relevant included automated root cause analysis, at
66% – much higher than the 58% survey average. In
industries as regulated as these, that makes absolute

57
Financial The financial services sector is obliged to be fast-
moving, which is why it tends to be at the forefront in
the adoption of new technologies and approaches. In
services last year’s World Quality Report, we noted the sector’s
capacity for innovation in quality assurance (QA), in
particular. We also mentioned that testing and QA were
Despite upheavals, financial services in many cases being absorbed into the mainstream of
the software development lifecycle.
organizations are continuing to evolve
QA dynamically – and to beat a path In these main respects, there are few signs of change in
for others to follow this year’s report for the financial sector, which at 19%
of our total survey cohort remains the single largest
sector we have consulted. Indeed, when asked about
testing and QA objectives, the highest-rated responses
from the sector included the contribution to business
Anand Moorthy growth; ensuring end-user satisfaction; detecting
Vice President, Financial Services, Capgemini software defects before go-live; and achieving quality
at speed. This last response is of particular importance
Nilesh Vaidya to financial services. Agile development environments
Executive Vice President can facilitate speed improvements, which is why moves
Financial Services, Capgemini in this direction are often an objective here.
These objectives for testing and QA are also important
for the IT strategy as a whole – but in this case,
factors such as the customer experience, business
responsiveness, and faster time to market are joined
by the need to enhance security, which, as one might
expect, is given the highest mean rating of all.
In general terms, we see a fair degree of optimism
as far as achieving various targets for application
development is concerned. Two-thirds (66%) of
financial services sector respondents said their
testing pretty much always covers everything that is
needed. Almost as many (63%) said their applications
development across distributed teams is almost always
well orchestrated and integrated, while 59% said their
end-to-end automation, from build to deployment, is
virtually always in place. These responses are a sign of
the high levels of maturity in this market, and they also
indirectly indicate the extent to which agile and DevOps
are being adopted.
The sector is equally upbeat about the ability of its
teams to meet various testing targets. In almost every
category – such as meeting quality goals, having the
right QA and test expertise, and having the right
testing strategy, process, or methodology – it returned
highly positive results that were higher than for survey
respondents as a whole. In only one instance was
it lower than the global average, and that, perhaps
unsurprisingly, was the availability of sufficient time
for testing.
It was interesting to see the extent to which financial
services QA teams use different methods of

58 World Quality Report I 2020-21

 Financial services

testing digital applications. Only a quarter (25%) of Looking ahead on AI – and to a

respondents in this sector said they use independent post-COVID world
validation teams to perform quality checks before A similar degree of confidence is evident with respect
go-live. By contrast, for the energy, utilities, and to the use of artificial intelligence (AI) and machine
chemicals (EUC) sector respondents, this figure was learning (ML) in quality assurance. More than four
34%. We feel this is an indicator of the extent to which in five (81%) financial services sector respondents
QA is more integrated, more optimized, and more emphatically agree that their current test strategies will
mature than is the case in some other sectors. As work for AI as well, while the same proportion (81%) felt
we noted last year, so here also: QA is becoming the that only small changes to their current test strategy
responsibility of the entire software development would be needed.
lifecycle team.
In general, our view is that the application of AL and ML
to quality assurance in this market is suitably crafted,
Development environments and is working well – but that its full potential has not
yet been reached.
Agile and DevOps have already featured in this
assessment, so it’s worth noting year-on-year This is corroborated by our assessment of responses
developments in this regard. For agile, 27% of financial to a question about plans for AI in testing. A very high
services respondents reported usage, up from 25% last proportion of our financial services sector respondents
year. For DevOps, 29% reported usage, up from 27% (93%) wholeheartedly agreed that AI is currently the
last year. And for waterfall, 20% reported usage, down strongest area of growth in their testing activities. This
considerably from 32% last year. It’s no surprise to see seems to be an acknowledgement that they are not yet
those increases for agile and DevOps, but as we can see where they feel they need to be, and that they are keen
here, the extent of the drop of waterfall doesn’t create to move ahead. Indeed, in our experience, AI is only
a balance. It could be a matter of perception: in other being implemented in pockets, rather than across the
words, the better outcomes from agile and DevOps full spectrum of QA activities.
are skewing the picture. It could also be related to the
We also asked people to look ahead to a post-COVID
extent to which respondents are validating across the
world. How did the financial services sector think QA
entire lifecycle. Whatever the case, we note that the
and testing might change? Significantly higher than
same imbalance can also be seen in the survey-wide
average numbers of them strongly agreed, first, that
figures, and not just for this sector.
they needed to measure the quality of work of teams
What approaches are QA teams using to accelerate and differently (88%); and second, that they needed to
optimize testing in agile and DevOps developments? automate more (79%). In a world in which working more
Half (50%) of financial services respondents said from home is likely to be normalized, the metrics by
they are always or almost always maximizing test which testing is assessed may need to be recalibrated.
automation. Interestingly, as many as 55% said they
are virtually always adopting shift-left techniques;
almost half (47%) said they are testing less during Taking stock
development, and focusing more on quality monitoring Overall, this year we feel that, despite the upheavals
and production testing; and a third (33%) said that 2020 has brought, quality engineering in the
they almost always implement automated quality financial services sector is continuing to evolve. It is
dashboards to enable continuous quality monitoring. becoming more mature, and more integrated, and while
These responses collectively suggest a more integrated to some extent organizations are still feeling their way
and continuous approach is developing, and that past on AI, they are moving towards a continuous testing
distinctions between QA and development are indeed environment, and they are adopting agile and DevOps
beginning to blur. What’s more, our survey suggests to help facilitate this.
there is a desire to maintain the momentum: when That said, though, we don’t anticipate the move to
identifying skills gaps, highlights for financial services agile and DevOps to be either rapid or wholesale. We
sector respondents included test automation skills rather expect a hybrid development environment, also
(34% of respondents) and development skills (31%). encompassing waterfall, to remain in place for some
These corroborate the shift we are seeing to the left, time to come.
as well as towards a continuous integration/continuous
delivery (CI/CD) approach.

59
Healthcare
In the healthcare and life sciences section of last year’s
World Quality Report, we noted that the sector was subject
to significant pressures as far as cost containment and time

and life to market were concerned. In quality assurance (QA) in

particular, we observed that the general pressure on costs
carried through to budget constraints for testing, and indeed

sciences for all of IT. Nonetheless, we said, there was a clear appetite
in this industry to invest in intelligent automation, and also in
artificial intelligence (AI) and machine learning (ML).

Healthcare and life sciences organizations In the year that has passed since then, we’ve grown
remain committed to automation and accustomed to seeing the major upheavals that have been
wrought by COVID-19, and so in the healthcare sector, it
AI in their QA efforts wouldn’t be surprising to see the same thing happening.
In fact, though, pretty much everything we see this year is
the same – pressures on cost, pressures on delivery, and
the pursuit of automation and AI in order to relieve those
pressures.
Anshu Saurabh
Practice Manager, Digital Assurance and The difference, of course, is in intensity: it seems everything
is happening harder, and faster. Healthcare and life sciences
Quality Engineering, Capgemini USA
organizations are looking for quality at speed and on a
Azfar Mallick budget, and they are turning to automation and AI to help
Vice President, Capgemini USA them get there.

Objectives, strategies, and successes

In fact, quality at speed are exactly the words used in one of
our key survey questions this year. We asked how important
various objectives were when it comes to testing and QA, and
almost two-thirds of healthcare and life sciences respondents
(64%) said that this was key. It was essential, they said,
that software should be released faster, but without
compromising quality. The three most popular responses to
this question were also to be expected: detecting software
defects before go-live (79%); contributing to business
growth (76%); and ensuring end-user satisfaction and a good
customer experience (also 76%).
We also asked people to rate the importance of different
elements of their IT strategy. The need to enhance
security was marked high, at 80%, but so, too, was the cost
optimization of IT, at 69%. Given the emphasis on quality at
speed that we just noted, we were a little surprised to see
faster time to market rank lower on this question, at 62% –
especially when time to market is directly related to higher
responsiveness to business demands, which was ranked high
by 71% of this sector’s respondents.
It was interesting to see how well healthcare and life sciences
survey participants seem to understand their success as
far as achieving their application development targets was
concerned. The categories for which they gave themselves

60 World Quality Report I 2020-21

 Healthcare and life sciences

the best ratings were, as we would expect, the completeness activities (rated highly by 68% of respondents), reduction of
of their testing, the sufficiency and availability of their tools test cycle time (64%), and reduction of test costs (63%).
and methods, and the extent to which their application
In our survey, plans for the use of automation in the coming
development across distributed teams is orchestrated and
year include the use of model-based testing tools, and also of
integrated. The categories for which fewer of them gave high
robotics automation, which we have ourselves seen used in
ratings were the achievement of end-to-end automation,
the field, especially for risk-based testing.
the clear definition of requirements, and testing efficiency
in terms of speed and cost. In all these cases, there is indeed
work still to do.
Adjusting to COVID-19
Inevitably, the focus for testing and QA has been affected
Development environments and budgets by this year’s pandemic. Respondents in this sector reported
that they expected to increase their emphasis on customer
This year’s survey also showed a gradual increase in this
experience validation and usability testing (rated highly by
sector in the use of agile and DevOps environments, and a
57%). In a world that has been working from home, it is no
marked decrease in the use of waterfall. We see these same
surprise to note that 55% said they needed more and better
trends in the field. To accelerate and optimize QA in agile
collaboration tools for their teams, and that they needed
and DevOps, techniques being mentioned by healthcare and
more remote access to test systems and test environments.
life sciences respondents include shift-left testing (used a
In both these latter categories, responses for the sector
great deal by 49%), and integrating tests as automatic quality
were significantly higher than for our survey respondents as
gates in the continuous integration/continuous delivery (CI/
a whole.
CD) pipeline (42%). We see interest in CI/CD growing in the
sector alongside that for automation, and this is reflected in Respondents felt the drop in QA and testing budgets that we
responses relating to skills needs: over a third (34%) of sector noted earlier is likely to continue, with an average predicted
survey participants said they needed to work on their skillsets decrease of 17% – one of the highest predicted sector
in these areas. drops we have seen this year. Once again, the benefits of
automation may be a factor here. Also, faster time-to-market
Given the cost containment pressures we mentioned at the
results may be reducing cycle times, and hence also reducing
outset, it is a surprise to see the extent to which QA and
pressures on budgets.
testing has dropped as a proportion of overall IT spend this
year. Last year, the figure was 25%; this year, it stands at a
reported 21%, which is one of the biggest differences we
have seen in any sector. One possible explanation is that test Looking ahead
automation is delivering cost efficiencies, and indeed, we do Many of the key points that have emerged in this year’s
see a slightly higher-than-average spend on software tool survey are trends that we have seen for ourselves in the
licenses in this sector. This theory might also be corroborated healthcare and life sciences sector as whole. Faster time to
by the number of sector respondents who see the market remains a goal; so, too, is the use of automation to
importance of increasing the level of test automation, which boost efficiency; and we also see the growing use of AI to
at over a third (34%) was also higher than average. improve and streamline processes. We don’t expect these
technologies to deliver fully zero-touch testing any time
soon, but designating that as an objective will deliver many
AI and automation benefits along the way.
A very high proportion of respondents in healthcare and life It’s all part of the sector’s interest in making continuous
sciences (86%) emphatically agreed that AI is currently the testing a reality – and in this respect, we anticipate that a real
strongest growth area in their test activities. push in DevOps will be a key enabler. Watch this space.
However, the area in which they feel they are making the
most progress is, as we have seen, test automation. Over
two-thirds (68%) of them said they have the required
automation tools, and the benefits they perceived included,
as we would expect, better control and transparency of test

61
High-tech
In the 2019 edition of the World Quality Report, we noted
high-tech was a trailblazer among sectors in its approach to
quality assurance (QA) and testing, and that is still very much
the case this year. It’s an approach that is seeing QA being
High-tech organizations continue gradually subsumed into the software development lifecycle,
with test engineers making way for multi-skilled Software
to blaze the trail with test automation Development Engineers in Test (SDETs), and with federated
and user experience testing – and they structures rather than centralized QA functions.
are making QA everyone’s responsibility Test automation is continuing to increase in significance, and
we also see increasing emphasis being placed on continuous
testing and DevOps environments, and also on user
experience testing, which is a major source of differentiation
in this industry.
Vivek Jaykrishnan
Senior Director
Digital Engineering and Manufacturing Services Business-oriented, user-oriented quality
Capgemini
The business-led nature of quality assurance in the high-
Jean-Claude Viollier tech sector is very much in evidence in this year’s survey
Head of North America High-Tech Unit responses. Contribution to business growth is ranked as a
Capgemini highly essential objective by as many as 85% of the sector’s
respondents, against 74% for the survey as a whole. End-user
satisfaction is similarly higher than average, at 74%. We also
see a marked difference between the sector and the total
cohort on the importance of supporting everybody in the
team to achieve higher quality. This is an essential objective
for two-thirds of sector respondents, against 60% for the
survey total. It’s a statistic that corroborates the principle of
the SDET approach.
We were surprised to see only half of high-tech respondents
(50%) saying they pretty much always achieve well-
orchestrated and integrated application development across
distributed teams. The cohort of respondents for this sector
includes people from hardware and aerospace industries,
and we suspect this may have skewed the figure. The high-
tech industry in general is very much accustomed to working
in distributed teams. Indeed, this holistic approach is an
essential part of what is being termed summative usability
testing, in which the usability of software design is tested
from end to end.
This year’s survey bears witness to this. It shows that user
acceptance and performance testing are conducted in high-
tech to a much higher degree than for survey respondents
as a whole. Similarly, almost a quarter (24%) of high-tech
respondents said they very frequently use crowd test services
before a software release, for in-the-field design evaluation.
The survey average for crowd testing was lower, at 19% – but
for high-tech, this is a hot area right now.

62 World Quality Report I 2020-21

 High-tech

Agile, DevOps, test automation, and AI Indeed, the benefit of test automation that this industry
In most other sectors covered in this report, the year-on- sector perceived to be highest is the reduction of test cycle
year increase in use of agile and DevOps environments has times, mentioned by 76% of our high-tech respondents.
been steady, but slight. That’s not the case here: our high- The benefit attracting the lowest response for high-tech
tech respondents reported a full four-point increase in use was the better detection of defects, at 53%. This is indeed a
of both environments, to a mean summary of 26% for agile, disadvantage of test automation: what its users gain in speed
and of 28% for DevOps. At the same time, the mean average of execution, they to some extent lose in effectiveness.
for waterfall dropped from 30% in 2019 to 20% this year.
In a trend-setting market such as this, these shifts aren’t A further downside to the growth of test automation is
surprising, although we suspect these reported figures may evidenced in a question relating to skills. A full third (33%)
not acknowledge the degree to which agile projects are being of high-tech respondents said that their organizations were
conducted within a broader waterfall legacy context. lacking in test case design skills. This is a core skill of legacy
manual testing, and in an environment that is shifting to
The sector is also setting the pace in the acceleration and automation, it tends to diminish.
optimization of testing in agile and DevOps developments.
We found that 29% of our survey respondents as a whole
said they were using tools to generate or update tests
The COVID-19 effect: more of the same
automatically – but as many as 39% of our high-tech
respondents said the same. We see this a great deal in It’s interesting to note that in the case of the high-tech sector,
the field: the use of self-healing and self-updating tests is this year’s pandemic has intensified current trends, rather
becoming much more prevalent. than changed them. Areas of focus that are predictably more
important for this sector post-COVID than for respondents as
This is why we see a significantly higher-than-average a whole include security validation, and customer experience
number of respondents in this sector assigning a high level validation and usability testing. Lower-than-average areas of
of importance to the removal of redundant tests (39%, over focus included the need for better team collaboration tools,
a survey average of 34%). The popularity of test automation and a greater emphasis on performance validation – but this
is also made clear in the importance attached to the may be because in these two areas, the high-tech sector feels
implementation of intelligent autonomous test solutions pretty much on top of things as they stand.
(64%, over a survey average of 54%), and increasing the level
of test automation (40%, over a survey-wide figure of 31%). In a post-pandemic world, the yardsticks by which the
industry currently measures itself will still be very much in
A further area in which high-tech is blazing a trail is in the use use. They reflect the trends we have seen throughout this
of artificial intelligence – not just in testing, but in general analysis, and include an emphasis on test automation, and a
application, too. Think, for instance, of all those voice- stronger, more collaborative QA community culture, in which
activated interpretative apps that can set your alarm for responsibility for quality is not assigned to a designated team,
next morning, or add milk to your shopping list, or tell you but is shared by everyone.
the capital of Mongolia (Ulaanbaatar, by the way). It’s why in
this year’s survey, we see more than three-quarters (77%) of
respondents in this sector strongly agreeing that they will
take advantage of AI technologies in testing for the testing
of AI. It’s also why 91% of them said that, whether they
are testing AI or testing with it, artificial intelligence is the
strongest growth area in their activities right now.

Time savings: the main test automation benefit

Time pressures are a perennial problem in IT, so it is at first
surprising to see that as many as three-quarters (75%) of
respondents in the high-tech sector said they have enough
time to build and maintain their automated tests, against
a survey average of just 63%. We feel this high figure may
be because of the extent to which QA teams in this sector
are adopting DevOps. Working in this environment means
these teams are likely to start writing their automated tests
very early in the cycle. This saves them time later, not least
because it makes those automated tests easier to maintain.

63
Government and In the introduction to last year’s public sector analysis
in this report, we noted that while the mechanisms of
government and of public sector organizations may

public sector vary by country or by region, what all such bodies have
in common is a duty towards the citizens they serve.
This has never been a more important truth to
Serving the public in challenging recognize. Like every other market and industry in this
times – with a new determination report, the public sector is working towards a smarter,
to change business-as-usual in more efficient, and more cohesive approach to quality
assurance (QA) – but in 2020, it’s doing so under
quality assurance the weight of public expectations surrounding the
global pandemic.
We’ll come to COVID-19. First, though, we’ll consider
the general picture, as seen by our public sector
Jill Harris respondents, who represent 15% of our total
Delivery Director, Capgemini UK survey sample.
Steve Williams Business growth – and customer experience
Delivery Lead, Capgemini UK As we also did last year, we asked participants to rate
the importance of various QA objectives. In line with
the survey as a whole, public sector respondents rated
the contribution to business growth (43%) and the
importance of the customer experience (39%) as the
two most important factors. Even so, and despite the
fact that the business growth response was up from
38% last year, the figures were lower than we expected.
The most marked difference between public sector
responses and the global total was for the importance
of protecting corporate image. The general figure was
36%, but for the public sector, it was 44%. This, for us, is
not surprising at all: the global pandemic has intensified
the spotlight on public services. Of less importance
to our public sector respondents was faster time to
market, at 21%. It’s true that overall quality is more
important in this sector than speed, but even so, we
thought this figure was somewhat low.
Generally speaking, public sector survey participants
were sanguine about their ability to meet their
application development targets. As many as 62%
of them said their testing covers all that is needed,
and 60% said end-to-end automation, from build to
deployment, is in place. In our experience, automation
in the sector is patchy, rather than end-to-end, and
there is some way to go, too, in clearly defining
requirements (47%).
Respondents were similarly optimistic about the extent
to which application development is orchestrated
(57%): we feel more effort is needed here.

Development environments and skills

Responses to preferred development environments
were mixed. A quarter (25%) of public sector survey
participants said they worked in agile, while 30% used

64 World Quality Report I 2020-21

 Government and public sector

DevOps, and 20% used waterfall. Last year, a third (33%) activities (47%). We regard all of these as key, and
of people in this category said they used waterfall. We expect to see further developments in these areas.
feel these numbers reflect the hybrid nature of the QA
environment in this sector: many individual delivery The public sector and COVID-19
groups are working in agile and DevOps, but – possibly In our experience, public sector organizations have been
without their knowledge – they are doing so within a responding well to the new operating circumstances
broader, waterfall-based context. Who is answering created by the global pandemic. The transition to
the question here may be a factor in the difference working from home was largely smooth, and there
in the figures – and another factor may be the lack of have been few blips in service. Team members have
professional test expertise in agile teams, cited by 43% embraced the need to communicate, and to work as
of public sector respondents. a group.
Indeed, deficient skills in general identified by this We’ve also noted how the ground rules for design,
sector all average out at around 25% of respondents, development and QA have shifted. Solutions are built
and include test case design skills, collaboration skills, to be simple and streamlined. It’s good, outcome-
and data analytics and AI skills. Knowledge of business oriented work, rather than innovation for its own
processes also featured, with 24% of public sector sake, and a key part of this is an increased focus on
respondents, which is a little disconcerting: business understanding the entire customer journey, building
knowledge is necessary in order to know what to test. and testing integration end-to-end so everything will
work first time.
Overall, these skills requirements tell us that what’s
needed is a plan – a means by which these gaps can be Public sector bodies have been quick to recognize these
identified and filled in an orchestrated way. There is benefits, and are looking at how the new working
clearly work to be done here. practices they’ve put in place in difficult circumstances
can become part of business-as-usual once pandemic
Testing efficiency and automation measures are eased.
In a new question for 2020, we asked people to rate
The survey data endorses our analysis. When asked to
various approaches to increasing testing efficiency.
look beyond COVID-19, just over half (51%) of public
Well over half of public sector respondents (58%) rated
sector respondents said they were highly likely to be
shift-left testing as essential or as highly important. This
focusing more on customer experience validation
is a trend we see in the field, and incidentally, we also
and usability testing. This was a higher response than
see some emphasis on shift-right testing, where lessons
for survey participants as a whole, and indeed, other
learned in production are played back into design
forward-looking statements were also higher for the
revisions and updates. Other high-scoring responses to
public sector, including improving team productivity
this question included test environment provisioning
monitoring; acquiring more and better collaboration
and test data generation and provisioning, which are
tools; and improving the orchestration of QA activities
indeed major factors for the public sector organizations
across teams. These trends are to be expected: more
we encounter.
than in many other kinds of organization, public sector
For test automation, the public sector picture is fairly bodies need to be seen to be delivering well, and in
positive, and is broadly in line with our expectations. particular, in terms of the customer experience.
Almost two-thirds (64%) said test data and test
Finally, we asked our respondents to take stock in a
environments are available at the right time, and 62%
broader way. Almost nine in ten (87%) public sector
said they have the right automation tools. As many
participants strongly agreed that they needed to build
as 70% of public sector respondents said they have
a stronger QA community culture. Over three-quarters
the right automation strategy – but of course, it’s
(78%) of them also strongly agreed that they need to
the implementation rather than the strategy that is
measure the quality of their teams’ work differently,
a challenge, particularly when there are pressures of
and also to empower teams with more responsibility for
time, a constant imperative to deliver policy change,
quality.
and a relentless requirement to modernize the sector.
It’s all true. What the pandemic has brought about is
The three most frequently cited automation techniques
not a blip. A change is needed to business-as-usual,
that public sector respondents are planning to
and we can expect to see public sector organizations
use this year were model-based testing tools (47%
continue to put their commitment in this respect at the
were extremely or highly likely), test environment
front and center of their QA efforts.
visualization (46%), and robotics automation for test

65
Telecoms,
In markets such as telecoms, media and entertainment (TME),
it’s highly likely that most of the trends we observed last year
will be true again not just now, but for the foreseeable future.

media, and For instance, in 2019, we noted that these were sectors
that were highly customer-driven, and that were therefore
complex. We also saw that they were under high time-

entertainment to-market pressures. We observed growth in the sector’s

installed base of internet of things (IoT) devices, which had
quality assurance (QA) implications, and we noted a need for
test automation to help address this workload.
TME organizations need to
develop a more cohesive strategy to None of these challenges are likely to diminish any time soon.
The one challenge of 2019 that is less of an issue at present
quality assurance – and they are is the effect that the advent of 5G might have on the QA
determined to do so picture. It’s a development that hasn’t really scaled yet, and so
we have yet to see any large-scale consequences for quality.
On the flip side, there is one challenge being faced by QA
teams in the TME sector this year that wasn’t an issue at all in
2019. That, of course, is the effect of COVID-19. We’ll return
Chris Landry to that later.
Vice President
Telecommunications, Media & Technology Objectives and successes
Capgemini
The perennial challenges we’ve noted perhaps explain why
Lior Auslander one of the most important objectives in our survey was
Vice President judged by our TME respondents to be to make QA and testing
a smarter, automated process (rated highly by 70% of the
Telecommunications, Media & Technology
sector’s participants, against a survey-wide average of 62%).
Capgemini
Telecoms systems are complex, and are highly coupled from
the back to the front end: it’s almost impossible to isolate
a system in this environment. What’s more, these systems
are disparate: many of them are new, but some of them
are legacy. It’s little wonder test automation is given such a
high priority.
How well do QA teams in the TME sector feel they are
doing in the achievement of their application development
objectives? They seem to feel pretty successful: against
most options, they posted higher-than-average ratings.
These options included the completeness of their testing;
the sufficiency and availability of their tools and methods;
the achievement of end-to-end automation; and the clear
definition of requirements. Perceived success in testing key
applications is also high.
In our view, the scores given are optimistic. While it’s true
that many organizations are achieving “wins” in individual
parts of the testing and QA process, that doesn’t necessarily
mean that the general successes being logged here can be
inferred. Nor does it necessarily mean that work is being
conducted in accordance with an overall plan. This may, in
turn, explain why TME survey participants posted lower-than-
average responses when asked about how efficient their
testing was in terms of speed and cost.

66 World Quality Report I 2020-21

 Telecoms, media and entertainment

Agile and DevOps: This increased activity may explain why the effect of the
more strategic thinking needed pandemic on QA budgets is anticipated to be less severe than
in some other sectors. Two-thirds (67%) of our respondents in
We see a gradual rise in TME organizations in their reported
these industries said they expected the budgetary impact to
use of agile and DevOps environments, and a simultaneous
be between zero and a decrease of 10%.
decline in their use of waterfall. This is broadly in line with
the trends we see across our survey as a whole – which, One of the most interesting areas for us was the extent
considering the complexity we have noted in these industries, to which TME organizations now seem to be reevaluating
is quite a heartening development. their approach and their goals in the context of COVID-19.
A higher-than-average proportion of them strongly agree
However, it’s not without its challenges. In agile
that they need to empower teams with more responsibility
development, more than half (53%) of TME respondents said
for quality (82%); that they need to build a stronger QA
they had difficulties with aligning the tools for automated
community culture (87%); and that they need to measure the
tests, and almost as many (48%) said they had a lack of
quality of work of their teams differently (also 87%).
professional test expertise in their agile teams. These
indicate a tactical approach, swapping tools in and out to see Until now, we’ve been growing accustomed to observing
what works, rather than working strategically, within a single, what might be termed a shop-floor mentality in QA in the
consolidated ecosystem. TME sector, where teams achieve outcomes via case-by-
case workarounds, and where a sense of overall success is
Our survey also shows that many TME respondents are trying
unreliably derived from these individual instances. But what
various ways to accelerate and improve testing in agile and
we’re seeing here in these responses seems to be a shift in
DevOps environments. Faster testing can, of course, be
attitude. TME organizations may have started to focus more
achieved in one of two ways: either simply by doing things
on the means by which they get to their destinations, rather
more rapidly, or by reducing non-productive time as much as
than on the mere fact of their arrival. In short, we may be
possible. This can be hard to do, and many of the routes being
seeing an evolution from a set of rough-and-ready QA tactics
adopted – such as integrating tests as automatic gates in the
to a more cohesive QA strategy.
CI/CD pipeline, or finding and removing redundant test cases
– can be hard to measure.
Summary
Moving forward in automation Last year, we concluded by saying that there were challenging
but exciting times ahead for TME organizations. At the time,
A higher number of TME respondents than our survey
of course, no one knew quite how challenging the last year
average reported they are getting a return on their
would be.
investment in test automation. We feel that, in this complex
environment, it is perhaps a little easier for some TME The good news is that, even though there are signs that the
organizations to find some low-hanging fruit. However, as industries have been over-optimistic in their assessments of
we have already noted, successes in individual areas don’t themselves, there are also indications that a greater capacity
necessarily denote a comprehensive plan. for clear-sightedness is emerging.
Other automation benefits mentioned by TME survey Telecoms, media and entertainment businesses are very
participants include better control and transparency of can-do. It’s in their DNA. If they can couple this with a more
test activities, and a reduction in test cycle times. Curiously, strategic and holistic approach to quality assurance, the
however, fewer of them noted reductions in test costs, and future could look very promising.
better test coverage. It’s hard to gauge how much of this is
real, and how much may be wishful thinking.
It’s possible the picture may change over the year to come:
when asked about their automation plans, more than
half (57%) of our respondents from these industries said
they would be using model-based testing tools. This is an
approach we have already started to see in practice.

COVID-19: re-evaluating the QA approach

We’ve seen some of the effects of COVID-19 on the TME
sector for ourselves in the field. Externally, for instance,
customer organizations have been obliged to reconfigure
their call centers to enable staff to work from home, which
has had significant implications for telecommunications
QA teams. Internally, we’ve seen many of last year’s
capex projects shelved in order to free up budget to fix
operational issues.

67
 About
the study
World Quality Report 2020-21

The World Quality Report 2020-21 is based on For this year’s research, we selected only
research findings from 1,750 interviews carried out organizations with more than 1,000 employees (in
during June and July 2020 using CATI (Computer the respondent’s national market) – an approach
Aided Telephone Interviews). The average used for the last five years to provide us with valid
length of each interview was 30 minutes and the trending data.
interviewees were all senior executives in corporate
Research participants were selected so as to ensure
IT management functions, working for companies
sufficient coverage of different regions and vertical
and public sector organizations across 32 countries.
markets to provide industry-specific insight into
The interviews this year were based on a the quality assurance and testing issues within each
questionnaire of 40 questions, with the actual sector.
interview consisting of a subset of these questions
With the inclusion of product heads/CTOs and VP/
depending on the interviewee’s role in the
Directors of Research & Development, we are able
organization. The quantitative research study was
to bring in their views and insights in the space of
complemented by additional in-depth interviews
product, engineering and digital manufacturing
to provide greater insight into certain subject areas
services for the automotive, healthcare and life
and to inform the analysis and commentary. The
sciences, and high-tech sectors.
main themes for all survey questions remained the
same, though a few objective responses were also The research sample consists mainly of senior-level
added for the first time this year. Quality measures IT executives.
were put in place to ensure the questionnaire was
To ensure a robust and substantive market research
understood, answered accurately and completed in
study, the recruited sample must be statistically
a timely manner by the interviewee.
representative of the population in terms of its size
and demographic profile.

68 World Quality Report I 2020-21

 About the study

The required sample size varies depending on During the interviews, the research questions
the population it represents – usually expressed asked of each participant were linked to the
as a ratio or incidence rate. In a business-to- respondent’s job title and the answers he/she
business (B2B) market research study, the average provided to previous questions where applicable.
recommended sample size is 100 companies. For this reason, the base number of respondents
This is lower than the average sample size used for each survey question shown in the graphs is not
for business-to-consumer (B2C) market research always the full 1,750 sample size.
because whole organizations are being researched,
The survey questionnaire was devised by Digital
rather than individuals.
Assurance and Quality Engineering experts in
As mentioned above, the B2B market research Capgemini, Sogeti and Micro Focus (sponsors of
conducted for the World Quality Report 2020- the research study), in consultation with Coleman
21 is based on a sample of 1,750 interviews from Parkes Research. The 40-question survey covered
enterprises with more than 1,000 employees (25%), a range of software quality engineering and digital
organizations with more than 5000 employees assurance subjects, enriched by qualitative data
(34%) and companies with more than 10000 obtained from the additional in-depth interviews.
employees (41%). The approach and sample size
used for the research this year enables direct
comparisons of the current results to be made with
previous research studies conducted for the report,
where the same question was asked.

69
Interviews by sectors

19% 15% 13% 10% 8%

Financial Services Public Sector Telecommunications, Consumer Products High-tech,

industry, including /Government Media and and including hardware
Capital Markets, Entertainment Retail/Distribution vendors
Banking and and Logistics + Aerospace and
Insurance Defense

8% 8% 7% 6% 6%

Healthcare Automotive Energy, Utilities, Manufacturing Transportation

and Life Sciences and Chemicals

Interviews by job title

25%
CIOs

20%
IT Directors

19%
QA/Testing Manager

17%
VP Applications

7%
CMO/CDO

6%
CTO/Product Head

6%
VP/Director of R&D

70 World Quality Report I 2020-21

 About the study

Interviews by regions 305

Western Europe

175
Nordics
125
Southern EU 170
90 South East Asia
Eastern Europe

1750
Total

350
North America

65
Middle
135 East Asia
BeNeLux
80 155 100
LATAM UK and Ireland Australia &
New Zealand

Interviews by number of employees

41%
34%
25%

1000 - 4999 5000 - 10000 10000+

71
 About the Sponsors
World Quality Report 2020-21

About About
Capgemini and Sogeti Micro Focus

Capgemini is a global leader in consulting, digital Micro Focus delivers enterprise software to
transformation, technology, and engineering empower our 40,000 customers worldwide to
services. The Group is at the forefront of innovation digitally transform. With a broad portfolio,
to address the entire breadth of clients’ underpinned by a robust analytics ecosystem, the
opportunities in the evolving world of cloud, digital company enables customers to address the four
and platforms. Building on its strong 50-year core pillars of digital transformation: Enterprise
heritage and deep industry-specific expertise, DevOps, Hybrid IT Management, Predictive
Capgemini enables organizations to realize their Analytics and Security, Risk & Governance. By
business ambitions through an array of services design, these tools bridge the gap between existing
from strategy to operations. A responsible and and emerging technologies so customers can run
multicultural company of 265,000 people in nearly and transform at the same time.
50 countries, Capgemini’s purpose is to unleash
human energy through technology for an inclusive When it comes to building and delivering better
and sustainable future. With Altran, the Group software faster, you can no longer choose between
reported 2019 combined global revenues of €17 speed, quality, and security. You need a “Quality
billion. everywhere” culture, making quality the
responsibility of everyone. Our continuous quality
Part of the Capgemini Group, Sogeti operates in and security solutions help you make this cultural
more than 100 locations globally. Working closely shift—offering ongoing and comprehensive testing
with clients and partners to take full advantage of of web, mobile, and enterprise applications from the
the opportunities of technology, Sogeti combines start. Deliver high-quality optimized experiences to
agility and speed of implementation to tailor keep, grow and expand your business by making
innovative future-focused solutions in Digital quality the technical, organization and cultural
Assurance and Testing, Cloud and Cybersecurity, all foundation of your strategy.
fueled by AI and automation. With its hands-on
‘value in the making’ approach and passion for For more information, visit
technology, Sogeti helps organizations implement www.microfocus.com
their digital journeys at speed.

Visit us at
www.capgemini.com
www.sogeti.com

People matter, results count.

72 World Quality Report I 2020-21

Thank you

Capgemini, Sogeti and Micro Focus would like to thank

The 1,750 IT executives who took part in the *All research carried out by Coleman Parkes
research study this year for their time and Research is conducted in compliance with the Code
contribution to the report. In accordance with the of Conduct and guidelines set out by the MRS in the
UK Market Research Society (MRS) Code of Conduct UK, as well as the legal obligations under the Data
(under which this survey was carried out) the Protection Act 1998.
identity of the participants in the research study
and their responses remain confidential and are not
available to the sponsors.
www.worldqualityreport.com
All the business leaders and subject matter experts
who provided valuable insight into their respective
areas of expertise and market experience, including
the authors of country and industry sections and
subject-matter experts from Capgemini, Sogeti and ©2020 Capgemini, Sogeti and Micro Focus.
Micro Focus. All Rights Reserved.

Main Report Authors Capgemini and Micro Focus, and their respective
marks and logos used herein, are trademarks or
Sathish Natarajan and Dhiraj Sinha
registered trademarks of their respective
Market Research companies. All other company, product and service
Coleman Parkes Research* names mentioned are the trademarks of their
respective owners and are used herein with no
Writer intention of trademark infringement. Rightshore® is
Robert Fenner a trademark belonging to Capgemini. TMap®, TMap
NEXT®, TPI® and TPI NEXT® are registered
Creative Design trademarks of Sogeti, part of the Capgemini Group.
Manas Kar

Proof-reader
Monica Kwiecinski

Program Management (Capgemini Group)

Manish Goyal

Marketing & Comms. (Micro Focus)

Malcolm Isaacs
Christine Ewing

73
 Previous editions
World Quality Report

2009-10 2010-11 2011-12 2012-13

1st edition 2nd edition 3rd edition 4th edition

2016-17 2015-16 2014-15 2013-14

8th edition 7thedition 6th edition 5th edition

2017-18 2018-19 2019-20

9th edition 10th edition 11th edition

74 World Quality Report I 2020-21

75
Micro Focus
Malcolm Isaacs
ADM Solutions Marketing Manager
[email protected]

Christine Ewing
Senior Director, Product Marketing
[email protected]

Sogeti
Sathish Natarajan
Group Vice President, Head of Digital
Assurance and Quality Engineering,
Capgemini North America
[email protected]

Mark Buenen
Global Leader, Digital Assurance
and Quality Engineering
Capgemini Group
[email protected]

Capgemini
Anand Moorthy Ajay Walgude
Vice President, Financial Services, Vice President, Head of Digital
Digital Assurance and Quality Assurance and Quality Engineering,
Engineering, North America Financial Services UK & Europe
[email protected] [email protected]

Sanjeev Deshmukh Dhiraj Sinha

Vice President, Digital Assurance Vice President, Financial Services,
and Quality Engineering, Digital Assurance and Quality
North America Engineering - APAC
[email protected] [email protected]

Parinita Patankar
Vice President,
Digital Assurance and Quality
Engineering – Capgemini India
[email protected]

In association with:
ODS_20201023_ManasK