ISSN 0361-7688, Programming and Computer Software, 2022, Vol. 48, No. 8, pp. 658–684. © Pleiades Publishing, Ltd.

, 2022.

Software Testing in the DevOps Context:

A Systematic Mapping Study
B. Pandoa,* and A. Dávilab,**
a
Faculty of Informatics and Systems, National Agrarian University of La Selva,
Tingo María, Huánuco, Peru
b
Department of Engineering, Pontifical Catholic University of Peru, Avenida Universitaria 1801 Distrito San Miguel,
Lima, Peru
*e-mail: [email protected]
**e-mail: [email protected]
Received June 12, 2022; revised July 18, 2022; accepted August 12, 2022

Abstract—DevOps is a philosophy and framework that allows software development and operations teams to
work in a coordinated manner, with the purpose of developing and releasing software quickly and cheaply.
However, the effectiveness and benefits of DevOps depend on several factors, as reported in the literature.
In particular, several studies have been published on software test automation, which is a cornerstone for the
continuous integration phase in DevOps, which needs to be identified and classified. This study consolidates
and classifies the existing literature on automated tests in the DevOps context. For the study, a systematic
mapping study was performed to identify and classify papers on automated testing in DevOps based on
8 research questions. In the query of 6 relevant databases, 3,312 were obtained; and then, after the selection
process, 299 papers were selected as primary studies. Researchers maintain a continuing and growing interest
in software testing in the DevOps context. Most of the research (71.2%) is carried out in the industry and is
done on web applications and SOA. The most reported types of tests are unit and integration tests.

DOI: 10.1134/S0361768822080175

1. INTRODUCTION resolve this challenge [18, 19]. On the other hand, in

the academic field, various literature review studies
The software market constantly demands strategies have been carried out where: (i) it is pointed out that
that allow it to deal with changes quickly [1, 2]. How- the concept of DevOps is not completely defined [20];
ever, these strategies must maintain quality and avoid (ii) the definitions, practices and benefits of DevOps
the costs of application downtime and failure [3]. are categorized [21]; (iii) the relevant aspects are
Although agile methods are presented as a good alter- determined [22, 23]; (iv) the factors that interrupt its
native; these do not close the cycle until the delivery adoption are identified [24]; (v) the influence on the
and operation of the software [4]. In this context, the product is presented [7]; and, (vi) in [2], a strong need
DevOps philosophy and framework extends the agile to respond quickly to the market is reported and that
methodology to deliver applications quickly and fre- DevOps helps to address this problem.
quently [5], improving performance and costs [6], and
taking care of the product quality [7]– [9]. So, with Since software testing is a critical factor for the
the support of top management [10], DevOps can rep- adoption of DevOps [25], it should be reviewed how it
resent a great opportunity for companies of any size to is being applied in the reported cases. For this reason,
gain a foothold in the market [11]. For this reason, var- this paper consolidates and classifies the literature on
ious companies have been adopting it [12] or have applied software testing in a DevOps context. The
adopted plans [13]. Also, DevOps is a key factor in the paper is organized as follows: in Section 2, the funda-
microservices architecture [14]. mental aspects of this study are presented; in Section 3,
the Systematic Mapping Study (SMS) is described; in
In the field of the software industry, the introduc- Section 4, the results of the SMS are presented; and,
tion of the term DevOps, in 2008 [15], made it possible in Section 5, the conclusions are established.
to articulate a set of practices that had already been
taking place. In particular, the continuous integration
practice that is based, among others, on automated 2. BACKGROUND
tests [16], which represents one of the vital factors for In this section, DevOps and software testing are
its adoption [17], despite long-standing efforts to briefly presented; as well as the works related to this study.

658
 SOFTWARE TESTING IN THE DEVOPS CONTEXT 659

2.1. DevOps tom-out, interface, configuration, usability, and inter-

action).
DevOps integrates the teams that are usually sepa-
In the agile context, agile tests have shown their
rated (development and operations), focusing on
benefits [33, 34], being necessary that the software-
delivering value quickly and continuously, based on 4
testers are present from the collection of requirements
dimensions [22]: collaboration, automation, measure-
[35] and maintain fluid communication, both formal
ment and monitoring. In DevOps [4], it has extended
and informal, with the programmers [36]. Regarding
the already known practices of agile methods, distrib-
automation, based on time and costs, [37] points out
uting them in 3 phases: construction phase, deploy-
that unit tests should be automated to a greater extent;
ment phase, and operation phase. In addition, it
and, to a lesser extent, user interface tests. In addition,
incorporates some existing practices such as: continu-
in [38], it is pointed out that: (i) unit testing and com-
ous integration [26], continuous deployment [27],
ponent testing are more automated; (ii) for the tests of
continuous delivery [28], and continuous testing [29].
quality characteristics such as security, reliability,
For each phase, there are software tools on the load, stress and the like, tools can be used; (iii) in rela-
market that help to carry out these activities quickly tion to functional tests, prototypes, stories, simula-
[21]. However, DevOps goes beyond just having soft- tions and the like can be carried out manually or auto-
ware tools for automation [23]. Therefore, the main matically; and, finally, (iv) the exploratory tests, sce-
objective of DevOps is to deliver value to the client, narios, usability, alpha, beta and user acceptance only
taking care of quality and based on committed teams, manually.
that is, focusing on empowering the team more than For the context of DevOps, according to the study
the individual. In consideration of this, in [30], strate- by [16], considering that DevOps focuses on speed
gies are described to gradually adapt the organization (Time to market oriented) maintaining high quality
to ensure that the teams work in an integrated manner. and low cost, it indicates that special attention is nec-
Studies on DevOps are being developed in various essary to the automation of tests during the software
axes. A fundamental axis, which still shows its evolu- development life cycle, since the time involved in car-
tion since 2008, are those referring to its definition and rying out the tests significantly affects the delivery
structure, such as: (i) in [20], which points of commu- time. In addition, [17] points out that DevOps is a new
nication, collaboration and teamwork as more used in approach in the software testing strategy that allows an
DevOps; (ii) in [22], which identifies 4 dimensions organization to improve its results, based on Lean
that are collaboration, automation, measurement and principles.
monitoring; (iii) in [21], in which it is mentioned that
DevOps emerged in agile development and that it is 2.3. Related Work
perceived as a chain of agile tools; and, (iv) in [23], the
cultural and organizational aspect is highlighted as a About DevOps, several literature review studies or
fundamental aspect in DevOps and that is included as surveys have been carried out, of which the following
CAMS (Cultural Automation, Monitoring and Shar- can be pointed out: (i) in [39], study efforts to consider
ing). Other axes are related to its applicability, tools, accessibility tests in continuous integration, noting
and quality, among others. In particular, in [2], it is that they can be integrated into the automated testing
pointed out that the pressure of the market and inter- process and represent an underdeveloped area; (ii) in
est groups for continuous change is a factor that moti- [40], they review the literature looking for gaps in
vates the adoption of DevOps; in [24] points out the DevOps, and their results point out that there are
existence of 5 categories of factors that inhibit the important challenges such as the automation of test
adoption of DevOps, and recommends proposing selection and specification, as well as the existence of
solutions that help take these factors into account; and strategies and frameworks for test automation; (iii) in
in [7], points out that DevOps practices of continuous [41], it is pointed out that more research is required for
deployment, automation of software tests, team coop- access control tests in the field of security, from the
eration, cloud computing and monitoring and feed- perspective of security and software engineering, and
back influence the quality of the software product. proposes a guide for integrating this type of testing into
the software process; (iv) in [42], it is pointed out that
inadequate tests are the most common technical rea-
2.2. Software Testing in Agile and DevOps Context son related to the breakdown of the software construc-
tion and that there is a lack of time to write and execute
Software testing [31] are activities in the software tests; (v) in [43] it is pointed out that performing tests
development process to determine that the software are one of the greatest challenges in continuous deliv-
has the expected behavior under a list of test cases. ery, that the most common tests are unit, integration,
Tests can be categorized, according to [32]: (i) object functional, load and stress tests, and that most fre-
of the test (unit, integration and system); and (ii) test quent non-functional tests in continuous delivery is
objective (acceptance, installation, alpha, beta, maintainability, performance and load; (vi) in [44], it
regression, performance, security, load, recovery, bot- is pointed out that the biggest problem is unitary and

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

660 PANDO, DÁVILA

integration testing, due to ambiguous or weak tests, RQ-5 What programming languages and software
and that a critical problem is the design of systems testing tools are used in DevOps? Possible answers, at
considering the tests, so, among others, they recom- least initially, are: Java, C, PHP, JS, Xunit, Selenium.
mend the use of the test driven development (TDD) RQ-6 In what types of applications are software
method; (vii) in [45], based on interviews and surveys, testing used in the DevOps context? The possible
it is pointed out that 25% have fully automated pipe- answers, at least initially, are: web, desktop, console,
lines and 45% have them partially automated, and mobile.
points out that one of the factors that prevents achieve
automation, is that those involved in acceptance test- RQ-7 What infrastructure tools are used for soft-
ing do not have the knowledge for this purpose; (viii) ware testing in DevOps? Possible answers are: Jenkins,
in another study [46], based on an SLR, points to soft- Travis, Docker, AWS, Azure.
ware testing as a critical factor for continuous integra- RQ-8 In what types of activities do software testing
tion and points to Jenkins and Git as the most frequent occur in DevOps? Possible answers are: Continuous
tools for continuous integration; and (ix) in [47], they Integration, Continuous Deployment, Continuous
are categorized into challenges of people, practices, Delivery. Also, are security tests mentioned?
tools and infrastructure in the context of DevSecOps
and the need to automate tests that are currently done
manually is pointed out, taking into account the bal- 3.2. Search Query
ance between fast delivery and security testing. Searches were performed according to a generated
search string of the population (P) and intervention (I)
as suggested [48]. The terms related to (P) are:
3. RESEARCH METODOLOGY DevOps, Continuous Integration, Continuous Test-
In this study, a Systematic Mapping Study (SMS) ing, Continuous Deployment, and Continuous Deliv-
was performed. The SMS proposed by [48] is a ery. The term related to I is: test. Then, the search
research technique to identify and characterize all string stayed as “P and I”: “(DevOps OR “continuous
available studies on a given topic, using a reliable and integration” OR “continuous deployment” OR “con-
verifiable methodology. tinuous delivery” OR “continuous testing”) AND
test*.” Although a string in English was searched,
papers written in Spanish and Portuguese were also
3.1. Scope and Research Questions considered. Also, to allow for as many results as possi-
ble, the date was not restricted. The digital databases
Software testing is one of the pillars to encourage are: IEEE Xplore, SCOPUS, ScienceDirect. ACM
good results in DevOps contexts [5, 8], and on which Digital Library, Web of Science and Willey, selected
various publications have been made that require for their scientific relevance and access to them.
identification, studied and classified. For this reason,
an SMS was performed with the purpose of identifying
the levels of software tests that are being used in these 3.3. Data Selection
contexts, as well as the authors, their evolution and the The selection process was defined in four stages,
regions where the subject is being investigated, among where the inclusion criteria (IC) and exclusion criteria
others. The research questions and considerations for (EC) are applied (see Table 1); and according to [48]
the answers are: the quality assessment is omitted since relevant digital
RQ-1 What is the evolution of the publication of databases were chosen. The defined selection process
papers on software testing in the DevOps contexts? has the following stages:
The year of publication was taken as relevant data. • In the first stage, obtaining the metadata, the
RQ-2 What kind of research has been done in soft- EC.1 and IC.2 criteria are used, and the Parsif.al web
ware testing in DevOps? The types of research, application to facilitate some operations, such as dis-
adapted from [49], are: (i) survey/interview, (ii) case carding duplicate papers in the different databases.
study, (iii) multiple case study, (iv) replication study, • In the second stage, the title is read and EC.2 is
(v) review or literature mapping, and, (vi) background applied, to rule out papers that are not related to the
theory. subject of software testing in the DevOps contexts.
RQ-3 What kinds of proposals have been presented • In the third stage, reading the summaries, IC.2,
on software testing in DevOps? The types of proposals IC.3, EC.3 is applied.
are an emerging classification and can be: methods, • In the fourth stage, a quick reading is made of the
tools, frameworks. content of the study to determine its relevance to the
RQ-4 What levels of software testing are used in subject of software testing in DevOps contexts and cri-
DevOps? The possible test levels, depending on the teria IC.2, IC.3, EC.3, and EC.4 are applied. Like-
object of the test, are: unit, integration, user, security wise, at this stage, the papers to which the full text is
and load/performance [32]. not available (EC.5) are withdrawn.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 661

Table 1. Inclusion Criteria (IC) and Exclusion Criteria To extract the data, a file was created (see Table 2)
(EC) to be used in a spreadsheet and collect the data from
Id Criteria the papers on it. For the independent classification of
the topic (a more generic classifier), the categorization
IC.1 IC.1 Paper in indexed journals or conferences presented in [49] was used, so the papers obtained
whose memories are indexed were classified into (i) survey/interview, (ii) case
IC.2 IC.2 Paper with content in English, Spanish or
study, (iii) multiple case study, (iv) replication study,
(v) literature review or mapping, and (vi) background
Portuguese
theory. For the specific classification of the topic, the
IC.3 IC.3 Paper that focuses on software testing in the one presented in the SWEBOK [32] was used, where it
DevOps context indicates that software tests are categorized according
EC.1 EC.1 Duplicate article to levels according to the object and objective of the
tests.
EC.2 EC.2 Paper outside the topic of software and
DevOps
EC.3 EC.3 Paper that does not mention software test- 4. RESULTS
ing levels or strategies The searches in the considered databases were car-
EC.4 EC.4 Secondary or tertiary articles ried out between June and July 2021. For each data-
EC.5 EC.5 Paper whose content is not available base, the search string was adapted according to its
own rules (see Table 3). Of the 3,312 papers found, it
was processed stage by stage until reaching a total of
In planning the SMS, an iteration (“iteration 0”) 299 primary studies. The process was based on the
inclusion and exclusion criteria according to the study
was carried out to help with the definition of the search planning. Table 4 shows the number of papers that
string, review the selection process (with inclusion and remained after each stage. In addition, 15 (5%) papers
exclusion criteria defined), and identify a preliminary were withdrawn because the full text was not available,
set of studies for analysis, verify that the research ques- even after having searched different sources. The list of
tions could be answered adequately. primary studies is available in Appendix A.

Table 2. Structure of the data extraction form

Data Detail Question
Id Study Unique identifier of the study created for the MSL General
Title Title of the paper RQ-1
Author List of authors of the paper RQ-1
The year Year in which the paper was published RQ-1
Type of publication Journal or conference where the paper was published RQ-1
Country Country of affiliation of the authors RQ-1
Research type Categorizes the type of research of the paper RQ-2
Context Categorizes between the academic or industrial context of the paper RQ-2
Domain Categorizes the business domain where the item was applied RQ-2
Type of proposal Categorizes the type of proposal of the paper, if applicable RQ-3
Test Level Categorizes the test levels mentioned in the paper RQ-3, RQ-4
Continuous phase Categorizes the continuous phase mentioned in the paper RQ-4
Method Identifies the method or good development practices RQ-4
Testing tool Identifies the testing tool used RQ-5
Version Control Identifies the tool used for code version management RQ-5
Programming language Programming language mentioned in the paper RQ-5, RQ-6
Type App Type of software developed in the paper RQ-6
Architecture type Type of the architecture of the application developed in the paper RQ-6
Infrastructure tool Collects the infrastructure tools used in the research presented in the paper RQ-7
Security Identifies if the paper mentions the security tests RQ-8
Teams in DevOps Identifies if the paper addresses Devs, Ops or both teams RQ-8

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

662 PANDO, DÁVILA

Table 3. Database search string

Source Search string Quantity
IEEE ((“All Metadata”:Devops) OR (“All Metadata”:"Continuous Integration") OR (“All
Metadata”:“Continuous Deployment”) OR (“All Metadata”:“Continuous Delivery”) 529
OR (“All Metadata”:“Continuous Testing”)) AND ((“All Metadata”:Test*))
Scopus TITLE-ABS-KEY ((devops OR “Continuous Integration” OR “Continuous Deploy-
1.561
ment” OR “Continuous Delivery” OR “Continuous Testing”) AND test*)
ACM Title: ((Devops OR “Continuous Integration” OR “Continuous Deployment” OR “Con-
tinuous Delivery” OR “Continuous Testing”) AND Test*) OR Abstract:((Devops OR
“Continuous Integration” OR “Continuous Deployment” OR “Continuous Delivery”
246
OR “Continuous Testing”) AND Test*) OR Keyword:((Devops OR “Continuous Inte-
gration” OR “Continuous Deployment” OR “Continuous Delivery” OR “Continuous
Testing”) AND Test*)
Science Direct Title-keyword-abstract (Devops OR “Continuous Integration” OR “Continuous Deploy-
462
ment” OR “Continuous Delivery” OR “Continuous Testing”) AND Test
Web of Science TITLE-ABS-KEY ((devops OR “Continuous Integration” OR “Continuous Deploy-
432
ment” OR “Continuous Delivery” OR “Continuous Testing”) AND test*)
Willey TITLE-ABS-KEY ((devops OR “Continuous Integration” OR “Continuous Deploy-
82
ment” OR “Continuous Delivery” OR “Continuous Testing”) AND test*)
Total 3.312

4.1. RQ-1 what is the Evolution of the Publication who also found a high percentage (20%) of papers at
of Papers on Software Testing in the DevOps Contexts? the industry level. Of the remaining group of research
From the selected primary studies, from 2011 to types, it can be pointed out that those related to opin-
Jun-2021 (see Fig. 1a), it is observed that the level of ion-research allow concepts, ideas, lessons to be pro-
publications has been increasing from the beginning, posed when dealing with software testing in DevOps.
which shows the importance of software testing in Likewise, the result of the research context shows that
DevOps contexts and that coincides with those indi- 213 (71.2%) according to Fig. 2b, are papers in the
cated by [42]. In addition, this growth is expected to industry, compared to 29 (9.7%) are papers in aca-
continue in the following years. demia; which reinforces the idea of the previous result.
Although the topic of DevOps is of global impor- Finally, from the perspective of the application
tance, it can be seen (see Fig.e 1b) that according to domain (see Fig. 2c), 185 (61.8%) papers have been
the Pareto rule 80% of the studies are concentrated in applied to commercial solutions, that is, applications
16 countries: USA (16.7%), Germany (10.7%), India to sell products, rent services, etc. Likewise, an inter-
(9.4%), Italy (6%), Canada (5%), Switzerland (4.7%), esting focus is seen in the education sector, where 27
China (3.7%), Sweden (3.7%) Australia (3.3%), Fin- (9%) primary studies have focused on applications for
land (3.3%), Brazil (2.7%), UK (2.7%), the Nether- education (support for the teaching/learning process).
lands (2%), Spain (2%), Ireland (2%), Korea (1.7%),
and Belgium (1.7%). 4.3. RQ-3 what Kinds of Proposals Have Been Presented
On the other hand, the publication media where on Software Testing in DevOps?
they have been published 4 or more primary studies In Fig. 3, it can be seen that 216 (72.2% primary
are 14 media and are presented in Table 5. studies) propose tools to support DevOps contexts,
incorporating software testing as part of them. Fur-
4.2. RQ-2 what Types of Research Have Been Done thermore, 40 (14%) and 3 (1%) papers propose meth-
on Software Testing in DevOps?
From the primary studies, on types of research (see Table 4. Search results by stage
Fig. 2a), there are two predominant types of research
Procedure Selection Criteria Total
(78.6%): 136 study cases (45.5%), and 99 experiments
(33.1%); which are mostly reported in the industry. First stage EC.1, IC.1 1.179
This orientation, towards the more empirical side, Second stage EC.2 928
makes sense, since the cases and experiments of inte-
grating Dev and Ops work teams materialize in real Third stage IC.2, IC.3, EC.3 344
projects. This result coincides with the study by [46], Fourth Stage IC.2, IC.3, EC.3, EC.4, EC5 299

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 663

80
(a) (b)
71
70
%
63 61 60 100
60
50
90
50 50 80
70
40 40
37
32
60
30 31 30 28 50
23
40
20 20 18
30
1514
11111010
20
10 10 88
66655
8
44333333
2222
10
4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 1 0
0 1 0 USA
0
Alemania
India
Italia

Portugal

ESlovenia
Corea del Sur
Republica Checa

Corea del Sur

Chile

Belgica
Canada
Sulza
China
Suecia
Australia
Finlandia
Brasil
UK
Paises Bajos
Irlanda
Espana
Corea
Belgica
Noruega
Israel
Rusia
Francia
Letonia
Sri Lanka
Austria
Austria
Rumania
Romania
Hungria
Serbia
Sudan
Ucrania
Mexico
Colombia
Turkia
Malasia
Dinamarca
Ecuador
Singapur
Iran
Marruecos
Tailandia
Grecia
Escocia
Japon
2011 2014 2016 2018 2020
2013 2015 2017 2019

Fig. 1. Evolution of publications per year (a), and publications by country (b) in DevOps software testing.

ods and frameworks respectively to support testing are 139 papers (46.5%). Despite this, these works do
work. These results are in agreement with the results indicate that software testing is a DevOps necessity,
obtained in the study by [40], they point out that tools but they do not specify the levels of testing in the
and frameworks have been proposed and that most are DevOps context. In the case of the primary studies,
based on unit tests and automated integration. which do indicate the levels of proof, it follows that:
(i) 122 papers (35.1%) have reported unit and user
interface tests; (ii) 33 papers (11%) have reported load
4.4. RQ-4 what Levels of Software Testing and stress; and, (iii) the rest are user tests and penetra-
are used in DevOps?
tion testing (pen-testing). The work of [43] and [45]
In relation to the levels of software testing used in agree that unit and integration tests are among the
DevOps (see Fig. 4a), the response of “not precise” most studied. Likewise, [43] adds functional, load and

Table 5. Frequency of primary studies by means of communication, which have 4 or more publications
Venue Count
Lecture Notes in Computer Science 11
Communications in Computer and Information Science 9
CEUR Workshop Proceedings 9
International Conference on Software Engineering 9
ACM International Conference Proceeding Series 7
International Workshop on Quality-Aware DevOps (QUDOS) 7
IEEE Software 5
Euromicro Conference on Software Engineering and Advanced Application (SEAA) 5
Information and Software Technology 5
IEEE International Conference on Software Maintenance and Evolution (ICSME) 5
Advances in Intelligent Systems and Computing 4
International Conference on Software Testing, Verification and Validation (ICSTW) 4
International Conference on Software Analysis, Evolution, and Reengineering (SANER) 4
Journal of System and Software 4

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

664 PANDO, DÁVILA

(a) 250 (b) (c)

213
Case Study 136
200
200 185
Experiment 99 180
150 160
140
Propose 27 120
100 100 76
Opinion Research 25 57 80
50 60
29 40 27
Survey 12 20 8 1 1 1
0 0
0 50 100 150 Industry Not Precise Academy

al

Ed ise

e
ive

Ba c

g
Au ativ

in
ni

ni
ci

ec

ot

nk
ha

io
er

uc

m
Pr

Av
m

ec
to
o.
om

M
N
C
Fig. 2. Distribution of primary studies of software testing in the DevOps context, by: (a) research type, (b) research context, and
(c) application domain.

Level
PenTesting: 5

User: 17

Load/Stress: 33

Unit/Int/UI: 105

No.Precise: 139

Tool: 216 Method: 40 No. Precise: 37 Framework: 3 Metric: 3

Proposal

Fig. 3. Types of proposals by test levels.

stress tests as the most studied with 63.6% of the total to TDD and XP with 5 (1.7%) papers, considering
studies reviewed; and, they consider that security tests both. In particular, in the case of TDD studies, they
are much less studied with 3.6%. According to reviews consider the method important for the success of soft-
from [44] and [39], GUI and accessibility tests are still ware testing in DevOps. This suggests that, for now,
pending challenges in continuous contexts. although TDD is a very good method, there are few
studies in this type of context. Similarly, the studies by
According to this Fig. 4a, in relation to the oppor-
[44] and [46] consider that TDD would help to better
tunity in the use of software tests in DevOps, it can be
conceptualize testing strategies and mitigate system
pointed out that 162 papers (54.2%) have been applied
design errors for help continuous testing.
during continuous integration; which, at first glance,
turns out to be the natural space for testing. However,
84 (28.1%) papers have also been identified that have 4.5. RQ-5 what Programming Languages and Software
used tests to solve activities in continuous delivery and Testing Tools are Used in DevOps?
44 (14.7%) in continuous deployment, which shows
Due to the nature and objectives of the primary
that 42.8% of the tests are outside continuous integra-
studies, in many cases, programming languages, test-
tion.
ing support tools, and version control tools are not
According to Fig. 4b, in relation to the software required. In the case of programming languages (see
development methodology, from the primary studies, Fig. 5), it is observed that Java is the most reported
it has been determined as “not precise” in 217 (72.6%) language with 90 (30%) papers. In the case of test sup-
papers. In the other cases, it shows 75 (25.1%) papers port tools, Junit with 25 (8.4%) and Selenium with 13
used agile methodologies, and more explicitly points (4.3%) papers are the most reported. Finally, in the

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 665

Level (a) (b)

PenTesting: 5 250
217

User: 17 200

Load/Stress: 33 150

Unit/Int/UI: 105
100
75

50
No.Precise: 139

4 2 1
0
Integration: Delivery: Deployment: No. Precise: No TDD Agiles BPM XP
162 84 44 9
Continuous Stage

Fig. 4. Test levels (a) grouped by continuous phase and (b) methods used in software testing in DevOps.

Tools
TestNG: 1

Pytest: 1

NoseTest: 1

Mocha: 1

Jmeter: 1

PHPUnits: 2

Selenium: 13

Junit: 25

No.Pse.: 254

No.Pse.: 139 Java: 90 Py: 27 JS: 23 PHP: 12 Ruby: 6 C: 2 Git: 179 No.Precise: 120
Lang. VC.

Fig. 5. Software testing tools in DevOps by programming languages and version control.

case of version control tools, Git is mentioned in 179 4.6. RQ-6 in what Types of Applications
(59.9%) of papers. and Architectures is Software Testing used
in the DevOps Context?
In the review of [46], it is agreed that Junit, Sele-
nium and Git are the most frequent tools in the In relation to the types of applications where soft-
DevOps software testing application. In addition [46], ware tests are used in DevOps (see Fig. 7a), reported
in the primary studies, web applications with 219
considers NUnit among the most frequent, however, (71.9%) papers have to be the most reported applica-
of the selected primary studies, no reference to said tions, and to a lesser extent, mobile applications with
tool was found. 13 (4.3%) papers. The identified console applications
are reported for cases in which they apply machine
According to Fig. 6a, Java is the most used lan- learning concepts and use this type of application to
guage over time with an average of 13 papers per year, display the results. In relation to the types of architec-
while Python has been considered in recent years, ture (see Fig. 7b), the primary studies indicate that 134
with an average of 4 papers per year as presented in (44.8%) are of the MVC type and 52 (17.4%) are of the
Fig. 6b. SOA type, and especially, of the latter, 14 studies

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

666 PANDO, DÁVILA

(b)
(a) Java 13.0
30
27 JS 4.6
25
20 20 Python 4.0
19

15 C 2.0

10 10
PHP 2.0
8
5 4
2 Ruby 1.5
0
2013 2014 2015 2016 2017 2018 2019 2020 0 5 10 15
C Java JS PHP Python Ruby

Fig. 6. Programming languages in software testing over time (a) and average per year (b).

250 160
215 (a) 140 134 (b)
200
120
150 100
85
80
100 60 52
61
40 28
50
13 20
6 4
0 0
Web No.Precise Mobile Embedded Console MVC Not Precise SOA Monolitic

Fig. 7. Type of applications (a) and architectures (b) in software testing in DevOps.

report REST as a technology communication. Despite tion to the average of the publications of papers per
this, 85 (28.4%) papers which represent a high per- year, which Docker has about 6.8 papers/year since
centage that does not need it. 2016, AWS is 3.3 since 2018 and GitLab is 4.8 since
For [46], 33% of their studies found are web appli- 2017. This result shows that Docker is being recur-
cations, being the most frequent for DevOps software rently reported in the selected primary studies. In the
tests; and it also agrees that few researches, that is, interviews conducted by [45], containerization is
1.6%, are reported on embedded applications. mentioned as one of the most studied solutions in con-
tinuous delivery.
4.7. RQ-7 what Tools are used In Fig. 9, it can be seen that Java appears in 40
for Software Testing in DevOps? (13.4%) primary studies, being used in conjunction
with Jenkins, becoming the most frequent language
Regarding the tools, it can be pointed out that they for Jenkins. Furthermore, in the case of Java, 19 (21%)
are not reported in 111 (37.1%) of the studies (see Fig. 8a). papers have been applied in industry and 3 (4%) in the
In the studies that are reported, Jenkins is present in 92 academic context.
(30.8%) primary studies. This result coincides with
the review by [46] who also found Jenkins to be the Figure 10 shows that 63 (21%) Jenkins primary
most studied tool. In the industry, Jenkins is known as studies have been studied in the industry and Docker
a very versatile tool that allows you to automatically with 34 (7.4%) is behind Jenkins. This shows that Jen-
run tests written by the development team, whether kins is the most studied software testing tool in
they are unit, integration, UI, loading and others. DevOps contexts.
Crossing these results with the years of publication, Figure 11a shows that although Java was often used
according to Fig. 8b, it can be seen that Jenkins has as a programming language, Junit was not necessarily
been increasingly reported in primary studies since mentioned in these studies. However, Junit does
2013. It is also observed, according to Fig. 8c, in rela- appear as the most mentioned testing tools in the pri-

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 667

(c)
(a) (b) Jenkins 11.5
120 111
Docker 6.8
25
100 92 23
20 GitLab 4.8
80 19
16 Travis 3.3
60 15
13 11 AWS 3.3
40 34 10 10
24 8 6 Azire 1.7
20 6 6
20 10 5 5
5 3 2 GitHub 1.0
1
0 0
2011 2013 2014 2015 2016 2017 2018 2019 2020 0 5 10 15
Je se

D ns

G er
ab

Tr S
is

G e
ub
ur
AW

av
k
i
i

irL

itH
ec
nk
oc

Az
Pr

AWS Azure Docker GitHub

o.

GitLab Jenkins Travis

N

Fig. 8. Software testing tools in DevOps (a) by years (b) and, distributed over time and average per year (c).

Languages

C: 2

Ruby: 6

PHP: 12

JS: 23

Py: 27

Java: 90

No.Pr.: 139

No.Pr. Jenkins Docker GitLab AWS Travis Azure Github Industry: 213 Academy: 29 No.Precise: 57
111 92 34 24 20 10 5 3
Tools Contexts

Fig. 9. Programming languages and tools in DevOps software testing.

mary studies. In addition, these, for the most part, 185 despite the fact that there are only 15 application test-
(61.8%) papers have been applied in commercial busi- ing papers. penetration (see Fig. 4a). These findings
ness domains. Figure 11b confirms that Junit is also are in the same direction as that indicated by [47, 41,
applied in the industrial context. 46], about the need to study more about the security
issues in Devops contexts, also known as DevSecOps.
This allows you to integrate these types of tests into
4.8. RQ-8 in what Types of Activities do Software Testing your development tools.
Occur in DevOps? Also, are Safety Tests Mentioned?
According to Fig. 12, the selected primary studies
show that more than 230 (75%) have concerned them- 4.9. Threats to Validity
selves with both what is needed in development and in The analysis of the threats to validity was based on
operation, be it with tools, methods, frameworks or the work and questions proposed by [50].
suggestions. 60 (20%) papers have studied the specific
activities of development teams. Finally, only 9 (3%) Study Selection Validation. During the planning of
have focused solely on operating activities. the research, in order to ensure the proper identifica-
According to Fig. 13, more than half of the papers tion of all relevant studies, the following was carried
found, that is 169 (56.6%), mention application secu- out: (i) a preliminary search to identify a relevant set of
rity as an important factor in the DevOps contexts, 20 “test” papers that allowed validating the research

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

668 PANDO, DÁVILA

Tool
Github: 3

Azure: 5

Travis: 10

AWS: 20

GitLab: 24

Docker: 34

Jenkins: 92

No.Precise: 111

Industry: 213 Academy: 29 No. Precise: 57

Context

Fig. 10. Tools in DevOps for software testing according to its context.

(a) Tool Tool (b)

TestNG TestNG

Pytest: 1 Pytest: 1

PhpUnit: 2 PhpUnit: 2

NoseTest: 1 NoseTest: 1

Mocha: 1 Mocha: 1

Jmeter: 1 Jmeter: 1

Docker: 34 Selenium: 13

Junit: 25 Junit: 25

No.Pr.: 254 No.Pr.: 254

No.Pr. Jenkins Docker GitLab AWS Travis Azure Github Mecha- Banking Avionic Auto- Edu No.Pr. Com- Industry: Academy: No.Precise:
111 92 34 24 20 10 5 3 nic 1 1 motive 27 76 mercial 213 29 57
1 8 185
Tool Domain Context

Fig. 11. Test tools, infrastructure in DevOps (a) and application context (b).

questions research, the search chain and selection pro- 250 230
cess; then, (ii) Population and Intervention was used,
according to [48], to structure a convenient search 200
chain, actually an iterative task; (iii) a chain test was 150
carried out with the “test” papers, and a check was
made if the data obtained from said “test” papers 100
allowed to answer the research questions; and (iv) it 60
50
was established to work with 6 relevant digital data-
9
bases. 0
DevOps Devs Ops
The selection was made using the methodology
proposed by [48]. Duplicate papers were filtered in the
exclusion criteria by DOI, title, authors and year. Fig. 12. Software testing in DevOps phases

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 669

130
5. CONCLUSIONS
This research presents a Systematic Mapping Study
(SMS) on software testing in the DevOps context. The
SMS is based on the proposal of [48]. In the selection
process, 3, 312 studies were obtained and at the end of
the process, 299 were selected as primary studies.
169 Based on the data obtained from the primary studies,
it was possible to answer the 8 research questions
No Yes raised.
The interest of research on software testing in the
Fig. 13. Mention of security in software testing in DevOps. DevOps context is current and continuously growing
since 2011. It is also appreciated that it is a global inter-
est, in particular, considering that there are 16 coun-
Inclusion/exclusion criteria were discussed by the tries from 3 regions (America, Europe, and Asia) who
authors based on similar research. At each stage, a have published 239 (80%) of the studies.
general criterion was applied, that, when in doubt of In accordance with the origin and empirical nature
acceptance or rejection, acceptance is chosen so that of DevOps, the majority of primary studies, which
the paper is subsequently evaluated. This reloads the mean 235 (78.6%) are of the type of case studies and
next stage, but reduces the risk of deleting relevant experiments. Likewise, 213 of these studies have been
papers. carried out in industry contexts (71.2%) and 185 in
commercial applications (61.8%). In addition, 216
Data validation. Taking into account what was indi- (72.2%) primary studies have proposed tools that sup-
cated in [48], it was decided to only work with relevant port test automation. The results also indicate that
digital databases. These databases usually already have software testing is considered an important factor in
evaluation schemes for the journals and reports of DevOps issues, but what levels of testing are being
events that they incorporate. In this context, it was used are not specified. But, in those that do specify,
decided not to make a quality assessment in the selec- unit and integration tests are the most studied, and to
tion process. a lesser extent, user, load and stress and security tests.
In the first 100 primary studies, a first consolida- In relation to technology, such as programming
tion was performed, and these studies were discussed language and test support tools, it can be noted that
between both authors. The evaluation also made it these issues are not explicitly reported in primary
possible to note the relationship of the results with the studies. In the cases that do report, it is pointed out
subject under research. The classification schemes that Java is the most reported language with 90 (30%)
were proposed during the planning of the SMS and both in academic and industrial environments; and in
were refined, in some cases, during the data the case of test development tools, 25 papers, that is
extraction. Additionally, the verification of the selec- means, more than 8.3% have been reported to Junit.
tion was carried out by the second author in a sample Other reported programming languages are: Python,
manner. Js and PHP respectively. Furthermore, it has to be
mentioned that Java is the most reported language in
Data validation. Both authors are related to the primary studies over time, with an average of 13 papers
research topic and the second author has more experi- per year.
ence in secondary studies. The work carried out is rep- The most studied types of applications are those of
licable since all the data collected during the research the Web type with 216 (72.2%), based on both SOA
are publicly accessible, phase by phase, as well as the and MVC. One of the most reported tools is Jenkins
general search string and the personalized ones for for both continuous integration, continuous deploy-
each database. At the beginning of the research, it was ment and continuous delivery. In addition, tools such
determined by the research questions and the results of as: Travis, Docker, GitLab, Github and AWS are also
the first stages, that the research would be a systematic reported, showing that the studies carried out are
mapping of literature due to the need to classify soft- applied to current market tools.
ware tests in DevOps contexts. The research can be The results of this research show research opportu-
generalized to all DevOps contexts because it collects nities in software testing for the DevOps contexts.
the information without considering specific regions, Likewise, it is clear that training in automated software
places or periods. In addition, it considers primary testing skills could help small companies to compete in
studies from both industry and academia. the world market with quality.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

670 PANDO, DÁVILA

Appendix A. List of Primary Studies

ID Authors Year Title
S01 Priyadarsini K., Fantin Irudaya Raj E., Yasmine 2020 Comparing DevOps procedures from the context
Begum A., Shanmugasundaram V. of a systems engineer
S02 Casola V., De Benedictis A., Rak M., Salzillo G. 2020 A cloud secdevops methodology: From design to
testing
S03 Fehlmann T., Kranich E. 2020 A Framework for Automated Testing
S04 Amaral C.J., Kampik T., Cranefield S. 2020 A framework for collaborative and interactive
agent-oriented developer operations
S05 Klammer C., Gmeiner J. 2020 A Lightweight Customized Build Chain Visual-
ization Approach Applied in Industry
S06 Casola V., De Benedictis A., Rak M., Villano U. 2020 A methodology for automated penetration test-
ing of cloud applications
S07 Guntha R., Rao S.N., Muccini H., Vinodini Ramesh M. 2020 A Novel Paradigm for Rapid Yet Robust Contin-
uous Delivery of Software for Disaster Manage-
ment Scenarios
S08 Hsu W., Lin J.-S., Chen Y.-C., Wang C.-Y., 2020 An Automatic Software Quality and Function
Huang C.-T. Assurance Case Study for Agile
S09 Cai Y.X., Shang Y.F., Tan Y.X., Tang Z.W., Zhao B. 2020 An Effective Solution for Application Orchestration
S10 Macarthy R.W., Bass J.M. 2020 An Empirical Taxonomy of DevOps in Practice
S11 Kanchana A., Murthy B.N.S. 2020 Automated Development and Testing of ECUs
in Automotive Industry with Jenkins
S12 Avritzer A. 2020 Automated scalability assessment in devops envi-
ronments
S13 Rakshith M.N., Shivaprasad N. 2020 Build Optimization Using Jenkins
S14 Karlaš B., Interlandi M., Renggli C., Wu W., Zhang C., 2020 Building Continuous Integration Services for
Mukunthu Iyappan Babu D., Edwards J., Lauren C., Machine Learning
Xu A., Weimer M.
S15 Ambrosino G., Fioccola G.B., Canonico R., Ventre G. 2020 Container Mapping and its Impact on Perfor-
mance in Containerized Cloud Environments
S16 Reiterer S.H., Balci S., Fu D., Benedikt M., Soppa A., 2020 Continuous Integration for Vehicle Simulations
Szczerbicka H.
S17 Gota L., Gota D., Miclea L. 2020 Continuous Integration in Automation Testing
S18 Gorsky S.A. 2020 Continuous integration, delivery, and deploy-
ment for scientific workflows in Orlando Tools
S19 Rangnau T., Buijtenen R. v., Fransen F., Turkmen F. 2020 Continuous Security Testing: A Case Study on
Integrating Dynamic Security Testing Tools in
CI/CD Pipelines
S20 Johnson M., Cummings D., Leinwand B., Elsberry C. 2020 Continuous Testing and Deployment for Urban
Air Mobility
S21 Angara J., Prasad S. 2020 Continuous testing real-time health analytics
dashboard
S22 Doležel M. 2020 Defining testops: Collaborative behaviors and
technology-driven workflows seen as enablers of
effective software testing in devops
S23 Török M., Pataki N. 2020 DevOps dashboard with heatmap
S24 Yang D., Wang D., Yang D., Dong Q., Wang Y., Zhou H., 2020 DevOps in practice for education management
Daocheng H. information system at ECNU

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 671

S25 Laaber C., Würsten S., Gall H.C., Leitner P. 2020 Dynamically reconfiguring software micro-
benchmarks: Reducing execution time without
sacrificing result quality
S26 Al-Sabbagh K.W., Staron M., Ochodek M., Meding W. 2020 Early prediction of test case verdict with bag-of-
words vs. word embeddings
S27 Couto L.D., Tran-Jørgensen P.W.V., Nilsson R.S., 2020 Enabling continuous integration in a formal
Larsen P.G. methods setting
S28 Karakasis V., Manitaras T., Rusu V.H., Sarmiento- 2020 Enabling Continuous Testing of HPC Systems
Pérez R., Bignamini C., Kraushaar M., Jocksch A., Using ReFrame
Omlin S., Peretti-Pezzi G., Augusto J.P.S.C., Friesen B.,
He Y., Gerhardt L., Cook B., You Z.-Q., Khuvis S.,
Tomko K.
S29 Vassallo C., Proksch S., Zemp T., Gall H.C. 2020 Every build you break: developer-oriented assis-
tance for build failure resolution
S30 Luzar A., Stanovnik S., Cankar M. 2020 Examination and comparison of tosca orchestra-
tion tools
S31 Meinicke J., Wong C.-P., Vasilescu B., Kästner C. 2020 Exploring differences and commonalities
between feature flags and configuration options
S32 Demeyer S., Parsai A., Vercammen S., van Bladel B., 2020 Formal Verification of Developer Tests: A
Abdi M. Research Agenda Inspired by Mutation Testing
S33 Mazkatli M., Monschein D., Grohmann J., Koziolek A. 2020 Incremental Calibration of Architectural Perfor-
mance Models with Parametric Dependencies
S34 Shin J.-S., Kim J. 2020 K-one playground: Reconfigurable clusters for a
cloud-native testbed
S35 Batra P., Jatain A. 2020 Measurement Based Performance Evaluation of
DevOps
S36 Eismann S., Bezemer C.-P., Shang W., Okanović D., 2020 Microservices: A performance tester’s dream or
Van Hoorn A. nightmare?
S37 van den Heuvel W.-J., Tamburri D.A. 2020 Model-driven ml-ops for intelligent enterprise
applications: vision, approaches and challenges
S38 Shahin M., Babar M.A. 2020 On the role of software architecture in DevOps
transformation: An industrial case study
S39 Mirhosseini S., Parnin C. 2020 Opunit: Sanity Checks for Computing Environ-
ments
S40 Gias A.U., Van Hoorn A., Zhu L., Casale G., Düll- 2020 Performance engineering for microservices and
mann T.F., Wurster M. serverless applications: The RADON approach
S41 Chen J. 2020 Performance Regression Detection in DevOps
S42 Raj P., Sinha P. 2020 Project management in era of agile and devops
methodolgies
S43 Cheriyan A., Gondkar R.R., Babu S.S. 2020 Quality Assurance Practices and Techniques
Used by QA Professional in Continuous Delivery
S44 Huang M., Fan W., Huang W., Cheng Y., Xiao H. 2020 Research on Building Exploitable Vulnerability
Database for Cloud-Native App
S45 Fayollas C., Bonnin H., Flebus O. 2020 SafeOps: A Concept of Continuous Safety
S46 Vishnu Vardhan Reddy B.S., Swamy B.K., Sai S.P.S., 2020 Securing web application by using qualitative
Kiran K.V.D. research methods for detection of vulnerabilities
in any application of DevSecOps
S47 Petrovic N., Tosic M. 2020 SMADA-Fog: Semantic model driven approach
to deployment and adaptivity in fog computing

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

672 PANDO, DÁVILA

S48 Orviz Fernández P., David M., Duma D.C., Ronch- 2020 Software Quality Assurance in INDIGO-Data-
ieri E., Gomes J., Salomoni D. Cloud Project: a Converging Evolution of Software
Engineering Practices to Support European
Research e-Infrastructures
S49 Wang Y., Mäntylä M.V., Demeyer S., Wiklund K., 2020 Software test automation maturity: A survey of
Eldh S., Kairi T. the state of the practice
S50 Bernard E., Ambert F., Legeard B. 2020 Supporting efficient test automation using light-
weight MBT
S51 Li R., Liu X., Zheng X., Zhang C., Liu H. 2020 TDD4Fog: A Test-Driven Software Develop-
ment Platform for Fog Computing Systems
S52 Wang Y., Pyhäjärvi M., Mäntylä M.V. 2020 Test Automation Process Improvement in a
DevOps Team: Experience Report
S53 Hasan M.M., Bhuiyan F.A., Rahman A. 2020 Testing practices for infrastructure as code
S54 Marlowe T.J., Kirova V., Chang G. 2020 The state of agile: Changes in the world of
change
S55 Klemets J., Storholmen T.C.B. 2020 Towards Super User-Centred Continuous Deliv-
ery: A Case Study
S56 Ding Z., Chen J., Shang W. 2020 Towards the use of the readily available tests from
the release pipeline as performance tests. Are we
there yet
S57 Leotta M., Cerioli M., Olianas D., Ricca F. 2020 Two experiments for evaluating the impact of
Hamcrest and AssertJ on assertion development
S58 Gallaba K., McIntosh S. 2020 Use and Misuse of Continuous Integration Fea-
tures: An Empirical Study of Projects That
(Mis)Use Travis CI
S59 Zhou Y., Su Y., Chen T., Huang Z., Gall H.C., Pan- 2020 User Review-Based Change File
ichella S.
S60 Yu L., Alégroth E., Chatzipetrou P., Gorschek T. 2020 Utilising CI environment for efficient and effec-
tive testing of NFRs
S61 Van Rossem S., Tavernier W., Colle D., Pickavet M., 2020 VNF Performance modelling: From stand-alone
Demeester P. to chained topologies
S62 Bertolino A., Angelis G. D., Guerriero A., Miranda B., 2019 DevOpRET: Continuous reliability testing in
Pietrantuono R., Russo S. DevOps
S63 Jacobsen D. M., Kleinman R., Longley H. 2019 Managing a Cray supercomputer as a git branch
S64 Meyers B., Gadeyne K., Oakes B., Bernaerts M., Van- 2019 A Model-Driven Engineering Framework to
gheluwe H., Denil J. Support the Functional Safety Process
S65 Zampetti F., Bavota G., Canfora G., Penta M.D. 2019 A Study on the Interplay between Pull Request
Review and Continuous Integration Builds
S66 Chhillar D., Sharma K. 2019 ACT Testbot and 4S Quality Metrics in XAAS
Framework
S67 Abbass M.K.A., Osman R.I.E., Mohammed A.M.H., 2019 Adopting Continuous Integeration and Continu-
Alshaikh M.W.A. ous Delivery for Small Teams
S68 Guerriero M., Garriga M., Tamburri D.A., Palomba F. 2019 Adoption, Support, and Challenges of Infra-
structure-as-Code: Insights from Industry
S69 Durieux T., Abreu R., Monperrus M., Bissyande T.F. 2019 An Analysis of 35+ Million Jobs of Travis CI
S70 Vasile T., Cane S., C. Bertram C., F. Jakob F. 2019 Applying Security Concepts to Continuous Inte-
gration for the Purpose of Testing Embedded
Systems
S71 Vassallo C., Proksch S., Gall H.C., Penta M. 2019 Automated Reporting of Anti-Patterns and
Decay in Continuous Integration

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 673

S72 Janes A., Russo B. 2019 Automatic Performance Monitoring and Regres-
sion Testing During the Transition from Mono-
lith to Microservices
S73 Krym T., Poniszewska-Marańda A., Markl E., Dupas R. 2019 Automatic Process of Continuous Integration of
Web Application
S74 Najafi A., Rigby P.C., Shang W. 2019 Bisecting commits and modeling commit risk
during testing
S75 Tomassi D.A., Dmeiri N., Wang Y., Bhowmick A., Liu Y., 2019 BugSwarm: Mining and Continuously Growing
Devanbu P.T., Vasilescu B., Rubio-González C. a Dataset of Reproducible Failures and Fixes
S76 Satyal S., Weber I., Paik H.-Y., Di Ciccio C., Mend- 2019 Business process improvement with the AB-
ling J. BPM methodology
S77 Gupta R.K., Venkatachalapathy M., Jeberla F.K. 2019 Challenges in Adopting Continuous Delivery
and DevOps in a Globally Distributed Product
Team: A Case Study of a Healthcare Organization
S78 Judvaitis J., Nesenbergs K., Balass R., Greitans M. 2019 Challenges of DevOps ready IoT testbed
S79 Nogueira A.F., Sergeant E., Ribeiro J.C.B., Zenha- 2019 Collecting data from continuous practices: An
Rela M.A., Craske A. infrastructure to support team development
S80 Singh C., Gaba N.S., Kaur M., Kaur B. 2019 Comparison of Different CI/CD Tools Inte-
grated with Cloud Platform
S81 Jawarneh I.M.A., Bellavista P., Bosi F., Foschini L., 2019 Container Orchestration Engines: A Thorough
Martuscelli G., Montanari R., Palopoli A. Functional and Performance Comparison
S82 Grambow M., Lehmann F., Bermbach D. 2019 Continuous Benchmarking: Using System
Benchmarking in Build Pipelines
S83 Glein R., Perloff A., Ulmer K. 2019 Continuous integration of FPGA designs for CMS
S84 Felidré W., Furtado L., d. Costa D.A., Cartaxo B., 2019 Continuous Integration Theater
Pinto G.
S85 Johanssen J.O., Kleebaum A., Paech B., Bruegge B. 2019 Continuous software engineering and its support
by usage and decision knowledge: An interview
study with practitioners
S86 Guşeila L., D. Bratu ., Moraru S. 2019 Continuous Testing in the Development of IoT
Applications
S87 Lescisin M., Mahmoud Q.H., Cioraca A. 2019 Design and implementation of SFCI: A tool for
security focused continuous integration
S88 O. Veres; N. Kunanets; V. Pasichnyk; N. Vereten- 2019 Development and Operations – the Modern
nikova; R. Korz; A. Leheza Paradigm of the Work of IT Project Teams
S89 Jennings R.A.K., Gannod G. 2019 DevOps – Preparing Students for Professional
Practice
S90 Heistand C., Thomas J., Tzeng N., Badger A.R., 2019 DevOps for Spacecraft Flight Software
Rodriguez L.M., Dalton A., Pai J., Bodzas A.,
Thompson D.
S91 Guşeila L., D. Bratu ., Moraru S 2019 DevOps Transformation for Multi-Cloud IoT
Applications
S92 Agrawal P., Rawat N. 2019 Devops, A New Approach To Cloud Develop-
ment & Testing
S93 Embury S.M., Page C. 2019 Effect of continuous integration on build health
in undergraduate team projects
S94 Vassallo C. 2019 Enabling Continuous Improvement of a Contin-
uous Integration Process
S95 K. Baral K., R. Mohod R., J. Flamm J., S. Goldrich, 2019 Evaluating a Test Automation Decision Support
P. Ammann Tool

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

674 PANDO, DÁVILA

S96 Huijgens H., Greuter E., Brons J., van Doorn E.A., 2019 Factors Affecting Cloud Infra-Service Develop-
Papadopoulos I., Morales Martinez F., Aniche M., ment Lead Times: A Case Study at ING
Visser O., van Deursen A.
S97 Suk T., Hwang J., Bulut M. F., Zeng Z. 2019 Failure-Aware Application Placement Modeling
and Optimization in High Turnover DevOps Envi-
ronment
S98 Bezemer C.-P., Eismann S., Ferme V., Grohmann J., 2019 How is performance addressed in DevOps?
Heinrich R., Jamshidi P., Shang W., Van Hoorn A., A survey on industrial practices
Villavicencio M., Walter J., Willnecker F.
S99 Chen B. 2019 Improving the Software Logging Practices in
DevOps
S100 Carturan S., Goya D. 2019 Major Challenges of Systems-of-Systems with
Cloud and DevOps – A Financial Experience
Report
S101 Shah J. A., Dubaria D. 2019 NetDevOps: A New Era Towards Networking
DevOps
S102 Haavisto J., Arif M., Lovén L., Leppänen T., Riekki J. 2019 Open-source RANs in Practice: an Over-The-
Air Deployment for 5G MEC
S103 Keahey K., Anderson J., Ruth P., Colleran J., Ham- 2019 Operational lessons from chameleon
mock C., Stubbs J., Zhen Z.
S104 Hakimzadeh K., Dowling J. 2019 Ops-Scale: Scalable and Elastic Cloud Opera-
tions by a Functional Abstraction and Feedback
Loops
S105 Salinas E. 2019 Pat Helland on Failure and Resilience in Dis-
tributed Systems
S106 Al-Sabbagh K.W., Staron M., Hebig R., Meding W. 2019 Predicting test case verdicts using textual analysis
of committed code churns
S107 Nuriddinov A., Tavernier W., Colle D., Pickavet M., 2019 Reproducible Functional Tests for Multi-scale
Peustery M., Schneidery S. Network Services
S108 Wiedemann A., Forsgren N., Wiesche M., Gewald H., 2019 Research for practice: The Devops phenomenon
Krcmar H.
S109 Sidhu P.K., Mussbacher G., McIntosh S. 2019 Reuse (or Lack Thereof) in Travis CI Specifica-
tions: An Empirical Study of CI Phases and
Commands
S110 Mäkinen S., Puonti M., Lehtonen T., Mikkonen T., 2019 Revisiting continuous deployment maturity:
Kilamo T., Männistö T. A two-year perspective
S111 Siewruk G., Mazurczyk W., Karpiński A. 2019 Security assurance in Devops methodologies and
related environments
S112 Vera-Rivera F.H., Vera-Rivera J.L., Gaona-Cuevas C.M. 2019 Sinplafut: A microservices - Based application
for soccer training
S113 Naik S. M., Fernandes M., Pushpakumar G., Pathak R. 2019 Smart Grid Communication Protocol Test Auto-
mation along with Protection Test Automation
S114 Risdianto A.C., Usman M., Kim J.W. 2019 SmartX box: Virtualized hyper-converged
resources for building an affordable playground
S115 Czarnecki K. 2019 Software Engineering for Automated Vehicles:
Addressing the Needs of Cars That Run on Soft-
ware and Data
S116 Keskin Kaynak İ., Çilden E., Aydin S. 2019 Software Quality Improvement Practices in
Continuous Integration
S117 Cunningham S., Gambo J., Lawless A., Moore D., 2019 Software Testing: A Changing Career
Yilmaz M., Clarke P.M., O’Connor R.V.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 675

S118 Rahman A., Williams L. 2019 Source code properties of defective infrastruc-
ture as code scripts
S119 Kapoor S., Sagar K., Reddy B.V.R. 2019 Speedroid: A novel automation testing tool for mobile
apps
S120 Arulkumar V., Lathamanju R. 2019 Start to Finish Automation Achieve on Cloud
with Build Channel: By DevOps Method
S121 Figalist I., Biesdorf A., Brand C., Feld S., Kiermeier M. 2019 Supporting the DevOps Feedback Loop using
Unsupervised Machine Learning
S122 G. Lim; M. Ham; J. Moon; W. Song; S. Woo; S. Oh 2019 TAOS-CI: Lightweight Modular Continuous
Integration System for Edge Computing
S123 Cruzes D.S., Melsnes K., Marczak S. 2019 Testing in a DevOps Era: Perceptions of Testers
in Norwegian Organisations
S124 Martin D., Panichella S. 2019 The Cloudification Perspectives of Search-
Based Software Testing
S125 Fazayeli H., Syed-Mohamad S.M., Md Akhir N.S. 2019 Towards auto-labelling issue reports for pull-
based software development using text mining
approach
S126 Meixner K., Winkler D., Biffl S. 2019 Towards combined process & tool variability
management in software testing
S127 Pietrantuono R., Bertolino A., Angelis G., Miranda B., 2019 Towards Continuous Software Reliability Testing
Russo S. in DevOps
S128 Giorgi F., Paulisch F. 2019 Transition Towards Continuous Delivery in the
Healthcare Domain
S129 Paule C., Düllmann T.F., Van Hoorn A. 2019 Vulnerabilities in Continuous Delivery Pipe-
lines? A Case Study
S130 Chwalisz M., Geissdoerfer K., Wolisz A. 2019 Walker: DevOps Inspired Workflow for Experimen-
tation
S131 Benni B., Blay-Fornarino M., Mosser S., Précisio F., 2019 When DevOps Meets Meta-Learning: A Portfo-
Jungbluth G. lio to Rule them all
S132 Daoudagh S., Lonetti F., Marchetti E. 2019 An automated framework for continuous devel-
opment and testing of access control systems
S133 Luz W. P., Pinto G., Bonifacio R. 2018 Building a Collaborative Culture: A Grounded
Theory of Well Succeeded Devops Adoption in
Practice
S134 Osses F., Marquez G., Astudillo H. 2018 Exploration of Academic and Industrial Evi-
dence about Architectural Tactics and Patterns in
Microservices
S135 Schulz, Henning and Angerstein, Tobias and van 2018 Towards Automating Representative Load Test-
Hoorn, André ing in Continuous Software Engineering
S136 Kuusinen K., Balakumar V., Jepsen S.C., Larsen S.H., 2018 A Large Agile Organization on Its Journey
Lemqvist T.A., Muric A., Nielsen A.Ø., Vestergaard O. Towards DevOps
S137 Sandobalin J. 2018 A Model-Driven Approach to Continuous
Delivery of Cloud Resources
S138 Sandobalin J., Insfran E., Abrahao S. 2018 A smart provisioning approach to cloud infra-
structure
S139 Baudry B., Harrand N., Schulte E., Timperley C., Tan 2018 A spoonful of DevOps helps the GI go down
S.H., Selakovic M., Ugherughe E.
S140 Shah J., Dubaria D., Widhalm J. 2018 A Survey of DevOps tools for Networking
S141 Li H., Chen T., Hassan A. E., Nasser M., Flora P. 2018 Adopting Autonomic Computing Capabilities in
Existing Large-Scale Systems
S142 Zykov S.V. 2018 Agile services

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

676 PANDO, DÁVILA

S143 Akman S., Aksuyek E.B., Kaynak O. 2018 ALM Tool Infrastructure with a Focus on
DevOps Culture
S144 Wiedemann A., Wiesche M. 2018 Are you ready for Devops? Required skill set for
Devops teams
S145 Rubasinghe I., Meedeniya D., Perera I. 2018 Automated Inter-artefact Traceability Establish-
ment for DevOps Practice
S146 Rosa R. V., Rothenberg C. E. 2018 Automated VNF Testing with Gym: A Bench-
marking Use Case
S147 Debroy V., Brimble L., Yost M., Erry A. 2018 Automating Web Application Testing from the
Ground Up: Experiences and Lessons Learned
in an Industrial Setting
S148 Kargar M. J., Hanifizade A. 2018 Automation of regression test in microservice
architecture
S149 Mohan V., Othmane ben L., Kres A. 2018 BP: Security Concerns and Best Practices for
Automation of Software Deployment Processes:
An Industrial Case Study
S150 Rahman A., Williams L. 2018 Characterizing Defective Configuration Scripts
Used for Continuous Deployment
S151 Rahman A., Agrawal A., Krishna R., Sobran A. 2018 Characterizing the influence of continuous inte-
gration: Empirical results from 250+ open source
and proprietary projects
S152 Agarwal A., Gupta S., Choudhury T. 2018 Continuous and Integrated Software Develop-
ment using DevOps
S153 Bai X., Li M., Pei D., Li S., Ye D. 2018 Continuous Delivery of Personalized Assessment
and Feedback in Agile Software Engineering
Projects
S154 Arachchi S. A. I. B. S., Perera I. 2018 Continuous Integration and Continuous Deliv-
ery Pipeline Automation for Agile Software Proj-
ect Management
S155 Williams L. 2018 Continuously Integrating Security
S156 Alshahwan N., Gao X., Harman M., Jia Y., Mao K., 2018 Deploying search based software engineering
Mols A., Tei T., Zorin I. with sapienz at facebook
S157 Marijan D., Sen S. 2018 Devops enhancement with continuous test optimi-
zation
S158 Marijan D., Liaaen M., Sen S. 2018 DevOps Improvements for Reduced Cycle
Times with Integrated Test Optimizations for
Continuous Integration
S159 Angara J., Gutta S., Prasad S. 2018 DevOps with continuous testing architecture and
its metrics model
S160 Park S., Huh J.-H. 2018 Effect of cooperation on manufacturing IT proj-
ect development and test bed for successful
industry 4.0 Project: Safety management for
security
S161 Mårtensson T., Ståhl D., Bosch J. 2018 Enable more frequent integration of software in
industry projects
S162 Casale G., Li C. 2018 Enhancing Big Data Application Design with the
DICE Framework
S163 Dülmann T. F., Paule C., van Hoorn A. 2018 Exploiting DevOps Practices for Dependable
and Secure Continuous Delivery Pipelines
S164 Loseva E., Obeid A., Richter H., Backes R., Eichhorn D. 2018 FIXIT - A semi-automatic software deployment
tool for arbitrary targets

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 677

S165 Jiang H., Chen X., He T., Chen Z., Li X. 2018 Fuzzy clustering of crowdsourced test reports for
apps
S166 Widder D., Vasilescu B., Hilton M., Kästner C. 2018 I’m Leaving You, Travis: A Continuous Integra-
tion Breakup Story
S167 Fernandes T. C. M., Costa I., Salvetti N., de 2018 Influence of DevOps practices in IT manage-
Magalhaes F. L. F., Fernandes A. A. ment processes according to the COBIT 5 model
S168 Soenen T., Van Rossem S., Tavernier W., Vicens F., 2018 Insights from SONATA: Implementing and inte-
Valocchi D., Trakadas P., Karkazis P., Xilouris G., grating a microservice-based NFV service plat-
Eardley P., Kolometsos S., Kourtis M.-A., Guija D., form with a DevOps methodology
Siddiqui S., Hasselmeyer P., Bonnet J., Lopez D.
S169 van Rossem S., Tavernier W., Colle D., Pickavet M., 2018 Introducing Development Features for Virtual-
Demeester P. ized Network Services
S170 Asha N., Mani P. 2018 Knowledge-based acceptance test driven agile
approach for quality software development
S171 Eickhoff F. L., McGrath M. L., Mayer C., 2018 Large-scale application of IBM Design Thinking
Bieswanger A., Wojciak P. A. and Agile development for IBM z14
S172 Staron M., Meding W., Söder O., Bäck M. 2018 Measurement and Impact Factors of Speed of
Reviews and Integration in Continuous Software
Engineering
S173 Chen L. 2018 Microservices: Architecting for Continuous
Delivery and DevOps
S174 Alipour H., Liu Y. 2018 Model Driven Deployment of Auto-Scaling Ser-
vices on Multiple Clouds
S175 Wurster M., Breitenbücher U., Kopp O., Leymann F. 2018 Modeling and Automated Execution of Applica-
tion Deployment Tests
S176 D’Ambrogio A., Falcone A., Garro A., Giglio A. 2018 On the importance of simulation in enabling
continuous delivery and evaluating deployment
pipeline performance
S177 Zhang Y., Vasilescu B., Wang H., Filkov V. 2018 One size does not fit all: An empirical study of
containerized continuous deployment workflows
S178 Cheriyan A., Gondkar R. R., Gopal T., Suresh B. S. 2018 Quality Assurance Practices in Continuous
Delivery - an implementation in Big Data
Domain
S179 Mijumbi R., Okumoto K., Asthana A., Meekel J. 2018 Recent Advances in Software Reliability Assur-
ance
S180 Kerzazi N., EL Asri I. 2018 Release engineering: From structural to func-
tional view
S181 Marquez G., Osses F., Astudillo H. 2018 Review of Architectural Patterns and Tactics for
Microservices in Academic and Industrial Liter-
ature
S182 Satyal S., Weber I., Paik H.-Y., Di Ciccio C., Mendling J. 2018 Shadow Testing for Business Process Improve-
ment
S183 Limoncelli T.A. 2018 SQL is no excuse to avoid DevOps
S184 Zimmerer P. 2018 Strategy for Continuous Testing in iDevOps
S185 Luhana K. K., Schindler C., Slany W. 2018 Streamlining mobile app deployment with Jen-
kins and Fastlane in the case of Catrobat’s
pocket code
S186 Bai X., Pei D., Li M., Li S. 2018 The DevOps Lab Platform for Managing Diver-
sified Projects in Educating Agile Software Engi-
neering

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

678 PANDO, DÁVILA

S187 Guamán D., Pérez J., Díaz J. 2018 Towards a (semi)-automatic reference process to
support the reverse engineering and reconstruc-
tion of software architectures
S188 Martin K., Önmer U., Florian M. 2018 Towards a Continuous Feedback Loop for Ser-
vice-Oriented Environments
S189 Steffens A., Lichter H., Moscher M. 2018 Towards data-driven continuous compliance
testing
S190 Klinaku F., Ferme V. 2018 Towards Generating Elastic Microservices:
A Declarative Specification for Consistent Elas-
ticity Configurations
S191 Peuster M., Karl H. 2018 Understand Your Chains and Keep Your Dead-
lines: Introducing Time-constrained Profiling
for NFV
S192 Kim C., Kim S., Kim J. 2018 Understanding automated continuous integra-
tion for containerized smart energy IoT-cloud
service
S193 Snyder B., Curtis B. 2018 Using Analytics to Guide Improvement during
an Agile – DevOps Transformation
S194 Schermann G., Cito J., Leitner P., Zdun U., Gall H.C. 2018 We’re doing it live: A multi-method empirical
study on continuous experimentation
S195 Pinto G., Castor F., Bonifacio R., Rebouças M. 2018 Work practices and challenges in continuous
integration: A survey with Travis CI users
S196 Fagerholm F., Sanchez Guinea A., Mäenpää H., 2017 The RIGHT model for Continuous Experimen-
Münch J. tation
S197 Ferme V., Pautasso C. 2017 Towards Holistic Continuous Software Perfor-
mance Assessment
S198 Eddy B. P., Wilde N., Cooper N. A., Mishra B., Gam- 2017 A Pilot Study on Introducing Continuous Inte-
boa V. S., Shah K. M., Deleon A. M., Shields N. A. gration and Delivery into Undergraduate Soft-
ware Engineering Courses
S199 Vassallo C., Schermann G., Zampetti F., Romano D., 2017 A Tale of CI Build Failures: An Open Source and
Leitner P., Zaidman A., Di Penta M., Panichella S. a Financial Organization Perspective
S200 Younge A. J., Pedretti K., Grant R. E., Brightwell R. 2017 A Tale of Two Systems: Using Containers to
Deploy HPC Applications on Supercomputers
and Clouds
S201 Wongkampoo S., Kiattisin S. 2017 Atom-Task Precondition Technique to Optimize
Large Scale GUI Testing Time based on Parallel
Scheduling Algorithm
S202 Wu C.-F.E., Burugula R.S., Yu H., Dubey N., Jann J., 2017 Automation of cloud node installation for testing
Nguyen M. and scalable provisioning
S203 Shahin M., Babar M. A., Zahedi M., Zhu L. 2017 Beyond Continuous Delivery: An Empirical
Investigation of Continuous Deployment Chal-
lenges
S204 Brooks T. T. 2017 Big Data Complex Event Processing for Internet
of Things Provenance: Benefits for Audit, Foren-
sics, and Safety
S205 Ståhl D., Bosch J. 2017 Cinders: The continuous integration and delivery
architecture framework
S206 Wettinger J., Breitenbücher U., Falkenthal M., Ley- 2017 Collaborative gathering and continuous delivery
mann F. of DevOps solutions through repositories
S207 Kao C. H. 2017 Continuous evaluation for application develop-
ment on cloud computing environments

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 679

S208 Stahl D., Martensson T., Bosch J. 2017 Continuous practices and devops: beyond the
buzz, what does it all mean?
S209 Fitzgerald B., Stol K.-J. 2017 Continuous software engineering: A roadmap
and agenda
S210 Metzger S., Durden D., Sturtevant C., Luo H., 2017 Eddy4R 0.2.0: A DevOps model for community-
Pingintha-Durden N., Sachs T., Serafimovich A., extensible processing and analysis of eddy-cova-
Hartmann J., Li J., Xu K., Desai A.R. riance data based on R, Git, Docker, and HDF5
S211 Kupsch J. A., Miller B. P., Basupalli V., Burger J. 2017 From continuous integration to continuous
assurance
S212 Perera P., Silva R., Perera I. 2017 Improve software quality through practicing
DevOps
S213 Vost S., Wagner S. 2017 Keeping Continuous Deliveries Safe
S214 Zimmermann O. 2017 Microservices tenets: Agile approach to service
development and deployment
S215 Chung S. 2017 Object-oriented programming with DevOps
S216 Heinrich R., Van Hoorn A., Knoche H., Li F., 2017 Performance engineering for microservices:
Lwakatare L.E., Pahl C., Schulte S., Wettinger J. Research challenges & directions
S217 Haili W., Renbin G., Congbin W., Lei G. 2017 Research and application of development model
of information service for IOT of oil and gas pro-
duction based on cloud architecture
S218 Farahmandpour Z., Versteeg S., Han J., Kameswaran A. 2017 Service Virtualisation of Internet-of-Things
Devices: Techniques and Challenges
S219 Bucena I., Kirikova M. 2017 Simplifying the devops adoption process
S220 van Deursen A. 2017 Software engineering without borders
S221 Spinellis D. 2017 State-of-the-Art Software Testing
S222 Martensson T., Stahl D., Bosch J. 2017 The EMFIS model – Enable more frequent
integration of software
S223 Zhao Y., Serebrenik A., Zhou Y., Filkov V., Vasilescu B. 2017 The impact of continuous integration on other soft-
ware development practices: A large-scale empiri-
cal study
S224 Parnin C., Helms E., Atlee C., Boughton H., Ghattas M., 2017 The Top 10 Adages in Continuous Deployment
Glover A., Holman J., Micco J., Murphy B., Savor T.,
Stumm M., Whitaker S., Williams L.,
S225 Palihawadana S., Wijeweera C. H., Sanjitha M. G. Y. N., 2017 Tool support for traceability management of
Liyanage V. K., Perera I., Meedeniya D. A. software artefacts with DevOps practices
S226 Laukkanen E., Paasivaara M., Itkonen J., Lassenius C., 2017 Towards Continuous Delivery by Reducing the
Arvonen T. Feature Freeze Period: A Case Study
S227 Ameller D., Farre C., Franch X., Valerio D., Cassa- 2017 Towards continuous software release planning
rino A.
S228 Duffau C., Grabiec B., Blay-Fornarino M. 2017 Towards Embedded System Agile Development
Challenging Verification, Validation and Accredi-
tation: Application in a Healthcare Company
S229 Nidagundi P., Novickis L. 2017 Towards utilization of lean canvas in the devops
software
S230 Hilton M., Nelson N., Tunnell T., Marinov D., Dig D. 2017 Trade-offs in continuous integration: Assurance,
security, and flexibility
S231 Morris D., Voutsinas S., Hambly N.C., Mann R.G. 2017 Use of Docker for deployment and testing of
astronomy software

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

680 PANDO, DÁVILA

S232 Zhao M., Le Gall F., Cousin P., Vilalta R., Munoz R., 2017 Verification and validation framework for 5G
Castro S., Peuster M., Schneider S., Siapera M., network services and apps
Kapassa E., Kyriazis D., Hasselmeyer P., Xilouris G.,
Tranoris C., Denazis S., Martrat J.
S233 Ur R., Akond A., Williams L. 2016 Security Practices in DevOps
S234 Calefato F., Lanubile F. 2016 A Hub-and-Spoke Model for Tool Integration in
Distributed Development
S235 Di Nitto E., Jamshidi P., Guerriero M., Spais I., Tam- 2016 A software architecture framework for quality-
burri D.A. aware devops
S236 Hanappi O., Hummer W., Dustdar S. 2016 Asserting reliable convergence for configuration
management scripts
S237 Bae J., Kim C., Kim J. 2016 Automated deployment of SmartX IoT-cloud
services based on continuous integration
S238 Makki M., Van D., Joosen L.W. 2016 Automated workflow regression testing for
multi-tenant SaaS: Integrated support in self-
service configuration dashboard
S239 Schermann G., Schöni D., Leitner P., Gall H.C. 2016 Bifrost: Supporting continuous deployment with
automated enactment of multi-phase live testing
strategies
S240 Risdianto A.C., Shin J., Kim J. 2016 Building and operating distributed SDN-cloud
testbed with hyper-convergent smartx boxes
S241 Liu D., Zhu H., Xu C., Bayley I., Lightfoot D., Green M., 2016 CIDE: An Integrated Development Environ-
Marshall P. ment for Microservices
S242 Vassallo C., Zampetti F., Romano D., Beller M., Pan- 2016 Continuous Delivery Practices in a Large Finan-
ichella A., Di Penta M., Zaidman A. cial Organization
S243 Savor T., Douglas M., Gentili M., Williams L., Beck K., 2016 Continuous Deployment at Facebook and
Stumm M. OANDA
S244 Rossi C., Shibley E., Su S., Beck K., Savor T., Stumm M. 2016 Continuous deployment of mobile software at
facebook (showcase)
S245 Pang C., Hindle A. 2016 Continuous Maintenance
S246 Staples M., Zhu L., Grundy J. 2016 Continuous Validation for Data Analytics Sys-
tems
S247 Hadar E., Hadar I. 2016 CURA: Complex-system Unified reference
architecture position paper: A practitioner view
S248 Riungu-Kalliosaari L., Mäkinen S., Lwakatare L.E., 2016 DevOps adoption benefits and challenges in
Tiihonen J., Männistö T. practice: A case study
S249 Colavita F. 2016 Devops movement of enterprise agile breakdown
silos, create collaboration, increase quality, and
application speed
S250 Callanan M., Spillane A. 2016 DevOps: Making It Easy to Do the Right Thing
S251 Sheridan C., Whigham D., Artac M. 2016 DICE fault injection tool
S252 Amith Raj M. P., Kumar A., Pai S. J., Gopal A. 2016 Enhancing security of Docker using Linux hard-
ening techniques
S253 Rahman M. T., Querel L., Rigby P. C., Adams B. 2016 Feature Toggles: Practitioner Practices and a
Case Study
S254 Mäkinen S., Leppänen M., Kilamo T., Mattila A.-L., 2016 Improving the delivery cycle: A multiple-case
Laukkanen E., Pagels M., Männistö T. study of the toolchains in Finnish software inten-
sive enterprises
S255 Jones S., Noppen J., Lettice F. 2016 Management challenges for devops adoption
within UK SMEs

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 681

S256 Artač M., Borovšak T., Di Nitto E., Guerriero M., 2016 Model-Driven continuous deployment for qual-
Tamburri D.A. ity devops
S257 Adams B., McIntosh S. 2016 Modern Release Engineering in a Nutshell –
Why Researchers Should Care
S258 Kroß J., Willnecker F., Zwickl T., Krcmar H. 2016 PET: Continuous performance evaluation tool
S259 Ohtsuki M., Ohta K., Kakeshita T. 2016 Software engineer education support system
ALECSS utilizing devOps tools
S260 Rahman A. A. U., Williams L. 2016 Software Security in DevOps: Synthesizing
Practitioners' Perceptions and Practices
S261 Cito J., Mazlami G., Leitner P. 2016 TemPerf: Temporal correlation between perfor-
mance metrics and source code
S262 Shahin M., Babar M.A., Zhu L. 2016 The Intersection of Continuous Deployment
and Architecting Process: Practitioners’ Per-
spectives
S263 Punjabi R., Bajaj R. 2016 User stories to user reality: A DevOps approach
for the cloud
S264 Gottesheim W. 2015 Challenges, Benefits and Best Practices of Per-
formance Focused DevOps
S265 Shtern M., Simmons B., Smit M., Lu, H., Litoiu M. 2015 Performance Management and Monitoring
S266 Stillwell M., Coutinho J.G.F. 2015 A DevOps approach to integration of software
components in an EU research project
S267 Salant E., Leitner P., Wallbom K., Ahtes J. 2015 A framework for a cost-efficient cloud ecosystem
S268 Bruneo D., Longo F., Merlino G., Peditto N., 2015 A Modular Approach to Collaborative Develop-
C. Romeo C., Verboso F., Puliafito A. ment in an OpenStack Testbed
S269 Rajagopalan S., Jamjoom H. 2015 App–Bisect: Autonomous healing for microser-
vice-based apps
S270 Chen H., Kazman R., Haziyev S., Kropov V., 2015 Architectural Support for DevOps in a Neo-
Chtchourov D. Metropolis BDaaS Platform
S271 Scheuner J., Cito J., Leitner P., Gall H. 2015 Cloud workBench: Benchmarking IaaS provid-
ers based on infrastructure-as-code
S272 Gebert S., Schwartz C., Zinner T., Tran-Gia P. 2015 Continuously delivering your network
S273 Lehtonen T., Suonsyrjä S., Kilamo T., Mikkonen T. 2015 Defining metrics for continuous delivery and
deployment pipeline
S274 Bruneo D., Longo F., Merlino G., Peditto N., Romeo C., 2015 Enabling Collaborative Development in an
Verboso F., Puliafito A. OpenStack Testbed: The CloudWave Use Case
S275 Wettinger J., Andrikopoulos V., Leymann F. 2015 Enabling devops collaboration and continuous
delivery using diverse application environments
S276 Soni M. 2015 End to End Automation on Cloud with Build
Pipeline: The Case for DevOps in Insurance
Industry, Continuous Integration, Continuous
Testing, and Continuous Delivery
S277 Segall I., Tzoref-Brill R. 2015 Feedback-driven combinatorial test design and
execution
S278 Vasilescu B., Yu Y., Wang H., Devanbu P., Filkov V. 2015 Quality and productivity outcomes relating to
continuous integration in GitHub
S279 de Bayser M., Azevedo L. G., Cerqueira R. 2015 ResearchOps: The case for DevOps in scientific
applications
S280 Laukkanen E., Paasivaara M., Arvonen T. 2015 Stakeholder Perceptions of the Adoption of Con-
tinuous Integration-A Case Study
S281 Rahman A.A.U., Helms E., Williams L., Parnin C. 2015 Synthesizing Continuous Deployment Practices
Used in Software Development

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

682 PANDO, DÁVILA

S282 Rathod N., Surve A. 2015 Test orchestration a framework for Continuous
Integration and Continuous deployment
S283 Wahaballa A., Wahballa O., Abdellatief M., Xiong H., 2015 Toward unified DevOps model
Qin Z.
S284 Virmani M. 2015 Understanding DevOps & bridging the gap from
continuous integration to continuous delivery
S285 Chen J., Xu X., Osterweil L.J., Zhu L., Brun Y., Bass L., 2015 Using simulation to evaluate error detection
Xiao J., Li M., Wang Q. strategies: A case study of cloud-based deploy-
ment processes
S286 Engblom J. 2015 Virtual to the (near) end: Using virtual platforms
for continuous integration
S287 Farroha B.S., Farroha D.L. 2014 A Framework for Managing Mission Needs, Com-
pliance, and Trust in the DevOps Environment
S288 Harrer S., Röck C., Wirtz G. 2014 Automated and Isolated Tests for Complex Mid-
dleware Products: The Case of BPEL Engines
S289 Fitzgerald B., Stol K. 2014 Continuous software engineering and beyond:
Trends and challenges
S290 Cois C.A., Yankel J., Connell A. 2014 Modern DevOps: Optimizing software develop-
ment through effective system interactions
S291 Wright S.A., Druta D. 2014 Open source and standards: The role of open
source in the dialogue between research and
standardization
S292 Hussaini S.W. 2014 Strengthening harmonization of Development
(Dev) and Operations (Ops) silos in IT environ-
ment through systems approach
S293 Bellomo S., Ernst N., Nord R., Kazman R. 2014 Toward Design Decisions to Enable Deployabil-
ity: Empirical Study of Three Projects Reaching
for the Continuous Delivery Holy Grail
S294 Erculiani F., Abeni L., Palopoli L. 2014 UBuild: Automated testing and performance
evaluation of embedded linux systems
S295 Neely S., Stolt S. 2013 Continuous Delivery? Easy! Just Change Every-
thing (Well, Maybe It Is Not That Easy)
S296 Schaefer A., Reichenbach M., Fey D. 2013 Continuous integration and automation for DevOps
S297 Feitelson D.G., Frachtenberg E., Beck K.L. 2013 Development and Deployment at Facebook
S298 Meyer S., Healy P., Lynn T., Morrison J. 2013 Quality Assurance for Open Source Software
Configuration Management
S299 Gohil K., Alapati N., Joglekar S. 2011 Towards behavior driven operations (BDOps)

ACKNOWLEDGMENTS Proc. 17th Int. Conf. on Advances in ICT for Emerging

Regions, ICTer 2017, Colombo, 2017, vol. 2018, pp. 19–
Authors recognize reviews from members of Grupo de
25.
Investigación y Desarrollo en Ingeniería de Software- Pon-
2. Nicolau de França, B.B., Jeronimo, H., and Travas-
tificia Universidad Católica del Perú GIDIS-PUCP).
sos, G.H., Characterizing DevOps by hearing multiple
voices, Proc. 30th Brazilian Symp. on Software Engineer-
ing, SBES’16, Maringa, 2016, pp. 53–62.
3. Elliot, S., DevOps and the Cost of Downtime: Fortune
CONFLICT OF INTEREST 1000 Best Practice Metrics Quantified, 2015.
The authors declare that they have no conflicts of interest. 4. Ebert, C., Gallardo, G., Hernantes, J., and Serrano, N.,
DevOps, IEEE Software, 2016, vol. 33, no. 3, pp. 94–
100.
REFERENCES 5. Riungu-Kalliosaari, L., Mäkinen, S., Lwakatare, L.E.,
1. Samarawickrama, S.S. and Perera, I., Continuous Tiihonen, J., and Männistö, T., DevOps adoption ben-
scrum: a framework to enhance scrum with DevOps, efits and challenges in practice: a case study, Proc. Int.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

 SOFTWARE TESTING IN THE DEVOPS CONTEXT 683

Conf. on Product-Focused Software Process Improve- Sci. Workshop of XP2016 (XP’16 Workshops), Edin-
ment, Trondheim, 2016, vol. 10027 LNCS, pp. 590– burgh, 2016, pp. 1–11.
597. 21. Ghantous, G.B. and Gill, A., DevOps: concepts, prac-
6. Stillwell, M. and Coutinho, J.G.F., A DevOps ap- tices, tools, benefits and challenges, Proc. PACIS 2017,
proach to integration of software components in an EU Langkawi, 2017, p. 96.
research project, Proc. 1st Int. Workshop on Quality- 22. Lwakatare, L.E., Kuvaja, P., and Oivo, M., Dimen-
Aware DevOps, QUDOS 2015, Bergamo, 2015, pp. 1–6. sions of DevOps, Proc. Int. Conf. on Agile Software De-
7. Céspedes, D., Angeleri, P., Melendez, K., and Dávila, A., velopment, Helsinki, 2015, pp. 212–217.
Software product quality in DevOps contexts: a system-
23. Katal, A., Bajoria, V., and Dahiya, S., DevOps: bridg-
atic literature review, Proc. Int. Conf. on Software Process
ing the gap between development and operations, Proc.
Improvement, 2020, vol. 1071, pp. 51–64.
3rd Int. Conf. on Computing Methodologies and Commu-
8. Perera, P., Silva, R., and Perera, I., Improve software nication, ICCMC 2019, Erode, 2019, pp. 1–7.
quality through practicing DevOps, Proc. 17th Int. Conf.
on Advances in ICT for Emerging Regions, ICTer 2017, 24. Kamuto, M.B. and Langerman, J.J., Factors inhibiting
Colombo, 2017, vol. 2018, pp. 13–18. the adoption of DevOps in large organisations: South
African context, Proc. 2nd IEEE Int. Conf. on Recent
9. Elberzhager, F., Arif, T., Naab, M., Suß, I., and Ko- Trends in Electronics, Information and Communication
ban, S., Software quality. complexity and challenges of Technology, Bangalore, 2017, vol. 2018, pp. 48–51.
software engineering in emerging technologies, Proc.
SWQD: Int. Conf. on Software Quality, Vienna, 2017, 25. Zimmerer, P., Strategy for continuous testing in
vol. 269, pp. 33–44. iDevOps, Proc. 40th IEEE/ACM Int. Conf. on Software
Engineering, Gothenburg, 2018, pp. 532–533.
10. Jones, S., Noppen, J., and Lettice, F., Management
challenges for devops adoption within UK SMEs, Proc. 26. Fowler, M., Continuous Integration, 2006.
2nd Int. Work. Qual. DevOps QUDOS 2016, Co-Located https://www.martinfowler.com/articles/continuousIn-
with ISSTA 2016, Saarbrücken, 2016, pp. 7–11. tegration.html. Accessed Nov. 28, 2020.
11. Soni, M., End to end automation on cloud with build 27. Parnin, C., et al., The Top 10 adages in continuous de-
pipeline: the case for DevOps in insurance industry, ployment, IEEE Software, 2017, vol. 34, no. 3, pp. 86–
continuous integration, continuous testing, and contin- 95.
uous delivery, Proc. IEEE Int. Conf. on Cloud Computing 28. Fowler, M., Continuous delivery, May 30, 2013.
in Emerging Markets, CCEM 2015, Bangalore, 2015, https://martinfowler.com/bliki/ContinuousDeliv-
pp. 85–89. ery.html. Accessed Nov. 28, 2020.
12. Senapathi, M., Buchan, J., and Osman, H., DevOps 29. Fitzgerald, B. and Stol, K.J., Continuous software en-
capabilities, practices, and challenges: insights from a gineering and beyond: trends and challenges, Proc. 1st
case study, ACM Int. Conf. Proc. Ser., 2018, part F1377, Int. Workshop on Rapid Continuous Software Engineer-
pp. 1–11. ing, RCoSE 2014, Hyderabad, 2014, pp. 1–9.
13. Chen, L., Continuous delivery: overcoming adoption 30. Hüttermann, M., DevOps for Developers, 1st ed., New
challenges, J. Syst. Software, 2017, vol. 128, pp. 72–86. York: Apress, 2012.
14. Valdivia, J.A., Lora-González, A., Limón, X., Cortes- 31. ISO/IEC/IEEE 24765:2017: Systems and Software En-
Verdin, K., and Ocharán-Hernández, J.O., Patterns re- gineering – Vocabulary, Geneva, 2017.
lated to microservice architecture: a multivocal litera-
ture review, Program. Comput. Software, 2020, vol. 46, 32. IEEE Computer Society, SWEBOK V3.0, IEE, 2014.
no. 8, pp. 594–608. 33. Gupta, R.K., Manikreddy, P., and Gv, A., Challenges
15. Debois, P., Agile infrastructure & operations, Proc. in adapting agile testing in a legacy product, Proc. 11th
2008 Agile Conf., Toronto, 2008. http://www.je- IEEE Int. Conf. on Global Software Engineering, ICGSE
di.be/presentations/agile-infrastructure-agile- 2016, Orange County, CA, 2016, pp. 104–108.
2008.pdf. Accessed June 28, 2020. 34. Jeeva Padmini, K.V., Kankanamge, P.S., Bandara,
16. Virmani, M., Understanding DevOps & bridging the H.M.N.D., and Perera, G.I.U.S., Challenges faced by
gap from continuous integration to continuous delivery, agile testers: a case study, Proc. 4th Int. Multidisciplinary
Proc. 5th Int. Conf. on Innovative Computing Technology, Moratuwa Engineering Research Conf. MERCon’2018,
INTECH 2015, Galicia, 2015, pp. 78–82. Moratuwa, 2018, pp. 431–436.
17. Mullaguru, S.N., Changing scenario of testing para- 35. Coutinho, J.C.S., Andrade, W.L., and Machado, P.D.L.,
digms using DevOps – a comparative study with classi- Requirements engineering and software testing in agile
cal models, Global J. Comput. Sci. Technol., 2015, methodologies: a systematic mapping, Proc. 23rd Bra-
vol. 15, no. 2, pp. 23–27. zilian Symp. on Software Engineering, SBES 2019, Sal-
18. Chernonozhkin, S.K., Automated test generation and vador, 2019, pp. 322–331.
static analysis, Program. Comput. Software, 2001, vol. 27, 36. Cruzes, D.S., Moe, N.B., and Dyba, T., Communica-
no. 2, pp. 86–94. tion between developers and testers in distributed con-
19. Kuliamin, V.V., Petrenko, A.K., Kossatchev, A.S., and tinuous agile testing, Proc. 11th IEEE Int. Conf. on
Burdonov, I.B., The UniTesK approach to designing Global Software Engineering, ICGSE 2016, Orange
test suites, Program. Comput. Software, 2003, vol. 29, County, CA, 2016, pp. 59–68.
no. 6, pp. 310–322. 37. Fowler, M., Test Pyramid, 2012. https://martinfowl-
20. Jabbari, R., Ali, N., and Petersen, K., What is DevOps? er.com/bliki/TestPyramid.html. Accessed July 12,
A systematic review on definitions and practices, Proc. 2020.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022

684 PANDO, DÁVILA

38. Crispin, L. and Gregory, J., Agile Testing: a Practical 45. Shahin, M., Babar, M.A., Zahedi, M., and Zhu, L.,
Guide for Testers and Agile Teams, 1st ed., Indiana: Ad- Beyond continuous delivery: an empirical investigation
dison-Wesley, 2008. of continuous deployment challenges, Proc. Int. Symp.
39. Sane, P., A brief survey of current software engineering on Empirical Software Engineering and Measurement,
practices in continuous integration and automated ac- Toronto, 2017, vol. 2017, pp. 111–120.
cessibility testing, Proc. Int. Conf. on Wireless Communi- 46. Shahin, M., Ali Babar, M., and Zhu, L., Continuous
cations, Signal Processing and Networking, WiSPNET integration, delivery and deployment: a systematic re-
2021, 2021, pp. 130–134. view on approaches, tools, challenges and practices,
40. Alnafessah, A., Gias, A.U., Wang, R., Zhu, L., Casale, G., IEEE Access, 2017, vol. 5, pp. 3909–3943.
and Filieri, A., Quality-aware DevOps research: where
do we stand?, IEEE Access, 2021, vol. 9, pp. 44476– 47. Rajapakse, R.N., Zahedi, M., Ali Babar, M., and
44489. Shen, H., Challenges and solutions when adopting
41. Daoudagh, S., Lonetti, F., and Marchetti, E., Contin- DevSecOps: a systematic review, Inf. Software Technol.,
uous development and testing of access and usage con- 2021, vol. 141, p. 106700.
trol, Proc. 2020 European Symp. on Software Engineer- 48. Petersen, K., Vakkalanka, S., and Kuzniarz, L., Guide-
ing, 2020, pp. 51–59. lines for conducting systematic mapping studies in soft-
42. Pinto, G., Castor, F., Bonifacio, R., and Rebouças, M., ware engineering: an update, Inf. Software Technol.,
Work practices and challenges in continuous integra- 2015, vol. 64, pp. 1–18.
tion: a survey with Travis CI users, Software Pract. Exp.,
2018, vol. 48, no. 12, pp. 2223–2236. 49. Kuhrmann, M., Diebold, P., and Münch, J., Software
43. Mascheroni, M.A. and Irrazábal, E., Continuous test- process improvement: a systematic mapping study on
ing and solutions for testing problems in continuous de- the state of the art, Peer J. Comput. Sci., 2016, vol. 2016,
livery: a systematic literature review, Comput. Sist., no. 5.
2018, vol. 22, no. 3, pp. 1009–1038. 50. Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M.,
44. Laukkanen, E., Itkonen, J., and Lassenius, C., Prob- and Chatzigeorgiou, A., Identifying, categorizing and
lems, causes and solutions when adopting continuous mitigating threats to validity in software engineering
delivery – a systematic literature review, Inf. Software secondary studies, Inf. Software Technol., 2019,
Technol., 2017, vol. 82, pp. 55–79. vol. 106, pp. 201–230.

PROGRAMMING AND COMPUTER SOFTWARE Vol. 48 No. 8 2022