Writing a literature review is a critical component of academic research, demanding a high level of

dedication, analytical skill, and an in-depth understanding of the subject matter. The process involves
sifting through an extensive array of academic literature to identify relevant studies, analyze their
findings, and synthesize the available information to provide new insights or contribute to the
existing body of knowledge. This task, however, is far from straightforward and presents several
challenges that can make it a daunting endeavor for many researchers, especially those early in their
careers or those juggling multiple commitments.

The complexity of writing a literature review stems from several key factors. Firstly, identifying the
right sources requires a thorough understanding of the research topic, including current trends, key
theories, and major debates within the field. Researchers must navigate through databases and
libraries, often wading through an overwhelming amount of information to find those pieces that are
truly relevant to their research question.

Once the relevant literature is identified, the process of data extraction begins. This involves
carefully reading and analyzing each source, identifying the methodologies used, the findings
reported, and the conclusions drawn by the authors. Researchers must then critically evaluate the
quality and reliability of these studies, comparing and contrasting different pieces of research to
identify patterns, themes, and gaps in the literature. This step is crucial for building a solid
foundation for the literature review but requires a high level of critical thinking and analytical skills.

Furthermore, synthesizing the extracted data into a coherent and compelling literature review is a
significant challenge. Researchers must not only summarize the existing research but also critically
analyze it, identifying connections between studies, assessing their contributions to the field, and
highlighting areas where further research is needed. This requires a deep understanding of the subject
area, excellent writing skills, and the ability to construct a persuasive argument that contributes to the
scholarly conversation.

Given these challenges, it's understandable why many researchers seek professional assistance with
their literature reviews. For those looking for reliable help, ⇒ StudyHub.vip ⇔ offers specialized
services tailored to the needs of academic researchers. This platform provides expert guidance
throughout the process, from identifying relevant literature to synthesizing the data into a well-
structured review. By leveraging the expertise of professionals who are well-versed in the intricacies
of academic writing, researchers can navigate the complexities of the literature review process more
efficiently, ensuring a high-quality output that contributes meaningfully to their field of study.
Cite this chapter as: Li T, Higgins JPT, Deeks JJ (editors). Heshmat MD; PhD candidate Stage 1
Planning the review Phase 0 Identification of the need for a review Phase 1 Preparation of a
proposal for a review Phase 2 Development of a review protocol R. If you wish to reuse any or all of
this article please use the link below which will take you to the Copyright Clearance Center’s
RightsLink service. The PRISMA statement for reporting systematic reviews and meta-analyses of
studies that evaluate health care interventions: explanation and elaboration. The full list of inclusion
and exclusion criteria for journal articles is published in additional file 2. Plagiarism of results data in
the form of duplicated publication (either by the same or by different authors) may, if undetected,
lead to study participants being double counted in a synthesis. Availability of corpora, and copyright
issues 4.5 There are several corpora described in the literature, many with manual gold-standard
labels (see Table 4 ). Choice of data extraction tools for systematic reviews depends on resources and
review complexity. Extraction templates and approaches should be determined by the needs of the
specific review. None of the handbooks makes an explicit recommendation on who should be
involved in piloting the data extraction form or their expertise. As previously mentioned, users of
systematic reviews need sufficient information on non-outcome data to make sense of the underlying
primary studies and assess their applicability. International Network of Agencies for Health
Technology Assessment (INAHTA). Because a study may be associated with multiple reports, it is
important to record the study ID as well as the report ID. Clear definition of perspectives and
decisions on indirect evidence, sampling and use of existing QES help in targeting eligibility criteria.
One might suspect that if particularly stellar performance were demonstrated by a project, those data
would be prominently advertised. A significant irony about the body of papers included in this
review is that there is a large amount of missingness related to the performance of such methods. The
Cochrane Handbook for Systematic Reviews is available online, and chapter 7.6 in Part 2 covers data
extraction. COPY CITATION DETAILS track receive updates on this article Track an article to
receive email alerts on any updates to this article. The authors included more than 50 publications in
this version of their review that addressed extraction of data from abstracts, while less (26%) used
full texts. Saldanha IJ, Li T, Yang C, Ugarte-Gil C, Rutherford GW, Dickersin K. BMC medical
research methodology. 2020;20(1):259. doi: 4. Mathes T, Klasen P, Pieper D. For each item in the
tool, a description of what happened in the study is required, which may include verbatim quotes
from study reports. Lastly, we note that some of the included sources referenced more
comprehensive guidance such as the Cochrane Handbook. PubMed Abstract 64. Amini I, Martinez
D, Aliod DM: Overview of the ALTA. Cochrane Handbook for Systematic Reviews of Interventions
version 6.4 (updated August 2023). Where applicable, the name will appear in both the original
language and in English. For sentence classification the NICTA gold-standard 52 is used by eight
others, and the automatically labelled corpus by Jin and Szolovits 96 is used by five others and was
adapted once. Avoid asking a question in a way that the response may be left blank. Partly Is the
statistical analysis and its interpretation appropriate. Manually annotated: 344 IC, 341 O, and 144 P
and more derived by automatic labelling.
For the updated review we used our living review web application to retrieve all publications with
the exception of the items retrieved by EPPI-Reviewer (these are added to the dataset separately).
Most included publications in the base-review did not report their hardware specifications, though
five publications (9%) did. We searched these websites for potentially relevant documents and
downloaded these. Agreement of coded items before reaching consensus can be quantified, for
example using kappa statistics (Orwin 1994), although this is not routinely done in Cochrane
Reviews. In accordance with other reviews they describe screening as the most frequently automated
step, while automated data extraction tools are lacking due to the complexity of the task. These
textbooks included information on data extraction in systematic reviews, but none of them focussed
on this topic exclusively. Systematic reviews in medical education: A practical approach: AMEE
Guide 94. We leave it for reference to publications that may still use these methods in the future.
3.4.2 Transparency of methods 3.4.2.1 Is there a description of the algorithms used. You can create
evidence and summary tables to describe study characteristics, results, or both. Our secondary
analysis on recommendations for non-interventional systematic reviews is included in Additional file
5 and the detailed results for the primary analysis in Additional file 6. HOW THIS STUDY MIGHT
AFFECT RESEARCH, PRACTICE OR POLICY This paper offers guidance for researchers when
conducting a rapid QES and informs commissioners of research and policy-makers what to expect
when commissioning such a review. Instead of using this theme as one of the review findings, the
reviewers should read and interpret beyond the given description in an article, compare and contrast
themes, findings from one article with findings and themes from another article to find similarities
and differences and to understand and explain bigger picture for their readers. Where feasible,
information should be sought (and presented in the review) that is sufficient for replication of the
interventions under study. With this living review we aim to review the literature continually. You
work at the same institute as any of the authors. The table below provides some software to help you
design data extraction forms using templates. Most often, the source of the disagreement is an error
by one of the extractors and is easily resolved. There were some notable exceptions to our findings.
Data extraction form and quality assessment methods used - Does therapeutic writing help people
with long-term conditions. Notwithstanding stereotypes, EDs affect individuals from all racial and
ethnic backgrounds. It is likely that the particular research context (e.g. see Pham et al., 2021 1 ) will
have a large degree of influence on the performance metrics to be had if they can be determined. Ip
S, Hadar N, Keefe S, Parkin C, Iovin R, Balk EM, Lau J. BMJ disclaims all liability and
responsibility arising from any reliance placed on the content. It is usually safest to assume that they
were not reported. Norman C, Leeflang M, Neveol A: Data Extraction and Synthesis in Systematic
Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process. The
scope of data extraction methods can be applied to the full text or to abstracts within each eligible
publication’s corpus. Review authors should familiarize themselves with reporting guidelines for
systematic reviews (see online Chapter III and the PRISMA statement; (Liberati et al 2009) to ensure
that relevant elements and sections are incorporated. Neither human nor machine can instantly
perform perfect data extraction or labelling, 37 and thus errors done in earlier classification steps can
be carried forwards and accumulate. To some extent the results reflect the empirical evidence from
comparative methods research. The reference month for the compilation of this list was January
2019, the list is included in additional file 1.
This creates opportunities for support through intelligent software, which identify and extract
information automatically. Therefore, the potential time-saving and utility of full text data extraction
is much higher because more time can be saved by automation and it provides automation that more
closely reflects the work done by systematic reviewers in practice. Because a study may be
associated with multiple reports, it is important to record the study ID as well as the report ID. Two
reviewers will independently screen titles and abstracts and full-texts against specified inclusion and
exclusion criteria. After initial testing, accuracy of the extracted data should be checked against the
source document or verified data to identify problematic areas. For the complete list of frameworks
please see Appendix A and D in Underlying data. 127 3.4.5 Internal and external validity of the
model 3.4.5.1 Does the dataset or assessment measure provide a possibility to compare to other tools
in the same domain. The authors did not carry out a systematic search for textbooks, but included
textbooks from a broad range of disciplines including medicine, nursing, education, health library
specialties and the social sciences published between 1998 and 2017. Reference 102 shows precision-
recall plots for different classification thresholds. Regardless of the collection methods, precise
definitions of adverse effect outcomes and their intensity should be recorded, since they may vary
between studies. However, there are templates and guidance available. Review authors should
familiarize themselves with reporting guidelines for systematic reviews (see online Chapter III and
the PRISMA statement; (Liberati et al 2009) to ensure that relevant elements and sections are
incorporated. In these cases, the comparability is limited because those publications used different
data sets, which can influence the difficulty of the data extraction task and lead to better results
within for example structured datasets or topic-specific datasets. 3.4.5.2 Are explanations for the
influence of both visible and hidden variables in the dataset given. PLoS Medicine 2009; 6. Safer DJ.
Design and reporting modifications in industry-sponsored comparative psychopharmacology trials.
This is true for most publications published before 2010, and about 50% of the literature published in
more recent years. Norman C, Leeflang M, Neveol A: Data Extraction and Synthesis in Systematic
Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process. Clinical
orthopaedics and related research. 2007;455:23-9. 2. Centre for Reviews and Dissemination. Here at
UNC, we use a systematic review software called Covidence. We therefore encourage developers of
guidelines documents for systematic reviews to provide more comprehensive recommendations on
developing and piloting data extraction forms and the data extraction process. Pham B, Jovanovic J,
Bagheri E, Antony J, et al.: Text mining to support abstract screening for knowledge syntheses: a
semi-automated workflow. Plagiarism of results data in the form of duplicated publication (either by
the same or by different authors) may, if undetected, lead to study participants being double counted
in a synthesis. In most cases, this was due to the text-types mined in the publications. The Systematic
Review Toolbox The SR Toolbox is a community-driven, searchable, web-based catalogue of tools
that support the systematic review process across multiple domains. The study report may not fully
reflect how the study was actually conducted. Typically, aspects that should be collected are those
that could (or are believed to) affect presence or magnitude of an intervention effect and those that
could help review users assess applicability to populations beyond the review. Problems with the data
collection form may surface after pilot testing has been completed, and the form may need to be
revised after data extraction has started. Thus, discussion among the authors is a sensible first step.
We found a variety of topics discussed in these publications and summarised them under seven
different domains. The Creative Commons Public Domain Dedication waiver ( ) applies to the data
made available in this article, unless otherwise stated in a credit line to the data. Development, testing
and use of data extraction forms in systematic Data extraction forms link systematic reviews with
primary research and provide the foundation for appraising, analysing, summarising and interpreting
a body of evidence. The (semi) automation of data extraction in systematic reviews is an advantage
for researchers and ultimately for evidence-based clinical practice.
The authors declare that they have no competing interests. Mayo-Wilson E, Fusco N, Li TJ, Hong H,
Canner JK, Dickersin K, MUDS Investigators. STRENGTHS AND LIMITATIONS OF THIS
STUDY This protocol will ensure transparency in methodology to reduce the likelihood of reviewing
bias. If the review includes non-randomized studies, appropriate features of the studies should be
described (see Chapter 24 ). A systematic review of machine-learning for systematic review
automation, published in Portuguese in 2020, included 35 publications. Provide space for notes,
regardless of whether paper or electronic forms are used. A comprehensive data charting form will
be developed in Excel and used to extract relevant information from studies. Therefore, the
literature for automation of data extraction focuses on RCTs, and their related PICO elements.
Furthermore, in the end, you have to map your results as per your research questions. A full
systematic review was completed in 2 weeks using automation tools: a case study. Extracted data
will be presented in tables, charts and visual maps. Of the included publications in the review
update, 73 out of 76 (97%) provided descriptions of their dataset and its characteristics. Because
CSRs also incorporate tables and figures, with appendices containing the protocol, statistical analysis
plan, sample case report forms, and patient data listings (including narratives of all serious adverse
events), they can be thousands of pages in length. The list of textbooks was also based on a previous
study not intended to cover the literature in full. It is likely that at least some HTA agencies have
internal documents that provide more specific recommendations. The arrival of transformer-based
methods in 2018 marked the last big change in the field, as documented by this LSR. Formulate the
review question Search the Literature Medical Study Types Quality assessment Data extraction
Analyse and synthesise Further reading Research Support This link opens in a new window EndNote
for Systematic Reviews This is the process where you take data from each of the studies and store it
in a data extraction form. You can mention the above research papers list in a spreadsheet or in
another tool. Those who conduct systematic reviews know well the degree of missing information
sought to summarize a group of studies. Keywords: Data extraction; Evidence synthesis; Systematic
review methods. Evidence in support of duplicate data extraction comes from several indirect
sources. Basic study parameters (diagnosis criteria, research duration, and total sample size),
interventions (drugs, dosages, formulae, and dosage form), and. Not approved Fundamental flaws in
the paper seriously undermine the findings and conclusions Reviewer Reports Invited Reviewers 1 2
3 Version 2 (update) 09 Oct 23 Version 1 19 May 21 read read read Emma McFarlane, National
Institute for Health and Care Excellence, London, UK Kathryn A. Such an approach prevents the
review authors from understanding the article’s conceptual contribution, developing rigorous
synthesis and drawing reasonable interpretations of results from an individual article. This web only
file has been produced by the BMJ Publishing Group from an electronic file supplied by the
author(s) and has not been edited for content. View their short introductions to data extraction and
analysis for more information. Typical settings of healthcare intervention studies include acute care
hospitals, emergency facilities, general practice, and extended care facilities such as nursing homes,
offices, schools, and communities. Yes Have the search and update schedule been clearly defined and
justified. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:210. Marshall IJ,
Kuiper J, Wallace BC. Carlisle JB, Dexter F, Pandit JJ, Shafer SL, Yentis SM.
Commonly, randomized controlled trials (RCT) text was at least one of the target text types used in
the included publications. 3.2.3.2 Data extraction targets Mining P, IC, and O elements is the most
common task performed in the literature of systematic review (semi-) automation (see Table A1 in
Underlying data, 127 and Figure 6 ). Society for Social Work and Research Washington, DC January
15, 2012. Mission. Researchers often use a form or table to capture the data they will then
summarize or analyze. Systematic Reviews as Topic Patient Care Data availability statement No data
are available. For thousands of years it has been the traditional territory of the Huron-Wendat, the
Haudenosaunee, and most recently, the Mississaugas of the Credit River. Demner-Fushman D, et al.:
Finding medication doses in the liteature. CRD’s guidance for undertaking reviews in health care.
Possible reasons for this are concerns over copyright, or malfunctioning download links from
websites mentioned in older publications. Of the included publications in the base-review, 17 out of
53 (32%) described trade-offs or provided plots or tables showing the development of evaluation
scores if certain parameters were altered or relaxed. Recall (i.e., sensitivity) is often described as the
most important metric for systematic review automation tasks, as it is a methodological demand that
systematic reviews do not exclude any eligible data. Data Extraction Excel is the most basic tool for
the management of the screening and data extraction stages of the systematic review process. Such
investigations should address how the tool will fit into existing workflows. A bibliography of the
included studies should always be created, particularly if you are intending to publish your review.
See Horton et al., (2010) 'Systematic review data extraction: cross-sectional study showed that
experience did not increase accuracy', and Jones et al., (2005) 'High prevalence but low impact of
data extraction and reporting errors were found in Cochrane systematic reviews' for more on this
topic'. Consequently, the findings of a systematic review depend critically on decisions relating to
which data from these studies are presented and analysed. Question 4: are there differences in
reported ED prevalence across ethnicity. To avoid any misconceptions, we emphasise that by
aggregating these results we by no means suggest that all items are of equal importance. Revisions of
the manuscript for important intellectual content: RBB, DP, AW. Macro-scores were used in one
publication. 37 Micro scores were used by Fiszman et al. 47 for class-level results. Finally, judicious
use of Grading of Recommendations Assessment, Development and Evaluation approach for
assessing the Confidence of Evidence from Reviews of Qualitative research assessments and of
software as appropriate help to achieve a timely and useful review product. Huang KC, et al.: PICO
element detection in medical text without metadata: are first sentences enough. Explainability and
interpretability of data extraction systems 4.4 The neural networks or machine-learning models from
publications included in this review learn to classify and extract data by adjusting numerical weights
and by applying mathematical functions to these sets of weights. However, this information was not
always clearly reported. 4.1.4 Target texts Reports of randomised controlled trials were the most
common texts used for data extraction. Stage 1: identifying the research questions The first stage of
a scoping review is to develop research questions which are informed by the purpose and objectives
of the scoping review. 31 Based on the aforementioned objectives of this scoping review, we have
formulated six research questions. SRDR (Systematic Review Data Repository) is a Web-based tool
for the extraction and management of data for systematic review or meta-analysis. Stage 2:
identifying the relevant literature Information sources. Several studies have shown that data
extraction errors are frequent in systematic reviews, especially regarding outcome data. Annals of
Internal Medicine 1997; 127: 531-537. Meinert CL. Clinical trials dictionary: Terminology and usage
recommendations. Electronic searches for text can provide a useful aid to locating information
within a report. Not approved Fundamental flaws in the paper seriously undermine the findings and
conclusions Reviewer Reports Invited Reviewers 1 2 3 Version 2 (update) 09 Oct 23 Version 1 19
May 21 read read read Emma McFarlane, National Institute for Health and Care Excellence, London,
UK Kathryn A. It provides information and tips on searching for literature and managing your
resources.
Rathbone et al., 28 for example, used hand-crafted Boolean searches specific to a systematic
review’s PICO criteria to support the screening process of a review within Endnote. Quality
Appraisal Evidence Synthesis Interpret Results Reporting with PRISMA 2020 Once the search and
selection of studies for inclusion is completed the next step is to read the full text of each article
identified for inclusion in the Systematic Review and extract the relevant data using a standardised
data extraction form. The search strategy for our database search was pragmatic for reasons stated in
the methods and may have missed some relevant articles. Details available here A good data
collection form should minimize the need to go back to the source documents. Three examples of a
data extraction form are below: Data Extraction Form Example (suitable for small-scale literature
review of a few dozen studies) This example was used to gather data for a poster reporting a
literature review of studies of interventions to increase Emergency Department throughput. For
example, if authors of an included article intended to develop a new scale and test its psychometric
properties. It may be important to record the range of analysis options available, even if not all are
extracted in detail. Electronic health records and non-trial data were common, and we created a list
of datasets that would be excluded in this category (see more information in Underlying data:
Appendix B 127 ). The (semi) automation of data extraction in systematic reviews is an advantage
for researchers and ultimately for evidence-based clinical practice. After title and abstract screening,
15 potentially relevant full texts remained. Related systematic reviews and overviews 1.1 This
review is, to the best of our knowledge, the only living systematic review (LSR) of data extraction
methods. Automating data extraction in systematic reviews: a systematic review. These newer
methods, 14 however, are used in contemporary systematic review automation software which will
be reviewed in the scope of this living review. Three examples of data extraction forms are below:
Data Extraction Form Example (suitable for small-scale literature review of a few dozen studies)
This form was used to gather data for a poster reporting a literature review of studies of
interventions to increase Emergency Department throughput. Additionally, the authors may want to
consider commenting on the topic areas covered by the included studies and whether that has an
impact on any of the metrics measured. Quantitative and qualitative summaries of extracted data will
be provided. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of
research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and
monitoring progress Phase 7 Data synthesis R. Reporting of the results will follow PRISMA ScR
guidelines. 32 Ethics and dissemination As scoping reviews involve the methodological examination
and integration of existing literature and resources, ethical approval is not required. You can mention
the above research papers list in a spreadsheet or in another tool. The following is an example of
elements to include in a standardised data extraction form. The Sto:lo have an intrinsic relationship
with what they refer to as S’olh Temexw (Our Sacred Land), therefore we express our gratitude and
respect for the honour of living and working in this territory. Download the ebook on Data
Extraction for Intervention Reviews Tailored for Intervention Systematic Review researchers and
shaped by insights from the global systematic review community, this tool-agnostic guide is a must-
have for both seasoned researchers and those just starting out. The dataset, code and weights of
trained models are available in Underlying data: Appendix C. 127 This includes plots of each
model’s evaluation in terms of area under the curve (AUC), accuracy, F1, recall, and variance of
cross-validation results for every metric. The final categories are a mixture of non-machine-leaning
automation (application programming interface (API) and metadata retrieval, PDF extraction, rule-
base), classic machine-learning (naive Bayes, decision trees, SVM, or other binary classifiers) and
neural or deep-learning approaches (convolutional neural network (CNN), LSTM, transformers, or
word embeddings). If a meta-analysis is also being completed, extract raw and refined data from
each result in the study. Six (8%) implemented publicly available tools Conclusions: This living
systematic review presents an overview of (semi)automated data-extraction literature of interest to
different types of literature review. It will often be necessary to perform calculations to obtain the
required statistics for presentation or synthesis. Additional information specific to our research
questions will also be extracted. An rQES team, in particular, cannot afford any extra time or
resource requirements that might arise from either a misunderstanding of the review question, an
unclear picture of user requirements or an inappropriate choice of methods. All abstracts were
screened by two independent reviewers.
P, population; I, intervention; C, comparison; O, outcome. Yes Are the rationale for, and objectives
of, the Systematic Review clearly stated. Federal government websites often end in.gov or.mil.
Before sharing sensitive information, make sure you're on a federal government site. Review authors
have the opportunity to work with trialists, journal editors, funders, regulators, and other
stakeholders to make study data (e.g. CSRs, IPD, and any other form of study data) publicly
available, increasing the transparency of research. Established frameworks for extracting data have
been created. You are an Editor for the journal in which the article is published. This follow-up task
will be a major point of interest for many who will follow updates to this paper. We have not
formally extracted information on cross-referencing between documents, however. Between review
updates, trends for sharing data and code increased strongly: in the base-review, data and code were
available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new
publications. Provenance and peer review Not commissioned; internally peer reviewed. Journal of
Nervous and Mental Disease 2002; 190: 583-592. The US Food and Drug and Administration had
historically avoided releasing CSRs but launched a pilot programme in 2018 whereby selected
portions of CSRs for new drug applications were posted on the agency’s website. Primary research
methods include interviews, internet surveys, and focus groups. Effects of a demand-led evidence
briefing service on the uptake and use of research evidence by commissioners of health services: a
controlled before-and-after study. Closed-ended questions do not require post hoc coding and
provide better control over data quality than open-ended questions. Theories may be unearthed
within the topic search or be already known to team members, fro example, Theory of Planned
Behaviour. 29 Options for managing the quantity and quality of studies and data emerge during the
scoping (see above). Including your themes in the summary tables (see figure 1 ) demonstrates to the
readers that a robust method of data extraction and synthesis has been followed. CSRs are
particularly useful for identifying outcomes assessed but not presented to the public. In addition,
when non-systematic adverse events are reported based on quantitative selection criteria (e.g. only
adverse events that occurred in at least 5% of participants were included in the publication), use of
reported data alone may bias the results of meta-analyses. HTA documents were retrieved in
February 2019 and database searches conducted in December 2019. Yes Are sufficient details of the
methods and analysis provided to allow replication by others. Zhang et al. 12 included 49 references
on automation of data extraction fields such as diseases, outcomes, or metadata. There were some
notable exceptions to our findings. Additional information specific to our research questions will also
be extracted. Consider the following examples, but note that this is not an exhaustive list: Examples
of 'Non-Financial Competing Interests' Within the past 4 years, you have held joint grants, published
or collaborated with any of the authors of the selected paper. Of the included publications in the
review update, 73 out of 76 (97%) provided descriptions of their dataset and its characteristics.
Supplemental material The search results for each database will be documented and references will
be imported into the citation management system EndNote, where duplicates will be removed before
studies are screened for inclusion. The form also included an open field for comments in case any
additional items of interest were identified. As a final note, some parts of systematic reviews can
now be assisted by automation methods. Of the included publications in the base-review, 47 out of
53 (88%) described using at least one third-party framework for their data extraction systems. The
following list is likely to be incomplete, due to non-available code and incomplete reporting in the
included publications.