Statistics

Study designs: Part 8 ‑ Meta‑analysis (I)

Priya Ranganathan, Rakesh Aggarwal1
Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, Maharashtra, 1Director, Jawaharlal Institute of Postgraduate Medical
Education and Research, Puducherry, India

Abstract A systematic review is a form of secondary research that answers a clearly formulated research question
using systematic and defined methods to identify, collect, appraise, and summarize all the primary research
evidence on that topic. In this article, we look at meta‑analysis – the statistical technique of combining the
results of studies included in a systematic review.

Keywords: Research design, review, systematic, meta‑analysis

Address for correspondence: Dr. Rakesh Aggarwal, Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry, India.
E‑mail: [email protected]
Received: 02-09-20, Accepted: 05-09-20, Published: 06-10-20.

In a previous article in this series, we looked at systematic involved in conducting a systematic review in a recent
reviews – their methods, uses, and limitations.[1] Systematic article.[1] If the systematic review reveals that there are an
reviews are frequently followed by the use of a mathematical adequate number of studies with shared characteristics,
or statistical technique to pool the data from individual then the reviewers may consider performing a formal
studies included in the review, to obtain a single summary meta‑analysis. The systematic review process ensures that
measure of effect. This technique, known as meta‑analysis, the meta‑analysis includes data from all the available studies
is the focus of this article. that fulfill certain selection and/or quality criteria, on the
question being studied, in an unbiased manner. If this is
DEFINITION not done, then the results of the meta‑analysis may be
biased and hence unreliable.
The term “meta‑analysis” was first defined in 1976 by Glass
as “The analysis of analyses …… The statistical analysis of UNDERSTANDING HETEROGENEITY
a large collection of analysis results from individual studies
for the purpose of integrating the findings.”[2] A more recent The term “heterogeneity” refers to variability between the
definition of meta‑analysis is: “a statistical analysis which results of different studies on a research question. On any
combines or integrates the results of several independent question, results of the available studies are unlikely to be
clinical trials considered by the analyst to be combinable”.[3] identical, and a certain amount of variation between these
is expected just by chance. This variation, or statistical
REQUIREMENTS FOR A META‑ANALYSIS heterogeneity, depends on two factors. One of these is
the sample size, with smaller‑sized studies showing a greater
The performance of a meta‑analysis is contingent upon variation. The second relates to variability of the outcome
a good‑quality systematic review. We discussed the steps variable, e.g., the random variation being larger if the event

Access this article online This is an open access journal, and articles are distributed under the terms of the Creative Commons
Attribution‑NonCommercial‑ShareAlike 4.0 License, which allows others to remix, tweak, and
Quick Response Code:
Website: build upon the work non‑commercially, as long as appropriate credit is given and the new creations
are licensed under the identical terms.
www.picronline.org
For reprints contact: [email protected]

DOI:
10.4103/picr.PICR_283_20 How to cite this article: Ranganathan P, Aggarwal R. Study designs:
Part 8 ‑ Meta‑analysis (I). Perspect Clin Res 2020;11:178-81.

178 © 2020 Perspectives in Clinical Research | Published by Wolters Kluwer - Medknow

 Ranganathan and Aggarwal: Meta‑analysis

rate is very low or very high, and smaller if the event rate Furthermore, it does not provide a quantitative measure
is close to 50% for dichotomous variables. of the extent of heterogeneity.

In addition, studies on a particular research question The Higgins I2 test provides a numeric value, known as the
included in a systematic review often differ from each I2 statistic, for the degree of heterogeneity between studies
other somewhat in population characteristics – e.g., age and beyond what would be expected by chance. It can vary from
gender distribution, ethnicity, baseline severity of disease, 0 to 100% (or 0 to 1.00 ), with lower values indicating less
and presence of comorbid conditions – these are examples marked heterogeneity. This value represents the proportion
of clinical heterogeneity. Furthermore, the studies may of total variation across studies that is due to heterogeneity
have methodological heterogeneity, i.e., variations in rather than chance. Higgins also specified empiric cutoffs
dose, frequency, or route of administration of a drug used, of 25%, 50%, and 75% to indicate low, moderate and high
use of different drugs for a particular condition (e.g., studies heterogeneity respectively.[4]
assessing the efficacy of beta‑blockers in treating portal
hypertension may use a variety of beta‑blockers such HETEROGENEITY AND CHOICE OF MODEL:
RANDOM‑EFFECTS VERSUS FIXED‑EFFECTS
as propranolol or nadolol), blinding techniques (open,
single‑blind, or double‑blind), tools or exact measures
In a meta‑analysis, results of individual studies are
used to evaluate the outcome (e.g., blood pressure may be
combined to produce a single overall result. This does
measured by intra‑arterial or noninvasive techniques), time
not imply that the number of events and subjects can
points used for outcome assessment, etc. Hence, the studies
be simply aggregated across studies. Instead, a statistical
included in a meta‑analysis would be expected to show a
method is used to obtain a summary measure. Further, since
greater variability than may be expected by chance alone.
the included studies vary in sample size, event rate, and
The presence, the degree, and the nature of heterogeneity in individual results, these cannot be given equal importance.
methodology and results of the available studies influence Thus, each study is given a different weightage, depending
the decisions about whether the studies can be combined on its characteristics, with a study with more stable value
using a meta‑analysis and about the statistical tools to of the outcome measure being awarded greater weight.
use. The available studies must be sufficiently related for
Broadly, there are two types of methods to pool data in a
those to be pooled. This is not a statistical decision but
meta‑analysis – the fixed‑effects and the random‑effects
needs a careful evaluation by experts in the subject area
models.
of the research question. If there is obvious clinical and/
or methodological heterogeneity, it is appropriate to not
The fixed‑effects model assumes that all the studies in a
proceed with the meta‑analysis. Once a decision to proceed
review have a single true effect, and that any variations
with meta‑analysis is taken, a more formal assessment of
between the results of these studies represent a random
heterogeneity is done.
error, which is largely a reflection of their relative
TESTS FOR HETEROGENEITY sample sizes and the observed event rates. Therefore,
the fixed‑effects model assigns greater weightage to
Inter-study heterogeneity can be formally assessed. This larger studies and those with nearly equal rate of events
is most often done using one of the two tests, namely the and non-events, and less weightage to smaller studies
Cochran’s Q test and the Higgins I2 test. and those with either too few or too many events.
For the fixed‑effects model, at least three different
The Cochran’s Q test (also known as the Chi‑square test calculation techniques are available to assign weightage
for heterogeneity or the Chi‑square test for homogeneity) to studies – the inverse variance technique, the Peto
looks at whether the results of individual studies differ odds ratio, and the Mantel‑Haenszel method, and any
from the expected average effect, i.e., differ by more than of these may be used.
what would have been expected by chance. If the Cochran’s
Q test is positive (shows a statistically significant result or The random‑effects model assumes that the studies included
low P value), then the heterogeneity between the studies in a review are drawn from somewhat different populations
exceeds the random expectation. This test is limited by the of studies, with slightly different treatment effects. In
fact that its result is heavily dependent on the number of this model, larger studies are given proportionately
studies; if this number is very small or very large, the test less weightage, whereas smaller studies are given
tends to under‑ or overestimate heterogeneity, respectively. proportionately larger weight than in the fixed-effects
Perspectives in Clinical Research | Volume 11 | Issue 4 | October-December 2020 179
 Ranganathan and Aggarwal: Meta-analysis

model; thus, study weights are more similar under this continuous data, where each individual’s outcome is a
model. The DerSimonian and Laird method is the most numerical quantity (e.g., weight gain in kg); count and
commonly used random‑effects technique. rate (e.g., number of events per unit time, such as number
of episodes of diarrhea per year) data; or survival data
The confidence intervals of the summary effect obtained (time until an event of interest occurs).
using the random‑effects model are wider than those for
the fixed‑effects methods. For dichotomous data, the effect of an intervention (i.e.,
the difference in outcomes between the treated and the
There is little consensus on which of the two models to use. control group) can be represented as risk ratio, odds ratio,
Although it appears tempting to choose between these based or risk difference.[5] For continuous data, the treatment
on the measure of heterogeneity, this is not recommended, effect is most often represented as mean difference (or
and the clinical and methodological heterogeneity also need to the difference between the means in the treatment and
be taken into account, with the random‑effects model being
the comparator group). An alternative is the use of
used if one believes that there is sufficient heterogeneity. Some
standardized mean difference (calculated as the mean
authors recommend that one should always analyze the data
difference between the groups divided by the standard
and report results using both the models. Others believe that
deviation of data for all participants). This is nothing but
it is safer to always use the random‑effects model.
the mean difference expressed using the standard deviation
CHOICE OF EFFECT MEASURE as a unit. Its use allows one to pool studies which measure
the same outcome using different scales (e.g., improvement
The outcome data being compared and pooled can be in depression using different psychometric scales).
of different types. The main types include dichotomous
data, where each individual’s outcome is one of only two For rates and survival data, the effect measures used are
possible categorical responses (e.g., cured or not cured); rate ratio and hazard ratio, respectively.[5,6]

Figure 1: A typical forest plot showing its various components. At times, some components (e.g., the columns with original data from the included
studies) are not included, or further details are added (Figure adapted from Aggarwal et al).[7]

180 Perspectives in Clinical Research | Volume 11 | Issue 4 | October-December 2020

 Ranganathan and Aggarwal: Meta‑analysis

THE FOREST PLOT The results of tests of heterogeneity – the Chi‑square test
and the I2 test – are also reported. In this example, the
The results of a meta‑analysis are depicted using a Chi‑square test was significant with a low P value, and the
graphic known as a forest plot. A forest plot includes I2 value was 77%, both suggesting statistical heterogeneity
identifiers for individual studies included in the analysis, and supporting the choice of the random‑effects model.
the results of each study in brief and the weightage
given to each study, and provides a visual representation The results of individual studies and of the meta‑analysis
of the degree of heterogeneity between the results of are read in relation to a vertical line, which represents the
individual studies. “line of no effect,” i.e., a situation where the treatment
does not lead to any change in the outcome measure.
As an example, Figure 1 shows the forest plot of a This line is drawn at the value of 1.0 in case of ratio
meta‑analysis of studies comparing zinc supplementation
measures (e.g., odds ratio, risk ratio, etc., as in this case)
with placebo for the prevention of diarrhea in
or at 0 in case of linear measures (e.g., mean difference
healthy children.[7] The first column lists unique study
or standardized mean difference). Any horizontal lines
identifiers (usually the last name of the first author and
or diamonds which do not cross this line are deemed as
the year of publication). The next four columns depict
statistically significant.
the results of individual studies – in this case, the number
of diarrheal episodes and cumulative years of follow‑up
The above text provides basic information on a meta‑analysis.
in each study arm, i.e., the intervention arm and the
The technique has several additional nuances, which we
comparator. For other types of studies, the data reported
propose to deal with in the next article in this series.
may include means and standard deviations (for continuous
outcomes) or the number of events and the number of Financial support and sponsorship
participants (for binary outcomes). The results of each Nil.
study are visually depicted in the forest plot, with the
summary statistic (in this case, rate ratio) represented by a Conflicts of interest
square and a horizontal line representing the confidence There are no conflicts of interest.
intervals for the summary statistic. The confidence intervals
used most often are 95%, but another value (e.g., 99%) can REFERENCES
be used depending on the author’s choice. The weightage
given to each study is shown as a percentage of the total, 1. Ranganathan P, Aggarwal R. Study designs: Part 7‑Systematic reviews.
Perspect Clin Res 2020;11:97‑100.
and sometimes also as the relative sizes of the squares on 2. Glass, Gene V. Primary, Secondary, and Meta‑Analysis of Research.
the bar for each study. Educational Researcher, Vol. 5, no. 10; 1976. p. 3‑8. JSTOR, Available
from: www.jstor.org/stable/1174772. [Last accessed on 2020 Aug
The combined treatment effect, as determined by the 20].
meta-analysis, is shown using a diamond, with its center 3. Huque MF. Experiences with meta‑analysis in NDA submissions. Proc
Biopharm Section Am Stat Assoc 1988;2:28-33.
representing the overall summary statistic and the 4. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta‑analysis.
horizontal limbs (the horizontal spread of the diamond) Stat Med 2002;21:1539‑58.
depicting the confidence intervals. The model used to 5. Ranganathan P, Aggarwal R, Pramesh CS. Common pitfalls in
calculate the weightage (random‑effects in this example) is statistical analysis: Odds versus risk. Perspect Clin Res 2015;6:222‑4.
6. Sedgwick P, Joekes K. Interpreting hazard ratios. BMJ 2015;351:h4631.
mentioned in the header of the last column. This column 7. Aggarwal R, Sentz J, Miller MA. Role of zinc administration in
also has numeric values for the summary statistics and their prevention of childhood diarrhea and respiratory illnesses: A
confidence intervals. meta‑analysis. Pediatrics 2007;119:1120‑30.

Perspectives in Clinical Research | Volume 11 | Issue 4 | October-December 2020 181