See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/51046128

Meta-analysis in medical research

Article in Hippokratia · December 2010

Source: PubMed

CITATIONS READS

1,027 7,850

1 author:

Anna-Bettina Haidich
Aristotle University of Thessaloniki
268 PUBLICATIONS 8,910 CITATIONS

SEE PROFILE

All content following this page was uploaded by Anna-Bettina Haidich on 05 June 2014.

The user has requested enhancement of the downloaded file.

HIPPOKRATIA 2010, 14 (Suppl 1): 29-37 PASCHOS
HIPPOKRATIA 2010,KA
14 (Suppl 1) 29

REVIEW ARTICLE

Meta-analysis in medical research

Haidich AB
Department of Hygiene and Epidemiology, Aristotle University of Thessaloniki School of Medicine, Thessaloniki,
Greece

Abstract
The objectives of this paper are to provide an introduction to meta-analysis and to discuss the rationale for this type of
research and other general considerations. Methods used to produce a rigorous meta-analysis are highlighted and some
aspects of presentation and interpretation of meta-analysis are discussed.
Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess previous research
studies to derive conclusions about that body of research. Outcomes from a meta-analysis may include a more precise
estimate of the effect of treatment or risk factor for disease, or other outcomes, than any individual study contributing to
the pooled analysis. The examination of variability or heterogeneity in study results is also a critical outcome. The ben-
efits of meta-analysis include a consolidated and quantitative review of a large, and often complex, sometimes apparently
conflicting, body of literature. The specification of the outcome and hypotheses that are tested is critical to the conduct
of meta-analyses, as is a sensitive literature search. A failure to identify the majority of existing studies can lead to er-
roneous conclusions; however, there are methods of examining data to identify the potential for studies to be missing; for
example, by the use of funnel plots. Rigorously conducted meta-analyses are useful tools in evidence-based medicine.
The need to integrate findings from many studies ensures that meta-analytic research is desirable and the large body of
research now generated makes the conduct of this research feasible. Hippokratia 2010; 14 (Suppl 1): 29-37

Key words: meta-analysis, systematic review, randomized clinical trial, bias, quality, evidence-based medicine
Corresponding author: Anna-Bettina Haidich, Department of Hygiene and Epidemiology Aristotle University of Thessaloniki, School of
Medicine, 54124 Thessaloniki, Greece, Tel: +302310-999143, Fax: +302310-999701, e-mail:[email protected]

Important medical questions are typically studied munity5-8. The objectives of this paper are to introduce
more than once, often by different research teams in dif- meta-analysis and to discuss the rationale for this type of
ferent locations. In many instances, the results of these research and other general considerations.
multiple small studies of an issue are diverse and con-
flicting, which makes the clinical decision-making dif- Meta-Analysis and Systematic Review
ficult. The need to arrive at decisions affecting clinical Glass first defined meta-analysis in the social science
practise fostered the momentum toward “evidence-based literature as “The statistical analysis of a large collection
medicine”1-2. Evidence-based medicine may be defined of analysis results from individual studies for the purpose
as the systematic, quantitative, preferentially experimen- of integrating the findings”9. Meta-analysis is a quanti-
tal approach to obtaining and using medical information. tative, formal, epidemiological study design used to sys-
Therefore, meta-analysis, a statistical procedure that in-
tegrates the results of several independent studies, plays
a central role in evidence-based medicine. In fact, in the
hierarchy of evidence (Figure 1), where clinical evidence
is ranked according to the strength of the freedom from
various biases that beset medical research, meta-analy-
ses are in the top. In contrast, animal research, laboratory
studies, case series and case reports have little clinical
value as proof, hence being in the bottom.
Meta-analysis did not begin to appear regularly in the
medical literature until the late 1970s but since then a
plethora of meta-analyses have emerged and the growth
is exponential over time (Figure 2)3. Moreover, it has
been shown that meta-analyses are the most frequently
cited form of clinical research4. The merits and perils of
the somewhat mysterious procedure of meta-analysis,
however, continue to be debated in the medical com- Figure 1: Hierarchy of evidence.
30 HAIDICH AB

are used to minimize bias, thus providing more reliable

findings from which conclusions can be drawn and de-
cisions made than traditional review methods14,15. Sys-
tematic reviews need not contain a meta-analysis—there
are times when it is not appropriate or possible; however,
many systematic reviews contain meta-analyses16.
The inclusion of observational medical studies in
meta-analyses led to considerable debate over the valid-
ity of meta-analytical approaches, as there was necessar-
ily a concern that the observational studies were likely
to be subject to unidentified sources of confounding and
risk modification17. Pooling such findings may not lead
to more certain outcomes. Moreover, an empirical study
showed that in meta-analyses were both randomized and
non-randomized was included, nonrandomized studies
tended to show larger treatment effects18.
Meta-analyses are conducted to assess the strength of
Figure 2: Cumulative number of publications about meta- evidence present on a disease and treatment. One aim is
analysis over time, until 17 December 2009 (results from to determine whether an effect exists; another aim is to
Medline search using text “meta-analysis”). determine whether the effect is positive or negative and,
ideally, to obtain a single summary estimate of the effect.
tematically assess the results of previous research to de- The results of a meta-analysis can improve precision of
rive conclusions about that body of research. Typically, estimates of effect, answer questions not posed by the in-
but not necessarily, the study is based on randomized, dividual studies, settle controversies arising from appar-
controlled clinical trials. Outcomes from a meta-analysis ently conflicting studies, and generate new hypotheses. In
may include a more precise estimate of the effect of treat- particular, the examination of heterogeneity is vital to the
ment or risk factor for disease, or other outcomes, than development of new hypotheses.
any individual study contributing to the pooled analysis.
Identifying sources of variation in responses; that is, ex- Individual or Aggregated Data
amining heterogeneity of a group of studies, and general- The majority of meta-analyses are based on a series
izability of responses can lead to more effective treatments of studies to produce a point estimate of an effect and
or modifications of management. Examination of hetero- measures of the precision of that estimate. However,
geneity is perhaps the most important task in meta-analy- methods have been developed for the meta-analyses to be
sis. The Cochrane collaboration has been a long-standing, conducted on data obtained from original trials19,20. This
rigorous, and innovative leader in developing methods in approach may be considered the “gold standard” in meta-
the field10. Major contributions include the development analysis because it offers advantages over analyses using
of protocols that provide structure for literature search aggregated data, including a greater ability to validate the
methods, and new and extended analytic and diagnostic quality of data and to conduct appropriate statistical anal-
methods for evaluating the output of meta-analyses. Use ysis. Further, it is easier to explore differences in effect
of the methods outlined in the handbook should provide across subgroups within the study population than with
a consistent approach to the conduct of meta-analysis. aggregated data. The use of standardized individual-level
Moreover, a useful guide to improve reporting of system- information may help to avoid the problems encountered
atic reviews and meta-analyses is the PRISMA (Preferred in meta-analyses of prognostic factors21,22. It is the best
Reporting Items for Systematic reviews and Meta-analy- way to obtain a more global picture of the natural his-
ses) statement that replaced the QUOROM (QUality Of tory and predictors of risk for major outcomes, such as
Reporting of Meta-analyses) statement11-13. in scleroderma23-26.This approach relies on cooperation
Meta-analyses are a subset of systematic review. A between researchers who conducted the relevant studies.
systematic review attempts to collate empirical evidence Researchers who are aware of the potential to contribute
that fits prespecified eligibility criteria to answer a specif- or conduct these studies will provide and obtain addition-
ic research question. The key characteristics of a system- al benefits by careful maintenance of original databases
atic review are a clearly stated set of objectives with pre- and making these available for future studies.
defined eligibility criteria for studies; an explicit, repro-
ducible methodology; a systematic search that attempts Literature Search
to identify all studies that meet the eligibility criteria; an A sound meta-analysis is characterized by a thor-
assessment of the validity of the findings of the included ough and disciplined literature search. A clear definition
studies (e.g., through the assessment of risk of bias); and of hypotheses to be investigated provides the framework
a systematic presentation and synthesis of the attributes for such an investigation. According to the PRISMA
and findings from the studies used. Systematic methods statement, an explicit statement of questions being ad-
 HIPPOKRATIA 2010, 14 (Suppl 1) 31

dressed with reference to participants, interventions, com- studies had shown that certain methodological character-
parisons, outcomes and study design (PICOS) should be istics, such as poor concealment of treatment allocation
provided11,12. It is important to obtain all relevant studies, or no blinding in studies exaggerate treatment effects27.
because loss of studies can lead to bias in the study. Typi- Therefore, it is important to critically appraise the quality
cally, published papers and abstracts are identified by a of studies in order to assess the risk of bias.
computerized literature search of electronic databases The study design, including details of the method of
that can include PubMed (www.ncbi.nlm.nih.gov./en- randomization of subjects to treatment groups, criteria
trez/query.fcgi), ScienceDirect (www.sciencedirect.com), for eligibility in the study, blinding, method of assess-
Scirus (www.scirus.com/srsapp ), ISI Web of Knowledge ing the outcome, and handling of protocol deviations are
(http://www.isiwebofknowledge.com), Google Scholar important features defining study quality. When studies
(http://scholar.google.com) and CENTRAL (Cochrane are excluded from a meta-analysis, reasons for exclusion
Central Register of Controlled Trials, http://www.mrw. should be provided for each excluded study. Usually, more
interscience.wiley.com/cochrane/cochrane_clcentral_ar- than one assessor decides independently which studies to
ticles_fs.htm). PRISMA statement recommends that a full include or exclude, together with a well-defined checklist
electronic search strategy for at least one major database and a procedure that is followed when the assessors dis-
to be presented12. Database searches should be augmented agree. Two people familiar with the study topic perform
with hand searches of library resources for relevant pa- the quality assessment for each study, independently.
pers, books, abstracts, and conference proceedings. Cross- This is followed by a consensus meeting to discuss the
checking of references, citations in review papers, and studies excluded or included. Practically, the blinding of
communication with scientists who have been working in reviewers from details of a study such as authorship and
the relevant field are important methods used to provide a journal source is difficult.
comprehensive search. Communication with pharmaceu- Before assessing study quality, a quality assessment
tical companies manufacturing and distributing test prod- protocol and data forms should be developed. The goal
ucts can be appropriate for studies examining the use of of this process is to reduce the risk of bias in the estimate
pharmaceutical interventions. of effect. Quality scores that summarize multiple compo-
It is not feasible to find absolutely every relevant nents into a single number exist but are misleading and
study on a subject. Some or even many studies may not unhelpful28. Rather, investigators should use individual
be published, and those that are might not be indexed in components of quality assessment and describe trials that
computer-searchable databases. Useful sources for un- do not meet the specified quality standards and probably
published trials are the clinical trials registers, such as the assess the effect on the overall results by excluding them,
National Library of Medicine’s ClinicalTrials.gov Web- as part of the sensitivity analyses.
site. The reviews should attempt to be sensitive; that is, Further, not all studies are completed, because of pro-
find as many studies as possible, to minimize bias and be tocol failure, treatment failure, or other factors. Nonethe-
efficient. It may be appropriate to frame a hypothesis that less, missing subjects and studies can provide important
considers the time over which a study is conducted or to evidence. It is desirable to obtain data from all relevant
target a particular subpopulation. The decision whether randomized trials, so that the most appropriate analysis
to include unpublished studies is difficult. Although lan- can be undertaken. Previous studies have discussed the
guage of publication can provide a difficulty, it is impor- significance of missing trials to the interpretation of in-
tant to overcome this difficulty, provided that the popula- tervention studies in medicine29,30. Journal editors and
tions studied are relevant to the hypothesis being tested. reviewers need to be aware of the existing bias toward
publishing positive findings and ensure that papers that
Inclusion or Exclusion Criteria and Potential for Bias publish negative or even failed trials be published, as
Studies are chosen for meta-analysis based on inclu- long as these meet the quality guidelines for publication.
sion criteria. If there is more than one hypothesis to be There are occasions when authors of the selected pa-
tested, separate selection criteria should be defined for pers have chosen different outcome criteria for their main
each hypothesis. Inclusion criteria are ideally defined at analysis. In practice, it may be necessary to revise the
the stage of initial development of the study protocol. The inclusion criteria for a meta-analysis after reviewing all
rationale for the criteria for study selection used should of the studies found through the search strategy. Varia-
be clearly stated. tion in studies reflects the type of study design used, type
One important potential source of bias in meta-analy- and application of experimental and control therapies,
sis is the loss of trials and subjects. Ideally, all random- whether or not the study was published, and, if published,
ized subjects in all studies satisfy all of the trial selection subjected to peer review, and the definition used for the
criteria, comply with all the trial procedures, and provide outcome of interest. There are no standardized criteria for
complete data. Under these conditions, an “intention-to- inclusion of studies in meta-analysis. Universal criteria
treat” analysis is straightforward to implement; that is, are not appropriate, however, because meta-analysis can
statistical analysis is conducted on all subjects that are be applied to a broad spectrum of topics. Published data
enrolled in a study rather than those that complete all in journal papers should also be cross-checked with con-
stages of study considered desirable. Some empirical ference papers to avoid repetition in presented data.
32 HAIDICH AB

Clearly, unpublished studies are not found by search- servative approach with wider confidence intervals than
ing the literature. It is possible that published studies are the fixed-effects model where the studies are weighted
systemically different from unpublished studies; for ex- only with the inverse of their variance. The most com-
ample, positive trial findings may be more likely to be monly used random-effects method is the DerSimonian
published. Therefore, a meta-analysis based on literature and Laird method35. Furthermore, it is suggested that
search results alone may lead to publication bias. comparing the fixed-effects and random-effect models
Efforts to minimize this potential bias include work- developed as this process can yield insights to the data36.
ing from the references in published studies, searching
computerized databases of unpublished material, and in- Heterogeneity
vestigating other sources of information including con- Arguably, the greatest benefit of conducting meta-
ference proceedings, graduate dissertations and clinical analysis is to examine sources of heterogeneity, if pres-
trial registers. ent, among studies. If heterogeneity is present, the sum-
mary measure must be interpreted with caution 37. When
Statistical analysis heterogeneity is present, one should question whether
The most common measures of effect used for dichot- and how to generalize the results. Understanding sources
omous data are the risk ratio (also called relative risk) and of heterogeneity will lead to more effective targeting of
the odds ratio. The dominant method used for continuous prevention and treatment strategies and will result in new
data are standardized mean difference (SMD) estimation. research topics being identified. Part of the strategy in
Methods used in meta-analysis for post hoc analysis of conducting a meta-analysis is to identify factors that may
findings are relatively specific to meta-analysis and in- be significant determinants of subpopulation analysis or
clude heterogeneity analysis, sensitivity analysis, and covariates that may be appropriate to explore in all stud-
evaluation of publication bias. ies.
All methods used should allow for the weighting of To understand the nature of variability in studies, it is
studies. The concept of weighting reflects the value of important to distinguish between different sources of het-
the evidence of any particular study. Usually, studies are erogeneity. Variability in the participants, interventions,
weighted according to the inverse of their variance31. It and outcomes studied has been described as clinical di-
is important to recognize that smaller studies, therefore, versity, and variability in study design and risk of bias has
usually contribute less to the estimates of overall effect. been described as methodological diversity10. Variability
However, well-conducted studies with tight control of in the intervention effects being evaluated among the dif-
measurement variation and sources of confounding con- ferent studies is known as statistical heterogeneity and is
tribute more to estimates of overall effect than a study of a consequence of clinical or methodological diversity, or
identical size less well conducted. both, among the studies. Statistical heterogeneity mani-
One of the foremost decisions to be made when fests itself in the observed intervention effects varying
conducting a meta-analysis is whether to use a fixed-ef- by more than the differences expected among studies that
fects or a random-effects model. A fixed-effects model is would be attributable to random error alone. Usually, in
based on the assumption that the sole source of variation the literature, statistical heterogeneity is simply referred
in observed outcomes is that occurring within the study; to as heterogeneity.
that is, the effect expected from each study is the same. Clinical variation will cause heterogeneity if the inter-
Consequently, it is assumed that the models are homoge- vention effect is modified by the factors that vary across
neous; there are no differences in the underlying study studies; most obviously, the specific interventions or par-
population, no differences in subject selection criteria, ticipant characteristics that are often reflected in different
and treatments are applied the same way32. Fixed-effect levels of risk in the control group when the outcome is
methods used for dichotomous data include most often dichotomous. In other words, the true intervention effect
the Mantel-Haenzel method33 and the Peto method 34(only will differ for different studies. Differences between stud-
for odds ratios). ies in terms of methods used, such as use of blinding or
Random-effects models have an underlying assump- differences between studies in the definition or measure-
tion that a distribution of effects exists, resulting in het- ment of outcomes, may lead to differences in observed
erogeneity among study results, known as τ2. Conse- effects. Significant statistical heterogeneity arising from
quently, as software has improved, random-effects mod- differences in methods used or differences in outcome as-
els that require greater computing power have become sessments suggests that the studies are not all estimating
more frequently conducted. This is desirable because the the same effect, but does not necessarily suggest that the
strong assumption that the effect of interest is the same true intervention effect varies. In particular, heterogene-
in all studies is frequently untenable. Moreover, the fixed ity associated solely with methodological diversity indi-
effects model is not appropriate when statistical het- cates that studies suffer from different degrees of bias.
erogeneity (τ2) is present in the results of studies in the Empirical evidence suggests that some aspects of design
meta-analysis. In the random-effects model, studies are can affect the result of clinical trials, although this may
weighted with the inverse of their variance and the het- not always be the case.
erogeneity parameter. Therefore, it is usually a more con- The scope of a meta-analysis will largely determine
 HIPPOKRATIA 2010, 14 (Suppl 1) 33

the extent to which studies included in a review are di- sion, studies are regarded as if they were individual pa-
verse. Meta-analysis should be conducted when a group tients, but their effects are properly weighted to account
of studies is sufficiently homogeneous in terms of sub- for their different variances44.
jects involved, interventions, and outcomes to provide a Sensitivity analyses have also been used to examine
meaningful summary. However, it is often appropriate to the effects of studies identified as being aberrant concern-
take a broader perspective in a meta-analysis than in a ing conduct or result, or being highly influential in the
single clinical trial. Combining studies that differ sub- analysis. Recently, another method has been proposed
stantially in design and other factors can yield a mean- that reduces the weight of studies that are outliers in
ingless summary result, but the evaluation of reasons for meta-analyses45. All of these methods for examining het-
the heterogeneity among studies can be very insightful. It erogeneity have merit, and the variety of methods avail-
may be argued that these studies are of intrinsic interest able reflects the importance of this activity.
on their own, even though it is not appropriate to produce
a single summary estimate of effect. Presentation of results
Variation among k trials is usually assessed using A useful graph, presented in the PRISMA statement11,
Cochran’s Q statistic, a chi-squared (χ2) test of heteroge- is the four-phase flow diagram (Figure 3).
neity with k-1 degrees of freedom. This test has relatively
poor power to detect heterogeneity among small numbers
of trials; consequently, an α-level of 0.10 is used to test
hypotheses38,39.
Heterogeneity of results among trials is better quanti-
fied using the inconsistency index I 2, which describes the
percentage of total variation across studies40. Uncertainty
intervals for I 2 (dependent on Q and k) are calculated us-
ing the method described by Higgins and Thompson41.
Negative values of I 2 are put equal to zero, consequently
I 2 lies between 0 and 100%. A value >75% may be con-
sidered substantial heterogeneity41. This statistic is less
influenced by the number of trials compared with other
methods used to estimate the heterogeneity and provides
a logical and readily interpretable metric but it still can be
unstable when only a few studies are combined42.
Given that there are several potential sources of het-
erogeneity in the data, several steps should be considered
in the investigation of the causes. Although random-ef-
fects models are appropriate, it may be still very desirable
to examine the data to identify sources of heterogeneity Υear of publication
and to take steps to produce models that have a lower lev-
el of heterogeneity, if appropriate. Further, if the studies Figure 3: PRISMA 2009 Flow Diagram (From Moher D,
examined are highly heterogeneous, it may be not appro- Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred
priate to present an overall summary estimate, even when reporting items for systematic reviews and meta-analyses:
random effects models are used. As Petiti notes43, statis- the PRISMA statement. J Clin Epidemiol 2009;62:1006-12,
tical analysis alone will not make contradictory studies For more information, visit www.prisma-statement.org).
agree; critically, however, one should use common sense
in decision-making. Despite heterogeneity in responses, This flow-diagram depicts the flow of information
if all studies had a positive point direction and the pooled through the different phases of a systematic review or
confidence interval did not include zero, it would not be meta-analysis. It maps out the number of records identi-
logical to conclude that there was not a positive effect, fied, included and excluded, and the reasons for exclu-
provided that sufficient studies and subject numbers were sions. The results of meta-analyses are often presented
present. The appropriateness of the point estimate of the in a forest plot, where each study is shown with its ef-
effect is much more in question. fect size and the corresponding 95% confidence interval
Some of the ways to investigate the reasons for het- (Figure 4).
erogeneity; are subgroup analysis and meta-regression. The pooled effect and 95% confidence interval is
The subgroup analysis approach, a variation on those shown in the bottom in the same line with “Overall”. In
described above, groups categories of subjects (e.g., by the right panel of Figure 4, the cumulative meta-analysis
age, sex) to compare effect sizes. The meta-regression is graphically displayed, where data are entered succes-
approach uses regression analysis to determine the influ- sively, typically in the order of their chronological ap-
ence of selected variables (the independent variables) on pearance46,47. Such cumulative meta-analysis can retro-
the effect size (the dependent variable). In a meta-regres- spectively identify the point in time when a treatment
34 HAIDICH AB

are more likely to be published and,

in the case of interventions with a
commercial value, to be promoted,
than studies with non-significant or
“negative” results (negative stud-
ies). Studies that produce a positive
result, especially large studies, are
more likely to have been published
and, conversely, there has been a re-
luctance to publish small studies that
have non-significant results. Further,
publication bias is not solely the
responsibility of editorial policy as
there is reluctance among research-
ers to publish results that were either
uninteresting or are not random-
ized48. There are, however, problems
with simply including all studies
that have failed to meet peer-review
standards. All methods of retrospec-
tively dealing with bias in studies
are imperfect.
It is important to examine the re-
sults of each meta-analysis for evi-
dence of publication bias. An estima-
tion of likely size of the publication
Figure 4: Forest plots of the meta-analysis addressing the use of antibiotic prophy- bias in the review and an approach
laxis compared with no treatment in colon surgery. The outcome is would infection to dealing with the bias is inherent
and 32 studies were included in the meta-analysis. Risk ratio <1 favors use of pro- to the conduct of many meta-analy-
phylactic antibiotics, whereas risk ratio > 1 suggests that no treatment is better. Left
ses. Several methods have been de-
panel, Studies displayed chronologically by year of publication, n represents study
veloped to provide an assessment of
size. Each study is represented by a filled circle (denoting its risk ratio estimate) and
publication bias; the most common-
the horizontal line denotes the corresponding 95% confidence interval. Studies that
ly used is the funnel plot. The funnel
intersect the vertical line of unity (RR=1), indicate no difference between the antibi-
plot provides a graphical evaluation
otic group and the control group. Pooled results from all studies are shown at bottom
with the random-effect model. Right panel,Cumulative meta-analysis of same stud- of the potential for bias and was de-
ies with random-effects model, where the summary risk ratio is re-estimated each veloped by Light and Pillemer49 and
time a study is added over time. It reveals that antibiotic prophylaxis efficacy could discussed in detail by Egger and col-
have been identified as early as 1971 after 5 studies involving about 300 patients (n leagues50,51. A funnel plot is a scat-
in this panel represents cumulative number of patients from included studies). (From terplot of treatment effect against a
Ioannidis JP, Lau J. State of the evidence: current status and prospects of meta- measure of study size. If publication
analysis in infectious diseases. Clin Infect Dis 1999;29:1178–85). bias is not present, the plot is ex-
pected to have a symmetric inverted
effect first reached conventional levels of significance. funnel shape, as shown in Figure 5A.
Cumulative meta-analysis is a compelling way to exam- In a study in which there is no publication bias, larger
ine trends in the evolution of the summary-effect size, studies (i.e., have lower standard error) tend to cluster
and to assess the impact of a specific study on the overall closely to the point estimate. As studies become less pre-
conclusions46. The figure shows that many studies were cise, such as in smaller trials (i.e., have a higher standard
performed long after cumulative meta-analysis would error), the results of the studies can be expected to be
have shown a significant beneficial effect of antibiotic more variable and are scattered to both sides of the more
prophylaxis in colon surgery. precise larger studies. Figure 5A shows that the smaller,
less precise studies are, indeed, scattered to both sides of
Biases in meta-analysis the point estimate of effect and that these seem to be sym-
Although the intent of a meta-analysis is to find and metrical, as an inverted funnel-plot, showing no evidence
assess all studies meeting the inclusion criteria, it is not of publication bias. In contrast to Figure 5A, Figure 5B
always possible to obtain these. A critical concern is the shows evidence of publication bias. There is evidence of
papers that may have been missed. There is good reason the possibility that studies using smaller numbers of sub-
to be concerned about this potential loss because stud- jects and showing an decrease in effect size (lower odds
ies with significant, positive results (positive studies) ratio) were not published.
 HIPPOKRATIA 2010, 14 (Suppl 1) 35

crual of patients are published sooner than negative trials

with low early accrual56. However, missing studies, either
due to publication bias or time-lag bias may increasingly
be identified from trials registries.
The selective reporting bias exists when published
articles have incomplete or inadequate reporting. Empiri-
cal studies have shown that this bias is widespread and
of considerable importance when published studies were
compared with their study protocols29,30. Furthermore, re-
cent evidence suggests that selective reporting might be
an issue in safety outcomes and the reporting of harms
in clinical trials is still suboptimal57. Therefore, it might
not be possible to use quantitative objective evidence for
harms in performing meta-analyses and making thera-
peutic decisions.
Excluding clinical trials reported in languages other
than English from meta-analyses may introduce the lan-
guage bias and reduce the precision of combined estimates
of treatment effects. Trials with statistically significant
results have been shown to be published in English58. In
contrast, a later more extensive investigation showed that
trials published in languages other than English tend to be
of lower quality and produce more favourable treatment
effects than trials published in English and concluded that
excluding non-English language trials has generally only
modest effects on summary treatment effect estimates but
the effect is difficult to predict for individual meta-analy-
ses59.

Evolution of meta-analyses
Figure 5: A) Symmetrical funnel plot. B) Asymmetrical fun- The classical meta-analysis compares two treatments
nel plot, the small negative studies in the bottom left corner while network meta-analysis (or multiple treatment meta-
is missing. analysis) can provide estimates of treatment efficacy of
multiple treatment regimens, even when direct compari-
Asymmetry of funnel plots is not solely attributable sons are unavailable by indirect comparisons60. An ex-
to publication bias, but may also result from clinical het- ample of a network analysis would be the following. An
erogeneity among studies. Sources of clinical heterogene- initial trial compares drug A to drug B. A different trial
ity include differences in control or exposure of subjects studying the same patient population compares drug B
to confounders or effect modifiers, or methodological to drug C. Assume that drug A is found to be superior
heterogeneity between studies; for example, a failure to to drug B in the first trial. Assume drug B is found to be
conceal treatment allocation. There are several statistical equivalent to drug C in a second trial. Network analysis
tests for detecting funnel plot asymmetry; for example, then, allows one to potentially say statistically that drug A
Egger’s linear regression test50, and Begg’s rank correla- is also superior to drug C for this particular patient popu-
tion test52 but these do not have considerable power and lation. (Since drug A is better than drug B, and drug B is
are rarely used. However, the funnel plot is not without equivalent to drug C, then drug A is also better to drug C
problems. If high precision studies really are different even though it was not directly tested against drug C.)
than low precision studies with respect to effect size (e.g., Meta-analysis can also be used to summarize the per-
due different populations examined) a funnel plot may formance of diagnostic and prognostic tests. However,
give a wrong impression of publication bias53. The ap- studies that evaluate the accuracy of tests have a unique
pearance of the funnel plot plot can change quite dramati- design requiring different criteria to appropriately assess
cally depending on the scale on the y-axis - whether it is the quality of studies and the potential for bias. Addition-
the inverse square error or the trial size54. ally, each study reports a pair of related summary statis-
Other types of biases in meta-analysis include the tics (for example, sensitivity and specificity) rather than
time lag bias, selective reporting bias and the language a single statistic (such as a risk ratio) and hence requires
bias. The time lag bias arises from the published stud- different statistical methods to pool the results of the stud-
ies, when those with striking results are published earlier ies61. Various techniques to summarize results from diag-
than those with non-significant findings55. Moreover, it nostic and prognostic test results have been proposed62-64.
has been shown that positive studies with high early ac- Furthermore, there are many methodologies for advanced
36 HAIDICH AB

meta-analysis that have been developed to address specif- impact of various study designs in the health sciences. JAMA.
ic concerns, such as multivariate meta-analysis65–67, and 2005; 293: 2362-2366.
5. Hennekens CH, DeMets D, Bairey-Merz CN, Borzak S, Borer
special types of meta-analysis in genetics68 but will not
J. Doing more good than harm: the need for a cease fire. Am J
be discussed here. Med. 2009; 122: 315-316.
Meta-analysis is no longer a novelty in medicine. Nu- 6. Naylor CD. Meta-analysis and the meta-epidemiology of clini-
merous meta-analyses have been conducted for the same cal research. BMJ. 1997; 315: 617-619.
medical topic by different researchers. Recently, there is 7. Bailar JC. The promise and problems of meta-analysis [edito-
rial]. N Engl J Med. 1997; 337: 559-561.
a trend to combine the results of different meta-analyses,
8. Meta-analysis under scrutiny [editorial]. Lancet 1997; 350:
known as a meta-epidemiological study, to assess the risk 675.
of bias79,70. 9. Glass GV. Primary, secondary and meta-analysis of research.
Educational Researcher. 1976; 5: 3–8.
Conclusions 10. Higgins JPT, Green S (editors). Cochrane Handbook for System-
The traditional basis of medical practice has been atic Reviews of Interventions Version 5.0.2 [updated September
2009]. The Cochrane Collaboration, 2009. Available from www.
changed by the use of randomized, blinded, multicenter cochrane-handbook.org.
clinical trials and meta-analysis, leading to the widely used 11. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group.
term “evidence-based medicine”. Leaders in initiating this Preferred reporting items for systematic reviews and meta-anal-
change have been the Cochrane Collaboration who have yses: the PRISMA statement. J Clin Epidemiol 2009;62:1006-
produced guidelines for conducting systematic reviews 1012.
12. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Götzsche PC, Io-
and meta-analyses10 and recently the PRISMA statement,
annidis JP, et al. The PRISMA statement for reporting systematic
a helpful resource to improve reporting of systematic re- reviews and meta-analyses of studies that evaluate health care
views and meta-analyses has been released11. Moreover, interventions: explanation and elaboration. J Clin Epidemiol.
standards by which to conduct and report meta-analyses 2009;62:e1-34.
of observational studies have been published to improve 13. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF.
Improving the quality of reports of metaanalyses of randomised
the quality of reporting71.
controlled trials: the QUOROM statement. Quality of Reporting
Meta-analysis of randomized clinical trials is not an of Meta-analyses. Lancet 1999; 354: 1896–1900.
infallible tool, however, and several examples exist of 14. Antman, EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A
meta-analyses which were later contradicted by single comparison of results of meta-analyses of randomized control
large randomized controlled trials, and of meta-analyses trials and recommendations of clinical experts: Treatments for
addressing the same issue which have reached opposite myocardial infarction. JAMA. 1992; 268: 240–248.
15. Oxman, AD, Guyatt GH. The science of reviewing research.
conclusions72. A recent example, was the controversy be- Ann NY Acad Sci. 1993; 703: 125–133.
tween a meta-analysis of 42 studies73 and the subsequent 16. Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in sys-
publication of the large-scale trial (RECORD trial) that tematic reviews. Ann Intern Med. 1997; 127: 820–826.
did not support the cardiovascular risk of rosiglitazone74. 17. Greenland S. Invited commentary: A critical look at some popu-
However, the reason for this controversy was explained lar meta-analytic methods. Am J Epidemiol. 1994; 140: 290–
296.
by the numerous methodological flaws found both in the
18. Ioannidis JPA, Haidich A-B, Pappa M, Pantazis N, Kokori SI,
meta-analysis and the large clinical trial75. Tektonidou MG, et al. Comparison of evidence of treatment ef-
No single study, whether meta-analytic or not, will fects in randomized and non-randomized studies. JAMA. 2001;
provide the definitive understanding of responses to treat- 286: 821-830.
ment, diagnostic tests, or risk factors influencing disease. 19. Simmonds MC, Higgins JPT, Stewart LA, Tierney JA, Clarke
MJ, Thompson SG. Meta-analysis of individual patient data
Despite this limitation, meta-analytic approaches have
from randomized trials: A review of methods used in practice.
demonstrable benefits in addressing the limitations of Clin Trials. 2005; 2: 209–217.
study size, can include diverse populations, provide the 20. Stewart LA, Clarke MJ. Practical methodology of meta-analy-
opportunity to evaluate new hypotheses, and are more ses (overviews) using updated individual patient data. Cochrane
valuable than any single study contributing to the analy- working group. Stat Med. 1995; 14: 2057–2079.
sis. The conduct of the studies is critical to the value of a 21. Altman DG. Systematic reviews of evaluations of prognostic
variables. BMJ. 2001; 323: 224 –228.
meta-analysis and the methods used need to be as rigor- 22. Simon R, Altman DG: Statistical aspects of prognostic factor
ous as any other study conducted. studies in oncology. Br J Cancer.1994; 69: 979 –985.
23. Ioannidis JPA, Vlachoyiannopoulos PG, Haidich AB, Medsger
References TA Jr., Lucas M, Michet CJ, et al. Mortality in systemic sclero-
1. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, sis: an international meta-analysis of individual patient data. Am
Cook RJ. User’s guide to the medical literature. IX. A method J Med. 2005; 118: 2-10.
for grading health care recommendations. Evidence-Based Med- 24. Oxman AD, Clarke MJ, Stewart LA. From science to practice.
icine Working Group. JAMA. 1995; 274: 1800–1804. Meta-analyses using individual patient data are needed. JAMA.
2. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson 1995; 274: 845- 846.
WS. Evidence based medicine: what it is and what it isn’t. BMJ. 25. Stewart LA, Tierney JF: To IPD or not to IPD? Advantages and
1996: 312: 71–72. disadvantages of systematic reviews using individual patient
3. Chalmers TC, Matta RJ, Smith H Jr, Kunzler AM. Evidence fa- data. Eval Health Prof. 2002; 25: 76 –97.
voring the use of anticoagulants in the hospital phase of acute 26. Trikalinos TA, Ioannidis JPA: Predictive modeling and hetero-
myocardial infarction. N Engl J Med. 1977; 297: 1091–1096. geneity of baseline risk in meta-analysis of individual patient
4. Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation data. J Clin Epidemiol. 2001; 54: 245–252.
 HIPPOKRATIA 2010, 14 (Suppl 1) 37

27. Pildal J, Hróbjartsson A, Jorgensen KJ, Hilden J, Altman DG, 54. Tang JL, Liu JL. Misleading funnel plot for detection of bias in
Götzsche PC. Impact of allocation concealment on conclusions meta-analysis. J Clin Epidemiol. 2000; 53: 477–484.
drawn from meta-analyses of randomized trials. Int J Epidemiol. 55. Ioannidis JP. Effect of the statistical significance of results on the
2007; 36: 847-857. time to completion and publication of randomized efficacy trials.
28. Juni P, Altman DG, Egger M. Systematic reviews in health care: JAMA. 1998; 279: 281-286.
Assessing the quality of controlled clinical trials. BMJ. 2001; 56. Haidich AB, Ioannidis JP. Effect of early patient enrollment on
323: 42–46. the time to completion and publication of randomized controlled
29. Chan AW, Hróbjartsson A , Haahr MT, Götzsche PC, Altman trials. Am J Epidemiol. 2001; 154: 873–880.
DG. Empirical evidence for selective reporting of outcomes in 57. Haidich AB, Charis B, Dardavessis T, Tirodimos I, Arvanitidou
randomized trials: comparison of protocols to published articles. M. The quality of safety reporting in trials is still suboptimal:
JAMA. 2004; 291: 2457–2465. Survey of major general medical journals. J Clin Epidemiol.
30. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A, Cronin E, et 2010; (in press).
al. Systematic review of the empirical evidence of study publica- 58. Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler
tion bias and outcome reporting bias. PLoS One. 2008; 3:e3081. C, Antes G. Language bias in randomised controlled trials pub-
31. Egger M, Smith GD, Phillips AN. Meta-analysis: principles and lished in English and German. Lancet. 1997; 350: 326–329.
procedures. BMJ. 1997; 315: 1533-1537. 59. Jόni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction
32. Stangl DK, Berry DA. Meta-analysis in medicine and health and impact of language bias in meta-analyses of controlled tri-
policy. Marcel Dekker, New York, NY. 2000. als: empirical study. Int J Epidemiol. 2002; 31:115–123.
33. Mantel N, Haenszel W. Statistical aspects of the analysis of data 60. Lumley T. Network meta-analysis for indirect treatment com-
from retrospective studies of disease. J Natl Cancer Inst. 1959; parisons. Stat Med. 2002; 21: 2313–2324.
22: 719–748. 61. Deeks JJ. Systematic reviews in health care: Systematic reviews
34. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade of evaluations of diagnostic and screening tests. BMJ. 2001;
during and after myocardial infarction: an overview of the ran- 323: 157–162.
domized trials. Prog Cardiovasc Dis. 1985; 27: 335–371. 62. Littenberg B, Moses LE. Estimating diagnostic accuracy from
35. DerSimonian R, Laird N. Meta-analysis in clinical trials. Con- multiple conflicting reports: a new meta-analytic method. Med
trol Clin Trials. 1986; 7: 177–188. Decis Making. 1993; 13: 313–321.
36. Whitehead A. Meta-analysis of controlled clinical trials. 63. Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM,
JohnWiley & Sons Ltd. Chichester, UK. 2002. Zwinderman AH. Bivariate analysis of sensitivity and specificity
37. Greenland S. Quantitative methods in the review of epidemio- produces informative summary measures in diagnostic reviews.
logic literature. Epidemiol Rev. 1987; 9: 1–30. J Clin Epidemiol. 2005; 58: 982-990.
38. Cochran W. The combination of estimates from different experi- 64. Rutter CM, Gatsonis CA. A hierarchical regression approach to
ments. Biometrics. 1954; 10: 101–129. meta-analysis of diagnostic test accuracy evaluations. Stat Med.
39. Hardy RJ, Thompson SG. Detecting and describing heterogene- 2001; 20: 2865–2884.
ity in meta-analysis. Stat Med.1998; 17: 841–856. 65. Arends LR. Multivariate meta-analysis: modelling the heteroge-
40. Higgins JP, Thompsom SG, Deeks JJ, Altman DG. Measuring neity. Rotterdam: Erasmus University; 2006.
inconsistency in meta-analysis. BMJ. 2003; 327: 557–560. 66. Riley RD, Abrams KR, Sutton AJ, Lambert PC, Thompson JR.
41. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta- Bivariate random-effects meta-analysis and the estimation of be-
analysis. Stat Med. 2002; 21:1539–1558. tween-study correlation. BMC Med Res Methodol. 2007; 7: 3.
42. Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F, Botella 67. Trikalinos TA, Olkin I. A method for the meta-analysis of mutu-
J. Assessing heterogeneity in meta-analysis: Q statistic or I2 in- ally exclusive binary outcomes. Stat Med. 2008; 27: 4279–4300.
dex? Psychol Methods. 2006; 11: 193–206. 68. Trikalinos TA, Salanti G, Zintzaras E, Ioannidis JP. Meta-analy-
43. Petiti DB. Meta-analysis, decision analysis and cost-effective- sis methods. Adv Genet. 2008; 60: 311–334.
ness analysis. Oxford University Press, New York, NY. 1994. 69. Nόesch E, Trelle S, Reichenbach S, Rutjes AW, Bόrgi E, Scherer
44. Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in sys- M, et al. The effects of excluding patients from the analysis in
tematic reviews. Ann Intern Med. 1997; 127: 820–826. randomized controlled trials: meta-epidemiological study. BMJ.
45. Baker R, Jackson D. A new approach to outliers in meta-analy- 2009; 339: b3244.
sis. Health Care Manage Sci. 2008; 11:121–131. 70. Wood L, Egger M, Gluud LL, Schulz KF, Jόni P, Altman DG,
46. Lau J, Schmid CH, Chalmers TC. Cumulative meta-analysis of et al. Empirical evidence of bias in treatment effect estimates
clinical trials builds evidence for exemplary medical care. J Clin in controlled trials with different interventions and outcomes:
Epidemiol. 1995; 48: 45–57. meta-epidemiological study. BMJ. 2008; 336: 601-605.
47. Ioannidis JP, Lau J. State of the evidence: current status and 71. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD,
prospects of meta-analysis in infectious diseases. Clin Infect Rennie D, et al. Meta-analysis of observational studies in epi-
Dis. 1999; 29: 1178–1185. demiology: a proposal for reporting. Meta-analysis of Observa-
48. Dickersin K, Berlin JA. Meta-analysis: State of the science. Epi- tional Studies in Epidemiology (MOOSE) group. JAMA. 2000;
demiol Rev. 1992; 14: 154–176. 283:2008–2012.
49. Light J, Pillemer DB. Summing up: The Science of Review- 72. LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Derderian F.
ing Research. Harvard University Press, Cambridge, Massachu- Discrepancies between meta-analyses and subsequent large ran-
setts,1984. domized, controlled trials. N Engl J Med. 1997; 337: 536–542.
50. Egger M, Smith GD, Schneider M, Minder C. Bias in meta- 73. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myo-
analysis detected by a simple, graphical test. BMJ. 1997; 315: cardial infarction and death from cardiovascular causes. N Engl
629–624. J Med. 2007; 356: 2457–2471.
51. Sterne JA, Egger M. Funnel plots for detecting bias in meta- 74. Home PD, Pocock SJ, Beck-Nielsen H, Curtis PS, Gomis R,
analysis: guidelines on choice of axis. J Clin Epidemiol. 2001; Hanefeld M, et al; RECORD study team. Rosiglitazone Evalu-
54: 1046–1045. ated for Cardiovascular Outcomes in Oral Agent Combination
52. Begg CB, Mazumdar M. Operating characteristics of a rank cor- Therapy for Type 2 Diabetes (RECORD). Lancet. 2009; 373:
relation test for publication bias. Biometrics. 1994; 50: 1088- 2125-2135.
1101. 75. Hlatky MA, Bravata DM. Review: rosiglitazone increases risk
53. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of of MI but does not differ from other drugs for CV death in type
the misleading funnel plot. BMJ. 2006; 333: 597–590. 2 diabetes. Evid Based Med. 2007; 12:169-170.

View publication stats