‘Real World’ Evidence: A Dangerous Delusion

Steven E. Nissen MD MACC

Chief Academic Officer

Heart Vascular and Thoracic Institute
Lewis
Lewis and
and Patricia
Patricia Dickey
Dickey Chair
Chair in
in Cardiovascular
Cardiovascular Medicine
Medicine
Professor
Professor of
of Medicine
Medicine
Cleveland
Cleveland Clinic
Clinic Lerner
Lerner College
College of
of Medicine
Medicine
 Terminology

The term “Real World Evidence” is an oxymoron

Real world studies should not

be viewed as conclusive “evidence”.

A better term is “real world data”

Randomization Provides Evidence about Treatment Effects That
Can Be Trusted
Randomization results in groups of patients that are balanced (give or
take the play of chance) with respect to their risks of all types of health
outcomes. Consequently, in sufficiently large randomized trials, the
effects of a treatment can be reliably assessed.
Nonrandomized observational studies may be able to detect large
treatment effects. However, the potential biases can be appreciable, so
such studies cannot be trusted when the benefits or harms of a
treatment are actually null or only moderate.
Big data is often bad data
In recent years, an increasing number of observational studies have  been
published that have made claims about the effectiveness of alternative
therapies. With the development of electronic medical records and
governmental databases, some observational studies have become very large,
often reporting on hundreds of thousands or even millions of patients. A recent
examination of the effects of body weight on mortality included 3.6 million
residents of the United . Kingdom (~9% of the population), an approach
described as ‘big data’. Such studies are eye-catching, often garnering huge
press attention. Some journalists and the public are seduced by the large size
of the trials, assuming that the massive sizes of the studies will automatically
ensure that the findings are accurate.
 Definition of a Clinical Trial
A Clinical Trial is a Controlled “Experiment”
• Experiment = series of observations made under conditions
controlled by the scientist
• Resolve uncertainty by isolating control variable
(i.e. treatment) and outcome from extraneous influences
• The Trial Design isolates the factor of interest by controlling:
 variability
 bias
 application of treatment
 ascertainment of outcome
 analysis
Randomized Clinical Trials: The Gold Standard
• Strengths
– Highest quality of evidence
– Can prove causation
– Controlled, precise administration of an intervention
– Randomization minimizes selection bias and confounding
– Allows for blinding of patients and physicians
– Uniformly accepted by regulatory authorities and payers

• Weaknesses
– Logistically Challenging
– Expensive
– Time consuming
Because of the high cost and time consuming
nature of randomized clinical trials,
investigators have sought to conduct simpler
and less expensive studies.

In many cases, the results have been

disastrous
 Observational Studies
Hormone Replacement Therapy (HRT)

• Hypothesis that HRT reduced coronary heart disease

• Supportive data:
- Marked increase in CHD risk with menopause
- Estrogen raises HDL (“good”) and lowers LDL (“bad”)
cholesterol levels
- Non-human primate studies
• Over 30 observational studies suggested 40-50%
reduction in CHD risk!
Classical “Real World” Hormone Replacement Study
Nurse’s Health Study
• 59,337 female nurses, 30-55 yrs old at baseline
• Voluntary biennial questionnaire
• 16 years follow-up 1976-1992, >400,000 patient-years

Major Coronary Disease

Hormone Use Person Years
Relative Risk (95% CI)
Multivariate
Age Adjusted
Adjusted

Never Used 304,744 1.0 1.0

Estrogen Alone 82,626 0.45 (0.34-0.60) 0.60 (0.43-0.83)

Estrogen + Progestin 27,161 0.22 (0.12-0.41) 0.39 (0.19-0.78)

Grodstein et al. NEJM 1996;335:453.

Results of ‘Real World ‘Evidence’: Use of HRT
• One-third of postmenopausal women used HRT
• Second most prescribed drug, $1 billion in sales
• In year 2000, 46 million prescriptions for estrogen,
22 million prescriptions for estrogen + progestin

• Notion that HRT was beneficial for CHD thought so strong that
best rationale for RCT was to “prove” benefit to remaining
doctors who did not prescribe!

• Many authorities advocated against an RCT because they

argued it was unethical since therapy was “obviously”
beneficial
Clinical Trial of HRT for Secondary Prevention
Heart and Estrogen-Progestin Replacement Study (HERS)

• Postmenopausal women
• Patients had documented cardiovascular disease
• Randomized to estrogen-progestin vs placebo
• Double-blinded
• Followed for average of 4.1 years
• Total 2763 patients enrolled
Hulley et al. JAMA 1998;280:605.
Secondary Prevention Randomized Trial of HRT
Primary Results of the HERS Trial

All CHD Events All Cause Mortality

Estrogen
+ Progestin
Estrogen
+ Progestin

Placebo

Placebo

Hulley et al. JAMA 1998;280:605.

 HERS: Secondary Prevention Trial of HRT
• Thromboembolic events more frequent with HRT
(6.3 vs 2.2 per 1000 women-years, p = 0.002)

• Gallbladder disease more frequent with HRT

(84 vs 62 women, p = 0.05)

• Many believed results applied only to secondary

prevention

• No perceptible impact on prescriptions since HRT

believed to have other benefits
 Primary Prevention Randomized Trial of HRT
Women’s Health Initiative (WHI) Trial

• Large trial evaluating HRT in women without CHD

• Double-blind, randomized to estrogen-progestin vs placebo

• 16,608 patients followed for average of 5.2 years

• Trial stopped in 2002 on recommendation of Data Safety

Monitoring Board due to excessive risk of breast cancer
with HRT

JAMA 2002;288:321.
WHI Trial Studying HRT for Primary Prevention

JAMA 2002;288:321.
WHI Trial Studying HRT for Primary Prevention

JAMA 2002;288:321.
The announcement yesterday that a hormone replacement
regimen taken by six million Americans did more harm than
good was met with puzzlement and disbelief by women
and their doctors across the country
Real World Studies: The Problem of Confounding

• Many strategies employed to overcome confounding, but

often fail because not all confounders are easily identified.
• Unidentifed confounders lead to the problem of residual
confounding
 Hormone Replacement Therapy:
What Went Wrong?
• Women who chose to use HRT were healthier, had better
dietary and lifestyle patterns, resulting in lower
cardiovascular event rates

• Despite “adjustment” for many variables, residual

confounding in the observational studies resulted in false
positive results.

• Based on real world evidence, millions of American

women were exposed to a harmful therapy.
 Real World Studies:
Often a Parade of Low Quality Science

A few favorite “real world evidence” studies

widely covered by the media.
“After adjusting for age, height, smoking, physical
activity, and total energy intake, women who consumed
a 28-g (1 oz) serving size of nuts x2 times per week
experienced a significantly lower risk of pancreatic
cancer (RR, 0.65; 95% CI, 0.47–0.92; P for trend 
0.007) when compared with those who largely
abstained from nuts.”
Consumption of ≥2 servings per day of artificially
sweetened (diet) soda was independently associated
with eGFR decline ≥30% (OR 2.02, 95% CI 1.36 to 3.01)
Big Data is Often Bad Data
 Large UK Study: Relationship of BMI to Mortality

3.6 million patients

followed for 11.6 years
9% of UK Population

10 20 30 40 50
Lancet Diabetes Endocrinol 2018 BMI (kg/m2)
 H R fo r A ll-C a u s e M o rta lity
Meta-Analysis: BMI and All Cause Mortality (2013)

1.6

1.4 1.29
1.18
1.2
1.00
1.0 0.94 0.95

0.8

0.6

0.4

0.2

0.0 18.5-25 25-30 All ≥30 Subgroup Subgroup

30-35 >35
BMI Category
JAMA. 2013;309(1):71-82
 Authors’ Conclusions

“Grade 1 obesity overall was not associated

with higher mortality, and overweight was associated
with significantly lower all-cause mortality.”
 N=140,835

Lifetime Risk Pooling Project (10 studies, 3.2 million patient years)
 Evidence-Based Medicine
Hierarchy of Strength of Evidence Concerning
Efficacy of Treatment

Anecdotal case reports

Case series without controls

Case series with literature controls

Case-control observational studies

Meta-analysis of smaller randomized, controlled clinical trials

Single large randomized randomized, controlled clinical trial

Multiple randomized, controlled clinical trials

 Evidence Based Medicine
• Randomized Clinical Trial (RCT) is the gold standard
• RCT minimizes bias
• Can’t do RCT’s for all important questions (time, funding,
ethics)
• Must make choices on what evidence to use for clinical
guidelines
• For important questions with mortality or serious morbidity,
we need RCTs
• If RCTs not possible, we must be cautious and vigilant about
treatments only based on theory or observation/association
 Observational Studies
• Issue - correlation vs. causation. Real world studies can
suggest an association, but do not prove causation.
• Examples of False Positives:
- High cholesterol diet and rectal cancer
- Smoking and breast cancer
- Vasectomy and prostate cancer
- Red meat and colon cancer
- Red meat and breast cancer
- Drinking water frequently and bladder cancer
- Not consuming olive oil and breast cancer
• Replication of Observational Studies
- May not overcome confounding and bias
- Beta-carotene
 When are Observational Studies Useful?
• As “hypothesis generating” studies to inform design
of randomized clinical trials.

• When RCTs are not feasible or the effect size is so

large that a clinical trial not needed, for example:

– No RCT possible to study smoking and lung cancer

– The relative risk in observational studies was >9 !

• It is reasonable to consider “real world data” as

potentially informative if relative risk > 2.0 or <0.5.