ORIGINAL RESEARCH • RECHERCHE ORIGINALE
METHODOLOGY
Understanding receiver operating characteristic
(ROC) curves
Jerome Fan, MD; Suneel Upadhye, MD, MSc; Andrew Worster, MD, MSc
S EE ALSO PAGE 13.
I n this issue of the Journal, Auer and colleagues1 con-
clude that serum levels of neuron-specific enolase
(NSE), a biochemical marker of ischemic brain injury, may
with high specificity allows the authors to “rule-in” the
outcome for all patients with a NSE value above the se-
lected cutoff.4 The study indicates that patients with a NSE
have clinical utility for the prediction of survival to hospi- level >30 µg/L will die before hospital discharge and those
tal discharge in patients experiencing the return of sponta- with a NSE level <29 µg/L will possibly survive to hospi-
neous circulation following at least 5 minutes of cardiopul- tal discharge.
monary resuscitation. The authors used a receiver The area under the ROC curve (AUC) is widely recog-
operating characteristic (ROC) curve to illustrate and eval- nized as the measure of a diagnotic test’s discriminatory
uate the diagnostic (prognostic) performance of NSE. We power.5 The maximum value for the AUC is 1.0, thereby
explain ROC curve analysis in the following paragraphs. indicating a (theoretically) perfect test (i.e., 100% sensitive
The term “receiver operating characteristic” came from
tests of the ability of World War II radar operators to deter-
Optimum
mine whether a blip on the radar screen represented an ob- 1.0 Cut-off value
ject (signal) or noise. The science of “signal detection the-
0.9
ory” was later applied to diagnostic medicine. 2 The
determination of an “ideal” cut-off value is almost always 0.8
a trade-off between sensitivity (true positives) and speci- 0.7
ficity (true negatives). As both change with each “cut-off”
Sensitivity
0.6
value it becomes difficult for the reader to imagine which
cut-off is ideal. The ROC curve offers a graphical illustra- 0.5 Area under the curve (AUC) = 0.97
tion of these trade-offs at each “cut-off” for any diagnostic 0.4
test that uses a continuous variable.3 Ideally, the best “cut-
0.3
off” value provides both the highest sensitivity and the
highest specificity, easily located on the ROC curve by 0.2
finding the highest point on the vertical axis and the fur- 0.1
thest to the left on the horizontal axis (upper left corner)
(Fig. 1). However, it is rare that this ideal can be achieved, 0.0
so that, for example, one may opt to choose a higher sensi- 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
1 – Specificity
tivity at the cost of lower specificity. In the NSE study,1 the
authors chose a cut-off point of >30 µg/L with a specificity Fig. 1. Receiver operating characteristic curve illustrating
of 100% and sensitivity of 79% (Fig. 2). A cut-off point high discriminatory power.
Division of Emergency Medicine, McMaster University, Hamilton, Ont.
Received: Nov. 15, 2005; accepted: Nov. 22, 2005
Can J Emerg Med 2006;8(1):19-20
January • janvier 2006; 8 (1) CJEM • JCMU 19
https://doi.org/10.1017/S1481803500013336 Published online by Cambridge University Press
Fan et al
and 100% specific). An AUC value of 0.5 indicates no dis- discriminatory power and, therefore potential utility as a
criminative value (i.e., 50% sensitive and 50% specific) diagnostic test in determining the non-survivors of a car-
and is represented by a straight, diagonal line extending diac arrest with return of spontaneous circulation.
from the lower left corner to the upper right (Fig. 3). There It is important to note that ROC performance may
are several scales for AUC value interpretation but, in gen- change when the diagnostic test is applied to different clin-
eral, ROC curves with an AUC ≤0.75 are not clinically ical situations (e.g., patient populations) or under different
useful and an AUC of 0.97 has a very high clinical value, phases of test development (derivation, validation). The
correlating with likelihood ratios of approximately 10 and most useful information from a diagnostic test likely origi-
0.1. The AUC for NSE was 0.87, demonstrating moderate nates by pooling the results of several studies examining
the same test in different situations, generating averaged
1.0
specificity, sensitivity and ROC, so as to be able to get a
true understanding of the diagnostic test’s utility.6
0.9 30 µg/L
In summary, ROC analysis provides important informa-
0.8 tion about diagnostic test performance: the closer the apex
0.7
of the curve toward the upper left corner, the greater the dis-
criminatory ability of the test (i.e., the true-positive rate is
Sensitivity
0.6
high and the false-positive [1 – Specificity] rate is low).
0.5 This is measured quantitatively by the AUC such that a
0.4
value of >0.96 indicates excellent discriminatory ability.
Like all summary measures, however, there are confidence
0.3
intervals around this value that must be taken into consider-
0.2 ation. In the end, it will be rare for a diagnostic test to have
0.1
both 100% specificity and sensitivity. The clinician will
have to decide which cut-off value will provide the likeli-
0.0 hood ratios and sensitivity and specificity values that have
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 the greatest clinical value in the diagnosis of any disorder.
1 – Specificity
Competing interests: None declared.
Fig. 2. Receiver operating characteristic curve for the overall
performance of neuron-specific enolase to predict survival Key words: receiver operating characteristic curve; diagnosis; sensi-
at 48 hours after return of spontaneous circulation. tivity; specificity
1.0 References
1. Auer J, Berent R, Weber T, et al. Ability of neuron-specific eno-
0.9
lase to predict survival to hospital discharge after successful car-
0.8 diopulmonary resuscitation. Can J Emerg Med 2006;8(1):13-8.
2. Lusted LB. Decision-making studies in patient management.
0.7 N Engl J Med 1971; 284:416-24.
Sensitivity
0.6 3. Grzybowski M, Younger JG. Statistical methodology: III. Re-
ceiver operating characteristic (ROC) curves. Acad Emerg Med
0.5 1997;4:818-26.
0.4
4. Worster A, Innes G, Abu-Laban RB. Diagnostic testing: an
emergency medicine perspective. Can J Emerg Med 2002;4(5):
0.3 348-54.
Area under the curve (AUC) = 0.5 5. Faraggi D, Reiser B. Estimation of the area under the ROC
0.2 curve. Stat Med 2002;21:3093-106.
0.1 6. Deeks J. Systematic reviews of evaluations of diagnostic and
screening tests. In: Egger M, Smith GD, Altman DG, editors.
0.0 Systematic reviews in health care: meta-analysis in context.
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 BMJ Publishing Group; 2001.
1 – Specificity
Correspondence to: Dr. Andrew Worster, Emergency Department, Hamil-
Fig. 3. Receiver operating characteristic curve illustrating no ton Health Sciences, McMaster University Medical Centre, 1200 Main St.
discriminatory power. W, Hamilton ON L8N 3Z5
20 CJEM • JCMU January • janvier 2006; 8 (1)
https://doi.org/10.1017/S1481803500013336 Published online by Cambridge University Press
