Evaluation Metrics for classifica-
tion
Confusion Matrix, Precision, Re-
call, F-score, Accuracy, ROC
curve
Machine Learning, CS354
Slides courtesy: Nuriev Sirojiddin, Tuong
1 Le
CONTENTS
BINARY CLASSIFIER
CONFUSION MATRIX
ACCURACY
PRECISION
RECALL
ROC CURVE
- HOW TO PLOT ROC CURVE
2
Binary Classifier
A binary classifier produces output with two class values or labels,
such as Yes/No, 1/0, Positive/Negative for given input data
For performance evaluation: Observed labels are used to compare
with the predicted labels after classification
The predicted labels will be exactly the same, if the performance of
a classifier is perfect.
But it is uncommon to be able to develop a perfect classifier.
3
Confusion Matrix
A confusion matrix is formed from the four outcomes
produced as a result of binary classification
True positive (TP): correct positive prediction
False positive (FP): incorrect positive prediction
True negative (TN): correct negative prediction
False negative (FN): incorrect negative prediction
4
Confusion Matrix
Binary Classification: Green Vs. Grey
• Rows represent what is predicted, and columns represent what
is the actual label.
5
Confusion Matrix
True Positives False Positives
Green examples Gray examples
correctly identified Confusion Matrix falsely identified as
as green green
False Negatives True Negatives
Green examples Gray examples cor-
falsely identified as rectly identified as
grey grey
6
Accuracy
Accuracy is calculated as the number of all correct predictions
divided by the total number of the dataset
The best ACC is 1.0, whereas the worst is 0.0
= (TP + TN) / (TP+ FP + TN + FN)
Accuracy = (9+8) / (9+2+1+8) = 0.85
7
Precision
Precision is calculated as the number of correct positive predictions
divided by the total number of positive predictions
The best precision is 1.0, whereas the worst is 0.0
Precision = 9 / (9+2) = 0.81
8
Recall
Sensitivity = Recall = True Positive Rate
Recall is calculated as the number of correct positive predictions
divided by the total number of positives
The best recall is 1.0, whereas the worst is 0.0
= TP / (TP+FN)
Recall = 9 / (9+1) = 0.9
9
Example 1
Example: The example to classify whether images contain either a dog
or a cat
The training data contains 25000 images of dogs and cats;
The training data 75% of 25000 images; (25000*0.75 = 18750)
Validation data 25% of training data; (25000*0.25 = 6250)
Test Data, 5 cats, 5 dogs
Precision = 2/(2 + 0) * 100% = 100%
Recall = 2/(2 + 3) * 100% = 40%
Accuracy = (2 + 5)/(2 + 0 + 3 + 5) * 100% = 70%
10
Example 2 (Multi-class classification)
= = = 170/300 = .556
= = .5
= = .5
= = .667
= 0.556
= = .3
= = .6
= = 0.8
P = 0.556
11
Precision vs Recall
There is always a tradeoff between precision and recall.
Higher precision means lower recall and vice versa.
Due to this tradeoff, precision/recall alone will not be very useful.
We need to compute both the measures to get a true picture.
There is another measure, which takes into account both precision and
recall, i.e. the F-measure.
12
The F-measure
F-measure: A combined measure that assesses the precision/recall
tradeoff
It is computed as a weighted harmonic mean of both preci-
sion and recall. 1 ( 2
1) PR
F 2
1 1 PR
(1 )
P R
People usually use balanced F1 measure, where F1 measure is an F measure with
β=1
i.e., with β = 1 or α = ½
2 𝑃𝑅
Thus, the F1-measure can be computed using the following equation:
𝐹 1=
𝑃+𝑅
For example 1, precision was 1.0 and recall was 0.4, therefore, F1-measure can
13
The F1-measure (Example)
In example 1:
Precision was 1.0 2 𝑃𝑅
𝐹 1=
Recall was 0.4
𝑃+𝑅
Therefore, F1-measure can be computed as:
2∗ 1.0 ∗0.4
𝐹 1=
1.0+ 0.4
0.8
𝐹 1= =0.57
1.4
Therefore, F1-measure = 0.57
Try yourself:
Precision=0.8, recall=0.3
Precision=0.4, recall=0.9
14
ROC Curve
Binary classification Confusion Ma-
trix
ROC Curve basics
The ROC curve is an evaluation measure that is based on two
basic evaluation measures:
- specificity and sensitivity
Specificity = True Negative Rate,
Sensitivity – It is the same as Recall = True Positive Rate,
Specificity
Specificity is calculated as the number of correct negative
predictions divided by the total number of negatives.
17
ROC Curve basics (contd.)
In order to understand, let us consider a disease detection (binary classification)
problem.
Here, positive class represents diseased people and negative class represents
healthy people.
Broadly speaking, these two quantities tell us how good we are at detecting
diseased people (sensitivity) and healthy people (specificity).
The sensitivity is the proportion of the diseased people (T P +F N) that
we correctly classify as being diseased (T P).
Specificity is the proportion of all of the healthy people (T N + F P) that
we correctly classify as being healthy (T N).
18
ROC Curve basics (contd.)
If we diagnosed everyone as healthy, we would have a specificity
of 1 (very good – we diagnose all healthy people correctly), but a
sensitivity of 0 (we diagnose all unhealthy people incorrectly)
which is very bad.
Ideally, we would like Se = Sp = 1, i.e. perfect sensitivity and
specificity.
It is often convenient to be able to combine sensitivity and specific-
ity into a single value.
This can be achieved through evaluating the area under the re-
ceiver operating characteristic (ROC) curve.
19
What is an ROC Curve?
The ROC curve stands for Receiver Operating Characteristics
curve.
Used in signal detection to show the tradeoff between hit rate and
false alarm rate over a noisy channel, hence the term ‘receiver’
operating characteristics.
It is a visual tool for evaluating the performance of a binary classifier.
It illustrates the diagnostic ability of a binary classification system
as its discrimination threshold is varied.
An example ROC curve is shown in the figure.
The blue line represents the ROC curve.
The dashed line is a reference curve.
20
ROC Curve presumption
Before, we can plot the ROC curve, it is assumed that the classifier produces a positivity score,
which can then be used to determine a discrete label.
E.g. in case of NB classifier, the positivity score is the probability of positive class given the
test example. Usually, a cut-off threshold v = 0.5 is applied on this positivity score for pro-
ducing the output label.
In case of ML algorithms, that produce a discrete label for the test examples, slight modifications
can be made to produce a positivity score, e.g.
1. In KNN algorithm with K=3, majority voting is used to produce a discrete output label.
However, a positive score can be produced by computing the probability/ratio of posi-
tive neighbors of the test example.
i.e. if K=3, and the labels for closest neighbors are: [1, 0, 1], then the positivity score
may be (1+0+1)/3 = 2/3 = 0.66. A cut-off threshold v can be applied on this positivity
score to produce an output label, usually, v = 0.5.
For decision tree algorithm, refer to the following link:
https://stats.stackexchange.com/questions/105760/how-we-can-draw-an-roc-curve-for-decision-trees
21
ROC Curve presumption
An ROC curve is plotted by varying the cut-off threshold v.
The y-axis represents the sensitivity (true positive rate).
The x-axis represents 1-specificity, also called the false positive rate.
The threshold v is varied (e.g. from 0 to 1), and a pair of sensitivity/specificity
value is achieved, which forms a single point on the ROC curve.
E.g. for v = 0.5, say sensitivity=0.8, specificity=0.6 (hence FPR=1-speci-
ficity=0.4), hence (x,y)=(0.8, 0.4). However, if v = 0.6, say sensitivity=0.7,
specificity=0.7 (hence FPR=1-specificity=0.3), hence (x,y) = (0.7, 0.3)
The final ROC curve is drawn by connecting all these points, e.g. given below:
22
How to Plot ROC Curve - Example
Dynamic cut-off thresh-
olds
Cut-off = 0.020 Cut-off = 0.015 Cut-off = 0.010
23
How to Plot ROC Curve - Example
True positive rate (TPR) = 𝑇𝑃/(𝑇𝑃+𝐹𝑁)
and False positive rate (FPR) = 𝐹𝑃/(𝐹𝑃+𝑇𝑁)
Use different cut-off thresholds (0.00, 0.01, 0.02,…, 1.00),
calculate the TPR and FPR, and plot them into graph.
That is receiver operating characteristic (ROC) curve.
Example
TPR = 0.5 TPR = 1 TPR = 1
FPR = 0 FPR = 0.167 FPR = 0.667
24
How to Plot ROC Curve
A ROC curve is created by connecting ROC points of a classifier
A ROC point is a point with a pair of x and y values
where x is 1-specificity and y is sensitivity
The curve starts at (0.0, 0.0) and ends at (1.0, 1.0)
25
ROC Curve
A classifier with the random performance level always shows a straight
line
Two areas separated by this ROC curve
ROC curves in the area with the top left corner indicate good performance
levels
ROC curves in the other area with the bottom right corner indicate poor per-
formance levels
26
ROC Curve
A classifier with the perfect performance level shows
a combination of two straight lines
It is important to notice that classifiers with meaningful performance
levels
usually lie in the area between the random ROC curve and the perfect
ROC curve
27
The AUC measure
AUC(Area under the ROC curve) score
An advantage of using ROC curve is a single measure called AUC score
As the name indicates, it is an area under the curve calculated in the ROC space
Although the theoretical range of AUC score is between 0 and 1, the actual scores of
meaningful classifiers are greater than 0.5, which is the AUC score of a random classi-
fier
ROC curves clearly shows classifiers A
outperforms classifier B
28
