Confusion Matrix
for evaluating the Knn model
KNN Algorithm
• Generates a model that can predict based on supervised learning.
• Trained on labelled data
• It is a nearest neighbour Algorithm
• Prediction of new data is done based on feature similarity with the
neighbours.
• Also called feature similarly algorithm
• We should choose a right ‘K’….usually sqrt(n)
Examples where KNN is used to predict
• Amazon uses it to recommend books to the new customers
• Bank uses it to approve loan
• Doctors use it to predict diabetics
• Predicting risk factor of prostrate cancer
Steps for Developing a KNN Algorithm
Step -1: Data Collection followed by importing data in R
Step -2 Prepare and explore data
Step 3: Normalizing numeric data
Step 4: Creating training and test data set
Step 5: Training a model on data
Step 6: Evaluate the model performance….Using Confusion Matrix
What is a Confusion Matrix (CM)
• useful tool for calibrating the output of a model.....a tool for evaluating the
performance of the model
• examines all possible outcomes...True Positive (TP), True Negative(TN), False
Positive(FP), False Negative(FN)
• categorises the predictions against the actual values....it is a 2-dim matrix of predicted
values X actual values
• gives lot of additional data in addition to the accuracy of the KNN model.
• has this name as it shows how confused the model is between predicted outcome
values and actual outcome values.
• The columns in CM represent actual classes and rows represent the predicted
classes….or VV
How does Confusion Matrix (CM) look like?
What does TP, TN, FP, FN stand for in the Confusion Matrix (CM)?
In all there can be 4 possible outcomes:
TP: these are cases where the model correctly predicts outcome Y
FP: these are cases where the model incorrectly predicts outcome Y
TN: these are cases where the model correctly predicts N
FN: these are cases where the model incorrectly predicts outcome N
We can have binary matrix with two levels or more.
In case of fibre identification problem, there are three level : Cotton, Silk, Wool for
both Actual and predicted values........CM in this case will be a 3X3 matrix.
Examples
Confusion Matrix (CM)
For predicting mails as Spam or Non-spam
Example: Confusion Matrix (CM)....
for predicting Cancer or No-Cancer in diagnoses
Example: Confusion Matrix (CM)....
for predicting Cancer is benign or malignant
For prediction of prostrate cancer among men
based on their medical report for test 1 and test 2.
• At random sample tests (say, test 1 and test 2)
performed on 500 men ,
• Of these, 50 actually have prostate cancer.
• I predicted 100 total cases of prostrate cancer
• 45 of which are actually having prostrate cancer.
Confusion Matrix
Actual Positive Actual Negative
Type 1
Predicted Positive TP 45 FP 55
Predicted Negative FN 5 TN 395
Type 2
We can measure other additional information in Confusion Matrix
Actual values
Predicted values
For prediction of prostrate cancer among men
based on their medical report for test 1 and test 2.
• At random sample tests performed on500 men ,
Calculate the statistics using
• Of these, 50 actually have prostate cancer. labelled confusion matrix:
• I predicted 100 total cases of prostrate cancer
1. Accuracy (all correct / all) = TP + TN / TP + TN + FP + FN
• 45 of which are actually having PC. (45 + 395) / 500 = 440 / 500 = 0.88 or 88% Accuracy
Actual Actual
2. Misclassification (all incorrect / all) = FP + FN / TP + TN + FP +
Positive Negative FN
Predicted TP 45 FP 55 (55 + 5) / 500 = 60 / 500 = 0.12 or 12% Misclassification
Precision
Positive You can also just do 1 — Accuracy,
so: 1–0.88 = 0.12 or 12% Misclassification
Predicted FN 5 TN 395
3. Precision (true positives / predicted positives) = TP / TP + FP
Negative 45 / (45 + 55) = 45 / 100 = 0.45 or 45% Precision
Sensitivity Specificity
4. Sensitivity aka Recall (true positives / all actual positives) = TP /
TP + FN
45 / (45 + 5) = 45 / 50 = 0.90 or 90% Sensitivity
5. Specificity (true negatives / all actual negatives) =TN / TN + FP
395 / (395 + 55) = 395 / 450 = 0.88 or 88% Specificity
Therefore, KNN model built for prediction of prostrate cancer among
men based on their medical report for test 1 and test 2
(hypothetical….data not provided) with outcome as “have prostrate
cancer” or “do not have prostrate cancer” is 88% accurate.
Interpretation of the Results
• A confusion matrix can help us evaluate the performance of our models, it
provides statistics ….Accuracy, Precision, Sufficiency, etc….one or more of
these can be used to evaluate the model.
• In the given example (prostrate cancer), it can be seen that there are high
incidences of false positives (45 out of 45+55=100). Therefore, precision
given by TP/TP+FP ……is just 45%. This means that we are falsely
predicting prostrate cancer 55% of the time. Our model is thus NOT
precise.
Command in R to execute the Confusion Matrix
confusionMatrix(data = m1,factor(fibre_test_target))
1. We specify the predicted data and the actual data as the arguments
2. Both the datasets should be of factor type….convert them to factor in case it is not
conf_matrix <- confusionMatrix(data = m1,factor(fibre_test_target))
Assign the output of CM to a variable
print (conf_matrix)
Print confusion matrix to evaluate the model based on the statistics
R programme for predicting the fibre type (cotton, silk, wool) using KNN Model
Result of confusion matrix in KNN Model
Model’s accuracy is 1
