Machine Learning
Unit 1
Performance Measures
Course: Machine Learning TY B. Tech(CSIT)
Faculty: Mrs. S. P. Patil, IT Dept., RIT
Performance metrics
■ We must carefully choose the metrics for
evaluating ML performance because −
– How the performance of ML algorithms is
measured and compared will be dependent
entirely on the metric you choose.
– How you weight the importance of various
characteristics in the result will be influenced
completely by the metric you choose.
Classification
Need for Confusion Matrices
Need for Confusion matrix
What is Confusion Matrix
Creating confusion Matrix
Creating Confusion Matrix
Type-I and Type-II Error
Confusion Matrix Metrics
■Accuracy
■Precision
■Recall
■F1-Score
Accuracy
Precision
Recall
F1-Score
Importance of precision VS.
recall
■ The importance of precision vs. recall depends on the
specific application.
– For instance, in a medical diagnosis system:
High recall might be crucial – catching as many
positive cases (diseases) as possible, even if it leads to
some false positives (unnecessary tests).
– a financial fraud detection system might prioritize high
precision – minimizing false positives (wrongly declined
transactions) to avoid inconveniencing customers.
Confusion Matix: MNIST data
example
Precision/Recall Trade-off
Precision/Recall Trade-off
Choice of metric and tradeoffs
Metric Guidance
• Accuracy Use as a rough indicator of model
training progress/convergence for
balanced datasets.
For model performance, use only in
combination with other metrics.
Avoid for imbalanced datasets. Consider
using another metric.
• Recall Use when false negatives are more
(True positive expensive than false positives.
rate)
• False positive rate Use when false positives are more
expensive than false negatives.
• Precision Use when it's very important for positive
predictions to be accurate.
ROC Curve
■ The ROC curve is a visual representation of model performance across all
thresholds.
■ The ROC curve is drawn by calculating the true positive rate (TPR) and false
positive rate (FPR) at every possible threshold (in practice, at selected
intervals), then graphing TPR over FPR.
■ A perfect model, which at some threshold has a TPR of 1.0 and a FPR of 0.0,
can be represented by either a point at (0, 1) if all other thresholds are
ignored, or by the following:
ROC and AUC of a hypothetical perfect model.
Area under the curve (AUC)
■ The area under the ROC curve (AUC) represents the probability that the
model, if given a randomly chosen positive and negative example, will
rank the positive higher than the negative.
■ The perfect model above, containing a square with sides of length 1, has
an area under the curve (AUC) of 1.0. This means there is a 100%
probability that the model will correctly rank a randomly chosen positive
example higher than a randomly chosen negative example.
■ In the spam classifier example, a spam classifier with AUC of 0.5 assigns
a random spam email a higher probability of being spam than a random
legitimate email only half the time.
ROC Curve
This ROC curve plots the false positive rate against the true positive rate for all
possible thresholds; the red circle highlights the chosen ratio (at 43.68% recall)
ROC-AUC
Today’s Task
■ Study Performance metrics
■ Solve one example in your notebook (Calculate the performance
measure values)
■ Perform the experiment considering one of the application (using
python programming.
References
■ https://www.youtube.com/watch?v=prWyZhcktn4&t=1079s
■ https://youtu.be/aWAnNHXIKww
