Scribd
Upload
20 views17 pages

Session 2 Evaluation Boosting Bagging Contemporary Business Anaytics

Uploaded by

Moneesh Basker
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
20 views17 pages

Session 2 Evaluation Boosting Bagging Contemporary Business Anaytics

Uploaded by

Moneesh Basker
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 17

Contemporary Business Analytics

Evaluation of Algorithms
Bagging and Boosting
What is Machine Learning?
 “A computer program is said to learn from experience E with respect to
some class of tasks T and performance measure P, if its performance at
tasks in T, as measured by P, improves with experience E.”
 Tasks
 Analysing medical reports for identifying patients who might be
suffering from cancer
 Predicting possibility of fraud by looking at profiles of loan applicants
 Identifying objects in a picture
 Examples of tasks are:
 Classification, Regression, Machine translation, anomaly detection…
…

2
Experience
 Consists of datasets from the past which may or may not be labelled

3
Performance Measure

 The measure P is specific to the task T being carried out


 It’s accuracy or error rate for classification tasks
 Efficacy of algorithms sometimes also measured in terms of precision and
recall based on the confusion matrix
 Confusion Matrix

Predicted Values
Negative Positive

Negative True Negative (TN) False Positive (FP)


Actual Precision
Values
Positive False Negative (FN) True Positive (TP)
Recall
4
Evaluation

Test Data Test

Model Prediction
Evaluation – Test Data

Predicted Actual
0.673536 0
Test Data
0.407434 0
0.582201 1
0.601923 0
… …
0.99839 0
0.950054 1
Evaluation – Test Data

Threshold
Predicted Predicted Actual
0.673536 1 0
0.407434 if (predicted_value ≤ 0.5) 0 0
0.582201 predicted = 0 1 1
0.601923 else 1 0
… predicted = 1 … …
0.99839 1 0
0.950054 1 1
The Four Values
 True Positive: Model predicts Positive and it is True

 True Negative: Model predicts Negative and it is True

 False Positive: Model predicts Positive and it is False (Type I


Error)

 False Negative: Model predicts Negative and it is False (Type


II Error)
Confusion Matrix

Predicted Values
Negative Positive

Negative True Negative (TN) False Positive (FP)


Actual
Values
Positive False Negative (FN) True Positive (TP)
Accuracy

Out of the total number of observations how many


the classifier could predict correctly
Precision

Out of the total number of positive predictions by


the classifier how many are correct

𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑏𝑦 𝑡h𝑒𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑟


𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑏𝑦 𝑡h𝑒 𝐶𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑟
Recall / True Positive Rate

Out of the total number of positive observations how many the


classifier could correctly predict

𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑏𝑦 𝑡h𝑒𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑟


𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐴𝑐𝑡𝑢𝑎𝑙 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠
False Positive Rate

Out of the total number of negative observations how many the


classifier incorrectly predicted as positive

𝐼𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑏𝑦 𝑡h𝑒 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑟


𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠
Specificity / True Negative Rate

Out of the total number of negative observations how many the


classifier correctly predicted as negative

𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑏𝑦 𝑡h𝑒𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑟


𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠
F1 / F2 Score
F1 score is the harmonic mean of precision and recall

If Recall is more important than precision, then we use the F 2 score – Recall has twice
the weightage of Precision

The general formula for the F-score is


Receiver Operating Characteristic Curve
Precision Recall Curve

You might also like