UNIT 5
DESIGN AND ANALYSIS OF
MACHINE LEARNING
EXPERIMENTS
CROSS-VALIDATION
Cross-validation is a statistical method used in machine learning to
evaluate how well a model performs on an independent data set. It
involves dividing the available data into multiple folds or subsets, using
one of these folds as a validation set and training the model on the
remaining folds. This process is repeated multiple times each time using
a different fold as the validation set. Finally the results from each
validation step are averaged to produce a more robust estimate of the
model’s performance.
Types of Cross-Validation
• There are several types of cross validation techniques including k-fold
cross validation, leave-one-out cross validation, Holdout validation
and Stratified Cross-Validation. The choice of technique depends on
the size and nature of the data, as well as the specific requirements of
the modeling problem.
1. Holdout Validation
In Holdout Validation we perform training on the 50% of the given
dataset and rest 50% is used for the testing purpose. It’s a simple and
quick way to evaluate a model.
The major drawback of this method is that we perform training on the
50% of the dataset, it may possible that the remaining 50% of the data
contains some important information which we are leaving while
training our model i.e. higher bias.
LOOCV (Leave One Out Cross Validation)
In this method we perform training on the whole dataset but leaves only
one data-point of the available dataset and then iterates for each data-
point. In LOOCV the model is trained on n−1n−1 samples and tested on
the one omitted sample repeating this process for each data point in the
dataset. It has some advantages as well as disadvantages also.
An advantage of using this method is that we make use of all data points
and hence it is low bias.
• The major drawback of this method is that it leads to higher variation in
the testing model as we are testing against one data point. If the data
point is an outlier it can lead to higher variation. Another drawback is it
takes a lot of execution time as it iterates over ‘the number of data
points’ times.
K-Fold Cross Validation
• In K-Fold Cross Validation we split the dataset into k number of
subsets (known as folds) then we perform training on the all the
subsets but leave one(k-1) subset for the evaluation of the trained
model. In this method, we iterate k times with a different subset
reserved for testing purpose each time.
Example of K Fold Cross Validation
The diagram below shows an example of the training subsets and
evaluation subsets generated in k-fold cross-validation. Here we have
total 25 instances. In first iteration we use the first 20 percent of data
for evaluation and the remaining 80 percent for training ([1-5] testing
and [5-25] training) while in the second iteration we use the second
subset of 20 percent for evaluation and the remaining three subsets of
the data for training ([5-10] testing and [1-5 and 10-25] training) and so
on.
Classification Metrics
In a classification task, our main task is to predict the target variable, which
is in the form of discrete values. To evaluate the performance of such a
model, following are the commonly used evaluation metrics:
• Accuracy
• Logarithmic Loss
• Area Under Curve
• Precision
• Recall
• F1 Score
• Confusion Matrix
Accuracy
Accuracy is a fundamental metric for evaluating the performance of a classification model,
providing a quick snapshot of how well the model is performing in terms of correct
predictions. It is calculated as the ratio of correct predictions to the total number of input
samples.
• It works great if there are an equal number of samples for each class. For example, we
have a 90% sample of class A and a 10% sample of class B in our training set. Then, our
model will predict with an accuracy of 90% by predicting all the training samples
belonging to class A. If we test the same model with a test set of 60% from class A and
40% from class B. Then the accuracy will fall, and we will get an accuracy of 60%.
• Accuracy is good but it gives a False Positive sense of achieving high accuracy. The
problem arises due to the possibility of misclassification of minor class samples being
very high.
Area Under Curve (AUC)
True Positive Rate:
• Also called or termed sensitivity. True Positive Rate is considered as a
portion of positive data points that are correctly considered as
positive, with respect to all data points that are positive.
True Negative Rate
• Also called or termed specificity. True Negative Rate is considered as a
portion of negative data points that are correctly considered as
negative, with respect to all data points that are negatives.
False Positive Rate
• False Negatives rate is actually the proportion of actual positives that
are incorrectly identified as negatives
Precision
• There is another metric named Precision. Precision is a measure of a
model’s performance that tells you how many of the positive
predictions made by the model are actually correct.
Recall
• Recall is the ratio of correctly predicted positive instances to the total
actual positive instances. It measures how well the model captures all
relevant positive cases.
