Underfitting
When the complexity of the model is too less for it to learn the data that is
given as input, the model is said to “Underfit”. In other words, the
excessively simple model fails to “Learn” the intricate patterns and
underlying trends of the given dataset. Underfitting occurs for a model with
Low Variance and High Bias.
Underfitting data Visualization: With the initial idea out of the way,
visualization of an underfitting model is important. This helps in determining
if the model is underfitting the given data during training. As already
discussed, supervised learning is of two types: Classification and
Regression. The following graphs show underfitting for both of these cases:
Classification: As shown in the figure below, the model is trained to
classify between the circles and crosses. However, it is unable to do so
properly due to the straight line, which fails to properly classify either of
the two classes.
Regression: As shown in the figure below, the data points are laid out in
a given pattern, but the model is unable to “Fit” properly to the given
data due to low model complexity.
Detection of underfitting model: The model may underfit the data,
but it is necessary to know when it does so. The following steps are the
checks that are used to determine if the model is underfitting or not.
1. Training and Validation Loss: During training and validation, it is
important to check the loss that is generated by the model. If the model is
underfitting, the loss for both training and validation will be significantly
high. In terms of Deep Learning, the loss will not decrease at the rate that
it is supposed to if the model has reached saturation or is underfitting.
2. Over Simplistic Prediction Graph: If a graph is plotted showing the
data points and the fitted curve, and the curve is over-simplistic (as
shown in the image above), then the model is suffering from underfitting.
A more complex model is to be tried out.
a. Classification: A lot of classes will be misclassified in the training
set as well as the validation set. On data visualization, the graph
would indicate that if there was a more complex model, more
classes would have been correctly classified.
b. Regression: The final “Best Fit” line will fail to fit the data points in
an effective manner. On visualization, it would clearly seem that a
more complex curve can fit the data better.
Fix for an underfitting model: If the model is underfitting, the
developer can take the following steps to recover from the underfitting state:
1. Train Longer: Since underfitting means less model complexity, training
longer can help in learning more complex patterns. This is especially true
in terms of Deep Learning.
2. Train a more complex model: The main reason behind the model to
underfit is using a model of lesser complexity than required for the data.
Hence, the most obvious fix is to use a more complex model. In terms of
Deep Learning, a deeper network can be used.
3. Obtain more features: If the data set lacks enough features to get a
clear inference, then Feature Engineering or collecting more features will
help fit the data better.
4. Decrease Regularization: Regularization is the process that helps
Generalize the model by avoiding overfitting. However, if the model is
learning less or underfitting, then it is better to decrease or completely
remove Regularization techniques so that the model can learn better.
5. New Model Architecture: Finally, if none of the above approaches work,
then a new model can be used, which may provide better results.
Overfitting
When the complexity of the model is too high as compared to the data that it
is trying to learn from, the model is said to “Overfit”. In other words, with
increasing model complexity, the model tends to fit the Noise present in data
(eg. Outliers). The model learns the data too well and hence fails to
Generalize. Overfitting occurs for a model with High Variance and Low Bias.
Overfitting data Visualization: With the initial idea out of the way,
visualization of an overfitting model is important. Similar to underfitting,
overfitting can also be showcased in two forms of supervised learning:
Classification and Regression. The following graphs show overfitting for
both of these cases:
Classification: As shown in the figure below, the model is trained to
classify between the circles and crosses, and unlike last time, this time
the model learns too well. It even tends to classify the noise in the data by
creating an excessively complex model (right).
Regression: As shown in the figure below, the data points are laid out in
a given pattern, and instead of determining the least complex model that
fits the data properly, the model on the right has fitted the data points too
well when compared to the appropriate fitting (left).
Detection of overfitting model: The parameters to look out for to
determine if the model is overfitting or not is similar to those of underfitting
ones. These are listed below:
1. Training and Validation Loss: As already mentioned, it is important to
measure the loss of the model during training and validation. A very low
training loss but a high validation loss would signify that the model is
overfitting. Additionally, in Deep Learning, if the training loss keeps on
decreasing but the validation loss remains stagnant or starts to increase,
it also signifies that the model is overfitting.
2. Too Complex Prediction Graph: If a graph is plotted showing the data
points and the fitted curve, and the curve is too complex to be the
simplest solution which fits the data points appropriately, then the model
is overfitting.
a. Classification: If every single class is properly classified on the
training set by forming a very complex decision boundary, then
there is a good chance that the model is overfitting.
b. Regression: If the final “Best Fit” line crosses over every single
data point by forming an unnecessarily complex curve, then the
model is likely overfitting.
Fix for an overfitting model: If the model is overfitting, the developer
can take the following steps to recover from the overfitting state:
1. Early Stopping during Training: This is especially prevalent in Deep
Learning. Allowing the model to train for a high number of epochs
(iterations) may lead to overfitting. Hence it is necessary to stop the
model from training when the model has started to overfit. This is done by
monitoring the validation loss and stopping the model when the loss stops
decreasing over a given number of epochs (or iterations).
2. Train with more data: Often, the data available for training is less when
compared to the model complexity. Hence, in order to get the model to fit
appropriately, it is often advisable to increase the training dataset size.
3. Train a less complex model: As mentioned earlier, the main reason
behind overfitting is excessive model complexity for a relatively less
complex dataset. Hence it is advisable to reduce the model complexity in
order to avoid overfitting. For Deep Learning, the model complexity can
be reduced by reducing the number of layers and neurons.
4. Remove features: As a contrast to the steps to avoid underfitting, if the
number of features is too many, then the model tends to overfit. Hence,
reducing the number of unnecessary or irrelevant features often leads to
a better and more generalized model. Deep Learning models are usually
not affected by this.
5. Regularization: Regularization is the process of simplification of the
model artificially, without losing the flexibility that it gains from having a
higher complexity. With the increase in regularization, the effective model
complexity decreases and hence prevents overfitting.
6. Ensembling: Ensembling is a Machine Learning method which is used to
combine the predictions from multiple separate models. It reduces the
model complexity and reduces the errors of each model by taking the
strengths of multiple models. Out of multiple ensembling methods, two of
the most commonly used are Bagging and Boosting
Overfitting and Underfitting are two crucial concepts in machine learning and are
the prevalent causes for the poor performance of a machine learning model. This
tutorial will explore Overfitting and Underfitting in machine learning, and help you
understand how to avoid them with a hands-on demonstration.
What is Overfitting?
When a model performs very well for training data but has poor performance with test data (new
data), it is known as overfitting. In this case, the machine learning model learns the details and
noise in the training data such that it negatively affects the performance of the model on test data.
Overfitting can happen due to low bias and high variance.
Reasons for Overfitting
Data used for training is not cleaned and contains noise (garbage values) in it
The model has a high variance
The size of the training dataset used is not enough
The model is too complex
Ways to Tackle Overfitting
Using K-fold cross-validation
Using Regularization techniques such as Lasso and Ridge
Training model with sufficient data
Adopting ensembling techniques
What is Underfitting?
When a model has not learned the patterns in the training data well and is unable to generalize
well on the new data, it is known as underfitting. An underfit model has poor performance on the
training data and will result in unreliable predictions. Underfitting occurs due to high bias and
low variance.
Reasons for Underfitting
Data used for training is not cleaned and contains noise (garbage values) in it
The model has a high bias
The size of the training dataset used is not enough
The model is too simple
Ways to Tackle Underfitting
Increase the number of features in the dataset
Increase model complexity
Reduce noise in the data
Increase the duration of training the data
Now that you have understood what overfitting and underfitting are, let’s see what is a good fit
model in this tutorial on overfitting and underfitting in machine learning.
ML | Underfitting and Overfitting
Difficulty Level : Medium
Last Updated : 20 Oct, 2021
Let us consider that we are designing a machine learning model.
A model is said to be a good machine learning model if it
generalizes any new input data from the problem domain in a
proper way. This helps us to make predictions in the future data,
that the data model has never seen. Now, suppose we want to
check how well our machine learning model learns and
generalizes to the new data. For that, we have overfitting and
underfitting, which are majorly responsible for the poor
performances of the machine learning algorithms.
Before diving further let’s understand two important terms:
Bias: Assumptions made by a model to make a function easier
to learn.
Variance: If you train your data on training data and obtain a
very low error, upon changing the data and then training the
same previous model you experience a high error, this is
variance.
Underfitting:
A statistical model or a machine learning algorithm is said to have
underfitting when it cannot capture the underlying trend of the
data. (It’s just like trying to fit undersized pants!) Underfitting
destroys the accuracy of our machine learning model. Its
occurrence simply means that our model or the algorithm does
not fit the data well enough. It usually happens when we have
fewer data to build an accurate model and also when we try to
build a linear model with fewer non-linear data. In such cases, the
rules of the machine learning model are too easy and flexible to
be applied on such minimal data and therefore the model will
probably make a lot of wrong predictions. Underfitting can be
avoided by using more data and also reducing the features by
feature selection.
In a nutshell, Underfitting – High bias and low variance
Techniques to reduce underfitting:
1. Increase model complexity
2. Increase the number of features, performing feature engineering
3. Remove noise from the data.
4. Increase the number of epochs or increase the duration of training to
get better results.
Overfitting:
A statistical model is said to be overfitted when we train it with a lot of
data (just like fitting ourselves in oversized pants!). When a model gets
trained with so much data, it starts learning from the noise and inaccurate
data entries in our data set. Then the model does not categorize the data
correctly, because of too many details and noise. The causes of
overfitting are the non-parametric and non-linear methods because these
types of machine learning algorithms have more freedom in building the
model based on the dataset and therefore they can really build unrealistic
models. A solution to avoid overfitting is using a linear algorithm if we
have linear data or using the parameters like the maximal depth if we are
using decision trees.
In a nutshell, Overfitting – High variance and low bias
Examples:
Techniques to reduce overfitting:
1. Increase training data.
2. Reduce model complexity.
3. Early stopping during the training phase (have an eye over the loss
over the training period as soon as loss begins to increase stop
training).
4. Ridge Regularization and Lasso Regularization
5. Use dropout for neural networks to tackle overfitting.
Good Fit in a Statistical Model:
Ideally, the case when the model makes the predictions with 0 error, is
said to have a good fit on the data. This situation is achievable at a spot
between overfitting and underfitting. In order to understand it, we will
have to look at the performance of our model with the passage of time,
while it is learning from training dataset.
With the passage of time, our model will keep on learning and thus the
error for the model on the training and testing data will keep on
decreasing. If it will learn for too long, the model will become more prone
to overfitting due to the presence of noise and less useful details. Hence
the performance of our model will decrease. In order to get a good fit, we
will stop at a point just before where the error starts increasing. At this
point, the model is said to have good skills on training datasets as well as
our unseen testing dataset.
Overfitting and Underfitting in Machine
Learning
Overfitting and Underfitting are the two main problems that occur in
machine learning and degrade the performance of the machine
learning models.
The main goal of each machine learning model is to generalize well.
Here generalization defines the ability of an ML model to provide a
suitable output by adapting the given set of unknown input. It means
after providing training on the dataset, it can produce reliable and
accurate output. Hence, the underfitting and overfitting are the two
terms that need to be checked for the performance of the model and
whether the model is generalizing well or not.
Before understanding the overfitting and underfitting, let's understand
some basic term that will help to understand this topic well:
o Signal: It refers to the true underlying pattern of the data that
helps the machine learning model to learn from the data.
o Noise: Noise is unnecessary and irrelevant data that reduces the
performance of the model.
o Bias: Bias is a prediction error that is introduced in the model
due to oversimplifying the machine learning algorithms. Or it is
the difference between the predicted values and the actual
values.
o Variance: If the machine learning model performs well with the
training dataset, but does not perform well with the test dataset,
then variance occurs.
Overfitting
Overfitting occurs when our machine learning
model tries to cover all the data points or more than the required data points present in the given
dataset. Because of this, the model starts caching noise and inaccurate values present in the dataset,
and all these factors reduce the efficiency and accuracy of the model. The overfitted model has low
bias and high variance.
The chances of occurrence of overfitting increase as much we provide
training to our model. It means the more we train our model, the more
chances of occurring the overfitted model.
Overfitting is the main problem that occurs in supervised learning
Example: The concept of the overfitting can be understood by the
below graph of the linear regression output:
As we can see from the above graph, the model tries to cover all the
data points present in the scatter plot. It may look efficient, but in
reality, it is not so. Because the goal of the regression model to find
the best fit line, but here we have not got any best fit, so, it will
generate the prediction errors.
How to avoid the Overfitting in Model
Both overfitting and underfitting cause the degraded performance of
the machine learning model. But the main cause is overfitting, so there
are some ways by which we can reduce the occurrence of overfitting in
our model.
o Cross-Validation
o Training with more data
o Removing features
o Early stopping the training
o Regularization
o Ensembling
Underfitting
Underfitting occurs when our machine learning model is not able to
capture the underlying trend of the data. To avoid the overfitting in the
model, the fed of training data can be stopped at an early stage, due
to which the model may not learn enough from the training data. As a
result, it may fail to find the best fit of the dominant trend in the data.
In the case of underfitting, the model is not able to learn enough from
the training data, and hence it reduces the accuracy and produces
unreliable predictions.
An underfitted model has high bias and low variance.
Example: We can understand the underfitting using below output of
the linear regression model:
As we can see from the above diagram, the model is unable to capture
the data points present in the plot.
How to avoid underfitting:
o By increasing the training time of the model.
o By increasing the number of features.
