SMOTE for Imbalanced Classification with Python

When a dataset has more samples of one class and very few of another, the model tends to predict the majority class more often. This problem is called class imbalance. The Synthetic Minority Over-sampling Technique (SMOTE) helps fix this issue by creating new synthetic samples for the smaller (minority) class. This makes the dataset more balanced and helps the model learn both classes properly.

• SMOTE creates new synthetic data instead of copying existing samples.
• It improves accuracy for the minority class.
• Variants like ADASYN, Borderline SMOTE, SMOTE-ENN and SMOTE-TOMEK make SMOTE even more effective.
• It can be easily used with the Python library imbalanced-learn (imblearn).

Synthetic Minority Over-Sampling Technique (SMOTE)

SMOTE is a data-level resampling technique that generates synthetic (artificial) samples for the minority class. Instead of simply duplicating existing examples, it creates new data points by interpolating between existing ones. This approach allows the model to learn broader patterns and reduces the risk of overfitting to repeated samples.

Working:

1. Identify the Minority Class: The process begins by detecting which class (or classes) have significantly fewer samples compared to others.
2. Find Nearest Neighbors: For each sample in the minority class, SMOTE locates its k nearest neighbours (based on distance in the feature space). The value of k is a user-defined parameter that controls how many neighbours are considered.
3. Generate Synthetic Samples: A random neighbour from these k nearest points is chosen and a new synthetic instance is created along the line segment connecting the original sample and the chosen neighbour. This ensures the new points are realistic yet distinct.
4. Control Oversampling Amount: The number of synthetic samples to be generated is determined by an oversampling ratio, which is chosen so that both classes reach a similar size or desired balance.
5. Repeat for All Minority Samples: Steps 2 to 4 are repeated for all minority class examples to produce enough synthetic data for balancing.
6. Form the Final Balanced Dataset: After generating these synthetic examples, the dataset becomes more balanced, helping machine learning models train more effectively and fairly across all classes.

Implementation

import matplotlib.pyplot as plt
import pandas as pd
from imblearn.over_sampling import SMOTE

data = pd.read_csv('diabetes.csv')
X = data.drop("Outcome", axis=1)
y = data["Outcome"]

count_class = y.value_counts()
plt.bar(count_class.index, count_class.values)
plt.xlabel('Class')
plt.ylabel('Count')
plt.title('Class Distribution (Before SMOTE)')
plt.xticks(count_class.index, ['Class 0', 'Class 1'])
plt.show()

smote = SMOTE(sampling_strategy='minority')
X_res, y_res = smote.fit_resample(X, y)
print("After SMOTE:\n", y_res.value_counts())

Output:

Dataset used can be downloaded from here.

Variants of SMOTE

SMOTE effectively addresses data imbalance by generating synthetic samples, enriching the minority class and refining decision boundaries. Despite its benefits, SMOTE's computational demands can escalate with larger datasets and high-dimensional feature spaces. To enhance SMOTE's capability to handle various data scenarios, several extensions have been developed:

1. ADASYN (Adaptive Synthetic Sampling)

ADASYN stands for Adaptive Synthetic Sampling. It is an improved version of SMOTE that automatically focuses more on minority samples that are difficult to learn. Instead of generating synthetic samples uniformly, ADASYN creates more new samples for minority points that are near the decision boundary where the model usually makes more mistakes.

Working:

1. Calculate the level of difficulty for each minority sample. Here samples surrounded by majority samples are considered harder to learn.
2. Assign higher weights to difficult samples so that more synthetic examples are created around them.
3. Generate synthetic samples by interpolating between each difficult sample and its nearest minority neighbors.
4. The final dataset has more new samples near the boundary, improving the model’s ability to classify challenging regions.

Implementation:

from imblearn.over_sampling import ADASYN

adasyn = ADASYN(sampling_strategy='minority')
X_res, y_res = adasyn.fit_resample(X, y)
print(y_res.value_counts())

Output:

2. Borderline SMOTE

Borderline SMOTE is a modified version of SMOTE that focuses only on minority samples that lie near the boundary between classes. These are the samples most likely to be misclassified, so generating synthetic samples around them helps strengthen the classifier’s performance near decision boundaries.

Working:

1. Identify minority samples that have many majority samples as their nearest neighbors and these are called borderline samples.
2. Generate synthetic samples only around these borderline points, avoiding areas deep inside the majority class.
3. This keeps the generated data clean and helps the model learn class boundaries more accurately.

Implementation:

from imblearn.over_sampling import BorderlineSMOTE

blsmote = BorderlineSMOTE(sampling_strategy='minority', kind='borderline-1')
X_res, y_res = blsmote.fit_resample(X, y)
print(y_res.value_counts())

Output:

3. SMOTE-ENN (Edited Nearest Neighbors)

SMOTE-ENN combines two techniques i.e SMOTE for oversampling and Edited Nearest Neighbors (ENN) for cleaning. First, SMOTE generates synthetic data to balance the dataset. Then, ENN removes noisy or misclassified points from both classes to make the dataset cleaner and more reliable.

Working:

1. Apply SMOTE to oversample the minority class.
2. For each sample, look at its nearest neighbors.
3. If most neighbors belong to a different class, remove that sample beacuse it’s likely a noise.
4. The result is a balanced and denoised dataset that helps improve model performance and generalization.

Implementation:

from imblearn.combine import SMOTEENN
smote_enn = SMOTEENN()
X_resampled, y_resampled = smote_enn.fit_resample(x, y)
y_resampled.value_counts()

Output:

4. SMOTE-TOMEK (Hybrid Method)

SMOTE-TOMEK is a hybrid resampling technique that combines SMOTE and Tomek Links. After oversampling with SMOTE, Tomek Links are identified and removed to eliminate overlapping or borderline points between classes.

Working:

1. SMOTE first oversamples the minority class by creating synthetic examples.
2. Find Tomek Links as these are pairs of samples from opposite classes that are each other’s nearest neighbors.
3. Remove those pairs, as they often lie in overlapping regions that confuse the model.
4. The final dataset becomes balanced and cleaner, improving the separation between classes.

Implementation:

from imblearn.combine import SMOTETomek
smt = SMOTETomek(sampling_strategy='auto')
X_resampled, y_resampled = smt.fit_resample(x, y)
y_resampled.value_counts()

Output:

5. SMOTE-NC (Nominal Continuous)

SMOTE-NC (Synthetic Minority Over-sampling Technique for Nominal and Continuous features) is a version of SMOTE designed for datasets that contain both numerical and categorical variables. Traditional SMOTE works by interpolating between numeric features, but it fails when applied directly to categorical data because we can’t interpolate between category labels. SMOTE-NC fixes this by treating categorical and continuous features differently during the generation of synthetic samples.

Working:

1. Identify which features in the dataset are categorical and which are continuous.
2. For continuous features, generate synthetic samples using interpolation just like standard SMOTE.
3. For categorical features, assign the most frequent category among the nearest neighbors instead of interpolating.
4. Combine the newly generated synthetic samples to create a balanced dataset that respects both numeric and categorical data integrity.

Note: If our dataset doesn’t include categorical features (like the diabetes dataset), use standard SMOTE instead of SMOTE-NC.

Implementation:

import numpy as np
from sklearn.datasets import make_classification
from imblearn.over_sampling import SMOTENC
from collections import Counter

X, y = make_classification(n_classes=2, class_sep=2,
                           weights=[0.1, 0.9], n_informative=3, n_redundant=1,
                           n_features=5, n_clusters_per_class=1, n_samples=100, random_state=42)

categorical_features = [0, 3]
print("Before SMOTE-NC:", Counter(y))
smote_nc = SMOTENC(categorical_features=categorical_features, random_state=42)
X_res, y_res = smote_nc.fit_resample(X, y)

print("After SMOTE-NC:", Counter(y_res))

Output:

When to Use Each SMOTE Variant

Let's discuss when to use each variant: