Deep multi-view feature fusion with data augmentation for improved diabetic retinopathy classification

2.1 Traditional machine learning methods

Machine learning algorithms have been widely employed to assist in the detection, diagnosis, and risk stratification of DR. These algorithms can analyze retinal images, extract specific features, or lesions associated with DR, and classify the condition’s severity. By automating these tasks, machine learning models enhance the screening process, mitigate human error, and increase overall efficiency.

A variety of machine learning methods have been explored in the literature to address this problem. For instance, Biran et al. [4] utilized the Star database, comprising 33 retinal images, to classify the images into three categories: normal, non-proliferative diabetic retinopathy, and proliferative diabetic retinopathy. They employed Gabor filters for the detection of exudates and circular Hough transform for hemorrhage region identification, followed by classification using a support vector machine (SVM) with a non-linear kernel function. Similarly, Roy et al. [5] used fundus images from the DIARET, DRIVE, and MESSIDOR datasets, employing Fuzzy C-means and convex hull techniques for feature extraction, with SVM as the classifier.

In another study [6], microaneurysm (MA) detectability was analyzed using small 25 × 25 pixel patches extracted from DIARETDB1 fundus images. The raw pixel intensities of these patches were used as direct inputs to various classifiers, including RF, neural networks (NNs), and SVM. The SVM model, employing a radial basis function kernel, achieved a high accuracy of 98.5% in terms of the receiver-operating characteristic and 92.6% for the area under the curve. Furthermore, in the study by Emon et al. [7], a comprehensive set of machine learning classifiers including naïve Bayes, sequential minimal optimization, logistic regression, stochastic gradient descent, bagging classifier, J48 classifier, decision tree classifier, and RF was applied to the Diabetes Retinopathy Debrecen dataset from the University of California, Irvine (UCI). The retinal image features are extracted using principal component analysis. Moreover, Safitri and Juniati [8] proposed a method to identify DR using fractal analysis and the k-nearest neighbor (kNN) algorithm, where the authors segmented images, calculated the fractal dimension of the segmented regions, and classified the images using kNN. This method was evaluated using cross-validation, with the optimal value of k determined to achieve the highest accuracy.

Despite the effectiveness of machine learning in DR classification, several challenges persist. These include the need for extensive feature engineering, the potential for model overfitting, and the difficulty in capturing complex patterns inherent in retinal images. As a consequence, deep learning has emerged as a compelling alternative, offering superior performance in handling intricate tasks and learning hierarchical representations directly from raw data. Unlike traditional machine learning approaches, deep learning models eliminate the need for manual feature extraction by automatically learning relevant features, which reduces human effort and mitigates potential biases. Additionally, advancements in hardware and the availability of large-scale datasets have further facilitated the adoption of deep learning models, allowing for end-to-end training and deployment at scale. While deep learning has demonstrated remarkable success in fields such as image recognition, speech processing, and natural language understanding, traditional machine learning remains valuable in scenarios with limited data availability or where model interpretability is a critical requirement.

2.2 Deep learning-based methods

Deep learning has dramatically transformed computer vision. Its capacity to autonomously learn intricate features from data, adapt to diverse tasks, scale effectively, and generalize across different domains has led to the development of highly accurate, efficient, and adaptable models. These models have set new benchmarks in the medical image analysis field, offering substantial improvements over traditional methods.

The classification of DR using deep learning approaches can be broadly categorized into two primary strategies: learning from scratch and transfer learning. Transfer learning, in turn, can be implemented in two distinct modes: utilizing a pre-trained model as a feature extractor and fine-tuning a pre-trained model on the specific dataset of interest. The following sections delve into the relevant literature within each of these categories.

2.3 Ensemble learning

Ensemble learning is the training of multiple classifiers for a single task. Recently, ensemble learning methods based on stacking, bagging, or boosting have proved that learning multiple classifiers outperforms a single classifier in almost all machine learning tasks. Motivated by the effectiveness of ensemble learning, Dondeti et al. [37] developed an ensemble model using the Adaboost classifier. Three individual models based on Inception-v3, ResNet152, and Inception-Resnet-v2 architectures were trained on a private dataset in collaboration with the Beijing Tongren Eye Centre. The obtained results showed that the ensemble model outperformed the individual models. Likewise, Jiang et al. [38] trained an ensemble of five deep CNN models (Resnet50, Inceptionv3, Xception, Dense121, and Dense169) using the Kaggle dataset to improve the classification of different stages of DR. The final output is obtained by averaging the class prediction obtained by the CNN models.

2.4 Gaps and contributions

The extensive review of existing literature highlights the substantial progress made in applying DNN architectures to DR classification. Despite these advancements, a critical gap remains unaddressed: the exploration of non-NN-based deep learning techniques for this task. While the prevailing research predominantly focuses on NN models, there is a conspicuous absence of studies that investigate alternative deep learning frameworks. These NN-based approaches often rely heavily on extensive hyper-parameter tuning, which can be computationally expensive and time-consuming, while also presenting challenges related to model interpretability and generalizability.

In this study, we present a deep learning-based classification framework for DR severity grading, exploring the use of non-NN-based models. Our method is designed to address the limitations identified in existing approaches, as presented in the following sections of this article.

3.1 Feature extraction

The accuracy and reliability of DR classification are largely contingent upon the ability to effectively detect and represent key retinal biomarkers. These crucial indicators, which include MAs, hemorrhages, and exudates, are instrumental in determining both the presence and progression of the disease. CNNs have become a powerful tool, offering state-of-the-art performance in generating rich, discriminative image representation. However, the limited availability of labeled retinal images for DR classification poses a significant challenge, making it difficult to train CNNs from scratch. To address this, our proposed method leverages transfer learning, utilizing pre-trained CNN models as robust feature extractors. Transfer learning capitalizes on the knowledge embedded in models pre-trained on large, diverse datasets, enabling these models to extract meaningful features even when applied to new, smaller datasets.

As shown in Figure 1, the first critical step in our proposed method is the extraction of features from the retinal images. We employ two highly effective pre-trained CNN models, ResNet-50 [39] and EfficientNet [40], as the backbone for this task. ResNet-50, with its deep architecture and residual connections, is renowned for its ability to learn hierarchical features across varying levels of abstraction, making it particularly adept at capturing complex patterns within retinal images. EfficientNet, on the other hand, utilizes a compound scaling method that balances network depth, width, and resolution, offering a highly efficient yet powerful feature extraction process.

The output of this step is a 3D tensor, where N represents the number of input retinal images. Each image is represented by a matrix ( M × L ), where M is the number of the used pre-trained CNN models, and L is the size of the extracted embedding features from each CNN model. By integrating features from multiple CNN models, this step not only enhances the representational power of the extracted features but also ensures that the extracted features are well suited for subsequent analysis, setting the foundation for improved DR classification.

3.2 Feature reduction

Following feature extraction, the dimensionality of the resulting feature space must be reduced to improve computational efficiency and focus on the most informative aspects of the data. We employ MWPCA [41] for this purpose, chosen for its effectiveness in handling multidimensional data.

MWPCA is particularly adept at minimizing noise and eliminating irrelevant variations within the feature space, which enhances the representation of retinal images. By preserving the most significant features while reducing dimensionality, MWPCA ensures that the retained features are both relevant and compact. The output of this step is a reduced 3D tensor, represented by a matrix of size ( M × L ′ ) , where L ′ is significantly smaller than the original feature size L (i.e., L ′ ≪ L ).

3.3 Feature fusion

To enhance the discriminative capability of the extracted features, we employ TEDA [41]. As illustrated in Figure 1, TEDA processes the tensor output generated by MWPCA by applying projection matrices through contraction operations along each M × L ′ feature matrix.

This operation compresses the high-dimensional tensor into a more compact form while preserving the most critical discriminative information. The outcome of this transformation is a set of one-dimensional feature vectors for each input image x i ( i ∈ { 1 … N } ) , with a significantly reduced dimensionality L ″ , such that L ″ ≪ l ′ × M . This compressed representation retains the essential characteristics of the retinal images as well as significantly reduces the computational costs, thereby facilitating more efficient and accurate DR classification.

3.4 Data augmentation

Addressing class imbalance is critical in machine learning, particularly in DR classification, where minority classes are often underrepresented, leading to biased decision boundaries and reduced model performance. To mitigate this issue, we employ the SMOTE [42], a widely recognized approach for rebalancing imbalanced datasets.

SMOTE generates synthetic samples for the minority class by interpolating between existing data points and their KNNs. By creating new data points along the lines connecting these neighbors, SMOTE effectively expands the decision boundary of the minority class, enabling the classifier to learn more robustly from the underrepresented instances. This method enhances the model’s ability to detect subtle patterns specific to the minority class.

Also, one of the key advantages of SMOTE is its ability to improve the quality of the dataset without requiring additional manual labeling, which is often resource-intensive. The augmentation process results in an expanded feature set N ′ with N ′ ≫ N , significantly boosting the representational capacity of the minority class and, consequently, the overall classification performance.

3.5 Feature classification

In this step, we aim to train a DRF classifier [43]. The motivation behind using DRF is that it is a deep architecture based on a decision tree as a base unit instead of the artificial neural. That makes the RDF classifier much easier to train than other DNNs with interpretable results. Furthermore, DRF does not require extensive hyperparameter tuning, which is one of the most challenging in the majority of our related works. Additionally, the DRF can work effectively even with small training data. On the other hand, the DRF is based on an ensemble learning technique, which is proven to outperform the use of a single classifier for various ranges of classification tasks [26,43–48].

Inspired by the deep network structure of the DNN, the RDF is based on in-model feature processing. The DRF structure uses layer-by-layer stacking to pass the information within each layer from the input layer to the last layer that outputs the final predicted class. Each level in the DRF architecture obtains the input received by its previous level and outputs the processing results to the next level.

Each level is an ensemble of forest classifiers, and each forest is an ensemble of decision trees. At a cascade level, each forest generates a class distribution vector. Then, it concatenates all the generated class distribution vectors from all the level’s forests with the original input feature vector. The obtained vector is passed to the next level till the final level. The final cascade level takes as input the class distribution vectors from the previous layer. It outputs the final predicted class by averaging the received vectors and taking the class that has the highest prediction probability.

The number of cascade levels is determined automatically during the training phase based on the early stopping technique: after adding a new level, the training process will be discontinued if the classification performance does not increase sufficiently.

4.1 Experimental setup

To ensure robust and unbiased assessment, we employed a 5-fold cross-validation evaluation protocol in our experiments. The dataset was divided into five equally sized subsets, with class distributions preserved to reflect the original dataset’s balance. In each iteration, one subset served as the test set, while the remaining four subsets were used for training. This process was repeated five times, and the final performance was reported as the average across all folds.

In our study, we employed two pre-trained CNN models, namely, ResNet-50 [39] and EfficientNet [40], to extract deep retinal features. Specifically, we extracted the feature vectors from the last fully connected layer of these pre-trained models. For each input image, we obtained the feature vectors from both models, resulting in a 2D matrix of size 2,048 × 2 . Subsequently, we collected the matrices of all the training samples and formed a third-order tensor. The purpose of this tensor was to estimate the projection matrices for subspace projection. To ensure convergence during the tensor projection process, we empirically set the maximum number of iterations to 16. Additionally, we set the convergence threshold for the MWPCA algorithm to 106, as employed in the study of Ouamane et al. [41].

For the classification phase, we employed the default parameters recommended in the original DRF paper [43], which have been shown to provide robust performance across a variety of tasks without extensive tuning [26,43–48]. These settings include using two RFs and two completely RFs for each level in the cascade structure, with each forest containing 500 trees. These trees are grown until they reach a pure leaf or a specified maximum depth of 100. The number of levels in the cascade is determined automatically based on performance via the early stopping mechanism; training halts when no further improvement is observed in the validation set, ensuring robust classification performance without requiring extensive hyperparameter tuning.

In our baseline comparison experiments, we used Scikit-Learn’s [49,50]^[1] implementation of SVM and RF classifiers with their default parameter settings. For the RF model, we set the number of trees to 100, maximum depth to 30, and the minimum samples per leaf to 2. The SVM was trained using a RBF kernel with a regularization parameter (C) of 1 and a gamma value of 0.01. This approach was chosen to ensure a fair and consistent comparison across methods under standardized conditions, as these defaults are widely accepted as reasonable baselines for machine learning tasks. All the experiments were run on a machine with an Intel Core i7-7700 CPU (2.80 GHz) and 16GB RAM.

4.2 APTOS dataset

In this study, we utilized the APTOS 2019 dataset, a publicly available resource from the APTOS Blindness Detection competition on Kaggle. Curated by the Asia Pacific Tele-Ophthalmology Society, this dataset comprises 3,662 high-resolution retinal images, captured using fundus photography at the Aravind Eye Hospital in India. The dataset is instrumental in advancing research on DR detection, providing a diverse set of real-world retinal images essential for developing robust diagnostic models.

The dataset is categorized into five distinct classes: no DR (Class 0), mild DR (Class 1), moderate DR (Class 2), severe DR (Class 3), and proliferative DR (Class 4). The class distribution reveals a significant imbalance: 49% of the images are classified as no DR, while only 5% represent severe DR, and 8% are categorized as proliferative DR. Specifically, out of the 3,662 images, 1,805 are labeled as normal (no DR), and the remaining 1,857 are distributed across the four DR classes. Figure 2 provides a representative sample of images from each class, highlighting the varying severity levels of DR.

Figure 2

Sample images from the five Kaggle APTOS 2019 dataset classes: (a) no DR, (b) mild, (c) moderate, (d), proliferate, and (e) severe.

4.3 Evaluation metrics

To evaluate the classification performance of our proposition, we use the well-known evaluation metrics as follows.

Accuracy is a commonly used metric to evaluate the performance of a classification model. The accuracy measures the proportion of correct predictions out of the total number of predictions made. The accuracy is calculated as follows:

Precision is a metric used to measure the proportion of correctly predicted positive instances (true positives) out of all instances that were predicted as positive (true positives and false positives). The precision is calculated as follows:

A high precision value indicates that the model has a low rate of false positives, meaning it is accurate when predicting positive instances. Conversely, a low precision value suggests a high rate of false positives, indicating that the model is prone to labeling negative instances as positive. Precision is particularly useful in situations where false positives are costly or have significant consequences. For example, in medical diagnostics, precision is valuable because it measures the ability of a model to accurately identify true positives (actual cases of disease) and avoid false positives (misdiagnosing healthy individuals as diseased).

Recall: (also called sensitivity and true-positive rate), the fraction of correctly classified positive observations over all the positive observations:

F1 score is a metric commonly used to evaluate the performance of a classification model, particularly in situations where class imbalance exists. It is a single value that combines precision and recall into a balanced measure of the model’s overall accuracy. The F1-score is calculated as the harmonic mean of precision and recall, providing a single value that takes into account both metrics. It is calculated using the following formula:

The F1 score ranges from 0 to 1, with 1 representing the best possible performance. A higher F1 score indicates a better balance between precision and recall, suggesting that the model is effective at both correctly identifying positive instances and avoiding false positives. Conversely, a lower F1-score suggests that the model may have issues with either precision or recall, or both.

Weighted averaged F1 score calculates the metric by assigning different weights to each class based on the number of samples in each class, making it useful to evaluate the model’s performance while considering class imbalances. As an example, the weighted averaged F1 score can be estimated as the following equation:

where N is the total number of samples, N i is the number of samples in class i , and F1 score i is the F1 score of class i .

Macro-averaged F1 score calculates the metric for each class independently and then computes the average of these individual class metrics, making it useful when assessing the model’s performance without considering class imbalances. As an example, the macro-averaged F1 score can be estimated as the following equation:

where N is the total number of classes and F 1- score i is the F1 score for class i .

4.4 Results

We conducted a comprehensive set of experiments on the APTOS dataset, following a standardized evaluation protocol. Although various classifiers were explored, we present the results from our proposed DRF classifier, which consistently outperformed conventional machine learning classifiers.

Our first set of experiments focused on evaluating the effectiveness of both single-view and multi-view deep feature representations (Table 1). The obtained results demonstrate that the fusion of deep features from ResNet50 (M1) and EfficientNet (M2) consistently outperformed individual feature sets across most DR severity classes. These results demonstrate the advantage of using complementary information from multiple deep learning models, improving the classifier ability to distinguish fine-grained changes in retinal images.

In the second phase of our experimental analysis, we evaluated the impact of integrating MWPCA for dimensionality reduction on the performance of our proposed method. The application of MWPCA resulted in a substantial enhancement across all evaluated metrics, as detailed in Table 2. This improvement can be primarily attributed to MWPCA’s ability to effectively reduce noise and eliminate irrelevant variations within the feature space. By focusing on the most informative feature, MWPCA enhances the model’s ability to generalize, thereby leading to more accurate predictions.

In the third set of experiments, we investigated the impact of incorporating TEDA as a feature fusion technique. The results, summarized in Table 3, demonstrate a significant improvement in classification accuracy across all evaluated metrics after the application of TEDA. The primary factor contributing to this enhancement is TEDA’s ability to increase the discriminative power of the features, effectively increasing the separability between different classes. Additionally, TEDA plays a crucial role in dimensionality reduction, which updates the feature space by efficiently combining multiple feature vector representations. This feature reduction and fusion strategy using TEDA method provides a more robust foundation for accurate classification.

We conducted a series of experiments to assess the effectiveness of integrating SMOTE as a data augmentation strategy. Our results, summarized in Table 4, demonstrate that SMOTE significantly enhances the model’s performance across all evaluation metrics. Notably, the F1-score experienced marked improvements of 2.54% for class 0 (no DR), 3.26% for class 1 (mild DR), 1.48% for class 2 (moderate DR), 1.4% for class 3 (severe DR), and 0.8% for class 4 (proliferative DR). These gains underscore SMOTE’s effectiveness in addressing data imbalance, leading to a more robust and generalized classifier able to better distinguish between different severity levels of DR.

Next, we conducted a performance comparison between our proposed DRF classifier and traditional classifiers such as SVM and RF. The experimental results, summarized in Table 5, clearly demonstrate the superiority of the DRF classifier. The DRF achieved notable improvements in classification performance, with an increase in the F1 score ranging from 2.13 to 3.23% for the macro-averaged metrics, and from 1.38 to 2.2% for the weighted average metrics. This enhanced performance can be attributed to DRF’s ability to effectively manage small datasets while leveraging the complementary strengths of ensemble learning and deep learning. Specifically, DRF combines the robustness and generalization capabilities of RF with the layer-by-layer in-model feature processing power of deep learning networks, resulting in a more resilient and accurate classifier. For a more comprehensive understanding, the overall confusion matrix of our DRF-based classification approach is illustrated in Figure 3. This matrix provides a detailed view of the classifier performance across different DR severity classes, further validating the effectiveness of the proposed method.

Figure 3

Confusion matrix of the proposed method. Source: This figure has been created by the authors.

Finally, we conducted a comparative analysis to evaluate the effectiveness of our proposed method against recent related works. Specifically, we benchmarked our approach against traditional machine learning, deep learning, and ensemble learning-based methods. The results, summarized in Table 6, demonstrate that our method consistently outperforms existing approaches across multiple metrics. One of the significant advantages of our proposed method is its reduced dependency on large volumes of labeled training data, a common limitation in many DL-based methods. While most contemporary DL approaches rely heavily on NNs, our method leverages an ensemble of RF. This not only simplifies the training process but also enhances the model’s robustness, avoiding the complexities associated with DNN architectures, and benefits of the advantages of ensembled learning technique.

Overall, as illustrated in Table 6, our method achieved the highest accuracy, weighted-averaged F1, and macro-averaged F1 scores of 94.8, 94.8, and 92.4%, respectively. Notably, the performance gains range from 2.9 to 13.6% in accuracy, 2.8 to 13.4% in weighted-averaged F1 score, and 5.2 to 17.9% in macro-averaged F1 score. To further validate the effectiveness of our proposition, we performed a paired t -test to compare the performance of our proposed method against the related works methods across all train/test folds. Experimental results demonstrated that the differences were significant at ( p < 0.05 ), indicating that our approach consistently outperforms existing methods across different evaluation metrics, proving its effectiveness in DR classification.