Intra Fusion Based CNN Technique for MRI
Multimodal Brain Tumor Classification and
Segmentation
Vasanthi Ravindran1, Kalaiselvi Thiruvengadam2, Anitha Thiyagarajan3
Department of Computer Science and Applications, The Gandhigram Rural Institute (Deemed to be
University), Gandhigram, Tamil Nadu, India1,2,3
Abstract. The proposed work segment the tumor portion with substructure from MRI
Multimodal brain tumor images using image fusion techniques. The preprocessing work
is done by using Piece-wise linear transformation, to enhance the tumor region. The
proposed work classify the brain tumor image as tumor or non-tumor by convolutional
neural network (CNN) model, then extracts the whole tumor portion by largest connected
component (LCC) and finally segments the substructures. The segmented substructure of
tumor portion is validated with ground truth in qualitative and quantitative analysis. The
experiments are done using BraTS datasets and performance metrics such as structural
similarity index measure (SSIM), accuracy, dice coefficient (DC), and peak signal to
noise ratio (PSNR). This metrics are used to validate the shape of the tumor portion. The
metrics gives better results for the proposed work.
Keywords: Deep Learning, Convolutional Neural Network, Image Fusion, Piece - wise
linear transformation, MRI, Brain Tumor, Substructure Segmentation,
Morphological Operation.
1 Introduction
In medical diagnosis, medical image plays a crucial role to analysis the defects in the human
body. The most popular imaging techniques are magnetic resonance imaging (MRI), Computed
tomography (CT) and X-rays[1]. In brain tumor analysis, MRI is the most admired technique
for segmentation and detection of tumor.
The unwanted growths of cells are formed in the brain are tends to brain tumor. Primary and
secondary are two categories [2]. The MRI brain tissues are gray matter (GM), white matter
(WM), cerebrospinal fluid (CSF). Flair, T1,T2 and T1c are the four modalities produced by MRI
as shown in Figure 1.
IACIDS 2023, November 23-25, Lavasa, India
Copyright © 2024 EAI
DOI 10.4108/eai.23-11-2023.2343286
Fig 1. MRI Multimodality of Brain Tumor image a) Flair b) T1 c) T1c d) T2
Tumor portion in these four modalities are emerge with different intensities. To combine these
intensities of four modalities and provide more informative MRI brain tumor image, image
fusion is introduced in this process. The method of combining different images that will produce
single image with good quality is referred as image fusion [3].
Two domains are performed in image fusion. One is spatial and another one is frequency
domain. Thus, in medical diagnosis for brain tumor, MRI produces four modalities with
different information. It leads to make the treatment process more difficult. To overcome this
medical image fusion is applied, which will merge the different modalities of the MRI medical
imaging to give an output fused medical image.
Now a days, in the field of AI, deep learning (DL) is a recent developments of research, helps
to analyze the brain cancer, its causes[5]. In DL, various segmentation and classification
techniques are available, among that CNN is the most flexible architecture to classify the images
and will produce the accurate result.
In the proposed method, preprocessing work is done with four modalities of MRI brain tumor
images are merged in the combination of Flair + T1c, Flair+T2+T1c and T1+T2+T1c+Flair
using image intra fusion rule Then the classification of tumor and non tumor process is
performed by CNN architecture with good accuracy. After that, the tumor portion and its
substructures are segmented using morphological operations. Then the ground truth image is
compared with output fused segmented image using evaluation metrics are SSIM, PSNR and
dice - co-efficient (DC). The intra fusion technique proposed work produce better results both
in qualitative and quantitative analysis.
This paper structured as follows. Literature survey based on brain tumor segmentation
techniques are described in second section. Methodology expressed in third section. Materials
and metrics are discussed in fourth section. Performance evaluation and results expressed in
section five. Finally, conclusion of the work and future enhancement are expressed in section
six.
2 Literature Review
Kaur et.al, [11] proposed a new method using multimodal image fusion techniques for brain
tumor detection using. Here they used the modalities are MRI and Flurodeoxyglucose images.
In the proposed work, pixel level image fusion techniques were used. Here they described that
the PCA techniques produced the result with blurred edges and result obtained from MSVD was
blurred, the content information also less. When compared to other techniques proposed method
was give good edge details and content information. The evaluation parameters was used for
evaluation are , structural similarity index metric, standard deviation (SD with 0.2), entropy (EN
with 4), Mutual Information, PSNR with 36.651. Thus the work gives good result when
compared to PCA.
Nandhini et.al [3], proposed a method for tumor segmentation. Here they used positron emission
tomography (PET) and MRI images of brain tumor In this paper, they illustrated that the PET
image gives functional information of the brain and the MRI produced the brain tissues. The
novel method produces the result to locate the tumor portion for medical diagnosis. The
experiment was done using PET and MRI images in 256 X 256 sizes. Datasets were taken from
med.harvard.edu database. Evaluation metrics SSIM, MI were used and produced the results
0.8551 and 2.8059 respectively.
Chandrashekar et.al [6] proposed a new using non – subsampled contourlet transform (NSCT)
and sparse representation (SR). Modalities are taken here was CT and MRI brain images. They
proposed a technique for fusion with NSCT and SR. The proposed result of fused CT and MRI
images gives better quality. The experiments were done using radiopedia brain tumor datasets.
Performance evaluation metrics used here are standard deviation, Entropy, phase congruency,
Execution Time, PSNR, universal image quality index. The authors conclude that the proposed
technique produce better and improved result with 7.23%, *%, 0.01% and 6.9% for PSNR,
SSIM, PC and UIQI respectively.
3 Methodology
The proposed intra fusion segmentation of multimodal MRI brain tumor region with
substructure is implemented by the following steps.
Step 1:
• Input the Flair image which is one of the multimodal MRI brain tumor image, process with
Piece - Wise Linear Transformation and produce contrast brain tumor image.
• Contrast brain tumor Flair image is combined with other modalities T1,T1c,T2
• Fusion rule is applied to the combinations Flair+T1c, Flair+T1+T2+T1c, Flair+T2+T1c
• Among the three combinaions, Flair+T2+T1c selected. It gives better fused image with good
contrast of tumor portion
• The fused images processed with CNN model to classify the tumor and non-tumor images.
• After classification, whole tumor extracted from the MRI brain tumor image using largest
connected componentes (LCC).
• From the resultant MRI fused brain tumor image , substructure of tumor portion such as
Edema, Necrosis and Enhancing Tumor are segmented using morphological operations.
• Output is fused segmented gray image with substructure
The work started with MRI multimodal brain tumor images Flair, T1, T1c, and T2, are appear
with different intensities and different structures of tumor region as shown in figure 4. Pre-
processing work is done with Flair image processed by Piece – Wise Linear Transformation to
increases the contrast of tumor portion and distinguish from other tissue as shown in figure 2.
Then the enhanced flair image and other modalities are fused by Pixel level image fusion rule
to produce a high quality and enhanced structure of tumor portion. These modalities are merged
based on the combinations such Flair+T1c, Flair+T1+T2+T1c, Flair+T2+T1c. Here
Flair+T2+T1c produce the grayscale output fused brain tumor image with good structure of
tumor region with substructure as shown in Figure 3(a). Therefore Flair+T2+T1c combination
is taken for post-processing. The fused images are processed with CNN to classify the tumor or
non-tumor image. In CNN model five layers are used, each layer consist of convolutional layer,
max pooling layer with ReLu activation function and the model end with dropout and fully
connected layer. Compilation of CNN is done by Adam optimizer. This model produces better
accuracy for the proposed method. If the output of CNN model for MRI fused image is tumor,
then binary threshold transformation is applied to segregate the whole tumor portion from other
pixel of the image as displayed in Figure 3(b). Then the threshold image is processed with LCC
to extract the tumor region, is shown in Figure 3(c), then it produce the whole tumor region is
displayed in Figure 3(d). Based on the morphological operations, tumor region substructure are
segmented as Enhancing Tumor, Edema, and Necrosis are presented in Figure 4 (a),(b)and (c) .
Figure 2. Flair with Piece - Wise Linear Transformation
Fig 3. (a) Fused Output Image (b) Threshold Binary Image (c) Segmented Tumor
Portion (d) Whole Tumor region
Fig 4. (a) Enhancing Tumor (b) Edema (c) Necrotics (d) Ground Truth Image
Finally, similarity measures are done by the metrics DC,SSIM and PSNR between output fused
segmented image and ground truth image. Proposed method produce good results in the
quantitative analysis when compared to other existing methods.
4 Material and Metrics
BraTS publicly available dataset is used in this experiment [4]. Dataset contains11 volumes,
Flair, T1, T1c, and T2 of each modality have 135 images in these volumes. The preprocessing
work intra fusion method produced the fused images of each volume is 4890 images. The
experiments are done in the device with configuration: Intel(R) Core(TM), 64 bit, Windows 10,
i5Processor, 8 GB RAM, Python3.10. The evaluation metrics PSNR, SSIM and DC, are used
here for evaluation and results are shown in section five. CNN model performance is measured
by accuracy.
5 Results and Discussions
The preprocessing work was done with pixel level intra fusion method, using MRI
multimodality brain tumor images. These images are fused based on the combinations Flair
+T1c, Flair + T1c +T2 and Flair+T2+T1c+T1. To evaluate the CNN model, accuracy metric is
used for each epochs. In the experiment, CNN model implemented with 50 epochs, the model
give better accuracy upto 90% - 98.9 % .
The evaluation metrics PSNR, SSIM and DC are calculated for each volume. The proposed
work compared with existing method listed in Table 1. This expressed that the proposed intra
fusion brain tumor classification gives better accuracy result when compared to other CNN
based image fusion techniques.
Table 1. Proposed Method Compared with Existing Method
Existing Methods Accuracy
Fuzzy C Means with Gaussian 96.5
mixture model – CNN [7]
CNN [8] 92
Transfer Learning based DCNN 98.93
Model [9]
Image fusion with CNN[10] 98.18
Proposed Intra fusion with CNN 98.97
classification
6 Conclusion
In this work multimodality of MRI brain tumor images are fused using pixel level fusion rule. The
output of this preprocessing work produce the fused image is implemented with the CNN model
with five layers and dropout to categorize the image as tumor and non-tumor images of MRI.
Finally, in the classified brain tumor image tumor portion is extracted and tumor regions are
segmented using morphological operation. Thus the experiment is analyzed by the metrics such
as DC, PSNR and SSIM. CNN model evaluated by the validation accuracy, it gives better accuracy
result with 98.9%. In future, there is intent to segment the tumor regions using other deep learning
techniques.
References
[1] Kalaiselvi, T., & Sriramakrishnan, P. Knowledge Based Brain Tumor Segmentation using Local
Maxima and Local Minima.
[2] Kalaiselvi, T. (2011). Brain portion extraction and brain abnormality detection from magnetic
resonance imaging of human head scans. Pallavi Publication.
[3] Nandhini Abirami, R., Durai Raj Vincent, P. M., Srinivasan, K., Manic, K. S., & Chang, C. Y.
(2022). Multimodal medical image fusion of positron emission tomography and magnetic resonance
imaging using generative adversarial networks. Behavioural Neurology, 2022.
[4] Sriramakrishnan, P., Kalaiselvi, T., Padmapriya, S. T., Ramkumar, S., & Ponmozhi, K. Online Brain
Image Repositories for Brain Disease Detection. In December 2019 International Journal of Innovative
Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075, Volume-9 Issue-2S2.
[5] Zeineldin, R. A., Karar, M. E., Elshaer, Z., Coburger, J., Wirtz, C. R., Burgert, O., & Mathis-Ullrich,
F. (2022). Explainability of deep neural networks for MRI analysis of brain tumors. International
journal of computer assisted radiology and surgery, 17(9), 1673-1683.
[6] Chandrashekar, L., & Sreedevi, A. (2018). A hybrid multimodal medical image fusion technique
for CT and MRI brain images. International Journal of Computer Vision and Image Processing
(IJCVIP), 8(3), 1-15.
[7] Wang, K., Zheng, M., Wei, H., Qi, G., & Li, Y. (2020). Multi-modality medical image fusion using
convolutional neural network and contrast pyramid. Sensors, 20(8), 2169.
[8] Vijendran, A. S., & Ramasamy, K. (2023). Optimal segmentation and fusion of multi-modal brain
images using clustering based deep learning algorithm. Measurement: Sensors, 27, 100691.
[9] Malla, P. P., Sahu, S., & Alutaibi, A. I. (2023). Classification of Tumor in Brain MR Images Using
Deep Convolutional Neural Network and Global Average Pooling. Processes, 11(3), 679.
[10] Mohamed, N. A., & Al-Tamimi, M. S. H. (2020). Image fusion using a convolutional neural
network. Solid State Technol, 63(6).
[11] Kaur, H., Koundal, D., Kadyan, V., Kaur, N., & Polat, K. (2021). Automated Multimodal image
fusion for brain tumor detection. Journal of Artificial Intelligence and Systems, 3(1), 68-82.
