Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Digital Object Identifier 10.1109/ACCESS.2020.2980266

Convolutional Neural Networks with

Intermediate Loss for 3D
Super-Resolution of CT and MRI Scans
MARIANA-IULIANA GEORGESCU1 , RADU TUDOR IONESCU1 , (MEMBER, IEEE), AND
arXiv:2001.01330v4 [cs.CV] 6 Apr 2023

NICOLAE VERGA2,3
1
University of Bucharest, Bucharest, Romania
2
“Carol Davila” University of Medicine and Pharmacy, Bucharest, Romania
3
Colt, ea Hospital, Bucharest, Romania
Corresponding author: Radu Tudor Ionescu (e-mail: [email protected]).
The research leading to these results has received funding from the EEA Grants 2014-2021, under Project contract no.
EEA-RO-NO-2018-0496.

ABSTRACT Computed Tomography (CT) scanners that are commonly-used in hospitals and medical
centers nowadays produce low-resolution images, e.g. one voxel in the image corresponds to at most one-
cubic millimeter of tissue. In order to accurately segment tumors and make treatment plans, radiologists and
oncologists need CT scans of higher resolution. The same problem appears in Magnetic Resonance Imaging
(MRI). In this paper, we propose an approach for the single-image super-resolution of 3D CT or MRI scans.
Our method is based on deep convolutional neural networks (CNNs) composed of 10 convolutional layers
and an intermediate upscaling layer that is placed after the first 6 convolutional layers. Our first CNN, which
increases the resolution on two axes (width and height), is followed by a second CNN, which increases the
resolution on the third axis (depth). Different from other methods, we compute the loss with respect to the
ground-truth high-resolution image right after the upscaling layer, in addition to computing the loss after
the last convolutional layer. The intermediate loss forces our network to produce a better output, closer
to the ground-truth. A widely-used approach to obtain sharp results is to add Gaussian blur using a fixed
standard deviation. In order to avoid overfitting to a fixed standard deviation, we apply Gaussian smoothing
with various standard deviations, unlike other approaches. We evaluate the proposed method in the context
of 2D and 3D super-resolution of CT and MRI scans from two databases, comparing it to related works
from the literature and baselines based on various interpolation schemes, using 2× and 4× scaling factors.
The empirical study shows that our approach attains superior results to all other methods. Moreover, our
subjective image quality assessment by human observers reveals that both doctors and regular annotators
chose our method in favor of Lanczos interpolation in 97.55% cases for an upscaling factor of 2× and in
96.69% cases for an upscaling factor of 4×. In order to allow others to reproduce our state-of-the-art results,
we provide our code as open source at https://github.com/lilygeorgescu/3d-super-res-cnn.

INDEX TERMS Convolutional neural networks, single-image super-resolution, CT images, MRI images,
medical image super-resolution.

I. INTRODUCTION so the 3D CT images are composed of volumetric pixels

(voxels) that usually correspond to one cubic millimeter (1 ×
EDICAL centers and hospitals around the globe are
M typically equipped with single-energy Computer To-
mography (CT) or Magnetic Resonance Imaging (MRI) scan-
1 × 1 mm3 ) of tissue. One of the main benefits of this non-
invasive scanning technique is that it allows doctors to see
if there are malignant tumors inside the body. Nevertheless,
ners that produce cross-sectional images (slices) of various doctors, and even machine learning systems [1], are not able
body parts. The resulting images are of low-resolution, since to accurately contour (segment) the tumor regions because
one pixel usually corresponds to at most one-millimeter piece of the low-resolution of CT or MRI scans. According to a
of tissue. The thickness of one slice is one millimeter at best,

VOLUME 8, 2020 1
 Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

FIGURE 1: Our method for 3D image super-resolution based on two subsequent fully convolutional neural networks. In the
first stage, the input volume is resized in two dimensions (width and height). In the second stage, the processed volume is
further resized in the third dimension (depth). Using a scale factor of 2×, an input volume of 256 × 256 × 64 components is
upsampled to 512 × 512 × 128 components (on all axes). Best viewed in color.

team of radiologists from Colt, ea Hospital in Bucharest, that the results and obtain sharper images, a common approach
provided a set of anonymized CT scans for our experiments, is to apply Gaussian smoothing on top of the input images,
the desired resolution is to have one voxel correspond to one using a fixed standard deviation. Different from other med-
cubic micrometer (a thousandth part of a cubic millimeter) of ical image super-resolution methods [3], [14], [17], we use
tissue. In other words, the goal is to increase the resolution of various standard deviations in order to avoid overfitting to
3D CT and MRI scans by a factor of 10× in each direction. a certain standard deviation and improve the generalization
The main motivation behind our work is to allow radiolo- capacity of our model.
gists and oncologists to accurately segment tumors and make We note that our model belongs to a class of deep neural
better treatment plans. In order to achieve the desired goal, networks known as fully convolutional neural networks. The
we propose a machine learning method that takes as input a main advantage of using such models, which do not include
3D image and increases the resolution of the input image by dense (fully-connected) layers, is that the input samples do
a factor of 2× or 4×, providing as output a high-resolution not have to be of the same size. This flexibilty enables a broad
3D image. To our knowledge, there are only a few previous range of applications such as image segmentation [20], object
works [1]–[18] that study the super-resolution of CT or MRI tracking [21], crowd detection [22], time series classification
images. Similar to some of these previous works [2]–[7], [23] and single-image super-resolution [3], [6], [16], [18].
[9], [11]–[18], we approach single-image super-resolution Different from other fully convolutional neural networks [3],
(SISR) of CT and MRI scans using deep convolutional neural [6], [16], [18], [20]–[23], our network is specifically designed
networks (CNNs). We propose a CNN architecture composed for SISR, having a custom architecture that includes an
of 10 convolutional layers and an intermediate sub-pixel upscaling layer [19] useful only for SISR, as well as a novel
convolutional (upscaling) layer [19] that is placed after the loss function.
first 6 convolutional layers. Different from related works [3], We conduct super-resolution experiments on two
[6], [16], [18] that use the sub-pixel convolutional layer of Shi databases of 3D CT and MRI images, one gathered from
et al. [19], we add 4 convolutional layers after the upscaling the Colt, ea Hospital (CH) and one that is publicly available
layer. In order to obtain 3D super-resolution, we employ two online, known as NAMIC1 . We compare our method with
CNNs with similar architectures, as illustrated in Figure 1. several interpolation baselines (nearest, bilinear, bicubic,
The first CNN increases the resolution on two axes (width Lanczos) and state-of-the-art methods [3], [13], [15], [17], in
and height), while the second CNN takes the output from terms of the peak signal-to-noise ratio (PSNR), the structural
the first CNN and further increases the resolution on the similarity index (SSIM) and the information fidelity criterion
third axis (depth). Different from related methods [3], [6], (IFC). We perform comparative experiments on both 2D
[16], [18], we compute the loss with respect to the ground- and 3D single-image super-resolution under commonly-used
truth high-resolution image right after the upscaling layer, upscaling factors, namely 2× and 4×. The empirical results
in addition to computing the loss after the last convolutional indicate that our approach is able to surpass all the other
layer. The intermediate loss forces our network to produce a
better output, closer to the ground-truth. In order to improve 1 Available at http://hdl.handle.net/1926/1687.

2 VOLUME 8, 2020
Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

methods included in the experiments. For example, on the it builds the HR image only at the very end. Other works,
NAMIC data set, we obtain a PSNR of 40.57 and an SSIM such as [29], proposed deeper architectures, focusing strictly
of 0.9835 for 3D super-resolution by a factor of 2×, while on accuracy. Indeed, Zhang et al. [29] presented one of the
Pham et al. [13] reported a PSNR of 38.28 and an SSIM deepest CNNs used for SR, composed of 400 layers. They
of 0.9781 in the same setting. Furthermore, we conduct a used a channel attention mechanism and residual blocks to
subjective image quality assessment by human observers, handle the depth of the network.
asking 6 doctors and 12 regular annotators to choose between For medical SISR, some researchers have focused on
the CT images produced by our method and those produced sparse representations [8], [10], while others on training
by Lanczos interpolation (the best interpolation method). convolutional neural networks [1]–[7], [9], [11], [12], [14]–
The annotators opted for our method in favor of Lanczos [16], [18].
interpolation in 97.55% cases for an upscaling factor of 2× The authors of [8] proposed a weakly-supervised joint
and in 96.69% cases for an upscaling factor of 4×. These convolutional sparse coding method to simultaneously solve
results indicate that our method is significantly better than the problems of super-resolution and cross-modal image
Lanczos interpolation. In order to allow further developments synthesis. In [10], the authors adopted a method based on
and results replication, we provide our code as open source compressed sensing and self-similarity constraint, obtaining
in a public repository2 . better results than [17] in terms of SSIM and PSNR.
To summarize, our contribution is threefold: Some works [1], [3], [5], [6], [9]–[11], [14]–[16], [18]
• We propose a novel CNN model for 3D super-resolution focused on 2D upsampling, i.e. on increasing the width and
of CT and MRI scans, which is based on an intermediate height of CT/MRI slices, while other works [2], [4], [8], [12]
loss added to the standard output loss and on smooth- focused on 3D upsampling, i.e. on increasing the resolution
ing the input using random standard deviations for the of full 3D CT/MRI scans on all three axes (width, height and
Gaussian blur. depth).
• We conduct a subjective image quality assessment by For 2D upsampling, some works [1], [5], [9], [14] used
human observers to determine the quality and the utility interpolated low resolution (ILR) images, while other works
of our super-resolution results, as in [15]. [3], [6], [16], [18] used the efficient sub-pixel convolutional
• We provide our code online for download, allowing our neural network (ESPCN) introduced in [19]. Similar to the
results to be easily replicated. latter approaches [3], [6], [16], [18], we employ the sub-pixel
We organize the rest of this paper as follows. We present convolutional layer of Shi et al. [19]. Different from these
related work in Section II. We describe our method in detail related works [3], [6], [16], [18], we add a convolutional
in Section III. We present experiments and results in Sec- block after the sub-pixel convolutional layer, in order to
tion IV. Finally, we draw our conclusions in Section V. enhance the HR output image. Furthermore, we propose a
novel loss function for our CNN model. Instead of computing
II. RELATED WORK the loss between the output image and the ground-truth high-
The purpose of SISR is to reconstruct a high-resolution (HR) resolution image, we also compute the loss between the inter-
image from its low-resolution (LR) counterpart. Before the mediate image given by the sub-pixel convolutional layer and
deep learning age, researchers have used exemplar-based or the high-resolution image. This forces our neural network
sparse coding methods for SISR. Exemplar-based methods to learn a better intermediate representation, increasing its
learn mapping functions from external LR and HR exemplar performance.
pairs [24]–[26]. Sparse coding methods [27] are representa- There are some works [2], [11], [15] that employed gener-
tive for external exemplar-based SR methods. For example, ative adversarial networks (GANs) [30] to upsample medical
the method of Yang et al. [27] builds a dictionary with LR images. Although our approach based on fully convolutional
patches and the corresponding HR patches. neural networks is less related to GAN-based SISR methods,
To our knowledge, the first work to present a deep learning we decided to include the approach of You et al. [15] in our
approach for SISR is [28]. Dong et al. [28] proposed a CNN experiments, as a recent and relevant baseline.
composed of 8 convolutional layers. The network was trained For 3D upsampling, Chen et al. [2] trained a CNN with 3D
in an end-to-end fashion, minimizing the reconstruction error convolutions and used a GAN–based loss function to produce
between the HR image and the output of the network. They sharper and more realistic images. In order to upsample a
used bicubic interpolation to resize the image, before giving 3D image, Du et al. [4] employed a deconvolutional layer
it as input to the network. Hence, the CNN takes a blurred HR composed of 3D filters to upsample the LR image, in an
image as input and learns how to make it sharper. Since the attempt to reduce the computational complexity. As [2], [4],
input is an HR image, this type of CNN is time consuming. [8], [12], we tackle the problem of 3D CT/MRI image super-
Therefore, Shi et al. [19] introduced a new method for up- resolution. However, instead of using inefficient 3D filters
sampling the image using the CNN activation maps produced to upsample the LR images in a single step, we propose
by the last layer. Their network is more efficient, because a more efficient two-stage approach that uses 2D filters.
Our approach employs a CNN to increase the resolution in
2 Available at https://github.com/lilygeorgescu/3d-super-res-cnn. width and height, and another CNN to further increase the
VOLUME 8, 2020 3
 Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

resolution depth-wise. the CNN used in the first stage resizes the image on two
Most SISR works [3], [14], [17], apply Gaussian smooth- axes, the CNN used in the second stage resizes the image
ing using a fixed standard deviation on the training images, on a single axis. Both CNNs share the same architecture,
thus training the models in more difficult conditions. How- the only difference being in the upsamling layer (the second
ever, we believe that using a fixed standard deviation can CNN upsamples in only one direction). At training time, our
harm the performance, as deep models tend to overfit to the CNNs operate on patches. However, since the architecture is
training data. Different from the standard methodology, each fully convolutional, the models can operate on entire slices at
time we apply smoothing on a training image, we chose a inference time, for efficiency reasons.
different standard deviation, randomly. This simple change We hereby note that 3D super-resolution is not equivalent
improves the generalization capacity of our model, yielding to 2D super-resolution in a slice-by-slice order. Our first
better performance at test time. CNN performs 2D super-resolution in a slice-by-slice order,
While many works focus only on the super-resolution task, increasing an input of size h×w×d to the size r·h×r·w×d,
the work of Sert et al. [1] is focused on the gain brought where r is the scale factor. Since we end up with the same
by the upsampled images in solving a different task. Indeed, number of slices (r), this is not enough. This is why we
the authors [1] obtained an improvement of 7.5% in the need the second CNN to further increase the image from
classification of segmented brain tumors when the upsampled r · h × r · w × d voxels to r · h × r · w × r · d voxels. The final
images were used. output of r · h × r · w × r · d voxels could also be obtained
We note that there is also some effort in designing and ob- by employing a single CNN with 3D convolutions. In most
taining CT scan results of higher resolution directly from CT of our convolutional layers, each 2D convolutional filter is
scanners. For example, X-ray microtomography (micro-CT) formed of 3 · 3 · 32 + 1 = 289 weights to be learned. As
[31], which is based on pixel sizes of the cross-sections in the we employ two networks for 3D super-resolution, we learn
micrometer range, has applications in medical imaging [32], 2 · 289 = 578 weights. For an equivalent model based on 3D
[33]. Another alternative to standard (single-energy) CT is convolutions, each 3D convolutional filter would be formed
dual-energy or multi-energy CT [34], [35]. Different from the of 3 · 3 · 3 · 32 + 1 = 865 weights. This analysis proves
expensive alternatives such as dual-energy CT and micro-CT, that our two CNNs put together have less weights than a
our approach to increasing the resolution of single-energy single 3D CNN. We thus conclude that our approach is more
CT images using a machine learning algorithm represents an efficient. We note that our approach is essentially based on
economical and accessible mode. decomposing the 3D convolutional filters in a product of two
2D convolutional filters. We note that the same principle is
III. METHOD applied in literature [40], [41] to build more efficient CNN
Our method for solving the 3D image super-resolution prob- models by decomposing 2D convolutional layers in two sub-
lem relies on deep neural networks. The universal approxi- sequent 1D convolutional layers that operate on independent
mation theorem [36], [37] states that neural networks with at dimensions.
least one hidden layer are universal function approximators. We further describe in detail the proposed CNN architec-
Hence, the hypothesis class represented by neural networks ture, loss function and data augmentation procedure.
is large enough to accommodate any hypothesis explaining
the data. This high model capacity seems to help deep neural A. ARCHITECTURE
networks in attaining state-of-the-art results in many domains Our architecture, depicted in Figure 2 and used for both
[38], including image super-resolution [28]. This is the main CNNs, is composed of 10 convolutional (conv) layers, each
reason behind our decision to use neural networks. Interest- followed by Rectified Liner Units (ReLU) [42] activations.
ingly, Dong et al. [28] show that deep convolutional neural We decided to use ReLU activations, as this represents the
networks for super-resolution are equivalent to previously- most popular choice of transfer function in current research
used sparse-coding methods [27]. However, the recent litera- based on deep learning. All convolutional layers contain
ture indicates that deep neural networks attain better results filters with a spatial support of 3×3. While older deep models
than handcrafted methods in practice [28], [39], mainly be- were based on larger filters, e.g. the AlexNet [43] architecture
cause the parameters (weights) are learned from data in an contains filters of 11 × 11, the recent trend is towards using
end-to-end fashion. Further specific decisions, such as the smaller filters, e.g. the ResNet [44] architecture does not
neural architecture or the loss function, are taken based on contain filters larger than 3 × 3.
empirical observations. Our 10 conv layers are divided into two blocks. The first
Our approach is divided into two stages, as illustrated in block, formed of the first 6 conv layers, starts with the input
Figure 1. In the first stage, we upsample the image on height of the neural network and ends just before the upscaling layer.
and width using a deep fully convolutional neural network. Each of the first 5 convolutional layers are formed of 32
Then, in the second stage, we further upsample the resulting filters. For the CNN used in the first stage, the number of
image on the depth axis using another fully convolutional filters in the sixth convolutional layer is equal to the square
neural network. Therefore, our complete method is designed of the scale factor, e.g. for a scale factor of 4× the number
for resizing the 3D input volume on all three axes. While of filters is 16. For the CNN used in the second stage, the
4 VOLUME 8, 2020
Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

FIGURE 2: Our convolutional neural network for super-resolution on two axes, height and width. The network is composed
of 10 convolutional layers and an upsampling (sub-pixel convolutional) layer. It takes as input low-resolution patches of 7 × 7
pixels and, for the r = 2 scale factor, it outputs high-resolution patches of 14 × 14 pixels. The convolutional layers are
represented by green arrows. The sub-pixel convolutional layer is represented by the red arrow. The long-skip and short-skip
connections are represented by blue arrows. Best viewed in color.

FIGURE 4: An example of low-resolution input activation

maps and the corresponding high-resolution output activation
map given by the sub-pixel convolutional layer for upscaling
on one axis. For a scaling factor of r = 2 in one direction, the
sub-pixel convolutional layer requires r = 2 activation maps
as input. Best viewed in color.
FIGURE 3: An example of low-resolution input activation
maps and the corresponding high-resolution output activation
map given by the sub-pixel convolutional layer for upscaling spatial size of the low-resolution input is preserved, i.e.
on two axes. For a scaling factor of r = 2 in both directions, the activation maps of the sixth conv layer have hI × wI
the sub-pixel convolutional layer requires r2 = 4 activation components, where hI and wI are the height and the width
maps as input. Best viewed in color. of the input image I. In order to increase the input r times
on both axes, the output of the sixth conv layer must be a
tensor of hI × wI × r2 components. The activation map
number of filters in the sixth convolutional layer is equal to resulting from the sub-pixel convolutional layer is a matrix
the scale factor, e.g. for a scale factor of 4× the number of of (hI · r) × (wI · r) components. For super-resolution on
filters is 4. The difference is caused by the fact that the first two axes, the pixels are rearranged as shown in Figure 3. In a
CNN upscales on two axes, while the second CNN upscales similar fashion, we can increase the input r times on one axis.
on one axis. The first convolutional block contains a short- In this case, the output of the sixth conv layer must be a tensor
skip connection, from the first conv layer to the third conv of hI × wI × r components. This time, the activation map
layer, and a long-skip connection, from the first conv layer to resulting from the sub-pixel convolutional layer can be either
the fifth conv layer. a matrix of (hI ·r)×wI components or a matrix of hI ×(wI ·r)
The first convolutional block is followed by a sub-pixel components, depending on the direction we aim to increase
convolutional (upscaling) layer, which was introduced in the resolution. For super-resolution on one axis, the pixels
[19]. In the upscaling layer, the activation maps produced are rearranged as shown in Figure 4. To our knowledge, we
by the sixth conv layer are assembled into a single acti- are the first to propose a sub-pixel convolutional (upscaling)
vation map. Throughout the first convolutional block, the layer for super-resolution in one direction.
VOLUME 8, 2020 5
 Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

When Shi et al. [19] introduced the sub-pixel convolutional With these notations, the transformation function f of our
layer, they used it as the last layer of their CNN. Hence, full CNN architecture can be written as follows:
the output depth of the upscaling layer is equal to the num-
f (θ, I) = f2 (θ2 , f1 (θ1 , I)), (2)
ber of channels in the output image. Different from Shi et
al. [19], we employ further convolutions after the upscaling where θ are the parameters of the full CNN, θ1 are the
layer. Nevertheless, since we are working with CT/MRI parameters of the first convolutional block and θ2 are the
(grayscale) images, our final output has a single channel. parameters of the second convolutional block, i.e. θ is a
In our architecture, the upscaling layer is followed by our concatenation of θ1 and θ2 . Having defined f1 and f2 as
second convolutional block, which starts with the seventh above, we can formally write our loss function as follows:
convolutional layer and ends with the tenth convolutional
layer. The first three conv layers in our second block are Lf ull = Lstandard + λ · Lintermediate , (3)
formed of 32 filters. The tenth conv layer contains a single where λ is a parameter that controls the importance of the
convolutional filter, since our output is a grayscale image intermediate loss with respect to the standard loss, Lstandard
that has a single channel. The second convolutional block is the standard loss defined in Equation (1) and Lintermediate
contains a short skip connection, from the seventh conv is defined as follows:
layer to the ninth conv layer. The spatial size of hO × wO wO X
X hO
components of the activation maps is preserved throughout Lintermediate (θ1 , I, O) = |f1 (θ1 , I) − O|. (4)
the second convolutional block, where hO and wO are the i=1 j=1
height and the width of the output image O.
In the experiments, we set λ = 1, since we did not find any
strong reason to assign a lower or higher importance to the
B. LOSSES AND OPTIMIZATION intermediate loss. By replacing λ with 1 and the loss values
In order to obtain a CNN model for single-image super- from Equation (1) and Equation (4), Equation (3) becomes:
resolution, the aim is to minimize the differences between wO X
hO
the ground-truth high-resolution image and the output image
X
Lf ull (θ, I, O) = |f (θ, I) − O|
provided by the CNN. Researchers typically employ the i=1 j=1
mean absolute difference as the objective function. Given a wO X
hO
(5)
low-resolution input image I and the corresponding ground-
X
+ |f1 (θ1 , I) − O|.
truth high-resolution image O, the loss based on the mean i=1 j=1
absolute value is formally defined as follows:
In order to optimize towards the objective defined in Equa-
wO X
X hO tion (5), we employ the Adam optimizer [45], which is known
L(θ, I, O) = |f (θ, I) − O|, (1) to converge faster than Stochastic Gradient Descent.
i=1 j=1
C. DATA AUGMENTATION
where θ are the CNN parameters (weights), f is the trans- A common approach to force the CNN to produce sharper
formation function learned by the CNN, and wO and hO output images is to apply Gaussian smoothing using a fixed
represent the width and the height of the ground-truth image standard deviation at training time [3], [14], [17]. By train-
O, respectively. ing the CNN on blurred low-resolution images, the super-
When we train our CNN model, we do not employ the resolution task becomes harder. During inference, when the
standard approach of minimizing the difference between the input images are no longer blurred, the task will be much
output provided by the CNN and the ground-truth HR image. easier. However, smoothing only the training images with
Instead, we propose a novel approach based on an interme- a fixed standard deviation will inherently generate a dis-
diate loss function. Since the conv layers after the upscaling tribution gap between training and test data. If the CNN
layer are meant to refine the high-resolution image without fits the training distribution well, it might not produce the
taking any additional information from the low-resolution desired results at inference time. This is because machine
input image, we note that the high-resolution image resulting learning models are based on the assumption that training
immediately after the upscaling layer should be as similar data and test data are sampled from the same distribution.
as possible to the ground-truth high-resolution image. There- We propose to solve this problem by using a randomly-
fore, we propose a loss function that aims to minimize the chosen standard deviation for each training image. Although
difference between the intermediately-obtained HR image the training data distribution will still be different from the
and the ground-truth HR image, in addition to minimizing testing data distribution, it will include the distribution of test
the difference between the final HR output image and the samples, as illustrated in Figure 5. In order to augment the
ground-truth HR image. Let f1 denote the transformation training data, we apply Gaussian blur with a probability of
function that corresponds to the first convolutional block 0.5 (only half of the images are smoothed) using a kernel of
and the upscaling layer, and let f2 denote the transformation 3 × 3 components and a randomly-chosen standard deviation
function that corresponds to the second convolutional block. between 0 and 0.5. In this way, we increase the variance of
6 VOLUME 8, 2020
Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

similarity index (SSIM). The PSNR is the ratio between

the maximum possible power of a signal and the power of
corrupting noise that affects the fidelity of its representation.
Although the PSNR is one of the most used metrics for
image reconstruction, some researchers [46], [47] argued
that it is not highly indicative of the perceived similarity.
The SSIM [46] aims to address this shortcoming by taking
contrast, luminance and texture into account. The result of
the SSIM is a number between -1 and 1, where a value of 1
means the ground-truth image and the reconstructed image
are identical. Similarly, a higher PSNR value indicates a
FIGURE 5: Distribution of training samples (represented by better reconstruction, although the PSNR does not have an
green triangles) and test samples (represented by red cir- upper bound.
cles), when the training samples are smoothed using a fixed Since PSNR and SSIM values cannot guarantee a visually
standard deviation (left-hand side) versus using a randomly- favorable result, we employ an additional metric for the final
chosen standard deviation (right-hand side). In example (a), results, namely the information fidelity criterion (IFC) [48].
overfitting on the training data leads to poor results on test Although IFC is scarcely used in literature [15], Yang et
data. In example (b), the danger of overfitting is diminished al. [49] pointed out that IFC is correlated well with the human
because the test distribution is included in the training distri- perception of SR images. As for PSNR and SSIM, higher IFC
bution. Best viewed in color. values indicate better results.

2) Image Quality Assessment by Human Observers

the training data without introducing any bias. In this case, Because the above metrics rely only on the pixel values and
if the CNN fits the training distribution well, it will produce not on the perceived visual quality, we decided to evaluate
good super-resolution outputs during inference, since there is our method with the help of human annotators. Although
no distribution gap between training and test. a deep learning method can provide better PSNR, SSIM or
IFC values, it might produce artifacts that could be mis-
IV. EXPERIMENTS leading for right diagnostics and treatment. We thus have
A. DATA SETS to make sure that our approach does not produce any un-
The first data set used in the experiments consists of 10 wanted artifacts visible to humans. We conducted the image
anonymized 3D images of brain CT provided by the Medical quality assessment on the CH data set, testing our CNN-
School at Colt, ea Hospital. We further refer to this data set as based method against Lanczos interpolation. We used CT
the Colt, ea Hospital (CH) data set. In order to fairly train and slices extracted from high-resolution 3D images resulting
test our CNN models and baselines, we randomly selected after applying super-resolution on all three axes. For each
6 images for training and used the remaining 4 images for upsampling factor, 2× and 4×, we extracted 100 CT slices at
testing. The training set has 359 slices (2D images) and the random from the test set. Hence, each human evaluator had to
testing set has 238 slices. The height and the width of the annotate 200 image pairs (100 for each upsampling factor).
slices vary between 192 and 512 pixels, while the depth of the For each evaluation sample, an annotator would have seen
3D images varies between 3 and 176 slices. The resolution of the original image in the middle and the two reconstructed
a voxel is 1 × 1 × 1 mm3 . images on its sides, one on the left side and the other on the
The second data set used in our experiments is the National right side. The annotators had a magnifying glass tool that
Alliance for Medical Image Computing (NAMIC) Brain allowed them to look at details and discover artifacts. The
Multimodality data set. The NAMIC data set consists of 20 locations (left or right) of the images reconstructed by our
3D MRI images, each composed of 176 slices of 256 × 256 CNN and by Lanczos interpolation were randomly picked
pixels. As for the CH data set, the resolution of a voxel is every time. To prevent any form of cheating, the randomly
1 × 1 × 1 mm3 . For our experiments, we used T1-weighted picked locations were unknown to the annotators. For each
(T1w) and T2-weighted (T2w) images independently. Fol- test sample, we asked each annotator to select the image
lowing [13], we split the NAMIC data set into a training set that best reconstructed the original image. Our experiment
containing 10 3D images and a test set containing the other was completed by 18 human annotators, 6 of them being
10 images. We kept the same split for T1w and T2w. doctors specialized in radiotherapy and oncology. In total, we
collected 3600 annotations (18 annotators × 200 samples).
B. EXPERIMENTAL SETUP
1) Evaluation Metrics 3) Baselines
As most previous works [3], [5], [10], [12]–[18], we em- We compared our method with standard resizing methods
ployed the two most common evaluation metrics, namely based on various interpolation schemes, namely nearest
the peak signal-to-noise ratio (PSNR) and the structural neighbors, bilinear, bicubic and Lanczos. In addition to these
VOLUME 8, 2020 7
 Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

TABLE 1: Preliminary 2D super-resolution results on the CH throughout all the experiments. In a set of preliminary exper-
data set for an upscaling factor of 2×. The PSNR and the iments, we did not observe any significant differences when
SSIM values are reported for various patch sizes and different using mini-batches of 64 or 256 images. In each experiment,
numbers of filters. For models with 7 × 7 patches, we report we trained the CNN for 40 epochs, starting with a learning
the inference time (in seconds) per CT slice measured on an rate (step size) of 10−3 and decreasing the learning rate to
Nvidia GeForce 940MX GPU with 2GB of RAM. 10−4 after the first 20 epochs.
Number of filters Input size SSIM PSNR Time (in seconds)
D. ABLATION STUDY RESULTS
32 4×4 0.9165 35.36 -
32 7×7 0.9270 36.22 0.05 We performed an ablation study to emphasize the effect of
32 14 × 14 0.8987 33.83 - various components over the overall performance. The abla-
32 17 × 17 0.8934 33.42 - tion results obtained on the CH data set for super-resolution
64 7×7 0.9279 36.28 0.08
128 7×7 0.9276 36.28 0.16 on height and width by a factor of 2× are presented in
Table 2.
In our first ablation experiment, we have eliminated all
the enhancements in order to show the performance level
baselines, we compared with three methods [3], [15], [17] of a baseline CNN on the CH data set. Since there are
that focused on 2D SISR and one method [13] that focused several SISR works [3], [6], [16], [18] based on the stan-
on 3D SISR. We note that You et al. [15] did not report results dard ESPCN model [19], we have eliminated the second
on NAMIC. Nonetheless, You et al. [15] were kind to provide convolutional block in the second ablation experiment, trans-
access to their source code. We thus applied their method on forming our architecture into a standard ESPCN architecture.
both CH and NAMIC data sets, keeping the same settings The performance drops from 0.9270 to 0.9236 in terms
and hyperparameters as recommended by the authors. As You of SSIM and from 36.22 to 35.94 in terms of PSNR. In
et al. [15], we employed their method only on 2D super- the subsequent ablation experiments, we have removed, in
resolution for the 2× upscaling factor. For the other three turns, the intermediate loss, the short-skip connections and
baselines, we included the NAMIC scores reported in the the long-skip connection. The results presented in Table 2
respective articles [3], [13], [17]. indicate that all these components are relevant to our model,
bringing significant performance benefits in terms of both
C. PARAMETER TUNING AND PRELIMINARY RESULTS SSIM and PSNR. In our last ablation experiment, we used
We conducted a series of preliminary experiments to deter- a fixed standard deviation instead of a variable one for the
mine the optimal patch size as well as the optimal width Gaussian blur added on training patches. We notice that our
(number of convolutional filters) for our CNN. In order to data augmentation approach based on a variable standard
find the optimal patch size, we tried out patches of 4 × 4, deviation brings the highest gains in terms of SSIM (from
7 × 7, 14 × 14 and 17 × 17 pixels. In term of the number 0.9236 to 0.9270) and PSNR (from 35.69 to 36.22), with
of filters, we tried out values in the set {32, 64, 128} for all respect to the other ablated components.
conv layers in our network. These parameters were tuned in Since the differences in terms of PSNR or SSIM for the
the context of 2D super-resolution on the CH data set. The ablated models are hard to quantify as small or large with
corresponding results are presented in Table 1. First of all, we respect to the complete CNN, we conducted paired McNe-
note that our method produces better SSIM and PSNR values, mar’s significance testing [50] to determine if the differences
i.e. 0.9270 and 36.22, for patches of 7 × 7 pixels. Second of are statistically significant or not. We considered a p-value
all, we observe that adding more filters on the conv layers threshold of 0.001 for our statistical testing. Every ablated
slightly increases the SSIM and PSNR values. However, the model that is significantly different from the complete model
gains in terms of SSIM and PSNR come with a great cost in is marked with † in Table 2. We note that the complete CNN
terms of time. For example, using 128 filters on each conv is significantly better than each ablated version, although the
layer triples the processing time in comparison with using 32 actual differences in terms of PSNR or SSIM might seem
filters on each conv layer. For the subsequent experiments, small. We thus conclude that all the proposed enhancements
we thus opted for patches of 7 × 7 pixels and conv layers provide significant performance gains.
with 32 filters.
We believe that it is important to note that, although the E. RESULTS ON CH DATA SET
number of training CT slices is typically in the range of a We first compared our CNN-based model with a series of
few hundreds, the number of training patches is typically in interpolation baselines and a state-of-the-art method [15] on
the range of hundreds of thousands. For instance, the number the CH data set. We present the results for super-resolution
of 7 × 7 training patches extracted from the CH data set for on two axes (width and height) in Table 3. Among the
the 2× upscaling factor is 326,000. We thus stress out that the considered baselines, it seems that the Lanczos interpolation
number of training samples is high enough to train highly- method provides better results than the bicubic, the bilinear
accurate deep learning models. or the nearest neighbor methods. Our CNN model is able
During training, we used mini-batches of 128 images to surpass all baselines for both upscaling factors, 2× and
8 VOLUME 8, 2020
Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

TABLE 2: Ablation 2D super-resolution results on the CH data set for an upscaling factor of 2×. The PSNR and the SSIM
values are reported for various ablated versions of our CNN model. The best results are highlighted in bold. Results of ablated
models marked with † are significantly worse than our complete model, according to paired McNemar’s testing [50] for the
significance level 0.001.

Second conv block Intermediate loss Short-skip connections Long-skip connection Variable standard deviation SSIM PSNR
7 7 7 7 7 0.9224† 35.58†
7 7 3 3 3 0.9236† 35.94†
3 7 3 3 3 0.9256† 36.15†
3 3 7 3 3 0.9260† 36.17†
3 3 3 7 3 0.9234† 36.11†
3 3 3 3 7 0.9236† 35.69†
3 3 3 3 3 0.9270 36.22

TABLE 3: 2D super-resolution results of our CNN model versus a state-of-the-art method [15] and several interpolation
baselines on the CH data set. The PSNR, the SSIM and the IFC values are reported for two upscaling factors, 2× and 4×.
The best result on each column is highlighted in bold.

Method 2× 4×
SSIM PSNR IFC SSIM PSNR IFC
Nearest neighbor 0.8893 32.71 4.40 0.7659 29.06 1.32
Bilinear 0.8835 33.34 3.73 0.7725 29.73 1.49
Bicubic 0.9077 34.71 4.59 0.7965 30.41 1.72
Lanczos 0.9111 35.08 4.93 0.8012 30.57 1.84
You et al. [15] 0.8874 32.73 4.40 - - -
Our CNN model 0.9291 36.39 5.36 0.8308 31.59 1.92

TABLE 4: 3D super-resolution results of our CNN model versus several interpolation baselines on the CH data set. The PSNR,
the SSIM and the IFC values are reported for two upscaling factors, 2× and 4×. The best result on each column is highlighted
in bold.

Method 2× 4×
SSIM PSNR IFC SSIM PSNR IFC
Nearest neighbor 0.8430 30.36 2.19 0.7152 27.32 0.72
Bilinear 0.8329 30.72 2.18 0.7206 27.93 0.95
Bicubic 0.8335 26.51 2.47 0.7200 24.05 1.04
Lanczos 0.8423 27.85 2.58 0.7263 25.06 1.09
Our CNN model 0.8926 33.04 2.83 0.7819 29.36 1.13

4×. Compared to the best interpolation method (Lanczos), output image is obtained by taking the median of the HR
our method is 0.0180 better in terms of SSIM, 1.31 better in images. In the following experiments on CH and NAMIC
terms of PSNR and 0.43 better in terms of IFC. Furthermore, data sets, the reported results always include the described
our CNN provides superior results to the GAN-based method self-ensemble strategy.
of You et al. [15]. We provide the results for super-resolution on all three
We note that, in Table 2, we reported an SSIM of 0.9270 axes in Table 4. First of all, we notice that the SSIM, the
and a PSNR of 36.22 for our method, while in Table 3, we PSNR and the IFC values are lower for all methods when
reported an SSIM of 0.9291 and a PSNR of 36.39. In order to dealing with 3D super-resolution (Table 4) instead of 2D
boost the performance of our method in accordance with the super-resolution (Table 3). This shows that the task of 3D
observed differences between Tables 2 and 3, we employed super-resolution is much harder than 2D super-resolution.
the self-ensemble strategy used by Lim et al. [51]. For each This is an expected result, considering that the dimension-
input image, the self-ensemble strategy consists in generat- ality of the reconstruction space increases significantly for
ing additional images using geometric transformations, e.g. 3D super-resolution, i.e. there are many more HR outputs
rotations and flips. Following Lim et al. [51], we generated 7 corresponding to a single LR input, while the training data
augmented images from the LR input image, upsampling all is the same. Nevertheless, our method exhibits smaller per-
8 images (the original image and the 7 additional ones) using formance drops when going from 2D super-resolution to 3D
our CNN. We then applied the inverse transformations to the super-resolution. As for the 2D super-resolution experiments
resulting 8 HR images in order to obtain 8 output images on CH data set, our CNN model for 3D super-resolution is
that are aligned with the ground-truth HR images. The final superior to all baselines for both upscaling factors. We thus
VOLUME 8, 2020 9
 Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

TABLE 5: 2D super-resolution results of our CNN model versus several state-of-the-art methods [3], [15], [17] and the Lanczos
interpolation baseline on the NAMIC data set. For Zeng et al. [17], we included results for both single-channel super-resolution
(SCSR) and multi-channel super-resolution (MCSR). The PSNR, the SSIM and the IFC values are reported for both T1w and
T2w images and for two upscaling factors, 2× and 4×. The best results on each column are highlighted in bold.

T1-weighted T2-weighted
Method 2× 4× 2× 4×
SSIM PSNR IFC SSIM PSNR IFC SSIM PSNR IFC SSIM PSNR IFC
Lanczos 0.9635 37.03 3.06 0.8955 32.71 1.28 0.9788 39.64 3.22 0.9143 33.80 1.37
Zeng et al. [17] (SCSR) 0.9220 36.86 − 0.7120 28.33 − − − − − − −
Zeng et al. [17] (MCSR) 0.9450 38.32 − 0.8110 30.84 − − − − − − −
Du et al. [3] 0.9390 37.21 − 0.7370 29.05 − − − − − − −
You et al. [15] 0.9448 35.09 2.80 − − − 0.9594 36.19 2.89 − − −
Our CNN model 0.9775 39.29 3.56 0.9193 33.74 1.29 0.9882 42.20 3.79 0.9382 34.86 1.39

TABLE 6: 3D super-resolution results of our CNN model versus a state-of-the-art method [13] and the Lanczos interpolation
baseline on the NAMIC data set. The PSNR, the SSIM and the IFC values are reported for both T1w and T2w images and for
two upscaling factors, 2× and 4×. The best results on each column are highlighted in bold.

T1-weighted T2-weighted
Method 2× 4× 2× 4×
SSIM PSNR IFC SSIM PSNR IFC SSIM PSNR IFC SSIM PSNR IFC
Lanczos 0.9423 35.72 2.00 0.8690 31.81 0.95 0.9615 37.80 2.29 0.8829 32.08 1.03
Pham et al. [13] − − − − − − 0.9781 38.28 − − − −
Our CNN model 0.9687 37.85 2.38 0.9050 32.88 0.99 0.9835 40.57 2.67 0.9251 33.54 1.10

conclude that our CNN model is better than all interpolation same explanation applies to the NAMIC data set, i.e. the
baselines on the CH data set, for both 2D and 3D super- CNNs have to produce likely reconstruction patterns in a
resolution and for all upscaling factors. much larger space.
While some of the considered state-of-the-art methods
F. RESULTS ON NAMIC DATA SET [3], [13], [17] presented results only for some cases on
On the NAMIC data set, we compared our method with NAMIC, either 2D super-resolution on T1w images or 3D
the best-performing interpolation method on the CH data super-resolution on T2w images, we provide our results for
set, namely Lanczos interpolation, as well as some state- all possible cases. We note that our CNN model surpasses
of-the-art 2D [3], [15], [17] and 3D [13] super-resolution Lanczos interpolation in each and every case. Furthermore,
methods. We note that most previous works, including [3], our model provides superior results than all the state-of-the-
[13], [17], used bicubic interpolation as a relevant baseline. art methods [3], [13], [15], [17] considered in our evaluation
Unlike these works, we opted for Lanczos interpolation, on the NAMIC data set.
which provided better results than bicubic interpolation and In addition to the quantitative results shown in Tables 5 and
other interpolation methods on the CH data set. 6, we present qualitative results in Figure 6. We selected 5 ex-
We first present the 2D super-resolution results in Table 5. amples of 2D super-resolution results generated by Lanczos
The 2D SR results indicate that the GAN-based method of interpolation, by the GAN-based method of You et al. [15]
You et al. [15] is superior to the CNN baselines [3], [17]. and by our CNN model. A close inspection reveals that our
However, none of the state-of-the-art methods [3], [15], [17] results are generally sharper than those of Lanczos interpo-
is able to attain better performance than Lanczos interpo- lation and those of You et al. [15]. As also confirmed by the
lation. This proves that Lanczos interpolation is a much SSIM, the PSNR and the IFC values presented in Tables 5
stronger baseline. Among the deep learning methods, our and 6, the images generated by our CNN are closer to the
CNN is the only one to surpass Lanczos interpolation for 2D ground-truth images. At the scale factor of 2× considered in
SR on NAMIC. We believe that this result is noteworthy. Figure 6, our CNN does not produce any patterns or artifacts
We also present the 3D super-resolution results in Table 6. that deviate from the ground-truth.
The 3D SR results show that the approach of Pham et al. [13]
is better than Lanczos interpolation, which is remarkable. G. IMAGE QUALITY ASSESSMENT RESULTS
Our CNN is even better, surpassing both the Lanczos inter- We provide the outcome of the subjective image quality
polation and the approach of Pham et al. [13]. assessment by human observers in Table 7. The study reveals
As for the CH data set, we observe that the PSNR, the that both doctors and regular annotators opted for our ap-
SSIM and the IFC scores for 2D super-resolution are higher proach in favor of Lanczos interpolation at an overwhelming
than the corresponding scores for 3D super-resolution. The rate (97.55% at the 2× scale factor and 96.69% at the 4×
10 VOLUME 8, 2020
Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

FIGURE 6: Image super-resolution examples selected from the NAMIC data set. In order to obtain the input images of 128×128
pixels, the original NAMIC images were downsampled by a scale factor of 2×. HR images of 256 × 256 pixels generated by
Lanczos interpolation, by the GAN-based method of You et al. [15] and by our CNN model are compared with the original
(ground-truth) HR images.

scale factor). For the 2× scale factor, 10 out of 18 annotators seems to be more prevalent at the 4× scale factor, although
preferred the output of our CNN in all the 100 presented the phenomenon is still rarely observed. This explains why
cases. We note that doctors #2 and #4 opted for Lanczos doctors #4 and #5 preferred Lanczos interpolation in more
interpolation in 15 and 14 cases (for the 2× scale factor), cases than the other doctors, although the majority of their
respectively, which was not typical to the other annotators. votes are still in favor of our CNN. When they opted for
Similarly, for the 4× scale factor, there are 3 annotators Lanczos interpolation, they considered that it is safer to
(doctor #4, doctor #5 and person #6) that seem to prefer consider its blurred and less informative output. In trying
Lanczos interpolation at a higher rate than the other annota- to find an explanation for these reconstruction patterns, we
tors. After discussing with the doctors about their choices, we analyzed the output of the CNN without data augmentation.
discovered that, in most cases, they prefer the sharper output We observed such reconstruction patterns even when training
of our CNN. However, the CNN seems to introduce some data augmentation was removed, ruling out this hypothesis.
reconstruction patterns (learned from training data) that do Given a low-resolution input patch, the CNN finds the most
not correspond exactly to the ground-truth. This phenomenon likely high-resolution patch corresponding to the input. This

VOLUME 8, 2020 11
 Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

TABLE 7: Image quality assessment results collected from agent is introduced).

6 doctors and 12 regular annotators, for the comparison be-
tween our CNN-based method versus Lanczos interpolation.
V. CONCLUSION
For each upscaling factor, each annotator had to select an
In this paper, we have presented an approach based on fully
option for a number of 100 image pairs. To prevent cheating,
convolutional neural networks for the super-resolution of
the randomly picked locations (left or right) for the generated
CT/MRI scans. Our method is able to reliably upscale 3D
HR images were unknown to the annotators.
CT/MRI image up to a scale factor of 4×. We have com-
Annotator ID 2× 4×
pared our approach with several baseline interpolation and
Our CNN Lanczos Our CNN Lanczos state-of-the-art methods [3], [13], [15], [17]. The empirical
Doctor #1 100 0 100 0 results indicated that our approach provides superior results
Doctor #2 85 15 100 0 on both CH and NAMIC data sets. We have also conducted
Doctor #3 100 0 100 0
a subjective image quality assessment by human observers,
Doctor #4 86 14 71 29
Doctor #5 100 0 68 32 showing that our method is significantly better than Lanczos
Doctor #6 95 5 95 5 interpolation. The subjective image quality assessment also
Person #1 100 0 99 1 revealed the limitations of a pure algorithmic approach. The
Person #2 97 3 100 0
Person #3 98 2 100 0
doctors invited to take part in our study concluded that going
Person #4 98 2 100 0 to a scale factor higher than 4× requires alternative solutions.
Person #5 100 0 98 2 In future work, we aim to continue our research by extending
Person #6 100 0 76 24 the proposed CNN method to multi-channel input. This will
Person #7 100 0 93 7
Person #8 99 1 98 2 likely help us in achieving higher upscaling factors, e.g. 10×,
Person #9 100 0 98 2 required for the accurate diagnostics and treatment of cancer,
Person #10 100 0 98 2 an actively studied and extremely important research topic
Person #11 98 2 99 1
Person #12 100 0 98 2
[52], [53].
Overall in % 97.55% 2.45% 96.69% 3.31%
ACKNOWLEDGMENT
The authors would like to thank the reviewers for their
helpful comments. In memory of Tatiana-Mihaela Ionescu,
likelihood is learned by the CNN when it is trying to mini-
who offered her CT scans to facilitate our study, but died of
mize the loss over the entire training set. Although producing
cancer.
the most probable output works well in most cases, using
a machine learning model, e.g. a CNN, is not the perfect
REFERENCES
solution. The explanation becomes clear if we consider that
[1] E. Sert, F. Özyurt, and A. Doğantekin, “A new approach for brain tumor
multiple HR patches can correspond to the same LR input diagnosis system: single image super resolution based maximum fuzzy
patch and that choosing the most likely HR patch is not entropy segmentation and convolutional neural network,” Medical Hy-
always the right answer. We thus conclude that our CNN potheses, vol. 133, p. 109413, 2019.
[2] Y. Chen, F. Shi, A. G. Christodoulou, Y. Xie, Z. Zhou, and D. Li, “Ef-
suffers from the same problem as any other machine learning ficient and accurate MRI super-resolution using a generative adversarial
model. Furthermore, we stress out that the reconstruction pat- network and 3D multi-level densely connected network,” in Proceedings
terns in question are plausible from a biological point of view, of MICCAI, 2018, pp. 91–99.
[3] X. Du and Y. He, “Gradient-Guided Convolutional Neural Network for
i.e. the doctors were able to spot them only by comparing MRI Image Super-Resolution,” Applied Sciences, vol. 9, no. 22, p. 4874,
the HR output with the ground-truth HR image. We note that 2019.
these patterns should not be mistaken with artifacts that could [4] J. Du, L. Wang, A. Gholipour, Z. He, and Y. Jia, “Accelerated Super-
be caused by underfitting or a poor architectural choice. Our resolution MR Image Reconstruction via a 3D Densely Connected Deep
Convolutional Neural Network,” in Proceedings of BIBM, 2018, pp. 349–
CNN does not introduce such artifacts. 355.
Based on our subjective image quality assessment, we con- [5] J. Du, Z. He, L. Wang, A. Gholipour, Z. Zhou, D. Chen, and Y. Jia,
“Super-resolution reconstruction of single anisotropic 3D MR images
cluded with the doctors that going beyond the 4× scale factor, using residual convolutional neural network,” Neurocomputing, 2019.
solely with a method based on algorithmic super-resolution, [6] J. Hatvani, A. Horváth, J. Michetti, A. Basarab, D. Kouamé, and
is neither safe (a CNN might introduce too many patterns far M. Gyöngy, “Deep learning-based super-resolution applied to dental com-
puted tomography,” IEEE Transactions on Radiation and Plasma Medical
from the ground-truth) nor helpful (a standard interpolation Sciences, vol. 3, no. 2, pp. 120–128, 2018.
method is not informative). However, the doctors agree that [7] J. Hatvani, A. Basarab, J.-Y. Tourneret, M. Gyöngy, and D. Kouamé, “A
either super-resolution method is desirable in favor of the Tensor Factorization Method for 3-D Super Resolution With Application
input low resolution images. Therefore, in order to reach to Dental CT,” IEEE Transactions on Medical Imaging, vol. 38, no. 6, pp.
1524–1531, 2018.
the scale factor of 10× desired by the doctors, we have to [8] Y. Huang, L. Shao, and A. F. Frangi, “Simultaneous Super-Resolution
look in other directions in future work. A promising direction and Cross-Modality Synthesis of 3D Medical Images Using Weakly-
is to combine multiple inputs, e.g. by using CT and MRI Supervised Joint Convolutional Sparse Coding,” in Proceedings of CVPR,
2017, pp. 5787–5796.
scans of the same person or by using CT/MRI scans taken [9] J. Jurek, M. Kocinski, A. Materka, M. Elgalal, and A. Majos, “Cnn-based
at different moments in time (before and after the contrast superresolution reconstruction of 3d mr images using thick-slice scans,”

12 VOLUME 8, 2020
Georgescu et al.: Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

Biocybernetics and Biomedical Engineering, vol. 40, no. 1, pp. 111–125, Destructive Ocular Visualization Methods by Improved X-Ray Imaging
2019. Techniques,” PLoS One, vol. 12, p. e0170633, 2017.
[10] Y. Li, B. Song, J. Guo, X. Du, and M. Guizani, “Super-Resolution of Brain [33] G. Tromba, R. Longo, A. Abrami, F. Arfelli, A. Astolfo, P. Bregant,
MRI Images Using Overcomplete Dictionaries and Nonlocal Similarity,” F. Brun, K. Casarin, V. Chenda, D. Dreossi, M. Hola, J. Kaiser, L. Mancini,
IEEE Access, vol. 7, pp. 25 897–25 907, 2019. R. H. Menk, E. Quai, E. Quaia, L. Rigon, T. Rokvic, N. Sodini, D. San-
[11] D. Mahapatra, B. Bozorgtabar, and R. Garnavi, “Image super-resolution abor, E. Schultke, M. Tonutti, A. Vascotto, F. Zanconati, M. Cova, and
using progressive generative adversarial networks for medical image anal- E. Castelli, “The SYRMEP Beamline of Elettra: Clinical Mammography
ysis,” Computerized Medical Imaging and Graphics, vol. 71, pp. 30–39, and Bio-medical Applications,” AIP Conference Proceedings, vol. 1266,
2019. no. 1, pp. 18–23, 2010.
[12] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero, [34] C. H. McCollough, S. Leng, L. Yu, and J. G. Fletcher, “Dual- and Multi-
A. de Marvao, S. Cook, D. O’Regan, and D. Rueckert, “Multi-input Energy CT: Principles, Technical Approaches, and Clinical Applications,”
Cardiac Image Super-Resolution Using Convolutional Neural Networks,” Radiology, vol. 276, no. 3, pp. 637–653, 2015.
in Proceedings of MICCAI, 2016, pp. 246–254. [35] J. Doerner, M. Hauger, T. Hickethier, J. Byrtus, C. Wybranski, N. G.
[13] C.-H. Pham, C. Tor-Díez, H. Meunier, N. Bednarek, R. Fablet, N. Passat, Hokamp, D. Maintz, and S. Haneder, “Image quality evaluation of dual-
and F. Rousseau, “Multiscale brain MRI super-resolution using deep 3D layer spectral detector CT of the chest and comparison with conventional
convolutional networks,” Computerized Medical Imaging and Graphics, CT imaging,” European Journal of Radiology, vol. 93, pp. 52–58, 2017.
vol. 77, no. 101647, 2019. [36] G. Cybenko, “Approximation by superpositions of a sigmoidal function,”
[14] J. Shi, Z. Li, S. Ying, C. Wang, Q. Liu, Q. Zhang, and P. Yan, “MR image Mathematics of Control, Signals and Systems, vol. 2, no. 4, pp. 303–314,
super-resolution via wide residual networks with fixed skip connection,” 1989.
IEEE Journal of Biomedical and Health Informatics, vol. 23, no. 3, pp. [37] K. Hornik, “Approximation Capabilities of Multilayer Feedforward Net-
1129–1140, 2018. works,” Neural Networks, vol. 4, no. 2, pp. 251–257, 1991.
[15] C. You, G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, [38] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
Z. Zhang, W. Cong et al., “CT super-resolution GAN constrained by the no. 7553, pp. 436–444, 05 2015.
identical, residual, and cycle learning ensemble (GAN-CIRCLE),” IEEE [39] R. T. Ionescu, F. S. Khan, M.-I. Georgescu, and L. Shao, “Object-centric
Transactions on Medical Imaging, 2019. Auto-encoders and Dummy Anomalies for Abnormal Event Detection in
[16] H. Yu, D. Liu, H. Shi, H. Yu, Z. Wang, X. Wang, B. Cross, M. Bramler, and Video,” in Proceedings of CVPR, 2019, pp. 7842–7851.
T. S. Huang, “Computed tomography super-resolution using convolutional [40] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
neural networks,” in Proceedings of ICIP, 2017, pp. 3944–3948. T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient Convo-
lutional Neural Networks for Mobile Vision Applications,” arXiv preprint
[17] K. Zeng, H. Zheng, C. Cai, Y. Yang, K. Zhang, and Z. Chen, “Simulta-
arXiv:1704.04861, 2017.
neous single- and multi-contrast super-resolution for brain MRI images
[41] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up Convolutional
based on a convolutional neural network,” Computers in Biology and
Neural Networks with Low Rank Expansions,” in Proceedings of BMVC,
Medicine, vol. 99, pp. 133–141, 2018.
2014.
[18] X. Zhao, Y. Zhang, T. Zhang, and X. Zou, “Channel splitting network
[42] V. Nair and G. E. Hinton, “Rectified Linear Units Improve Restricted
for single MR image super-resolution,” IEEE Transactions on Image
Boltzmann Machines,” in Proceedings of ICML, 2010, pp. 807–814.
Processing, vol. 28, no. 11, pp. 5649–5662, 2019.
[43] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification
[19] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueck- with Deep Convolutional Neural Networks,” in Proceedings of NIPS,
ert, and Z. Wang, “Real-Time Single Image and Video Super-Resolution 2012, pp. 1106–1114.
Using an Efficient Sub-Pixel Convolutional Neural Network,” in Proceed- [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image
ings of CVPR, 2016, pp. 1874–1883. Recognition,” in Proceedings of CVPR, June 2016, pp. 770–778.
[20] J. Long, E. Shelhamer, and T. Darrell, “Fully Convolutional Networks for [45] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in
Semantic Segmentation,” in Proceedings of CVPR, 2015, pp. 3431–3440. Proceedings of ICLR, 2015.
[21] L. Wang, W. Ouyang, X. Wang, and H. Lu, “Visual Tracking with Fully [46] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image
Convolutional Networks,” in Proceedings of ICCV, 2015, pp. 3119–3127. quality assessment: from error visibility to structural similarity,” IEEE
[22] G. Castellano, C. Castiello, C. Mencar, and G. Vessio, “Crowd Detection Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.
for Drone Safe Landing Through Fully-Convolutional Neural Networks,” [47] Z. Wang and A. C. Bovik, “Mean squared error: Love it or leave it? A
in Proceedings of SOFSEM, 2020, pp. 301–312. new look at signal fidelity measures,” Signal Processing Magazine, vol. 26,
[23] F. Karim, S. Majumdar, H. Darabi, and S. Chen, “LSTM Fully Convolu- no. 1, pp. 98–117, 2009.
tional Networks for Time Series Classification,” IEEE access, vol. 6, pp. [48] H. R. Sheikh, A. C. Bovik, and G. de Veciana, “An information fidelity
1662–1669, 2017. criterion for image quality assessment using natural scene statistics,” IEEE
[24] M. Bevilacqua, A. Roumy, C. Guillemot, and M. line Alberi Morel, Transactions on Image Processing, vol. 14, no. 12, pp. 2117–2128, 2005.
“Low-Complexity Single-Image Super-Resolution based on Nonnegative [49] C.-Y. Yang, C. Ma, and M.-H. Yang, “Single-Image Super-Resolution: A
Neighbor Embedding,” in Proceedings of BMVC, 2012, pp. 135.1–135.10. Benchmark,” in Proceedings of ECCV, 2014, pp. 372–386.
[25] Hong Chang, Dit-Yan Yeung, and Yimin Xiong, “Super-resolution through [50] T. G. Dietterich, “Approximate Statistical Tests for Comparing Supervised
neighbor embedding,” in Proceedings of CVPR, vol. 1, 2004, pp. I–I. Classification Learning Algorithms,” Neural Computation, vol. 10, no. 7,
[26] D. Dai, R. Timofte, and L. V. Gool, “Jointly Optimized Regressors for pp. 1895–1923, 1998.
Image Super-resolution,” Computer Graphics Forum, vol. 34, pp. 95–104, [51] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced Deep Residual
2015. Networks for Single Image Super-Resolution,” in Proceedings of CVPRW,
[27] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image Super-Resolution Via 2017, pp. 1132–1140.
Sparse Representation,” IEEE Transactions on Image Processing, vol. 19, [52] C. Popa, N. Verga, M. Patachia, S. Banita, and C. Matei, “Advantages of
no. 11, pp. 2861–2873, 2010. laser photoacoustic spectroscopy in radiotherapy characterization,” Roma-
[28] C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using nian Reports in Physics, vol. 66, pp. 120–126, 2014.
Deep Convolutional Networks,” IEEE Transactions on Pattern Analysis [53] D. Sardari and N. Verga, “Calculation of externally applied electric field
and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. intensity for disruption of cancer cell proliferation,” Electromagnetic Biol-
[29] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image Super- ogy and Medicine, vol. 29, no. 1–12, pp. 26–30, 2010.
Resolution Using Very Deep Residual Channel Attention Networks,” in
Proceedings of ECCV, 2018, pp. 294–310.
[30] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in
Proceedings of NIPS, 2014, pp. 2672–2680.
[31] J. C. Elliott and S. D. Dover, “X-ray microtomography,” Journal of
Microscopy, vol. 126, no. 2, pp. 211–213, 1982.
[32] C. Enders, E.-M. Braig, K. Scherer, J. U. Werner, G. K. Lang, G. E.
Lang, F. Pfeiffer, P. Noël, E. Rummeny, and J. Herzen, “Advanced Non-

VOLUME 8, 2020 13