Neurocomputing 409 (2020) 244–258

Contents lists available at ScienceDirect

Neurocomputing
journal homepage: www.elsevier.com/locate/neucom

A survey on U-shaped networks in medical image segmentations

Liangliang Liu a,d, Jianhong Cheng a, Quan Quan a, Fang-Xiang Wu b, Yu-Ping Wang c,
Jianxin Wang a,⇑
a
Hunan Provincial Key Lab on Bioinformatics, School of Computer Science and Engineering, Central South University, Changsha 410083, PR China
b
Department of Mechanical Engineering and Division of Biomedical Engineering, University of Saskatchewan, Saskatoon, SK S7N5A9, Canada
c
School of Biomedical Engineering Department, Tulane University, New Orleans, LA 70118, USA
d
Department of Network Center, Pingdingshan University, Pingdingshan 467000, PR China

a r t i c l e i n f o a b s t r a c t

Article history: The U-shaped network is one of the end-to-end convolutional neural networks (CNNs). In electron micro-
Received 14 January 2020 scope segmentation of ISBI challenge 2012, the concise architecture and outstanding performance of the
Revised 14 April 2020 U-shaped network are impressive. Then, a variety of segmentation models based on this architecture
Accepted 26 May 2020
have been proposed for medical image segmentations. We present a comprehensive literature review
Available online 1 June 2020
Communicated by Xiaofeng Zhu
of U-shaped networks applied to medical image segmentation tasks, focusing on the architectures,
extended mechanisms and application areas in these studies. The aim of this survey is twofold. First,
we report the different extended U-shaped networks, discuss main state-of-the-art extended mecha-
Keywords:
U-shaped network
nisms, including residual mechanism, dense mechanism, dilated mechanism, attention mechanism,
Convolutional neural networks multi-module mechanism, and ensemble mechanism, analyze their pros and cons. Second, this survey
Medical image segmentation provides the overview of studies in main application areas of U-shaped networks, including brain tumor,
Extended mechanism stroke, white matter hyperintensities (WMHs), eye, cardiac, liver, musculoskeletal, skin cancer, and neu-
ronal pathology. Finally, we summarize the current U-shaped networks, point out the open challenges
and directions for future research.
Ó 2020 Elsevier B.V. All rights reserved.

1. Introduction mated methods which can locate, segment, and quantify the lesion
tissues. The segmentation methods are expected to have a broad
Medical imaging play an important role in clinical diagnosis, impact by supporting clinician decisions.
especially in treatment planning, surgery and prognosis evaluation. In adjuvant treatment decision, an automatic segmentation
Medical images are used to collect potential life-saving information method can provide reliable diagnostic evidence for radiologists.
by non-invasive peering at human organs [1]. Rapid and accurate It will not only help radiologists examine lesion tissues and make
identifications and segmentations of lesions tissues are the basis treatment plans for predicting potential treatment risks and bene-
of diagnosis. With the rapid development and popularization of fits, but will also qualitatively and quantitatively analyze damaged
medical imaging equipments, medical imaging information is pro- tissues, which makes medical trials more reliable and repeatable
vided by magnetic resonance imaging (MRI), computed tomography [4,5]. Another beneficiary is the patients as automatic segmenta-
(CT), positron emission tomography (PET), digital pathology and tions can reduce trauma and help them intuitively understand
microscopy. These medical images can provide exceptional anatom- the condition [6,7].
ical views of organs. However, it is not easy for radiologists to quan- However, there are 3 challenges in medical segmentation tasks.
tify organs or lesions from these medical images. For instance, CT First, the limited number of image samples for specific disease is
and MRI images are 2 important diagnostic basis in clinical diagno- one of the challenges to segmentation methods. Second, the similar
sis. Both images are frequently in the 3D format, and segmentation intensity and the changeable position shape and the size of interest
of lesions tissues need to be done slice by slice on 2D images. If all lesions case the difficulty in segmentation methods. Third, many
medical images are hand-marked by radiologists, it will take up to factors of capturing images, such as sampling artifacts, spatial alias-
15 min per image [2], which is a time-consuming task with low ing and noise factors may cause the boundary of structures to be
inter-rater agreement [3]. Therefore, it is necessary to develop auto- indistinct and disconnected. In fact, many researchers have pro-
posed a variety of segmentation methods. In particular, in recent
⇑ Corresponding author. years the developments of deep learning methods have made
E-mail address: [email protected] (J. Wang). remarkable achievements in medical image segmentation field.

https://doi.org/10.1016/j.neucom.2020.05.070
0925-2312/Ó 2020 Elsevier B.V. All rights reserved.
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 245

Deep learning in medical applications have aroused great used for the segmentation task, but for compressing images and
interest in past few years. These methods attempt to use multiple denoising functions. For example, encoder is a down-sampling pro-
non-linear transformations to abstract input data at multiple levels cess, and obtains smaller features from input image in the down-
and map predicted results. These methods not only learn the non- sampling process. This process is equivalent to image compression.
linear mapping between input data and predicted results, but also Decoder is an up-sampling process. During this process, the com-
learn the hidden features [8]. So far, the most successful image anal- pressed image is restored to the original image.
ysis methods are convolution neural networks (CNNs) [9–14]. The In 2015, Ronneberger et al. [23] and Long et al. [24] introduced
breakthrough of CNNs is very important for the development of an encoder-decoder architecture into image segmentation, and
the medical image segmentations [15]. In these CNN methods, U- proposed a U-Net and a Fully Convolutional Network (FCN),
shaped networks have made outstanding achievements and thus respectively. U-Net was proposed later than FCN, and both net-
become the popular technology for medical image segmentations. work architectures are similar: there are 2 main differences
Some reviews and researchs on the applications of deep learning between these 2 networks. The first difference is that U-Net is a
to medical image analysis were published by Liu et al. [16], Litjens symmetric architecture, which consists of 2 parts (encoder and
et al. [17], Shen et al. [18], Suzuki et al. [19], Yu et al. [20], and Chang decoder). However, the decoder of FCN is relatively simple. There
et al. [21]. They covered a lot of research work. However, the contri- is only one deconvolution operation in the decoder process, and
bution of U-shaped networks in the medical image segmentation does not keep up with the convolution architecture. The second
field was not reflected in these existing reviews. Since the U- difference is the manner of skip connections: FCN uses addðÞ as
shaped network (U-Net) was made public in 2015, in less than the skip connection to connect encoder and decoder, while U-Net
4 years, it has received more than 5400 references. Most medical uses concatðÞ as the skip connection to connect encoder and deco-
image segmentation methods have chosen U-shaped architecture der. Compared with addðÞ method, conatðÞ method can help U-Net
as the basic segmentation framework or chosen the U-Net as the model to obtain more feature information. This is a key to improve
basic comparison method. The motivation of our review is to provide the performance of segmentation methods.
a comprehensive analysis of medical image segmentation methods U-Net is a neat end-to-end neural network architecture which
which are related U-Net/ U-shaped networks. This also include over- is known for its ‘‘U” shape. Meanwhile, U-Net achieved a remark-
view tables of most publications, so readers can use to quickly assess able success in cell segmentation with the light microscope
the method. Finally, we leverage our own experience with the appli- images. U-Net won the championship with great advantage in neu-
cations of related U-shaped methods and public datasets in medical ronal structures segmentation at ISBI challenge 2012. In addition,
image segmentations. We also provide readers with a dedicated dis- U-Net got the top ranked in ISBI 2014 challenge for segmentation
cussion on the state-of-the-art methods, and research directions. of Glioblastoma-astrocytoma U373 cells.1 It achieved an average
To identify relevant papers, we used the Google scholar search intersection over union (IOU) of 92%, as opposed to the second best
engine to query papers with titles or abstracts containing (‘‘U- algorithm with an IOU of 83%. Besides, U-Net won the Grand Chal-
Net”, ‘‘V-Net”, ‘‘U shape” or ‘‘U-shaped”). The start time of the lenge for Computer-Automated Detection of Caries in Bitewing
included articles is 2015, and the deadline is May 1, 2019. This sur- Radiography at ISBI 2015.2 It outperformed the second best method
vey comes from more than 100 papers related to medical image by a large margin. Furthermore, U-Net also won the HeLa Cells
segmentation tasks in the search results. Some papers come from Tracking Challenge at ISBI 2015.3 It achieved an average IOU of
academic journals, such as PNAS, IEEE Transactions on Image Pro- 77.5% which is better than the second best method with an IOU of
cessing, Neuroimage and IEEE Transactions on Medical Imaging 46%. In 2016, Ronneberger et al. further extended the previous U-
and some are from related conferences, such as CVPR, ICCV, MICCAI Net from 2-dimensional (2D) space to 3-dimensional (3D) space
and ISBI, while others are from ArXiv. We re-checked all papers and utilized the proposed 3D U-Net to segment dense volumetric
and excluded the papers that had nothing to do with medical images from a sparse annotation in MICCAI 2016 [25]. Moreover,
images or U-shaped networks. If overlapping work were reported another volumetric convolutional neural network called V-Net was
in multiple publications, only the papers with the highest citation proposed by Milletari et al. in 2016. This network was used to seg-
rate were included. ment prostate volumes [26]. Based on 2D U-Net architecture, Isensee
With this survey we aim to: et al. improved the 3D U-Net to segment brain tumors with multi-
subregion architectures and achieved competitive results in BraTS
1. Summarize the advancement of U-shaped networks in the field 2017 and 2018 challenges [27,28], respectively. A recent study
of medical image segmentations. showed that the methods based on the U-shaped architecture
2. Highlight specific contributions of extended mechanisms based achieved the best performance on 6 publicly available segmentation
on U-shaped architectures. tasks and can automatically adapt to any given dataset [29]. Due to
3. Present some public datasets and demonstrate the achieve- its excellent performance and elegant architecture, the U-shaped
ments of these extended networks in medical image segmenta- networks have shown the best performance in segmentations of
tion application areas. medical images [30], and quickly become a benchmark in medical
image segmentations.
The rest of this survey is structured as followed. Section 2
reviews the history and architecture of U-shaped networks. Sec-
tion 3 describes the extended mechanisms based on U-shaped 2.1. Network architecture
architectures. Section 4 introduces the public datasets used in
medical image segmentations. Section 5 provides the main appli- A classical architecture of U-Net is shown in Fig. 1. U-Net is
cations. We end with a summary, a critical discussion and an out- composed of a contracting path and an expanding path, which cor-
look for future research in Section 6. respond to an encoder and a decoder, respectively. The contracting
path consists of a series of down-sampling operations, which is
used to reduce the scale of images and obtain context information.
2. U-net neural network
1
https://cs.adelaide.edu.au/c~arneiro/isbi14_challenge/.
In 2006, Hinton et al. proposed a classic architecture: encoder- 2
www-o.ntust.edu.tw/c~weiwang/ISBI2015/challenge2/index.html.
decoder [22]. The initial encoder-decoder architecture was not 3
https://cs.adelaide.edu.au/z~hi/isbi15_challenge/index.html.
246 L. Liu et al. / Neurocomputing 409 (2020) 244–258

Fig. 1. The architecture of a U-Net.

The expanding path consists of a series of up-sampling operations, address this drawback, Milletari et al. [26] and Isensee et al. [27]
which is used to precisely locate pixels/ voxels and restore images. used convolutional blocks with a stride of 2 instead of the pooling
Specifically, as an encoder-decoder network, U-Net architecture operations. The advantages of this strategy is that it not only
can be divided into 4 parts: a) convolution operations, b) retains the image details but also reduces the image resolution to
down-sampling operations, c) up-sampling operations and d) con- be suitable for understanding and learning.
catenation operation. The contracting path consists of repeated
operations of convolution and down-sampling layers. Symmetri- 2.1.3. Up-sampling operation
cally, the expanding path is made up of convolution and Up-sampling aims to magnify the original image and display it
up-sampling layers. In order to fuse more context information, at a higher resolution. Compared with down-sampling operation,
concatenation operations are adopted between contracting and up-sampling operation mainly increases the resolution of feature
expanding paths at the same depth level. maps in neural networks. That is to say, it allows network to trans-
fer context information from the low spatial resolution layer to the
2.1.1. Convolution operation higher spatial resolution layer in the up-sampling stage. The com-
Convolution is the most important operation in deep learning monly used up-sampling methods are: bilinear, deconvolution and
methods. It is mainly used to extract features from input data. unpooling. Almost all image segmentation methods use an inter-
However, a simple convolution is not enough in medical image polation algorithm (one of bilinear algorithms) in the up-
segmentation. Because a simple convolution is insufficient for sampling operation [39], and so is the U-Net. The interpolation
extracting abstract features information from input images, since algorithm inserts new elements between the pixels in image
a lot of image features are lost in the convolution process. In order matrix. This algorithm is mainly used to repair the uneven lines
to alleviate this problem, a convolution is frequently combined in images. Compared with other methods, the interpolation algo-
with several normalization methods such as Batch Normalization rithm is more suitable for image segmentations.
[31], Layer Normalization [32], Instance Normalization [33] and
Group Normalization [34]. In addition, activation function is also 2.1.4. Concatenation operation
an important part of a neural network. Activation function is used Concatenation is one essential operation in U-Net network. It is
to realize nonlinearity between input and output of neurons. Com- used to pinpoint pixel locations. In U-Net network, the high resolu-
mon activation functions include: ReLU [35], LReLU [36], PReLU tion feature maps from the contracting path are combined with
[37] and RReLU [38]. Some of them not only speed up the conver- that from the expanding path in the same depth layers. Some
gence, but also avoid the gradient vanishing problem. In the classi- researchers extended the concatenation operation in U-Net. For
cal architecture of the U-Net, each convolutional operation is instance, the studies of Milletari et al. have shown that the
usually followed by a normalization and an activation function, extended concatenation operations not only improve the quality
which constitutes a high-performance convolutional block [27]. of the final result, but also shorten the convergence time of the
training [26]. In addition, the attention concatenation is the latest
2.1.2. Down-sampling operation and efficient way of concatenation operations [40].
In image segmentations, down-sampling operation reduces the
resolution of feature maps while retaining the significant informa- 2.2. Advantage
tion. Max pooling and average pooling are 2 common strategies
used in down-sampling operation. The pooling strategy uses higher There are usually 2 challenges for CNNs in medical image seg-
level abstraction to represent image features. Neural networks use mentations: One is the limited number of samples for training.
pooling strategy to integrate feature points within small neighbor- Another is that increase of the depth and width of CNNs is likely
hoods after several convolution layers to get new features. The max to cause the problem of gradient vanishing problem or gradient
pooling is beneficial for extracting the extreme features such as exploding problem. Most researchers tried to change the network
edges, while the average pooling makes features smoother. The architectures to alleviate these 2 challenges. The appearance of
max pooling was used in original U-Net. Although pooling can be U-Net alleviates those 2 challenges. Compared with other CNN
positive to have invariance and to remove irrelevant details, it also models, U-Net has the following advantages: 1. Supporting a
has a negative impact on removing important information. To limited number of samples to train network. 2. Realizing image
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 247

features with a multi-scale recognition and fusion. 3. Having a sim- 3.1. Residual mechanism
ple and flexible structure. 4. Obtaining high quality pixel level seg-
mentation results. He et al. proposed a residual deep convolution network
The original U-Net not only shows a good performance in med- (ResNet) in 2015 [41]. ResNet won the championship of image
ical segmentation applications, but also provides a flexible and classification, detection and location in ImageNet challenge.
extensible structure. So the improved model can adapt to new ResNet used residual convolutions to improve feature utilization
tasks. In the next section, we introduce some U-Net-related and thus increase the performance of networks. In addition, resid-
models. ual mechanism adds the depth of the network while ensuring good
performance. With the increase of network depth, this mechanism
3. Improved mechanisms also alleviates the gradient vanishing problem in deep networks.
ResNet was originally used for image classification [41]. After
Despite the original U-Net network has shown outstanding per- that, many researchers introduced the essence of ResNets into
formance in segmenting medical images, the classical U-Net archi- the field of medical image segmentations by combining residual
tecture also have some defects in certain aspects. Therefore, based blocks and U-shaped architectures. As shown in Table 1, Fakhry
on the U-shaped architecture, researchers have proposed various et al. proposed a residual deconvolutional network (RDN) based
improved mechanisms to improve the performance of the model, on U-Net [42]. RDN consists of 2 information pathways that cap-
such as: residual mechanism, dense mechanism, dilated mecha- ture full-resolution features and contextual information, respec-
nism, attention mechanism, multi-module mechanism, and tively. Residual convolution not only deepens RDN network, but
ensemble mechanism. The segmentation models built on these also contains enough context information and provides multi-
extension mechanisms are the potential successors of the U-Net scale full resolution features. Furthermore, Chen et al. embedded
architecture. In the following section, we introduce these improved residual blocks into a U-shaped segmentation network for obtain-
mechanisms based on U-shaped architecture since 2015. This sur- ing more contextual information [43]. Residual blocks consist of
vey includes over 100 papers, and the pie chart of various improve- different size kernels which can aggregate feature mappings from
ment mechanisms in the literature is shown in Fig. 2. It is easy to different branches. Similarly, Alom et al. also embedded residual
observe that the residual and ensemble mechanisms are mostly blocks into the U-shaped architecture for training a deep network
used in these papers. and ensuring better feature representation for medical image seg-
mentation tasks [44]. Chen et al. proposed a VoxResNet basis on
the U-shaped architecture [45]. They took advantage of the charac-
teristics of residual networks to alleviate the degradation of train-
ing deep networks, and increased the network depth to improve
segmentation performance. Likewise, in order to alleviate the gra-
dient vanishing problem of the back-propagation flow and improve
the convergence speed, Guerrero et al. also applied the residual
structure to WMH lesion segmentations [46]. In addition, Drozdzal
et al. [47], Clèrigues et al. [48], Ibtehaz et al. [49] and Liu et al. [50]
also used the residual mechanisms in their studies.

3.2. Dense mechanism

Huang et al. published the best paper on CVPR conference in

2017 [61]. They proposed a deep network, named DenseNet, which
breaks away the traditional idea of the ResNet and expands recep-
Fig. 2. The pie chart of various improvement mechanisms in these literatures.
tion to improve network performance. Compared with ResNet,

Table 1
Overview of papers using residual mechanism techniques in U-shaped networks.

Model name Image type Target

FCNs [51] (2016) EM images Neural structures
RDN [42] (2017) EM images Neural structures
DRINet [43] (2018) CT, MRI CSF, Multi-organ, Brain tumor
FC-ResNet [47] (2018) CT, MRI Liver, Prostate
R2U-Net [52] (2018) Nuclei images Nuclei
RA-UNet [53] (2018) CT Liver tumor
R2U-Net [52] (2018) Retina image, Photograph, CT Blood vessel, Skin cancer, Lung
SUNet [48] (2018) MRI Stroke lesion
VoxResNet [45] (2018) MRI Brain structures
ResU-Net [54] (2018) MRI WMH
uResNet [46] (2018) MRI WMH
MultiResUNet [49] (2019) Microscopy, dermoscopy, endoscopy imageS, MRI Neural structures, Electron microscopy, Skin lesions,
Colonoscopy, Brain tumor
CNN [55] (2019) MRI Brain tumor
DRUNet [56] (2019) MRI Brain tumor
CE-Net [57] (2019) Retina image, CT, Endoscopy Image, Retina image Optic disc, Lung, Cell contour, Retinal OCT layer
CNN [58] (2019) MRI Brain tumor
Res-CNN [50] (2019) MRI Stroke
STRAINet [59] (2019) MRI Prostate
AnatomyNet [60] (2019) CT Head and Neck
248 L. Liu et al. / Neurocomputing 409 (2020) 244–258

DenseNet was embedded a more radical dense connection strat- images, which can reduce input image size and increase the recep-
egy: interconnecting all layers, specifically each layer accepts all tive field in the down-sampling process. However, image segmen-
the layers in front of it as its additional input. From feature point tation prediction is the output of pixel-wise (voxel-wise), it is
of view, by feature reusing and bypass settings, DenseNet not only necessary to restore small-sized images to the size of input image
greatly reduces the amount of network parameters, but also allevi- by up-sampling process. In the process of image size reduction and
ates the gradient vanishing problem when the back-propagation enlargement, some feature information is lost. The advantage of
flows in deep networks. dilated convolution is without pooling operations, which can alle-
DenseNet is a convolutional neural network with dense connec- viate the loss of feature information in down-sampling and up-
tions. Dense mechanism is the essence of this network. Dense sampling processes. Dilated convolution also enlarges the recep-
mechanism allows a network to obtain the maximum information tive field size, so that the output of each convolution contains a lar-
flow and the gradient flow. At the same time, this mechanism can ger range of feature information.
inhibit the problem of gradient vanishing in the process of iteration Compared with an ordinary convolution, there is an additional
and reduce the number of parameters when training networks. parameter named dilation rate in dilated convolution. Dilated rate
Dense block is the presentation of the dense mechanism. In a dense is mainly used to represent the size of expansion. The similarity
block, there is a direct connection between any 2 layers, this means between dilated convolution and ordinary convolution is that the
that input of each layer of network is the union of the output of all size of convolution core is the same. In a neural network that uses
the previous layers, and feature maps learned by this layer are dilated convolution instead of ordinary convolution, a number of
passed directly to all layers behind it as input. training parameters do not increase. However, dilated convolution
Embedding dense blocks into a network not only alleviates the can make the model obtain larger receptive field size and reduce
gradient vanishing problem, but also makes segmentation net- the loss of feature information, it will help to improve the perfor-
works much wider and deeper. In many studies of lesion segmen- mance of the model.
tations, dense blocks were embedded into U-shaped networks to Dilated convolution first comes from semantic image segmenta-
improve the segmentation performance. As shown in Table 2, Li tions [71]. Dilated convolution not only has a simple and elegant
et al. proposed a hybrid densely connected U-Net (H-DenseUNet) structure, but also helps to improve the performance of segmenta-
for liver and tumor segmentation [62]. H-DenseUNet inherits the tion models. As shown in Table 3, Nie et al. applied dilated convo-
advantages of both densely connected paths [61] and U-shaped lution in a novel deep network (STRAINet) to segment of pelvic
network [23]. H-DenseUNet realizes hierarchical aggregation vol- organs from MRI images [59]. Dilated convolution helps STRAINet
ume contexts by combining 2D and 3D DenseUNets. In addition, to gain a larger receptive field at the lowest cost of memory. In
Liu et al. embedded dense blocks into 2 symmetrical U-shaped net- addition, Dolz et al. embedded 2 different scale dilated blocks into
works [63]. They used multi-kernels to extract effective features a dense network for ischemic stroke lesion segmentations [70],
from sparse pixels, and verified the performance of the proposed they used multiple dilated convolutions to handle the variability
network on 2 open challenges: sub-acute ischemic stroke lesion size of lesions and prevent them from losing during training. Like-
segmentation (SISS) and acute stroke outcome/penumbra estima- wise, Tureckova et al. proposed a 3D U-shaped segmentation
tion (SPES). Similarly, Zhang et al. proposed a gland segmentation model that was derived from the model presented by Isensee
network, which consists of 3 different multi-scale dense connec- et al. [27]. Tureckova et al. used dilated convolution as input con-
tions with U-shaped architectures [64]. Likewise, some researchers volution of the network, which outperforms the original architec-
also embedded dense blocks and residual blocks into U-shaped ture in nearly all used evaluation metrics [72]. Furthermore,
architectures for medical image segmentations. For instance, Chen Vesal et al. [73] and Li et al. [56] also used the dilated convolution
et al. proposed a DRINet for 3 different segmentation tasks [43]. to segment left atrial and brain tumors, respectively.
DRINet inherits the architecture of U-Net, and has an analysis path Attention mechanism in deep learning originates from the
and a synthesis path. The analysis path is composed of the stack attention mechanism of human brain [75]. When human brain
dense blocks rather than the traditional convolution layers. The receives external information, such as visual information and audi-
synthesis path consists of residual blocks and unpooling blocks. tory information. Brain does not processes and understand all
In addition to the above extended models, the other models in information, but only focuses on some significant or interesting
Table 2 also show good performance in different organ segmenta- information, which helps to filter out the unimportant information
tion tasks. and improve the efficiency of information processing. In deep
learning methods, attention mechanism was first used to improve
3.3. Dilated mechanism the distinguish ability on the recurrent neural network (RNN).
Then it was widely used in many fields such as machine transla-
Dilated convolution was first introduced into the field of tion, speech recognition, image segmentation and so on [76–79].
semantic image segmentation by Yu et al. in 2015 [71]. Traditional In 2014, a recurrent model of visual attention (RAM) was proposed
CNNs perform convolution and pooling operations on the input by the Google Mind team, which attracted the attention from a
large number of researchers [75]. Subsequently, Bahdanau et al.
introduced attention mechanism into NLP field [76]. Then RNN
Table 2
models consisting of the attention mechanism are applied to vari-
Overview of papers using dense mechanism techniques in U-shaped networks.
ous NLP tasks [80].
Model name Image type Target Recently, how to use attention mechanism in medical image
Dense-CRFs [65] (2017) MRI Stroke segmentations has became a hot research topic. As shown in
DFCN [66] (2017) MRI Cardiac Table 4, Jin et al. were the first researchers to introduce attention
DRINet [43] (2018) CT, MRI CSF, Multi-organ, Brain tumor mechanism into medical image segmentations [40]. They proposed
DenseNet [67] (2018) MRI Brain tumor
3D-SkipDenseSeg [68] (2018) MRI Brain tumor
a 3D hybrid residual attention-aware segmentation method (RA-
DenseNet [69] (2018) MRI Brain tumor UNet). Attention mechanism was used in RA-UNet to extract con-
H-DenseUNet [62] (2018) MRI Intervertebral disc (IVD) textual information by combining low-level feature maps with
MDU-Net [64] (2018) EM images Neural structures high-level ones, which helps models to compensate the missing
MK-DCNN [63] (2019) MRI Stroke and penumbra
feature information. Similarly, Oktay et al. proposed a novel atten-
CNN [70] (2018) MRI Stroke and penumbra
tion gate (AG) model for segmenting abdominal organs from 2
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 249

Table 3 and a multi-scale 3D U-Net were used for brain tumor segmenta-
Overview of papers using dilated mechanism techniques in U-shaped networks. tions [84,86,87], respectively.
Model name Image type Target
CNN [72] (2019) CT Stroke 3.5. Ensemble mechanism
CSAU [74] (2019) Eye-fundus image Retinal vessel
RA-UNet [53] (2018) CT Liver tumor
In deep CNNs, ensemble mechanisms are also used to reduce
the over-fitting problem in the training process [89]. An ensemble
module consists of a set of parallel neural networks, which can pro-
Table 4
cess the same input data in parallel and finally their outputs are
Overview of papers using attention mechanism techniques in U-shaped networks.
combined to complete the segmentations [89]. It consists of a main
Model name Image type Target segmentation network and a pre-processing module or a post-
Attention U-Net [81] (2018) CT Liver and pancreas processing module. The pre-processing module is used to roughly
DRUnet [56] (2019) MRI Brain tumor process input images, the main segmentation network is used to
3D CNN [73] (2018) LGE-MRI Left atrial chamber
segment target lesions, and the post-processing module is used
RA-UNet [53] (2018) CT Liver tumor
STRAINet [59] (2019) MRI Prostate to refine segmentation results. Most times, the performance of
ensemble modules is better than that of a single subnet. In an
ensemble network, subnets can learn different attributes from
large CT medical datasets [81]. AG model can automatically learn the training data during the batch learning processing [90].
to focus on target of varying shapes and sizes. In addition, Li As shown in Table 6, there are various ensembles models which
et al. developed a connection sensitive attention U-Net (CSAU) consist of U-shaped networks and other models. For example,
for retinal vessel segmentation [74]. Before outputting predicted Kamnitsas et al. constructed an ensemble of multiple models and
result, CSAU integrated connection sensitive loss and AG concate- architectures (EMMA) for brain tumor segmentations [92]. EMMA
nated attention weights to features. Experiments illustrated that won the brain tumor segmentation challenge (BraTs) 2016. Simi-
this strategy can improve the accuracy on detailed vessels. larly, Li et al. presented a study using deep neural networks and
n U-shaped models to automatically detect WMH from multi-
modality MRI images [90]. In order to further study the segmenta-
3.4. Multi-module mechanism tion performance of ensemble models, Han et al. investigated 2
types of novel U-shaped architectures for medical image segmen-
Multi-module mechanism is a general term for a class of meth- tations [96]. One is a dual frame U-shaped architectures that can
ods, which uses multiple neural networks to improve segmenta- get residual signals in the low-resolution path. Other is a tight
tion performance. It is widely used in many deep learning frame U-shaped architecture with the orthogonal wavelet frame.
methods [63,64,82–86]. These methods are usually structured in Both architectures exhibited better performance than that of a
containing multi-scale or multi-kernel formats, which contain standard U-Net [95]. In addition, Jin et al. proposed an end-to-
encode informative and discriminative features covering multiple end Deformable U-Net (DUNet) for retinal vessel segmentations
granularity levels. The internal representations of these methods [40], which exploited the local features of retinal vessels with a
have different regions of interest, and allow the construction of U-shaped architecture. DUNet captures the retinal vessels at vari-
multi-grained feature information. ous shapes and scales by adaptively adjusting the receptive fields
Table 5 shows some multi-module U-shaped networks which according to the scales and shapes of vessels. Likewise, Fu et al.
were used in medical image segmentation field. Multi-module net- proposed an ensemble deep learning system for the cup of glau-
works can process high and low resolution images at the same coma segmentation [99]. They introduced the polar transformation
time, which helps networks to obtain complementary feature technology to the ensemble system, and combined with a multi-
information. These segmentation networks are usually used to scale U-shaped convolutional network.
conjunct with other mechanisms. For example, Zhang et al. pro-
posed a multi-scale U-shaped network for gland segmentations
[64], which also consists of some dense convolutions. The proposed 4. Public datasets
network can directly fuse the neighboring different scale feature
maps from both higher layers and lower layers, which can There are lots of medical image segmentation methods pro-
strengthen the feature propagation in current layer and reduce posed during last years. However, some datasets used in these
over-fitting problem. Similarly, an end-to-end multi-kernel deep methods are not available. In order to get a deeper understanding,
convolution neural network (MK-DCNN) was proposed for stroke we discuss some of the most widely used public datasets on med-
MRI segmentation [63]. MK-DCNN consists of 2 sub-densenets that ical image segmentations, which are summarised in Table 7 (Med-
with different convolution kernels. MK-DCNN can extract more ical datasets correspond to websites are shown in Appendix A). All
image features than that of the single DCNN, which helps improve datasets listed in Table 7 and each dataset provides the ground
the performance of the model. In addition, 2 cascade of 3D U-Nets truth. These datasets can be divided into 2 parts according to the
nature of the data: 2D or 3D. X-ray images are in 2D from and most
CT and MRI images are in 3D form.
Table 5 The traditional segmentation methods frequently use 2D
Overview of papers using Multi-module mechanism techniques in U-shaped images as inputs, such as threshold method [103–105], fuzzy tech-
networks.
nique [106], modified context-sensitive gaussian mixture model
Model name Image type Target [107], adaptive outlier detection method [108], morphological seg-
CNNs [70] (2018) MRI Stroke and penumbra mentation method [109], and region-growing technique [110].
CNNs [84] (2019) MRI Brain tumor These segmentation methods only consider the information in a
V-Net [87] (2019) MRI Brain tumor single image and treat each 2D images as a separate sample. Most
MS-DCNN [88] (2018) CT Stroke
deep learning segmentation methods also used 2D medical images
MK-DCNN [63] (2019) MRI Stroke and penumbra
Multi-scale Masked U-Net [86] (2018) MRI Brain tumor as inputs [23,42,43,47,57,111]. This is a simple way to segment
medical images. With the advent of computer hardware and GPU
250 L. Liu et al. / Neurocomputing 409 (2020) 244–258

Table 6 parameters for training 3D network is much more than that for
Overview of papers using Ensemble mechanism techniques in U-shaped networks. training 2D networks, which leads to the large amount of time
Model name Image type Target required for training models. In addition, processing 3D images
M-net [91] (2017) MRI CSF requires high-performance GPU and larger video memory server,
EMMA [92] (2017) MRI Brain tumors which are also the challenge of 3D image segmentations. Therefore,
UNet++ [93] (2018) CT Liver, lung nodule it is necessary to cut 3D images into 3D patches, then apply 3D
Two-stream U-Net [94] (2018) EM images Neural structures CNNs on each patch [121–126]. The advantage is that the global
DUNet [40] (2018) Photographs Retinal vessel
3D U-Nets [95] (2019) MRI Gliomas brain tumors
slice sequence information can be taken into account, while the
Dual and tight U-Nets [96] (2018) CT Abdomen disadvantage is that they lack the pre-trained model parameters
Ensemble FCNs [90] (2018) MRI WMH for 3D CNNs.
InfiNet [97] (2018) MRI CSF 2D images and 3D images have their own advantages in seg-
Multi-stream 3D U-Net [98] (2018) MRI Prostate
mentation tasks. In the same segmentation task, using 3D images
M-Net [99] (2019) Images Glaucoma
3D-ESPNet [100,101] (2019) MRI Brain tumors as input can result in slightly better segmentation than using 2D
MHL [102] (2019) DCE-MRI Breast tumor images, but it takes longer training time, and needs much powerful
hardware devices. In the study, whether to use 2D images or 3D
images as the input of the segmentation models depends on the
Table 7 nature of images and the condition of the hardware devices.
Overview of medical datasets used in U-shaped networks.

Dataset Sample number, Image types Target 5. Application areas

BraTS 2015 274/110, MRI (3D) Brain tumor
BraTS 2016 200/191, MRI (3D) Brain tumor Based on these datasets introduced in the previous section, in
BraTS 2017 285/146, MRI (3D) Brain tumor this section, we present an overview of the U-shaped networks
BRATS 2018 285/191, MRI (3D) Brain tumor
applications to the various areas in medical imaging. We highlight
MRBrainS13 36/15, MRI (3D) WM, GM and CSF
MRBrainS18 7/23, MRI (3D) WM, GM and CSF some key contributions and discuss the performance in different
SPES 2015 30/20, MRI (3D) Acute stroke lesion application areas. The main application areas are shown in Fig. 3.
SISS 2015 28/36, MRI (3D) Ischemic stroke
lesion
ISLES2016 35/40, MRI(3D) Ischemic stroke 5.1. Brain
lesion
ISLES2017 43/32, MRI(3D) Ischemic stroke As shown in Tables 1–6, U-shaped networks have been exten-
lesion
sively used for brain image segmentations in several different
ISLES2018 63/40, CT(3D) Ischemic stroke
lesion
application domains, such as: brain tumors, stroke lesions and
iSeg 2017 10/13, MRI (3D) WM, GM and CSF WMH lesions. The challenges for brain tissue segmentations in
WMH 2017 60/110, MRI(3D) WMH medical images are twofold: 1. Segmentation methods cannot pre-
IBSR 18, MRI (2D) NITRC cisely segment the boundaries of different lesions. 2. Patients often
Head&Neck 25/15, CT (2D) Head and Neck
have other diseases, which have similar intensities to target
ACDC 2017 100/50, MRI (3D) Cardiac
DSB 2015 191612/171920, MRI (2D) Cardiac lesions. However, the U-shaped segmentation networks have
LIDC-IDRI 244527, CT (2D) Lung achieved remarkable results in brain tissue segmentations.
LowDoseCT 10/20, CT (2D) Low dose abdomen
PROMISE 50/30, MRI (3D) Prostate
2012 5.1.1. Brain tumor
LiTS 2017 130/70, CT (2D) Liver tumor Brain tumor is one of the most dreadful types of cancer. The
ISIC 2017 2000/600, Dermoscopic image (2D) Skin noninvasive evaluation of brain tumors can provide valuable infor-
ISIC 2018 2594/1000, Dermoscopic image Skin
mation for diagnosis and prognosis. Due to unpredictable appear-
(2D)
GlaS 85/80, Histology image (2D) Gland
ance and shape of brain tumors, accurate segmentation of brain
NIH-TCIA 19328, CT (3D) Pancreas tumors from multi-modality medical images is one of the most
IVD 2018 8/6, MRI (3D) Intervertebral disc challenging tasks in medical image studies.
STARE 400, Photograph (2D) Retina From 2012 to 2018, Medical Image Computing and Computer
HRF 45, Photograph (2D) Retina
Assisted Intervention (MICCAI) provides a series of available data-
IDRiD 516, Fundus images (2D) Retinal
ORIGA 650, Fundus images (2D) Glaucoma sets about brain tumor in MRI scans (BraTS 2012–2018). Many
ISBI 2012 30/30, Microscopy images (2D) Neuronal researchers have devoted themselves to this research. For instance,
DRIVE 20/20, Photograph (2D) Blood vessels Dong et al. proposed a 3D U-shaped network on BraTS 2015 chal-
lenge [127], which obtained the outstanding segmentation perfor-
mance. Similarly, Isensee et al. developed a 3D network that was
servers, more and more researchers use 3D medical images as also inspired by the U-shaped network. The proposed network
input of segmentation methods. There are 2 main ideas for using not only achieved the stat-of-the-art on BraTS 2015, but also was
3D medical images as deep learning methods inputs: (1) Convers- one of the top ranked methods on BraTS 2017 [27]. In addition,
ing the 3D image into 2D image slices, then using 2D slices as in 2017, Kamnitsas et al. proposed a dual pathway 3D CNN for
inputs of segmentation methods [46,50,63,88,112–115]. Finally brain tumor segmentations in BraTS 2015 challenge [122]. They
the predicted 2D results are stitched into 3D form. This approach employed the dual-pathway architecture to obtain multiple scales
has a drawback is that although 2D slice makes full use of the feature information and used fully connected conditional random
information in the entire 2D slice, it ignores the relationship field to remove false positives. The proposed network was the
between several adjacent slices and loses the correlation between top ranked methods on BraTS 2015. Likewise, Hussain et al. pro-
global slice sequences. (2) Using 3D images as inputs [26,73,116– posed a 3D DCNN segmentation network and evaluated it on BraTS
120]. Compared with using 2D images, using 3D images as input 2013 and BraTS 2015 challenge [128]. In all above studies, the
can obtain more semantic information, which helps improve the input MRI images are in 3D. As far as we know, there is no 2D seg-
performance of segmentation models. However, the number of mentation network for these brain tumor segmentation challenges.
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 251

Fig. 3. Collage of several medical imaging applications with U-shaped networks. The first and third columns are original images, and the second and fourth columns are
ground truths.

5.1.2. Stroke and penumbra model was the top ranked method in ISLES 2017 challenge. Song
Stroke is the second cause of death and disability worldwide. et al. proposed a 3D multi-scale U-Net that obtained the top ranked
Ischemic stroke is the main sub-type of stroke diseases. It is often in ISLES 2018 challenge [132]. Although these challenges were very
caused by local thrombosis or embolic causes. Penumbra is a lesion successful, but the best performance segmentation methods was
tissue that surrounds stroke lesion. Correct assessment of the pres- unsatisfactory in these challenges. For example, the presence of
ence, location, extent and evolution of stroke and penumbra similar lesions in the limited number of images the performance
lesions is great significance for treatment and prognosis. As shown of the models [129]. How to cross this gap is still one of the chal-
in Table 7, Ischemic Stroke Lesion Segmentation (ISLES) challenge lenges we should focus on.
provides 5 public stroke image datasets for researchers and a fair
and direct platform for segmentation methods. 5.1.3. WMH
ISLES 2015 provides 2 sub-challenges (SPES and SISS). SPES is a White matter hyperintensities (WMH) is one of the main small
challenge that aims to segment penumbra lesion of the acute vessel disease in brain [133]. A dedicated review about automated
stroke, while SISS is a challenge that aims to segment stroke lesion WMH segmentation techniques on medical image analysis [134]
of the ischemic stroke. CH-Insel (the team of McKinley et al.) and has noted that segmentation and quantification of WMH volume
UK-Imp2 (the team of Konstantinos et al.) are the top ranked meth- plays an importance role in clinical studies and practices. Correct
ods on 2 sub-challenges [129], respectively. In addition, the net- assessment of WMH also is a way to support diagnosis, prognosis,
works proposed by participating teams lianl1 and clera1 showed and monitor of treatment for dementia and other neurovegetative
the best robustness on both sub-challenges [63]. In 2016, Choi diseases.
et al. proposed an ensemble segmentation model that consisted As show in Table 7, there are a limited number of public data-
of 2 deep convolutional neural networks. The ensemble model sets about WMH segmentations. Where WMH challenge 2017 pro-
inherited the advantages of FCN and U-Net [130], which was the vides a prominent WMH segmentation platform. In 2017, an
top ranked method in ISLES 2015 challenge. Furthermore, Choi extended U-shaped network was used to automatically detect
et al. proposed another ensemble segmentation model that con- and segment WMH from Flair and T1 MRI images [90]. This model
sists of residual U-Net [130] and spatial pyramid pooling [131], this was evaluated and ranked the first in the WMH segmentation chal-
252 L. Liu et al. / Neurocomputing 409 (2020) 244–258

lenge. Similarly, Zhang et al. used an extended U-shaped network 5.3. Cardiac
to accomplish WMH segmentation in the same challenge [135].
They used randomly initialized weights to improve the segmenta- In medical image analysis, accurate location and segmentation
tion performance. In order to study WMH deeply, Guerrero et al. of abdominal organs has always been one of the themes of medical
proposed an uResNet to segmentation and differentiation between image segmentations. The high mortality rate of cardiovascular
WMH and stroke lesions [46]. The extracted WMH volumes were diseases has always been one of the hot research topics, especially
found to correlate with the Fazekas visual rating score, which is segmentation of left ventricle (LV) organ.
very meaningful work for clinicians to study these 2 kinds of In all cardiovascular segmentation methods, U-shaped architec-
lesions. In addition, Liu et al. [136] also used the deep CNN model tures were used by all top performing models. In automated car-
to segment WMH tissue from stroke MRI images. Although these diac diagnosis challenge (ACDC) 2017, the most of top-ranking
studies yielded good results in WMH segmentations, but they segmentation methods are related to U-shaped networks
ignored to distinguish the similar lesions from WMH, such as [66,119,120,140–142]. For example, Poudel et al. proposed an U-
stroke lesions and multiple sclerosis lesions. shaped network that segmented LV from full stack of 2D slices in
MICCAI 2009 LV Segmentation Challenge [143]. Similarly, Veasl
et al. proposed a 3D U-shaped CNN for volumetric segmentation
5.1.4. Other
of the LV chamber in STACOM MICCAI 2018 challenge[73]. In addi-
Segmentation of normal brain structures is also an important
tion, Isensee et al. introduced an nnU-Net (‘‘no-new-Net”), which is
research direction. Some studies focus on the segmentation of
a robust and self-adapting framework on basis of 2D and 3D vanilla
white matter (WM), gray matter (GM) and cerebrospinal fluid
U-Nets [28]. Compared with the state-of-the-art methods, nnU-Net
(CSF), which is critical to study the early brain development in
achieved the highest mean dice scores in a heart disease in–house
infants or neurodevelopmental or neuropsychiatric disorders, such
dataset.
as schizophrenia and autism. More and more attention has been
paid to these 3 brain tissues.
5.4. Liver
MICCAI have provided 3 relevant challenges: MRBrainS13,
iSeg2017 and MRBrainS18. The U-shape-related segmentation
Liver cancer is a disease with a high incidence and the relevant
methods perform well on these 3 segmentation challenges. For
medical images are readily available. There are massive amounts of
example, Chen et al. proposed a segmentation model (VoxResNet)
data available for the demand of deep learning model. Deep learn-
for the brain segmentation on 3D MRI images [45]. VoxResNet
ing methods have shown output performance on liver and tumor
achieved the first place in the challenge out of 37 competitors
segmentation tasks. Among these models, U-shaped network is
including several state-of-the-art brain segmentation methods.
the most used one.
Moreover, Li et al. adopted deep dilated a residual U-shaped net-
LiTS 2017 is a public challenge that about liver and tumor in CT
work to segment brain tissues from multi-modality MRI images
scans. It provides a fair competition platform for many medical
in MRBrainS18 [56]. Similarly, Luna et al. also provided a 3D
image segmentation methods. Deep learning segmentation meth-
patch-wise U-shaped network in MRBrainS18 [137]. In addition
ods perform well in this challenge. Especially H-DenseUNet [62]
to public datasets, Chen et al. used DRINet method to segment
and RA-UNet [53] segmentation methods outperformed other
the CSF tissues on 2D CT images from an in-house dataset [43].
state-of-the-art segmentation methods. These 2 methods achieved
Moreover, multiple sclerosis [138], diencephalon [91] and others
very competitive performance on liver segmentations. In addition,
are also the subjects of brain imaging research.
a novel UNet++ segmentation model was proposed in LiTS 2018
challenge [93]. UNet++ was also used in nuclei and polyp segmen-
5.2. Eye tation tasks. Results showed that UNet++ could achieve better per-
formance than U-Net.
Fundus image is a specialized form of medical images. It is
taken by fundus camera operated. Retinal fundus images are 5.5. Prostate
mostly stored in the form of 2D images. Ophthalmologists can
diagnose eye diseases with retinal fundus images such as: glau- Prostate images most come from radiography or computed
coma, optic disc, blood vessel and retina. Diagnosis of eye diseases tomography. It is an easily accessible medical kind of diagnosis
is not only time-consuming but also inefficient. Therefore, using images. There are a large number of unlabeled prostate images that
computer technologies to segment eye tissues for early screening need to be processed by clinicians [144]. It is a tedious task for clin-
and detection of eye diseases is critical to maintain healthy vision icians to locate and mark the shape of target tissues in images.
and quality of life. With the development of deep learning in the field of image anal-
U-shaped networks have shown its excellence in medical image ysis, more and more researchers try to use deep learning methods
segmentation tasks, it also attracts more attention compared with to assist in localization and segmentation of prostate lesions. U-
the traditional CNN due to its ability to obtain a coarse-to-fine rep- shaped segmentation models is one of them.
resentation. For example, Fu et al. [99] proposed a deep U-shaped PROMISE12 is a well know challenge that is used to compare
network (M-Net) for segmenting optic disc (OD) and optic cup (OC) interactive and (semi)-automatic segmentation algorithms of pros-
on ORIGA dataset [139]. The experiments showed that M-Net not tate from MRI images. In this challenge, top ranked methods taken
only achieved the state-of-the-art segmentation result, but also by a 3D U-shaped segmentation method fCNN, which was pro-
could diagnose glaucoma. Similarly, Alom et al. developed 2 recur- posed by Yu et al. [118]. The fCNN model is a hybrid network
rent convolutional neural networks based on the U-shaped archi- which consists of ResNet and U-net architecture. In addition, by
tecture [44] and applied them to blood vessel segmentations in using the volumetric convolutional strategy, a 3D segmentation
retina images. In addition, Li et al. developed a connection sensitive method (V-Net) also achieved good performances in PROMISE12
attention U-shaped network (CSAU) for vessel segmentations from challenge [26]. Furthermore, Drozdzal et al. proposed a 3D FC-
retinal fundus images [74]. CSAU achieved the leading position on ResNet network for liver lesions and prostate segmentations on
all 3 public fundus image datasets (DRIVE, STARE and HRF). Like- in–house dataset and PROMISE12, respectively [47]. Different from
wise, Jin et al. also confirmed the good performance of their pro- above studies, Chen et al. proposed a bridge U-shaped network for
posed segmentation model on these 3 fundus image datasets [40]. prostate segmentation which is based on 2D images [111]. In
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 253

PROMISE12 challenge, they converted 3D prostate MRIs to 2D 2018 challenge. Furthermore, Kawahara et al. also made outstand-
slices. This proposed network not only got the highest score among ing achievement in an in-house dataset of skin cancer [147]. In
2D methods, but also performed better than the contrastive 3D addition, Codella et al. provided a review of the ISIC challenge
methods in 2 metrics. [148]. They affirmed that the advance of deep learning methods
on skin segmentation tasks, at the same time, they also pointed
5.6. Musculoskeletal (intervertebral disc) out that these methods generate little interpretable evidence of
disease diagnosis.
Accurate localization and segmentation of musculoskeletal are
crucial for the assessment of bone diseases. Deep learning methods
5.9. Other
have been extensively used for musculoskeletal image analysis in
several different application domains: bone, spinal and interverte-
In addition to above studies, this final section briefly describes
bral discs. These images always show in X-ray or MRI images.
the application of U-shaped networks in other diseases. It is
Table 7 shows some public datasets.
remarkable that U-Nets or U-shaped networks can be applied in
The latest public musculoskeletal dataset is IVDM3Seg, which is
different medical images segmentation tasks, which marks the ver-
published in MICCAI 2018. The goal of IVDM3Seg is to provides a
satility and applicability of U-shaped networks.
standard evaluation framework for investigating (semi-) automatic
In GlaS challenge of MICCAI 2015, Li et al. proposed a bi-
intervertebral discs (IVDs) localization and segmentation algo-
directional recurrent U-Net based on probabilistic map guidance
rithms. Dolz et al. extended the U-Net architecture and proposed
(PBR-UNet) and used the proposed method to segment pancreatic
IVD-Net for localization and segmentation of IVDs from multi
cancer [149]. Zhang et al. proposed an ensemble U-shaped network
MRI images in IVDM3Seg [145]. They demonstrated that naive fea-
for pancreatic cancer segmentation [64]. The proposed network
ture fusion strategies in IVD-Net can improve the representation
consisted of 3 different multi-scale dense connections. In addition,
power and boost discriminative performance. Similarly, on the
Chen et al. proposed a U-SEG-NET segmentation network for hip-
same dataset, Li et al. also proposed a segmentation network for
pocampus segmentations [150]. U-SEG-NET combined U-net with
IVDs from multi-scale and modality MRI images [124], the pro-
multiple decision maps. The performance of U-SEG-NET was vali-
posed method achieved the first place in the IVDM3Seg 2018
dated by the brain dataset of ADNI project. Furthermore, Zhu
challenge.
et al. applied an improved 3D U-Net (AnatomyNet) to the segmen-
tation of head and neck (HAN) cancer organs in MICCAI challenge
5.7. Neuronal
2015 [60]. AnatomyNet was able to process whole-volume CT
images and delineate all organs-at-risks in one pass, which only
Neuronal segmentation is an important part of medical image
need a few pre- or post-processing operations. In addition, U-
segmentations. U-Net was firstly evaluated on neuronal dataset
shaped networks were also used in polyp and cell nuclei segmen-
(ISBI 2012) [23]. ISBI challenge 2012 is the first challenge that aims
tation task [93], kidney segmentation task [151], lung segmenta-
to segment neurons on 2D EM images. Since then, a wide variety of
tion task[44], and breast tumor segmentation task [102] and so on.
depth learning segmentation methods used this challenge as the
benchmark.
In ISBI 2012 challenge, a residual deconvolutional U-shaped 6. Discussion
network (RDN) was proposed for 2D EM image segmentations
[42], which consisted of 2 information pathways that captured From more than 100 papers reviewed in this survey, it can be
full-resolution features and contextual information, respectively. clearly seen that U-shaped networks have played a significant role
RDN achieved one of the top results on this challenge. Further- in the study of medical image segmentations. U-shaped networks
more, Drozdzal et al. provided a pipeline which was combined have made a profound impact on current medical image segmenta-
FCNs with FC-ResNets for medical image segmentations [47]. Com- tion research. As shown in Fig. 1, the number of related papers has
pared with other 2D methods, the proposed method exhibited the grown very rapidly. Among them, combination of U-shaped net-
state-of-the-art performance on EM images challenging. In addi- works with residual, dense and dilated mechanisms not only deliv-
tion, R2U-Net [52], MultiResUNet [49] and CE-Net [57] and so on ered a variety of deep networks, but also promoted the
also showed outstanding performance in ISBI 2012 challenge. performance of medical image segmentation. It can be found that
in the past 3 years, the end-to-end CNN networks with U-shaped
5.8. Skin cancer architectures have become the preferred method for medical
image segmentations.
Skin cancer is the most common malignant tumour in human After reviewing of these papers, we hope to provide the com-
beings. The degree of skin lesions in dermatoscopy images varies prehensive information for developing better U-shaped segmenta-
greatly. At present, diagnosis of benign and malignant skin diseases tion methods for each individual lesion or organ segmentation. The
is mainly performed by doctors on dermatoscopy images. This is a derived U-shaped networks are outstanding in many medical
time-consuming and error-prone process. In 2017, the researchers image segmentation challenges, 2 significant conclusions we can
in Stanford University showed that deep neural networks could draw are: 1. The same network is difficult to apply to the segmen-
analyze skin cancer, which has greatly inspired the research in this tation of multiple tasks, and the generalization performance of the
field [146]. More and more deep learning methods are applied to segmentation methods needs to be improved. For example,
this field. embedding residual mechanism into U-Net network is an effective
In the field of skin lesion segmentations, ISIC 2017 and 2018 are way to segment the individual lesion, but the results are quite dif-
2 public challenges which concern skin cancer. Alom et al. ferent in other segmentation tasks with the same architecture and
proposed a recurrent residual convolutional neural network parameters. 2. Increasing the depth of network structures can
(R2U-Net) for skin image segmentations. R2U-Net have made an improve network performance, but the precise structure is not
impressive achievement in ISIC 2017 challenge. Similarly, Ibtehaz the most important determinant. The prior knowledge of doctors
et al. developed a novel architecture MultiResUNet as the potential on medical images is one of the important keys. These prior knowl-
successor to successful U-Net architecture [49]. Compared with the edge can provide advantages to improve the performance of seg-
U-Net, MultiResUNet obtained a remarkable performance in ISIC mentation methods. In addition, data augmentation methods are
254 L. Liu et al. / Neurocomputing 409 (2020) 244–258

also an aid way to improve the performance of segmentation net- Quan Quan: Investigation. Fang-Xiang Wu: Writing - review &
works, such as increasing the number of medical images and editing. Yu-Ping Wang: Writing - review & editing. Jianxin Wang:
enhancing image quality by image pre-processing methods. Conceptualization, Funding acquisition, Writing - review & editing.
Of course, data augmentation and preprocessing are not the only
key factors for a good solution. More and more researchers are using Acknowledgment
hybrid architectures instead of single architecture to achieve better
segmentation results. For example, multi-scale, multi-modality and The work described in this paper was supported by the National
multi-model are integrated into segmentation methods. More con- Natural Science Foundation of China under Grant Nos. U1909208,
textual information can be obtained by controlling the size or 61772552; the 111 Project (No.B18059); the Hunan Provincial
modality of input images, or the same image information can be Science and Technology Program (2018WK4001); the Natural
processed by multiple models and then be merged to obtain better Science Foundation of Hunan Province in China (2018JJ2534).
feature maps. It is also an effective way to improve model perfor-
mance by using 3D images instead of 2D images as data source. Appendix A
In addition, most researchers proposed U-shaped methods
which retains the architecture of the original U-Net with 4 levels Overview of medical datasets and websites.
of down-sampling and 4 levels of up-sampling. The improvements
on original U-Net architecture have also been made. For example, Dataset Website
Zhou et al. [93] used redesigned skip paths and deep supervision
[152] to improve segmentation performance. The proposed method BraTS 2015 https://www.med.upenn.edu/sbia/brats2015.
was validated on 4 different medical segmentation tasks. The html
experiments illustrated that the design of network architectures BraTS 2016 https://www.med.upenn.edu/sbia/brats2016.
need to consider specific situation of segmentation tasks. For most html
simple segmentation tasks, a segment network does not need to be BraTS 2017 https://www.med.upenn.edu/sbia/brats2017.
very deep or complex, and a very wide network can achieve good html
performance. However, more complex segmentation tasks require BRATS 2018 https://www.med.upenn.edu/sbia/brats2018.
deeper and more complex networks to achieve satisfactory results. html
Finally, U-shaped networks and other deep learning methods MRBrainS13 https://mrbrains13.isi.uu.nl/index.php
lack the interpretability of the ‘‘black box” problem. Although the MRBrainS18 https://mrbrains18.isi.uu.nl/
performance of some segmentation methods have reached or SPES 2015 http://www.isles-challenge.org/ISLES2015/
exceeded that of radiologists in some areas, radiologists and SISS 2015 http://www.isles-challenge.org/ISLES2015/
patients do not intuitively understand the operating mechanisms ISLES2016 http://www.isles-challenge.org/ISLES2016/
of models, which makes them suspicious to deep learning meth- ISLES2017 http://www.isles-challenge.org/ISLES2017/
ods. In order to solve this problem, some researchers have devel- ISLES2018 http://www.isles-challenge.org/
oped strategies to interpret the operating mechanism of the iSeg 2017 http://iseg2017.web.unc.edu/
inner layers of neural networks, such as deconvolution networks WMH 2017 http://wmh.isi.uu.nl/
[153], guided back-propagation [154]. Some researchers combined IBSR https://www.nitrc.org/projects/ibsr/
images with electronic medical records, and they tried to use tex- Head&Neck http://www.imagenglab.com/newsite/pddca/
tual information in electronic medical records to explain the inter- ACDC 2017 https://www.creatis.insa-lyon.fr/Challenge/
nal mechanisms of networks [155,156]. Other researchers have acdc/index.html
tried to visualize the hidden layers of neural networks, and they DSB 2015 https://www.kaggle.com/c/s-annual-data-
used visual intermediate layer images to explain image processing science-bowl/overview
in neural networks [157]. These methods allow radiologists to LIDC-IDRI https://wiki.cancerimagingarchive.net/display/
assess the reliability of network and accelerate the acceptance of Public/LIDC-IDRI
deep learning applications in the process of clinical auxiliary diag- LowDoseCT https://www.aapm.org/grandchallenge/
nosis. The interpretability of deep learning networks in medical lowdosect/
image segmentation field is unexplored and needs to be further PROMISE https://promise12.grand-challenge.org/
improved in future. 2012
Due to the complexity of medical image segmentation tasks and LiTS 2017 https://competitions.codalab.org/competi
the variety of target tissues, current segmentation methods are dif- tions/15595
ficult to be fully used for clinical diagnosis. It is not feasible to use ISIC 2017 http://challenge2017.isic-archive.com/
fully automatic, reliable and accurate segmentation methods to ISIC 2018 https://challenge2018.isic-archive.com/task1/
completely replace the work of physicians and clinical researchers. GlaS https://warwick.ac.uk/fac/sci/dcs/research/tia/g
Instead, according to different segmentation tasks, physicians and lascontest
clinical researchers can choose appropriate segmentation method NIH-TCIA https://www.cancerimagingarchive.net/
to assist diagnosis: In particular, clearly outlined, large lesions IVD 2018 https://ivdm3seg.weebly.com/
are already segmented with good results, which are usually tedious STARE http://cecas.clemson.edu/ ahoover/stare/
to outline by hand. For smaller and less pronounced lesions the HRF http://www5.cs.fau.de/research/data/fundus-
manual approach is still recommended. images/
IDRiD https://idrid.grand-challenge.org/
ORIGA http://www.moh.gov.sg/mohcorp/publications.
aspx?id = 16320
CRediT authorship contribution statement
ISBI 2012 http://brainiac2.mit.edu/isbi_challenge/
DRIVE http://www.isi.uu.nl/Research/Databases/
Liangliang Liu: Conceptualization, Data curation, Formal
DRIVE/
analysis, Writing - original draft. Jianhong Cheng: Investigation.
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 255

References [27] F. Isensee, P. Kickingereder, W. Wick, M. Bendszus, K.H. Maier-Hein, Brain

tumor segmentation and radiomics survival prediction: contribution to the
brats challenge, in: International MICCAI Brainlesion Workshop, Springer,
[1] T. Mcinerney, D. Terzopoulos, Deformable models in medical image analysis:
2017, pp. 287–297.
a survey, Medical Image Analysis 1 (2) (1996) 91–108.
[28] F. Isensee, J. Petersen, A. Klein, D. Zimmerer, P.F. Jaeger, S. Kohl, J. Wasserthal,
[2] A.L. Martel, S.J. Allder, G.S. Delay, P.S. Morgan, A.R. Moody, Measurement of
G. Koehler, T. Norajitra, S. Wirkert, et al., nnu-net: Self-adapting framework
infarct volume in stroke patients using adaptive segmentation of diffusion
for u-net-based medical image segmentation, arXiv preprint
weighted mr images, in: International Conference on Medical Image
arXiv:1809.10486.
Computing & Computer-assisted Intervention, 1999.
[29] F. Isensee, J. Petersen, S.A. Kohl, P.F. Jäger, K.H. Maier-Hein, nnu-net: Breaking
[3] A.B. Neumann, K.Y. Jonsdottir, M. Kim, H. Niels, G. Carsten, B. Alberto, F. Jens,
the spell on successful medical image segmentation, arXiv preprint
G. Roberto, J.H. Gillard, H. Marc, Interrater agreement for final infarct MRI
arXiv:1904.08128.
lesion delineation, Stroke; A Journal of Cerebral Circulation 40 (12) (2009)
[30] T. Falk, D. Mai, R. Bensch, Ö. Çiçek, A. Abdulkadir, Y. Marrakchi, A. Böhm, J.
3768–3771.
Deubner, Z. Jäckel, K. Seiwald, et al., U-net: deep learning for cell counting,
[4] H.M. Wheeler, M. Mlynash, M. Inoue, A. Tipirneni, J. Liggins, G. Zaharchuk, M.
detection, and morphometry, Nature Methods 16 (1) (2019) 67.
Straka, S. Kemp, R. Bammer, M.G. Lansberg, et al., Early diffusion-weighted
[31] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training
imaging and perfusion-weighted imaging lesion volumes forecast final infarct
by reducing internal covariate shift, arXiv preprint arXiv:1502.03167.
size in defuse 2, Stroke 44 (3) (2013) 681–685.
[32] J. Lei Ba, J.R. Kiros, G.E. Hinton, Layer normalization, arXiv preprint
[5] G.W. Albers, V.N. Thijs, L. Wechsler, S. Kemp, G. Schlaug, E. Skalabrin, R.
arXiv:1607.06450.
Bammer, W. Kakuda, M.G. Lansberg, A. Shuaib, et al., Magnetic resonance
[33] D. Ulyanov, A. Vedaldi, V. Lempitsky, Instance normalization: The missing
imaging profiles predict clinical response to early reperfusion: the diffusion
ingredient for fast stylization, arXiv preprint arXiv:1607.08022.
and perfusion imaging evaluation for understanding stroke evolution (defuse)
[34] Y. Wu, K. He, Group normalization, in: Proceedings of the European
study, Annals of Neurology: Official Journal of the American Neurological
Conference on Computer Vision (ECCV), 2018, pp. 3–19.
Association and the Child Neurology Society 60 (5) (2006) 508–517.
[35] V. Nair, G.E. Hinton, Rectified linear units improve restricted boltzmann
[6] A. Kemmling, F. Flottmann, N.D. Forkert, J. Minnerup, W. Heindel, G.
machines, in: Proceedings of the 27th International Conference on Machine
Thomalla, B. Eckert, M. Knauth, M. Psychogios, S. Langner, et al.,
Learning (ICML-10), 2010, pp. 807–814.
Multivariate dynamic prediction of ischemic infarction and tissue salvage as
[36] A.L. Maas, A.Y. Hannun, A.Y. Ng, Rectifier nonlinearities improve neural
a function of time and degree of recanalization, Journal of Cerebral Blood Flow
network acoustic models, in: Proc. ICML, vol. 30, 2013, p. 3.
& Metabolism 35 (9) (2015) 1397–1405.
[37] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing
[7] M.G. Lansberg, M. Straka, S. Kemp, M. Mlynash, L.R. Wechsler, T.G. Jovin, M.J.
human-level performance on imagenet classification, in: Proceedings
Wilder, H.L. Lutsep, T.J. Czartoski, R.A. Bernstein, et al., MRI profile and
of the IEEE International Conference on Computer Vision, 2015, pp. 1026–
response to endovascular reperfusion after stroke (defuse 2): a prospective
1034.
cohort study, The Lancet Neurology 11 (10) (2012) 860–867.
[38] B. Xu, N. Wang, T. Chen, M. Li, Empirical evaluation of rectified activations in
[8] I. Arel, D.C. Rose, T.P. Karnowski, Deep machine learning – a new frontier in
convolutional network, arXiv preprint arXiv:1505.00853.
artificial intelligence research [research frontier], Computational Intelligence
[39] D.S. Shepard, Computer mapping: the symap interpolation algorithm, Spatial
Magazine IEEE 5 (4) (2010) 13–18.
Statistics & Models 40 (1984) 133–145.
[9] Y. Lecun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) 436.
[40] Q. Jin, Z. Meng, T.D. Pham, Q. Chen, L. Wei, R. Su, Dunet: A deformable
[10] K. Fukushima, Neocognitron: A self-organizing neural network model for a
network for retinal vessel segmentation, arXiv preprint arXiv:1811.01206.
mechanism of pattern recognition unaffected by shift in position, Biological
[41] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition,
Cybernetics 36 (4) (1980) 193–202.
in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.
[11] S.B. Lo, S.A. Lou, J.S. Lin, M.T. Freedman, M.V. Chien, S.K. Mun, Artificial
770–778.
convolution neural network techniques and applications for lung nodule
[42] A. Fakhry, T. Zeng, S. Ji, Residual deconvolutional networks for brain electron
detection, IEEE Transactions on Medical Imaging 14 (4) (1995) 711–718.
microscopy image segmentation, IEEE Transactions on Medical Imaging 36
[12] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al., Gradient-based learning
(2) (2017) 447–456.
applied to document recognition, Proceedings of the IEEE 86 (11) (1998)
[43] L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, D. Rueckert, Drinet for
2278–2324.
medical image segmentation, IEEE Transactions on Medical Imaging 37 (11)
[13] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A.
(2018) 2453–2462.
Karpathy, A. Khosla, M. Bernstein, Imagenet large scale visual recognition
[44] M.Z. Alom, M. Hasan, C. Yakopcic, T.M. Taha, V.K. Asari, Recurrent residual
challenge, International Journal of Computer Vision 115 (3) (2015) 211–252.
convolutional neural network based on u-net (r2u-net) for medical image
[14] D.M. Pelt, J.A. Sethian, A mixed-scale dense convolutional neural network for
segmentation, arXiv preprint arXiv:1802.06955.
image analysis, Proceedings of the National Academy of Sciences 115 (2)
[45] H. Chen, Q. Dou, L. Yu, J. Qin, P.-A. Heng, Voxresnet: Deep voxelwise residual
(2018) 254–259.
networks for brain segmentation from 3d mr images, NeuroImage 170 (2018)
[15] M. Lai, Deep learning for medical image segmentation, arXiv preprint
446–455.
arXiv:1505.02000.
[46] R. Guerrero, C. Qin, O. Oktay, C. Bowles, L. Chen, R. Joules, R. Wolz, M. d. C.
[16] J. Liu, M. Li, J. Wang, F. Wu, T. Liu, Y. Pan, A survey of MRI-based brain tumor
Valdés-Hernández, D. Dickie, J. Wardlaw, et al., White matter hyperintensity
segmentation methods, Tsinghua Science & Technology 19 (6) (2014) 578–
and stroke lesion segmentation and differentiation using convolutional
595.
neural networks, NeuroImage: Clinical 17 (2018) 918–934.
[17] G. Litjens, T. Kooi, B.E. Bejnordi, A.A.A. Setio, F. Ciompi, M. Ghafoorian, J.A. Van
[47] M. Drozdzal, G. Chartrand, E. Vorontsov, M. Shakeri, L. Di Jorio, A. Tang, A.
Der Laak, B. Van Ginneken, C.I. Sánchez, A survey on deep learning in medical
Romero, Y. Bengio, C. Pal, S. Kadoury, Learning normalized inputs for iterative
image analysis, Medical Image Analysis 42 (2017) 60–88.
estimation in medical image segmentation, Medical Image Analysis 44 (2018)
[18] D. Shen, G. Wu, H.-I. Suk, Deep learning in medical image analysis, Annual
1–13.
Review of Biomedical Engineering 19 (2017) 221–248.
[48] A. Clèrigues, S. Valverde, J. Bernal, J. Freixenet, A. Oliver, X. Lladó, Sunet: a
[19] K. Suzuki, Overview of deep learning in medical imaging, Radiological Physics
deep learning architecture for acute stroke lesion segmentation and outcome
and Technology 10 (3) (2017) 257–273.
prediction in multimodal MRI, arXiv preprint arXiv:1810.13304.
[20] Y. Yu, L. Min, L. Liu, Y. Li, J. Wang, Clinical big data and deep learning:
[49] N. Ibtehaz, M.S. Rahman, Multiresunet: Rethinking the u-net architecture for
applications, challenges and future outlooks, Big Data Mining and Analytics
multimodal biomedical image segmentation, arXiv preprint
(2019), https://doi.org/10.26599/BDMA.2018.902.
arXiv:1902.04049.
[21] K. Chang, A.L. Beers, H.X. Bai, J.M. Brown, K.I. Ly, X. Li, J.T. Senders, V.K.
[50] L. Liu, S. Chen, F. Zhang, F.-X. Wu, Y. Pan, J. Wang, Deep convolutional neural
Kavouridis, A. Boaro, C. Su, et al., Automatic assessment of glioma burden: a
network for automatically segmenting acute ischemic stroke lesion in multi-
deep learning algorithm for fully automated volumetric and bidimensional
modality MRI, Neural Computing and Applications (2019), https://doi.org/
measurement, Neuro-oncology 21 (11) (2019) 1412–1422.
10.1007/s00521-019-04096-x.
[22] G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with
[51] M. Drozdzal, E. Vorontsov, G. Chartrand, S. Kadoury, C. Pal, The importance of
neural networks, science 313 (5786) (2006) 504–507.
skip connections in biomedical image segmentation, in: Deep Learning and
[23] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for
Data Labeling for Medical Applications, Springer, 2016, pp. 179–187.
biomedical image segmentation, International Conference on Medical
[52] M.Z. Alom, C. Yakopcic, T.M. Taha, V.K. Asari, Nuclei segmentation with
Image Computing and Computer-Assisted Intervention (2015) 234–241.
recurrent residual convolutional neural networks based u-net (r2u-net), in:
[24] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic
NAECON 2018-IEEE National Aerospace and Electronics Conference, IEEE,
segmentation, IEEE Conference on Computer Vision and Pattern Recognition
2018, pp. 228–233.
(2015) 3431–3440.
[53] Q. Jin, Z. Meng, C. Sun, L. Wei, R. Su, Ra-unet: A hybrid deep attention-aware
[25] Ö. Çiçek, A. Abdulkadir, S.S. Lienkamp, T. Brox, O. Ronneberger, 3d u-net:
network to extract liver and tumor in ct scans, arXiv preprint
learning dense volumetric segmentation from sparse annotation, in:
arXiv:1811.01328.
International Conference on Medical Image Computing and Computer-
[54] D. Jin, Z. Xu, A.P. Harrison, D.J. Mollura, White matter hyperintensity
Assisted Intervention, Springer, 2016, pp. 424–432.
segmentation from t1 and flair images using fully convolutional neural
[26] F. Milletari, N. Navab, S.-A. Ahmadi, V-net: Fully convolutional neural
networks enhanced with residual connections, in: IEEE 15th International
networks for volumetric medical image segmentation, in: 2016 Fourth
Symposium on Biomedical Imaging (ISBI 2018), IEEE, 2018, pp. 1060–1064.
International Conference on 3D Vision (3DV), IEEE, 2016, pp. 565–571.
256 L. Liu et al. / Neurocomputing 409 (2020) 244–258

[55] A. Myronenko, 3d mri brain tumor segmentation using autoencoder [85] Y. Kong, J. Gao, Y. Xu, Y. Pan, J. Wang, J. Liu, Classification of autism spectrum
regularization, in: International MICCAI Brainlesion Workshop, Springer, disorder by combining brain connectivity and deep neural network classifier,
2018, pp. 311–320. Neurocomputing 324 (2019) 63–68.
[56] H. Li, A. Zhygallo, B. Menze, Automatic brain structures segmentation using [86] Y. Xu, M. Gong, H. Fu, D. Tao, K. Zhang, K. Batmanghelich, Multi-scale masked
deep residual dilated u-net, in: International MICCAI Brainlesion Workshop, 3-d u-net for brain tumor segmentation, in: International MICCAI Brainlesion
Springer, 2018, pp. 385–393. Workshop, Springer, 2019, pp. 222–233.
[57] Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, J. Liu, Ce-net: [87] A. Casamitjana, M. Catà, I. Sánchez, M. Combalia, V. Vilaplana, Cascaded v-net
Context encoder network for 2d medical image segmentation, IEEE using roi masks for brain tumor segmentation, in: International MICCAI
Transactions on Medical Imaging. Brainlesion Workshop, Springer, 2017, pp. 381–391.
[58] A. Kermi, I. Mahmoudi, M.T. Khadir, Deep convolutional neural networks [88] L. Liu, S. Yang, L. Meng, M. Li, J. Wang, Multi-scale deep convolutional neural
using u-net for automatic brain tumor segmentation in multimodal MRI network for stroke lesions segmentation on ct images, in: International
volumes, in: International MICCAI Brainlesion Workshop, Springer, 2018, pp. MICCAI Brainlesion Workshop, Springer, 2019, pp. 283–291.
37–48. [89] D. Opitz, R. Maclin, Popular ensemble methods: an empirical study, Journal of
[59] D. Nie, L. Wang, Y. Gao, J. Lian, D. Shen, Strainet: Spatially varying stochastic Artificial Intelligence Research 11 (1999) 169–198.
residual adversarial networks for MRI pelvic organ segmentation, IEEE [90] H. Li, G. Jiang, J. Zhang, R. Wang, Z. Wang, W.-S. Zheng, B. Menze, Fully
Transactions on Neural Networks and Learning Systems (2019) 1552–1564. convolutional network ensembles for white matter hyperintensities
[60] W. Zhu, Y. Huang, L. Zeng, X. Chen, Y. Liu, Z. Qian, N. Du, W. Fan, X. Xie, segmentation in mr images, NeuroImage 183 (2018) 650–665.
Anatomynet: Deep learning for fast and fully automated whole-volume [91] R. Mehta, J. Sivaswamy, M-net: A convolutional neural network for deep brain
segmentation of head and neck anatomy, Medical Physics 46 (2) (2019) 576– structure segmentation, in: IEEE 14th International Symposium on
589. Biomedical Imaging (ISBI 2017), IEEE, 2017, pp. 437–440.
[61] G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected [92] K. Kamnitsas, W. Bai, E. Ferrante, S. McDonagh, M. Sinclair, N. Pawlowski, M.
convolutional networks, in: Proceedings of the IEEE Conference on Computer Rajchl, M. Lee, B. Kainz, D. Rueckert, et al., Ensembles of multiple models and
Vision and Pattern Recognition, 2017, pp. 4700–4708. architectures for robust brain tumour segmentation, arXiv preprint
[62] X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, P.-A. Heng, H-denseunet: hybrid densely arXiv:1711.01468.
connected unet for liver and tumor segmentation from ct volumes, IEEE [93] Z. Zhou, M.M.R. Siddiquee, N. Tajbakhsh, J. Liang, Unet++: A nested u-net
Transactions on Medical Imaging 37 (12) (2018) 2663–2674. architecture for medical image segmentation, in: Deep Learning in Medical
[63] L. Liu, F.-X. Wu, J. Wang, Efficient multi-kernel dcnn with pixel dropout for Image Analysis and Multimodal Learning for Clinical Decision Support,
stroke MRI segmentation, Neurocomputing 350 (2019) 117–127, https://doi. Springer, 2018, pp. 3–11.
org/10.1016/j.neucom.2019.03.049. [94] R. Bermúdez-Chacón, P. Márquez-Neila, M. Salzmann, P. Fua, A domain-
[64] J. Zhang, Y. Jin, J. Xu, X. Xu, Y. Zhang, Mdu-net: Multi-scale densely connected adaptive two-stream u-net for electron microscopy image segmentation, in:
u-net for biomedical image segmentation, arXiv preprint arXiv:1812.00352. IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), IEEE,
[65] M. Shaikh, G. Anand, G. Acharya, A. Amrutkar, V. Alex, G. Krishnamurthi, Brain 2018, pp. 400–404.
tumor segmentation using dense fully convolutional neural network, in: [95] X. Feng, N. Tustison, C. Meyer, Brain tumor segmentation using an ensemble
International MICCAI Brainlesion Workshop, Springer, 2017, pp. 309–319. of 3d u-nets and overall survival prediction using radiomic features, in:
[66] M. Khened, V. Alex, G. Krishnamurthi, Densely connected fully convolutional International MICCAI Brainlesion Workshop, Springer, 2018, pp. 279–288.
network for short-axis cardiac cine mr image segmentation and heart [96] Y. Han, J.C. Ye, Framing u-net via deep convolutional framelets: application to
diagnosis using random forest, in: International Workshop on Statistical sparse-view ct, IEEE Transactions on Medical Imaging 37 (6) (2018) 1418–
Atlases and Computational Models of the Heart, Springer, 2017, pp. 140–151. 1429.
[67] J. Stawiaski, A pretrained densenet encoder for brain tumor segmentation, in: [97] S. Kumar, S. Conjeti, A.G. Roy, C. Wachinger, N. Navab, Infinet: Fully
International MICCAI Brainlesion Workshop, Springer, 2018, pp. 105–115. convolutional networks for infant brain MRI segmentation, in: IEEE 15th
[68] T.D. Bui, S.-I. Ahn, Y. Lee, J. Shin, A skip-connected 3d densenet networks with International Symposium on Biomedical Imaging (ISBI 2018), IEEE, 2018, pp.
adversarial training for volumetric segmentation, in: International MICCAI 145–148.
Brainlesion Workshop, Springer, 2018, pp. 378–384. [98] A. Meyer, A. Mehrtash, M. Rak, D. Schindele, M. Schostak, C. Tempany, T.
[69] G. Kim, Brain tumor segmentation using deep fully convolutional neural Kapur, P. Abolmaesumi, A. Fedorov, C. Hansen, Automatic high resolution
networks, in: International MICCAI Brainlesion Workshop, Springer, 2017, pp. segmentation of the prostate from multi-planar mri, in: 2018 IEEE 15th
344–357. International Symposium on Biomedical Imaging (ISBI 2018), IEEE, 2018, pp.
[70] J. Dolz, I.B. Ayed, C. Desrosiers, Dense multi-path u-net for ischemic stroke 177–181.
lesion segmentation in multiple image modalities, in: International MICCAI [99] H. Fu, J. Cheng, Y. Xu, D.W.K. Wong, J. Liu, X. Cao, Joint optic disc and cup
Brainlesion Workshop, Springer, 2018, pp. 271–282. segmentation based on multi-label deep network and polar transformation,
[71] F. Yu, V. Koltun, Multi-scale context aggregation by dilated convolutions, IEEE Transactions on Medical Imaging 37 (7) (2018) 1597–1605.
arXiv preprint arXiv:1511.07122. [100] N. Nuechterlein, S. Mehta, 3d-espnet with pyramidal refinement for
[72] A. Tureckova, A.J. Rodríguez-Sánchez, Isles challenge: U-shaped convolution volumetric brain tumor image segmentation, in: International MICCAI
neural network with dilated convolution for 3d stroke lesion segmentation, Brainlesion Workshop, Springer, 2019, pp. 245–253.
in: International MICCAI Brainlesion Workshop, Springer, 2018, pp. 319–327. [101] S. Mehta, M. Rastegari, A. Caspi, L. Shapiro, H. Hajishirzi, Espnet: Efficient
[73] S. Vesal, N. Ravikumar, A. Maier, Dilated convolutions in neural networks for spatial pyramid of dilated convolutions for semantic segmentation, in:
left atrial segmentation in 3d gadolinium enhanced-MRI, in: International Proceedings of the European Conference on Computer Vision (ECCV), 2018,
Workshop on Statistical Atlases and Computational Models of the Heart, pp. 552–568.
Springer, 2018, pp. 319–328. [102] J. Zhang, A. Saha, Z. Zhe, M.A. Mazurowski, Hierarchical convolutional neural
[74] R. Li, M. Li, J. Li, Connection sensitive attention u-net for accurate retinal networks for segmentation of breast tumors in mri with application to
vessel segmentation, arXiv preprint arXiv:1903.05558. radiogenomics, IEEE Transactions on Medical Imaging 38 (2) (2019) 435–447.
[75] V. Mnih, N. Heess, A. Graves, et al., Recurrent models of visual attention, [103] R.D. Boer, H.A. Vrooman, F.V.D. Lijn, M.W. Vernooij, M.A. Ikram, A.V.D. Lugt,
Advances in Neural Information Processing Systems (2014) 2204–2212. M.M.B. Breteler, W.J. Niessen, White matter lesion extension to automatic
[76] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning brain tissue segmentation on MRI, Neuroimage 45 (4) (2009) 1151–1161.
to align and translate, arXiv preprint arXiv:1409.0473. [104] C.J. Jack, P. O’Brien, D. Rettman, M. Shiung, Y. Xu, R. Muthupillai, A. Manduca,
[77] K.M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, R. Avula, B. Erickson, Flair histogram segmentation for measurement of
P. Blunsom, Teaching machines to read and comprehend, Advances in Neural leukoaraiosis volume, Journal of Magnetic Resonance Imaging 14 (6) (2010)
Information Processing Systems (2015) 1693–1701. 668–676.
[78] J.K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, Y. Bengio, Attention-based [105] B.I. Yoo, J.J. Lee, J.W. Han, S.Y.W. Oh, E.Y. Lee, J.R. Macfall, M.E. Payne, T.H. Kim,
models for speech recognition, Advances in Neural Information Processing J.H. Kim, K.W. Kim, Application of variable threshold intensity to
Systems (2015) 577–585. segmentation for white matter hyperintensities in fluid attenuated
[79] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, inversion recovery magnetic resonance images, Neuroradiology 56 (4)
I. Polosukhin, Attention is all you need, Advances in Neural Information (2014) 265–281.
Processing Systems (2017) 5998–6008. [106] F. Admiraal-Behloul, D.M.J.V.D. Heuvel, H. Olofsen, M.J.P.V. Osch, J.V.D. Grond,
[80] H. Salehinejad, S. Sankar, J. Barfett, E. Colak, S. Valaee, Recent advances in M.A.V. Buchem, J.H.C. Reiber, Fully automatic segmentation of white matter
recurrent neural networks, arXiv preprint arXiv:1801.01078. hyperintensities in mr images of the elderly, Neuroimage 28 (3) (2005) 607–
[81] O. Oktay, J. Schlemper, L.L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. 617.
McDonagh, N.Y. Hammerla, B. Kainz, et al., Attention u-net: learning where to [107] R. Simões, C. Mönninghoff, M. Dlugaj, C. Weimar, I. Wanke, V.C.V.W. Am, C.
look for the pancreas, arXiv preprint arXiv:1804.03999. Slump, Automatic segmentation of cerebral white matter hyperintensities
[82] J. Liu, M. Li, W. Lan, F.-X. Wu, Y. Pan, J. Wang, Classification of alzheimer’s using only 3d flair images, Magnetic Resonance Imaging 31 (7) (2013) 1182–
disease using whole brain hierarchical network, IEEE/ACM Transactions on 1189.
Computational Biology and Bioinformatics 15 (2) (2016) 624–632. [108] K.H. Ong, D. Ramachandram, R. Mandava, I.L. Shuaib, Automatic white matter
[83] J. Liu, J. Wang, Z. Tang, B. Hu, F.-X. Wu, Y. Pan, Improving alzheimer’s disease lesion segmentation using an adaptive outlier detection method, Magnetic
classification by combining multiple measures, IEEE/ACM Transactions on Resonance Imaging 30 (6) (2012) 807–823.
Computational Biology and Bioinformatics 15 (5) (2017) 1649–1659. [109] R. Beare, V. Srikanth, J. Chen, T.G. Phan, J. Stapleton, R. Lipshut, D. Reutens,
[84] T.A. Tuan et al., Brain tumor segmentation using bit-plane and unet, in: Development and validation of morphological segmentation of age-related
International MICCAI Brainlesion Workshop, Springer, 2018, pp. 466–475. cerebral white matter hyperintensities, Neuroimage 47 (1) (2009) 199–203.
 L. Liu et al. / Neurocomputing 409 (2020) 244–258 257

[110] Y. Kawata, H. Arimura, Y. Yamashita, T. Magome, M. Ohki, F. Toyofuku, Y. [134] M.E. Caligiuri, P. Perrotta, A. Augimeri, F. Rocca, A. Quattrone, A. Cherubini,
Higashida, K. Tsuchiya, Computer-aided evaluation method of white matter Automatic detection of white matter hyperintensities in healthy aging and
hyperintensities related to subcortical vascular dementia based on magnetic pathology using magnetic resonance imaging: a review, Neuroinformatics 13
resonance imaging, Computerized Medical Imaging & Graphics 34 (5) (2010) (3) (2015) 261–276.
370–376. [135] Y. Zhang, W. Chen, Y. Chen, X. Tang, A post-processing method to improve the
[111] W. Chen, Y. Zhang, J. He, Y. Qiao, Y. Chen, H. Shi, X. Tang, Prostate white matter hyperintensity segmentation accuracy for randomly-initialized
segmentation using 2d bridged u-net, arXiv preprint arXiv:1807.04459v2. u-net, in: 2018 IEEE 23rd International Conference on Digital Signal
[112] W. Zhang, R. Li, H. Deng, L. Wang, W. Lin, S. Ji, D. Shen, Deep convolutional Processing (DSP), IEEE, 2018, pp. 1–5.
neural networks for multi-modality isointense infant brain image [136] L. Liu, S. Chen, X. Zhu, X.-M. Zhao, F.-X. Wu, J. Wang, Deep convolutional
segmentation, NeuroImage 108 (2015) 214–224. neural network for accurate segmentation and quantification of white matter
[113] H. Chen, X. Qi, L. Yu, P.-A. Heng, Dcan: deep contour-aware networks for hyperintensities, Neurocomputing 384 (2020) 231–242.
accurate gland segmentation, in: Proceedings of the IEEE conference on [137] M. Luna, S.H. Park, 3d patchwise u-net with transition layers for mr brain
Computer Vision and Pattern Recognition, 2016, pp. 2487–2496. segmentation, in: International MICCAI Brainlesion Workshop, Springer,
[114] C. Qin, R. Guerrero, C. Bowles, L. Chen, D.A. Dickie, M.d.C. Valdes-Hernandez, J. 2018, pp. 394–403.
Wardlaw, D. Rueckert, A large margin algorithm for automated segmentation [138] D. García-Lorenzo, S. Francis, S. Narayanan, D.L. Arnold, D.L. Collins, Review of
of white matter hyperintensity, Pattern Recognition 77 (2018) 150–159. automatic segmentation methods of multiple sclerosis white matter lesions
[115] J. Wasserthal, P. Neher, K.H. Maier-Hein, Tractseg-fast and accurate white on conventional magnetic resonance imaging, Medical Image Analysis 17 (1)
matter tract segmentation, NeuroImage 183 (2018) 239–253. (2013) 1–18.
[116] Z. Lao, D. Shen, D. Liu, A.F. Jawad, E.R. Melhem, L.J. Launer, R.N. Bryan, C. [139] Z. Zhang, F.S. Yin, J. Liu, W.K. Wong, N.M. Tan, B.H. Lee, J. Cheng, T.Y. Wong,
Davatzikos, Computer-assisted segmentation of white matter lesions in 3d Origa-light: an online retinal fundus image database for glaucoma analysis
mr images using support vector machine, Academic Radiology 15 (3) (2008) and research, in: Annual International Conference of the IEEE Engineering in
300–313. Medicine and Biology, IEEE, 2010, pp. 3065–3068.
[117] R. Simões, C. Mönninghoff, M. Dlugaj, C. Weimar, I. Wanke, A.-M.v.C. van [140] F. Isensee, P.F. Jaeger, P.M. Full, I. Wolf, S. Engelhardt, K.H. Maier-Hein,
Walsum, C. Slump, Automatic segmentation of cerebral white matter Automatic cardiac disease assessment on cine-mri via time-series
hyperintensities using only 3d flair images, Magnetic Resonance Imaging segmentation and domain specific features, in: International workshop on
31 (7) (2013) 1182–1189. statistical atlases and computational models of the heart, Springer, 2017, pp.
[118] L. Yu, X. Yang, H. Chen, J. Qin, P.A. Heng, Volumetric convnets with mixed 120–129.
residual connections for automated prostate segmentation from 3d mr [141] Y. Jang, Y. Hong, S. Ha, S. Kim, H.-J. Chang, Automatic segmentation of lv and
images, in: Thirty-first AAAI Conference on Artificial Intelligence, 2017. rv in cardiac mri, in: International Workshop on Statistical Atlases and
[119] J. Patravali, S. Jain, S. Chilamkurthy, 2d–3d fully convolutional neural Computational Models of the Heart, Springer, 2017, pp. 161–169.
networks for cardiac mr segmentation, in: International Workshop on [142] X. Yang, C. Bian, L. Yu, D. Ni, P.-A. Heng, Class-balanced deep neural network
Statistical Atlases and Computational Models of the Heart, Springer, 2017, for automatic ventricular structure segmentation, in: International
pp. 130–139. Workshop on Statistical Atlases and Computational Models of the Heart,
[120] C.F. Baumgartner, L.M. Koch, M. Pollefeys, E. Konukoglu, An exploration of 2d Springer, 2017, pp. 152–160.
and 3d deep learning techniques for cardiac mr image segmentation, in: [143] R.P. Poudel, P. Lamata, G. Montana, Recurrent fully convolutional neural
International Workshop on Statistical Atlases and Computational Models of networks for multi-slice MRI cardiac segmentation, in: Reconstruction,
the Heart, Springer, 2017, pp. 111–119. Segmentation, and Analysis of Medical Images, Springer, 2016, pp. 83–94.
[121] J. Dolz, C. Desrosiers, A.I. Ben, 3d fully convolutional networks for subcortical [144] L. Geert, T. Robert, V.D.V. Wendy, H. Caroline, K. Sjoerd, V.G. Bram, V. Graham,
segmentation in mri: a large-scale study, Neuroimage 170 (2016) 456–470. G. Gwenael, B. Neil, Z. Jindang, Evaluation of prostate segmentation
[122] K. Kamnitsas, C. Ledig, V.F. Newcombe, J.P. Simpson, A.D. Kane, D.K. Menon, algorithms for MRI: the promise12 challenge, Medical Image Analysis 18
D. Rueckert, B. Glocker, Efficient multi-scale 3d cnn with fully connected crf (2) (2014) 359–373.
for accurate brain lesion segmentation, Medical Image Analysis 36 (2017) [145] J. Dolz, C. Desrosiers, I.B. Ayed, Ivd-net: Intervertebral disc localization and
61–78. segmentation in mri with a multi-modal unet, in: International Workshop
[123] P. Moeskops, J.D. Bresser, H.J. Kuijf, A.M. Mendrik, G.J. Biessels, J.P.W. Pluim, I. and Challenge on Computational Methods and Clinical Applications for Spine
Išgum, Evaluation of a deep learning approach for the segmentation of brain Imaging, Springer, 2018, pp. 130–143.
tissues and white matter hyperintensities of presumed vascular origin in [146] A. Esteva, B. Kuprel, R.A. Novoa, J. Ko, S.M. Swetter, H.M. Blau, S. Thrun,
MRI, Neuroimage Clinical 17 (C) (2018) 251–262. Dermatologist-level classification of skin cancer with deep neural networks,
[124] X. Li, Q. Dou, H. Chen, C.-W. Fu, X. Qi, D.L. Belavý, G. Armbrecht, D. Felsenberg, Nature 542 (7639) (2017) 115–118.
G. Zheng, P.-A. Heng, 3d multi-scale fcn with random modality voxel dropout [147] J. Kawahara, A. BenTaieb, G. Hamarneh, Deep features to classify skin lesions,
learning for intervertebral disc localization and segmentation from multi- in: IEEE 13th International Symposium on Biomedical Imaging (ISBI), IEEE,
modality mr images, Medical Image Analysis 45 (2018) 41–54. 2016, pp. 1397–1400.
[125] Q. Dou, L. Yu, H. Chen, Y. Jin, X. Yang, J. Qin, P.A. Heng, 3d deeply supervised [148] N.C. Codella, D. Gutman, M.E. Celebi, B. Helba, M.A. Marchetti, S.W. Dusza, A.
network for automated segmentation of volumetric medical images, Medical Kalloo, K. Liopyris, N. Mishra, H. Kittler, et al., Skin lesion analysis toward
Image Analysis 41 (2017) 40–54. melanoma detection: a challenge at the 2017 international symposium on
[126] H. Chen, X. Qi, L. Yu, Q. Dou, J. Qin, P.A. Heng, Dcan: Deep contour-aware biomedical imaging (isbi), hosted by the international skin imaging
networks for object instance segmentation from histology images, Medical collaboration (isic), in: 2018 IEEE 15th International Symposium on
Image Analysis 36 (2017) 135–146. Biomedical Imaging (ISBI 2018), IEEE, 2018, pp. 168–172.
[127] H. Dong, G. Yang, F. Liu, Y. Mo, Y. Guo, Automatic brain tumor detection and [149] J. Li, X. Lin, H. Che, H. Li, X. Qian, Probability map guided bi-directional
segmentation using u-net based fully convolutional networks, in: Annual recurrent unet for pancreas segmentation, arXiv preprint arXiv:1903.00923.
Conference on Medical Image Understanding and Analysis, Springer, 2017, [150] Y. Chen, B. Shi, Z. Wang, P. Zhang, C.D. Smith, J. Liu, Hippocampus
pp. 506–517. segmentation through multi-view ensemble convnets, in: IEEE 14th
[128] S. Hussain, S.M. Anwar, M. Majid, Segmentation of glioma tumors in brain International Symposium on Biomedical Imaging (ISBI 2017), IEEE, 2017,
using deep convolutional neural network, Neurocomputing 282 (2018) 248– pp. 192–196.
261. [151] H. Ravishankar, S. Thiruvenkadam, R. Venkataramani, V. Vaidya, Joint deep
[129] O. Maier, B.H. Menze, J. von der Gablentz, L. Häni, M.P. Heinrich, M. Liebrand, learning of foreground, background and shape for robust contextual
S. Winzeck, A. Basit, P. Bentley, L. Chen, et al., Isles 2015-a public evaluation segmentation, in: International Conference on Information Processing in
benchmark for ischemic stroke lesion segmentation from multispectral MRI, Medical Imaging, Springer, 2017, pp. 622–632.
Medical Image Analysis 35 (2017) 250–269. [152] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, Z. Tu, Deeply-supervised nets, in:
[130] Y. Choi, Y. Kwon, H. Lee, B.J. Kim, M.C. Paik, J.-H. Won, Ensemble of deep Artificial Intelligence and Statistics, 2015, pp. 562–570.
convolutional neural networks for prognosis of ischemic stroke, in: [153] M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional
International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke networks, in: European Conference on Computer Vision, Springer, 2014, pp.
and Traumatic Brain Injuries, Springer, 2016, pp. 231–243. 818–833.
[131] K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional [154] J.T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller, Striving for
networks for visual recognition, IEEE Transactions on Pattern Analysis and simplicity: The all convolutional net, arXiv preprint arXiv:1412.6806.
Machine Intelligence 37 (9) (2015) 1904–1916. [155] Z. Zhang, Y. Xie, F. Xing, M. McGough, L. Yang, Mdnet: A semantically and
[132] S. Tao, 3d multi-scale u-net with atrous convolution for ischemic stroke visually interpretable medical image diagnosis network, in: Proceedings of
lesion segmentation, in: International MICCAI Brainlesion Workshop, the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.
(BrainLes 2018), LNCS 11383 37 (9) (2015) 1904–1916. 6428–6436.
[133] J.M. Wardlaw, E.E. Smith, G.J. Biessels, C. Cordonnier, F. Fazekas, R. Frayne, R.I. [156] A. Karpathy, L. Fei-Fei, Deep visual-semantic alignments for generating image
Lindley, J.T. O’Brien, F. Barkhof, O.R. Benavente, et al., Neuroimaging descriptions, in: Proceedings of the IEEE Conference on Computer Vision and
standards for research into small vessel disease and its contribution to Pattern Recognition, 2015, pp. 3128–3137.
ageing and neurodegeneration, The Lancet Neurology 12 (8) (2013) 822–838. [157] G. Wang, A perspective on deep imaging, IEEE Access 4 (2016) 8914–8924.
258 L. Liu et al. / Neurocomputing 409 (2020) 244–258

Liangliang Liu received his M.S. degree from Henan biological system identification and parameter estimation, applications of control
University in 2014. He is currently a Ph.D. candidate in theory to biological systems. Dr. Wu is serving as the editorial board member of five
School of Computer Science and Engineering, Central international journals, the guest editor of several international journals, and as the
South University, Changsha, Hunan, P.R. China. His program committee chair or member of several international conferences. He has
research interests include machine learning, deep also reviewed papers for many international journals.
learning and medical image analysis.

Yu-Ping Wang received the BS degree in applied

mathematics from Tianjin University, China, in 1990,
and the MS degree in computational mathematics and
the PhD degree in communications and electronic sys-
tems from Xian Jiaotong University, China, in 1993 and
1996, respectively. After his graduation, he had visiting
positions at the Center for Wavelets, Approximation and
Jianhong Cheng received the B.S. degree from Liaoning Information Processing of the National University of
Technical University, Fuxin, China, in 2014, the M.S. Singapore and Washington University Medical School in
degree in software engineering from Central South St. Louis. From 2000 to 2003, he worked as a senior
University, Changsha, China, in 2017. And he is cur- research engineer at Perceptive Scientific Instruments,
rently a Ph.D. candidate in School of Computer Science Inc., and then Advanced Digital Imaging Research, LLC,
and Engineering, Central South University, Changsha, Houston, Texas. In the fall of 2003, he returned to academia as an assistant pro-
China. His research interests include machine learning, fessor of computer science and electrical engineering at the University of Missouri-
deep learning and medical image analysis. Kansas City. He is currently a full Professor of Biomedical Engineering and Global
Biostatistics & Data Sciences at Tulane University School of Science and Engineering
& School of Public Health and Tropical Medicine. He has been on numerous program
committees and NSF and NIH review panels, and served as editors for several
journals such as J. Neuroscience Methods, IEEE Trans. Medical Imaging and IEEE/
ACM Trans. Computational Biology and Bioinformatics (TCBB).

Quan Quan received the B.S. degree from Central South Jianxin Wang received his B.S. and M.S. degree in
University of, Materials Science and Engineering, Computer Science from Central South University of
ChangSha,China, in 2017, currently a M.S degree can- Technology, P. R. China, and his PhD degree in Computer
didate in School of Computer Science and Engineering, Science from Central South University. Currently, he is
Central South University, Changsha, China. His research the vice dean and a professor in School of Computer
interests include machine learning, deep learning and Science and Engineering, Central South University,
medical image analysis. Changsha, Hunan, P.R. China. He is currently serving as
the executive editor of International Journal of Bioin-
formatics Research and Applications and serving in the
editorial board of International Journal of Data Mining
and Bioinformatics. He has also served as the program
committee member for many international conferences.
His current research interests include algorithm analysis and optimization,
parameterized algorithm, bioinformatics and computer network. He has published
Fang-Xiang Wu received the B.Sc. degree and the M.Sc. more than 200 papers in various International journals and refereed conferences.
degree in applied mathematics, both from Dalian He is a senior member of the IEEE.
University of Technology, Dalian, China, in 1990 and
1993, respectively, the first Ph.D. degree in control
theory and its applications from Northwestern
Polytechnical University, Xi’an, China, in 1998, and the
second Ph.D. degree in biomedical engineering from
University of Saskatchewan (U of S), Saskatoon, Canada,
in 2004. During 2004–2005, he worked as a Postdoctoral
Fellow in the Laval University Medical Research Center
(CHUL), Quebec City, Canada. He is currently a Professor
of the Division of Biomedical Engineering and the
Department of Mechanical Engineering at the U of S. His current research interests
include computational and systems biology, genomic and proteomic data analysis,