See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/330203264

A guide to deep learning in healthcare

Article  in  Nature Medicine · January 2019

DOI: 10.1038/s41591-018-0316-z

CITATIONS READS

162 4,802

10 authors, including:

Alexandre Robicquet Bharath Ramsundar

Stanford University Stanford University
6 PUBLICATIONS   1,101 CITATIONS    17 PUBLICATIONS   905 CITATIONS   

SEE PROFILE SEE PROFILE

Katherine Chou
Google Inc.
7 PUBLICATIONS   571 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Human trajectory forecasting View project

Tsunamis Modeling View project

All content following this page was uploaded by Alexandre Robicquet on 25 October 2019.

The user has requested enhancement of the downloaded file.

Perspective | FOCUS
Perspective | FOCUS
https://doi.org/10.1038/s41591-018-0316-z

https://doi.org/10.1038/s41591-018-0316-z

A guide to deep learning in healthcare

Andre Esteva1,3*, Alexandre Robicquet1,3, Bharath Ramsundar1, Volodymyr Kuleshov1, Mark DePristo2,
Katherine Chou2, Claire Cui2, Greg Corrado2, Sebastian Thrun1 and Jeff Dean2

Here we present deep-learning techniques for healthcare, centering our discussion on deep learning in computer vision, natural
language processing, reinforcement learning, and generalized methods. We describe how these computational techniques can
impact a few key areas of medicine and explore how to build end-to-end systems. Our discussion of computer vision focuses
largely on medical imaging, and we describe the application of natural language processing to domains such as electronic health
record data. Similarly, reinforcement learning is discussed in the context of robotic-assisted surgery, and generalized deep-
learning methods for genomics are reviewed.

D
eep learning1, a subfield of machine learning (ML), has seen Computer vision
a dramatic resurgence in the past 6 years, largely driven by Some of the greatest successes of deep learning have been in the field
increases in computational power and the availability of mas- of computer vision (CV)2. CV focuses on image and video under-
sive new datasets. The field has witnessed striking advances in the standing, and deals with tasks such as object classification, detec-
ability of machines to understand and manipulate data, including tion, and segmentation—which are useful in determining whether a
images2, language3, and speech4. Healthcare and medicine stand to patient’s radiograph contains malignant tumors. Convolutional neu-
benefit immensely from deep learning because of the sheer volume ral networks (CNNs)1,2, a type of deep-learning algorithm designed
of data being generated (150 exabytes or 1018 bytes in United States to process data that exhibits natural spatial invariance (e.g., images,
alone, growing 48% annually5) as well as the increasing proliferation whose meanings do not change under translation), have grown to
of medical devices and digital record systems. be central in this field.
ML is distinct from other types of computer programming in Medical imaging, for instance, can greatly benefit from recent
that it transforms the inputs of an algorithm into outputs using advances in image classification and object detection2,8. Many stud-
statistical, data-driven rules that are automatically derived from ies have demonstrated promising results in complex diagnostics
a large set of examples, rather than being explicitly specified by spanning dermatology9,10, radiology11–14, ophthalmology15–17, and
humans. Historically, constructing a ML system required domain pathology18–21 (Fig. 2). Deep-learning systems could aid physi-
expertise and human engineering to design feature extractors that cians by offering second opinions and flagging concerning areas
transformed raw data into suitable representations from which a in images.
learning algorithm could detect patterns. In contrast, deep learn- Image-level diagnostics have been quite successful at employing
ing is a form of representation learning—in which a machine is CNN-based methods (Fig. 2). This is largely due to the fact that
fed with raw data and develops its own representations needed for CNNs have achieved human-level performance in object-classifica-
pattern recognition—that is composed of multiple layers of rep- tion tasks2, in which a CNN learns to classify the object contained
resentations. These layers are typically arranged sequentially and in an image. These same networks have demonstrated strong per-
composed of a large number of primitive, nonlinear operations, formance in transfer learning22, in which a CNN initially trained
such that the representation of one layer (beginning with the raw on a massive dataset that is unrelated to the task of interest (e.g.,
data input) is fed into the next layer and transformed into a more ImageNet2, a dataset of millions of common everyday objects) is
abstract representation1. As data flows through the layers of the sys- further fine-tuned on a much smaller dataset related to the task
tem, the input space becomes iteratively warped until data points of interest (e.g., medical images). In the first step, the algorithm
become distinguishable (Fig. 1a). In this manner, highly complex leverages large amounts of data to learn of the natural statistics in
functions can be learned. images—straight lines, curves, colorations, etc.—and in the second
Deep-learning models scale to large datasets—in part owing step, the higher-level layers of the algorithm are retrained to dis-
to their ability to run on specialized computing hardware—and tinguish between diagnostic cases. Similarly, object detection and
continue to improve with more data, enabling them to outper- segmentation algorithms identify specific parts of an image that
form many classical ML approaches. Deep-learning systems can correspond to particular objects. CNN methods take image data as
accept multiple data types as input—an aspect of particular rel- input and iteratively warp it through a series of convolutional and
evance for heterogeneous healthcare data (Fig. 1b). The most nonlinear operations until the original raw data matrix is trans-
common models are trained using supervised learning, in which formed into a probability distribution over potential image classes
datasets are composed of input data points (e.g., skin lesion (e.g., medical diagnostic cases) (Fig. 2).
images) and corresponding output data labels (e.g., ‘benign’ or Remarkably, deep-learning models have achieved physician-level
‘malignant’). Reinforcement learning (RL), in which computa- accuracy at a broad variety of diagnostic tasks, including identifying
tional agents learn by trial and error or by expert demonstration, moles from melanomas9,10, diabetic retinopathy, cardiovascular risk,
has progressed with the adoption of deep learning, achieving and referrals from fundus15,16 and optical coherence tomography
remarkable feats in areas such as game playing (e.g., Go6). RL (OCT)17 images of the eye, breast lesion detection in mammograms13,
can be useful in healthcare whenever learning requires physician and spinal analysis with magnetic resonance imaging23. A single
demonstration, for instance in learning to suture wounds for deep-learning model has even been shown to be effective at diagno-
robotic-assisted surgery7. sis across medical modalities (e.g., radiology and ophthalmology)24.

1
Stanford University, Stanford, CA, USA. 2Google Research, San Jose, CA, USA. 3These authors contributed equally: Andre Esteva, Alexandre Robicquet.
*e-mail: [email protected]

24 Nature Medicine | VOL 25 | JANUARY 2019 | 24–29 | www.nature.com/naturemedicine

 FOCUS | Perspective
|
Nature MediCine FOCUS Perspective
https://doi.org/10.1038/s41591-018-0316-z

a Neural network layers make data linearly separable even been used to discover biological features of tissue associated
with survival probability21.
The primary limitation to building a supervised deep-learning
system for a new medical imaging task is access to a sufficiently
large, labeled dataset. Small and labeled datasets for specific tasks
are easier to collect, but result in algorithms that tend to perform
poorly on new data. In these cases, techniques for heavy data
augmentation have been shown to be effective at helping algo-
rithms generalize25. Similarly, large but unlabeled datasets are
also easier to collect, but will require a shift towards improved
Input data Hidden layer 1 Hidden layer 2 Output semisupervised and unsupervised techniques, such as generative
adversarial networks26.
b Deep learning can featurize and learn from a variety of data types
Natural language processing
Natural language processing (NLP) focuses on analyzing text and
speech to infer meaning from words. Recurrent neural networks
(RNNs)—deep learning algorithms effective at processing sequen-
Image inputs
tial inputs such as language, speech, and time-series data27—play
an important role in this field. Notable successes of NLP include
machine translation28, text generation29, and image captioning30.
0.8
0.6
In healthcare, sequential deep learning and language technolo-
0.4
0.2 gies power applications within domains such as electronic health
0.0
–0.2 records (EHRs).
–0.4
0 1 2 3 4 5 6 EHRs are rapidly becoming ubiquitous31. The EHR of a large
Temporal inputs
medical organization can capture the medical transactions of over
10 million patients throughout the course of a decade. A single
hospitalization alone typically generates ∼​150,000 pieces of data.
The potential benefits derived from this data are significant. In
aggregate, an EHR of this scale represents 200,000 years of doctor
wisdom and 100 million years of patient outcome data, covering a
Multimodal
inputs plethora of rare conditions and maladies. As such, applyication of
Input Hidden Hidden Hidden Hidden Output
layer layer 1 layer 2 layer 3 layer 4 layer deep-learning methods to EHR data is a rapidly expanding area32,33.
Figure 3 outlines the technical steps in building deep-learning
Fig. 1 | Deep learning. a, A simple, multilayer deep neural network takes systems for EHRs. Raw data are first aggregated across across
two classes of data, denoted by the different colors, and makes them institutions in order to ensure that a generalizable system is built.
linearly separable by iteratively distorting the data as it flows from layer The data are then standardized and parsed temporally and across
to layer. The final output layer serves as a classifier by outputting the patients, which makes them suitable for deep-learning training.
probability of either one of the classes. This example illustrates the basic From this, we can then infer answers to high-level medical ques-
concept used by large scale networks. Conceptual illustration adapted tions, such as ‘What past history is relevant to the patient’s current
with permission from http://colah.github.io/. b, Example large-scale diagnosis?’, ‘What is the patient’s current problem list?’, and ‘What
network that accepts as input a variety of data types (images, time-series, opportunities are there to intervene?’
etc.), and for each data type learns a useful featurization in its lower- When making predictions, most work to date uses supervised
level towers. The data from each tower is then merged and flows through learning on limited sets of structured data, including lab results,
higher levels, allowing the DNN to perform inference across data types—a vitals, diagnostic codes, and demographics. To account for the
capability that is increasingly important in healthcare. structured and unstructured data contained in EHRs, researchers
are beginning to employ unsupervised learning approaches, such as
auto-encoders—in which networks are first trained to learn useful
However, a key limitation across studies that compare human to representations by compressing and then reconstructing unlabeled
algorithmic performance has been a lack of clinical context—they data—to predict specific diagnoses34. Recent uses of deep learning
constrain the diagnosis to be performed using just the images at model the temporal sequence of structured events that occurred
hand. This often increases the difficulty of the diagnostic task for the in a patient’s record with convolutional and recurrent neural net-
human reader, who in real-world clinical settings has access to both works in order to predict future medical incidents35–38. Much of this
the medical imagery and supplemental data, including the patient work focuses on the Medical Information Mart for Intensive Care
history and health record, additional tests, patient testimony, etc. (MIMIC) dataset39 (e.g., for the prediction of sepsis40), which con-
Clinics are beginning to employ object detection and segmenta- tains intensive care unit (ICU) patients from a single center. While
tion in images for urgent and easily missed cases, such as flagging ICU patients generate more EHR data than non-ICU patients, they
large-artery occlusion in the brain using radiological images14, dur- are significantly outnumbered by non-ICU patients. As such, it is
ing which patients have a limited amount of time (a few minutes) still uncertain how well techniques derived from this data will gen-
before permanent brain damage occurs. Further, cancer histopa- eralize to broader populations.
thology reads, which require human experts to laboriously scan The next generation of automatic speech recognition32 and infor-
and diagnose gigapixel images (or equivalently large physical slides) mation extraction models will likely develop clinical voice assistants
can be supplemented with CNNs trained to detect mitotic cells18 to accurately transcribe patient visits. Doctors easily spend 6 hours
or tumor regions19. They can be trained to quantify the amount of in an 11-hour workday working on documentation in the EHR,
PD-L1 present in a histopathology image20—a task important in which leads to burnout and reduces time with patients31. Automated
determining which type of immuno-oncology drug a patient would transcription will alleviate this and facilitate more affordable
be receptive to. Combined with pixel-level analyses, CNNs have scribing services. Consider RNN-based language translation27,

Nature Medicine | VOL 25 | JANUARY 2019 | 24–29 | www.nature.com/naturemedicine 25

Perspective | FOCUS | FOCUS
Perspective
https://doi.org/10.1038/s41591-018-0316-z Nature MediCine

Input Convolution Pooling Convolution Pooling Fully connected Softmax

Feature learning Classification

Fig. 2 | Medical Imaging. CNNs can be trained on a variety of medical imagery, including radiology, pathology, dermatology, and ophthalmology.
Information flows left to right. CNNs take input images and sequentially transform them, using simple operations such as convolutional, pooling, and fully
connected layers, into flattened vectors. The elements of the output vector (softmax layer) represent the probabilities of the presence of disease. During
the training process, the internal parameters of the network layers are iteratively adjusted to improve accuracy. Typically, lower layers (left) learn simple
image features—edges and basic shapes—which influence the high-level representations (right). Prediction tasks include both classification of the images
(i.e., cancerous versus benign) as well as localization of medical features such as tumors.

which uses an end-to-end technique to translate directly from knot-tying7. For instance, computer vision techniques (e.g., CNNs
speech in one language to text in another. Adapted to EHRs, this for object detection/segmentation and stereovision) can reconstruct
technique could translate a patient–provider conversation directly the landscape of an open wound from image data, and a suturing or
into a transcribed text record. The key challenge lies in classifying knot-tying trajectory can be generated by solving a path optimization
the attributes and status of each medical entity from the conversa- problem that attempts to find an optimal trajectory while account-
tion while accurately summarizing the dialogue. Though promising ing for external constraints, such as joint limits and obstacles43.
in early human–computer interaction experiments, these tech- Similarly, image-trained RNNs can learn to tie knots autonomously
niques have yet to be widely deployed in medical practice. by learning sequences of events, in this case physical maneuvers,
Future work will likely focus on developing algorithms to bet- from surgeons44.
ter leverage some of the information-rich yet unstructured data in These techniques are particularly advantageous for fully auton-
EHRs. Clinical notes, for instance, are often omitted or redacted omous robotic surgery or minimally invasive surgery. Consider
when developing predictive systems. Here, large-scale RNNs are modern laparoscopic surgery (MLS)—in which several small inci-
beginning to demonstrate impressive predictive results by combin- sions are used to insert a number of instruments into the body,
ing structured and unstructured data in a semisupervised way33. including cameras and surgical tools, which surgeons then teleop-
This data combination allows them to learn from broader popu- erate. Deep imitation learning, RNNs, and trajectory transfer algo-
lations across more diverse data types, outperforming other tech- rithms can fully automate certain teleoperated manipulation tasks
niques across tasks including mortality, readmission, length of stay, of the surgical procedure7. In MLS, the automation of repetitive
and diagnosis predictions. tasks is even more time-critical than in open surgery. For instance,
it may take 3 minutes to tie a knot in MLS instead of a few seconds,
Reinforcement learning as in open surgery.
Reinforcement learning (RL) refers to a class of techniques designed One of the main challenges during semiautonomous teleopera-
to train computational agents to successfully interact with their tion is correctly localizing an instrument’s position and orientation
environment, typically to achieve specific goals. This learning can in the vicinity of surgical scenes. Here, recent pixel-wise instrument
happen through trial and error, through demonstration, or through segmentation techniques45, developed using an improved U-Net
a hybrid approach. As an agent takes actions within its environ- architecture CNN25,46, begin to show promise. Another challenge for
ment, an iterative feedback loop of reward and consequence trains the progression of deep learning in surgical robotics is data collec-
the agent to better accomplish the goals at hand. Learning from tion. Deep imitation learning requires large training datasets with
expert demonstration is accomplished either by learning to predict many examples per surgical action. Given that many surgeries are
the expert’s actions directly via supervised learning (i.e., imitation nuanced and unique, it remains difficult to collect sufficient data for
learning) or by inferring the expert’s objective (i.e., inverse RL). more general surgical tasks. Further, it remains difficult for auton-
To successfully train an agent, it is critical to have a model function omous systems to adapt to completely unknown and unobserved
that can take as input sensory signals from the environment and situations highly dissimilar from anything previously seen, such as
output the next actions for the agent to take. Deep RL, in which a an anomalous surgical accident.
deep-learning model serves as the model function, shows promise.
One healthcare domain that can benefit from deep RL is Generalized deep learning
robotic-assisted surgery (RAS). Currently, RAS largely depends Beyond CV, NLP, and RL tasks, deep learning is adaptable to
on a surgeon guiding a robot’s instruments in a teleoperated fash- domains in which input data is nuanced and requires specialized
ion. Deep learning can enhance the robustness and adaptability treatment. For illustrative purposes, here we consider genomics, an
of RAS by using computer vision models (e.g., CNNs) to perceive example domain in which deep learning has been adapted beyond
surgical environments and RL methods to learn from a surgeons conventional (e.g., CNN- or RNN-based) approaches to work with
physical motions41,42. unique (e.g., nonimage, nontemporal) data representations.
These techniques support the automation and speed of highly Modern genomic technologies collect a wide variety of mea-
repetitive and time-sensitive surgical tasks, such as suturing and surements, from an individual’s DNA sequence to the quantity of

26 Nature Medicine | VOL 25 | JANUARY 2019 | 24–29 | www.nature.com/naturemedicine

 FOCUS | Perspective
|
Nature MediCine FOCUS Perspective
https://doi.org/10.1038/s41591-018-0316-z

a Unstructured EHR
data

Models that work at hospital 1 don’t work at hospital 2

b Map to common
format (FHIR)

Models can be used across health systems

c Temporal
sequencing

Sequence models have access to the entire patient’s record

Use AI to answer
questions What parts of a patient's past What about a patient's current What are opportunities What are the risks
history should be reviewed? state needs to be known? to intervene? of future outcomes?

Fig. 3 | Making predictions using EHRs. a, Unstructured EHR data. Medical records are stored in idiosyncratic data structures and formats such that
models built on a given hospital’s record do not necessarily work with data from a different hospital. b, Data standardization. By mapping data from
multiple sites to a single format based on FHIR, data are standardized into a homogeneous format. c, Sequencing. By temporally sequencing all data into a
patient timeline, time-based deep-learning techniques can be applied on the entirety of EHR datasets for making predictions about single patients.

b c d

3
2 4

1 5

Identification of
Pharmacogenomics pathogenic
variants

Molecular diagnostics

High-throughput
sequencing AA AB BB

Medical record
ClinVar DNA sequencing Gene splicing
Pedigree
Test results Input Hidden Hidden Output
Gene conservation layer layer 1 layer 2 layer Experimental data interpretation
Amino acid sequences

Input data Example data tensor DNN Biomedical applications

transformed into used to train that can be used for

Fig. 4 | ML in genomics. a, Input data. Genomic data consists of experimental measurements from which certain properties or outcomes of interest
may be predicted. This data is often diverse and may include sequencing, gene expression, and functional data as well as other forms of molecular data.
b Example data tensors. Raw experimental measurements need to be transformed into a form that is suitable for consumption by deep-learning
algorithms, which take as input multidimensional data tensors and associated target labels. c, DNN. Labeled tensors are used to train DNNs to predict
the label from the input data tensor. d, Biomedical applications. Trained DNNs can be used in biomedical applications, such as in predicting labels
for previously unseen data tensors or examining the relationship between input data and output labels. Example applications include interpreting
experimental data (e.g., inferring DNA sequences from the output of a sequencing instrument or inferring the effects of DNA mutations on gene splicing)
and molecular diagnostics (e.g., predicting the effects of genetic mutations on disease risk or drug response), among many others.

various proteins in their blood. There are many opportunities for deep-learning system in genomics involves taking raw data (e.g.,
deep learning to improve the methods used to analyze these mea- gene expression data), converting this raw data into input data ten-
surements, which will ultimately help clinicians provide more accu- sors, and feeding these tensors through neural networks which then
rate treatments and diagnoses. The typical pipeline for building a power specific biomedical applications (Fig. 4).

Nature Medicine | VOL 25 | JANUARY 2019 | 24–29 | www.nature.com/naturemedicine 27

Perspective | FOCUS | FOCUS
Perspective
https://doi.org/10.1038/s41591-018-0316-z Nature MediCine

One set of opportunities centers on genome-wide association 7. Yohannes Kassahun, et al. Surgical robotics beyond enhanced dexterity
(GWA) studies—large case-control studies that seek to discover instrumentation: a survey of machine learning techniques and their role in
intelligent and autonomous surgical actions. Int. J. Comput. Assist. Radio. Surg. 11,
causal genetic mutations affecting specific traits. Analyzing GWA 553–568 (2016).
studies requires algorithms that scale to very large patient cohorts 8. Litjens, G. et al. A survey on deep learning in medical image analysis.
and that deal with latent confounders. These challenges can be Med. Image Anal. 42, 60–88 (2017).
addressed via optimization tools and techniques developed for 9. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep
deep learning—including stochastic optimization and other mod- neural networks. Nature 542, 115–118 (2017).
10. Haenssle, H. A. et al. Man against machine: diagnostic performance
ern algorithms47 combined with software frameworks for scaling of a deep learning convolutional neural network for dermoscopic melanoma
computation in parallel48—as well as through modeling techniques recognition in comparison to 58 dermatologists. Ann. Oncol. 29,
that handle unseen confounders49. In the near future, models that 1836–1842 (2018).
integrate external modalities and additional sources of biological 11. Cheng, J.-Z. et al. Computer aided diagnosis with deep learning architecture:
data into GWA studies—e.g., medical images or measurements of applications to breast lesions in us images and pulmonary nodules
in CT scans. Sci. Rep. 6, 24454 (2016).
splicing and other intermediary molecular phenotypes—may also 12. Cicero, M. et al. Training and validating a deep convolutional neural network
benefit from deep learning to more accurately identify disease-asso- for computer-aided detection and classification of abnormalities on frontal
ciated causal mutations50. chest radiographs. Invest. Radiol. 52, 281–287 (2017).
Understanding the genetics of disease allows clinicians to recom- 13. Kooi, T. et al. Large scale deep learning for computer aided detection of
mend treatments and provide more accurate diagnoses. A key chal- mammographic lesions. Med. Image Anal. 35, 303–312 (2017).
14. Barreira, C. M. et al. Abstract WP61: Automated large artery occlusion
lenge for physicians is determining if novel variants in a patient’s detection in st roke imaging-paladin study. Stroke 49, AWP61 (2018).
genome are medically relevant. In part, this decision relies on pre- 15. Gulshan, V. et al. Development and validation of a deep learning algorithm
dicting the pathogenicity of mutations; a task which already uses for detection of diabetic retinopathy in retinal fundus photographs.
features like protein structure and evolutionary conservation to JAMA 316, 2402–2410 (2016).
train learning algorithms51. Given their greater power and ability to 16. Poplin, R. et al. Webster. Prediction of cardiovascular risk factors from
retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2,
effectively integrate disparate data types, deep-learning techniques 158–164 (2018).
are likely to provide more accurate pathogenicity predictions than 17. Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral
are possible today52. in retinal disease. Nat. Med. 24, 1342 (2018).
Machine learning also plays a role in phenotype prediction 18. Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis
from genetic data, including complex traits such as height as well detection in breast cancer histology images with deep neural networks. In
International Conference on Medical Image Computing and Computer-assisted
as disease risk. Deep learning can further enhance such models Intervention 411–418 (Springer, 2013).
by integrating additional modalities such as medical images, clini- 19. Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images.
cal history, and wearable device data53. A particularly promising Preprint at https://arxiv.org/abs/1703.02442 (2017).
approach to phenotype prediction is to predict intermediate molec- 20. Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal
ular phenotypes—e.g., gene expression or gene splicing—which genotype–immunophenotype relationships and predictors of response to
checkpoint blockade. Cell Rep. 18, 248–262 (2017).
then feed into downstream disease predictors54. Intermediate 21. Beck, A. H. et al. Systematic analysis of breast cancer morphology
molecular states can be easier to predict than human traits because uncovers stromal features associated with survival. Sci. Transl. Med. 3,
of larger, more proximal signals and more extensive training data. 108ra113 (2011).
These two features make the problem a good fit for deep learning, 22. Yosinski, J., Clune, J., Bengio, Y. and Lipson, L. How transferable are features
which has shown success at predicting splicing55 and transcription in deep neural networks? In Advances in Neural Information Processing Systems
3320–3328 (2014).
factor binding56. 23. Jamaludin, A., Kadir, T. and Zisserman, A. Spinenet: automatically
Genomic data can also directly serve as a biomarker for the pinpointing classification evidence in spinal mris. In International Conference
onset and progression of disease. For example, blood contains small on Medical Image Computing and Computer-Assisted Intervention
fragments of cell-free DNA released from cells present elsewhere 166–175 (Springer, 2016).
in the body. These fragments are noninvasive indicators of organ 24. Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by
image-based deep learning. Cell 172, 1122–1131 (2018).
rejection (i.e., the immune system attacking graft cells57), bacterial 25. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for
infection58, and early-stage cancer59. Cell-free DNA is successfully biomedical image segmentation. In International Conference on Medical Image
used in prenatal diagnostics: fetal DNA present in the mother’s Computing and Computer-Assisted Intervention (2015).
blood indicates chromosomal aberrations and can reveal the whole 26. Goodfellow, I. et al. Generative adversarial nets.
genome of the fetus60. Biomarker data are often noisy and requires In Advances in Neural Information Processing Systems 2672–2680 (2014).
27. Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning
sophisticated analysis (e.g., to determine whether cell-free DNA is with neural networks. In Advances in Neural Information Processing Systems
indicative of cancer); deep-learning systems can enhance the quality 3104–3112 (2014).
of biomarker assays targeting DNA sequences61, methylation62, gene 28. Wu, Y. et al. Google’s neural machine translation system: bridging the gap
expression63, chromatin64 profiles, and many other measurements. between human and machine translation. Preprint at https://arxiv.org/
abs/1609.08144 (2016).
29. Kannan, A. et al. Smart reply: automated response suggestion for email. In
Received: 17 October 2018; Accepted: 28 November 2018;
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
Published online: 7 January 2019 Discovery and Data Mining (ACM, 2016).
30. Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. Show and tell: a neural image
References caption generator. In Proceedings of the IEEE Conference on Computer Vision
1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015). and Pattern Recognition 3156–3164 (2015).
2. Russakovsky, O.et al. Imagenet large scale visual recognition challenge. Int. J. 31. The Office of the National Coordinator for Health Information Technology.
Compute. Vis. 115, 211–252 (2015). Quick stats: health IT dashboard. https://dashboard.healthit.gov/quickstats/
3. Hirschberg, J. & Manning, C. D. Advances in natural language processing. quickstats.php (2017).
Science 349, 261–266 2015). 32. Shickel, B., Tighe, P. J., Bihorac, A. & Rashidi, P. Deep EHR: a survey of
4. Geoffrey Hinton, et al. Deep neural networks for acoustic modeling in speech recent advances in deep learning techniques for electronic health record
recognition: the shared views of four research groups. IEEE Signal Process. (EHR) analysis. IEEE J. Biomed. Health Inform. 22, 1589–1604 (2017).
Mag. 29, 82–97 (2012). 33. Rajkomar, A. et al. Scalable and accurate deep learning with
5. Stanford Health. Harnessing the power of data in health. Stanford Medicine electronic health records. NPJ Digit. Med. 1, 18 (2018).
2017 Health Trends Report (2017). 34. Miotto, R. et al. Deep patient: an unsupervised representation to predict
6. Silver, D. et al. Mastering the game of go with deep neural networks and tree the future of patients from the electronic health records. Sci. Rep. 6,
search. Nature 529, 484–489 (2016). 26094 (2016).

28 Nature Medicine | VOL 25 | JANUARY 2019 | 24–29 | www.nature.com/naturemedicine

 FOCUS | Perspective
|
Nature MediCine FOCUS Perspective
https://doi.org/10.1038/s41591-018-0316-z

35. Liu, V., Kipnis, P., Gould, M. K. & Escobar, G. J. Length of stay predictions: 56. Alipanahi, B. et al. Predicting the sequence specificities of dna-and
improvements through the use of automated laboratory and rna-binding proteins by deep learning. Nature Biotechnol. 33, 831–838 (2015).
comorbidity variables. Med. Care 48, 739–744 (2010). 57. Snyder, T. M., Khush, K. K., Valantine, H. A. & Quake, S. R. Universal
36. Choi, E. et al. Doctor AI: predicting clinical events via recurrent neural networks. noninvasive detection of solid organ transplant rejection.
In Machine Learning for Healthcare 301–318 (2016). Proc. Natl Acad. Sci. USA 108, 6229–6234 (2011).
37. Che, Z. et al. Recurrent neural networks for multivariate time series with 58. Abril, M. K. et al. Diagnosis of capnocytophaga canimorsus sepsis by
missing values. Rep. 8, 1–12 (2018). whole-genome next-generation sequencing. In Open Forum Infectious Diseases
38. Suresh, H. et al. Clinical intervention prediction and understanding with Vol. 3, ofw144 (Oxford University Press, 2016).
deep neural networks. PMLR 68, 322–377 (2017). 59. Forshew, T. et al. Noninvasive identification and monitoring of cancer
39. Johnson, A. E. W.et al. Mimic-iii, a freely accessible critical care database. mutations by targeted deep sequencing of plasma dna. Sci. Transl. Med. 4,
Sci. Data 3, 160035 (2016). 136ra68–136ra68 (2012).
40. Mao, Q.et al. Multicentre validation of a sepsis prediction algorithm using 60. Fan, H. C. et al. Non-invasive prenatal measurement of the fetal genome.
only vital sign data in the emergency department, general ward and icu. Nature 487, 320–324 (2012).
BMJ Open 8, e017833 (2018). 61. Ching, T. et al. Opportunities and obstacles for deep learning in biology and
41. Abbeel, P. & Ng, A. Y. Apprenticeship learning via inverse reinforcement medicine. Preprint at https://doi.org/10.1101/142760 (2017).
learning. In Proceedings of the Twenty-First International Conference on 62. Angermueller, C., Lee, H. J., Reik, W. & Stegle, O. Deepcpg: accurate
Machine Learning 1 (ACM, 2004). prediction of single-cell dna methylation states using deep learning.
42. Ratliff, N. D., Silver, D. & Bagnell, J. A. Learning to search: functional Genome Biol. 18, 67 (2017).
gradient techniques for imitation learning. Autonomous Robots 27, 63. Chen, Y. et al. Gene expression inference with deep learning.
25–53 (2009). Bioinformatics 32, 1832–1839 (2016).
43. Schulman, J. et al. A case study of trajectory transfer through non-rigid 64. Koh, P. W., Pierson, E. & Kundaje, A. Denoising genome-wide histone chip-seq
registration for a simplified suturing scenario. In 2013 IEEE/RSJ International with convolutional neural networks. Bioinformatics 33, i225–i233 (2017).
Conference on Intelligent Robots and Systems (IROS) 4111–4117 (IEEE, 2013).
44. Mayer, H.et al. A system for robotic heart surgery that learns to tie knots Acknowledgements
using recurrent neural networks. Adv. Robot 22, 1521–1537 (2008). The authors would like to thank D. Wang, E. Dorfman, and A. Rajkomar for the visual
45. Shvets, A., Rakhlin, A., Kalinin, A. A. and Iglovikov, V. Automatic instrument design of the figures in this paper and P. Nejad for insightful conversation and ideas.
segmentation in robot-assisted surgery using deep learning. Preprint at
https://arxiv.org/abs/1803.01207 (2018).
46. Jin, A. et al. Tool detection and operative skill assessment in surgical videos Author contributions
using region-based convolutional neural networks. Preprint at https://arxiv. B.R., V.K., M.D., and K.C. share second authorship. C.C., G.C., and S.T. share third
org/abs/1802.08774 (2018). authorship. J.D. is the principal investigator. A.E. and A.R. conceptualized the structure
47. Loh, P.-R. et al. Efficient bayesian mixed-model analysis increases association of the review and contributed to the computer vision and reinforcement learning
power in large cohorts. Nat. Genet. 47, 284 (2015). sections. V.K., B.R., and M.D. contributed to the generalized deep learning section. K.C.
48. Abadi, M. et al. Tensorflow: Large-scale machine learning on heterogeneous and J.D. contributed to the natural language processing section. C.C., G.C., S.T., and J.D.
distributed systems. http://download.tensorflow.org/paper/whitepaper2015.pdf oversaw the work. All authors contributed to multiple parts of the review, as well as the
(2015). style and overall contents.
49. Tran, D. and Blei, D. M. Implicit causal models for genome-wide association
studies. In International Conference on Learning Representations (2018). Competing interests
50. Lee, S.-I. et al. Learning a prior on regulatory potential from eqtl data. M.D., C.C., K.C., G.C. and J.D. are employees of Google Inc. This work was internally
PLoS Genet. 5, e1000358 (2009). funded by Google Inc. G.C. is a board member at the Partnership on AI to Benefit People
51. Kircher, M. et al. A general framework for estimating the relative and Society. S.T. is an employee of Udacity, Inc. and the Kitty Hawk Corporation. He
pathogenicity of human genetic variants. Nat. Genet. 46, 310–315 (2014). is on the faculty of Stanford University and Georgia Institute of Technology. B.R. is a
52. Quang, D., Chen, Y. & Xie, X. Dann: a deep learning approach for annotating partner of Computable LLC.​
the pathogenicity of genetic variants. Bioinformatics 31, 761–3 (2015).
53. Dudley, J. T. et al. Personalized medicine: from genotypes, molecular
phenotypes and the quantified self, towards improved medicine. Additional information
In Pacific Symposium on Biocomputing 342–346 (2014). Reprints and permissions information is available at www.nature.com/reprints.
54. Leung, M. K. K., Delong, A., Alipanahi, B. & Frey, B. J. Machine learning in Correspondence should be addressed to A.E.
genomic medicine: a review of computational problems and data sets.
Proc. IEEE 104, 176–197 (2016). Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in
55. Xiong, H. Y. et al. The human splicing code reveals new insights into the published maps and institutional affiliations.
genetic determinants of disease. Science 347, 1254806 (2015). © Springer Nature America, Inc. 2019

Nature Medicine | VOL 25 | JANUARY 2019 | 24–29 | www.nature.com/naturemedicine 29

View publication stats