Introducton
Deep Learnint o
g
M
1-SA ST
2- JU IE
L
3-LA S
4-NA
VISUAL 1
Types of Learning
Supervised: Learning with a labeled training set
Example: email classification with already labeled emails
Unsupervised: Discover patterns in unlabeled data
Example: cluster similar documents based on text
Reinforcement learning: learn to act based on feedback/reward
Example: learn to play Go, reward: win or lose
class A
class A
Classificaton Regression Clustering
Anomaly Detecton
Sequence labeling
[Link]
…
ARTIFICIAL INTELLIGENCE
The INTELLIGENCE exhibited by machines or software, and the branch of computer
science that develops machines and software with INTELLIGENCE.
ARTIFICIAL INTELLIGENCE AI- is the intelligence of machines and the branch of
computer science that aims to create it.
It is the study and design of intelligent agents.
AI is a broad discipline that promises to simulate numerous innate human skills such as
automatic programming, case-based reasoning, neural networks, decision-making, expert
systems, natural language processing, pattern recognition and speech recognition etc.
MACHINE LEARNING
A branch of artificial intelligence, concerns the construction and study of systems that can
LEARN from data
SOURCE WIKIPEDIA
Machine Learning Basics
Machine learning is a field of computer science that gives computers the ability to
learn without being explicitly programmed
Machine Learning
Labeled Data algorithm
Training
Prediction
Labeled Data Learned model Predicton
Methods that can learn from and make predictons on data
ML vs. Deep Learning
Most machine learning methods work well because of human-designed
representations and input features
ML becomes just optimizing weights to best make a final predicton
What is Deep Learning (DL) ?
A machine learning subfield of learning representations of data. Exceptonal effectve
at learning patterns.
Deep learning algorithms attempt to learn (multple levels of) representaton by using
a hierarchy of multiple layers
If you provide the system tons of information, it begins to understand it and respond
in useful ways.
[Link]
Why is DL useful?
o Manually designed features are often over-specified, incomplete and take a long time
to design and validate
o Learned Features are easy to adapt, fast to learn
o Deep learning provides a very flexible, (almost?) universal, learnable framework for
representng world, visual and linguistc informaton.
o Can learn both unsupervised and supervised
o Effectve end-to-end joint system learning
o Utlize large amounts of training data
In ~2010 DL started outperforming other
ML techniques
first in speech and vision, then NLP
Neural Network Intro
Weights
Actvaton functons
How do we train?
4 + 2 = 6 neurons (not countng inputs)
[3 x 4] + [4 x 2] = 20 weights
4 + 2 = 6 biases
26 learnable parameters
Demo
Hype or Reality?
Quotes
I have worked all my life in Machine Learning, and I’ve never seen one
algorithm knock over benchmarks like Deep Learning
– Andrew Ng (Stanford & Baidu)
Deep Learning is an algorithm which has no theoretical limitations
of what it can learn; the more data you give and the more
computational time you provide, the better it is – Geoffrey Hinton (Google)
Human-level artificial intelligence has the potential to help humanity
thrive more than any invention that has come before it – Dileep George
(Co-Founder Vicarious)
For a very long time it will be a complementary tool that human
scientists and human experts can use to help them with the things
that humans are not naturally good – Demis Hassabis (Co-Founder DeepMind)
Hype or Reality?
Deep Learning at Google
Hype or Reality?
NIPS (Computational Neuroscience Conference) Growth
What is Artificial Intelligence?
Input: Output:
Artificial
Sensors Movement
Data
Intelligence Text
Machine Learning - Basics
Introduction
Machine Learning is a type of Artificial Intelligence that provides
computers with the ability to learn without being explicitly
programmed.
Machine Learning
Algorithm
Labeled Data
Training
Prediction
Learned Model Prediction
Data
Provides various techniques that can learn from and make predictions on data
Machine Learning - Basics
Learning Approaches
Supervised Learning: Learning with a labeled training set
Example: email spam detector with training set of already labeled emails
Unsupervised Learning: Discovering patterns in unlabeled data
Example: cluster similar documents based on the text content
Reinforcement Learning: learning based on feedback or reward
Example: learn to play chess by winning or losing
Machine Learning - Basics
Problem Types
Classification Regression
(supervised – predictive) (supervised – predictive)
Clustering Anomaly Detection
(unsupervised – descriptive) (unsupervised– descriptive)
What is Deep Learning?
Part of the machine learning field of learning representations of
data. Exceptional effective at learning patterns.
Utilizes learning algorithms that derive meaning out of data by using
a hierarchy of multiple layers that mimic the neural networks of our
brain.
If you provide the system tons of information, it begins to
understand it and respond in useful ways.
Inspired by the Brain
The first hierarchy of neurons
that receives information in the
visual cortex are sensitive to
specific edges while brain regions
further down the visual pipeline
are sensitive to more complex
structures such as faces.
Our brain has lots of neurons connected together and the strength of
the connections between neurons represents long term knowledge.
1
One learning algorithm hypothesis: all significant mental algorithms
are learned except for the learning and reward machinery itself.
Why Deep Learning?
Applications
Speech Computer Natural Language
Recognition Vision Processing
What changed?
Old wine in new bottles
Big Data Computation Algorithmic
(Digitalization) (Moore’s Law, GPUs) Progress
The Big Players
Superstar Researchers
Geoffrey Hinton: University of Toronto & Google
Yann LeCun: New York University & Facebook
Andrew Ng: Stanford & Baidu
Yoshua Bengio: University of Montreal
Jürgen Schmidhuber: Swiss AI Lab & NNAISENSE
The Big Players
Companies
The Big Players
Startups
DNNresearch
Acquired
Deep Learning - Basics
No more feature engineering
Traditional
Feature
Learning
Engineering
Input Data Algorithm
Costs lots of time
Deep
Learning
Input Data Algorithm
Deep Learning - Basics
Architecture
A deep neural network consists of a hierarchy of layers, whereby each layer
transforms the input data into more abstract representations (e.g. edge ->
nose -> face). The output layer combines those features to make predictions .
Deep Learning - Basics
What did it learn?
Edges Nose, Eye… Faces
Deep Learning - Basics
Artificial Neural Networks
Consists of one input, one output and multiple fully-connected hidden layers in-
between. Each layer is represented as a series of neurons and progressively extracts
higher and higher-level features of the input until the final layer essentially makes a
decision about what the input shows. The more layers the network has, the higher-
level features it will learn.
Deep Learning - Basics
The Neuron
An artificial neuron contains a nonlinear activation function and has
several incoming and outgoing weighted connections.
Neurons are trained to filter and detect specific features or patterns
(e.g. edge, nose) by receiving weighted input, transforming it with
the activation function und passing it to the outgoing connections.
Deep Learning - Basics
The Training Process
Forward it trough
the network to get
Sample labeled data predictions
Update the Backpropagate
connection weights the errors
Learns by generating an error signal that measures the difference between the
predictions of the network and the desired values and then using this error signal
to change the weights (or parameters) so that predictions get more accurate.
Deep Learning - Basics
Usage Requirements
Large data set with good quality (input-output mappings)
Measurable and describable goals (define the cost)
Enough computing power (AWS GPU Instance)
Excels in tasks where the basic unit (pixel, word) has very little meaning
in itself, but the combination of such units has a useful meaning
Deep Learning - Tools
Computing is affordable
AWS EC2 GPU Spot Instance: g2.2xlarge - $0.0782 per Hour
The DIGITS DevBox combines the
world’s best hardware (4 GPUs),
software, and systems engineering
for deep learning in a powerful
solution that can fit under your
desk. Cost: $15k
Outlook
NVIDIA Pascal
NVIDIA’s Pascal GPU architecture will accelerate
deep learning applications up to 10X beyond the
speed of its current-generation Maxwell processors.
Outlook
Artificial Quantum Intelligence
Quantum Artificial Intelligence Lab is a joint initiative of NASA and Google to
study how quantum computing might advance machine learning. This type of
computing may provide the most creative and parallelized problem-solving
process under the known laws of physics.
Quantum computers handle what are called quantum bits
or qubits that can readily have a value of one or zero or
anything in between.
Quantum computing represents a paradigm shift, a radical
change in the way we do computing and at a scale that has
unimaginable power – Eric Ladizinsky (Co-founder D-Wave)
Outlook
Neuromorphic Chips
IBM TrueNorth is a brain-inspired computer chip that implements
networks of integrate-and-fire spiking artificial neurons and uses
only a tiny 70 mw of power –orders of magnitude less energy
than traditional chips. The system is designed to be able to run
deep-learning algorithms.
1 million 256 million 4096
Programmable Programmable Neurosynaptic
Neurons Synapses Cores
Outlook
Deep Learning
Significant advances in deep reinforcement and unsupervised
learning
Bigger and more complex architectures based on various
interchangeable modules/techniques
Deeper models that can learn from much fewer training cases
Harder problems such as video understanding and natural language
processing will be successfully tackled by deep learning algorithms
Takeaways
Machines that learn to represent the world from experience.
Deep Learning is no magic! Just statistics in a black box, but
exceptional effective at learning patterns.
We haven’t figured out creativity and human-empathy.
Transitioning from research to consumer products. Will make the
tools you use every day work better, faster and smarter.
