Introduction to Neural Networks

Deep Learning for NLP

Kevin Patel

ICON 2017

December 21, 2017

Overview

1 Motivation

2 Perceptron

3 Feed Forward Neural Networks

4 Deep Learning

5 Conclusion

Kevin Patel Neural Networks 2/53

Our brains are so awesome, that we cannot replicate their
computation
 Our brains are so awesome, that we cannot replicate their
computation
Our brains’ capability is so limited, that we have failed to replicate
their computation
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Purpose of Artificial Intelligence

Human Like AI: An AI which functions like a human and has

similar characteristics
Beneficial AI: An AI which works in a fundamentally different
manner, but gets the job done

Kevin Patel Neural Networks 4/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

The Human Brain

Brain - a large network of neurons

Neuron - a cell capable of receiving, processing and
transmitting information via electric and chemical signals

Dendrites receive input

Axon transmits output
Kevin Patel Neural Networks 5/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptron

A simple artificial neuron Rosenblatt (1958)

Input: one or more binary values xi
Output: single binary value y
Output computed using weighted sum of inputs and a
threshold
Giving different weights to different features while making a
decision

Kevin Patel Neural Networks 6/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptron (Contd.)

x1
w1
x2 w2 y
w3
x3

{ ∑
1, if wi xi > threshold
y=
0 otherwise

Kevin Patel Neural Networks 7/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Some Conventions

Inputs also treated as neurons (no input, output is the actual

value of the feature)
∑
Rewrite wi xi as w.x
Move threshold to the other side of the equation; Call it bias
b = −threshold
{
1, if w.x + b > 0
y=
0 otherwise

Kevin Patel Neural Networks 8/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Bias

An indication of how easy it is for a neuron to fire

The higher the value of bias, the easier it is for the neuron to
fire
Consider it as a prior inclination towards some decision
The higher your initial inclination, the smaller the amount of
extra push needed to finally make some decision

Kevin Patel Neural Networks 9/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptron Example

Should I go to lab?
x1 : My guide is here
x2 : Collaborators are in lab
x3 : The buses are running
x1 x4 : Tasty tiffin in the mess
10 b: My inclination towards
x2 going to the lab no matter
2 y
2 what
x3
-3
x4

Kevin Patel Neural Networks 10/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptron Example

Should I go to lab?
x1 : My guide is here
x2 : Collaborators are in lab
x3 : The buses are running
x1 x4 : Tasty tiffin in the mess
10 b: My inclination towards
x2 going to the lab no matter
2 y
2 what
x3
-3 What if b = −3?
x4

Kevin Patel Neural Networks 10/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptron Example

Should I go to lab?
x1 : My guide is here
x2 : Collaborators are in lab
x3 : The buses are running
x1 x4 : Tasty tiffin in the mess
10 b: My inclination towards
x2 going to the lab no matter
2 y
2 what
x3
-3 What if b = −3?
x4 What if b = 1?

Kevin Patel Neural Networks 10/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Network of Perceptrons

w111
x1 w112
21 w211
w122
31
13 w212
x2 w123
41
32 w213 y
w133
42
51 w214
x3 w143
52 w215
w153

Outputs of perceptrons fed into next layer

Simpler decisions made at initial layers used as features for
making complex decisions.
Kevin Patel Neural Networks 11/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptrons and Logic Gates

OR Gate AND Gate

x1 x1
2 2
−1 −3
x2 2 x2 2

NAND Gate
NOT Gate

x1
-2
x1 3
-2 1 x2 -2

Kevin Patel Neural Networks 12/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptrons and Logic Gates (contd.)

XOR Gate
x1 −2 3
−2 −2 −2

3 3
−2 −2 −2

x2 −2 3

Kevin Patel Neural Networks 13/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Perceptrons and Logic Gates (contd.)

NAND gates are universal gates

Perceptrons can simulate NAND gates
Therefore, Perceptrons are universal for computation
Good News: Network of perceptrons is as capable as any
other computing device
Bad News: Is it just another fancy NAND gate?
Silver Lining: Learning algorithms can automatically figure out
weights and biases, whereas we need to explicitly design
circuits using NAND gates

Kevin Patel Neural Networks 14/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Learning

Simple weight update rule to learn parameters of a single

perceptron
Perceptron Convergence Theorem guarantees that learning
will converge to a correct solution in case of linearly separable
data.
However, learning is difficult in case of network of perceptrons
Ideally, a learning process involves changing one of the input
parameters by a small value, hoping that it will change the
output by a small value.
For instance, in handwritten digit recognition, if the network is
misclassifying 9 as 8, then we want
Here, a small change in parameters of a single perceptron ⇒
flipped output ⇒ change behavior of entire network
Need some machinery such that gradual change in parameters
lead to gradual change in output
Kevin Patel Neural Networks 15/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Learning (contd.)
If y is a function of x, then change in y i.e. ∆y is related to
change in x i.e. ∆y as follows (Linear Approximation)
dy
∆y ≈ ∆x
dx

Example

f(x) = x2
f′ (x) = 2x
f(4.01) ≈ f(4) + f′ (4)(4.01 − 4)
= 16 + 2 × 4 × 0.01
= 16.08

Kevin Patel Neural Networks 16/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Learning (contd.)

For a fixed datapoint with two features (x1 , x2 ), the change in

output of the perceptron depends on the corresponding
changes in weights w1 and w2 and the bias b
Thus, change in y - ∆y is
∂y ∂y ∂y
∆y ≈ ∆w1 + ∆w2 + ∆b
∂w1 ∂w2 ∂b

Partial derivative ill-defined in case of perceptron, which

creates hurdle for learning

Kevin Patel Neural Networks 17/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Sigmoid Neurons

Another simple artificial neuron McCulloch and Pitts (1943)

Input: one or more real values xi
Output: single real value y
Output computed by applying sigmoid function σ on the
weighted sum of inputs and bias
1
y = σ(w.x + b) = −(w.x+b)
1+e

Decision making using sigmoid:

Given real valued output, use threshold
If y > 0.5, output 1, else 0

Kevin Patel Neural Networks 18/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Sigmoid vs. Perceptron

When z = w.x + b >> 0

e−z ≈ 0
σ(z) ≈ 1
Similar to perceptron producing 1 as output when z is large
and positive
When z = w.x + b << 0
e−z ≈ ∞
σ(z) ≈ 0
Similar to perceptron producing 0 as output when z is large
and negative
Different primarily when absolute value of z is small.

Kevin Patel Neural Networks 19/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Sigmoid vs. Perceptron (contd.)

Step (Perceptron) Sigmoid

Sigmoid is continuous and differentiable over its domain
Learning is possible via small changes in parameters and using
linear approximation

Kevin Patel Neural Networks 20/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Activation Functions

Linear Sigmoid

Tanh ReLU
Kevin Patel Neural Networks 21/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Notations

x: Input
wlij : weight from jth neuron in (l − 1)th layer to ith neuron in
lth layer
blj : bias of the jt h neuron in the lth layer
zlj : wl .al−1 + blj
alj : f(zlj )

Kevin Patel Neural Networks 22/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Neural Network Architecture

w111
x1 w211
w112
21
w212
21
y1
w131
22
13
x2 w213
22
w123
41
32
w214
23
y2
1
w42
33
x3 w224
w143

For MNIST Digit recognition

Input Layer 28 × 28 = 784 neurons
Output Layer?

Kevin Patel Neural Networks 23/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

One-hot output vs. Binary encoded output

Given 10 digits, we fixed 10 neurons in output layer

Why not 4 neurons, and generate binary representation of
digits?
The task is to observe features and learn to decide whether it
is a particular digit
Observing visual features and trying to predict, say, most
significant bit, will be hard
Almost no correlation there

Kevin Patel Neural Networks 24/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Feed Forward Computation

Given input x
z1 = w1 .x + b1
a1 = σ(z1 )
zl = wl .al−1 + b1
al = σ(zl )
aL is the output, where L is the last layer
Note that output contains real numbers (due to σ function)

Kevin Patel Neural Networks 25/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Loss functions
Consider a network with parameter setting P1
True Predicted Correct
0 1 0 0.3 0.4 0.3 yes
1 0 0 0.1 0.2 0.7 no
0 0 1 0.3 0.3 0.4 yes
2
Number of correctly classified examples = 3
Classification error = 1 − 23 = 13
Consider same network with parameter setting P2
True Predicted Correct
0 1 0 0.1 0.7 0.2 yes
1 0 0 0.3 0.4 0.3 no
0 0 1 0.1 0.2 0.7 yes

Classification error still the same

Kevin Patel Neural Networks 26/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Loss functions
∑
Mean Squared Error : MSE = 1
M (yi − ti )2

True Predicted Correct

0 1 0 0.3 0.4 0.3 yes
1 0 0 0.1 0.2 0.7 no
0 0 1 0.3 0.3 0.4 yes

Mean Squared Error = (0.54 + 0.54 + 1.34)/3 = 0.81

True Predicted Correct

0 1 0 0.1 0.7 0.2 yes
1 0 0 0.3 0.4 0.3 no
0 0 1 0.1 0.2 0.7 yes

Mean Squared Error = (0.14 + 0.14 + 0.74)/3 = 0.34

Indicates that second parameter setting is better
Kevin Patel Neural Networks 27/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Loss functions
Mean Cross∑Entropy
MCE = M1
(−ti logyi − (1 − ti )log(1 − yi ))

True Predicted Correct

0 1 0 0.3 0.4 0.3 yes
1 0 0 0.1 0.2 0.7 no
0 0 1 0.3 0.3 0.4 yes

Mean Cross Entropy = −(ln(0.4) + ln(0.4) + ln(0.1))/3 = 1.38

True Predicted Correct
0 1 0 0.1 0.7 0.2 yes
1 0 0 0.3 0.4 0.3 no
0 0 1 0.1 0.2 0.7 yes

Mean Cross Entropy = −(ln(0.7) + ln(0.7) + ln(0.3))/3 = 0.64

Indicates that second parameter setting is better
Kevin Patel Neural Networks 28/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Minimizing Loss
Consider a function C that depends on some parameter x as
shown below:

How to find the value of x for which C is minimum?

Idea: Choose a random value for x, place an imaginary ball
there. It will eventually lead to a valley
Kevin Patel Neural Networks 29/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Gradient Descent
Recall ∆C ≈ dC
dx .∆x
We want to change x such that C is reduced i.e. ∆C has to
be always negative
What if we choose ∆x = −η dC
dx ?

dC
∆C ≈ .∆x
dx
dC dC
= .(−η )
dx dx
dC 2
= −η.( )
dx
≤0

Gradient Descent: xt+1 = xt − η dC

dx
Kevin Patel Neural Networks 30/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Gradient Descent

Kevin Patel Neural Networks 31/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Back Propagation

Effective use shown in Williams et al. (1986)

Every neuron taking part in the decision
Every neuron shares the blame for error
Decision made by a neuron dependent on its weights and
biases
Thus error is caused due to these weights and biases
Need to change weights and biases such that overall error is
reduced
Can use gradient descent here
For a weight wkij , the weight update will be

∂C
wkij ← wkij − η
∂wkij

Kevin Patel Neural Networks 32/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Back Propagation (contd.)

Will use calculus chain rule to obtain the partial derivatives
1 First find error for each neuron on the last layer

δ L = ∇a C ⊙ σ ′ (zL )

2 Then find error for each neuron on the interior neurons

δ l = ((wl+1 )T δ l+1 ) ⊙ σ ′ (zl )

3 Update weights and biases using following gradients:

∂C
= δjl
∂blj
∂C
= al−1
k δj
l
∂wljk
Kevin Patel Neural Networks 33/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Vanishing Gradient Problem

Observed by Hochreiter (1998)

Important point: the σ ′ (zl ) term in the previous steps
Derivative of the activation function
Will be multiplied at each layer during back propagation
Example: 3 layer network
δ 4 = A.σ ′ ()
δ 3 = X.δ 4 .σ ′ () = X.A.σ ′ ().σ ′ ()
δ 2 = Y.δ 3 .σ ′ () = Y.X.A.σ ′ ().σ ′ ().σ ′ ()
δ 1 = Z.δ 2 .σ ′ () = Z.Y.X.A.σ ′ ().σ ′ ().σ ′ ().σ ′ ()

Kevin Patel Neural Networks 34/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Vanishing Gradient Problem (contd.)

Sigmoid Derivative of Sigmoid

Maximum value of sigmoid’s derivative = 0.25
0.25n ≈ 0 as n → ∞
Gradient tends to 0 i.e. vanishes

Kevin Patel Neural Networks 35/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Deep Learning

Set of techniques and architectures that tackles such learning

problems and helps to reach optimal parameters faster
Various methods:
Start at near optimal values of parameters so smaller updates
due to vanishing gradients is not much of a problem
Use better activation functions which can avoid such problems
Use better optimizers than standard gradient descent
Use novel architectures

Kevin Patel Neural Networks 36/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Greedy Unsupervised

Pretraining
x1
y1
x2
y2
x3
y3
x4
y4
x5

Proposed by Bengio et al. (2007)

Kevin Patel Neural Networks 37/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Greedy Unsupervised

Pretraining
x1

x2

x3

x4

x5

Proposed by Bengio et al. (2007)

Kevin Patel Neural Networks 38/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Greedy Unsupervised

Pretraining

Proposed by Bengio et al. (2007)

Kevin Patel Neural Networks 39/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Greedy Unsupervised

Pretraining

Proposed by Bengio et al. (2007)

Kevin Patel Neural Networks 40/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Greedy Unsupervised

Pretraining
x1
y1
x2
y2
x3
y3
x4
y4
x5

Proposed by Bengio et al. (2007)

Kevin Patel Neural Networks 41/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Novel Activation

Functions

Rectified Linear Unit Nair and Hinton (2010): Derivative = 1

when non-zero, else 0
Product of derivatives does not vanish
But once a ReLU gets to 0, it is difficult to get it to one again
(Dead ReLU problem)
Addressed by better variants such as Leaky ReLU Maas et al.
(2013), Parametric ReLU He et al. (2015) etc.

Kevin Patel Neural Networks 42/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Types of Gradient Descent

Based on the amount of data used for training

Batch GD: all training data per update, slow, not applicable in
online setting, but guaranteed to converge to global minimum
for convex and local minimum for non-convex
Stochastic GD: one training datapoint per update, fluctuates
a lot, allows to jump to new and potentially better local
minima, this complicates convergence, has been shown that
by decreasing learning rate almost certainly converges to
global in convex and local in non-convex
Mini-batch GD: batch of n datapoints per update, best of
both worlds - relatively stable convergence and can use matrix
operations for batches

Kevin Patel Neural Networks 43/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Learning Difficulties via Optimizers

SGD mainly used for a long time

Converges slowly
Can get stuck in local minima

Kevin Patel Neural Networks 44/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

SGD

Kevin Patel Neural Networks 45/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

SGD + Momentum

Kevin Patel Neural Networks 46/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Nesterov Accelerated Gradient

Developed by Nesterov (1983)

Kevin Patel Neural Networks 47/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Other Optimizers

AdaGrad: Adapts the learning rate to the parameters,

performing larger updates for infrequent and smaller updates
for frequent parameters Duchi et al. (2011)
AdaDelta: Does not need an initial learning rate Zeiler (2012)
RMSProp: Good with recurrent networks; Unpublished
method from Hinton’s Coursera Lectures
Adam: Benefits of RMSProp and AdaDelta mixed with
momemtum tricks Kingma and Ba (2014)

Kevin Patel Neural Networks 48/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Tackling Vanishing Gradients via Novel Architectures

Novel architectures made to specific problems

Example:
Derivative of activation function in LSTM is identity function
is 1. Gradient does not vanish
Effective weight depends on forget gate activation, whose
derivative is never > 1.0. So Gradient does not explode
Will be covered in other sessions

Kevin Patel Neural Networks 49/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Conclusion

Exciting area of research

Heavy involvement from industry
Many developments in each of those subareas: Activation
functions, Optimizers, Architectures, etc.

Kevin Patel Neural Networks 50/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

References I

Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. (2007).

Greedy layer-wise training of deep networks. In Advances in
neural information processing systems, pages 153–160.
Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient
methods for online learning and stochastic optimization. Journal
of Machine Learning Research, 12(Jul):2121–2159.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. In Proceedings of the IEEE international
conference on computer vision, pages 1026–1034.
Hochreiter, S. (1998). Recurrent neural net learning and vanishing
gradient. International Journal Of Uncertainity, Fuzziness and
Knowledge-Based Systems, 6(2):107–116.
Kevin Patel Neural Networks 50/53
 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

References II

Kingma, D. and Ba, J. (2014). Adam: A method for stochastic

optimization. arXiv preprint arXiv:1412.6980.
Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). Rectifier
nonlinearities improve neural network acoustic models. In Proc.
ICML, volume 30.
McCulloch, W. S. and Pitts, W. (1943). A logical calculus of the
ideas immanent in nervous activity. The bulletin of
mathematical biophysics, 5(4):115–133.
Nair, V. and Hinton, G. E. (2010). Rectified linear units improve
restricted boltzmann machines. In Proceedings of the 27th
international conference on machine learning (ICML-10), pages
807–814.

Kevin Patel Neural Networks 51/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

References III

Nesterov, Y. (1983). A method for unconstrained convex

minimization problem with the rate of convergence o (1/k̂ 2). In
Doklady AN USSR, volume 269, pages 543–547.
Rosenblatt, F. (1958). The perceptron: A probabilistic model for
information storage and organization in the brain. Psychological
review, 65(6):386.
Williams, R. J., Rumelhart, D. E., and Hinton, G. E. (1986).
Learning representations by back-propagating errors. Nature,
323(6088):533–538.
Zeiler, M. D. (2012). Adadelta: an adaptive learning rate method.
arXiv preprint arXiv:1212.5701.

Kevin Patel Neural Networks 52/53

 Motivation Perceptron Feed Forward Neural Deep Learning Conclusion
Networks

Thank You

Kevin Patel Neural Networks 53/53