Artificial
Intelligence
Introduction to Neural Networks and
Deep Learning Frameworks
Copyright Intellipaat. All rights reserved.
Agenda
Topology of Neural
01 Networks 02 Perceptrons
Activation Functions Perceptron Training
03 and Their Types
04 Algorithm
Deep Learning 06 What Are Tensors?
05 Frameworks
Program Elements
07 Computational Graph 08 in TensorFlow
Copyright Intellipaat. All rights reserved.
Topology of a Neural Network
Typically, artificial neural networks have a layered structure. The Input Layer picks up the input signals and passes them on to the next layer, also
known as the ‘Hidden’ Layer (there may be more than one Hidden Layer in a neural network). Last comes the Output Layer that delivers the result
Input layer Hidden layer Output layer
Copyright Intellipaat. All rights reserved.
Well, everyone has heard about AI, but how
many of you know that the inspiration behind
artificial neural networks came from the
biological neurons that are found within
human brains?
Copyright Intellipaat. All rights reserved.
Let us first understand the architecture of
our biological neurons which is very similar to
that of artificial neurons
Copyright Intellipaat. All rights reserved.
Neurons: How Do They Work?
A neural network is a computer simulation of the way biological neurons work within a human brain
Dendrites: These branch-like structures extending away from the cell body
receive messages from other neurons and allow them travel to the cell body
Cell Body: It contains a nucleus, smooth and rough endoplasmic reticulum,
Golgi apparatus, mitochondria, and other cellular components
Axon: An axon carries an electrical impulse from the cell body to another
neuron
Copyright Intellipaat. All rights reserved.
Now, let us understand about artificial
neurons in detail!
Copyright Intellipaat. All rights reserved.
Artificial Neurons
▪ The most fundamental unit of a deep neural network is
called as an artificial neuron
▪ It takes an input, processes it, passes it through an
activation function, and returns the output
▪ Such type of artificial neurons are called as perceptrons
▪ A perceptron is a linear model used for binary
classification
Schematic Representation of a Neuron in a Neural Network
Copyright Intellipaat. All rights reserved.
Perceptron: How Does It Work?
▪ The three arrows correspond to the three inputs coming into the network
▪ Values [0.7, 0.6, and 1.4] are weights assigned to the corresponding input
▪ Inputs get multiplied with their respective weights and their sum is taken
▪ Consider the three inputs as x1, x2, and x3
▪ Let the three weights be w1, w2, and w3
Sum = x1w1 + x2w2+x3w3
Sum=x1(0.7) + x2(0.6) + x3(1.4)
▪ An offset is added to this sum. This offset is called Bias
▪ It is just a constant number, say 1, which is added for scaling purposes
New_Sum = x1(0.7) + x2(0.6) + x3(1.4) + bias
Copyright Intellipaat. All rights reserved.
Why Do We Need Weights?
▪ Statistically, weights determine the relative importance of input
▪ Mathematically, they are just the slope of the line
Copyright Intellipaat. All rights reserved.
Why Do We Need Weights?
Will it rain, if I
wear a blue
shirt?
Humidity x1
Output
Will it rain? (0/1)
Blue shirt x2
w2 is assigned a lower value because significance of the
input ‘blue shirt’ is less than ‘humidity’
Copyright Intellipaat. All rights reserved.
Why Do We Need Activation Functions?
We have two classes. One set is
represented with triangles and
the other with circles
Copyright Intellipaat. All rights reserved.
Why Do We Need Activation Functions?
Draw me a linear decision
boundary which can separate
these two classes
Copyright Intellipaat. All rights reserved.
Why Do We Need Activation Functions?
We will have to add a third
dimension to create a linearly
separable model which is easy to
deal with
Copyright Intellipaat. All rights reserved.
Activation Functions
▪ They are used to convert an input signal of a node in an
artificial neural network to an output signal
▪ That output signal now is used as an input in the next layer in
the stack
▪ Activation functions introduce non-linear properties to our
network
▪ A neural network without an activation function is essentially
just a linear regression model
▪ The activation function does non-linear transformation to the
input making it capable to learn and perform more complex
tasks
Copyright Intellipaat. All rights reserved.
Identity Types of Activation Functions
Binary Step
Sigmoid
Tanh
ReLU
Leaky ReLU
Softmax
Copyright Intellipaat. All rights reserved.
Identity Function
• A straight line function where activation is proportional to input
• No matter how many layers we have, if all of them are linear in nature, the final activation function of the last
layer will be nothing but just a linear function of the input of the first layer
• We use a linear function to solve a linear regression problem
• Range: (−∞,∞)
𝒇 𝒙 =𝒙
Copyright Intellipaat. All rights reserved.
Binary Step Function
• It is also known as the Heaviside step function, or the unit step function, which is usually denoted by H or θ, is a
discontinuous function
• Its value is 0 for the negative argument and 1 for the positive argument
• It depends on the threshold value we define
• We use the binary step function to solve a binary classification problem
𝟎 𝒇𝒐𝒓 𝒙 < 𝟎 • Range: (0,1)
𝒇 𝒙 =ቐ
𝟏 𝒇𝒐𝒓 𝒙 = > 𝟎
Copyright Intellipaat. All rights reserved.
Sigmoid Function
• The sigmoid function is an activation function where it scales values between 0 and 1 by applying a threshold
• When we apply the weighted sum in the place of x, the values are scaled in between 0 and 1
• Large negative numbers are scaled toward 0, and large positive numbers are scaled toward 1
• Range: (0,1)
𝟏
𝒇 𝒙 =
𝟏 + ⅇ−𝒙
Copyright Intellipaat. All rights reserved.
Tanh Function
• It is a hyperbolic trigonometric function
• The Tanh activation works almost always better than sigmoid functions as optimization is easier in this method
• The advantage of Tanh is that it can deal more easily with negative numbers
• It is actually a mathematically shifted version of the sigmoid function
• Range: (−1,1)
𝟐
𝒇 𝒙 : 𝒕𝒂𝒏 𝒉 𝒙 = 𝒙 −𝟏
𝟏+ⅇ−𝟐
Copyright Intellipaat. All rights reserved.
ReLU Function
• ReLU stands for rectified linear unit
• It is the most widely used activation function
• It is primarily implemented in Hidden Layers of the neural network
• This function allows only the maximum values to pass during the front propagation as shown in the graph below
• Range: (0,∞)
𝟎 𝒇𝒐𝒓 𝒙 < 𝟎
𝒇 𝒙 =ቐ
𝒙 𝒇𝒐𝒓 𝒙 = > 𝟎
Copyright Intellipaat. All rights reserved.
Leaky ReLU Function
• Leaky ReLU allows a small negative value to pass during the back propagation if we have a dead ReLU problem
• This eventually activates the neuron and brings it down
• Range: (−∞,∞)
𝟎. 𝟎𝟏𝒙 𝒇𝒐𝒓 𝒙 < 𝟎
𝒇 𝒙 =ቐ
𝒙 𝒇𝒐𝒓 𝒙 = > 𝟎
Copyright Intellipaat. All rights reserved.
Softmax Function
• The Softmax function is used when we have multiple classes
• It is useful for finding out the class which has the max. probability
• The Softmax function is ideally used in the Output Layer of the classifier where we are actually trying to attain the
probabilities to define the class of each input
• Range: (0,1)
ⅇ𝒛 𝒋
𝜎 𝒛 𝒋 = σ𝑲 𝒛 , 𝒋 = 𝟏, 𝟐, . . 𝑲
𝒌=𝟏 ⅇ 𝒌
Copyright Intellipaat. All rights reserved.
Got bored again? Let us get back to
perceptrons and try to understand them in a
better way!
Copyright Intellipaat. All rights reserved.
Training a Perceptron
By training a perceptron, we try to find a line, plane, or some hyperplane which can accurately separate two
classes by adjusting weights and biases
Error = 2 Error = 1 Error = 0
Copyright Intellipaat. All rights reserved.
Perceptron Training Algorithm
Input
W1
Bias
X1
Activation Initialize weights,
X2 W2 Function bias, and threshold
Output
Calculate the sum
Update weights
X3 W3 and pass through an
activation function 𝜕𝐸
Wnew = Wold – LR*( )
𝜕𝑤
Error Produce the output
(Error)
Xn Wn
(Correct)
Stop
Copyright Intellipaat. All rights reserved.
Benefits of Using Artificial Neural Networks
Organic Non-linear Data
Fault Tolerance Self-repairing
Learning Processing
Copyright Intellipaat. All rights reserved.
Let us now move toward Deep
Learning frameworks!
Copyright Intellipaat. All rights reserved.
Deep Learning Frameworks
These Deep Learning libraries help in implementing artificial neural networks
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
TensorFlow is an open-source software library for high-performance numerical computations
MXNet
Developed by Google
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet Natural language
Forecasting
processing
Text classification Tagging
Google Translate
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Tensor
Used for visualizing TensorFlow computations and graphs
Board
TensorFlow Used for rapid deployment of new algorithms/experiments
Serving while retaining the same server architecture and APIs
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
A high-Level API which can run on top of
TensorFlow, Theano, or CNTK
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
A recurrent neural network
A convolutional neural network
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Stacks layers on top
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
A scientific computing framework developed by
Facebook
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
‘Pythonic’ in nature
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Offers dynamic computational
graphs
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
A Deep Learning programming
library written for Java
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Image recognition Fraud detection
Parts of speech
Text mining
tagging
Natural language
processing
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Developed by Apache
Software Foundation
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Copyright Intellipaat. All rights reserved.
TensorFlow
Keras
PyTorch
DL4J
MXNet
Speech
Imaging
recognition
Forecasting NLP
Copyright Intellipaat. All rights reserved.
What Are Tensors?
A tensor is a multi-dimensional array in which data is stored
Tensor is given
as an input to a
neural network
Tensor
Copyright Intellipaat. All rights reserved.
Tensor Rank
Tensor rank represents the dimension of the n-dimensional array
Rank Math Entity Example
0 Scalar (magnitude only) s = 483
1 Vector (magnitude and v = [1.1, 2.2, 3.3]
direction)
2 Matrix (table of numbers)
m = [1, 2, 3], [4, 5, 6], [7, 8, 9]]
3 3-Tensor (cube of numbers) t=
[[[2], [4], [6]], [[8], [10], [12]], [
[14], [16], [18]]]
n n-Tensor ……
Copyright Intellipaat. All rights reserved.
Computational Graph
Computation is done in the form of a graph
a = 10
b = 20 Addition
c = 30 c
ℎ = 𝑎∗𝑏 +𝑐
Multiplication
a b
Copyright Intellipaat. All rights reserved.
Computational Graph
The computational graph is executed inside a session
h h
a = 10
b = 20 Addition Addition
c = 30 c c
ℎ = 𝑎∗𝑏 +𝑐
Multiplication Multiplication
a b a b
Session
Copyright Intellipaat. All rights reserved.
Computational Graph
The computational graph is executed inside a session
a = 10 Node -> Mathematical operation
b = 20 Addition
c = 30 c
ℎ = 𝑎∗𝑏 +𝑐
Multiplication
Edge -> Tensor
a b
Copyright Intellipaat. All rights reserved.
Program Elements in TensorFlow
Constant Placeholder Variable
Copyright Intellipaat. All rights reserved.
Constant Constants are program elements whose values do not change
Placeholder
Variable a=tf.constant(10) b=tf.constant(20)
Copyright Intellipaat. All rights reserved.
Constant A placeholder is a program element to which we can assign data at a later
time
Placeholder
Variable x=tf.placeholder(tf.float32) y=tf.placeholder(tf.string)
Copyright Intellipaat. All rights reserved.
Constant A variable is a program element which allows us to add new trainable
parameters to the graph
Placeholder
Variable W=tf.Variable([3],tf.float32) b=tf.Variable([0.4],tf.float32)
Copyright Intellipaat. All rights reserved.
Quiz
Copyright Intellipaat. All rights reserved.
Quiz 1
A tensor is a single-dimensional array in
which data is stored
A True
B False
Copyright Intellipaat. All rights reserved.
Answer 1
A tensor is a single-dimensional array in
which data is stored
A True
B False
Copyright Intellipaat. All rights reserved.
Quiz 2
How many layers does a standard Neural
Network has?
A 1
B 2
C 3
D 4 or more
Copyright Intellipaat. All rights reserved.
Answer 2
How many layers does a standard Neural
Network has?
A 1
B 2
C 3
D 4 or more
Copyright Intellipaat. All rights reserved.
Quiz 3
Perceptron is other name of Neurons
A Yes
B No
Copyright Intellipaat. All rights reserved.
Answer 3
Perceptron is other name of Neurons
A Yes
B No
Copyright Intellipaat. All rights reserved.
Thank you!
Copyright Intellipaat. All rights reserved.
India: +91-7847955955
US: 1-800-216-8930 (TOLL FREE)
[email protected]
24/7 Chat with Our Course Advisor
Copyright Intellipaat. All rights reserved.
