PERCEPTRON

NETWORK,
BACKPROPAGATION
NETWORK
• The Perceptron is one of the simplest artificial 2
neural
network architectures, introduced by Frank Rosenblatt in 1957. It is primarily
used for binary classification.
• The Perceptron model proved to be highly effective in solving specific
classification problems.
• Perceptron is a type of neural network that performs binary classification that
maps input features to an output decision, usually classifying data into one of
two categories, such as 0 or 1.
• Perceptron consists of a single layer of input nodes that are fully connected to a
layer of output nodes. It is particularly good at learning linearly separable
patterns. It utilizes a variation of artificial neurons called Threshold Logic
Units (TLU), which were first introduced by McCulloch and Walter Pitts in the
1940s.
TYPES OF PERCEPTRON

1.Single layer Perceptron: It is a type of perceptron is limited to

learning linearly separable patterns. It is effective for tasks where the
data can be divided into distinct categories through a straight line.
While powerful in its simplicity, it struggles with more complex
problems where the relationship between inputs and outputs is non-
linear.
2.Multi layer Perceptron: It possess enhanced processing capabilities
as they consist of two or more layers, adept at handling more
complex patterns and relationships within the data.
BASIC COMPONENTS OF PERCEPTRON 4

A Perceptron is composed of key components that work together to process

information and make predictions.
• Input Features: The perceptron takes multiple input features, each
representing a characteristic of the input data.
• Weights: Each input feature is assigned a weight that determines its
influence on the output. These weights are adjusted during training to find
the optimal values.
• Summation Function: The perceptron calculates the weighted sum of its
inputs, combining them with their respective weights.
• Activation Functions: The weighted sum is passed through
the Heaviside step function, comparing it to a threshold to produce a
binary output (0 or 1).
• Output: The final output is determined by the activation function, often
used for binary classification tasks.
• Bias: The bias term helps the perceptron make adjustments independent
of the input, improving its flexibility in learning.
• Learning Algorithm: The perceptron adjusts its weights and bias using a
learning algorithm, such as the Perceptron Learning Rule, to minimize
Single-layer perceptron 5
 6
HOW DOES PERCEPTRON WORK?
• A weight is assigned to each input node of a perceptron, indicating
the importance of that input in determining the output. The
Perceptron’s output is calculated as a weighted sum of the inputs,
which is then passed through an activation function to decide
whether the Perceptron will fire.
• The output of perceptron is computed as:

Here, the is the step activation function.

• The step function compares this weighted sum to a threshold. If
the input is larger than the threshold value, the output is 1;
otherwise, it's 0. This is the most common activation function used
in Perceptron's is represented by the Heaviside step function:
 7
The weight updating
• If y≠t then we have to update the weights
w(new)=w(old)+αtx
• Else:
w(new)=w(old)
8
AND Gate- Perceptron Training Rule 9
10
11
12
13
 14

BACKPROPAGATION IN NEURAL NETWORKS

• Back Propagation, also known as "Backward Propagation of Errors,"
is a method used to train a neural network. Its goal is to reduce the
difference between the model’s predicted output and the actual
output by adjusting the weights and biases in the network.
• It works iteratively to adjust weights and bias to minimize the cost
function. In each epoch, the model adapts these parameters by
reducing loss by following the error gradient. It often uses
optimization algorithms like gradient descent or stochastic
gradient descent. The algorithm computes the gradient using the
chain rule from calculus, allowing it to effectively navigate complex
layers in the neural network to minimize the cost function.
 15

BACKPROPAGATION IN NEURAL NETWORKS

 16

BACKPROPAGATION IN NEURAL NETWORKS

• Back Propagation plays a critical role in how neural networks
improve over time. Here's why:
• Efficient Weight Update: It computes the gradient of the loss
function with respect to each weight using the chain rule making it
possible to update weights efficiently.
• Scalability: The Back Propagation algorithm scales well to networks
with multiple layers and complex architectures making deep learning
feasible.
• Automated Learning: With Back Propagation the learning process
becomes automated and the model can adjust itself to optimize its
performance.
WORKING OF BACK PROPAGATION ALGORITHM 17
The Back Propagation algorithm involves two main steps: the Forward Pass and
the Backward Pass.
1. Forward Pass Work
In forward pass the input data is fed into the input layer. These inputs combined
with their respective weights are passed to hidden layers. For example in a network
with two hidden layers (h1 and h2) the output from h1 serves as the input to h2.
Before applying an activation function, a bias is added to the weighted inputs.
Each hidden layer computes the weighted sum (`a`) of the inputs then applies an
activation function like ReLU to obtain the output (`o`). The output is passed to the
next layer where an activation function such as softmax converts the weighted
outputs into probabilities for classification.
 18
2. Backward Pass
In the backward pass the error (the difference between the predicted
and actual output) is propagated back through the network to adjust
the weights and biases. One common method for error calculation is
the Mean Square Error (MSE) given by:

Once the error is calculated the network adjusts weights

using gradients which are computed with the chain rule. These
gradients indicate how much each weight and bias should be adjusted
to minimize the error in the next iteration. The backward pass
continues layer by layer ensuring that the network learns and improves
its performance. The activation function through its derivative plays a
crucial role in computing these gradients during Back Propagation.
 19
Example of Back Propagation
 20
• Typically, the input layer (green) are the input variables from a
dataset, and the output layer (red) is the neural
network’s prediction value. Within the hidden and output layers,
a weighted sum (denoted by s) is taken for the each node, followed
by the application of an activation function (denoted by a) which
normalises the value according to a desired activation function.
• The process of feeding data through a network from input to output
is called forward propagation. The process of observing the error
rate of the forward propagation and feeding the error backward to
fine-tune the weights of the neural network is called back
propagation. We forward propagate before we back propagate.
 21
1. Forward Propagation:
Using the sigmoid function as the activation function for this example (activation
functions are used as mappings for inputs to be within certain ranges in the case of
sigmoid, the range is (0, 1)).
22
23
24
 25
Mean Squared Error (MSE) Calculation:
The mean squared error is a measure of the difference between the
expected and actual outputs. We are looking for a low MSE score,
which indicates a better fit of the model to the data. We will be
using gradient descent as a way to decrease this value.
 26
2. Back Propagation
• Now that the predicted value is calculated, the neural network needs to adjust its
weights based on the prediction error. This is done through back propagation.
• For this example, consider a learning rate of 0.1 i.e., α = 0.1
• The general mathematical idea behind back-propagation is to apply the chain rule
to find the change in the error function over the change in a weight. Consider
weight 7 for this example:

• All three partial equations can be derived from our work

27
 28

This formula can be done for all weights connecting the hidden layer to the
output layer.
Now we have the gradient of the error function. 29

We want to apply gradient descent to get a new value of weight w₇. The new w₇
(we can symbolise this w₇’) can be obtained by subtracting the learning rate
multiplied by the gradient from w₇

So in general, for an output neuron:

 30
Output Layer
Now, applying real numbers from the example to find new values of w₇
through w₁₂
31
THANK
YOU
Brita Tamm
502-555-0152
[email protected]
www.firstupconsultants.com