Subject: Neural Networks and Deep Learning

(UNIT-1 Class Notes)

Prepared
by
Dr K Madan Mohan
Asst. Professor
Department of CSE (AI&ML)
Sreyas Institute of Engineering and Technology,
Nagole, Bandlaguda, Hyderabad

Sreyas Institute of Engineering and Technology – Autonomous

9-39, Sy No 107 Tattiannaram, GSI Rd, beside INDU ARANYA HARITHA,
Bandlaguda, Nagole, Hyderabad, Telangana 500068
8/20/23, 10:23 PM Introduction - Neural Networks and Deep Learning Tutorial | Study Glance

 NN&DL Menu

Introduction

Neural Networks
The Artificial Neural Networks (ANN) began with Warren McCulloch and Walter Pitts (1943) who created a computational model
for neural networks based on algorithms called threshold logic

Computational model that mimes the functional of human brain to perform various tasks faster than traditional system. ANN is an
efficient information processing system which resembles the characteristics of biological neural network

Warren McCulloch & Walter Pitts

https://studyglance.in/nn/display.php?tno=1&topic=Introduction 1/4
8/20/23, 10:23 PM Introduction - Neural Networks and Deep Learning Tutorial | Study Glance

Poor cell growth?

Ad Merck Life Science

Biological Neurons
Dendrites : Responsible for receiving information from other neurons and bring it to the cell body(soma)
Soma : Responsible for processing of information, they have received from dendrites.
Axon : It is just like a cable through which send information from the cell body to other neurons.
Synapses : It is the connection between the axon and other neuron dendrites.

Ads by
Stop seeing this ad

Why this ad?

https://studyglance.in/nn/display.php?tno=1&topic=Introduction 2/4
8/20/23, 10:23 PM Introduction - Neural Networks and Deep Learning Tutorial | Study Glance
The input signals is received by dendrites, and processing generally to the cell body(soma). Incoming signals can be either
excitatory which means they tend to make the neuron fire (generate an electrical impulse).

Most neurons receive many input signals throughout their dendritic trees. A single neuron may have more than one set of dendrites
and may receive many thousands of input signals. To decide whether a neuron is excited to fire an impulse depends on the sum of
all of the excitatory and inhibitory signals it receives. The processing of this information takes place in soma which is neuron cell
body. If the neuron does end up firing, the nerve impulse, or action potential, is conducted down the axon.

Towards its end, the axon splits up into many branches known as axon terminals (or nerve terminals), which makes connections on
target cells.

The junctions that allow signal transmission between the axons terminals and dendrites are called synapses. The process of
transmission is by diffusion of chemicals called neuro transmitters across the synaptic cleft

Artificial Neural Networks Model

Ads by
Stop seeing this ad

Why this ad?

The artificial neuron model has N input, denoted as x1, x2, ...xn. Each line connecting these inputs to the neuron is assigned a
weight, which are denoted as w1, w2, .., wn respectively. Weights in the artificial model correspond to the synaptic connections in
biological neurons. The threshold in artificial neuron is usually represented by Θ and the activation corresponding to the graded
potential is given by the formula:

yin = x 1 . w1 + x 2 . w2 + x 3 . w3 … x n . wn

i.e., Net input

yin = ∑ x i . wi + b

The output can be calculated by applying the activation function over the net input.

Y = F (yin )

Next Topic : Basic models

ABOUT

https://studyglance.in/nn/display.php?tno=1&topic=Introduction 3/4
8/20/23, 10:23 PM Introduction - Neural Networks and Deep Learning Tutorial | Study Glance

Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=1&topic=Introduction 4/4
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

3D Interior Design Made Easy

Design your 3D dream home like a Pro. Interior Design & 4K Rend

Coohom 3D Design Tool

 NN&DL Menu

Basic models

The models of ANN are specified by the three basic entities namely:
1. The model's synaptic interconnection.
2. The training rules or learning rules adopted for updating and adjusting the connection weights.
3. Activation functions.

Network Architectures
The manner in which the neurons of a neural network are structured is intimately linked with the learning algorithm used to train
the network.

There are three fundamentally different classes of network architectures:

1. Single-Layer Feedforward Networks

2. Multilayer Feedforward Networks
3. Recurrent Networks

Single-Layer Feedforward Networks

Ads by
Stop seeing this ad Why this ad?

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 1/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

In this type of network, we have input layer and output layer but the input layer does not count because no computation is
performed in this layer.

Output Layer is formed when different weights are applied on input nodes and the cumulative effect per node is taken.

After this, the neurons collectively give the output layer to compute the output signals.

Multilayer Feedforward Networks

This network has one or more hidden layers, the term "hidden" refers to the fact that this part of the neural network is not seen
directly from either the input or output of the network.The function of hidden neurons is to intervene between the external input
and the network output in some useful manner.

The existence of one or more hidden layers enables the network to be computationally stronger.

Recurrent Networks
Arecurrent neural network distinguishes itself from a feedforward neural network in that it has at least one feedback loop

Single Layer Recurrent Network

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 2/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

Vertex Bougainvillea
Ad Vertex bougainvillea Open

This network is a single-layer network with a feedback connection in which the processing element's output can be directed back to
itself or to other processing elements or both.

A recurrent neural network is a class of artificial neural network where the connection between nodes forms a directed graph along
a sequence.

This allows is it to exhibit dynamic temporal behavior for a time sequence. Unlike feedforward neural networks, RNNs can use their
internal state (memory) to process sequences of inputs.

Multilayer Recurrent Network

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 3/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

In this type of network, processing element output can be directed to the processing element in the same layer and in the preceding
layer forming a multilayer recurrent network.

They perform the same task for every element of the sequence, with the output being dependent on the previous computations.
Inputs are not needed at each time step.

The main feature of a multilayer recurrent network is its hidden state, which captures information about a sequence.

Learning rules
Just as there are different ways in which we ourselves learn from our own surrounding environments, neural networks learning
processes as follows: learning with a teacher (supervised learning) and learning without a teacher (unsupervised learning and
reinforcement learning).

Supervised Learning

Learning with a teacher is also referred to as supervised learning. In conceptual terms, we may think of the teacher as having
knowledge of the environment and unknown to the neural network. the teacher is able to provide the neural network with a desired
response for that training vector. Indeed, the desired response represents the "optimum" action to be performed by the neural
network. The network parameters are adjusted under the combined influence of the training vector and the error signal. The error
signal is defined as the difference between the desired response and the actual response of the network. This adjustment is carried
out iteratively in a step-by-step fashion with the aim of eventually making the neural network emulate the teacher; the emulation is
presumed to be optimum in some statistical sense. In this way, knowledge of the environment available to the teacher is transferred

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 4/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance
to the neural network through training and stored in the form of "fixed" synaptic weights, representing long-term memory.When this
condition is reached, we may then dispense with the teacher and let the neural network deal with the environment completely by
itself.

Unsupervised Learning

In unsupervised, or self-organized, learning is done without the supervision of a teacher. The goal of unsupervised learning is to find
the underlying structure of dataset, group that data according to similarities, and represent that dataset in a compressed format.

To perform unsupervised learning, we may use a competitive-learning rule. For example, we may use a neural network that consists
of two layers, an input layer and a competitive layer. The input layer receives the available data. The competitive layer consists of
neurons that compete with each other (in accordance with a learning rule) for the "opportunity" to respond to features contained in
the input data. In its simplest form, the network operates in accordance with a "winner-takes-all" strategy. In such a strategy, the
neuron with the greatest total input "wins" the competition and turns on; all the other neurons in the network then switch off.

Reinforcement Learning

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 5/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

Reinforcement Learning is a feedback-based Network technique in which an agent learns to behave in an environment by
performing the actions and seeing the results of actions. For each good action, the agent gets positive feedback, and for each bad
action, the agent gets negative feedback or penalty.

Since there is no labeled data, so the agent is bound to learn by its experience only.

The agent interacts with the environment and explores it by itself. The primary goal of an agent in reinforcement learning is to
improve the performance by getting the maximum positive rewards.

The goal of reinforcement learning is to minimize a cost-to-go function, defined as the expectation of the cumulative cost of actions
taken over a sequence of steps instead of simply the immediate cost.

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 6/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

Activation functions
The activation function is applied over the net input to calculate the output of an ANN. An integration function (say f) is associated
with the input of a processing element. This function serves to combine activation, information or evidence from an external source
or other processing elements into a net input to the processing element.

When a signal is fed through a multilayer network with linear activation functions, the output obtained remains same as that could
be obtained using a single-layer network. Due to this reason, nonlinear functions are widely used in multilayer networks compared
to linear functions.
There are many activation functions available. In this part, we'll look at a few:

Identity function:
It is a linear function and can be defined as f (x) = x for all x
The output here remains the same as input. The input layer uses the identity activation function.

Binary step function:

This function can be defined as

1 if x ≥ Θ
f (x) = {
0 if x < Θ

where q represents the threshold value. This function is most widely used in single-layer nets to convert the net input to an output
that is a binary (1 or 0).

Bipolar step function:

This function can be defined as

1 if x ≥ Θ
f (x) = {
−1 if x < Θ

where q represents the threshold value. This function is also used in single-layer nets to convert the net input to an output that is
bipolar (+1 or –1).

Sigmoidal functions:
The sigmoidal functions are widely used in back-propagation nets because of the relationship between the value of the functions at
a point and the value of the derivative at that point which reduces the computational burden during training.
Sigmoidal functions are of two types:

1. Binary sigmoid function:

It is also termed as logistic sigmoid function or unipolar sigmoid function. The range of the sigmoid function is from 0 to 1. It can be
defined as

1
f (x) =
-λx
1 + e

where λ is the steepness parameter. The derivative of this function is

f´(x) = λf (x) [1 - f (x)]

2. Bipolar sigmoid function:

The range of the Bipolar sigmoid function is from -1 to 1. It can be defined as

-λx
2 1 − e
f (x) = −1 =
-λx -λx
1 + e 1 + e

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 7/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

where λ is the steepness parameter. The derivative of this function is

λ
f´(x) = [1 + f (x)] [1 - f (x)]
2

The bipolar sigmoidal function is closely related to hyperbolic tangent function, which is written as

x -x -2x
e − e 1 − e
f (x) = =
x -x -2x
e + e 1 + e

The derivative of the hyperbolic tangent function is

h´(x) = [1 + h(x)] [1 - h(x)]

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 8/9
8/20/23, 10:24 PM Basic models - Neural Networks and Deep Learning Tutorial | Study Glance

Next Topic : Terminologies

ABOUT
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=2&topic=Basic-models 9/9
8/20/23, 10:25 PM Terminologies - Neural Networks and Deep Learning Tutorial | Study Glance

AI program for your startup

Increase mileage on your cloud investment with Google Cloud

Google Cloud Startups Ap

 NN&DL Menu

Terminologies

Weights
In ANN architecture, every neuron is connected to other neurons by means of a directed communication link and every link is
associated with weights. Weight is a parameter which contains information about the input signal. This information is used by the
net to solve a problem.

Wij is the weight from processing element ´i´ source node to processing element ´j´ destination node.

Bias (b)
The bias is a constant value included in the network. Its impact is seen in calculating the net input. The bias is included by adding a
component x0 =1 to the input vector X.

The bias can also be explained as follows: Consider an equation of straight line, y = mx + c where x is the input, m is the weight, c
is the bias and y is the output. Thus, bias plays a major role in determining the output of the network.

Bias can be positive or negative. The positive bias helps in increasing the net input of the network. The negative bias helps in
decreasing the net input of the network.

Threshold (Θ)
Threshold is a set value used in the activation function. In ANN, based on the threshold value the activation functions are defined
and the output is calculated.

Learning Rate (α)

The learning rate is used to control the amount of weight adjustment at each step of training. The learning rate ranges from 0 to 1.
It determines the rate of learning at each time step

Next Topic : Perceptron Networks

ABOUT
Ads
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important by
& previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects. Stop seeing this ad Why this ad?

https://studyglance.in/nn/display.php?tno=3&topic=Terminologies 1/2
8/20/23, 10:25 PM Terminologies - Neural Networks and Deep Learning Tutorial | Study Glance

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=3&topic=Terminologies 2/2
8/20/23, 10:21 PM Perceptron Networks - Neural Networks and Deep Learning Tutorial | Study Glance

Buy 3 Shirts Combo | 899/-

Find a Large Selection of Exclusive Designs an
Prices!

JackDumbell
 NN&DL Menu

Perceptron Networks

The perceptron was first introduced by Mr. Frank Rosenblatt in 1957.

A Perceptron is an algorithm for supervised learning of binary classifiers.

There are two types of Perceptrons: Single layer and Multilayer.

Single layer - Single layer perceptrons can learn only linearly separable patterns
Multilayer - Multilayer perceptrons or feedforward neural networks with two or more layers have the greater processing power
The Perceptron algorithm learns the weights for the input signals in order to draw a linear decision boundary.

This enables you to distinguish between the two linearly separable classes +1 and -1.

Perceptron algorithm (Single layer)

Step 0: initialize the weights and the bias(for easy calculation they can be set to zero). also initialize the learning rate α(0, α, 1) for
simpicity α is set to 1.

Step 1: Perform step 2 to 6 until the final stopping condition is false.

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Step 2: Perform step 3 to 5 each training pari indicated by s:t

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Step 3: The input layer containing input units is applied with identity activation function: xi = si

Step 4: calculate the ourput of the network. to do so first obtain the net input:

yin = ∑ x i . wi + b

https://studyglance.in/nn/display.php?tno=4&topic=Perceptron-Networks 1/5
8/20/23, 10:21 PM Perceptron Networks - Neural Networks and Deep Learning Tutorial | Study Glance

where n is the number of inputs neurons in the input layer. Then apply activation over the net input calculated to obrain the output

⎧1 if yin > θ

f (yin ) = ⎨ 0 if − θ ⩽ yin ⩽ θ
⎩
−1 if yin < −θ

Step 5: weight and bias adjustment: compare the value of the actual (calculated) output and desire(targer) output

if y ≠ t, then
wi(new) = wi(old)+ αtxi
b(new) = b(old)+ αt
else we have
wi(new) = wi(old)
b(new) = b(old)

Step 6: Train the network until there is no weight change. This is the stopping condition for the network. If this condition is not
met, then start again form step 2.

Perceptron algorithm (Multiple layer)

Step 0: Initialize the weights, biases and learning rate suitably.

Step 1: Check for stopping condition; if it is false, perform Steps 2–6.

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Step 2: Perform Steps 3–5 for each bipolar or binary training vector pair s : t.

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Step 3: Set activation (identity) of each input unit i = 1 to n: xi = si

Step 4: Calculate output response of each output unit j = 1 to m: First, the net input is calculated as
n

yinj = ∑ xi . w ij + bj

Then activations are applied over the net input to calculate the output response:

⎧1 if yin > θ

f (yin ) = ⎨ 0 if − θ ⩽ yin ⩽ θ
⎩
−1 if yin < −θ

https://studyglance.in/nn/display.php?tno=4&topic=Perceptron-Networks 2/5
8/20/23, 10:21 PM Perceptron Networks - Neural Networks and Deep Learning Tutorial | Study Glance

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Step 5: Make adjustment in weights and bias for j = l to m and i = l to n.

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

if yj ≠ tj, then
wij(new) = wij(old)+ αtjxi
bj(new) = bj(old)+ αtj
else we have
wij(new) = wij(old)
bj(new) = bj(old)

Step 6: Test for the stopping condition, i.e., if there is no change in weights then stop the training process, else start again from
Step 2.

Example of Single layer Perceptron

We need to understand that the output of an AND gate is 1 only if both inputs (in this case, x1 and x2) are 1.

Truth table for AND function with bipolar inputs and targets.

x1 x2 Target
1 1 1
1 -1 -1
-1 1 -1
-1 -1 -1

Row-1
initializing w1, w2, and b as 0, α=1, and θ=0.

we get;

x1(0)+x2(0)+0

Passing the first row of the AND logic table (x1=1, x2=1), we get;

1*0+1*0+0 = 0

yin = 0

Y = f(yin) => f(0) => 0

check Y is equal to t or not, that is 0 ≠ 1, Hence weights change is required.

wi(new) = wi(old)+ αtxi

b(new) = b(old)+ αt

w1(new) = w1(old)+ αtx1

= 0+ 1*1*1 = 1

w2(new) = w2(old)+ αtx2

https://studyglance.in/nn/display.php?tno=4&topic=Perceptron-Networks 3/5
8/20/23, 10:21 PM Perceptron Networks - Neural Networks and Deep Learning Tutorial | Study Glance
= 0+ 1*1*1 = 1

b(new) = 0+ 1*1 = 1

new updated weightes are w1=w2=b=1

yin = 1*1 + 1*1 + 1 = 3

Y = f(yin) => f(3) => 1

check Y is equal to t or not, that is 1 = 1, Hence weights change is Not required.

Row-2
w1, w2, and b as 1, α=1, and θ=0.

we get;

x1(1)+x2(1)+1

Passing the second row of the AND logic table (x1=1, x2=-1), we get;

1*1+-1*1+1 = 1

yin = 1

Y = f(yin) => f(1) => 1

check Y is equal to t or not, that is 1 ≠ -1, Hence weights change is required.

wi(new) = wi(old)+ αtxi

b(new) = b(old)+ αt

w1(new) = w1(old)+ αtx1

= 1+ 1*-1*1 = 0

w2(new) = w2(old)+ αtx2

= 1+ 1*-1*-1 = 2

b(new) = 1+ 1*-1 = 0

new updated weightes are w1=0, w2= 2, b=0

yin = 1*0 + -1*2 + 0 = -2

Y = f(yin) => f(-2) => -1

check Y is equal to t or not, that is -1 = -1, Hence weights change is Not required.

Row-3
w1=0, w2= 2, b=0, α=1, and θ=0.

we get;

x1(0)+x2(2)+0

Passing the third row of the AND logic table (x1=-1, x2=1), we get;

-1*0+1*2+0 = 2

yin = 2

Y = f(yin) => f(2) => 1

check Y is equal to t or not, that is 1 ≠ -1, Hence weights change is required.

wi(new) = wi(old)+ αtxi

b(new) = b(old)+ αt

w1(new) = w1(old)+ αtx1

= 0+ 1*-1*-1 = 1

w2(new) = w2(old)+ αtx2

= 2+ 1*-1*1 = 1

https://studyglance.in/nn/display.php?tno=4&topic=Perceptron-Networks 4/5
8/20/23, 10:21 PM Perceptron Networks - Neural Networks and Deep Learning Tutorial | Study Glance
b(new) = 0+ 1*-1 = -1

new updated weightes are w1=1, w2= 1, b=-1

yin = -1*1 + 1*1 + -1 = -1

Y = f(yin) => f(-1) => -1

check Y is equal to t or not, that is -1 = -1, Hence weights change is Not required.

Row-4
w1=1, w2= 1, b=-1, α=1, and θ=0.

we get;

x1(1)+x2(1)+-1

Passing the fourth row of the AND logic table (x1=-1, x2=-1), we get;

-1*1+-1*1+(-1) = -3

yin = -3

Y = f(yin) => f(-3) => -1

check Y is equal to t or not, that is -1 = -1, Hence weights change is not required.

Next Topic : Adaptive Linear Neuron

ABOUT
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=4&topic=Perceptron-Networks 5/5
8/20/23, 10:27 PM Adaptive Linear Neuron - Neural Networks and Deep Learning Tutorial | Study Glance

 NN&DL Menu

Adaptive Linear Neuron

Adaptive Linear Neuron (Adaline)

Adaline which stands for Adaptive Linear Neuron, is a network having a single linear unit. It was developed by Widrow and Hoff in
1960. Some important points about Adaline are as follows −

It uses bipolar activation function.

Adaline neuron can be trained using Delta rule or Least Mean Square(LMS) rule or widrow-hoff rule
The net input is compared with the target value to compute the error signal.
on the basis of adaptive training algoritham weights are adjusted
The basic structure of Adaline is similar to perceptron having an extra feedback loop with the help of which the actual output is
compared with the desired/target output. After comparison on the basis of training algorithm, the weights and bias will be updated.

Ads by
Send feedback

Why this ad?

Adaptive Linear Neuron Learning algorithm

Step 0: initialize the weights and the bias are set to some random values but not to zero, also initialize the learning rate α.

Step 1 − perform steps 2-7 when stopping condition is false.

Find Out More

Comprehensive solutions for thriving cell cultures

https://studyglance.in/nn/display.php?tno=5&topic=Adaptive-Linear-Neuron 1/4
8/20/23, 10:27 PM Adaptive Linear Neuron - Neural Networks and Deep Learning Tutorial | Study Glance

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Step 2 − perform steps 3-5 for each bipolar training pair s:t.

Step 3 − Activate each input unit as follows −

xi = s i (i = 1 to n)

Step 4 − Obtain the net input with the following relation −

yin = ∑ xi . wi + b

Here ‘b’ is bias and ‘n’ is the total number of input neurons.

Step 5 Until least mean square is obtained (t - yin), Adjust the weight and bias as follows −

wi(new) = wi(old) + α(t - yin)xi

b(new) = b(old) + α(t - yin)

Now calculate the error using => E = (t - yin)2

Step 7 − Test for the stopping condition, if error generated is less then or equal to specified tolerance then stop.

Ads by
Stop seeing this ad Why this ad?

Multiple Adaptive Linear Neuron (Madaline)

Madaline which stands for Multiple Adaptive Linear Neuron, is a network which consists of many Adalines in parallel. It will have a
single output unit. Some important points about Madaline are as follows −

It is just like a multilayer perceptron, where Adaline will act as a hidden unit between the input and the Madaline layer.
The weights and the bias between the input and Adaline layers, as in we see in the Adaline architecture, are adjustable.
The Adaline and Madaline layers have fixed weights and bias of 1.
Training can be done with the help of Delta rule.
It consists of “n” units of input layer and “m” units of Adaline layer and “1” unit of the Madaline layer. Each neuron in the Adaline
and Madaline layers has a bias of excitation “1”. The Adaline layer is present between the input layer and the Madaline layer; the
Adaline layer is considered as the hidden layer.

https://studyglance.in/nn/display.php?tno=5&topic=Adaptive-Linear-Neuron 2/4
8/20/23, 10:27 PM Adaptive Linear Neuron - Neural Networks and Deep Learning Tutorial | Study Glance

Multiple Adaptive Linear Neuron (Madaline) Training Algorithm

By now we know that only the weights and bias between the input and the Adaline layer are to be adjusted, and the weights and
bias between the Adaline and the Madaline layer are fixed.

Step 0 − initialize the weights and the bias(for easy calculation they can be set to zero). also initialize the learning rate α(0, α, 1)
for simpicity α is set to 1.

Step 1 − perform steps 2-6 when stopping condition is false.

Step 2 − perform steps 3-5 for each bipolar training pair s:t

Step 3 − Activate each input unit as follows −

xi = s i (i = 1 to n)

Step 4 − Obtain the net input at each hidden layer, i.e. the Adaline layer with the following relation −
n

Q inj = bj + ∑ x i wij (j = 1 to m)

Here ‘b’ is bias and ‘n’ is the total number of input neurons.

Step 5 − Apply the following activation function to obtain the final output at the Adaline and the Madaline layer −

1 if x ⩾ 0
f (x) = {
−1 if x < 0

Output at the hidden (Adaline) unit

Qj = f (Q inj )

https://studyglance.in/nn/display.php?tno=5&topic=Adaptive-Linear-Neuron 3/4
8/20/23, 10:27 PM Adaptive Linear Neuron - Neural Networks and Deep Learning Tutorial | Study Glance
Final output of the network

i.e.
m
yinj = b0 + ∑ Qj vj
j=1

y = f (yin )

Step 6 − Calculate the error and adjust the weights as follows −

If t ≠ y and t = +1, update weights on Zj, where net input is closest to 0 (zero)
wij(new) = wij(old) + α(1 - Qinj)xi
bj(new) = bj(old) + α(1 - Qinj)

else If t ≠ y and t = -1, update weights on Zk, whose net input is positive
wik(new) = wik(old) + α(-1 - Qink)xi
bk(new) = bk(old) + α(-1 - Qink)

else if y = t then
no weight updation is required.

Step 7 − Test for the stopping condition, which will happen when there is no change in weight or the highest weight change
occurred during training is smaller than the specified tolerance.

Next Topic : Back Propagation

ABOUT
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=5&topic=Adaptive-Linear-Neuron 4/4
8/20/23, 10:28 PM Back Propagation - Neural Networks and Deep Learning Tutorial | Study Glance

 NN&DL Menu

Back Propagation

Back Propagation Neural Networks

Back Propagation Neural (BPN) was first introduced in 1960s, It is a multilayer neural network consisting of the input layer, at least
one hidden layer and output layer. It is a multi-layer feed forward neural network using Gradient descent approach which exploits
the chain rule to optimize the parameter. The error which is calculated at the output layer, by comparing the target output and the
actual output, will be propagated back towards the input layer.

The main features of Backpropagation are the iterative, recursive and efficient method through which it calculates the updated
weight to improve the network until it is not able to perform the task for which it is being trained.

Back Propagation Neural Networks Training Algorithm

Back Propagation Neural will use binary sigmoid activation function. The training will have the following three phases.

Phase 1 − Feed Forward Phase

Phase 2 − Back Propagation of error
Phase 3 − Updating of weights
Step 0 − initialize the weights and the bias(For easy calculation and simplicity, take some small random values but not zero). also
initialize the learning rate α(0, α, 1).

Step 1 − Continue step 2-10 when the stopping condition is not true.

Ads by
Stop seeing this ad Why this ad?

https://studyglance.in/nn/display.php?tno=6&topic=Back-Propagation 1/4
8/20/23, 10:28 PM Back Propagation - Neural Networks and Deep Learning Tutorial | Study Glance

Ads by
Stop seeing this ad Why this ad?
AI program for your startup
Increase mileage on your cloud investment with Google
Cloud

Google Cloud Startups Apply Now

Step 2 − Continue step 3-9 for every training pair.

Phase 1
Step 3 − Each input unit receives input signal xi and sends it to the hidden unit for all i = 1 to n

Step 4Calculate the net input at the hidden unit using the following relation −
n

Qinj = ∑ xi vij + bj j = 1 to p

i=1

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Now calculate the net output by applying the following sigmoidal activation function

Qj = f (Qinj )

Step 5 Calculate the net input at the output layer unit using the following relation
p

yink = ∑ Qj w jk + bk k = 1 to m

j=1

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Calculate the net output by applying the following sigmoidal activation function

yk = f (yink )

Phase 2
Step 6 − Compute the error correcting term, in correspondence with the target pattern received at each output unit, as follows
′

δk = (tk − yk )f (yink )

The derivative f (yink ) can be calculated as:

if Binary sigmoid function:

https://studyglance.in/nn/display.php?tno=6&topic=Back-Propagation 2/4
8/20/23, 10:28 PM Back Propagation - Neural Networks and Deep Learning Tutorial | Study Glance
′

f (yink ) = f (yink )(1 − f (yink ))

if Bipolar sigmoid function:

′

f (yink ) = (1 + f (yink ))(1 − f (yink ))

Then, send δk back to the hidden layer

Step 7 − Now each hidden unit will be the sum of its delta inputs from the output units.
m

δinj = ∑ δk w jk

k=1

Error term can be calculated as follows −

′

δj = δinj f (Qinj )

Phase 3
Step 8 − Each output unit yk (k = 1 to m) updates the weight and bias as follows

w jk (new) = w jk (old) + Δw jk

bk (new) = bk (old) + Δbk

Δw jk = αδk Qj

Δb0k = αδk

Step 9 − Each Hidden unit qj (j = 1 to p) updates the weight and bias as follows

vij (new) = vij (old) + Δvij

bj (new) = bj (old) + Δbj

Δvij = αδj xi

Δbj = αδj

Step 10 − Check for the stopping condition, which may be either the number of epochs reached or the target output matches the
actual output.

Next Topic : Associate Memory Network

ABOUT
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=6&topic=Back-Propagation 3/4
8/20/23, 10:28 PM Back Propagation - Neural Networks and Deep Learning Tutorial | Study Glance

https://studyglance.in/nn/display.php?tno=6&topic=Back-Propagation 4/4
8/20/23, 10:30 PM Associate Memory Network - Neural Networks and Deep Learning Tutorial | Study Glance

 NN&DL Menu

Associate Memory Network

Associative Memory Networks

An associative memory network can store a set of patterns as memories. When the associative memory is being presented with a
key pattern, it responds by producing one of the stored patterns, which closely resembles or relates to the key panern. Thus, the
recall is through association of the key pattern, with the help of informtion memorized. These types of memories are also called as
content-addressable memories (CAM). The CAM can also be viewed as associating data to address, i.e.; for every data in the
memory there is a corresponding unique address. Also, it can be viewed as data correlator. Here inpur data is correlated with that of
the stored data in the CAM. It should be nored that the stored patterns must be unique, i.e., different patterns in each location. If
the same pattern exists in more than one locacion in the CAM, then, even though the correlation is correct, the address is noted to
be ambiguous. Associative memory makes a parallel search within a stored data file. The concept behind this search is to Output
any one or all stored items Which match the given search argumemt.

Ads by
Stop seeing this ad Why this ad?
3D Interior Design Made Easy
Design your 3D dream home like a Pro. Interior Design & 4K
Render

Coohom 3D Design Tool Open

Training Algorithms for Pattern Association

There are two algorithms developed for training of pattern association nets.

1. Hebb Rule
2. Outer Products Rule

1. Hebb Rule
The Hebb rule is widely used for finding the weights of an associative memory neural network. The training vector pairs here are
denoted as s:t. The weights are updated unril there is no weight change.

Hebb Rule Algorithmic

Step 0: Set all the initial weights to zero, i.e.,
Wij = 0 (i = 1 to n, j = 1 to m)

Step 1: For each training target input output vector pairs s:t, perform Steps 2-4.

Ads by

Ideate, innovate, & integrate

Stop seeing this ad Why this ad?

Enable agile development from product launch to feature

pivots

Google Cloud Startups Ads by Apply Now

Find Out More
Stop seeing this ad Why this ad?
Comprehensive solutions for thriving cell cultures

https://studyglance.in/nn/display.php?tno=7&topic=Associate-Memory-Network 1/6
8/20/23, 10:30 PM Associate Memory Network - Neural Networks and Deep Learning Tutorial | Study Glance
Step 2: Activate the input layer units to current training input, Xi=Si (for i = 1 to n)

Ads by
Stop seeing this ad Why this ad?
AI program for your startup
Increase mileage on your cloud investment with Google
Cloud

Google Cloud Startups Apply Now

Step 3: Activate the output layer units to current target output,
yj = tj (for j = 1 to m)

Step 4: Start the weight adjustment

w ij (new) = w ij (old) + xi yj (i = 1 to n j = 1 to m)

2. Outer Products Rule

Outer products rule is a method for finding weights of an associative net.

Input=> s = (s1, ... ,si, ... ,sn)

Output=> t= (t1, ... ,tj, ... ,tm)

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

The outer product of the two vectors is the product of the matrices S = sT and T = t, i.e., between [n X 1] marrix and [1 x m]
matrix. The transpose is to be taken for the input matrix given.

ST = sTt =>

s1
⎡ ⎤

⎢ . ⎥
⎢ ⎥
⎢ . ⎥
⎢ ⎥
⎢ ⎥
⎢ si ⎥ * [ t1 . . tj . . tm ]
⎢ ⎥
⎢ ⎥
⎢ . ⎥
⎢ ⎥
⎢ . ⎥

⎣ ⎦
sn

This weight matrix is same as the weight matrix obtained by Hebb rule to store the pattern association s:t. For storing a set of
associations, s(p):t(p), p = 1 to P, wherein,
s(p) = (s1 (p}, ... , si(p), ... , sn(p))
t(p) = (t1 (p), · · ·' tj(p), · · · 'tm(p))

the weight matrix W = {wij} can be given as

n

T
w = ∑S (p) . S(p)

p=1

https://studyglance.in/nn/display.php?tno=7&topic=Associate-Memory-Network 2/6
8/20/23, 10:30 PM Associate Memory Network - Neural Networks and Deep Learning Tutorial | Study Glance

Ads by
Stop seeing this ad Why this ad?

There two types of associative memories

Auto Associative Memory

Hetero Associative memory

Auto Associative Memory

An auto-associative memory recovers a previously stored pattern that most closely relates to the current pattern. It is also known
as an auto-associative correlator.

In the auto associative memory network, the training input vector and training output vector are the same

Vertex Bougainvillea
Ad Vertex bougainvillea Open

https://studyglance.in/nn/display.php?tno=7&topic=Associate-Memory-Network 3/6
8/20/23, 10:30 PM Associate Memory Network - Neural Networks and Deep Learning Tutorial | Study Glance

Auto Associative Memory Algorithm

Training Algorithm
For training, this network is using the Hebb or Delta learning rule.

Step 1 − Initialize all the weights to zero as wij = 0 i=1ton, j=1ton

Step 2 − Perform steps 3-4 for each input vector.

Step 3 − Activate each input unit as follows −

xi = si (i = 1 to n)

Step 4 − Activate each output unit as follows −

yj = sj (j = 1 to n)

Step 5 − Adjust the weights as follows −

w ij (new) = w ij (old) + xi yj

The weight can also be determine form the Hebb Rule or Outer Products Rule learning

T
w = ∑S (p) . S(p)

p=1

Testing Algorithm
Step 1 − Set the weights obtained during training for Hebb’s rule.

Step 2 − Perform steps 3-5 for each input vector.

Step 3 − Set the activation of the input units equal to that of the input vector.

Step 4 − Calculate the net input to each output unit j = 1 to n;

n

yinj = ∑ xi w ij

i=1

Step 5 − Apply the following activation function to calculate the output

+1 if yinj > 0
yj = f (yinj ) = {
−1 if yinj ⩽ 0

Hetero Associative memory

In a hetero-associate memory, the training input and the target output vectors are different. The weights are determined in a way
that the network can store a set of pattern associations. The association here is a pair of training input target output vector pairs
(s(p), t(p)), with p = 1,2,…p. Each vector s(p) has n components and each vector t(p) has m components. The determination of
weights is done either by using Hebb rule or delta rule. The net finds an appropriate output vector, which corresponds to an input
vector x, that may be either one of the stored patterns or a new pattern.

https://studyglance.in/nn/display.php?tno=7&topic=Associate-Memory-Network 4/6
8/20/23, 10:30 PM Associate Memory Network - Neural Networks and Deep Learning Tutorial | Study Glance

3D Interior Design Made Easy

Ad Coohom 3D Design Tool Open

Hetero Associative Memory Algorithm

Training Algorithm
Step 1 − Initialize all the weights to zero as wij = 0 i= 1 to n, j= 1 to m

Step 2 − Perform steps 3-4 for each input vector.

Step 3 − Activate each input unit as follows −

xi = si (i = 1 to n)

Step 4 − Activate each output unit as follows −

yj = sj (j = 1 to m)

Step 5 − Adjust the weights as follows −

w ij (new) = w ij (old) + xi yj

The weight can also be determine form the Hebb Rule or Outer Products Rule learning

T
w = ∑S (p) . S(p)

p=1

Testing Algorithm
Step 1 − Set the weights obtained during training for Hebb’s rule.

Step 2 − Perform steps 3-5 for each input vector.

https://studyglance.in/nn/display.php?tno=7&topic=Associate-Memory-Network 5/6
8/20/23, 10:30 PM Associate Memory Network - Neural Networks and Deep Learning Tutorial | Study Glance
Step 3 − Set the activation of the input units equal to that of the input vector.

Step 4 − Calculate the net input to each output unit j = 1 to m;

n

yinj = ∑ xi w ij

i=1

Step 5 − Apply the following activation function to calculate the output

⎧ +1
⎪ if yinj > 0

yj = f (yinj ) = ⎨ 0 if yinj = 0
⎩
⎪
−1 if yinj < 0

Next Topic : Bidirectional Associative Memory

ABOUT
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=7&topic=Associate-Memory-Network 6/6
8/20/23, 10:31 PM Bidirectional Associative Memory - Neural Networks and Deep Learning Tutorial | Study Glance

 NN&DL Menu

Bidirectional Associative Memory

Bidirectional Associative Memory (BAM)

Bidirectional associative memory (BAM), first proposed by Bart Kosko in the year 1988. The BAM network performs forward and
backward associative searches for stored stimulus responses. The BAM is a recurrent hetero associative pattern-marching nerwork
that encodes binary or bipolar patterns using Hebbian learning rule. It associates patterns, say from set A to patterns from set B
and vice versa is also performed. BAM neural nets can respond to input from either layers (input layer and output layer).

Ads by
Stop seeing this ad Why this ad?
Buy 3 Shirts Combo | 899/-
Find a Large Selection of Exclusive Designs and Beautiful
Shirts at Great Prices!

JackDumbell Shop Now

Bidirectional Associative Memory Architecture

The architecture of BAM network consists of two layers of neurons which are connected by directed weighted pare interconnecrions.
The network dynamics involve two layers of interaction. The BAM network iterates by sending the signals back and forth between
the two layers until all the neurons reach equilibrium. The weights associated with the network are bidirectional. Thus, BAM can
respond to the inputs in either layer.

Ads by
Stop seeing this ad Why this ad?

https://studyglance.in/nn/display.php?tno=8&topic=Bidirectional-Associative-Memory 1/4
8/20/23, 10:31 PM Bidirectional Associative Memory - Neural Networks and Deep Learning Tutorial | Study Glance

Ads by
Stop seeing this ad

Why this ad?

Figure shows a BAM network consisting of n units in X layer and m units in Y layer. The layers can be connected in both
directions(bidirectional) with the result the weight matrix sent from the X layer to the Y layer is W and the weight matrix for signals
sent from the Y layer to the X layer is WT. Thus, the Weight matrix is calculated in both directions.

Determination of Weights
Let the input vectors be denoted by s(p) and target vectors by t(p). p = 1, ... , P. Then the weight matrix to store a set of input and
target vectors, where

s(p) = (s1(p), .. , si(p), ... , sn(p))

t(p) = (t1(p), .. , tj(p), ... , tm(p))

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

can be determined by Hebb rule training a1gorithm. In case of input vectors being binary, the weight matrix W = {wij} is given by

https://studyglance.in/nn/display.php?tno=8&topic=Bidirectional-Associative-Memory 2/4
8/20/23, 10:31 PM Bidirectional Associative Memory - Neural Networks and Deep Learning Tutorial | Study Glance
P

w ij = ∑[2si (p) − 1][2tj (p) − 1]

p=1

When the input vectors are bipolar, the weight matrix W = {wij} can be defined as

w ij = ∑[si (p)][tj (p)]

p=1

The activation function is based on whether the input target vector pairs used are binary or bipolar

The activation function for the Y-layer

1. With binary input vectors is

⎪1
⎧ if yinj > 0

yj = ⎨ yj if yinj = 0
⎩
⎪
0 if yinj < 0

2. With bipolar input vectors is

⎧ 1 if yinj > θi
⎪

yj = ⎨ yj if yinj = θj
⎩
⎪
−1 if yinj < θj

The activation function for the X-layer

1. With binary input vectors is

⎧1 if x ini > 0

xi = ⎨ xi if x ini = 0
⎩
0 if x ini < 0

2. With bipolar input vectors is

⎧1 if x ini > θi

xi = ⎨x if x ini = θi
i
⎩
−1 if x ini < θi

Ads by
Stop seeing this ad Why this ad?
Buy 3 Shirts Combo | 899/-
Find a Large Selection of Exclusive Designs and Beautiful
Shirts at Great Prices!

JackDumbell Shop Now

Testing Algorithm for Discrete Bidirectional Associative Memory

Step 0: Initialize the weights to srore p vectors. Also initialize all the activations to zero.

Step 1: Perform Steps 2-6 for each testing input.

Step 2: Ser the activations of X layer to current input pauern, i.e., presenting the input pattern x to X layer and similarly presenting
the input pattern y to Y layer. Even though, it is bidirectional memory, at one time step, signals can be sent from only one layer. So,
either of the input patterns may be the zero vector

Step 3: Perform Steps 4-6 when the acrivacions are not converged.

Step 4: Update the activations of units in Y layer. Calculate the net input,
n

yinj = ∑ x i wij

i=1

https://studyglance.in/nn/display.php?tno=8&topic=Bidirectional-Associative-Memory 3/4
8/20/23, 10:31 PM Bidirectional Associative Memory - Neural Networks and Deep Learning Tutorial | Study Glance
Applying ilie activations, we obtain

yj = f (yinj )

Send this signal to the X layer.

Step 5: Updare the activations of unirs in X layer. Calculate the net input,
m

x ini = ∑ yj wij

j=1

Applying ilie activations, we obtain

xi = f (x ini )

Send this signal to the Y layer.

Step 6: Test for convergence of the net. The convergence occurs if the activation vectors x and y reach equilibrium. If this occurs
then stop, Otherwise, continue.

Next Topic : Hopfield Networks

ABOUT
Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=8&topic=Bidirectional-Associative-Memory 4/4
8/20/23, 10:33 PM Hopfield Networks - Neural Networks and Deep Learning Tutorial | Study Glance

Ads by
Stop seeing this ad Why this ad?
Import from Germany to Mexico
MSC offers the best shipping solution at the best market price. G
Instant Quote now

MSC Cargo Ge

 NN&DL Menu

Hopfield Networks

Hopfield Neural Network

Hopfield neural network was Proposed by John J. Hopfield in 1982. It is an auto-associative fully interconnected single layer
feedback network. It is a symmetrically weighted network(i.e., Wij = Wji). The Hopfield network is commonly used for auto-
association and optimization tasks.

The Hopfield network is of two types

1. Discrete Hopfield Network

2. Continuous Hopfield Network

Discrete Hopfield Network

When this is operated in discrete line fashion it is called as discrete Hopfield network

The network takes two-valued inputs: binary (0, 1) or bipolar (+1, -1); the use of bipolar inpurs makes the analysis easier. The
network has symmetrical weights with no self-connections, i.e.,
Wij = Wji;
Wij = 0 if i = j

Architecture of Discrete Hopfield Network

The Hopfield's model consists of processing elements with two outputs, one inverting and the other non-inverting. The outputs from
each processing element are fed back to the input of other processing elements but not to itself.

Ads by
Stop seeing this ad Why this ad?

https://studyglance.in/nn/display.php?tno=9&topic=Hopfield-Networks 1/4
8/20/23, 10:33 PM Hopfield Networks - Neural Networks and Deep Learning Tutorial | Study Glance

Training Algorithm of Discrete Hopfield Network

During training of discrete Hopfield network, weights will be updated. As we know that we can have the binary input vectors as well
as bipolar input vectors.

Let the input vectors be denoted by s(p), p = 1, ... , P. Then the weight matrix W to store a set of input vectors, where

Ads by
Stop seeing this ad Why this ad?

In case of input vectors being binary, the weight matrix W = {wij} is given by

Ads by
Stop seeing this ad Why this ad?

https://studyglance.in/nn/display.php?tno=9&topic=Hopfield-Networks 2/4
8/20/23, 10:33 PM Hopfield Networks - Neural Networks and Deep Learning Tutorial | Study Glance
P

w ij = ∑[2si (p) − 1][2sj (p) − 1] f or i ≠ j

p=1

When the input vectors are bipolar, the weight matrix W = {wij} can be defined as

w ij = ∑[si (p)][sj (p)] f or i ≠ j

p=1

Ads by
Stop seeing this ad Why this ad?
Vertex Bougainvillea
Bougainvillea offers 65 exclusive villas surrounded by open
space with abundance of nature

Vertex bougainvillea Open

Testing Algorithm of Discrete Hopfield Net

Step 0: Initialize the weights to store patterns, i.e., weights obtained from training algorithm using Hebb rule.

Step 1: When the activations of the net are not converged, then perform Steps 2-8.

Step 2: Perform Steps 3-7 for each input vector X.

Step 3: Make the initial activations of the net equal to the external input vector X:

yi = xi f or i = 1 to n

Step 4: Perform Steps 5-7 for each unit yi. (Here, the units are updated in random order.)

Step 5: Calculate the net input of the network:

yini = xi + ∑ yj w ji

Step 6: Apply the activations over the net input to calculate the output:

⎧1 if yini > θi

yi = ⎨y if yini = θi
i
⎩
0 if yini < θi

where θi is the threshold and is normally taken as zero.

Step 7: Now feed back the obtained output yi to all other units. Thus, the activation vectors are updated.

Step 8: Finally, test the network for convergence.

Continuous Hopfield Network

Continuous network has time as a continuous variable, and can be used for associative memory problems or optimization problems
like traveling salesman problem. The nodes of this nerwork have a continuous, graded output rather than a two state binary ourput.
Thus, the energy of the network decreases continuously with time.

Next Topic : Fixed Weight Competitive Networks

ABOUT

https://studyglance.in/nn/display.php?tno=9&topic=Hopfield-Networks 3/4
8/20/23, 10:33 PM Hopfield Networks - Neural Networks and Deep Learning Tutorial | Study Glance

Study Glance provides Tutorials , Power point Presentations(ppts), Lecture Notes, Important & previously asked questions, Objective Type
questions, Laboratory programs and we provide Syllabus of various subjects.

CATEGORIES
Tutorials Questions
PPTs Lab Programs
Lecture Notes Syllabus

Copyright © 2020 All Rights Reserved by StudyGlance.

   

https://studyglance.in/nn/display.php?tno=9&topic=Hopfield-Networks 4/4