Unit-5

Artificial Neural Networks

Introduction to Artificial Neural Networks
► From Biological to Artificial Neurons,
► Biological Neurons,
► Logical Computations with Neurons,
► The Perceptron,
► Multi-Layer Perceptron and Backpropagation.
From Biological to Artificial Neurons
► The term "Artificial neural network" refers to a biologically inspired sub-field of
artificial intelligence modeled after the brain.
► An Artificial neural network is usually a computational network based on
biological neural networks that construct the structure of the human brain.
► Similar to a human brain has neurons interconnected to each other, artificial
neural networks also have neurons that are linked to each other in various layers
of the networks.
► These neurons are known as nodes.
.
What is Artificial Neural Network?
► The Term "Artificial Neural Network" is derived from Biological neural networks that develop the
structure of a human brain.
Biological Neuron
► Dendrites from Biological Neural Network represent inputs in Artificial Neural Networks, cell nucleus
represents Nodes, synapse represents Weights, and Axon represents Output.
► ANNs are composed of multiple nodes, which imitate biological neurons of
human brain.
► The neurons are connected by links and they interact with each other.
► The nodes can take input data and perform simple operations on the data.
► The result of these operations is passed to other neurons. The output at each
node is called its activation or node value.
► Each link is associated with weight. ANNs are capable of learning, which takes
place by altering weight values.
► An Artificial Neural Network in the field of Artificial intelligence where it attempts to mimic
the network of neurons makes up a human brain so that computers will have an option to
understand things and make decisions in a human-like manner.
► The artificial neural network is designed by programming computers to behave simply like
interconnected brain cells.
► There are around 1000 billion neurons in the human brain. Each neuron has an association point
somewhere in the range of 1,000 and 100,000.
► In the human brain, data is stored in such a manner as to be distributed, and we can extract more
than one piece of this data when necessary from our memory parallelly.
► the human brain is made up of incredibly amazing parallel processors.
Logical Computations with Neurons
► There are three layers in the network architecture: the input layer, the hidden layer (more than one), and the
output layer.
► Because of the numerous layers are sometimes referred to as the MLP (Multi-Layer Perceptron).
► Input Layer:
► As the name suggests, it accepts inputs in several different formats provided by the
programmer.
► Hidden Layer:
► The hidden layer presents in-between input and output layers. It performs all the calculations
to find hidden features and patterns.
► Output Layer:
► The input goes through a series of transformations using the hidden layer, which finally
results in output that is conveyed using this layer.
► The artificial neural network takes input and computes the weighted sum of the inputs and
includes a bias.
Advantages of Artificial Neural Networks
1. Attribute-value pairs are used to represent problems in ANN.
2. The output of ANNs can be discrete-valued, real-valued, or a vector of multiple real or
discrete-valued characteristics, while the target function can be discrete-valued, real-valued, or a
vector of numerous real or discrete-valued attributes.
3. Noise in the training data is not a problem for ANN learning techniques. There may be mistakes
in the training samples, but they will not affect the final result.
4. It’s utilized when a quick assessment of the taught target function is necessary.
5. The number of weights in the network, the number of training instances evaluated, and the
settings of different learning algorithm parameters can all contribute to extended training periods
for ANNs.
Disadvantages of Artificial Neural Networks
1. Hardware Dependence:
• The construction of Artificial Neural Networks necessitates the use of parallel processors.
• As a result, the equipment’s realization is contingent.
2. Understanding the network’s operation:
• This is the most serious issue with ANN.
• When ANN provides a probing answer, it does not explain why or how it was chosen.
• As a result, the network’s confidence is eroded.
3. Assured network structure:
• Any precise rule does not determine the structure of artificial neural networks.
• Experience and trial and error are used to develop a suitable network structure.
4. Difficulty in presenting the issue to the network:
• ANNs are capable of working with numerical data.
• Before being introduced to ANN, problems must be converted into numerical values.
• The display method that is chosen will have a direct impact on the network’s performance.
• The user’s skill is a factor here.
5. The network’s lifetime is unknown:
• When the network’s error on the sample is decreased to a specific amount, the training is complete.
• The value does not produce the best outcomes.
The Perceptron
► Perceptron is a building block of an Artificial Neural Network.
► Initially, in the mid of 19th century, Mr. Frank Rosenblatt invented the Perceptron for performing certain calculations to
detect input data capabilities or business intelligence.
► Perceptron is a linear Machine Learning algorithm used for supervised learning for various binary classifiers.
► This algorithm enables neurons to learn elements and processes them one by one during preparation.
► Perceptron model is also treated as one of the best and simplest types of Artificial Neural networks.
► It is a supervised learning algorithm of binary classifiers. Hence, we can consider it as a single-layer neural network with
four main parameters,
► i.e., input values,
weights and Bias,
sum
an activation function.
Basic Components of Perceptron
• Input Nodes or Input Layer:
► This is the primary component of Perceptron which accepts the initial data into the system for further
processing. Each input node contains a real numerical value.
• Wight and Bias:
► Weight parameter represents the strength of the connection between units. This is another most important
parameter of Perceptron components. Weight is directly proportional to the strength of the associated input
neuron in deciding the output. Further, Bias can be considered as the line of intercept in a linear equation.
• Activation Function:
► These are the final and important components that help to determine whether the neuron will fire or not.
Activation Function can be considered primarily as a step function.
► Types of Activation functions:
• Sign function
• Step function, and
• Sigmoid function
Types of Activation Functions
How does Perceptron work?
► In Machine Learning, Perceptron is considered as a single-layer neural network
that consists of four main parameters named input values (Input nodes), weights
and Bias, net sum, and an activation function.
► The perceptron model begins with the multiplication of all input values and their
weights, then adds these values together to create the weighted sum.
► Then this weighted sum is applied to the activation function 'f' to obtain the
desired output. This activation function is also known as the step function and is
represented by 'f'.
Activation function plays a vital role in ensuring that output is mapped between required values
(0,1) or (-1,1).
Perceptron model works in two important steps as follows:

► Step-1
► In the first step first, multiply all input values with corresponding weight values and then add them to
determine the weighted sum. Mathematically, we can calculate the weighted sum as follows:
∑ wi*xi = x1*w1 + x2*w2 +…wn*xn
► Add a special term called bias 'b' to this weighted sum to improve the model's performance.
∑ wi*xi + b
► Step-2
► an activation function is applied with the above-mentioned weighted sum, which gives us output either in
binary form or a continuous value as follows:
Y = f ( ∑wi*xi + b)
Types of Perceptron Models
► Based on the layers, Perceptron models are divided into two types. These are as follows:
1. Single-layer Perceptron Model
2. Multi-layer Perceptron model

► Single Layer Perceptron Model:

► This is one of the easiest Artificial neural networks (ANN) types. A single-layered perceptron
model consists feed-forward network and also includes a threshold transfer function inside the
model. The main objective of the single-layer perceptron model is to analyze the linearly
separable objects with binary outcomes.
► Single-layer perceptron can learn only linearly separable patterns."
► Multi-Layered Perceptron Model:
► Like a single-layer perceptron model, a multi-layer perceptron model also has the same
model structure but has a greater number of hidden layers.
► The multi-layer perceptron model is also known as the Backpropagation algorithm,
which executes in two stages as follows:
• Forward Stage: Activation functions start from the input layer in the forward stage
and terminate on the output layer.
• Backward Stage: In the backward stage, weight and bias values are modified as per
the model's requirement. In this stage, the error between actual output and demanded
originated backward on the output layer and ended on the input layer.
• A multi-layer perceptron model has greater processing power and can process linear
and non-linear patterns. Further, it can also implement logic gates such as AND, OR,
XOR, NAND, NOT, XNOR, NOR.
Advantages:
• A multi-layered perceptron model can solve complex non-linear problems.
• It works well with both small and large input data.
• Helps us to obtain quick predictions after the training.
• Helps us obtain the same accuracy ratio with big and small data.
Disadvantages:
• In multi-layered perceptron model, computations are time-consuming and complex.
• It is tough to predict how much the dependent variable affects each independent variable.
• The model functioning depends on the quality of training.
Characteristics of the Perceptron Model
► The following are the characteristics of a Perceptron Model:
1. It is a machine learning algorithm that uses supervised learning of binary classifiers.
2. In Perceptron, the weight coefficient is automatically learned.
3. Initially, weights are multiplied with input features, and then the decision is made whether the
neuron is fired or not.
4. The activation function applies a step rule to check whether the function is more significant than
zero.
5. The linear decision boundary is drawn, enabling the distinction between the two linearly separable
classes +1 and -1.
6. If the added sum of all input values is more than the threshold value, it must have an output signal;
otherwise, no output will be shown.
Limitation of Perceptron Model

► The following are the limitation of a Perceptron model:

1. The output of a perceptron can only be a binary number (0 or 1) due to the
hard-edge transfer function.
2. It can only be used to classify the linearly separable sets of input vectors. If the
input vectors are non-linear, it is not easy to classify them correctly.
Perceptron Learning Rule
► Perceptron Learning Rule states that the algorithm would automatically learn the optimal weight
coefficients. The input features are then multiplied with these weights to determine if a neuron
fires or not.
► The Perceptron receives multiple input signals, and if the sum of the input signals exceeds a
certain threshold, it either outputs a signal or does not return an output.
Perceptron Function
► Perceptron is a function that maps its input “x,” which is multiplied with the learned weight coefficient; an
output value ”f(x)”is generated.

In the equation given above:

•“w” = vector of real-valued weights

•“b” = bias (an element that adjusts the boundary away from origin without any dependence on the input value)

•“x” = vector of input x values

•“m” = number of inputs to the Perceptron

The output can be represented as “1” or “0.” It can also be represented as “1” or “-1” depending on which
activation function is used.
Multi-Layer Perceptron
► A multi-layered perceptron (MLP) is one of the most common neural network models used in the field
of deep learning.
► The multilayer perceptron (MLP) is used for a variety of tasks, such as stock analysis, image identification,
spam detection, and election voting predictions.
► The Basic Structure
► A multi-layered perceptron consists of interconnected neurons transferring information to each other, much
like the human brain. Each neuron is assigned a value. The network can be divided into three main layers.
► Input Layer
► This is the initial layer of the network which takes in an input which will be used to produce an output.
► Hidden Layer(s)
► The network needs to have at least one hidden layer. The hidden layer(s) perform computations and
operations on the input data to produce something meaningful.
► Output Layer
► The neurons in this layer display a meaningful output.
Connections
► The MLP is a feedforward neural network, which means that the data is transmitted from the
input layer to the output layer in the forward direction.
► The connections between the layers are assigned weights. The weight of a connection specifies its
importance. This concept is the backbone of an MLP’s learning process.
► While the inputs take their values from the surroundings, the values of all the
other neurons are calculated through a mathematical function involving the
weights and values of the layer before it.
Backpropagation
► Backpropagation is the essence of neural network training. It is the
method of fine-tuning the weights of a neural network based on the error
rate obtained in the previous epoch (i.e., iteration).
► Proper tuning of the weights allows you to reduce error rates and make
the model reliable by increasing its generalization.
► Backpropagation in neural network is a short form for “backward
propagation of errors.” It is a standard method of training artificial neural
networks.
► This method helps calculate the gradient of a loss function with respect to
all the weights in the network.
How Backpropagation Algorithm Works
► The Back propagation algorithm in neural network computes the gradient of the loss function for a
single weight by the chain rule. It efficiently computes one layer at a time, unlike a native direct
computation.
► It computes the gradient, but it does not define how the gradient is used. It generalizes the
computation in the delta rule.
 1. Inputs X, arrive through the preconnected path
2. Input is modeled using real weights W. The weights are usually randomly
selected.
3. Calculate the output for every neuron from the input layer, to the hidden
layers, to the output layer.
4. Calculate the error in the outputs

5.Travel back from the output layer to the hidden layer to adjust the
weights such that the error is decreased.
Why We Need Backpropagation?

• Backpropagation is fast, simple and easy to program

• It has no parameters to tune apart from the numbers of input
• It is a flexible method as it does not require prior knowledge about the
network
• It is a standard method that generally works well
• It does not need any special mention of the features of the function to be
learned.