Neural Networks
Artificial neuron
• An artificial neuron is a mathematical function conceived as a model
of biological neurons, a neural network. Artificial neurons are
elementary units in an artificial neural network.
• The artificial neuron receives one or more inputs (representing
dendrites) and sums them to produce an output (or activation)
(representing a neuron's axon). Usually each input is separately
weighted, and the sum is passed through a non-linear function known
as an activation function or transfer function
ARTIFICIAL NEURON
Biological Neuron
About 100 billion neuron in the human brain
ARTIFICIAL NEURON
Biological Neuron
ARTIFICIAL NEURON
Biological Neuron
The main body of the cell collects the incoming signals from the
other neurons through its dendrites
The incoming signals are constantly being summed in the cell
body
If the result of the summation crosses as certain threshold, the
cell body emits a signal of its own (called firing of the neuron)
This signal passes through the neuron’s axon, from where the
dendrites of other neurons pick it up
ARTIFICIAL NEURON
Biological Neuron
There are 1,000 to 10,000 dendrites in each neuron (few
millimeters long)
There is only one axon (several centimeters long)
The connection between dendrites and axon is electrochemical
and it is called synapses
The synapses modify the signal while passing it on to dendrites
The human learning is stored in these synapses, and the
connection of neurons with other neurons
ARTIFICIAL NEURON
Biological Neuron
The human learning is stored in these synapses, and the
connection of neurons with other neurons
If stimulus at a dendrite causes the neuron to fire, then the
connection between that dendrite and axon is strengthened
If the arrival of stimulus does not cause the neuron to fire, the
connection weakens over time
ARTIFICIAL NEURON
Artificial Neuron Model
ARTIFICIAL NEURON
-1
Artificial Neuron Model
Implementation of AND
function
Let W1 = W2 = 1
X1 X2 X1W1 + X2W2 +(-1xtheta) Y f(x)>=0
theta=2
0 0 0-2 =-2 0
0 1 1 -2=-1 0
1 0 1-2=-1 0
1 1 2-2=0 1
If we make = 2 (or any value >1 but <=2), we will get correct
results with a unit step activation function
ARTIFICIAL NEURON X1 x2 y
0 0 0
0 1 0
Artificial Neuron Model 1 0 0
1 1 1
If we place the 4 points in a two
coordinate system (X1 and X2),
we have drawn a line from (2, 0)
to (0, 2) in the resulting plane
x2
Any new data falling on the left
side of the line will give an
output of zero and the data on
the right side of the line will be x1
classified as one
ARTIFICIAL NEURON
x0=-1
Artificial Neuron Model
Implementation of OR
function
Let W1 = W2 = 1
f(x)>=0
X1 X2 X1W1 + X2W2 +X0 Y theta=1
0 0 0-1=-1 0
0 1 1-1=0 1
1 0 1-1=0 1
1 1 2-1=1 1
If we make = 1 (or any value >0 but <=1), we will get correct
results with a unit step activation function
ARTIFICIAL NEURON X1 x2 y
0 0 0
0 1 1
Artificial Neuron Model 1 0 1
1 1 1
If we place the 4 points in a two
coordinate system (X1 and X2),
we have drawn a line from (1, 0)
to (0, 1) in the resulting plane
Any new data falling on the left
side of the line will give an
output of zero and the data on
the right side of the line will be
classified as one
ARTIFICIAL NEURON
Artificial Neuron Model
If we want to utilize a unit step function centered at zero for both
AND and OR neurons, we can incorporate another input X0
constantly set at –1
The weight W0 corresponding to this input would be the ,
calculated previously
ARTIFICIAL NEURON
W1=0.75, w2=0.5, w0=-0.6
Setting of weights (Training)
X1W1+X2W2+x0w0
X1 X2 Y(Desired output) (9.4)(0.75)+(6.4)(0.5)+(1)(-0.6)=0.95
1.0 1.0 1 1 f(x)>=0
9.4 6.4 -1 -1
2.5 2.1 1 X0=1 Error=Desired-Actual=-1-1=-2
8.0 7.7 -1
0.5 2.2 x11
w0 Changing weights:
w1 Wnew = Wold + (Error)xi
7.9 8.4 -1 W0=-0.6+(0.2)(-2)(1)=-1
7.0 7.0 -1 W1=0.75+(0.2)(-2)(9.4)=-3.01
W2=0.5+(0.2)(-2)(6.4)=-2.06
2.8 0.8 x2 1 w2
1.2 3.0 1 X1W1+X2W2+x0w0
7.8 6.1 -1 (9.4)(-3.01)+(6.4)(-2.06)+(1)(-
1)=-28.294-13.184-1=-40.478
ARTIFICIAL NEURON
Training of weights
Supervised training: the classes of training samples are known
Random initialization of weights
(threshold or bias is considered as a weight and its input is fixed
at 1)
ARTIFICIAL NEURON
Training of weights: Algorithm
For each input, calculate the output with the current weights
The error will be equal to ydesired – actual
How to reduce this error?
Input is fixed
We have to change weights
Which weight to change?
Since we don’t know which weight is contributing how
much to the error, hence we change all weights
ARTIFICIAL NEURON
Training of weights: Algorithm
How much will be the change in a weight?
The change in each connection weight has to be
proportional to the error and the magnitude of its input
(we assume that each weight is contributing to the error
a value proportional to its input)
Change each weight by wi = (Error)xi
The learning rate is fixed at a small value, so that we may not
only learn the current data sample, but also the retain the
previous learning
ARTIFICIAL NEURON
Training of weights: Algorithm
Wnew = Wold + (Error)xi
Formula for changing weights
Iterate repeatedly over the whole data set.
Stop when the combined error of all the inputs is below a
certain threshold
ARTIFICIAL NEURON
Training of weights
For our example let [0.75, 0.5, –0.6] be the initial weights (set
randomly)
Let the learning rate be 0.2
The first data sample is classified correctly
The second data sample gives an error of –2 and the weight
vector should be updated
Wnew = Wold + (Error)xi
= -3.01
-2.06
-1.00
ARTIFICIAL NEURON
Training of weights
After 500 iterations the weights converge to [-1.3, -1.1, 10.9]
i.e. output = 1.3x1 –1.1x2 + 10.9 = 0
ARTIFICIAL NEURON
Training of weights
This training rule is called
Error-Correction Learning Rule
It is usually used for step functions
Weights are only adjusted when the neurons give an error at the
output
wi = (y – f(xiwi))xi
ARTIFICIAL NEURON
Linearly Separable Problems
If the data can be correctly divided into two categories by a line
or hyper-plane
ARTIFICIAL NEURON
Non Linear Problems
If the data cannot be correctly separated by a single line or plane,
then it is not linearly separable. A single layer of neurons cannot
classify the input patterns that are not linearly separable; e.g.
exclusive OR problem
X1 X2 Y
0 0 0
0 1 1
1 0 1
1 1 0
ARTIFICIAL NEURON N1: -1(w0)+x1(w1)+x2(w3)
X1 X2
0 0 -1(0.5)+(0)(1)+(0)(-1)=-0.5=0
Non Linear Problems 0 1 -1(0.5)+(0)(1)+(1)(-1)=-1.5=0
1 0 -1(0.5)+(1)(1)+(0)(-1)=0.5 =1
1 1 -1(0.5)+(1)(1)+(1)(-1)=-0.5 =0
For such problems we need two or more layers of neurons
N2:-1(w5)+x1(w2)+x2(w4)
-1 w0=0.5 X1 X2
w1=1 -1 0 0 -1(-0.5)+(0)(1)+(0)(-1)=0.5=1
0 1 -1(-0.5)+(0)(1)+(1)(-1)=-0.5=0
w3=-1 w8=-1 1 0 -1(-0.5)+(1)(1)+(0)(-1)=1.5=1
x1 w6=1
1 1 -1(-0.5)+(1)(1)+(1)(-1)=0.5 =1
N3:N1(w6)+N2(W7)+(-1)(w8)
N1 N2
w5=-0.5 0 1 (0)(1)+(1)(-1)+(-1)(-1)=0=1
x2
w2=1 0 0 (0)(1)+(0)(-1)+(-1)(-1)=1=0
1 1 (1)(1)+(1)(-1)+(-1)(-1)=1=0
w4=-1 w7=-1 0 1 (0)(1)+(1)(-1)+(-1)(-1)=0=1
f(x)<=0
AND NAND
OR NOR
XOR NOT
XNOR
ARTIFICIAL NEURON
Non Linear Problems
The first neuron’s output is
Point (0, 0) => Sum = -0.5 and Y1 = 0
Point (0, 1) => Sum = -1.5 and Y1 = 0
Point (1, 0) => Sum = 0.5 and Y1 = 1
Point (1, 1) => Sum = -0.5 and Y1 = 0
It places a line
ARTIFICIAL NEURON
Non Linear Problems
The second neuron’s output is
Point (0, 0) => Sum = 0.5 and Y2 = 1
Point (0, 1) => Sum = -0.5 and Y2 = 0
Point (1, 0) => Sum = 1.5 and Y2 = 1
Point (1, 1) => Sum = 0.5 and Y2 = 1
It places a line
ARTIFICIAL NEURON
The input sum for the neuron of the 2nd layer = Y1 – Y2 + 1
For point (0,1) Y1 = 0, Y2 = 0, Sum = 1, Y3 = 0
For point (1,0) Y1 = 1, Y2 = 1, Sum = 1, Y3 = 0
For point (0,0) Y1 = 0, Y2 = 1, Sum = 0, Y3 = 1
For point (1,1) Y1 = 0, Y2 = 1, Sum = 0, Y3 = 1
Thus the neuron of the 2nd layer separates the data correctly
ARTIFICIAL NEURON
The first layer projects the input data (x1, x2) into another
dimension (Y1, Y2)
x2 Y2
x1 Y1
In dimension (Y1, Y2), the data becomes linearly seperable
ARTIFICIAL NEURON
Multi Layer Perceptron: Utilization
Hence a single neuron with unit step activation function can
classify the input into two categories
A
A=0
B=1 B
ARTIFICIAL NEURON
Multi Layer Perceptron: Utilization
Two neurons with unit step activation function can classify the
input into four categories
A=0
B=1 00
10
A=0
B=1
01
11
ARTIFICIAL NEURON
Multi Layer Perceptron: Utilization
Two layers of neurons can classify complex types of inputs
B A
