CIS Faculty
Department of Information Systems
Mansoura University
Artificial Neural Networks
Artificial Neural Networks
Dr. Mohammed Elmogy
Mansoura University
Faculty of Computers & Information Sciences
Department of Information Systems
[email protected]
5. October 2011
Artificial Neural Networks 1
CIS Faculty
Department of Information Systems
Mansoura University
Artificial Neural Networks
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?
Artificial Neural Networks 2
CIS Faculty
Department of Information Systems
Mansoura University
Introduction Artificial Neural Networks
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?
Artificial Neural Networks 3
CIS Faculty
Department of Information Systems
Mansoura University
Introduction Artificial Neural Networks
A New Sort of Computer
What are computer systems good at... and not so good at?
Artificial Neural Networks 4
CIS Faculty
Department of Information Systems
Mansoura University
Introduction Artificial Neural Networks
Artificial Neural Network (ANN)
I An ANN is a mathematical model or computational model that
is inspired by the structure and/or functional aspects of
biological neural networks.
I An ANN consists of an interconnected group of artificial
neurons, and it processes information using a connectionist
approach to computation.
I In most cases an ANN is an adaptive system that changes its
structure based on external or internal information that flows
through the network during the learning phase.
Artificial Neural Networks 5
CIS Faculty
Department of Information Systems
Mansoura University
Introduction Artificial Neural Networks
Artificial Neural Network (ANN) (cont.)
I Modern neural networks are non-linear statistical data modeling
tools. They are usually used to model complex relationships
between inputs and outputs or to find patterns in data.
I Structure: large number of highly interconnected processing
elements (neurons) working together.
I Like people, they learn from experience (by example).
Artificial Neural Networks 6
CIS Faculty
Department of Information Systems
Mansoura University
Introduction Artificial Neural Networks
Where Can ANNS Help?
I When we cannot formulate an algorithmic solution.
I When we can get lots of examples of the behavior we require
(learning from experience).
I When we need to pick out the structure from existing data.
Artificial Neural Networks 7
CIS Faculty
Department of Information Systems
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?
Artificial Neural Networks 8
CIS Faculty
Department of Information Systems
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
Inspiration from Neurobiology
I A neuron: many-inputs / one-output unit.
I output can be excited or not excited.
I incoming signals from other neurons determine if the neuron
shall excite (fire).
I Output subject to attenuation in the synapses, which are
junction parts of the neuron.
Artificial Neural Networks 9
CIS Faculty
Department of Information Systems
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
Inspiration from Neurobiology (cont.)
I Dendrites: Carry signals in.
I Synapse: Size chages in response to learning. The synapse
resistance to the incoming signal can be changed during a
”learning” process.
I Axon:Carries signals away
I Nucleus: How to deal with the input?
Artificial Neural Networks 10
CIS Faculty
Department of Information Systems
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
Artificial Neurons
I The neuron calculates a weighted sum of inputs and compares
it to a threshold. If the sum is higher than the threshold, the
output is set to 1, otherwise to 0.
Artificial Neural Networks 11
CIS Faculty
Department of Information Systems
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
A Simple Perceptron
Artificial Neural Networks 12
CIS Faculty
Department of Information Systems
Mansoura University
Natural and Artificial Neurons Artificial Neural Networks
Example: A Simple Single Unit Adaptive Network
Artificial Neural Networks 13
CIS Faculty
Department of Information Systems
Mansoura University
ANN Learning Artificial Neural Networks
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?
Artificial Neural Networks 14
CIS Faculty
Department of Information Systems
Mansoura University
ANN Learning Artificial Neural Networks
Learning
I From experience: examples / training data.
I Strength of connection between the neurons is stored as a
weight-value for the specific connection.
I Learning the solution to a problem = changing the
connection weights.
Artificial Neural Networks 15
CIS Faculty
Department of Information Systems
Mansoura University
ANN Learning Artificial Neural Networks
Operation Mode
I Fix weights (unless in online learning).
I Network simulation = input signals flow through network to
outputs.
I Output is often a binary decision.
I Inherently parallel.
I Simple operations and threshold: fast decisions and real-time
response.
Artificial Neural Networks 16
CIS Faculty
Department of Information Systems
Mansoura University
ANN Learning Artificial Neural Networks
Operation Mode (cont.)
I Adaptive interaction between individual neurons.
I Power: collective behavior of interconnected neurons.
Artificial Neural Networks 17
CIS Faculty
Department of Information Systems
Mansoura University
ANN Learning Artificial Neural Networks
Evolving Networks
I Continuous process of:
I Evaluate output.
I Adapt weights.
I Take new inputs.
I ANN evolving causes stable state of the weights, but neurons
continue working: network has learned dealing with the
problem.
Artificial Neural Networks 18
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?
Artificial Neural Networks 19
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Learning Performance
I Network architecture.
I Learning method: Unsupervised, Reinforcement learning, and
Backpropagation.
Artificial Neural Networks 20
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Unsupervised Learning
I No help from the outside.
I No training data, no information available on the desired
output.
I Learning by doing.
I Used to pick out structure in the input:
I Clustering
I Reduction of dimensionality =⇒ compression
I Example: Kohonen’s Learning Law.
Artificial Neural Networks 21
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Competitive learning: Example
I Example: Kohonen network or Self-organizing Map (SOM)
(Winner takes all).
I Only update weights of winning neuron:
I Network topology.
I Training patterns
I Activation rule
I Neighbourhood
I Learning
Artificial Neural Networks 22
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
SOM
Artificial Neural Networks 23
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
SOM (cont.)
Artificial Neural Networks 24
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Competitive learning: Example (cont.)
I SOM is trained using unsupervised learning to produce a
low-dimensional (typically two-dimensional), discretized
representation of the input space of the training samples, called
a map.
I SOMs use a neighborhood function to preserve the topological
properties of the input space.
I The goal of learning is to cause different parts of the network
to respond similarly to certain input patterns. This is partly
motivated by how visual, auditory or other sensory information
is handled in separate parts of the cerebral cortex in the human
brain.
Artificial Neural Networks 25
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Reinforcement Learning
I Teacher: training data.
I The teacher scores the performance of the training examples.
I Use performance score to shuffle weights randomly.
I Relatively slow learning due to randomness.
Artificial Neural Networks 26
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Back Propagation
I It is a supervised learning method. It requires a teacher that
knows, or can calculate, the desired output for any input in the
training set.
I It is most useful for feed-forward networks. The term is an
abbreviation for backward propagation of errors.
Backpropagation requires that the activation function used by
the artificial neurons (or nodes) be differentiable
Artificial Neural Networks 27
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Back Propagation (cont.)
Artificial Neural Networks 28
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Back Propagation (cont.)
Artificial Neural Networks 29
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Back Propagation (cont.)
I Desired output of the training examples.
I Error = difference between actual & desired output.
I Change weight relative to error size.
I Calculate output layer error, then propagate back to previous
layer.
I Improved performance, very common!
Artificial Neural Networks 30
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Online / Offline
I Offline
I Weights fixed in operation mode.
I Most common.
I Online
I System learns while in operation mode.
I Requires a more complex network architecture.
Artificial Neural Networks 31
CIS Faculty
Department of Information Systems
Mansoura University
ANNs Types Artificial Neural Networks
Generalization vs. Specialization
Optimal number of hidden neurons
I Too many hidden neurons: you get an over fit, training set is
memorized, thus making the network useless on new data sets.
I Not enough hidden neurons: network is unable to learn problem
concept. The network’s language isn’t able to express the
problem solution.
I Overtraining: Too much examples, the ANN memorizes the
examples instead of the general idea.
Artificial Neural Networks 32
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
Outline
Introduction
Natural and Artificial Neurons
ANN Learning
ANNs Types
Where are NN Used?
Artificial Neural Networks 33
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
Where are NN Used?
I Recognizing and matching complicated, vague, or incomplete
patterns.
I Data is unreliable.
I Problems with noisy data.
I Prediction
I Classification
I Data association
I Data conceptualization
I Filtering
I Planning
Artificial Neural Networks 34
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
Where are NN Used? (cont.)
I Prediction: learning from past experience, pick the best stocks
in the market, predict weather, identify people with cancer risk.
I Classification: Image processing, Predict bankruptcy for credit
card companies, Risk assessment.
I Recognition: Pattern recognition: SNOOPE (bomb detector in
U.S. airports), Character recognition, Handwriting: processing
checks.
Artificial Neural Networks 35
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
Where are NN Used? (cont.)
I Data association: Not only identify the characters that were
scanned but identify when the scanner is not working properly.
I Data Conceptualization: infer grouping relationships e.g.
extract from a database the names of those most likely to buy
a particular product.
I Data Filtering: e.g. take the noise out of a telephone signal,
signal smoothing.
I Planning: Unknown environments, Sensor data is noisy, Fairly
new approach to planning.
Artificial Neural Networks 36
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
Strengths of a Neural Network
I Power: Model complex functions, nonlinearity built into the
network.
I Ease of use: Learn by example, Very little user domain-specific
expertise needed.
I Intuitively appealing: based on model of biology, will it lead to
genuinely intelligent computers/robots?
I Neural networks cannot do anything that cannot be done using
traditional computing techniques, BUT they can do some
things which would otherwise be very difficult.
Artificial Neural Networks 37
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
General Advantages
I Advantages:
I Adapt to unknown situations.
I Robustness: fault tolerance due to network redundancy.
I Autonomous learning and generalization.
I Disadvantages:
I Not exact.
I Large complexity of the network structure.
I For motion planning?
Artificial Neural Networks 38
CIS Faculty
Department of Information Systems
Mansoura University
Where are NN Used? Artificial Neural Networks
Status of Neural Networks
I Most of the reported applications are still in research stage.
I No formal proofs, but they seem to have useful applications
that work.
Artificial Neural Networks 39
