SOFT COMPUTING

UNIT - 1
Soft Computing :
 It is an important branch of study in the area of intelligent and knowledge-
based systems.
 The idea of soft computing was initiated in 1981 when Lotfi A. Zadeh
published his first paper on soft data analysis “What is Soft Computing”.
 Zadeh, defined Soft Computing into one multidisciplinary system as the fusion
of the fields of Fuzzy Logic, Neuro-Computing, Evolutionary and Genetic
Computing, and Probabilistic Computing.
 Soft computing uses a combination of Neural Networks, Fuzzy logic and
Genetic Algorithm.
 Soft Computing is the fusion of methodologies designed to model and enable

solutions to real world problems, which are not modeled or too difficult to model

mathematically.

 The aim of Soft Computing is to exploit the tolerance for imprecision, uncertainty,

approximate reasoning, and partial truth in order to achieve close resemblance with

human like decision making.

 The ultimate goal of soft computing is to be able to emulate the human mind as

closely as possible.
 Zadeh defined Soft Computing as –

“Soft Computing is an emerging approach to computing which parallel the remarkable

ability of the human mind to reason and learn in a environment of uncertainty and

imprecision”.

Comparison between Soft Computing and Hard Computing

• Soft computing deals with approximate models and gives solution to complex problems.

• Hard computing deals with precise models where accurate solutions are achieved quickly.
 Constituents of Soft Computing
Fuzzy logic, neural networks, probabilistic reasoning, and genetic algorithms are cooperatively
used in soft computing for knowledge representation and for mimicking the reasoning and
decision-making processes of a human.
 Fuzzy Logic
o FL was introduced in the year 1965 by Lofti A. Zadeh.
o It is a mathematical tool for dealing with uncertainty.
o It provides a technique to deal with imprecision and information granularity.
o FL is a superset of conventional (or Boolean) logic and contains similarities and
differences with Boolean logic.
o FL is similar to Boolean logic results are returned by FL operations when all fuzzy
memberships are restricted to 0 and 1.
o FL differs from Boolean logic in that it is permissive of natural queries and is more like
human thinking; it is based on degrees of truth.
o FL is a form of multi-valued logic to deal with reasoning that is approximate rather than
precise.
o Zadeh presented FL as a way of processing data by allowing partial set membership rather
than crisp set membership or non-membership.
o Fuzzy logic is a problem-solving control system methodology that lends itself to
implementation in systems ranging from simple, small, embedded microcontrollers to large,
networked, multichannel PC or workstation based data acquisition and control systems.
 Artificial Neural Networks
o ANNs are massively connected networks of computational neurons and represent parallel-
distributed processing structures.
o The inspiration for NN has come from the biological architecture of neurons in the human
brain.
o A key characteristic of neural networks is their ability to approximate arbitrary nonlinear
functions.
o The main objective of neural network research is to develop a computational device for
modeling the brain to perform various computational tasks at a faster rate tehn the
traditional systems.
o An ANN is composed of a large number of highly interconnected processing elements,
called neurons, working to solve specific problems.

7
o A neural network consists of a set of nodes, usually organized into layers, and connected
through weight elements called synapses.
o At each node, the weighted inputs are summed (aggregated), threshold, and subjected to an
activation function in order to generate the output of that node.
o Through the use of neural networks, an intelligent system would be able to learn and
perform high-level cognitive tasks.
o ANNs perform various tasks such as pattern matching and classification, optimization
function, approximation, vector quantization, and data clustering.
 Probabilistic Reasoning

o Probabilistic reasoning may be viewed as similar to the fuzzy logic reasoning, considering

uncertainty in the place of fuzziness as the concept of approximation that is applicable.

o Probability distribution and density functions are employed in place of membership

functions.

o The Bayesian approach is commonly used.

o The Bayes’ relation states-

[Note

P(A|B) is posterior conditional probability

P (B|A) is class conditional probability

P(A) is priori probability ] 10

 Genetic Algorithm
o The word genetics is derived from the Greek word genesis meaning to grow or to become.
o Genetics is the science that deals with the mechanism responsible for similarities and
differences in a species.
o The concepts of GAs are directly derived from natural evolution and heredity.
o GAs are adaptive heuristic search algorithms based on the evolutionary ideas of natural
selection and genetics.
o GAs belong to the area of evolutionary computing.
o Evolutionary computing has the following characteristics-
• It is based on multiple searching points or solution candidates (population-based search).
• It uses evolutionary operations such as crossover and mutation.
• It is based on probabilistic operations.
o Evolutionary computing can play an important role in the development of an optimal and
self-improving intelligent machine.
o GAs are better than conventional algorithms in that they are more robust.
o GAs are powerful adaptive methods to solve search and optimization problems.
o A genetic algorithm works with a population of individuals, each representing a possible
solution to a given problem.
o Each individual is assigned a fitness score according to how good its solution to the
problem is.
o The highly fit individuals are given opportunities to reproduce by cross breeding with other
individuals in the population.
o This produces new individuals as offspring, who share the some features taken from each
parent.

12
 Hybrid Systems
These systems can be classified into three different systems –
o Neuro-Fuzzy Hybrid Systems
• Neuro-fuzzy systems represent a newly developed class of hybrid intelligent systems
combining the main features of artificial neural networks with those of fuzzy logic
systems.
• Goal of these systems is to avoid difficulties encountered in applying fuzzy logic for
systems represented by numerical knowledge (data sets), or conversely in applying
neural networks for systems represented by linguistics information(fuzzy sets).
• It is recognized that neither fuzzy reasoning systems nor neural networks are by
themselves capable of solving problems involving at the same time both linguistic and
numerical knowledge.
o Neuro-Genetic Hybrid Systems

• Neuro-fuzzy hybrid system is a system in which neural network employs a genetic

algorithm to optimize its structural parameters that define its architecture.

• The parameters of neural networks are encoded by genetic algorithms as a string of

properties of the network or chromosomes.

• A large population of chromosomes is generated, which represent the many possible

parameter sets for the given neural network.

• This hybrid system has the ability to locate the neighborhood of the optimal solution

quickly as compared to other traditional search strategies.

o Fuzzy-Genetic Hybrid Systems
The integration of fuzzy systems and genetic systems can be performed in the following
two ways –
• By the use of fuzzy logic based techniques for improving genetic algorithm behavior and
modelling GA components. This is called Fuzzy genetic algorithms (FGAs).
• By the application of genetic algorithms in various optimization and search problems
involving fuzzy systems.
 Application areas of Soft Computing

 Communication Systems - In communication systems Soft Computing can be used

effectively to find solutions that have not yet been resolved by Hard Computing. Neural

Networking and Fuzzy Logic combine and provide approaches from Neuro-Fuzzy to use for

information purposes and representation.

 Robotics - Robotics is an emerging field which is based on human thinking and behavior.

Fuzzy Logic and Expert System techniques integrate in a way to develop useful real world

applications. Also neuro-fuzzy system hybridization is the representation of fuzzy logic with

learning capabilities. Neuro-fuzzy approach learns obstacle avoidance and wall-following

behavior on a small size robot. Present day intelligence is considered to be interactive

information processing among humans and artificial objects.

 Transportation - Soft Computing is applicable in constructing intelligent vehicles and

provide efficient environment to each other i.e. to machines and drivers. Intelligent vehicle

control requires recognition of the driving environment and planning of driving that is easily

acceptable for drivers. The field of transportation deals with passengers, logistics operations,

fault diagnosis etc. Fuzzy Logic and Evolutionary Computing are often used in elevator

control systems.

 Healthcare - Health care environment is very much reliant to on computer technology. With

the advancement in computer technology, the use of Soft Computing methods provide better

and advance aids that assists the physician in many cases, rapid identification of diseases and

diagnosis in real time. Soft Computing techniques are used by various medical applications
 such as Medical Image Registration Using Genetic Algorithm, Machine Learning techniques

to solve prognostic problems in medical domain, Artificial Neural Networks in diagnosing

cancer and Fuzzy Logic in various diseases.

 Data Mining - Data Mining is a form of knowledge discovery used for solving problems in a

particular area. Data sets may be gathered and implemented collectively for purposes others

that those for which they were originally created. The three basic methodologies of Soft

Computing are widely applied in the data mining. Fuzzy Logic plays important role in

modeling different functions of data mining.

 Clustering - Data Mining extracts information from large database in order to reveal useful

information for decision making by a user. Fuzzy sets focused on a specified search and help

to discover dependencies between the data in qualitative format. Various fuzzy clustering

algorithms for mining telecommunications customer and business customer market share.
 Education - Soft computing techniques are used in the field of education for improving the

performance of students in academics. NN and fuzzy logic helps in doing the evaluation of

students’ grades in different courses. Bayesian network helps in detecting the learning style of

students. This enables the teachers to forecast the GPA of students and take effective

measures in improving the score.

 Association Rule - Association rules are the describing rules that associate relationship

among different attributes. The use of fuzzy technique has been considered the key features

of data mining systems. Neural Network contributes towards the data mining from rule

extraction and clustering. Neural Network has also been used for a variety of classification

and regression tasks. In addition, combination of neural network and fuzzy logic is one of the

most popular hybridization that integrates the merits of neural and fuzzy approaches.

 Crime forecasting - Machine learning algorithms can be used to reduce the crime rates. ML
helps to analyze the crime data and make predictions on the type of crime that might occur in

future at some location. Thus, ML and data mining helps in minimizing the crime rate levels in

the society.
 ARTIFICIAL NEURAL NETWORKS

 An Artificial Neural Network is an information-processing system that has certain

performance characteristics in common with biological neural networks.

 A neural network is a massively parallel distributed processor made up of simple

processing units that has a natural propensity for storing experiential knowledge and

making it available for use. It resembles the brain in two respects:

o Knowledge is acquired by the network from its environment through a learning

process.

o Interneuron connection strengths, known as synaptic weights, are used to store the

acquired knowledge.
 Artificial Neural Networks have been developed as generalizations of mathematical models

of human cognition or neural biology, based on the assumptions that –

o Information processing occurs at many simple elements called neuron. Signals are

passed between neurons over connection links.

o Each connection link has an associated weight.

o Each neuron applies an activation function to its net input to determine its output signal.
 Biological Neural Network
 In1911, Ramόn y Cajál introduced the idea of neurons as structural
constituents of the brain.
 The brain consists of a large number (approximately of highly connected
neurons and approximately connections per neuron.
 The net result is that the energetic efficiency of the brain is approximately
10-16 joules (J) per operation per second.
 According to Arbib (1987) : Neurons communicate with each other by
means of electrical impulses.
 Every neuron has three components:
o Dendrites,
o Cell body, and
o Axon.
 Dendrites : They form a dendritic tree, which is a very fine bush of thin
fibers around the neuron's body. They receive information from neurons
through axons. These are receptive networks of nerve fibers that carry
electrical signals into the cell body.
 Cell body : It is also called soma. Nucleus cell is located in soma. It
effectively sums and thresholds these incoming signals.
 The axon : It is a single long fiber that carries the signal from the cell body
out to other neurons.
• The point of contact between an axon of one cell and a dendrite of another
cell is called a synapse.
• The signals reaching a synapse and received by dendrites are electrical
impulses.
• The inter-neuronal transmission is sometimes electrical but is usually effected
by the release of chemical transmitters at the synapse. Thus, terminal buttons
generate the chemical that affects the receiving neuron.
• The receiving neuron either generates an impulse to its axon, or produces no
response.
Conditions necessary for the firing of a neuron:
 Incoming impulses can be excitatory if they cause the firing, or Inhibitory if
they hinder the firing of the response.
 Excitation should exceed the inhibition by the amount called the threshold of

the neuron.

 In 1987, Arbib stated that typically a value of threshold is about 40mV.

 Since a synaptic connection causes the excitatory or inhibitory reactions of

the receiving neuron, thus assign positive and negative unity weight values

are assigned respectively to them.

 The neuron fires when the total of the weights to receive impulses exceeds

the threshold value during the latent summation period.

 Comparison between the Artificial Neural Network and Biological Neural Network
Characteristics ANN BNN

Speed ANNs are faster in processing Biological neurons are slow in processing
information. Cycle time corresponding information. Cycle time corresponding
to execution of one time step of a to a neural event prompted by an
program in the CPU is in the range of external stimulus occurs in a
few nanoseconds. milliseconds range.

Processing Operate in sequential mode one BNNs can perform massively parallel
instruction after another. operations.
Size and ANNs do not involve much The number of neurons in the brain is
Complexity computational neurons. Hence it is estimated to about the total number of
difficult to perform complex pattern interconnections to be around The size
recognition. and complexity of connections gives the
brain the power of performing complex
pattern recognition tasks.
Storage Information is stored in memory BNNs store information in the
which is addressed by location. Any strength of the interconnections.
new information in the same location Information in the brain is
destroys the old information. Hence it adaptable, because new information
is strictly replaceable. is added by adjusting the
interconnection strengths, without
destroying the old information.

Fault tolerance ANNs are not inherently fault BNNs exhibit fault tolerance since
tolerant, since the information the information is distributed in the
corrupted in the memory cannot be connections throughout the network.
retrieved.

Control There is a control unit, which There is no central control for

mechanism monitors all the activities of processing information in the brain.
computing.
 HISTORY of ANN
The history of neural networks begins in the early 1940’s:
 1943: Warren McCulloch and Walter Pitts outlined the first formal model of
an elementary computing neuron. The model included all necessary elements to
perform logic operations, and thus it could function as an arithmetic logic
computing element. The formal neuron model was not widely adopted for the
vacuum tube computing hardware description, and the model never became
technically significant.
 1949: Donald Hebb first proposed a learning scheme for updating neuron's
connections known as Hebbian learning rule. The rule implies that the
 connection between two neurons is strengthened when both neurons are
active at the same time. This change in strength is proportional to the product
of the two activities.
 1950: The neuropsychologist Karl Lashley stated in his thesis that brain
information storage is realized as a distributed system.
 1954: Marvin Minsky built and tested the first neuro-computer called as
Snark. which has already been capable to adjust its weights automatically. But
it has never been practically implemented.
 1956: Well-known scientists met at the Dartmouth Summer Research
Project and discussed how to simulate a brain.
 1957-1958: At the MIT, Frank Rosenblatt, Charles Wightman and their
coworkers developed the first successful neurocomputer - the Mark I
perceptron. It which was capable to recognize simple numerics by means of
a 20 × 20 pixel image sensor and electromechanically worked with 512 motor
driven potentiometers, where each potentiometer representing one variable
weight.
 1958: Neuron-like element called perceptron was invented by Frank
Rosenblatt. It was a trainable machine capable of learning to classify certain
patterns by modifying connections to the threshold elements.
 1959: Frank Rosenblatt described different versions of the perceptron,
formulated and verified his perceptron convergence theorem.
 1960: Bernard Widrow and Marcian E. Hoff introduced the ADALINE
(ADAptive LInear NEuron) and a new powerful learning rule called the
Widrow-Hoff (1960, 1962). The rule minimized the summed square error
during training involving pattern classification.
 1969: Marvin Minsky and Seymour Papert published a precise mathematical
analysis of the perceptron to show that the perceptron model was not capable of
representing many important problems such as XOR problem and linear
separability.
 1972: Teuvo Kohonen introduced a model of the linear associator, a model of
an associative memory.
 1973: Christoph von der Malsburg used a neuron model that was nonlinear
and biologically more motivated.
 1974: For his dissertation in Harvard Paul Werbos developed a learning
procedure called backpropagation of error.
 1976-1980 and thereafter: Stephen Grossberg along with Gail Carpenter
presented neural models of adaptive resonance theory (ART).
 1982: Teuvo Kohonen described the self-organizing feature maps also known
as Kohonen maps to study and analyze the mechanisms involving self-
organization in the brain. In this year, John Hopfield also invented the
Hopfield networks which are inspired by the laws of magnetism in physics.
 1983: Fukushima, Miyake and Ito introduced the neural model of the
Neocognitron which could recognize handwritten characters and was an
extension of the Cognitron network already developed in 1975.
 1985: John Hopfield published an article describing a way of finding
acceptable solutions for the Travelling Salesman problem by using Hopfield
nets.
 1986: The backpropagation of error learning procedure as a generalization of
the delta rule was separately developed and widely published by the Parallel
Distributed Processing Group.
 1987: The first open conference on neural networks – the IEEE International
Conference on Neural Networks was held in San Diego and the International
Neural Network Society (INNS) was formed.
 1988: INNS journal Neural Networks was founded.
 1988: Grossberg developed a learning rule similar to that of Kohonen, which is
widely used in Counter Propagation network. this learning occurs for all the
units in a particular layer and no competition among these units is assumed.
 1988: Broomhead and Lowe developed Radial Basis Functions (RBF), a
multilayer network quiet similar to the back propagation net.
 1989: Yann LeCun provided the first practical demonstration of
Backpropagation at Bell Labs. He combined Convolutional Neural Networks
with backpropagation on handwritten digits.
 1989: INNS journal Neural Computation was founded.
 1990: Vladimir Vapnik and Dana Cortes developed the Support Vector
Machine (SVM).
 1990: INNS journal IEEE Transactions on Neural Networks was founded.
 1999: Computers started becoming faster at processing data and GPU
(graphics processing units) were developed. Faster processing, with
GPUs processing pictures, increased computational speeds by 1000 times
over a 10 year span.

 2001:A research report by META Group (now called Gartner) described

challenges and opportunities of data growth as three-dimensional. The
report described the increasing volume of data and increasing speed of
data as increasing the range of data sources and types.

 2009:AI professor Fei-Fei Li at Stanford launched ImageNet, assembled

a free database of more than 14 million labeled images. The Internet is
full of unlabeled images. Labeled images are required to train neural nets.

 2009 - 2012 : The recurrent neural networks and deep feedforward

neural networks were developed in the research group of Schmidhuber.
37
 2011: The speed of GPUs (graphics processing units) had increased
significantly, making it possible to train convolutional neural
networks without the layer-by-layer pre-training.

 2014: Scientists from IBM introduced the processor (TrueNorth),

with the architecture similar to that existing in the brain. IBM
presented the integrated circuit the size of postage stamp able to
simulate the work of the millions of neurons and 256 million of
synapses in a real time. The system is able to execute from 46 to 400
billion synaptic operations per second.

38
 Artificial neuron
A neuron is an information-processing unit that is fundamental to the operation
of a neural network.
 A set of synapses or connecting links, each of which is characterized by a

weight or strength of its own. Specifically, a signal at the input of synapse j

connected to neuron k is multiplied by the synaptic weight

 An adder for summing the input signals, weighted by the respective synapses

of the neuron;

 An activation function (also referred to as a squashing function) for limiting

the amplitude of the output of a neuron. It squashes or limits the permissible

amplitude range of the output signal to some finite value.

 Weights – In the architecture of an ANN, each neuron is connected to other neurons by
means of direct communication links, and each communication link is associated with
weights. The weight contain information about the input signal.
 Bias - the bias is considered as another weight. The bias is included by adding a component
x0 =1 to the input vector. The bias included in the network has its impact in calculating the
net input. The bias can be of two types- positive bias and negative bias. The positive bias
helps in increasing the net input of the network and the negative bias helps in decreasing the
net input of the network.
 Threshold – the threshold value is used in the activation function. A comparison is made
between the calculated net input and the threshold to obtain the net output. Thus, threshold is
a set value based upon which the final output of the network may be calculated.

42
 An externally applied bias, denoted by which has the effect of increasing or lowering
the net input of the activation function, depending on whether it is positive or
negative, respectively.
Mathematically a neuron may be described by the equations:

Effect of bias can be shown as :

43
 Neural Network Architectures

The arrangement of neurons to form layers and connection pattern formed within and between

layers is called the network architecture.

There exist 5 types of neuron connection architectures-

(a) Single layer feed forward network

(b) Multilayer feed-forward network

(c) Single node with its own feedback

(d) Single-layer recurrent network

(e) Multilayer recurrent network

44
(a) Single layer feed forward network

• This type of network comprise of two layers – input layer and output layer

• The input layer neurons receive the input signals and the output layer neurons receive the

output signals.

• The synaptic links carrying the weights connect every input neuron to the output neuron

but not vice-versa.

• If no neuron in the output layer is an input to a node in the same layer or the proceeding

layer, then network is said to be the Feed forward network.

• Thus, in a single layer feed forward network, the input layer merely transmits the signals

to the output layer.

45
Single layer feed forward network

46
(b) Multilayer feed forward network

• This network is made up of multiple layers.

• Thus, architecture of a multilayer network besides possessing an input and an output layer

also have one or more intermediary layers called hidden layers.

• The computational units of the hidden layer are known as hidden neurons or hidden units.

• The hidden layer aids in performing useful intermediary computations before directing the

input to the output layer.

• The input layer neurons are linked to the hidden layer neurons and the weights on these links

are referred to as input-hidden layer weights.

47
• The hidden layer neurons are linked to the output layer neurons and the weights on these

links are referred to as hidden-output layer weights.

Multilayer feed forward network

48
(c) Single node with its own feedback

• It is a simple recurrent neural network having a single neuron with feedback to itself.

• A network is called a Recurrent network, if there is atleast one feedback loop in the network.

• If outputs are directed back as input to the processing elements in the same layer or in the

proceeding layer, then network is called the feedback network.

Single node with own feedback

49
(d) Single layer recurrent network

• It is a single layer network with feedback connection in which a processing element’s output

can be directed back to the processing element itself or to other processing element or to

both.

Single layer recurrent network

50
(e) Multilayer recurrent network

• Processing element output can be directed back to the nodes in the preceding layer, forming a

multilayer recurrent network.

Multilayer recurrent network

51
Activation Functions
 CHARCTERISTICS of ANN

 It is a neutrally implemented mathematical model.

 There exist a large number of highly interconnected processing elements called neurons.
 The interconnections with their weighted linkages hold the informative knowledge.
 The input signals arrive at the processing elements through connections and connecting
weights.
 The neural networks can map input patterns to their associated output patterns.
 Artificial neural networks learn by example, therefore they can be trained with known
examples of a problem to acquire knowledge about it.
 ANNs possess the capability to generalize. They can predict new outcomes from past trends.
 ANNs are robust systems and are fault tolerant. Therefore, they can recall full patterns
from incomplete, partial or noisy patterns.
 Artificial Neural Networks can process information in parallel and in a distributed manner.
 The computational power can be demonstrated only by the collective behavior of neurons
and no single neuron carries specific information.
 Benefits of Artificial Neural Network

1. Nonlinearity: An ANN can be linear or nonlinear. Nonlinearity is distributed

throughout the network.
2. Input-Output mapping: In most cases, learning involves modification of the
synaptic weights of a neural network by applying a set of labeled training
samples or task examples. Thus, the network learns from the examples by
constructing an input-output mapping for the problem at hand.
3. Adaptivity: ANNs have a built-in capability to adapt their synaptic weights
to changes in the surrounding environment.
4. Contextual Information: Contextual information is dealt with naturally by a
neural network as knowledge is represented by the very structure and
activation state of a neural network. hence, every neuron in the network is
potentially affected by the global activity of all other neurons in the network.

5. Fault Tolerance: A neural network has the potential to be inherently fault

tolerant, or capable of robust computation. Its performance degrades
gracefully
under adverse operating conditions. Thus, in principle, a neural network
exhibits a graceful degradation in performance rather than catastrophic failure.
6. VLSI Implementation : The massively parallel nature of a neural network
makes it potentially fast for the computation of certain tasks. This same feature
makes a neural network well suited for implementation using very-large-scale-
integrated (VLSI) technology.
7. Uniformity of Analysis and Design: Neural networks enjoy universality as
information processors and same notation is used in all domains involving the
application of neural networks such as neurons, in one form or another, represent
an ingredient common to all neural networks. This commonality makes it
possible to share theories and learning algorithms in different applications of
neural networks. Modular networks can be built through a seamless integration of
modules.
8. Neurobiological Analogy: The design of a neural network is motivated by analogy
with the brain, which makes it fast and powerful for fault-tolerant parallel processing.
Neurobiologists look to (artificial) neural networks as a research tool for the
interpretation of neurobiological phenomena as well as engineers look to neurobiology
for new ideas to solve problems more complex than those based on conventional
hardwired design techniques.
9. Self-organization: An ANN can create its own organization or representation of the
information it receives during learning time.
10. Real-time operation: ANN computations may be carried out in parallel, using
special hardware devices designed and manufactured to perform real-time processing.