Supervised vs Unsupervised Learning

Bachelor of Technology
Computer Science and Engineering

Submitted By

ABHISHEK KUMAR (13000116143)

OCTOBER 2019

Techno India
EM-4/1, Sector-V, Salt Lake
Kolkata- 700091
West Bengal
India
 TABLE OF CONTENTS

1. Abstract
2. Introduction
3. Body
i. Supervised Learning:
Ii. Forward Pass:

Iii. Unsupervised Learning

Iv. CLASSIFICATION

V. EXPERIMENTAL OBSERVATION
4. Conclusion
5. References

TISL/CSE/Term-Paper/Semester- 2
 Abstract

This paper presents a comparative account of unsupervised and supervised learning models
and their pattern classification evaluations as applied to the higher education scenario.
Classification plays a vital role in machine based learning algorithms and in the present
study, we found that, though the error back-propagation learning algorithm as provided by
supervised learning model is very efficient for a number of non-linear real-time problems,
KSOM of unsupervised learning model, offers efficient solution and classification in the
present study.

TISL/CSE/Term-Paper/Semester- 3
 Introduction

Introduction of cognitive reasoning into a conventional computer can solve problems by example
mapping like pattern recognition, classification and forecasting. Artificial Neural Networks
(ANN) provides these types of models. These are essentially mathematical models describing a
function; but, they are associated with a particular learning algorithm or a rule to emulate human
actions. ANN is characterized by three types of parameters.

All these ANN models are unique in nature and each offers advantages of its own. The profound
theoretical and practical implications of ANN have diverse applications. Among these, much of
the research effort on ANN has focused on pattern classification. ANN performs classification
tasks obviously and efficiently because of its structural design and learning methods. There is no
unique algorithm to design and train ANN models because, learning algorithm differs from each
other in their learning ability and degree of inference. Hence, in this paper, we try to evaluate the
supervised and unsupervised learning rules and their classification efficiency using specific
example.

The overall organization of the paper is as follows. After the introduction, we present the various
learning algorithms used in ANN for pattern classification problems and more specifically the
learning strategies of supervised and unsupervised algorithms in section II.

TISL/CSE/Term-Paper/Semester- 4
 Body

Learning can refer to either acquiring or enhancing knowledge. As Herbert Simon says, Machine
Learning denotes changes in the system that are adaptive in the sense that they enable the system
to do the same task or tasks drawn from the same population more efficiently and more
effectively the next time. ANN learning paradigms can be classified as supervised, unsupervised
and reinforcement learning. Supervised learning model assumes the availability of a teacher or
supervisor who classifies the training examples into classes and utilizes the information on the
class membership of each training instance, whereas, Unsupervised learning model identify the
pattern class information heuristically and Reinforcement learning learns through trial and error
interactions with its environment (reward/penalty assignment). Though these models address
learning in different ways, learning depends on the space of interconnection neurons. That is,
supervised learning learns by adjusting its inter connection weight combinations with the help of
error signals where as unsupervised learning uses information associated with a group of neurons
and reinforcement learning uses reinforcement function to modify local weight parameters. Thus,
learning occurs in an ANN by adjusting the free parameters of the network that are adapted
where the ANN is embedded. This parameter adjustment plays key role in differentiating the
learning algorithm as supervised or unsupervised models or other models. Also, these learning
algorithms are facilitated by various learning rules as shown in the Fig bellow.

FIG-Learning Rules Of ANN

TISL/CSE/Term-Paper/Semester- 5
 Supervised Learning:
Supervised learning is based on training a data sample from data source with correct
classification already assigned. Such techniques are utilized in feed forward or Multi Layer
Perceptron (MLP) models. These MLP has three distinctive characteristics:

1. One or more layers of hidden neurons that are not part of the input or output layers of the
network that enable the network to learn and solve any complex problems

2. The nonlinearity reflected in the neuronal activity is differentiable and,

3. The interconnection model of the network exhibits a high degree of connectivity

These characteristics along with learning through training solve difficult and diverse problems.
Learning through training in a supervised ANN model also called as error back propagation
algorithm. The error correction-learning algorithm trains the network based on the input-output
samples and finds error signal, which is the difference of the output calculated and the desired
output and adjusts the synaptic weights of the neurons that is proportional to the product of the
error signal and the input instance of the synaptic weight. Based on this principle, error back
propagation learning occurs in two passes:

Forward Pass:
Here, input vector is presented to the network. This input signal propagates forward, neuron by
neuron through the network and emerges at the output end of the network as output signal: y(n) =
φ(v(n)) where v(n) is the induced local field of a neuron defined by v(n) =Σ w(n)y(n). The output
that is calculated at the output layer o(n) is compared with the desired response d(n) and finds the
error e(n) for that neuron. The synaptic weights of the network during this pass are remains
same. Backward Pass: The error signal that is originated at the output neuron of that layer is
propagated backward through network. This calculates the local gradient for each neuron in each
layer and allows the synaptic weights of the network to undergo changes in accordance with the
delta rule as:

Δw(n) = η * δ(n) * y(n).

This recursive computation is continued, with forward pass followed by the backward pass for
each input pattern till the network is converged [4-7].

Supervised learning paradigm of an ANN is efficient and finds solutions to several linear and
non-linear problems such as classification, plant control, forecasting, prediction, robotics etc [8-
9]

TISL/CSE/Term-Paper/Semester- 6
 Unsupervised Learning
Self-Organizing neural networks learn using unsupervised learning algorithm to identify hidden
patterns in unlabelled input data. This unsupervised refers to the ability to learn and organize
information without providing an error signal to evaluate the potential solution. The lack of
direction for the learning algorithm in unsupervised learning can sometime be advantageous,
since it lets the algorithm to look back for patterns that have not been previously considered [10].
The main characteristics of Self-Organizing Maps (SOM) are:

1. It transforms an incoming signal pattern of arbitrary dimension into one or 2 dimensional map
and perform this transformation adaptively
2. The network represents feed forward structure with a single computational layer consisting of
neurons arranged in rows and columns.
3. At each stage of representation, each input signal is kept in its proper context and,
4. Neurons dealing with closely related pieces of information are close together and they
communicate through synaptic connections

The computational layer is also called as competitive layer since the neurons in the layer
compete with each other to become active. Hence, this learning algorithm is called competitive
algorithm. Unsupervised algorithm in SOM works in three phases:

Competition phase: for each input pattern x, presented to the network, inner product with synaptic
weight w is calculated and the neurons in the competitive layer finds a discriminant function that induce
competition among the neurons and the synaptic weight vector that is close to the input vector in the
Euclidean distance is announced as winner in the competition. That neuron is called best matching
neuron, i.e. x = arg min ║x - w║.

Cooperative phase: the winning neuron determines the center of a topological neighborhood h of
cooperating neurons. This is performed by the lateral interaction d among the cooperative neurons. This
topological neighborhood reduces its size over a time period.

Adaptive phase: enables the winning neuron and its neighborhood neurons to increase their
individual values of the discriminant function in relation to the input pattern through suitable synaptic
weight adjustments, Δw = ηh(x)(x – w).

Upon repeated presentation of the training patterns, the synaptic weight vectors tend to follow the
distribution of the input patterns due to the neighborhood updating and thus ANN learns without
supervisor

Self-Organizing Model naturally represents the neurobiological behavior, and hence is used in
many real world applications such as clustering, speech recognition, texture segmentation, vector
coding etc [11-13].

TISL/CSE/Term-Paper/Semester- 7
 CLASSIFICATION
Classification is one of the most frequently encountered decision making tasks of human activity.
A classification problem occurs when an object needs to be assigned into a predefined group or
class based on a number of observed attributes related to that object. There are many industrial
problems identified as classification problems. For examples, Stock market prediction, Weather
forecasting, Bankruptcy prediction, Medical diagnosis, Speech recognition, Character
recognitions to name a few [14-18]. These classification problems can be solved both
mathematically and in a nonlinear fashion. The difficulty of solving such problem
mathematically lies in the accuracy and distribution of data properties and model capabilities
[19].

The recent research activities in ANN prove, ANN as best classification model due to the non-
linear, adaptive and functional approximation principles. A Neural Network classifies a given
object according to the output activation. In a MLP, when a set of input patterns are presented to
the network, the nodes in the hidden layers of the network extract the features of the pattern
presented. For example, in a 2 hidden layers ANN model, the hidden nodes in the first hidden
layer forms boundaries between the pattern classes and the hidden nodes in the second layer
forms a decision region of the hyper planes that was formed in the previous layer. Now, the
nodes in the output layer logically combines the decision region made by the nodes in the hidden
layer and classifies them into class 1 or class 2 according to the number of classes described in
the training with fewest errors on average. Similarly, in SOM, classification happens by
extracting features by transforming of m-dimensional observation input pattern into q-
dimensional feature output space and thus grouping of objects according to the similarity of the
input pattern.

The purpose of this study is to present the conceptual framework of well known Supervised and
Unsupervised learning algorithms in pattern classification scenario and to discuss the efficiency
of these models in an education industry as a sample study. Since any classification system seeks
a functional relationship between the group association and attribute of the object, grouping of
students in a course for their enhancement can be viewed as a classification problem [20-22]. As
higher education has gained increasing importance due to competitive environment, both the
students as well as the education institutions are at crossroads to evaluate the performance and
ranking respectively. While trying to retain its high ranking in the education industry, each
institution is trying to identify potential students and their skill sets and group them in order to
improve their performance and hence improve their own ranking. Therefore, we take this
classification problem and study how the two learning algorithms are addressing this problem.

In any ANN model that is used for classification problem, the principle is learning from
observation. As the objective of the paper is to observe the pattern classification properties of
those two algorithms, we developed Supervised ANN and Unsupervised ANN for the problem
mentioned above. A Data set consists of 10 important attributes that are observed as qualification

TISL/CSE/Term-Paper/Semester- 8
 to pursue Master of Computer Applications (MCA), by a university/institution is taken. These
attributes explains, the students’ academic scores, priori mathematics knowledge, score of
eligibility test conducted by the university. Three classes of groups are discovered by the input
observation [3]. Following sections presents the structural design of ANN models, their training
process and observed results of those learning ANN model.

EXPERIMENTAL OBSERVATION
Supervised ANN A 11-4-3 fully connected MLP was designed with error back-propagation
learning algorithm. The ANN was trained with 300 data set taken from the domain and 50 were
used to test and verify the performance of the system. A pattern is randomly selected and
presented to the input layer along with bias and the desired output at the output layer. Initially
each synaptic weight vectors to the neurons are assigned randomly between the range [-1,1] and
modified during backward pass according to the local error, and at each epoch the values are
normalized.

Hyperbolic tangent function is used as a non-linear activation function. Different learning rate
were tested and finally assigned between [0.05 - 0.1] and sequential mode of back propagation
learning is implemented. The convergence of the learning algorithm is tested with average
squared error per epoch that lies in the range of [0.01 – 0.1]. The input patterns are classified into
the three output patterns available in the output layer. Table I shows the different trial and error
process that was carried out to model the ANN architecture.

TISL/CSE/Term-Paper/Semester- 9
 Unsupervised ANN Kohonen’s Self Organizing Model (KSOM), which is an unsupervised
ANN, designed with 10 input neurons and 3 output neurons. Data set used in supervised model is
used to train the network. The synaptic weights are initialized with 1/√ (number of input
attributes) to have a unit length initially and modified according to the adaptability.

Results of the network depends on the presentation pattern of the input vector for small amount
of training data hence, the training patterns are presented sequentially to the NN.

Euclidean distance measure was calculated at each iteration to find the winning neuron. The
learning rate parameter initially set to 0.1, decreased over time, but not decreased below 0.01. At
convergence phase it was maintained to 0.01 [11]. As the competitive layer is one dimensional
vector of 3 neurons, the neighborhood parameter has not much influence on the activation. The
convergence of the network is calculated when there were no considerable changes in the
adaptation. The following table illustrates the results:

TISL/CSE/Term-Paper/Semester- 10
 Conclusion

Designing a classification network of given patterns is a form of learning from observation. Such
observation can declare a new class or assign a new class to an existing class. This classification
facilitates new theories and knowledge that is embedded in the input patterns. Learning behavior
of the neural network model enhances the classification properties. This paper considered the
two learning algorithms namely supervised and unsupervised and investigated its properties in
the classification of post graduate students according to their performance during the admission
period. We found out that though the error back-propagation supervised learning algorithm is
very efficient for many non- linear real time problems, in the case of student classification
KSOM – the unsupervised model performs efficiently than the supervised learning algorithm.

TISL/CSE/Term-Paper/Semester- 11
 References

 L. Fu., Neural Networks in Computer Intelligence, Tata McGraw-Hill, 2003.

 S. Haykin, Neural Networks- A Comprehensive Foundation, 2nd ed., Pearson Prentice
Hall, 2005.
 R. Sathya and A. Abraham, “Application of Kohonen SOM in Prediction”, In
Proceedings of ICT 2010, Springer-Verlag Berlin Heidelberg, pp. 313–318, 2010.
 R. Rojas, The Backpropagation Algorithm, Chapter 7: Neural Networks, Springer-
Verlag, Berlin, pp. 151-184, 1996
 M. K. S. Alsmadi, K. B. Omar, and S. A. Noah, “Back Propagation Algorithm: The Best
Algorithm Among the Multi-layer Perceptron Algorithm”, International Journal of
Computer Science and Network Security, Vol.9, No.4, 2009, pp. 378 – 383.
 X. Yu, M. O. Efe, and O. Kaynak, “A General Backpropagation Algorithm for
Feedforward Neural Networks Learning”, IEEE Trans. On Neural Networks, Vol. 13,
No. 1, 2002, pp. 251 – 254.
 Jin-Song Pei, E. Mai, and K. Piyawat, “Multilayer Feedforward Neural Network
Initialization Methodology for Modeling Nonlinear Restoring Forces and Beyond”, 4th
World Conference on Structural Control and Monitoring, 2006, pp. 1-8.
 Awodele, and O. Jegede, “Neural Networks and Its Application in Engineering”,
Proceedings of Informing Science & IT Education Conference (InSITE) 2009, pp. 83-95.
 Z. Rao, and F. Alvarruiz, “Use of an Artificial Neural Network to Capture the Domain
Knowledge of a Conventional Hydraulic Simulation Model”, Journal of
HydroInformatics, 2007, pg.no 15-24.

TISL/CSE/Term-Paper/Semester- 12