Unit 3

Supervised

Machine Learning
Learning
Machine learning is an application of AI that enables systems to learn and improve
from experience without being explicitly programmed. Machine learning focuses on
developing computer programs that can access data and use it to learn for themselves.
The machine learning process begins with observations or data, such as examples,
direct experience or instruction. It looks for patterns in data so it can later make inferences
based on the examples provided. The primary aim of ML is to allow computers to learn
autonomously without human intervention or assistance and adjust actions
accordingly. Learning system of a machine learning algorithms can be broken down
into three main parts.
 Decision Process: In general, machine learning algorithms are used to
make a prediction or classification. Based on some input data, which can be
labelled or unlabeled, ML algorithm will produce an estimate about a pattern in
the data.
 Error Function: An error function serves to evaluate the prediction of the
model. If there are known examples, an error function can make comparison to
assess the accuracy of the model.
 Model Optimization Process: If the model can’t fit better to the data points
in the training set, then parameters are adjusted to reduce the discrepancy between
the known example and the model estimate. The algorithm will repeat this evaluate
and optimize process, until a threshold of accuracy has been met.

Types of Machine Learning

Machine learning methods fall into three primary categories: Supervised, Unsupervised,
and Reinforcement.
Supervised Learning
In this learning paradigm, we present examples of correct input-output pairs to the ML
algorithms during the training phase. This training set of examples is equivalent to the
teacher for the ML algorithms. During the training of ML algorithm under supervised
learning, then it takes input vector and computes output vector. An error signal is
generated, if there is a difference between the computed output and the desired output
vector. On the basis of this error signal, the model parameters are adjusted until the actual
output is matched with the desired output. Supervised machine learning is used for
performing tasks like: Regression and Classification. Naïve Bayes classifier, Logistic
Prepared By: Arjun Singh Saud
regression, decision tree examples of classification algorithm. Linear regression and
multiple regression are examples of regression algorithms.

Unsupervised Learning
In unsupervised learning ML algorithm is provided with dataset without desired
output. The ML algorithm then attempts to find patterns in the data by extracting
useful features and analyzing its structure. Unsupervised learning algorithms are
widely used for tasks like: clustering, dimensionality reduction, association mining etc. K-
Means algorithm, K-Medoid algorithm, Agglomerative algorithm etc. are examples of
clustering algorithms.

Reinforcement Learning
In reinforcement learning, we do not provide the machine with examples of correct
input-output pairs, but we do provide a method for the machine to quantify its
performance in the form of a reward signal. Reinforcement learning methods resemble
how humans and animals learn: the machine tries a bunch of different things and is

Prepared By: Arjun Singh Saud

rewarded with performance signal. Reinforcement learning algorithms are widely used
for training agents interacting with its environment.

Classification vs. Prediction

Classification and Prediction are two major categories of prediction problems which
are usually dealt with Data mining and machine learning. The terms prediction and
regression are used synonymously in in data mining. Both of them are supervised
learning approaches. Classification is the process of finding or discovering a model or
function which helps to predict class label for a given data. Prediction is the process of
finding a model or function which is used to predict continuous real-valued output for
a given data.

For example, we can build a classification model to categorize bank loan applications as
either safe or risky. We can also construct a classification model to identify digits. On the
other hand, we can build a regression model to predict the expenditures of a potential
customers on computer equipment given their income and occupation. We can also build
a prediction model to predict stock price given historical trading data.
Working of Classification Algorithms
The Classification process works in following two steps: Learning Step and Testing Step.
 Learning Step: This step is also called training step. In this step the learning
algorithms build a model on the basis of relationship between input and output in
the training dataset. This dataset contains input attributes along with class label
for every input tuple. Because the class label of each training tuple is provided, this
step is also known as supervised learning.
 Testing Step: In this step, the model is used for prediction. Here the test dataset is
used to estimate the accuracy of the model. This dataset contains values of input
attributes along with class label of the output attribute. However, the model
only takes values of input attributes and predicts class label of each input tuple.
Then, accuracy of the model is computed by looking at predicted class labels and
actual
Prepared By: Arjun Singh Saud
 class labels of test dataset. The model can be applied to the new data tuples if the
accuracy is considered acceptable.

Linear Regression
Regression analysis is the process of curve fitting in which the relationship between the
independent variables and dependent variables are modeled in the mth degree
polynomial. Polynomial Regression models are usually fit with the method of least mean
square (LMS). If we assume that the relationship is a linear one and only one variable,
then we can use linear equation given as below.

𝑦 = 𝑓(𝑥) = 𝑤0 + 𝑤1𝑥
In the above equation, y is dependent variable and x is independent variable, w0 and w1
are coefficient that needs to be determined through training of the model. If we have
two independent variable, linear regression equation can be written as below.

𝑦 = 𝑓(𝑥) = 𝑤0 + 𝑤1𝑥1 + 𝑤2𝑥2

We use linear_model.LinearRegression() to create a linear regression object. We then use
the fit() method to train the linear regression model. This method takes dependent and
independent variables as input. Finally, we predict values of dependent variable by
providing values of independent variable as input.
Example
import numpy as np
from sklearn.linear_model import LinearRegression
x = np.array([2,3,5,8,9,11,15,12,19,17])
y = np.array([5,7,11,17,19,23,31,25,39,35])
test=np.array([4,10,13])

Prepared By: Arjun Singh Saud

#reshaping data in column vector form
x=x.reshape((len(x),1))
test=test.reshape((len(test),1))
lr=LinearRegression()
lr.fit(x,y)
pred=lr.predict(test)
print("Test Data:",test)
print("Predicted Values:",pred)

Logistic Regression
Logistic regression is one of the most popular machine learning algorithms for binary

range of problems. We want to predict a variable 𝑦̂ ∈ {0,1} , where 0 is called

classification. This is because it is a simple algorithm that performs very well on a wide

negative class, while 1 is called positive class. Such task is known as binary classification.
The heart of the logistic regression technique is logistic function and is defined as given in
equation
(1). Logistic function transforms the input into the range [0, 1]. Smallest negative numbers
results in values close to zero and the larger positive numbers results in values close to
one.
1
f (x)
 1 ex

If there are two input variable, logistic regression has two coefficients just like linear
regression.

y  w 0  w 1 x1  w 2 x2

Unlike linear regression, the output is transformed into a probability using the logistic
function.

1
yˆ   (
y)  1 ey

If the probability is > 0.5 we can take the output as a prediction for the class 1, otherwise
the prediction is for the class 0. The job of the learning algorithm will be to discover the
best values for the coefficients (w0, w1, and w2) based on the training data.

We use linear_model.LogisticRegression() to create a logistic regression object. We then use

the fit() method to train the logistic regression model. This method takes target and
independent variables as input. Finally, we predict values of target variable by providing
values of independent variable as input.
Prepared By: Arjun Singh Saud
Example
import numpy as np
from sklearn.linear_model import LogisticRegression
x = np.array([3.78, 2.44, 2.09, 0.14, 1.72, 1.65, 4.92, 4.37, 4.96, 4.52,
3.69, 5.88])
y= np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
test=np.array([2.93,1.86,5.24,6.32])

#reshaping data in column vector form

x=x.reshape((len(x),1))
test=test.reshape((len(test),1))
lr=LogisticRegression()
lr.fit(x,y)
pred=lr.predict(test)
print("Test Data:",test)
print("Predicted Values:",pred)

Naïve Bayes Classification

Naïve Bayes classification also called Bayesian classification or Naïve Bayes classification.
It is a classification technique based on Bayes’ Theorem which assumes that input
features of the model and independent of each other. Bayes’ theorem is sated as below.

P( X | H )P(H )
P(H | X )  P( X )

Bayes’ theorem is useful in that it provides a way of calculating the posterior

probability, P(H|X), from P(H), P(X|H), and P(X). Here P(X) and P(H) are prior
probability and P(X|H) is likelihood. Let D be a database and C1,C2……Cm are m classes.
Now above Bayes rule can be written as below.

P(Ci | X )  P( X | Ci )P(Ci )
P( X )

Given a tuple, X, the classifier will predict that X belongs to the class having the highest
posterior probability, conditioned on X. Thus we need to maximize P(Ci|X). As P(X) is
constant for all classes, only P(X|Ci)P(Ci) need to be maximized. Let X is the set of
attributes {x1, x2, x3…….xn} where attributes are independent of one another. Now the
probability P(X|Ci) is given by the equation given below.
n

P( X | Ci )   P(xk | Ci )  P(x1 | Ci )  P(x2 | Ci )  P(xn | Ci )

k 1

Prepared By: Arjun Singh Saud

In puthon, GaussianNB class of sklearn.naive_bayes module is used to create Naïve
Bayes classifiers instance and perform classification using the model. Consider the
dataset given below.

Example given below creates Naïve Bayes classifier model using the above training
data and then predicts class level of the tuple: X = (age = youth, income = medium, student =
yes, credit_rating = fair) using the model.
Example
from sklearn.naive_bayes import GaussianNB
from sklearn import preprocessing
import pandas as pd
Age=['Youth','Youth','Middle_Aged','Senior','Senior','Senior','Middle_Aged
','Youth','Youth','Senior','Youth','Middle_Aged','Middle_Aged','Senior','Y
outh']
Income=['High','High','High','Medium','Low','Low','Low','Medium','Low','Me
dium','Medium','Medium','High','Medium','Medium']
Student=['No','No','No','No','Yes','Yes','Yes','No','Yes','Yes','Yes','No'
,'Yes','No','Yes']
Credit_Rating=['Fair','Excellent','Fair','Fair','Fair','Excellent','Excell
ent','Fair','Fair','Fair','Excellent','Excellent','Fair','Excellent','Fair
']
Buys=['No','No','Yes','Yes','Yes','No','Yes','No','Yes','Yes','Yes','Yes',
'Yes','No',"?"]

Prepared By: Arjun Singh Saud

le = preprocessing.LabelEncoder()
a=list(le.fit_transform(Age))
i=list(le.fit_transform(Income))
s=list(le.fit_transform(Student))
cr=list(le.fit_transform(Credit_Rating))
b=list(le.fit_transform(Buys))
d={'Age':a,'Income':i,'Student':s, 'Credit_Rating':cr, 'Buys_Computer':b}
df=pd.DataFrame(d)
print(df)
x=df[['Age','Income','Student','Credit_Rating']]
y=df['Buys_Computer']
trainx=x[0:14]
trainy=y[0:14]
testx=x[14:15]

model = GaussianNB()
model.fit(trainx,trainy)
predicted= model.predict(testx)
if(predicted==1):
pred='No'
else:
pred='Yes'
print("Predicted Value:", pred)

Decision Tree Classifier

Decision tree classification algorithm is the learning of decision trees from class labeled
training tuples. Decision tree is a flowchart-like tree structure where internal nodes (non
leaf node) denotes a test on an attribute, branches represent outcomes of tests, and Leaf
nodes (terminal nodes) hold class labels.

Prepared By: Arjun Singh Saud

Once the decision tree is learned, in order to make prediction for a tuple, the attributes
of a tuple are tested against the decision tree. A path is traced from the root to a leaf
node which determines the predicted class for that tuple. Constructing a Decision tree
uses greedy algorithm. Tree is constructed in a top-down divide-and-conquer manner.
High level algorithm for decision tree algorithm is presented below.

1. At start, all the training tuples are at the root

2. Tuples are partitioned recursively based on selected attributes

3. If all samples for a given node belong to the same class

• Label the class

4. Else if there are no remaining attributes for further partitioning

• Majority voting is employed for assigning class label to the leaf

5. Else

• Got to step 2

There are many variations of decision-tree algorithms. Some of them are: ID3 (Iterative
Dichotomiser 3), C4.5 (successor of ID3), CART (Classification and Regression Tree) etc. There
are different attribute selection measures used by decision tree classifiers. Some of them
are: Information Gain, Gain Ratio, Gini Index etc. ID3 stands for Iterative Dichotomiser 3. It
uses top-down greedy approach to build decision tree model. This algorithm computes
information gain for each attribute and then selects the attribute with the highest

Prepared By: Arjun Singh Saud

information gain. Information gain measures reduction in entropy after data
transformation. It is calculated by comparing entropy of the dataset before and after
transformation. Entropy is the measure of homogeneity of the sample. Entropy or
expected information of dataset D is calculated by using equation(1) given below.
m

E(D)    p i log 2 pi (1)

i1

Where pi is the probability of a tuple in D belonging to class Ci and is estimated

using Equation (2).

pi  Ci, D (2)
D

Where Ci,D is the number of tuples in D belonging to class Ci and is the number
of tuples in D.

Suppose we have to partition the tuples in D on some attribute A having v distinct

values. The attribute A can be used to split D into v partitions {D1,D2,..,Dv}. Now, the
total entropy of data partitions while partitioning D around attribute A is calculated
using Equation (3).
Dj
E (D)  
v  E(D
(3)
)A j

j1 D

Finally, the information gain achieved after partitioning D on attribute A is calculated

using Equation (4).
IG(A)  E(D)  EA
(4)
(D)
In python, we create instance of DecisionTreeClassifier class in sklearn.tree module. The
instance can be trained using training dataset for learning predictive model in the form
of decision tree structure. The model can then be used to predict class label for the new
data tuples. We have to supply the parameter value criterion = "entropy" to create
instance of ID3 decision tree.
Example
Use the dataset given below to train ID3 decision tree classifier and predict class label
for the input tuple {Outlook=Sunny, Temperature=Hot, Humidity=Normal,
Windy=Strong}.

Prepared By: Arjun Singh Saud

from sklearn.tree import DecisionTreeClassifier
from sklearn import preprocessing
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn import tree

outlook=['Sunny','Sunny','Overcast','Rainy','Rainy','Rainy','Overcast','Su
nny','Sunny', 'Rainy','Sunny','Overcast','Overcast','Rainy','Sunny']
temp=['Hot','Hot','Hot','Mild','Cool','Cool','Cool','Mild','Cool','Mild','
Mild','Mild','Hot','Mild','Hot']
humidity=['High','High','High','High','Normal','Normal','Normal','High','N
ormal','Normal','Normal','High','Normal','High','Normal']
wind=['Weak','Strong','Weak','Weak','Weak','Strong','Strong','Weak','Weak'
,'Weak','Strong','Strong','Weak','Strong','Strong']
play=['No','No','Yes','Yes','Yes','No','Yes','No','Yes','Yes','Yes','Yes',
'Yes','No','?']

d={'Outlook':outlook,'Temperature':temp,'Humidity':humidity, 'Windy':wind,
'Play_Tennis':play}
df=pd.DataFrame(d)
Le = LabelEncoder()
df['Outlook'] = Le.fit_transform(df['Outlook'])
df['Temperature'] = Le.fit_transform(df['Temperature'])
df['Humidity'] = Le.fit_transform(df['Humidity'])
df['Windy'] = Le.fit_transform(df['Windy'])

Prepared By: Arjun Singh Saud

df['Play_Tennis'] = Le.fit_transform(df['Play_Tennis'])
print(df)
x=df[['Outlook','Temperature','Humidity','Windy']]
y=df['Play_Tennis']
trainx,trainy=x[0:14],y[0:14]
testx,testy=x[14:15],y[14:15]

dt = DecisionTreeClassifier(criterion = 'entropy')
dt.fit(trainx,trainy)
p= dt.predict(testx)
p=Le.inverse_transform(p)
print("Predicted Label:",p)

Prepared By: Arjun Singh Saud