Artificial Intelligence
Institute of Technology
University of Gondar
Biomedical Engineering Department
By Ewunate Assaye (MSc.)
Supervised Learning
Outlines: -
» ANN
» CNN
» RNN
» Prediction/Regression
2
Artificial Neural Network (ANN)
» The study of artificial neural networks (ANN) was inspired by attempts to simulate
biological neural systems.
» The human brain consists primarily of nerve cells called neurons, linked together with
other neurons via strands of fiber called axons.
» Analogous to human brain structure, an ANN is composed of an interconnected
assembly of nodes and directed links with Weights.
3
Artificial Neural Network (ANN)
» Neural Network has different layers connected to each others and work on the structure
and functions of a human brain. It learns from huge volume of data and uses complex
algorithms to train a neural net.
» ANN structure
4
Artificial Neural Network (ANN)
» You can see how they are similarity
VS
Biological Neural Network Artificial neural Network
Application of ANN
» Handwriting recognition- used to convert handwritten characters into digital characters
that the system can recognize.
» Stock Exchange prediction- ANN can examine a lot of factors and predict the prices on a
daily basis helping the stock brokers.
» Image analysis- to identify age of person
Future of ANN:
o More personalized choices for users and customers all over the world
o Hyper intelligent virtual assistant will make life easier
o ANN will be used in the field of medicine, agriculture etc.
How does ANN works?
7
How does ANN works?
8
How does ANN works?
9
How does ANN works?
Example
11
Example
12
Example
13
Loss Function
14
Gradient Descent
» The graphical methods of finding the minimum of a function is called gradient descent.
» The plot graph for weight versus loss
Back propagation
Back Propagation
Back propagation
Popular Neural Network
Feed Forward NN
Convolutional Neural Network(CNN)
Convolutional Neural Network(CNN)
How CNN recognize Image?
Convolutional Neural Network(CNN)
» How CNN recognize an image of a bird ?
Convolutional Neural Network(CNN)
» How CNN recognize an image of a bird ?
Convolutional Neural Network(CNN)
26
Why Recurrent Neural Network?
Why Recurrent Neural Network?
Application of Recurrent Neural Network
Application of Recurrent Neural Network
Application of Recurrent Neural Network
Application of Recurrent Neural Network
Type of Recurrent Neural
Type of Recurrent Neural
Neural Network as a Classifier
» Strength
o High tolerance to noisy data
o Ability to classify untrained patterns
o Well-suited for continuous-valued inputs and outputs
o Successful on an array of real-world data, e.g., hand-written letters
o Algorithms are inherently parallel
o Techniques have recently been developed for the extraction of rules from trained neural networks
» Weakness
o Long training time
o Require a number of parameters typically best determined empirically, e.g., the network topology or
“structure.”
o Poor interpretability: Difficult to interpret the symbolic meaning behind the learned weights and of
“hidden units” in the network
35
Quiz
Prediction-Regression
» (Numerical) prediction is similar to classification
o construct a model
o use model to predict continuous or ordered value for a given input
• Prediction is different from classification
o Classification refers to predict categorical class label
o Prediction models continuous-valued functions
• Major method for prediction: regression
o model the relationship between one or more independent or predictor variables and
a dependent or response variable
37
Prediction-Regression
• Regression analysis
o Linear and multiple regression
o Non-linear regression
o Other regression methods: generalized linear model, Poisson
regression, log-linear models, regression trees
39
Linear Regression
» Linear regression
o It is a type of statistical analysis that attempts to show the relationship between two variables.
o It assumes a linear relationship between the input variables (x) and the single output variable (y).
o Creates the predictive model on any data, showing trends in data
» Linear regression is a useful tool for predicting a quantitative response.
» Example, for credit approval-the bank has customer records that contain information fields
related to personal credit, such as annual salary, years in residence , outstanding loans, etc.
» Then the bank wants to set a proper credit limit for each approved customer
» Credit limits are traditionally determined by human experts. The bank wants to automate
this task, as it did with credit approval.
40
Linear Regression
» The choice of a linear model for this problem presumes that there is a linear
combination of the customer information fields that would properly approximate
the credit limit as determined by human experts.
41
Linear Regression
» The bank uses historical records to construct a model.
» Examples of dataset is (X1, Y1 ), (X2 , Y2 ), ... , (XN , YN ) , where Xn is customer information
and Yn is the credit limit set by one of the human experts in the bank.
» The bank wants to use learning to find a hypothesis g that replicates how human experts
determine credit limits.
» The label Yn comes from some distribution P(y I x) instead of a deterministic function f (x) .
» We have an unknown distribution P(x, y) that generates each ( Xn , Yn) , and we want to find a
hypothesis g that minimizes the error between g(x) and y with respect to that distribution.
» Learning a linear regression model means estimating the values of the coefficients used in the
representation with the data that we have available
42
Simple Linear Regression
» It is a very straightforward approach for predicting a quantitative response Y on
the basis of a single predictor variable X.
» It assumes that there is approximately a linear relationship between X and Y
» Mathematically: γ ≈ 𝛽0 + 𝛽1 𝑥
» ≈ is “approximately modeled as”
» For example, X may represent TV advertising and Y may represent sales.
» Then we can regress sales onto TV by fitting the model.
Sales ≈ 𝛽0 + 𝛽1 𝑇𝑉
Simple Linear Regression…
» 𝛽0 and 𝛽1 are two unknown constants that represent the intercept and slope terms
in the linear model.
» 𝛽0 and 𝛽1 are known as the model coefficients or parameters
» The training used to produce estimates 𝛽መ0 and 𝛽መ1 for the model coefficients
» We can predict future sales on the basis of a particular value of TV advertising by
computing
» 𝛾ො ≈ 𝛽መ0 + 𝛽መ1 𝑋
Simple Linear Regression
o where 𝑦ො indicates a prediction of Y on the basis of X = x
» Estimating the Coefficients
o To make predictions, we must use data to estimate the coefficients
o Let 𝑥1 , 𝑦1 , 𝑥2 , 𝑦2 , … , (𝑥𝑛 , 𝑦𝑛 )
✓Represent n observation pairs, each of which consists of a measurement of X and
a measurement of Y
o In the Advertising example, this data set consists of the TV advertising budget and
product sales in n = 200 different markets.
Simple Linear Regression
» The next figure shows the Advertising data set and the plot displays sales, in thousands of
units, as a function of TV budgets, in thousands of dollars, for 200 different markets.
» In each plot show the simple least squares fit of sales to that variable, and , the blue line
represents a simple model that can be used to predict sales using TV.
» The goal is to obtain coefficient estimates 𝛽መ0 and 𝛽መ1 such that the linear model fit the
available data i.e. 𝑦𝑖 ≈ 𝛽መ0 + 𝛽መ1 𝑥𝑖 for i=1….n.
» Or, find an intercept 𝛽መ0 and a slope 𝛽መ1 , such that the resulting line is as close as possible
to the n = 200 data points.
Simple Linear Regression
» For the Advertising data, the least squares
fit for the regression of sales onto TV is
shown.
» The fit is found by minimizing the sum of
squared errors.
» Each grey line segment represents an
error, and the fit makes a compromise by
averaging their squares.
» In this case a linear fit captures the
essence of the relationship, although it is
somewhat deficient in the left of the plot
8/2/
202
Simple Linear Regression
» Let 𝑦𝑖 = 𝛽መ0 + 𝛽መ1 𝑥𝑖 be the prediction for Y based on the 𝑖𝑡ℎ value of X.
» Then 𝑒𝑖 = 𝑦𝑖 − 𝑦ො𝑖 represents the ith residual—this is the difference between the
𝑖𝑡ℎ observed response value and the 𝑖𝑡ℎ response value that is predicted by our
linear model
Multiple Linear Regression
» In practice, there are more than one predictors
» For example, in the Advertising data, we have examined the relationship between
sales and TV advertising.
» There may have data on money spent for advertising on radio and in newspapers, and
we may want to know whether either of these two media is associated with sales.
» This accommodates multiple predictors.
» We can do this by giving each predictor a separate slope coefficient in a single model.
» suppose that we have p distinct predictors.
49
Multiple Linear Regression
» The multiple linear regression model takes the form:
𝑦 = 𝛽0 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + ⋯ + 𝛽𝑝 𝑥𝑝 + 𝜀 ,
where 𝑥𝑗 represents the 𝑗𝑡ℎ predictor and 𝛽𝑗 quantifies the association between that variable
and the response.
» We interpret 𝛽𝑗 as the average effect on 𝑦 of a one unit increase in 𝑥𝑗 , holding all other
predictors fixed.
» In the advertising example
o 𝑠𝑎𝑙𝑒𝑠 = 𝛽0 + 𝛽1 × 𝑇𝑉 + 𝛽2 × 𝑟𝑎𝑑𝑖𝑜 + 𝛽3 × 𝑛𝑒𝑤𝑠𝑝𝑎𝑝𝑒𝑟 + 𝜀
Logistic Regression(Classification)
» Logistic regression is a method used to predict a categorical dependent
variable, given a set of independent variables.
» It is a classification method which is based on the probability for a sample
to belong to a class
» Dependent variable(y): the response binary variable holding values like 0 or
1, yes or no.
» Independent variable(x): the predictor variables used to predict the response
variable
51
Logistic Regression
Logistic Regression
» Example: To predict if a student will get admitted to a school based on GPA.
53
Time series Analysis
» A time series is a series of data points indexed (listed or graphed) in time order.
» Most commonly, a time series is a sequence taken at successive equally spaced points in time.
» Time series forecasting is the use of a model to predict future values based on previously
observed values.
Example:
o stock market prediction- time series of the daily prices of various stocks
o weather forecasting- average monthly temperature for a specific place during the years 2000 to
2019
54
Time series Analysis
When Not to use Time series analysis:
o When values are constant
o When values can be represented using known functions such as Sin x
Components Time series:
o General trend- a general direction in which something is developing or changing.
o Seasonal- a peak or dip which is seen in a time interval
o Irregular fluctuation-the uncontrolled situations which arises due to which the value
changes
55
Assignment 2
1. Write python algorithm for SVM, Clustering and value based machine
learning methods.
Quiz 3
