Regression
analysis
▷ A regression problem is composed of
• an outcome or response variable 𝑌
• a number of risk factors or predictor variables 𝑋𝑖 that affect 𝑌
• also called explanatory variables, or features in the machine learning community
• a question about 𝑌, such as How to predict 𝑌 under different conditions?
▷𝑌 is sometimes called the dependent variable and 𝑋𝑖 the independent
variables
• not the same meaning as statistical independence
• experimental setting where the 𝑋𝑖 variables can be modified and
changes in 𝑌
can be observed
1 / 68
Regression analysis:
objectives
Prediction Model inference
We want to learn about the relationship
We want to estimate 𝑌 at some specific between 𝑌 and 𝑋𝑖, such as the combination
values of 𝑋𝑖 of predictor variables which has the most
effect on 𝑌
2 / 68
Univariate linear
regression
(when all you have is a single predictor
variable)
3 / 68
Linear
regression
▷ Linear regression: one of the simplest and most commonly used
statistical modeling techniques
▷ Makes strong assumptions about the relationship between the
predictor variables (𝑋𝑖) and the response (𝑌) 0.8
0.6
• (a linear relationship, a straight line when plotted)
0.4
• only valid for continuous outcome variables (not applicable to
0.2
category outcomes such as success/failure)
0
0 0.5 1 1.5 2 2.5 3 3.5 4
outcome predictor
variable variable
“Fitting a line
𝑦 = 𝛽 0 + 𝛽1 × 𝑥 + error
through data”
intercept slope
4 / 68
Linear
regression
▷ Assumption: 𝑦 = 𝛽 0 + 𝛽 1 × 𝑥 + error
▷ Our task: estimate 𝛽 0 and 𝛽 1 based on the available data
▷ Resulting model is 𝑦̂ = 𝛽0̂ + 𝛽1̂ × 𝑥
• the “hats” on the variables represent the fact that they are
estimated from the available data
• 𝑦̂ is read as “the estimator for 𝑦”
▷ 𝛽 0 and 𝛽 1 are called the model parameters or coefficients
180
▷ Objective: minimize the error, the difference between our 170
observations and the predictions made by our linear 160
model 150
• minimize the length of the red lines in the figure to the right
(called the “residuals”)
140
40 45 50 55 60 65 70
5 / 68
Ordinary Least Squares
regression
▷ Ordinary Least-Squares (ols) regression: a
method for selecting the model parameters 190
• β₀ and β₁ are chosen to minimize the square of the
distance between the predicted values and the 180
actual values
170
• equivalent to minimizing the size of the red
rectangles in the figure to the right
160
▷ An application of a quadratic loss function
150
• in statistics and optimization theory, a loss function, 40 45 50 55 60 65 70 75 80 85
90
or cost function, maps from an observation or event
to a number that represents some form of “cost”
6 / 68
Simple linear regression:
example
▷ The British Doctors’ Study followed the health of a large number of
physicians in the uk over the period 1951–2001
▷ Provided conclusive evidence of linkage between smoking and lung
cancer, myocardial infarction, respiratory disease and other illnesses
▷ Provides data on annual mortality for a variety of diseases at four levels
of cigarette smoking:
1 never smoked
2 1-14 per day
3 15-24 per day
4 > 25 per day
More information: ctsu.ox.ac.uk/research/british-doctors-study
7 / 68
Simple linear regression: the
data
cigarettes smoked CVD mortality lung cancer mortality
(per day) (per 100 000 men per year) (per 100 000 men per year)
0 572 14
10 (actually 1-14) 802 105
20 (actually 15-24) 892 208
30 (actually >24) 1025 355
sease
CVD: cardiovascular di
Source: British Doctors’ Study
8 / 68
Simple linear regression:
plots
Deaths for different smoking intensities
1000
import pandas
900 import matplotlib.pyplot as plt
CVD deaths
data = pandas.DataFrame({"cigarettes": [0,10,20,30],
800
"CVD": [572,802,892,1025],
"lung": [14,105,208,355]});
700 data.plot("cigarettes", "CVD", kind="scatter")
plt.title("Deaths for different smoking intensities")
plt.xlabel("Cigarettes smoked per day")
600
plt.ylabel("CVD deaths")
0 5 10 15 20 25 30
Cigarettes smoked per day
lude that
Quite tempting to conc
deaths
cardiovascular disease
cigarette
increase linearly with
consumption…
9 / 68
Aside: beware assumptions of
causality
1964: the US Surgeon General issues a
report claiming that cigarette
smoking causes lung cancer, based
mostly on correlation data similar to
the previous slide.
lung
smoking
cancer
12 /
Aside: beware assumptions of
causality
1964: the US Surgeon General issues a
report claiming that cigarette
hidden
smoking causes lung cancer, based
mostly on correlation data similar to factor?
the previous slide.
However, correlation is not sufficient
to demonstrate causality. There might
lung
be some hidden genetic factor that smoking
cancer
causes both lung cancer and desire
for nicotine.
12 /
Beware assumptions of
causality
▷ To demonstrate the causality, you need a randomized controlled
experiment
▷ Assume we have the power to force people to smoke or not smoke
• and ignore moral issues for now!
▷ Take a large group of people and divide them into two groups
• one group is obliged to smoke
• other group not allowed to smoke (the “control” group)
▷ Observe whether smoker group develops more lung cancer than the
control group
▷ We have eliminated any possible hidden factor causing both smoking and
lung cancer
▷ More information: read about design of experiments
13 /
Fitting a linear model in
Python
▷ In these examples, we use the statsmodels library for statistics in
Python
• other possibility: the scikit-learn library for machine learning
▷ We use the formula interface to ols regression, in
statsmodels.formula.api
▷ Formulas are written outcome ~ observation
• meaning “build a linear model that predicts variable outcome as a function
of input data on variable observation”
14 /
Fitting a linear
model
import numpy, pandas
CVD deaths for different smoking intensities
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
1000
df = pandas.DataFrame({"cigarettes": [0,10,20,30],
900
"CVD": [572,802,892,1025],
CVD deaths
"lung": [14,105,208,355]});
800
df.plot("cigarettes", "CVD", kind="scatter")
lm = smf.ols("CVD ~ cigarettes", data=df).fit()
700 xmin = df.cigarettes.min()
xmax = df.cigarettes.max()
600 X = numpy.linspace(xmin, xmax, 100)
# params[0] i s th e i n t e r c e p t ( b e t a ₀ )
0 5 10 15 20 25 30 # params[1] i s th e s l o p e ( b e t a ₁ )
Cigarettes smoked per day Y = lm.params[0] + lm.params[1] * X
plt.plot(X, Y, color="darkgreen")
15 /
Parameters of the linear
model
60
▷ 𝛽0 is the intercept of the regression line
(where it meets the 𝑋 = 0 axis)
40
▷ 𝛽 1 is the slope of the regression line
▷ Interpretation of 𝛽 1 = 0.0475: a “unit” 20
𝛽 = Δ𝑦
1
increase in cigarette smoking is associated 𝛽0 { Δ𝑥
with a 0.0475 “unit” increase in deaths 0
from lung cancer 0 5 10 15 20 25 30
16 /
Scatterplot of lung cancer
deaths
Lung cancer deaths for different smoking
intensities
350
300 import pandas
250 import matplotlib.pyplot as plt
200
Lung cancer
data = pandas.DataFrame({"cigarettes": [0,10,20,30],
150 "CVD": [572,802,892,1025],
deaths
"lung": [14,105,208,355]});
100 data.plot("cigarettes", "lung", kind="scatter")
plt.xlabel("Cigarettes smoked per day")
50
plt.ylabel("Lung cancer deaths")
0
0 5 10 20 30
15 Cigarettes smoked per
25 day
lude that lung
Quite tempting to conc
linearly with
cancer deaths increase
cigarette consumption…
17 /
Fitting a linear
model
import numpy, pandas
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
Lung cancer deaths for different smoking intensities
df = pandas.DataFrame({"cigarettes": [0,10,20,30],
350
"CVD": [572,802,892,1025],
300 "lung": [14,105,208,355]});
df.plot("cigarettes", "lung", kind="scatter")
250
lm = smf.ols("lung ~ cigarettes", data=df).fit()
200
Lung cancer
xmin = df.cigarettes.min()
150 xmax = df.cigarettes.max()
deaths
X = numpy.linspace(xmin, xmax, 100)
100
# params[0] i s th e i n t e r c e p t ( b e t a ₀ )
50 # params[1] i s th e s l o p e ( b e t a ₁ )
Y = lm.params[0] + lm.params[1] * X
0
plt.plot(X, Y, color="darkgreen")
0 5 10 20 30
15 Cigarettes smoked per
25 day
d
Download the associate
Python notebook at
rg
risk-engineering.o
18 /
Using the model for
prediction
Q: What is the expected lung cancer mortality risk for a group of people
who smoke 15 cigarettes per day?
import numpy, pandas
import statsmodels.formula.api as smf
df = pandas.DataFrame({"cigarettes": [0,10,20,30],
"CVD": [572,802,892,1025],
"lung": [14,105,208,355]});
# create and f i t the l i n e a r model
lm = smf.ols(formula="lung ~ cigarettes",
data=df).fit() # use the f i t t e d model f o r p r e d i c t i o n
lm.predict({"cigarettes": [15]}) / 100000.0
# p r o b a b i l i t y o f mo r t a lit y from lung cancer, per person
per year
array([ 0.001705])
19 /
▷ How do we assess how well the linear model fits our observations?
Assessing
model
quality
• make a visual check on a scatterplot
• use a quantitative measure of “goodness of fit”
For simple linear regression, 𝑟2 is simply the square of the sample
▷
correlation coefficient 𝑟
• ▷ Coefficient of determination 𝑟2: a number that indicates how well data fit a statistical model
• it’s the proportion of total variation of outcomes explained by the m
• 𝑟2 = 1: regression line fits perfectly
• 𝑟2 = 0: regression line does not fit at all
20 /
