Introduction to Linear
Regression and
Correlation Analysis
Scatter Diagrams
A scatter plot is a graph that may be
used to represent the relationship
between two variables. Also
referred to as a scatter diagram.
diagram
Dependent and Independent
Variables
A dependent variable is the variable to be
predicted or explained in a regression
model. This variable is assumed to be
functionally related to the independent
variable.
Dependent and Independent
Variables
An independent variable is the variable
related to the dependent variable in a
regression equation. The independent
variable is used in a regression model to
estimate the value of the dependent
variable.
Best Fit
Represents our model. It is the line that
“best fits” our data points. The line
represents the best estimate of the y value
for every given input of x.
What is Simple Linear Regression?
• Simple Linear Regression is a method used to fit
the best straight line between a set of data points.
• After a graph is properly scaled, the data points
must “look” like they would fit a straight line, not
a parabola, or any other shape.
• The line is used as a model in order to predict a
variable y from another variable x. A regression
line must involve 2 variables, the dependent and
the independent variable.
• Finding the “best-fit” line is the goal of simple
linear regression.
Two Variable Relationships
(a) Linear
Two Variable Relationships
(b) Linear
Two Variable Relationships
(c) Curvilinear
Two Variable Relationships
(d) Curvilinear
Two Variable Relationships
(e) No Relationship
Correlation
The correlation coefficient is a quantitative
measure of the strength of the linear
relationship between two variables. The
correlation ranges from + 1.0 to - 1.0. A
correlation of 1.0 indicates a perfect linear
relationship, whereas a correlation of 0
indicates no linear relationship.
Correlation
SAMPLE CORRELATION COEFFICIENT
r
( x x )( y y )
[ ( x x ) ][ ( y y )
2 2
]
where:
r = Sample correlation coefficient
n = Sample size
x = Value of the independent variable
y = Value of the dependent variable
Correlation
SAMPLE CORRELATION COEFFICIENT
or the algebraic equivalent:
n xy x y
r
[n( x 2 ) ( x) 2 ][n( y 2 ) ( y ) 2 ]
Correlation
Sales Years
y x yx y2 x2
487 3 1,461 237,169 9
445 5 2,225 198,025 25
272 2 544 73,984 4
641 8 5,128 410,881 64
187 2 374 34,969 4
440 6 2,640 193,600 36
346 7 2,422 119,716 49
238 1 238 56,644 1
312 4 1,248 97,344 16
269 2 538 72,361 4
655 9 5,895 429,025 81
563 6 3,378 316,969 36
4,855 55 26,091 2,240,687 4,855
Correlation
n xy x y
r
[n( x ) ( x) ][n( y ) ( y ) ]
2 2 2 2
12(26,091) 55(4,855)
r
[12(329) (55) 2 ][12(2,240,687) (4,855) 2 ]
0.8325
Correlation
Sales Years with Midwest
Sales 1
Years with Midwest 0.832534056 1
Correlation between Years and Sales
Excel Correlation Output
Correlation
TEST STATISTIC FOR CORRELATION
r
t
2
1 r
n2
df n 2
where:
t = Number of standard deviations r is from 0
r = Simple correlation coefficient
n = Sample size
Correlation Significance Test
H 0 : 0.0 (no correlatio n)
H A : 0.0
0.05
Rejection Region Rejection Region
/2 = 0.025 /2 = 0.025
t.025 2.228 0 t.025 2.228
10 n2
1 0.6931 1 r 2
4.752 t
0.8325 r
Since t=4.752 > 2.048, reject H0, there is a significant
linear relationship
Correlation
Spurious correlation occurs when
there is a correlation between two
otherwise unrelated variables.
Simple Linear Regression
Analysis
Simple linear regression analysis
analyzes the linear relationship that
exists between a dependent variable
and a single independent variable.
Simple Linear Regression
One Variable
• Problem: A waiter wants to predict his next tip, but he forgot to record
the bill amounts for previous tips.
• Here is a graph of his tips. The tips is the only variable. Let’s call it the
y variable.
• Meal# is not a variable. It is simply used to identify a tip.
y variable
Can we come up with a model for this problem with only 1
variable?
𝒚=
10
• Now, let’s talk about goodness of fit. This will
tell us how good our data points fit the line.
• We need to calculate the residuals (errors) for
each point.
+7
+1 +4
-2 𝒚 = 10
-5 -5
Residual
A residual is the difference between
the actual value of the dependent
variable and the value predicted by
the regression model.
y yˆ
• The best fit line is the one that minimizes the sum of the squares of the
residuals (errors).
• The error is the difference between the actual data point and the point on
the line.
• SSE (Sum Of Squared Errors) = (-5)2 + 72 + 12 + (-2) 2 + 42 + (-5) 2 = 120
+
7 +
+
1 4
- 𝒚=
- 2 -
10
5 5
• SST (Sum Of Squared Total) = SSR (Sum Of Squared Regression) + SSE is
the Sum Of Squares Equation.
• Since there is no regression line (as we only have 1 variable), we can
not make the SSE any smaller than 120, because SSR = 0.
Two Variables
Simple Linear Regression
Analysis
SIMPLE LINEAR REGRESSION MODEL
(POPULATION MODEL)
y 0 1 x
where:
y = Value of the dependent variable
x = Value of the independent variable
0= Population’s y-intercept
1 = Slope of the population regression line
= Error term, or residual
Simple Linear Regression
Analysis
The simple linear regression model has four
assumptions:
Individual values if the error terms, i, are
statistically independent of one another.
The distribution of all possible values of is normal.
The distributions of possible i values have equal
variances for all value of x.
The means of the dependent variable, for all specified
values of the independent variable, y, can be
connected by a straight line called the population
regression model.
Simple Linear Regression
Analysis
REGRESSION COEFFICIENTS
In the simple regression model, there
are two coefficients: the intercept and
the slope.
Simple Linear Regression
Analysis
The interpretation of the regression slope
coefficient is that is gives the average change
in the dependent variable for a unit increase
in the independent variable. The slope
coefficient may be positive or negative,
depending on the relationship between the
two variables.
Simple Linear Regression
Analysis
The least squares criterion is used
for determining a regression line
that minimizes the sum of squared
residuals.
Another Example…
Experience and Sales
Simple Linear Regression
Analysis
Y yˆ 150 60 x
Sales in Thousands
390
400
300
312
200
Residual = 312 - 390 = -78
100
4 X
Years with Company
Simple Linear Regression
Analysis
ESTIMATED REGRESSION MODEL
(SAMPLE MODEL)
yˆ i b0 b1 x
where:
ŷ= Estimated, or predicted, y value
b0 = Unbiased estimate of the regression intercept
b1 = Unbiased estimate of the regression slope
x = Value of the independent variable
Simple Linear Regression
Analysis
LEAST SQUARES EQUATIONS
b1
( x x )( y y )
(x x) 2
algebraic equivalent:
xy x y
b1 n
( x ) 2
x 2
n
and
b0 y b1 x
Simple Linear Regression
Analysis
SUM OF SQUARED ERRORS
SSE y b0 y b1 xy
2
Simple Linear Regression Analysis
Sales Years
y x xy y2 x2
487 3 1,461 237,169 9
445 5 2,225 198,025 25
272 2 544 73,984 4
641 8 5,128 410,881 64
187 2 374 34,969 4
440 6 2,640 193,600 36
346 7 2,422 119,716 49
238 1 238 56,644 1
312 4 1,248 97,344 16
269 2 538 72,361 4
655 9 5,895 429,025 81
563 6 3,378 316,969 36
4,855 55 26,091 2,240,687 4,855
Simple Linear Regression
Analysis
xy x y
26,091
55(4,855)
b1 n 12 49.9101
x 2
( x ) 2
329
(55) 2
n 12
b0 y b1 x 404.5833 49.9101(4.5833) 175.8288
The least squares regression line is:
yˆ 175.8288 49.9101( x)
Simple Linear Regression
Analysis
SUMMARY OUTPUT
Regression Statistics
Multiple R 0.832534056
R Square 0.693112955
Adjusted R Square 0.662424251
Standard Error 92.10553441
Observations 12
ANOVA
df SS MS F Significance F
Regression 1 191600.622 191600.622 22.58527906 0.000777416
Residual 10 84834.29469 8483.429469
Total 11 276434.9167
Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0%
Intercept 175.8288191 54.98988674 3.197475563 0.00953244 53.30369475 298.3539434 53.30369475 298.3539434
Years with Midwest 49.91007584 10.50208428 4.752397191 0.000777416 26.50996978 73.3101819 26.50996978 73.3101819
Excel Midwest Distribution Results
Least Squares Regression
Properties
The sum of the residuals from the least
squares regression line is 0.
The sum of the squared residuals is a
minimum.
The simple regression line always passes
through the mean of the y variable and the
mean of the x variable.
The least squares coefficients are unbiased
estimates of 0 and 1.
Simple Linear Regression
Analysis
SUM OF RESIDUALS
( y yˆ ) 0
SUM OF SQUARED RESIDUALS
( y yˆ ) 2
Simple Linear Regression
Analysis
TOTAL SUM OF SQUARES
TSS ( y y ) 2
where:
TSS = Total sum of squares
n = Sample size
y = Values of the dependent variable
y= Average value of the dependent variable
Simple Linear Regression
Analysis
SUM OF SQUARES ERROR (RESIDUALS)
SSE ( y yˆ ) 2
where:
SSE = Sum of squares error
n = Sample size
y = Values of the dependent variable
ŷ= Estimated value for the average of y for the
given x value
Simple Linear Regression
Analysis
SUM OF SQUARES REGRESSION
SSR ( yˆ y ) 2
where:
SSR = Sum of squares regression
y= Average value of the dependent variable
y = Values of the dependent variable
ŷ= Estimated value for the average of y for the
given x value
Simple Linear Regression
Analysis
SUMS OF SQUARES
TSS SSE SSR
Simple Linear Regression
Analysis
The coefficient of determination is the
portion of the total variation in the
dependent variable that is explained by its
relationship with the independent variable.
The coefficient of determination is also
called R-squared and is denoted as R2.
Simple Linear Regression
Analysis
COEFFICIENT OF DETERMINATION (R2)
2 SSR
R
TSS
Simple Linear Regression
Analysis
COEFFICIENT OF DETERMINATION (R2)
2SSR 191,600.62
R 0.6931
TSS 276,434.90
69.31% of the variation in the sales data for this
sample can be explained by the linear relationship
between sales and years of experience.
Simple Linear Regression
Analysis
COEFFICIENT OF DETERMINATION
SINGLE INDEPENDENT VARIABLE CASE
2 2
R r
where:
R2 = Coefficient of determination
r = Simple correlation coefficient
Simple Linear Regression
Analysis
STANDARD DEVIATION OF THE
REGRESSION SLOPE COEFFICIENT
(POPULATION)
b1
where:
(x x) 2
b=1 Standard deviation of the regression slope
(Called the standard error of the slope)
= Population standard error of the estimate
Simple Linear Regression
Analysis
ESTIMATOR FOR THE STANDARD ERROR
OF THE ESTIMATE
SSE
s
n k 1
where:
SSE = Sum of squares error
n = Sample size
k = number of independent variables in the model
Simple Linear Regression
Analysis
ESTIMATOR FOR THE STANDARD
DEVIATION OF THE REGRESSION SLOPE
s s
sb1
(x x) 2
x 2
( x ) 2
where: n
sb1= Estimate of the standard error of the least squares
slope
s = SSE Sample standard error of the estimate
n2
Simple Linear Regression
Analysis
TEST STATISTIC FOR TEST OF
SIGNIFICANCE OF THE REGRESSION SLOPE
b1 1
t df n 2
where: sb1
b1 = Sample regression slope coefficient
1 = Hypothesized slope
sb1 = Estimator of the standard error of the slope
Significance Test of
Regression Slope
H 0 : 1 0. 0
H A : 1 0.0
0.05
Rejection Region Rejection Region
/2 = 0.025 /2 = 0.025
t.025 2.228 0 t.025 2.228
sb1 10.50
t 1 1
4.753
b 49.91 0
Since t=4.753 > 2.048, reject H0: conclude that the
true slope is not zero
Simple Linear Regression
Analysis
MEAN SQUARE REGRESSION
SSR
MSR
where: k
SSR = Sum of squares regression
k = Number of independent variables in the model
Simple Linear Regression
Analysis
MEAN SQUARE ERROR
SSE
MSE
where: n k 1
SSE = Sum of squares error
n = Sample size
k = Number of independent variables in the model
Significance Test
H 0 : 1 0.0 F Ratio
H A : 1 0.0 MSR 191,600.6
22.59
0.05 MSE 8,483.43
Rejection Region
= 0.05
F 4.96
Since F= 22.59 > 4.96, reject H0: conclude that the
regression model explains a significant amount of the
variation in the dependent variable
Simple Regression Steps
Develop a scatter plot of y and x. You are
looking for a linear relationship between
the two variables.
Calculate the least squares regression line
for the sample data.
Calculate the correlation coefficient and the
simple coefficient of determination, R2.
Conduct one of the significance tests.
Simple Linear Regression
Analysis
CONFIDENCE INTERVAL ESTIMATE FOR
THE REGRESSION SLOPE
b1 t / 2 sb1
or equivalently: df n 2
s
b1 t / 2
where:
(x x) 2
sb1 = Standard error of the regression slope
coefficient
s = Standard error of the estimate
Simple Linear Regression
Analysis
CONFIDENCE INTERVAL FOR y | xp
1 ( x p x )2
yˆ t / 2 s
n (x x) 2
where:
ŷ = Point estimate of the dependent variable
t = Critical value with n - 2 d.f.
s = Standard error of the estimate
n = Sample size
xp = Specific value of the independent variable
x = Mean of independent variable observations
Simple Linear Regression
Analysis
PREDICTION INTERVAL FOR Y | xp
2
1 (xp x)
yˆ t / 2 s 1
n (x x) 2
Residual Analysis
Before using a regression model for
description or prediction, you should do a
check to see if the assumptions concerning
the normal distribution and constant
variance of the error terms have been
satisfied. One way to do this is through
the use of residual plots.
plots
Simple Linear Regression Model
- Assumptions
Assumptions of CLRM
CLRM – ASSUMPTION 1
CLRM – ASSUMPTION 2
X values are fixed in repeated sampling
Values taken by the regressor, X, are
considered fixed in repeated samples. More
technically, X is assumed to be non-
stochastic (so that Xi and ui are also
uncorrelated)
CLRM – ASSUMPTION 3
Error term εi has ZERO MEAN VALUE given the
value of X
Thus, the Conditional mean value of εi is
Zero.
That is, E(εi Xi) = 0
CLRM – ASSUMPTION 4
Homoscedasticty or Equal variance of εi
Given the value of X, the variance of εi is
same for all observations
Thus, Var(εi Xi) = E[εi - E(εi Xi)]2
CLRM – ASSUMPTION 5
No autocorrelation between the error terms
Given any two X values, Xi and Xj, the correlation
between any two εi and εj is Zero
Covar(εi,εj Xi,Xj) = E[εi - E(εj)] Xi [εj - E(εi)] Xj
= E(εi Xi)(εj Xj) = 0
CLRM – ASSUMPTION 6
Zero covariance between εi and Xi or
E(εiXi) = 0
CLRM – ASSUMPTION 7
The number of observations ‘n’ must be
greater than the number of parameters
to be estimated
That is, N > P
CLRM – ASSUMPTION 8
Variability in X values: The X values in a
given sample MUST NOT be same for all
Thus, Variance (X) must be a finite positive
number
CLRM – ASSUMPTION 9
The regression model should be
correctly specified
Therefore, there is NO SPECIFICATION
BIAS or ERROR in the model used for
empirical analysis
CLRM – ASSUMPTION 10
There is NO PERFECT MULTI-COLLINEARITY
There are no perfect linear relationships
among the explanatory variables
Example, X2 = A + B*X1
