Scribd
Upload
341 views1 page

OLS Estimator Derivation Guide

This document derives the ordinary least squares (OLS) estimator (β̂) in three steps: 1. It defines the population model as y = Xβ + u and writes the estimated equation as ŷ = Xβ̂. 2. It expresses the sum of squared residuals S(β̂) in terms of y, X, and β̂. 3. It takes the first order condition of S(β̂) with respect to β̂ and manipulates it to arrive at the OLS estimator: β̂ = (X'X)^-1 X'y.

Uploaded by

Anthony Sabarillo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
341 views1 page

OLS Estimator Derivation Guide

This document derives the ordinary least squares (OLS) estimator (β̂) in three steps: 1. It defines the population model as y = Xβ + u and writes the estimated equation as ŷ = Xβ̂. 2. It expresses the sum of squared residuals S(β̂) in terms of y, X, and β̂. 3. It takes the first order condition of S(β̂) with respect to β̂ and manipulates it to arrive at the OLS estimator: β̂ = (X'X)^-1 X'y.

Uploaded by

Anthony Sabarillo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

A derivation of the OLS estimator (𝜷)̂

Definitions:

1 𝑥 𝑥 ⋯ 𝑥 𝑥
𝑦
⎡1 𝑥 𝑥 ⋯ 𝑥 ⎤ ⎡𝑥 ⎤
𝒚≡ ⋮ 𝑿≡⎢
⎢⋮ ⎥
⎥ = [𝟏 𝒙 𝒙 ⋯ 𝒙 ] where 𝒙 ≡ ⎢ ⋮ ⎥
𝑦 ⋮ ⋮ ⋱ ⋮
× ⎣1 𝑥 𝑥 ⋯ 𝑥 ⎦ ⎣𝑥 ⎦ ×
×( + )

𝛽 𝑢
⎡𝛽 ⎤ ⎡𝑢 ⎤
𝜷≡⎢
⎢⋮ ⎥
⎥ 𝒖≡⎢ ⋮ ⎥
⎣𝛽 ⎦( ⎣𝑢 ⎦ ×
+ )×

Let the population model be 𝒚 = 𝑿𝜷 + 𝒖. The estimated equation is 𝒚̂ = 𝑿𝜷.̂

We can thus write 𝒖 = 𝒚 − 𝒚̂ = 𝒚 − 𝑿𝜷.̂

The sum of squared residuals is

𝑆 𝜷̂ ≡ 𝑢̂ = 𝒖 𝒖 = 𝒚 − 𝑿𝜷 ̂ 𝒚 − 𝑿𝜷 ̂ = 𝒚 − 𝜷 ̂ 𝑿 𝒚 − 𝑿𝜷 ̂
=
= 𝒚 𝒚 − 𝒚 𝑿𝜷 ̂ − 𝜷 ̂ 𝑿 𝒚 + 𝜷 ̂ 𝑿 𝑿𝜷.̂

The expressions in red are equal scalars. We can thus write them together as −2𝒚 𝑿𝜷 ̂ and re-express 𝑆 𝜷 ̂ :

𝑆 𝜷 ̂ = 𝒚 𝒚 − 2𝒚 𝑿𝜷 ̂ + 𝜷 ̂ 𝑿 𝑿𝜷.̂

We want to choose the 𝜷 ̂ that minimizes 𝑆 𝜷 ̂ . The first order condition is

𝜕𝑆 𝜷 ̂
=𝟎 ×( + ) ⟹ −2𝒚 𝑿 + 2𝜷 ̂ 𝑿 𝑿 = 𝟎 ×( + ) .
𝜕𝜷 ̂

[Note: By rules of matrix differentiation, 𝜕 −2𝒚 𝑿𝜷 ̂ /𝜕𝜷 ̂ = −2𝒚 𝑿 and 𝜕(𝜷 ̂ 𝑿 𝑿𝜷)/𝜕𝜷
̂ ̂ = 2𝜷 ̂ 𝑿 𝑿.]

Manipulating the first order condition above, we get

𝜷 ̂ 𝑿 𝑿 = 𝒚 𝑿.

Getting the transpose of both sides of the equation above yields

𝑿 𝑿𝜷 ̂ = 𝑿 𝒚.

Pre-multiplying both sides by (𝑿 𝑿)− , we get

𝜷 ̂ = (𝑿 𝑿)− 𝑿 𝒚 ∎

You might also like