Series and series

representation
Practical Time Series Analysis
Thistleton and Sadigov
Objectives
• Recall infinite series and their convergence
• Examine geometric series
• Represent rational functions as a geometric series
 Sequence and series
• Sequence 𝑎𝑛 is list of numbers in definite order

𝑎1 , 𝑎2 , 𝑎3 , … 𝑎𝑛 , …
• If the limit of the sequence exists, i..e,

lim 𝑎𝑛 = 𝑎
𝑛→∞
then we say the sequence is convergent.
 Examples
𝑛
• 𝑎𝑛 =
𝑛+1
1 2 3 𝑛
, , ,…, ,… → 1
2 3 4 𝑛+1
• 𝑎𝑛 = 3𝑛
3, 9, 27, … , 3𝑛 , …
• 𝑎𝑛 = 𝑛
1, 2, 3, … , 𝑛, …
1
• 𝑎𝑛 =
𝑛2
1 1 1
1, , , … , 2 , … → 0
4 9 𝑛
Partial sums
• Partial sums of a sequence 𝑎𝑛 are defined as

𝑠𝑛 = 𝑎1 + 𝑎2 + ⋯ + 𝑎𝑛
• 𝑠1 = 𝑎1
• 𝑠2 = 𝑎1 + 𝑎2
• 𝑠3 = 𝑎1 + 𝑎2 + 𝑎3
.
.
.
 Series
• If the partial sums 𝑠𝑛 is convergent to a number s, then we
say

∞
the infinite series 𝑘=1 𝑎𝑘 is convergent, and is equal to s.
∞

𝑎𝑘 = lim 𝑠𝑛 = lim (𝑎1 + 𝑎2 + ⋯ + 𝑎𝑛 ) = 𝑠

𝑛→∞ 𝑛→∞
𝑘=1

∞
• Otherwise, we say 𝑘=1 𝑎𝑘 is divergent.
Some convergent series
∞ 1
• 𝑘=1 2𝑘 =1

∞ 1 𝜋2
• 𝑘=1 𝑘 2 =
6

∞ −1 𝑘+1
• 𝑘=1 = ln 2
𝑘
Some divergent series
∞ 𝑘
• 𝑘=1 3

∞
• 𝑘=1(2𝑘 + 1)

∞ 1
• 𝑘=1 𝑘
 Absolute convergence
• Series is absolutely convergent if
∞

|𝑎𝑘 |
𝑘=1

is convergent.

• Absolute convergence implies convergence.

Convergence tests
• Integral test
• Comparison test
• Limit comparison test
• Alternating series test
• Ratio test
• Root test
Geometric series
• Geometric sequence
∞
𝑎𝑟 𝑛−1 𝑛=1 = {𝑎, 𝑎𝑟, 𝑎𝑟 2 , 𝑎𝑟 3 , … }

• Geometric series

∞ 𝑘−1 𝑎
𝑘=1 𝑎𝑟 =
1−𝑟
if 𝑟 < 1.

1
∞ 1 1 1 1 2
• 𝑘=1 2𝑘 = + + +⋯= 1 =1
2 4 8 1−
2
1 1
since 𝑎 = , 𝑟 = .
2 2
Series representation
1
• Series representation for
1−𝑥
where 𝑎 = 1, 𝑟 = 𝑥.

1
= 1 + 𝑥 + 𝑥2 + 𝑥3 + ⋯
1−𝑥

if 𝑥 < 1.
 Series representation cont.
1
• Series representation for 𝑥
1−𝑥 1−2

∞
1 2 −1 1 𝑘
𝑥 = + 𝑥 = 2 − 𝑘
𝑥
1−𝑥 1− 1 − 𝑥 1− 2
2 2 𝑘=0

𝑥
If 𝑥 < 1 and < 1, i.e., if 𝑥 < 1.
2
 Complex functions
Assume z is a complex number

𝑎 ∞
= 𝑎 + 𝑎𝑧 + 𝑎𝑧 2 + ⋯ = 𝑘=1 𝑎𝑧 𝑘−1
1−𝑧

if |z| < 1.
 What We’ve Learned

• The definition of infinite series and their convergence

• Geometric series is convergent if the multiplier has norm less

than 1

• How to represent some rational functions as a geometric

series
Backward shift operator
Practical Time Series Analysis
Thistleton and Sadigov
 Objectives

• Define and utilize backward shift operator

Definition
• 𝑋1 , 𝑋2 , 𝑋3 , …
• Backward shift operator is defined as

𝐵𝑋𝑡 = 𝑋𝑡−1

• 𝐵2 𝑋𝑡 = 𝐵𝐵𝑋𝑡 = 𝐵𝑋𝑡−1 = 𝑋𝑡−2

• 𝐵𝑘 𝑋𝑡 = 𝑋𝑡−𝑘
Example – Random Walk
𝑋𝑡 = 𝑋𝑡−1 + 𝑍𝑡

𝑋𝑡 = 𝐵𝑋𝑡 + 𝑍𝑡

1 − 𝐵 𝑋𝑡 = 𝑍𝑡

𝜙 𝐵 𝑋𝑡 = 𝑍𝑡
where

𝜙 𝐵 =1−𝐵
Example – MA(2) process
𝑋𝑡 = 𝑍𝑡 + 0.2𝑍𝑡−1 + 0.04𝑍𝑡−2

𝑋𝑡 = 𝑍𝑡 + 0.2𝐵𝑍𝑡 + 0.04𝐵2 𝑍𝑡

𝑋𝑡 = (1 + 0.2𝐵 + 0.04𝐵2 ) 𝑍𝑡

𝑋𝑡 = 𝛽 𝐵 𝑍𝑡
where

𝛽 𝐵 = 1 + 0.2𝐵 + 0.04𝐵2
Example – AR(2) process
𝑋𝑡 = 0.2𝑋𝑡−1 + 0.3𝑋𝑡−2 + 𝑍𝑡

𝑋𝑡 = 0.2𝐵𝑋𝑡 + 0.3𝐵2 𝑋𝑡 + 𝑍𝑡

(1 − 0.2𝐵 − 0.3𝐵2 ) 𝑋𝑡 = 𝑍_𝑡

𝜙(𝐵)𝑋𝑡 = 𝑍𝑡
where

𝜙 𝐵 = 1 − 0.2𝐵 − 0.3𝐵2
MA(q) process (with a drift)
𝑋𝑡 = 𝜇 + 𝛽0 𝑍𝑡 + 𝛽1 𝑍𝑡−1 + ⋯ + 𝛽𝑞 𝑍𝑡−𝑞 ,

Then,

𝑋𝑡 = 𝜇 + 𝛽0 𝑍𝑡 + 𝛽1 𝐵1 𝑍𝑡 + ⋯ + 𝛽𝑞 𝐵𝑞 𝑍𝑡 ,

𝑋𝑡 − 𝜇 = 𝛽 𝐵 𝑍𝑡 ,
where
𝛽 𝐵 = 𝛽0 + 𝛽1 𝐵 + ⋯ + 𝛽𝑞 𝐵𝑞 .
 AR(p) process

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ + 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

Then,
𝑋𝑡 − 𝜙1 𝑋𝑡−1 − 𝜙2 𝑋𝑡−2 − ⋯ − 𝜙𝑝 𝑋𝑡−𝑝 = 𝑍𝑡

𝑋𝑡 − 𝜙1 𝐵𝑋𝑡 − 𝜙2 𝐵2 𝑋𝑡 − ⋯ − 𝜙𝑝 𝐵𝑝 𝑋𝑡 = 𝑍𝑡

𝜙 𝐵 𝑋𝑡 = 𝑍𝑡 ,
Where
𝜙 𝐵 = 1 − 𝜙1 𝐵 − 𝜙2 𝐵2 − ⋯ − 𝜙𝑝 𝐵𝑝 .
 What We’ve Learned
• The definition of the Backward shift operator

• How to utilize backward shift operator to write

MA(q) and AR(p) processes
Introduction to Invertibility
Practical Time Series Analysis
Thistleton and Sadigov
 Objectives

• Learn invertibility of a stochastic process

Two MA(1) models
• Model 1

𝑋𝑡 = 𝑍𝑡 + 2𝑍𝑡−1

• Model 2

1
𝑋𝑡 = 𝑍𝑡 + 𝑍𝑡−1
2
 Theoretical Auto Covariance Function of
Model 1
𝛾 𝑘 = 𝐶𝑜𝑣 𝑋𝑡+𝑘 , 𝑋𝑡 = 𝐶𝑜𝑣 𝑍𝑡+𝑘 + 2𝑍𝑡+𝑘−1 , 𝑍𝑡 + 2𝑍𝑡−1

If 𝑘 > 1, then 𝑡 + 𝑘 − 1 > 𝑡, so all 𝑍’s are uncorrelated, thus 𝛾 𝑘 = 0.

If 𝑘 = 0, then
𝛾 0 = 𝐶𝑜𝑣 𝑍𝑡 + 2𝑍𝑡−1 , 𝑍𝑡 + 2𝑍𝑡−1 =

𝐶𝑜𝑣 𝑍𝑡 , 𝑍𝑡 + 4𝐶𝑜𝑣 𝑍𝑡−1 , 𝑍𝑡−1 = 𝜎𝑍2 + 4𝜎𝑍2 = 5𝜎𝑍2 .

If 𝑘 = 1, then

𝛾 1 = 𝐶𝑜𝑣 𝑍𝑡+1 + 2𝑍𝑡 , 𝑍𝑡 + 2𝑍𝑡−1 = 𝐶𝑜𝑣 2𝑍𝑡 , 𝑍𝑡

= 2𝜎𝑍2

If 𝑘 < 0, then
𝛾 𝑘 = 𝛾(−𝑘)
Auto Covariance Function and ACF of Model 1
0, 𝑘>1
2𝜎𝑍2 , 𝑘=1
𝛾 𝑘 =
5𝜎𝑍2 , 𝑘=0
𝛾 −𝑘 , 𝑘<0

𝛾 𝑘
Then, since 𝜌 𝑘 = ,
𝛾 0

0, 𝑘>1
2
, 𝑘=1
𝜌 𝑘 = 5
1, 𝑘=0
𝜌 −𝑘 , 𝑘<0
ACF
 ACF of Model 2
1 1 1
𝛾 1 𝐶𝑜𝑣 𝑍𝑡+1 + 𝑍𝑡 , 𝑍𝑡 + 𝑍𝑡−1 2
𝜌 1 = = 2 2 = 2 = .
𝛾 0 1 1 1 5
𝐶𝑜𝑣[𝑍𝑡 + 𝑍𝑡−1 , 𝑍𝑡 + 𝑍𝑡−1 ] 1 +
2 2 4

Thus we obtain the same ACF:

0, 𝑘>1
2
, 𝑘=1
𝜌 𝑘 = 5
1, 𝑘=0
𝜌 −𝑘 , 𝑘<0
ACFs are same!
 Inverting through backward substitution

MA(1) process
𝑋𝑡 = 𝑍𝑡 + 𝛽𝑍𝑡−1 ,

𝑍𝑡 = 𝑋𝑡 − 𝛽𝑍𝑡−1 = 𝑋𝑡 − 𝛽 𝑋𝑡−1 − 𝛽𝑍𝑡−2 = 𝑋𝑡 − 𝛽𝑋𝑡−1 + 𝛽 2 𝑍𝑡−2

In this manner,

𝑍𝑡 = 𝑋𝑡 − 𝛽𝑋𝑡−1 + 𝛽 2 𝑋𝑡−2 − 𝛽 3 𝑋𝑡−3 + ⋯

i.e.,
𝑋𝑡 = 𝑍𝑡 + 𝛽𝑋𝑡−1 − 𝛽 2 𝑋𝑡−2 + 𝛽 3 𝑋𝑡−3 − ⋯

We ‘inverted’ MA(1) process to AR(∞).

 Inverting using Backward shift operator

𝑋𝑡 = 𝛽 𝐵 𝑍𝑡

where

𝛽 𝐵 = 1 + 𝛽𝐵

Then, we find 𝑍𝑡 by inverting the polynomial operator 𝛽 𝐵 :

𝛽 𝐵 −1 𝑋 = 𝑍𝑡
𝑡
 Inverse of 𝛽(𝐵)

−1
1
𝛽 𝐵 = = 1 − 𝛽𝐵 + 𝛽2 𝐵2 − 𝛽3 𝐵3 + ⋯
1 + 𝛽𝐵

Here we expand the inverse of the polynomial operator as a

‘rational function where 𝛽𝐵 is a complex number’.

Thus we obtain,

−1
𝛽 𝐵 𝑋𝑡 = 1 − 𝛽𝑋𝑡−1 + 𝛽2 𝑋𝑡−2 − 𝛽3 𝑋𝑡−3 + ⋯
 ∞

𝑍𝑡 = −𝛽 𝑛 𝑋𝑡−𝑛
𝑛=0

In order to make sure that the sum on the right is

convergent (in the mean-square sense), we need 𝛽 < 1.

There is an optional reading titled “Mean-square

convergence” where we explain this result.
 Invertibility - Definition
Definition:

𝑋𝑡 is a stochastic process.

𝑍𝑡 is innovations, i.e., random disturbances or white noise.

∞
𝑋𝑡 is called invertible, if 𝑍𝑡 = 𝑘=0 𝜋𝑘 𝑋𝑡−𝑘 where
∞
𝑘=0 𝜋𝑘 is convergent.
Model 1 vs Model 2
• Model 1 is not invertible since
∞ ∞

|𝜋𝑘 | = 2𝑘 , 𝐷𝑖𝑣𝑒𝑟𝑔𝑒𝑛𝑡
𝑘=0 𝑘=0

• Model 2 is invertible since

∞ ∞
1
|𝜋𝑘 | = 𝑘
, 𝐺𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐 𝑆𝑒𝑟𝑖𝑒𝑠, 𝐶𝑜𝑛𝑣𝑒𝑟𝑔𝑒𝑛𝑡
2
𝑘=0 𝑘=0
 Model choice

• For ‘invertibility’ to hold, we choose Model 2, since

1
< 1.
2

• This way, ACF uniquely determines the MA process.

 What We’ve Learned
• Definition of invertibility of a stochastic process

• Invertibility condition guarantees unique MA

process corresponding to observed ACF
Invertibility and stationarity
conditions
Practical Time Series Analysis
Thistleton and Sadigov
 Objectives

• Articulate invertibility condition for MA(q) processes

• Discover stationarity condition for AR(p) processes

• Relate MA and AR processes through duality

 MA(q) process

𝑋𝑡 = 𝛽0 𝑍𝑡 + 𝛽1 𝑍𝑡−1 + ⋯ 𝛽𝑞 𝑍𝑡−𝑞

Using Backward shift operator,

𝑋𝑡 = 𝛽0 + 𝛽1 𝐵 + ⋯ + 𝛽𝑞 𝐵𝑞 𝑍𝑡 = 𝛽 𝐵 𝑍𝑡

We obtain innovations 𝑍𝑡 in terms of present and past values of 𝑋𝑡 ,

−1
𝑍𝑡 = 𝛽 𝐵 𝑋𝑡 = 𝛼0 + 𝛼1 𝐵 + 𝛼2 𝐵2 + ⋯ 𝑋𝑡

For this to hold, “complex roots of the polynomial 𝛽(𝐵) must lie
outside of the unit circle where 𝐵 is regarded as complex variable”.
 Invertibility condition for MA(q)
MA(q) process is invertible if the roots of the polynomial

𝛽 𝐵 = 𝛽0 + 𝛽1 𝐵 + ⋯ + 𝛽𝑞 𝐵𝑞

all lie outside the unit circle, where we regard 𝐵 as a

complex variable (not an operator).

(Proof is done using mean-square convergence, see optional

reading)
 EX: MA(1) process
• 𝑋𝑡 = 𝑍𝑡 + 𝛽𝑍𝑡−1

• 𝛽 𝐵 = 1 + 𝛽𝐵

1
• In this case only one (real) root 𝐵 = −
𝛽
1
• − > 1 ⇒ 𝛽 < 1.
𝛽

∞ 𝑘 𝐵𝑘 ∞ 𝑘𝑋
• Then, 𝑍𝑡 = 𝑘=0 (−𝛽) 𝑋𝑡 = 𝑘=0 (−𝛽) 𝑡−𝑘
Example – MA(2) process
5 1
𝑋𝑡 = 𝑍𝑡 + 𝑍𝑡−1 + 𝑍𝑡−2
6 6
Then,

𝑋𝑡 = 𝛽 𝐵 𝑍𝑡

Where
5 1 2
𝛽 𝐵 =1+ 𝐵+ 𝐵
6 6
Example cont.
5 1 2
1+ 𝑧+ 𝑧 =0
6 6

𝑧1 = 2, 𝑧2 = 3
Example cont.
−1 1 3 2
𝛽 𝐵 = 5 1 = 1 − 1
1+ 𝐵+ 𝐵2 1+ 𝐵
2
1+ 𝐵
3
6 6

∞ 𝑘 𝑘
−1
1 1
𝛽 𝐵 = 3 − −2 − 𝐵𝑘
2 3
𝑘=0
 ∞ 𝑘 𝑘
1 1
𝑍𝑡 = 3 − −2 − 𝐵𝑘 𝑋𝑡
2 3
𝑘=0
∞ ∞

𝑍𝑡 = 𝜋𝑘 𝐵𝑘 𝑋𝑡 = 𝜋𝑘 𝑋𝑡−𝑘
𝑘=1 𝑘=1

Where
𝑘 𝑘
1 1
𝜋𝑘 = 3 − −2 −
2 3
MA(2) process ⟹ AR(∞) process
 Stationarity condition for AR(p)
AR(p) process
𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ + 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

is (weakly) stationary if the roots of the polynomial

𝜙 𝐵 = 1 − 𝜙1 𝐵 − 𝜙2 𝐵2 − ⋯ − 𝜙𝑝 𝐵𝑝 .

all lie outside the unit circle, where we regard 𝐵 as a

complex variable (not an operator).
AR(1) process
𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝑍𝑡 ⟹ 1 − 𝜙1 𝐵 𝑋𝑡 = 𝑍𝑡

𝜙 𝐵 = 1 − 𝜙1 𝐵

1
𝜙 𝑧 = 1 − 𝜙1 𝑧 = 0 ⟹ 𝑧 =
𝜙1
 1
𝑧 = > 1 ⇒ 𝜙1 < 1
𝜙1
Thus, when 𝜙1 < 1, the AR(1) process is stationary.

1
𝑋𝑡 = 𝑍𝑡 = 1 + 𝜙1 𝐵 + 𝜙1 𝐵2 − ⋯ 𝑍𝑡
∞
1 − 𝜙1 𝐵
= 𝜙1𝑘 𝑍𝑡−𝑘
𝑘=0
Another look at 𝜙1
Take Variance from both side,
∞ ∞ ∞

𝑉𝑎𝑟 𝑋𝑡 = 𝑉𝑎𝑟 𝜙1𝑘 𝑍𝑡−𝑘 = 𝜙12𝑘 𝜎𝑍2 = 𝜎𝑍2 𝜙12𝑘

𝑘=0 𝑘=0 𝑘=0

which is a convergent geometric series if 𝜙12 < 1, i.e.,

𝜙1 < 1.
AR(𝑝) process ⟹ MA(∞) process
 Duality between AR and MA processes

Under invertibility condition of MA(q),

MA(q) ⟹ AR(∞)

Under stationarity condition of AR(p)

AR(p) ⟹ MA(∞)
What We’ve Learned

• Invertibility condition for MA(q) processes

• Stationarity condition for AR(p) processes
• Duality MA and AR processes
Mean Square Convergence
Practical Time Series Analysis
Thistleton and Sadigov
 Objectives

• Learn mean-square convergence

• Formulate necessary and sufficient condition for

invertibility of MA(1) process
 Mean-square convergence
Let
𝑋1 , 𝑋2 , 𝑋3 , …

be a sequence of random variables (i.e. a stochastic process).

We say 𝑋𝑛 converge to a random variable 𝑋 in the mean-square

sense
if

2
𝐸 𝑋𝑛 − 𝑋 → 0 𝑎𝑠 𝑛 → ∞
MA(1) model
We inverted MA(1) model
𝑋𝑡 = 𝑍𝑡 + 𝛽𝑍𝑡−1

as
∞

𝑍𝑡 = −𝛽 𝑘 𝑋𝑡−𝑘
𝑘=0

Infinite sum above is convergent in mean-square sense under some

condition on 𝛽.
Auto covariance function

0, 𝑘>1
𝛽𝜎𝑍2 , 𝑘=1
𝛾 𝑘 =
(1 + 𝛽2 )𝜎𝑍2 , 𝑘=0
𝛾 −𝑘 , 𝑘<0
 Series convergence
Lets find 𝛽′ 𝑠 that partial sum

−𝛽 𝑘 𝑋𝑡−𝑘
𝑘=0

converges to 𝑍𝑡 in mean-square sense.

 𝑛 2 𝑛 2 𝑛

𝐸 −𝛽 𝑘 𝑋𝑡−𝑘 − 𝑍𝑡 =𝐸 −𝛽 𝑘 𝑋𝑡−𝑘 − 2𝐸 −𝛽 𝑘 𝑋𝑡−𝑘 𝑍𝑡 + 𝐸 𝑍𝑡2

𝑘=0 𝑘=0 𝑘=0
𝑛 𝑛−1
2
=𝐸 𝛽 2𝑘 𝑋𝑡−𝑘 + 2𝐸 −𝛽 2𝑘+1
𝑋𝑡−𝑘 𝑋𝑡−𝑘+1 − 2𝐸 𝑋𝑡 𝑍𝑡 + 𝜎𝑍2
𝑘=0 𝑘=0
𝑛 𝑛−1
2
= 𝛽 2𝑘 𝐸 𝑋𝑡−𝑘 −2 𝛽 2𝑘+1 𝐸 𝑋𝑡−𝑘 𝑋𝑡−𝑘+1 − 2𝐸 [𝑍𝑡2
𝑘=0 𝑘=0
To get

𝑛
𝐸 𝑘=0 −𝛽 𝑘 𝑋𝑡−𝑘 −
i.e.,
1
− >1
𝛽

i.e., zero of the polynomial

𝛽 𝐵 = 1 + 𝛽𝐵

Lies outside of the unit circle.

 What We’ve Learned
• Definition of the mean square convergence

• Necessary and sufficient condition for invertibility

of MA(1) process
Difference equations
Practical Time Series Analysis
Thistleton and Sadigov
Objectives

• Recall and solve difference equations

Difference equation
• General term of a sequence is given, ex: 𝑎𝑛 = 2𝑛 + 1. So,

3, 5, 7, …

• General term not given, but a relation is given, ex:

𝑎𝑛 = 5𝑎𝑛−1 − 6𝑎𝑛−2

• This is a difference equation (recursive relation)

 How to solve difference equations?
• We look for a solution in the format

𝑎𝑛 = 𝜆𝑛

• For the previous problem,

𝜆𝑛 = 5𝜆𝑛−1 − 6𝜆𝑛−2

We simplify

𝜆2 − 5𝜆 + 6 = 0

• Auxiliary equation or characteristic equation.

• 𝜆 = 2, 𝜆 = 3
• 𝑎𝑛 = 𝑐1 2𝑛 + 𝑐2 3𝑛
• With some initial conditions, say 𝑎0 = 3, 𝑎1 = 8.

We get
𝑐1 + 𝑐2 = 3
2𝑐1 + 3𝑐2 = 8
Thus,

𝑐1 = 1, 𝑐2 = 2.
 Solution
𝑎𝑛 = 2𝑛 + 2 ∙ 3𝑛

Is the solution of 2nd order difference equation

𝑎𝑛 = 5𝑎𝑛−1 − 6𝑎𝑛−2
 𝑘-th order difference equation
𝑎𝑛 = 𝛽1 𝑎𝑛−1 + 𝛽2 𝑎𝑛−2 + ⋯ + 𝛽𝑘 𝑎𝑛−𝑘

Its characteristic equation

𝜆𝑘 − 𝛽1 𝜆𝑘−1 − ⋯ − 𝛽𝑘−1 𝜆 − 𝛽𝑘 = 0

Then we look for the solutions of the characteristic equation. Say,

all k solutions are distinct real numbers, 𝜆1 , 𝜆2 , … , 𝜆𝑘 , then

𝑎𝑛 = 𝑐1 𝜆1𝑛 + 𝑐2 𝜆𝑛2 + ⋯ + 𝑐𝑛 𝜆𝑛𝑘

Coefficients 𝑐𝑗′ 𝑠 are determined using initial values.

 Example - Fibonacci sequence

Fibonacci sequence is defined as follows:

1, 1, 2, 3, 5, 8, 13, 21, …

i.e., every term starting from the 3rd term is addition of the
previous two terms.

Question: What is the general term, 𝑎𝑛 , of the Fibonacci

sequence?
Formulation
∞
We are looking for a sequence an n=0 , such that

𝑎𝑛 = 𝑎𝑛−1 + 𝑎𝑛−2

where 𝑎0 = 1, 𝑎1 = 1.

Characteristic equation becomes

𝜆2 − 𝜆 − 1 = 0
 1− 5 1+ 5
Then 𝜆1 = and 𝜆2 = .
2 2

Thus
𝑛 𝑛
1− 5 1+ 5
𝑎𝑛 = 𝑐1 + 𝑐2
2 2

Use initial data

𝑐1 + 𝑐2 = 1

1− 5 1+ 5
𝑐1 + 𝑐2 =1
2 2
 General term of Fibonacci sequence

We obtain

5− 5 1 1− 5
𝑐1 = =−
10 5 2

5+ 5 1 1+ 5
𝑐2 = =
10 5 2

𝑛+1 𝑛+1
1 1− 5 1 1+ 5
𝑎𝑛 = − +
Relation to differential equations
𝑘 −th order linear ordinary equation

𝑦 𝑘 = 𝛽1 𝑦 𝑘−1 + ⋯ 𝛽𝑘−1 𝑦 + 𝛽𝑘

Solution format 𝑦 = 𝑒 𝜆𝑡 gives characteristic equation

𝜆𝑘 − 𝛽1 𝜆𝑘−1 − ⋯ − 𝛽𝑘−1 𝜆 − 𝛽𝑘 = 0

Then we solve the characteristic equation.

What We’ve Learned

• Definition of difference equations and how to solve

them
Yule-Walker Equations
Practical Time Series Analysis
Thistleton and Sadigov
Objectives
• Introduce Yule – Walker equations

• Obtain ACF of AR processes using Yule – Walker equations

 Procedure
• We assume stationarity in advance (a priori assumption)
• Take product of the AR model with 𝑋𝑛−𝑘
• Take expectation of both sides
• Use the definition of covariance, and divide by 𝛾 0 = 𝜎𝑋2
• Get difference equation for 𝜌(𝑘), ACF of the process
• This set of equations is called Yule-Walker equations
• Solve the difference equation
 Example
We have an AR(2) process
1 1
𝑋𝑡 = 𝑋𝑡−1 + 𝑋𝑡−2 + 𝑍𝑡 … (∗)
3 2

Polynomial

1 1 2
𝜙 𝐵 =1− 𝐵− 𝐵
3 2

−2 ± 76
has real roots both of which has magnitude greater than 1, so
6
roots are outside of the unit circle in ℝ2 . Thus, this AR(2) process is a
stationary process.
 Example cont.
Note that if 𝐸 𝑋𝑡 = 𝜇, then

1 1
𝐸(𝑋𝑡 ) = 𝐸(𝑋𝑡−1 ) + 𝐸(𝑋𝑡−2 ) + 𝐸(𝑍𝑡 )
3 2
1 1
𝜇= 𝜇+ 𝜇
3 2
𝜇=0
Multiply both side of ∗ with 𝑋𝑡−𝑘 , and take expectation

1 1
𝐸 𝑋𝑡−𝑘 𝑋𝑡 = 𝐸 𝑋𝑡−𝑘 𝑋𝑡−1 + 𝐸 𝑋𝑡−𝑘 𝑋𝑡−2 + 𝐸(𝑋𝑡−𝑘 𝑍𝑡 )
3 2
 Example cont.
Since 𝜇 = 0, and assume 𝐸 𝑋𝑡−𝑘 𝑍𝑡 = 0,

1 1
𝛾 −𝑘 = 𝛾 −𝑘 + 1 + 𝛾(−𝑘 + 2)
3 2
Since 𝛾 𝑘 = 𝛾 −𝑘 for any 𝑘,

1 1
𝛾 𝑘 = 𝛾 𝑘 − 1 + 𝛾(𝑘 − 2)
3 2
Divide by 𝛾 0 = 𝜎𝑋2

1 1
𝜌 𝑘 = 𝜌 𝑘 − 1 + 𝜌(𝑘 − 2)
3 2

This set of equations is called Yule-Walker equations.

Solve the difference equation
We look for a solution in the format of 𝜌 𝑘 = 𝜆𝑘 .

2
1 1
𝜆 − 𝜆− =0
3 2

2+ 76 2− 76
Roots are 𝜆1 = and 𝜆2 = , thus
12 12

𝑘 𝑘
2 + 76 2 − 76
𝜌 𝑘 = 𝑐1 + 𝑐2
12 12
 Finding 𝑐1 , 𝑐2
Use constraints to obtain coefficients

𝜌 0 = 1 ⇒ 𝑐1 + 𝑐2 = 1

And for 𝑘 = 𝑝 − 1 = 2 − 1 = 1,

𝜌 𝑘 = 𝜌 −𝑘

Thus,
1 1 2 2 + 76 2 − 76 2
𝜌 1 = 𝜌 0 + 𝜌 −1 ⇒ 𝜌 1 = ⇒ 𝑐1 + 𝑐2 =
3 2 3 12 12 3
Solve the system for 𝑐1 , 𝑐2
𝑐1 + 𝑐2 = 1
2 + 76 2 − 76 2
𝑐1 + 𝑐2 =
12 12 3

Then,

4+ 6 4− 6
𝑐1 = and 𝑐2 =
8 8
ACF of the AR(2) model
For any 𝑘 ≥ 0,
𝑘 𝑘
4 + 6 2 + 76 4 − 6 2 − 76
𝜌 𝑘 = +
8 12 8 12

And
𝜌 𝑘 = 𝜌 −𝑘
Simulation
 What We’ve Learned

• Yule- Walker equations is set of difference equations

governing ACF of the underlying AR process

• How to find the ACF of an AR process using Yule-

Walker equations