ARIMA & SARIMA

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Agenda
▪ Business Problem

▪ ARIMA Models for Non - Stationary Time Series

▪ Forecasting ARIMA Models

[email protected]
Y21IHWS8GO
▪ Seasonal ARIMA Models

▪ Forecasting SARIMA Models

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
[email protected]
ARIMA
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Business problem: predict the amount of Carbon Dioxide (in
parts per million)
- It is important for nation to develop models that accurately forecast CO2 emission and
[email protected]
can take an primitive measures accordingly.
Y21IHWS8GO

- This model forecast can be used to create premium tables that can assist/guide the nation
to control the emission.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Dependent Variable
▪ The variable we wish to explain or predict

▪ Usually denoted by Y

[email protected]
Y21IHWS8GO
▪ Dependent Variable = Response Variable = Target
Variable

▪ Here ‘CO2 ppm’ is our target variable

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Independent Variable
▪ The variables used to explain the dependent variable

▪ Usually denoted by X

[email protected]
Y21IHWS8GO
▪ Independent Variable = Predictor Variable

▪ In our example, Year-Month are the independent

variables

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
[email protected]
Y21IHWS8GO Visiting Basics

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Variable that contributes to Carbon Dioxide (in parts
per million)
[email protected]
Y21IHWS8GO Year-Month
CO2 ppm
(Independent
(Target Variable)
Variable)

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Data
Let us consider the following data.

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA Models for Non-stationary Time Series
• Stationary Process :-

- Autoregressive Process: AR ( p )
- Moving Average Process: MA ( q )
- ARMA ( p, q )
[email protected]
Y21IHWS8GO

• Integrated Non-stationary Process :-

- ARIMA ( p, d, q )

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Decomposition of Time-Series Data

[email protected]
Y21IHWS8GO

- We can infer that the existence of trend and seasonality in the data, hence
stationary data doesn’t have trend, seasonality.
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Stationary test - (Dickey-Fuller Test)

[email protected]
Y21IHWS8GO

- We can infer that p-value greater than 0.5, hence null hypothesis rejected the data is
non stationary. This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA Model
• ARIMA :- autoregressive integrated moving average. it is an a way of modelling time series
data for forecasting or predicting future points in a series.

• ARIMA models consists of three components in it :-

- AR model
[email protected]
Y21IHWS8GO

- Integrated component

- MA model

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA Model
● A pattern of growth/decline in the data is accounted for (hence the ‘autoregressive’/’AR’
part).
● The rate of change of the growth/decline in the data is accounted for (hence the
“integrated”/’I’ part)
● The noise between the consecutive time points is accounted for (hence the “moving
[email protected]
Y21IHWS8GO
average” part)

Note :- time series data is an data that is made up of a sequence of data points taken at
successive equally spaced points in time.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA Model
• Few Key notes to be known about ARIMA models:

- The ARIMA model is denoted ARIMA as ( p , d , q ).

- p : order of AR model.
[email protected]
Y21IHWS8GO
- d : times to difference the data.

- q : order of MA model.

- p, d, and q are non-negative integers.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
● Differencing is an non-stationary time series data one or more times that can convert it
into stationary. Hence the integrated ‘ I ‘ is an component of ARIMA.

● d is an number of times to perform an lag-1 to the difference on data.

- d = 0: no differencing
[email protected]
- d = 1: difference of once
Y21IHWS8GO

- d = 2: difference of twice

𝑌𝑖 = 𝑍𝑖 − 𝑍𝑖−1

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
The second component of ARIMA model, where I as ‘integration’ , is used to replace the series
with the difference between their current values and the previous values and this differencing
process can be performed more than once as per the requirement .

● Equation of first order differencing is : 𝒚𝒕 = 𝒚𝒕 − 𝒚𝒕-1

[email protected]
Y21IHWS8GO

● Hence, for 𝒚𝒕 =2 and 𝒚𝒕-1= 1 ; 𝒚𝒕 will be 1

● As same, second order differencing , 𝒚𝒕 = (𝒚𝒕 − 𝒚𝒕-1) −(𝒚𝒕-1− 𝒚𝒕-2)

● Order of this component (order of differencing) is applied by parameter d while fitting a

model : ARIMA ( p, d, q )

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
● Differencing (lag difference) is the process of transforming a time series to stabilize the
mean.

● We have several ways to identify the time series, such as a line plot, which represents
the series over time.
[email protected]
Y21IHWS8GO
● Within these trends, seasonality and random walk can be observed, which is a change
over a period of time, and this behavior is considered a nonstationary time series.

● The clear rule states that it needs to remove trends and seasonality in the data from the
training time series forecasting model.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
The following are the reasons for differencing:

● To convert non-stationary data into a stationary time series.

● To remove seasonal trends.
[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
FIRST-ORDER DIFFERENCING (TREND DIFFERENCING):
● Change between two consecutive observations in the time series can be written as
follows:
Y’t = Yt - Yt-1
● When the differenced series contains white noise (εt), the formula can be written as
follows:
[email protected]
Y21IHWS8GO
Yt - Yt-1 = εt
Yt = Yt-1 + εt
where εt = white noise. This is known as a normal random walk .
● A random walk represents that a time series is nonstationary.
● This is mostly seen in financial, economic, and microeconomics data. A random walk has
mostly long durations of trend ups and down and uncertain and unpredictable changes.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
FIRST-ORDER DIFFERENCING (TREND DIFFERENCING):
● For instance, the stock price of some XYZ Company has gone up and down for
the last six months. This is random walk behavior. Any future moment is not
predictable because of the haphazard ups and downs.
● So, here it is clear that the random time series has nonzero mean values. In that
[email protected]
Y21IHWS8GO
case, the final formula is known as random walk with drift, as shown here:
Yt = c + Yt-1 + εt

where c is the average of change between two observations.

● If c is positive, then the average change will rise in Yt, and Yt will move upward.
● If c is negative, the Yt value will move downward.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing

• Few Key notes to be known about Differencing:

○ First-order differencing in a time series will remove a linear trend (i.e.,

differences=1).
[email protected]
Y21IHWS8GO
○ Second-order differencing will remove a quadratic trend (i.e., differences=2).

○ In addition, first-order differencing in a time series at a lag equal to the period

will remove a seasonal trend.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ARIMA (d) - Differencing
SECOND-ORDER DIFFERENCING (TREND DIFFERENCING)

● Second-order differencing is a technique used to make first-order differencing data

stationary when the first-order differencing has failed.
[email protected]
Y21IHWS8GO
● So, it’s necessary to apply second-order differencing to obtain a stationary series.
Y”t = Y’t – Y’t-1
= (Yt - Yt-1) – (Yt-1 - Yt-2)
= Yt - 2Yt-1 + Yt-2

● This is second-order differencing, Y”, which has t-2 values. This is known as double
changes in the original series.
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Distinguish p, d, q values
● The Values of p, q are determine based on the auto-correlation (ACF) and partial
autocorrelation (PACF) plots and values of d depends on the level of stationarity in data.

● Where, as in PACF plot the number of peaks indicates the order of the autoregression/AR
(value of p in ARIMA( p, d, q )).
[email protected]
Y21IHWS8GO

● As we can see in the right figure (Next slide), As there is an one peak falling out of range,
hence, the order of AR , i.e. value of p would be 1.

● As per the ACF plot, number of peaks indicates the order of the moving average (value
of q in ARIMA ( p, d, q )) .

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Distinguish p, d, q values
• Where, as in p, d and q parameters in ARIMA (p , d , q) are substituted with
integer values where p and q take any values between 0 to 5 and value of d is set
between 0 to 2

• For example, ARIMA(2,1,1) means that you have a second order autoregressive
[email protected]
Y21IHWS8GO
model with a first order moving average component and series has been
differenced once to induce stationarity

• A value of 0 can be used for any of the above mentioned parameters indicating
that particular component (AR/ I/ MA) should not be used. This way, the ARIMA
model can be configured to perform the function of an ARMA model, and even a
simple AR, I, or MA model depending on the data
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Distinguish p, d, q values

● We can observe that in the left figure there was five raised peaks falling out of range,
hence, the order of MA i.e. value of q would be 5.

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Parameter combinations of p, d, q values of the
Model

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Akaike Information Criterion (AIC)
• The best fit model is selected based on Akaike Information Criterion (AIC) , and
Bayesian Information Criterion (BIC) values. The idea is to choose a model with
minimum AIC and BIC values.

• AIC is an effort to balance the model between goodness-of-fit and number of

[email protected]
parameters used in the model, This is similar to the balancing act between income
Y21IHWS8GO
and cost of a company so that the debs of the company is optimized (Debt = Cost -
Income).

• As a modeler, we care about the maximum goodness of fit (income) with the

minimum number of parameters (cost).

AIC=2K−2ln(L)

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Akaike Information Criterion (AIC)
• For the given model, L in the above formula is the maximized value of the likelihood
function representing goodness-of-fit, and K the number of estimated parameters.
Like our debts, we want to keep AIC value at the minimum to choose the best
possible model.

• Bayesian Information Criterion (BIC) is another variant of AIC and is used for the
[email protected]
Y21IHWS8GO
same purpose of best fit model selection. For the best possible model selection, we
want to look at AIC, BIC, and AICc (AIC with sample correction) if all these values
are minimum for a given model.

• With increasing parameters K will increase and hence AIC increases. While with the
goodness of the fit L increases thus decreasing AIC.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Best parameters are selected in accordance with
the lowest Akaike Information Criteria (AIC)

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Best parameters are selected in accordance with
the lowest Akaike Information Criteria (AIC)

- We can interpret that the

parameters (p,d,q) (2,1,3) are the
[email protected]
Y21IHWS8GO optimal with the lowest AIC to

treat the non stationary data and

predict the time-series data.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
[email protected]
Y21IHWS8GO Model Evaluation

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution
Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Limitations of ARIMA
● For an ARIMA model, we can see the predictions(previous slide) with 95%
confidence interval bands. The seasonality was unable to be captured. Let us try
out a SARIMA model.

● Seasonality in a time series data can be captured with SARIMA Model.

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Summary of ARIMA
● ARIMA is a method among several used for forecasting univariate variables, which
uses information obtained from the variable itself to predict its trend. The variables
are regressed on its own past values.

● AR(p) is where p equals the order of autocorrelation (designates weighted moving

[email protected]
Y21IHWS8GO
average over past observations) z I (d), where d is the order of integration
(differencing), which indicates linear trend or polynomial trend z.

● MA(q) is where q equals the order of moving averages (designates weighted

moving average over past errors).

● ARIMA is made up of two models: AR and MA.

This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
[email protected]
Y21IHWS8GO
SARIMA

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 SARIMA Model
• In general the economic, agricultural and geophysical time series have cycle
components within a specific time period.
• The smallest time period for this repetitive phenomenon is called a seasonal period
(s).
• For example,
[email protected]
Y21IHWS8GOmonthly temperature have a 12-month cycle, s = 12.
the quarterly ice cream sales have a 4-quarterly cycle, s = 4.
It may be useful to use a s-fold difference operator
with s = 4 to remove the cycle component from quarterly data,
s = 12 to remove annual fluctuations from monthly data.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 SARIMA Model
• The ARIMA models can be extended to handle seasonal components of a data
series.

• The multiplicative seasonal autoregressive moving average model, SARIMA (p, d,

q)(P, D, Q)s is given by where { }is Gaussian white noise is ordinary autoregressive
and moving average components; and are seasonal autoregressive and moving
[email protected]
Y21IHWS8GO
average components, respectively, and are the ordinary and seasonal difference
component of order d and D.

• ARIMA(p, d, q)x(P, D, Q)[freq]

• Seasonal difference = D
• Appropriate for seasonal series

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Parameter combinations of ( p, d, q ) x
(P, D, Q)[frequency/s] values of the Model

[email protected]
Y21IHWS8GO

Hence, as per the business problem we have a data for an year-wise split which composed
of months .so, our frequency or (s) is 12
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Best parameters are selected in accordance with
the lowest Akaike Information Criteria (AIC)

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Best parameters are selected in accordance with
the lowest Akaike Information Criteria (AIC)

- We can interpret that the

[email protected]
Y21IHWS8GO parameters (p,d,q)x(P,D,Q)[s]
(1,1,0) (1,0,2,12) are the optimal
with the lowest AIC to treat the
non stationary data and predict
the time-series data.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
[email protected]
Y21IHWS8GO Model Evaluation

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation

Model RMSE MAPE

ARIMA(2,1,3) 1.17 0.28

[email protected](1,1,0)(0,1,2,12) 0.81 0.18
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation
● From an SARIMA model, we can see the predictions. The seasonality was been
captured.

● The evaluation metric for Autoregression models are MAPE.

● Hence, the data consist of seasonality we are able to secure the least/optimal
[email protected]
Y21IHWS8GO
MAPE in SARIMA model.

● It’s been evident accordingly such a way when to be used ARIMA ,SARIMA
Models.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Actual forecast

[email protected]
Y21IHWS8GO

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Summary
● Based on accuracy choose the model which works best

● Must have proper interpretability

[email protected]
Y21IHWS8GO

● Often a simple model works better

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Further study
● Property sale data ● Stock price data

Year Qtr Sale Home Loan Interest Rate Date Open High Low Close Volume
2019 Q1 2049 8.55 10-01-2015 49.5 49.7 47.43 47.98 4572964
2019 Q2 1842 8.55
10-02-2015 47.12 49.54 46.99 49.51 4423982
[email protected]
2019
Y21IHWS8GO Q3 1769 8.5
10-05-2015 49.77 49.97 48.83 49.23 3689865
2019 Q4 1880 8.35
10-06-2015 48 48.61 47.12 48.29 5235897
2020 Q1 1760 8.2
2020 Q2 2041 8.05 10-07-2015 47.33 47.54 45.82 46.39 6813959

2020 Q3 1920 7.9 10-08-2015 46.02 46.14 44.26 45.34 6133216

2020 Q4 1742 7.75 10-09-2015 44.19 44.87 43.67 44.14 6158370
2021 Q1 to be forecast 7.6
10-12-2015 44.6 44.6 43.05 43.12 3836303

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Appendix
Akaike's Information Criterion:-

Akaike's information criterion (AIC) is known in the statistics trade as a penalized

log-likelihood. If you have a model for which a log-likelihood value can be obtained, then

[email protected]
Y21IHWS8GO
where p is the number of parameters in the model, and 1 is added for the estimated
variance (you could call this another parameter if you wanted to). To demystify AIC let's
calculate it by hand. We revisit the regression data for which we calculated the
log-likelihood.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Appendix
Akaike's Information Criterion:-
attach(regression)
names(regression)
[1] "speed" "time"
growth
[email protected]
Y21IHWS8GO
[1] 12 10 8 11 6 7 2 3 3
The are nine values of the response variable, growth, and we calculated the log-likelihood
as −23.98941 earlier. There was only one parameter estimated from the data for these
calculations (the mean value of y), so p = 1. This means that AIC should be

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Appendix
Akaike's Information Criterion:-
Fortunately, we do not need to carry out these calculations, because there is a built-in
function for calculating AIC. It takes a model object as its argument, so we need to fit a
one-parameter model to the speed data like this:
model<-lm(speed~1)
[email protected]
Y21IHWS8GO

Then we can get the AIC directly:

AIC(model)
[1] 51.97882

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Appendix
AIC AS A MEASURE OF THE FIT OF A MODEL:
● The more parameters that there are in the model, the better the fit. You could obtain a
perfect fit if you had a separate parameter for every data point, but this model would
have absolutely no explanatory power.
● There is always going to be a trade-off between the goodness of fit and the number of
parameters required by parsimony. AIC is useful because it explicitly penalizes any
[email protected]
Y21IHWS8GO
superfluous parameters in the model, by adding 2(p + 1) to the deviance.
● When comparing two models, the smaller the AIC, the better the fit. This is the basis
of automated model simplification using step.
● You can use the function AIC to compare two models, in exactly the same way as you
can use anova. Here we develop an analysis of covariance.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Appendix
AIC AS A MEASURE OF THE FIT OF A MODEL:

model.1<-lm(Car ~ Grazing*Root)
model.2<-lm(Car ~ Grazing+Root)
AIC(model.1, model.2)
[email protected]
Y21IHWS8GO
df AIC

model.1 5 273.0135
model.2 4 271.1279

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Appendix
AIC AS A MEASURE OF THE FIT OF A MODEL:
● Because model.2 has the lower AIC, we prefer it to model. l. The log-likelihood
was penalized by 2 × (4 + 1) = 10 in model 1 because that model contained 4
parameters (2 slopes and 2 intercepts) and by 2 × (3 + 1) = 8 in model.2 because
that model had 3 parameters (two intercepts and a common slope).
[email protected]
Y21IHWS8GO
● You can see where the two values of AIC come from by calculation:
-2*logLik(model.1)+2*(4+1)
[1] 273.0135
-2*logLik(model.2)+2*(3+1)
[1] 271.1279

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
 References
Aitkin, M., Anderson, D., Francis, B. and Hinde, J. (1989) Statistical Modelling in GlIM.
Oxford: Clarendon Press.
Atkinson, A.C. (1985) Plots, Transformations, and Regression. Oxford: Clarendon Press.
Box, G.E.P. and Jenkins, G.M. (1976) Time Series Analysis: Forecasting and Control.
[email protected]
Oakland, CA: Holden-Day.
Y21IHWS8GO

Hicks, C.R. (1973) Fundamental Concepts in the Design of Experiments. New York, Holt:
Rinehart and Winston.
Johnson, N.L. and Kotz, S. (1970) Continuous Univariate Distributions. Volume 2. New
York: John Wiley.
Priestley, M.B. (1981) Spectral Analysis and Time Series. London: Academic Press.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.