Nonlinear Time Series Models

Time series can have many patterns. These include trends, seasonality, cycles, and irregularity. When analyzing time series data, it's crucial to detect these patterns. You must also understand their possible causes and relationships. You must also know which algorithms can model and forecast each pattern. A trend behaviour can be linear or nonlinear. A linear trend refers to a consistent upward or downward movement in the data over a period of time. A Nonlinear trend in time series data refers to a pattern of change that deviates from a straight line. In this article, we will discuss the same.

What are Non Linear Time Series?

Nonlinear time series models are indispensable for analyzing and predicting data where the relationship between variables is not linear. These models adeptly capture intricate patterns and dependencies in time series data, making them the ideal choice for various real-world phenomena where linear models are insufficient.

Non-linear time series models are used to analyze and predict data where the relationship between variables is not linear. These models capture more complex patterns and dependencies in time series data, making them suitable for various real-world phenomena where linear models fall short.

Key Concepts of Nonlinear Time Series

• Non-linearity: Non-linear time series models are used to capture intricate relationships in time series data that linear models are unable to capture. These models are essential for accurately representing and predicting behaviors in data where changes are not proportional to the inputs. In this discussion, we will explore common types of non-linear time series models, provide a detailed example with code, and visualize the results. The common types of non-linear time series models include Threshold Autoregressive (TAR) Models, Smooth Transition Autoregressive (STAR) Models, Autoregressive Conditional Heteroskedasticity (ARCH) and Generalized ARCH (GARCH) Models, Markov Switching Models, and Neural Network Models.
• Deterministic vs. Stochastic Non-linearity: The connection between variables is deterministic and can be defined by mathematical functions such as polynomials or exponentials. Deterministic non-linearity enables the capture of complex, non-linear relationships that linear models cannot adequately describe. Polynomial models are a simple yet powerful way to model such non-linear relationships. By fitting a polynomial to the data, we can effectively capture and predict non-linear trends in the time series. Examples of Deterministic Non-linear Models is Threshold Models, Polynomial Models, Exponential Models, Logistic Models, Smooth Transition Models.
• Stochastic Non-linearity: Stochastic non-linearity in time series models refers to situations in which the relationship between variables is non-linear and involves randomness. Unlike deterministic non-linear models, which have predictable outcomes based solely on past values, stochastic non-linear models incorporate random components. This makes future values inherently uncertain, even if the past values are known. Examples of Stochastic Non-linear Models is Autoregressive Conditional Heteroskedasticity (ARCH) and Generalized ARCH (GARCH) Models, Smooth Transition Autoregressive (STAR) Models, Markov Switching Models, Non-linear Moving Average (NMA) Models.
• Stationarity: "Stationarity" refers to the property where the statistical characteristics of a time series remain constant over time. This means that the mean, variance, and autocorrelation structure of the time series do not change. Non-linear time series models can be used to account for non-linear relationships while maintaining stationarity. Examples of stationary non-linear time series models include Autoregressive Conditional Heteroskedasticity (ARCH) and Generalized ARCH (GARCH) Models, Threshold Autoregressive (TAR) Models, Smooth Transition Autoregressive (STAR) Models, Markov Switching Models, and Non-linear Moving Average (NMA) Models.

Stationary non-linear time series models, such as the TAR model, are powerful tools for capturing complex relationships while maintaining stationarity. These models are suitable for data that exhibits non-linear behavior without long-term trends or changing variance. By understanding and applying these models, analysts can effectively model and predict time series with intricate, non-linear dynamics.

Types of Non-linear Time Series Models

1. Threshold Autoregressive (TAR) Models:

Threshold Autoregressive (TAR) models are a type of non-linear time series model. These models switch between different regimes or behaviors based on the value of an observed variable relative to certain thresholds. This approach allows the model to capture non-linear relationships by dividing the data into different regimes and fitting a separate autoregressive model to each regime. The TAR package in R provides Bayesian modeling of autoregressive threshold time series models. It identifies the number of regimes, thresholds, and autoregressive orders, as well as estimates remaining parameters.

It consists of two parts: one for observations below the threshold and another for observations above the threshold.

The two-part TAR model is given by the following formula:

• For observations below the threshold:

Y_t = \phi_{1,0} + \phi_{1,1}y_{t-1} + \phi_{1,2}y_{t-2} +... + \phi_{1,p}y_{t-p} + \epsilon_t, if y_{t-d} ≤ \tau

• For observations above the threshold:

Y_t = \phi_{2,0} + \phi_{2,1}y_{t-1} + \phi_{2,2}y_{t-2} + ... + \phi_{2,p}y_{t-p} + \epsilon_t, if y_{t-d} > \tau

Where:

• y_t is the observed time series at time 𝑡.
• \phi_{i,j} are the coefficients for the AR model in the 𝑖-th regime, with 𝑖 = 1,2 denoting the regimes.
• \epsilon_t is the error term.
• \tau is the threshold.
• d is the delay parameter, indicating the lag that the threshold depends on.
• p represents the order of the autoregressive process.

Estimation:

The threshold \tau can be determined by methods such as grid search, where various potential thresholds are tested, and the one that minimizes a chosen criterion (e.g., AIC or BIC) is selected.

The delay parameter d and the autoregressive coefficients \phi_{i,j} are typically estimated using standard regression techniques within each regime.

2. Autoregressive Conditional Heteroskedasticity (ARCH) and Generalized ARCH (GARCH) Models:

Autoregressive Conditional Heteroskedasticity (ARCH) and Generalized ARCH (GARCH) models are essential for modeling the conditional variance of a time series, particularly in financial econometrics. They are indispensable for capturing the volatility clustering phenomenon observed in many financial time series, where periods of high volatility are consistently followed by similar periods, and vice versa.

Autoregressive Conditional Heteroskedasticity (ARCH) Model:

Model Structure:

The ARCH(q) model specifies that the conditional variance of a time series is a function of its past squared residuals. Mathematically, it can be represented as:

\sigma_t^2 = \alpha_0 + \Sigma_{i=1}^{p}\alpha_i\varepsilon_{t-i}^{2}

Where:

• \sigma_t^2 is the conditional variance of the time series at time t.
• \alpha_0 is a constant term.
• \alpha_i are the parameters of the model.
• \varepsilon_{t-i}^{2} are the squared residuals of the time series up to lag q.

Estimation:

Estimating the parameters \alpha_i of the ARCH model involves methods such as maximum likelihood estimation (MLE). Typically, the sum of squared residuals is minimized to find the optimal parameters.

3. Generalized Autoregressive Conditional Heteroskedasticity (GARCH) Model:

Model Structure:

The GARCH(p, q) model extends the ARCH model by incorporating both autoregressive and moving average terms for the conditional variance. The GARCH(p, q) model can be represented as:

\sigma_t^2 = \alpha_0 + \Sigma_{i=1}^{p}\alpha_i\varepsilon_{t-i}^{2} + \Sigma_{j=1}^{q}\beta_j\sigma_{t-j}^{2}

Where:

• \sigma_t^2 is the conditional variance of the time series at time t.
• \alpha_i and \beta_j are the parameters of the model.
• \varepsilon_{t-i}^{2} are the squared residuals of the time series up to lag p.
• \sigma_{t-j}^{2} are the conditional variances up to lag q.

Estimation:

Estimating the parameters \alpha_i and \beta_j of the GARCH model also involves methods such as maximum likelihood estimation (MLE). The process is similar to that of the ARCH model but involves optimizing the likelihood function with respect to both sets of parameters.

Applications:

ARCH and GARCH models are widely used in financial modeling for:

• Forecasting volatility in asset returns.
• Risk management and portfolio optimization.
• Option pricing and hedging strategies.
• Evaluating the impact of news and events on financial markets.

4. Smooth Transition Autoregressive (STAR) Models:

Smooth Transition Autoregressive (STAR) models represent a type of nonlinear time series model that facilitates smooth transitions between different regimes. In contrast to Threshold Autoregressive (TAR) models, which switch abruptly between regimes, STAR models transition smoothly from one regime to another based on an underlying transition function.

Model Structure

A basic STAR model can be written as:

y_t = \phi_{1,0} + \Sigma_{i=1}^p\phi_{1,i}y_{t-i} + (\phi_{2,0} + \Sigma_{i=1}^p\phi_{2,i}y_{t-i})G(s_{t-d}; \gamma, c) + \epsilon_t

Where:

• y_t is the observed time series at time t.
• \phi_{1,i} are the parameters of the linear part of the model.
• \phi_{2,i} are the parameters associated with the nonlinear part of the model.
• G(s_{t-d}; \gamma, c) is the transition function.
• s_{t-d} is the transition variable with delay d.
• \gamma is the smoothness parameter.
• c is the threshold parameter.
• \epsilon_t is the error term.

Note: The transition function G(s_{t-d}; \gamma, c) determines how smoothly the model transitions between regimes.

Estimation:

1. Identifying the appropriate transition variable s_{t-d}.
2. Estimating the linear and nonlinear parameters (\phi_{1,i} and \phi_{2,i}).
3. Estimating the parameters of the transition function (\gamma and c).

Applications:

STAR models are useful in various contexts where smooth transitions between different regimes are expected. Common applications include:

• Economic and financial time series, where market conditions change gradually.
• Environmental data, where changes can be gradual and influenced by multiple factors.
• Any scenario where a smooth transition between states is more realistic than an abrupt switch.

4. Non-linear Moving Average (NMA) Models:

Non-Moving Average (NMA) models are not a standard class of time series models like AR (Autoregressive), MA (Moving Average), ARMA (Autoregressive Moving Average), or ARIMA (Autoregressive Integrated Moving Average) models. However, the term "Non-Moving Average" can be interpreted to refer to time series models that do not include a moving average component. In this sense, NMA models would encompass purely autoregressive models or other models that do not explicitly incorporate moving average terms.

Purely Autoregressive (AR) Models:

The AR model is a classic example of a time series model that does not include a moving average component.

Model Structure:

An AR(p) model, where p is the order of the autoregressive process, can be written as:

y_t = \phi_0 + \phi_1y_{t-1} + \phi_2y_{t-2} + ... + \phi_py_{t-p} + \epsilon_t

Where:

• y_t is the value of the time series at time t.
• \phi_0 is the intercept term (often assumed to be zero in many formulations).
• \phi_1, \phi_2,..., \phi_p are the parameters of the model.
• \epsilon_t is the error term or white noise.

Estimation:

The parameters of the AR model can be estimated using methods such as:

• Ordinary Least Squares (OLS): Minimizing the sum of squared residuals to estimate the coefficients.
• Yule-Walker Equations: Using the autocorrelation function to estimate the parameters.
• Maximum Likelihood Estimation (MLE): Maximizing the likelihood function for the observed data.

Applications:

AR models are widely used in many fields for various purposes, such as:

• Financial Econometrics: Modeling stock prices, interest rates, and other financial variables.
• Economics: Forecasting economic indicators like GDP, inflation, and unemployment rates.
• Engineering: Signal processing and control systems.
• Environmental Science: Modeling temperature, precipitation, and other environmental variables.

5. Neural Networks and Deep Learning Models:

Neural networks are a class of machine learning models inspired by the human brain. They consist of layers of interconnected nodes (neurons) where each connection has a weight. The basic types include: Feedforward Neural Networks (FNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs).

Deep learning refers to neural networks with many layers (deep neural networks), enabling the learning of complex features and representations. The basic types include: Deep Feedforward Networks, Deep Convolutional Networks, Deep Recurrent Networks.

Applications:

Neural networks and deep learning models are applied in various fields, including:

• Computer Vision: Image classification, object detection, and facial recognition.
• Natural Language Processing (NLP): Text classification, language translation, and chatbots.
• Speech Recognition: Converting speech to text.
• Time Series Forecasting: Predicting future values based on historical data.
• Healthcare: Diagnosing diseases from medical images, predicting patient outcomes.
• Autonomous Vehicles: Enabling perception and decision-making in self-driving cars.

These models are trained using backpropagation and optimization algorithms, enabling them to learn from large datasets and enhance their performance on complex tasks.

6. Polynomial and Exponential Models:

Polynomial Models:

Polynomial models capture nonlinear relationships by including polynomial terms of the independent variable(s), fitting data with curves for more flexibility than linear models.

Model Structure

A polynomial regression model of degree n can be written as:

y = \beta_0 + \beta_1x + \beta_2x^2 + ... + \beta_nx^n + \epsilon

Where:

y is the dependent variable.

x is the independent variable.

\beta_0, \beta_1,..., \beta_n are the coefficients of the model.

\epsilon is the error term.

Estimation:

The coefficients \beta_i are typically estimated using ordinary least squares (OLS) regression.

Applications:

Polynomial models are used in various fields to model complex relationships, such as:

• Economics: Modeling cost functions and production curves.
• Engineering: Modeling material properties and system behaviors.
• Natural Sciences: Fitting growth curves and decay processes.

Here's a simple example using Python and scikit-learn:

Exponential Models:

Exponential models are used to describe processes that grow or decay at a constant relative rate. These models are characterized by an exponential function of the independent variable.

Model Structure

An exponential growth model can be written as:

y = \beta_0e^{\beta_1x} + \epsilon

An exponential decay model can be written as:

y = \beta_0e^{-\beta_1x} + \epsilon

Where:

• y is the dependent variable.
• x is the independent variable.
• \beta_0 and \beta_1 are the parameters of the model.
• \epsilon is the error term.

Estimation:

The parameters \beta_0 and \beta_1 can be estimated using nonlinear regression techniques.

Applications:

Exponential models are used in various fields for modeling growth and decay processes, such as:

• Biology: Modeling population growth and decay of substances.
• Finance: Modeling compound interest and investment growth.
• Physics: Modeling radioactive decay and charging/discharging of capacitors.

Here's a simple example using Python and scipy.optimize:

It is important to remember that in both polynomial and exponential models, the key is to choose the appropriate model structure that best captures the underlying relationship in the data. Polynomial models are flexible and can fit a wide range of curves, while exponential models are ideal for processes with constant relative growth or decay rates.

Conclusion

Non-linear time series models are powerful tools for capturing complex relationships in data that linear models cannot adequately describe. By choosing the appropriate non-linear model and carefully estimating its parameters, analysts can make more accurate predictions and gain deeper insights into the underlying processes driving the time series.