Modelling Time Series with Neural

Networks
Volker Tresp
Summer 2017

1
 Modelling of Time Series

• The next figure shows a time series (DAX)

• Other interesting time-series: energy prize, energy consumption, gas consumption,

copper prize, ...

2
 Neural Networks for Time-Series Modelling

• Let zt, t = 1, 2, . . . be the time-discrete time-series of interest (example: DAX)

• Let xt, t = 1, 2, . . . denote a second time-series, that contains information on zt

(Example: Dow Jones)

• For simplicity, we assume that both zt and xt are scalar. The goal is the prediction of
the next value of the time-series

• We assume a system of the form

zt = f (zt−1, . . . , zt−T , xt−1, . . . , xt−T ) + t

with i.i.d. random numbers t, t = 1, 2, . . . which model unknown disturbances.

3
 Neural Networks for Time-Series Modelling (cont’d)

• We approximate, using a neural network,

f (zt−1, . . . , zt−T , xt−1, . . . , xt−T )

≈ fw,V (zt−1, . . . , zt−T , xt−1, . . . , xt−T )

and obtain the cost function
N
X
cost(w, V ) = (zt − fw,V (zt−1, . . . , zt−T , xt−1, . . . , xt−T ))2
t=1

• The neural network can be trained as before with simple back propagation if in training
all zt and all xt are known!

• This is a NARX model: Nonlinear Auto Regressive Model with external inputs. Ano-
ther name: TDNN (time-delay neural network).

• Note the ”convolutional“ idea in TDNNs

4
• Language model: last T words as input. The task is to predict the next word in a
sentence. One-hot encoding.
 Mutiple-Step Prediction

• Predicting more than one time step in the future is not trivial

• The future inputs are not available

• The model noise needs to be properly considered in multiple step prediction (for
example by a stochastic simulation); if possible one could also simulate future inputs
(multivariate prediction)

• Teacher forcing: Use the predicted output as input (can be risky)

5
 Recurrent Neural Network

• Recurrent Neural Networks are powerful methods for time series and sequence model-
ling

6
 Generic Recurrent Neural Network Architecture

• Consider a feedforward neural network where there are connections between the hidden
units
zt,h = sig(zT T
t−1 ah + xt vh )
and, as before,
ŷt = sig(zT
t w)

• Here, zt = (zt,1, zt,2, . . . , zt,H )T , xt = (xt,0, xt,1, . . . , xt,M −1)T

• In Recurrent Neural Networks (RNNs) the next state of the neurons in the hidden
layer depends on their last state and both are not directly measured

• ah, w, vh are weight vectors

• Note that in must applications, one is interested in the output yt (and not in zt,h)

• The next figure shows an example. Only some of the recurrent connections are shown
(blue). The blue connections also model a time lag. Without recurrent connections
(ah = 0, ∀h), we obtain a regular feedforward network
7
• Note, that a recurrent neural network has an internal memory
A Recurrent Neural Network Architecture unfolded in Time

• The same RNN but with a different intuition

• Consider that at each time-step a feedforward Neural Network predicts outputs based
on some inputs

• In addition, the hidden layer also receives input from the hidden layer of the previous
time step

• Without the nonlinearities in the transfer functions, this is a linear state-space model;
thus a RNN is a nonlinear state-space model

8
 Training of Recurrent Neural Network Architecture

• Backpropagation through time (BPTT): essentially backpropagation applied to the

unfolded network; note that all that happened before time t influences ŷt, so the error
needs to be backpropagated backwards in time, in principle until the beginning of
the experiment! In reality, one typically truncates the gradient calculation (review in:
Werbos (1990))

• Real-Time Recurrent Learning (RTRL) (Williams and Zipser (1989))

• Time-Dependent Recurrent Back-Propagation: learning with continuous time (Lag-

rangian approach) (Pearlmutter 1998)

9
 Echo-State Network

• Recurrent Neural Networks are notoriously difficult to train

• A simple alternative is to initialize A and V randomly (according to some recipe) and

only train w, e.g., with the ADALINE learning rule

• This works surprisingly well and is done in the Echo-State Network (ESN)

10
 Iterative Prediction

• Assume a trained model where the prediction is: ŷt → (xt, yt) → ŷt+1, ...

• Thus we predict (e.g., the DAX of the next day) and the obtain a measurement of
the next day

• A RNN would ignore the new measurement. What can be done

• 1: In probabilistic models, the measurement can change the hidden state estimates
accordingly (HMM, Kalman filter, particle filter, ....)

• 2: We can use yt as an input to the RNN (as in TDNN)

• 3: We add a (linear) noise model

11
 Bidirectional RNNs

• The predictions in bidirectional RNNs depend on past and future inputs

• Useful for sequence labelling problems: handwriting recognition, speech recognition,

bioinformatics, ...

12
 Long Short Term Memory (LSTM)

• As a recurrent structure the Long Short Term Memory (LSTM) approach has been
very successful

• Basic idea: at time T a newspaper announces that the Siemens stock is labelled as
“buy”. This information will influence the development of the stock in the next days.
A standard RNN will not remember this information for very long. One solution is to
define an extra input to represent that fact and that is on as along as “buy” is valid. But
this is handcrafted and does not exploit the flexibility of the RNN. A flexible construct
which can hold the information is a long short term memory (LSTM) block.

• The LSTM was used very successful for reading handwritten text and is the basis for
many applications involving sequential data (NLP, translation of text, ...)

13
 LSTM in Detail

• The LSTM block replaces one hidden unit zh, together with its input weights ah and
vh. In general all H hidden units are replaced by H LSTM blocks. It produces one
output zh (in the figure it is called y)
• All inputs in the figure are weighted inputs
• Thus in the figure z would be the regular RNN-neuron output with a tanh transfer
function
• Three gates are used that control the information flow
• The input gate (one parameter) determines if z should be attenuated
• The forget gate (one parameter) determines if the last z should be added and with
which weight
• Then another tanh, modulated by the output gate
• See http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-
a-grulstm-rnn-with-python-and-theano/

14
 LSTM Applications

• Wiki: LSTM achieved the best known results in unsegmented connected handwriting
recognition, and in 2009 won the ICDAR handwriting competition. LSTM networks
have also been used for automatic speech recognition, and were a major component
of a network that in 2013 achieved a record 17.7% phoneme error rate on the classic
TIMIT natural speech dataset

• Applications: Robot control, Time series prediction, Speech recognition, Rhythm lear-
ning, Music composition, Grammar learning, Handwriting recognition, Human action
recognition, Protein Homology Detection

15
 Gated Recurrent Units (GRUs)

• Some people found LSTMs too complicated and invented GRUs with fewer gates

16
 Encoder Decoder Architecture

• For example, used in machine translation

17