World Academy of Science, Engineering and Technology 42 2008

Comparing Autoregressive Moving Average (ARMA) coefcients determination using Articial Neural Networks with other techniques
Abiodun M. Aibinu, Momoh J. E. Salami, Amir A. Shae and Athaur Rahman Najeeb

AbstractAutoregressive Moving average (ARMA) is a parametric based method of signal representation. It is suitable for problems in which the signal can be modeled by explicit known source functions with a few adjustable parameters. Various methods have been suggested for the coefcients determination among which are Prony, Pade, Autocorrelation, Covariance and most recently, the use of Articial Neural Network technique. In this paper, the method of using Articial Neural network (ANN) technique is compared with some known and widely acceptable techniques. The comparisons is entirely based on the value of the coefcients obtained. Result obtained shows that the use of ANN also gives accurate in computing the coefcients of an ARMA system. KeywordsAutoregressive Moving Average, Coefcients, Back Propagation, Model Parameters, Neural Network, Weight.

technique is the estimation of the model coefcients. Some of the existing methods of determining model coefcients includes Prony, Pade, Least Square, Shank, Autocorrelation, Autocovariance methods [9]. The organization of this paper is as follows, section I gives a brief and concise introduction to signal modeling and its associated challenges. In section II, some of the variations or types of modeling will be discussed while section III gives brief introduction to various methods of estimating model coefcients. In section IV, the detail of using Neural network reported in [7], [8] will be discussed. Section V will discuss the result obtained while the conclusion is as presented in section VI. II. S IGNAL M ODELING T YPES Some of the known modeling methods include:

I. I NTRODUCTION The use of modeling technique to predict or reconstruct a data sequence is concerned with the representation of data in an efcient technique [1][4], [6], [10], [13]. Signal modeling have been used in radar application, geophysical application, Medical signal processing, ultrasonic tissue backscatter coefcient estimation, speech processing, music understanding and more recently in the eld of Magnetic Resonance Imaging (MRI) reconstruction [2], [4], [6], [10], [13], [14], [16], [17]. Signal modeling involves two steps steps [2], these are; 1) Model selection: Choosing an appropriate parametric form for the model data 2) Model Parameter determination: This include model order and model coefcients determination. Despite the success reported in the use of modeling technique, two important problems constitutes challenges to the applicability of this method, these are: 1) Estimation of Model order: There have been various effort in determining a workable criteria for the determination of an appropriate model order. The use of a model with an order too high over ts the data while the use of a model with a low order leads to insensitivity to noise [2], [4], [6], [15]. 2) Estimation of model coefcient : The second important challenges mitigating against the use of modeling
Athaur Rahman Najeeb is with the Kulliyah of Engineering, International Islamic University Malaysia (IIUM), email: athaur@[Link] Momoh.J.E Salami is with the Kulliyah of Engineering, International Islamic University Malaysia (IIUM), email: momoh@[Link] Amir A. Shae is with the Kulliyah of Engineering, International Islamic University Malaysia (IIUM), email: aashae@[Link] Abiodun. M. Aibinu, Malaysia, email: maibinu@[Link]

Autoregressive modeling Technique (AR): Consider a system describe by a linear constant coefcient difference equation (LCCDE) given by (1), the output y(n) is obtained by using only previous outputs i.e y(n 1), y(n 2), y(n 3) . . . y(n p)and the current input i.e x(n), which means that b(k) = 0 for k > 0 and only a(k) and b(0) must be determined, such a system are called Autoregressive (AR) model. AR model equation is given
p

y(n) =
k=1

ak y(n k) + b(0)x(n)

(1)

and the equivalent z-domain is

Y (z) =
k=1

ak Y (z)z k + b(0)X(z) Y (z) = X(z) b(0)

(2)

H(z) =

(3)

1+
k=1

ak z k

Autoregressive with external input(s) modeling Technique (ARX): From (4), the output y(n) is given by
p

y(n) =
k=1

ak y(n k) + b(0)x(n)

(4)

adding an external input yields

173

World Academy of Science, Engineering and Technology 42 2008

y(n) =
k=1

ak y(n k) + b(0)x(n) +
k=1

ck u(n k) (5)

taking z-transform of both sides results in

Y (z) =
k=1

ak Y (z)z k + b(0)X(z) +
k=1

ck U (z)z k

Fig. 1.

Direct Method of least square method of ARMA Model

(6) Autoregressive moving Average (ARMA) modeling technique The general ARMA equation is given by
p q

y(n) =
k=1

ak y(n k) +
k=0

bk x(n k)

(7)

modeling techniques is as contained in the Wold decomposition theorem [12]. This theorem shows that any stationary ARMA or MA process of nite variance can be represented as a unique AR process of possibly innite order; likewise any ARMA or AR process can be represented as a MA process of possibly innite order [12]. III. M ETHODS OF C OEFFICIENTS DETERMINATION

taking z-transform of both sides results in

Y (z) =
k=1

ak Y (z)z k +
k=0 q

bk X(z)z k

(8)

Various methods have been reported in literatures for determining the AR/ARMA model coefcients, among which are: A. Direct least square method

so that H(z) = Y (z) = X(z)

bk z k
k=0 p

(9) ak z k

The block diagram for direct method of least square solution is as shown in g. 1 The modeling error can be written as e(n) = x(n) h(n) in Frequency domain, we have

1+
k=1

in which X(z) =

x(n)z n
k=

(10)

E(ej ) = X(ej )

Bq (ej ) Ap (ej )

(15)

and Y (z) =

y(n)z n
k=

(11)

In this method, the signal modeling to be minimized is the squared error,

Autoregressive moving Average with external input(s) (ARMAX) modeling technique The general ARMA equation is given by 12,i.e
p q

ls =
n=0

|e(n)|2

A necessary condition for the coefcients ap (k) and bq (k) to minimize the error is that the partial derivative of ls with respect to each of the coefcients vanishes. i.e ls = 0; a (k) p ls = 0; b (k) p k = 1, 2, . . . , p

y(n) =
k=1

ak y(n k) +
k=0

bk x(n k)

(12)

the addition of external input to the system results in

y(n) =
k=1

ak y(nk)+
k=0

bk x(nk)+
k=1

ck u(nk) (13)
r

k = 1, 2, . . . , q

taking z-transform of both sides results in

Using Parsevals theorem and taking the fourier transform of the error e(n), we have ls = 1 ls = (k) ap 2 1 ls = a (k) 2 p and

Y (z) =
k=1

ak Y (z)z k +
k=0

bk X(z)z k +
k=1

ck U (z)z k

1 2

|E(ej )|2 d

(16)

(14) other known modeling techniques include Autoregressive Integrated Moving Model (ARIMA) Autoregressive Integrated Moving with external input Model (ARIMAX) Autoregressive Fractionally Integrated Moving Model (AFRIMA) In this report, The general ARMA given by eqn. 12 will be discussed and the relationship between AR, MA and ARMA

ap (k)

[E(ej )E (ej )]d = 0

(17)

substituting (15) to (17), gives [X(ej )

174

World Academy of Science, Engineering and Technology 42 2008

1 ls = b (k) 2 q

[X(ej )

Bq (ej ) ejk ] d = 0 Ap (ej ) A (ej ) p (19)

B. Pade Approximation The Pade approximation can be developed using g. 1. The system function is
q

Fig. 2.

Method of Prony Method

Bq (z) = H(z) = Aq (z)

bq z k
k=0 p

substituting eqn. (23) in eqn. (26),

1+
k=1

ap z k

(x(n) +
n=q+1 l=1

ap (l)x(n l))x (n k) = 0

(27)

which leads to the difference equation

or equivalently, ap (k)h(n k) = bq (n) (20)

h(n) +
k=1

ap (l)[
l=1 n=q+1

(x(nl)x (nk)] =
n=q+1

(x(n)x (nk)] (28)

setting h(n) = x(n) for n = 0, 1, 2 . . . , p + q in (20) yields a set of p + q + 1 linear equations in p + q + 1 unknowns, given by
q

becomes
p

ap (l)rx (k, l) = rx (k, 0); ap (k)h(nk) = bq (n) 0 n = 1, 2, . . . , q n = q + 1, . . . q + p (21)

k = 1, 2 . . . , p

(29)

h(n)+
k=1

where rx (k, l) =

(x(n l)x (n k)
n=q+1

C. Prony method Multiplying both sides of (15) by Ap (z) yields E(z) = Ap E (z) Bq (z) in time domain, e(n) = ap (n) x(n) bq (n) = q (n) bq (n), b since bq (n) = 0 for n > q, then, x(n) +
p

D. Shank method (Modied Prony) Shank method is a modied Prony method in the sense that the moving average coefcients is obtained by nding the least square minimization of the model error over the entire data length [9]. The model can be viewed as a cascade of two lters, Bq (z) and Ap (z) The combine transfer function is given by, H(z) = Bq (z) 1 Ap (z)

(22)

ap (l)x(n l) bq (n)
l=1 p

e(n) =

x(n) +
l=1

ap (l)x(n l)

n = 1, 2, . . . , q The output of y(n) can be computed using n=n>q (23)

g(n) = (n)

ap (k)g(n k)

l=1

ls =
n=0

|e(n)|2 =
n=0

|x(n) +
l=1

ap (l)x(n l)|2

(24)

The numerator coefcient is obtained by minimizing the square of the error.

setting the partial derivatives of ls with respect to a (k) equal p to zero, gives e (n) [e(n)e (n)] p,q = = =0 e(n) a (k) n=q+1 a (k) ap (k) p p n=q+1 (25) p,q = e(n)x (n k) = 0 a (k) n=q+1 p

S =
n=0

|e(n)|2

minimizing the error in Prony method gives

bp (l)ry (k, l) = rxy (k);

k = 1, 2 . . . , q

(30)

where k = 1, 2, . . . p (26) ry (k l) =

(y(n l)y (n k)
n=0

175

World Academy of Science, Engineering and Technology 42 2008

and rxy (k) =

(x(n)y (n k)
n=0

IV. AR/ARMA C OEFFICIENTS DETERMINATION USING A RTIFICIAL N EURAL N ETWORK T ECHNIQUE The general three layer neural network of obtaining the coefcients of ARMA and NARMA reported in [7], [8] is as shown in gure 4

The block diagram for Shanks method is as shown in g. 3

Fig. 3.

Method of Shank method

E. Autocorrelation method In this method, the signal x(n) is only known over a nite data sequence [0, N ], this is obtained by multiplying an innite data sequence by a window function w(n) to obtain another signal x (n). i.e x (n) = s(n)w(n) 0 0nN 1 otherwise (31)
Fig. 4. Neural Network Technique for obtaining ARMA/NARMA coefcients. [7], [8]

Using Pronys method to nd an all pole model for x (n) by minimizing ap (k) coefcients as stated in (26), (28), (29), yields
p

For a system dene by,

y(n) =
k=1

ak y(n k) +
k=0

bk x(n k)

(35)

ap (l)rx (k, l) = rx (k, 0);

k = 1, 2 . . . , p

(32)

the coefcients are obtained from the neural network weights value and polynomial coefcients given by eq. 54 and eq. 55.
M

where rx (k) =

x(n)x (n k)
n=k

ai =
j=1 M

wj1 a1j vij y(n i)

(36)

The autocorrelation matrix formed is a symmetric Toeplitz matrix. F. Covariance method In contrast to Autocorrelation method discussed in section III-E, the error in (24) is minimized over a denite interval [p, N ] , that is
N C p = n=p

bi =
j=1

wj1 a1j vij x(n i)

(37)

A. ARMA-ANN Coefcients determination Algorithm 1) Stopping criteria: Set the stopping criteria; Epoch or Mean Square Error (MSE) 2) Initialize: Number of input nodes equal model order. i.e (Input nodes = p + q) Output node is one(1), y(n) Initialize the weight vectors Intialize Polynomial Order, R (R=2) Intialize all Polynomial coefcients,azi where
R

|e(n)|2

(33)

The only difference between covariance method and Prony method is in the summation of the error term [9], the covariance normal equations can be written as
p

ap (l)rx (k, l) = rx (k, 0);

k = 1, 2 . . . , p

(34)

where rx (k, l) =

Pi (t) =
N z=0

azi tz

(38)

(x(n l)x (n k)
n=p

3) Training Pattern : Select the training input and output pairs for the network.

176

World Academy of Science, Engineering and Technology 42 2008

Input pattern, AR section: y(n1), y(n2), . . . y(np) Input pattern, MA section: x(n), x(n 1), . . . x(n q) Target pattern, T (n) 4) Run selected pattern The output node y(n) is given by,
M

For Output neuron, the activation function is P (t) = t so the derivative P1 = P2 = 1. t t So, (53) wk(new) = wk(old) + Pk 10) Test for Stopping criteria: Test for stopping criteria. If completion criteria is not satised, go to step 4 else calculate and output the ARMA coefcients
M

y(n) =
k=1

wk1 hi (t)

(39)

and the hidden nodes output is given as,

ai = azi tz (40) bi =
j=1 j=1 M

wj1 a1j vij y(n j)

(54)

hi (t) =
z=0

where the AR part is given by

wj1 a1j vij x(n j)

(55)

t(n) =
j=1

vk1 yi (n j) + (j)

(41)

11) Ends V. R ESULT OBTAINED In this paper, results obtained by the use of Pade, Prony Shank, Autocorrelation and Covariance method coefcients determination types will be reported. 1) Autoregressive Equation 1 y(n) = y(n 1) + 0.24y(n 2) + w(n) where w(n) is white noise. 2) Autoregressive Equation 2 (56)

5) Evaluate the Error The output Error is (n) = T (n) y(n) 6) Back Propagate the Error k (n) = wk (n) 7) Weight Update (Input to Hidden layer) vjk(new) = vjk(old) + k where Pk = t
2

(42)

(43)

Pk (t) y(n j) t

(44)

y(n) =

0.51y(n 1) + 0.315y(n 2) 0.23y(n 3) 0.56y(n 4) + 0.1y(n 5) 0.045y(n 6) + w(n)

babk t
b=1

(b1)

(45)

where w(n) is white noise. 3) Autoregressive Moving Average y(n) = 0.11y(n 1) + 0.52y(n 2) +x(n) 0.3x(n 1) 0.078x(n 2)

8) Polynomial Coefcient Update (Hidden layer) ajknew = ajkold + 1 where Pi =1 a0i Pi = ti a1i Pi = t2 i a2i which can be simply put as Pi = tr i ari (51)
Methods Actual Value Pade Prony Shank Autocorrelation Autocovariance NN

Pi ajk

(46)

TABLE I R ESULT OBTAINED FROM EQN . 56 Methods Actual Value Pade Prony Shank Autocorrelation Autocovariance NN a(1) 1.000 0.670 0.833 0.872 0.903 1.001 1.000 a(2) 0.240 0.203 0.215 0.233 0.240 0.240 0.241

(47) (48) (49) (50)

TABLE II R ESULT OBTAINED FROM EQN . 57 a(1) 0.510 0.704 0.504 0.513 0.530 0.510 0.510 a(2) 0.315 0.033 0.331 0.331 0.285 0.322 0.314 a(3) 0.230 0.202 0.202 0.202 0.202 0.230 0.230 a(4) 0.560 0.167 0.667 0.668 0.667 0.561 0.560 a(5) 0.100 0.488 0.188 0.103 0.089 0.104 0.102 a(6) 0.045 0.279 0.178 0.179 0.179 0.046 0.045

for r = 0, 1 and 2 , i is the hidden layer number and ti is as dened in 41 for the AR section. 9) Weight Update (Hidden layer to Output Layer) wk1new = wk1old + Pk Pk t (52)

177

World Academy of Science, Engineering and Technology 42 2008

TABLE III R ESULT OBTAINED FROM EQN . 57 Methods Actual Value Pade Prony Shank Autocorrelation Autocovariance NN a(1) 0.110 0.421 0.221 0.221 0.121 0.111 0.110 a(2) 0.500 0.341 0.600 0.5980 0.631 0.500 0.500 b(1) 0.411 0.223 0.366 0.380 0.372 0.411 0.411 b(2) 0.390 0.219 0.431 0.439 0.343 0.390 0.390 b(3) 0.685 0.355 0.534 0.534 0.639 0.680 0.685

[13] R. Palaniappan, Towards Optimal Model Oreder Selection for Autoregressive Spectral Analysis of Mental Tasks Using Genetic Algorithm, IJCSNS International Journal of Computer Science and Network Security, Vol. 6 No. 1A, January 2006. [14] H. Akaike, Power Spectrum Estimation through Autoregression Model Fitting, Annals of the Institute of Statistical Mathematics, vol. 21, pp. 407-419, 1969. [15] H. Akaike, A New Look at the Statistical Model Identication, IEEE Trans. Autom. Control, vol. AC-19, pp. 716-723, 1974. [16] J. Rissanen, Modelling by shortest data description, Automatica, vol.14, pp.465-471, 1978. [17] M.J. Salami, A. R. Najeeb, O. Khalifa, K. Arrin, MR Reconsturction with Autoregressive Moving Average, International Conference on Biotechnology Engineering, Kuala Lumpur, pp 676 - 704, May, 2007.

VI. C ONCLUSION In this paper, different methods of determining ARMA coefcients have been evaluated based on a simulated data. An algorithm in achieving the reported methods in [7], [8] have also been discussed. MATLAB implementation of this have also been carried out . This work only consider the accuracy of the coefcients and not the time of completion of each of the method. Result obtained shows that this method efciently and accurately compute ARMA coefcients. Result obtained also shows that the result perform better than some of the existing method of ARMA coefcient beacuse of the non linearity nature of ANN. Area of application of this proposed algorithm include Magnetic Resonance Imaging reconstruction using parametric technique [4], Signal modeling, Adaptive control system and PID tunning. Acknowledgement: We shall like to express our appreciation to Prof. K. H. Chon for all his suggestion, advice and contribution to this work. R EFERENCES
[1] Z. P. Liang, P. C. Lauterbur, Principles of Magnetic Resonance Imaging, A signal processing perspective, IEEE Press, New York, 2000. [2] A. M. Aibinu, M. J.E. Salami, A. A. Shae and A. R. Najeeb, Model Order Determination for MRI Signal, accepted for publication , 2008. [3] D. G. Nishimura, Principles of Magnetic Resonance Imaging, April 1996. [4] M. R. Smith, S. T. Nichols, R. M. Henkelman and M. L. Wood, Application of Autoregressive Moving Average Parametric Modeling in Magnetic Resonance Image Reconstruction, IEEE Transactions on Medical Imaging, Vol. M1-5:3, pp 257 - 261, 1986. [5] F. J. Harris,On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform, Proceedings of the IEEE. Vol. 66, January 1978. [6] M. R. Smith, S. T. Nichols, R. Constable and R. Henkelman, A quantitative comparison of the TERA modeling and DFT magnetic resonance image reconstruction techniques, Magn. Reson. Med., Vol. 19 pp. 1-19, 1991. [7] K. H. Chon, R. J. Cohen, Linear and Non-Linear ARMA Model Parameter Estimation Using Articial Neural Network, IEEE Transactions on BioMedical Engineering, Vol. 44, No 3, pp 168 - 174, 1997. [8] K. H. Chon, D. Hoyer, A. A Armoundas, N-H Holstein-Rathlou and D. J Marsh, Robust Nonlinear Autoregressive Moving Average Model Parameter Estimation Using Recurrent Articial Neural Network, Annals of BioMedical Engineering, Vol. 27, pp 538-547, 1999. [9] M. H. Hayes, Staitical Digital Signal processing and Modelling, John Wiley & Sons, Canada, 1996. [10] Z. P. Liang, F. E. Boada, R. T. Constable, E. M. Haacke, P. C. Lauterbur, and M. R. Smith, Constrained Reconstruction Methods in MR Imaging, Reviews of MRM, vol. 4, pp.67 - 185, 1992. [11] E. Hackle and Z. Liang, Superresolution Reconstruction Through Object Modeling and Estimation, IEEE transactions in A.S.S.P, 37: 592 - 595, 1989. [12] E. C. Whitman, The spectral analysis of discrete time series in terms of linear regressive models, Naval Ordinance Labs Rep., NOLTR-070109, White Oak, MD, June 23, 1974.

178