1

ESSE 4020 / ESS 5020 Time Series and Spectral Analysis 27 Sept 2019 Notes 2: AR, MA, ARMA and ARIMA processes CM2009, Chapters 4, 6, 7; SS2017, Chapter 3, B&D, Chapters 2, 3 Recall autocovariance and autocorrelation of Xt at time lag h

Note no time dependence for stationary time series, but some have time dependence γ(h,t).

Note γ(h) for autocovariance, ρ(h) = γ(h)/ γ(0) for autocorrelation. H is lag.

2

CM 2009, Section 4.5 - AR models. Backward shift operator: xt-1 = B(xt) etc.

4.5.2 Stationary and non-stationary AR processes The equation θp(B) = 0, where B is formally treated as a number (real or complex), is called the characteristic equation. The roots of the characteristic equation (i.e., the polynomial θp(B) from Equation (4.16)) must all exceed unity in absolute value for the process to be stationary. Notice that the random walk has θ1(B) = 1−B with root B = 1 and is non-stationary. Random Walk examples - non-stationary

3

Variance increasing with time, so non-stationary, variance increasing with time. Modify rwalk6.m to rwalk7.m

Note that histogram used more bins and shows that only the odd numbers are occupied after 100 steps, or actually 99 steps since we start at 2, (with Y1 = 0). We want to show that the variance is increasing with time. We have all the position data in the array YA(r,s) so we want the variance of the columns of YA. “V = var(A) returns the variance of the elements of A along the first array dimension whose size does not equal 1.” Same with mean and sum and std(A,1) So plot(std(YA,1)); hold on; plot(mean(YA)); xlabel('X distance'); ylabel('Y displacement');

4

5

But that was a short test, can increase STEPS and RUNS. RUNS=1000; STEPS=200;

Mean now closer to 0, standard deviation approximately √n, note √200 = 14.14

----------------------------------------

6

Lets modify and create an AR(1) process with |α1| < 1. With the rwalk code we can introduce a coefficient alpha1 so that the process becomes Xt = alpha1 * X t-1 + Wt And set alpha1 = + or - 0.75 for example. Code is ar1.m

Note: θ1(B) = 1− α1B has root 1/α1, so +/- 4/3 and requirement is |root| > 1, so stationary.

7

Key component in code is, for s = (2:STEPS) Y(s) = alpha1*Y(s-1) + ssize*2*(R0(s)-0.5); % uniform random numbers between -1 and 1. YA(r,s) = Y(s); end

What should we expect the variance to be? 1/(1-α12) = 2.286, var(2*R0-0.5) = 1/3; leads to STD(YA) = 0.873. Also checking histograms of R0 and YF.

8

Can also look at the case with alpha1 = -0.75 Try some AR(2) examples? This code in R xt = alpha1*xt-1 + alpha2*xt-2 + wt. and maybe wt = rnorm(n, mean = 0, sd = 1) alpha1 = 0.75; alpha2 = 0.25 # will generate 2000 series SARR <- matrix(data = NA, nrow = 2000, ncol = 100, byrow = FALSE) S <- vector("numeric",100); S[1]=0 ; S[2]= 0 for (i in 1:2000) {SRS <- rnorm(100, mean=0, sd=1); # normally distributed for (t in 3:100) {S[t] <- S[t-1] *alpha1 + alpha2*S[t-2] + SRS[t]; SARR[i,t] = S[t]}} plot(SARR[1,])

9

for (i in 1:20) {plot(SARR[100*i,], type="l",col=i,ylim=c(-20,20),xlab="timesteps"); par(new=T)}

? Are roots > 1? xt = alpha1*xt-1 + alpha2*xt-2 + wt. : alpha1 = 0.75; alpha2 = 0.25 Characteristic Equation is B2+3B-4=0 with roots -4 and +1 so should NOT be stationary. For (t in 2:100) {COLVAR[t] <- var(SARR[,t])}; plot(COLVAR) Lets adapt our matlab code again (arm2.m), or we could do arm5.m or armK.m and set some alpha values =0. Not computationally efficient but we are dealing with small numbers of calculations. This is assignment 2. Useful notes (by Mingda Zhang) at http://people.cs.pitt.edu/~milos/courses/cs3750/lectures/class16.pdf And https://faculty.chicagobooth.edu/jeffrey.russell/teaching/bstats/timeseries.pdf

10

The Characteristic Equation The roots of the characteristic equation (i.e., the polynomial θp(B) from Equation (4.16)) must all exceed unity in absolute value for the process to be stationary. Can we see why? Simple AR(1) case (1-α1B)xt = wt so root is 1/α1 and need α1 < 1 for stationarity. If α1 ≥ 1 non-stationary and variance will ? increase. Suppose we write xt = (1-α1B)-1 wt and expand (1-α1B)-1 = 1 + α1B + α12B2 + α13B3 …. With Bnwt = wt-n we have xt = wt + α1wt-1 + α12wt-2 …. Provided the wt are IID we then have that the variance of xt will be (1 + α1 + α12 + α13 ….) * variance of wt. If α1 < 1 this converges to 1/(1- α11) and diverges if α1 ≥ 1. For a higher order AR(p) process the roots of the characteristic equation can be considered as 1/β1, 1/β2 …. 1/βk which may be complex, but the AR(p) process can be written as (1-β1B) (1-β2B)….. (1-βkB)xt = wt and similar arguments can be made to show that the variance increases if any βi has absolute value ≥ 1, i.e. |1/ βi| ≤1. Examples AR(1): (1-α1B)xt = wt ; α1 = +/-0.9, +/-1.1 wt as normally distributed White noise randn([NR 1]) will give NR values. Check, R= 100, RN=randn([NR 1]); mean(RN); -0.0690; std(RN); 0.9499 So can use variations of code ar1a.m

11

RUNS = 1000; STEPS = 200 alpha1 = 0.9

Can we determine expected VAR and STD? xt = α1xt-1 + wt E[xt2] = α12 E[xt-12] + E[wt2] stationary? E[xt2] = E[xt-12] so (1- α12)E[xt2] = E[wt2] = 1 and E[xt2] = 5.26, =(2.29)2. Should be same with α1 = -0.9?

and can try α1 = +/-1.1

12

With an ar2 model alpha1 = 0.5000; alpha2 = 0.4000; RUNS = 1000; STEPS = 200 – more steps better!

Can we compute E[Yt2]? And E[YtYt-1], E[YtYt-k]? IF stationary. And how to check stationarity? xt = α1xt-1 + α2xt-2 + wt - gets more difficult, but possible if stationary, and it looks stationary. Is there a simple check? p=[-alpha2 -alpha1 1]; r=roots (p); r = -2.3252; 1.0752 E[xt2] = α12 E[xt-12] + α22 E[xt-22] + 2α1α2 E[xt-1xt-2] + E[wt2] : (1-α12-α22) E[xt2] =2α1α2 E[xt-1xt] + E[wt2] E[xt-1xt] = α1 E[xt-12] + α2 E[xt-1xt-2] : (1- α2)E[xt-1xt] = α1 E[xt2] (1-α12-α22-2α12α2/(1- α2)) E[xt2] = E[wt2] and E[xt2] = 3.8961, so STD = 1.9739 And also E[xt-1xt] / E[xt2] = α1 / (1- α2) = 0.8333

13

Histogram and sample autocorrelation is just for the final run. Could do for all runs and average. Note our calculated value for ρ(1) was 0.8333. Try for a few other runs, or for all and average. Lengthening STEPS gives ρ(1) = 8382, and another run gave the average acf. YACC = 1.0000 0.8253 0.8098 0.7343 0.6911 0.6400 0.5965 0.5533 0.5147 0.4776

14

CM2009, Exercises 4.8. Do some! e.g. #5. wt can be normally distributed, rnorm(1000, 0,1)

------------------------------------------------------------

15

Another look at ar(2) clear RUNS=5; STEPS=50; alpha1 = 0.5;alpha2 = 0.5; alpha1 alpha2 RUNS STEPS p=[-alpha2 -alpha1 1]; rts=roots (p); rts if min(abs(rts))<=1; 'rts <=1' pause ; % Would prefer to stop? end; R1= 1:RUNS; YF = 1:RUNS; YA = ones(RUNS,STEPS); Y=1:STEPS; YACA=ones(RUNS,10); for r = (1:RUNS) rng(r);%for repeatability RN = randn(STEPS, 1); YA(r,1)=0; YA(r,2)=0; Y(1)=0; Y(2)=0; for s = (3:STEPS) Y(s) = alpha1*Y(s-1) + alpha2*Y(s-2) + RN(s);

YA(r,s) = Y(s); end YF(r) = YA(r,STEPS); acf = autocorr(Y); for k = (1:10) YACA(r,k)=acf(k); end plot(Y) hold on r%just a check to see how things are going end YR=10; xlim([0,STEPS]); ylim([-YR,YR]) title('AR2 Examples') hold off pause YACC=mean(YACA); YACC pause plot(std(YA,1)); hold on; plot(mean(YA)) xlabel('X step'); ylabel('Y displacement properties'); title('AR2 process, Mean and STD(YA,1)'); hold off

Test with a range of alpha1, alpha2 values, stationary and non-stationary

16

ARMA models. "Auto-Regressive Moving Average"

A FEW BASICS (from B&D)

or,

SS2016 use this BUT I and CM2009 usually interchange θ and ϕ

17

CM 2009, Section 6.3 Moving average models 6.3.1 MA(q) process: Definition and properties A moving average (MA) process of order q is a linear combination of the current white noise term and the q most recent past white noise terms and is defined by MA(q) xt = wt + β1wt−1 + . . . + βqwt−q (6.1) where {wt} is white noise with zero mean and variance σw2. Equation (6.1) can be rewritten in terms of the backward shift operator B xt = (β0 + β1B + β2B2 + · · · + βqBq)wt = Φq(B)wt with β0 = 1. (6.2)

where Φq is a polynomial of order q. Because MA processes consist of a finite sum of stationary white noise terms, they are stationary and hence have a time-invariant mean and autocovariance.

18

Some examples in R – an MA(3) case wt<- rnorm(1000, 0,1); xt[1:3]=0 for (i in 4:1000) {xt[i] <- wt[i] + wt[i-1] + wt [i-2] + wt[i-3]} plot(xt,type="l"); xt[1:20] [1] 0.00000000 0.00000000 0.00000000 -0.08433462 -2.32828574 -2.38131712 [7] -0.47701596 -0.90236979 0.09260226 0.49226734 0.24246025 -0.76357704 [13] -0.33308547 -0.83848288 0.50881789 0.60260623 0.76167261 0.52655829 [19] -1.38800848 -0.14792698 mean(xt[4:1000]); [1] -0.08895812 var(xt[4:1000]) ; [1] 3.733957 wt<- rnorm(10000, 0,1); xt[1:3] =0.0; for (i in 4:10000) {xt[i] <- wt[i] + 2*wt[i-1] + 3*wt [i-2] + 4*wt[i-3]} var(xt [4:10000]) [1] 30.11514 # 1 + 4 + 9 + 16 = 30 mean(xt[4:10000]) [1] 0.04345046 plot(acf(xt[4:10000])) xt.acf <- acf(xt[4:10000]) xt.acf[0:10] Autocorrelations of series ‘xt[4:10000]’, by lag 0 1 2 3 4 5 6 7 8 9 10 1.000 0.670 0.368 0.136 0.004 -0.002 -0.002 -0.002 0.001 -0.003 -0.008 plot(xt.acf)

19

Compute acf2 by hand? (3+8)/30 > 11/30 : [1] 0.3666667 Invertible?

20

And with Matlab – my MA(4) example, stripped down a bit – just 1 run clear STEPS=5000; RUNS=1; beta1=1.0; beta2=1.0;beta3=1.0;beta4=1.0; beta1 beta2 beta3 beta4 RUNS STEPS R1= 1:RUNS; YF = 1:RUNS; YA = ones(RUNS,STEPS); Y=1:STEPS; for r = (1:RUNS) rng(r);%for repeatability RN = randn(STEPS, 1); YA(r,1)=0; YA(r,2)=0; Y(1)=0; Y(2)=0; YA(r,3)=0; YA(r,4)=0; Y(3)=0; Y(4)=0; for s = (5:STEPS)

Y(s)=beta1*RN(s-1)+beta2*RN(s-2) + beta3*RN(s-3)+beta4*RN(s-4)+RN(s);

YA(r,s) = Y(s); end YF(r) = YA(r,STEPS); acf = autocorr(Y); for k = (1:10) YACA(r,k)=acf(k); end plot(Y) hold on end YR=10; xlim([0,STEPS]); ylim([-YR,YR]) title('ARMA(0,4) Example') hold off pause autocorr(Y) pause autocorr(RN) mean(Y) var(Y,1)

acf = Columns 1 through 6; 1.0000 0.3093 0.4423 0.2864 0.2728 -0.0068 mean(Y); var(Y,1); -0.0318; 6.7238

21

Run again with more steps, 50,000 acf = Columns 1 through 6; 1.0000 0.2827 0.4271 0.2758 0.2724 -0.0109 mean(Y); var(Y,1); ans = -0.0232; ans = 6.9709 Check var (RN,1) = 1.0013 mean(RN) = -0.0047 acf (RN) acr = 1.0000 0.0010 0.0039 -0.0043 -0.0121 -0.0062 What values should we expect? Mean = 0: Variance = 5? ρ(1) = 4/5, ρ(2) = 3/5 ???? Try a MA(3) run? And I eventually found my typo, With the typo, Y(s)=beta1*RN(s-1)+beta2*RN(s-2) + beta3*RN(s-4)+beta4*RN(s-4)+RN(s); So E[Y(s)2] = 1+ beta12 + beta22 +(beta3 + beta4)2 = 7 and explains things!

22

ARMA

in the MA case θ(B) = 1 so the issue is to determine ϕ-1(B)

-------------------------------------------------------

23

ARMA from CM 2009

24

-------------------------------------------------------

25

A simple ARMA(2,1) example: (1 – 0.75B + 0.125B2)xt = wt – 0.5 wt-1. Roots of Characteristic Equation, z = 2, 4, so stationary. xt = 0.75xt-1 – 0.125xt-2 -0.5wt-1 + wt. Compute γ(0), γ(1) etc., and run ARMA(2,1) code. STEPS = 50,000. >> var(Y) = 3.5136 >> mean(Y) = 0.0104

Look at equations for variance and autocovariance…… they get complicated.