Statistics 203: Introduction to Regression and Analysis...

- p. 1/15

Statistics 203: Introduction to Regressionand Analysis of Variance

Time Series: Brief Introduction

Jonathan Taylor

● Today’s class

● Modelling correlation

● Other models of correlation

● Autoregressive models

● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2

● Moving average &

ARMA(p, q) models

● ARMA(2, 4)

● Stationary time series

● Estimating autocovariance /

correlation● Estimating power spectrum

● Diagnostics

- p. 2/15

Today’s class

■ Models for time-correlated noise.■ Stationary time series.■ ARMA models.■ Autocovariance, power spectrum.■ Diagnostics.

http://www-stat.stanford.edu/~jtaylo/courses/stats203/R

● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 3/15

Modelling correlation

■ In the mixed effects model

Y = Xβ + Zγ + ε

with ε ∼ N(0, σ2I) and γ ∼ N(0, D) we were essentiallysaying

Y ∼ N(Xβ, ZDZt + σ2I)

■ We then estimated D from the data (more precisely, R doesthis for us).

■ We can impose structure on D if necessary. For example, intwo-way random effects ANOVA, we assumed thatαi, βj , (αβ)ij were independent mean zero normal randomvariables.

■ In summary, a mixed effect model can be thought of asmodelling the correlation in the errors of Y coming from“sampling from a population.”


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 4/15

Other models of correlation

■ Not all correlations come from sampling.■ Another common source is correlation in time.■ Example: imagine modelling monthly temperature in a given

location over many years.◆ Yt = µt%12 + εt, 1 ≤ t ≤ T◆ Clearly, µ will vary smoothly as a function of t, but there

will also be correlation in εt due to “weather systems” thatlast more than one day.

◆ To estimate µ “optimally” and (especially to) makeinferences about µ we should take these correlations intoaccount.

■ Time series models are models of such (auto)correlation.Good references: Priestley, “Spectral Theory and TimeSeries”; Brockwell and Davis, “Introduction to Time Seriesand Forecasting.”

■ Nottingham temperature example.■ Today we will just talk about time series in general.


http://www-stat.stanford.edu/~jtaylo/courses/stats203/R/time.series/temperature

● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 5/15

Autoregressive models

■ Simplest stationary “auto”correlation

εt = α · εt−1 + ηt

where η ∼ N(0, σ2I) are i.i.d. Normal random variables,|α| < 1.

■ This is called an auto-regressive process: “auto” because εis like a regression of ε on its past.

■ It is called AR(1) because it only goes 1 time point into thepast.

■ Covariance / correlation function

Cov(εt, εt+j) =σ2α|j|

1 − α2, Cor(εt, εt+j) = α|j|

■ Model is “stationary” because Cov(εt, εt+j) depends only on|j|.


- p. 6/15

AR(1), α = 0.95

Time

ar1.

sim

1

0 50 100 150 200

−6

−4

−2

02

46

8


- p. 7/15

AR(1), α = 0.5

Time

ar1.

sim

2

0 50 100 150 200

−2

−1

01

2


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 8/15

AR(k) models

■ The AR(1) model can be easily generalized to the AR(p)model:

εt =

p∑

j=1

αjεt−j + ηt

where η ∼ N(0, σ2I) are i.i.d. Normal random variables.■ Condition on α’s: all roots of the (complex) polynomial

φα(z) = 1 −

p∑

j=1

αjzj

are within the unit disc in the complex plane.


- p. 9/15

AR(2), α1 = 0.9, α2 = −0.2

Time

ar2.

sim

0 50 100 150 200

−4

−2

02

4


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 10/15

Moving average & ARMA(p, q) models

■ MA(q) is another stationary model:

εt =

q∑

j=0

βjηt−q

where η ∼ N(0, σ2I) are i.i.d. Normal random variables.■ No conditions on β’s – this is always stationary.■ ARMA(p, q) model:

εt =

p∑

l=1

αlXt−l +

q∑

j=0

βjηt−q

where η ∼ N(0, σ2I) are i.i.d. Normal random variables.


- p. 11/15

ARMA(2, 4)

Time

arm

a.si

m

0 50 100 150 200

−20

020

40


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 12/15

Stationary time series

■ In general a (Normally distributed) time series (εt) isstationary if

Cov(εt, εt+j) = R(|j|)

for some “covariance” function R.■ If errors are not normally distributed then the process is

called weakly stationary, or stationary in mean-square.■ The function R(t) can generally be expressed as the Fourier

transform of a spectral density

R(t) =1

2π

∫ π

−π

eitωfR(ω) dω

where f is called the “spectral” density of the process.■ Conversely

fR(t) =∑

t

e−itωR(t)

■ The function fR is sometimes called the “power spectrum” ofε.


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 13/15

Estimating autocovariance / correlation

■ Natural estimate of covariance function for t ≥ 0 based onobserving (ε1, . . . , εn)

R(t) =1

n

n−t∑

j=1

(εj+t − ε)(εj − ε).

■ Estimate of correlation function

Cor(t) =R(t)

R(0).


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 14/15

Estimating power spectrum

■ Estimate of power spectrum based on observing (ε1, . . . , εn)is called the periodogram, the discrete Fourier transform of ε

I(ω) =

∣∣∑n

t=1 e−iωtεt

∣∣2

n

■ In factI(ω) =

∑

t

R(t)e−iωt

i.e. it is the Fourier transform of R(t).■ It is customary to use a smoothed periodogram as an

estimate of fR

fR(ω) =

∫Kh((λ − ω)/h)I(λ) dλ.

for some kernel Kh.■ If ε’s are i.i.d. (hence stationary), then

E(I(ω)) = Var(ε).


● Today’s class




● AR(1), α = 0.95

● AR(1), α = 0.5

● AR(k) models

● AR(2), α1 =

0.9, α2 = −0.2


ARMA(p, q) models

● ARMA(2, 4)




● Diagnostics

- p. 15/15

Diagnostics

■ Suppose we fit an ARMA(p, q) model to observations(ε1, . . . , εn): how can we tell if the fit is “good”?

■ How do we do this? By residuals of course. In an AR(p)model, for instance, define

ηt = Xt −

p∑

j=1

αjXt−j.

■ These should look like an i.i.d. sequence, at least roughly.■ Can plot residuals themselves, autocorrelation function of

residuals, and cumulative periodogram.


Statistics 203: Introduction to Regression and Analysis...

Documents

Transcript of Statistics 203: Introduction to Regression and Analysis...