STA 6857---Difference Equations & Autocorrelation and ...

28
STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4)

Transcript of STA 6857---Difference Equations & Autocorrelation and ...

Page 1: STA 6857---Difference Equations & Autocorrelation and ...

STA 6857—Difference Equations & Autocorrelation andPartial Autocorrelation Functions (§3.3 & 3.4)

Page 2: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Outline

1 Difference Equations

2 Autocorrelation Function

3 Homework 3c

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 2/ 23

Page 3: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Outline

1 Difference Equations

2 Autocorrelation Function

3 Homework 3c

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 3/ 23

Page 4: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

ACF of AR(1)

From last time, we found the ACF of the AR(1) model given by

xt = φxt−1 + wt

to be ρ(h) = φ|h| which satisfies the difference equation

un = φun−1

with initial condition u0 = 1.

This difference equation is classified as homogeneous of order 1.

In the following, we will consider homogeneous difference equations of ageneral order p.

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 4/ 23

Page 5: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Difference Equation

Definition (Difference Equation)An equation which expresses a value of a sequence as a function of the otherterms in the sequence is called a difference equation.

ExampleConsider the difference equation

an = an−1 + an−2

where a0 = 0 and a1 = 1. The solution to this is the famed Fibonacci numbersgiven by

an =(1 +

√5)n − (1−

√5)n

2n√

5

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 5/ 23

Page 6: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Difference Equations are Fun!

ExampleConsider the difference equation

an = an−1 + n

where a0 = 0.What’s the solution to this difference equation?Prize awarded to the first correct respondent!!

Answer:

an = 1 + 2 + · · ·+ n =n(n + 1)

2Winner receives ...

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 6/ 23

Page 7: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Difference Equations are Fun!

ExampleConsider the difference equation

an = an−1 + n

where a0 = 0.What’s the solution to this difference equation?Prize awarded to the first correct respondent!!Answer:

an = 1 + 2 + · · ·+ n =n(n + 1)

2

Winner receives ...

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 6/ 23

Page 8: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Difference Equations are Fun!

ExampleConsider the difference equation

an = an−1 + n

where a0 = 0.What’s the solution to this difference equation?Prize awarded to the first correct respondent!!Answer:

an = 1 + 2 + · · ·+ n =n(n + 1)

2Winner receives ...

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 6/ 23

Page 9: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Second-Order Homogeneous Difference Equation

Solving difference equations often similar to solving differential equations.We start with a general second-order homogeneous difference equation given by

un = α1un−1 + α2un−2 (α2 6= 0)

which can be written as

(1− α1B− α2B2)︸ ︷︷ ︸α(B)

un = 0

Here α(z) is called the characteristic polynomial.

Since deg(α(z)) = 2, α(z) will have two complex roots by the fundamentaltheorem of algebra.

Call these two roots z1 and z2.

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 7/ 23

Page 10: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Two Cases

General second order difference equation:

un = α1un−1 + α2un−2 (α2 6= 0) (?)

Case 1: z1 6= z2 In this case, the solution to (?) is

un = c1

(1z1

)n

+ c2

(1z2

)n

Case 2: z1 = z2 In this case, the solution to (?) is

un =(

1z1

)n

(c1 + c2n)

In each case, the constants c1 and c2 are determined from the initial conditions.Feel free to check these solutions by plugging into (?)!

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 8/ 23

Page 11: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Back to the Fibonacci Numbers

an − an−1 − an−2 = 0⇒ (1− B− B2)︸ ︷︷ ︸α(B)

an = 0

The characteristic equation is α(z) = 1− z− z2 which has two roots

z1 = −√

5 + 12

and z2 =√

5− 12

Notice the following interesting relationships:

1z1

= − 21 +√

(1−√

51−√

5

)=−2 + 2

√5

1− 5=

1−√

52

= −z2

This follows from the general fact that if z1 and z2 are roots of the quadraticequation ax2 + bx + c, then z1z2 = c

a (and also z1 + z2 = −ba ).

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 9/ 23

Page 12: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Back to the Fibonacci Numbers

an − an−1 − an−2 = 0⇒ (1− B− B2)︸ ︷︷ ︸α(B)

an = 0

The characteristic equation is α(z) = 1− z− z2 which has two roots

z1 = −√

5 + 12

and z2 =√

5− 12

Notice the following interesting relationships:

1z1

= − 21 +√

(1−√

51−√

5

)=−2 + 2

√5

1− 5=

1−√

52

= −z2

This follows from the general fact that if z1 and z2 are roots of the quadraticequation ax2 + bx + c, then z1z2 = c

a (and also z1 + z2 = −ba ).

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 9/ 23

Page 13: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Fibonacci Recursion – Solved

Utilizing the general solution of a second-order homogeneous differenceequation with two distinct roots gives

an = c1

(1z1

)n

+ c2

(1z2

)n

= c1(−z2)n + c2(−z1)n

a0 = c1 + c2 = 0⇒ c2 = −c1

a1 = −c1z2 − c2z1 = 1

⇒ −c1z2 + c1z1 = 1

⇒ c1(z1 − z2) = 1

⇒ c1 =1

z1 − z2

=−1√

5Note that

z1 − z2 = −√

5 + 12

−√

5− 12

= −√

5

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 10/ 23

Page 14: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Fibonacci Recursion – Solved

Utilizing the general solution of a second-order homogeneous differenceequation with two distinct roots gives

an = c1

(1z1

)n

+ c2

(1z2

)n

= c1(−z2)n + c2(−z1)n

a0 = c1 + c2 = 0⇒ c2 = −c1

a1 = −c1z2 − c2z1 = 1

⇒ −c1z2 + c1z1 = 1

⇒ c1(z1 − z2) = 1

⇒ c1 =1

z1 − z2

=−1√

5

Note that

z1 − z2 = −√

5 + 12

−√

5− 12

= −√

5

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 10/ 23

Page 15: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Fibonacci Recursion – Solved

Utilizing the general solution of a second-order homogeneous differenceequation with two distinct roots gives

an = c1

(1z1

)n

+ c2

(1z2

)n

= c1(−z2)n + c2(−z1)n

a0 = c1 + c2 = 0⇒ c2 = −c1

a1 = −c1z2 − c2z1 = 1

⇒ −c1z2 + c1z1 = 1

⇒ c1(z1 − z2) = 1

⇒ c1 =1

z1 − z2=−1√

5Note that

z1 − z2 = −√

5 + 12

−√

5− 12

= −√

5

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 10/ 23

Page 16: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Fibonacci Solution Simplified

an = c1(−z2)n + c2(−z1)n

= c1(−z2)n − c1(−z1)n

=(−z1)n − (−z2)n

√5

=

(1+√

52

)n−(

1−√

52

)n

√5

=(1 +

√5)n − (1−

√5)n

2n√

5

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 11/ 23

Page 17: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Homogeneous Difference Equation of Order pNow consider the general equation:

un − α1un−1 − · · · − αpun−p = 0 (αp 6= 0)

The associated characteristic equation is

α(z) = 1− α1z− · · · − αpzp

Suppose α(z) has r distinct roots: z1 with multiplicity m1, z2 withmultiplicity m2,. . . , and zr with multiplicity mr. (So thatm1 + m2 + · · ·+ mr = p.)Here are the roots of α(z) listed out:

z1, z1, . . . , z1︸ ︷︷ ︸m1

, z2, z2, . . . , z2︸ ︷︷ ︸m2

, . . . , zr, zr, . . . , zr︸ ︷︷ ︸mr

Then the solution is

un = z−n1 P1(n) + z−n

2 P2(n) + · · ·+ z−nr Pr(n)

where Pj(n) is a polynomial in n of degree mj − 1 whose coefficients arededuced from the initial conditions.

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 12/ 23

Page 18: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Our Applications of Difference Equations

Determine the causal representation of an ARMA(p,q) — cf. Example 3.10(p.102–103)

Compute the ACF of AR(p) and ARMA(p,q) model...next section.

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 13/ 23

Page 19: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Outline

1 Difference Equations

2 Autocorrelation Function

3 Homework 3c

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 14/ 23

Page 20: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

ACF of an MA(q)

We computed the ACF of an MA(1) last time and also computed theautocovariance function for a general MA(∞) last Friday. Brushing over thecomputations, the ACF of an MA(q) with representation

xt = µ+ wt + θ1wt−1 + θ2wt−2 + · · ·+ θqwt−q

is given as

ρ(h) =

∑q−h

j=0 θjθj+h

1+θ21+···+θ2

q1 ≤ h ≤ q

0 h > q

No statistical significance in the correlelogram (ACF plot) after lag q indicatesthe process may be an MA(q) process.

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 15/ 23

Page 21: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Autocovariance function of an ARMA(p,q)

The computation of the autocovariance function of the ARMA modelφ(B)xt = θ(B)wt begins as

γ(h) = cov(xt+h, xt)

= E

p∑j=1

φjxt+h−j +q∑

j=0

θjwt+h−j

xt

=

p∑j=1

φjγ(h− j) +q∑

j=0

θj E(wt+h−jxt).

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 16/ 23

Page 22: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Autocovariance function of a Causal ARMA(p,q)

For a causal ARMA(p,q) model, we have

xt =∞∑

j=0

ψjwt−j

so that

E(wt+h−jxt) = E

[wt+h−j

( ∞∑k=0

ψkwt−k

)]= ψj−hσ

2

Therefore we can continue the calculation of the autocovariance function as

γ(h) =p∑

j=1

φjγ(h− j) +q∑

j=0

θj E(wt+h−jxt)

=p∑

j=1

φjγ(h− j) + σ2q∑

j=h

θjψj−h

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 17/ 23

Page 23: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Autocovariance Recurrence

Note that θj = 0 when j > q + 1, so we have the following recurrencerelationship

γ(h)− φ1γ(h− 1)− · · · − φp(h− p) = 0, h ≥ max(p, q + 1)

with initial conditions

γ(h)−p∑

j=1

φjγ(h− j) = σ2q∑

j=h

θjψj−h, 0 ≤ h < max(p, q + 1)

Dividing through by γ(0) gives a similar recursion on ρ(h) = γ(h)/γ(0).

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 18/ 23

Page 24: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

ACF of Causal ARMA(1,1)

Example (ACF of Causal ARMA(1,1))Start with the causal ARMA(1,1) given by xt = φxt−1 + θwt−1 + wt.From the previous slide, the following recursion holds

γ(h)− φγ(h− 1) = 0

so the general solution is γ(h) = cφ|h|.

We use the initial conditions from the previous slide to determine theconstant c.

To continue, we need to know the values of ψ0 and ψ1.

Recall ψ(z)φ(z) = θ(z).Multiplying out this equation gives the relationships ψ0 = 1 andψ1 = θ1 + φ1φ0. (Refer Example 3.10 pp.102-103 for more details.)

This gives two equations and two unknowns (γ(0) and γ(1))...

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 19/ 23

Page 25: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

ACF of Causal ARMA(1,1) (cont.)

Example ((cont.))...

γ(0) = φγ(1) + σ2(1 + θφ+ θ2)

γ(1) = φγ(0) + σ2θ

we can continue to solve for these unknowns to produce

γ(h) = σ2 (1 + θφ)(φ+ θ)1− φ2 φ|h|−1,

and for h ≥ 1,

ρ(h) =(1 + θφ)(φ+ θ)

1 + 2θφ+ θ2 φ|h|−1

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 20/ 23

Page 26: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Outline

1 Difference Equations

2 Autocorrelation Function

3 Homework 3c

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 21/ 23

Page 27: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Textbook Reading

Read the following section from the textbook§3.5 (Forecasting)

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 22/ 23

Page 28: STA 6857---Difference Equations & Autocorrelation and ...

Difference Equations Autocorrelation Function Homework 3c

Textbook Problems

Do the following exercise from the textbook3.6 (a)

3.8

Arthur Berg STA 6857—Difference Equations & Autocorrelation and Partial Autocorrelation Functions (§3.3 & 3.4) 23/ 23