Examples of Stationary Processes

1) Strong Sense White Noise: A process ǫtis strong sense white noise if ǫt is iid with mean

0 and finite variance σ2.

2) Weak Sense (or second order or wide

sense) White Noise: ǫt is second order sta-

tionary with

E(ǫt) = 0

and

Cov(ǫt, ǫs) =

σ2 s = t

0 s 6= t

In this course: ǫt denotes white noise; σ2 de-

notes variance of ǫt. Use subscripts for vari-

ances of other things.

16

Example Graphics:

White noise: iid N(0,1) data

IID N(0,1)

0 200 400 600 800 1000

White noise: Xt = ǫt · · · ǫt+9

Wide Sense White Noise

0 200 400 600 800 1000

17

2) Moving Averages: if ǫt is white noise then

Xt = (ǫt + ǫt−1)/2 is stationary. (If you use

second order white noise you get second order

stationary. If the white noise is iid you get

strict stationarity.)

Example proof: E(Xt) =[

E(ǫt) + E(ǫt−1)]

/2 =

0 which is constant as required. Moreover:

Cov(Xt, Xs) is

Var(ǫt)+Var(ǫt−1)4 s = t

14Cov(ǫt + ǫt−1, ǫt+1 + ǫt) s = t+ 114Cov(ǫt + ǫt−1, ǫt+2 + ǫt+1) s = t+ 2

...

Most of these covariances are 0. For instance

Cov(ǫt + ǫt−1, ǫt+2 + ǫt+1) =

Cov(ǫt, ǫt+2) + Cov(ǫt, ǫt+1)

+ Cov(ǫt−1, ǫt+2) + Cov(ǫt−1, ǫt+1) = 0

because the ǫs are uncorrelated by assumption.

18

The only non-zero covariances occur for s = t

and s = t± 1. Since Cov(ǫt, ǫt) = σ2 we get

Cov(Xt, Xs) =

σ2

2 s = t

σ2

4 |s− t| = 1

0 otherwise

Notice that this depends only on |s− t| so that

the process is stationary.

The proof that X is strictly stationary when

the ǫs are iid is in your homework; it is quite

different.

19

Example Graphics:

Xt = (ǫt + ǫt−1)/2

MA(1)Process

0 200 400 600 800 1000

Xt = ǫt + 6ǫt−1 + 15ǫt−2 + 20ǫt−3

+15ǫt−4 + 6ǫt−5 + ǫt−6

MA(6) Process

0 200 400 600 800 1000

20

The trajectory of X can be made quite smooth

(compared to that of white noise) by averaging

over many ǫs.

3) Autoregressive Processes:

AR(1) process X: process satisfying equations:

Xt = µ+ ρ(Xt−1 − µ) + ǫt (1)

where ǫ is white noise. If Xt is second order

stationary with E(Xt) = θ, say, then take ex-

pected values of (1) to get

θ = µ+ ρ(θ − µ)

which we solve to get

θ(1 − ρ) = µ(1 − ρ) .

Thus either ρ = 1 (later – X not stationary)

or θ = µ. Calculate variances:

Var(Xt) = Var(µ+ ρ(Xt−1 − µ) + ǫt)

= Var(ǫt) + 2ρCov(Xt−1, ǫt)

+ ρ2Var(Xt−1)

21

Now assume that the meaning of (1) is that

ǫt is uncorrelated with Xt−1, Xt−2, · · · .

Strictly stationary case: imagining somehow

Xt−1 is built up out of past values of ǫs which

are independent of ǫt.

Weakly stationary case: imagining that Xt−1 is

actually a linear function of these past values.

Either case: Cov(Xt−1, ǫt) = 0.

If X is stationary: Var(Xt) = Var(Xt−1) ≡ σ2X

so

σ2X = σ2 + ρ2σ2

X

whose solution is

σ2X =

σ2

1 − ρ2

22

Notice that this variance is negative or unde-

fined unless |ρ| < 1. There is no stationary

process satisfying (1) for |ρ| ≥ 1.

Now for |ρ| < 1 how is Xt determined from the

ǫs? (We want to solve the equations (1) to get

an explicit formula for Xt.) The case µ = 0 is

notationally simpler. We get

Xt = ǫt + ρXt−1

= ǫt + ρ(ǫt−1 + ρXt−2)...

= ǫt + ρǫt−1 + · · · + ρk−1ǫt−k+1

+ ρkXt−k

Since |ρ| < 1 it seems reasonable to suppose

that ρkXt−k → 0 and for a stationary series

X this is true in the appropriate mathematical

sense. This leads to taking the limit as k → ∞

to get

Xt =∞∑

j=0

ρjǫt−j .

23

Claim: if ǫ is a weakly stationary series then

Xt =∑∞j=0 ρ

jǫt−j converges (technically it con-

verges in mean square) and is a second order

stationary solution to the equation (1).

If ǫ is a strictly stationary process then under

some weak assumptions about how heavy the

tails of ǫ are Xt =∑∞j=0 ρ

jǫt−j converges almost

surely and is a strongly stationary solution of

(1).

In fact; if . . . , a−1, a0, a1, a2, . . . are constants

such that∑

a2j < ∞ and ǫ is weak sense white

noise (respectively strong sense white noise with

finite variance) then

Xt =∞∑

j=−∞

ajǫt−j

is weakly stationary (respectively strongly sta-

tionary with finite variance). In this case we

call X a linear filter of ǫ.

24

Example Graphics:

AR(1)Process: Rho=0.99

0 200 400 600 800 1000

AR(1) Process: Rho=0.5

0 200 400 600 800 1000

25

Motivation of the jargon “filter” comes from

physics.

Consider an electric circuit with a resistance R

in series with a capacitance C.

Apply “input” voltage U(t) across the two el-

ements.

Measure voltage drop across capacitor.

Call this voltage drop “output” voltage; denote

output voltage by Xt.

26

The relevant physical rules are these:

1. The total voltage drop around the circuit is

0. This drop is −U(t) plus the voltage drop

across the resistor plus X(t). (The nega-

tive sign is a convention; the input voltage

is not a “drop”.)

2. Voltage drop across resistor is Ri(t) where

i is current flowing in circuit.

3. If the capacitor starts off with no charge

on its plates then the voltage drop across

its plates at time t is

X(t) =

∫ t0 i(s) ds

C

These rules give

U(t) = Ri(t) +

∫ t0 i(s) ds

C

27

Differentiate the definition of X to get

X ′(t) = i(t)/C

so that

U(t) = RCX ′(t) +X(t) .

Multiply by et/RC/RC to see that

et/RCU(t)

RC=

(

et/RCX(t))′

whose solution, remembering X(0) = 0, is ob-

tained by integrating from 0 to s to get

es/RCX(s) =1

RC

∫ s

0et/RCU(t) dt

leading to

X(s) =1

RC

∫ s

0e(t−s)/RCU(t) dt

=1

RC

∫ s

0e−u/RCU(s− u) du

This formula is the integral equivalent of our

definition of filter and shows X = filter(U).

28

Defn: If {ǫt} is a white noise series and µ andb0, . . . , bp are constants then

Xt = µ+ b0ǫt + b1ǫt−1 + · · · + bpǫt−p

is a moving average of order p; write MA(p).

Defn: A process X is an autoregression oforder p (written AR(p)) if

Xt =p

∑

1

ajXt−j + ǫt.

Defn: Process X is an ARMA(p, q) (mixedautoregressive of order p and moving averageof order q) if for some white noise ǫ:

φ(B)X = ψ(B)ǫ

φ(B) = I −p

∑

1

ajBj

and

ψ(B) = I −p

∑

1

bjBj

Problems: existence, stationarity, estimation,etc.

29

Other Stationary Processes:

Periodic processes: Suppose Z1 and Z2 are in-

dependent N(0, σ2) random variables and that

ω is a constant. Then

Xt = Z1 cos(ωt) + Z2 sin(ωt)

has mean 0 and

Cov(Xt, Xt+h) = σ2 [cos(ωt) cos(ω(t+ h))

+sin(ωt) sin(ω(t+ h))]

= σ2 cos(ωh)

Since X is Gaussian we find that X is second

order and strictly stationary. In fact (see your

homework) You can write

Xt = R sin(ωt+ Φ)

where R and Φ are suitable random variables

so that the trajectory of X is just a sine wave.

30

Poisson shot noise processes:

Poisson process is a process N(A) indexed by

subsets A of R such that each N(A) has a Pois-

son distribution with parameter λlength(A) and

if A1, . . . Ap are any non-overlapping subsets of

R then N(A1), . . . , N(Ap) are independent. We

often use N(t) for N([0, t]).

Shot noise process: X(t) = 1 at those t where

there is a jump in N and 0 elsewhere; X is

stationary.

If g a function defined on [0,∞) and decreasing

sufficiently quickly to 0 (like say g(x) = e−x)then the process

Y (t) =∑

g(t− τ)1(X(τ) = 1)1(τ ≤ t)

is stationary.

Y jumps every time t passes a jump in Poisson

process; otherwise follows trajectory of sum of

several copies of g (shifted around in time).

We commonly write

Y (t) =

∫ ∞

0g(t− τ)dN(τ)

31

ARCH Processes: (Autoregressive Conditional

Heteroscedastic)

Defn: Mean 0 process X is ARCH(p) if

Var(Xt+1|Xt,Xt−1, · · · ) ∼ N(0, Ht)

where

Ht = α0 +p

∑

1

αiX2t+1−i

GARCH Processes: (Generalized Autoregres-

sive Conditional Heteroscedastic)

Defn: The process X is GARCH(p, q) if X has

mean 0 and

Var(Xt+1|Xt,Xt−1, · · · ) ∼ N(0, Ht)

where

Ht = α0 +p

∑

1

αiX2t+1−i +

q∑

1

βjHt−j

Used to model series with patches of high and

low variability.

32

Markov Chains

Defn: A transition kernel is a function P(A, x)

which is, for each x in a set X (the state space),

a probability on X .

Defn: A sequence Xt is Markov (with station-

ary transitions P) if

P(Xt+1 ∈ A|Xt, Xt−1, · · · ) = P(A,Xt)

That is, the conditional distribution of Xt+1

given all history to time t depends only on value

of Xt.

Fact: under some conditions as t→ ∞ Xt, Xt+1, . . .

becomes stationary.

Fact: under similar conditions can give X0 a

distribution (called stationary initial distribu-

tion) so that X is a strictly stationary process.

33