Chapter 1
Basics
1.1 Definition
A time series (or stochastic process) is a function Xpt, ωq such that for
each fixed t, Xpt, ωq is a random variable [denoted by Xtpωq]. For a fixed ω,
Xpt, ωq is simply a function of t and is called a realization of the stochastic
process.
$''&''%Time Domain Approach
Frequency Domain Approach
$''&''%Continuous Time
Discrete Time
11
12 CHAPTER 1. BASICS
1.2 Characterization of a time series
A time series is characterized by the joint distribution function of any
subset Xt1 , � � � , Xtn that is FXt1 , ��� , Xtnpxt1 , � � � , xtnq.
Using the joint distribution function, define
µt � EpXtq
γt,s � CovpXt, Xsq, provided that they exist.
The first two monents, {µt}, {γt,s} completely characterize a Gaussian process.
FXt1 , ��� , Xtnis in general, very difficult to analyze. In particular, esti-
mation of {µt}, {γt,s} appears to be impossible unless a number of different
realizations, i.e. repeated observations are available.
1.3 Stationarity
Assume FXt1 ,��� ,Xtnis invariant under transformation of the time indices,
that is
FXt1�h, ��� , Xtn�hpxt1 , � � � , xtnq � FXt1 , ��� ,Xtn
pxt1 , � � � , xtnq p1q
for all sets of indices (t1, � � � , tn). This is called the strict stationarity.
1.3. STATIONARITY 13
Under this assumption, the joint distribution function depends only on
the distance between the elements in the index set.
If tXtu is strictly stationary and E|Xt| 8, then
EpXtq � µ, @t p2q
CovpXt, Xsq � γ|t�s|, @t, s p3q
tXtu is said to be weakly stationary (or ”covariance stationary” or sim-
ply, ”stationary”) if (2) and (3) holds.
REMARKS
1) (Weak) stationarity does not imply strict stationarity. Nor does
strict stationarity imply (weak) stationarity. e.g. A strict
stationary process may not possess finite moments (e.g. Cauchy).
2) For a Gaussian process, weak and strict stationarity are equivalent.
14 CHAPTER 1. BASICS
µ and γ|t�s| can be estimated by
pµ � sX � 1T
T
t�1Xt
pγh � ch � 1T
T
t�h�1pXt � sXqpXt�h � sXq, h � 0, 1, � � � .
If the process is also ergodic (average asymptotic independence), sX and
ch are consistent estimators of µ and γh.
1.4 Autocovariance and Autocorrelation Func-
tions
The sequence tγhu viewed as a function of h is called the autocovariance function.
The autocorrelation function is defined by
ρh � γhγ0,
note ρ0 � 1
Example 1. (White noise process)
tXtu, Xt � iid p0, σ2q, 0 σ2 8.
µ � 0, γh �
$''&''%σ2 if h � 0,
0 otherwise.
1.4. AUTOCOVARIANCE AND AUTOCORRELATION FUNCTIONS 15
Example 2. (MA(1) process)
Let tεtu be a white noise process with finite variance σ2. Let
Xt � εt � θεt�1.
Then
µ � EpXtq � 0
γ0 � Epεt � θεt�1q2 � Epε2t q � θ2Epε2
t�1q
� p1� θ2qσ2
γh � Erpεt � θεt�1qpεt�h � θεt�h�1qs
�
$''&''%θσ2 if |h| � 1,
0 if |h| ¥ 2.
Therefore,
ρh �
$''''''&''''''%1 if h � 0,θ
1� θ2 if |h| � 1,
0 otherwise.
16 CHAPTER 1. BASICS
Suppose θ � 0.6. Then ρ1 � 0.61� 0.62 � 0.44.
1 2 3 4 5
1
0.5
0 h
ρh
ρh � ρphq : the autocorrelation function.
Sample mean of tXtuTt�1 � sXT
Sample variance = c0 pγ0
Sample autocovariance = ch, h � 1, 2, � � � pγhSample autocorrelation = rhpor pρhq � ch
c0, h � 1, 2, � � �
Note r0 � 1.
1.5. LAG OPERATOR 17
A plot of rh against h � 0, 1, � � � is called correlogram.
1 2 3 4 5
1
0.5
0 h
rh
The sample autocorrelations are estimates of the corresponding theo-
retical autocorrelations and are therefore subject to sampling errors.
1.5 Lag Operator
The operator L (sometimes denoted by B) is defined by
LXt � Xt�1.
18 CHAPTER 1. BASICS
Formally, L operates on the whole sequence.
L :
����������������
...
Xt�1
Xt
Xt�1
...
����������������ÝÑ
����������������
...
Xt�2
Xt�1
Xt
...
����������������The lead operator L�1 (sometimes denoted by F ) is defined by
L�1Xt � Xt�1.
Successive application of L yields LhXt � Xt�h, h � 1, 2, � � �
We define L0Xt � Xt.
The lag operator is a linear operator :
LpcXtq � cLXt
LpXt � Ytq � LXt � LYt
and can be manipulated like a usual algebraic quantity.
For example, suppose
yt � φyt�1 � εt, and |φ| 1.
1.6. GENERAL LINEAR PROCESS 19
Then, p1� φLqyt � εt
So, yt � εt1� φL
�8
j�0pφLqjεt
∆ � 1� L is called the difference operator.
∆2yt � p1� Lq2yt � p1� 2L� L2qyt
= yt � 2yt�1 � yt�2
∆2yt � ∆p∆ytq � ∆pyt � yt�1q
= pyt � yt�1q � pyt�1 � yt�2q � yt � 2yt�1 � yt�2
1.6 General Linear Process
yt � εt � ψ1εt�1 � ψ2εt�2 � � � �
� ψpLqεt ÐÝ linear process
where εt � iid p0, σ2q and
ψpLq � 1� ψ1L� ψ2L2 � � � �
polynomial in lag operators ψpLq is sometimes called transfer function
20 CHAPTER 1. BASICS
A time series tytu can be viewed as the result of applying a backward (linear)
filter to a white noise process.
εtinputÝÑ
linear filter
ψpLqoutputÝÑ Yt � ψpLqεt
The sequence tψj : j � 0, 1, � � � u can be finite or infinite. If it is finite
of order q, we obtain MApqq process. This is clearly a stationary process. If
tψju is infinite, we usually assume it is absolutely summable. i.e.8
j�0|ψj| 8.
Then the resulting process is stationary.
To see this,
µ � 0 �8
j�0ψj � 0
γ0 � σ28
j�0ψ2j σ2
� 8
j�0|ψj|
2
8
γh � σ28
j�0ψjψj�h σ2
8
j�0|ψj||ψj�h| σ2
� 8
j�0|ψj|
2
8
1.7. AUTOREGRESSIVE PROCESS 21
The stationary condition is embodied in the condition that ψpzq must
converge for |z| ¤ 1, i.e. for z on or within the unit circle.
Note that absolute summability of tψju is sufficient but not necessary
for stationarity.
7
8
j�0ψjz
j 8
j�0|ψj| 8
1.7 Autoregressive Process
The process yt defined by
yt � φ1yt�1 � � � � � φpyt�p � εt p�q
is called a p-th order autoregressive process and is denoted by
yt � ARppq
The equation p�q is sometimes called a stochastic difference equation.
22 CHAPTER 1. BASICS
1.7.1 First - Order Autoregressive Process
ARp1q is given by
yt � φyt�1 � εt.
By successive substitution,
yt � εt � φyt�1
� εt � φεt�1 � φ2yt�2
� εt � φεt�1 � φ2εt�2 � φ3yt�3
�J�1
j�0φjεt�j � φJyt�J ,
implying Epyt|yt�Jq � φJyt�J
If |φ| ¥ 1, the value of yt�J can affect the prediction of future yt, no
matter how far ahead.
If |φ| 1,
yt � limJÑ8
J�1
j�0φjεt�j � lim
JÑ8φJyt�J
�8
j�0φjεt�j.
Note that8
j�0|φj| �
8
j�0|φ|j � 1
1� |φ| 8, if |φ| 1,
1.7. AUTOREGRESSIVE PROCESS 23
so, tφju is absolutely summable and tytu is a linear process.
ψpLq � p1� φLq�1 �8
j�0φjLj, or ψj � φj
Now,
Epy2t q � Epφ2y2
t�1 � ε2t � 2φyt�1εtq
γ0 � φ2γ0 � σ2
because Epyt�1εtq
� ErEpyt�1εt|yt�1qs
� Eryt�1Epεt|yt�1qs
� Epyt�1 � 0q � 0
Therefore, γ0 � σ2
1� φ2 8, if |φ| 1.
1.7.2 Second - Order Autoregressive Process
ARp2q is given by
yt � φ1yt�1 � φ2yt�2 � εt
or
φpLqyt � εt,
where, φpLq � 1� φ1L� φ2L2.
24 CHAPTER 1. BASICS
Now, suppose φpzq can be written as
φpzq � p1� λ1zqp1� λ2zq.
Then
ψpzq � φ�1pzq � 1p1� λ1zqp1� λ2zq �
K1
1� λ1z� K2
1� λ2z,
where K1 � λ1
λ1 � λ2and K2 � �λ2
λ1 � λ2.
Therefore, ψpzq converges for |z| ¤ 1, iff |λ1| 1 and |λ2| 1.
In other words, for ARp2q process to be stationary, the roots of
φpzq � 1� φ1z � φ2z2 must lie outside the unit circle.
Note that φp0q � 1 ¡ 0. Let m1 and m2 be the roots. The neces-
sary and sufficient condition for |m1| ¡ 1 and |m2| ¡ 1 are
|m1m2| ����� 1φ2
���� ¡ 1 ðñ |φ2| 1
φp1q � 1� φ1 � φ2 ¡ 0
φp�1q � 1� φ1 � φ2 ¡ 0
1.7. AUTOREGRESSIVE PROCESS 25
Õy � 1� φ1z � φ2z
2
�1 1 2 3
�1
1
2
z
y
To have real roots, φ21 � 4φ2 ¥ 0
Õreal roots
×complex roots
�2 �1 1 2
�2
�1
1
2
φ1
φ2
26 CHAPTER 1. BASICS
1.7.3 p - th Order Autoregressive Process
ARppq is given by
φpLqyt � εt,
where φpLq � 1� φ1L� � � � � φpLp.
This process is stationary if all characteristic roots of φpzq � 0 lie outside
the unit circle.
1.8 Autocovariance and Autocorrelation Func-
tions
1.8.1 AR(1) Process
yt � φyt�1 � εt
Then, Epyt�1ytq � Epφy2t�1 � εtyt�1q � φEpy2
t�1q.
So that
γ1 � φγ0,
ρ1 � φρ0 � φ .
1.8. AUTOCOVARIANCE AND AUTOCORRELATION FUNCTIONS 27
Similarly, and noticing that Epyt�hytq � Erφyt�hyt�1 � yt�hεts � φγh�1,
we have
γh � φγh�1 � φhγ0
ρh � φh.
1 2 3 4 5
1
0.5
0
exponential decay
φ � 0.6
h
ρτ
1 2 3 4 5
1
0.5
0
�0.5
damped oscillationφ � �0.6
h
ρτ
1.8.2 AR(2) Process
yt � φ1yt�1 � φ2yt�2 � εt
Then
Epytyt�hq � φ1Epyt�1yt�hq � φ2Epyt�2yt�hq � Epεtyt�hq p1q
γh � φ1γh�1 � φ2γh�2, h � 1, 2, � � �
ρh � φ1ρh�1 � φ2ρh�2, h � 1, 2, � � � . p2q
28 CHAPTER 1. BASICS
Setting h � 1 in (2) yields
ρ1 � φ1ρ0 � φ2ρ1 ρ2 � φ1ρ1 � φ2
� φ1
1� φ2� φ2
11� φ2
� φ2
Setting h � 0 in (1) yields
γ0 � φ1γ1 � φ2γ2 � σ2
γ0p1� φ1ρ1 � φ2ρ2q � σ2
γ0p1� φ21
1� φ2� φ2
1φ2
1� φ2� φ2
2q � σ2
γ0
�1� φ2
1� φ2
rp1� φ2q2 � φ2
1s � σ2
6 γ0 ��
1� φ2
1� φ2
σ2
rp1� φ2q2 � φ21s.
1.8.3 AR(p) Process
Epytyt�1q � Epφ1y2t�1 � � � � � φpyt�pyt�1 � εtyt�1q.
γ1 � φ1γ0 � � � � � φpγp�1
γ0 � φ1γ1 � � � � � φpγp � σ2.
1.9. MOVING AVERAGE PROCESS 29
If h ¥ p,
γh � φ1γh�1 � � � � � φpγh�p
ρh � φ1ρh�1 � � � � � φpρh�p
Note
ρh ÝÑ 0, as h ÝÑ 8 for AR(p) process.
1.9 Moving Average Process
The process yt which is defined by
yt � εt � θ1εt�1 � � � � � θqεt�q
is called a q -th order moving average process or MApqq process.
Write
yt � θpLqεt, where, θpLq � 1� θ1L� � � � � θqLq
Finite MA process can be regarded as the output yt from a linear filter
with transfer function θpLq when the input is WN (White Noise), εt. Finite
MA process is therefore stationary.
30 CHAPTER 1. BASICS
The autocovariances are
γ0 � Ey2t � Epεt � θ1εt�1 � � � � � θqεt�qq2
� σ2p1� θ21 � � � � � θ2
qq � σ2q
j�0θ2j , (where θ0 � 1).
γh � Epytyt�hq
� Erpεt � θ1εt�1 � � � � � θqεt�qqpεt�h � θ1εt�h�1 � � � � � θqεt�h�qqs
� σ2pθh � θ1θh�1 � � � � � θq�hθqq, for h ¤ q
� σ2q
j�h
θj�hθj (convolution).
γh � 0, for h ¡ q.
Note that there is a cut off q. This contrasts with ARppq process, for
which γh � 0, for any h.
The h� th order autocorrelation function
ρh � θh � θ1θh�1 � � � � � θq�hθq1� θ2
1 � � � � � θ2q
, for h � 1, 2, � � � , q
ρh � 0 for h ¡ q
The autocorrelation function for MA process has a cut - off at h � q.
Write
ΠpLqyt � εt
yt � θpLqεt
1.9. MOVING AVERAGE PROCESS 31
If MA process is invertible,
ΠpLq � θpLq � 1.
The invertiblity condition for MApqq process is that all roots of charac-
teristic equation θpzq � 0 lie outside the unit circle.
Write
yt �q¹j�1p1� ωjLqεt,
If it is invertible,q¹j�1p1� ωjL� ω2
jL2 � � � � qyt � εt,
that is, the finite order MA process is transformed into the infinite order AR
process.
1.9.1 MA(1) Process
yt � εt � θεt�1
� p1� θLqεt.
θ must lie in the range (-1, 1) for the process to be invertible.
γ0 � p1� θ2qσ2
γ1 � θσ2 γh � 0 for h � 2, 3, � � �
32 CHAPTER 1. BASICS
ρ1 � θ
1� θ2 ρh � 0 for h � 2, 3, � � � p�q
It follows from p�q that
θ2 � ρ�11 θ � 1 � 0
which implies that if θ � sθ is a solution, then so is θ � sθ�1.
So, if θ � sθ satisfies the invertibility condition, the other root θ � sθ�1
will not satisfy the condition.
1.10 Mixed Autoregressive - Moving Average
Model
The process yt which is defined by
yt � φ1yt�1 � � � � � φpyt�p � εt � θ1εt�1 � � � � � θqεt�q p1q
is called a (mixed) autoregressive - moving - average progress of p and q -th order,
or ARMApp, qq process. This may be thought of as a p-th order autoregres-
sive process, φpLqyt � et, with et following the q-th order moving average
process, et � θpLqεt, or alternatively, as an MApqq process yt � θpLqηt, with
ηt following ARppq process φpLqηt � εt.
1.10. MIXED AUTOREGRESSIVE - MOVING AVERAGE MODEL 33
ARMApp, qq process (1) is stationary if φpzq � 0 has all its roots lying
outside the unit circle, and is invertible if all roots of θpzq � 0 lie outside the
unit circle.
On multiplying throughout in (1) by yt�h and taking expectations, we see
that the autocovariance function satisfies the deterministic difference equa-
tion
γh � φ1γh�1 � � � � � φpγh�p
�γyεphq � θ1γyεph� 1q � � � � � θqγyεph� qq p2q
where γyεp�q is the cross covariance function between y and ε, and is defined
as γyεphq � Epyt�hεtq.
It is easy to see
γyεphq � 0, for h ¡ 0
γyεphq � 0, for h ¤ 0.
Hence,
γh � φ1γh�1 � � � � � φpγh�p, for h ¥ q � 1
and,
ρh � φ1ρh�1 � � � � � φpρh�p, for h ¥ q � 1
or,
φpLqρh � 0, for h ¥ q � 1 ÐÝ look like ARppq p3q
34 CHAPTER 1. BASICS
Setting h � 0 in (2), we have
γ0 � φ1γ1 � � � � � φpγp � σ2 � θ1γyεp�1q � � � � � θqγyεp�qq p4q
From (3), we see that the autocorrelation function for the mixed process
eventually takes the same shape as that of AR process φpLqyt � εt.
1.10.1 ARMA(1, 1) process
yt � φyt�1 � εt � θεt�1 p5q
p1� φLqyt � p1� θLqεt
The process is stationary if |φ| 1, and invertible if |θ| 1.
Using (2) and (4), we obtain
γ0 � φγ1 � σ2 � θγyεp�1q
γ1 � φγ0 � θσ2
γh � φγh�1, for h � 2, 3, � � �
Note that γyεp�1q � Epytεt�1q � Epyt�1εtq
So, on multiplying throughout (5) by εt�1, and taking expectations, we
obtain
γyεp�1q � pφ� θqσ2
1.10. MIXED AUTOREGRESSIVE - MOVING AVERAGE MODEL 35
Hence,
γ0 � φpφγ0 � θσ2q � σ2 � θpφ� θqσ2
� 1� θ2 � 2φθ1� φ2 σ2
γ1 � φp1� θ2 � 2φθq � θp1� φ2q1� φ2 σ2 � pφ� θqp1� φθq
1� φ2 σ2
γh � φγh�1, for h � 2, 3, � � � Ð look like ARp1q
and so
ρ1 � pφ� θqp1� φθq1� θ2 � 2φθ
ρh � φρh�1, for h � 2, 3, � � �
1 2 3 4 5
1
0.5
0
exponential decay
φ � 0.6θ � 0.3
h
ρh
[ρ1 � 0.337, ρ2 � 0.202, ρ3 � 0.121, ρ4 � 0.073, ρ5 � 0.044]
36 CHAPTER 1. BASICS
Theorem (Granger’s Lemma) IfXt � ARMApp,mq and Yt � ARMApq, nq,
if Xt and Yt are independent, and if Zt � Xt � Yt, then Zt � ARMApr, lq,
where r ¤ p� q, and l ¤ maxpp� n, q �mq.
P roof. Let a1pLqXt � b1pLqεt and a2pLqYt � b2pLqηt, where a1, a2, b1, b2
are polynomials in L of order p, q,m, n respectively, and εt, ηt are indepen-
dent WN processes. Multiplying Zt � Xt � Yt by a1pLqa2pLq, we obtain
a1pLqa2pLqZt � a1pLqa2pLqXt � a1pLqa2pLqYt
� a2pLqb1pLqεt � a1pLqb2pLqηt.
Since a2pLqb1pLqεt �MApq�mq and a1pLqb2pLqηt �MApp�nq, RHS �
MAplq, where l ¤ maxpp � n, q �mq. The order of a1pLqa2pLq is not more
than p� q, and hence the theorem follows. �
The inequalities are needed in the theorem since a1pLq and a2pLq may
contain common roots.
REMARKS
(1) The theorem implies that if the series analyzed is the sum of two
independent ARp1q, then the series will be ARMAp2, 1q,
1.11. AUTOCOVARIANCE GENERATING FUNCTION 37
for example. If the observed series is the sum of a ”true” ARppq
process plus a WN measurement error, then an ARMApp, pq process
results.
(2) Mixed model may achieve as good a fit as the AR model but using
fewer parameters. Principle of parsimony (Box and Jenkins)
1.11 Autocovariance Generating Function
A compact and convenient way of recording the information contained
in a sequence taju is by means of the generating function : apzq �¸j
ajzj,
where z is a possibly complex variable. The individual members of the se-
quence can easily be recovered from the coefficients associated with the zj’s.
The quantity z does not necessarily have any interpretation and should be
simply considered as the carries of the information in the sequence.
Now define
γpzq �8
h��8
γhzh autocovariance generating function
ρpzq �8
h��8
ρhzh autocorrelation generating function
mptq � Epetxq Moment Generating Function
φptq � Epeitxq Characteristic Function
38 CHAPTER 1. BASICS
Useful properties of generating functions :
(1) Additivity cj � aj � bj ÝÑ cpzq � apzq � bpzq
(2) Convolution∗ cj �j
k�0akbj�k ÝÑ cpzq � apzqbpzq
apzq �8
j�0ajz
j, j � 0, 1, 2, � � �
Given X1, X2, � � � , z - transform Xpzq �¸t
Xtzt
Now, let XT pzq �T
t�1Xtz
t,
then, ErXT pzqXT pz�1qs �T
h��T
pT � |h|qγhzh.
Dividing by T and passing to the limit, we obtain,
γpzq � limTÑ8
1T
ErXT pzqXT pz�1qs.
Let S � t�N
yt � yNS �S
j�0ψjε
NS�j �
S
j�0ψjεS�N�j �
t�N
j�0ψjεt�j
Ñ8
j�0ψjεt�j as N ÝÑ 8
∗c0 � a0b0c1 � a0b1 � a1b0c2 � a0b2 � a1b1 � a2b0pa0 � a1z � a2z2 � � � � qpb0 � b1z � b2z2 � � � � q � a0b0 � pa0b1 � a1b0qz � pa0b2 � a1b1 �a2b0qz
2 � � � �
1.11. AUTOCOVARIANCE GENERATING FUNCTION 39
y0 � ε0
y1 � ε1 � ψ1ε0
y2 � ε2 � ψ1ε1 � ψ2ε0
...
yT �T
j�0ψjεt�j
Define, yT pzq �T
t�0ytz
t
εT pzq �T
t�0εtz
t
ψT pzq �T
t�0ψtz
t
Suppose tYtu is a linear process.
Then, Y pzq � ψpzqεpzq.
Hence, γY pzq � limTÑ8
1T
ErψT pzqεT pzqψT pz�1qεT pz�1qs
� ψpzqψpz�1qσ2
1.11.1 MA(q) process
yt � θpLqεt
γpzq � σ2θpzqθpz�1q
40 CHAPTER 1. BASICS
Note that
θpzqθpz�1q �q
j�0
q
k�0θjθkz
j�k
�q
h��q
zhq
j�0θjθj�h
by setting j � k � h, and taking θj � 0 for j ¡ q.
For MA(1), yt � εt � θεt�1
θpzqθpz�1q � σ2p1� θzqp1� θz�1q
� σ2rp1� θ2q � θz � θz�1s
$''''''&''''''%γ0 � p1� θ2qσ2
γ1 � θσ2
γh � 0, for h � 2, 3, � � �
q � 1
$''''''&''''''%θ0 � 1
θ1 � θ
θh � 0 for h � 0, h � 1
1
h��1zh
1
j�0θjθj�h � pθ2
0 � θ21q � pθ0θ1 � θ1θ2qz � pθ0θ�1 � θ1θ0qz�1
� p1� θ2q � θz � θz�1
1.11.2 AR(p) process
φpLqyt � εt ùñ yt � ψpLqεt
and in generating function form
1.11. AUTOCOVARIANCE GENERATING FUNCTION 41
φpzqypzq � εpzq ùñ ypzq � ψpzqεpzq,
where, φpzqψpzq � 1.
Therefore,
γpzq � σ2ψpzqψpz�1q
� σ2
φpzqφpz�1q
For AR(1), yt � φyt�1 � εt
γpzq � σ2
p1� φzqp1� φz�1q� p1� φz � φ2z2 � � � � qp1� φz�1 � φ2z�2 � � � � qσ2
� rp1� φ2 � φ4 � � � � q � pφ� φ3 � φ5 � � � � qz � � � � sσ2
γ0 � σ2
1� φ2 , γh � φhγ0
1.11.3 ARMA(p, q) process
γpzq � σ2 θpzqθpz�1qφpzqφpz�1q
42 CHAPTER 1. BASICS