Stat 310B/Math 230B Theory of Probability

Homework 8 SolutionsAndrea Montanari Due on 3/12/2014

Exercise [6.2.5]

1. For any x ∈ S\C, our assumption that Px(τC < ∞) > 0 implies that Px(TC ≤ Nx) = Px(τC ≤ Nx) > 0for finite non-random constants Nx. Since S \ C is finite, the constant N = maxx/∈C Nx is finite andε = minx/∈C Px(TC ≤ N) is positive. Since τC is a stopping time, fixing y ∈ S and applying the Markovproperty for the indicator function h(ω) = I{TC(ω)≤N}, we find that for any non-negative integer n,

Py(τC ≤ n + N |FXn ) = IτC≤n + Ey[h(θnω)|FX

n ]IτC>n = IτC≤n + PXn(TC ≤ N)IτC>n .

If τC > n then necessarily Xn /∈ C, resulting with PXn(TC ≤ N) ≥ ε. Thus, Py(τC ≤ n + N |FX

n ) ≥ εfor all n and the stated bound Py(τC > kN) ≤ (1− ε)k follows from part (a) of Exercise 5.1.15.

2. If ω0 /∈ C = A ∪ B then h(θω) = h(ω) for the indicator function h(ω) = IτA<τB(ω) (which is in bSSc

since τA and τB are stopping times). Considering its expected value, we get by the tower and Markovproperties that for any x /∈ C,

g(x) = Px(τA < τB) = Ex[h(θω)] = Ex[Ex(h(θω)|FX1 )]

= Ex[PX1(τA < τB)] = Ex[g(X1)] =∑y∈S

p(x, y)g(y).

3. Since g(·) is bounded and τC is a stopping time for the canonical filtration FXn , the process Yn =

g(Xn∧τC) is integrable and adapted to this filtration. Further, it is easy to see that

Dn+1 = Yn+1 − Yn = [g(Xn+1)− g(Xn)]I{τC>n} .

Hence, for any initial distribution ν and all n, by the Markov property and taking out what is known,

Eν [Dn+1|FXn ] = (EXn [g(X1)]− g(Xn))I{τC>n} .

As mentioned before, if τC > n then Xn /∈ C in which case we have from part (b) that EXn [g(X1)] =g(Xn). Consequently, Eν [Dn+1|FX

n ] = 0 for all n and hence {Yn} is a MG (see Proposition 5.1.5).

4. We already saw that Px(τA < τB) has the stated properties. Conversely, suppose that a boundedfunction g(x) is harmonic at every x /∈ C. Then, by part (c) g(Xn∧τC

) is a MG for any initialmeasure ν. In particular, g(y) = Ey[g(Xn∧τC

)] for any y ∈ S and all n. Further, in view of part (a)τC < ∞ almost surely and our assumption that g(x) = 1 when x ∈ A and g(x) = 0 when x ∈ Bimplies that g(XτC

) = I{τA<τB}. Hence, taking n → ∞ it follows by bounded convergence thatg(y) = Ey[g(XτC

)] = Py(τA < τB), as claimed.

5. With τC an FXn -stopping time, the non-negative Mn = f(Xn∧τC

)+n∧τC is FXn -adapted. Suppose first

that τC > n. In this case Mn+1 = n+1+f(Xn+1) and Xn /∈ C, hence f(Xn) = 1+∑

y∈S p(Xn, y)f(y).Consequently, for any n ≥ 0,

E[Mn+1I{τC>n}|FXn ] = [n + 1 +

∑y∈S

p(Xn, y)f(y)]I{τC>n} = MnI{τC>n} . (∗)

1

Taking expectations it follows that E[Mn+1I{τC>n}] = E[MnI{τC>n}]. Further, if τC ≤ n then Mn+1 =Mn so in particular EMn+1 = EMn (possibly infinite). This applies for all n ≥ 0 and EM0 = Ef(X0)is finite by our assumption that X0 is supported on the finite set S \ C. Hence, Mn is integrable andby virtue of (*), it is also a martingale. Next, starting at X0 = x /∈ C, recall part (a), that τC isintegrable, so if in addition f(x) = 0 for x ∈ C, then Mn ≤ τC + supx/∈C f(x) is a U.I. martingale.Further, Mn = τC for all n large enough, hence Mn → τC in L1. In particular, ExMn → ExτC and weare done (since f(x) = ExM0 = ExMn for all n).

Exercise [6.2.11]

Each of the random vectors Y k takes values in S =⋃∞

r=1{r} × Sr. Let SS = σ({r} × SSr, r ≥ 1) be thecorresponding σ-algebra on S and FX

n = σ(Xi, i ≤ n) the canonical filtration of the Markov chain {Xn}.

1. Fixing k > 1 and A ∈ SS consider the stopping time τ = Rk−1. Since y is recurrent, Py(τ < ∞ andXτ = y) = 1. Hence, applying the strong Markov property for the a.s. finite stopping time τ and theindicator function h(ω) = I{Y1∈A} which is in bSSc (since r1 = Ty is a stopping time), we find that

Py(Y k ∈ A|FXτ ) = Ey[h(θτω)|FX

τ ] = EXτ (I{Y 1∈A}) = Py(Y 1 ∈ A) .

Note that for any 1 ≤ ` < k, the random vector Y ` is measurable on FXR`⊆ FX

τ . Thus, for any A ∈ SS

and B ∈ SSk−1

, by the tower property, taking out what is known and the preceding identity

Py(Y k ∈ A, (Y 1, . . . , Y k−1) ∈ B) = Ey[I{(Y 1,...,Y k−1)∈B}Py(Y k ∈ A|FXτ )]

= Py((Y 1, . . . , Y k−1) ∈ B)Py(Y 1 ∈ A) .

Taking B = Sk−1, we see that Y kd= Y 1 for any k. Therefore, Y k are identically distributed and upon

substituting Y k for Y 1 in the right hand side of the last display, we further deduce that these randomvectors are mutually independent.

2. In case of the measure Pν , the condition {Ty < ∞} assures us that also Rk < ∞ for all k (up to a setof probability zero). Then, a re-run of the argument of part (a) results with

Pν((Y 1, . . . , Y k) ∈ B ×A, Ty < ∞) = Pν((Y 1, . . . , Y k−1) ∈ B, Ty < ∞)Py(Y 1 ∈ A) ,

for all k > 1, A ∈ SS and B ∈ SSk−1

. Considering B = Sk−1 this shows that conditional on {Ty < ∞}the random vectors {Y k, k ≥ 2} are identically distributed under Pν , having each the same law as Y 1

has under Py, which upon substituting in the right hand side of the last display also proves our claimthat conditional on {Ty < ∞} the vectors Y k are mutually independent (for all k ≥ 1).

Exercise [6.2.29]

1. By the definition of invariant measure, for any x ∈ S,∑y∈S

q(x, y) =1

µ(x)

∑y∈S

µ(y)p(y, x) =µ(x)µ(x)

= 1.

Therefore, the non-negative q(·, ·) is also a transition probability on S.

2. Since ν(·) is an excessive measure for p(·, ·), for any x ∈ S,∑y∈S

q(x, y)h(y) =1

µ(x)

∑y∈S

p(y, x)ν(y) ≤ ν(x)µ(x)

= h(x) .

Thus, h(x) is a non-negative super-harmonic function for q(·, ·).

2

3. Let P and Q denote the laws of the homogeneous Markov chains {Xn} and {Yn} with transitionprobabilities p(·, ·) and q(·, ·), respectively. We proceed to show by induction on n ≥ 1 that the dualityrelation

Qx(Yn = y) =µ(y)µ(x)

Py(Xn = x) (∗)

(c.f. Definition 6.2.34), holds for any x, y ∈ S. Indeed, for n = 1 this identity is merely the definitionof q(·, ·), and if it holds up to some k ≥ 1, then,

Qx(Yk+1 = y) =∑z∈S

Qx(Yk = z)q(z, y) =∑z∈S

µ(z)µ(x)

Pz(Xk = x)µ(y)µ(z)

p(y, z)

=µ(y)µ(x)

∑z∈S

Py(X1 = z)Pz(Xk = x) =µ(y)µ(x)

Py(Xk+1 = x),

establishing the validity of (*) for n = k + 1. Since {Xn} is irreducible, for any x, y ∈ S there exists nfinite such that Py(Xn = x) > 0. By (*) this implies that Qx(Yn = y) > 0. Hence, Qx(Ty < ∞) > 0for all x, y ∈ S, i.e. q(·, ·) is irreducible. With p(·, ·) recurrent it further follows from (*) that for anyy ∈ S,

Qy[N∞(y)] =∞∑

n=1

Qy(Yn = y) =∞∑

n=1

Py(Xn = y) = Py[N∞(y)] = ∞ ,

so q(·, ·) is also recurrent. Recall part (b) that the non-negative function h(x) = ν(x)/µ(x) is super-harmonic for the irreducible, recurrent transition probability q(·, ·). By Exercise 6.2.23 it must thenbe a constant function, namely, ν(x) = cµ(x) for some constant c > 0 and all x ∈ S.

Exercise [6.2.31]

1. The proof of the stated identity is by induction over k, starting at k = 1 with the characterization(6.2.5) of µ(·) being an invariant measure. For the induction step, assuming that the claim holds forall z ∈ S and up to some finite k, you find that∑

x

µ(x)Px(Xk+1 = y) =∑

x

µ(x)∑

z

Px(Xk = z)p(z, y)

=∑

z

p(z, y)∑

x

µ(x)Px(Xk = z) =∑

z

p(z, y)µ(z) = µ(y) .

2. If y is accessible from v then Pv(Xk = y) > 0 for some finite k, so when in addition µ(v) > 0, we haveby part (a) that

µ(y) =∑

x

µ(x)Px(Xk = y) ≥ µ(v)Pv(Xk = y) > 0 .

3. Following the proof of Proposition 6.2.30, the restriction pR(·, ·) of p(·, ·) to R is a family of transitionprobabilities on an irreducible, recurrent state space R (having a strictly positive invariant measure,see Proposition 6.2.27). Since µ is an invariant measure for p(·, ·),∑

x∈R

µ(x)pR(x, y) ≤∑x∈S

µ(x)p(x, y) = µ(y) . (∗)

Consequently, either µ(·) : R 7→ [0,∞) is identically zero, or it is an excessive measure for pR(·, ·)(see Definition 6.2.25). Applying Exercise 6.2.29 for the chain of transitions pR(·, ·) we deduce that ineither case the inequality in (*) holds with equality if y ∈ R. Hence, µ(x)p(x, y) = 0 for all x /∈ R andy ∈ R.

3

4. Suppose R is accessible from v /∈ R, that is, there exists y ∈ R and a finite path v = x0, x1, . . . , xk = ysuch that p(xi, xi+1) > 0 for i = 0, . . . , k − 1. Since xk ∈ R and x0 /∈ R, there exists 0 ≤ i < k suchthat xi /∈ R and xi+1 ∈ R. If µ(v) > 0 then by part (b) also µ(xi) > 0, contradicting the conclusionof part (c).

Exercise [6.2.20]

1. Fixing 0 < |s| < 1, note that sT 0x = s0 = 1, hence u(s) = 1 +

∑k≥0 Ex[sT k+1

x ]. Further, if T kx = ∞

then also T k+1x = ∞ with sT k

x = sT k+1x = 0, so u(s) = 1 +

∑k≥0 Ex[sT k+1

x IT kx <∞]. When the stopping

time T kx is finite we know that XT k

x= x, so by the strong Markov property at the stopping time T k

x

(for the function h(ω) = sTx(ω) ∈ bSSc),

Ex[sT k+1x I{T k

x <∞}] = Ex[sT kx I{T k

x <∞}]Ex[sTx ] = Ex[sT kx ]Ex[sTx ] .

Summing over k ≥ 0 we get that u(s) = 1 + u(s)f(s) as stated.

2. Recall that T kx < ∞ if and only if N∞(x) ≥ k. Further, as s ↑ 1, we have that sT k

x ↑ I{T kx <∞} =

I{N∞(x)≥k}. Consequently, by monotone convergence, both f(s) ↑ Px(Tx < ∞) = ρxx and

lims↑1

u(s) =∞∑

k=0

Px(T kx < ∞) =

∞∑k=0

Px(N∞(x) ≥ k) = 1 + Ex[N∞(x)].

For any 0 < s < 1 we have that 0 ≤ f(s) < 1, and so from part (a) we get that u(s) − 1 =f(s)/(1 − f(s)). Since f 7→ f/(1 − f) is continuous on [0, 1], considering s ↑ 1 we thus deduce thatEx[N∞(x)] = ρxx/(1− ρxx). Note that we did not assume that ρxx < 1, nor that Ex[N∞(x)] < ∞.

3. Recall that for any n ≥ 0 the event {Xn = x} is the union of the disjoint events {T kx = n}. Thus, by

Fubini’s theorem

u(s) =∞∑

k=0

∞∑n=0

Px(T kx = n)sn =

∞∑n=0

sn∞∑

k=0

Px(T kx = n) =

∞∑n=0

Px(Xn = x)sn.

For the SRW Sn on Z we have that Px(Sn = x) = 0 for n odd, whereas Px(S2m = x) =(2mm

)pmqm.

Hence, applying the hint for t = 4pqs2 ∈ [0, 1), we find that

u(s) =∞∑

m=0

Px(S2m = x)s2m =∞∑

m=0

(2m

m

)2−2m(4pqs2)m = (1− 4pqs2)−1/2.

4. From parts (a), (b) and (c) we deduce that for the SRW on Z,

ρxx = lims↑1

f(s) = lims↑1

u(s)− 1u(s)

= 1−√

1− 4pq = 1− |p− q| = 2min(p, q) ,

so this irreducible chain is recurrent if p = 1/2 and transient for any other value of 0 < p < 1.

4