STAT 516: Multivariate Distributions

22
STAT 516: Multivariate Distributions Lecture 7: Convergence in Probability and Convergence in Distribution Prof. Michael Levine November 12, 2015 Levine STAT 516: Multivariate Distributions

Transcript of STAT 516: Multivariate Distributions

Page 1: STAT 516: Multivariate Distributions

STAT 516: Multivariate DistributionsLecture 7: Convergence in Probability and Convergence in

Distribution

Prof. Michael Levine

November 12, 2015

Levine STAT 516: Multivariate Distributions

Page 2: STAT 516: Multivariate Distributions

Convergence in Probability

I A sequence Xnp→ X (converges in probability to X ) if, for

any ε > 0limn→∞

P(|Xn − X | ≥ ε) = 0

I In this context, X may be a constant a - a degenerate randomvariable

I Chebyshev’s inequality is a common way of showingconvergence in probability

Levine STAT 516: Multivariate Distributions

Page 3: STAT 516: Multivariate Distributions

Examples

1. Let Xn = X + 1n where X ∼ N(0, 1)

2. Easy to verify that (by Chebyshev’s inequality) Xnp→ X

1. For Xn s.t. the mean µ and variance σ2 are finite

Xnp→ µ

2. The weak law of large numbers - second moment must exist;strong law does not require that - will not be proved in thiscourse

Levine STAT 516: Multivariate Distributions

Page 4: STAT 516: Multivariate Distributions

Convergence in probability is closed under linearity

I Xnp→ X and Yn

p→ Y implies Xn + Ynp→ X + Y

I If a is a constant and Xnp→ X aXn

p→ aX

I Conclusion: convergence in probability is closed under linearity

Levine STAT 516: Multivariate Distributions

Page 5: STAT 516: Multivariate Distributions

Continuous Mapping Theorem for Convergence inProbability

I If g is a continuous function, Xnp→ X then g(Xn)

p→ g(X )

I We only prove a more limited version: if, for some constant a,g(x) is continuous at a, g(Xn)

p→ g(a)

I Can be viewed as one of the statements of Slutsky theorem -the full theorem to be stated later

Levine STAT 516: Multivariate Distributions

Page 6: STAT 516: Multivariate Distributions

Another useful property

I If Xnp→ X and Yn

p→ Y , XnYnp→ XY

I Only prove in this form but can be generalized tog(Xn,Yn)

p→ g(X ,Y )

Levine STAT 516: Multivariate Distributions

Page 7: STAT 516: Multivariate Distributions

Consistency and convergence in probability

I For X ∼ F (x ; θ), θ ∈ Ω a statistic Tn is a consistentestimator of θ if

Tnp→ θ

I Weak Law of Large Numbers → X is a consistent estimator ofµ

I A sample variance S2 = 1n−1

∑ni=1(Xi − X )2

p→ σ2

I By continuous mapping theorem Sp→ σ

Levine STAT 516: Multivariate Distributions

Page 8: STAT 516: Multivariate Distributions

Example

I Let X1, . . . ,Xn ∼ Unif [0, 1] and Yn = maxX1, . . . ,XnI The cdf of Yn is FYn(t) =

(tθ

)nfor 0 < t ≤ θ

I Check that EYn =(

nn+1

)θ - Yn is a biased estimator

I Direct computation implies that Yn is consistent...and so isthe unbiased estimator

(n+1n

)Yn

Levine STAT 516: Multivariate Distributions

Page 9: STAT 516: Multivariate Distributions

Convergence in Distribution

I If for a sequence Xn with cdf FXn(x) , and a randomvariable X ∼ FX (x)

limn→∞

FXn(x) = FX (x)

for all points of continuity of FX (x), XnD→ X - Xn converges

in distribution or in law to X

I We say that FX (x) is the asymptotic distribution or thelimiting distribution of Xn

I Occasional abuse of notation: Xn → N(0, 1)

Levine STAT 516: Multivariate Distributions

Page 10: STAT 516: Multivariate Distributions

Convergence in distribution and convergence in probability

I Convergence in distribution is only concerned withdistributions and not at all with random variables

I For a symmetric fX (x), X and −X have the same distribution

I Let

Xn =

X if n is odd−X if n is even

I Clearly, Xn

D→ X but there is no convergence in probability!

Levine STAT 516: Multivariate Distributions

Page 11: STAT 516: Multivariate Distributions

Example

I Let X ∼ N(0, σ2/n)

I Check that

limn→∞

Fn(x) =

0 x < 012 x = 01 x > 0

I Conclude that Fn(x) converges to the point mass at zero

Levine STAT 516: Multivariate Distributions

Page 12: STAT 516: Multivariate Distributions

Example

I Convergence of pdfs/pmfs does NOT mean convergence indistribution!

I Define the pmf

pn(x) =

1 x = 2 + 1

n0 elsewhere

I limn→∞ pn(x) = 0 for any x

I However, the limiting function of cdf’s is F (x) = 0 if x < 2and F (x) = 1 if x ≥ 2 which is a cdf!

I Convergence in distribution does take place!

Levine STAT 516: Multivariate Distributions

Page 13: STAT 516: Multivariate Distributions

Example

I However...for Tn ∼ tn we have

Fn(t) =

∫ t

−∞

Γ[(n + 12)]

√πnΓ

(n2

) 1

(1 + y2/n)(n+1)/2dy

I Stirling’s formula:

Γ(k + 1) ≈√

2πkk+1/2 exp (−k)

I The limit under the sign of integral is the normal pdf...so

limn→∞

Fn(t) =

∫ t

−∞

1√2π

exp (−y2/2) dy

I The limiting distribution of tn is N(0, 1)

Levine STAT 516: Multivariate Distributions

Page 14: STAT 516: Multivariate Distributions

Example

I Recall that for X1, . . . ,Xn ∼ Unif [0, θ] Yn = max1≤i≤n is theconsistent estimator of θ

I Now we can say more...let Zn = n(θ − Yn)

I For any t ∈ (0, nθ)

P(Zn ≤ t) = P(Yn ≥ θ − (t/θ)) = 1−(

1− t/θ

n

)n

I Since limn→∞ P(Zn ≤ t) = 1− exp (−t/θ) for some

Z ∼ exp(θ) ZnD→ Z

Levine STAT 516: Multivariate Distributions

Page 15: STAT 516: Multivariate Distributions

Relationship between convergence in probability andconvergence in distribution

I If Xnp→ X , Xn

D→ X

I The converse is not true in general - see an earlier example!

I However, if for a constant b XnD→ b it also true that Xn

p→ b

Levine STAT 516: Multivariate Distributions

Page 16: STAT 516: Multivariate Distributions

Basic properties of convergence in distribution

I If XnD→ X and Yn

p→ 0, Xn + YnD→ X

I This is the magic wand if it is hard to show that XnD→ X but

easy to show that some other YnD→ X and Xn − Yn

p→ 0

I If XnD→ X and g(x) is a continuous function on the support

of X ,

g(Xn)D→ g(X )

I If a and b are constants, XnD→ X , An

p→ a, and Bnp→ b,

An + BnXnD→ a + bX

Levine STAT 516: Multivariate Distributions

Page 17: STAT 516: Multivariate Distributions

Boundedness in probability

I For any X ∼ FX (x), we can always find η1 and η2 s.t.FX (x) < ε/2 for x ≤ η1 and FX (x) > 1− ε/2 for x ≥ η2

I Thus, for η = max|η1|, |η2|

P[|X | ≤ η] ≥ 1− ε

I Formal definition: Xn is bounded in probability if for anyε > 0 there exist Bε > 0 and an integer Nε s.t. if n ≥ NεP[Xn ≤ Bε] ≥ 1− ε

I Can show immediately that if XnD→ X then Xn is bounded

in probability...the converse is not always true

Levine STAT 516: Multivariate Distributions

Page 18: STAT 516: Multivariate Distributions

Why a sequence that is bounded in probability may notconverge

I Define X2m = 2 + 12m and X2m−1 = 1 + 1

2m w.p.1

I All of the mass of this sequence is concentrated in [1, 2.5] andso it is bounded in probability

I Xn consists of two subsequences that converge to degenerateRV’s Y = 2 and W = 1 in distribution

Levine STAT 516: Multivariate Distributions

Page 19: STAT 516: Multivariate Distributions

A useful property

I Xn a sequence of random variables bounded in prob. and Yn asequence that converges to zero in probability

I Then, XnYnp→ 0

I Analog from the world of calculus: if A is a constant andξn = 1

n then limn→∞1n = 0 and limn→∞

An = 0

Levine STAT 516: Multivariate Distributions

Page 20: STAT 516: Multivariate Distributions

MGF technique

I If Xn has mgf MXn(t) for |t| ≤ h, X has mgf MX (t) for|t| ≤ h1 < h, and limn→∞MXn(t) = M(t) for |t| ≤ h1, then

XnD→ X

I Take Yn ∼ b(n, p) with fixed µ = np for every n

I Check that MYn(t) = [(1− p) + pet ]n =[1 + µ(et−1)

n

]n

Levine STAT 516: Multivariate Distributions

Page 21: STAT 516: Multivariate Distributions

Poisson approximation of the binomial: an example

I Y ∼ b(50, 1

25

);

I P(Y ≤ 1) =(2425

)50+ 50

(125

) (2425

)49= 0.4000

I Since µ = np = 2, we have the Poisson approximation

e−2 + 2e−2 = 0.406

Levine STAT 516: Multivariate Distributions

Page 22: STAT 516: Multivariate Distributions

Central Limit Theorem (CLT)

I If X1, . . . ,Xn ∼ N(µ, σ2) we know that X ∼ N(µ, σ

2

n

)I CLT: if X1, . . . ,Xn are independent, bE Xi = µ and

Var Xi = σ2, we have

√n(X − µ)

σ∼ N(0, 1)

Levine STAT 516: Multivariate Distributions