5.  Limit  Theorems,  Part  I:  Weak  Law  of  Large  Numbers  

ECE  302  Spring  2012  Purdue  University,  School  of  ECE  

Prof.  Ilya  Pollak    

Some  moJvaJon  

•  Suppose  X1,  …,  Xn  are  i.i.d.  (independent,  idenJcally  distributed)  with  mean  μ  and  variance  σ2  

•  The  sample  mean  is  Mn  =  (X1  +  …  +  Xn)/n  

•  What  happens  as  n  −>  ∞  ?  

•  Why  is  answering  this  quesJon  important?  – Will  allow  us  to  interpret  expectaJons  and  probabiliJes  in  terms  of  a  long  sequence  of  independent  idenJcal  experiments.  

– Will  facilitate  approximate  analysis  of  r.v.’s  such  as  Mn  whose  distribuJon  is  difficult  to  compute  for  large  n.  

Ilya Pollak

Weak  law  of  large  numbers  

•  Suppose  X1,  …,  Xn  are  i.i.d.  (independent,  idenJcally  distributed)  with  mean  μ  and  variance  σ2  

•  The  sample  mean  is  Mn  =  (X1  +  …  +  Xn)/n  

•  This  says  that  the  bulk  of  the  distribuJon  of  Mn  is  concentrated  near  the  actual  mean  μ.  

P( Mn − µ ≥ ε)→ 0 as n→∞,for every ε > 0

Ilya Pollak

Plan  for  proving  WLLN  

•  Tools:  Markov  and  Chebyshev  inequaliJes.  •  Meaning  of  “          “  →

Ilya Pollak

Markov  Inequality  If P(X < 0) = 0, then

P(X ≥ a) ≤ E[X]a

, for all a > 0.

Proof . Fix a > 0, and let ga (x) = 0, if 0 ≤ x < aa, if x ≥ a

⎧⎨⎪

⎩⎪Then ga (x) ≤ x for all x ≥ 0, and soE ga (X)[ ] ≤ E[X]

But E ga (X)[ ] = 0 ⋅P(X < a) + a ⋅P(X ≥ a) = a ⋅P(X ≥ a)Hence, a ⋅P(X ≥ a) ≤ E[X].

Ilya Pollak

Chebyshev  Inequality  If E[X] = µ and var(X) = σ 2 , then

P X − µ ≥ c( ) ≤ σ 2

c2 , for any c > 0.

Alternative form: P X − µ ≥ kσ( ) ≤ 1k2 , for any k > 0.

Proof . Since (X − µ)2 ≥ 0 with probability 1, we can apply Markov inequality.Let a = c2 . Then

P (X − µ)2 ≥ c2( ) ≤ E (X − µ)2⎡⎣ ⎤⎦c2 =

σ 2

c2

But P (X − µ)2 ≥ c2( ) = P X − µ ≥ c( ).If c = kσ , then

P X − µ ≥ kσ( ) ≤ σ 2

k2σ 2 =1k2

Ilya Pollak

Review  of  determinisJc  limits  

Suppose a1,a2 ,a3,… is a sequence of numbersA number a is the limit of an (or an converges to a) if, for everyδ > 0, there exists n0 such that for all n ≥ n0 we have an − a ≤ δNotation: an → a as n→∞, or lim

n→∞an = a

Ilya Pollak

Convergence  in  probability  Suppose Y1,Y2 ,Y3,… is a sequence of random variables.Yn converges in probability to a number a if, for every ε > 0,

limn→∞P Yn − a ≥ ε( ) = 0

---in other words, if, for every δ > 0 and ε > 0, there existsn0 such that for all n ≥ n0 ,

P Yn − a ≥ ε( ) ≤ δ .Interpretation: For any given levels of accuracy ε and confidence δ ,the random variable Yn is likely to be approximately equal to a,within these levels of accuracy and confidence, providedthat n is large enough.

Ilya Pollak

Ex.  5.8:  Convergence  in  probability  

0   n2   y  

1/n  

1  −  1/n  pYn (y)

P Yn − 0 ≥ ε( ) =0, ε > n2 1n

, 0 < ε ≤ n2

⎧

⎨⎪

⎩⎪

≤1n→ 0 as n→∞, for any ε > 0.

Therefore, Yn converges in probability to zero.

But note: E Yn[ ] = n2 ⋅1n= n→∞ as n→∞.

Ilya Pollak

Weak  law  of  large  numbers  

•  Suppose  X1,  …,  Xn  are  i.i.d.  (independent,  idenJcally  distributed)  with  mean  μ  and  variance  σ2  

•  The  sample  mean  Mn  =  (X1  +  …  +  Xn)/n  converges  to  the  actual  mean  μ  in  probability:  

P( Mn − µ ≥ ε)→ 0 as n→∞,for every ε > 0

Ilya Pollak

Weak  law  of  large  numbers:  Proof  

•  X1,  …,  Xn  are  i.i.d.  with  mean  μ  and  variance  σ2  •  The  sample  mean  Mn  =  (X1  +  …  +  Xn)/n  converges  to  the  actual  mean  μ  in  probability.  

Proof . Chebyshev inequality:

P Mn − E Mn[ ] ≥ ε( ) ≤ var Mn( )ε 2 .

But E Mn[ ] = µ and var Mn( ) = σ 2

n. Therefore,

P Mn − µ ≥ ε( ) ≤ σ 2

nε 2 → 0 as n→∞, for every ε > 0.

Ilya Pollak

Ex.  5.4:  ProbabiliJes  and  Frequencies  •  Suppose  that  during  each  repeJJon  of  an  experiment,  event  A  occurs

 with  probability  p  =  P(A).  

•  Consider  n  independent  repeJJons.  •  Let  Mn  =  (number  of  occurrences  of  A)/n  

Let Xi =1, if A occurs on the i-th experiment 0, if A does not occur on the i-th experiment

⎧⎨⎪

⎩⎪Then Xi's are independent Bernoulli random variables, with

pXi (xi ) =p, if xi=1 1− p, if xi=0

⎧⎨⎪

⎩⎪

Mn =X1 + X2 +…+ Xn

nE[Xi ] = p, hence E[Mn ] = pWLLN: when n is large, the empirical frequency Mn of Ais very likely to be close to p, the probability of A. Ilya Pollak

Ex.  5.5:  Polling  Estimate President Obama's approval rating by asking n persons drawn at randomfrom the voter population. Let

Xi =1, if the i-th person approves0, otherwise

⎧⎨⎪

⎩⎪Model X1,X2 ,…,Xn as independent Bernoulli r.v.'s with mean p and variance p(1-p),where p is the "true" approval rating.

We estimate p as the sample mean: Mn =X1 + X2 +…+ Xn

n

E[Mn ] = p, var(Mn ) = p(1− p)n

By WLLN, Mn → p in probability as n→∞.

By Chebyshev inequality, P Mn − p ≥ ε( ) ≤ p(1− p)nε 2 ≤

14nε 2

p(1−p)

p 0 1/2 1

1/4

Ilya Pollak

Ex.  5.5:  Polling  Estimate President Obama's approval rating by asking n persons drawn at randomfrom the voter population. Let

Xi =1, if the i-th person approves0, otherwise

⎧⎨⎪

⎩⎪Model X1,X2 ,…,Xn as independent Bernoulli r.v.'s with mean p and variance p(1-p),where p is the "true" approval rating.

We estimate p as the sample mean: Mn =X1 + X2 +…+ Xn

n

E[Mn ] = p, var(Mn ) = p(1− p)n

By WLLN, Mn → p in probability as n→∞.

By Chebyshev inequality, P Mn − p ≥ ε( ) ≤ p(1− p)nε 2 ≤

14nε 2

Suppose we want P Mn − p ≥ 0.01( ) ≤ 0.05. I.e., we want to be 95% confidentthat we are within 0.01 of the actual approval rating. We can guarantee this

if we take n ≥ 14ε 2 0.05

=1

4 ⋅0.012 ⋅0.05= 50,000.

Ilya Pollak

Ex.  5.5:  Polling  Estimate President Obama's approval rating by asking n persons drawn at randomfrom the voter population. Let

Xi =1, if the i-th person approves0, otherwise

⎧⎨⎪

⎩⎪

Estimate p, the "true" approval rating, as the sample mean: Mn =X1 + X2 +…+ Xn

n

E[Mn ] = p, var(Mn ) = p(1− p)n

By WLLN, Mn → p in probability as n→∞.

By Chebyshev inequality, P Mn − p ≥ ε( ) ≤ p(1− p)nε 2 ≤

14nε 2

Suppose we want P Mn − p ≥ 0.01( ) ≤ 0.05. I.e., we want to be 95% confidentthat we are within 0.01 of the actual approval rating. We can guarantee this

if we take n ≥ 14ε 2 0.05

=1

4 ⋅0.012 ⋅0.05= 50,000. Note: This is very conservative

because the Chebyshev inequality is loose!

Ilya Pollak