1. Axiomatic definition of probability 1.1. Probability space. Let ©
EECS 126: Probability & Random Processes Fall 2020
Transcript of EECS 126: Probability & Random Processes Fall 2020
EECS 126: Probability & Random ProcessesFall 2021
PageRank
Shyam Parekh
• Originally used by Google for ranking the pages from a keyword search.
𝜋 𝑖 =
𝑗∈𝑋
𝜋 𝑗 𝑃 𝑗, 𝑖 , ∀ 𝑖 ∈ 𝒳 ⟺ 𝜋 = 𝜋𝑃
• Original algorithm also used a damping factor.
• With the normalization condition 𝑖𝜖𝑋 𝜋(𝑖) = 1, π =1
39[12, 9, 10, 6, 2]
• This is similar to a Markov Chain (defined next).
PageRank
𝑃 𝐴, 𝐵 = 1/2, 𝑃 𝐷, 𝐸 = 1/3, …
• DTMC 𝑋(𝑛 , 𝑛 ≥ 0} over state space 𝒳 with 𝑃 = [𝑃(𝑖, 𝑗)] as the transition probability matrix with 𝑃 𝑖, 𝑗 = 𝑃 𝑋 𝑛 + 1 = 𝑗 𝑋 𝑛 = 𝑖].
– State transition diagram.
– Memoryless Property: 𝑃 𝑋 𝑛 + 1 = 𝑗 𝑋 𝑛 = 𝑖, 𝑋 𝑚 ,𝑚 < 𝑛] = 𝑃 𝑖, 𝑗 ∀ 𝑖, 𝑗.
• 𝜋𝑛+1 i = 𝑗∈𝑋 𝜋𝑛 𝑗 𝑃(𝑗, 𝑖) ⟺ 𝜋𝑛+1 = 𝜋𝑛𝑃 ⟺ 𝜋𝑛 = 𝜋0𝑃𝑛, ∀ 𝑛 ≥ 0.
– A normalized 𝜋 is an invariant distribution ⟺ 𝜋 = 𝜋P.
– Irreducible: MC can reach any state from any other state (possibly in multiple steps).
– Aperiodic: Let 𝑑 𝑖 ≔ 𝑔. 𝑐. 𝑑. 𝑛 ≥ 1 𝑃𝑛 𝑖, 𝑖 > 0}. An irreducible DTMC is aperiodic if 𝑑 𝑖 = 1, ∀ 𝑖. (Fact: In an irreducible DTMC, 𝑑 𝑖 is same ∀ i.)
Discrete Time Markov Chain (DTMC)
Irreducible & Periodic
Irreducible & Aperiodic
Reducible
• Theorem: Consider an irreducible DTMC over a finite state space. Then,
(a) There is a unique invariant distribution 𝜋.
(b) Long-term fraction of time (𝑋 𝑛 = 𝑖) ≔ lim𝑁→∞
1
𝑁 𝑛=0𝑁−11{𝑋 𝑛 = 𝑖} = π(𝑖).
(c) If the DTMC is aperiodic, 𝜋𝑛 → 𝜋.
• Example showing aperiodicity is necessary for (c).
• In part (c), 𝜋𝑛 → 𝜋 irrespective of the initial distribution 𝜋0.
– This implies each row of 𝑃𝑛 converges to 𝜋 as 𝑛 → ∞.
Big Theorem for Finite DTMC
0 1
Illustrations
1. Mean Hitting Time β 𝑖 ≔ 𝐸 𝑇𝐴 𝑋0 = 𝑖], 𝑖 ∈ 𝒳, A ⊂ 𝒳.
– First Step Equations (FSEs) are: 𝛽 𝑖 = 1 + 𝑗 𝑃 𝑖, 𝑗 𝛽 𝑗 , 𝑖𝑓 𝑖 ∉ 𝐴
0, 𝑖𝑓 𝑖 ∈ 𝐴
2. Probability of hitting a state before another α 𝑖 ≔ 𝑃 𝑇𝐴 < 𝑇𝐵 𝑋0 = 𝑖], 𝑖 ∈ 𝒳, 𝐴, 𝐵 ⊂ 𝒳, 𝐴 ∩ 𝐵 = ∅.
– FSEs are: 𝛼 𝑖 =
𝑗 𝑃 𝑖, 𝑗 𝛼 𝑗 , 𝑖𝑓 𝑖 ∉ 𝐴 ∪ 𝐵
1, 𝑖𝑓 𝑖 ∈ 𝐴0, 𝑖𝑓 𝑖 ∈ 𝐵
3. Average discounted reward 𝛿 𝑖 ≔ 𝐸 𝑍 𝑋0 = 𝑖], 𝑖 ∈ 𝒳, where 𝑍 = 𝑛=0
𝑇𝐴 𝛽𝑛ℎ 𝑋 𝑛 ,𝐴 ⊂ 𝒳 with 0 < 𝛽 ≤ 1 as the discount factor.
– FSEs are: 𝛿 𝑖 = ℎ 𝑖 + 𝛽 𝑗 𝑃 𝑖, 𝑗 𝛿 𝑗 , 𝑖𝑓 𝑖 ∉ 𝐴
ℎ 𝑖 , 𝑖𝑓 𝑖 ∈ 𝐴
Hitting Times
• Canonical Sample Space with outcome 𝜔 being the Markov Chain trajectory/realization:
• 𝑃 𝑋0 = 𝑖0, 𝑋1 = 𝑖1, … , 𝑋𝑛 = 𝑖𝑛 = 𝜋 𝑖0 𝑃 𝑖0, 𝑖1 …𝑃(𝑖𝑛−1, 𝑖𝑛)
• There exists a consistent probability measure over the sample space.
Markov Chain Trajectory/Realization
• Let {𝑋 𝑖 , 𝑖 ≥ 1} be a sequence of Independent and Identically Distributed (IID) RVs with mean 𝜇, and let 𝑆(𝑛) = 𝑖=1
𝑛 𝑋(𝑖). Assume 𝐸 𝑋 𝑖 < ∞.
• Strong Law of Large Numbers (SLLN): 𝑃( lim𝑛→∞
𝑆(𝑛)
𝑛= μ) = 1.
–𝑆(𝑛)
𝑛converges to 𝜇 almost surely.
• Weak Law of Large Numbers (WLLN): lim𝑛→∞𝑃(𝑆(𝑛)
𝑛− 𝜇 > 𝜖) = 0, ∀ 𝜖 > 0.
–𝑆(𝑛)
𝑛converges to 𝜇 in probability.
• Almost sure convergence implies convergence in probability.
• SLLN Example: 𝑌 𝑛 =𝑆(𝑛)
𝑛, where 𝑋 𝑖 = outcome of ith roll of a balanced die.
• Part (b) of the Big Theorem indicates almost sure convergence.
– For an irreducible DTMC over a finite numerical state space, 1
𝑛 𝑖=0𝑛−1𝑋(𝑖) converges almost
surely to 𝐸(𝑋) where 𝑋 is an RV with the distribution same as the invariant distribution 𝜋.
Laws of Large Numbers
• Let {𝑋 𝑖 , 𝑖 ≥ 1} be a sequence of Independent and Identically Distributed (IID) RVs with mean 𝜇, and let 𝑆(𝑛) = 𝑖=1
𝑛 𝑋(𝑖). Assume 𝐸 𝑋(𝑖)4 < ∞.
• Strong Law of Large Numbers (SLLN): 𝑃( lim𝑛→∞
𝑆(𝑛)
𝑛= μ) = 1.
SLLN vs. WLLN, SLLN Proof
• Ti = Time to return to i. Ti’s are IID.
Key Points from the Big Theorem Proof
T0 T1 T2 T3 T4 T5