ECE 302: Chapter 02 Probability Model - Purdue University › ChanGroup › ECE302 › files ›...

c©Stanley Chan 2019. All Rights Reserved.

ECE 302: Chapter 02 Probability Model

Fall 2019

Prof Stanley Chan

School of Electrical and Computer EngineeringPurdue University

1 / 52


1. Set Theory

2 / 52


Set

Definition

A set is a collection of objects. We denote A = ξ1, ξ2, . . . , ξn as a set,and ξi be the i-th element in the set.

Notation:

ξ ∈ A: An object ξ is in set A.

ξ 6∈ A: An object ξ is not in set A.

Finite, Countable, Uncountable.

Finite: A = 0, 1Countable: A = 2, 4, 6, 8, . . .Uncountable: A = x | 0 < x < 1

Open and Closed Intervals.

(a, b) = x | a < x < b[a, b] = x | a ≤ x ≤ b

3 / 52


Subset

Definition

A subset of A is sub-collection of objects in A. That is, B ⊆ A if for anyξ ∈ B, this ξ is also in A.

Proper subset: B ⊂ A.

Example. If A = 1, 2, 3, 4, 5, 6, then B = 1, 2, is a proper subset of A.Example. If A = 1, 2, 3, then the set B = 1, 2, 3 is an impropersubset of A.

Theorem

If A ⊆ B and B ⊆ A, then A = B.

Proof. Suppose A ⊂ B (which means A 6= B. You may also considerB ⊂ A). Then there exist x ∈ B but 6∈ A. But B ⊆ A requires all x ∈ Balso ∈ A. So we reach a contradiction. The only way to resolve thecontradiction is to make A = B.

4 / 52


Empty and Universal Set

Definition (Empty Set)

A set is empty if it contains no element. We denote an empty set as ∅.

An empty set is a subset of any set.

Definition

The universal set is the set containing all elements. We denote auniversal set as Ω.

Any set is a subset of Ω, including Ω itself.

5 / 52


Finite Union

Definition (Finite Union)

The finite union of two sets A and B contains all elements in A or in B.That is,

A ∪ B = ξ | ξ ∈ A or ξ ∈ B.

Example. If A = 1, 2, 3, 4, B = 1, 5, 6, then A ∪ B =

Example. If A = t | 3 < t ≤ 4, B = t | t ≥ 3.5, then A ∪ B =

6 / 52


Infinite Union

Definition (Infinite Union)

The infinite union of A1,A2, . . . ,An is denoted as

Adef=

∞⋃n=1

An.

It holds that x ∈ A if x is in at least one of A1,A2, . . . ,An.

Example. If An =[0, 1− 1

n

], then

(a) A = [0, 1](b) A = [0, 1)

7 / 52


Finite Intersection

Definition

The finite intersection of two sets A and B contains all elements in Aand in B. That is,

A ∩ B = ξ | ξ ∈ A and ξ ∈ B.

Example. If A = 1, 2, 3, 4, B = 1, 5, 6, then A ∩ B =

Example. If A = t | 3 < t ≤ 4, B = t | t ≥ 3.5, then A ∩ B =

8 / 52


Infinite Intersection

Definition

Define the infinite intersection of A1,A2, . . . ,An as

A =∞⋂n=1

An

Then, x ∈ A if x is in all of A1,A2, . . . ,An.

Example. If An = [0, 1 + 1n ), then

(a) A = [0, 1](b) A = [0, 1)

9 / 52


Complement

Definition

The complement of a set A is the set containing all elements in Ω but notin A. That is

Ac = ξ | ξ ∈ Ω and ξ 6∈ A.

Example. Let A = even integers, Ω = integers, then Ac =

10 / 52


Difference

Definition

The difference A\B is the set containing all elements in A but not in B.

A\B = ξ | ξ ∈ A and ξ 6∈ B.

11 / 52


Disjoint

Definition

Two sets A and B are disjoint if A ∩ B = ∅. For a collection of setsA1,A2, . . . ,An, we say that the collection is disjoint if Ai ∩ Aj = ∅.

12 / 52


Partition

Definition

A collection of sets A1, . . . ,An is a partition to the universal set Ω if itsatisfies the following conditions:

(non-overlap) A1, . . . ,An is disjoint;

(decompose) A1 ∪ A2 ∪ . . . ∪ An = Ω.

13 / 52


Set Operations

There are four set operations:

Commutative (Order does not matter).

A ∩ B = B ∩ A

A ∪ B = B ∪ A

Associative (How to do multiple union and intersection)

A ∪ (B ∪ C ) = (A ∪ B) ∪ C

A ∩ (B ∩ C ) = (A ∩ B) ∩ C

14 / 52


Set Operations

Distributive (How to mix union and intersection)

A ∩ (B ∪ C ) = (A ∩ B) ∪ (A ∩ C )

A ∪ (B ∩ C ) = (A ∪ B) ∩ (A ∪ C )

De Morgan’s Law (How to complement over intersection and union)

(A ∩ B)c = Ac ∪ Bc

(A ∪ B)c = Ac ∩ Bc

15 / 52


2. Probability Model

16 / 52


What is Probability?

It is a number.

Always between 0 and 1.

Always the probability of an event.

Example. The probability of getting a Head when tossing a coin:

P(“H”) =

17 / 52


Three Elements of a Probability Model

1 Sample Space

2 Event

3 Probability Law

18 / 52


Sample Space

Definition (Sample Space)

A sample space Ω is the collection of all possible outcomes.

We denote ω as an element in Ω.

Example.

Coin flip:Ω =

Throw a dice:Ω =

Waiting time for a bus in West Lafayette:Ω =

19 / 52


Event

Definition (Event)

An event F is a subset in the sample space Ω.

Outcome VS Event:

Example. Throw a dice. Let Ω = 1, 2, 3, 4, 5, 6.

F1 = even numbers = 2, 4, 6.F2 = less than 3 = 1, 2.

Example. Wait a bus. Let Ω = 0 ≤ t ≤ 30.F1 = 0 ≤ t < 10F2 = 0 ≤ t < 5 ∪ 20 < t ≤ 30.

20 / 52


Event Space

Definition (Event Space)

The collection of all possible events is called the

Event Space or σ-field

denoted as F . F satisfies the following two properties:

If F ∈ F , then F c ∈ F

If F1,F2, . . . ∈ F , then Fi ∩ Fj ∈ F and Fi ∪ Fj ∈ F .

Example. Ω = H,T, the event space is ∅,H,T ,Ω.

21 / 52


Probability Law

Definition

A probability law is a function P : F → [0, 1] that maps an event A to areal number in [0, 1]. The function must satisfy three axioms known asProbability Axioms.

I. Non-negativity:

II. Normalization:

22 / 52


Probability Law

III. Additivity:

For any disjoint subsets A1,A2, . . ., it holds that

P

[ ∞⋃n=1

An

]=

∞∑n=1

P[An].

23 / 52


Properties of Probability

1 P[Ac ] = 1− P[A].

2 For any A ⊆ Ω, P[A] ≤ 1.

3 P[∅] = 0.

24 / 52



4 For any A and B,

P[A ∪ B] = P[A] + P[B]− P[A ∩ B].

25 / 52



5 (Union Bound) For any A and B,

P[A ∪ B] ≤ P[A] + P[B].

26 / 52



6 If A ⊆ B, then P[A] ≤ P[B]

Example. A = t ≤ 5, and B = t ≤ 10, then P[A] ≤ P[B].

27 / 52


Example

Let the events A and B have P[A] = x , P[B] = y and P[A ∪ B] = z . Findthe following probabilities.

(a) P[A ∩ B]

(b) P[Ac ∩ Bc ]

28 / 52


Example

(c) P[Ac ∪ Bc ]

(d) P[A ∩ Bc ]

29 / 52


Probability as a Measure

P[·] is a measure of the size of the event.

Ω = 1, 2, 3, 4, 5, 6The measure P[·] is acounter

Counts the number of events

Ω = [0, 1]× [0, 1]

The measure P[·] is anintegration

Integrates the area of events

30 / 52


Probability as a Measure

P[·] is a measure of the size of the event.

Ω = 1, 2, 3, 4, 5, 6E = 1, P[E ] = 1/6

E = 1, 3, P[E ] = 2/6

P[E1] ≤ P[E2] if E1 ⊆ E2.

Ω = [0, 1]× [0, 1]

E = shaded region, P[E ] = area.

E = (x0, y0), P[E ] = 0.

P[E ] can be 0 even if E 6= ∅.

31 / 52


Measure Zero

Definition

Let Ω be the sample space. A set A ∈ Ω is said to have measure zero iffor any given ε > 0,

There exists a countable number of subsets An such thatA ⊆ ∪∞n=1An, and∑∞

n=1 P[An] < ε.

32 / 52


Measure Zero Example

Example. Let Ω = [0, 1]. Then the set 0.5 ⊂ Ω has measure zero.

Fix any ε.

Create a very small interval around 0.5.

Shrink the radius of the interval to make sure it is less than ε.

Facts.

Discrete space: No issue about measure zero.

Example. 1 in 1, . . . , 6 has non-zero probability.

Continuous space: Isolated points have measure zero.

Example. 1 in [1, 6] has zero probability.

33 / 52


Almost Surely

Definition

An event A ∈ R is said to hold almost surely (a.s.) if

P[A] = 1,

except for all measure-zero sets in R.

Example. Let Ω = [0, 1]. Then

P[(0, 1)] = 1 (almost surely)

because the points 0 and 1 have measure zero in Ω.

Note: In this course we will skip “a.s.” if the context is clear.

34 / 52


3. Conditional Probability

35 / 52


Definition (Conditional Probability)

Assume P[B] 6= 0. The conditional probability of A given B is

Difference:

P[A |B] =P[A ∩ B]

P[B]and P[A ∩ B] =

P[A ∩ B]

P[Ω].

36 / 52


Understanding Conditional Probability

https://ocw.mit.edu/resources/

res-6-012-introduction-to-probability-spring-2018/part-i-the-fundamentals/

37 / 52

https://ocw.mit.edu/resources/res-6-012-introduction-to-probability-spring-2018/part-i-the-fundamentals/



Example

https://ocw.mit.edu/resources/

res-6-012-introduction-to-probability-spring-2018/part-i-the-fundamentals/

38 / 52




Example

Conditional probability can appear in daily life:

A = You are admitted to Harvard

B = Your SAT score is 2200 or above

The probabilities are:

P[A] very small (4.5% in 2019 1)

P[B] hard, but not too bad

P[A|B] may not be very high, because a lot of applicants areoutstanding

P[B|A] probably higher, because you will not be reviewed if your SATis low

And do not forget, the 4.5% is a conditional probability conditioned onthose who applied only.

1https://www.thecrimson.com/article/2019/3/29/2023-admit-numbers/39 / 52

https://www.thecrimson.com/article/2019/3/29/2023-admit-numbers/


Bayes Theorem

Theorem (Bayes Theorem)

For any two events A and B such that P[A] > 0 and P[B] > 0, it holdsthat

P[A |B] =P[B |A]P[A]

P[B].

40 / 52


Law of Total Probability

Theorem (Law of Total Probability)

Let A1,A2, . . . ,An be a partition of Ω, i.e., A1, . . . ,An are disjoint andΩ = A1 ∪ A2 ∪ . . . ∪ An. Then, for any B ⊆ Ω,

P[B] =n∑

i=1

P[B |Ai ] P[Ai ].

41 / 52


Law of Total Probability

Figure: Law of total probability decomposes the probability P[B] into multipleconditional probabilities P[B |Ai ]. The probability of obtaining each P[B |Ai ] isP[Ai ].

42 / 52


Example 1: Tennis Tournament

Consider a tennis tournament. Your probability of winning the game is

0.3 against1

2of the players (Event A).

0.4 against1

4of the players (Event B).

0.5 against1

4of the players (Event C).

What is the probability of winning the game?

43 / 52


Example 2: Communication Channel

Consider a communication channel shown below. The probability ofsending a 1 is p and the probability of sending a 0 is 1− p. Given that 1 issent, the probability of receiving 1 is 1− η. Given that 0 is sent, theprobability of receiving 0 is 1− ε.

Find P[Receive 1]

Find P[Send 1]

Find P[Receive 1 | Send 1]

Find P[Send 1 | Receive 1]

44 / 52


Properties of Conditional Probability

Theorem

Let P[B] > 0. The conditional probability P[A |B] satisfies Axiom I toAxiom III.

Proof.

Axiom I:

Axiom II:

Axiom III:

45 / 52


Independence

Definition

Two events A and B are statistically independent if

Disjoint VS Independent.

46 / 52


Examples of Independence

Example 1. Throw a dice twice. Let

A = 1st dice is 3 and B = 2nd dice is 4.Are A and B independent?

47 / 52




A = 1st dice is 1 and B = sum is 7.

Are A and B independent?

Think about P[A|B].

If you know the sum is 7, then the pair has to be (1,6), (2,5), (3,4),(4,3), (5,2), (6,1).

The chance of getting first dice = 1 is still 1/6. It has been not beenchanged by B.

So independent.

How about you change B = sum is 8?

48 / 52




A = 1st dice is 2 and B = sum is 8.

Are A and B independent?

Think about P[A|B].

If you know the sum is 8, then the pair has to be (2,6), (3,5), (4,4),(5,3), (6,2).

The chance of getting first dice = 2 is no longer 1/6. It has beenchanged by B.

So dependent.

49 / 52




A = max is 2 and B = min is 2.Are A and B independent?

50 / 52


Independence Via Conditional Probability

Recall that P[A |B] = P[A∩B]P[B] .

If A and B are independent, then P[A ∩ B] = P[A]P[B]

Therefore,

P[A |B] =P[A ∩ B]

P[B]=

P[A] P[B]

P[B]= P[A].

Interpretation.

51 / 52


Prisoner’s Dilemma

Three Prisoners: A, B, C. The King decides to release 2 and kill 1.You were A.Your chance of release is 2/3.Suppose you know the guard well. You can ask him about which of Bor C will be released.But if you find out B (or C) is released, your chance becomes 1/2.How come!!

52 / 52

ECE 302: Chapter 02 Probability Model - Purdue University › ChanGroup › ECE302 › files ›...

Documents

Transcript of ECE 302: Chapter 02 Probability Model - Purdue University › ChanGroup › ECE302 › files ›...