Ruben Zamar Department of Statistics UBCruben/Stat302Website/Lectures/Conditiona… · Department...

Conditional Probability and Independence

Ruben ZamarDepartment of Statistics

January 22, 2019

Ruben Zamar Department of Statistics UBC ()Chapter 3 January 22, 2019 1 / 63

MOTIVATION

The outcome could be any element in the Sample Space, Ω.

Sometimes the range of possibilities is restricted because of “partialinformation”

Examples

number of shots:

partial info: we know it wasn’t an “ace”

ELEC 321 final grade:

partial info: we know it is at least a “B”

CONDITIONING EVENT

The event B representing the “partial information” is called“conditioning event”

Denote by A the event of interest

Example (Number of Shots)

B = 2, 3, ... = not an “ace” (conditioning event)

A = 1, 3, 5, ... = server wins (event of interest)

Example (Final Grade)

B = [70, 100] = at least a “B” (conditioning event)

A = [80, 100] = an “A” (event of interest)

DEFINITION OF CONDITIONAL PROBABILITY

Suppose that P (B) > 0

P (A|B) =P (A∩ B)P (B)

The left hand side is read as “probability of A given B”

Useful formulas:

P (A∩ B) = P (B)P (A|B)

= P (A)P (B |A)

CONDITIONAL PROBABILITY

P (A|B) , as a function of A (and for B fixed) satisfies all theprobability axioms:

- P (Ω|B) = P (Ω ∩ B) /P (B) = P (B) /P (B) = 1

- P (A|B) ≥ 0

- If Ai are disjoint then

P (∪Ai |B) =P [(∪Ai ) ∩ B ]

=P [∪ (Ai ∩ B)]

=∑P (Ai ∩ B)

P (B)= ∑P (Ai |B)

EXAMPLE: NUMBER OF SHOTS

For simplicity, suppose that points are decided in at most 8 shots,with probabilities:

Shots 1 2 3 4 5 6 7 8Prob. 0.05 0.05 0.15 0.10 0.20 0.10 0.20 0.15

Using the table above:

P (Sever wins | Not an ace) =P (3, 5, 7)

P (2, 3, 4, 5, 6, 7, 8)

=0.550.95

= 0.579

EXAMPLE: FINAL GRADE

Suppose that

P (Grade is larger than x) =100− x100

= 1− x100

Using the formula above:

P (To get an “A” | To get at least a “B”) =P ([80, 100])P ([70, 100])

=100− 80100− 70 =

= 0.667

SCREENING TESTS

Items are submitted to a screening test before shipment

The screening test can result in either

POSITIVE (indicating that the item may have a defect)

NEGATIVE (indicating that the item doesn’t have a defect)

Screening tests face two types of errors

FALSE POSITIVE

FALSE NEGATIVE

SCREENING TESTS (continued)

For each item we have 4 possible events

Item true status:

D = item is defective

Dc = item is not defective

Test result:

B = test is positive

Bc = test is negative

SCREENING TESTS (continued)

The following conditional probabilities are normally known

Sensitivity of the test: P (B |D) = 0.95 (say)

Specificity of the test: P (Bc |Dc ) = 0.99 (say)

which implies

P (Bc |D) = 0.05 and P (B |Dc ) = 0.01

The proportion of defective items is also normally known

P (D) = 0.02 (say)

TEST PERFORMANCE

The following questions may be of interest:

What is the probability that a randomly chosen item tests positive?

What is the probability of defective given that the test resultednegative?

What is the probability of defective given that the test resultedpositive?

What is the probability of screening error?

We will compute these probabilities

PROBABILITY OF TESTING POSITIVE

P (B) = P (B ∩D) + P (B ∩Dc )

= P (D)P (B |D) + P (Dc )P (B |Dc )

= 0.02× 0.95+ (1− 0.02)× 0.01

= 0.0288

PROB OF DEFECTIVE GIVEN A POSITIVE TEST

P (D |B) =P (D ∩ B)P (B)

=P (D)P (B |D)

=0.02× 0.950.0288

= 0.65972

PROB OF DEFECTIVE GIVEN A NEGATIVE TEST

P (D |Bc ) =P (D ∩ Bc )P (Bc )

=P (D)P (Bc |D)1− P (B)

=0.0098

1− 0.0288

= 0.01

SCREENING ERROR

P (Error) = P (D ∩ Bc ) + P (Dc ∩ B)

= P (D)P (Bc |D) + P (Dc )P (B |Dc )

= 0.02× (1− 0.95) + (1− 0.02)× 0.01

= 0.0108

BAYES’FORMULA

The formula

P (D |B) =P (D ∩ B)P (B)

=P (B |D)P (D)

P (B |D)P (D) + P (B |Dc )P (Dc )

is the simple form of Bayes’formula.

This has been used in the "Screening Example”presented before.

BAYES’FORMULA (continued)

The general form of Bayes’Formula is given by

P (Di |B) =P (Di ∩ B)P (B)

=P (B |Di )P (Di )

∑kj=1 P (B |Dj )P (Dj )

where D1, D2, ..., Dk is a partition of the sample space Ω:

Ω = D1 ∪D2 ∪ · · · ∪Dk

Di ∩Dj = φ, for i 6= j

EXAMPLE: THREE PRISONERS

Prisoners A, B and C are to be executed

The governor has selected one of them at random to be pardoned

The warden knows who is pardoned, but is not allowed to tell

Prisoner A begs the warden to let him know which one of the othertwo prisoners is not pardoned

Prisoner A tells the warden: “Since I already know that one of theother two prisioners is not pardoned, you could just tell me who isthat”

Prisoner A adds: “If B is pardoned, you could give me C’s name. IfC is pardoned, you could give me B’s name. And if I’m pardoned, youcould flip a coin to decide whether to name B or C.”

The warden is convinced by prisoner A’s arguments and tells him: “B isnot pardoned”

Result: Given the information provided by the Warden, C is nowtwice more likely to be pardoned than A!

Why? Check the derivations below:

NOTATION:A = A is pardonedB = B is pardonedC = C is pardoned

b = The warden says “B is not pardoned”Clearly

P (A) = P (B) = P (C ) =13

P (b|B) = 0 (warden never lies)

P (b|A) = 1/2 (warden flips a coin)

P (b|C ) = 1 (warden cannot name A)

By the Bayes’formula:

P (A|b) =P (b|A)P (A)

P (b|A)P (A) + P (b|B)P (B) + P (b|C )P (C )

=12 ×

13 + 0×

13 + 1×

HenceP (C |b) = 1− P (A|b) = 1− 1

SCREENING EXAMPLE II

The tested items have two components: “c1”and “c2”

Suppose

D1 = Only component “c1” is defective , P (D1) = 0.01

D2 = Only component “c2” is defective , P (D2) = 0.008

D3 = Both components are defective , P (D3) = 0.002

D4 = Both components are non defective , P (D4) = 0.98

SCREENING EXAMPLE II (continued)

LetB = Screening test is positive

Suppose

P (B |D1) = 0.95

P (B |D2) = 0.96

P (B |D3) = 0.99

P (B |D4) = 0.01

SOME QUESTIONS OF INTEREST

The following questions may be of interest:

What is the probability of testing positive?

What is the probability that component “ci” (i = 1, 2) is defectivewhen the test resulted positive?

What is the probability that the item is defective when the test resultednegative?

What is the probability both components are defective when the testresulted positive?

What is the probability of testing error?

We will compute these probabilities

PROB OF TESTING POSITIVE

P (B) = P (B ∩D1) + P (B ∩D2) + P (B ∩D3) + P (B ∩D4)

= 0.01× 0.95+ 0.008× 0.96+ 0.002× 0.99+ 0.98× 0.01

= 0.02896

Notice that the probability of defective is

P (D) = 0.01+ 0.008+ 0.002 = 0.02

PROB OF TESTING NEGATIVE

P (Bc ) = P (Bc ∩D1) + P (Bc ∩D2) + P (Bc ∩D3) + P (Bc ∩D4)

= 0.01× 0.05+ 0.008× 0.04+ 0.002× 0.01+ 0.98× 0.99

= 0.97104

Naturally,P (B) + P (Bc ) = 0.02896+ 0.97104 = 1

TEST RESULTED POSITIVE

The posterior probabilities given this “data”are:

P (D1|B) =0.01× 0.950.02896

= 0.32804

P (D2|B) =0.008× 0.960.02896

= 0.26519

P (D3|B) =0.002× 0.990.02896

= 0.06837

P (D4|B) =0.98× 0.010.02896

= 0.338 40

P (defective|B) = 1− P (D4|B) = 1− 0.338 40 = 0.6616

TEST RESULTED NEGATIVE

The posterior probabilities given this “data”are:

P (D1|Bc ) =0.01× 0.050.97104

= 0.00051491

P (D2|Bc ) =0.008× 0.040.97104

= 0.00032954

P (D3|Bc ) =0.002× 0.010.97104

= 0.000020596

P (D4|Bc ) =0.98× 0.990.97104

= 0.99913

P (defective|Bc ) = 1− P (D4|Bc ) = 1− 0.99913 = 0.00087

CONCLUSION

Prior reliability of items being sold:

P (Defective) = P (D1) + P (D2) + P (D3) = 0.01+ 0.008+ 0.002

= 0.02

Posterior reliability of items being sold:

P (Defective|Bc ) = 1− P (D4|Bc ) = 1− 0.99913= 0.00087 < 0.001

COST - BENEFIT ANALYSIS

Cost: possibly discarding a small percentage of non-defective items

P (“Testing Positive”∩ “Non-Defective”) = P (B ∩D4)= P (D4|B)P (B)= 0.33840× 0.02896 < 0.01

Benefit: Relative reliability improvement:

P (Defective)− P (Defective|Bc ) = 0.02− 0.00087= 0.019 13

Notice that0.02/0.00087 > 22

INDEPENDENCE

DEFINITION: Events A and B are independent if

P (A∩ B) = P (A)P (B)

If A and B are independent then

P (A|B) =P (A∩ B)P (B)

=P (A)P (B)P (B)

= P (A)

P (B |A) =P (A∩ B)P (A)

=P (A)P (B)P (A)

= P (B)

DISCUSSION

If P (A) = 1, then A is independent of all B.

P (A∩ B) = P (A∩ B) +=0︷︸︸︷

P (Ac ∩ B) = P (B)

P (A∩ B) =

=1︷︸︸︷P (A)P (B)

DISCUSSION (Cont)

Suppose that A and B are non-trivial events ( 0 < P (A) < 1 and0 < P (B) < 1 )

If A and B are mutually exclusive ( A∩ B = φ ) then they cannot beindependent because

P (A|B) = 0 < P (A)

If A ⊂ B then they cannot be independent because

P (A|B) = P (A∩ B)P (B)

=P (A)P (B)

> P (A)

DISCUSSION (Cont)

Suppose Ω = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 and the numbers are equallylikely.

A = 1, 2, 3, 4, 5 and B = 2, 4, 6, 8

P (A∩ B) = P (2, 4) = 0.20, P (A)P (B) = 0.5× 0.4 = 0.20

Hence, A and B are independent

In terms of probabilities A is half of Ω. On the other hand A∩ B ishalf of B.

DISCUSSION (continued)

What happens if

P (i) =i55

P (A∩ B) = P (2, 4) = 6/55 = 0.10909P (A)P (B) = (15/55)× (20/55) = 0.099174

Hence, A and B are not independent in this case.

MORE THAN TWO EVENTS

Definition: We say that the events A1,A2, ...,An are independent if

P (Ai1 ∩ Ai2 ∩ · · · ∩ Aik ) = P (Ai1)P (Ai2) · · ·P (Aik )

for all 1 ≤ i1 < i2 < · · · < ik ≤ n, and all 1 ≤ k ≤ n.

For example, if n = 3, then

P (A1 ∩ A2) = P (A1 )P (A2)

P (A1 ∩ A3) = P (A1)P (A3)

P (A2 ∩ A3) = P (A2)P (A3)

P (A1 ∩ A2 ∩ A3) = P (A1)P (A2)P (A3)

SYSTEM OF INDEPENDENT COMPONENTS

In series

→ a → b → c →

In parallel

NOTATION

A = Component a works

B = Component b works

C = Component c works

INDEPENDENT COMPONENTS

We assume that A, B and C are independent, that is

P (A∩ B ∩ C ) = P (A)P (B)P (C )

P (A∩ B) = P (A)P (B) ,

P (B ∩ C ) = P (B)P (C ) ,

P (A∩ C ) = P (A)P (C )

RELIABILITY CALCULATION

Problem 1: Suppose that

P (A) = P (B) = P (C ) = 0.95.

Calculate the reliability of the system

→ a → b → c →

Solution:

P (System Works) = P (A∩ B ∩ C )

= P (A)P (B)P (C )

= 0.953 = 0.857

PRACTICE

P (A) = P (B) = P (C ) = 0.95.

PROBLEM 2 (Solution)

P (System works) = 1− P (System fails)

= 1− P (Ac ∩ Bc ∩ C c )

= 1− P (Ac )P (Bc )P (C c )

= 1− (1− P (A)) (1− P (B)) (1− P (C ))

= 1− 0.053 = 0.99988

PRACTICE

P (A) = P (B) = P (C ) = P (D) = 0.95.

subsys I

subsys II

PROBLEM 3 (Solution)

P (System works) = P (subsys I works ∩ subsys II works)

= P (subsys I works )P (subsys II works)

= [1− P (subsys I fails )] [1− P (subsys II fails )]

= [1− P (Ac ∩ Bc )] [1− P (C c ∩Dc )]

= [1− P (Ac )P (Bc )] [1− P (C c )P (Dc )]

= [1− (1− P (A)) (1− P (B))] [1− (1− P (C )) (1− P (D))]

=(1− 0.052

)2= 0.99501

CONDITIONAL INDEPENDENCE

Definition: We say that the events T1,T2, ...,Tn are conditionallyindependent given the event B if

P (Ti1 ∩ Ti2 ∩ · · · ∩ Tik | B) = P (Ti1 | B)P (Ti2 | B) · · ·P (Tik | B)

for all 1 ≤ i1 < i2 < · · · < ik ≤ n, and all 1 ≤ k ≤ n.

For example, if n = 3, then

P (T1 ∩ T2 | B) = P (T1 | B)P (T2 | B)

P (T1 ∩ T3 | B) = P (T1 | B)P (T3 | B)

P (T2 ∩ T3 | B) = P (T2 | B)P (T3 | B)

P (T1 ∩ T2 ∩ T3 | B) = P (T1 | B)P (T2 | B)P (T3 | B)

Conditional independence doesn’t imply unconditional independenceand vice versa

Conditional independence given B doesn’t imply conditionalindependence given Bc

However, usually both conditional independences are assumedtogether in applications

We will apply this concept in Bayesian probability updating

SEQUENTIAL BAYES’FORMULA (Bonus material)

Let Si be the outcome of the ith test. For instance

S1 =The 1th test is positive

The 2th test is negative

The 3th test is negative

and so on

The outcomes Si (i = 1, 2, ..., n) are available in a sequential fashion.

Let Ik = S1 ∩ S2 ∩ · · · ∩ Sk (data available at step k) and set

π0 = P (D) Prior prob of an item being defective

π1 = P (D |I1) = P (D |S1) Posterior prob given S1

π2 = P (E |I2) = P (E |S1 ∩ S2) Posterior prob given S1 and S2

π3 = P (E |I3) = P (E |S1 ∩ S2 ∩ S3) Posterior prob given S1,S2 and S3

and so on

CONDITIONAL INDEPENDENCE ASSUMPTION

Assume that the Si (i = 1, 2, ..., n) are independent given E and alsogiven E c .

Then, for k = 1, 2, ..., n

πk =P (Sk |D)πk−1

P (Sk |D)πk−1 + P (Sk |Dc ) (1− πk−1)

πk =P (Ik |D)π0

P (Ik |D)π0 + P (Ik |Dc ) (1− π0)

=P (Ik−1 ∩ Sk |D)π0

P (Ik−1 ∩ Sk |D)π0 + P (Ik−1 ∩ Sk |Dc ) (1− π0)

=P (Sk |D)P (Ik−1|D)π0

P (Sk |D)P (Ik−1|D)π0 + P (Sk |Dc )P (Ik−1|Dc ) (1− π0)

(By the cond. independence assumption)

Proof (continued)

πk =P (Sk |D)P (Ik−1 ∩D)

P (Sk |E )P (Ik−1 ∩ E ) + P (Sk |E c )P (Ik−1 ∩ E c )

=P (Sk |E )P (Ik−1 ∩ E ) /P (Ik−1)

[P (Sk |E )P (Ik−1 ∩ E ) + P (Sk |E c )P (Ik−1 ∩ E c )] /P (Ik−1)

=P (Sk |E )πk−1

P (Sk |E )πk−1 + P (Sk |E c ) (1− πk−1), πk−1 = P (E |Ik−1)

Pseudo Code

Input:

(S1, S2, S3, ...,Sn) = (1, 0, 1, ..., 0) (outcomes for the n tests)

π = P (E ) (prob of event of interest, for instance E= "the part isdefective")

pk = P (Sk = +|E ) k = 1, 2, ..., n (Sensitivity of kth test)

qk = P (Sk = −|E c ) k = 1, 2, ..., n (Specificity of kth test)

Output πk = P (E |S1 ∩ S2 ∩ · · · ∩ Sk ) , k = 1, 2, ..., n

Pseudo Code (continued)

Example of Input:

n = 4, π = 0.05

Test Results = (1, 1, 0, 1)

k pk = P (1|Defective) 1− qk = P (1|Non Defective)1 p1 = 0.80 1− q1 = 0.052 p2 = 0.78 1− q2 = 0.103 p3 = 0.85 1− q3 = 0.204 p4 = 0.82 1− q4 = 0.15

Pseudo Code - Computation

Computation of πk

1) Initialization: Set π0 = π

2) k-step:

If Sk = 1, set a = pk and b = 1− qk

If Sk = 0, set a = 1− pk and b = qk

3) Computing πk :

πk =aπk−1

aπk−1 + b (1− πk−1)

Example: Simple Spam Email Detection

When you receive an email, your spam fillter uses Bayes rule to decidewhether it is spam or not.

Basic spam filters check whether some pre-specified words appear inthe email; e.g.

diplomat,lottery,money,inheritance,president,sincerely,huge,....

We consider n events Wi telling us whether the ith pre-specified wordis in the message

E = e-mail is spam

Wi = word i is in the message , i = 1, 2, ..., n

Assume that W1,W2, ...,Wn are conditionally independent given Eand also E c .

Human examination of a large number of messages is used estimateπ0 = P (E )

The training data is also used to estimate pi = P (Wi |E ) and1− qi = P (Wi |E c )

Let In = S1 ∩ S2 ∩ · · · ∩ Sn, where Si is either Wi or W ci .

The spam filter assumes that the Wi are conditionally independent(given E and given E c ) to compute

P (E |In) =P (In |E )P (E )

P (In |E )P (E ) + P (In |E c )P (E c )

Sequential Updating

The posterior probs πk = P (E |Ik ) (k = 1, 2, ..., n− 1) can becomputed sequentially using the formula

πk = P (E | Ik ) =P (Sk |E )P (E |Ik−1)

P (Sk |E )P (E |Ik−1) + P (Sk |E c )P (E c |Ik−1)

=P (Sk |E )πk−1

P (Sk |E )πk−1 + P (Sk |E c ) (1− πk−1)

An early decision to classify the e-mail as spam can be made ifP (E | Ik ) becomes too large (or too small).

Numerical Example

For a simple numerical example consider a case with

n = 8 words, P (Spam) = 0.10

Word P(Word Present | Spam) P(Word Absent | No Spam)Sensitivity Specificity

W1 0.74 0.98W2 0.83 0.88W3 0.88 0.89W4 0.75 0.99W5 0.82 0.85W6 0.73 0.89W7 0.77 0.93W8 0.86 0.92

Word Word Status P(Spam | Ik )W1 1 0.804W2 0 0.443W3 0 0.097W4 1 0.889W5 0 0.630W6 1 0.919W7 1 0.992W8 1 0.999

Ruben Zamar Department of Statistics UBCruben/Stat302Website/Lectures/Conditiona… · Department...

Documents

Transcript of Ruben Zamar Department of Statistics UBCruben/Stat302Website/Lectures/Conditiona… · Department...

Statistics 101

Practical Statistics for Physicists

Summary Boltzman statistics: Fermi-Dirac statistics: Bose-Einstein statistics: Maxwell-Boltzmann statistics: Problem 13-4: Show that for a system of N.

Introduction to Statistics

Ruben Zamar Department of Statistics UBCruben/Stat321Website/Lectures/Slides2.pdf · Ruben Zamar Department of Statistics UBC ()Module 2 January 16, 2017 3 / 61. DEFINITION OF CONDITIONAL

Kollias Statistics Libre

Theory of Statistics - University of Arizonamath.arizona.edu/~jwatkins/notests.pdf · Theory of Statistics ... 1.2 Bayesian Statistics ... 5.2 Classical Decision Theory ...

Statistics II Factorial ANOVA

Instrumental-Variable - University of British Columbiaruben/website/Santalo_School_2014_I.pdf · Instrumental-Variable Ruben Zamar Department of Statistics UBC March 12, 2014 Ruben

Applied statistics lecture_4

AP STATISTICS EXAM REVIEW

Statistics Consulting Cheat Sheet - statweb.stanford.edustatweb.stanford.edu/~kriss1/statistics-consulting-cheat.pdf · Statistics Consulting Cheat Sheet Kris Sankaran October 1,

Basic Business Statistics - Αρχικήmba.teipir.gr/files/2nd_lecture.pdf · Basic Business Statistics 11th Edition ... In this chapter, you learn: ... 2 Basic Business Statistics,

Inferential statistics 1

Population statistics at regional level/el Statistics ...ec.europa.eu/eurostat/statistics-explained/pdfscache/19876.pdf · Population statistics at regional level/el Statistics Explained

GREECE - statistics

F. Branzoli ¶ , P. Carretta ¶ , M. Filibian ¶ , S. Klytaksaya ‡ and M. Ruben ‡

Statistics 2 Exercises - WU

Ecological Statistics

LGIR SMS STATISTICS 2015