Sample space and probability - Purdue Engineering · 1. Sample Space and Probability Part I ECE 302...

1.SampleSpaceandProbabilityPartI

ECE302Fall2009TR3‐4:15pmPurdueUniversity,SchoolofECE

Prof.IlyaPollak

ProbabilisJcModel

•  SamplespaceΩ=thesetofallpossibleoutcomesofanexperiment– E.g.,{+$1,‐$1}foraone‐monthstockmovementintheopJonexample;{$102,$100,$98}forthestockvalueaUertwomonths;{Obama,McCain,neither}foravoter’spreference,etc.

•  ProbabilitylawwhichassignstoasetAofpossibleoutcomes(alsocalledanevent)anumberP(A),calledtheprobabilityofA.

Sets:basictermsandnotaJon,1

€

A countably infinite (or countable) set is a set with infinitelymany elements which can be enumerated in a list, e.g.,the set of all integers 0,−1,1,−2,2,…{ }An example of an uncountable set is the set of all real numbersbetween 0 and 1, denoted [0,1]

Sets:basictermsandnotaJon,2

€

The set of all x that have a certain property P is denoted byx | x satisfies P{ },

e.g., the interval [0,1] can alternatively be written as x | 0 ≤ x ≤1{ }.

SetoperaJons:complement

SetoperaJons:union

€

More generally,

Snn=1

∞

= S1∪ S2 ∪… = x | x ∈ Sn for some n{ }

SetoperaJons:intersecJon

€

More generally,

Snn=1

∞

= S1∩ S2 ∩… = x | x ∈ Sn for all n{ }

Disjointsets

ParJJon

CollecJvelyexhausJvesets

Example:threecointosses

H

H

H

T T

T

H

T

H

H

T

H

T

T

HHH

HHT

HTH

HTT

THH

THT

TTH

TTT

€

Sample space Ω = HHH,HHT,HTH,HTT,THH,THT,TTH,TTT{ }Event "heads on the first and second toss" is the set HHH,HHT{ }

Example:presidenJalelecJon% for Obama

% for McCain

100%

100%


% for McCain

100%

100%

Note: Even though the sample space is discrete in reality (the quantum is 1 voter out of 130M, or 0.00000077%), it is more conveniently modeled as continuous.


% for McCain

100%

100%

Is this sample space enough for answering these questions: (1) Will Obama win the popular vote? (2) Will Obama win the election?


% for McCain

100%

100%

Is this sample space enough for answering these questions: (1) Will Obama win the popular vote? (2) Will Obama win the election? (1) No: need to include 3rd-party candidates (2) No: need both 3rd-party candidates and electoral

(not popular) vote


% for McCain

100%

100%

ProbabilisJcModel

•  SamplespaceΩ•  Probabilitylaw:assignstoaneventAanumberP(A)saJsfyingthefollowingprobabilityaxioms:

€

1. P(A) ≥ 02. If A∩ B =∅, then P(A∪ B) = P(A) + P(B)

If A1,A2,… are disjoint, then P Ann=1

∞

= P(An )

n=1

∞

∑

3. P(Ω) =1

RelaJonshipbetweenprobabilityandrelaJvefrequencyofoccurrence


€

Suppose all outcomes are equally likelyP(Ω) =1 by axiom 3P HHH{ }( ) + P HHT{ }( ) +…+ P TTT{ }( ) = P(Ω) by axiom 2

P HHH{ }( ) = P HHT{ }( ) =… = P TTT{ }( ) =1/8

P(S4 ) = P("two tails in a row") = P HTT,TTH,TTT{ }( ) = 3/8 by axiom 2

€

Discrete uniform probability law : if Ω consists of N equally likelyoutcomes, then, for any event A,

P(A) =number of elements in A

N


% for McCain

100%

100%

Example:someproperJesofprobabilitylaws


€

(a) If A ⊂ B then P(A) ≤ P(B)Proof : Let C = Ac ∩ BThen B = A∪Cand C∩ A =∅ (since C ⊂ Ac ).Therefore,

P(B) = P(A∪C) =Axiom 2

P(A) + P(C) ≥Axiom 1 applied to P (C )

P(A)


€

(c) P(A∪ B) ≤ P(A) + P(B)Proof :Use property (b) : P(A∪ B) = P(A) + P(B) − P(A∩ B)

≥0, by Axiom 1

≤ P(A) + P(B)


€

(d) P(A∪ B∪C) = P(A) + P(Ac ∩ B) + P(Ac ∩ Bc ∩C)Proof :Since A, Ac ∩ B, Ac ∩ Bc ∩C form a partition for A∪ B∪C,the statement follows from Axiom 2.

CondiJonalProbability•  NotaJon:P(A|B)•  ProbabilityofeventA,giventhateventBoccurred•  DefiniJon:assumingP(B)≠0,

•  CondiJonalprobabiliJesspecifyaprobabilitylawonthenewuniverseB(exercise)

Example:ProblemofPoints•  Besttwooutofthreefaircoinflips•  HelenbetsonH,TombetsonT

(a) What’stheprobabilitythatHelenwins1stround?

(b) What’stheprobabilitythatHelenwinsoverall?(c)  ThegameisinterruptedaUerHelenwins1stround.

What’sthecondiJonalprobabilitythatshewouldhavewonoverall?

•  SoluJon(a)  ½‐‐followsdirectlyfromthefactthatthecoinisfair.

ProblemofPoints,SoluJon

H

H

H

T T

T

H

T

H

H

T

H

T

T

HHH

HHT

HTH

HTT

THH

THT

TTH

TTT

H wins 1st round

H wins overall

P(H wins 1st round) = 1/2 P(H wins 1st round and H wins overall) = 3/8

P(H wins overall | H wins 1st round) = P(H wins 1st round and H wins overall)/ P(H wins 1st round) = (3/8)/(1/2) = 3/4

AlternaJvely,…

HHH

HHT

HTH

HTT

H wins 1st round

H wins overall View the blue set as the new sample space. Three out of four equally likely outcomes result in H’s overall win. Therefore, P(H wins overall | H wins 1st round) = 3/4.

Example1.9:IntrusionDetecJonEvent A: intrusion Event B: alarm

Suppose that, from past experiences, we know that P(A) = 0.02, P(B|Ac) = 0.05, P(Bc|A) = 0.01

Find P(B), P(A|B), the missed detection probability P(Bc and A) , the false alarm probability P(B and Ac)

Example:IntrusionDetecJon

A Bc∩A

Event A: intrusion Event B: alarm



Ac

B∩A

B∩Ac

Bc∩Ac


P(A) = 0.02 A

Bc∩A = missed detection

P(B|Ac)=0.05




Ac

B∩A

B∩Ac = false alarm

Bc∩Ac

P(Bc|A)=0.01


P(A) = 0.02 A


P(B|Ac)=0.05




Ac

B∩A


Bc∩Ac

P(Bc|A)=0.01

P(Bc∩A) = P(A)P(Bc|A) = 0.02·0.01 = 0.0002


P(A) = 0.02 A


P(B|Ac)=0.05




P(Ac) = 0.98 Ac

B∩A


Bc∩Ac

P(Bc|A)=0.01

P(Bc∩A) = P(A)P(Bc|A) = 0.02·0.01 = 0.0002

Example1.18:False‐PosiJvePuzzle

•  Acrimecommikedonanisland,populaJon5000

•  Apriori,allareequallylikelytohavecommikedit•  Basedonaforensictest,asuspectisarrested•  Apartfromthetest,nootherevidence

•  Accuracyofthetestis99.9%,i.e.,P(testisposiJve|suspectisguilty)=0.999andP(testisnegaJve|suspectisinnocent)=0.999

•  Youareonthejury.Doyouhave“reasonabledoubt”?

False‐PosiJvePuzzle:SoluJon

•  Proceedsimilartotheintruderexample

•  P(+)=P(+andguilty)+P(+andinnocent)=P(+|guilty)P(guilty)+P(+|innocent)P(innocent)=0.999∙0.0002+0.001∙0.9998=0.0011996

•  P(guilty|+)=P(+andguilty)/P(+)=0.999∙0.0002/0.0011996≈0.167!!!

•  Theseeminglyreliabletestisnotveryreliableatall!

•  Supposethisisrepeatedon1000islands•  Supposewetestall5,000,000peopleon1000islands

False‐PosiJvePuzzle:SomeIntuiJon

•  Supposethisisrepeatedon1000islands•  Supposewetestall5,000,000peopleon1000islands•  Onaverage,weexpectroughlythefollowingtestresults:


5,000,000 people

1000 guilty

4,999,000 innocent

~1 tests -

~999 test +



5,000,000 people

1000 guilty

4,999,000 innocent

~1 tests -

~999 test +

~4999 test +

~4,994,001 test -



5,000,000 people

1000 guilty

4,999,000 innocent

~1 tests -

~999 test +

~4999 test +

~4,994,001 test -

~999 criminals out of ~5998 who tested + i.e., roughly 1/6


•  I.e.,therearesofewcriminalsthatthebulkofpeoplewhotestposiJveareinnocent!


5,000,000 people

1000 guilty

4,999,000 innocent

~1 tests -

~999 test +

~4999 test +

~4,994,001 test -

~999 criminals out of ~5998 who tested + i.e., roughly 1/6

TotalProbabilityTheorem

A1

A2 A3B


•  OnewayofcompuJngP(B):P(B)=P(B∩A1)+P(B∩A2)+P(B∩A3)

=P(B|A1)P(A1)+P(B|A2)P(A2)+P(B|A3)P(A3)

A1

A2 A3B


•  OnewayofcompuJngP(B):P(B)=P(B∩A1)+P(B∩A2)+P(B∩A3)

=P(B|A1)P(A1)+P(B|A2)P(A2)+P(B|A3)P(A3)

•  Moregenerally,

P(B)=ΣiP(B|Ai)P(Ai)ifAi’saremutuallyexclusiveandBisasubsetoftheunionofAi’s

A1

A2 A3B

Bayes’Rule•  Priormodel:probabiliJesP(Ai)

Bayes’Rule•  Priormodel:probabiliJesP(Ai)•  Measurementmodel:P(B|Ai)

– condiJonalprobabilitytoobservedataBgiventhatthetruthisAi



•  WanttocomputeposteriorprobabiliBesP(Ai|B)– CondiJonalprobabilitythatthetruthisAigiventhatweobserveddataB

MulJplicaJonRule

€

P Aii=1

n

= P(A1)P(A2 | A1)P(A3 | A1∩ A2) ⋅… ⋅ P An Ai

i=1

n−1

,

provided all the conditioning events have nonzero probability

Example1.10•  Threecardsaredrawnfromadeckof52cards,without

replacement.Ateachstep,eachoneoftheremainingcardsisequallylikelytobepicked.What’stheprobabilitythatnoneofthethreecardsisaheart?

•  LetAi={i‐thcardisnotaheart},i=1,2,3

Example•  Threecardsaredrawnfromadeckof52cards,without



€

P(A1) =3952

€

P(A1c ) =

1352

€

A1

€

A1c




€

P(A1) =3952

€

P(A1c ) =

1352

€

A1

€

A1c

€

A1∩ A2

€

A1∩ A2c€

P(A2 | A1) =3851

€

1351




€

P(A1) =3952

€

P(A1c ) =

1352

€

A1

€

A1c

€

A1∩ A2

€

A1∩ A2c

€

A1∩ A2∩ A3

€

A1∩ A2∩ A3c

€

P(A2 | A1) =3851

€

1351

€

1350

€

P(A3 | A1∩ A2) =3750




€

P(A1) =3952

€

P(A1c ) =

1352

€

A1

€

A1c

€

A1∩ A2

€

A1∩ A2c

€

A1∩ A2∩ A3

€

A1∩ A2∩ A3c

€

P(A2 | A1) =3851

€

1351

€

1350

€

P(A3 | A1∩ A2) =3750

€

P(A1∩ A2∩ A3) = P(A1)P(A2 | A1)P(A3 | A1∩ A2) =3952

⋅3851⋅3750

≈ 0.41

Example:Prisoner’sDilemma(p.58)•  Threeprisoners:A,B,andC.•  Onewillbeexecutednextday,tworeleased.•  PrisonerAaskstheguardtotellhimthenameofoneoftheothertwowhowillbereleased.

•  GuardsaysthatBwillbereleased.(Assumethattheguardistellingthetruth.)

•  Aargues:beforemychancestobeexecutedwere1/3,nowtheyare1/2sinceIknowit’seithermeorC.What’swrongwithhisreasoning?

Prisoner’sDilemma:Afewremarks

•  Theprisonersonlyknowthatoneofthemwillbeexecuted,butdonotknowwhichone.Thus,fromtheirpointofview,areasonablemodelisadiscreteuniformprobabilitylawthatassignseachofthemprobability1/3tobeexecuted.

•  Theguardknowswhichoneofthemwillbeexecuted.•  SinceAalreadyknowsthateitherBorCwillbereleased,itdoesnotseemlikeknowingthatBwillbereleasedshouldinfluenceA’schancesinanyway.Infact,A’sprobabilitytobeexecutedissJll1/3(fromA’spointofview),asshowninthenextfewslides.

Prisoner’sDilemma:SoluJon•  Theguard’sresponseneedstobeincludedintheprobabilisJc

model.

•  LetEi={prisoneriwillbeexecuted}fori=A,B,C•  LetGj={guardnamesprisonerj}forj=B,C


model.


EA

EB

EC

1/3

1/3

1/3


model.


EA

EB

EC

1/3

1/3

1/3

1/2

1/2

EA∩GB

EA∩GC


model.


EA

EB

EC

1/3

1/3

1/3

1/2

1/2

0

1

EA∩GB

EA∩GC EB∩GB

EB∩GC


model.


EA

EB

EC

1/3

1/3

1/3

1/2

1/2

0

1

10

EA∩GB

EA∩GC EB∩GB

EB∩GC

EC∩GB

EC∩GC


model.


EA

EB

EC

1/3

1/3

1/3

1/2

1/2

0

1

10

EA∩GB

EA∩GC EB∩GB

EB∩GC

EC∩GB

EC∩GC

1/6

1/6

0

1/3

1/3

0


model.


EA

EB

EC

1/3

1/3

1/3

1/2

1/2

0

1

10

EA∩GB

EA∩GC EB∩GB

EB∩GC

EC∩GB

EC∩GC

1/6

1/6

0

1/3

1/3

0

P(EA|GB) = P(EA∩GB)/P(GB) = (1/6)/(1/6 + 0 + 1/3) = 1/3

TheMontyHallPuzzle(Ex.1.12)

•  Prizebehindoneofthreedoors.•  Contestantpicksadoor.•  Host(whoknowswheretheprizeis)opensoneoftheremainingtwodoorswhichdoesnothavetheprize.

•  Contestantisofferedanopportunitytostaywithhisdoor,ortoswitchtoanotherdoor.

•  Stayorswitch?

TheMontyHallPuzzle:SoluJon

•  Ifstay,P(win)=1/3


•  Ifstay,P(win)=1/3•  Ifswitch,theonlywaytoloseisifiniJallypointedtothedoorwithprize:P(lose)=1/3andsoP(win)=2/3


•  Ifstay,P(win)=1/3•  Ifswitch,theonlywaytoloseisifiniJallypointedtothedoorwithprize:P(lose)=1/3andsoP(win)=2/3

•  Conclusion:mustswitch!

•  Switchingisadvantageousbecausethehost’sacJontellsyousomething.IfyouiniJallypickedadoorwithnoprize,heisforcedtoopentheotherdoorwithnoprize.

TheMontyHallPuzzle:Discussion

•  Crucialpartsoftheproblemstatement:–  thehostknowswheretheprizeis– hemustopenthedoorwithnoprize

TheMontyHallPuzzle:Discussion

•  Crucialpartsoftheproblemstatement:–  thehostknowswheretheprizeis– hemustopenthedoorwithnoprize

•  Thus,ifyouhavechosenadoorwithnoprize,youareforcinghimtoopentheonlyotherdoorwithnoprizeandthusshowyouwheretheprizeis.

TheMontyHallPuzzle:AnotherSoluJon

•  UsethetotalprobabilitytheoremtoevaluatetheprobabiliJesofwinningunderthetwostrategies.



•  LetW=“win”andN=“originallypointtoadoorwithnoprize”.



•  LetW=“win”andN=“originallypointtoadoorwithnoprize”.•  P(N)=2/3;P(Nc)=1/3.




•  Ifyoudonotswitch,P(W|N)=0andP(W|Nc)=1.




•  Ifyoudonotswitch,P(W|N)=0andP(W|Nc)=1.•  So,ifyoudonotswitch,P(W)=P(W|N)P(N)+P(W|Nc)P(Nc)=1/3.





•  Ifyouswitch,P(W|N)=1becauseifyouoriginallychooseadoorwithnoprizethehostisforcedtoopentheonlyotherdoorwithnoprize.






•  Ifyouswitch,P(W|Nc)=0becauseifyouoriginallychoosethedoorwiththeprize,youwillswitchoutofitandlose.






•  Ifyouswitch,P(W|Nc)=0becauseifyouoriginallychoosethedoorwiththeprize,youwillswitchoutofitandlose.

•  So,ifyouswitch,P(W)=P(W|N)P(N)+P(W|Nc)P(Nc)=2/3.

Two‐EnvelopesPuzzle(p.58)

•  Youarehandedtwoenvelopes,eachcontaininganintegernumberofdollars,unknowntoyou.

•  Thetwoamountsaredifferent.

•  Youselectatrandomoneenvelopeandlookinside.

•  YoucaneithersJckwiththisenvelopeortaketheotherenvelope.YourobjecJveistogetthelargeramount.

•  Doesitmakerwhatyoudo?

Two‐EnvelopesPuzzle:SoluJon•  MakeindependentflipsofafaircoinunJlheadscomeupforthe

firstJme.LetX=1/2+numberoftossestogetfirstH.

•  Strategy:–  Iftheamountinyourenvelope>X,stay

–  Iftheamountinyourenvelope<X,switch





•  DenotethetwoamountsdandD,d<D





•  DenotethetwoamountsdandD,d<D•  Case1:X<d‐‐‐alwaysstay‐‐‐winwithprobability1/2





•  DenotethetwoamountsdandD,d<D•  Case1:X<d‐‐‐alwaysstay‐‐‐winwithprobability1/2•  Case2:X>D‐‐‐alwaysswitch‐‐‐winwithprobability1/2





•  DenotethetwoamountsdandD,d<D•  Case1:X<d‐‐‐alwaysstay‐‐‐winwithprobability1/2•  Case2:X>D‐‐‐alwaysswitch‐‐‐winwithprobability1/2•  Case3:d<X<D‐‐‐stayifpickD,switchifpickd‐‐‐win!





•  DenotethetwoamountsdandD,d<D•  Case1:X<d‐‐‐alwaysstay‐‐‐winwithprobability1/2•  Case2:X>D‐‐‐alwaysswitch‐‐‐winwithprobability1/2•  Case3:d<X<D‐‐‐stayifpickD,switchifpickd‐‐‐win!•  P(win)=P(win|X<d)P(X<d)+P(win|X>D)P(X>D)+P(win|d<X<D)P(d<X<D)(bytotalprobabilitytheorem)





•  DenotethetwoamountsdandD,d<D•  Case1:X<d‐‐‐alwaysstay‐‐‐winwithprobability1/2•  Case2:X>D‐‐‐alwaysswitch‐‐‐winwithprobability1/2•  Case3:d<X<D‐‐‐stayifpickD,switchifpickd‐‐‐win!•  P(win)=P(win|X<d)P(X<d)+P(win|X>D)P(X>D)+P(win|d<X<D)P(d<X<D)=1/2P(X<d)+1/2P(X>D)+P(d<X<D)






=1/2(P(X<d)+P(X>D)+P(d<X<D))+1/2P(d<X<D) =1






=1/2(P(X<d)+P(X>D)+P(d<X<D))+1/2P(d<X<D)=1/2(1+P(d<X<D))>1/2

Two‐EnvelopesPuzzle:Comment

•  NotethatXcanbeanyrandomvariablehavingnon‐zerovaluesat3/2,5/2,7/2(or,infact,atanypoint(s)on]1,2[,atanypoint(s)on]2,3[,etc.)

Sample space and probability - Purdue Engineering · 1. Sample Space and Probability Part I ECE 302...

Documents

Transcript of Sample space and probability - Purdue Engineering · 1. Sample Space and Probability Part I ECE 302...