A First Course in Stochastic Processes Chapter Two: Markov Chains.

34
A First Course in Stochastic Processes Chapter Two: Markov Chains

Transcript of A First Course in Stochastic Processes Chapter Two: Markov Chains.

Page 1: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A First Course in Stochastic Processes

Chapter Two: Markov Chains

Page 2: A First Course in Stochastic Processes Chapter Two: Markov Chains.

xNx

N

p

N

p

x

NpXxX

1|Pr 12

N

XNX 1

2 ,Bin~

X2=X1=1 X2=2

N

XNX 2

3 ,Bin~

X3=1 X4=3

Page 3: A First Course in Stochastic Processes Chapter Two: Markov Chains.

X1

N

XN 1,Bin X2

N

XN 2,Bin X3

N

XN 3,Bin

X4

N

XN 4,Bin X5

N

XN 5,Bin etc

xNx

tt N

p

N

p

x

NpXxX

1|Pr 1

Page 4: A First Course in Stochastic Processes Chapter Two: Markov Chains.

P =

jitt iXjX ,1 |Pr P

tX1tX 0 1 2 3 4 5 6 … 20

0 1 0 0 0 0 0 0 … 0

1 0.358 0.377 0.189 0.0596 0.0133 0.00224 0.000295 … 0.000

2 0.122 0.27 0.285 0.19 0.0898 0.0319 0.00887 … 0.000

3 0.0388 0.137 0.229 0.243 0.182 0.103 0.0454 … 0.000

4 0.0115 0.0576 0.137 0.205 0.218 0.175 0.109 … 0.000

5 0.00317 0.0211 0.0669 0.134 0.19 0.202 0.169 … 0.000

6 0.000798 0.00684 0.0278 0.0716 0.13 0.179 0.192 … 0.000

… … … … … … … … …

20 0 0 0 0 0 0 0 … 1

Page 5: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Example Two: Nucleotide evolution

A

G

C

T

Page 6: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Types of point mutation

A G

T C

Purine

Pyramidine Transitions

Transitions

Transversions

α

β β β β

α

Page 7: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

G

Kimura’s 2 parameter model (K2P)

P =

jitt iXjX ,1 |Pr P

tX

1tX

Page 8: A First Course in Stochastic Processes Chapter Two: Markov Chains.

GG TCAC A

G

C

T

A

G

C

T

CTATGA AG TTCGC

Page 9: A First Course in Stochastic Processes Chapter Two: Markov Chains.

The Markov Property

GG TCAC A

G

C

T

A

G

C

T

CTATGA

Page 10: A First Course in Stochastic Processes Chapter Two: Markov Chains.

The Markov Property

etc

XXXXX

XXXXX

XX

ttttt

ttttt

tt

,...,,,|Pr

,...,,,|Pr

|Pr

3211

3211

1

ATCAG

CTGAG

AG

tt

tttt

XxX

XXXXxX

|Pr

,...,,,|Pr

1

0211

Page 11: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Markov Chain properties

accessible aperiodic

communicate recurrent

irreducible transient

Page 12: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

G

Accessible

P =

tX

1tX

0

0but 0 2,

1, ACAC PP

Page 13: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

G

Accessible

P =

tX

1tX

0 0

0 0

A (and G) are no longer accessible from C (or T).

0, nACP

Page 14: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

G

Accessible

P =

tX

1tX

0 0

0 0

But C (and T) are still accessible from A (or G).

0, nCAP

Page 15: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

G

Communicate

P =

tX

1tX

Reciprocal accessibility

CA

Page 16: A First Course in Stochastic Processes Chapter Two: Markov Chains.

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

G

Irreducible

P =

tX

1tX

All elements communicate

0, njiP jiji , allfor

Page 17: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Non-irreducible

A . α β β

α . β β

β β . α

β β α .

A G C T

T

C

GP =

tX

1tX

0 0

0 0

0 0

0 0 =P1

P2

0

0

. α

α .

P1 = A

G

A G

C

T

C T

P2 = . α

α .

Page 18: A First Course in Stochastic Processes Chapter Two: Markov Chains.

• Reflexivity

• Symmetry

• Transitivity

Repercussions of communication

ii ,definitionBy

ijji then If

kikjji then and If

Page 19: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Periodicity

0 1 2 3 4 5 6

0 0 1 0 0 0 0 0

1 0 0 1 0 0 0 0

2 0 0 0 1 0 0 0

3 0 0 0 0 1 0 0

4 0 0 0 0 0 1 0

5 0 0 0 0 0 0 1

6 1 0 0 0 0 0 0

P =

tX 1tX

Page 20: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Periodicity

• The period d(i) of an element i is defined as the greatest common divisor of the numbers of the generations in which the element is visited.

• Most Markov Chains that we deal with do not exhibit periodicity.

• A Markov Chain is aperiodicif d(i) = 1 for all i.

Page 21: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Recurrence

1,

n

niiP

1,

n

niiP

recurrent11

,

n

niif

transient11

,

n

niif

Page 22: A First Course in Stochastic Processes Chapter Two: Markov Chains.

More on Recurrence

•and i is recurrent then j is recurrent

• In a one-dimensional symmetric random walk the origin is recurrent

• In a two-dimensional symmetric random walk the origin is recurrent

• In a three-dimensional symmetric random walk the origin is transient

ji If

Page 23: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Markov Chain properties

accessible aperiodic

communicate recurrent

irreducible transient

Page 24: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Markov Chains

Examples

Page 25: A First Course in Stochastic Processes Chapter Two: Markov Chains.

X1=1

X1 X2 X3

X4 X5

N

XN 1,Bin

N

XN 2,Bin

N

XN 3,Bin

N

XN 4,Bin

N

XN 5,Bin etc

pN ,Bin pN ,Bin pN ,Bin

pN ,Bin pN ,Bin

Page 26: A First Course in Stochastic Processes Chapter Two: Markov Chains.

P =

0 1 2 3 4 5 6 … 20

0 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

1 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

2 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

3 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

4 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

5 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

6 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

… … … … … … … … … …

20 p 0 p 1 p 2 p 3 p 4 p 5 p 6 … p 20

jittt jXiXjX ,11 Pr|Pr P

Page 27: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Diffusion across a permeable membrane (1D random walk)

otherwise0

1 if2

1 if2

,ij

a

ia

ija

ia

jiP

aX t ,...,2,1,0

Page 28: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Brownian motion (2D random walk)

otherwise0

1 if2

1

1 if2

1

,,ij

ij

jiji YX

max

max

,...,2,1,0

,...,2,1,0

YY

XX

t

t

jijiji ,,, YXP

Page 29: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Wright-Fisher allele frequency model

X1=1

Page 30: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Haldane (1927) branching process model of fixation probability

,...2,1,0tX

tX 2 3 4 4 4 4 2

Page 31: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Haldane (1927) branching process model of fixation probability

mXT

m

XXXX

YXYX

x

xX

sGsG

XXXT

sGsGsG

sGsGsG

sssxXsG

)()(

...

)()()(

)()()(

...)2Pr()1Pr()0Pr()Pr()(

21

0

2

21

Page 32: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Haldane (1927) branching process model of fixation probability

xtt

x

tttt

tttt

XXX

sXxX

sXXsXX

sXXXX

sGsG t

tt

)|Pr(

...)|3Pr()|2Pr(

)|1Pr()|0Pr(

)()(

10

31

21

11

1|1

Pi,j = coefficient of sj in the above generating function

Page 33: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Haldane (1927) branching process model of fixation probability

Probability of fixation = 2s

Page 34: A First Course in Stochastic Processes Chapter Two: Markov Chains.

Markov Chain properties

accessible aperiodic

communicate recurrent

irreducible transient