1. Consider a Markov process with states 0, 1 and 2 and with the following transition ratematrix Q:

Q =

!

"#!! ! 0µ !!! µ !µ 0 !µ

$

%&

where ! > 0 and µ > 0.

a. Derive the parameters vi and Pij for this Markov process.

b. Determine the expected time to go from state 1 to state 0.

2. Exercise 6.1.3. Exercise 6.3.4. Exercise 6.5.5. Exercise 6.6 (a), (b).

1

1.

a. We havev0 = !, v1 = ! + µ, v2 = µ,

andP01 = P20 = 1, P10 = 1! P12 =

µ

! + µ.

b. Let Ti(i = 1, 2) denote the time to go from state i to 0. Then E[T2] = 1/µ and

E[T1] =1

! + µ+

!

! + µ· E[T2] =

1

µ.

2. Let us assume that the state is (n, m). Male i mates at a rate ! with female j, andtherefore it mates at a rate !m. Since there are n males, mating occurs at a rate !nm.Therefore

vn,m = !nm.

Since any mating is equally likely to result in a female as in a male, we have

P(n,m);(n+1,m) = P(n,m);(n+1,m) =1

2.

3. This is not a birth and death process since we need more information than just thenumber working. We must also know which machine is working. We can analyze it byletting the states be

b : both machines are working1 : 1 is working, 2 is down2 : 2 is working, 1 is downd1 : both are down, 1 is being repairedd2 : both are down, 2 is being repaired.

Thenvb = µ1 + µ2, v1 = µ1 + µ, v2 = µ2 + µ, vd1 = vd2 = µ,

and

Pb,1 = 1! Pb,2 =µ2

µ1 + µ2,

P1,b = 1! P1,d2 =µ

µ1 + µ,

P2,b = 1! P2,d1 =µ

µ + µ2,

Pd1,1 = Pd2,2 = 1.

1

4.

a. Yes.

b. It is a pure birth process.

c. If there are i infected individuals then since a contact will involve an infected andan uninfected individual with probability i(n! i)/

!n2

", it follows that the birth rates

are !i = !i(n! i)/!

n2

", i = 1, . . . , n. Hence,

E[time all infected] =n(n! 1)

2!

n#

i=1

1

i(n! i).

5. Starting with E[T0] = 1!0

= 1! , employ the identity

E[Ti] =1

!i+

µi

!iE[Ti!1]

to succesively compute E[Ti] for i = 1, 2, 3, 4.

a. E[T0] + · · · + E[T3].

b. E[T2] + E[T3] + E[T4].

2

1. Consider the following queueing model: customers arrive at a service station accordingto a Poisson process with rate !. There are c servers; the service times are exponential withrate µ. If an arriving customer finds c servers busy, then he leaves the system immediately.

a. Model this system as a birth and death process.

b. Suppose now that there are infinitely many servers (c =!). Again model this systemas a birth and death process.

2. In Example 6.11 it is shown, using the backward equations, that

P !00(t) = µ" (µ + !)P00(t).

a. Derive this result using the forward equations.

b. Derive a di!erential equation for P11(t) in two ways: using the forward and backwardequations.

c. Suppose the machine is working at time 0. What is the probability that the machineis also working at time t?

3. Exercise 6.8.

4. Exercise 6.10 (but you do not have to verify that the transition probabilities satisfy theforward and backward equations).

1

1.

a. The state space is {0, 1, . . . , c! 1, c}, and the birth rates are

!n = !, n = 0, 1, . . . , c! 1,

and the death ratesµn = nµ, n = 1, 2, . . . , c.

Hence the transition rate matrix Q is given by

Q =

!

"""""""""""#

!! ! 0 0 0 · · · 0µ !(! + µ) ! 0 0 · · · 00 2µ !(! + 2µ) ! 0 · · · 0...

. . . . . . . . . . . . . . ....

0 · · · 0 (c! 2)µ !(! + (c! 2)µ) ! 00 · · · 0 0 (c! 1)µ !(! + (c! 1)µ) !0 · · · 0 0 0 cµ !cµ

$

%%%%%%%%%%%&

b. Now the state space does not stop at c, i.e., it is {0, 1, . . .} and the birth rates are

!n = !, n = 0, 1, . . . ,

and the death ratesµn = nµ, n = 1, 2, . . . .

2.

a. The forward equations are

P!

00(t) = µP01(t)! !P00(t)

P!

01(t) = !P00(t)! µP01(t).

Using P01(t) = 1! P00(t), we obtain the following di!erential equation,

P!

00(t) = µ! (µ + !)P00(t).

b. The backward equations are

P!

11(t) = µP01(t)! µP11(t)

P!

01(t) = !P11(t)! !P01(t).

Multiplying these equations by ! and µ, respectively, and then adding them, weobtain

!P!

11(t) + µP!

01(t) = 0.

1

Hence, for some c,!P11(t) + µP01(t) = c.

Substituting t = 0 yields c = !. Using the above equation, we get

P!

11(t) = !! (! + µ)P11(t).

The forward equations are

P!

11(t) = !P10(t)! µP11(t)

P!

10(t) = µP11(t)! !P10(t).

By substituting P10(t) = 1!P11(t), we obtain the same di!erential equation as before.

c. Solving the di!erential equation for P00(t) with initial condition P00(0) = 1 yields

P00(t) =µ

! + µ+

!

! + µe!(!+µ)t, t " 0.

3. The number of failed machines is a birth and death process with

!0 = 2!, !1 = !, !n = 0, n > 1, µ1 = µ2 = µ, µn = 0, n #= 1, 2.

Now substitute into the backward equations.

4. Let

Ij(t) =

!0, if machine j is working at time t,1, otherwise.

Also, let the state be (I1(t), I2(t)). This is clearly a continuous-time Markov chain with

v(0,0) = !1 + !2, !(0,0);(0,1) = !2, !(0,0);(1,0) = !1,

v(0,1) = !1 + µ2, !(0,1);(0,0) = µ2, !(0,1);(1,1) = !1,

v(1,0) = µ1 + !2, !(1,0);(0,0) = µ1, !(1,0);(1,1) = !2,

v(1,1) = µ1 + µ2, !(1,1);(0,1) = µ1, !(1,1);(1,0) = µ2.

By the independence assumption we have

P(i,j),(k,l)(t) = Pi,k(t)Qj,l(t), (1)

where Pi,k(t) is the probability that the first machine is in state k at time t given that itwas at state i at time 0; Qj,l(t) is defined similarly for the second machine. By example4.11 we have

P0,0(t) =µ1

!1 + µ1+

!1

!1 + µ1e!(!1+µ1)t,

P1,0(t) =µ1

!1 + µ1! µ1

!1 + µ1e!(!1+µ1)t,

2

and by the same argument,

P1,1(t) =!1

!1 + µ1+

µ1

!1 + µ1e!(!1+µ1)t,

P0,1(t) =!1

!1 + µ1! !1

!1 + µ1e!(!1+µ1)t.

Of course, similar expressions for the second machine are obtained by replacing (!1, µ1) by(!2, µ2). We then get P(i,j),(k,l)(t) by formula (??). For instance,

P(0,0),(0,0)(t) = P0,0(t)Q0,0(t) =µ1 + !1e!(!1+µ1)t

!1 + µ1· µ2 + !2e!(!2+µ2)t

!2 + µ2.

3

1.

a. If the state is the number of individuals at time t, we get a birth and death processswith

!n = n! + ", n < N,

!n = n! n ! N,

µn = nµ.

b. Let Pi be the long-run probability that the system is in state i. Since this is also theproportion of time the system is in state i, we are looking for

!!

i=3

Pi.

We havePkµk = Pk"1!k"1, k = 1, 2, . . . .

This yields

P1 ="

µP0 =

1

2P0,

P2 =" + !

2µP1 =

1

2P1 =

1

4P0,

P3 =" + 2!

3µP2 =

1

2P2 =

1

8P0,

and for k ! 3,

Pk =(k " 1)!

kµPk"1 =

k " 1

k

1

2Pk"1 = · · · =

3

k

"1

2

#k"3

P3.

Hence!!

k=3

Pk = P3

!!

k=3

3

k

"1

2

#k"3

= 24P3

!!

k=3

1

k

"1

2

#k

.

Since !!

k=1

1

kxk = " log(1" x),

we get!!

k=3

Pk = P3(24 log 2" 15).

Because all probabilities add up to 1, we have

P0

"1 +

1

2+

1

4+

1

8(24 log 2" 15)

#= 1.

1

So

P!10 = 3 log 2! 1

8,

and thus finally,"!

k=3

Pk =24 log 2! 15

24 log 2! 1" 0.105.

2. With the number of customers in the shop as the state, we get a birth and death processwith

!0 = !1 = 3, µ1 = µ2 = 4.

Therefore

P1 =3

4P0, P2 =

3

4P1 =

"3

4

#2

P0.

And since P0 + P1 + P2 = 1, we get

P0 =16

37.

a. The average number of customers in the shop is

P1 + 2P2 =30

37.

b. The proportion of the customers that enter the shop is

!(1! P2)

!= 1! P2 =

28

37.

c. Now µ = 8, so

P0 =64

97.

So the proportion of the customers that enter the shop is

1! P2 =88

97.

The rate of added customers is therefore

!"

88

97

#! !

"28

37

#" 0.45.

The business he does would improve by 0.45 customers per hour.

3. With the number of customers in the system as the state, we get a birth and deathprocess with

!0 = !1 = !2 = 3, !i = 0, i # 4, µ1 = 2, µ2 = µ3 = 4.

2

Therefore the balance equations reduce to

P1 =3

2P0, P2 =

3

4P1 =

9

8P0, P3 =

3

4P2 =

27

32P0.

And therefore,

P0 =!1 +

3

2+

9

8+

27

32

"!1

=32

143.

a. The fraction of potential custoemrs that enter the system is

!(1! P3)

!= 1! P3 =

116

143.

b. With a server working twice as fast we would get

P1 =3

4P0, P2 =

3

4P1 =

!3

4

"2

P0, P3 =!

3

4

"3

P0,

and

P0 =

#

1 +3

4+

!3

4

"2

+!

3

4

"3$!1

=64

175.

So that now

1! P3 =148

175.

4. Say the state is 0 if the machine is up, say it is in state i when it is down due to a typei failure, i = 1, 2. The balance equations for the limiting probabilities are as follows.

!P0 = µ1P1 + µ2P2,

µ1P1 = !pP0,

µ2P2 = !(1! p)P0,

P0 + P1 + P2 = 1.

These equations are easily solved to give the results

P0 = (1 + !p/µ1 + !(1! p)/µ2)!1 , P1 = !pP0/µ1, P2 = !(1! p)P0/µ2.

3

1. There are k + 1 states: state 0 means the machine is working, state i means that it isin repair phase i, i = 1, . . . , k. The balance equations for the limiting probabilities are

!P0 = µkPk,

µ1P1 = !P0,

µiPi = µi!1Pi!1, i = 2, . . . , k,

and the normalization equation is

P0 + · · · + Pk = 1.

To solve, note thatµiPi = µi!1Pi!1 = µi!2Pi!2 = · · · = !P0.

Hence,Pi = (!/µi)P0,

and, upon summing,

1 = P0

!

1 +k"

i=1

!

µi

#

.

Therefore,

P0 =

!

1 +k"

i=1

!

µi

#!1

, Pi = (!/µi)P0, i = 1, . . . , k.

a. Pi.

b. P0.

2. The number in the system is a birth and death process with parameters

!n = !/(n + 1), n ! 0, µn = µ, n ! 1.

From the equation above (6.20),

1/P0 = 1 +""

n=1

(!/µ)n/n! = e!/µ

and from (6.20),

Pn = P0(!/µ)n/n! = e!!/µ(!/µ)n/n!, n ! 0.

3. Let the state denote the number of machines that are down. This yields a birth anddeath process with

!0 =3

10, !1 =

2

10, !2 =

1

10, !i = 0, i ! 3,

1

and

µ1 =1

8, µ2 =

2

8, µ3 =

2

8.

The balance equations reduce to

P1 =3/10

1/8P0 =

12

5P0,

P2 =2/10

2/8P1 =

4

5P1 =

48

25P0,

P3 =1/10

2/8P2 =

4

10P2 =

192

250P0.

Hence, using P0 + P1 + P2 + P3 = 1, yields

P0 =!1 +

12

5+

48

25+

192

250

"!1

=250

1522.

a. Average number not in use is

P1 + 2P2 + 3P3 =2136

1522=

1068

761.

b. Proportion of time both repairmen are busy is

P2 + P3 =336

761.

4.

a. !0 = 13 .

b. !0 = 1.

c. !0 = (!

3" 1)/!

2.

2