Probability and Statistics with Reliability, Queuing and Computer Science Applications: Chapter 6 on Stochastic ProcessesKishor S. TrivediVisiting ProfessorDept. of Computer Science and EngineeringIndian Institute of Technology, Kanpur

What is a Stochastic Process?Stochastic Process: is a family of random variables {X(t) | t T} (T is an index set; it may be discrete or continuous)Values assumed by X(t) are called states. State space (I): set of all possible statesSometimes called a random process or a chance process

If x and y are mutually independent, then, p(y|x) = p(y).

Stochastic Process CharacterizationAt a fixed time t=t1, we have a random variable X(t1). Similarly, we have X(t2), .., X(tk). X(t1) can be characterized by its distribution function,

We can also consider the joint distribution function,

Discrete and continuous cases: States X(t) (i.e. time t) may be discrete/continuousState space I may be discrete/continuous

Classification of Stochastic ProcessesFour classes of stochastic processes:

discrete-state process chain discrete-time process stochastic sequence {Xn | n T} (e.g., probing a system every 10 ms.)

Example: a Queuing System

Interarrival times Y1, Y2, (common dist. Fn. FY) Service times: S1, S2, (iid with a common cdf FS) Notation for a queuing system: FY /FS/m Some interarrival/service time distributions types are: M: Memoryless (i.e., EXP) D: Deterministic Ek: k-stage Erlang etc. Hk: k-stage Hyper exponential distribution G: General distribution GI: General independent inter arrival times M/M/1 Memoryless interarrival/service times with a single server

Discrete/Continuous Stochastic ProcessesNk: Number of jobs waiting in the system at the time of kth jobs departure Stochastic process {Nk| k=1,2,}: Discrete time, discrete state

NkkDiscreteDiscrete

Continuous Time, Discrete SpaceX(t): Number of jobs in the system at time t. {X(t) | t T} forms a continuous-time, discrete-state stochastic process, with,

X(t)DiscreteContinuous

Discrete Time, Continuous SpaceWk: waiting time for the kth job. Then {Wk | k T} forms a Discrete-time, Continuous-state stochastic process, where,

WkkDiscreteContinuous

Continuous Time, Continuous SpaceY(t): total service time for all jobs in the system at time t. Y(t) forms a continuous-time, continuous-state stochastic process, Where,

Y(t)t

Further Classification Similarly, we can define nth order distribution:

Formidable task to provide nth order distribution for all n.

(1st order distribution)(2nd order distribution)

Further Classification (contd.)Can the nth order distribution be simplified?Yes. Under some simplifying assumptions:Independence

As example, we have the Renewal ProcessDiscrete time independent process {Xn | n=1,2,} (X1, X2, .. are iid, non-negative rvs), e.g., repair/replacement after a failure. Markov process introduces a limited form of dependence Markov ProcessStochastic proc. {X(t) | t T} is Markov if for any t0 < t1< < tn< t, the conditional distribution satisfies the Markov property:

Stationary: E[x(t)] = E[x] ensemble average. When the pdf or the CDF exhibits stationarity property, then, the process is said to strictly stationary. If only the first moment satisfies this property, then, the process is said to stationary in the mean etc.

Markov ProcessWe will only deal with discrete state Markov processes i.e., Markov chainsIn some situations, a Markov chain may also exhibit time-homogeneity

Future of process (probabilistically) determined by its current state, independent of how it reached this particular state; but in a non homogeneous case, current time can also determine the future.For a homogeneous Markov chain current time is also not needed to determine the future. Let Y: time spent in a given state in a hom. CTMC

Homogeneous CTMC-Sojourn timeSince Y, the sojourn time, has the memoryless prop.

This result says that for a homogeneous continuous time Markov chain, sojourn time in a state follows EXP( ) distribution (not true for non-hom CTMC)Hom. DTMC sojourn time dist. Is geometric.Semi-Markov process is one in which the sojourn time in a state is generally distributed.

Bernoulli ProcessA sequence of iid Bernoulli rvs, {Yi | i=1,2,3,..}, Yi =1 or 0{Yi} forms a Bernoulli Process, an example of a renewal process. Define another stochastic process , {Sn | n=1,2,3,..}, where Sn = Y1 + Y2 ++ Yn (i.e. Sn :sequence of partial sums)Sn = Sn-1+ Yn (recursive form)P[Sn = k | Sn-1= k] = P[Yn = 0] = (1-p) and,P[Sn = k | Sn-1= k-1] = P[Yn = 1] = p {Sn |n=1,2,3,..}, forms a Binomial process, an example of a homogeneous DTMC

{Yi} forms a discrete-time process.

Renewal Counting ProcessRenewal counting process: # of renewals (repairs, replacements, arrivals) by time t: a continuous time process: If time interval between two renewals follows EXP distribution, then Poisson Process

Note:For a fixed t, N(t) is a random variable (in this case a discrete random variable known as the Poisson random variable) The family {N(t), t 0} is a stochastic process, in this case, the homogeneous Poisson process{N(t), t 0} is a homogeneous CTMC as well

Poisson ProcessA continuous time, discrete state process.N(t): no. of events occurring in time (0, t]. Events may be,

# of packets arriving at a router port# of incoming telephone calls at a switch# of jobs arriving at file/compute server Number of component failures Events occurs successively and that intervals between these successive events are iid rvs, each following EXP( )

: arrival rate (1/ : average time between arrivals): failure rate (1/ : average time between failures)

Poisson Process (contd.)N(t) forms a Poisson process provided:

N(0) = 0Events within non-overlapping intervals are independentIn a very small interval h, only one event may occur (prob. p(h))

Letting, pn(t) = P[N(t)=n],

For a Poisson process, interarrival times follow EXP( ) (memoryless) distribution. E[N(t)] = Var[N(t)] = t ; What about E[N(t)/t], as t infinity?

Merged Multiple Poisson Process StreamsConsider the system,

Proof: Using z-transform. Letting, = t,

+

Decomposing a Poisson StreamDecompose a Poisson process using a prob. switch

N arrivals decomposed into {N1, N2, .., Nk}; N= N1+N2, ..,+NkCond. pmf

Since, The uncond. pmf

Generalizing the Poisson Process

Non-Homogeneous Poisson

Process (NHPP)Poisson Process

Non-Homogeneous Poisson Process (NHPP)If the expected number of events per unit time, l, changes with age (time), we have a non-homogeneous Poisson model. We assume that:1.If 0 t, the pmf of N(t) is given by:

where m(t) 0 is the expected number of events in the time period [0, t]2.Counts of events in non-overlapping time periods are mutually independent.

m(t) : the mean value function. l(x) :the time-dependent rate of occurrence of events or time-dependent failure rate

NHPP(cont.)

Generalizing Poisson Process

Non-Homogeneous Poisson

Process (NHPP)Poisson ProcessRenewal CountingProcess

Renewal Counting ProcessPoisson process EXP( ) distributed interarrival times.What if the EXP( ) assumption is removed renewal proc.Renewal proc. : {Xi | i=1,2,} (Xis are iid non-EXP rvs) Xi : time gap between the occurrence of (i-1) st and ith event

Sk = X1 + X2 + .. + Xk time to occurrence of the kth event.

N(t)- Renewal counting process is a discrete-state, continuous-time stochastic process. N(t) denotes no. of renewals in the interval (0, t].

Renewal Counting Processes (contd.)For N(t), what is P(N(t) = n)?

F(n+1) (t): prob(time taken for n-renewals + time for one more renewal)= tn + t

Renewal Counting Process ExpectationLet, m(t) = E[N(t)]. Then, m(t) = mean no. of arrivals in time (0,t]. m(t) is called the renewal function.

Renewal Density FunctionRenewal density function:

For example, if the renewal interval X is EXP(), thend(t) = , t >= 0 and m(t) = t , t >= 0.P[N(t)=n] = Fn(t) will turn out to be n-stage Erlang

e t ( t)n/n! i.e Poisson pmf

Alternating Renewal ProcessWhere:Failure times T1, T2, are mutually independent with a common distribution function WRestoration times D1, D2, are mutually independent with a common distribution function GThe sequences {Tn} and {Dn} are independent

10I(t)OperatingRestorationTime

Availability Analysis Availability: is defined is the ability of a system to provide the desired service.If no repair/replacement,Availability(t)=Reliability(t)If repairs are possible, then above is pessimistic.

MTBF = E[Di+Ti+1] = E[Ti+Di]=E[Xi]=MTTF+MTTR

Availability Analysis (contd.)

Two mutually exclusive situations:

System does not fail before time t A(t) = R(t)System fails, but the repair is completed before time tTherefore, A(t) = sum of these two probabilities

Availability Expression dA(x) : Incremental availability

dA(x) = Prob(that after renewal, life time is > (t-x) & that the renewal occurs in the interval (x,x+dx])

Availability Expression (contd.)A(t) can also be expressed in the Laplace domain.

Since, R(t) = 1-W(t) or LR(s) = 1/s LW(s) = 1/s Lw(s)/s

What happens when t becomes very large?

However,

Availability, MTTF and MTTRSteady state availability A is:

Taking the expression of sLA(s) and taking the limit via LHospital rule and using the moment generating property of the LT, we get the required result for the steady-state

A=MTTF/(MTTF+MTTR)

Availability ExampleAssuming EXP( ) density fn for g(t) and w(t)

Generalizing Poisson Process

Non-Homogeneous Poisson

Process (NHPP)Poisson ProcessRenewal CountingProcessHomogeneousContinuous TimeMarkov ChainHomogeneousDiscrete TimeMarkov ChainSemi-Markov ProcessMarkov RegenerativeProcessNon-HomogeneousContinuous TimeMarkov Chain

Compound Poisson ProcessBernoulli Process

If x and y are mutually independent, then, p(y|x) = p(y).Stationary: E[x(t)] = E[x] ensemble average. When the pdf or the CDF exhibits stationarity property, then, the process is said to strictly stationary. If only the first moment satisfies this property, then, the process is said to stationary in the mean etc. {Yi} forms a discrete-time process.F(n+1) (t): prob(time taken for n-renewals + time for one more renewal)= tn + t