Page 180 - A First Course In Stochastic Models

TRANSIENT DISTRIBUTION OF CUMULATIVE REWARDS 173

for this model. Throughout this section it is assumed that the continuous-time
Markov chain {X(t)} has a finite state space I.

4.6.1 Transient Distribution of Cumulative Sojourn Times
Consider a continuous-time Markov chain {X(t)} whose finite state space I is
divided into two disjoint subsets I 0 and I f with

I 0 = the set of operational states and I f = the set of failed states.
Define for any t > 0 the indicator random variable I (t) by

1 if X(t) ∈ I 0 ,
I (t) =
0 otherwise.
Then the random variable

t
O(t) = I (u) du
0
represents the cumulative operational time of the system during (0, t). The transient
probability distribution of O(t) can be calculated by a nice probabilistic algorithm
that is based on the uniformization method.
To find P {O(t) ≤ x}, we first uniformize the continuous-time Markov chain
{X(t)} according to (4.5.3). Denoting by O(t) the cumulative operational time in
the uniformized process {X(t)}, we have

P {O(t) ≤ x} = P {O(t) ≤ x},

since the uniformized process is probabilistically equivalent with the original pro-
cess. By conditioning on the number of state transitions of the uniformized process
during (0, t), we have
∞

P {O(t) ≤ x} = P {O(t) ≤ x | the uniformized process makes n state
n=0
(νt) n
−νt
transitions in (0, t)} e .
n!
The next key step in the analysis is the relation between the Poisson process
and the uniform distribution. In the uniformized process the epochs of the state
transitions are determined by a Poisson process that is independent of the discrete-
time Markov chain governing the state transitions. Under the condition that the
uniformized process has made n state transitions during (0, t), the joint distribution
of the epochs of these state transitions is the same as the joint distribution of the
order statistics U (1) , . . . , U (n) of n independent random variables U 1 , . . . , U n that
are uniformly distributed on (0, t); see Theorem 1.1.5. Note that U (k) is the smallest