Page 163 - A First Course In Stochastic Models

156 CONTINUOUS-TIME MARKOV CHAINS

cycle be defined as the time elapsed between two consecutive visits of the process
to state r. Using Wald’s equation, it is readily seen that

1

E(length of one cycle) = E(number of visits to state j in one cycle) × .
ν j
j∈I
Thus, by Lemma 3.5.10,
1 π j
E(length of one cycle) = .
π r ν j
j∈I
Since E(length of one cycle) is finite by Assumption 4.2.1, the result now follows.
This completes the proof.
Next it is not difficult to prove Theorem 4.2.2

Proof of Theorem 4.2.2 We first prove the result for initial state X(0) = r, where
r is the regeneration state from Assumptions 4.2.1 and 4.2.2. The process {X(t)}
regenerates itself each time the process makes a transition into state r. Let a cycle
be defined as the time elapsed between two consecutive visits of the process to
state r. In the proof of the above theorem we have already shown
1 π k
E(length of one cycle) = .
π r ν k
k∈I
The expected length of a cycle is finite. Next apply the renewal-reward theorem
from Chapter 2. This gives
R(t) E(reward earned during one cycle)
lim = (4.3.4)
t→∞ t E(length of one cycle)
with probability 1. Using Wald’s equation, E(reward earned during one cycle) is
 
r(j)
 .
E(number of visits to state j during one cycle) ×  + p jk F jk
ν j
j∈I k =j
Hence, by Lemma 3.5.10 and relation (4.3.1),
 
π j r(j)
E(reward earned during one cycle) =  + p jk F jk 
π r ν j
j∈I k =j
 
1 π j
=  r(j) + q jk F jk  .
π r ν j
j∈I k =j
Taking the ratio of the expressions for the expected reward earned during one cycle
and the expected length of one cycle and using relation (4.3.2), we get the result