Page 137 - A First Course In Stochastic Models

THEORETICAL CONSIDERATIONS 129

Taking x j = π j for all j and using (3.5.10), it follows that
π j = 1. This ends
j∈I
the proof.
Though we are mainly concerned with the Cesaro limit of the n-step transition
probabilities, we also state a result about the ordinary limit. If the regeneration
(n)
state r from Assumption 3.3.1 is aperiodic, then by Theorem 2.2.4, lim n→∞ p
rj
exists for all j. From this result it is not difficult to obtain that
(n)
lim p ij = π j , i, j ∈ I (3.5.11)
n→∞
when the positive recurrent state r from Assumption 3.3.1 is aperiodic.
Before giving the remaining proof of Theorem 3.3.2, we give an interesting
interpretation of the ratio π i /π j for two recurrent states i and j.

Lemma 3.5.10 Suppose that the Markov chain {X n } satisfies Assumption 3.3.1.
Then for any two recurrent states s and ℓ

π ℓ
E(number of visits to state ℓ between two successive visits to state s) = .
π s
Proof Fix states ℓ, s ∈ R. The Markov chain can be considered as a regenerative
process with the epochs at which the process visits state s as regeneration epochs.
Defining a cycle as the time elapsed between two successive visits to state s, it
follows from the definition of the mean recurrence time µ ss that

E(length of one cycle) = µ ss .
By Lemma 3.5.8 the mean cycle length µ ss is finite. Imagine that the Markov chain
earns a reward of 1 each time the process visits state ℓ. Assuming that the process
starts in state s, we have by the renewal-reward theorem from Chapter 2 that

the long-run average reward per time unit
E(reward earned during one cycle)
=
E(length of one cycle)
1
= E(number of visits to state ℓ in one cycle) (3.5.12)
µ ss
with probability 1. On the other hand,

the long-run average reward per time unit
= the long-run average number of visits to state ℓ per time unit.

In the proof of Theorem 3.3.1 we have seen that
the long-run average number of visits to state ℓ per time unit

= π ℓ with probability 1 (3.5.13)