Page 105 - A First Course In Stochastic Models

THE EQUILIBRIUM PROBABILITIES 97

for transient states j. Fix now a recurrent state j. By the definition of recurrence,
we have f jj = 1. The times between successive visits to state j are independent
and identically distributed random variables with mean µ jj . In other words, visits
of the Markov chain to state j can be seen as renewals. Denote by N(t) the number
of visits of the Markov chain to state j during the first t transition epochs. Then,
by Lemma 2.2.2,

N(t) 1
lim = with probability 1. (3.3.3)
t→∞ t µ jj
This limiting result holds for both µ jj < ∞ and µ jj = ∞. In other words, the
long-run average number of transitions to state j per time unit equals 1/µ jj with
probability 1 when the process starts in state j. Define the indicator variable

1 if the process visits state j at time k,
I k =
0 otherwise.
Since N(n) = I 1 + · · · + I n , we can rewrite (3.3.3) as

n
1 1
lim I k = with probability 1. (3.3.4)
n→∞ n µ jj
k=1
Obviously,
(k)
E(I k | X 0 = j) = P {X k = j | X 0 = j} = p .
jj

n
Noting that (1/n) k=1 k is bounded by 1 and using the bounded convergence
I
theorem from Appendix A, it follows from (3.3.4) that
n n
1 1 1
= E lim I k | X 0 = j = lim E I k | X 0 = j
µ jj n→∞ n n→∞ n
k=1 k=1
n n
1 1 (k)
= lim E (I k | X 0 = j) = lim p jj .
n→∞ n n→∞ n
k=1 k=1
It remains to prove that (3.3.2) holds for any state i = j. To do so, we use the
relation (3.2.12) which was derived in the proof of Lemma 3.2.3. Averaging this
relation over n = 1, . . . , m, interchanging the order of summation and letting
m → ∞, the relation (3.3.2) follows in the same way as (3.2.13).

Another natural question is under which condition the effect of the initial state

n (k)
of the process fades away as time increases so that lim n→∞ (1/n) k=1 p ij does
not depend on the initial state X 0 = i for each j ∈ I. We need some condition as
the following example shows. Take a Markov chain with state space I = {1, 2} and
the one-step transition probabilities p ij with p 11 = p 22 = 1 and p 12 = p 21 = 0. In
(n) (n)
n (k)
this example p = 1 and p = 0 for all n ≥ 1 so that lim n→∞ (1/n) p
11 21 k=1 i1