Page 198 - Probability, Random Variables and Random Processes

190 RANDOM PROCESSES [CHAP 5

5.37. Consider a Markov chain with state space (0, 1,2) and transition probability matrix

Show that state 0 is periodic with period 2.

The characteristic equation of P is given by

Thus, by the Cayley-Hamilton theorem (in matrix analysis), we have P3 = P. Thus, for n 2 1,

Therefore d(0) = gcd{n 2 1 : poo(") > 0) = gcd(2, 5, 6, . . .) = 2

Thus, state 0 is periodic with period 2.
Note that the state transition diagram corresponding to the given P is shown in Fig. 5-11. From Fig.
5-11, it is clear that state 0 is periodic with period 2.

Fig. 5-11

5.38. Let two gamblers, A and B, initially have k dollars and m dollars, respectively. Suppose that at
each round of their game, A wins one dollar from B with probability p and loses one dollar to B
with probability q = 1 - p. Assume that A and B play until one of them has no money left. (This
is known as the Gambler's Ruin problem.) Let X, be A's capital after round n, where n = 0, 1,
2, . . . and X, = k.

(a) Show that X(n) = (X,, n 2 0) is a Markov chain with absorbing states.
(b) Find its transition probability matrix P.