Page 174 - Probability, Random Variables and Random Processes

166 RANDOM PROCESSES [CHAP 5

In the case where the state space E is finite and equal to (1, 2, . . . , m), P is m x m dimensional; that is,

where

A square matrix whose elements satisfy Eq. (5.34) or (5.35) is called a Markov matrix or stochastic
matrix.

B. Higher-Order Transition Probabilities-Chapman-Kolmogorov Equation:
Tractability of Markov chain models is based on the fact that the probability distribution of
(X,, 2 0) can be computed by matrix manipulations.
n
Let P = pi,] be the transition probability matrix of a Markov chain {X,, n 2 0). Matrix powers
of P are defined by

with the (i, j)th element given by

Note that when the state space E is infinite, the series above converges, since by Eq. (5.34),

Similarly, p3 = PP~ has the (i, j)th element

and in general, Pn + = PPn has the (i, j)th element

Finally, we define PO = I, where I is the identity matrix.
The n-step transition probabilities for the homogeneous Markov chain (X,, n 2 0) are defined
by

Then we can show that (Prob. 5.70)

We compute pip1 by taking matrix powers.
The matrix identity

when written in terms of elements