Page 174 - Schaum's Outlines - Probability, Random Variables And Random Processes

CHAP. 51 RANDOM PROCESSES 167

is known as the Chapman-Kolmogorou equation. It expresses the fact that a transition from i to j in
n + m steps can be achieved by moving from i to an intermediate k in n steps (with probability pik(n)),
and then proceeding to j from k in m steps (with probability p,]")). Furthermore, the events "go from i
to k in n steps" and "go from k to j in m steps" are independent. Hence the probability of the
transition from i to j in n + rn steps via i, k, j is pik(")pk]"). Finally, the probability of the transition
from i to j is obtained by summing over the intermediate state k.

C. The Probability Distribution of {X, , n 2 0) :
Let pi(n) = P(X, = i) and

Then pi(0) = P(Xo = i) are the initial-state probabilities,

is called the initial-state probability vector, and p(n) is called the state probability vector after n tran-
sitions or the probability distribution of X,. Now it can be shown that (Prob. 5.29)

which indicates that the probability distribution of a homogeneous Markov chain is completely
determined by the one-step transition probability matrix P and the initial-state probability vector
HO).

D. Classification of States:
1. Accessible States :
State j is said to be accessible from state i if for some n 2 0, pi,.('" z 0, and we write i -+ j. Two
states i and j accessible to each other are said to communicate, and we write i-j. If all states commu-
nicate with each other, then we say that the Markov chain is irreducible.
2. Recurrent States:
Let be the time (or the number of steps) of the first visit to state j after time zero, unless state j
is never visited, in which case we set Tj = oo. Then IT;. is a discrete r.v. taking values in (1, 2, . . . , m}.
Let
,...,
f;:~m)=~(T,=ml~,=i)=~(~,=j,~,#j,k=l,2 m-llXo=i) (5.40)
and&iO) = 0 since 7j 2 1. Then

and
The probability of visiting j in finite time, starting from i, is given by

Now state j is said to be recurrent if
fjj=P(T,< coIX,=j)= 1