Page 152 - A First Course In Stochastic Models

THE MODEL 145

∗
Here we run into subtleties that are well beyond the scope of this book. Note
that fundamental difficulties may arise when the state space is infinite, but these
difficulties are absent in almost all practical applications. To avoid the technical
problems, we make the following assumption for the given data q ij .

Assumption 4.1.2 The rates ν i = q ij are positive and bounded in i ∈ I.
j =i
The boundedness assumption is trivially satisfied when I is finite and holds in
most applications with an infinite state space. Using very deep mathematics it can
be shown that under Assumption 4.1.2 the infinitesimal transition rates determine a
unique continuous-time Markov chain {X(t)}. This continuous-time Markov chain
is precisely the Markov jump process constructed according to the above rules (a)

and (b), where the leaving rates are given by ν i = j =i q ij and the p ij by p ij =
q ij /ν i . The continuous-time Markov chain {X(t)} does indeed have the property

q ij t + o( t), j = i,
P {X(t + t) = j | X(t) = i} = (4.1.1)
1 − ν i t + o( t), j = i.
It is noted that Assumption 4.1.2 implies that the constructed continuous-time
Markov chain {X(t)} automatically satisfies Assumption 4.1.1.
In solving specific problems it suffices to specify the infinitesimal transition rates
q ij . We now give two examples. In these examples the q ij are determined as the
result of the interaction of several elementary processes of the Poisson type. The
q ij are found by using the interpretation that q ij t represents the probability of
making a transition to state j in the next t time units when the current state is i
and t is very small.

Example 4.1.1 (continued) Inventory control for an inflammable product

The stochastic process {X(t), t ≥ 0} with X(t) denoting the stock on hand at
time t is a continuous-time Markov chain with state space I = {0, 1, . . . , Q}. Its
infinitesimal transition rates q ij are the result of the interaction of the two indepen-
dent Poisson processes for the demands and the replenishment opportunities. The
q ij are given by
q i,i−1 = λ for i = 1, . . . , Q,

q 0Q = µ and the other q ij = 0.
To see this, note that for any state i with i ≥ 1,

P {X(t + t) = i − 1 | X(t) = i}
= P {a demand occurs in (t, t + t]} + o( t)
= λ t + o( t)

∗ Conditions under which the infinitesimal parameters determine a unique continuous-time Markov chain
are discussed in depth in Chung (1967).