Page 160 - A First Course In Stochastic Models

THE FLOW RATE EQUATION METHOD 153

has a unique solution. A brute-force method for solving the equilibrium equations
is to truncate this infinite system through a sufficiently large integer N (to be found
by trial and error) such that ∞ [p(i, 0) + p(i, 1)] ≤ ε for some prespecified
i=N+1
accuracy number ε. In Section 4.4 we discuss a more sophisticated method to
solve the infinite system of linear equations. Once the state probabilities have been
computed, we find

∞

the long-run average number of ships in the harbour = i[p(i, 0) + p(i, 1)],
i=1
∞

the fraction of time the unloader is in repair = p(i, 0),
i=1

the long-run average amount of time spent in the harbour per ship
∞
1
= i[p(i, 0) + p(i, 1)].
λ
i=1
The latter result uses Little’s formula L = λW.

Continuous-time Markov chains with rewards

In many applications a reward structure is imposed on the continuous-time Markov
chain model. Let us assume the following reward structure. A reward at a rate of
r(j) per time unit is earned whenever the process is in state j, while a lump
reward of F jk is earned each time the process jumps from state j to state k ( = j).
In addition to Assumption 4.2.1 involving the regeneration state r, we make the
following assumption.

Assumption 4.2.2 (a) The total reward earned between two visits of the process
{X(t)} to state r has a finite expectation and

|r(j)| p j + p j q jk |F jk | < ∞.
j∈I j∈I k =j

(b) For each initial state X(0) = i with i = r, the total reward earned until the
first visit of the process {X(t)} to state r is finite with probability 1.

This assumption is automatically satisfied when the state space I is finite and
Assumption 4.2.1 holds. For each t > 0, define the random variable R(t) by

R(t) = the total reward earned up to time t.

The following very useful result holds for the long-run average reward.