Page 7 - A First Course In Stochastic Models

CONTENTS vii

5.6 Queueing Networks 214
5.6.1 Open Network Model 215
5.6.2 Closed Network Model 219
Exercises 224
Bibliographic Notes 230
References 231

6 Discrete-Time Markov Decision Processes 233
6.0 Introduction 233
6.1 The Model 234
6.2 The Policy-Improvement Idea 237
6.3 The Relative Value Function 243
6.4 Policy-Iteration Algorithm 247
6.5 Linear Programming Approach 252
6.6 Value-Iteration Algorithm 259
6.7 Convergence Proofs 267
Exercises 272
Bibliographic Notes 275
References 276
7 Semi-Markov Decision Processes 279

7.0 Introduction 279
7.1 The Semi-Markov Decision Model 280
7.2 Algorithms for an Optimal Policy 284
7.3 Value Iteration and Fictitious Decisions 287
7.4 Optimization of Queues 290
7.5 One-Step Policy Improvement 295
Exercises 300
Bibliographic Notes 304
References 305

8 Advanced Renewal Theory 307
8.0 Introduction 307
8.1 The Renewal Function 307
8.1.1 The Renewal Equation 308
8.1.2 Computation of the Renewal Function 310
8.2 Asymptotic Expansions 313
8.3 Alternating Renewal Processes 321
8.4 Ruin Probabilities 326
Exercises 334
Bibliographic Notes 337
References 338
9 Algorithmic Analysis of Queueing Models 339

9.0 Introduction 339
9.1 Basic Concepts 341