Discrete Time Processes

This unique text for beginning graduate students gives a self-contained introduction to the mathematical properties of stochastics and presents their applications to Markov processes, coding theory, population dynamics, and search engine design. The book is ideal for a newly designed course in an introduction to probability and information theory. Prerequisites include working knowledge of linear algebra, calculus, and probability theory.

This unique text for beginning graduate students gives a self-contained introduction to the mathematical properties of stochastics and presents their applications to Markov processes, coding theory, population dynamics, and search engine design. The book is ideal for a newly designed course in an introduction to probability and information theory. Prerequisites include working knowledge of linear algebra, calculus, and probability theory.

More and more digital devices are being used for informa­ tion processing and control purposes in a variety of systems applications, including industrial processes, power networks, biological systems and communication networks. This trend has been helped by the advent of microprocessors and the consequent availability of cheap distributed computing power. For those applications, where digital devices are used, it is reasonable to model the system in discrete-time. In addition there are other application areas, e.g. econometric systems, business systems, certain command and control systems, environmental systems, where the underlying models are in discrete-time and here discrete-time approaches to analysis and control are the most appropriate.

Devoted to a systematic exposition of some recent developments in the theory of discrete-time Markov control processes, the text is mainly confined to MCPs with Borel state and control spaces.

Stochastic control theory is a relatively young branch of mathematics. The beginning of its intensive development falls in the late 1950s and early 1960s. ~urin~ that period an extensive literature appeared on optimal stochastic control using the quadratic performance criterion (see references in Wonham [76]). At the same time, Girsanov [25] and Howard [26] made the first steps in constructing a general theory, based on Bellman's technique of dynamic programming, developed by him somewhat earlier [4]. Two types of engineering problems engendered two different parts of stochastic control theory. Problems of the first type are associated with multistep decision making in discrete time, and are treated in the theory of discrete stochastic dynamic programming.