This book gives a comprehensive and self-contained introduction to the theory of symmetric Markov processes and symmetric quasi-regular Dirichlet forms. In a detailed and accessible manner, Zhen-Qing Chen and Masatoshi Fukushima cover the essential elements and applications of the theory of symmetric Markov processes, including recurrence/transience criteria, probabilistic potential theory, additive functional theory, and time change theory. The authors develop the theory in a general framework of symmetric quasi-regular Dirichlet forms in a unified manner with that of regular Dirichlet forms, emphasizing the role of extended Dirichlet spaces and the rich interplay between the probabilistic and analytic aspects of the theory. Chen and Fukushima then address the latest advances in the theory, presented here for the first time in any book. Topics include the characterization of time-changed Markov processes in terms of Douglas integrals and a systematic account of reflected Dirichlet spaces, and the important roles such advances play in the boundary theory of symmetric Markov processes. This volume is an ideal resource for researchers and practitioners, and can also serve as a textbook for advanced graduate students. It includes examples, appendixes, and exercises with solutions.

This introduction to some of the principal models in the theory of disordered systems leads the reader through the basics, to the very edge of contemporary research, with the minimum of technical fuss. Topics covered include random walk, percolation, self-avoiding walk, interacting particle systems, uniform spanning tree, random graphs, as well as the Ising, Potts, and random-cluster models for ferromagnetism, and the Lorentz model for motion in a random medium. This new edition features accounts of major recent progress, including the exact value of the connective constant of the hexagonal lattice, and the critical point of the random-cluster model on the square lattice. The choice of topics is strongly motivated by modern applications, and focuses on areas that merit further research. Accessible to a wide audience of mathematicians and physicists, this book can be used as a graduate course text. Each chapter ends with a range of exercises.

This book covers formulation, algorithms, and structural results of partially observed Markov decision processes, whilst linking theory to real-world applications in controlled sensing. Computations are kept to a minimum, enabling students and researchers in engineering, operations research, and economics to understand the methods and determine the structure of their optimal solution.

DIVAn investigation of the logical foundations of the theory behind Markov random processes, this text explores subprocesses, transition functions, and conditions for boundedness and continuity. 1961 edition. /div

The significantly expanded and updated new edition of a widely used text on reinforcement learning, one of the most active research areas in artificial intelligence. Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives while interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the field's key ideas and algorithms. This second edition has been significantly expanded and updated, presenting new topics and updating coverage of other topics. Like the first edition, this second edition focuses on core online learning algorithms, with the more mathematical material set off in shaded boxes. Part I covers as much of reinforcement learning as possible without going beyond the tabular case for which exact solutions can be found. Many algorithms presented in this part are new to the second edition, including UCB, Expected Sarsa, and Double Learning. Part II extends these ideas to function approximation, with new sections on such topics as artificial neural networks and the Fourier basis, and offers expanded treatment of off-policy learning and policy-gradient methods. Part III has new chapters on reinforcement learning's relationships to psychology and neuroscience, as well as an updated case-studies chapter including AlphaGo and AlphaGo Zero, Atari game playing, and IBM Watson's wagering strategy. The final chapter discusses the future societal impacts of reinforcement learning.

The main theme of this book is the interplay between the behaviour of a class of stochastic processes (random walks) and discrete structure theory. The author considers Markov chains whose state space is equipped with the structure of an infinite, locally finite graph, or as a particular case, of a finitely generated group. The transition probabilities are assumed to be adapted to the underlying structure in some way that must be specified precisely in each case. From the probabilistic viewpoint, the question is what impact the particular type of structure has on various aspects of the behaviour of the random walk. Vice-versa, random walks may also be seen as useful tools for classifying, or at least describing the structure of graphs and groups. Links with spectral theory and discrete potential theory are also discussed. This book will be essential reading for all researchers working in stochastic process and related topics.

The Application of Hidden Markov Models in Speech Recognition presents the core architecture of a HMM-based LVCSR system and proceeds to describe the various refinements which are needed to achieve state-of-the-art performance.

Since the publication of the first edition in 1994, this book has attracted constant interests from readers and is by now regarded as a standard reference for the theory of Dirichlet forms. For the present second edition, the authors not only revise