Reinforcement learning and stochastic optimization: a unified framework for sequential decisions

"The first step in sequential decision problems is to understand what decisions are being made. It is surprising how often it is that people faced with complex problems, which spans scientists in a lab to people trying to solve major health problems, are not able to identify the decisions they face. We then want to find a method for making decisions. There are at least 45 words in the English language that are equivalent to "method for making a decision," but the one we have settled on is policy. The term policy is very familiar to fields such as Markov decision processes and reinforcement learning, but with a much narrower interpretation than we will use. Other fields do not use the term at all. Designing effective policies will be the focus of most of this book. Even more subtle is identifying the different sources of uncertainty. It can be hard enough trying to identify potential decisions, but thinking about all the random events that might affect whatever it is that you are managing, whether it is reducing disease, managing inventories, or making investments, can seem like a hopeless challenge"--

Preface xxv -- Acknowledgments xxxi -- Part I – Introduction 1 -- 1 Sequential Decision Problems 3 -- 1.1 The Audience 7 -- 1.2 The Communities of Sequential Decision Problems 8 -- 1.3 Our Universal Modeling Framework 10 -- 1.4 Designing Policies for Sequential Decision Problems 15 -- 1.5 Learning 20 -- 1.6 Themes 21 -- 1.7 Our Modeling Approach 27 -- 1.8 How to Read this Book 27 -- 1.9 Bibliographic Notes 33 -- Exercises 34 -- Bibliography 38 -- 2 Canonical Problems and Applications 39 -- 2.1 Canonical Problems 39 -- 2.2 A Universal Modeling Framework for Sequential Decision Problems 64 -- 2.3 Applications 69 -- 2.4 Bibliographic Notes 85 -- Exercises 90 -- Bibliography 93 -- 3 Online Learning 101 -- 3.1 Machine Learning for Sequential Decisions 102 -- 3.2 Adaptive Learning Using Exponential Smoothing 110 -- 3.3 Lookup Tables with Frequentist Updating 111 -- 3.4 Lookup Tables with Bayesian Updating 112 -- 3.5 Computing Bias and Variance* 118 -- 3.6 Lookup Tables and Aggregation* 121 -- 3.7 Linear Parametric Models 131 -- 3.8 Recursive Least Squares for Linear Models 136 -- 3.9 Nonlinear Parametric Models 140 -- 3.10 Nonparametric Models* 149 -- 3.11 Nonstationary Learning* 159 --…