Bellman

dynamic programming is two things wearing one name. to bellman it was a mathematical theory of multistage decision processes — sibling to linear programming in the “programming means planning” sense.¹ 𐃏 to a computer scientist it is a technique: solve a problem by combining solutions to subproblems, and never solve the same subproblem twice (Cormen, Thomas H. and Leiserson, Charles E. and Rivest, Ronald L. and Stein, Clifford, 2009). the two are the same idea at different altitudes, and this page covers both. worked implementations also live in this github repo.

q-learning is the algorithm that made reinforcement learning feel inevitable: interact with an unknown world, nudge a table of numbers after every step, and the table converges to the value of optimal behaviour — even while you behave suboptimally the entire time. 𐃏 everything runs on one line of arithmetic, and the rest of this page is the machinery needed to say precisely why that line works. the canonical reference for all of it is sutton & barto’s reinforcement learning: an introduction, free at http://incompleteideas.net/book/the-book-2nd.html.