15.2 Continuous-Time Dynamic Programming

Dynamic programming has been a recurring theme throughout most of this
book. So far, it has always taken the form of computing optimal
cost-to-go (or cost-to-come) functions over some sequence of stages.
Both value iteration and Dijkstra-like algorithms have emerged. In
computer science, dynamic programming is a fundamental insight in the
development of algorithms that compute optimal solutions to problems.
In its original form, however, dynamic programming was developed to
solve the optimal control problem [84]. In this setting, a
discrete set of stages is replaced by a continuum of stages, known as
*time*. The dynamic programming recurrence is instead a partial
differential equation, called the Hamilton-Jacobi-Bellman (HJB)
equation. The HJB equation can be solved using numerical algorithms;
however, in some cases, it can be *solved
analytically*.^{15.3} Section
15.2.2 briefly describes an analytical solution in the case of
linear systems. Section 15.2.3 covers Pontryagin's minimum
principle, which can be derived from the dynamic programming
principle, and generalizes the optimization performed in Hamiltonian
mechanics (recall Section 13.4.4).