Title:
Feynman-Kac Numerical Techniques for Stochastic Optimal Control
Feynman-Kac Numerical Techniques for Stochastic Optimal Control
Author(s)
Hawkins, Kelsey Pal
Advisor(s)
Tsiotras, Panagiotis
Editor(s)
Collections
Supplementary to
Permanent Link
Abstract
Three significant advancements are proposed for improving numerical methods in the solution of forward-backward stochastic differential equations (FBSDEs) appearing in the Feynman-Kac representation of the value function in stochastic optimal control (SOC) problems. First, we propose a novel characterization of FBSDE estimators as either on-policy or off-policy, highlighting the intuition for these techniques that the distribution over which value functions are approximated should, to some extent, match the distribution the policies generate.
Second, two novel numerical estimators are proposed for improving the accuracy of single-timestep updates. In the case of LQR problems, we demonstrate both in theory and in numerical simulation that our estimators result in near machine-precision level accuracy, in contrast to previously proposed methods that can potentially diverge on the same problems.
Third, we propose a new method for accelerating the global convergence of FBSDE methods. By the repeated use of the Girsanov change of probability measures, it is demonstrated how a McKean-Markov branched sampling method can be utilized for the forward integration pass, as long as the controlled drift terms are appropriately compensated in the backward integration pass. Subsequently, a numerical approximation of the value function is proposed by solving a series of function approximation problems backwards in time along the edges of a space-filling tree.
Sponsor
Date Issued
2021-08-25
Extent
Resource Type
Text
Resource Subtype
Dissertation