Revisiting the ODE Method for Recursive Algorithms: Fast Convergence Using Quasi Stochastic Approximation
Several decades ago, Profs. Sean Meyn and Lei Guo were postdoctoral fellows at ANU, where they shared interest in recursive algorithms. It seems fitting to celebrate Lei Guo's 60th birthday with a review of the ODE Method and its recent evolution. The method has been regarded as a technique for algorithm analysis. It is argued that this viewpoint is backwards: The original stochastic approximation method was surely motivated by an ODE, and tools for analysis came much later (based on establishing robustness of Euler approximations). The paper presents a brief survey of recent research in machine learning that shows the power of algorithm design in continuous time, following by careful approximation to obtain a practical recursive algorithm. While these methods are usually presented in a stochastic setting, this is not a prerequisite. In fact, recent theory shows that rates of convergence can be dramatically accelerated by applying techniques inspired by quasi Monte-Carlo. Subject to conditions, the optimal rate of convergence can be obtained by applying the averaging technique of Polyak and Ruppert. The conditions are not universal, but theory suggests alternatives to achieve acceleration. The theory is illustrated with applications to gradient-free optimization, and policy gradient algorithms for reinforcement learning.
- Research Organization:
- National Renewable Energy Laboratory (NREL), Golden, CO (United States)
- Sponsoring Organization:
- USDOE National Renewable Energy Laboratory (NREL), Laboratory Directed Research and Development (LDRD) Program
- DOE Contract Number:
- AC36-08GO28308
- OSTI ID:
- 1866774
- Report Number(s):
- NREL/JA-5D00-82852; MainId:83625; UUID:7b1f0e19-78b1-4858-b2db-e2141be91986; MainAdminID:64465
- Journal Information:
- Journal of Systems Science and Complexity, Vol. 34
- Country of Publication:
- United States
- Language:
- English
Similar Records
Topics in Theoretical Physics
Recursive least-squares learning algorithms for neural networks