Reinforcement learning of motor skills in high dimensions: A path integral approach

— Reinforcement learning (RL) is one of the most general approaches to learning control. Its applicability to complex motor systems, however, has been largely impossible so far due to the computational difficulties that reinforcement learning encounters in high dimensionsal continuous stateaction spaces. In this paper, we derive a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals. While solidly grounded in optimal control theory and estimation theory, the update equations for learning are surprisingly simple and have no danger of numerical instabilites as neither matrix inversions nor gradient learning rates are required. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to highdimensional control problems. Finally, a learning experiment on a robot dog illustrates the functionality of our algorithm in a real-world scenario. We belie...

• Type: Journal
• Year: 2010
• Where: ICRA
• Authors: Evangelos Theodorou, Jonas Buchli, Stefan Schaal
• Keywords: Gradient Learning Rates, ICRA 2010, Optimal Control, Path Integrals, Robotics