Residual Algorithms: Reinforcement Learning with Function Approximation

14 years 8 months ago
Residual Algorithms: Reinforcement Learning with Function Approximation
A number of reinforcement learning algorithms have been developed that are guaranteed to converge to the optimal solution when used with lookup tables. It is shown, however, that these algorithms can easily become unstable when implemented directly with a general function-approximation system, such as a sigmoidal multilayer perceptron, a radial-basisfunction system, a memory-based learning system, or even a linear function-approximation system. A new class of algorithms, residual gradient algorithms, is proposed, which perform gradient descent on the mean squared Bellman residual, guaranteeing convergence. I shown, however, that they may learn very slowly in some cases. A larger class of algorithms, residual algorithms, is proposed that has the guaranteed convergence of the residual gradient algorithms, yet can retain the fast learning speed of direct algorithms. In fact, both direct and residual gradient algorithms are shown to be special cases of residual algorithms, and it is shown...
Leemon C. Baird III
Added 17 Nov 2009
Updated 17 Nov 2009
Type Conference
Year 1995
Where ICML
Authors Leemon C. Baird III
Comments (0)