Sciweavers

Share
CDC
2010
IEEE

Pathologies of temporal difference methods in approximate dynamic programming

10 years 6 months ago
Pathologies of temporal difference methods in approximate dynamic programming
Approximate policy iteration methods based on temporal differences are popular in practice, and have been tested extensively, dating to the early nineties, but the associated convergence behavior is complex, and not well understood at present. An important question is whether the policy iteration process is seriously hampered by oscillations between poor policies, roughly similar to the attraction of gradient methods to poor local minima. There has been little apparent concern in the approximate DP/reinforcement learning literature about this possibility, even though it has been documented with several simple examples. Recent computational experimentation with the game of tetris, a popular testbed for approximate DP algorithms over a 15-year period, has brought the issue to sharp focus. In particular, using a standard set of 22 features and temporal difference methods, an average score of a few thousands was achieved. Using the same features and a random search method, an overwhelmingl...
Dimitri P. Bertsekas
Added 13 May 2011
Updated 13 May 2011
Type Journal
Year 2010
Where CDC
Authors Dimitri P. Bertsekas
Comments (0)
books