Sciweavers

NIPS
2007

Topmoumoute Online Natural Gradient Algorithm

13 years 5 months ago
Topmoumoute Online Natural Gradient Algorithm
Guided by the goal of obtaining an optimization algorithm that is both fast and yields good generalization, we study the descent direction maximizing the decrease in generalization error or the probability of not increasing generalization error. The surprising result is that from both the Bayesian and frequentist perspectives this can yield the natural gradient direction. Although that direction can be very expensive to compute we develop an efficient, general, online approximation to the natural gradient descent which is suited to large scale problems. We report experimental results showing much faster convergence in computation time and in number of iterations with TONGA (Topmoumoute Online natural Gradient Algorithm) than with stochastic gradient descent, even on very large datasets.
Nicolas Le Roux, Pierre-Antoine Manzagol, Yoshua B
Added 30 Oct 2010
Updated 30 Oct 2010
Type Conference
Year 2007
Where NIPS
Authors Nicolas Le Roux, Pierre-Antoine Manzagol, Yoshua Bengio
Comments (0)