Practical Riemannian Neural Networks

4 years 4 months ago
Practical Riemannian Neural Networks
We provide the first experimental results on non-synthetic datasets for the quasidiagonal Riemannian gradient descents for neural networks introduced in [Oll15]. These include the MNIST, SVHN, and FACE datasets as well as a previously unpublished electroencephalogram dataset. The quasi-diagonal Riemannian algorithms consistently beat simple stochastic gradient gradient descents by a varying margin. The computational overhead with respect to simple backpropagation is around a factor 2. Perhaps more interestingly, these methods also reach their final performance quickly, thus requiring fewer training epochs and a smaller total computation time. We also present an implementation guide to these Riemannian gradient descents for neural networks, showing how the quasi-diagonal versions can be implemented with minimal effort on top of existing routines which compute gradients. We present a practical and efficient implementation of invariant stochastic gradient descent algorithms for neural...
Gaétan Marceau-Caron, Yann Ollivier
Added 31 Mar 2016
Updated 31 Mar 2016
Type Journal
Year 2016
Where CORR
Authors Gaétan Marceau-Caron, Yann Ollivier
Comments (0)