Sciweavers

Share
ICDM
2010
IEEE

Averaged Stochastic Gradient Descent with Feedback: An Accurate, Robust, and Fast Training Method

8 years 3 months ago
Averaged Stochastic Gradient Descent with Feedback: An Accurate, Robust, and Fast Training Method
On large datasets, the popular training approach has been stochastic gradient descent (SGD). This paper proposes a modification of SGD, called averaged SGD with feedback (ASF), that significantly improves the performance (robustness, accuracy, and training speed) over the traditional SGD. The proposal is based on three simple ideas: averaging the weight vectors across SGD iterations, feeding the averaged weights back into the SGD update process, and deciding when to perform the feedback (linearly slowing down feedback). Theoretically, we demonstrate the reasonable convergence properties of the ASF. Empirically, the ASF outperforms several strong baselines in terms of accuracy, robustness over the noise, and the training speed. To our knowledge, this is the first study of "feedback" in stochastic gradient learning. Although we choose latent conditional models for verifying the ASF in this paper, the ASF is a general purpose technique just like SGD, and can be directly applied ...
Xu Sun, Hisashi Kashima, Takuya Matsuzaki, Naonori
Added 12 Feb 2011
Updated 12 Feb 2011
Type Journal
Year 2010
Where ICDM
Authors Xu Sun, Hisashi Kashima, Takuya Matsuzaki, Naonori Ueda
Comments (0)
books