Maximal Causes for Non-linear Component Extraction

9 years 10 months ago
Maximal Causes for Non-linear Component Extraction
We study a generative model in which hidden causes combine competitively to produce observations. Multiple active causes combine to determine the value of an observed variable through a max function, in the place where algorithms such as sparse coding, independent component analysis, or non-negative matrix factorization would use a sum. This max rule can represent a more realistic model of non-linear interaction between basic components in many settings, including acoustic and image data. While exact maximum-likelihood learning of the parameters of this model proves to be intractable, we show that efficient approximations to expectation-maximization (EM) can be found in the case of sparsely active hidden causes. One of these approximations can be formulated as a neural network model with a generalized softmax activation function and Hebbian learning. Thus, we show that learning in recent softmax-like neural networks may be interpreted as approximate maximization of a data likelihood. ...
Jörg Lücke, Maneesh Sahani
Added 27 Dec 2010
Updated 27 Dec 2010
Type Journal
Year 2008
Where JMLR
Authors Jörg Lücke, Maneesh Sahani
Comments (0)