Sciweavers

Share
NIPS
2004

Hierarchical Distributed Representations for Statistical Language Modeling

9 years 21 days ago
Hierarchical Distributed Representations for Statistical Language Modeling
Statistical language models estimate the probability of a word occurring in a given context. The most common language models rely on a discrete enumeration of predictive contexts (e.g., n-grams) and consequently fail to capture and exploit statistical regularities across these contexts. In this paper, we show how to learn hierarchical, distributed representations of word contexts that maximize the predictive value of a statistical language model. The representations are initialized by unsupervised algorithms for linear and nonlinear dimensionality reduction [14], then fed as input into a hierarchical mixture of experts, where each expert is a multinomial distribution over predicted words [12]. While the distributed representations in our model are inspired by the neural probabilistic language model of Bengio et al. [2, 3], our particular architecture enables us to work with significantly larger vocabularies and training corpora. For example, on a large-scale bigram modeling task invol...
John Blitzer, Kilian Q. Weinberger, Lawrence K. Sa
Added 31 Oct 2010
Updated 31 Oct 2010
Type Conference
Year 2004
Where NIPS
Authors John Blitzer, Kilian Q. Weinberger, Lawrence K. Saul, Fernando Pereira
Comments (0)
books