Sciweavers

Share
CORR
1998
Springer

Similarity-Based Models of Word Cooccurrence Probabilities

8 years 10 months ago
Similarity-Based Models of Word Cooccurrence Probabilities
Abstract. In many applications of natural language processing (NLP) it is necessary to determine the likelihood of a given word combination. For example, a speech recognizer may need to determine which of the two word combinations “eat a peach” and “eat a beach” is more likely. Statistical NLP methods determine the likelihood of a word combination from its frequency in a training corpus. However, the nature of language is such that many word combinations are infrequent and do not occur in any given corpus. In this work we propose a method for estimating the probability of such previously unseen word combinations using available information on “most similar” words. We describe probabilistic word association models based on distributional word similarity, and apply them to two tasks, language modeling and pseudo-word disambiguation. In the language modeling task, a similarity-based model is used to improve probability estimates for unseen bigrams in a back-off language model...
Ido Dagan, Lillian Lee, Fernando C. N. Pereira
Added 22 Dec 2010
Updated 22 Dec 2010
Type Journal
Year 1998
Where CORR
Authors Ido Dagan, Lillian Lee, Fernando C. N. Pereira
Comments (0)
books