A Stochastic Language Model using Dependency and Its Improvement by Word Clustering

In this paper, we present a stochastic language model for Japanese using dependency. The prediction unit in thismodel isallattributeof "bunsetsu". This isrepresented by the product of the head of content words and that of function words. The relation between the attributes of "bunsetsu" is ruled by a context-free grammar. The word sequences axe predicted from the attributeusing word n-gram model. The spellof Unknow word ispredicted using character n-grainmodel. This model isrobust in that itcan compute the probability of an arbitrary string and iscomplete in that itmodels from unknown word to dependency at the same time.

• Type: Conference
• Year: 1998
• Where: ACL
• Authors: Shinsuke Mori, Makoto Nagao
• Keywords: ACL 1998, ACL 2007, Spellof Unknow Word, Stochastic Language Model, Word N-gram Model