Sciweavers

Share
ACL
2006

Discriminative Pruning of Language Models for Chinese Word Segmentation

9 years 5 months ago
Discriminative Pruning of Language Models for Chinese Word Segmentation
This paper presents a discriminative pruning method of n-gram language model for Chinese word segmentation. To reduce the size of the language model that is used in a Chinese word segmentation system, importance of each bigram is computed in terms of discriminative pruning criterion that is related to the performance loss caused by pruning the bigram. Then we propose a step-by-step growing algorithm to build the language model of desired size. Experimental results show that the discriminative pruning method leads to a much smaller model compared with the model pruned using the state-of-the-art method. At the same Chinese word segmentation F-measure, the number of bigrams in the model can be reduced by up to 90%. Correlation between language model perplexity and word segmentation performance is also discussed.
Jianfeng Li, Haifeng Wang, Dengjun Ren, Guohua Li
Added 30 Oct 2010
Updated 30 Oct 2010
Type Conference
Year 2006
Where ACL
Authors Jianfeng Li, Haifeng Wang, Dengjun Ren, Guohua Li
Comments (0)
books