Trans-dimensional Random Fields for Language Modeling

8 years 9 days ago

Download stat.rutgers.edu

Language modeling (LM) involves determining the joint probability of words in a sentence. The conditional approach is dominant, representing the joint probability in terms of conditionals. Examples include n-gram LMs and neural network LMs. An alternative approach, called the random ﬁeld (RF) approach, is used in whole-sentence maximum entropy (WSME) LMs. Although the RF approach has potential beneﬁts, the empirical results of previous WSME models are not satisfactory. In this paper, we revisit the RF approach for language modeling, with a number of innovations. We propose a trans-dimensional RF (TDRF) model and develop a training algorithm using joint stochastic approximation and trans-dimensional mixture sampling. We perform speech recognition experiments on Wall Street Journal data, and ﬁnd that our TDRF models lead to performances as good as the recurrent neural network LMs but are computationally more efﬁcient in computing sentence probability.

Bin Wang, Zhijian Ou, Zhiqiang Tan

Real-time Traffic