A Probabilistic Model for Online Document Clustering with Application to Novelty Detection

In this paper we propose a probabilistic model for online document clustering. We use non-parametric Dirichlet process prior to model the growing number of clusters, and use a prior of general English language model as the base distribution to handle the generation of novel clusters. Furthermore, cluster uncertainty is modeled with a Bayesian Dirichletmultinomial distribution. We use empirical Bayes method to estimate hyperparameters based on a historical dataset. Our probabilistic model is applied to the novelty detection task in Topic Detection and Tracking (TDT) and compared with existing approaches in the literature.

• Type: Conference
• Year: 2004
• Where: NIPS
• Authors: Jian Zhang 0003, Zoubin Ghahramani, Yiming Yang
• Keywords: English Language Model, NIPS 2004, NIPS 2007, Non-parametric Dirichlet Process, Probabilistic Model