Text segmentation via topic modeling: an analytical study

In this paper, the task of text segmentation is approached from a topic modeling perspective. We investigate the use of latent Dirichlet allocation (LDA) topic model to segment a text into semantically coherent segments. A major benefit of the proposed approach is that along with the segment boundaries, it outputs the topic distribution associated with each segment. This information is of potential use in applications like segment retrieval and discourse analysis. The new approach outperforms a standard baseline method and yields significantly better performance than most of the available unsupervised methods on a benchmark dataset. Categories and Subject Descriptors: I.5.4 [Pattern Recognition]: Applications - text processing General Terms: Algorithms, Experimentation, Performance

• Type: Conference
• Year: 2009
• Where: CIKM
• Authors: Hemant Misra, François Yvon, Joemon M. Jose, Olivier Cappé
• Keywords: CIKM 2009, Database, Latent Dirichlet Allocation, Segment Retrieval, Topic Modeling Perspective