Sciweavers

Share
KDD
2008
ACM

Unsupervised deduplication using cross-field dependencies

9 years 2 months ago
Unsupervised deduplication using cross-field dependencies
Recent work in deduplication has shown that collective deduplication of different attribute types can improve performance. But although these techniques cluster the attributes collectively, they do not model them collectively. For example, in citations in the research literature, canonical venue strings and title strings are dependent--because venues tend to focus on a few research areas--but this dependence is not modeled by current unsupervised techniques. We call this dependence between fields in a record a cross-field dependence. In this paper, we present an unsupervised generative model for the deduplication problem that explicitly models cross-field dependence. Our model uses a single set of latent variables to control two disparate clustering models: a Dirichlet-multinomial model over titles, and a non-exchangeable string-edit model over venues. We show that modeling cross-field dependence yields a substantial improvement in performance--a 58% reduction in error over a standard...
Robert Hall, Charles A. Sutton, Andrew McCallum
Added 30 Nov 2009
Updated 30 Nov 2009
Type Conference
Year 2008
Where KDD
Authors Robert Hall, Charles A. Sutton, Andrew McCallum
Comments (0)
books