Word Length n-Grams for Text Re-use Detection

10 years 7 months ago
Word Length n-Grams for Text Re-use Detection
Abstract. The automatic detection of shared content in written documents –which includes text reuse and its unacknowledged commitment, plagiarism– has become an important problem in Information Retrieval. This task requires exhaustive comparison of texts in order to determine how similar they are. However, such comparison is impossible in those cases where the amount of documents is too high. Therefore, we have designed a model for the proper pre-selection of closely related documents in order to perform the exhaustive comparison afterwards. We use a similarity measure based on word-level n-grams, which proved to be quite effective in many applications As this approach becomes normally impracticable for real-world large datasets, we propose a method based on a preliminary word-length encoding of texts, substituting a word by its length, providing three important advantages: (i) being the alphabet of the documents reduced to nine symbols, the space needed to store n-gram lists is r...
Alberto Barrón-Cedeño, Chiara Basile
Added 12 Aug 2010
Updated 12 Aug 2010
Type Conference
Year 2010
Authors Alberto Barrón-Cedeño, Chiara Basile, Mirko Degli Esposti, Paolo Rosso
Comments (0)