Document Length Normalization by Statistical Regression

The document-length normalization problem has been widely studied in the field of Information Retrieval. The Cosine Normalization [2], the Maximum tf Normalization [1] and the Byte Length Normalization [12] are the most commonly used normalization techniques. In [14], authors studied the retrieval probability of documents w.r.t. their size, using different similarity measures. They have shown that none of existing measures retrieve the documents of different lengths with the same probability. We first show here that the document and query sizes are indeed very influent on the similarity score expectation. Therefore, we propose to realize a statistical regression of the similarity scores distribution w.r.t. document and query sizes in order to normalize them. Experimental results appear to indicate that our approach, as well in the field of classical Information Retrieval as when applied to a document clustering process, allows to judge similarities really more fairly.

• Type: Conference
• Year: 2007
• Where: ICTAI
• Authors: Sylvain Lamprier, Tassadit Amghar, Bernard Levrat, Frédéric Saubion
• Keywords: Artificial Intelligence, Byte Length Normalization, Document-length Normalization Problem, ICTAI 2007, Maximum Tf Normalization