A New Measure of the Cluster Hypothesis

Abstract. We have found that the nearest neighbor (NN) test is an insufficient measure of the cluster hypothesis. The NN test is a local measure of the cluster hypothesis. Designers of new document-to-document similarity measures may incorrectly report effective clustering of relevant documents if they use the NN test alone. Utilizing a measure from network analysis, we present a new, global measure of the cluster hypothesis: normalized mean reciprocal distance. When used together with a local measure, such as the NN test, this new global measure allows researchers to better measure the cluster hypothesis. Key words: Cluster hypothesis, nearest neighbor test, relevant document networks, normalized mean reciprocal distance

• Type: Conference
• Year: 2009
• Where: ICTIR
• Authors: Mark D. Smucker, James Allan
• Keywords: Cluster Hypothesis, ICTIR 2009, Information Retrieval, Local Measure, NN Test