Scalability of findability: effective and efficient IR operations in large information networks

It is crucial to study basic principles that support adaptive and scalable retrieval functions in large networked environments such as the Web, where information is distributed among dynamic systems. We conducted experiments on decentralized IR operations on various scales of information networks and analyzed effectiveness, efficiency, and scalability of various search methods. Results showed network structure, i.e., how distributed systems connect to one another, is crucial for retrieval performance. Relying on partial indexes of distributed systems, some level of network clustering enabled very efficient and effective discovery of relevant information in large scale networks. For a given network clustering level, search time was well explained by a polylogarithmic relation to network size (i.e., the number of distributed systems), indicating a high scalability potential for searching in a growing information space. In addition, network clustering only involved local self-organizatio...

• Type: Conference
• Year: 2010
• Where: SIGIR
• Authors: Weimao Ke, Javed Mostafa
• Keywords: Distributed Systems, Information, Information Technology, SIGIR 2010, Various Scales