Sciweavers

Share
JASIS
2007

Exploiting parallelism to support scalable hierarchical clustering

8 years 10 months ago
Exploiting parallelism to support scalable hierarchical clustering
A distributed memory parallel version of the group average Hierarchical Agglomerative Clustering algorithm is proposed to enable scaling the document clustering problem to large collections. Using standard message passing operations reduces interprocess communication while maintaining efficient load balancing. In a series of experiments using a subset of a standard TREC test collection, our parallel hierarchical clustering algorithm is shown to be scalable in terms of processors efficiently used and the collection size. Results show that our algorithm performs close to the expected O(n2 /p) time on p processors, rather than the worst-case O(n3 /p) time . Furthermore, the O(n2 /p) memory complexity per node allows larger collections to be clustered as the number of nodes increases. While partitioning algorithms such as k-means are trivially parallelizable, our results confirm those of other studies showing that hierarchical algorithms produce significantly tighter clusters in the d...
Rebecca Cathey, Eric C. Jensen, Steven M. Beitzel,
Added 15 Dec 2010
Updated 15 Dec 2010
Type Journal
Year 2007
Where JASIS
Authors Rebecca Cathey, Eric C. Jensen, Steven M. Beitzel, Ophir Frieder, David A. Grossman
Comments (0)
books