Data Management on Grid Filesystem for Data-Intensive Computing

10 years 2 months ago
Data Management on Grid Filesystem for Data-Intensive Computing
In parallel computing environments such as HPC clusters and the Grid, data-intensive applications involve large overhead costs due to a concentration of access to the files on common nodes. To avoid this problem in traditional distributed filesystems, users have to distribute the file access manually. However, such solution has some difficulties for users in the Grid environment. We propose a data management mechanism for data-intensive computing on Grid filesystem. Our technique improves the file access performance by automatically scheduling the file access and the data management on the filesystem. The filesystem is based on dynamically configured node groups corresponding to the network topology. Utilizing the configuration, it monitors file access to detect concentrated situations, creates the file replica, and schedules its placement and access. We applied the proposal technique to the Gfarm, a filesystem that scales to the Grid. We emulate real application workloa...
Hitoshi Sato, Satoshi Matsuoka
Added 04 Jun 2010
Updated 04 Jun 2010
Type Conference
Year 2007
Authors Hitoshi Sato, Satoshi Matsuoka
Comments (0)