Sciweavers

8
Voted
CLUSTER
2009
IEEE

Combining I/O operations for multiple array variables in parallel netCDF

13 years 9 months ago
Combining I/O operations for multiple array variables in parallel netCDF
—Parallel netCDF (PnetCDF) is a popular library used in many scientific applications to store scientific datasets. It provides high-performance parallel I/O while maintaining file-format compatibility with Unidata’s netCDF. Array variables comprise the bulk of the data in a netCDF dataset, and for accesses to large regions of single array variables, PnetCDF attains very high performance. However, the current PnetCDF interface only allows access to one array variable per call. If an application instead accesses a large number of small-sized array variables, this interface limitation can cause significant performance degradation, because high end network and storage systems deliver much higher performance with larger request sizes. Moreover, the record variables data is stored interleaved by record, and the contiguity information is lost, so the existing MPI-IO collective I/O optimization can not help. This paper presents a new mechanism for PnetCDF to combine multiple I/O operat...
Kui Gao, Wei-keng Liao, Alok N. Choudhary, Robert
Added 20 May 2010
Updated 20 May 2010
Type Conference
Year 2009
Where CLUSTER
Authors Kui Gao, Wei-keng Liao, Alok N. Choudhary, Robert B. Ross, Robert Latham
Comments (0)