Sciweavers

Share
HPDC
2010
IEEE

A data transfer framework for large-scale science experiments

10 years 29 days ago
A data transfer framework for large-scale science experiments
Modern scientific experiments can generate hundreds of gigabytes to terabytes or even petabytes of data that may furthermore be maintained in large numbers of relatively small files. Frequently, this data must be disseminated to remote collaborators or computational centers for data analysis. Moving this data with high performance and strong robustness and providing a simple interface for users are challenging tasks. We present a data transfer framework comprising a high-performance data transfer library based on GridFTP, a data scheduler, and a graphical user interface that allows users to transfer their data easily, reliably, and securely. This system incorporates automatic tuning mechanisms to select at runtime the number of concurrent threads to be used for transfers. Also included are restart mechanisms capable of dealing with client, network, and server failures. Experimental results indicate that our data transfer system can significantly improve data transfer performance and c...
Wantao Liu, Brian Tieman, Rajkumar Kettimuthu, Ian
Added 09 Nov 2010
Updated 09 Nov 2010
Type Conference
Year 2010
Where HPDC
Authors Wantao Liu, Brian Tieman, Rajkumar Kettimuthu, Ian T. Foster
Comments (0)
books