Reconciling scratch space consumption, exposure, and volatility to achieve timely staging of job input data

Innovative scientific applications and emerging dense data sources are creating a data deluge for highend computing systems. Processing such large input data typically involves copying (or staging) onto the supercomputer's specialized high-speed storage, scratch space, for sustained high I/O throughput. The current practice of conservatively staging data as early as possible makes the data vulnerable to storage failures, which may entail re-staging and consequently reduced job throughput. To address this, we present a timely staging framework that uses a combination of job startup time predictions, user-specified intermediate nodes, and decentralized data delivery to coincide input data staging with job start-up. By delaying staging to when it is necessary, the exposure to failures and its effects can be reduced. Evaluation using both PlanetLab and simulations based on three years of Jaguar (No. 1 in Top500) job logs show as much as 85.9% reduction in staging times compared to dir...

• Type: Journal
• Year: 2010
• Where: IPPS
• Authors: Henry M. Monti, Ali Raza Butt, Sudharshan S. Vazhkudai
• Keywords: Data Delivery, Dense Data Sources, Distributed And Parallel Computing, Input Data, IPPS 2010