Server-Side Parallel Data Reduction and Analysis

13 years 11 months ago

Download dust.ess.uci.edu

Abstract. Geoscience analysis is currently limited by cumbersome access and manipulation of large datasets from remote sources. Due to their data-heavy and compute-light nature, these analysis workloads represent a class of applications unsuited to a computational grid optimized for compute-intensive applications. We present the Script Workﬂow Analysis for MultiProcessing (SWAMP) system, which relocates data-intensive workﬂows from scientists’ workstations to the hosting datacenters in order to reduce data transfer and exploit locality. Our colocation of computation and data leverages the typically reductive characteristics of these workﬂows, allowing SWAMP to complete workﬂows in a fraction of the time and with much less data transfer. We describe SWAMP’s implementation and interface, which is designed to leverage scientists’ existing script-based workﬂows. Tests with a production geoscience workﬂow show drastic improvements not only in overall execution time, but in...

Daniel L. Wang, Charles S. Zender, Stephen F. Jenk

Real-time Traffic