DiSK: A distributed shared disk cache for HPC environments

Abstract—Data movement within high performance environments can be a large bottleneck to the overall performance of programs. With the addition of continuous storage and usage of older data, the back end storage is becoming a larger problem than the improving network and computational nodes. This has led us to develop a Distributed Shared Disk Cache, DiSK, to reduce the dependence on these back end storage systems. With DiSK requested files will be distributed across nodes in order to reduce the amount of requests directed to the archives. DiSK has two key components. One is a Distributed Metadata Management, DIMM, scheme that allows a centralized manager to access what data is available in the system. This is accomplished through the use of a counterbased bloomfilter with locality checks in order to reduce false positives and false negatives. The second component is a method of replication called Differentiable Replication, DiR. The novelty of DiR is that the requirements of the �...

• Type: Conference
• Year: 2009
• Where: COLCOM
• Authors: Brandon Szeliga, Tung Nguyen, Weisong Shi
• Keywords: COLCOM 2009, DiSK Requested files, Distributed And Parallel Computing, End Storage, End Storage Systems