A distributed shared buffer space for data-intensive applications

Efficient memory allocation and data transfer for cluster-based data-intensive applications is a difficult task. Both changes in cluster interconnects and application workloads usually require tuning of the application and network code. We propose separating control and data transfer traffic by accessing data through a DSM-like cluster-wide shared buffer space and only including buffer references in the control messages. Using a generic API for accessing buffers allows for tuning data transfer without changing the application code. A prototype, implemented in the context of a distributed storage system, has been validated with several networking technologies, showing that such a framework can combine performance and flexibility.

• Type: Conference
• Year: 2005
• Where: CCGRID
• Authors: Renaud Lachaize, Jorgen S. Hansen
• Keywords: CCGRID 2005, Cluster Computing, Cluster-based Data-intensive Applications, Data Transfer Traffic, Efficient Memory Allocation