Improving Communication Performance on InfiniBand by Using Efficient Data Placement Strategies

Despite using high-speed network interconnection systems like InfiniBand, the communication overhead for parallel applications is still high. In this paper we show, how such costs can be reduced by choosing appropriate data placement strategies. For large buffers, we propose a transparent placing in hugepages as it can dramatically decrease memory registration overhead and may increase network bandwidth. Thus, we developed a new library that can be preloaded for applications at load time and cares about drawbacks of using hugepages. So we believe that it is the most suitable one in the HPC area for Linux today. But we do not only refer to large buffers as small communication buffers also play a significant role for application behaviour. We show that transfer latencies vary, depending on data placement. All current communication library implementations for InfiniBand do not utilize scatter-gather lists for send and receive operations, but we show that this feature can have a positive ...

• Type: Conference
• Year: 2006
• Where: CLUSTER
• Authors: Robert Rex, Frank Mietke, Wolfgang Rehm, Christoph Raisch, Hoang Nam Nguyen
• Keywords: CLUSTER 2006, Data Placement, Distributed And Parallel Computing, Large Buffers, Small Communication Buffers