Optimization and Analysis of MPI Collective Communication on Fat-Tree Networks

Abstract

We explore new collective algorithms to optimize MPIBcast, MPIReduce and MPIAllreduce on InfiniBand clusters. Our algorithms are specifically designed for fat-tree networks. We present multi-color k-ary trees with a novel mapping scheme to map the colors to fat-tree network nodes. Our multi-color tree algorithms result in better utilization of network links over traditional algorithms on fat-tree networks. We also present optimizations for clusters of SMP nodes as we explore both hybrid and Multi Leader SMP techniques to achieve the best performance. We show the benefits of our algorithms with performance results from micro-benchmarks on POWER8 and X86 InfiniBand clusters. We also show performance optimizations from our algorithms in the PARATEC and QBOX applications.