Reducing communication overhead under parallel list processing in multicore clusters

Abstract

The Data List Management Library (DLML) processes data lists in parallel, balancing the workload transparently to programmers. Its first design was targeted at clusters of uniprocessor nodes, and based on multiprocess parallelism and on message-passing communication. This paper presents a multithreaded design of DLML aimed at clusters of multicore nodes to better capitalise on intra-node parallelism. On applications tested, MultiCore DLML runs twice as fast as DLML when message-passing communication is not excessive. Good performance was achieved only after addressing issues relating to MPI communication overhead, cache locality and memory consumption. © 2011 IEEE.