Experience with a concurrency bugs benchmark

Abstract

We describe a benchmark of publicly-available multi-threaded programs with documented bugs in them. This project was initiated a few years ago with the goal of helping research groups in the fields of concurrent testing and debugging to develop tools and algorithms that improve the quality of concurrent programs. We present a survey of usage of the benchmark, concluding that the benchmark had an impact on the research in the field of testing and debugging of concurrent programs. We also present new possible directions to foster a discussion about new goals to be set for this initiative. © 2008 IEEE.