Compiling a benchmark of documented multi-threaded bugs

Abstract

Testing multi-threaded, concurrent, or distributed programs is acknowledged to be a very difficult task. We decided to create a benchmark of programs containing documented multi-threaded bugs that can be used in the development of testing tool for the domain. In order to augment the benchmark with a sizable number of programs, we assigned students in a software testing class to write buggy multi-threaded Java programs and document the bugs. This paper documents this experiment. We explain the task that was given to the students, go over the bugs that they put into the programs both intentionally and unintentionally, and show our findings. We believe this part of the benchmark shows typical programming practices, including bugs, of novice programmers. In grading the assignments, we used our 'technologies to look for undocumented bugs. In addition to finding many undocumented bugs, which was not surprising given that writing correct multi-threaded code is difficult, we also found a number of bugs in our tools. We think this is a good indication of the expected utility of the benchmark for multi-threaded testing tool creators.