Multi-Object Classification via Crowdsourcing with a Reject Option

Abstract

Consider designing an effective crowdsourcing system for $M$-ary classification where crowd workers complete simple binary microtasks, which are aggregated to give the final result. We consider the novel scenario where workers have a reject option, so they may skip microtasks they are unable or unwilling to do. For example, in mismatched speech transcription, workers who do not know the language may be unable to respond to microtasks in phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Furthermore, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.