About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
Transactions of the Japanese Society for Artificial Intelligence
Paper
Learning from crowds and experts
Abstract
Crowdsourcing services are often used to collect a large amount of labeled data for machine learning. Although they provide us an easy way to get labels at very low cost in a short period, they have serious limitations. One of them is the variable quality of the crowd-generated data. There have been many attempts to increase the reliability of crowdgenerated data and the quality of classifiers obtained from such data. However, in these problem settings, relatively few researchers have tried using expert-generated data to achieve further improvements. In this paper, we apply three models that deal with the problem of learning from crowds to this problem: a latent class model, a personal classifier model, and a data-dependent error model. We evaluate these methods against two baseline methods on a real data set to demonstrate the effectiveness of combining crowd-generated data and expert-generated data.