Sciweavers

Share
CIKM
2011
Springer

Worker types and personality traits in crowdsourcing relevance labels

10 years 1 months ago
Worker types and personality traits in crowdsourcing relevance labels
Crowdsourcing platforms offer unprecedented opportunities for creating evaluation benchmarks, but suffer from varied output quality from crowd workers who possess different levels of competence and aspiration. This raises new challenges for quality control and requires an in-depth understanding of how workers’ characteristics relate to the quality of their work. In this paper, we use behavioral observations (HIT completion time, fraction of useful labels, label accuracy) to define five worker types: Spammer, Sloppy, Incompetent, Competent, Diligent. Using data collected from workers engaged in the crowdsourced evaluation of the INEX 2010 Book Track Prove It task, we relate the worker types to label accuracy and personality trait information along the ‘Big Five’ personality dimensions. We expect that these new insights about the types of crowd workers and the quality of their work will inform how to design HITs to attract the best workers to a task and explain why certain HIT d...
Gabriella Kazai, Jaap Kamps, Natasa Milic-Frayling
Added 13 Dec 2011
Updated 13 Dec 2011
Type Journal
Year 2011
Where CIKM
Authors Gabriella Kazai, Jaap Kamps, Natasa Milic-Frayling
Comments (0)
books