Sciweavers

Share
CSCW
2016
ACM

Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks

3 years 18 days ago
Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks
Crowdsourcing is a common strategy for collecting the “gold standard” labels required for many natural language applications. Crowdworkers differ in their responses for many reasons, but existing approaches often treat disagreements as “noise” to be removed through filtering or aggregation. In this paper, we introduce the workflow design pattern of crowd parting: separating workers based on shared patterns in responses to a crowdsourcing task. We illustrate this idea using an automated clustering-based method to identify divergent, but valid, worker interpretations in crowdsourced entity annotations collected over two distinct corpora – Wikipedia articles and Tweets. We demonstrate how the intermediatelevel view provide by crowd-parting analysis provides insight into sources of disagreement not easily gleaned from viewing either individual annotation sets or aggregated results. We discuss several concrete applications for how this approach could be applied directly to impr...
Sanjay Kairam, Jeffrey Heer
Added 01 Apr 2016
Updated 01 Apr 2016
Type Journal
Year 2016
Where CSCW
Authors Sanjay Kairam, Jeffrey Heer
Comments (0)
books