Sciweavers

Share
SIGIR
2012
ACM

Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization

9 years 2 months ago
Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization
In crowdsourced relevance judging, each crowd worker typically judges only a small number of examples, yielding a sparse and imbalanced set of judgments in which relatively few workers influence output consensus labels, particularly with simple consensus methods like majority voting. We show how probabilistic matrix factorization, a standard approach in collaborative filtering, can be used to infer missing worker judgments such that all workers influence output labels. Given complete worker judgments inferred by PMF, we evaluate impact in unsupervised and supervised scenarios. In the supervised case, we consider both weighted voting and worker selection strategies based on worker accuracy. Experiments on crowd judgments from the 2010 TREC Relevance Feedback Track show promise of the PMF approach merits further investigation and analysis. Categories and Subject Descriptors I.2.6 [Artificial Intelligence]: Learning General Terms Algorithms, Design, Experimentation, Performance Keywo...
Hyun Joon Jung, Matthew Lease
Added 28 Sep 2012
Updated 28 Sep 2012
Type Journal
Year 2012
Where SIGIR
Authors Hyun Joon Jung, Matthew Lease
Comments (0)
books