Sciweavers

Share
AAAI
2012

Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests

8 years 6 months ago
Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests
We examine designs for crowdsourcing contests, where participants compete for rewards given to superior solutions of a task. We theoretically analyze tradeoffs between the expectation and variance of the principal’s utility (i.e. the best solution’s quality), and empirically test our theoretical predictions using a controlled experiment on Amazon Mechanical Turk. Our evaluation method is also crowdsourcing based and relies on the peer prediction mechanism. Our theoretical analysis shows an expectation-variance tradeoff of the principal’s utility in such contests through a Pareto efficient frontier. In particular, we show that the simple contest with 2 authors and the 2-pair contest have good theoretical properties. In contrast, our empirical results show that the 2-pair contest is the superior design among all designs tested, achieving the highest expectation and lowest variance of the principal’s utility.
Xi Alice Gao, Yoram Bachrach, Peter Key, Thore Gra
Added 29 Sep 2012
Updated 29 Sep 2012
Type Journal
Year 2012
Where AAAI
Authors Xi Alice Gao, Yoram Bachrach, Peter Key, Thore Graepel
Comments (0)
books