Sciweavers

Share
ICASSP
2011
IEEE

CROWDMOS: An approach for crowdsourcing mean opinion score studies

10 years 2 months ago
CROWDMOS: An approach for crowdsourcing mean opinion score studies
MOS (mean opinion score) subjective quality studies are used to evaluate many signal processing methods. Since laboratory quality studies are time consuming and expensive, researchers often run small studies with less statistical significance or use objective measures which only approximate human perception. We propose a cost-effective and convenient measure called crowdMOS, obtained by having internet users participate in a MOS-like listening study. Workers listen and rate sentences at their leisure, using their own hardware, in an environment of their choice. Since these individuals cannot be supervised, we propose methods for detecting and discarding inaccurate scores. To automate crowdMOS testing, we offer a set of freely distributable, open-source tools for Amazon Mechanical Turk, a platform designed to facilitate crowdsourcing. These tools implement the MOS testing methodology described in this paper, providing researchers with a user-friendly means of performing subjective qua...
Flavio Ribeiro, Dinei A. F. Florêncio, Cha Z
Added 21 Aug 2011
Updated 21 Aug 2011
Type Journal
Year 2011
Where ICASSP
Authors Flavio Ribeiro, Dinei A. F. Florêncio, Cha Zhang, Michael L. Seltzer
Comments (0)
books