Are your participants gaming the system?: screening mechanical turk workers

In this paper we discuss a screening process used in conjunction with a survey administered via Amazon.com’s Mechanical Turk. We sought an easily implementable method to disqualify those people who participate but don’t take the study tasks seriously. By using two previously pilot tested screening questions, we identified 764 of 1,962 people who did not answer conscientiously. Young men seem to be most likely to fail the qualification task. Those that are professionals, students, and non-workers seem to be more likely to take the task seriously than financial workers, hourly workers, and other workers. Men over 30 and women were more likely to answer seriously. Author Keywords screening, Mechanical Turk, survey, crowdsourcing ACM Classification Keywords

• Type: Conference
• Year: 2010
• Where: CHI
• Authors: Julie S. Downs, Mandy B. Holbrook, Steve Sheng, Lorrie Faith Cranor
• Keywords: Amazon.com’s Mechanical Turk, CHI 2010, Human Computer Interaction, Mechanical Turk, Screening Process