Using Amazon's Mechanical Turk as a tool for a global survey: Lessons learned from a large-scale implementation
Objective. To share lessons learned from implementing a health survey to a global sample of mTWs.
Methods. Two phases included: 1) pilot survey targeting seven global regions (n=20/region), and 2) large-scale implementation in US, India, and other countries. Workers were paid $0.50 for taking a 15-20 minute (English) survey about attitudes and beliefs towards genetic testing and health. Administrative and descriptive information were collected and analyzed by region/country including: completions by location, demographics, time to complete, and survey satisfaction.
Results. There are four key lessons. 1) MTurk is fast. The US sample (n=505) accrual took <2 days and the Indian sample (n=505) took 11 days, while the response from other countries (n=118) generally exceeded 30 days. 2) Using Amazon country specification reduced “expat effect.” In the pilot, workers self-identifying from countries other than the US or India were frequently not citizens of those countries. To address this “expat effect,” the final survey identified workers based on their Amazon account country. 3) Demographic differences exist in mTWs between countries. For example, US mTWs were significantly more likely female (60.1%) compared to India (30.2%) and other countries (34.2%). 4) mTWs found the survey understandable/acceptable. mTWs report high understandability and acceptability of the survey, which did not vary significantly across countries.
Conclusions. MTurk provides an efficient platform for survey research from diverse US and Indian samples. In other countries, the mTurk mechanism yielded a smaller sample more slowly.
Explain how mTurk works as a research tool. Differentiate the uses of mTurk depending on what type of sample one is seeking. Describe the lessons learned from this study.
Keyword(s): Methodology, Epidemiology
Qualified on the content I am responsible for because: I implement, analyze, and disseminate qualitative and quantitative research projects, especially related to large quantitative and qualitative datasets.
Any relevant financial relationships? No
I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.