Using Amazon's Mechanical Turk as a tool for a global survey: Lessons learned from a large-scale implementation

Background. Amazon Web Services’ Mechanical Turk (mTurk) is a cloud-based, crowdsourcing marketplace that allows global mTurk workers (mTWs) to be paid to complete tasks. The benefits of accessing mTWs for survey research include an efficient mechanism for conducting online surveys, operationalizing inclusion and exclusion criteria, and the ability to pay workers globally. Survey researchers increasingly access mTWs but few have explicitly sought a global sample.

Objective. To share lessons learned from implementing a health survey to a global sample of mTWs.

Methods. Two phases included: 1) pilot survey targeting seven global regions (n=20/region), and 2) large-scale implementation in US, India, and other countries. Workers were paid $0.50 for taking a 15-20 minute (English) survey about attitudes and beliefs towards genetic testing and health. Administrative and descriptive information were collected and analyzed by region/country including: completions by location, demographics, time to complete, and survey satisfaction.

Results. There are four key lessons. 1) MTurk is fast. The US sample (n=505) accrual took <2 days and the Indian sample (n=505) took 11 days, while the response from other countries (n=118) generally exceeded 30 days. 2) Using Amazon country specification reduced “expat effect.” In the pilot, workers self-identifying from countries other than the US or India were frequently not citizens of those countries. To address this “expat effect,” the final survey identified workers based on their Amazon account country. 3) Demographic differences exist in mTWs between countries. For example, US mTWs were significantly more likely female (60.1%) compared to India (30.2%) and other countries (34.2%). 4) mTWs found the survey understandable/acceptable. mTWs report high understandability and acceptability of the survey, which did not vary significantly across countries.

Conclusions. MTurk provides an efficient platform for survey research from diverse US and Indian samples. In other countries, the mTurk mechanism yielded a smaller sample more slowly.