Online Program

Using Amazon's Mechanical Turk as a tool for a global survey: Lessons learned from a large-scale implementation

Wednesday, November 4, 2015 : 10:30 a.m. - 10:50 a.m.

Margaret Demment, PhD, Clinical and Translational Science Institute, University of Rochester School of Medicine, Rochester, NY
Susan Groth, PhD, RN, WHNP-BC, School of Nursing, University of Rochester, Rochester, NY
I. Diana Fernandez, MD, MPH, PhD, Department of Community and Preventive Medicine, University of Rochester, Rochester, NY
Ann M. Dozier, PhD, RN, Public Health Sciences/Social and Behavioral Sciences, University of Rochester, Rochester, NY
Jack Chang, MS, University of Rochester, Rochester, NY
Timothy Dye, PhD, Obstetrics and Gynecology, Pediatrics, Public Health Sciences, and Medical Informatics, University of Rochester, Rochester, NY
Background. Amazon Web Services’ Mechanical Turk (mTurk) is a cloud-based, crowdsourcing marketplace that allows global mTurk workers (mTWs) to be paid to complete tasks. The benefits of accessing mTWs for survey research include an efficient mechanism for conducting online surveys, operationalizing inclusion and exclusion criteria, and the ability to pay workers globally. Survey researchers increasingly access mTWs but few have explicitly sought a global sample.

Objective. To share lessons learned from implementing a health survey to a global sample of mTWs.

Methods. Two phases included: 1) pilot survey targeting seven global regions (n=20/region), and 2) large-scale implementation in US, India, and other countries. Workers were paid $0.50 for taking a 15-20 minute (English) survey about attitudes and beliefs towards genetic testing and health. Administrative and descriptive information were collected and analyzed by region/country including: completions by location, demographics, time to complete, and survey satisfaction.

Results. There are four key lessons. 1) MTurk is fast. The US sample (n=505) accrual took <2 days and the Indian sample (n=505) took 11 days, while the response from other countries (n=118) generally exceeded 30 days. 2) Using Amazon country specification reduced “expat effect.” In the pilot, workers self-identifying from countries other than the US or India were frequently not citizens of those countries. To address this “expat effect,” the final survey identified workers based on their Amazon account country. 3) Demographic differences exist in mTWs between countries. For example, US mTWs were significantly more likely female (60.1%) compared to India (30.2%) and other countries (34.2%). 4) mTWs found the survey understandable/acceptable. mTWs report high understandability and acceptability of the survey, which did not vary significantly across countries.

Conclusions. MTurk provides an efficient platform for survey research from diverse US and Indian samples. In other countries, the mTurk mechanism yielded a smaller sample more slowly.   

Learning Areas:


Learning Objectives:
Explain how mTurk works as a research tool. Differentiate the uses of mTurk depending on what type of sample one is seeking. Describe the lessons learned from this study.

Keyword(s): Methodology, Epidemiology

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: I implement, analyze, and disseminate qualitative and quantitative research projects, especially related to large quantitative and qualitative datasets.
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.