Integrating Automatic Flagging and Manual Inspection to Efficiently Identify Fraudulent Entries in Online Survey Research

The Sexually Explicit Media (SEM) Study aims to explore the relationship between SEM and HIV risk behavior in men who have sex with men (MSM). In 2011, male U.S. residents over the age of 18 who had had sex with a man in the previous five years were recruited for an online survey of SEM use. To help researchers identify invalid entries, Stata was used to automate a protocol for flagging potentially fraudulent responses. Entries were reviewed for duplicate IP, email, and payment addresses as well as last names. Responses to similar items at the beginning and end of the survey (e.g., age, sexual behavior, and zip code) were also compared. For visual inspection, a map was generated using plots of participants' zip codes in order to detect whether an abnormally high frequency of survey responses appeared in one geographic area. With liberal criteria, such as using Soundex to match different spellings, 240 of the 324 participants (74%) were flagged as potentially fraudulent. Using exact matches to flag duplicates, the number of possibly invalid entries fell to 134 (41%), of which 97 (72%) were due to the time taken to complete the survey. After reviewing the flagged entries manually, only three were determined to be invalid: one for having a non-U.S. PayPal address and two for having duplicate IP addresses. Practical guidelines for the combined use of Stata for automated fraud checking and manual examination of survey responses to efficiently improve validity in Internet-based LGBT health surveys will be discussed.