Using an experimental pretest to examine the impact of question wording changes on a national survey measuring college student health

Background

Health surveys are often revised in response to respondent feedback and changing priorities, yet the impact of such changes on survey reliability and validity is rarely evaluated. The American College Health Association-National College Health Assessment (ACHA-NCHA) is a large-scale survey measuring college student risk behaviors that underwent an extensive revisions process from 2005-2007. Common changes to survey questions included modified stem wording, response options, and instructions.

Methods

This study is a preliminary analysis of results from an experimental field pretest comparing patterns of student responses on survey items from the original to the revised version. Seven universities self-selected to participate, and students from each institution were randomized to complete either the revised (final n=6216) or original (final n=6120) ACHA-NCHA. Results were compared using chi-square tests for differences in proportions in the matched samples.

Results

Even minor changes in questions produced statistically significant differences in student response patterns (p<0.001). Effect sizes ranged from small to moderate. Survey questions between versions are currently being evaluated for evidence of reliability and criterion- and construct-related validity. It is expected that data collected from the revised survey will demonstrate evidence of higher levels of reliability and validity than from the original.

Implications

Survey data are used for numerous purposes in public health and it is crucial that their reliability and validity are maximized. This study demonstrates the importance of evaluating how minor question modifications can produce significant differences in responses and means through which researchers can subsequently evaluate which are more reliable and valid.