Appropriate use of design effects and sample weights in complex health survey data: A review of articles published using data from Add Health, MTF, and YRBS

Secondary data analysis of nationally representative health surveys is commonly conducted by public health researchers and can be extremely useful when investigating risk and protective factors associated with health-related outcomes. However, the increased use of such data also requires methodological skills that many researchers were not trained in as graduate students. For example, national health surveys often involve elements of differential sampling, clustering, stratification, and multiple stages of sample selection. If the design effect is not accounted for and data derived from a survey with a complex sample design are analyzed with statistical techniques that assume simple random sampling, the standard errors are often too small and can lead to incorrect inferences being made by the researcher. The purpose of this paper is to assess the prevalence of researchers reporting the use of statistical techniques that take into account the sampling structure of complex health survey data and sample weights in published peer-reviewed articles using data from three commonly utilized large-scale adolescent health surveys - National Longitudinal Study of Adolescent Health, Monitoring the Future, and Youth Risk Behavior Surveillance System. Articles for this study were selected through a systematic review of published empirical research articles, which used data from the three surveys, indexed in ERIC, PsychINFO, PubMed, and Web of Science during the 10 year period of January 1995 to December 2005. A total of 481 articles met the inclusion criteria. Differences in reporting the use of appropriate analytic techniques across the three surveys are discussed.