Conducting propensity score analyses in complex survey data with design weights: Recommendations

Introduction: Investigators frequently wish to evaluate the effectiveness of health disparity policies. However, one cannot randomly assign individuals to policy conditions (policy exposure vs. no exposure). This limits causal conclusions, as confounding variables influence whether individuals are exposed to a policy. For example, state policies differ. Simultaneously, sociodemographic variables predict individuals' state of residence. As a result, the propensity to benefit from a policy correlates with characteristics of individuals exposed to the policy. Propensity scores offer a method for handing this bias.

Surveys often offer excellent data for policy research. However, survey data frequently include design weights. Little guidance exists on on how to include weights in propensity score methods. In this presentation, I address this. I offer practical advice for researchers seeking to incorporate propensity score methods in the presence of design weights.

Methods: Using data from the 2005-2006 National Survey of Children with Special Health Care Needs (n = 40,723), a complex survey with design weights, I compare the performance of unweighted and weighted propensity score methods across four propensity score matching methods: nearest-neighbor, radius, kernel, and stratification matching.

Results: Failure to include weights in the propensity score's development resulted in appreciably different propensity scores compared scores developed including weights. Likewise, across methods, failure to include weights in propensity score analyses resulted in biased results.

Discussion: When conducting propensity scores in data with design weights, analysts should include the weights in the propensity score development and analyses. Researchers should not ignore weights, as it results in biased estimates.