Conducting multilevel model public health research in complex survey data with design weights

Background: Large scale survey data offer premier opportunities to conduct multilevel modeling studies. However, little summarized guidance exists with regard to fitting multilevel models in complex survey data with design weights. This has impeded multilevel models' application in health research. Simulation work suggests that analysts should scale design weights using two methods and fit the multilevel models using unweighted and scaled-weighted data. My research summarizes this advice, examines its performance across a variety of multilevel models using real data, and develops best practice recommendations.

Methods: Using data from the 2005-2006 National Survey of Children with Special Health Care Needs (NS-CSHCN: n = 40,723) that used a complex sampling design to collect data from children clustered within states, I examine the performance of weight scaling methods across outcome (categorical vs. continuous) and model type (level-1 only: individual only; level-2: contextual; or combined: individual and contextual).

Results: Scaled weighted estimates and standard errors differed slightly from unweighted analyses, agreeing more with each other than with unweighted analyses. Consistent with previous simulations, scaling weights consistently led to slightly larger, less biased standard errors. However, in these data, observed differences were minimal and did not lead to different inferential conclusions.

Conclusions: If including weights, analysts should scale the weights using the two described scaling methods and use software that properly includes the scaled weights in the estimation. Researchers should not include raw, unscaled weights in any analyses. Otherwise, they may achieve erroneous results.