Using calibration weighting in Web-based health surveys to combine probability and volunteer samples for studying small areas or low prevalence phenomena

Health surveys depend on probability-based methods to deliver representative samples whether the data collection mode is in-person, telephone, mail or online. Samples drawn from Internet panels recruited through probability-based methods, after post-stratification weighting, can reliably generalize to the population of interest. Due to finite panel size, however, there are instances of too few panel members to meet sample size requirements, particularly for rare health conditions, certain minority groups, high-risk behaviors or small geographic areas. In such cases, a carefully designed supplemental sample from a non-probability volunteer Internet panel source may be added. This paper will first show that when both samples are profiled with five Likert-type questions on early adopter (EA) attitudes, non-probability samples tend to have proportionally more EA characteristics compared to probability samples. This finding is consistent over different demographic groups. Secondly, taking advantage of these EA differences, this paper describes a statistical weighting technique using a SAS raking procedure for calibrating the combined samples with the probability-based sample using EA characteristics. Results from a state-wide tobacco survey and an emergency preparedness survey will be demonstrated. As an example, data from one probability sample (n=611) and one volunteer sample (n=750), showed that a reduction in the average mean squared error from 3.8 to 1.8 can be achieved with calibration. The average estimated bias was also reduced from 2.056 to 0.064. This calibration approach can be a viable methodology for combining probability and non-probability Web panel samples for health surveys, especially where rare elements challenge affordable samples.