Findings from the HCAHPS mode experiment

Research Objective: To determine the effects of mode of survey administration on patient responses to the HCAHPS (CAHPS Hospital) Survey. Study Design: Within each of 45 randomly sampled hospitals, 27,229 patients were randomized to four modes of survey administration: Mail Only, Telephone Only, Mixed Mode (mail with telephone follow-up), or Active IVR (interactive voice response, in which patients respond via telephone keypads). These patients completed the CAHPS (Consumer Assessment of Healthcare Providerslans and Systems) Hospital Survey, in which recently discharged patients evaluate aspects of their hospital care. Population Studied: English- or Spanish-speaking adult patients with non-psychiatric primary diagnoses discharged alive after at least one overnight stay in a general acute care US hospital with at least 1200 annual discharges. Principal Findings: Linear regression was used to model each HCAHPS outcome (2 overall ratings and 7 composites) from fixed effects for mode, hospital identifiers, and patient characteristics. Substantial and statistically significant (p<0.05) mode effects were found for both overall ratings and 4 of 7 composites (Cleanliness & Quiet, Responsiveness of Hospital Staff , Pain Management, and Discharge Information); mode effects were smaller for the 3 composites involving communication (with doctors, nurses, and about medications). Patients provided more positive evaluations in the Telephone Only and Active IVR modes than in the Mail Only and Mixed modes. Differences between Telephone Only and Active IVR responses and between Mail Only and Mixed Mode responses were small. The Telephone Only and Active IVR scores for the 6 most affected outcomes noted above were at least 0.4-0.5 hospital-level standard deviations higher and sometimes as much as 1 standard deviation higher than scores from Mail Only and Mixed modes. These mode effects varied little by hospital. Response rates varied strongly by randomized mode (p<0.0001), ranging from 41.2% for Mixed Mode to 20.7% for Active IVR; these patterns were consistent across hospitals. Conclusions: Because a hospital's choice of vendor or survey mode may be confounded with factors related to underlying quality, an external mode experiment is necessary to estimate mode effects for subsequent fieldings of the survey. We conclude that adjustments for mode effects are necessary to make the reported HCAHPS scores comparable; in the absence of such adjustments a hospital that would have ranked at the 50th percentile in the Mail Only mode would be ranked at the 66th to 84th percentile in the Telephone Only mode for a majority of outcomes.