Does risk adjustment of the Quality Measures included in the Nursing Home Compare Report card matter?

The Centers for Medicare & Medicaid Services (CMS) publishes quality measures (QMs) for nursing homes. We examined several of the measures to determine if a more comprehensive risk adjustment changes conclusions about the quality of individual nursing homes. The analysis included measures for pressure sores, physical restraints, pain for long and for short stay patients. We used the Minimum Data Set (MDS) to obtain information about all nursing home residents in the period 2001-2005. We estimated random effects logistic models. The dependent variables were defined based on the CMS definition of its QMs. The independent variables included individual level risk factors that were identified by a geriatrician as those likely to influence the health outcome. These models were used to calculate facility-level risk-adjusted quality measures that were then compared to the CMS QMs. To measure the agreement between the two types of quality measures we calculated Kappa statistics, nationally and stratified by state. Kappa values for pressure sores averaged 0.89 for low quality outliers and 0.59 for high quality outliers (defined as those in the 5th percentile of the distribution), pain for long stay patients averaged 0.84 and 0.79, and for short stay patients 0.80 and 0.61, respectively. Because of the skewed distribution of physical restraints we calculated Kappa for terciles and it averaged 0.86. These Kappa values suggest a moderate to very good agreement between the CMS QMs and the risk adjusted QMs. However, these findings also suggest that the CMS measures could be improved by additional risk adjustment.