170106 Comparison of Combining Methods for Multiple Quality Indicators on the VBP Measures

Monday, October 27, 2008: 2:30 PM

Jacob Jen-Hao Cheng, PhD, MS , Research & Development, Quality Indicator Project / Maryland Hospital Association, Elkridge, MD
Yong Chen , Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD
Nikolas Matthes, MD, PhD, MPH , Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD
Samuel Ogunbo, PhD , Research & Development, Quality Indicator Project / Maryland Hospital Association, Elkridge, MD
Carlos Alzola, MS , Data Insights, Vienna, VA
Currently different composite scores are used to summarize the overall performance from multiple indicators for internal and public reporting of National Hospital Quality measures. As CMS proposes Value-Based Purchasing (VBP) as a potential model for determining rewards and incentives for pay-for-performance program based on single summary score for each hospital, we compare the VBP composite score with four others composite scores: overall pass rate (OPR), average patient rate (APR), average indicator rate (AIR) and perfect patient rate (PPR).

APR and PPR belong to the patient adjusted method that calculates patient pass rates first and AIR and VBP to the indicator adjusted method that calculates indicator (measure) pass rates first. With PPR as a derived method from APR requiring a perfect criterion on the patient rates, VBP can be viewed as a derived method from AIR scaling the indicator rates into “attainment” or “improvement” scores.

This study includes 387,624 patients from 453 hospitals participating National Hospital Quality Measures through the Quality Indicator Project for 16 process-of-care VBP measures for 2006. The comparisons were carried over in three ways: distribution analysis, pairwise agreement analysis, and linear regression on hospital characteristics. Five statistics were computed to measure the ranking agreement between any pair of two methods: rank correlation coefficient, number of >25% changes, average of rank changes, percentage of inconsistent pairs, kappa coefficient and misclassification rate. The results of agreement analysis can reflect the structure of the data.

PPR and VBP (derived methods) are distributed with lower centrals and wider variances than other methods, suggesting more opportunity for quality improvement. On the other hand, OPR, APR and AIR are distributed closed with higher centrals and smaller variances. However, similar distributions do not necessarily result in similar hospital rankings. High ranking agreement between OPR and APR suggests that the non-eligible cases are distributed quite evenly along patient arrays and that the variance of patient pass rates is small. Moderate agreement between OPR and AIR suggests variant indicator pass rates along the indicator arrays. High agreement between APR and PPR implies that the perfect patients are distributed quite randomly between hospitals. Disparity between AIR and VBP results from hospitals with credits gained by improvements. Lower predictive power of the VBP model reflects the loosened associations between the hospital characteristics and the rewarding criteria, encouraging more hospitals to improve quality regardless of their characteristics.

Learning Objectives:
Discuss both data aggregating methods and data structure that greatly affect distributions and rankings of composite scores. List the comparison methods needed to understand the different composite scores and the data structure. Describe the properties of different composite scores and their implications for public reporting and financial rewards.

Keywords: Health Care Quality, Quality Improvement

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: This study had no financial relationship with any organization.
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.