Positive predictive value of AHRQ Patient Safety Indicators in a national sample of hospitals

BACKGROUND. The AHRQ Patient Safety Indicators (PSI) have become a widely used tool for identifying potential safety-related events in acute care hospitals, using ICD-9-CM coded administrative data. Little is known about the criterion validity of these indicators across multiple hospitals. The PSI Validation Pilot Project was designed to gather evidence on criterion validity based on medical record review, to improve guidance about how to interpret PSI rates, to evaluate potential specification changes, and to pilot a system for ongoing validation studies. METHODS. In response to a national call for volunteers, 47 hospitals from 29 states agreed to abstract up to 30 records from 2005-06 using standard tools and guidelines. PSI rates at these 47 hospitals were similar to mean PSI rates in the Nationwide Inpatient Sample. Ongoing support was provided through training webinars, written documents, electronic discussions, and feedback. Phase 1 focused on five PSIs specified below; phase 2 will address five additional PSIs. Positive predictive value (PPV) was defined as the crude percentage of PSI-flagged cases that were confirmed by detailed record review. False positive cases were classified as either miscoded diagnoses or correctly coded diagnoses that predated admission. RESULTS. For "acccidental puncture and laceration" (N=250), PPV was 90% (9% miscoded, 1% predated admission) and about 69% of confirmed events required a reparative procedure. For "iatrogenic pneumothorax" (N=175), PPV was 88% (4% miscoded, 8% predated admission) and about 66% of confirmed events were treated with tube thoracostomy. For "postoperative DVT/PE" (N=123), PPV was 84% (7% miscoded, 10% predated admission), but about 22% of true positives involved arm/neck veins and 4% involved superficial leg veins. For "postoperative sepsis" (N=98), PPV was 68% (29% miscoded, 3% predated admission). For "selected infections due to medical care" (N=189), PPV was 63% (19% miscoded, 18% present at admission) and about 78% of confirmed events were catheter-related. These estimates do not include additional cases that were identified by abstraction as meeting AHRQ exclusion criteria. DISCUSSION. The PPV of five PSIs in a nonrandom but representative sample of US hospitals varied from 49% to 90%, depending on the specific PSI and on how denominator exclusions are handled. Incorporating "present at admission" data would substantially improve most of these PPVs. The observed variability in PPV should be considered in selecting indicators for public reporting and pay-for-performance. We successfully pilot-tested a mechanism for ongoing validation, which can be applied to estimate sensitivity in future work.