Use of technology to improve inter-rater reliability in medical record review: The added-value of mobile devices

Medical record review is a widely used methodology for clinical research, for health care program evaluation and for collecting data to assess quality of care. Measuring inter-rater reliability among independent chart abstractors (raters) is an essential component in assuring data collection has high correlation of agreement among abstractors across responses. In 2011/12, nine nurse abstractors from the University of California, San Francisco's Bixby Center for Global Reproductive Health collected data from nearly 5,000 medical records in 216 provider sites in 18 counties, for the California Department of Public Health Office of Family Planning's Family PACT (Planning, Access, Care, and Treatment) Program. The abstraction tool was developed in Microsoft Excel with software written for a wireless tablet. The tool was designed to minimize response variation by using yes/no and multiple choice responses as well as date and number dials. We identified a subset of key responses among the more than 150 data points to compare with responses of each of the abstractors. Ten charts at three different provider sites were re-abstracted for each of the abstractors by the quality assurance nurse, considered to be the true score. In this presentation, we will discuss findings suggesting that the combination of an intensive abstractor training with baseline homework, the ease of use of the wireless tablet, and an abstraction tool that supports standardized responses serves to minimize the effect of abstractor variability on data quality results. These strategies are cost efficient, maximize available resources, and are reproducible on a large scale.