Analysis of impact scores obtained as a function of review cycles (From June 2010 To June 2012 Review Cycles)

Research Objective

To examine the trend of scientific quality of research grant applications submitted to AHRQ with time, we assessed the overall impact scores provided by each of the 5 standing study sections from June 2010 to June 2012.

Study Design

The Scientific Peer Review Committees at AHRQ use 9 points overall impact score system to evaluate the scientific merit of research grant applications submitted to AHRQ for funding opportunity. The data used in this analysis was from the National Institutes of Health (NIH) Query View & Report Database (QVR System). Microsoft Excel and SPSS were used for analysis.

Population Studied

Research grant applications submitted to AHRQ were reviewed by one of the 5 standing study section committees or by the Special Emphasis Panel (SEP). The 5 study section review committees at AHRQ are: Healthcare Systems & Value Research, Healthcare Information Technology Research, Healthcare Effectiveness and Outcomes Research, Healthcare Safety and Quality Improvement Research, Healthcare Research Training. AHRQ also has convened many SEPs to evaluate grant applications responding to the RFAs.

Principal Findings

The means impact scores tend to increase with time. For the HCRT, HQER or HSR study section, the means impact scores decrease to about 31-35 with the exception of February 2011 review cycle meeting. The means of overall impact scores gradually increased to higher values at the subsequent review cycles to about 38-53 for all 5 study sections. The means for the June 2010 cycle ranged from 36.26–40.76 and rose upward for the June 2012 cycle at 38.48-53.46. Although the mean values of impact scores implicated an upward trend, there was no statistical significance among these mean values (P-value > .05).

Conclusions

Our analysis of impact scores indicate that the AHRQ peer review process is consistent in terms of quality of the scientific review among study sections.