Analysis of overall impact scoring trends within AHRQ peer review study sections

Objective: Determine whether trends exist in the scoring of research grant applications submitted for funding and assess potential differences in scoring trends between study sections at the Agency for Healthcare Research and Quality (AHRQ).

Methods: Peer review scoring data was obtained, using NIH eRA Commons, from applications received in FY 2010-2014 from five AHRQ study sections. Subgroup analysis was conducted on applications considered under general research mechanisms R01, R03, and R18 in FY 2011-2014. Means (SD) and medians (range) were calculated for each AHRQ review meeting and fiscal year. Score trends were assessed for each study section, as well as comparisons of score trends between study sections. Percentile standardized scores were used to make adequate comparisons between study sections. Descriptive statistics and linear regression were conducted with SAS 9.3 statistical software.

Results: The study analyzed 3,370 applications, of which 1,752 (52%) were discussed and received a score. Over time, slight trends towards lower (better) median scores were found in four out of five AHRQ study sections. The r² values ranged 0.08-0.23. The subgroup analysis included 1,086 applications (57% discussed) considered under general research mechanisms R01, R03, and R18. Triaging of applications was high among R03 (54.8%) and R18 (57.5%) applications across all study sections, compared to R01 applications (11.4%). In comparing study sections, R01 mean scores ranged from 34.2±13.3 to 42.8±12.5; R03 from 36.4±15.2 to 41.9±12.8; and R18 from 38.0±13.6 to 43.1±18.0. Percentile scores did not differ by study section, adjusting for FY, for R01 (F=0.74,p=0.53), R03 (F=0.31,p=0.82), and R18 (F=0.22,p=0.88).

Conclusions: AHRQ study sections perform consistently over time, reflecting both the assessments of the reviewers and the quality of the applications. These results show that careful selection of subject-matter experts, and consistency and uniformity in conducting the evaluation of research grant applications, are the best practices for peer review.