Analysis of overall impact scoring trends within AHRQ peer review study sections
Methods: Peer review scoring data was obtained, using NIH eRA Commons, from applications received in FY 2010-2014 from five AHRQ study sections. Subgroup analysis was conducted on applications considered under general research mechanisms R01, R03, and R18 in FY 2011-2014. Means (SD) and medians (range) were calculated for each AHRQ review meeting and fiscal year. Score trends were assessed for each study section, as well as comparisons of score trends between study sections. Percentile standardized scores were used to make adequate comparisons between study sections. Descriptive statistics and linear regression were conducted with SAS 9.3 statistical software.
Results: The study analyzed 3,370 applications, of which 1,752 (52%) were discussed and received a score. Over time, slight trends towards lower (better) median scores were found in four out of five AHRQ study sections. The r2 values ranged 0.08-0.23. The subgroup analysis included 1,086 applications (57% discussed) considered under general research mechanisms R01, R03, and R18. Triaging of applications was high among R03 (54.8%) and R18 (57.5%) applications across all study sections, compared to R01 applications (11.4%). In comparing study sections, R01 mean scores ranged from 34.2±13.3 to 42.8±12.5; R03 from 36.4±15.2 to 41.9±12.8; and R18 from 38.0±13.6 to 43.1±18.0. Percentile scores did not differ by study section, adjusting for FY, for R01 (F=0.74,p=0.53), R03 (F=0.31,p=0.82), and R18 (F=0.22,p=0.88).
Conclusions: AHRQ study sections perform consistently over time, reflecting both the assessments of the reviewers and the quality of the applications. These results show that careful selection of subject-matter experts, and consistency and uniformity in conducting the evaluation of research grant applications, are the best practices for peer review.
Learning Areas:Conduct evaluation related to programs, research, and other areas of practice
Other professions or practice related to public health
Public health or related organizational policy, standards, or other guidelines
Public health or related research
Identify characteristics of the AHRQ peer review process. Describe trends in scoring of research grant applications submitted for funding (FY 2010-2014). Compare scoring trends between five AHRQ study sections.
Keyword(s): Funding/Financing, Evaluation
Qualified on the content I am responsible for because: As a Director of the Division of Scientific Review at AHRQ, I directed and supervised this project.
Any relevant financial relationships? No
I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.