258860 Evaluation of the Criteria Adopted by AHRQ in the Peer Review of Extramural Research Proposals

Monday, October 29, 2012

Shyam Misra, MD, PhD , Health and Human Services (HHS), Agency for Healthcare Research and Quality, Rockville, MD
Rebecca Trocki, MSHAI , Office of Extramural Research, Education, and Priority Populations, Agency for Healthcare Research and Quality, Rockville, MD
Kishena Wadhwani, PhD, MPH , Department of Health and Human Services, Agency for Healthcare Research and Quality, AHRQ, Rockville, MD
Francis Chesley, MD , Office of Extramural Research, Education, and Priority Populations, Agency for Healthcare Research and Quality, Rockville, MD
Background: Since June 2010, AHRQ is implementing a revised peer review process to evaluate the scientific and technical merit of new research grant applications http://grants.nih.gov/grants/guide/notice-files/NOT-HS-10-002.html. Each proposal is reviewed and scored independently by three reviewers based on five criteria. Based on a predetermined cutoff percentile of the reviewer score (RS), the best proposals are selected for discussion by the review committee (RC) and given a final impact score (IS).

Study Design: The study analyzed 1279 applications of FY2010-2011 of which 804 were discussed. The study included only new applications considered under research mechanisms R01, R03, R18 and R21 and training mechanisms K01, K02, K08, F32 and R36. Factor and regression analysis was conducted with SAS9.0 statistical software.

Principal Findings: As expected the IS scores were strongly correlated (r2 =0.93) with RS and individual Criterion Scores (CS) (r2 ≥ 0.65). High correlation among CS suggested the presence of common factors and Scree plot suggested the presence of two latent factors which adequately represented the CS. Although all criteria independently influenced the RS and IS, the ‘Approach' criterion with everything else held constant had a stronger influence on the IS. The findings suggest the need to further refine the scoring of proposals with unique criteria for a given study area.

Conclusion: The significance of the ‘approach' criterion is emphasized for all applicants to take note. However, review criteria evaluation must be conducted periodically to ensure fairness in the award process and also assess the construct validity and uniqueness of the review criteria.

Learning Areas:
Administer health education strategies, interventions and programs
Administration, management, leadership
Biostatistics, economics
Program planning
Public health administration or related administration
Public health or related organizational policy, standards, or other guidelines

Learning Objectives:
To assess the correlation of each of the five peer reviewed criterion scores and their influence on the overall impact score that largely determines the awarding of a research grant application by the Agency for Healthcare Research and Quality (AHRQ).

Keywords: Health Care Delivery, Government

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: I am a medical epidemiologist by training and experience and been involved in public health and assosiated operational research for over 20 years. I am the co-author of this reasearch and was responsible for data analysis and interpretation. I have no conflicts of interest with regard to this presentation.
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.

Back to: 3189.0: Poster Session II - Contextual Factors and Their Impact on Public Health Agencies