Online Program

Analysis of impact scores obtained as a function of review cycles (From June 2010 To June 2012 Review Cycles)

Monday, November 4, 2013

Debra Tan, MPH, School of Public Health, University of North Texas Health Science Center, Fort Worth, TX
Kishena Wadhwani, PhD, MPH, OEREP/Division of Scientific Review, Agency for Healthcare Research and Quality, Rockville, MD
Rebecca Trocki, MSHAI, Office of Extramural Research, Education, and Priority Populations, Agency for Healthcare Research and Quality, Rockville, MD
Shyam Misra, MD, PhD, Health and Human Services (HHS), Agency for Healthcare Research and Quality, Rockville, MD
Research Objective

To examine the trend of scientific quality of research grant applications submitted to AHRQ with time, we assessed the overall impact scores provided by each of the 5 standing study sections from June 2010 to June 2012.

Study Design

The Scientific Peer Review Committees at AHRQ use 9 points overall impact score system to evaluate the scientific merit of research grant applications submitted to AHRQ for funding opportunity. The data used in this analysis was from the National Institutes of Health (NIH) Query View & Report Database (QVR System). Microsoft Excel and SPSS were used for analysis.

Population Studied

Research grant applications submitted to AHRQ were reviewed by one of the 5 standing study section committees or by the Special Emphasis Panel (SEP). The 5 study section review committees at AHRQ are: Healthcare Systems & Value Research, Healthcare Information Technology Research, Healthcare Effectiveness and Outcomes Research, Healthcare Safety and Quality Improvement Research, Healthcare Research Training. AHRQ also has convened many SEPs to evaluate grant applications responding to the RFAs.

Principal Findings

The means impact scores tend to increase with time. For the HCRT, HQER or HSR study section, the means impact scores decrease to about 31-35 with the exception of February 2011 review cycle meeting. The means of overall impact scores gradually increased to higher values at the subsequent review cycles to about 38-53 for all 5 study sections. The means for the June 2010 cycle ranged from 36.26–40.76 and rose upward for the June 2012 cycle at 38.48-53.46. Although the mean values of impact scores implicated an upward trend, there was no statistical significance among these mean values (P-value > .05).


Our analysis of impact scores indicate that the AHRQ peer review process is consistent in terms of quality of the scientific review among study sections.

Learning Areas:

Administration, management, leadership
Conduct evaluation related to programs, research, and other areas of practice
Public health or related research

Learning Objectives:
Discuss future recommendations in the extramural health services research community regarding the scientific peer process for grant applications submitted to AHRQ for funding opportunity.

Keyword(s): Public Health Research, Public Health Agency Roles

Presenting author's disclosure statement:

Not Answered