Text mining of National Institutes of Health (NIH) R01 grant critiques: Does the sex and race of the principal investigator make a difference?

Advancing women and racial/ethnic minorities in health science research careers is important for ensuring the future competitiveness of U.S. science and technology and for addressing persistent health disparities. The National Institutes of Health's (NIH) R01 grant mechanism funds the majority of U.S. research programs focused on improving human health and treatment of disease. Women and racial/ethnic minorities have lower success rates than men and Whites, respectively, for R01 grants, yet no study has examined the extent to which applicant sex and race may differentially advantage scientists in NIH's peer review process. Conventional analyses of application scores and funding outcomes reveal little about reviewers’ evaluative judgments. Text mining of reviewers’ narrative critiques can provide a window into reviewers’ cognitive processing, and when used in combination with traditional comparisons, can reveal the extent to which reviewers’ evaluative judgments align with scores and funding outcomes across applicants. We conducted a longitudinal analysis of unfunded and funded grant application critiques from a nationally representative sample of 27,000 NIH R01 investigators from 2010 to 2014. We applied text mining algorithms to analyze reviewers' evaluative commentary in critiques, analyzed the relationship between text mining outcomes and application scores and funding outcomes, and used multi-level regression models to compare results by applicant sex and race. Results from this study suggest that applicant sex and race may, however inadvertently, influence the evaluation of NIH grant applications, and point to areas for policy intervention.