Title: Grant Peer Review: Improving Inter-Rater Reliability with Training

Abstract

In this study, we developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and the amount of time reading the review criteria compared to the no video condition. The increase in reliability for experienced reviewers is notable because it is commonly assumed that reviewers—especially those with experience—have good understanding of the grant review rating scale. Our findings suggest that both experienced and novice reviewers who had not received the type of training developed in this study may not have appropriate understanding of the definitions and meaning for each value of the rating scale and that experienced reviewers may overestimate their knowledge of the rating scale. Lastly, the results underscore the benefits ofmore » and need for specialized peer reviewer training.« less

Authors:

Sattler, David N. ^[1]; McKnight, Patrick E. ^[2]; Naney, Linda ^[3]; Mathis, Randy ^[3]

---

+ Show Author Affiliations

1. Western Washington Univ. Bellingham, WA (United States)
2. George Mason Univ., Fairfax, VA (United States)
3. Oak Ridge Associated Universities (ORAU), Oak Ridge, TN (United States)

Publication Date:

Mon Jun 15 00:00:00 EDT 2015

Research Org.:

Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:

USDOE Office of Science (SC)

OSTI Identifier:

1260644

Grant/Contract Number:  

AC05-06OR23100

Resource Type:

Accepted Manuscript

Journal Name:

PLoS ONE

Additional Journal Information:

Journal Volume: 10; Journal Issue: 6; Journal ID: ISSN 1932-6203

Publisher:

Public Library of Science

Country of Publication:

United States

Language:

English

Subject:

99 GENERAL AND MISCELLANEOUS