A New Way To Find Reviewers

DOE program managers are routinely called upon to identify peer reviewer candidates for grant and field work proposals.  Each proposal requires a minimum of three reviewers and often more to cover separate aspects of the proposal.  To generate reviewer candidates, program managers draw upon their subject matter expertise and manually scour journal literature.  Although this process is facilitated by the availability of electronic journals, it is labor intensive and represents a major cost.

Information experts at OSTI have demonstrated the capability to identify high quality potential reviewers by analyzing the scientific content underlying its R&D information collections and other integrated sources. This analytical procedure combines sophisticated semantic algorithms with informed judgments. Not only is it efficient, it also raises the prospect of broadening the reviewer base by finding new qualified reviewers.

In non-technical language, the OSTI Reviewer Finder works as follows. First a semantic technique is used to find a core set of papers that are directly related to the proposal in question. This can also be done for groups of proposals or other topic specific needs.

Second, a sophisticated semantic algorithm, developed by OSTI, is used to find all those papers that are closely related to the core papers. In fact these papers are ranked according to the degree of closeness. This means the pool of related papers can be made larger or smaller as needed.

This two step process may be repeated when a proposal involves several distinct topics, methods or other aspects, which often happens. For example, in addition to subject matter experts one might want to have reviewers who are experts in the methods used.

All of the authors of the core and closely related papers are potential reviewers, so they are abstracted and listed. However, some authors must be excluded, for various reasons. For example, it is common practice to exclude those authors who have co-authored a paper with the principle investigator of the proposal to be reviewed. Others may be have retired or be otherwise unavailable. Other restrictions may apply, so the final step is to filter the list and locate the remaining, qualified reviewers. The OSTI list of potential reviewers is submitted for consideration to the program manager who determines the reviewers to be contacted and invited to participate in the review.

While algorithmically sophisticated, the Reviewer Finder procedure is simple to execute, and therefore quite efficient. It is also comprehensive, thus providing the broadest possible pool of qualified reviewers. This breadth in itself is a valuable product of the procedure.

With the Reviewer Finder successfully demonstrated, it is ready for serious deployment by information specialists.  Ultimately, the Reviewer Finder procedure could be made sufficiently user friendly that it could be used directly by the program manager.  A small investment would be needed to develop appropriate software.