In the past decades, many countries started funding academic institutions based on the evaluation of their scientific performance. Post-publication peer review is often used in this context. Bibliometric indicators have been suggested as an alternative to peer review. A recurrent question is whether peer review and metrics yield similar outcomes. In this paper, we study this question based on a sample of publications submitted to the national Italian research assessment exercise (2011–2014). In particular, we study the agreement between peer review and metrics at the institutional level, and compare this to the internal agreement of peer review. We base our analysis on a hierarchical Bayesian model using cross-validation. We find that the level of agreement is higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is low, but on par with the similarly low internal agreement among two reviewers for certain fields of science. The low agreement between metrics and peer review is no reason for rejecting the use of metrics for some fields in the Italian national research assessment exercise. Our results provide input to the broader discussion of research evaluation, in which other factors also play an important role.

Peer Review

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00352

This content is only available as a PDF.

Author notes

Handling Editor: Vincent Larivière

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode.

Article PDF first page preview

Article PDF first page preview