Informed peer review for publication assessments: Are improved impact measures worth the hassle?

Evaluative scientometrics is mainly aimed at measuring and comparing the research performance of entities. In general, a research entity is said to perform better than another if, all production factors being equal, its total output has higher impact. The question then is how to measure the impact of output. Citation-based indicators are more apt to assess scholarly impact than social impact, although it is reasonable to expect that a certain correlation between scholarly and social impact exists (Abramo, 2018).

As far as scholarly impact is concerned, three approaches are available to assess the impact of publications: human judgment (peer review); the use of citation-based indicators (bibliometrics); or drawing on both, whereby bibliometrics informs peer review judgment (informed peer review).

The axiom underlying citation-based indicators is that when a publication is cited, it has contributed to (has had an impact on) the new knowledge encoded in the citing publications—normative theory (Bornmann & Daniel, 2008; Kaplan, 1965; Merton, 1973). There are strong distinctions and objections to the above axiom argued by the social constructivism school, holding that that citing to give credit is the exception, while persuasion is the major motivation for citing (Bloor, 1976; Brooks, 1985, 1986; Gilbert, 1977; Latour, 1987; MacRoberts & MacRoberts, 1984, 1987, 1988, 1989a, 1989b, 1996, 2018; Mulkay, 1976; Teplitskiy, Dueder, et al., 2019).

Although scientometricians, as a shorthand, say that they “measure” scholarly impact, what they actually do is “predict” impact. The reason is that to serve its purpose, any research assessment aimed at informing policy and management decisions cannot wait for the publications life cycle to be completed (i.e., the publications stop being cited), which may take decades (Song, Situ, et al., 2018; Teixeira, Vieira, & Abreu, 2017; van Raan, 2004).

As a consequence, scientometricians count early citations, not overall citations. The question then is how long should the citation time window be in order for the early citations to be considered an accurate and robust proxy of overall scholarly impact. The longer the citation time window, the more accurate the prediction. In the end, the answer is subjective, because of the embedded trade-off: The appropriate choice of citation time window is a compromise between the two objectives of accuracy and timeliness in measurement, and the relative solutions differ from one discipline to another. The topic has been extensively examined in the literature (Abramo, Cicero, & D’Angelo, 2011; Adams, 2005; Glänzel, Schlemmer, & Thijs, 2003; Nederhof, Van Leeuwen, & Clancy, 2012; Onodera, 2016; Rousseau, 1988; Stringer, Sales-Pardo, & Amaral, 2008; Wang, 2013).

Most studies in evaluative scientometrics focus on providing new creative solutions to the problem of how to best support the measurement of research performance. An extraordinary number of performance indicators continue to be proposed. It suffices to say that at the recent 17^th International Society of Scientometrics and Informetrics Conference (ISSI 2019), a special plenary session and five parallel sessions, including 25 contributions altogether (leaving aside poster presentations), were devoted to “novel bibliometric indicators.”

Far fewer studies have tackled the problem of how to improve the impact prediction power of early citations, given the inevitable citation short time windows. A number of scholars have proposed combining citation counts with other independent variables related to the publication. Whatever the combination, there is a common awareness that it cannot be the same across disciplines, because the citation accumulation speed and distribution curves vary across disciplines (Baumgartner & Leydesdorff, 2014; Garfield, 1972; Mingers, 2008; Wang, 2013).

It has been shown that in mathematics (and with weaker evidence in biology and earth sciences), for citation windows of 2 years or less the journal’s 2-year impact factor (IF) is a better predictor of long-term impact than early citations are (Abramo, D’Angelo, & Di Costa, 2010). In every science discipline apart from mathematics, for citation windows of 0 or 1 year only a combination of IF and citations is recommended (Bornmann, Leydesdorff, & Wang 2014; Levitt & Thelwall, 2011). The same seems to be valid in the social sciences as well (Stern, 2014). A model based on IF and citations to predict long-term citations was proposed by Stegehuis, Litvak, and Waltman (2015). The weighted combination of citations and journal metric percentiles adopted in the Italian research assessment exercise, VQR 2011–2014 (Anfossi, Ciolfi, et al., 2016), proved to be a worse predictor of future impact than citations only (Abramo & D’Angelo, 2016).

To provide practitioners and decision makers with a better predictor of overall impact, and awareness of how the predicting power varies with the citation time window, Abramo, D’Angelo, and Felici (2019) made available, in each of the 170 subject categories (SCs) in the sciences and economics with more than 100 Italian 2004–2006 publications: (a) the weighted combinations of 2-year IF and citations, as a function of the citation time window, which best predict overall impact; and (b) the predictive power of each combination.

It emerged that the IF has a nonnegligible role only with very short citation time windows (0–2 years); for longer ones, the weight of early citations dominates and the IF is not informative in explaining the difference between long-term and short-term citations.

The calibration of the weights by citation time window and SC, and the measurement of the impact indicator, are not as straightforward as the simple measurement of normalized citations.

In this study, we want to find out whether the extra work involved in improving the predicting power of early citations is worthwhile. We ask whether and to what extent applying a predictor of overall impact that is more accurate than early citations has an effect on the research performance ranks of individuals. In this specific case, as a performance indicator we refer to the total impact of individuals. This indicator is particularly appropriate if one needs to identify the top experts in a particular field, for consultancy work or the like. Using an authors’ name disambiguation algorithm for Italian academics, we measure the total impact of Italian professors (assistant, associate, and full) in the sciences and in economics over, valuing their publications first by the early citations and then by the weighted citation-IF combination provided by Abramo et al. (2019). At this point, we can analyze the extent of variations in rank of individuals in each discipline and field in which they are classified¹.

The rest of the paper is organized as follows. In Section 2, we present the data and method. In Section 3, we report the comparison of the rankings by the two methods of valuing overall impact at field and discipline level. The discussion of results in Section 4 concludes the work.

For the purpose of this study, we are interested in how a different measure of impact affects the ranking of Italian professors by total impact in 2015–2017.

Data on the faculty at each university were extracted from the database of Italian university personnel maintained by the Ministry of Universities and Research (MUR). For each professor this database provides information on their gender, affiliation, field classification, and academic rank at the end of each year². In the Italian university system all academics are classified in one and only one field, a named scientific disciplinary sector (SDS), of which there are 370. SDSs are grouped into 14 disciplines, named university disciplinary areas (UDAs).

Data on output and relevant citations are extracted from the Italian Observatory of Public Research, a database developed and maintained by Abramo and D’Angelo, and derived under license from the Clarivate Analytics Web of Science (WoS) Core Collection. Beginning with the raw data of the WoS, and applying a complex algorithm to reconcile the authors’ affiliations and disambiguation of the true identity of the authors, each publication (article, letter, review, and conference proceeding) is attributed to the university professor who produced it³. Thanks to this algorithm, we can produce rankings by total impact at the individual level on a national scale. Based on the value of total impact we obtain a ranking list expressed on a percentile scale of 0–100 (worst to best) of all Italian academics of the same academic rank and SDS.

We limit our field of analysis to the sciences and economics, where the WoS coverage is acceptable for bibliometric assessment. The data set thus formed consists of 38,456 professors from 11 UDAs (mathematics and computer sciences, physics, chemistry, earth sciences, biology, medicine, agricultural and veterinary sciences, civil engineering, industrial and information engineering, psychology, and economics and statistics) and 218 SDSs, as shown in Table 1. Nine point three percent of professors are unproductive (0 publications), and as a consequence their scores but not necessarily their ranks remain unchanged by the two indicators. In fact, the scores and ranks of uncited productive professors (4.2% in all) will change (because IF is always above 0). Measuring the latter’s impact by citations only, their score (0) and rank would be the same as for unproductive professors, But it would not when measured by the weighted combination of normalized citations and IF.

We measure impact in two ways: One values publications by early citations only, and the other by the weighted combinations of citations and IF⁴, as a function of the citation time window and field of research, which best predict future impact (Abramo et al., 2019).

Because citation behavior varies across fields, we standardize the citations for each publication with respect to the average of the distribution of citations for all publications indexed in the same year and the same SC⁵. We apply the same procedure to the IF.

Furthermore, research projects frequently involve a team of scientists, which is registered in the coauthorship of publications. In this case, we account for the fractional contributions of scientists to outputs, which is sometimes further signaled by the position of the authors in the list of authors.

The fractional contribution equals the inverse of the number of authors in those fields where the practice is to place the authors in simple alphabetical order, but assumes different weights in other cases. For the life sciences, the widespread practice in Italy is for the authors to indicate the various contributions to the published research by the order of names in the listing of the authors. For the life sciences, we give different weights to each coauthor according to position in the list of authors and the character of the coauthorship (intramural or extramural).⁶

For reasons of significance, the analysis is limited to those professors who held formal faculty roles for at least 2 years over the 2015–2017 period.

Citations are observed at December 31, 2018, implying citation time windows ranging from 1–4 years.

In the following, we present the score and rank of performance by total impact of Italian professors, by SDS and UDA, as measured respectively by

• • 

  early citations (TI_C)

• • 

  the weighted combination of citations and IF of the hosting journal (TI_WC)

As already noted, no variations will occur for professors with no publications in the period under observation. We expect instead significant variations in score and rank for professors with uncited publications. In fact, while TI_C is zero, TI_WC will be above zero.

As an example, Table 2 shows the scores and ranks by TI_C and TI_WC for the 26 Italian professors in the SDS Aerospace propulsion. The score variation is zero for the two unproductive professors at the bottom of the list, while it is a maximum for the uncited productive professors (ID 49113 and 2592). Twelve professors experience no shift, among them the top five in ranking. A few pairs swap positions (e.g., ID 78162 and ID 49106). The maximum shift is three positions.

The SDS Industrial chemistry consists of 114 professors, mostly productive and cited. Figure 1 shows the dispersion of their impact. The very strong correlation of scores (Pearson ρ = 0.999) and ranks (Spearman ρ = 0.998) by TI_C and TI_WC are as expected.

Higher dispersion (Figure 2) occurs instead for the 73 professors in the SDS Complementary mathematics, whereby about two-thirds (50) of professors present zero TI_C, and 20% (15), while productive, are uncited (TI_C above 0). As a matter of fact, noticeable shifts in relative scores occur for high performers too (top right of the diagram), notwithstanding a very strong score correlation (Pearson ρ = 0.988). The ability of TI_WC to discriminate the impact of uncited publications, and therefore the relevant performance of uncited professors, explains the lower rank correlation (Spearman ρ = 0.915). Although variations in score are not that noticeable, those in rank are. To better show that, Figure 3 reports the share of professors experiencing a rank shift in both SDSs. In Complementary mathematics, over 60% of professors do not change rank (50% could not, as they were unproductive). The remaining 40% present shifts that are in some cases quite noticeable: Five professors improve their rank by no fewer than 10 positions. Rank shifts are less evident in Industrial chemistry: The average shift is 1.47 positions, as compared to 1.89 in Complementary mathematics. In the former SDS, because of the lower number of unproductive professors, shifts affect a higher share of the population, namely 70%.

For a better appreciation of the rank variations in the whole SDS spectrum, Figure 4 shows the box plots of the average percentile shifts in the SDSs of each UDA, while Table 3 presents some relevant descriptive statistics.

Economics and statistics is the UDA with the highest average percentile shift (6.5), the highest dispersion among SDSs (3.6 standard deviations), and the widest range of percentile shift, from 1.4 of SECS-P/13 (Commodity science) to 15.9 of SECS-P/04 (History of economic thought). It is followed by Mathematics and computer science, whose range of variation of the percentile shift is between 2.0 of MAT/08 (Numerical analysis) and 12.6 of MAT/04 (Complementary mathematics). In contrast, UDAs 4 (Earth sciences) and 5 (Biology) show the lowest dispersion among SDSs (0.3 standard deviation) and quite low average percentile shifts. In UDA Medicine, a peculiar case occurs: In SDS MED/47 (Nursing and midwifery) the two ranking lists are exactly the same. The same occurs also in two SDSs of Industrial and information engineering: ING-IND/29 (Raw materials engineering) and ING-IND/30 (Hydrocarburants and fluids of the subsoil). In general, in 17 out of 218 SDSs, the average percentile shift is never below five percentiles.

In general, the correlation between TI_C and TI_WC is very strong. Table 4 presents some descriptive statistics of both Pearson ρ (score) and Spearman ρ (rank) for the SDSs of each UDA. As for the scores, the minimum correlation (0.957) occurs in an SDS of Medicine (MED/02 – History of medicine). As for the ranks, the minimum occurs (0.884) in an SDS of Economics and statistics, SECS-P/04 (History of economic thought), which also stands out for the maximum average percentile shift among all SDSs (Table 3). It is a relatively small SDS, 35 professors in all, two-thirds of whom have zero TI_C.

The rank variations in general appear strongly correlated with the share of productive professors with zero TI_C (i.e., with only uncited publications). The correlation between the two variables is shown in Figure 5 (Pearson ρ = 0.791).

A typical way to report performance is by quartile ranking. We then analyze the performance quartile shifts by the two indicators of impact. Table 5 represents the contingency matrix of the performance quartile by TI_C and TI_WC of all 38,456 professors of the data set. Because of the strong correlation between the two indicators, we observe an equally strong concentration of frequencies along the main diagonal: In 93.3% of cases (24.52% + 22.90% + 20.96% + 24.94%), the performance quartile remains unchanged. 0.7% of professors in Q1 by TI_C shift to Q2 by TI_WC and 0.8% of professors in Q4 by TI_C shift to Q3 by TI_WC.

Table 6 shows the shift distributions by UDA. Economics and statistics presents the highest share of professors (17.1%) shifting quartile, followed by Mathematics (9%). In the remaining UDAs shares range between 4% and 7%. It must be noted that 0.4% of professors (154) experience two quartile shifts, and all but two shift from the bottom to above the median. They are mainly in Economics and statistics, and in Mathematics.

Evaluative scientometrics is mainly aimed at measuring and comparing the research performance of individuals and organizations. A critical issue in the process is the accurate prediction of the scholarly impact of publications when citation short time windows are allotted. This is often the case when the evaluation is geared to informed decision-making.

Better impact prediction accuracy often involves complex, costly, and time consuming measurements. Pragmatism requires an analysis of the effects of improved indicators on the performance ranking of the subjects under evaluation. This study follows up the work by the same authors (Abramo et al., 2019), which demonstrated that especially with very short time windows (0–2 years) the IF can be combined with early citations as a powerful covariate for predicting long-term impact.

Using the outcomes of such work (i.e., the weighted combinations of IF and citations as a function of the citation time window) that best predict the overall impact of single publications in each SC, we have been able to measure the 2015–2017 total impact of all Italian professors in the sciences and economics, and to analyze the variations in performance ranks when using early citations only.

As expected, scores and ranks by the two indicators show a very strong correlation. Nevertheless, in 7% of SDSs, the average shift never goes below 5 percentiles and is 15.6 and 12.9 on average in the SDSs, respectively, of Economics and statistics and Mathematics and computer science.

In terms of quartile shifts, almost 7% of professors undergo them. In Economics and statistics, 3% of professors shift from Q4 to above the median.

A strong correlation can be seen between the rate of shifts in rank and the share of uncited professors in the SDS. The total impact of uncited professors is in fact nil by TI_C, but above zero by TI_WC. In short, TI_WC can better discriminate the performance of professors in the left tail of the distribution. The higher the share of uncited professors in an SDS, the more recourse to TI_WC is recommended. Furthermore, the shorter the citation time window, the heavier the relative weight of IF in predicting the long-term impact. TI_WC is then highly recommended when citation time windows are short and the rate of uncited professors is high.

In the case of national research assessment exercises based on informed peer review or on bibliometrics only, the weighted combination of normalized citations and IF to rank publications might be adopted, as the weights can be made available by the authors for all SCs and citation time windows up to 6 years.

Possible future investigations within this stream of research might concern the effect of the improved indicator of publications’ impact on the performance score and rank of research organizations and research units.

Giovanni Abramo: Conceptualization; Formal analysis; Investigation; Project administration; Resources; Software; Supervision; Validation; Visualization; Writing—original draft; Writing—review & editing. Ciriaco Andrea D’Angelo: Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Resources; Software; Validation; Visualization; Writing—original draft; Writing—review & editing. Giovanni Felici: Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Resources; Software; Validation; Visualization; Writing—original draft; Writing—review & editing.

The authors have no competing interests.

No funding has been received for this research.

The authors use data licensed from Clarivate Analytics. They are not allowed to publish this data.