Unveiling the distinctive traits of a nation’s research performance: The case of Italy and Norway

https://publons.com/publon/10.1162/qss_a_00198

One of the principal objectives that governments and institutions pursue through research assessment is improving research effectiveness and efficiency. Effectiveness can be deployed along various dimensions, depending on the strategic goals of the country. These include among others, sustainability goals, research integrity, responsible research and innovation, open science, gender equality in research, geographical balance, international research collaboration, public–private collaboration, and rapid and widespread diffusion of new knowledge to potential users.

Efficiency essentially deals with the relation between research output and inputs to produce it. All inputs being equal, individual and organizational performance can be increased by producing more research products (alone or in collaboration), or of better quality (higher impact), or both. Providing the government, research institutions, and scientists with information along each single dimension of their overall research performance might serve better the aim of stimulating improvement. In this way, the single actors are informed about the dimensions with higher margins of efficiency gains.

Whether associated or not with financial rewards or other competitive mechanisms, the simple communication to individuals of their performance scores and ranks, if properly channeled, can be instrumental in continuous improvement. At the large-scale level of national scientific systems, governments and policymakers might also benefit from such information to complement that of their comparative world performance rank. It can inform decisions on the dimensions along which to focus and orient policy measures.

This study deals with research efficiency. The quintessential indicator of efficiency in any production systems is productivity (i.e., the rate of output per unit of input). In other words, it measures how efficiently production inputs are being used. Most bibliometricians define productivity as the number of publications in the period of observation. Because publications have different values (impact), and the resources employed for research are not homogeneous across individuals and organizations, we adopt the definition of productivity extracted from the economic theory of production: the value of output per euro spent in research (i.e., costs of labor and all other production factors, referred to as capital in the following).

Following on from our previous studies on the comparison of research productivity of Italian and Norwegian academics (Abramo, Aksnes, & D’Angelo, 2020, 2021), in this work, we present a methodology to measure the single components of research productivity, and apply it to further scan and contrast the two countries’ academic systems.

In terms of their research production systems, Italy and Norway are rather different. First, the total scientific output of Italy measured by number of journal articles is approximately five times as large as Norway’s. In 2018 (latest available data at time of first of our previous studies), Italy ranked as the eighth largest science producing country in the world and Norway was in 29th position (Norges forskningsråd, 2019). However, compared with the population size, per capita production is 150% higher for Norway than for Italy. Furthermore, the overall citation impact of research output per capita is also higher for Norway. Thus, Norway may be considered a more research-intensive country than Italy. The underlying reasons for these differences are interesting to analyze further, and we aim to contribute to additional knowledge on the issue.

Abramo et al. (2020) for the first time ever applied the Fractional Scientific Strength, or FSS, indicator of research performance of professors and universities to a country other than Italy (i.e., Norway). FSS (Abramo & D’Angelo, 2014) differs from the most popular indicators, such as the h-index (Hirsch, 2005), the MNCS (Waltman, van Eck et al., 2011), and their variants (Alonso, Cabrerizo et al., 2009; Waltman, 2016; Wildgaard, Schneider, & Larsen, 2014), essentially because it accounts for the resources used to produce research output. Furthermore, the FSS adopts the publications’ fractional counting method, and values them through citation indicators.

We refer the reader to Abramo et al. (2020) for details about the academic systems in the two countries and the difficulties of achieving comparable measurements of performance, and the ways to overcome them. The reader will also find the procedure for operationalizing the measurements and all the limits and assumptions involved. Where appropriate, we extrapolate from that study (and report in this) the performance scores and ranks in the two countries, across disciplines.

In summary, the previous study showed the following:

• hardly any differences in the average research productivity of Italian and Norwegian professors;

• a higher concentration of Norwegian professors in the top and the bottom tails of the productivity distribution; and

• higher productivity of Norway in Mathematics and Earth and space sciences, and of Italy in Biomedical research and Engineering.

In this study, we complement the above results with more in-depth analysis of performance along the single dimensions of overall research productivity of Italian and Norwegian professors. In particular, we contrast the academics of the two countries in terms of yearly output, fractional output, average citations per paper, and average impact factor (IF) per paper.

The elaborations are aimed at answering the following research questions:

1. Do the academics of one country tend to publish more than those of the other?

2. In doing so, do they engage in more collaborative work?

3. Do they pursue higher quality research products in terms of citation impact?

4. Do they publish their results in more prestigious journals?

Such questions are important in the context of evaluative use of bibliometrics. For example, in an upcoming evaluation of Norwegian STEM research¹, the panels are asked to assess the production of scholarly publications, the research quality, and national and international cooperation—in addition to several other dimensions, such as research career and mobility, leadership, and research infrastructure. The Italian national research assessment exercise series, VQR², of which the results of the latest edition have just been released, emphasizes the importance of the quality of scientific production over volume, international research collaboration over national, etc. Thus, in-depth analyses of bibliometric patterns may be useful to assess the aspects related to research output, where a unit of analysis may perform differently across the dimensions. For example, high publication volume does not necessarily imply high citation impact and vice versa. Moreover, different indicators reflect different dimensions of performance, which are required to provide reliable assessments, a point which has long been emphasized by bibliometric professionals (van Raan, 1993). Therefore, we wish to underline that none of the indicators that we present and measure in this paper can be considered alone as performance indicators. Each one represents more or less a dimension of performance, but not the overall performance. They convey complementary information, useful to orient scientists in focusing their efforts for continuous improvement, and managers and for policymakers in formulating their interventions to the same aim.

Generally, a large number of factors may explain patterns identified through bibliometric analyses. Some can be investigated bibliometrically, but often other approaches are needed. For example, some might be related to different policy and behavioral mechanisms. Such explanatory attempts are not straightforward, as the systems are influenced by a large number of factors, and at best indirect evidence can be provided. For example, higher publication volume or quality may be due to more financial resources, more time for research, better management, changes in the staff composition, recruitment of talented individuals, responses to funding incentives, and so forth.

In this paper, we therefore mainly focus on the empirical aspects, and second-order interpretations are addressed to a limited extent. In presenting our study, we devote special attention to the definition of each single indicator and the description of the operationalization of their measurement to make hopeful future replications of the exercise in other countries more straightforward.

The rest of the paper is structured as follows. In the next section we present the construction of the data set. In Section 3, we will recall the FSS definition and present those of the complementary indicators we are going to measure. In Section 4, we will present the results of the assessment. Section 5 will conclude the work with our considerations.

To follow on from the abovementioned previous studies of ours and their findings (Abramo et al., 2020, 2021), we observe the research activity of Italian and Norwegian professors in the same period, 2011–2015.

Data on Italian professors have been retrieved from the database on university personnel, which is operated by the Italian Ministry of Universities and Research (MIUR). This database contains information on the name of each individual, their gender, affiliation, field classification, and academic rank at the end of each year³.

Data on the Norwegian professors have been retrieved from a similar database, the Norwegian Research Personnel Register, operated by Nordic Institute for Studies in Innovation, Research and Education (NIFU) (the database underlying the official Norwegian R&D statistics).

For reasons of significance, the analysis is limited to those professors who held formal faculty positions for at least three years over the 2011–2015 period. Furthermore, the data set is limited to individuals with at least one publication during the time period (nonpublishing personnel are not registered in Norwegian databases).

The data on the publication output of the Italian professors are retrieved from the Italian Observatory of Public Research (ORP), a bibliometric database where data are available at the level of authors. The database is developed and maintained by Abramo and D’Angelo, and derived under license from the Clarivate Analytics Web of Science (WoS) Core Collection.

For the Norwegian professors, WoS publication data have been retrieved from a bibliographic database called Cristin (Current Research Information System in Norway). This is a national information infrastructure, providing data for all institutions in the higher education sector, research institutes and hospitals.

To perform a comparative assessment without distortions (see below), each individual is classified in one and only one WoS subject category (SC)⁴.

The analysis covers professors in all fields, with the exception of arts and humanities and a few SCs in the social sciences, where WoS has largest limitations in terms of coverage (Aksnes & Sivertsen, 2019; Hicks, 1999; Larivière, Archambault et al., 2006). There are large variations in the size of the different SCs. In some cases there are few observations, which might cause random fluctuations in performance. Therefore, we have further removed the SCs that did not meet a minimum threshold of at least 10 professors in total, of both nationalities.

The final data set consists of 34,009 Italian and 4,327 Norwegian professors, falling in 177 SCs. Their distribution per academic rank and discipline⁵ is shown in Table 1.

As already said, in the economic theory of production, productivity is defined as the rate of output per unit of input. Because publications (output) have different values (impact), and the resources employed for research are not homogeneous across individuals and organizations, in research systems, a more appropriate definition of productivity is the value of output per euro spent in research. The FSS indicator is a proxy measure of research productivity. A thorough description of the FSS indicator, and the theory underlying it, can be found in Abramo and D’Angelo (2014).

In the fields where the common practice is to place authors in simple alphabetical order, the fractional contribution equals the inverse number of authors, while in other cases different weights are applied (see Waltman, 2012). More specifically, for Biology, Biomedical research, and Clinical medicine, the usual practice is to indicate the contributions of each author by their position in the bylines. Thus, for these disciplines, we have applied a system where different weights are credited to each coauthor depending on their position in the list of authors and the type of coauthorship (intramural or extramural)¹⁰.

Looking at the formula, it can be seen that productivity is a function of output, or more precisely of individual contribution to output, its quality (impact), and the resources used for production.

In this work we want to assess how Italian professors compare to Norwegian professors along each dimension of productivity, namely, in terms of output, fractional output, average citations per paper, and, although not a dimension of productivity, average IF per paper. The average IF per paper is informative about the degree of prestige of the journals chosen by researchers to have their manuscripts published. The breakdown of the productivity indicator along the above dimensions allows us to unveil the possible different traits of professors in conducting research activities.

The relevant indicators are the following:

• Output (O), average yearly publications authored by the professor per euro spent in research:

  $O=1wr2+kNt$

  (2)

  where

  • w = average yearly salary of the professor

  • k = average yearly capital available for research to the professor

  • N = number of publications by the professor in period under observation

  • t = number of years on staff of the professor during the period under observation

• Fractional Output (FO), average yearly total contribution to publications authored by the professor per euro spent in research:

  $FO=1wr2+k1t∑i=1Nfi$

  (3)

  where

  • f_i = fractional contribution of professor to publication i.

• Average Citation (AC), average standardized citations per publication:

  $AC=1N∑i=1Ncic¯$

  (4)

  where

  • c_i = citations received by publication i

  • $c¯$ = average of distribution of citations received for all WoS-cited publications in same year and SC of publication i.

• Average IF (AIF), average standardized IF per publication:

  $AIF=1N∑i=1NIFiIF¯$

  (5)

  where

  • IF = IF of journal hosting publication i

  • $IF¯$ = average of distribution of IFs of journals hosting all WoS-cited publications in same year and SC of publication i.

The performance scores of professors belonging to different SCs cannot be compared directly. In fact, scientists’ intensity of publication remarkably varies across fields, in general (Lillquist & Green, 2010; Sandström & Sandström, 2009; Sorzano, Vargas et al., 2014), and in both countries in particular (D’Angelo & Abramo, 2015; Piro, Aksnes & Rørstad, 2013); citation behavior varies across fields (Stringer, Sales-Pardo, & Amaral, 2010; Vieira & Gomes, 2010); and the intensity of collaboration (i.e., the average number of coauthors per publication) also varies across fields (Abramo, D’Angelo, & Murgia, 2013; Glanzel & Schubert, 2004; Yoshikane & Kageura, 2004).

To avoid distortions, then, the performance rankings of professors are constructed at the SC level.

For comparisons at higher levels of aggregation (i.e., discipline and overall), we normalize performance scores to the average score of all professors of the same SC. To exemplify, an FSS score of 1.10 means that the professor’s performance is 10% above average, in his or her own SC¹¹.

In the following tables, figures and text, all performance scores are normalized, but we keep the same denomination for the relevant indicators.

In the following, we present the comparison of Italian and Norwegian professors along each of the above indicators, per discipline and overall.

Figure 1 shows the average normalized scores of each indicator, for the overall 34,009 Italian and 4,327 Norwegian professors. While the two populations show practically the same productivity, FSS¹², noticeable differences occur for the other indicators. On average, Italian professors publish more (1.4% above average) than Norwegian professors (11.1% below average). Accounting for the real contribution to each publication, the gap decreases, but it is still there: Italy’s FO equals 1.00, while Norway’s is 0.97.

Because the FSS is the same, it follows that Norway’s AC needs to be higher (1.03) than Italy’s (1.00). Furthermore, Norwegian academics on average publish in more prestigious journals (AIF = 1.07) than Italians (AIF = 0.99).

In short, while the two countries present the same research productivity, Italians publish more, with larger research teams, while Norwegians publish higher quality products in more prestigious journals.

We wondered whether the same traits could be found among top-performing scientists as well. Limiting the analysis to top 10% professors by FSS at SC level, we observe similar differences for all considered indicators (Figure 2). Norwegian top professors (FSS = 4.15) are slightly more productive than Italians (FSS = 4.10). The first build up their supremacy by producing higher impact publications. In fact, the average O by Italian top professors is 6.8% higher than Norwegian (2.57 vs. 2.39), and although it diminishes by FO to 5.2% (2.77 vs. 2.62), the roles swap in terms of impact, whereby Norwegian top professors register an AC (AJI) 11.4% (12.4%) higher than Italians.

In the following, we present the results at field level (discipline and SC), to assess possible differences across fields between the two populations. We will start with output indicators; impact will follow.

Italian professors’ O is higher than that of Norwegians in all 11 disciplines except Mathematics, as shown in Table 2. The biggest gap occurs in Biomedical research, where the 3,707 Italian professors’ O is 1.02, while the 245 Norwegian professors’ is 0.77. Noticeable differences occur also in Psychology (1.05 vs. 0.84), Biology (1.02 vs. 0.82), Engineering (1.01 vs. 0.82), and Clinical medicine (1.02 vs. 0.85).

When accounting for the real contribution to each paper, Italian professors’ contribution (FO) is greater in only five disciplines, especially in Biomedical research (1.01 vs. 0.88) and in Biology (1.01 vs. 0.90). Norwegians noticeably prevail in Mathematics (1.12 vs. 0.99), Chemistry (1.13 vs. 0.99), and Earth and space sciences (1.08 vs. 0.98).

Next, we analyze the research output at the level of SCs. As noted in Section 2, SCs differ considerably in size, and some of them include a rather small number of individuals, particularly for Norway. Nevertheless, an analysis at this level may reveal interesting differences across the nations.

Table 3 shows how the differences by O and FO observed at discipline level vary across the SCs of each single discipline. The FSS scores are reported as a reference. We note that Norwegian professors outperform Italians’ FSS in 40% of SCs, in 24% by O, and in 38% by FO. In particular, by O Norwegians never outperform Italians in Psychology, and they do so in only two of the 28 SCs of Biology and in two of the 14 SCs of Biomedical research. On the other hand, in Mathematics, Italians outperform Norwegians in two out of 6 SCs. In terms of FO, Norway recovers and, differently from other disciplines, in Chemistry it outperforms Italy in five out of seven SCs.

Table 4 shows the 10 SCs where the gap by O between Norway and Italy is highest, and vice versa. At the top of the list, in line with the above findings, we find an SC of Mathematics (Mathematics, interdisciplinary applications), where the six Norwegian professors show an O (3.77) about six times as high as that of their 46 Italian counterparts (0.64). The other top SCs fall in four disciplines, namely, Engineering (Remote sensing; Construction & building technology), Clinical medicine (Anesthesiology; Substance abuse), Physics (Acoustics; Thermodynamics; Physics, particles & fields), and Political and social sciences (Area studies; Political science). On the other side, the 10 SCs with the highest gap in favor of Italy fall in Engineering (seven), Clinical medicine (two), and Biomedical research (one).

Table 5 presents the same analysis by FO. The results are similar to those by O. Six of the top 10 SCs where Norway outperforms Italy remain the same, while only one changes among those where Italy outperforms Norway.

We now turn to the average impact analysis per discipline. Figure 3 presents the normalized scores of AC. Norwegian academics outperform Italians in six disciplines, most noticeably in Earth and space sciences (0.96 for Italy vs. 1.16 for Norway), Mathematics (0.98 vs. 1.15), and Clinical medicine (0.99 vs. 1.09). Italians outperform Norwegians in five, most noticeably in Engineering (1.01 vs. 0.90), Psychology (1.01 vs. 0.95), and Physics (1.00 vs. 0.95).

Norwegian professors publish on average in more prestigious journals than Italians in eight disciplines, the only exceptions being Physics, Chemistry, and Engineering (Figure 4).

As shown in Figure 1, Norway produces higher impact publications on average, but from Table 6 we see that this occurs in fewer than 50% of the SCs under observations (86 out of 177): in particular, in 10 of the 11 SCs of Earth and space sciences, in four of the six SCs of Mathematics, and in seven of the 11 SCs of Political and social sciences. Conversely, Italians outperform Norwegians in six of the eight SCs of Economics, five of the seven SCs of Chemistry, and 10 of the 14 SCs of Biomedical research.

Norwegian research products are published in higher IF journals in 112 SCs (63%).

Table 7 reports the 10 SCs with the highest difference of AC for each country. In five SCs the average AC of Norwegian professors is above 2, while for Italians it is below 1. The highest gap is registered in History & philosophy of science, with an average AC for 62 Italian professors of 0.74, against 2.59 for 10 Norwegians. Of the top 10 SCs where Italy outperforms Norway, five SCs fall in Engineering (Robotics; Engineering, aerospace; Transportation science & technology; Materials science, composites; Computer science, cybernetics), and three in Economics/Political and social sciences (Public administration; Urban studies; Area studies).

Table 8 proposes the same view by the average IF of hosting journals per paper. Along this indicator, five of the top 10 SCs where Norway outperforms Italy fall in Clinical medicine (Andrology; Urology & nephrology; Medicine, general & internal; Anesthesiology; Medicine, legal). Similarly, five of the top 10 SCs for Italy fall in Engineering (Robotics; Medical informatics; Computer science, interdisciplinary applications; Nanoscience & nanotechnology; Transportation science & technology).

The results of any evaluation exercise depend on the adopted methodology and, in particular, on the assumptions underlying the measurement indicators used. As reported in Section 3, in this study we used both output to input indicators (O, FO, and FSS) and average impact indicators (AC and AIF) that are independent of input.

The typical limits embedded in bibliometric analyses apply to our study too. In particular, the new knowledge produced is not only that encoded in publications. Moreover, bibliographic repertories (such as WoS, used here) do not register all publications. The measurement of the impact of publications, before the end of their life cycle, using citation-based indicators is a prediction of their overall impact, and therefore not definitive. Citations can also be negative or inappropriate, and in any case they certify only scholarly impact, forgoing other types of impact. Finally, the results could be sensitive to the field classification schemes used for publications and, especially in the proposed work, for professors.

To these limits, for O, FO, and FSS we must add those associated to the assumptions on input data. When assessing the performance of individual scientists, if there are differences in the production factors available to each, then one should normalize for them. Unfortunately, relevant data are not available at the individual level, either in Italy or in Norway. So, we are forced to assume that the same resources are available to all professors within the same field. The second assumption is that the hours devoted to research are the same for all professors.

Given the characteristics of the Italian and Norwegian academic systems, this assumption appears to be acceptable. In fact, in keeping with the Humboldtian model, there are no “teaching-only” universities included in the analysis, as all professors are required to carry out both research and teaching, with a breakdown of the overall workload that, on average, seems to justify the assumption that time available for research is 50% of the total work time. However, lacking a normative reference or a qualified on-field survey attesting the validity of this assumption, we here propose a specific analysis to test how sensitive evaluation outcomes are to such an assumption. In particular, for O, FO and FSS we repeat the analysis at an overall level in four distinct scenarios:

• Scenario 0: the one based on the assumption adopted in this study, and whose results are shown in Figure 1;

• Scenario 1: where the time available for research is set at 40% of the total work hours for all professors in the data set;

• Scenario 2: where the time available for research is set to 60% of the total work hours for all professors in the data set;

• Scenario 3: where the time available for research is set to 40% of the total work hours for full professors, 50% for associate and 60% for assistant (assuming that the responsibilities for management, coordination and institutional representation activities (and therefore the time dedicated to them) grows with academic rank.

It should be noted that differentiation by country would have the sole effect of increasing the productivity of professors in one country with respect to the other.

In Figure 5 we show the results of the sensitivity analysis. The variability of the average value of the indicators among the four scenarios is very limited for Italy, while it is more evident for Norway. This result certainly depends on the different size of the two countries: Norwegian professors represent 11% of the total data set, Italians 89%. In particular, for Norway Scenario 0 is certainly the most favorable. With respect to the single indicators, for O and FO the differences between the two countries are always quite noticeable. In contrast, for FSS, Scenario 0 shows practically no difference in average performance between the two countries, while in the other three scenarios the difference is more noticeable.

We repeat the same analysis at the discipline level. For each country and discipline, we provide the average value and error bars of normalized research output (Figure 6), fractional output (Figure 7), and productivity (Figure 8) for the four scenarios. The graphs confirm that the variability among the four scenarios is practically undetectable for Italy, while more evident for Norway. However, in no case does a ranking inversion between the two countries emerge. We can conclude that at the discipline level, the sensitivity of results of an evaluation exercise to assumptions about the time available for research is very limited.

Instead, at the overall level, the analysis reveals a higher sensitivity to input data and suggests attention when setting assumptions and caution in the interpretation of results.

Cross-national comparison of research performance is one of the main areas of application of bibliometrics. For example, such analyses have a central position in science and technology indicator reports for monitoring scientific development and performance (see, for example, National Science Board, 2020; Norges forskningsråd, 2019; OECD, 2019). Through the great attention that these reports receive, the bibliometric indicators play an important role in the public perception of the scientific performance of countries. Our study provides new insight into more advanced methods for comparing national research performance than is provided through such reports.

Despite Italy and Norway differing considerably along such dimensions as research size and profile, and the organization of their research systems, there are hardly any differences at all in the average research productivity (FSS) of Italian and Norwegian professors (Abramo et al., 2020).

In this work, we delve into the various dimensions of research performance. Overall, the study shows that an average Italian professor publishes more papers than a Norwegian. This difference is quite large when measuring publications on a whole count basis, but is smaller using fractionalized measures (in which credit for a publication is allocated according to the contribution of the participating coauthors). This means that Italian publications must have more coauthors, which suggests that Italians tend to work in larger teams. This might be related to the fact that the Italian research system is much larger than Norway’s. Possibly, Italian researchers may benefit from larger universities, which can favor larger institutional collaborations. The different collaboration behavior of Italian and Norwegian professors might reveal an interesting topic for future investigation. Further future research may repeat this study, restricting the observation to top performers only, to assess whether their distinctive traits along each productivity component differ from the average population.

The discussion of the appropriateness of whole versus fractionalized measures of publication output has a long history in bibliometrics, dating back to the dispute on the decline of British science during the 1980s (Braun, Glänzel, & Schubert, 1989; Irvine, Martin et al., 1985). Moed (2005) argues that the two measurement principles should be seen as complementary, where whole counts measure participation and fractionalized counts measure the number of creditable papers. Our analysis shows that the different measurement principles yield quite different results, which has also been documented in previous studies (Aksnes, Sivertsen et al., 2016; Gauffriau, Olesen Larsen et al., 2008). Currently, fractionalized measurement seems to have gained increased popularity (Waltman & van Eck, 2015), and this is also the principle underlying the FSS indicator.

While the number of publications produced per individual is higher on average for Italian than for Norwegian professors, the pattern is reversed when looking at the citation impact and journal profile. We observe national patterns where Italian academics produce more papers but with lower impact and in less prestigious journals. Thus, there is a quantity versus quality tension in the national publication patterns, where the first dimension apparently is more emphasized by Italians and the latter by Norwegians. However in the combined FSS indicators these differences tend to level out.

We are not able to provide any final answer on the possible reasons underlying these differences. The quality of the research is one obvious dimension, but there are also mechanisms in the research systems that might favor certain publication behaviors more than others. We will discuss this further below.

In Norway, there is a performance-based funding system where a bibliometric indicator is applied for the allocation of funding across institutions (Sivertsen, 2018). Although this model is designed to work at an overall national level, previous research has shown that it is sometimes applied at lower levels and may have an incentive effect at the level of individual researchers (Aagaard, 2015). Two important elements of the model should be emphasized. First, extra points are given for publications in the most prestigious publication channels; and second, all journal and book publications in accredited publication channels are included, not only publications indexed in WoS. This means that there is an incentive to publish in prestigious publication outlets and no incentive to publish in WoS journals specifically. Moreover, citations are not included as indicators in the Norwegian model.

In Italy, a proportion of public funds started being allocated to universities on the basis of the outcome of the second national research assessment exercise, VQR 2006–2010¹³. The evaluation was based on a restricted number of research products submitted by universities, and the bibliometric indicator applied to assess their relative quality was a combination of citations and IF. The underlying incentive seems to have professors focus on the production of a few very high-quality outputs, to be published in prestigious journals. In the same period, the “national scientific habilitation” (ASN) for university appointments was introduced in Italy¹⁴. The ASN required passing the threshold values in all or part of (depending on the field) three bibliometric indicators, namely number of articles, number of citations, and contemporary h-index¹⁵.

Based on these national differences, where citations are used as a component indicator in Italy only, one might have expected Italian professors to be more highly cited. However, our overall results show the contrary. The empirical paradox may suggest that the incentive effect at the level of individual researchers may be limited, or that it takes a longer time for incentive systems to show effects. Still, it should be emphasized that the Norwegian division of publication channels by quality levels means that the Norwegian model is not a pure productivity measure. Moreover, the funding models are only a part of the picture, and citations indicators are sometimes used in other contexts as well, such as in evaluations of research and in the assessment of research proposals and candidates (Wilsdon, Allen et al., 2015). The national differences in the number of publications are also relevant to discuss in the light of the two funding models. While the Italian model is limited to WoS-indexed publications, the Norwegian system covers all publications in channels accredited as scientific or scholarly. A previous analysis showed that the WoS Core Collection covered 69% of the total Norwegian publication output, with large variations across fields (Aksnes & Sivertsen, 2019). This means that a relatively large number of the publications of the professors analyzed would not be included in the present analysis. This presumably holds for both Italy and Norway. However, as there is no specific incentive to publish in WoS journals in Norway, this issue might affect Norway more than Italy.

Furthermore, previous research has shown that there is a language bias in the coverage of WoS, which affects various countries differently (van Leeuwen, Moed et al., 2001). Although journals in national languages as a general rule are not covered by WoS, at least not in the core collection, there are many exceptions. While coverage of national journals will increase the publication numbers of a country, it will have an opposite effect on the citation impact, as articles in these journals generally tend to be little cited. There are hardly any national Norwegian journals indexed in the core collection of WoS, while for Italy there are several, and becoming more and more numerous, especially after the introduction of the VQR. Possibly, this factor too might explain some of the observed country differences across the indicators analyzed.

In interpreting the results of the performance analysis, all the usual limits, caveats, assumptions, and qualifications of evaluative scientometrics apply. In addition to the bibliographic data source and relative country coverage, these include general bibliometric issues, such as that publications are not representative of all knowledge produced, and that citations have limitations as performance measures. In addition, the results are sensitive to the classification schemes applied for the publication output and professors. Finally, due to limitations in the availability of comparable input data, some adaptations have been introduced. The sensitivity of results to varying input data suggests care when setting assumptions and caution in interpreting findings. We have shown in fact that varying the assumed 50% of professors’ time devoted to research to 40% or 60% would tilt research productivity in the two countries from being equal to favoring Italy. This is particularly true at overall level, while at discipline level the sensitivity of results is very limited and the relative performance of the two countries seems very robust.

From this, there arises a call for those governments and research institutions that intend to benefit from ever more precise and reliable performance evaluations, useful for their decisions and policy-making: They should be prepared to give scientometricians the underlying input data necessary for the job. Subject to the availability of input data, the present study can be replicated to include other countries.

Studies of the output dimension in other countries would be of particular interest. For a long time, various normalization procedures have been applied for calculating citation indicators (Moed, 2005). However, there are no cross-national reference standards for comparing publication output, although previous studies have shown the field dimension to be of particular importance. For example, a study of publication patterns in social sciences and humanities in eight European countries showed large differences across fields but also across nations (Kulczycki, Engels et al., 2018). In addition, publication output has been shown to be influenced by individual variables such as gender (Abramo et al., 2021; Sugimoto, Larivière et al., 2013; Elsevier, 2020), age (Abramo, D’Angelo, & Murgia, 2016; Gingras, Larivière et al., 2008; Kyvik, 1990; Levin & Stephan, 1989), and academic rank (Abramo, D’Angelo, & Di Costa, 2011; Blackburn, Behymer, & Hall, 1978; Ventura & Mombrù, 2006). Thus, further research would be required to provide better fundamentals for cross-national analyses along this dimension of performance.

Analyses based on comparable research assessment data may give stakeholders, such as governments, policymakers, universities, business enterprises, and prospective students, valuable information on how various higher education institutions perform. In particular, in-depth investigations along single dimensions of research performance can reveal those with higher potential efficiency gains, and therefore orient the incentive systems for continuous improvement.

Giovanni Abramo: Conceptualization, Investigation, Methodology, Supervision, Validation, Writing—original draft, Writing—review & editing. Dag W. Aksnes: Data curation, Funding acquisition, Investigation, Methodology, Supervision, Validation, Visualization, Writing—review & editing. Ciriaco Andrea D’Angelo: Data curation, Formal analysis, Investigation, Methodology, Validation, Writing—original draft.

The authors have no competing interests.

The research project received funding by the Nordic Institute for Studies in Innovation, Research and Education (NIFU), in Oslo, Norway for open access publishing.

Being subject to Clarivate-WoS license restrictions, the raw data cannot be made publicly available, but are available from the authors upon request by the reader for personal interest only.