University rankings in the context of research evaluation: A state-of-the-art review

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00317

The Academic Ranking of World Universities (ARWU) first appeared in 2003 and marked the beginning of the era¹ of global university rankings. Prospective students and their parents make decisions on applying to a particular university based on rankings; thus, university administrators try to construct university strategies around indicators of university rankings (Beine, Noël, & Ragot, 2014; Fauzi, Tan et al., 2020; Olcay & Bulu, 2017). Global university rankings “shape the allocation of hundreds of millions of dollars in public funding and private investment” (Nassiri-Ansari & McCoy, 2023, p. 4). Moreover, the governments of various countries consider the number of universities in a particular ranking (or rather, in its tier – e.g., top 100 or top 200) as an indicator of the competitiveness of national higher education systems and the object of national pride (Hazelkorn, Loukkola, & Zhang, 2014). Thus, global university rankings have become part of national strategies in less than 20 years (e.g., in China, France, and Russia). Even academics, despite the harsh criticism of rankings from the community, sometimes make decisions about where to build a career based on university rankings (from personal conversations with colleagues).

The flip side of the popularity of university rankings has been an ongoing debate in academic and policy discourses that touches on the validity of rankings in general. The main message of university rankings is that they are absolute indicators of excellence (merit, performance, quality, etc). Indeed, in our time, anyone can get an answer to the question of which university is higher in ranking in a matter of seconds with only an Internet connection. However, the validity of this premise has been constantly debated. Based on the ARWU and Times Higher Education (THE) rankings, Safón (2013) found that the basis of global university rankings is a research university from the United States or an English-speaking country with a high reputation and large financial resources focused on hard science. The Matthew effect leads to a significant bias in evaluation. Soh (2017) outlined seven “deadly sins” of university rankings: spurious precision, weight discrepancies, assumed mutual compensation, indicator redundancy, intersystem discrepancy, negligence of indicator scores, and inconsistency between changes in ranking and overall. Lim (2018) introduced the concept of “weak expertise,” which is a compromise between reliability and robustness, on the one hand, and relationships with key stakeholders, on the other. Most global university rankings use Scopus or Web of Science publication data, so there is a certain bias towards journals in English. Additionally, quantitative data can be easily manipulated and may not reflect the real quality of a university’s research/innovation activity (Vernon, Balas, & Momani, 2018). The use of indicator weightings in different rankings is usually arbitrary, which calls into question the validity of the final composite index (Johnes, 2018).

Espeland and Sauder (2016) explored the cultural, economic, political, and social implications of academic rankings in the context of higher education. The authors delved into the historical development of rankings and how they have come to shape accountability mechanisms within academia. Through extensive research and analysis of primary sources, Espeland and Sauder argued that rankings have become powerful engines of anxiety that drive universities to prioritize reputation management over educational quality. They discussed how rankings influence university decision-making, resource allocation, and even individual career trajectories. The book challenges the prevailing belief that rankings reflect objective measures of excellence and instead presents them as socially constructed systems that perpetuate inequalities and narrow definitions of success in academia.

A little earlier than the global rankings, a special type of policy initiative called excellence initiatives appeared. These initiatives initially originated in East Asian countries (China, Korea), but then spread to the European continent. Some initiatives were aimed directly at increasing the number of national universities in a particular ranking tier. For example, the Russian excellence initiative was named 5top100, which meant five Russian universities were supposed to be in the top 100 of global university rankings in 2020. Historically, rankings have played a significant role in China’s excellence initiatives. In some countries there has recently been a major shift away from rankings in favor of national and local relevance. For example, in 2021, the Russian government launched the Priority 2030 program, which, unlike the previous initiative Project 5-100, declared the rejection of university rankings as the basis for assessing universities (Kochetkov, 2022).

There is a close relationship between rankings and research evaluation (Hazelkorn & Gibson, 2017). However, some studies showed that the emphasis of global university rankings on the quantity of publications rather than their quality has led to an overemphasis on research output, which in turn has led to researchers being evaluated primarily on the number of publications they produce (Altbach & Salmi, 2011).

The aim of this study is to provide an overview of the literature on the use of university rankings in research evaluation and excellence initiatives, focusing both on how university rankings are used in practice in research evaluation and on how different authors believe university rankings should or should not be used in research evaluation. Table 1 presents a list of the rankings most frequently mentioned in this study.

I conducted a comprehensive review of the academic literature on university rankings within the framework of research evaluation practices and policies (Section 2). The main part of the review is based on English-language sources indexed in Web of Science. Section 3 analyzes gray literature². These pieces describe the global picture; however, the devil is in the details. Research evaluation policies and practices vary widely across countries and regions of the world, so context does matter here. Therefore, I added a review of academic literature in Russian (Section 4). My choice of Russia is intentional: Here, university rankings played a key role in the implementation of the Project 5top100 flagship excellence initiative (2013–2020). The project itself was so named because five Russian universities were supposed to be in the top 100 of global university rankings by 2020. Finally, in Section 5, I discuss university rankings in a broader social context.

To identify the literature discussed in this section, I used the following search query in the Web of Science Core Collection: “ranking* NEAR/2 university AND (“research evaluation” OR “research assessment” OR “research performance” OR “research quality” OR “excellence initiative*”)”. The query returned 161 results from 2005–2022. In the next step, I manually screened all the articles to remove off-topic results. The “core” included 33 articles. After that I used a forward snowballing technique to find additional papers (i.e., I looked through the 915 articles citing the core). This exercise returned five more papers. Thus, the final selection comprised 38 documents (Figure 1).

Most academic articles on university rankings and their use in research evaluation are critical. Iesbik Valmorbida, Rolim Ensslin et al. (2016) conducted a literature review on university rankings using the ProKnow-C tool (in Portuguese). The authors identified 20 university rankings and primary areas of criticism. The main criticisms involve certain ranking indicators, arbitrary assignment of weights, lack of transparency and reproducibility, and frequent changes in the ranking methodology. The authors also noted the problem of the lack of universality, because different rankings are based on completely different concepts that determine the methodology³. Earlier, Huang (2012) presented an excellent overview of debates around the QS ranking. These two reviews served as a good starting point for this article. At the same time, I would like to note that in addition to supplementing with more recent sources, the scope of the current review is more focused.

The literature review that I conducted highlights five primary areas of criticism:

• Technical errors in the methodology and lack of transparency and reproducibility of rankings.

• Incomplete coverage of university performance indicators and shift towards research.

• Territorial, linguistic, and geographic biases and Matthew effect.

• Risks of losing the national and organizational identity of higher education.

• Conflicts of interests.

Van Raan (2005) argued that the use of bibliometric indicators in the ARWU ranking faced a number of technical and methodological problems. Among the technical problems, the author highlighted errors in the identification of the citing/cited publication, as well as the loss of affiliation due to the plurality of spelling of the university name⁴ (see also Dimzov, Matošić, & Urem, 2021). Technical errors can significantly distort the final ranking given the large weight of citation indicators. Van Raan warned that nonexpert use of rankings based on bibliometric indicators may lead to “a bibliometrically limited view of a complex reality” (van Raan, 2005, p. 134). At the very least, the analysis of publications and citations should consider differences in the traditions of scientific communication between disciplines. The author also pointed to the problematic use of bibliometric data in many areas (engineering, social and behavioral sciences, and especially in the humanities) and the difference in citation levels between various documents. The author argued that if bibliometric data were used to evaluate universities, lists of publications should have been validated by the institutions under evaluation. Van Raan concluded that the ARWU ranking cannot be used, not only for evaluation purposes but even for benchmarking. More globally, the problem of university rankings based on bibliometric indicators comes down to an attempt by administrators of science to reduce everything to simple “fast” indicators.

Ioannidis, Patsopoulos et al. (2007) identified key design challenges for university rankings: adjustment for institutional size, identification of institutions, measurements of average versus measurements of extremes, adjustments for the scientific field, timeframe of measurement, and allocation of credit for excellence. Importantly, Ioannidis et al. drafted their article before the advent of the Leiden Ranking and subject rankings of various producers, in which these challenges were met, at least in part.

Billaut, Bouyssou, and Vincke (2010) reviewed the ARWU criteria and showed several issues:

• A considerable time lag exists between conducting research and awarding Nobel and Fields Prizes. Besides, these awards stand for only a small part of the scientific field spectrum.

• Highly cited researchers tend to be old and have changed universities several times over their careers. Again, there is a time lag, due to which the relationship between the criterion and quality of research in the assessed institution is not obvious.

• The weighting scheme for authors of articles in Nature and Science is illogical.

• The authors highlighted the same technical problems of paper counting in Web of Science as van Raan (2005) did earlier. The number of articles does not say anything about the quality of research. A significant proportion of the articles has not been cited by anyone.

• The indicator “the number of Full Time Equivalent (FTE) academic staff” is not clearly defined, so the Productivity criterion is questionable.

Huang (2012) mainly focused on the shortcomings of the QS rankings associated with the use of reputation questionnaires. The QS methodology suggests that questionnaires can serve as an indicator of a university’s quality, but they are only an indicator of reputation. The return rate and lack of control over the parameters of the experience and qualifications of the respondents cast doubt on the sample’s representativeness. In addition, the author questioned the validity of the Faculty to Student Ratio indicator, because it is easily manipulated by universities. The picture is complemented by an equal full score in the values of individual indicators and unexplained significant fluctuations in the ranking positions.

Pandiella-Dominique, Moreno-Lorente et al. (2018) analyzed the ARWU ranking methodology and highlighted two major flaws: incorrect data retrieval and the problem of reproducibility. The authors proposed to reduce the methodology to only two bibliometric indicators: the number of papers in Nature and Science (N&S) and total number of documents in Web of Science, based on a size-dependent counting method.

Krauskopf (2021) analyzed the ARWU subject ranking methodology and named key issues that need improvement:

• uneven distribution of Web of Science categories across ARWU subjects, and the complete absence of 54 categories in ARWU;

• use of different indicators for different subjects;

• language and territorial bias when completing the Shanghai Ranking’s Academic Excellence Survey; and

• arbitrariness of top journals identification.

Huang, Neylon et al. (2020) conducted a comparative analysis of the main indexing databases (Scopus, Web of Science, and Google Scholar). The analysis revealed discrepancies in bibliometric data, which can have a significant impact at the institutional level. Thus, the use of a single source of bibliometric data (the case of university rankings) is beneficial for some universities and disadvantageous for others. This impact is more significant for low-ranking universities in terms of possible position changes.

Fauzi et al. (2020) and Bellantuono, Monaco et al. (2022) highlighted the arbitrariness of the weights used in the calculation of a composite index. It is a potential source of bias in rankings that are based on a composite index. At the same time, many studies considered technical improvements of ranking methodologies and composite indicators. For example, Benito and Romera (2011) proposed an algorithm for university ranking based on simulation techniques, and applied it to French and German universities involved in excellence initiatives.

Ioannidis et al. (2007) reviewed two ranking methodologies (QS/THE and ARWU) and concluded that none was sufficiently valid to evaluate both research and teaching excellence. Iesbik Valmorbida et al. (2016) also argued that most ranking methodologies are mainly focused on the research activities of universities, and little attention is paid to other important aspects such as teaching, learning, professional development, and university management. García-Berro, Roca et al. (2016) used the example of Universitat Politècnica de Catalunya to show how university rankings directly affect academic evaluation by shifting the focus from teaching to research.

Vernon et al. (2018) concluded that none of the global university rankings covered university performance, although indicators of some rankings might be useful in improving research. U-Multirank offers the most comprehensive methodology; however, the ranking structure itself makes trend analysis difficult. The authors suggested further development of rankings with new indicators, primarily in scientific impact, economic outcomes, and public health impact.

Robinson-Garcia, Torres-Salinas et al. (2019) conducted an analysis of university rankings using principal component analysis and concluded that the results of all analyzed rankings correlate with three bibliometric indicators: publication count, h-index, and Category Normalized Citation Impact (CNCI). Other aspects of ranking methodologies do not significantly affect the final position of universities in the rankings. Thus, rankings are subject to all the shortcomings inherent in bibliometric databases (size dependency, disciplinary biases, Matthew effect).

Van Raan (2005) highlighted U.S. bias in citation data (as well as English language bias) among methodological issues of rankings. Billaut et al. (2010) raised the issue of incomplete coverage and English language bias of Web of Science. Van Raan, van Leeuwen, and Visser (2011) drew attention to the “language effect,” due to which publications in French and German receive significantly fewer citations than publications of French and German researchers in English. This problem also exists in other national languages (Spanish, Russian, and Chinese).

Huang (2012) revealed a correlation between the number of returned reputation questionnaires of a given country and the number of universities in that country in the QS ranking. The division of universities into two groups (the former is present in the domestic and international lists, the latter only in the domestic list) creates an advantage for the former and can be considered as an inconsistency of methodology.

Safón (2013) argued that the basis of all global university rankings is the profile of a U.S. Ivy League university with good reputation and large financial resources. This has been empirically confirmed by an analysis of the top 10, 20, 50, and 100 universities in university rankings, where the majority are consistently U.S. universities. Why do we consider such a university a template for all other universities in the world? Is the definition of world-class university really that simple? Safón and Docampo (2020) explored the impact of reputational bias on the ARWU ranking indicators. The analysis showed that reputational bias affected the number of papers published in Nature and Science but was not present in the case of highly cited researchers. Of course, this phenomenon plays into the hands of top universities. The media usually concentrates only on the “podium” (top 3 or even top 1) of the rankings; the rest of the universities “disappear” from the public discourse (Blasi, Romagnosi, & Bonaccorsi, 2018). Moreover, the impact of online video is positively related to research performance for highly prestigious universities, but not for the rest (Meseguer-Martinez, Ros-Galvez, & Rosa-Garcia, 2019). This “limited visibility” supports the Matthew effect in university rankings: The winners of the rankings further strengthen their reputation and gain an advantage in the future.

Fauzi et al. (2020) also pointed to the presence of territorial and linguistic biases in the rankings, as well as the Matthew reputation effect. Owing to the diversity of universities and their missions, no single ranking can be considered an ideal assessment tool. The authors see a solution in the development of a more holistic ranking methodology. Bellantuono et al. (2022) investigated territorial bias in university rankings and suggested ways to overcome it. The main idea is to divide universities into similarity groups based on multifaceted data using network analysis and to compare the performance of each institution with the expected level based on other universities in the similarity group.

Li (2016) analyzed the impact of university rankings on the institutional reconfiguration of East Asian universities using top universities in Mainland China, Hong Kong, and Japan as examples. The author argued that global university rankings have become a tool for determining the dominance of the Western higher education system. Certainly, a number of Asian universities have been beneficiaries of the promotion in university rankings (remember the formation of the C9 League in China). However, most universities face negative consequences, among which one can single out a departure from national traditions, as well as a shift in emphasis towards research (to the detriment of teaching and service). The use of publications as the main quantitative indicator of productivity opens the door to manipulative and corrupt behavior. The author concluded that Asian universities are yet to find their mission, which will simultaneously meet global challenges and local tasks (this thesis was largely implemented in the World-Class 2.0 project in China).

Hazelkorn and Gibson (2017) considered globalization to be an irreversible phenomenon, so the conflict between the international mission and the nation and community service is in many ways a conflict of “before” and “after.” University rankings and research evaluation procedures often use the same data sources (primarily bibliometric data) but have different methodologies and purposes. Country-specific data are usually unavailable for global rankings. The authors noted the exceptional role of rankings as drivers of excellence initiatives in higher education.

Global university rankings are closely related to excellence initiatives in higher education. For example, Luque-Martínez, Doña-Toledo, and Docampo (2016) noted the positive impact of the Campus de Excelencia Internacional initiative (CIE) on the promotion of Spanish universities in the ARWU ranking. Global university rankings have been a keynote of Russia’s Project 5top100 excellence initiative, making it a good example of the implementation of the neoliberal higher education paradigm (Tsvetkova & Lomer, 2019).

Global university rankings, as part of the archetype of the Western educational system, have played a significant role in the Taiwanese government’s excellence initiatives (Shreeve, 2020). However, promotion in rankings as a company slogan may differ from the real ambitions of the initiative. On the one hand, the use of global university rankings creates a threat to national identity in higher education; on the other hand, the refusal to integrate into the global academic system can be critical for the development of the national system of higher education. Gao and Zheng (2020) focused on the impact of excellence initiatives and new managerialism on the humanities and social sciences (HSS) in Chinese universities. Scholars in HSS are under tremendous pressure to address local issues while simultaneously publishing internationally. The centralization of decisions on the content of research and the choice of source for publication violates academic autonomy. Political and ideological discourses have a considerable influence on the HSS publication landscape.

Marques and Powell (2020), using the example of British Schools of Education, showed that university rankings affected not only the strategic behavior and organizational structures of universities, but also organizational subunits.

In the case of most global university rankings (such as THE, QS, ARWU, and U.S. News & World Report), the league tables are compiled by profit-seeking organizations that generate revenue by selling to universities additional data, services (consulting), and other types of subscription-based content (Lim, 2021). Chirikov (2023) analyzed the ranking positions of 28 Russian universities that contracted Quacquarelli Symonds for the provision of consulting services in 2016–2021. The results showed an anomalous increase in ranking positions, not supported by changes in the characteristics of universities in national statistics. Thus, it can be assumed that QS acts as a bias for itself.

Very few authors favor the use of rankings in research evaluation or are neutral. Most of these studies are cross-country comparisons or an attempt to determine the underlying factors that constitute the statics and dynamics of a university’s position in the rankings. Docampo (2011) considered the ARWU ranking to be a transparent tool for assessing research performance because the ranking was based solely on objective data. The author analyzed the research performance of national higher education systems using principal component analysis (PCA) based on the ARWU ranking. Docampo and Cram (2014) introduced the distinction between score-driven and rank-driven tables. ARWU belongs to the first type, as its final indicator is determined by the actual values of the indicators (taking into account normalization) and not by ranks. These indicators are so coherent that the aggregate scale turns out to be one-dimensional. Using PCA, the authors showed that the first principal component explained most of the variance in the raw scores. Thus, the authors considered the choice of ARWU methodology and indicators to be optimal given the regressive nature of research performance measures. Klumpp (2019) conducted a dynamic longitudinal efficiency analysis of European universities based on THE and Leiden Ranking indicators. Selten, Neylon et al. (2020) explored three major university rankings (ARWU, THE, and QS) and, based on the PCA, concluded that rankings are based on two factors: a university’s reputation and its research performance. At the same time, the choice of indicators of a particular ranking often does not coincide with the stated goals. Pakkan, Sudhakar et al. (2021) reduced the analysis of university rankings to bibliometric variables.

Rankings can be used as a benchmarking tool. Thus, Tuesta, Bolaños-Pizarro et al. (2020) proposed a methodology to identify a group of competitors for a university in three ranking systems (ARWU, THE, and QS). The authors used complex network theory and data envelopment analysis. Such an approach to identifying a group of closest competitors can be used as a decision-making tool. At the same time, the methodology has its limitations (e.g., the static nature of the analysis). The analysis shows a group of competing universities at a certain point in time but does not show those universities that have dropped out of this group, either up or down, which may be of increased interest to university management.

I now look at how university rankings and their use in research evaluation are considered in gray literature. The review is based on an expert selection of policy documents and expert pieces related to global university rankings and their interpretation. The selection of materials was mainly conducted based on an Internet search (search terms “university rankings” and “research evaluation,” manual filtering). Furthermore, I used the Overton database to search for policy documents that cite the research articles identified in the previous section. However, it should be acknowledged that this selection is somewhat arbitrary and subjective. Besides, it may have a natural bias towards policy documents from Europe, as I primarily work in European contexts. The selection can be enriched by the gray literature from other parts of the world in further studies.

For this section, I used practically the same structure as in Section 2: negative vs. positive or neutral attitudes. However, in the case of gray literature, it is possible to trace the evolution of attitudes towards rankings from a more positive view in the 2000s–early 2010s to a completely critical perspective in recent documents. My review is structured in the same way.

In gray literature, positive or neutral views turned out to be rare. I managed to find only a few works where rankings are considered as an objective phenomenon (for obvious reasons, I did not use the marketing materials of the ranking compilers).

In 2002, the International Ranking Expert Group (IREG) was created to develop principles for ranking universities. In 2006, the second IREG conference established the Berlin principles on ranking of higher education institutions (2006), which set rules in four areas: purposes and goals of rankings; design and weighting of indicators; collection and processing of data; and presentation of ranking results. This document is a basic statement for ranking universities; thus, it generally interprets rankings as a self-evident phenomenon. Unfortunately, the framework remained mostly on paper (e.g., not a single ranking known to me “recognizes the diversity of institutions and take the different missions and goals of institutions into account”).

In the following years, rankings were discussed quite regularly in academic news media and also in the popular press, usually in a relatively uncritical way. In 2007, for example, Simon Marginson (“Global University Rankings 2007: Interview with Simon Marginson,” 2007) in a representative interview criticized QS/THE, primarily for the low percentage of returned reputation questionnaires. At the same time, he praised ARWU as an objective measure of university quality. Moran (2010) argued that given the difference in population, the United Kingdom, not the United States, is the leader in higher education. The author uncritically accepted rankings as an objective measure of the performance of university systems.

Guttenplan (2013) believed that despite methodological criticism by the academic community, university rankings cannot be abandoned because of their huge popularity. Luxbacher (2013) acknowledged the flaws and limitations of ranking methodologies but also argued that “there is consensus that global university rankings are here to stay, that no ranking is all encompassing and that their influence is growing.” Usher (2013) emphasized the value of rankings in enabling an objective comparison of universities and in placing higher education at the center of the policy agenda.

Hazelkorn et al. (2014) published the final report on the Rankings in Institutional Strategies and Processes (RISP) project. RISP is a project implemented by the European University Association (EUA) with the support of the EU from 2012 to 2015. Its goal was to understand how universities use rankings and how they, in turn, influence university strategies. The authors acknowledged the exceptional role of rankings in shaping university strategies and rather perceived them as a natural phenomenon. This process affects both external and internal stakeholders. At the same time, it is impossible to single out common reaction patterns. In general, universities use rankings

• as a source of information;

• for benchmarking purposes;

• as evidence for decision-making; and

• as a marketing promotion tool.

In 2010, the League of European Research Universities (LERU) critically reviewed two European Commission initiatives (Boulton, Maes, & van Asten, 2010): U-Multirank and U-Map (U-Map, n.d.). Neither initiative allows transparent comparisons between countries. At the same time, the U-Map was rated more positively as an attempt to assess the activities rather than the quality of universities based on “monotonous tables.” Thus, the U-Map reflected the diversity of universities to a greater extent.

Since 2010, there has been growing concern in the academic community about the responsible use of quantitative metrics for research evaluation. This concern has resulted in a few significant interventions (Hicks, Wouters et al., 2015; San Francisco Declaration on Research Assessment (DORA), 2013; Wilsdon, Allen et al., 2015) and eventually culminated in a professional movement for research assessment reform (Rushforth & Hammarfelt, 2022). The growing criticism of global university rankings and their negative role in the evaluation of research can be considered as one of the areas of concern addressed by this movement.

In 2011 and 2013, the EUA published two reports on university rankings (Rauhvargers, 2011, 2013). While acknowledging the potential benefits of rankings in fostering healthy competition and international collaboration, the reports raised concerns about their limitations. They emphasized that rankings should be seen as tools rather than definitive measures of quality, as they often prioritize research outputs and internationalization while overlooking other important aspects of universities’ roles, such as societal impact and teaching excellence. The reports suggested that universities should approach rankings critically and focus on their unique missions, and advocated for more comprehensive assessment frameworks that reflect the diverse goals and contributions of higher education institutions. It is interesting to note that these reports are much more critical towards rankings than the abovementioned report by Hazelkorn et al. (2014) that was also commissioned by the EUA.

Holmes (2015) highlighted that major ranking compilers experience serious problems with maintaining the balance between popularity and stakeholders’ interest, on the one hand, and reliability and validity on the other. Yudkevich, Altbach, and Rumbley (2015) compared university rankings to the Olympics, which is a zero-sum game. This approach to evaluation is suitable for athletes, but not suitable for assessing the complex social constructions that are universities. Therefore, governments and universities themselves must approach the ranking game with caution.

Brankovic (2020) highlighted how rankings have become a powerful tool in various industries, from education to sports. The author urged readers to approach rankings critically, questioning their methodologies and understanding the potential biases that may affect the results.

Waltman, Wouters, and van Eck (2020) suggested 10 principles of ranking university responsibly. It is worth mentioning that all three authors represent the Centre for Science and Technology Studies of Leiden University, which is the producer of the Leiden Ranking. These principles are grouped into three areas: design, use, and interpretation of university rankings.

The International Network of Research Management Societies (INORMS) aims at information exchange and sharing best practices in research management across communities. The INORMS Research Evaluation Working Group (2022) proposed a set of principles for responsible ranking, grouped into four domains: good governance, transparency, measure what matters, and rigor. In October 2022, INORMS launched the More Than Our Rank initiative (INORMS, 2022), which was developed in response to problems in the use of university rankings. Any ranking methodology has limitations, and a university is much more than a set of indicators. In an opinion piece, Gadd (2021), one of the coordinators of More Than Our Rank, compared university rankings to GDP as a single measure of economic growth. She argued that just as GDP does not reflect all dimensions of social welfare (e.g., poverty, wealth, social stratification, hunger), university rankings do not reflect all dimensions of the performance of universities. The funds allocated by governments under policy initiatives of excellence are used to advance in the rankings, but not to create strong universities. Global rankings stimulate inequality both between countries and within one country. Ranking indicators do not correspond to the missions of universities and the mission of higher education in general. Only a united movement against the irresponsible use of university rankings in research evaluation can change the situation.

In 2022, the Harnessing the Metric Tide report was released (Curry, Gadd, & Wilsdon, 2022), which critically revisited the findings of the 2015 report (Wilsdon et al., 2015). This report was intended to supply material for discussion of the UK Research Excellence Framework (REF). One of the report’s headline recommendations is to critically rethink the role of university rankings in research assessment and to evaluate their impact on the research culture of UK universities.

A big step towards research assessment reform in Europe was the development of the Agreement on Reforming Research Assessment (CoARA, 2022), which, as of April 9, 2023, has been signed by 513 universities. One of the commitments in the agreement states that the use of rankings should be avoided in research assessment. At the same time, the agreement admits that rankings may be used for benchmarking purposes, but in such a case the limitations of the methodology should be acknowledged.

In February 2023, the United Nations University published a report entitled World-class university? Interrogating the biases and coloniality of global university rankings (Nassiri-Ansari & McCoy, 2023). This report summarized the major flaws and limitations of rankings; thus, it brought together the perspectives between academic and gray literature. The report was prepared in Asia, and it distinguishes itself from European documents by a strongly expressed colonial discourse in evaluating the impact of rankings on higher education. The authors of the report argued that major global university rankings “reflect a colonial hierarchy, with well-endowed and historically privileged universities from the Global North dominating” (Nassiri-Ansari & McCoy, 2023, p. 18).

A recent report published by Universities of the Netherlands (Ranking the University: On the Effects of Rankings on the Academic Community and How to Overcome Them, 2023) provided specific recommendations for cultural change in the use of university rankings, focusing mainly one-dimensional league tables. The proposed strategy for achieving the desired cultural change involves implementing initiatives at three distinct levels. Firstly, in the short term, universities need to undertake initiatives individually. Secondly, in the medium term, there need to be coordinated initiatives implemented at the national level, for example joint initiatives by all universities in the Netherlands. Finally, long-term, coordinated initiatives must be set up at the international level, for example at the European level.

The most recent EUA briefing takes a more critical approach to university rankings (compared to previous EUA reports), though it does not call for an immediate rejection of them (European University Association, 2023). Interestingly, this document suggests avoiding rankings in research assessments, referring to the CoARA Agreement. Universities have a responsibility to educate users on the responsible use of rankings, according to the briefing.

In 2013–2020, the Russian government implemented a large-scale excellence initiative in higher education, which was focused almost exclusively on promotion in university rankings. The very name of the political initiative (5top100) reveals the main goal—five Russian universities were to enter the top 100 of global institutional rankings by 2020. Already in the course of implementation, it became clear that the goal was unachievable; however, the rankings somehow played a key role in the assessment of participating universities’ progress within the framework of this project.

I used the Russian Index of Scientific Citation (RISC) to identify relevant academic literature in the largest database of publications in Russian. The search was conducted only by the document type “journal article” for the key phrases “rankings of universities” and “university rankings” considering morphology. The query returned 199 responses. In the next stage, I filtered out publications that did not belong to global rankings or described the case of a particular university as well as anonymous publications and translations into Russian. I also selected only those papers that address the use of global university rankings in research evaluation, especially in the context of excellence initiatives⁵. The final sample consisted of only 14 publications. This number may seem exceedingly small, but the mainstream Russian academic discourse considers global university rankings as a tool for assessing and improving the competitiveness of both universities and countries (Leonova, Malanicheva, & Malanicheva, 2017; Puzatykh, 2019).

Several authors have studied the problems of increasing the competitiveness of Russian universities and the national higher education system as a whole in the context of implementing the Project 5top100 excellence initiative (Arefiev, 2014, 2015; Guzikova & Plotnikova, 2014; Kushneva, Rudskaya, & Fersman, 2014). This is quite natural because the main goal of this initiative was the entry of five Russian universities into the top 100 of global rankings. In other words, rankings are considered an indicator of competitiveness. In the literature, there is also the concept of a “world-class university,” which is also viewed through the prism of university rankings (Pankova, 2015). In general, such studies aim to identify common characteristics of universities in the top 30 (50, 100) of global university rankings (Nikolenko, Vyalkov et al., 2014). Based on the 2008 THE ranking, Milkevich (2008) identified the key components of a world-class university:

• the level of faculty;

• the best students; and

• individual approach to the education process.

Efimov and Lapteva (2017) expanded the concept of “world-class university” by introducing the concept of “frontier university.” The authors defined the concept as “a university that operates at the forefront of development processes: new areas of knowledge, modern technologies; social development and human development” (p. 7). The authors referred to universities such as the University of Berlin, Higher School of Economics, Singularity University, and some others. The authors also proposed a typology for university rankings:

• Model 1. Ranking based on the ideal of an academic university with a focus on basic science (QS, ARWU, USNWR).

• Model 2. Ranking based on the ideal of the university as a center of higher education (Russian ranking Interfax).

• Model 3. Ranking based on the ideal of the university as a partner for business—a “workforce factory,” a technology and innovation development center (the authors gave the example of Professional Ranking of World Universities, and I would also add the Forbes ranking).

• Model 4. Ranking based on the ideal of the university as a “social elevator” and center for socially significant projects (ranking of the Washington Monthly magazine).

Against the backdrop of the geopolitical situation and sanctions of Western countries, the position of developing a specific ranking for Russia is becoming increasingly widespread. Bolsherotov (2013) pointed to the initially losing position of Russia in the “ranking game.” The author blamed global rankings for US bias in evaluation. Besides, the well-established tradition of publishing results in the national language, as well as the “removal” of the scientific sector from higher education, has an effect. The author suggested not sending the data of Russian universities to international rankings and not taking them into account. In later articles, Bolsherotov did not show such radical positions. For example, in 2020, he provided just an overview of three global university rankings (ARWU, THE, and QS) and analyzed case studies of successful universities (Bolsherotov, 2020). In particular, the author listed the California Institute of Technology and the role of investment by Chinese companies in promoting national universities in international rankings.

Lazar (2019) criticized the excessive bias towards quantitative performance indicators and the bureaucratization of science and higher education in general, and the use of rankings to assess the effectiveness (performance) of universities in particular. Lazar’s article is more of an emotional journalistic piece than a scientific one. Nevertheless, the author made an important remark on the differences between the Russian and Western systems of science and higher education. In the West, science and higher education originally coexisted in universities, and applied research developed through the active participation of businesses. In Russia, since the times of the USSR, there has been a “research triad”—the Academy of Sciences, industry design bureaus, and research institutes—while universities and industry institutes have been mainly engaged in training personnel with budget funding. Therefore, historically, both fundamental and applied research has largely moved outside universities. Over the past 10 years, there have been several attempts to break this system and return science to universities (the reform of the Russian Academy of Sciences, the creation of consortiums of universities, the Academy of Sciences, and industry within the framework of the new Russian program of academic excellence Priority 2030, etc), but a change in rooted institutional practice takes more time.

Pyatenko (2019) argued that global university rankings do not reflect the quality of education, but the advantage of one educational system (Western) over all others. Due to the limited number of bibliometric sources, there is an English-language bias in the assessment. Besides, the author noted that high scientific performance did not necessarily correspond to high education quality. Moreover, the rankings do not correlate with the indicators of demand for graduates by employers.

In this section, I am moving from academic literature to gray literature in Russian. The results of the implementation of the 5top100 project and the promotion of national universities in global university rankings have occupied a principal place in Russian public discourse in recent years. These topics were widely covered by leading business media. Thus, Forbes Education (2020) collected the opinions of experts about the program. It is not surprising that the majority of opinions turned out to be positive, given that most of the interviewed experts were participants of Project 5top100 or its successor Priority 2030 (for more details on this program, see Kochetkov, 2022). At the same time, the Vice-Rector of the private Russian Economic School, Zarema Kasabieva, noted that the project also had negative aspects, among which the expert emphasized the absence of a ripple effect on the entire national higher education system. The report of the Accounts Chamber on the results of the implementation of 5top100 caused a significant resonance (Bulletin of the Accounts Chamber Vol. 2 (279), 2021). Among the positive effects, the authors of the report noted the increase in the number of publications of the universities taking part in the project, as well as their promotion in global university rankings, primarily field ones. The author of the report, auditor Dmitry Zaitsev, believes that the project caused a radical rethinking by universities of their role, functions, and tasks. At the same time, the author had to admit that the main goal of the project, which was originally formulated as the entry of five Russian universities into the top100 of institutional rankings, was not achieved.

My objective is to present a comprehensive survey of the literature covering the use of university rankings in research evaluation and excellence initiatives. In Section 2, I have analyzed the relevant academic literature retrieved from Web of Science (mostly in English with one paper in Portuguese and two papers in Spanish). Most of the literature in this section consists of critiques of global university rankings. Section 3 reviews expert-selected gray literature. Finally, Section 4 is a review of relevant literature in Russian (with one paper in Ukrainian indexed in RISC). Mostly, this is academic literature, but some gray literature in Russian is included as well.

My analysis showed that under the guise of apparent transparency and objectivity, there are a lot of shortcomings that can positively affect a small group of elite universities and at the same time negatively affect the whole higher education sector and society in general.

This section concludes the review and at the same time discusses the results. I have summarized the main insights from the literature covered in the review in Figure 2.

It was a surprise to me that the various sections of the review (English language vs. Russian language, academic literature vs. gray literature) have only one single thesis at the intersection of all the perspectives (i.e., threat to national identities in higher education). The English-language sources cover the widest range of problems. At the same time, some authors have a positive or neutral view of global university rankings. Rankings are often seen as a source of “objective” data that can be used to compare countries and universities. However, authors do not always disclose the limitations associated with using purely quantitative data. Many contributions to the literature focus on fixing technical details of the ranking methodology, while the rankings themselves, and their interpretation and use in research evaluation, are not discussed in any way. Some researchers note the role of university rankings in increasing publication activities of universities and their integration into the global scientific and educational space.

It is interesting to note that over the past two decades, the attitude towards global university rankings in gray literature has undergone a significant transformation. In the 2000s and early 2010s, there used to be a positive view of university rankings, with many policymakers and some academics endorsing, at least to some extent, the use of rankings as a tool to measure the quality of higher education institutions globally (“Global University Rankings 2007: Interview with Simon Marginson,” 2007; Guttenplan, 2013; Luxbacher, 2013; Moran, 2010; Usher, 2013). Marginson criticizes QS/THE for low response rates but praises ARWU; Moran (2010), informed by rankings and statistical data, claims that the United Kingdom, not the United States, is a leader in higher education; Guttenplan (2013) believes rankings are popular despite criticism; Luxbacher (2013) acknowledges flaws but states rankings are here to stay; and Usher (2013) sees rankings as a way to objectively compare universities and prioritize higher education in policy. However, in recent years, the perspective on university rankings in gray literature has become more critical. For example, reports commissioned by the EUA have changed their perspective on rankings over time, from neutral and slightly positive to more critical. There is growing concern about the limitations of rankings, their flawed methodologies, and the negative consequences of relying too heavily on rankings. A few documents have highlighted the need for a responsible use of rankings and have warned against the misuse of rankings to make decisions that could have far-reaching implications on education policy. Instead, the gray literature calls for multiple performance measures that capture various dimensions of university excellence.

Most of the recent gray literature is in agreement with the authors of research articles in a negative (sometimes neutral) attitude towards global university rankings. Here, authors also consider ranking design issues. One of the main points of intersection between academic and gray literature is the emphasis on lack of transparency and reproducibility of rankings. Even those rankings that use a sound methodology (Leiden Ranking, U-Multirank) do not meet these requirements, because the raw data are closed due to the provider’s license (Web of Science) and the methodology is too complex for the average user to replicate on their own. However, the main focus of gray literature is on interpretation and use. Most opinions agree that current practice needs to be critically rethought.

Russian public discourse intersects very closely with academic literature in Russian, but the range of issues discussed is extremely narrow. In addition, both academic and gray literature in Russian overlap extraordinarily little with English-language sources. Unfortunately, this suggests that Russian science and educational policy exist in some kind of isolated space, which, of course, negatively affects development prospects.

The idea of increasing the number of publications to integrate into the global educational space intersects with Russian-language literature. The focus of literature in Russian is significantly more one dimensional than the reviewed English-language sources. The vast majority of authors consider university rankings in a discourse of competitiveness; accordingly, a university’s entry into a global university ranking is taken as an indicator of competitiveness. High positions in rankings allow a university to attract the best students and professors (including from abroad) and to receive additional funding through government excellence initiatives (e.g., Project 5top100). Positions in rankings also increase the value of a diploma in the eyes of employers, although the latter thesis is questioned periodically. It is important to note that some Russian researchers are promoting the thesis of rankings as a weapon of “soft power” to promote Western values in higher education, somewhat in line with discussions about “risks to national identity” in English-language literature. Recently, these thoughts have been increasingly heard in public discussions, and this cannot but cause concern, because such ideas lead, in fact, not to a revision of research evaluation practices, but to isolationism.

The way in which rankings are compiled and used is still far removed from the good practices promoted by statements on responsible use of rankings. The use and interpretation of rankings are often believed to be outside the responsibility of ranking compilers. However, some ranking compilers promote their rankings in quite aggressive ways. This aggressive promotion includes dissemination of a simplistic approach to interpreting their products, which is taken on board now by many university administrators and governments⁶. Therefore, the discussion of rankings and their use (especially in the context of research evaluation and policy initiatives) should involve not only ranking compilers and universities, but also the general public and academics. It is extremely important to involve governments in this discussion, because in many countries the higher education system is predominantly state-owned, and it is government bodies that form the assessment standards. This is especially true for East Asian countries and Russia.

What should be done in this context? Commitment 4 in the Agreement on Reforming Research Assessment explicitly states that rankings should not be used in the evaluation of research (CoARA, 2022). However, only institutions sign the agreement. Insufficient government involvement could potentially limit the implementation of the agreement. Besides, it must be considered that in many countries the higher education system is predominantly public and most of the funding depends on governments. At the same time, government intervention may be seen as a violation of university autonomy. This is a complex issue that does not have a clear answer, but we must acknowledge this complexity.

Moreover, the complexity of rejecting rankings must be acknowledged. Rankings are uncritically taken by too many as an indicator of, if not quality, then reputation. Therefore, the refusal of a university to be present in the rankings carries serious reputational risks. The marketing function of rankings makes them hard to refuse. In addition, most of the design and interpretation problems are associated with the use of composite indicators, which combine various aspects of a university’s performance in an arbitrary way. Rankings free from this issue can be used for benchmarking purposes (e.g., Leiden Ranking and U-Multirank). Therefore, the process of rejecting rankings must be evolutionary, not revolutionary.

Despite the difficulties of the transition, it is particularly important to take concrete steps. At the end of 2022, some of the prestigious U.S schools of law and medicine announced their withdrawal from the U.S. News & World Report (Hamann & Ringel, 2023). This, of course, is not yet the end of the “ranking power,” but such steps, coupled with the ongoing criticism from the academic community, are already forcing ranking compilers to make climbdowns in terms of improving the methodology⁷. Only in this way will the community regain control over the academic environment, which is increasingly passing into commercial organizations’ hands.

The rejection of rankings goes beyond legal or political aspects; it is a cultural change. Cultural change refers to the transformation and evolution of beliefs, values, customs, and practices within a society or a particular group over time (Varnum & Grossmann, 2017). Collective action must be taken at all levels, from individual researchers to governments and supranational organizations. If we all stop perceiving rankings in a certain way, they will no longer determine the evaluation of research. Therefore, I would focus on several specific stops:

• Stop evaluating academics based on university ranking indicators. Start rewarding the contributions of faculty and researchers in all areas of university activity.

• Stop constructing university strategies based on university rankings. Do not use ranking tiers in analytical reports for management decision-making; instead, focus on the actual contributions made by a university (scientific, educational, and societal).

• Stop evaluating universities based on ranking indicators. Every university has a unique mission, and only fulfillment of this mission really matters.

• Stop using ranking information in national strategies and other kinds of ambitions. Only universities’ contributions to national and global goals should be considered.

• Stop paying money for consulting services to ranking compliers. This is a pure conflict of interests.

I would like to thank my wife Irina, who has always acted as the first critic of my papers and tolerated me throughout the entire process of writing this paper. With my advisor Ludo Waltman, we have passed through dozens of R&R rounds during the preparation of this manuscript. Ludo’s contribution to this study is invaluable. I also want to thank all the members of the Evaluation & Culture focal area at CWTS for their recommendations that made this work much better.

The author is affiliated with the Centre for Science and Technology Studies (CWTS) at Leiden University, which is the producer of the CWTS Leiden Ranking and is engaged in U-Multirank. The author is also a former employee of the Ministry of Science and Higher Education of the Russian Federation that implemented the Project 5top100 excellence initiative. No confidential information received by the author during the period of public service in the Ministry was used in this paper.

The APC for publishing this article in open access was paid by the Centre for Science and Technology Studies of Leiden University.

Not applicable.