Performance-based publisher ratings and the visibility/impact of books: Small fish in a big pond, or big fish in a small pond?

The evaluation of scholarly books is a growing theme across Europe, mainly because of their significance to SSH scholarly communities (Zuccala & Robinson-Garcia, 2019) but also due to their registration in national databases (Giménez-Toledo, Mañana-Rodríguez et al., 2016). This has inspired a number of scholars to investigate book publishing more intensively, often in relation to national performance-based research funding systems (PRFS) (Zuccala, Giménez-Toledo, & Peruginelli, 2018). Some have investigated the prestige of publishers (Giménez-Toledo & Román-Román, 2009) as well as their specialization (Mannana-Rodriguez & Giménez-Toledo, 2018), and others have explained how peer-review labels in books can be used as an aid to evaluation procedures (Kulczycki, Rozkosz et al., 2019; Verleysen & Engels, 2014).

Peer review is as vital to book publishing as it is to the production of journal articles; hence, the need to identify “quality” publishing outlets (i.e., those adhering to review standards) is becoming a big incentive for establishing a European-wide publisher registry (Giménez-Toledo, Sivertsen, & Mañana-Rodríguez, 2019). Before we introduce the purpose and potential design of this proposed registry (in Section 3), we begin with a brief explanation of what we call a top-down (performance-based) versus bottom-up (individual level) approach to the evaluation of scholarly books. Following this, we introduce our study framework and research hypotheses.

A top-down versus bottom-up evaluation can be explained on the basis of what is known currently about evaluations in general, and about research articles. For instance, a bottom-up approach to evaluation starts at a low level of aggregation, where the lowest unit is the research paper, or the oeuvre of a particular researcher. Conversely, we can also take a top-down approach, or higher level of aggregation, where the focus is on a department, university, or journal (van Leeuwen, 2007). Different approaches lead to different insights into scholarly performance and impact (e.g., Zuccala, Costas, & van Leeuwen, 2010).

At the aggregate level of a journal, the most commonly used proxy for “quality” is the journal impact factor (JIF). However, it is well known that the JIF is a relatively poor predictor of the citation impact for individual articles (Hegarty & Walton, 2012). According to Pudovkin (2018) it is “a great mistake to consider the JIF value as a proxy for the citedness of individual papers in a journal” or “as a proxy for the influence of an author” (p. 1). Each research paper is likely to achieve its own degree of impact regardless of the journal in which it was published, with a “minority receiving a high number of citations, and a majority of documents receiving the few remaining citations” (Sugimoto & Larivière, 2018, p. 96). Whilst Waltman and Traag (2021) statistically show that the impact factor can be an accurate indicator of the value of an individual article (i.e., more than the number of citations it has received), emphasis is still placed on this indicator’s sociotechnical implications, not the statistics.

In the PRFS of the Nordic countries, specifically Norway, Denmark, and Finland, attention has been given to the development of journal lists with level assignments as opposed to JIFs or other journal-level indicators. Yet similar to the JIF, even if a correlation is found at the macrolevel between journal level ratings and citation scores, there are still many highly cited individual articles in lower rated journals, and articles without citations in higher level journals (Aksnes, 2017; Auranen & Pölönen, 2012).

In the case of book publishers, there is no established “publisher impact factor” because commercial citation databases (i.e., Web of Science/Scopus) do not accurately index publishers, international monographs, and book chapters. Journals are also far more homogeneous as publication channels than publishers are for books. To account for book publications, many European nations have, like journal ratings, developed publisher level ratings for their PRFSs (Hicks, 2012). In Denmark, Norway, and Finland, book publishers are assigned to level categories by experts in a given field (e.g., levels 0 to 3), and the levels determine the number of performance points (e.g., level 1 = 5 points) (Sivertsen, 2016). All points are allocated on an institutional level, that is, to a university, which in turn leads to performance-based funding from the country’s government. As indicators, these ratings are complementary to journal ratings, and also comparable to the JIF in the sense that they attempt to account for differences in the average “quality” of outputs at the level of the publication channel (see more in Section 2.2).

Little is known; however, about the extent to which publisher levels relate to the visibility or impact of individual books. Accordingly, we introduce a small fish in a big pond versus big fish in a small pond metaphor, where a “fish” is a book and “the pond” represents its publishing house (see Figure 1). Utilizing this metaphor, we investigate book titles:

• that were published by a variety of presses and publishing houses; some from the “big pond” of level 2 to level 3 publishing (presumably international), and others from the “small pond” of level 0 to level 1 publishing (presumably national or regional), and

• that can be evaluated according to different measures (indicators) of an individual book’s overall visibility and impact as a “big fish” or “small fish.” The indicators that we have retrieved include WorldCat holding and country distribution counts, Google Scholar and Scopus citations, and PlumX usage, captures, mentions, and social media counts.

Our assumption, based on data from three Nordic PRFS databases (Denmark, Norway, and Finland), is that there can be smaller fish books from the big pond of publishing as well as bigger fish books in the small pond of publishing. Figure 1 illustrates this on two axes, with high visibility or impact versus lower visibility or impact along the vertical axis, and publisher levels from 0 to 2 (national/regional) versus levels 2 to 3 (international) on the horizontal axis. Note that, within these categories, books will, in reality, exist on a continuum. We also expect that the publisher level, like the JIF, will be a limited predictor of an individual book’s impact and visibility.

This research has implications for the use of book publisher ratings in research evaluation, where the book as scholarly output is valued according to its publisher. A book’s value, however, can also be determined by an expert review of its content, by librarians who select them for library collections, and through the application of alternative metrics. There are considerable implications for researchers as well. Many are concerned with the overall visibility, perceived scholarly/cultural relevance, and impact of their book titles. With this, there are three primary aims to this study.

The first aim is to determine the degree to which a PRFS rating is related to an individual book title’s visibility or impact. The second aim is to identify the characteristics of a publisher or book, in addition to the PRFS rating, which may play a role in a book’s visibility or impact. And finally, the third aim is to provide new insight into the worldwide library distribution potential of publishers, as they fit into the overall schema of PRFSs. Some of what is clarified about performance-based systems, via our individual level analyses, might help to identify the essential criteria for a publisher’s inclusion in the registry. Currently established as well as future PRFSs might also benefit from this registry’s development.

For the scientist preparing a research article, emphasis is often placed on selecting a “high-impact” journal. The choice of publisher is equally as important to the scholar who writes a book. Yet, the prestige, specialization, or quality of a scholarly book publisher is not easily recognized, and has been less transparent to evaluation communities than the accounting and ranking systems currently in place for journals (e.g., the JIF, SNIP, SJR; Archambault & Larivière, 2009; Garfield, 2006; González-Pereira, Guerrero-Bote, & Moya-Anegón, 2010; Moed, 2011). Moreover, in some fields a consensus regarding publisher prestige, specialization, and quality can be reached, while in others not. As publishers’ names are not adequately standardized, categorized, or ranked in sources such as the Book Citation Index, or in other commercial databases (e.g., Scopus), it is therefore useful to review how the book publishing industry works.

Thomson’s (2005) volume Books in the digital age provides a thorough overview, though it focuses primarily on English-language publishing across Great Britain and the United States. Little is known about scholarly book publishing from a broader international or multilanguage perspective. What the volume does do, however, is convey how prestige, specialization, and quality cannot be easily extricated from other factors, such as a publisher’s economic status, historical context, approach to commercialization or lack of commercialization, and symbolic capital. There is and continues to be a difference between the directives of a university press versus a commercial publisher, with the latter “subject to the commercial constraints of the organization and the financial objectives set by its management” (Thomson, 2005, p. 87). Details pertaining to these different units could potentially be added to a European publisher registry, and this is discussed in more detail in Section 3. First, we explore some of the background literature relevant to the registry objective.

Publisher prestige is often described as a feature that scholars intuitively recognize, if not think about, when preparing a new book. Studies designed to investigate this concept have either utilized interviews or distributed surveys to academic communities concerning scholars’ rating of specific publishers (Garand & Giles, 2011; Giménez-Toledo & Román-Román, 2009; Goodson, Dillman, & Hira, 1999). Research by Goodson et al. (1999) found that interviewees were candid about their perceptions, explaining that there can be “a decided ‘unwritten’ hierarchy of publishers, not only in political science, indeed, not only in academia” (p. 257). With this notion of prestige, there is often an implied ranking. A publishing house that is at the “top” of the hierarchy is usually prestigious, but how a publisher is perceived in the minds of scholars might not necessarily turn up in the exploration of hard data. Zuccala, Guns et al. (2015) have therefore studied this by examining how citations could potentially be the focus of a publisher ranking exercise. Their research has shown that citations do, to some extent, reflect prestige, though they can also indicate the economic strength of the publishing house, based on revenues and international status (e.g., a head office in one country, and subsidiaries in others).

The most lucrative, historically famous, and productive publishing houses, across many scholarly subjects, produce books that are cited frequently, with the corollary that works published by smaller publishers might almost never be cited at all (Torres-Salinas et al., 2014). This is because publishing houses in general tend to be concentrated. In fact, the largest and more prolific publishers of journals are now concentrated in an oligopoly (Larivière, Haustein, & Mongeon, 2015). Research by Guns (2018) found a similar tendency with publishers of peer-reviewed books registered in the Flemish VABB-SHW. Still, when assessing book publishers who dominate market share—such as Cambridge University Press, Harvard University Press, or Oxford University Press—rank changes have been observed (at least in the field of history), with a shift from overall citation counts to average citation counts per book (Zuccala et al., 2015). Certain books and published topics are thus likely to receive more attention than others, even when produced by a top publisher.

Specialization offers a different perspective on publishing and can occur in environments of interorganizational rivalry. To avoid unnecessary rivalry, some publishers establish a market niche by producing texts for specific academic fields and subjects or install multiple imprints. Brill, for example, is a Leiden-based publishing company, known for producing social sciences and humanities books (in history, philosophy, religious studies, theology and world Christianity, literature and cultural studies, etc.).

A publisher may also become specialized or differentiated for other reasons. Thomson (2005) describes how AC2, a medium-sized academic publisher from the United Kingdom, experienced a decline in scholarly monograph sales and thus focused on cutting costs by both simplifying and standardizing production processes. A separate organizational unit was developed at this firm, which was designed to focus “exclusively on publishing monographs”: a strategy known as “sequestering the monograph” (Thomson, 2005, p. 163).

For collection development librarians in particular, publisher specialization is said to be problematic (Kousha & Thelwall, 2015; Metz & Stemmer, 1996). According to Metz and Stemmer “judgements are subjective and vary greatly from one library to another.” Moreover, “publishers are so specialized that only those familiar with a field can evaluate their work” (p. 246).

Thus far, little research attention has been given to the link between publisher specialization and research evaluation (e.g., Giménez-Toledo, Mañana-Rodríguez, & Tejada-Artigas, 2015; Mannana-Rodriguez & Giménez-Toledo, 2018), though scholars primarily understand that “few presses can successfully publish in all fields” (Parsons, 1990, p. 3). However, when comparisons are made between commercial publishers and university presses, it is the latter that are less likely to become fully specialized. University presses in general cannot afford to be specialized when required to serve the research areas or fields covered by their home institutions (Lockett & Speicher, 2016; Mannana-Rodriguez & Giménez-Toledo, 2018). At the same time, there are drawbacks, if not challenges, associated with the opposite “multidisciplinary” publishing approach, many of which relate to the editor’s role. When editors can successfully concentrate on books from specialized research areas, they can “become more knowledgeable concerning (new research) developments,” build reputations that will attract the best authors, and grow “more effective in cultivating and nurturing personal acquaintance with scholars” from specialty areas (Parsons, 1990, p. 4).

To find out if university presses are indeed less specialized, or “multidisciplinary,” Mannana-Rodriguez and Giménez-Toledo (2018) developed and applied to a set of Spanish publishers (n = 1952), what they call an “entropy-based indicator” (p. 19). All publishers in the list, which was downloaded from the DILVE database (developed by the Federation of Publishers’ Guilds of Spain), had been producing texts for the social sciences and humanities. Moreover, all, including commercial and university presses, were tested for their specialization on the basis of the unevenness of their distribution of titles across various disciplines. The following explanation was given:

The results of this study showed that many commercial publishers in Spain tended to have a very high concentration of titles by field of knowledge. This was especially true for those producing books for fields such as law and education. With other research fields, production processes tended to be less concentrated (e.g., fields such as Archaeology, Library and Information Studies, Anthropology). As expected, Mannana-Rodriguez and Giménez-Toledo (2018) found that many university presses from Spain were significantly more multidisciplinary than their commercial counterparts. While the authors recognize that the “mission” of a university press is to represent the intellectual range of interests of its home institution’s researchers, they also suggest that with “efficient decision-making processes” many presses might “benefit from the opportunities brought by specialization” (Mannana-Rodriguez & Giménez-Toledo, 2018, p. 28).

The term quality when applied to books or monographs, is based on “the overall intellectual and editorial quality of a publisher’s monographic offerings, reflecting the expertise of typical authors; the persuasiveness of evidence; the intellectual level of discourse; the tendency of a publisher’s titles to be influential in their fields; and the degree of editorial care” (Metz & Stemmer, 1996). While “editorial care” has always been the sine qua non for book quality, it is only recently that scholars have become more concerned with peer review, and isolating review procedures as a standard of quality for book publishers. To this end, there has been a movement towards peer review stamping or labeling, given that “there is no straightforward way to decide whether [books] have been subjected to [Peer Review]” (Verleysen & Engels, 2013, p. 428). The impetus for labeling books stems from the fact that “virtually all high-esteem journals use some type of [Peer Review] to assure content quality of published articles” (p. 428). In this sense the production of a new book is likened to a research article, with all “double-blind, single blind, open, and signed variants” that are “regarded by many as the quintessential mechanism to safeguard academic publishing standards” for journals (p. 428).

In Flanders, which is the Northern Dutch-speaking region of Belgium, it was decided that to achieve a comprehensive coverage of regional academic outputs in the Flemish Academic Bibliographic Database for the Social Sciences and Humanities (VABB-SHW), both books and articles should be included. Thus, with the inclusion of books it was considered necessary to know which book publishers were adhering to the criteria set forth in the legislation, including the use of peer review. A limited number of mostly international publishers are currently considered to use peer review, but in 2010 the Publishers Association of Flanders “introduced a quality label for individual books guaranteeing the peer-reviewed character of their contents, the so-called “GPRC label” (Benelux Trademark No. 0916696)” (Verleysen & Engels, 2013, p. 429). In Finland, the Federation of Finnish Learned Societies (TSV) has also developed a similar label, whereby scholarly publishers indicate themselves which of their books and articles have been peer reviewed (Kulczycki et al., 2019).

Although individual-level indicators for articles and authors have been studied frequently (Wildegaard, 2019), there is less certainty about the reliability of indicators at the individual level of a book (Zuccala & Robinson-Garcia, 2019, pp. 720–723). Even if citation counts accrue for books, there has not been any formal measure of a citation window. We do not know the time period in which books receive their peak citations, or if and when there is a normal decline. Moreover, the same problem with correlating journal article citations and social media counts holds true for books (see Thelwall, Haustein et al., 2013). Social media mentions occur more rapidly than citations, and thus each measures different phenomena: visibility with the former (Leonardi, 2014), and impact with the latter (Waltman, 2016).

The growth and development of individual-level book indicators depends also on how well data sources are compiled and updated, and how reliable they are for extracting “clean” counts for producing valid statistics. If the databases are incomplete or inaccurate, an indicator is less likely to be a sound proxy for the concept it measures. We need to test and retest counts or statistics for books at an individual level, with the hope that data sources are improving. For instance, it has been known for a long time that books, in comparison to journal articles, represent a lower percentage of what is indexed on social media platforms. However, “as of July 2018, Altmetric.com has tracked attention for more than 829,000 books and 80,000 book chapters across a wide range of subjects” (Konkiel & Adie, 2018, p. 2). Similar to journal articles, statistics show that “more than 70% of [mentions to books] occur on Twitter” (p. 3). At the time of this study, we did not have access to Altmetric.com, and thus chose to use PlumX with the understanding that it is of similar value (Torres-Salinas et al., 2017a, 2017b).

Denmark (in 2011), Finland (in 2012), and Norway (in 2005) have all introduced PRFS where part of the annual core-funding from each state is allocated to universities based on research output (Hicks, 2012; Norwegian Association of Higher Education Institutions, 2004; Uddannelses-og-Forskningsministeriet, 2018). In all of these countries, institutional publication data are used at a national level, because unlike the commercial databases (i.e., Web of Science and Scopus), national databases and Current Research Information systems (CRIS systems) support a more comprehensive coverage of outputs in all publication languages.

For all three countries, the PRFS relies on research communities to identify peer-reviewed publications channels, and to indicate differences in their perceived quality, prestige, and impact. Nonpeer-reviewed channels are either excluded from the authority list or indicated as level 0 or [−], whereas channels approved to be peer-reviewed are categorized as level 1. The leading international channels are categorized as level 2, and in Finland and Denmark a small subset of top channels is also categorized as level 3. However, in Denmark the level 3 categorization does not concern book publishers; only journals.

There are some minor differences, among the Nordic countries, in how the channels for book publications are handled. All countries have an authority list of book publishers with level ratings, but the Danish list is slightly more exclusive, as it does not contain level 0/−, whereas Finland and Norway do include a “0” category. Note that for Norway, this category “0” is made up of publishers that have either not applied for inclusion on the publishers list or have not been approved. Also, for Denmark, the number of publishers assigned to levels 1 and 2 is smaller than in Finland and Norway (see Table 1), while Finland, unlike the other countries, has chosen to include the registration of academic textbooks as part of its PRFS.

The level 2 list of book publishers was first produced in Norway, and it was also used as the basis for the level 2 categorization of publishers in Denmark and Finland. However, there are some differences. For example, Elsevier and Springer are rated as level 2 in Finland but have been assigned to level 1 in Norway and Denmark. Only Finland has a small subset of level 2 publishers categorized as level 3.

Another difference is that Denmark also lists more than 1,500 book series of mainly international book publishers, while Finland lists book series mainly from Finnish book publishers. Norway also lists book series, and this is done in two different ways, depending on the presence or absence of a unique ISSN. If a book series has a unique ISSN, the series is included in the field-specific journal list. This also means that a book series with an ISSN may have a different level from its publisher. If a book series does not have a unique ISSN (only an ISBN), it will have the same level as its publisher. If articles in a book are connected to a series with an ISSN, the articles have the same weights as in journals. All of this means that in some countries the weight of monographs and articles in books in the PRFS are determined on the basis of the book series, and not only on the publisher. Moreover, there are differences in the treatment of national languages. In Finland, four leading Finnish and Swedish language SSH publishers, and a few national language book series, are categorized as level 2, while in Norway and Denmark channels for national language books are placed at level 1.

There are some notable similarities and differences between the countries in the PRFS weighting of book publications. Denmark, Norway, and Finland all give monographs a higher weight than journal articles. However, the difference between level 1 and level 2 is stronger in Finland (3-fold, and 4-fold for level 3) than in Denmark and Norway (1.6-fold; see Table 2). On the other hand, the weight of level 1 monographs compared to level 1 journal articles is slightly stronger in Denmark and Norway (5-fold) than in Finland (4-fold). In Finland, the PRFS includes all publications that the universities report to the ministry as being peer-reviewed (i.e., self-reported by researchers and checked by data-collection personnel), irrespective of the publisher levels. Therefore, monographs published with a level 0 publisher may generate points (0.4) to a university, while in Denmark and Norway, monographs that have not been approved at either level 1 or level 2 are excluded from their PRFS.

With respect to publisher evaluation, there are more differences. In all three countries the categorization of journals/series as level 2 (or 3) is the responsibility of field-specific expert panels. However, for book publishers at level 2, the assignment processes in Norway and in Denmark are carried out by an academic committee (i.e., a select group of appointed researchers across all fields), not by the expert panels themselves. This is done because publishers in most contexts publish books in many different fields. Field-specific journal experts are invited to suggest levels also for publishers, but the final decision is made with an interdisciplinary perspective. In Finland, all book publishers approved to a level 2 and level 3 assignment are done collectively by panel chairs, based on a preliminary proposal by SSH fields.

There are also some limitations relevant to the number of book publishers assigned to and evaluated at level 2 (or 3) in the three countries. In the case of journals and series, level 2 can only be comprised of at most 20% of the world’s output of each panel’s channels. In the case of book publishers, level 2 is determined somewhat differently. In Norway, there is a balanced approach to fields when a larger share of its publications comes as books. Because books are more important to scholars from the SSH than the other fields, most of the suggestions to change level for publishers are received from the panels in SSH. These are also fields that are not sufficiently covered by international statistics, such as Scopus and Web of Science. In these fields, level 2 is calculated by including book and journal publications from national output. The limitation of book publishers on level 2 is then determined by a simulation across fields. For both Norway and Denmark, the primary rule is that no national publisher is to be categorized at level 2. In Finland, it has been broadly stated that around 10% of level 1 publishers can be placed on level 2, of which 1% are assigned to level 3.

The publication channel ratings in Denmark, Finland, and Norway attempt to stimulate both quality and quantity publishing at an institutional level. The purpose is to make it more rewarding for a university to have publication activity that focuses on channels with “more stringent requirements related to the originality and quality of submitted manuscripts” (Norwegian Association of Higher Education Institutions, 2004). To a smaller extent, they describe differences in the average quality and impact of outputs at the level of publication channel, assuming that individual outputs can have a higher or lower quality than this average. The presumption is—from the perspective of a PRFS—that on average, output from level 2 publishers is of higher quality than that from level 1 publishers, but there are individual exceptions to both directions.

In 2016, the COST Action ENRESSH (European Network for Research Evaluation in the Social Sciences and Humanities) was established, and throughout its 4-year period it has served as a critical forum for European researchers with an interest in how the social sciences and humanities (SSH) generate and disseminate knowledge; how scientific and societal interactions characterize different SSH disciplines; and the types of transparent, adapted methods needed for the evaluation of these different disciplines. The Action is comprised of four working groups, with one dedicated specifically to databases and the uses of data for understanding SSH research. In light of this topic, several ENRESSH members have recently conferred about the development of an international publisher registry.

While this registry has not been officially established, some principal ideas have been put forth; for example, that it shall be dynamic and interactive, and that publisher entries would include those that support the research quality standards of SSH on the basis of their peer review and publishing practices (ENRESSH, 2019; Giménez-Toledo et al., 2019). The information registered would come from the publishers themselves, from national bibliographic databases and legal deposit libraries where publications from research institutions are recorded, and from the scholarly community by feedback given either directly by the authors or through CRIS systems (ENRESSH, 2019). Moreover, registered publishing firms can be nationally or internationally oriented, albeit there is still a need to decide upon the types of information added and updated, such as procedures for peer review and statistics based on bibliographic information.

This idea of a publisher registry has strong implications for a top-down approach to the evaluation of books, currently used by Nordic PRFS. It may also add value to bottom-up, individual level book assessments.

One of the main issues linked to PRFS is that different countries have unique and often very different publisher lists. Research by Mañana-Rodriguez and Pölönen (2018) has examined this earlier, thus providing the groundwork for what they call “future efforts toward supra-national combinations of publishers’ lists” (p. 643). A preliminary test of how to “merge” the national publisher lists of Spain and Finland showed that both were quite different, but one of the added problems that they found concerned the way that publishers may be rated from a field-specific standpoint.

Despite the fact that some publisher lists are coordinated to be similar (e.g., Norway’s and Denmark’s), while others are disparate (i.e., Spain’s and Finland’s), we are not at the stage of “supra-national” mergers. Hence, the initial benefit of a publisher registry is to first provide scholars, administrators, and research evaluators, including publishers and their associations, with a comprehensive resource comprised of international academic publishing houses. There is an ongoing discussion pertaining to the type of publishers to be included, and the general consensus is that they should be “scholarly,” with established and accepted peer review standards. For an individual scholar, this would make for a valuable advisory system, with publishing standards and transparent peer review processes clearly flagged.

Although more emphasis has been placed on the publisher registry’s benefit to PRFS (i.e., potentially unifying PRFS), it is still critical to understand what scholars expect at an individual level after they select a publisher, and their book is available to both scholarly and lay communities. By investigating the visibility and impact of an individual title, we are better able to explain potential misuses of publisher levels and ratings, and point to reasonable uses.

This study focuses on 743 academic/scholarly books published in 2017 by researchers affiliated with Danish, Finnish, and Norwegian universities. We collected 263 book titles (monographs; books in series; textbooks) registered in the Danish National database of scholarly outputs for the year 2017, 221 book titles from the Finnish registry, and 259 from the Norwegian registry for the same year. These data were chosen based on the fact that all published titles can be matched to a national publisher list, and all receive performance points vis-à-vis the different research evaluation systems. We have also limited the published book titles to the year 2017 to ensure a similar time frame for the collection of broad impact indicators.

Data collection first involved a manual retrieval process for all Google Scholar citations and PlumX indicators; for library holdings and country distribution counts, data were delivered directly from the OCLC WorldCat. All of the individual level book indicators are outlined and described in Table 3.

In addition to the statistics, each title was coded according to language of publication, subject area, and type of publisher, with private firms categorized as “commercial” and university presses, research institutes, the World Bank, museums, and foundations categorized as “noncommercial.” As the subject area or discipline of each book title was not recorded in a uniform way in the Finnish and Norwegian registries, and not at all in the Danish registry, we obtained Library of Congress or Dewey Decimal Classification numbers for the PRFS from OCLC WorldCat. Together, these two types of classifications accounted for only 49% of the titles. We therefore used a combination of information (i.e., book title, LC or Dewey Class; WorldCat) to categorize each book title according to one primary subject area.

A date stamp (e.g., 2019.05.22) was added to our data table to indicate when PlumX counts and Google Scholar citation counts were retrieved manually, as well as URLs. In some cases, the PlumX indicators for an individual book were obtained using a DOI, and in other cases with an ISBN (the ISBN was only useful if the book’s DOI had already been traced by PlumX). If a Google Scholar or PlumX link led to no record or an inactive page, we left the associated data cell null. If the URL was active and the book title was “traceable” via the platform, we recorded a value of zero or greater.

For the publication year of 2017 (Figure 2), 167 (63%) books from the Danish National repository were published with a level 1 publisher, and 96 (37%) were published with a level 2 publisher. A similar distribution can be seen in Norway, with 184 (71%) published at level 1 and 75 (29%) published at level 2. By comparison, the 2017 set of books retrieved from the Finnish National Repository, includes those produced by a level 0 (n = 52; 24%), level 1 (n = 59; 27%), level 2 (n = 63; 28%) and level 3 (n = 47; 21%) publisher.

Table 4 and Figure 3 show that the WorldCat holdings indicator yielded the most counts overall (n = 692) with the lowest missing values (n = 51). A large number of titles were traced on Google Scholar, but citations per book were typically much lower (Mean = 8 and Median = 2) than WorldCat holdings. However, both low and missing values are likely to occur when collecting citations for books within a 2-year citation window.

PlumX yielded relatively high usage counts (Mean = 85 and Median = 26) and captures (Mean = 16 and Median = 4) for some individual books, but a particularly high percentage of data values were missing from this platform (i.e., approximately 63%). For journal articles, social media counts tend to accrue more rapidly than citations, but in the case of books, we recognize the problem of DOIs. Results similar to ours can be seen in previous research (see Torres-Salinas et al., 2017a, 2017b; White & Zuccala, 2018; Zuccala & White, 2015), where library holdings or “libcitations” were also found to be more prevalent than Google Scholar and Scopus citations, with much lower PlumX usage counts, captures, mentions, and social media counts. Except for the correlation analyses, shown below, we present no further analyses pertaining to the PlumX indicators.

Table 5 presents the Spearman rho correlation coefficients for each of the indicators tested in this study. The strongest correlation was found for WorldCat holding counts and WorldCat country distribution counts (rho = .96, p < 0.01). PlumX usage counts and captures are also strongly correlated (rho = .51, p < 0.01), followed by WorldCat country distribution counts and Google Scholar citations (rho = .44, p < 0.01), as well as WorldCat holdings and Google Scholar citations (rho = .43, p < 0.01).

Figure 4 presents the full frequency distributions of WorldCat holdings for the different PRFS system levels. Again, Finland is the only registry country that has level 0 and level 3 publisher categories. The cumulative relative distributions show that the highest percentage of WorldCat holding counts are concentrated at level 3, as well as at the tail end of level 2. For books published at level 1, there is a much broader distribution of holding counts.

In Figures 5, 6, and 7, boxplots are shown for book titles assigned to two publisher levels for Denmark and Norway, and four publisher levels for Finland. For each level, we see the collected book titles’ WorldCat library holding counts (“libcitations” as per White, Boell et al., 2009). Here, the broader “perceived cultural relevance” and visibility of each title is indicated by how many libraries worldwide hold at least one copy.

Starting from Figure 5 (Denmark), a benchmark has been added, based on the combined median for levels 1 and 2 (Denmark and Norway), and the combined median for levels 0 to 3 (Finland) for WorldCat holdings. These benchmarks are in some sense arbitrary, as alternative values may be selected, but they are useful for explaining our metaphor. Books produced with a level 1 publisher that have achieved greater than or equal to the visibility benchmark (≥ 39 holdings) are what we call bigger fish in the small pond of national/regional publishing. Likewise, the books that fall below the benchmark median (< 39) are smaller fish in the big pond of international publishing (see Figure 3). In Figure 6 and Figure 7 we apply the same description of bigger fish and smaller fish in the small or large ponds for Norway, with a benchmark median of 32, and for Finland, with a benchmark median of 111.

Figure 8 shows the full cumulative frequency distributions of Google Scholar citations for the different PRFS levels. The relative distributions show that for all levels, 20–80% are concentrated within a range of 0 to 10 citations. Books published by a level 3 publisher generally outperform those from level 1, where we see some with greater than 100 citations, but there is generally an equal distribution of books published at level 2 that perform just as well as level 3.

In Figures 9, 10, and 11 boxplots and combined median benchmarks are shown for book titles produced and assigned to two publisher levels for Denmark and Norway, and four publisher levels for Finland. For each level, we can observe the collected book titles’ Google Scholar citations. Here, the “impact” of each title is indicated by its count of citations.

Again, for each of the boxplot figures (Figures 5–7 and 9–11) a benchmark median is shown for WorldCat holdings, as well as Google Scholar citations. Conceptually, we use these median values to explain how a PRFS “rewards” a book at an individual level.

All book titles at level 1 that fall below the benchmark perform “as expected.” This means that they correspond sufficiently with the five reward points (or in Finland four points) earned as a result of their country’s PRFS. Moreover, all books at level 2 that are higher than the benchmark have also performed as “expected.” This means that these titles correspond sufficiently with the eight reward points earned as a result of their country’s PRFS.

Then, there are some level 1 books that are higher than the benchmark (as bigger fish in the smaller pond) and level 2 books that are lower than the benchmark median (as smaller fish in the bigger pond). These individual books are of interest because they all represent reward point discrepancies.

Consider the following example. There are two books from our data set: Census and census takers: A global history and Culture war: Affective cultural politics, tepid nationalism and art activism. The first title acquired five PRFS reward points, and the second acquired eight PRFS reward points, yet both have achieved the same degree of international visibility based on WorldCat holdings (n = 86). At an individual level of assessment there is no difference between the two books based on this one indicator, yet the second title was “rewarded” with more points than the other, simply because of its publisher. There is also an example for two book titles based on citations: Expanding welfare in an age of austerity: Increasing protection in an unprotected world was published at level 1, with five PRFS points and eight citations, and Visible hands: Government regulation and international business responsibility was published at level 2 with eight PRFS points, also having received eight citations. Again, there are no differences between these titles, except for reward points.

In Table 6 and Table 7, all data pertaining to “reward” discrepancies for both indicators are summarized as percentages. Note that for the Danish registry 25% of the books earned five PRFS points for having been produced by a level 1 publisher, yet have been individually and culturally more visible via WorldCat library holdings than 5% of the level 2 titles rewarded 8 points. Also, 32% of the level 1 titles from this same registry have had a higher citation impact within a 2-year period than 24% of the titles published in the same year at level 2. For Finland, the share of smaller fish for WorldCat holdings in the bigger pond of international publishing at level 2 (44%) is much larger than it is for Denmark and Norway. The most likely explanation for this is that Finland includes regional publishers at level 2, whereas the other countries only assign international publishers to level 1.

A Negative Binomial regression model was used to determine if additional features of a publisher or book, including the PRFS level, are predictors of WorldCat holding counts and Google Scholar Citations. With the boxplots, we only see the extent to which PRFS level ratings correspond with impact or visibility, but there may be other factors that put an individual book in a bigger fish − smaller pond or smaller fish − bigger pond category. For this analysis we use additional data pertaining to Publisher Type (Commercial/Noncommercial), Publisher Language (English/Not English), and Subject Area (see categories in Table 3).

As a Poisson model often precedes the use of a Negative Binomial model, a Poisson regression was used first to see if the variances in the distributions of our two predictor variables were equal to the means of their distributions (i.e., the mean counts of WorldCat holding counts and mean counts of Google Scholar citations). The variances were greater than the means, and in our case pointed to overdispersion in the data (WorldCat mean = 135; Variance = 50,643), Google Scholar mean = 8; Variance = 1,463). The ratio of the deviance to the degrees of freedom also signaled overdispersion (df = 131 for WorldCat holdings; df = 25 for Google Scholar citations). Note that the further this value is away from 1, the more the data are overdispersed.

A Negative Binomial regression essentially adjusts the Poisson model when there is a high amount of overdispersion. The predicted change for our outcome or dependent variables is in expected log counts. A positive regression value conveys a positive predictor relationship between the predictor variable and the expected count outcome. A negative value reflects a negative relationship between the predictor and the expected count. Results from the Negative Binomial model point to a degrees of freedom ratio that improved the overdispersion (df = 1.55 for WorldCat holdings; df = 2.38 for Google Scholar citations). The omnibus tests also indicated a significant improvement in fit over a null model (p < .001; Sig .000), which would include no predictors.

Table 8 shows the parameter estimates for WorldCat holdings as the dependent variable. Here we see that the Publisher Language (English versus Non-English) is positive and significant (B = 2.30; p < .001). The Publisher Level (Level 0, 1, 2, or 3) is also positive, but only to a limited degree (B = .193; p < .05), while the Subject Area shows a negative but nonsignificant relationship to holding counts (B = −.029; p > .05). A PRFS publisher level therefore relates to some degree to WorldCat holdings, but it is still not the strongest predictor of the log change in outcome for these counts when other variables are held constant.

Table 9 shows the parameter estimates for Google Scholar citations as our dependent variable. Here, the Publisher Language is the strongest predictor variable (B = .746; p < .000) followed by the Publisher Level (B = .699; p < .000), which is also positive and significant. The Subject Area of a book is also a positive predictive variable in relation to citations (B = .196; p < .000), and more so when measuring WorldCat holdings. Again, a PRFS publisher level relates to some degree to Google Scholar citations, but it is still not the strongest predictor of the log change in outcome for these counts when other variables are held constant.

When PRFS in Denmark, Norway, and Finland assign levels to publishers, suggestions and decisions are normally based on the overall reputation of the publisher in a research field, where reputation is closely connected to how the publisher performs in terms of peer review. Because our Negative Binomial model shows that type of publisher (i.e., commercial versus noncommercial) does not have any predictive value for WorldCat holdings, the present analysis is carried out to examine how a publisher functions as a worldwide library distributor. This can help to explain, in part, the bigger/smaller fish phenomenon, where books classified at certain level may have been produced by publishers with broader distribution potential than others.

The scatterplots in Figures 12 and 13 show the relationship between mean publisher WorldCat holdings and mean country distributions, both of which are strongly and significantly correlated (rho =.96, p < 0.01). For both figures we have selected a set of level 1, 2, and 3 publishing houses/presses that have produced three or morebooks.

Due to an overlap in publisher names added to Figures 12 and 13, we further specify in Table 10 the primary differences in level assignments for the three countries. The assignments noted in brackets indicate that there was no book title associated with that particular publisher for Denmark, and/or Norway in our 2017 data set.

Again, publishers facilitate editorial, peer review, and printing processes but also serve as distribution centers to libraries worldwide. In Figure 12, where the scale is logarithmic, most of the publishers at level 1 fit within the quadrant representing the lowest worldwide distribution. The same quadrant is outlined in Figure 13, which utilizes a normal scale. Still, there are a few publishing outlets at level 1 (Amsterdam University Press, Reclam Verlag, Bloomsbury) with a relatively strong distribution potential. Figure 13 shows that level 2 and 3 publishers tend to have a much higher distribution potential compared to those at level 1, with John Wiley & Sons, Manchester University Press, John Benjamins, and Wiley-Blackwell representing the highest from these two levels.

The following conclusions may be drawn from this study. The first is that WorldCat yields substantial library holding and country distribution counts for most book titles (i.e., 93% of n = 743 titles had at least one holding). Approximately 60% of the books in our data set also had at least one Google Scholar citation, but with many titles observed on the platform with a zero count, we might have obtained additional citations with a longer time period for data collection. With PlumX as a data source, some capture and usage counts were found, but the mentions and social media counts yielded the least amount of useful data. This is mainly because PlumX depends highly on the availability of DOIs for books. Overall, the study results point clearly to the reliability of the two WorldCat indicators.

The second conclusion relates specifically to the PRFS publisher level assignments. As the main aim of the Nordic systems is to stimulate high “quality” publishing, the focus is top down, with an emphasis on aggregate publishing patterns, leading to rewards at an institutional level. Thus, at an institutional level the expectation is that there will be an “average” quality of outputs. With all of our boxplot distributions (Figures 5–7 and Figures 9–11) we show that a lower publisher level generally corresponds with lower visibility (WorldCat counts) and lower impact (Google Scholar citations), while a higher publisher level generally corresponds with higher visibility and higher impact. But within each of these distributions there are a number of exceptions.

Approximately 25% to 41% of the books from each of the Danish, Norwegian, and Finnish data sets (2017) have higher degrees of visibility or impact as bigger fish in smaller ponds. A broader range of 5% to 44% present a lower degree of visibility or impact as smaller fish in bigger ponds. Many other factors are therefore contributing to a book’s achievement relative to this metaphorical “big-to-small fish” continuum. Our Negative Binomial regression model, which focuses on a few additional variables, shows that publisher type (i.e., commercial versus noncommercial) is not a strong predictor of either WorldCat holdings or Google Scholar citations, but that publication language is (i.e., English/Non-English). Clearly a book published in English is more likely to be cited and accrue broad international WorldCat holdings, and this makes sense, given the predominance of English in academia.

There are also differences in the distribution potential of every publisher. Publishers classified at level 1 tend to demonstrate a lower WorldCat distribution effect than those categorized at level 2; however, our study points to a few classification discrepancies (i.e., with Amsterdam University Press, Brill and Bloomsbury classified differently per PRFS). Still, authors may expect to have their book distributed more widely by publishers such as Oxford University Press, Princeton University Press, or Wiley-Blackwell.

While categories for journals (e.g., the Web of Science Subject Classification) are well-established and used often in journal-based analyses, it is still problematic to assess books on the basis of subjects. Again, only 49% of the titles had been assigned a real subject classification (i.e., with an OCLC Library of Congress or Dewey Decimal Number), and this meant that we could only make a manual subject assignment to cover the whole data set. With each title “freely” classified according to a main subject area, we found that the subject of a book has predictive value only in terms of log changes in Google Scholar citation rates, but not WorldCat holdings. While books pertaining to many subjects can be held in libraries worldwide, certain research areas tend to cite new books more than others.

PRFS committees as well as individual authors might more or less agree upon a publisher’s degree of specialization, quality, or prestige. An author’s choice of publisher can for instance, depend on the degree to which it is highly specialized or relevant to a very small field of specialization. In this instance, it might be the book series itself, the editorial staff, and review process that the author finds attractive about the publisher, more than the publisher’s level. Still, what an individual author might also want is his or her work to be exceptionally “visible” and generate some “impact,” based on a high uptake in the scholarly communication system. If relevant, authors would hope to see this take place on an international scale. As White et al. (2009) suggest: “esteem goes to researchers for their perceived impacts on culture and the life of the mind” (i.e., for scholars as well as the lay public [p. 1086]). What we know from journals, and now also with book publishers, is that there is no guarantee that a choice of publishing outlet will lead to academic and/or broad cultural impact.

Nevertheless, it is possible that PRFS in their current form may be having an influence on Nordic scholars’ publishing choices. Perhaps some authors are selecting a publisher merely because of its assigned level, without considering other factors. Some studies have already investigated the influence of PRFS on scholars’ journal publishing habits (Ingwersen & Larsen, 2014; Pölönen, Auranen et al., 2018), and more research is needed to understand the effect of PRFS levels on book authors.

As noted earlier, national publishers in Denmark are always categorized as level 1, with many international presses assigned to level 2. This means that national or regional publishing houses are at a disadvantage, particularly on an international scale, where the publishing world is intensely competitive. It is a routine feature of the publishing world to compete both in a “market for content and market for customers” (Thomson, 2005, p. 35). Regional specificity is a characteristic that can “make it more difficult for [a] press to attract the very best international scholars in international disciplines” (Thomson, 2005, p. 158). If authors are too concerned about the internationalization of their research and focus too intently on the acquisition of level 2 points, regional publishers, including research institutions, learned societies, and commercial publishers, may suffer.

Publishing more books with international publishers (in English) also makes it increasingly difficult to establish reliable indicators for books published at a local or national level (not in English). Thus far, national visibility can best be seen via social media platforms, such as LinkedIn, Twitter and/or other local news outlets. For example, a new book entitled Grov Konfækt. Tre vilde år med trykkefrihed 1770–73 (Horstbøll, Langen, & Stjernfelt, 2020), which examines the freedom of the press in Denmark, was confirmed a “masterpiece” by the Carlsberg Foundation. It was written and published in Danish with a Copenhagen-based press (Gyldendal.dk). An announcement on LinkedIn (Budtz-Pedersen, 2020) was made to indicate that a book reception was held in Copenhagen, where the Danish Prime Minister (Mette Frederiksen) and the Chairman and Editor in Chief of Information from the Carlsberg Foundation attended. Metric-type indicators have not (yet) been collected to determine its international visibility or impact, but there is clear evidence via social media that it is of high national importance.

As our study focuses on publisher assignments and what they do at a national level, certain findings have implications for the proposed international publisher registry (ENRESSH, 2019; Giménez-Toledo et al., 2019). Although there are many reasons for its development, we are specifically interested in what it can do to enhance PRFS.

There may be two benefits. The first relates to the lack of uniformity in publisher assignments, and in this paper we show that this is an issue even when the PRFS, like that of Denmark, Norway, and Finland, are relatively similar (see Table 10). Other publisher systems, like the one used in Spain, present even more incongruities (see Mañana-Rodriguez & Pölönen, 2018); hence it is still not clear what constitutes a good publisher from a specific geographical region. How can this be established? By gathering and registering more detailed information about publishers in one data source, it can be easier to promote a more unified approach to their classification. This, in turn, could give other countries in Europe a stronger motivation to adopt and coordinate more comprehensive PRFS.

The second benefit is that it will afford scholarly researchers not only more transparent, but also more detailed information about book publishers. So far, PRFS have not been designed to give authors opportunities to identify precisely what it is that they need from a publisher. PRFS committees are broadly classifying them, yet as we show from our individual level analyses, broad classifications can never account completely for what happens to a book at an individual level. This depends on a myriad of factors, such as peer review and editorial procedures, publishing/print language options, publisher specialization, and the distribution potential of the publisher itself. Transparency of information will not only be good for publishers, but valuable to authors. For instance, if scholars are given an opportunity to interact with one another via the system, and obtain recommendations about their publishing experiences (i.e., like a publisher Trip Advisor), this could help them to determine which press is best for their own publishing strategy. Recommendations would also allow authors to make more “organic” choices, particularly if they are writing about subjects pertaining to regional or national concerns (Kulczycki, Guns et al., 2020).

Last but not least, because the Nordic PRFS were originally created to be used as indicators for funding allocation at a macroevel, it is first and foremost critical to state that level ratings should not be applied at the level of individual researchers (Aagaard, 2018; Pölönen et al., 2018; Sivertsen, 2018). As our findings show (Sections 5.2 and 5.3), PRFS publisher levels do not necessarily relate to higher rates of visibility or impact for individual books. Still, some universities and departments have been known to (mis)use PRFS indicators for their internal evaluation and funding procedures (Aagaard, 2015; Hicks, 2012; Krog Lind, 2019; Sivertsen & Schneider, 2012; Wahlfors & Pölönen, 2018). In Sweden, for example, there is no PRFS, yet many Swedish universities use the Norwegian indicator and publication channel list for their internal evaluation/assessment/funding procedures (Hammarfelt, Nelhans et al., 2016). Similarly, in Poland, a new model was introduced in 2018 that utilizes publisher lists in a scholar’s habilitation procedure (Kulczycki & Korytkowski, 2018). When a candidate presents a book as his or her main achievement, it has to be from a publisher on Poland’s new publisher list. In this case, potential misuse occurs if the publisher alone signals the candidate’s achievement, to the exclusion of assessing the book’s content, visibility, and cultural impact.

Finally, we look to the near future, and beyond. Perhaps there may be an opportunity to develop interoperability between the international publisher registry and WorldCat. By this, we mean that perhaps holding and country distribution counts from WorldCat could be incorporated into the registry, in much the same way that PlumX indicators are linked to the Scopus index. This would make it easier and more appropriate to conduct performance evaluations at an individual level if the visibility or impact of a book is significant for a scholar’s promotion and/or tenure. All of this depends also on the progress of Open Access publishing. WorldCat holdings and country distribution counts seem to be reliable indicators at the moment, but may not be as useful in the future if open access publishing means that libraries no longer “hold” books. Publishers may develop stronger marketing approaches for open access books via social media (e.g., Wang & Zuccala, 2021) and there could be new opportunities to rethink metric indicators for books related to capture, publisher downloads, and usage statistics.

The authors would like to thank the reviewers and the editor Ludo Waltman in particular for his valuable comments, as well as Adrian Díaz-Faes for his informal reviews of earlier drafts of this paper.

The authors have no competing interests.

No funding has been received for this research.

Data for this project are available on Figshare (https://doi.org/10.6084/m9.figshare.14472645).