Abstract
This mapping review addresses scientometric indicators that quantify open scholarship. The goal is to determine what open scholarship metrics are currently being applied and which are discussed (e.g., in policy papers). The paper contributes to a better understanding of how open scholarship is quantitatively recorded in research assessment and where gaps can be identified. The review is based on a search in four databases, each with 22 queries. Out of 3,385 hits, we coded 248 documents chosen according to the research questions. The review discusses the open scholarship metrics of the documents as well as the topics addressed in the publications, the disciplines the publications come from, and the journals in which they were published. The results indicate that research and teaching practices are unequally represented regarding open scholarship metrics. Open research material is a central and exhausted topic in publications. Open teaching practices, on the other hand, play a role in the discussion and strategy papers of the review, but open teaching material is not recorded using concrete scientometric indicators. Here, we see a research gap and discuss the potential for further research and investigation.
PEER REVIEW
1. INTRODUCTION
Open scholarship means making all practices, processes, and products of the scientific world open and freely accessible. The term refers to both academic research and academic teaching (Scanlon, 2013; Tennant, Beamer et al., 2019). Regarding research, open materials are open access publications, open data, open code, open software, and any other resource evolving from a research process. Open materials in academic teaching are represented by open educational resources (OER) (UNESCO, 2019). OER are freely accessible teaching and learning materials, which are openly licensed and can be reused and further processed (Wiley, n.d.). In parallel with the new open practices, researchers started to quantify their advancement. Similarly to other popular metrics applied to scientific literature, researchers, and institutions, open scholarship metrics aim at measuring the “openness” in academia (e.g., monitor its development, compare the movement to traditional research outputs, or assess its contribution to research in general [Wouters, Ràfols et al., 2019]). Currently, the applications of such metrics and the development of concrete open scholarship indicators are work in progress. In the following, we aim at giving an overview of studies that either discuss or apply open scholarship indicators to show how far the quantification of open scholarship has progressed.
The research questions are
Which open scholarship indicators are currently applied and discussed?
Which subjects address and which disciplines and journals shed light on open scholarship indicators?
An analysis of the open scholarship indicators is important, as these have an impact on higher education policy decisions (see, for example, the Open Science Monitor [CWTS, 2019] or the Open Access Monitor [Barbers, Stanzel, & Mittermaier, 2022]). For this reason, it is important to have an overview of existing indicators and to understand how they are applied and discussed.
For the present mapping review, scientific documents were collected following a Preferred Reporting Items for Systematic Reviews (PRISMA) workflow and then coded according to three categories. The first category distinguishes between empirical studies and discussion papers. The second category labels each indicator used and the third category assesses the subject matter to which the indicators were applied in the empirical studies (e.g., articles/books or journals). The results section presents the results of the coding in the first two sections. The indicators used in the empirical studies are listed and presented (Section 4.1). This is followed by the recommendations and presentations of the indicators in the discussion papers (Section 4.2). To conclude the results section, the keywords, thematic disciplines, and journals of the publications are illustrated (Section 4.3). Finally, the discussion looks at the share of higher education teaching as part of open science in the context of open scholarship indicators.
2. THEORETICAL BACKGROUND
2.1. Open Scholarship
Open scholarship activities strive for the least restricted possible access to scientific processes and products, the possibility of reuse, and participation. The goal is to make scholarship comprehensible, more reliable, and efficient (Burgelman, Pascu et al., 2019). The systematic literature review by Vicente-Saez and Martinez-Fuentes (2018) focuses on the area of open research and concludes with the following definition: “Open Science is transparent and accessible knowledge that is shared and developed through collaborative networks.” In their outlook, the reviewers argue for the inclusion of open learning in the debate on openness and call for follow-up research in this regard.
The goals of open scholarship and open research dovetail with the goals of the openness movement in higher education and OER. OER are “teaching, learning and research materials in any medium—digital or otherwise—that reside in the public domain or have been released under an open license that permits no-cost access, use, adaptation and redistribution by others with no or limited restrictions” (UNESCO, 2012). OER have an open license (e.g. Creative Commons: https://creativecommons.org/) and can be retained, revised, remixed, reused and/or redistributed depending on the license (Wiley, n.d.; Wiley & Green, 2012). OER are discussed from different perspectives; for example, from an educational policy perspective aiming at designing high-quality teaching material, from a sociopolitical perspective aiming at reducing costs of educational material, and from a pragmatic perspective that discusses how digital educational materials can be transformed into OER in practice in a simple and straightforward way (Bellinger & Mayrberger, 2019). OER are the most tangible and measurable entities among teaching materials. They are the most likely to be recorded scientometrically.
2.2. Quantification of Open Scholarship
Scientometrics uses statistical methods to measure academic output, such as publications and citations, and to map patterns of scholarly structures and relations by visualizing scientific fields in time. If these are applied and interpreted appropriately, the indicators can provide clues to scientific performance (Ball, 2021).
Open scholarship indicators are also based on scientometric measurement methods and can be used for science evaluation as well as for monitoring. The indicators record openly accessible products and practices and can provide insights into how openly a person, institution, or country is working. The Open Science Monitor, for example, tracks the development of open practices in Europe by monitoring open access publications, open research data, and open collaboration on the level of EU countries and individual disciplines (CWTS, 2019). Besides evaluating and monitoring the development of open scholarship, those indicators can and should act as an incentive, according to a discussion paper of the Helmholtz Association (Pampel, Ferguson et al., 2020)1. Scientometric indicators are known to have a steering effect. Products and practices that are recorded scientometrically in academic performance measurements are made visible and possibly also rewarded. Consequently, there are positive effects for academics if they orient their academic practices to the existing incentives (Hicks, Wouters et al., 2015).
Because of the power that scientometric indicators have in science governance, an overview of them is important, so as to understand in which thematic areas and disciplines research is conducted on them. This mapping review aims to identify existing research gaps on open scholarship indicators and provide a basis for further research.
3. METHOD
3.1. Data Sources
The present study was designed as a mapping review/systematic map and serves to categorize existing literature about open scholarship indicators. We decided against conducting a qualitative systematic review because we are interested in mapping and categorizing the existing literature rather than analyzing the literature thematically. The mapping review explicitly serves to provide an overview and to identify research gaps. Methodologically, the literature search, screening, and coding are as structured and comprehensive as in a systematic review, but we do not assess the content of the literature in the mapping review in detail, which would be the methodology of a systematic review (Grant & Booth, 2009, p. 94). The data basis for the literature search consists of two international literature databases: one German literature database with a focus on educational research, and one preprint server that seems to contain relevant preprints on the review’s topic. No manual search was carried out because the selected literature databases broadly cover the topic of the mapping review in terms of content.
The Web of Science (WoS; published by Clarivate Analytics: https://webofknowledge.com) is an international bibliographic database containing cross-disciplinary publication and citation metadata with a bias towards the Anglo-American language area (van Leeuwen, Moed et al., 2001). For this review, we accessed WoS via the license of the Competence Centre for Bibliometrics (https://www.bibliometrie.info/). The license includes all indexes of WoS.
The database Bielefeld Academic Search Engine (BASE; published by Bielefeld University Library: https://www.base-search.net/) is a meta search engine containing a variety of scientific publications from different sources. Publications include scientific articles and books, as well as information on research data, doctorates and postdoctoral theses, teaching material, and other types of documents. BASE accesses 9,104 data providers. Relevant providers are, for example
ArXiv.org (Cornell University Library)—Coverage in BASE: 100%
SSOAR—Social Science Open Access Repository—Coverage in BASE: 99%
Zenodo—Coverage in BASE: 98%
The Fachportal Pädagogik (FP; published by DIPF: https://www.fachportal-paedagogik.de/en/literatur/index.html) is of great interest for the present work because of its broad coverage in the field of educational research in German-speaking countries. Beyond its national collection of literature references (FIS Bildung as part of FP), FP also obtains information from other data sources (e.g., ERIC). For this search, all data sources except BASE, which was searched in at its own search site, were included. We considered FP due to its educational focus, including literature on open and educational science assessment and infrastructures.
We also take into account the preprint server of the Open Science Framework (OSF; published by Center for Open Science, https://osf.io/preprints/) because we expect to find relevant preprints on the review’s topics. Similar to BASE and FP, OSF uses different data providers. In this case, the search was limited to OSF’s own preprint server.
3.2. Search Terms and Search Queries
A first literature search was conducted in October 2021. It served to validate the search terms. The search queries consist of two blocks (see Table 1). For block A, the two umbrella terms “open scholarship” and “open science” were chosen, as well as openness terms that primarily refer to quantifiable entities (“open access”; “open data”; “open educational resources” and the corresponding abbreviation “oer”). Softer openness terms were excluded from the search syntax, such as “open scholarly communication,” “open collaboration,” or “open method,” as the mapping review focuses on countable entities. Part B of the search syntax includes terms that capture scientometric indicators. For the search term validation, 18 terms were tested (see Table 1, search terms 1–18). The results of the test search were subjected to a keyword analysis to validate the search terms (see data set: https://doi.org/10.5281/zenodo.8128130). In the keyword analysis, all keywords assigned to documents returned in the test search were examined for their relevance for the present work. As a result, search terms 19 to 22 were added to the final search. The search terms 1–18 as well as the division into parts A and B of the search query were retained.
Search terms
Search terms in block A | (“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”) | ||
AND (a search term from block B) | |||
Search terms in block B | English (for all databases) | German (only for FP) | |
1 | indicator* | indikator* | |
2 | metric* | metrik* | |
3 | scientometric* | szientometri* | |
4 | bibliometric* | bibliometri* | |
5 | webometric* | webometri* | |
6 | altmetric* | altmetri* | |
7 | sociometric* | soziometri* | |
8 | “research assessment” | (forschungsbewertung* ODER forschungsbilanz*) | |
9 | Reward | belohnung | |
10 | Recognition | anerkennung | |
11 | Ranking | rangliste | |
12 | monitor* | ||
13 | “evidence-based policy” | (“evidenzbasierte politik” ODER “faktenbasierte politik” ODER “faktengestützte politik”) | |
14 | “impact analysis” | (wirkungsanalyse ODER folgenanalyse) | |
15 | “academic impact” | (“wissenschaftliche bedeutung” ODER “wissenschaftliche auswirkung” ODER “akademische bedeutung” ODER “akademische auswirkung”) | |
16 | “research impact” | forschungswirkung* | |
17 | “performance measurement” | leistungsmessung* | |
18 | Incentive | anreiz* | |
19 | “statistical analysis” | “statistische Analyse” | |
20 | “research evaluation” | “Forschungsevaluation” ODER “Wissenschaftsevaluation” | |
21 | informetric* | informetri* | |
22 | “data-driven policy*” | “datenbasierte politik” |
Search terms in block A | (“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”) | ||
AND (a search term from block B) | |||
Search terms in block B | English (for all databases) | German (only for FP) | |
1 | indicator* | indikator* | |
2 | metric* | metrik* | |
3 | scientometric* | szientometri* | |
4 | bibliometric* | bibliometri* | |
5 | webometric* | webometri* | |
6 | altmetric* | altmetri* | |
7 | sociometric* | soziometri* | |
8 | “research assessment” | (forschungsbewertung* ODER forschungsbilanz*) | |
9 | Reward | belohnung | |
10 | Recognition | anerkennung | |
11 | Ranking | rangliste | |
12 | monitor* | ||
13 | “evidence-based policy” | (“evidenzbasierte politik” ODER “faktenbasierte politik” ODER “faktengestützte politik”) | |
14 | “impact analysis” | (wirkungsanalyse ODER folgenanalyse) | |
15 | “academic impact” | (“wissenschaftliche bedeutung” ODER “wissenschaftliche auswirkung” ODER “akademische bedeutung” ODER “akademische auswirkung”) | |
16 | “research impact” | forschungswirkung* | |
17 | “performance measurement” | leistungsmessung* | |
18 | Incentive | anreiz* | |
19 | “statistical analysis” | “statistische Analyse” | |
20 | “research evaluation” | “Forschungsevaluation” ODER “Wissenschaftsevaluation” | |
21 | informetric* | informetri* | |
22 | “data-driven policy*” | “datenbasierte politik” |
The final search was conducted between February 20 and March 18, 2022. The German translations were only used in the FP database. For each data source, the search queries were adapted according to the database field options (Table 2). For WoS, we chose the fields title and keywords (author keywords & keywords plus). In BASE, we did two separate searches in the fields title (=tit) and keywords (=subj). In FP and OSF, the free text search was chosen, because the test searches showed that hit rates were not much higher than for searches in specified, such as title. Table 2 shows the concrete search queries for each database. The wildcard […] marks the location for each term from the search block B.
Search queries per data source
Database . | Search query . |
---|---|
WoS | (ti=(“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”) OR ak=(“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”) OR kp=(“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”)) AND (ti=[…] OR ak=[…] OR kp=[…]) |
BASE | Tit:(“open scholarship” “open science” “open access” “open data” “open educational resources” “oer”) tit:[…] |
Subj:(“open scholarship” “open science” “open access” “open data” “open educational resources” “oer”) subj:[…] | |
FP | (Freitext: “OPEN SCHOLARSHIP” oder “OPEN SCIENCE” ode “OPEN ACCESS” oder “OPEN DATA” oder “OPEN EDUCATIONAL RESOURCES” oder “OER”) und (Freitext: […]) |
OSF | (“open scholarship” “open science” “open access” “open data” “open educational resources” “oer”) AND […] |
Database . | Search query . |
---|---|
WoS | (ti=(“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”) OR ak=(“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”) OR kp=(“open scholarship” OR “open science” OR “open access” OR “open data” OR “open educational resources” OR “oer”)) AND (ti=[…] OR ak=[…] OR kp=[…]) |
BASE | Tit:(“open scholarship” “open science” “open access” “open data” “open educational resources” “oer”) tit:[…] |
Subj:(“open scholarship” “open science” “open access” “open data” “open educational resources” “oer”) subj:[…] | |
FP | (Freitext: “OPEN SCHOLARSHIP” oder “OPEN SCIENCE” ode “OPEN ACCESS” oder “OPEN DATA” oder “OPEN EDUCATIONAL RESOURCES” oder “OER”) und (Freitext: […]) |
OSF | (“open scholarship” “open science” “open access” “open data” “open educational resources” “oer”) AND […] |
3.3. Search Results and Screening
Table 3 presents the search results of the individual queries per search term and database.
Following the extraction of the search results, the data set was cleaned of duplicates. The first step was performed internally in the individual database results. In WoS, duplicates were identified and removed via the WoS internal database ID (=UT). In OSF, duplicates were identified via the DOI, in BASE via DOI and title, and in the FP via the database internal ID and title. The second step of the duplicate cleaning process involved the cross-database cleaning of duplicates. This was also done using the DOI and the titles. The number of duplicates identified and removed can be seen in the PRISMA diagram (Figure 1).
The PRISMA diagram also shows the two screening steps conducted after the duplicate cleaning. Both screening steps were performed by the first author. A total of 2,262 documents were included in Screening 1. Title and abstract were checked for relevance. First, we excluded documents that lack data completeness (i.e., all hits with no data about title and author were excluded). Second, all hits not available in German or English were excluded, as the authors are only familiar with these two languages. Documents were also excluded in Screening 1 if their content did not cover the topics performance measurement, openness, and/or indicators (content exclusion criteria). Documents were dismissed if the inappropriateness of the documents was obvious; otherwise, they were kept for Screening 2. One example of an excluded document addressed fishing performance in open waters.
After the first screening, we kept 424 documents for Screening 2, where we analyzed the full texts for relevance. Formal exclusion criteria related to inappropriate document types, such as short sketches or conference abstracts with no available full text. The content exclusion criteria are identical to Screening 1, but with a detailed assessment of the exclusion criteria to record on which criterion the documents were excluded. In Screening 2, a total of 158 documents were excluded for content-related reasons. The PRISMA diagram shows which of the three exclusion criteria were applied and how often; exclusion based on more than one criterion is possible. The final data set used for coding includes 248 documents (see data set: https://doi.org/10.5281/zenodo.8128130). The 248 documents were published between 2004 and 2022, with more than 50% published in 2018, 2019, and 2021.
3.4. Coding and Data Analysis
The inductive coding was divided into three parts and was also carried out by the first author. In a first step, a distinction was made between empirical applications of the indicators and discussion papers regarding the indicators. In the following step, the applied or discussed indicators were identified. We distinguish between known bibliometric and altmetric indicators (e.g., journal impact factor) applied to open scholarship objects (e.g., open access journals) and open scholarship indicators explicitly designed to measure open scholarship objects. The last coding aspect referred to the documents to which the open scholarship indicators were applied (e.g., open access publications, open data, open code). The result of the coding can be taken from the research data (https://doi.org/10.5281/zenodo.8128130).
To address the second research question, the topics, disciplines, and journals of the data set are analyzed. For this purpose, the data provided by the databases (keywords, subject categories, and source) are processed. A descriptive analysis is performed to get an overview of all disciplines the publications relate to (see Tables 6 and 7 in Section 4.3). Furthermore, the data is visualized with VOSviewer (https://www.vosviewer.com/). For this purpose, the program extracts noun phrases from the titles and abstracts. These are represented by nodes and related by means of links. The connection between two nodes indicates that both terms occurred together in a document.
3.5. Limitations
The first limitation of this study is that the subject categories are not available for all documents, but only for those provided by WoS (171 out of 248 documents). Furthermore, it must be noted that search queries must always be restricted with regard to terminology. It is conceivable that the quantification of open teaching material is not discussed within scientometric terminology, but within educational science terminology. We have approached the subject matter here from a scientometric perspective and thus disregarded potentially interesting search terms. Follow-up studies could broaden this perspective and, in addition to scientometric approaches, also consider forms of evaluation such as student evaluations. Accordingly, the search queries would have to be adapted and expanded.
4. RESULTS
The coding of the mapping review revealed that 203 documents specifically apply the indicators in empirical studies, and 45 documents present, discuss, or issue recommendations regarding the indicators. Concerning the first research question of this paper, Section 4.1 presents the indicators of the empirical studies and Section 4.2 the discussion/strategy papers. Regarding the second research question, the keywords and disciplines of the documents are analyzed in Section 4.3.
4.1. Application of the Indicators
We identified 82 indicators in the empirical studies of the mapping review. Thirty-seven of these are bibliometric indicators (e.g., the h-index), 37 are altmetric indicators (e.g., the altmetrics attention score [AAS]), and eight indicators relate specifically to the openness of the materials (e.g., the openness status). Due to the search queries of the review, all indicators are related to openness in some way. For example, surveys are among the studies that examine data sets fo “open access citation advantage.” For this purpose, the indicator “times cited count” (tcs) was applied to open access and nonopen access documents (Langham-Putrow, Bakker, & Riegelman, 2021). At the material level, there is a link to openness, but the tcs indicator is a classic bibliometric indicator. Therefore, the umbrella category of openness indicators was not chosen in this case. We classified the indicators into the three categories ourselves. Table 4 shows the 10 most frequently used bibliometric and altmetric indicators, as well as all openness indicators, as this is the main focus of this paper (for the complete list see data set: https://doi.org/10.5281/zenodo.8128130).
Indicators used in the empirical studies (10 most used bibliometric and altmetric indicators, and all openness indicators; extract from the codebook (https://doi.org/10.5281/zenodo.8128130); for the indicator descriptions see van Leeuwen and Tatum (2018), Waris, Naseer et al. (2021), and Moed (2017))

Furthermore, we identified the open access objects to which the indicators were applied. Only one label was assigned per document and the label “mixed document types” was assigned as soon as no clear distinction could be made between OA articles, OA books, and other forms of publication. For example, this label was given to the document that gives the following information about the data basis: “All document types for the 2000–2019 period were retrieved” (De Filippo & Mañana-Rodríguez, 2020). Some 49.26% of the documents (n = 100) refer to open access articles (Table 5). It is interesting to note that the indicators are applied to OER or MOOCs four times. The following documents are those dealing with teaching materials:
Wang, Xiaochen; Liu, Mengrong; Li, Qianhui; Gao, Yuan (2017): A bibliometric analysis of 15 years of research on open educational resources
Zancanaro, Airton; Amiel, Tel (2017): The academic production on open educational resources in Portuguese
Zancanaro, Airton; Todesco, Jose Leomar; Ramos, Fernando (2015): A bibliometric mapping of open educational resources
Wahid, Ratnaria; Ahmi, Aidi; Alam, A. S. A. Ferdous (2020): Growth and collaboration in massive open online courses: A bibliometric analysis
Open scholarship objects to which the indicators were applied
Objects . | Frequency . |
---|---|
OA articles | 100 |
OA journals | 61 |
Mixed document types | 9 |
Open data | 8 |
OA books | 7 |
OA repositories | 7 |
OER/MOOCs | 4 |
Preprints | 4 |
OA citation data | 1 |
Open bibliometrics | 1 |
Open code | 1 |
Sum | 203 |
Objects . | Frequency . |
---|---|
OA articles | 100 |
OA journals | 61 |
Mixed document types | 9 |
Open data | 8 |
OA books | 7 |
OA repositories | 7 |
OER/MOOCs | 4 |
Preprints | 4 |
OA citation data | 1 |
Open bibliometrics | 1 |
Open code | 1 |
Sum | 203 |
On the one hand, the four represent a small number; on the other hand, it should be noted that OER do not appear in the specific naming of the indicators (Table 4). In contrast to studies which, for example, specifically state in the methods section that they use the indicator “number of open data,” this is not explicitly addressed in the studies on OER. There, reference is made to the use of general bibliometric or altmetric indicators. These are then applied to the object of the OER without addressing this methodologically.
4.2. Discussion and Policy Papers
Some indicators do not appear in the empirical studies but are presented in the discussion and policy papers. Nichols and Twidale (2017) introduce and discuss various open scholarship indicators. The Practical Openness Index (POI) is a simple indicator at the author level that divides the number of open articles and conference papers by the number of all articles and conference papers. The OI-Broad follows the same approach but also includes book chapters as a third type of publication in the formula. The Effective Openness Index divides the number of open publications by the number of all publications minus the number of publications with copyright restrictions. The Preservation-Friendly Openness Index (PFOI) takes into account the repositories of deposited open documents. Only documents deposited in environments that are considered “long-term locations” (e.g., institutional/disciplinary repositories or library archives) are counted. Publication locations such as personal websites are not counted. The Acce$$ Index records the sum of the prices of all nonopen items. The calculation can be done by simply adding up the costs, or it can be done in a more complex way by weighting them differently. The Actual Individual Purchase Index represents actual costs paid by readers to access their work (cost of consumption). The Openness Cost Index, on the other hand, calculates the costs incurred in publishing open access documents (costs of publication). Finally, the Open Reference Index (ORI) is the proportion of all the cited works of a paper that are themselves open access.
Like the indicators discussed by Nichols and Twidale (2017), three other open scholarship indicators are also discussed in the discussion papers, but do not appear as use cases in the empirical studies and therefore do not appear in Section 4.1. The data index takes into account both the number of data sets and the data set citations. The calculation procedure is based on the calculation of the h-index (Hood & Sutherland, 2021). Metric for the evaluation of open data (Meloda 5) is an indicator that records the reuse of open data. The metric has eight dimensions. They include the legal licensing of the data, the mechanisms for accessing the data, the technical standards of the data sets, the data model, the geographical content of the data, the frequency of updating, the dissemination, and the reputation (Abella, Ortiz-de-Urbina-Criado, & Blos-Heredero, 2019). Finally, the Danish open access indicator is discussed. This policy tool captures the publication output of Danish universities and thus monitors the national open access strategy. Publications are categorized according to whether they are open access, have untapped open access potential (publications that are not open access but have been published in journals that allow green open access with an embargo period of up to 1 year), or have unclear open access potential (Elbæk, 2016).
Some discussion papers have concrete policy dimensions and issue recommendations. In each one, the introduction of open scholarship indicators is accompanied by a call for more development of new indicators as well as a caution against the misapplication of these indicators.
Three policy papers of the mapping review data set operate at the European level. The European Commission published a framework dedicated to the development and use of metrics for open science (Next-Generation Metrics: European Commission et al., 2017a). Listing 12 recommendations, the framework aims to achieve four overarching goals:
Fostering open science;
Removing barriers to open science;
Developing research infrastructures for open science; and
Embedding open science in society.
The aim is to shift the paradigm from “publish as early as possible” to “share your knowledge as early as possible.” For example, the recommendations include a call for recognizing and rewarding open science principles and practices (recommendation 4) and also encourage the development of new openness indicators and the further development of existing indicators (recommendation 2). (European Commission et al., 2017a).
Another strategy paper of the European Commission, also from 2017, is dedicated to concrete implementation strategies. The Open Science Career Assessment Matrix (OS-CAM) represents the range of assessment criteria for the evaluation of open science activities. The framework can be applied to both early career scientists and experienced senior researchers and is intended to drive a culture change that promotes open practices. OS-CAM presents possible evaluation criteria regarding openness. Depending on the application, criteria can be taken from the framework and be applied. The entire framework does not always have to be taken into account, but it must be applied to the use cases. The indicators can be used on an individual level as well as on a group level: for example, for the “purpose of recruitment and promotion” or for monitoring purposes. Classically, research output can be mapped with the indicators but also with the aspect “service and leadership” (e.g., peer review or networking) or the aspect “teaching and supervision” (e.g., mentoring or supervision) (European Commission et al., 2017b).
The European Commission's third policy paper (OpenEdu) is about openness in teaching. “Open Education” in this case goes beyond OER and research output to include policy decisions, teaching methods, collaborations, open learning, and making all content public. The OpenEdu framework is aimed at higher education institutions and presents 10 dimensions for opening education. It mentions six core dimensions (access, content, pedagogy, recognition, collaboration, and research) and four transversal dimensions (strategy, technology, quality, and leadership). The core dimensions focus on what should be implemented and the transversal dimensions deal with the question of how it can be implemented. All dimensions are interconnected and overlap (Inamorato dos Santos, Punie, & Castaño-Muñoz, 2016).
Knowledge Exchange (KE) is another science policy organization that serves to support and promote higher education and research, including, in particular, open scholarship. It is a consortium of six national research funding organizations (from Germany, the United Kingdom, Denmark, the Netherlands, Finland, and France). Part of the open scholarship promotion also lies in changes of the recognition and reward system and discusses the implementation of an “Openness Profile”. This could be included, for example, in the ORCID website, increasing the visibility of open scholarship practices. Part of the position paper is also a synthesis of different open scholarship taxonomies. In this context, it is interesting to note that teaching and OER are mentioned more frequently. For example, collaborative authoring of OER is mentioned as an activity, as is sharing OER in open development environments and platforms (Jones & Murphy, 2021).
Finally, the Stifterverband in Germany should be mentioned, which is a nonprofit association that advises science. The Stifterverband calls for the development of new indicators that reflect open research and innovation processes, supports the development of new data sources and calls for reflexive indicator impact assessments to cushion unintended consequences (Blümel, 2019).
4.3. Presentation of the Keywords, Disciplines, and Journals
Regarding the second research question, we took a closer look at the keywords, disciplines, and journals represented in the documents. For this purpose, the co-occurrence of the keywords of all coded literature data was evaluated and displayed by a topic modeling procedure of VOSviewer. VOSviewer’s topic modeling extracted 154 noun phrases from the 248 titles and abstracts and identified four clusters (Figure 2). Each keyword corresponds to a node in the network, where the size of the node represents the frequency of the keyword. The links between nodes (n = 4,068) indicate that associated keywords were shared at least once for a publication. The total link strength (n = 5,910) indicates that some links have multiple assignments and are weighted accordingly. All nodes are very close to each other; only the green cluster has a slight tendency to separate. The many connections between the clusters show that they are not to be considered excessively separate from one another, but rather merge into one another.
Visual representation of the clusters (also available here: https://app.vosviewer.com/?json=https://drive.google.com/uc?id=12SpWTi_nPlrh4jLkebLPqhTzblEmea54).
Visual representation of the clusters (also available here: https://app.vosviewer.com/?json=https://drive.google.com/uc?id=12SpWTi_nPlrh4jLkebLPqhTzblEmea54).
A closer look at the nodes and connections supports the impression that the network is very closely interwoven thematically. Figure 3 shows which items were assigned to which cluster, how often they were assigned in the data set, and how many links they have to other items (both the weighted (= total link strength) and the unweighted number (= links)). The presentation reveals that two umbrella categories stand out. These are bibliometrics-related terminology and openness-related terminology. Bibliometrics-related terms include, for example, “citation analysis” in cluster 1, “bibliometric indicator” in cluster 2, “h-index” in cluster 3, and “bibliometric study” in cluster 4. Thematically related to openness are, for example, “open data” in cluster 1, “oa status” in cluster 2, “oa journal” in cluster 3, and “oa article” in cluster 4. The predominance of these topics is not a coincidence but a logical consequence due to the search query of the mapping review (Section 3.2). The topics are distributed over all four clusters and no umbrella terms can be found for the individual clusters.
In the context of open scholarship, some research-relevant terms are apparent (e.g., “research community” (cluster 1) or “survey” (cluster 4) and many more), but only one term that is directly related to university teaching, which is “education” in cluster 1. In our view, this is a bias that cannot be explained by the search query, but rather indicates a skew in the appreciation of research and teaching achievements in the context of open scholarship. It seems that the quantification of open scholarship addresses research and neglects university teaching.
The keyword analysis was followed by an analysis of the disciplines. In total, the documents with available subject categories (n = 171) were classified into 36 different categories. However, the largest share came from the categories “Information and Library Science” and “Computer Science” (together 69%) (Table 6).
Subject categories
Subject category . | Frequency (total) . | Frequency (%) . |
---|---|---|
Information and Library Science | 108 | 41.38 |
Computer Science | 72 | 27.59 |
Science & Technology | 14 | 5.36 |
Communication | 6 | 2.3 |
Social Sciences | 6 | 2.3 |
Education & Educational Research | 6 | 2.3 |
Business & Economics | 5 | 1.92 |
Health Care Sciences & Services | 5 | 1.92 |
General & Internal Medicine | 4 | 1.53 |
Medical Informatics | 4 | 1.53 |
Environmental Sciences & Ecology | 2 | 0.77 |
Psychology | 2 | 0.77 |
Radiology, Nuclear Medicine & Medical Imaging | 2 | 0.77 |
Arts & Humanities | 2 | 0.77 |
Dentistry, Oral Surgery & Medicine | 2 | 0.77 |
Automation & Control Systems | 1 | 0.38 |
Biochemistry & Molecular Biology | 1 | 0.38 |
Rheumatology | 1 | 0.38 |
Engineering | 1 | 0.38 |
Pathology | 1 | 0.38 |
Linguistics | 1 | 0.38 |
Orthopedics | 1 | 0.38 |
Evolutionary Biology | 1 | 0.38 |
Geography | 1 | 0.38 |
Government & Law | 1 | 0.38 |
History & Philosophy of Science | 1 | 0.38 |
Life Sciences & Biomedicine | 1 | 0.38 |
Mathematical Methods in Social Sciences | 1 | 0.38 |
Mathematics | 1 | 0.38 |
Neurosciences & Neurology | 1 | 0.38 |
Sociology | 1 | 0.38 |
Surgery | 1 | 0.38 |
Physiology | 1 | 0.38 |
Cell Biology | 1 | 0.38 |
Public Administration | 1 | 0.38 |
Urban Studies | 1 | 0.38 |
Subject category . | Frequency (total) . | Frequency (%) . |
---|---|---|
Information and Library Science | 108 | 41.38 |
Computer Science | 72 | 27.59 |
Science & Technology | 14 | 5.36 |
Communication | 6 | 2.3 |
Social Sciences | 6 | 2.3 |
Education & Educational Research | 6 | 2.3 |
Business & Economics | 5 | 1.92 |
Health Care Sciences & Services | 5 | 1.92 |
General & Internal Medicine | 4 | 1.53 |
Medical Informatics | 4 | 1.53 |
Environmental Sciences & Ecology | 2 | 0.77 |
Psychology | 2 | 0.77 |
Radiology, Nuclear Medicine & Medical Imaging | 2 | 0.77 |
Arts & Humanities | 2 | 0.77 |
Dentistry, Oral Surgery & Medicine | 2 | 0.77 |
Automation & Control Systems | 1 | 0.38 |
Biochemistry & Molecular Biology | 1 | 0.38 |
Rheumatology | 1 | 0.38 |
Engineering | 1 | 0.38 |
Pathology | 1 | 0.38 |
Linguistics | 1 | 0.38 |
Orthopedics | 1 | 0.38 |
Evolutionary Biology | 1 | 0.38 |
Geography | 1 | 0.38 |
Government & Law | 1 | 0.38 |
History & Philosophy of Science | 1 | 0.38 |
Life Sciences & Biomedicine | 1 | 0.38 |
Mathematical Methods in Social Sciences | 1 | 0.38 |
Mathematics | 1 | 0.38 |
Neurosciences & Neurology | 1 | 0.38 |
Sociology | 1 | 0.38 |
Surgery | 1 | 0.38 |
Physiology | 1 | 0.38 |
Cell Biology | 1 | 0.38 |
Public Administration | 1 | 0.38 |
Urban Studies | 1 | 0.38 |
Figure 4 visualizes the most represented discipline, “information and library science.” The yellow nodes are strongly connected to the discipline, and the blue ones have no relation to it. It is interesting to note that the second cluster (the green cluster in Figure 2) is most strongly connected to the discipline of Information and Library Science (Figure 4).
Visual representation of the subject category “Information and Library Science.”
Visual representation of the subject category “Information and Library Science.”
As a final aspect, we consider the journals in which the articles of the mapping review were published (n = 170). A total of 101 journals are listed as sources in the data set, of which 79 occur only once. Table 7 shows which journals occur more than twice.
Representation of the journals in the clusters (in %)
Journal . | Frequency . | Red . | Green . | Blue . | Yellow . |
---|---|---|---|---|---|
Scientometrics | 47 | 29.21 | 38.33 | 24.82 | 7.54 |
Journal of the Association for Information Science and Technology (JASIST) | 7 | 12.5 | 62.5 | 0.00 | 25 |
PLOS ONE | 5 | 27.75 | 36.25 | 30.75 | 5.00 |
Publications | 5 | 50.00 | 12.50 | 25.00 | 12.50 |
Journal of Informetrics | 4 | 33.33 | 33.33 | 33.33 | 0.00 |
PeerJ | 4 | 8.25 | 27.00 | 52.00 | 12.50 |
Professional de la Informacion | 3 | 16.67 | 16.67 | 66.67 | 0.00 |
College & Research Libraries | 3 | 45.00 | 22.50 | 0.00 | 32.50 |
Journal . | Frequency . | Red . | Green . | Blue . | Yellow . |
---|---|---|---|---|---|
Scientometrics | 47 | 29.21 | 38.33 | 24.82 | 7.54 |
Journal of the Association for Information Science and Technology (JASIST) | 7 | 12.5 | 62.5 | 0.00 | 25 |
PLOS ONE | 5 | 27.75 | 36.25 | 30.75 | 5.00 |
Publications | 5 | 50.00 | 12.50 | 25.00 | 12.50 |
Journal of Informetrics | 4 | 33.33 | 33.33 | 33.33 | 0.00 |
PeerJ | 4 | 8.25 | 27.00 | 52.00 | 12.50 |
Professional de la Informacion | 3 | 16.67 | 16.67 | 66.67 | 0.00 |
College & Research Libraries | 3 | 45.00 | 22.50 | 0.00 | 32.50 |
The journal Scientometrics is the most common, which was expected due to its focus on quantitative research assessment. Besides Scientometrics, several other journals are present that also cover scientometric or informetric areas (e.g., the Journal of Informetrics and JASIST). Those top journals indicate that educational science perspectives on open scholarship metrics are very weakly represented. Only four journal titles indicate such a perspective:
International Review of Research in Open and Distributed Learning (n = 2)
Ried-revista Iberoamericana de Educacion a Distancia (n = 1)
Journal of Distance Education (n = 1)
Educational Technology & Society (n = 1)
In a further step, we examined whether the journals can be assigned to certain clusters of the network analysis. For this purpose, four of the most frequent words were checked for their presence in titles and abstracts of the documents. As a result, it was possible to determine which of the nine most frequently mentioned journals were represented in the different clusters and to what extent. The analysis showed that the journal Scientometrics belongs to the red cluster with 29.21%, to the green cluster with 38.33%, to the blue cluster with 24.82%, and to the yellow cluster with 7.54% (Table 7).
The analysis shows that the distribution across the clusters is very diverse and does not follow a pattern. For some journals, the distribution is very even (for example, the Journal of Informetrics), but for others it is very heavily weighted (for example, JASIST). The combination of journal and cluster analysis does not provide any revealing insights regarding the content orientation of the clusters or journals. What we have seen very clearly in the journal analysis, however, is that educational science journals play only a very minor role, and other social science or humanities fields are not among the most frequent sources.
5. DISCUSSION
Regarding the first research question, the mapping review revealed the following. First, we only found eight indicators explicitly designed for open scholarship assessment. All other indicators are bibliometric or altmetric indicators that are also applicable to nonopen research objects. Second, the open scholarship objects most investigated are those that correspond to research practices traditionally examined in bibliometrics and altmetrics (i.e., OA articles or OA journals). Indicators related to open education are rarely applied in the data set conducted. As a subject, OER/MOOCS were only addressed four times by the open scholarship indicators. Even though OER were simply developed later than other OA documents and therefore had less time to establish themselves as scientometric subjects, the empirical evidence shows that open teaching plays almost no role in scientometric measurements. In our view, this is problematic insofar as we understand open scholarship as a unit of open research and open teaching. Due to the weak presence of open teaching materials, we see their visibility in scientometric measurements as too low. Our assessment is supported by our empirical evidence in that the review shows that there is a political demand for broader coverage of academic performance (European Commission et al., 2017a) and for open teaching (Inamorato dos Santos et al., 2016). The scientometric recording of OER is thus likely to be politically desired.
Furthermore, we noticed that the trivial indicators are most frequently used (i.e., indicators without time and field normalization or weighting; see Table 4 or data set: https://doi.org/10.5281/zenodo.8128130). The bibliometric indicators include simple count indicators such as the “times cited score” (tcs) or the “number of papers” (p), as well as indicators combining these two, such as the “journal impact factor” (JIF) or the “Hirsch index” (h-index). In comparison, more complex indicators with normalization, such as the “source normalized impact per paper” (SNIP), appear rather rarely. This is even more true in the case of altmetric indicators, which consist exclusively of simple count indicators without time and field normalization and/or weighting. All altmetric indicators mentioned are based on the context-free counting of mentions or appearances on different websites. Among the openness indicators, only one normalized indicator can be found. The “normalized open access indicator” (NOAI) takes into account field-specific differences in open access publishing, but also appears only once in the data set. All other openness indicators are also limited to the mere collection of contextless information. Leydesdorff, Wouters, and Bornmann (2016) speak of citizen bibliometrics, meaning “simple but invalid indicators that are widely used (e.g., the h-index)” on the one hand and professional bibliometrics, meaning, for example, “more sophisticated indicators that are not used or cannot be used in evaluation practices because they are not transparent for users, cannot be calculated, or are difficult to interpret” on the other. We also see that professional bibliometrics is little used in the literature on open scholarship indicators. This is unfortunate in that much of the criticism of the use of scientometric indicators relates to their limited explanatory power. This point of view could be countered by the use of complex indicators, which is only done to a limited extent.
Regarding the second research question, the review discloses that the indexing of the documents shows a strong bias towards research-related topics. Keywords related to university teaching hardly ever occur (“education” only once). Documents that address open scholarship indicators are predominantly (74.33%) located in the disciplines “Information and Library Science,” “Computer Science,” and “Science and Technology.” Other disciplines do not play a major role and are only sparsely represented (≤ 2.3%). The latter is also confirmed by the journal analysis.
We note that open research and open teaching are thought of together in discussions and debates on open scholarship. At the same time, the application of quantifying open scholarship indicators concentrates on research-related outputs, such as publications, journals, and data. Education-related outputs are rarely represented in our data set. This might be, as mentioned above, due to the focused search based on scientometric terms. However, we see an imbalance here. If open scholarship refers to scholarly research and education in equal measure, open teaching and learning would need to become visible through quantifying indicators to incentivize open practices and balance the scholarly reward and recognition system. In summary, our review showed that teaching evaluation is not considered in scientometrics. Research on teaching evaluation exists, but concentrates on aspects such as student event critique, university rankings, teaching reports, and peer reviews (Bargel & El Hage, 2000). However, this is different from quantitatively measuring open practices in education, as research and discussion on open scholarship aim at. We see a research gap at this point and opportunities for future research.
6. CONCLUSION
In summary, the 248 coded documents of the mapping review contain 203 empirical studies that apply indicators and 45 discussion papers that provide recommendations or discussions about open scholarship indicators. Furthermore, it can be noted that the empirical papers use both classic bibliometric and altmetric indicators, as well as indicators that are exclusively related to openness. The keywords of the publications cover a wide range of topics, but there is a tendency towards technical and scientometric terminology. Publications on the topic of open scholarship indicators are viewed particularly highly from a library and information science perspective. Other disciplinary approaches did not find their way into our mapping review in a significant way.
Likewise, it has been found that although open scholarship indicators take into account both research and academic teaching, they have a very strong bias towards research-related topics when it comes to the application of these indicators. Our further work is dedicated to the analysis of potential open scholarship indicators, which serve to quantify open teaching/learning material. We also aim to develop an educational perspective on indicators for open scholarship in order to make the debate on quantifying openness more holistic. We see this as an added value, as open scholarship is strongly related to higher education teaching.
AUTHOR CONTRIBUTIONS
Verena Weimer: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Validation, Visualization, Writing—original draft, Writing—review & editing. Tamara Heck: Data curation, Methodology, Validation, Writing—review & editing. Thed van Leuuwen: Formal analysis, Methodology, Validation, Visualization, Writing—review & editing. Marc Rittberger: Methodology, Validation, Writing—review & editing.
COMPETING INTERESTS
The authors have no competing interests.
FUNDING INFORMATION
No funding was received for this research.
DATA AVAILABILITY
The research data are available at https://doi.org/10.5281/zenodo.8128130.
Note
The Helmholtz Association, together with the Max Planck Society, the Fraunhofer Society, and the Leibniz Association, is one of the four major nonuniversity research organizations in Germany.
REFERENCES
Author notes
Handling Editor: Vincent Larivière